16:00:00 <nirik> #startmeeting Infrastructure (2021-04-08)
16:00:00 <zodbot> Meeting started Thu Apr  8 16:00:00 2021 UTC.
16:00:00 <zodbot> This meeting is logged and archived in a public location.
16:00:00 <zodbot> The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:00 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:00 <zodbot> The meeting name has been set to 'infrastructure_(2021-04-08)'
16:00:01 <nirik> #meetingname infrastructure
16:00:01 <zodbot> The meeting name has been set to 'infrastructure'
16:00:01 <nirik> #chair nirik smooge siddharthvipul mobrien zlopez pingou bodanel dtometzki
16:00:01 <nirik> #info Agenda is at: https://board.net/p/fedora-infra
16:00:01 <zodbot> Current chairs: bodanel dtometzki mobrien nirik pingou siddharthvipul smooge zlopez
16:00:01 <nirik> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:00:02 <nirik> #topic aloha
16:00:14 <nirik> morning everyone!
16:00:15 <mobrien> .hi
16:00:16 <zodbot> mobrien: mobrien 'Mark O'Brien' <markobri@redhat.com>
16:00:22 <Zlopez[m]> .hello zlopez
16:00:22 <dtometzki> .hi
16:00:22 <zodbot> Zlopez[m]: zlopez 'Michal Konečný' <michal.konecny@psmail.xyz>
16:00:26 <zodbot> dtometzki: dtometzki 'Damian Tometzki' <linux@tometzki.de>
16:00:59 <nirik> #topic New folks introductions
16:00:59 <nirik> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:00:59 <nirik> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:01:04 <nirik> Any new folks today?
16:01:12 * nirik will wait a few for more folks to arrive.
16:01:24 <darknao> .hi
16:01:25 <zodbot> darknao: darknao 'Francois Andrieu' <naolwen@gmail.com>
16:04:00 <nirik> small group today. :) ok
16:04:16 <nirik> #topic Next chair
16:04:16 <nirik> #info magic eight ball says:
16:04:16 <nirik> #info chair 2021-04-15- dtometzki
16:04:16 <nirik> #info chair 2021-04-22- mobrien
16:04:17 <nirik> #info chair 2021-04-29 - ?
16:04:23 <nirik> anyone want to take the 29th?
16:04:33 <smooge> hello
16:04:37 <nirik> hey smooge
16:04:44 <Zlopez[m]> I can take it
16:04:50 <smooge> thanks Zlopez[m]
16:04:53 <nirik> Zlopez[m]: thanks!
16:04:58 <bodanel> .hi
16:04:59 <zodbot> bodanel: bodanel 'Bogdan Benea' <benea_bogdan@yahoo.com>
16:05:08 <nirik> #topic announcements and information
16:05:09 <nirik> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:05:09 <nirik> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:05:09 <nirik> #info work is being done on getting COPR proper powerPC systems
16:05:09 <nirik> #info final freeze started on April 06th at 1400 UTC
16:05:17 <computerkid> .hello computerkid
16:05:18 <nirik> any other announcements or info?
16:05:18 <zodbot> computerkid: computerkid 'Grayson Penland' <gpenland06@gmail.com>
16:05:37 <computerkid> Sorry, I'm late
16:05:45 <nirik> welcome computerkid. No worries. :)
16:05:52 <dtometzki> Fedora freeze
16:06:11 <nirik> Yep, we are in freeze now...
16:06:27 <mobrien> maybe put in an info to remind all sysadmin-* users to add an otp token in noggin
16:06:54 <computerkid> 1+ mobrien
16:07:39 <nirik> mobrien: good idea
16:07:52 <dtometzki> noggin is a other auth system ?
16:08:02 <nirik> #info all members of sysadmin* should make sure and add otp token(s). One primary and one spare
16:08:19 <nirik> dtometzki: noggin is the web app/frontend of the new account system.
16:08:27 <nirik> https://accounts.fedoraproject.org/
16:08:45 <dtometzki> ahh ok is the same
16:08:47 <nirik> it talks to IPA on the backend... which stores all the info and manages it.
16:09:08 <nirik> that reminds me... how long are we planning to keep the old account system up read-only?
16:09:16 <nirik> we should set a sunset date
16:10:14 <dtometzki> perhaps after freeze
16:10:17 <dtometzki> ?
16:10:41 <smooge> I would say May 1st 2021
16:10:53 <mobrien> nirik: We originally suggested 6 months as a grace period but I think it should be much shorter
16:11:04 <Zlopez[m]> nirik: What happens if you don't have any spare OTP token and lost the primary one
16:11:22 <mobrien> Zlopez: raise a ticket to get it removed
16:11:26 <nirik> yeah, we still need to port a few things tho, like zodbot...
16:11:39 <Zlopez[m]> mobrien:  Sounds easy to fix
16:11:49 <nirik> I think may might be ok, but later in may? I guess we should discuss on list to get a wider audience?
16:12:25 <nirik> Zlopez[m]: yeah, a ticket, but... we also need to verify you are you, which is a lot harder. Also, it's anoying processing those from lots of people manually.
16:12:31 <mobrien> Zlopez: the biggest issue at the moment is verifying the identity of the person requesting the otp token to be removed
16:13:09 <Zlopez[m]> The last time I think we used mail encrypted by my PGP key
16:13:23 <nirik> and actually, ticket won't work anymore... has to be email... since you need otp to login to pagure now...
16:13:39 <mobrien> oh yes, true
16:14:20 <nirik> I'll start a thread on the list... unless someone else would like to? ;)
16:14:48 <Zlopez[m]> Oh, I didn't knew that when you add OTP now, you need to use it for every login
16:15:35 <nirik> I think so... ipsilon will only ask for password, but you need to enter password + token there.
16:15:46 <mobrien> Zlopez: yep and as of yet no yubikey support
16:15:54 <copperi_> 2FA works that way
16:16:05 <Zlopez[m]> I have my OTP synced between multiple devices, so I should be without issue there
16:16:24 <dtometzki> yubikey works n my side
16:16:37 <Zlopez[m]> Started after my last phone died without warning
16:17:01 <nirik> yeah, thats fine too... just have a way to access it if you loose your device, etc.
16:17:24 <nirik> #topic Monitoring discussion [nirik]
16:17:24 <nirik> #info https://nagios.fedoraproject.org/nagios
16:17:24 <nirik> #info Go over existing out items and fix
16:17:39 <nirik> so, I managed to find and fix 3 machines that were down...
16:18:34 <nirik> otherwise we have a mustang down, a emag with a bad drive still, vmhost-x86-12 (not sure what it's state is)
16:18:53 <mobrien> nicely done
16:19:00 <nirik> and a mgmt interface on a dell chassis showing down. Not sure what to do about that one... power cycle the entire thing?
16:19:45 <nirik> all our registries are running low on disk. We should increase disk size after the freeze.
16:20:17 <nirik> there's been no bugzilla messages on our bus for a while. Its likely broken at the bugzilla side again. ;(
16:21:14 <nirik> the others are all stg ones or things we should fix someday
16:21:55 <nirik> also, 3 machines: pdc-web01, pdc-web02 and resultsdb01... all frequently alert. We should set nagios to restart httpd on those before alerting.
16:22:05 <nirik> thats it. Any questions or comments on nagios?
16:22:36 <smooge> not from me'
16:23:10 <nirik> #topic Learning topic discussion
16:23:30 <nirik> I'm signed up today to talk about our IRC bot overlord: zodbot
16:23:46 <nirik> #info  "Zodbot/bots"
16:24:20 <nirik> zodbot is a Limnoria bot (which is a fork of supybot). It's python based and has a plugin setup.
16:24:36 <nirik> it's managed from irc itself. ie, you send it commands and it writes them to it's config.
16:24:40 <smooge> benign overlord
16:25:19 <nirik> history of the name: in super man (2?) the villans were a trio of super people from supermans home planet. Their leader was 'Zod'
16:25:37 <nirik> zodbot runs on our value01 server (value added services)
16:26:04 <nirik> The most important thing zodbot does is meetings (like this one).
16:26:18 <nirik> It's using a plugin called "supybot-Meetbot" for this.
16:26:56 <nirik> It was orig developed by a debian developer, but then they disappeared. I have a fork of it on pagure.io that we use...(well, we use the packaged version rpm, but upstream is the fork)
16:27:24 <nirik> zodbot keeps track of meetings and then writes the logs/summary to value01. It's then served/searched by the mote application.
16:27:45 <nirik> zodbot also has a supybot-Fedora plugin to interact with Fedora services.
16:28:07 <nirik> Currently it can look up people in fas (old account system). I think someone is working on updating it to talk to the new one
16:28:34 <nirik> many of it's functions are simply using its 'alias' ability. You can alias something to a command/string of commands.
16:28:39 <nirik> so for example:
16:28:42 <nirik> .ticket 1
16:28:43 <zodbot> nirik: Issue #1: This is a bug - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/1
16:28:50 <nirik> .alias list ticket
16:28:50 <zodbot> nirik: (alias list [--locked|--unlocked]) -- Lists alias names of a particular type, defaults to all aliases if no --locked or --unlocked option is given.
16:29:04 <nirik> .misc help ticket
16:29:04 <zodbot> nirik: (ticket <an alias, 1 argument>) -- Alias for "showticket https://pagure.io/fedora-infrastructure/issue/%s $1".
16:29:14 <nirik> so thats just an alias
16:29:19 <nirik> .misc help showticket
16:29:19 <zodbot> nirik: (showticket <baseurl> <number>) -- Return the name and URL of a trac ticket or bugzilla bug.
16:29:41 <nirik> There is also a koji plugin, which doesn't get too much use.
16:30:30 <smooge> i didn't know that one worked anymore
16:30:40 <nirik> It's in something like 150 channels... which is pretty crazy
16:30:58 <dtometzki> can we anything do when zdbot on a fedora-meeting room isnt available ?
16:31:02 <nirik> .buildload
16:31:03 <zodbot> nirik: Load: 843.0 Total: 1553.0 Use: 54.3% (Medium Load)
16:31:07 <nirik> .builders
16:31:08 <zodbot> nirik: Enabled: 187 Ready: 183 Disabled: 325
16:32:01 <nirik> dtometzki: so, there seems to be some bug or issue with it rejoining all it's channels after a network issue... so sometimes it doesn't end up in all of them. Not sure why... fixing that would fix that problem
16:32:25 <nirik> so in the future, we probibly will be looking at making some new bot(s) on matrix...
16:32:47 <dtometzki> is it possible that i can fix such issue ?
16:32:48 <nirik> for right now zodbot works ok for matrix people due to the bridge, but there's some issues.
16:33:13 <Zlopez[m]> Like multiline paste
16:33:52 <nirik> dtometzki: I'm not sure if we have been able to duplicate it. You could try with a Limnoria instance of your own?
16:34:03 <nirik> Zlopez[m]: yeah, the <you have sent a long message> thing
16:34:18 <nirik> Also, someday it might be nice to add features for meetings in particular. Like voting.
16:35:22 <dtometzki> no i mean there was some questions on monday morning (CET) zobot isnt available can anyone help
16:35:35 <nirik> As a slight aside... there's another set of bots we run you may have seen:
16:36:13 <nirik> fm-admin and fm-stg-admin in #fedora-admin, etc... those are fedmsg bots, also running on value01. They are 'dumb' in that all they do is print messages that match to the channel they are in.
16:36:38 <nirik> dtometzki: it requires an 'owner' of the bot. Basically anyone who has been around a long while.
16:37:17 <mobrien> Another bot is used for oncall right :)
16:37:46 <nirik> we could look and see if there is a way to make that perm wider... but we don't want anyone to do it as it could then join a bunch of channels where it's not wanted.
16:37:56 <nirik> mobrien: no, zodbot has that too.. it's just another alias.
16:37:59 <nirik> .misc help oncall
16:37:59 <zodbot> nirik: (oncall <an alias, 0 arguments>) -- Alias for "echo dtometzki is oncall. My normal hours are 13:00 UTC to 21:00 UTC Monday through Friday. If I do not answer or it is outside those hours, please file a ticket (https://pagure.io/fedora-infrastructure/issues)".
16:38:14 <nirik> oh, I missed the oncall section didn't I? oops
16:38:49 <nirik> any bot questions? I can go back to oncall section if not, or after...
16:38:52 <dtometzki> i take over
16:39:35 <nirik> thanks dtometzki
16:39:51 <Zlopez[m]> nirik: I only wanted to add that I don't see the long message warning on matrix, it's just not processed by zodbot
16:41:26 <nirik> Zlopez[m]: right, it's the bridge. You send some lines the bridge sees them all as a bundle and says on the irc side "a long line was sent..." and a link to your lines...
16:41:42 <nirik> where zodbot operates on lines. It can't decode the url and see commands in it
16:42:56 <nirik> ok, on oncall, dtometzki took over. I only saw 2 oncall pings... one filed a ticket and not sure what the other one wanted. ;)
16:43:05 <nirik> #topic Open Floor
16:43:13 <nirik> any other business?
16:43:17 <smooge> that's an awfully clean floor
16:43:22 <smooge> what do you use?
16:43:46 <mobrien> nirik: you can stick me in for the next available oncall
16:43:56 <nirik> it's a floor cleaning and a dessert toping!
16:44:14 <pingou> one question, you've mentioned having a spare otp for AAA
16:44:15 <nirik> mobrien: sure. 23rd to 30th
16:44:21 <pingou> do we have any suggestions/recommendations?
16:44:22 <mobrien> +1
16:44:46 * pingou has freeotp on the phone he has
16:45:02 <mobrien> pingou, the simplest solution is another device. a tablet or spare phone
16:45:04 <nirik> pingou: not sure. I use andotp and it lets you also do backups
16:45:20 <dtometzki> is it possible to todo an online meeting to get to know each other better ?
16:45:33 <Zlopez[m]> pingou: I'm using FreeOTP and doing backups
16:45:41 <darknao> i personnaly use authy, that let you recover your backup if you loose your phone
16:45:42 <dtometzki> for example teams or jitsi
16:45:56 <pingou> Zlopez[m]: how do you do backups?
16:46:12 <Zlopez[m]> Export to my own nextcloud
16:46:17 <computerkid> I'm doing learning next week right?
16:46:36 <pingou> darknao: syncs to a cloud provider or?
16:46:47 <Zlopez[m]> pingou: The FreeOTP+ has Export/Import button
16:46:52 <nirik> dtometzki: we could yeah... I think video is bad for actual business meetings for lots of reasons, but I agree it's nice to meet people... a more informal meeting might be nice
16:47:00 <nirik> computerkid: yep :)
16:47:04 <pingou> Zlopez[m]: iOS or Android?
16:47:14 <Zlopez[m]> pingou: But be careful, the Import just replace everything
16:47:19 <computerkid> I need to start making some notes nirik.....
16:47:20 * pingou like the idea of an open floor infra meetup
16:47:32 <nirik> dtometzki: might open a thread on the list and we can find a time?
16:47:41 <Zlopez[m]> pingou: Android like /e/ OS
16:47:46 <dtometzki> yes
16:47:50 <dtometzki> great
16:47:53 <pingou> Zlopez[m]: I don't see an option to do export here :(
16:48:04 <pingou> (android)
16:48:15 <pingou> and iirc the iOS version was upgraded recently
16:48:15 <nirik> pingou: freeotp cant. freeotp+ can
16:48:25 <pingou> ah! that may be it
16:48:26 <nirik> freeotp+ is a fork I think
16:48:50 <nirik> andotp is nice because it's also 100% open source. I think freeotp+ is also tho
16:48:55 <Zlopez[m]> I have FreeOTP+ 2.3 (14)
16:49:01 <darknao> pingou: Authy servers are used for backup, which are encrypted with your password
16:49:02 <pingou> somehow I think that this may be a nice fedoramagazine article :]
16:49:11 <nirik> Also bitwarden might be an option...
16:49:14 <Zlopez[m]> nirik: Yeah FreeOTP+ is fully open
16:49:27 <pingou> oh bitwarden does otp?
16:49:30 <Zlopez[m]> pingou: +1 for the article
16:49:44 <computerkid> I use 1password and FreeOTP as a backup
16:50:04 <nirik> I think so, but not 100% sure. I don't use it.
16:50:19 <darknao> bitwarden require a subscription for otp i think
16:50:33 <nirik> https://bitwarden.com/help/article/authenticator-keys/
16:50:44 <Zlopez[m]> I used LastPass in past, but I didn't like that my passwords are actually in some third party database
16:50:52 <Zlopez[m]> Now I'm using KeepassXC + nextcloud
16:51:05 <nirik> looks like adding a key is free, generating requires paid
16:51:39 <nirik> (and for Red Had employees I think there's a site agreement or something, but don't know the details, consult your intranet)
16:52:06 <nirik> ok, any other items before we close out?
16:52:57 <nirik> thanks everyone so much for coming!
16:53:00 <nirik> #endmeeting