16:00:02 <nirik> #startmeeting Infrastructure (2021-08-12)
16:00:02 <zodbot> Meeting started Thu Aug 12 16:00:02 2021 UTC.
16:00:02 <zodbot> This meeting is logged and archived in a public location.
16:00:02 <zodbot> The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:02 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:02 <zodbot> The meeting name has been set to 'infrastructure_(2021-08-12)'
16:00:02 <nirik> #meetingname infrastructure
16:00:02 <zodbot> The meeting name has been set to 'infrastructure'
16:00:02 <nirik> #chair nirik siddharthvipul mobrien zlopez pingou bodanel dtometzki jnsamyak computerkid
16:00:02 <nirik> #info Agenda is at: https://board.net/p/fedora-infra
16:00:02 <nirik> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:00:02 <zodbot> Current chairs: bodanel computerkid dtometzki jnsamyak mobrien nirik pingou siddharthvipul zlopez
16:00:02 <nirik> #topic ahoy!
16:00:14 <eddiejennings> .hi
16:00:15 <zodbot> eddiejennings: eddiejennings 'Eddie Jennings, Jr.' <eddie@eddiejennings.net>
16:00:28 <mkonecny> .hello zlopez
16:00:29 <zodbot> mkonecny: zlopez 'Michal Konečný' <michal.konecny@psmail.xyz>
16:00:34 <lenkaseg> .hello lenkaseg
16:00:35 <zodbot> lenkaseg: lenkaseg 'Lenka Segura' <lenka@sepu.cz>
16:00:36 <petebuffon[m]> .hello petebuffon
16:00:38 <zodbot> petebuffon[m]: petebuffon 'Peter Buffon' <pabuffon@gmail.com>
16:00:45 <eddiejennings> This time tomorrow I'll be in hour three of my RHCE exam :D
16:00:55 <nirik> #topic New folks introductions
16:00:55 <nirik> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:00:55 <nirik> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:00:58 <mobrien> .hi
16:00:59 <zodbot> mobrien: mobrien 'Mark O'Brien' <markobri@redhat.com>
16:01:03 <nirik> eddiejennings: good luck on it!
16:01:06 <nirik> any new folks today?
16:01:08 <eddiejennings> Thank you :D
16:01:56 <darknao> .hi
16:01:57 <zodbot> darknao: darknao 'Francois Andrieu' <naolwen@gmail.com>
16:02:02 <bcapper> .hi
16:02:02 <zodbot> bcapper: Sorry, but you don't exist
16:02:17 <bcapper> Client: HexChat 2.14.3 • OS: Fedora release 34 (Thirty Four) • CPU: Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz (2.10GHz) • Memory: Physical: 30.4 GiB Total (27.2 GiB Free) Swap: 7.8 GiB Total (7.8 GiB Free) • Storage: 39.3 GB / 498.4 GB (459.1 GB Free) • VGA: Intel Corporation WhiskeyLake-U GT2 [UHD Graphics 620] @ Intel Corporation Coffee Lake HOST and DRAM Controller • Uptime:
16:02:17 <bcapper> 8h 12m 19s
16:02:23 <bcapper> sry
16:03:14 <nirik> no problems. ;) nice OS there.
16:03:49 <bcapper> Only the best will do :)
16:04:38 <eddiejennings> The RHCE remote exam ISO is Fedora :D
16:04:40 <nirik> ok, if no new folks will sail this ship onward!
16:04:55 <nirik> #topic Next chair
16:04:55 <nirik> #info magic eight ball says:
16:04:55 <nirik> #info chair 2021-08-19 - eddiejennings
16:04:55 <nirik> #info chair 2021-08-26 - ???
16:05:05 <nirik> anyone want to take the 08-26 meeting?
16:05:11 <eddiejennings> You're all doomed! XD
16:06:02 <mkonecny> I can take it
16:06:19 <nirik> mkonecny: great
16:07:00 <nirik> #topic announcements and information
16:07:00 <nirik> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:07:00 <nirik> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:07:00 <nirik> #info If your team wants support from the Fedora Program Management Team, file an isssue: https://pagure.io/fedora-pgm/pgm_team/issues?template=support_request
16:07:02 <nirik> #info Nest with Fedora is over, videos to be posted in sept
16:07:04 <nirik> #info tomorrow (2021-08-13) is a Red Hat "recharge day" so many Red Hat employees will be away
16:07:11 <nirik> any other announcements from anyone?
16:08:09 <nirik> #topic Oncall
16:08:09 <nirik> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:08:09 <nirik> #info darknao is on call for 2021-08-05 to 2021-08-12
16:08:09 <nirik> #info ?? is on call for 2021-08-12 to 2021-08-19
16:08:09 <nirik> ## .oncalltakeeu .oncalltakeus
16:08:29 <nirik> anyone want the job of hanging out in the crows nest and telling us about incoming ships? :)
16:08:51 <petebuffon[m]> I'll take it
16:09:06 <petebuffon[m]> lots of nautical analogies today
16:09:45 <nirik> excellent. yeah.
16:10:00 <nirik> #info Summary of last week: (from current oncall )
16:10:10 <nirik> darknao: much in the way of oncall notices?
16:10:23 <darknao> couple of pings
16:10:26 <mkonecny> petebuffon:
16:10:34 <mkonecny> Everyone aboard!
16:10:36 <darknao> # 08.07: login issue on pagure (OpenID transaction canceled): solved by itself moments later
16:10:38 <darknao> # 08.07: someone was unable to upload ssh key to its account: was using the old fas page
16:11:46 <nirik> cool. Thanks for handling those...
16:12:26 <nirik> #topic Monitoring discussion [nirik]
16:12:26 <nirik> #info https://nagios.fedoraproject.org/nagios
16:12:26 <nirik> #info Go over existing out items and fix
16:12:39 <nirik> lets away to nagios island and look for burried treasure
16:12:52 <nirik> oh no, it's just a swamp. :)
16:13:04 <mkonecny> Aye, Aye Capt'n!
16:13:51 <nirik> so, there's a bunch of 'down' nodes that are the new ocp4 cluster instances... I'm sure that will get sorted out once everything is fully up and running.
16:14:50 <nirik> otherwise we have the other known down systems and just 7 other acked alerts.
16:15:22 <nirik> Does anyone know what the "queue gitlab-centos" in rabbitmq is consumed by?
16:15:29 <nirik> RABBITMQ_QUEUE WARNING - messages WARNING (12008), messages_ready OK (12008) messages_unacknowledged OK (0) consumers OK (0)
16:16:26 <mkonecny> By the name it looks like CentOS Stream related
16:16:26 <nirik> otherwise nothing new and exciting
16:16:52 <mkonecny> I will let them know
16:16:57 <nirik> mkonecny: thanks!
16:17:41 <nirik> So, we don't have a learning topic lined up today I don't think? so we want to do some backlog discussion? or call it a quick meeting?
16:18:27 <mkonecny> I would go for the backlog discussion
16:18:42 <nirik> sure, we can...
16:18:50 <nirik> #topic Fedora Infra backlog refinement
16:18:50 <nirik> #info Refine oldest tickets on https://pagure.io/fedora-infrastructure/issues
16:19:09 <nirik> .ticket 5478
16:19:10 <zodbot> nirik: Issue #5478: Establishing a new FAS group for Python-SIG - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/5478
16:19:24 <nirik> this ticket is still wanted, it's blocked by upgrading mailman so we can rename a list
16:19:42 <nirik> .ticket 8455
16:19:44 <zodbot> nirik: Issue #8455: Move mailman to newer release of Fedora or CentOS - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8455
16:20:03 <mkonecny> Can we do it?
16:21:11 <nirik> if we get mailman upgraded sure.
16:22:09 <nirik> so it might be nice to get more folks working on that...
16:22:26 <mkonecny> Are we waiting for upstream on this one?
16:22:40 <eddiejennings> After tomorrow, I can start looking into things again
16:22:55 <nirik> no, I don't think so. I think newer versions (if we could ever deploy them) would let us do this (but I am not 100% sure of that)
16:24:11 <nirik> so, let me move to a more interesting one to discuss...
16:24:25 <nirik> .ticket 7377
16:24:26 <zodbot> nirik: Issue #7377: SSH keys length can prevent user from login in Fedora infrastructure - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/7377
16:25:13 <nirik> so this is because pagure is using openid instead of oidc... and we haven't moved pagure to pidc because ipsilon needs some changes.
16:25:43 <nirik> many of these old tickets have a tangled solution
16:25:45 <mkonecny> So this is a complex issue to solve :-/
16:26:37 <mkonecny> I looked at the pagure today and there are plenty of open PRs and some issues with test runs under F34
16:26:40 <nirik> but perhaps we should just close it and say 'don't use such large keys for now, we will fix it when we can'
16:27:04 <mkonecny> It should still be open, so we knew about the issue
16:27:06 <petebuffon[m]> Would it be possible to use client side form validation and limit the number of characters?
16:27:08 <Eighth_Doctor> issues with test runs on F34?
16:27:15 <Eighth_Doctor> the only issues I'm aware of are on C7
16:27:42 <nirik> petebuffon[m]: sure, but that doesn't really 'solve' the issue...
16:27:49 <Eighth_Doctor> and I have a whole bunch of cleanup planned (including the elimination of the C7 CI jobs) as part of dropping Python 2 support
16:27:52 <nirik> and larger keys/number of keys works fine in many other places.
16:27:58 <mkonecny> Conan Kudo: I tried to run it today and just got plenty of string encoding doesn't work
16:28:17 <Eighth_Doctor> is there a ticket for that, because I didn't know that was an issue
16:28:18 <petebuffon[m]> nirik: True, just thought it might be a quick fix.
16:28:32 <nirik> Eighth_Doctor: can you say off hand what the mailman3 status is? I know mm3 is in fedora now, but what are next steps? hyperkitty packaging? other?
16:28:48 <Eighth_Doctor> nirik: hyperkitty and postorius need to be packaged
16:28:53 <Eighth_Doctor> once that's done, we're ready to go
16:28:59 <mkonecny> I couldn't find a way to actually run the tests on local changes
16:29:12 <nirik> cool. I'll once again try and find time to look into them...
16:29:45 <Eighth_Doctor> Zlopez: yeah I need to adjust the apparatus to default to local working copy
16:30:14 <Eighth_Doctor> the more urgent issue for me is to rip out the Python 2 support, because a couple of things are only barely working with the polyglot code
16:30:27 <mkonecny> But the pagure definitely need some love
16:30:54 <Eighth_Doctor> the CI has been failing because of CentOS 7 for a while
16:30:58 <mkonecny> I would just drop it, if we don't need it
16:31:14 <Eighth_Doctor> and I'm just at the point of just ripping it all out despite CentOS being a laggard here
16:31:19 <nirik> pagure? we do need it. ;)
16:31:27 <Eighth_Doctor> Zlopez: you can also run the tests with tox
16:31:33 <mkonecny> I meant the Python 2 support
16:31:33 <Eighth_Doctor> and that does work last I checked
16:31:43 <nirik> ah yes, death to py2
16:31:46 <Eighth_Doctor> yeah, this weekend I'm yanking it all out, the py2 support
16:31:49 <mkonecny> Conan Kudo:  This was what I did and everything was failing for me
16:31:51 <Eighth_Doctor> I'm tired of it
16:32:11 <Eighth_Doctor> it's making it too hard for me to do anything
16:34:07 <mkonecny> Conan Kudo:  This is the error I'm getting when running tox `TypeError: not all arguments converted during string formatting`
16:34:14 <Eighth_Doctor> hmm
16:34:30 <Eighth_Doctor> I'll look into it this evening then, after my work hours are over
16:35:05 <mkonecny> Conan Kudo: Don't rush it
16:35:23 <mkonecny> Otherwise I think we can look at the next ticket
16:35:59 <mkonecny> I'm not sure how nirik is selecting them
16:36:00 <nirik> mkonecny: pick one you are interested in and I can tell you it's long and sad tale over some grog...
16:36:27 <mkonecny> Let's go with the oldest ones
16:36:37 <mkonecny> .ticket 7361
16:36:38 <zodbot> mkonecny: Issue #7361: Signing infrastructure for aarch64 - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/7361
16:37:06 <nirik> so, this needs me talking to pjones and figuring out how we can support this.
16:37:17 <nirik> which I should do I guess. It's been waiting a long long time.
16:37:53 <mkonecny> Or maybe it will be better to go from the oldest last update
16:38:05 <mkonecny> .ticket 8768
16:38:06 <zodbot> mkonecny: Issue #8768: perl-sig should watch all perl packages - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8768
16:38:20 <mkonecny> There is no response at all on this ticket
16:39:25 <nirik> I sent email to peter about that last ticket.
16:39:57 <nirik> mkonecny: well, we haven't asked them anything?
16:40:06 <nirik> I guess we could ask if it still matters/is needed?
16:40:48 <nirik> https://src.fedoraproject.org/group/perl-maint-sig <- 49 packages
16:41:11 <mkonecny> I will update the ticket and ask if this is still relevant
16:41:13 <nirik> https://src.fedoraproject.org/user/perl-sig/projects <- 33 packages
16:41:31 <nirik> so, I am not sure if they meant the user 'perl-sig' or the group 'perl-maint-sig'
16:42:15 <nirik> ie, do they need actual rights to commit/admin, or just want to be watching them all.
16:42:22 <mkonecny> They asked for perl-sig in the ticket
16:42:47 <mkonecny> Let's see if there will be response
16:43:10 <mkonecny> Comment added
16:44:02 <mkonecny> And I switched it to Waiting on Reporter
16:44:17 <mkonecny> Ok, next one
16:44:24 <mkonecny> .ticket 8729
16:44:25 <zodbot> mkonecny: Issue #8729: Do we want to have a copy of translate.fp.o backups? - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8729
16:44:59 <nirik> so, we have backups, but it's manual... we need some automated way to get it.
16:45:53 <eddiejennings> Is there a specific product / solution used for backing up our infrastructure?
16:46:05 <mkonecny> I'm for automation all the way
16:46:44 <nirik> eddiejennings: rdiff-backup currently is used to backup things to a netapp volume. That volume is then snapmirrored to another datacenter/netapp.
16:47:08 * eddiejennings nods.
16:48:05 <mkonecny> Are the manual steps documented somewhere?
16:49:19 <nirik> https://pagure.io/fedora-infrastructure/issue/8729#comment-655073 perhaps?
16:49:36 <nirik> not sure. I did get a copy at one point, but I'd have to look again...
16:50:39 <mkonecny> This probably wouldn't be that much trouble to automate, but let's see
16:50:58 <nirik> I could try and get a copy and update the ticket on the steps? or ?
16:51:54 <mkonecny> I saw that the latest archive was downloaded by borg, could we use it for automation?
16:53:20 <nirik> well, yeah, we would have to use borg to get the backup... which is kind of anoying if we are only using it for this one thing.
16:53:32 <nirik> I am not sure if borg is available in epel...
16:53:46 <mkonecny> Me neither
16:54:54 <mkonecny> It looks like the only ones who have access to this right now are jibecfed and pingou
16:55:46 <nirik> he added me too. I also have access. (Or did)
16:56:07 <mkonecny> But otherwise it looks like it could be only running the borg extract in cron
16:56:36 <nirik> but it also needs the passphrase and such... which is anoying to store in cron.
16:56:51 <mkonecny> I understand :-/
16:57:30 <nirik> so, we are almost out of time here...
16:57:30 <mkonecny> Not much we can do right now than
16:57:43 <nirik> how about we ping on the ticket and see if pingou wants to drive it forward.
16:57:46 <nirik> if not, I will try to
16:58:06 <mkonecny> Ok, I can ping him on the ticket
16:58:39 <nirik> +1
16:58:50 <nirik> #topic Open Floor / AOB / walk the plank
16:58:56 <nirik> any other business / discussion items?
16:58:57 <mkonecny> On the other side I got response about the gitlab-centos queue from bstinson, it's a GitLab webhook, but there is probably no consumer for it
16:59:15 <nirik> ah, should we clear it?
17:00:21 <mkonecny> lrossetti responded that they maybe use it for automotive, but he wasn't sure
17:00:33 <nirik> ok. we can deal with it out of meeting.
17:00:44 <nirik> Thanks everyone! fair winds!
17:00:47 <nirik> #endmeeting