2024-04-25 08:25:06 <@meetbot:fedora.im> HTML Minutes: https://meetbot.fedoraproject.org/meeting-3_matrix_fedoraproject-org/2024-04-25/cpe-infra-releng-daily-standup.2024-04-25-08.00.html
2024-04-25 16:00:07 <@lenkaseg:fedora.im> !startmeeting Infrastructure (2024-04-25)
2024-04-25 16:00:08 <@meetbot:fedora.im> Meeting started at 2024-04-25 16:00:07 UTC
2024-04-25 16:00:08 <@meetbot:fedora.im> The Meeting name is 'Infrastructure (2024-04-25)'
2024-04-25 16:00:16 <@lenkaseg:fedora.im> !meetingname infrastructure
2024-04-25 16:00:21 <@lenkaseg:fedora.im> !chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
2024-04-25 16:00:26 <@lenkaseg:fedora.im> !info Agenda is at: https://board.net/p/fedora-infra
2024-04-25 16:00:33 <@lenkaseg:fedora.im> About our team: https://docs.fedoraproject.org/en-US/cpe/
2024-04-25 16:00:37 <@lenkaseg:fedora.im> Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
2024-04-25 16:00:43 <@lenkaseg:fedora.im> !topic namaste
2024-04-25 16:00:49 <@lenkaseg:fedora.im> Hi everybody!
2024-04-25 16:01:14 <@freedisch:matrix.org> Hello ๐๐ฝ
2024-04-25 16:01:20 <@aggraxis:fedora.im> Hello!
2024-04-25 16:01:47 <@Zlopez:matrix.org> !hi
2024-04-25 16:01:49 <@nirik:matrix.scrye.com> morning
2024-04-25 16:01:50 <@zodbot:fedora.im> Michal Konecny (zlopez)
2024-04-25 16:02:12 <@leo:fedora.im> hey!
2024-04-25 16:02:23 <@patrikp:matrix.org> Good evening. ๐
2024-04-25 16:02:38 <@lenkaseg:fedora.im> Nice to see so many folks today!
2024-04-25 16:02:44 <@lenkaseg:fedora.im> !topic New folks introductions
2024-04-25 16:02:54 <@lenkaseg:fedora.im> !info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
2024-04-25 16:03:22 <@lenkaseg:fedora.im> We have a new contributor here today, please welcome Paul Maconi !
2024-04-25 16:03:58 <@aggraxis:fedora.im> Thanks! I'm excited to be here.
2024-04-25 16:03:59 <@nirik:matrix.scrye.com> welcome!
2024-04-25 16:04:05 <@leo:fedora.im> welcome!
2024-04-25 16:04:42 <@lenkaseg:fedora.im> Paul Maconi: Would you like introduce yourself?
2024-04-25 16:05:16 <@nirik:matrix.scrye.com> are you more interested in operations type things? or development? we have both. ;)
2024-04-25 16:05:33 <@aggraxis:fedora.im> Sure! I have a longer introduction up on my discussion profile and in the join-sig mailing list that I am happy to summarize on your list later, but I'm probably what you guys call more of an operations type sysad.
2024-04-25 16:05:59 <@aggraxis:fedora.im> I work for a company that provides embedded service to the US govt, and our customer runs a series of software test beds on both open and closed networks.
2024-04-25 16:06:51 <@aggraxis:fedora.im> I sponsored and integrated the RHEL product (starting with RHEL 7) into our software baseline maybe 6 or so years ago, I forget. We work a lot with Windows and RHEL 7, 8, and 9.
2024-04-25 16:07:07 <@aggraxis:fedora.im> Especially with integration, automation, security compliance, automation. We love ansible
2024-04-25 16:07:34 <@aggraxis:fedora.im> Anyways, I use Fedora at home because it keeps me playing in a platform similar to what I work with.
2024-04-25 16:08:02 <@aggraxis:fedora.im> After all these years I saw the join us message and decided I had ignored it long enough, so here I am
2024-04-25 16:08:03 <@nirik:matrix.scrye.com> awesome.
2024-04-25 16:08:24 <@aggraxis:fedora.im> The people at the join-sig helped me find my way around a bit, and I've been peeking at your group for a couple of days.
2024-04-25 16:08:38 <@lenkaseg:fedora.im> good to see you here
2024-04-25 16:08:41 <@aggraxis:fedora.im> I'm hoping we can find a way for me to be of use to you
2024-04-25 16:09:48 <@lenkaseg:fedora.im> Sure, in case you see something that would interest you, let us know!
2024-04-25 16:10:03 <@nirik:matrix.scrye.com> do feel free to hang out in our rooms and ask questions or chime in on things that seem interesting to you...
2024-04-25 16:10:43 <@aheath1992:matrix.org> .hi
2024-04-25 16:10:47 <@aheath1992:matrix.org> !hi
2024-04-25 16:10:49 <@zodbot:fedora.im> Andrew Heath (aheath1992)
2024-04-25 16:10:50 <@aggraxis:fedora.im> Thanks! I appreciate the warm welcome
2024-04-25 16:11:06 <@lenkaseg:fedora.im> Let's move on to....
2024-04-25 16:11:11 <@lenkaseg:fedora.im> !topic Next chair
2024-04-25 16:11:16 <@lenkaseg:fedora.im> !info magic eight ball says:
2024-04-25 16:11:22 <@lenkaseg:fedora.im> !info chair 2024-04-25 lenkaseg
2024-04-25 16:11:38 <@lenkaseg:fedora.im> that's me today and we son't have anybody for the next week
2024-04-25 16:11:42 <@lenkaseg:fedora.im> any volunteers?
2024-04-25 16:11:55 <@lenkaseg:fedora.im> 2024-05-02 ???
2024-04-25 16:12:27 <@lenkaseg:fedora.im> or 2024-05-09 ???
2024-04-25 16:13:01 <@lenkaseg:fedora.im> I can take next week again then
2024-04-25 16:13:11 <@lenkaseg:fedora.im> not the one after that
2024-04-25 16:13:21 <@lenkaseg:fedora.im> !info chair 2024-05-02 lenkaseg
2024-04-25 16:14:24 <@Zlopez:matrix.org> Unfortunately I will not be around for those two
2024-04-25 16:14:41 <@nirik:matrix.scrye.com> Yeah, I am not sure I will be around for the one after next week either.
2024-04-25 16:14:53 <@lenkaseg:fedora.im> the same here :)
2024-04-25 16:15:08 <@leo:fedora.im> it's at a bad time for me :(
2024-04-25 16:15:13 <@lenkaseg:fedora.im> If almost nobody's around, maybe we should consider cancelling
2024-04-25 16:15:19 <@nirik:matrix.scrye.com> perhaps we cancel... yeah.
2024-04-25 16:15:53 <@lenkaseg:fedora.im> 2024-05-02, 2024-05-09 or both?
2024-04-25 16:16:45 <@aheath1992:matrix.org> I can cover 2024-05-09 if we don't cancel
2024-04-25 16:16:48 <@nirik:matrix.scrye.com> I will be here next week...
2024-04-25 16:17:12 <@nirik:matrix.scrye.com> so I was thnking 05-09... but if aheath1992 wants to run it, lets go ahead ?
2024-04-25 16:17:52 <@lenkaseg:fedora.im> sold to aheath1992! Congrats!
2024-04-25 16:18:10 <@lenkaseg:fedora.im> !info chair 2024-05-09 aheath1992
2024-04-25 16:18:24 <@nirik:matrix.scrye.com> I _might_ be there, not sure, we will see. ;)
2024-04-25 16:18:39 <@lenkaseg:fedora.im> Let's move on!
2024-04-25 16:18:40 <@lenkaseg:fedora.im> !topic announcements and information
2024-04-25 16:18:49 <@lenkaseg:fedora.im> !info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2024-04-25 16:18:56 <@lenkaseg:fedora.im> !info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
2024-04-25 16:19:01 <@lenkaseg:fedora.im> !info F40 released 2024-04-23
2024-04-25 16:19:08 <@lenkaseg:fedora.im> ๐
2024-04-25 16:19:21 <@leo:fedora.im> hooray!
2024-04-25 16:19:41 <@nirik:matrix.scrye.com> Congrats to everyone on f40. :) Great work...
2024-04-25 16:20:03 <@Zlopez:matrix.org> Congrats!
2024-04-25 16:21:04 <@lenkaseg:fedora.im> any other announcements?
2024-04-25 16:22:21 <@Zlopez:matrix.org> !info ARC investigation for dist-git is getting started
2024-04-25 16:22:55 <@Zlopez:matrix.org> !link https://pagure.io/fedora-infra/arc/issue/164
2024-04-25 16:23:40 <@nirik:matrix.scrye.com> going to be interesting to see...
2024-04-25 16:23:45 <@nirik:matrix.scrye.com> lots of work ahead there.
2024-04-25 16:24:23 <@aheath1992:matrix.org> Zlopez: who is heading up this investigation?
2024-04-25 16:25:27 <@nirik:matrix.scrye.com> jednorozec I think is...
2024-04-25 16:26:22 <@lenkaseg:fedora.im> Let's move to oncall =>
2024-04-25 16:26:23 <@lenkaseg:fedora.im> !topic Oncall
2024-04-25 16:26:32 <@lenkaseg:fedora.im> !info https://fedoraproject.org/wiki/Infrastructure/Oncall
2024-04-25 16:26:37 <@lenkaseg:fedora.im> https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
2024-04-25 16:26:43 <@lenkaseg:fedora.im> ##!oncall add
2024-04-25 16:27:13 <@lenkaseg:fedora.im> Whoa, did we have an oncall person this week?
2024-04-25 16:27:23 <@leo:fedora.im> yeah, Aurรฉlien B
2024-04-25 16:27:26 <@leo:fedora.im> yeah, Aurรฉlien B i think
2024-04-25 16:28:12 <@Zlopez:matrix.org> aheath1992: Akashdeep Dhar should be the one leading it
2024-04-25 16:28:56 <@Zlopez:matrix.org> jednorozec is gathering all the requirements for it
2024-04-25 16:29:00 <@lenkaseg:fedora.im> I don't think Aurelien is here today
2024-04-25 16:29:13 <@nirik:matrix.scrye.com> ah, ok. that makes sense.
2024-04-25 16:30:41 <@lenkaseg:fedora.im> Do we have a oncall volunteer for 2024-04-25 to 2024-05-02?
2024-04-25 16:30:52 <@nirik:matrix.scrye.com> I can take it...
2024-04-25 16:30:53 <@lenkaseg:fedora.im> Do we have an oncall volunteer for 2024-04-25 to 2024-05-02?
2024-04-25 16:31:08 <@lenkaseg:fedora.im> Thanks nirik!
2024-04-25 16:31:21 <@lenkaseg:fedora.im> !info nirik is on call from 2024-04-25 to 2024-05-02
2024-04-25 16:32:17 <@lenkaseg:fedora.im> Any volunteer for 2024-05-02 to 2024-05-09?
2024-04-25 16:32:20 <@leo:fedora.im> i'll take it
2024-04-25 16:32:34 <@lenkaseg:fedora.im> leo: Excellent, thanks!
2024-04-25 16:33:09 <@lenkaseg:fedora.im> !info leo is on call from 2024-05-02 to 2024-05-09
2024-04-25 16:33:29 <@lenkaseg:fedora.im> Let's move to
2024-04-25 16:33:30 <@lenkaseg:fedora.im> !topic Monitoring discussion [nirik]
2024-04-25 16:33:36 <@lenkaseg:fedora.im> Over to you nirik!
2024-04-25 16:33:59 <@nirik:matrix.scrye.com> so, I was hoping to say things were much better this week... but sadly...
2024-04-25 16:34:17 <@nirik:matrix.scrye.com> there's some badges alerts, but those are all on the old system. It was moved to openshift a few days ago.
2024-04-25 16:34:39 <@nirik:matrix.scrye.com> There's some space issues on proxies. We aren't fully sure whats causing it... just seems to be a bunch of smaller things adding up.
2024-04-25 16:35:23 <@nirik:matrix.scrye.com> finally there's some queue alerts... hopefully those will clear up on there own here now.
2024-04-25 16:35:35 <@nirik:matrix.scrye.com> Otherwise, no hosts down! :)
2024-04-25 16:35:58 <@nirik:matrix.scrye.com> nothing more off hand, unless folks have questions...
2024-04-25 16:37:23 <@lenkaseg:fedora.im> Forgot to paste the link, here it is
2024-04-25 16:37:23 <@lenkaseg:fedora.im> !info https://nagios.fedoraproject.org/nagios
2024-04-25 16:38:13 <@lenkaseg:fedora.im> I see no learning topic for today
2024-04-25 16:38:25 <@lenkaseg:fedora.im> That means we have 3 options probably:
2024-04-25 16:38:32 <@lenkaseg:fedora.im> RHEL7 EOL
2024-04-25 16:38:38 <@lenkaseg:fedora.im> Revisit blocked tickets
2024-04-25 16:38:47 <@lenkaseg:fedora.im> Fedora iNfra backlog refinement
2024-04-25 16:39:02 <@lenkaseg:fedora.im> The one with most ๐๏ธ wins
2024-04-25 16:39:34 <@lenkaseg:fedora.im> Unless we have something more important to discuss?
2024-04-25 16:39:55 <@leo:fedora.im> i don't think we have learning topic planned out anyway
2024-04-25 16:40:11 <@lenkaseg:fedora.im> Yeah, we didn't have any for a long time
2024-04-25 16:40:25 <@lenkaseg:fedora.im> !topic RHEL7 EOL
2024-04-25 16:40:31 <@lenkaseg:fedora.im> !info Check the progress on tickets related to RHEL7 EOL (end of June)
2024-04-25 16:40:34 <@nirik:matrix.scrye.com> We should do more of them... I can try. ;)
2024-04-25 16:40:37 <@lenkaseg:fedora.im> !ticket https://pagure.io/fedora-infrastructure/issue/11815
2024-04-25 16:41:03 <@Zlopez:matrix.org> That is the meta ticket, let me go one by one
2024-04-25 16:41:11 <@Zlopez:matrix.org> !ticket 11817
2024-04-25 16:41:14 <@zodbot:fedora.im> **fedora-infrastructure #11817** (https://pagure.io/fedora-infrastructure/issue/11817):**rhel7 - db-fas01.iad2.fedoraproject.org**
โ **Opened:** a month ago by zlopez
โ **Last Updated:** a month ago
โ **Assignee:** kevin
2024-04-25 16:41:59 <@Zlopez:matrix.org> I assume this could now be done when the freeze is over
2024-04-25 16:42:51 <@nirik:matrix.scrye.com> yes, it's on my list
2024-04-25 16:43:02 <@nirik:matrix.scrye.com> it will need a ipsilon outage.
2024-04-25 16:43:12 <@nirik:matrix.scrye.com> (although it could be pretty short I hope)
2024-04-25 16:43:32 <@nirik:matrix.scrye.com> so, yes, on track. I just need to do it.
2024-04-25 16:43:39 <@lenkaseg:fedora.im> Do we have an SOP for this? Would like to see how is it done
2024-04-25 16:44:23 <@aggraxis:fedora.im> We have a bunch of this kind of work in our backlog at the office as well. Some of our stuff is easier to re-plumb than others.
2024-04-25 16:44:30 <@nirik:matrix.scrye.com> I can write up something, but it is pretty infrequent... like every... 3-6 years?
2024-04-25 16:45:15 <@Zlopez:matrix.org> Maybe some general guide how to update VM in Fedora infra?
2024-04-25 16:45:18 <@nirik:matrix.scrye.com> short summary: install new rhel9 vm, rsync /var/lib/pgsql from old one, run pg-upgrade, upgrade postgres again, run pg-upgrade, take down old vm, change new vm to use same host/ip, reboot
2024-04-25 16:45:19 <@lenkaseg:fedora.im> You see, it's very rare learning opportunity! Now or in years! ๐
2024-04-25 16:46:01 <@nirik:matrix.scrye.com> database servers are a more complex case.
2024-04-25 16:46:14 <@james:fedora.im> Do we archive old VMs somewhere for a time, or nah?
2024-04-25 16:46:25 <@nirik:matrix.scrye.com> most vm's have storage elsewhere, so you can just: delete old vm, change ansible, redepoy
2024-04-25 16:46:46 <@nirik:matrix.scrye.com> depends on the use. Some of them we do yeah. Things like builders, nope
2024-04-25 16:47:33 <@nirik:matrix.scrye.com> speaking of db servers... Zlopez: I was gonna reinstall db01.stg the other day, but saw you were using it... can I do that sometime, or are you in the middle of something there?
2024-04-25 16:49:10 <@Zlopez:matrix.org> I finished the script for migrating mailman socialaccount to OIDC, I was just checking if everything went smoothly
2024-04-25 16:49:13 <@Zlopez:matrix.org> I can logout
2024-04-25 16:49:46 <@Zlopez:matrix.org> Feel free to reinstall it
2024-04-25 16:49:52 <@nirik:matrix.scrye.com> ok
2024-04-25 16:50:19 <@nirik:matrix.scrye.com> I can try and write up a thing based on that... and share it around/make it into a guide.
2024-04-25 16:51:07 <@Zlopez:matrix.org> That would be great, I can start some draft as I did updated few VMs by myself
2024-04-25 16:51:42 <@Zlopez:matrix.org> Let's go to next ticket
2024-04-25 16:51:46 <@Zlopez:matrix.org> !11821
2024-04-25 16:51:51 <@Zlopez:matrix.org> !ticket 11821
2024-04-25 16:51:53 <@zodbot:fedora.im> **fedora-infrastructure #11821** (https://pagure.io/fedora-infrastructure/issue/11821):**rhel7 - sundries servers**
โ **Opened:** a month ago by zlopez
โ **Last Updated:** a month ago
โ **Assignee:** kevin
2024-04-25 16:52:42 <@Zlopez:matrix.org> Isn't this used by anaconda geoip service?
2024-04-25 16:53:00 <@nirik:matrix.scrye.com> yes.
2024-04-25 16:53:08 <@nirik:matrix.scrye.com> this also is on my list...
2024-04-25 16:53:25 <@nirik:matrix.scrye.com> if someone wanted to look at the items it's doing and see if they need adjustment for rhel9 that would be nice.
2024-04-25 16:53:46 <@nirik:matrix.scrye.com> most of the things are just nfs mounts and rsync... that should work fine.
2024-04-25 16:53:56 <@nirik:matrix.scrye.com> geoip needs checking
2024-04-25 16:54:21 <@Zlopez:matrix.org> Do we have a playbook for sundries servers?
2024-04-25 16:54:48 <@nirik:matrix.scrye.com> yes. in the usual place. ;) playbooks/groups/sundries.
2024-04-25 16:55:26 <@nirik:matrix.scrye.com> I guess it might be good to make a sundries02.stg instance thats rhel9 and see what breaks. ;)
2024-04-25 16:55:38 <@nirik:matrix.scrye.com> if someone would like to take that on I can assist...
2024-04-25 16:56:12 <@Zlopez:matrix.org> I can look into that once the mailman is done, but I have plenty of work around that
2024-04-25 16:56:25 <@nirik:matrix.scrye.com> yeah. I can update the ticket on it.
2024-04-25 16:56:47 <@Zlopez:matrix.org> We have 4 more minutes
2024-04-25 16:57:02 <@Zlopez:matrix.org> Do we want to look at the next ticket or go directly to open floor?
2024-04-25 16:57:41 <@lenkaseg:fedora.im> 3 mins... I guess it's open floor
2024-04-25 16:57:44 <@lenkaseg:fedora.im> !topic Open Floor
2024-04-25 16:58:14 <@lenkaseg:fedora.im> How do we stand on the RHEL7 EOL? Are we making it?
2024-04-25 16:58:41 <@nirik:matrix.scrye.com> I think things are looking reasonable... but it's hard to tell if something show stopper will hit us. ;)
2024-04-25 16:58:51 <@Zlopez:matrix.org> We have plenty of things in progress and few already done
2024-04-25 16:59:29 <@Zlopez:matrix.org> We are on good track
2024-04-25 17:00:11 <@lenkaseg:fedora.im> Time's up, thanks for attending everybody!
2024-04-25 17:00:17 <@lenkaseg:fedora.im> !endmeeting