<@meetbot:fedora.im>
08:25:06
HTML Minutes: https://meetbot.fedoraproject.org/meeting-3_matrix_fedoraproject-org/2024-04-25/cpe-infra-releng-daily-standup.2024-04-25-08.00.html
<@lenkaseg:fedora.im>
16:00:07
!startmeeting Infrastructure (2024-04-25)
<@meetbot:fedora.im>
16:00:08
Meeting started at 2024-04-25 16:00:07 UTC
<@meetbot:fedora.im>
16:00:08
The Meeting name is 'Infrastructure (2024-04-25)'
<@lenkaseg:fedora.im>
16:00:16
!meetingname infrastructure
<@lenkaseg:fedora.im>
16:00:21
!chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
<@lenkaseg:fedora.im>
16:00:26
!info Agenda is at: https://board.net/p/fedora-infra
<@lenkaseg:fedora.im>
16:00:33
About our team: https://docs.fedoraproject.org/en-US/cpe/
<@lenkaseg:fedora.im>
16:00:37
Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
<@lenkaseg:fedora.im>
16:00:43
!topic namaste
<@lenkaseg:fedora.im>
16:00:49
Hi everybody!
<@freedisch:matrix.org>
16:01:14
Hello 👋🏽
<@aggraxis:fedora.im>
16:01:20
Hello!
<@Zlopez:matrix.org>
16:01:47
!hi
<@nirik:matrix.scrye.com>
16:01:49
morning
<@zodbot:fedora.im>
16:01:50
Michal Konecny (zlopez)
<@leo:fedora.im>
16:02:12
hey!
<@patrikp:matrix.org>
16:02:23
Good evening. 👋
<@lenkaseg:fedora.im>
16:02:38
Nice to see so many folks today!
<@lenkaseg:fedora.im>
16:02:44
!topic New folks introductions
<@lenkaseg:fedora.im>
16:02:54
!info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
<@lenkaseg:fedora.im>
16:03:22
We have a new contributor here today, please welcome Paul Maconi !
<@aggraxis:fedora.im>
16:03:58
Thanks! I'm excited to be here.
<@nirik:matrix.scrye.com>
16:03:59
welcome!
<@leo:fedora.im>
16:04:05
welcome!
<@lenkaseg:fedora.im>
16:04:42
Paul Maconi: Would you like introduce yourself?
<@nirik:matrix.scrye.com>
16:05:16
are you more interested in operations type things? or development? we have both. ;)
<@aggraxis:fedora.im>
16:05:33
Sure! I have a longer introduction up on my discussion profile and in the join-sig mailing list that I am happy to summarize on your list later, but I'm probably what you guys call more of an operations type sysad.
<@aggraxis:fedora.im>
16:05:59
I work for a company that provides embedded service to the US govt, and our customer runs a series of software test beds on both open and closed networks.
<@aggraxis:fedora.im>
16:06:51
I sponsored and integrated the RHEL product (starting with RHEL 7) into our software baseline maybe 6 or so years ago, I forget. We work a lot with Windows and RHEL 7, 8, and 9.
<@aggraxis:fedora.im>
16:07:07
Especially with integration, automation, security compliance, automation. We love ansible
<@aggraxis:fedora.im>
16:07:34
Anyways, I use Fedora at home because it keeps me playing in a platform similar to what I work with.
<@aggraxis:fedora.im>
16:08:02
After all these years I saw the join us message and decided I had ignored it long enough, so here I am
<@nirik:matrix.scrye.com>
16:08:03
awesome.
<@aggraxis:fedora.im>
16:08:24
The people at the join-sig helped me find my way around a bit, and I've been peeking at your group for a couple of days.
<@lenkaseg:fedora.im>
16:08:38
good to see you here
<@aggraxis:fedora.im>
16:08:41
I'm hoping we can find a way for me to be of use to you
<@lenkaseg:fedora.im>
16:09:48
Sure, in case you see something that would interest you, let us know!
<@nirik:matrix.scrye.com>
16:10:03
do feel free to hang out in our rooms and ask questions or chime in on things that seem interesting to you...
<@aheath1992:matrix.org>
16:10:43
.hi
<@aheath1992:matrix.org>
16:10:47
!hi
<@zodbot:fedora.im>
16:10:49
Andrew Heath (aheath1992)
<@aggraxis:fedora.im>
16:10:50
Thanks! I appreciate the warm welcome
<@lenkaseg:fedora.im>
16:11:06
Let's move on to....
<@lenkaseg:fedora.im>
16:11:11
!topic Next chair
<@lenkaseg:fedora.im>
16:11:16
!info magic eight ball says:
<@lenkaseg:fedora.im>
16:11:22
!info chair 2024-04-25 lenkaseg
<@lenkaseg:fedora.im>
16:11:38
that's me today and we son't have anybody for the next week
<@lenkaseg:fedora.im>
16:11:42
any volunteers?
<@lenkaseg:fedora.im>
16:11:55
2024-05-02 ???
<@lenkaseg:fedora.im>
16:12:27
or 2024-05-09 ???
<@lenkaseg:fedora.im>
16:13:01
I can take next week again then
<@lenkaseg:fedora.im>
16:13:11
not the one after that
<@lenkaseg:fedora.im>
16:13:21
!info chair 2024-05-02 lenkaseg
<@Zlopez:matrix.org>
16:14:24
Unfortunately I will not be around for those two
<@nirik:matrix.scrye.com>
16:14:41
Yeah, I am not sure I will be around for the one after next week either.
<@lenkaseg:fedora.im>
16:14:53
the same here :)
<@leo:fedora.im>
16:15:08
it's at a bad time for me :(
<@lenkaseg:fedora.im>
16:15:13
If almost nobody's around, maybe we should consider cancelling
<@nirik:matrix.scrye.com>
16:15:19
perhaps we cancel... yeah.
<@lenkaseg:fedora.im>
16:15:53
2024-05-02, 2024-05-09 or both?
<@aheath1992:matrix.org>
16:16:45
I can cover 2024-05-09 if we don't cancel
<@nirik:matrix.scrye.com>
16:16:48
I will be here next week...
<@nirik:matrix.scrye.com>
16:17:12
so I was thnking 05-09... but if aheath1992 wants to run it, lets go ahead ?
<@lenkaseg:fedora.im>
16:17:52
sold to aheath1992! Congrats!
<@lenkaseg:fedora.im>
16:18:10
!info chair 2024-05-09 aheath1992
<@nirik:matrix.scrye.com>
16:18:24
I _might_ be there, not sure, we will see. ;)
<@lenkaseg:fedora.im>
16:18:39
Let's move on!
<@lenkaseg:fedora.im>
16:18:40
!topic announcements and information
<@lenkaseg:fedora.im>
16:18:49
!info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
<@lenkaseg:fedora.im>
16:18:56
!info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
<@lenkaseg:fedora.im>
16:19:01
!info F40 released 2024-04-23
<@lenkaseg:fedora.im>
16:19:08
🎉
<@leo:fedora.im>
16:19:21
hooray!
<@nirik:matrix.scrye.com>
16:19:41
Congrats to everyone on f40. :) Great work...
<@Zlopez:matrix.org>
16:20:03
Congrats!
<@lenkaseg:fedora.im>
16:21:04
any other announcements?
<@Zlopez:matrix.org>
16:22:21
!info ARC investigation for dist-git is getting started
<@Zlopez:matrix.org>
16:22:55
!link https://pagure.io/fedora-infra/arc/issue/164
<@nirik:matrix.scrye.com>
16:23:40
going to be interesting to see...
<@nirik:matrix.scrye.com>
16:23:45
lots of work ahead there.
<@aheath1992:matrix.org>
16:24:23
Zlopez: who is heading up this investigation?
<@nirik:matrix.scrye.com>
16:25:27
jednorozec I think is...
<@lenkaseg:fedora.im>
16:26:22
Let's move to oncall =>
<@lenkaseg:fedora.im>
16:26:23
!topic Oncall
<@lenkaseg:fedora.im>
16:26:32
!info https://fedoraproject.org/wiki/Infrastructure/Oncall
<@lenkaseg:fedora.im>
16:26:37
https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
<@lenkaseg:fedora.im>
16:26:43
##!oncall add <fas_name>
<@lenkaseg:fedora.im>
16:27:13
Whoa, did we have an oncall person this week?
<@leo:fedora.im>
16:27:23
yeah, Aurélien B
<@leo:fedora.im>
16:27:26
yeah, Aurélien B i think
<@Zlopez:matrix.org>
16:28:12
aheath1992: Akashdeep Dhar should be the one leading it
<@Zlopez:matrix.org>
16:28:56
jednorozec is gathering all the requirements for it
<@lenkaseg:fedora.im>
16:29:00
I don't think Aurelien is here today
<@nirik:matrix.scrye.com>
16:29:13
ah, ok. that makes sense.
<@lenkaseg:fedora.im>
16:30:41
Do we have a oncall volunteer for 2024-04-25 to 2024-05-02?
<@nirik:matrix.scrye.com>
16:30:52
I can take it...
<@lenkaseg:fedora.im>
16:30:53
Do we have an oncall volunteer for 2024-04-25 to 2024-05-02?
<@lenkaseg:fedora.im>
16:31:08
Thanks nirik!
<@lenkaseg:fedora.im>
16:31:21
!info nirik is on call from 2024-04-25 to 2024-05-02
<@lenkaseg:fedora.im>
16:32:17
Any volunteer for 2024-05-02 to 2024-05-09?
<@leo:fedora.im>
16:32:20
i'll take it
<@lenkaseg:fedora.im>
16:32:34
leo: Excellent, thanks!
<@lenkaseg:fedora.im>
16:33:09
!info leo is on call from 2024-05-02 to 2024-05-09
<@lenkaseg:fedora.im>
16:33:29
Let's move to
<@lenkaseg:fedora.im>
16:33:30
!topic Monitoring discussion [nirik]
<@lenkaseg:fedora.im>
16:33:36
Over to you nirik!
<@nirik:matrix.scrye.com>
16:33:59
so, I was hoping to say things were much better this week... but sadly...
<@nirik:matrix.scrye.com>
16:34:17
there's some badges alerts, but those are all on the old system. It was moved to openshift a few days ago.
<@nirik:matrix.scrye.com>
16:34:39
There's some space issues on proxies. We aren't fully sure whats causing it... just seems to be a bunch of smaller things adding up.
<@nirik:matrix.scrye.com>
16:35:23
finally there's some queue alerts... hopefully those will clear up on there own here now.
<@nirik:matrix.scrye.com>
16:35:35
Otherwise, no hosts down! :)
<@nirik:matrix.scrye.com>
16:35:58
nothing more off hand, unless folks have questions...
<@lenkaseg:fedora.im>
16:37:23
Forgot to paste the link, here it is
<@lenkaseg:fedora.im>
16:37:23
!info https://nagios.fedoraproject.org/nagios
<@lenkaseg:fedora.im>
16:38:13
I see no learning topic for today
<@lenkaseg:fedora.im>
16:38:25
That means we have 3 options probably:
<@lenkaseg:fedora.im>
16:38:32
RHEL7 EOL
<@lenkaseg:fedora.im>
16:38:38
Revisit blocked tickets
<@lenkaseg:fedora.im>
16:38:47
Fedora iNfra backlog refinement
<@lenkaseg:fedora.im>
16:39:02
The one with most 👍️ wins
<@lenkaseg:fedora.im>
16:39:34
Unless we have something more important to discuss?
<@leo:fedora.im>
16:39:55
i don't think we have learning topic planned out anyway
<@lenkaseg:fedora.im>
16:40:11
Yeah, we didn't have any for a long time
<@lenkaseg:fedora.im>
16:40:25
!topic RHEL7 EOL
<@lenkaseg:fedora.im>
16:40:31
!info Check the progress on tickets related to RHEL7 EOL (end of June)
<@nirik:matrix.scrye.com>
16:40:34
We should do more of them... I can try. ;)
<@lenkaseg:fedora.im>
16:40:37
!ticket https://pagure.io/fedora-infrastructure/issue/11815
<@Zlopez:matrix.org>
16:41:03
That is the meta ticket, let me go one by one
<@Zlopez:matrix.org>
16:41:11
!ticket 11817
<@zodbot:fedora.im>
16:41:14
**fedora-infrastructure #11817** (https://pagure.io/fedora-infrastructure/issue/11817):**rhel7 - db-fas01.iad2.fedoraproject.org**
● **Opened:** a month ago by zlopez
● **Last Updated:** a month ago
● **Assignee:** kevin
<@Zlopez:matrix.org>
16:41:59
I assume this could now be done when the freeze is over
<@nirik:matrix.scrye.com>
16:42:51
yes, it's on my list
<@nirik:matrix.scrye.com>
16:43:02
it will need a ipsilon outage.
<@nirik:matrix.scrye.com>
16:43:12
(although it could be pretty short I hope)
<@nirik:matrix.scrye.com>
16:43:32
so, yes, on track. I just need to do it.
<@lenkaseg:fedora.im>
16:43:39
Do we have an SOP for this? Would like to see how is it done
<@aggraxis:fedora.im>
16:44:23
We have a bunch of this kind of work in our backlog at the office as well. Some of our stuff is easier to re-plumb than others.
<@nirik:matrix.scrye.com>
16:44:30
I can write up something, but it is pretty infrequent... like every... 3-6 years?
<@Zlopez:matrix.org>
16:45:15
Maybe some general guide how to update VM in Fedora infra?
<@nirik:matrix.scrye.com>
16:45:18
short summary: install new rhel9 vm, rsync /var/lib/pgsql from old one, run pg-upgrade, upgrade postgres again, run pg-upgrade, take down old vm, change new vm to use same host/ip, reboot
<@lenkaseg:fedora.im>
16:45:19
You see, it's very rare learning opportunity! Now or in years! 😀
<@nirik:matrix.scrye.com>
16:46:01
database servers are a more complex case.
<@james:fedora.im>
16:46:14
Do we archive old VMs somewhere for a time, or nah?
<@nirik:matrix.scrye.com>
16:46:25
most vm's have storage elsewhere, so you can just: delete old vm, change ansible, redepoy
<@nirik:matrix.scrye.com>
16:46:46
depends on the use. Some of them we do yeah. Things like builders, nope
<@nirik:matrix.scrye.com>
16:47:33
speaking of db servers... Zlopez: I was gonna reinstall db01.stg the other day, but saw you were using it... can I do that sometime, or are you in the middle of something there?
<@Zlopez:matrix.org>
16:49:10
I finished the script for migrating mailman socialaccount to OIDC, I was just checking if everything went smoothly
<@Zlopez:matrix.org>
16:49:13
I can logout
<@Zlopez:matrix.org>
16:49:46
Feel free to reinstall it
<@nirik:matrix.scrye.com>
16:49:52
ok
<@nirik:matrix.scrye.com>
16:50:19
I can try and write up a thing based on that... and share it around/make it into a guide.
<@Zlopez:matrix.org>
16:51:07
That would be great, I can start some draft as I did updated few VMs by myself
<@Zlopez:matrix.org>
16:51:42
Let's go to next ticket
<@Zlopez:matrix.org>
16:51:46
!11821
<@Zlopez:matrix.org>
16:51:51
!ticket 11821
<@zodbot:fedora.im>
16:51:53
**fedora-infrastructure #11821** (https://pagure.io/fedora-infrastructure/issue/11821):**rhel7 - sundries servers**
● **Opened:** a month ago by zlopez
● **Last Updated:** a month ago
● **Assignee:** kevin
<@Zlopez:matrix.org>
16:52:42
Isn't this used by anaconda geoip service?
<@nirik:matrix.scrye.com>
16:53:00
yes.
<@nirik:matrix.scrye.com>
16:53:08
this also is on my list...
<@nirik:matrix.scrye.com>
16:53:25
if someone wanted to look at the items it's doing and see if they need adjustment for rhel9 that would be nice.
<@nirik:matrix.scrye.com>
16:53:46
most of the things are just nfs mounts and rsync... that should work fine.
<@nirik:matrix.scrye.com>
16:53:56
geoip needs checking
<@Zlopez:matrix.org>
16:54:21
Do we have a playbook for sundries servers?
<@nirik:matrix.scrye.com>
16:54:48
yes. in the usual place. ;) playbooks/groups/sundries.
<@nirik:matrix.scrye.com>
16:55:26
I guess it might be good to make a sundries02.stg instance thats rhel9 and see what breaks. ;)
<@nirik:matrix.scrye.com>
16:55:38
if someone would like to take that on I can assist...
<@Zlopez:matrix.org>
16:56:12
I can look into that once the mailman is done, but I have plenty of work around that
<@nirik:matrix.scrye.com>
16:56:25
yeah. I can update the ticket on it.
<@Zlopez:matrix.org>
16:56:47
We have 4 more minutes
<@Zlopez:matrix.org>
16:57:02
Do we want to look at the next ticket or go directly to open floor?
<@lenkaseg:fedora.im>
16:57:41
3 mins... I guess it's open floor
<@lenkaseg:fedora.im>
16:57:44
!topic Open Floor
<@lenkaseg:fedora.im>
16:58:14
How do we stand on the RHEL7 EOL? Are we making it?
<@nirik:matrix.scrye.com>
16:58:41
I think things are looking reasonable... but it's hard to tell if something show stopper will hit us. ;)
<@Zlopez:matrix.org>
16:58:51
We have plenty of things in progress and few already done
<@Zlopez:matrix.org>
16:59:29
We are on good track
<@lenkaseg:fedora.im>
17:00:11
Time's up, thanks for attending everybody!
<@lenkaseg:fedora.im>
17:00:17
!endmeeting