2025-05-22 17:00:03 <@james:fedora.im> !startmeeting Infrastructure (2025-05-22)

2025-05-22 17:00:04 <@meetbot:fedora.im> Meeting started at 2025-05-22 17:00:03 UTC
2025-05-22 17:00:04 <@meetbot:fedora.im> The Meeting name is 'Infrastructure (2025-05-22)'
2025-05-22 17:00:07 <@james:fedora.im> !meetingname infrastructure
2025-05-22 17:00:07 <@meetbot:fedora.im> The Meeting Name is now infrastructure
2025-05-22 17:00:13 <@nirik:matrix.scrye.com> morning everyone.
2025-05-22 17:00:30 <@markrosenbaum:fedora.im> !hi
2025-05-22 17:00:31 <@zodbot:fedora.im> Mark Rosenbaum (markrosenbaum)
2025-05-22 17:00:31 <@james:fedora.im> !chair nirik  Zlopez  nb  bodanel dtometzki  jnsamyak  lenkaseg  patrikp 
2025-05-22 17:00:38 <@james:fedora.im> !info Agenda is at: https://board.net/p/fedora-infra
2025-05-22 17:00:41 <@james:fedora.im> !info About our team: https://docs.fedoraproject.org/en-US/cle/
2025-05-22 17:00:44 <@james:fedora.im> !info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
2025-05-22 17:00:47 <@james:fedora.im> !topic Hola y bienvenido
2025-05-22 17:02:11 <@nirik:matrix.scrye.com> ¿Cómo estás?
2025-05-22 17:02:20 <@james:fedora.im> Bien
2025-05-22 17:03:21 <@james:fedora.im> !topic New folks introductions
2025-05-22 17:03:27 <@james:fedora.im> !info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
2025-05-22 17:03:31 <@james:fedora.im> !info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
2025-05-22 17:03:40 <@james:fedora.im> Anyone new this week?
2025-05-22 17:04:14 <@nirik:matrix.scrye.com> probibly everyone is already gone for the weekend. ;)
2025-05-22 17:04:37 <@james:fedora.im> They are indeed the intelligent ones
2025-05-22 17:04:40 <@james:fedora.im> !topic Next chair
2025-05-22 17:05:18 <@james:fedora.im> Is there a command to see?
2025-05-22 17:05:35 <@james:fedora.im> The board has nobody for next week.
2025-05-22 17:05:38 <@nirik:matrix.scrye.com> I don't think we have anyone set yet...
2025-05-22 17:06:12 <@nirik:matrix.scrye.com> I guess I might be able to, but was hoping to avoid too much else with all the dc stuff going on.
2025-05-22 17:06:27 <@nirik:matrix.scrye.com> (and I think we should cancel the one after next week since flock will be happening)
2025-05-22 17:06:48 <@james:fedora.im> On the 5th of June, yeh that makes sense.
2025-05-22 17:07:58 <@james:fedora.im> Mark Rosenbaum: How do you feel about running it next week? :)
2025-05-22 17:08:33 <@markrosenbaum:fedora.im> Uhhhh, hmmmmm
2025-05-22 17:08:48 <@markrosenbaum:fedora.im> We may have a big conference next week at work
2025-05-22 17:09:01 <@markrosenbaum:fedora.im> If we don't end up going sure I can try
2025-05-22 17:09:43 <@james:fedora.im> Done ;)
2025-05-22 17:09:44 <@james:fedora.im> !topic announcements and information
2025-05-22 17:09:48 <@james:fedora.im> !info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0815 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2025-05-22 17:09:52 <@james:fedora.im> !info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1900 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2025-05-22 17:09:56 <@james:fedora.im> !info OpenID EOL in Fedora Infra is TBD as we changed the approach. See https://communityblog.fedoraproject.org/end-of-openid-authentication-in-fedora-account-system-may-update/
2025-05-22 17:10:16 <@james:fedora.im> !info Big upgrade today, starting at 21:00 UTC ... see status.
2025-05-22 17:10:29 <@nirik:matrix.scrye.com> !info tomorrow is a Red Hat recharge day
2025-05-22 17:10:29 <@james:fedora.im> !info Everyone is already on holiday until Tuesday.
2025-05-22 17:10:36 <@nirik:matrix.scrye.com> ha. yep
2025-05-22 17:11:16 <@james:fedora.im> !topic Oncall
2025-05-22 17:11:20 <@james:fedora.im> !info https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team
2025-05-22 17:11:23 <@james:fedora.im> !info nirik is on call from 2025-05-22 to 2025-05-29
2025-05-22 17:11:27 <@james:fedora.im> !oncall
2025-05-22 17:11:28 <@zodbot:fedora.im> ● @Zlopez:matrix.org (zlopez) Current Time for them: 19:11 (Europe/Prague)
2025-05-22 17:11:28 <@zodbot:fedora.im> The following people are oncall:
2025-05-22 17:11:28 <@zodbot:fedora.im> 
2025-05-22 17:11:28 <@zodbot:fedora.im> If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)
2025-05-22 17:12:10 <@james:fedora.im> Anyone want to take the week after (when everyone is at flock)
2025-05-22 17:13:02 <@nirik:matrix.scrye.com> !oncall
2025-05-22 17:13:03 <@zodbot:fedora.im> If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)
2025-05-22 17:13:03 <@zodbot:fedora.im> The following people are oncall:
2025-05-22 17:13:03 <@zodbot:fedora.im> 
2025-05-22 17:13:03 <@zodbot:fedora.im> ● @nirik:matrix.scrye.com (kevin) Current Time for them: 10:13 (US/Pacific)
2025-05-22 17:13:51 <@james:fedora.im> Can leave it ... I might be able to take the flock week, if nobody else can do it.
2025-05-22 17:13:56 <@james:fedora.im> !info Summary of last week: (from current oncall)
2025-05-22 17:14:33 <@nirik:matrix.scrye.com> I didn't see any pings off hand FWIW...
2025-05-22 17:14:50 <@james:fedora.im> Yeh, I don't remember seeing anything from Zlopez  ... but maybe
2025-05-22 17:14:58 <@james:fedora.im> !topic Monitoring discussion [nirik]
2025-05-22 17:15:01 <@james:fedora.im> !info https://nagios.fedoraproject.org/nagios 
2025-05-22 17:15:10 <@Zlopez:matrix.org> !hi
2025-05-22 17:15:12 <@zodbot:fedora.im> Michal Konecny (zlopez)
2025-05-22 17:15:15 <@Zlopez:matrix.org> I'm little late
2025-05-22 17:15:35 <@nirik:matrix.scrye.com> noting new on nagios I don't think... same as last week pretty much
2025-05-22 17:16:00 <@Zlopez:matrix.org> There were two pings for oncall, both of them related to IPA not working correctly after automatic dnf update
2025-05-22 17:16:25 <@Zlopez:matrix.org> I will start working on the datanommer checks next week, the reviews are finally done :-)
2025-05-22 17:16:33 <@nirik:matrix.scrye.com> oh nice.
2025-05-22 17:16:55 <@nirik:matrix.scrye.com> I am installing a noc01.rdu3... but I have nagios not set to install there. We will need to do some work to make the nagios templating work there.
2025-05-22 17:17:36 <@james:fedora.im> One thing a bit related to this ... we had a weird thing where backup01 lost the NFS mount and then tried to backup to / ... seems like it'd be better to have a nagios check that the NFS mount is there.
2025-05-22 17:18:13 <@Zlopez:matrix.org> What about going straight for zabbix?
2025-05-22 17:18:14 <@james:fedora.im> Why is nagios different there?
2025-05-22 17:18:59 <@nirik:matrix.scrye.com> we could make a nagios check. There's just not enough time to add all the checks that we might want.
2025-05-22 17:19:12 <@nirik:matrix.scrye.com> we could move to zabbix, but it's another thing changing at the same time we are trying to migrate.
2025-05-22 17:19:39 <@Zlopez:matrix.org> That is true, it will be simpler to go with what we have now and experiment after
2025-05-22 17:19:48 <@nirik:matrix.scrye.com> nagios generates it's configuration from ansible vars. There's almost surely some iad2 hard coding of groups or things in there.
2025-05-22 17:20:33 <@Zlopez:matrix.org> I wouldn't be surprised
2025-05-22 17:21:20 <@james:fedora.im> Ahh, yeh. Also I see enough swap/CPU utilization still for zabbix ... and I'm not sure how close zabbix is with porting our existing custom rules.
2025-05-22 17:21:52 <@nirik:matrix.scrye.com> Gwmngilfen got staging in much better shape, but there's probibly still checks to check and port...
2025-05-22 17:22:57 <@james:fedora.im> Did we want to look at backlog or blocked tickets?
2025-05-22 17:24:09 <@Zlopez:matrix.org> I would rather hear about the next steps in dc move, when we finally have network ready
2025-05-22 17:24:26 <@james:fedora.im> !topic Open Floor
2025-05-22 17:24:46 <@james:fedora.im> Seems fair ... you want to give an update nirik?
2025-05-22 17:25:12 <@nirik:matrix.scrye.com> Sure, I can
2025-05-22 17:25:49 <@nirik:matrix.scrye.com> so, we have network now ( 🎊 ). I have install a vmhost-x86-01.rdu3, and on it a bastion01 and ns01 and am working on a noc01...
2025-05-22 17:25:57 <@nirik:matrix.scrye.com> so, we have network now ( 🎊 ). I have installed a vmhost-x86-01.rdu3, and on it a bastion01 and ns01 and am working on a noc01...
2025-05-22 17:26:19 <@nirik:matrix.scrye.com> next week we will install more servers and bootstrap up more things.
2025-05-22 17:26:37 <@nirik:matrix.scrye.com> Hopefully then we can make some ipa replicas, some proxies and I can look at installing openshift clusters.
2025-05-22 17:26:41 <@Zlopez:matrix.org> I expect a call to arms soon :-)
2025-05-22 17:27:08 <@nirik:matrix.scrye.com> yeah. After clusters are up we need folks to look at what could be migrated before our big switch...
2025-05-22 17:27:31 <@nirik:matrix.scrye.com> then after flock we will look at moving anything we can early... and finish prep work... and week of june 30th is the switcharoo.
2025-05-22 17:27:47 <@Zlopez:matrix.org> That sounds doable
2025-05-22 17:28:04 <@farchord:fedora.im> Curious, what is the expected downtime? I assume there'll be a certain amount of time where systems will be down
2025-05-22 17:29:04 <@nirik:matrix.scrye.com> the week of june 30th... don't try and do anything monday or tuesday. :)
2025-05-22 17:29:14 <@nirik:matrix.scrye.com> wed hopefully things will be up again and we can start fixing
2025-05-22 17:29:18 <@farchord:fedora.im> ahhh fair enough 🙂
2025-05-22 17:30:10 <@nirik:matrix.scrye.com> I'd love to move things in parts completely, but a lot of things are intertwined: They have netapp storage, or databases that we need to make sure to migrate, etc.
2025-05-22 17:30:13 <@james:fedora.im> !info The week of June 30th, is the planned DC switchover... don't try and do anything Monday or Tuesday.
2025-05-22 17:30:52 <@farchord:fedora.im> Yeah might be worth announcing that in some shape or form, as a banner on src.fp.o/koji/bodhi/... or something
2025-05-22 17:31:08 <@nirik:matrix.scrye.com> I have pushed community blog articles all the while. ;)
2025-05-22 17:31:20 <@nirik:matrix.scrye.com> but yeah, we can after flock make more noise.
2025-05-22 17:32:22 <@nirik:matrix.scrye.com> Happy to answer any other questions folks have... gonna be a busy month.
2025-05-22 17:33:05 <@farchord:fedora.im> For the record -- I feel for you 😆 And I get it! I just think it should be plastered pretty much everywhere. Maybe it's just me 😆
2025-05-22 17:33:07 <@james:fedora.im> I'll look at doing a status PR on Tuesday, give people a month to see it.
2025-05-22 17:33:26 <@nirik:matrix.scrye.com> 👍
2025-05-22 17:33:31 <@james:fedora.im> Not sure how we get something on koji/bodhi/etc.
2025-05-22 17:33:45 <@nirik:matrix.scrye.com> note that it was going to be the 16th and slipped... so we need to make sure and keep any announcements/status up to date
2025-05-22 17:34:02 <@james:fedora.im> Yeh
2025-05-22 17:34:07 <@farchord:fedora.im> At minimum, can't a <div> be inserted in the theme?
2025-05-22 17:34:07 <@nirik:matrix.scrye.com> not sure we have much ability there. We can do banners on src/pagure I know
2025-05-22 17:35:07 <@Zlopez:matrix.org> Maybe article on fedora magazine would be fine as well
2025-05-22 17:35:49 <@farchord:fedora.im> Just to save us/you from answering 100s of messages in #admin:fedoraproject.org "Hey, uh, I'm trying to build package X and it's n0t w0rk1ng11!11"
2025-05-22 17:35:51 <@james:fedora.im> Huh, maybe obvious to others but checking the calendar ... The week of the 30th is Fri 4th July.
2025-05-22 17:35:59 <@nirik:matrix.scrye.com> yeah, although... if all goes as I plan, end users won't actually notice much
2025-05-22 17:36:09 <@nirik:matrix.scrye.com> yes, yes it is
2025-05-22 17:36:24 <@farchord:fedora.im> You know, I don't know if you are a gamer at all, but.... I feel like you just set a flag 😆
2025-05-22 17:36:41 <@nirik:matrix.scrye.com> Oh I love to tempt fate that way. :)
2025-05-22 17:38:52 <@nirik:matrix.scrye.com> that was 20 years ago now... ;) Saw something about it recently
2025-05-22 17:39:34 <@james:fedora.im> Yeh, I saw that go past too recently ... Time marches on. I've found it annoying that old people look like me now.
2025-05-22 17:39:37 <@farchord:fedora.im> I was playing WoW back then so I know the meme very well 😆
2025-05-22 17:39:53 <@nirik:matrix.scrye.com> it's pretty amusing. Or was at the time
2025-05-22 17:40:03 <@farchord:fedora.im> like the -50DKP meme
2025-05-22 17:40:17 <@farchord:fedora.im> But anyway -- Off topic
2025-05-22 17:40:28 <@Zlopez:matrix.org> I'm dealing with the same :-D
2025-05-22 17:40:37 <@nirik:matrix.scrye.com> anyhow, I hope we can move a number of things early, move things that tue/wed and get it all over with. ;)
2025-05-22 17:41:02 <@Zlopez:matrix.org> Could we also host a WoW server? :-D
2025-05-22 17:41:12 <@farchord:fedora.im> Not legally.
2025-05-22 17:41:23 <@Zlopez:matrix.org> It was a joke
2025-05-22 17:41:28 <@farchord:fedora.im> I know 😆
2025-05-22 17:43:20 <@nirik:matrix.scrye.com> Things should be nice once we are moved. The new HW is nice and fast and shiny.
2025-05-22 17:43:44 <@james:fedora.im> Okay, unless anyone speaks up I'll close the meeting and give everyone a few extra minutes back.
2025-05-22 17:43:44 <@farchord:fedora.im> So you're saying the kernel will now build in 10 seconds flat? Awesome!
2025-05-22 17:44:07 <@james:fedora.im> He's saying the servers might boot in 10 minutes flat ;)
2025-05-22 17:44:20 <@nirik:matrix.scrye.com> perhaps even 8 or 9!
2025-05-22 17:44:25 <@farchord:fedora.im> lol
2025-05-22 17:45:16 <@Zlopez:matrix.org> The power will be over 9000!
2025-05-22 17:45:32 <@james:fedora.im> !endmeeting