#meeting-3:fedoraproject.org log

<@james:fedora.im>

17:00:03

!startmeeting Infrastructure (2025-05-22)

<@meetbot:fedora.im>

17:00:04

Meeting started at 2025-05-22 17:00:03 UTC

<@meetbot:fedora.im>

17:00:04

The Meeting name is 'Infrastructure (2025-05-22)'

<@james:fedora.im>

17:00:07

!meetingname infrastructure

<@meetbot:fedora.im>

17:00:07

The Meeting Name is now infrastructure

<@nirik:matrix.scrye.com>

17:00:13

morning everyone.

<@markrosenbaum:fedora.im>

17:00:30

!hi

<@zodbot:fedora.im>

17:00:31

Mark Rosenbaum (markrosenbaum)

<@james:fedora.im>

17:00:31

!chair nirik Zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp

<@james:fedora.im>

17:00:38

!info Agenda is at: https://board.net/p/fedora-infra

<@james:fedora.im>

17:00:41

!info About our team: https://docs.fedoraproject.org/en-US/cle/

<@james:fedora.im>

17:00:44

!info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra

<@james:fedora.im>

17:00:47

!topic Hola y bienvenido

<@nirik:matrix.scrye.com>

17:02:11

¿Cómo estás?

<@james:fedora.im>

17:02:20

Bien

<@james:fedora.im>

17:03:21

!topic New folks introductions

<@james:fedora.im>

17:03:27

!info This is a place where people who are interested in Fedora Infrastructure can introduce themselves

<@james:fedora.im>

17:03:31

!info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/

<@james:fedora.im>

17:03:40

Anyone new this week?

<@nirik:matrix.scrye.com>

17:04:14

probibly everyone is already gone for the weekend. ;)

<@james:fedora.im>

17:04:37

They are indeed the intelligent ones

<@james:fedora.im>

17:04:40

!topic Next chair

<@james:fedora.im>

17:05:18

Is there a command to see?

<@james:fedora.im>

17:05:35

The board has nobody for next week.

<@nirik:matrix.scrye.com>

17:05:38

I don't think we have anyone set yet...

<@nirik:matrix.scrye.com>

17:06:12

I guess I might be able to, but was hoping to avoid too much else with all the dc stuff going on.

<@nirik:matrix.scrye.com>

17:06:27

(and I think we should cancel the one after next week since flock will be happening)

<@james:fedora.im>

17:06:48

On the 5th of June, yeh that makes sense.

<@james:fedora.im>

17:07:58

Mark Rosenbaum: How do you feel about running it next week? :)

<@markrosenbaum:fedora.im>

17:08:33

Uhhhh, hmmmmm

<@markrosenbaum:fedora.im>

17:08:48

We may have a big conference next week at work

<@markrosenbaum:fedora.im>

17:09:01

If we don't end up going sure I can try

<@james:fedora.im>

17:09:43

Done ;)

<@james:fedora.im>

17:09:44

!topic announcements and information

<@james:fedora.im>

17:09:48

!info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0815 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org

<@james:fedora.im>

17:09:52

!info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1900 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org

<@james:fedora.im>

17:09:56

!info OpenID EOL in Fedora Infra is TBD as we changed the approach. See https://communityblog.fedoraproject.org/end-of-openid-authentication-in-fedora-account-system-may-update/

<@james:fedora.im>

17:10:16

!info Big upgrade today, starting at 21:00 UTC ... see status.

<@nirik:matrix.scrye.com>

17:10:29

!info tomorrow is a Red Hat recharge day

<@james:fedora.im>

17:10:29

!info Everyone is already on holiday until Tuesday.

<@nirik:matrix.scrye.com>

17:10:36

ha. yep

<@james:fedora.im>

17:11:16

!topic Oncall

<@james:fedora.im>

17:11:20

!info https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team

<@james:fedora.im>

17:11:23

!info nirik is on call from 2025-05-22 to 2025-05-29

<@james:fedora.im>

17:11:27

!oncall

<@zodbot:fedora.im>

17:11:28

● @Zlopez:matrix.org (zlopez) Current Time for them: 19:11 (Europe/Prague)

<@zodbot:fedora.im>

17:11:28

The following people are oncall:

<@zodbot:fedora.im>

17:11:28

<@zodbot:fedora.im>

17:11:28

If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)

<@james:fedora.im>

17:12:10

Anyone want to take the week after (when everyone is at flock)

<@nirik:matrix.scrye.com>

17:13:02

!oncall

<@zodbot:fedora.im>

17:13:03

If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)

<@zodbot:fedora.im>

17:13:03

The following people are oncall:

<@zodbot:fedora.im>

17:13:03

<@zodbot:fedora.im>

17:13:03

● @nirik:matrix.scrye.com (kevin) Current Time for them: 10:13 (US/Pacific)

<@james:fedora.im>

17:13:51

Can leave it ... I might be able to take the flock week, if nobody else can do it.

<@james:fedora.im>

17:13:56

!info Summary of last week: (from current oncall)

<@nirik:matrix.scrye.com>

17:14:33

I didn't see any pings off hand FWIW...

<@james:fedora.im>

17:14:50

Yeh, I don't remember seeing anything from Zlopez ... but maybe

<@james:fedora.im>

17:14:58

!topic Monitoring discussion [nirik]

<@james:fedora.im>

17:15:01

!info https://nagios.fedoraproject.org/nagios

<@Zlopez:matrix.org>

17:15:10

!hi

<@zodbot:fedora.im>

17:15:12

Michal Konecny (zlopez)

<@Zlopez:matrix.org>

17:15:15

I'm little late

<@nirik:matrix.scrye.com>

17:15:35

noting new on nagios I don't think... same as last week pretty much

<@Zlopez:matrix.org>

17:16:00

There were two pings for oncall, both of them related to IPA not working correctly after automatic dnf update

<@Zlopez:matrix.org>

17:16:25

I will start working on the datanommer checks next week, the reviews are finally done :-)

<@nirik:matrix.scrye.com>

17:16:33

oh nice.

<@nirik:matrix.scrye.com>

17:16:55

I am installing a noc01.rdu3... but I have nagios not set to install there. We will need to do some work to make the nagios templating work there.

<@james:fedora.im>

17:17:36

One thing a bit related to this ... we had a weird thing where backup01 lost the NFS mount and then tried to backup to / ... seems like it'd be better to have a nagios check that the NFS mount is there.

<@Zlopez:matrix.org>

17:18:13

What about going straight for zabbix?

<@james:fedora.im>

17:18:14

Why is nagios different there?

<@nirik:matrix.scrye.com>

17:18:59

we could make a nagios check. There's just not enough time to add all the checks that we might want.

<@nirik:matrix.scrye.com>

17:19:12

we could move to zabbix, but it's another thing changing at the same time we are trying to migrate.

<@Zlopez:matrix.org>

17:19:39

That is true, it will be simpler to go with what we have now and experiment after

<@nirik:matrix.scrye.com>

17:19:48

nagios generates it's configuration from ansible vars. There's almost surely some iad2 hard coding of groups or things in there.

<@Zlopez:matrix.org>

17:20:33

I wouldn't be surprised

<@james:fedora.im>

17:21:20

Ahh, yeh. Also I see enough swap/CPU utilization still for zabbix ... and I'm not sure how close zabbix is with porting our existing custom rules.

<@nirik:matrix.scrye.com>

17:21:52

Gwmngilfen got staging in much better shape, but there's probibly still checks to check and port...

<@james:fedora.im>

17:22:57

Did we want to look at backlog or blocked tickets?

<@Zlopez:matrix.org>

17:24:09

I would rather hear about the next steps in dc move, when we finally have network ready

<@james:fedora.im>

17:24:26

!topic Open Floor

<@james:fedora.im>

17:24:46

Seems fair ... you want to give an update nirik?

<@nirik:matrix.scrye.com>

17:25:12

Sure, I can

<@nirik:matrix.scrye.com>

17:25:49

so, we have network now ( 🎊 ). I have install a vmhost-x86-01.rdu3, and on it a bastion01 and ns01 and am working on a noc01...

<@nirik:matrix.scrye.com>

17:25:57

so, we have network now ( 🎊 ). I have installed a vmhost-x86-01.rdu3, and on it a bastion01 and ns01 and am working on a noc01...

<@nirik:matrix.scrye.com>

17:26:19

next week we will install more servers and bootstrap up more things.

<@nirik:matrix.scrye.com>

17:26:37

Hopefully then we can make some ipa replicas, some proxies and I can look at installing openshift clusters.

<@Zlopez:matrix.org>

17:26:41

I expect a call to arms soon :-)

<@nirik:matrix.scrye.com>

17:27:08

yeah. After clusters are up we need folks to look at what could be migrated before our big switch...

<@nirik:matrix.scrye.com>

17:27:31

then after flock we will look at moving anything we can early... and finish prep work... and week of june 30th is the switcharoo.

<@Zlopez:matrix.org>

17:27:47

That sounds doable

<@farchord:fedora.im>

17:28:04

Curious, what is the expected downtime? I assume there'll be a certain amount of time where systems will be down

<@nirik:matrix.scrye.com>

17:29:04

the week of june 30th... don't try and do anything monday or tuesday. :)

<@nirik:matrix.scrye.com>

17:29:14

wed hopefully things will be up again and we can start fixing

<@farchord:fedora.im>

17:29:18

ahhh fair enough 🙂

<@nirik:matrix.scrye.com>

17:30:10

I'd love to move things in parts completely, but a lot of things are intertwined: They have netapp storage, or databases that we need to make sure to migrate, etc.

<@james:fedora.im>

17:30:13

!info The week of June 30th, is the planned DC switchover... don't try and do anything Monday or Tuesday.

<@farchord:fedora.im>

17:30:52

Yeah might be worth announcing that in some shape or form, as a banner on src.fp.o/koji/bodhi/... or something

<@nirik:matrix.scrye.com>

17:31:08

I have pushed community blog articles all the while. ;)

<@nirik:matrix.scrye.com>

17:31:20

but yeah, we can after flock make more noise.

<@nirik:matrix.scrye.com>

17:32:22

Happy to answer any other questions folks have... gonna be a busy month.

<@farchord:fedora.im>

17:33:05

For the record -- I feel for you 😆 And I get it! I just think it should be plastered pretty much everywhere. Maybe it's just me 😆

<@james:fedora.im>

17:33:07

I'll look at doing a status PR on Tuesday, give people a month to see it.

<@nirik:matrix.scrye.com>

17:33:26

👍

<@james:fedora.im>

17:33:31

Not sure how we get something on koji/bodhi/etc.

<@nirik:matrix.scrye.com>

17:33:45

note that it was going to be the 16th and slipped... so we need to make sure and keep any announcements/status up to date

<@james:fedora.im>

17:34:02

Yeh

<@farchord:fedora.im>

17:34:07

At minimum, can't a <div> be inserted in the theme?

<@nirik:matrix.scrye.com>

17:34:07

not sure we have much ability there. We can do banners on src/pagure I know

<@Zlopez:matrix.org>

17:35:07

Maybe article on fedora magazine would be fine as well

<@farchord:fedora.im>

17:35:49

Just to save us/you from answering 100s of messages in #admin:fedoraproject.org "Hey, uh, I'm trying to build package X and it's n0t w0rk1ng11!11"

<@james:fedora.im>

17:35:51

Huh, maybe obvious to others but checking the calendar ... The week of the 30th is Fri 4th July.

<@nirik:matrix.scrye.com>

17:35:59

yeah, although... if all goes as I plan, end users won't actually notice much

<@nirik:matrix.scrye.com>

17:36:09

yes, yes it is

<@farchord:fedora.im>

17:36:24

You know, I don't know if you are a gamer at all, but.... I feel like you just set a flag 😆

<@nirik:matrix.scrye.com>

17:36:41

Oh I love to tempt fate that way. :)

<@nirik:matrix.scrye.com>

17:38:52

that was 20 years ago now... ;) Saw something about it recently

<@james:fedora.im>

17:39:34

Yeh, I saw that go past too recently ... Time marches on. I've found it annoying that old people look like me now.

<@farchord:fedora.im>

17:39:37

I was playing WoW back then so I know the meme very well 😆

<@nirik:matrix.scrye.com>

17:39:53

it's pretty amusing. Or was at the time

<@farchord:fedora.im>

17:40:03

like the -50DKP meme

<@farchord:fedora.im>

17:40:17

But anyway -- Off topic

<@Zlopez:matrix.org>

17:40:28

I'm dealing with the same :-D

<@nirik:matrix.scrye.com>

17:40:37

anyhow, I hope we can move a number of things early, move things that tue/wed and get it all over with. ;)

<@Zlopez:matrix.org>

17:41:02

Could we also host a WoW server? :-D

<@farchord:fedora.im>

17:41:12

Not legally.

<@Zlopez:matrix.org>

17:41:23

It was a joke

<@farchord:fedora.im>

17:41:28

I know 😆

<@nirik:matrix.scrye.com>

17:43:20

Things should be nice once we are moved. The new HW is nice and fast and shiny.

<@james:fedora.im>

17:43:44

Okay, unless anyone speaks up I'll close the meeting and give everyone a few extra minutes back.

<@farchord:fedora.im>

17:43:44

So you're saying the kernel will now build in 10 seconds flat? Awesome!

<@james:fedora.im>

17:44:07

He's saying the servers might boot in 10 minutes flat ;)

<@nirik:matrix.scrye.com>

17:44:20

perhaps even 8 or 9!

<@farchord:fedora.im>

17:44:25

lol

<@Zlopez:matrix.org>

17:45:16

The power will be over 9000!

<@james:fedora.im>

17:45:32

!endmeeting