<@zlopez:fedora.im>
17:00:23
!startmeeting Infrastructure (2025-11-06)
<@meetbot:fedora.im>
17:00:24
Meeting started at 2025-11-06 17:00:23 UTC
<@meetbot:fedora.im>
17:00:24
The Meeting name is 'Infrastructure (2025-11-06)'
<@zlopez:fedora.im>
17:00:30
!chair @nirik:matrix.scrye.com @zlopez:fedora.im @jnsamyak:matrix.org @james:fedora.im @gwmngilfen:fedora.im
<@zlopez:fedora.im>
17:00:30
!topic Hola y bienvenido
<@zlopez:fedora.im>
17:00:30
!info Agenda is at: https://board.net/p/fedora-infra
<@zlopez:fedora.im>
17:00:30
!info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
<@zlopez:fedora.im>
17:00:30
!meetingname infrastructure
<@zlopez:fedora.im>
17:00:30
!info About our team: https://docs.fedoraproject.org/en-US/cle/
<@meetbot:fedora.im>
17:00:30
The Meeting Name is now infrastructure
<@zlopez:fedora.im>
17:00:39
!hi
<@zodbot:fedora.im>
17:00:40
Michal Konecny (zlopez)
<@zlopez:fedora.im>
17:00:48
Welcome everyone π
<@nirik:matrix.scrye.com>
17:01:35
morning
<@zlopez:fedora.im>
17:02:26
Morning nirik
<@nirik:matrix.scrye.com>
17:03:22
hows things?
<@zlopez:fedora.im>
17:04:36
Finished few things today and enjoying WARHW in the meantime π
<@zlopez:fedora.im>
17:04:59
Let's see if there is somebody new today
<@zlopez:fedora.im>
17:05:01
!topic New folks introductions
<@zlopez:fedora.im>
17:05:01
!info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
<@zlopez:fedora.im>
17:05:01
!info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
<@zlopez:fedora.im>
17:08:04
It doesn't seem there is anybody
<@zlopez:fedora.im>
17:08:14
!topic Next chair
<@zlopez:fedora.im>
17:08:14
!info magic eight ball says:
<@zlopez:fedora.im>
17:08:14
!info chair 2025-11-06 - Michal KoneΔnΓ½
<@zlopez:fedora.im>
17:08:14
!info chair 2025-11-13 - ???
<@zlopez:fedora.im>
17:08:30
Who wants to chair next week?
<@nirik:matrix.scrye.com>
17:08:45
I'm not sure I want to, but I can if no one else can...
<@zlopez:fedora.im>
17:10:02
I probably can
<@zlopez:fedora.im>
17:10:49
I can probably go twice in a row
<@zlopez:fedora.im>
17:11:21
!info chair 2025-11-13 - zlopez
<@zlopez:fedora.im>
17:12:23
!topic announcements and information
<@zlopez:fedora.im>
17:12:23
!info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1900 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
<@zlopez:fedora.im>
17:12:23
!info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0815 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
<@nirik:matrix.scrye.com>
17:12:57
don't think i have anything to announce off hand
<@zlopez:fedora.im>
17:13:33
Me neither right now
<@zlopez:fedora.im>
17:13:53
!topic Oncall
<@zlopez:fedora.im>
17:13:53
!info https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team
<@zlopez:fedora.im>
17:13:53
!info on call from 2025-10-24 to 2025-11-06 - nirik
<@zlopez:fedora.im>
17:13:53
!info on call from 2025-11-07 to 2025-11-13 - ???
<@zlopez:fedora.im>
17:14:11
Is there any progress on oncall changes?
<@nirik:matrix.scrye.com>
17:15:19
nope. ;( sorry...
<@zlopez:fedora.im>
17:15:32
Ok, so let me take next week
<@zlopez:fedora.im>
17:16:10
!info on call from 2025-11-07 to 2025-11-13 - zlopez
<@zlopez:fedora.im>
17:16:18
!info Summary of last week: (from current oncall)
<@zlopez:fedora.im>
17:17:24
nirik: Did you see any ping?
<@nirik:matrix.scrye.com>
17:17:40
nope, I didn't
<@zlopez:fedora.im>
17:18:09
!topic Monitoring discussion [nirik]
<@zlopez:fedora.im>
17:18:09
!info Go over existing items and fix them
<@zlopez:fedora.im>
17:18:09
!info https://nagios.fedoraproject.org/nagios & https://zabbix.fedoraproject.org (top 100 triggers: https://zabbix.fedoraproject.org/zabbix.php?action=toptriggers.list)
<@nirik:matrix.scrye.com>
17:19:00
lets see
<@nirik:matrix.scrye.com>
17:20:07
mostly the same things as always... there's a alert on bvmhost-p10-01, because we are reconfiguring that server completely. There's some mail queue alerts for copr machines, I haven't had a chance to look at that.
<@zlopez:fedora.im>
17:20:49
I saw the bvmhost alert, but I remembered that you were doing something with it
<@nirik:matrix.scrye.com>
17:21:01
it should be set in downtime.
<@nirik:matrix.scrye.com>
17:21:29
for zabbix, there's a bunch of haproxy alerts for ipsilon02. That machine isn't working right and I can't see why... so httpd is off on it for now.
<@zlopez:fedora.im>
17:22:08
I will start working on ipsilon03, so I can check out ipsilon02 as well
<@nirik:matrix.scrye.com>
17:22:54
it configures fine, but then when it gets requests it shows a success, then a session failure.
<@zlopez:fedora.im>
17:23:23
That is strange π
<@nirik:matrix.scrye.com>
17:23:35
I tried both f42 and f43... same thing in both. (ipsilon01 works and is f42)
<@zlopez:fedora.im>
17:23:59
And the configuration is the same I assume
<@nirik:matrix.scrye.com>
17:24:19
well, in ansible at least. I wasn't able to find a difference...
<@zlopez:fedora.im>
17:25:01
I will check it out, maybe I will find something
<@nirik:matrix.scrye.com>
17:25:29
please do. Thanks!
<@zlopez:fedora.im>
17:25:37
Now let's go to next topic
<@zlopez:fedora.im>
17:25:49
!topic Fedora Infra backlog refinement
<@zlopez:fedora.im>
17:25:49
!link https://pagure.io/fedora-infrastructure/issues?status=Open&order_key=last_updated&order=asc
<@zlopez:fedora.im>
17:25:49
!info Refine oldest tickets on Fedora Infra tracker
<@zlopez:fedora.im>
17:26:12
!ticket 11833
<@zodbot:fedora.im>
17:26:15
β **Opened:** a year ago by sgallagh
<@zodbot:fedora.im>
17:26:15
β **Assignee:** jnsamyak
<@zodbot:fedora.im>
17:26:15
β **Last Updated:** 9 months ago
<@zodbot:fedora.im>
17:26:15
<@zodbot:fedora.im>
17:26:15
**fedora-infrastructure #11833** (https://pagure.io/fedora-infrastructure/issue/11833):**Add yselkowitz to list to notify when ELN builds fail**
<@james:fedora.im>
17:26:34
Need some way to test this :(
<@james:fedora.im>
17:27:22
I tried to fix it with nirik's help, but it didn't seem to work.
<@zlopez:fedora.im>
17:28:27
At least he see all the failures now, so he just needs to filter the correct mails
<@nirik:matrix.scrye.com>
17:29:35
so yeah... not sure here.
<@zlopez:fedora.im>
17:29:43
James: Do you want to update the ticket?
<@james:fedora.im>
17:30:08
Sure, I can add something
<@zlopez:fedora.im>
17:31:08
Let's go to next ticket than
<@zlopez:fedora.im>
17:31:20
!ticket 12142
<@zodbot:fedora.im>
17:31:20
β **Last Updated:** 9 months ago
<@zodbot:fedora.im>
17:31:20
**fedora-infrastructure #12142** (https://pagure.io/fedora-infrastructure/issue/12142):**Move OpenShift apps from deploymentconfig to deployment**
<@zodbot:fedora.im>
17:31:20
<@zodbot:fedora.im>
17:31:20
β **Opened:** a year ago by kevin
<@zodbot:fedora.im>
17:31:20
β **Assignee:** ryanlerch
<@zlopez:fedora.im>
17:32:10
I think this just needs more time as there is a lot of apps
<@nirik:matrix.scrye.com>
17:32:13
still needs doing for a lot of apps
<@nirik:matrix.scrye.com>
17:32:18
yeah
<@zlopez:fedora.im>
17:32:42
I can probably look at some, when I have some spare time
<@james:fedora.im>
17:33:31
Looks scarey, and it's openshift ... but there's documentation!
<@nirik:matrix.scrye.com>
17:34:15
its just moving from the openshift specifc thing to the more general k8s thing
<@zlopez:fedora.im>
17:34:24
It shouldn't be that hard, it just needs work
<@nirik:matrix.scrye.com>
17:34:25
its just moving from the old openshift specifc thing to the more general k8s thing
<@nirik:matrix.scrye.com>
17:34:48
someday deploymentconfig will finally go away
<@zlopez:fedora.im>
17:35:37
I still have release-monitoring using it, should look into that
<@zlopez:fedora.im>
17:37:01
Let's go to next one
<@zlopez:fedora.im>
17:37:40
!ticket 11683
<@zodbot:fedora.im>
17:37:41
<@zodbot:fedora.im>
17:37:41
**fedora-infrastructure #11683** (https://pagure.io/fedora-infrastructure/issue/11683):**discourse s3 backup buckets no longer active**
<@zodbot:fedora.im>
17:37:41
β **Opened:** a year ago by mattdm
<@zodbot:fedora.im>
17:37:41
β **Last Updated:** 9 months ago
<@zodbot:fedora.im>
17:37:41
β **Assignee:** Not Assigned
<@zlopez:fedora.im>
17:39:17
Anything for this one?
<@zlopez:fedora.im>
17:39:41
From what I can see it's waiting for @mattdm
<@nirik:matrix.scrye.com>
17:40:22
yeah. ;( but I wonder if it's still the case... seems odd that the normal backup wouldn't back up those things.
<@zlopez:fedora.im>
17:41:13
Maybe it's working now, but we don't know
<@zlopez:fedora.im>
17:41:19
The tickets is a year old
<@zlopez:fedora.im>
17:41:25
The ticket is a year old
<@nirik:matrix.scrye.com>
17:42:25
yeah. I guess let me ask mattdm?
<@nirik:matrix.scrye.com>
17:43:34
I sent him a dm. ;)
<@zlopez:fedora.im>
17:43:57
Do we want to look at another one?
<@nirik:matrix.scrye.com>
17:44:29
sure
<@zlopez:fedora.im>
17:44:43
!ticket 12415
<@zodbot:fedora.im>
17:44:44
**fedora-infrastructure #12415** (https://pagure.io/fedora-infrastructure/issue/12415):**Update the "Common Bugs" footer URI to F41, not F38.**
<@zodbot:fedora.im>
17:44:44
β **Assignee:** Not Assigned
<@zodbot:fedora.im>
17:44:44
β **Last Updated:** 9 months ago
<@zodbot:fedora.im>
17:44:44
β **Opened:** 9 months ago by rokejulianlockhart
<@zodbot:fedora.im>
17:44:44
<@zlopez:fedora.im>
17:44:51
Is this still relevant?
<@zlopez:fedora.im>
17:45:40
Hm, it seems that this is still there
<@zlopez:fedora.im>
17:45:51
It's something in koji or in our deployment?
<@nirik:matrix.scrye.com>
17:47:08
I think it's in our koji theme? ryanlerch might know more...
<@nirik:matrix.scrye.com>
17:47:40
https://koji.fedoraproject.org/koji/buildinfo?buildID=2738975
<@nirik:matrix.scrye.com>
17:48:42
so, if it is, someone would just have to modify that and build in infra-tags...
<@zlopez:fedora.im>
17:48:55
OK
<@zlopez:fedora.im>
17:50:34
Will probably need some digging, as I'm not sure where the theme even is hosted
<@zlopez:fedora.im>
17:50:43
Let me update the ticket
<@nirik:matrix.scrye.com>
17:50:57
I think upstream is:
<@nirik:matrix.scrye.com>
17:51:26
https://gitlab.com/fedora/websites-apps/themes/koji-theme-fedora
<@nirik:matrix.scrye.com>
17:51:49
and we also need f43/f44 builds in infra tags while we are at it
<@zlopez:fedora.im>
17:51:51
I didn't checked gitlab
<@zlopez:fedora.im>
17:52:23
I can see it there https://gitlab.com/fedora/websites-apps/themes/koji-theme-fedora/-/blob/main/templates/footer.chtml?ref_type=heads#L37
<@nirik:matrix.scrye.com>
17:52:51
yeah. probibly we should just point to some generic common bugs thing, not a specific version
<@zlopez:fedora.im>
17:53:29
I updated the ticket with the info
<@nirik:matrix.scrye.com>
17:53:35
https://discussion.fedoraproject.org/c/ask/common-issues/82/all perhaps
<@zlopez:fedora.im>
17:53:44
We just need somebody to pick it up
<@zlopez:fedora.im>
17:54:10
Let me switch to open floor as we have only few minutes left
<@zlopez:fedora.im>
17:54:19
!topic Open Floor
<@nirik:matrix.scrye.com>
17:54:31
I have one thing to mention:
<@nirik:matrix.scrye.com>
17:54:55
fedora-messaging is busted in f43+...
<@nirik:matrix.scrye.com>
17:55:14
so we should make sure NOT to deploy any applications/vm's on f43 that need to use fedora-messaging
<@nirik:matrix.scrye.com>
17:55:17
https://github.com/fedora-infra/fedora-messaging/issues/531
<@nirik:matrix.scrye.com>
17:55:49
!info do not deploy fedora-messaging applications on f43+ yet until https://github.com/fedora-infra/fedora-messaging/issues/531 is fixed.
<@james:fedora.im>
17:56:06
Huh
<@zlopez:fedora.im>
17:56:35
Should we make that a priority?
<@james:fedora.im>
17:56:48
Ahh, was looking for an F43 upgrade ticket ... but it's proxies.
<@nirik:matrix.scrye.com>
17:57:08
yeah, proxies I think are ok, no emitting or listening for messages there.
<@nirik:matrix.scrye.com>
17:57:21
ipsilon is also fine I think
<@nirik:matrix.scrye.com>
17:57:42
but not koji hubs or bodhi or ...
<@nirik:matrix.scrye.com>
17:58:25
anyhow, just wanted to note it.
<@zlopez:fedora.im>
17:58:38
We have a lot of things using fedora messaging
<@zlopez:fedora.im>
17:58:50
We should probably resolve this soon
<@nirik:matrix.scrye.com>
17:59:40
yeah, would be good.
<@james:fedora.im>
17:59:44
Do we have any idea how big the problem is?
<@zlopez:fedora.im>
18:01:18
It seems to affect twisted with python 3.14, so I assume it will affect more things than just fedora messaging
<@nirik:matrix.scrye.com>
18:01:19
not sure what you mean? it just means we can't use fedora-messaging on f43+... so a problem, but not a bad one if we don't accidentally upgrade something. ;)
<@nirik:matrix.scrye.com>
18:01:33
oh, ok. I thought it was more fedora-messaging specific.
<@zlopez:fedora.im>
18:01:49
I was just looking at the discussion in the ticket you shared
<@zlopez:fedora.im>
18:02:03
And we are already at the end of our time
<@zlopez:fedora.im>
18:02:09
Thanks everybody for coming
<@zlopez:fedora.im>
18:02:13
!endmeeting