2025-11-06 17:00:23 <@zlopez:fedora.im> !startmeeting Infrastructure (2025-11-06)
2025-11-06 17:00:24 <@meetbot:fedora.im> Meeting started at 2025-11-06 17:00:23 UTC
2025-11-06 17:00:24 <@meetbot:fedora.im> The Meeting name is 'Infrastructure (2025-11-06)'
2025-11-06 17:00:30 <@zlopez:fedora.im> !chair @nirik:matrix.scrye.com @zlopez:fedora.im  @jnsamyak:matrix.org  @james:fedora.im @gwmngilfen:fedora.im
2025-11-06 17:00:30 <@zlopez:fedora.im> !topic Hola y bienvenido
2025-11-06 17:00:30 <@zlopez:fedora.im> !info Agenda is at: https://board.net/p/fedora-infra
2025-11-06 17:00:30 <@zlopez:fedora.im> !info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
2025-11-06 17:00:30 <@zlopez:fedora.im> !meetingname infrastructure
2025-11-06 17:00:30 <@zlopez:fedora.im> !info About our team: https://docs.fedoraproject.org/en-US/cle/
2025-11-06 17:00:30 <@meetbot:fedora.im> The Meeting Name is now infrastructure
2025-11-06 17:00:39 <@zlopez:fedora.im> !hi
2025-11-06 17:00:40 <@zodbot:fedora.im> Michal Konecny (zlopez)
2025-11-06 17:00:48 <@zlopez:fedora.im> Welcome everyone 🙂
2025-11-06 17:01:35 <@nirik:matrix.scrye.com> morning
2025-11-06 17:02:26 <@zlopez:fedora.im> Morning nirik 
2025-11-06 17:03:22 <@nirik:matrix.scrye.com> hows things?
2025-11-06 17:04:36 <@zlopez:fedora.im> Finished few things today and enjoying WARHW in the meantime 🙂
2025-11-06 17:04:59 <@zlopez:fedora.im> Let's see if there is somebody new today
2025-11-06 17:05:01 <@zlopez:fedora.im> !topic New folks introductions
2025-11-06 17:05:01 <@zlopez:fedora.im> !info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
2025-11-06 17:05:01 <@zlopez:fedora.im> !info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
2025-11-06 17:08:04 <@zlopez:fedora.im> It doesn't seem there is anybody
2025-11-06 17:08:14 <@zlopez:fedora.im> !topic Next chair
2025-11-06 17:08:14 <@zlopez:fedora.im> !info magic eight ball says:
2025-11-06 17:08:14 <@zlopez:fedora.im> !info chair 2025-11-06 - Michal Konečný
2025-11-06 17:08:14 <@zlopez:fedora.im> !info chair 2025-11-13 - ???
2025-11-06 17:08:30 <@zlopez:fedora.im> Who wants to chair next week?
2025-11-06 17:08:45 <@nirik:matrix.scrye.com> I'm not sure I want to, but I can if no one else can...
2025-11-06 17:10:02 <@zlopez:fedora.im> I probably can
2025-11-06 17:10:49 <@zlopez:fedora.im> I can probably go twice in a row
2025-11-06 17:11:21 <@zlopez:fedora.im> !info chair 2025-11-13 - zlopez
2025-11-06 17:12:23 <@zlopez:fedora.im> !topic announcements and information
2025-11-06 17:12:23 <@zlopez:fedora.im> !info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1900 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2025-11-06 17:12:23 <@zlopez:fedora.im> !info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0815 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2025-11-06 17:12:57 <@nirik:matrix.scrye.com> don't think i have anything to announce off hand
2025-11-06 17:13:33 <@zlopez:fedora.im> Me neither right now
2025-11-06 17:13:53 <@zlopez:fedora.im> !topic Oncall
2025-11-06 17:13:53 <@zlopez:fedora.im> !info https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team
2025-11-06 17:13:53 <@zlopez:fedora.im> !info on call from 2025-10-24 to 2025-11-06 - nirik
2025-11-06 17:13:53 <@zlopez:fedora.im> !info on call from 2025-11-07 to 2025-11-13 - ???
2025-11-06 17:14:11 <@zlopez:fedora.im> Is there any progress on oncall changes?
2025-11-06 17:15:19 <@nirik:matrix.scrye.com> nope. ;( sorry...
2025-11-06 17:15:32 <@zlopez:fedora.im> Ok, so let me take next week
2025-11-06 17:16:10 <@zlopez:fedora.im> !info on call from 2025-11-07 to 2025-11-13 - zlopez
2025-11-06 17:16:18 <@zlopez:fedora.im> !info Summary of last week: (from current oncall)
2025-11-06 17:17:24 <@zlopez:fedora.im> nirik: Did you see any ping?
2025-11-06 17:17:40 <@nirik:matrix.scrye.com> nope, I didn't
2025-11-06 17:18:09 <@zlopez:fedora.im> !topic Monitoring discussion [nirik]
2025-11-06 17:18:09 <@zlopez:fedora.im> !info Go over existing items and fix them
2025-11-06 17:18:09 <@zlopez:fedora.im> !info https://nagios.fedoraproject.org/nagios & https://zabbix.fedoraproject.org (top 100 triggers: https://zabbix.fedoraproject.org/zabbix.php?action=toptriggers.list)
2025-11-06 17:19:00 <@nirik:matrix.scrye.com> lets see
2025-11-06 17:20:07 <@nirik:matrix.scrye.com> mostly the same things as always... there's a alert on bvmhost-p10-01, because we are reconfiguring that server completely. There's some mail queue alerts for copr machines, I haven't had a chance to look at that.
2025-11-06 17:20:49 <@zlopez:fedora.im> I saw the bvmhost alert, but I remembered that you were doing something with it
2025-11-06 17:21:01 <@nirik:matrix.scrye.com> it should be set in downtime.
2025-11-06 17:21:29 <@nirik:matrix.scrye.com> for zabbix, there's a bunch of haproxy alerts for ipsilon02. That machine isn't working right and I can't see why... so httpd is off on it for now.
2025-11-06 17:22:08 <@zlopez:fedora.im> I will start working on ipsilon03, so I can check out ipsilon02 as well
2025-11-06 17:22:54 <@nirik:matrix.scrye.com> it configures fine, but then when it gets requests it shows a success, then a session failure.
2025-11-06 17:23:23 <@zlopez:fedora.im> That is strange 😕
2025-11-06 17:23:35 <@nirik:matrix.scrye.com> I tried both f42 and f43... same thing in both. (ipsilon01 works and is f42)
2025-11-06 17:23:59 <@zlopez:fedora.im> And the configuration is the same I assume
2025-11-06 17:24:19 <@nirik:matrix.scrye.com> well, in ansible at least. I wasn't able to find a difference...
2025-11-06 17:25:01 <@zlopez:fedora.im> I will check it out, maybe I will find something
2025-11-06 17:25:29 <@nirik:matrix.scrye.com> please do. Thanks!
2025-11-06 17:25:37 <@zlopez:fedora.im> Now let's go to next topic
2025-11-06 17:25:49 <@zlopez:fedora.im> !topic Fedora Infra backlog refinement
2025-11-06 17:25:49 <@zlopez:fedora.im> !link https://pagure.io/fedora-infrastructure/issues?status=Open&order_key=last_updated&order=asc
2025-11-06 17:25:49 <@zlopez:fedora.im> !info Refine oldest tickets on Fedora Infra tracker
2025-11-06 17:26:12 <@zlopez:fedora.im> !ticket 11833
2025-11-06 17:26:15 <@zodbot:fedora.im> ● **Opened:** a year ago by sgallagh
2025-11-06 17:26:15 <@zodbot:fedora.im> ● **Assignee:** jnsamyak
2025-11-06 17:26:15 <@zodbot:fedora.im> ● **Last Updated:** 9 months ago
2025-11-06 17:26:15 <@zodbot:fedora.im> 
2025-11-06 17:26:15 <@zodbot:fedora.im> **fedora-infrastructure #11833** (https://pagure.io/fedora-infrastructure/issue/11833):**Add yselkowitz to list to notify when ELN builds fail**
2025-11-06 17:26:34 <@james:fedora.im> Need some way to test this :(
2025-11-06 17:27:22 <@james:fedora.im> I tried to fix it with nirik's help, but it didn't seem to work.
2025-11-06 17:28:27 <@zlopez:fedora.im> At least he see all the failures now, so he just needs to filter the correct mails 
2025-11-06 17:29:35 <@nirik:matrix.scrye.com> so yeah... not sure here.
2025-11-06 17:29:43 <@zlopez:fedora.im> James: Do you want to update the ticket?
2025-11-06 17:30:08 <@james:fedora.im> Sure, I can add something
2025-11-06 17:31:08 <@zlopez:fedora.im> Let's go to next ticket than
2025-11-06 17:31:20 <@zlopez:fedora.im> !ticket 12142
2025-11-06 17:31:20 <@zodbot:fedora.im> ● **Last Updated:** 9 months ago
2025-11-06 17:31:20 <@zodbot:fedora.im> **fedora-infrastructure #12142** (https://pagure.io/fedora-infrastructure/issue/12142):**Move OpenShift apps from deploymentconfig to deployment**
2025-11-06 17:31:20 <@zodbot:fedora.im> 
2025-11-06 17:31:20 <@zodbot:fedora.im> ● **Opened:** a year ago by kevin
2025-11-06 17:31:20 <@zodbot:fedora.im> ● **Assignee:** ryanlerch
2025-11-06 17:32:10 <@zlopez:fedora.im> I think this just needs more time as there is a lot of apps
2025-11-06 17:32:13 <@nirik:matrix.scrye.com> still needs doing for a lot of apps
2025-11-06 17:32:18 <@nirik:matrix.scrye.com> yeah
2025-11-06 17:32:42 <@zlopez:fedora.im> I can probably look at some, when I have some spare time
2025-11-06 17:33:31 <@james:fedora.im> Looks scarey, and it's openshift ... but there's documentation!
2025-11-06 17:34:15 <@nirik:matrix.scrye.com> its just moving from the openshift specifc thing to the more general k8s thing
2025-11-06 17:34:24 <@zlopez:fedora.im> It shouldn't be that hard, it just needs work
2025-11-06 17:34:25 <@nirik:matrix.scrye.com> its just moving from the old openshift specifc thing to the more general k8s thing
2025-11-06 17:34:48 <@nirik:matrix.scrye.com> someday deploymentconfig will finally go away
2025-11-06 17:35:37 <@zlopez:fedora.im> I still have release-monitoring using it, should look into that
2025-11-06 17:37:01 <@zlopez:fedora.im> Let's go to next one
2025-11-06 17:37:40 <@zlopez:fedora.im> !ticket 11683
2025-11-06 17:37:41 <@zodbot:fedora.im> 
2025-11-06 17:37:41 <@zodbot:fedora.im> **fedora-infrastructure #11683** (https://pagure.io/fedora-infrastructure/issue/11683):**discourse s3 backup buckets no longer active**
2025-11-06 17:37:41 <@zodbot:fedora.im> ● **Opened:** a year ago by mattdm
2025-11-06 17:37:41 <@zodbot:fedora.im> ● **Last Updated:** 9 months ago
2025-11-06 17:37:41 <@zodbot:fedora.im> ● **Assignee:** Not Assigned
2025-11-06 17:39:17 <@zlopez:fedora.im> Anything for this one?
2025-11-06 17:39:41 <@zlopez:fedora.im> From what I can see it's waiting for @mattdm
2025-11-06 17:40:22 <@nirik:matrix.scrye.com> yeah. ;( but I wonder if it's still the case... seems odd that the normal backup wouldn't back up those things.
2025-11-06 17:41:13 <@zlopez:fedora.im> Maybe it's working now, but we don't know
2025-11-06 17:41:19 <@zlopez:fedora.im> The tickets is a year old
2025-11-06 17:41:25 <@zlopez:fedora.im> The ticket is a year old
2025-11-06 17:42:25 <@nirik:matrix.scrye.com> yeah. I guess let me ask mattdm?
2025-11-06 17:43:34 <@nirik:matrix.scrye.com> I sent him a dm. ;)
2025-11-06 17:43:57 <@zlopez:fedora.im> Do we want to look at another one?
2025-11-06 17:44:29 <@nirik:matrix.scrye.com> sure
2025-11-06 17:44:43 <@zlopez:fedora.im> !ticket 12415
2025-11-06 17:44:44 <@zodbot:fedora.im> **fedora-infrastructure #12415** (https://pagure.io/fedora-infrastructure/issue/12415):**Update the "Common Bugs" footer URI to F41, not F38.**
2025-11-06 17:44:44 <@zodbot:fedora.im> ● **Assignee:** Not Assigned
2025-11-06 17:44:44 <@zodbot:fedora.im> ● **Last Updated:** 9 months ago
2025-11-06 17:44:44 <@zodbot:fedora.im> ● **Opened:** 9 months ago by rokejulianlockhart
2025-11-06 17:44:44 <@zodbot:fedora.im> 
2025-11-06 17:44:51 <@zlopez:fedora.im> Is this still relevant?
2025-11-06 17:45:40 <@zlopez:fedora.im> Hm, it seems that this is still there
2025-11-06 17:45:51 <@zlopez:fedora.im> It's something in koji or in our deployment?
2025-11-06 17:47:08 <@nirik:matrix.scrye.com> I think it's in our koji theme? ryanlerch might know more...
2025-11-06 17:47:40 <@nirik:matrix.scrye.com> https://koji.fedoraproject.org/koji/buildinfo?buildID=2738975
2025-11-06 17:48:42 <@nirik:matrix.scrye.com> so, if it is, someone would just have to modify that and build in infra-tags...
2025-11-06 17:48:55 <@zlopez:fedora.im> OK
2025-11-06 17:50:34 <@zlopez:fedora.im> Will probably need some digging, as I'm not sure where the theme even is hosted
2025-11-06 17:50:43 <@zlopez:fedora.im> Let me update the ticket
2025-11-06 17:50:57 <@nirik:matrix.scrye.com> I think upstream is:
2025-11-06 17:51:26 <@nirik:matrix.scrye.com> https://gitlab.com/fedora/websites-apps/themes/koji-theme-fedora
2025-11-06 17:51:49 <@nirik:matrix.scrye.com> and we also need f43/f44 builds in infra tags while we are at it
2025-11-06 17:51:51 <@zlopez:fedora.im> I didn't checked gitlab
2025-11-06 17:52:23 <@zlopez:fedora.im> I can see it there https://gitlab.com/fedora/websites-apps/themes/koji-theme-fedora/-/blob/main/templates/footer.chtml?ref_type=heads#L37
2025-11-06 17:52:51 <@nirik:matrix.scrye.com> yeah. probibly we should just point to some generic common bugs thing, not a specific version
2025-11-06 17:53:29 <@zlopez:fedora.im> I updated the ticket with the info
2025-11-06 17:53:35 <@nirik:matrix.scrye.com> https://discussion.fedoraproject.org/c/ask/common-issues/82/all perhaps
2025-11-06 17:53:44 <@zlopez:fedora.im> We just need somebody to pick it up
2025-11-06 17:54:10 <@zlopez:fedora.im> Let me switch to open floor as we have only few minutes left
2025-11-06 17:54:19 <@zlopez:fedora.im> !topic Open Floor
2025-11-06 17:54:31 <@nirik:matrix.scrye.com> I have one thing to mention:
2025-11-06 17:54:55 <@nirik:matrix.scrye.com> fedora-messaging is busted in f43+...
2025-11-06 17:55:14 <@nirik:matrix.scrye.com> so we should make sure NOT to deploy any applications/vm's on f43 that need to use fedora-messaging
2025-11-06 17:55:17 <@nirik:matrix.scrye.com> https://github.com/fedora-infra/fedora-messaging/issues/531
2025-11-06 17:55:49 <@nirik:matrix.scrye.com> !info do not deploy fedora-messaging applications on f43+ yet until https://github.com/fedora-infra/fedora-messaging/issues/531 is fixed.
2025-11-06 17:56:06 <@james:fedora.im> Huh
2025-11-06 17:56:35 <@zlopez:fedora.im> Should we make that a priority?
2025-11-06 17:56:48 <@james:fedora.im> Ahh, was looking for an F43 upgrade ticket ... but it's proxies.
2025-11-06 17:57:08 <@nirik:matrix.scrye.com> yeah, proxies I think are ok, no emitting or listening for messages there.
2025-11-06 17:57:21 <@nirik:matrix.scrye.com> ipsilon is also fine I think
2025-11-06 17:57:42 <@nirik:matrix.scrye.com> but not koji hubs or bodhi or ...
2025-11-06 17:58:25 <@nirik:matrix.scrye.com> anyhow, just wanted to note it.
2025-11-06 17:58:38 <@zlopez:fedora.im> We have a lot of things using fedora messaging
2025-11-06 17:58:50 <@zlopez:fedora.im> We should probably resolve this soon
2025-11-06 17:59:40 <@nirik:matrix.scrye.com> yeah, would be good.
2025-11-06 17:59:44 <@james:fedora.im> Do we have any idea how big the problem is?
2025-11-06 18:01:18 <@zlopez:fedora.im> It seems to affect twisted with python 3.14, so I assume it will affect more things than just fedora messaging
2025-11-06 18:01:19 <@nirik:matrix.scrye.com> not sure what you mean? it just means we can't use fedora-messaging on f43+... so a problem, but not a bad one if we don't accidentally upgrade something. ;)
2025-11-06 18:01:33 <@nirik:matrix.scrye.com> oh, ok. I thought it was more fedora-messaging specific.
2025-11-06 18:01:49 <@zlopez:fedora.im> I was just looking at the discussion in the ticket you shared
2025-11-06 18:02:03 <@zlopez:fedora.im> And we are already at the end of our time
2025-11-06 18:02:09 <@zlopez:fedora.im> Thanks everybody for coming
2025-11-06 18:02:13 <@zlopez:fedora.im> !endmeeting