2026-04-02 16:09:53 <@patrikp:matrix.org> !startmeeting Infrastructure (2026-04-02)
2026-04-02 16:09:55 <@meetbot:fedora.im> Meeting started at 2026-04-02 16:09:53 UTC
2026-04-02 16:09:55 <@meetbot:fedora.im> The Meeting name is 'Infrastructure (2026-04-02)'
2026-04-02 16:10:03 <@gwmngilfen:fedora.im> !hi
2026-04-02 16:10:04 <@zodbot:fedora.im> Greg Sutcliffe (gwmngilfen) - he / him / his
2026-04-02 16:10:05 <@patrikp:matrix.org> !info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
2026-04-02 16:10:05 <@patrikp:matrix.org> !info Agenda is at: https://board.net/p/fedora-infra
2026-04-02 16:10:05 <@patrikp:matrix.org> !chair @nirik:matrix.scrye.com @zlopez:fedora.im  @jnsamyak:matrix.org  @james:fedora.im @gwmngilfen:fedora.im @patrikp:matrix.org
2026-04-02 16:10:05 <@patrikp:matrix.org> !meetingname infrastructure
2026-04-02 16:10:05 <@patrikp:matrix.org> !topic Hola y bienvenido
2026-04-02 16:10:05 <@patrikp:matrix.org> !info About our team: https://docs.fedoraproject.org/en-US/cle/
2026-04-02 16:10:06 <@smoliicek:fedora.im> !hi
2026-04-02 16:10:06 <@meetbot:fedora.im> The Meeting Name is now infrastructure
2026-04-02 16:10:07 <@zodbot:fedora.im> Vít Smolík (smoliicek) - he / him / his
2026-04-02 16:10:31 <@patrikp:matrix.org> Hello and welcome.
2026-04-02 16:10:32 <@smoliicek:fedora.im> hello 👋
2026-04-02 16:10:34 <@nirik:matrix.scrye.com> morning
2026-04-02 16:10:46 <@nirik:matrix.scrye.com> sorry for meeting time confusion. I blame daylight savings time
2026-04-02 16:11:34 <@nirik:matrix.scrye.com> but we should talk about meeting time at some point here too.
2026-04-02 16:11:52 <@patrikp:matrix.org> Earlier works better for me too.
2026-04-02 16:12:15 <@james:fedora.im> So nirik mentioned moving it an hour earlier than it is now ... which would be better for me.
2026-04-02 16:12:34 <@james:fedora.im> So 15:00 UTC, to be clear.
2026-04-02 16:12:45 <@gwmngilfen:fedora.im> 15utc works for me
2026-04-02 16:12:52 <@nirik:matrix.scrye.com> Its early for me, but I can do it.
2026-04-02 16:12:58 <@smoliicek:fedora.im> the later time works better for me, but seems like you got the majority
2026-04-02 16:13:02 <@nirik:matrix.scrye.com> I think I used to have another meeting there, but don't anymore. ;)
2026-04-02 16:14:16 <@nirik:matrix.scrye.com> I can send a list/discussion post on it I guess? to catch people not here right now?
2026-04-02 16:14:21 <@nirik:matrix.scrye.com> and we can change it next week?
2026-04-02 16:14:24 <@gwmngilfen:fedora.im> +1
2026-04-02 16:14:32 <@patrikp:matrix.org> Sure.
2026-04-02 16:14:37 <@patrikp:matrix.org> OK, let's get started then.
2026-04-02 16:15:08 <@patrikp:matrix.org> !topic Next chair
2026-04-02 16:15:08 <@patrikp:matrix.org> !info chair 2026-04-16 - ???
2026-04-02 16:15:08 <@patrikp:matrix.org> !info chair 2026-04-09 - mkonecny
2026-04-02 16:15:08 <@patrikp:matrix.org> !info magic eight ball says
2026-04-02 16:15:38 <@nirik:matrix.scrye.com> !action nirik to send out mailing list / discussion on new meeting time, proposing 15UTC while dst is active.
2026-04-02 16:16:12 <@gwmngilfen:fedora.im> 16th is in the easter hols here. i can probably run the 23rd if we do change the time
2026-04-02 16:16:32 <@patrikp:matrix.org> Alright, we don't need to decide it now.
2026-04-02 16:16:40 <@patrikp:matrix.org> !topic announcements and information
2026-04-02 16:16:40 <@patrikp:matrix.org> !info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1900 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2026-04-02 16:16:50 <@patrikp:matrix.org> !info We are now in final freeze.
2026-04-02 16:17:49 <@patrikp:matrix.org> Anybody have anything to announce? Let me give a minute or two.
2026-04-02 16:18:34 <@patrikp:matrix.org> Moving on...
2026-04-02 16:18:40 <@patrikp:matrix.org> !info Go over existing items and fix them
2026-04-02 16:18:40 <@patrikp:matrix.org> !info https://nagios.fedoraproject.org/nagios & https://zabbix.fedoraproject.org (top 100 triggers: https://zabbix.fedoraproject.org/zabbix.php?action=toptriggers.list)
2026-04-02 16:18:40 <@patrikp:matrix.org> !topic Monitoring discussion [nirik / gwmngilfen]
2026-04-02 16:18:56 <@gwmngilfen:fedora.im> so firstly, 400 ,ore items gone from Nagios 🎉
2026-04-02 16:19:01 <@gwmngilfen:fedora.im> so firstly, 400 more items gone from Nagios 🎉
2026-04-02 16:19:21 <@nirik:matrix.scrye.com> Hurray!
2026-04-02 16:19:24 <@gwmngilfen:fedora.im> having deployed the ping & mgmt checks to Zabbix, they could go from Nagios. progess 🙂
2026-04-02 16:19:40 <@nirik:matrix.scrye.com> bvmhost-a64-03 is still down. I submitted a support request, but no reply yet oddly. ;(
2026-04-02 16:20:06 <@gwmngilfen:fedora.im> i've also disabled the trigger on proxy11 for now, because we're not getting anywhere with it, and we know how to replicate it. I opened a ticket so we don't forget
2026-04-02 16:20:46 <@gwmngilfen:fedora.im> the only really notable thing on the top100 is the wiki backend in haproxy - it seemed to fail entirely, but I think it was scraper load
2026-04-02 16:21:13 <@nirik:matrix.scrye.com> yeah, the wiki is more challenging to put behind anubis... because it's tied to the main fedoraproject.org site...
2026-04-02 16:21:23 <@gwmngilfen:fedora.im> the rest is db locks / haproxy 5xx errors / ping response times. all of which are mostly because I'm gathering data on sensible thresholds. they'll go soon.
2026-04-02 16:21:59 <@nirik:matrix.scrye.com> the ipsilon 5xx errors do seem to be a real problem, but not sure if anyone can ding into it and figure them out or we just wait for keycloak...
2026-04-02 16:22:17 <@nirik:matrix.scrye.com> perhaps Aurélien B would be able to look.
2026-04-02 16:22:23 <@gwmngilfen:fedora.im> i spoke to Aurelien a bit in our morning standup. not sure how far off keycloak is 😉
2026-04-02 16:23:01 <@nirik:matrix.scrye.com> yeah, that would be good to know too. Or if we could help with blockers.
2026-04-02 16:23:26 <@abompard:fedora.im> It's far. I can try to investigate the ipsilon errors if it's causing issues
2026-04-02 16:24:15 <@nirik:matrix.scrye.com> ah. bummer. Whats the blockers? or just time?
2026-04-02 16:24:31 <@nirik:matrix.scrye.com> yeah, that would be great if you can... it seems to be tracebacks talking saml2 to bugzilla. ;(
2026-04-02 16:25:08 <@smoliicek:fedora.im> i can look into stuff (as far as my access goes) if needed, not working on anything currently
2026-04-02 16:25:33 <@nirik:matrix.scrye.com> I think the main issue with the 5xx ipsilon errors is that they are so often they drown out the other ones we want to see how often/if they have a pattern.
2026-04-02 16:26:34 <@gwmngilfen:fedora.im> interestingly
2026-04-02 16:26:52 <@gwmngilfen:fedora.im> the error rate seems to have dropped this morning
2026-04-02 16:26:57 <@gwmngilfen:fedora.im> for proxy01/10
2026-04-02 16:27:43 <@gwmngilfen:fedora.im> we we're getting ~10-20/min until ~5am this morning
2026-04-02 16:27:47 <@gwmngilfen:fedora.im> we were getting ~10-20/min until ~5am this morning
2026-04-02 16:27:51 <@nirik:matrix.scrye.com> huh
2026-04-02 16:28:11 <@gwmngilfen:fedora.im> proxy10 last 7 days
2026-04-02 16:28:45 <@nirik:matrix.scrye.com> there was a python update that rolled out to fedora machines today.
2026-04-02 16:29:53 <@nirik:matrix.scrye.com> Aurélien B: do we have a ticket to track the keycloak move? should we ?
2026-04-02 16:30:31 <@smoliicek:fedora.im> we have something like that iirc
2026-04-02 16:30:48 <@abompard:fedora.im> looks like there are integrity errors when adding stuff to the `transactions` table. This table contains items that were added in 2017. I'm prettysure the oidc transaction is over now
2026-04-02 16:30:57 <@smoliicek:fedora.im> https://forge.fedoraproject.org/infra/tickets/issues/13188
2026-04-02 16:31:07 <@abompard:fedora.im> yes, thanks Vit
2026-04-02 16:31:46 <@abompard:fedora.im> I should write a script to clean up that table
2026-04-02 16:32:19 <@nirik:matrix.scrye.com> yeah, it's likely large... there might even be some script shipped with it to do that we haven't run?
2026-04-02 16:32:25 <@gwmngilfen:fedora.im> i can log a ticket for it if wanted
2026-04-02 16:32:30 <@nirik:matrix.scrye.com> but not sure why it would cause integrety errors
2026-04-02 16:33:03 <@nirik:matrix.scrye.com> ah right, thanks for that Vít Smolík. I didn't find it...
2026-04-02 16:33:48 <@nirik:matrix.scrye.com> so, ok, anything else on monitoring then?
2026-04-02 16:34:10 <@abompard:fedora.im> Maybe it happens if the page is reloaded during the transaction
2026-04-02 16:34:17 <@gwmngilfen:fedora.im> i think thats all from my side
2026-04-02 16:34:32 <@abompard:fedora.im> It's possible that the sql insert code doesn't handle this situation
2026-04-02 16:34:58 <@gwmngilfen:fedora.im> #13251 for that
2026-04-02 16:35:04 <@nirik:matrix.scrye.com> it's also possibly it's just bot hits...doing crazy things.
2026-04-02 16:35:13 <@abompard:fedora.im> besides the 500 errors, do we have user reports about this issue?
2026-04-02 16:35:46 <@nirik:matrix.scrye.com> yes.
2026-04-02 16:35:49 <@abompard:fedora.im> ah
2026-04-02 16:36:05 <@smoliicek:fedora.im> also the windows popup windows? is that still an issue?
2026-04-02 16:36:05 <@nirik:matrix.scrye.com> people often report getting weird errors with ipsilon, especially logging into bugzilla.
2026-04-02 16:36:18 <@smoliicek:fedora.im> also the windows (operating system) popup windows? is that still an issue?
2026-04-02 16:36:35 <@rahulks:fedora.im> Hi Folks, I am new here, can one share me bugzilla URL
2026-04-02 16:36:37 <@nirik:matrix.scrye.com> yes, it is.
2026-04-02 16:37:15 <@abompard:fedora.im> I can try to wrap the sql insertion code with a try/except to avoid the crash
2026-04-02 16:37:15 <@nirik:matrix.scrye.com> Rahul Singh: welcome! which url? just https://bugzilla.redhat.com ?
2026-04-02 16:37:35 <@rahulks:fedora.im> Fedora Bugzilla
2026-04-02 16:37:50 <@nirik:matrix.scrye.com> yes, it's that one. we use the redhat.com bugzilla.
2026-04-02 16:37:56 <@smoliicek:fedora.im> We use the RedHat bugzilla
2026-04-02 16:39:02 <@rahulks:fedora.im> Thanks nirik , I am able to login. Can we have small call to understand the things. I am struggling to pick issue. I am into Infrastracture & QA
2026-04-02 16:39:52 <@gwmngilfen:fedora.im> for Fedora infra tickets, we use https://forge.fedoraproject.org/infra/tickets/issues
2026-04-02 16:40:15 <@nirik:matrix.scrye.com> Rahul Singh: this is probibly not a great issue to start with. We know the problem and Aurélien B is planning to fix it. ;)
2026-04-02 16:41:26 <@gwmngilfen:fedora.im> i've got to jump. might be afk next week, easter is here. we'll see. thanks all, patrikp++ for running
2026-04-02 16:41:38 <@patrikp:matrix.org> Let's move on then?
2026-04-02 16:41:53 <@patrikp:matrix.org> !link https://forge.fedoraproject.org/infra/tickets/issues?q=&type=all&sort=leastupdate&state=open&labels=&milestone=0&project=0&assignee=0&poster=0
2026-04-02 16:41:53 <@patrikp:matrix.org> !info Refine oldest tickets on Fedora Infra tracker
2026-04-02 16:41:53 <@patrikp:matrix.org> !topic Fedora Infra backlog refinement
2026-04-02 16:42:17 <@patrikp:matrix.org> Bye bye. 👋
2026-04-02 16:42:37 <@rahulks:fedora.im> thank you for sharing this
2026-04-02 16:43:14 <@patrikp:matrix.org> !link https://forge.fedoraproject.org/infra/tickets/issues/12761
2026-04-02 16:43:30 <@nirik:matrix.scrye.com> ok, so I asked about this and got no answer.
2026-04-02 16:43:35 <@nirik:matrix.scrye.com> I will ask again?
2026-04-02 16:44:08 <@nirik:matrix.scrye.com> done. next
2026-04-02 16:44:45 <@patrikp:matrix.org> !link https://forge.fedoraproject.org/infra/tickets/issues/11884
2026-04-02 16:45:17 <@smoliicek:fedora.im> i could look into this
2026-04-02 16:45:18 <@patrikp:matrix.org> Should we do stuff with the labels, maybe?
2026-04-02 16:45:27 <@nirik:matrix.scrye.com> phone call
2026-04-02 16:45:33 <@patrikp:matrix.org> They still use gain/trouble.
2026-04-02 16:46:56 <@nirik:matrix.scrye.com> yes, we should clean out the old labels and add new
2026-04-02 16:47:05 <@nirik:matrix.scrye.com> so... on this one...
2026-04-02 16:47:05 <@smoliicek:fedora.im> ill try to do that in the next sprint
2026-04-02 16:47:21 <@nirik:matrix.scrye.com> we have a way to do this.
2026-04-02 16:47:37 <@nirik:matrix.scrye.com> we just need to document it as the way to do this and make sure all our apps do it.
2026-04-02 16:48:00 <@nirik:matrix.scrye.com> it's done with ansible tags.
2026-04-02 16:49:05 <@nirik:matrix.scrye.com> look at the bodhi playbook for example. It has 'rollout' and 'build' tags. So, running the playbook and passing -t will do that...
2026-04-02 16:49:17 <@smoliicek:fedora.im> so write an SOP for this?
2026-04-02 16:49:30 <@nirik:matrix.scrye.com> well, or put it in our openshift guide:
2026-04-02 16:49:44 <@nirik:matrix.scrye.com> https://docs.fedoraproject.org/en-US/infra/developer_guide/openshift/#_openshift
2026-04-02 16:50:04 <@nirik:matrix.scrye.com> and I know a few apps use this pattern, but many don't.
2026-04-02 16:50:10 <@nirik:matrix.scrye.com> so we should make them all use it. ;)
2026-04-02 16:50:26 <@nirik:matrix.scrye.com> at least I think this will work for the need... well, we may need to add more tags?
2026-04-02 16:50:52 <@smoliicek:fedora.im> something like a redeploy tag could be useful?
2026-04-02 16:51:08 <@nirik:matrix.scrye.com> yeah, 'restart deployment'
2026-04-02 16:51:34 <@nirik:matrix.scrye.com> should I add this info to the ticket? or would someone else like to?
2026-04-02 16:51:39 <@Zlopez:matrix.org> It seems the DST caught up with me, I thought the meeting is starting in 10 minutes 😃
2026-04-02 16:51:55 <@smoliicek:fedora.im> should we create a new ticket to track where it is/isn't? (talking about the tags)
2026-04-02 16:52:16 <@nirik:matrix.scrye.com> Zlopez: yeah, we were all confused. ;(
2026-04-02 16:52:36 <@nirik:matrix.scrye.com> Vít Smolík: sure.
2026-04-02 16:52:59 <@rahulks:fedora.im> is there any meeting today ?
2026-04-02 16:53:04 <@nirik:matrix.scrye.com> pull requests for these might be a good thing for Rahul Singh to look at helping with? :)
2026-04-02 16:53:16 <@nirik:matrix.scrye.com> Bheda Rahul: you are in one right now.
2026-04-02 16:53:17 <@smoliicek:fedora.im> you are currently participating in an infrastructure meeting :)
2026-04-02 16:53:50 <@rahulks:fedora.im> is there any meeting link ?
2026-04-02 16:54:11 <@smoliicek:fedora.im> we use chat for meetings mostly
2026-04-02 16:54:11 <@smoliicek:fedora.im> this is the meeting
2026-04-02 16:54:24 <@rahulks:fedora.im> I understood
2026-04-02 16:54:24 <@patrikp:matrix.org> It's a text meeting.
2026-04-02 16:55:09 <@smoliicek:fedora.im> will you do it? or should I?
2026-04-02 16:55:18 <@rahulks:fedora.im> This group is for Infrastracture, I understood now. Thank you for helping. I will check all issues.
2026-04-02 16:55:18 <@rahulks:fedora.im> If you can give me any issues also ok for me.
2026-04-02 16:55:35 <@nirik:matrix.scrye.com> Vít Smolík: actually, lets just use the existing ticket.
2026-04-02 16:56:06 <@nirik:matrix.scrye.com> I added a note there. next ticket? well, we don't have much time left...
2026-04-02 16:56:37 <@patrikp:matrix.org> Let's do a couple of minutes of open floor then I'll stop it.
2026-04-02 16:56:43 <@patrikp:matrix.org> !topic Open Floor
2026-04-02 16:57:17 <@nirik:matrix.scrye.com> anyone have any questions/comments/ideas? :)
2026-04-02 16:57:58 <@rahulks:fedora.im> I am able to see the issues but where can i validate it to investigate and perform some tests
2026-04-02 16:58:22 <@smoliicek:fedora.im> i added the matrix outage to statusfpo, and will send out the email about it too
2026-04-02 16:59:41 <@nirik:matrix.scrye.com> rahulks: you can feel free to ask about issues in #admin:fedoraproject.org or #noc:fedoraproject.org and when someone can they will answer...
2026-04-02 16:59:48 <@nirik:matrix.scrye.com> Vít Smolík: thanks.
2026-04-02 17:00:08 <@patrikp:matrix.org> And we're at time
2026-04-02 17:00:14 <@patrikp:matrix.org> !info Thank you all for coming!
2026-04-02 17:00:14 <@patrikp:matrix.org> !endmeeting