<@patrikp:matrix.org>
16:09:53
!startmeeting Infrastructure (2026-04-02)
<@meetbot:fedora.im>
16:09:55
Meeting started at 2026-04-02 16:09:53 UTC
<@meetbot:fedora.im>
16:09:55
The Meeting name is 'Infrastructure (2026-04-02)'
<@gwmngilfen:fedora.im>
16:10:03
!hi
<@zodbot:fedora.im>
16:10:04
Greg Sutcliffe (gwmngilfen) - he / him / his
<@patrikp:matrix.org>
16:10:05
!info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
<@patrikp:matrix.org>
16:10:05
!info Agenda is at: https://board.net/p/fedora-infra
<@patrikp:matrix.org>
16:10:05
!chair @nirik:matrix.scrye.com @zlopez:fedora.im @jnsamyak:matrix.org @james:fedora.im @gwmngilfen:fedora.im @patrikp:matrix.org
<@patrikp:matrix.org>
16:10:05
!meetingname infrastructure
<@patrikp:matrix.org>
16:10:05
!topic Hola y bienvenido
<@patrikp:matrix.org>
16:10:05
!info About our team: https://docs.fedoraproject.org/en-US/cle/
<@smoliicek:fedora.im>
16:10:06
!hi
<@meetbot:fedora.im>
16:10:06
The Meeting Name is now infrastructure
<@zodbot:fedora.im>
16:10:07
Vít Smolík (smoliicek) - he / him / his
<@patrikp:matrix.org>
16:10:31
Hello and welcome.
<@smoliicek:fedora.im>
16:10:32
hello 👋
<@nirik:matrix.scrye.com>
16:10:34
morning
<@nirik:matrix.scrye.com>
16:10:46
sorry for meeting time confusion. I blame daylight savings time
<@nirik:matrix.scrye.com>
16:11:34
but we should talk about meeting time at some point here too.
<@patrikp:matrix.org>
16:11:52
Earlier works better for me too.
<@james:fedora.im>
16:12:15
So nirik mentioned moving it an hour earlier than it is now ... which would be better for me.
<@james:fedora.im>
16:12:34
So 15:00 UTC, to be clear.
<@gwmngilfen:fedora.im>
16:12:45
15utc works for me
<@nirik:matrix.scrye.com>
16:12:52
Its early for me, but I can do it.
<@smoliicek:fedora.im>
16:12:58
the later time works better for me, but seems like you got the majority
<@nirik:matrix.scrye.com>
16:13:02
I think I used to have another meeting there, but don't anymore. ;)
<@nirik:matrix.scrye.com>
16:14:16
I can send a list/discussion post on it I guess? to catch people not here right now?
<@nirik:matrix.scrye.com>
16:14:21
and we can change it next week?
<@gwmngilfen:fedora.im>
16:14:24
+1
<@patrikp:matrix.org>
16:14:32
Sure.
<@patrikp:matrix.org>
16:14:37
OK, let's get started then.
<@patrikp:matrix.org>
16:15:08
!topic Next chair
<@patrikp:matrix.org>
16:15:08
!info chair 2026-04-16 - ???
<@patrikp:matrix.org>
16:15:08
!info chair 2026-04-09 - mkonecny
<@patrikp:matrix.org>
16:15:08
!info magic eight ball says
<@nirik:matrix.scrye.com>
16:15:38
!action nirik to send out mailing list / discussion on new meeting time, proposing 15UTC while dst is active.
<@gwmngilfen:fedora.im>
16:16:12
16th is in the easter hols here. i can probably run the 23rd if we do change the time
<@patrikp:matrix.org>
16:16:32
Alright, we don't need to decide it now.
<@patrikp:matrix.org>
16:16:40
!topic announcements and information
<@patrikp:matrix.org>
16:16:40
!info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1900 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
<@patrikp:matrix.org>
16:16:50
!info We are now in final freeze.
<@patrikp:matrix.org>
16:17:49
Anybody have anything to announce? Let me give a minute or two.
<@patrikp:matrix.org>
16:18:34
Moving on...
<@patrikp:matrix.org>
16:18:40
!info Go over existing items and fix them
<@patrikp:matrix.org>
16:18:40
!info https://nagios.fedoraproject.org/nagios & https://zabbix.fedoraproject.org (top 100 triggers: https://zabbix.fedoraproject.org/zabbix.php?action=toptriggers.list)
<@patrikp:matrix.org>
16:18:40
!topic Monitoring discussion [nirik / gwmngilfen]
<@gwmngilfen:fedora.im>
16:18:56
so firstly, 400 ,ore items gone from Nagios 🎉
<@gwmngilfen:fedora.im>
16:19:01
so firstly, 400 more items gone from Nagios 🎉
<@nirik:matrix.scrye.com>
16:19:21
Hurray!
<@gwmngilfen:fedora.im>
16:19:24
having deployed the ping & mgmt checks to Zabbix, they could go from Nagios. progess 🙂
<@nirik:matrix.scrye.com>
16:19:40
bvmhost-a64-03 is still down. I submitted a support request, but no reply yet oddly. ;(
<@gwmngilfen:fedora.im>
16:20:06
i've also disabled the trigger on proxy11 for now, because we're not getting anywhere with it, and we know how to replicate it. I opened a ticket so we don't forget
<@gwmngilfen:fedora.im>
16:20:46
the only really notable thing on the top100 is the wiki backend in haproxy - it seemed to fail entirely, but I think it was scraper load
<@nirik:matrix.scrye.com>
16:21:13
yeah, the wiki is more challenging to put behind anubis... because it's tied to the main fedoraproject.org site...
<@gwmngilfen:fedora.im>
16:21:23
the rest is db locks / haproxy 5xx errors / ping response times. all of which are mostly because I'm gathering data on sensible thresholds. they'll go soon.
<@nirik:matrix.scrye.com>
16:21:59
the ipsilon 5xx errors do seem to be a real problem, but not sure if anyone can ding into it and figure them out or we just wait for keycloak...
<@nirik:matrix.scrye.com>
16:22:17
perhaps Aurélien B would be able to look.
<@gwmngilfen:fedora.im>
16:22:23
i spoke to Aurelien a bit in our morning standup. not sure how far off keycloak is 😉
<@nirik:matrix.scrye.com>
16:23:01
yeah, that would be good to know too. Or if we could help with blockers.
<@abompard:fedora.im>
16:23:26
It's far. I can try to investigate the ipsilon errors if it's causing issues
<@nirik:matrix.scrye.com>
16:24:15
ah. bummer. Whats the blockers? or just time?
<@nirik:matrix.scrye.com>
16:24:31
yeah, that would be great if you can... it seems to be tracebacks talking saml2 to bugzilla. ;(
<@smoliicek:fedora.im>
16:25:08
i can look into stuff (as far as my access goes) if needed, not working on anything currently
<@nirik:matrix.scrye.com>
16:25:33
I think the main issue with the 5xx ipsilon errors is that they are so often they drown out the other ones we want to see how often/if they have a pattern.
<@gwmngilfen:fedora.im>
16:26:34
interestingly
<@gwmngilfen:fedora.im>
16:26:52
the error rate seems to have dropped this morning
<@gwmngilfen:fedora.im>
16:26:57
for proxy01/10
<@gwmngilfen:fedora.im>
16:27:43
we we're getting ~10-20/min until ~5am this morning
<@gwmngilfen:fedora.im>
16:27:47
we were getting ~10-20/min until ~5am this morning
<@nirik:matrix.scrye.com>
16:27:51
huh
<@gwmngilfen:fedora.im>
16:28:11
proxy10 last 7 days
<@nirik:matrix.scrye.com>
16:28:45
there was a python update that rolled out to fedora machines today.
<@nirik:matrix.scrye.com>
16:29:53
Aurélien B: do we have a ticket to track the keycloak move? should we ?
<@smoliicek:fedora.im>
16:30:31
we have something like that iirc
<@abompard:fedora.im>
16:30:48
looks like there are integrity errors when adding stuff to the `transactions` table. This table contains items that were added in 2017. I'm prettysure the oidc transaction is over now
<@smoliicek:fedora.im>
16:30:57
https://forge.fedoraproject.org/infra/tickets/issues/13188
<@abompard:fedora.im>
16:31:07
yes, thanks Vit
<@abompard:fedora.im>
16:31:46
I should write a script to clean up that table
<@nirik:matrix.scrye.com>
16:32:19
yeah, it's likely large... there might even be some script shipped with it to do that we haven't run?
<@gwmngilfen:fedora.im>
16:32:25
i can log a ticket for it if wanted
<@nirik:matrix.scrye.com>
16:32:30
but not sure why it would cause integrety errors
<@nirik:matrix.scrye.com>
16:33:03
ah right, thanks for that Vít Smolík. I didn't find it...
<@nirik:matrix.scrye.com>
16:33:48
so, ok, anything else on monitoring then?
<@abompard:fedora.im>
16:34:10
Maybe it happens if the page is reloaded during the transaction
<@gwmngilfen:fedora.im>
16:34:17
i think thats all from my side
<@abompard:fedora.im>
16:34:32
It's possible that the sql insert code doesn't handle this situation
<@gwmngilfen:fedora.im>
16:34:58
#13251 for that
<@nirik:matrix.scrye.com>
16:35:04
it's also possibly it's just bot hits...doing crazy things.
<@abompard:fedora.im>
16:35:13
besides the 500 errors, do we have user reports about this issue?
<@nirik:matrix.scrye.com>
16:35:46
yes.
<@abompard:fedora.im>
16:35:49
ah
<@smoliicek:fedora.im>
16:36:05
also the windows popup windows? is that still an issue?
<@nirik:matrix.scrye.com>
16:36:05
people often report getting weird errors with ipsilon, especially logging into bugzilla.
<@smoliicek:fedora.im>
16:36:18
also the windows (operating system) popup windows? is that still an issue?
<@rahulks:fedora.im>
16:36:35
Hi Folks, I am new here, can one share me bugzilla URL
<@nirik:matrix.scrye.com>
16:36:37
yes, it is.
<@abompard:fedora.im>
16:37:15
I can try to wrap the sql insertion code with a try/except to avoid the crash
<@nirik:matrix.scrye.com>
16:37:15
Rahul Singh: welcome! which url? just https://bugzilla.redhat.com ?
<@rahulks:fedora.im>
16:37:35
Fedora Bugzilla
<@nirik:matrix.scrye.com>
16:37:50
yes, it's that one. we use the redhat.com bugzilla.
<@smoliicek:fedora.im>
16:37:56
We use the RedHat bugzilla
<@rahulks:fedora.im>
16:39:02
Thanks nirik , I am able to login. Can we have small call to understand the things. I am struggling to pick issue. I am into Infrastracture & QA
<@gwmngilfen:fedora.im>
16:39:52
for Fedora infra tickets, we use https://forge.fedoraproject.org/infra/tickets/issues
<@nirik:matrix.scrye.com>
16:40:15
Rahul Singh: this is probibly not a great issue to start with. We know the problem and Aurélien B is planning to fix it. ;)
<@gwmngilfen:fedora.im>
16:41:26
i've got to jump. might be afk next week, easter is here. we'll see. thanks all, patrikp++ for running
<@patrikp:matrix.org>
16:41:38
Let's move on then?
<@patrikp:matrix.org>
16:41:53
!link https://forge.fedoraproject.org/infra/tickets/issues?q=&type=all&sort=leastupdate&state=open&labels=&milestone=0&project=0&assignee=0&poster=0
<@patrikp:matrix.org>
16:41:53
!info Refine oldest tickets on Fedora Infra tracker
<@patrikp:matrix.org>
16:41:53
!topic Fedora Infra backlog refinement
<@patrikp:matrix.org>
16:42:17
Bye bye. 👋
<@rahulks:fedora.im>
16:42:37
thank you for sharing this
<@patrikp:matrix.org>
16:43:14
!link https://forge.fedoraproject.org/infra/tickets/issues/12761
<@nirik:matrix.scrye.com>
16:43:30
ok, so I asked about this and got no answer.
<@nirik:matrix.scrye.com>
16:43:35
I will ask again?
<@nirik:matrix.scrye.com>
16:44:08
done. next
<@patrikp:matrix.org>
16:44:45
!link https://forge.fedoraproject.org/infra/tickets/issues/11884
<@smoliicek:fedora.im>
16:45:17
i could look into this
<@patrikp:matrix.org>
16:45:18
Should we do stuff with the labels, maybe?
<@nirik:matrix.scrye.com>
16:45:27
phone call
<@patrikp:matrix.org>
16:45:33
They still use gain/trouble.
<@nirik:matrix.scrye.com>
16:46:56
yes, we should clean out the old labels and add new
<@nirik:matrix.scrye.com>
16:47:05
so... on this one...
<@smoliicek:fedora.im>
16:47:05
ill try to do that in the next sprint
<@nirik:matrix.scrye.com>
16:47:21
we have a way to do this.
<@nirik:matrix.scrye.com>
16:47:37
we just need to document it as the way to do this and make sure all our apps do it.
<@nirik:matrix.scrye.com>
16:48:00
it's done with ansible tags.
<@nirik:matrix.scrye.com>
16:49:05
look at the bodhi playbook for example. It has 'rollout' and 'build' tags. So, running the playbook and passing -t will do that...
<@smoliicek:fedora.im>
16:49:17
so write an SOP for this?
<@nirik:matrix.scrye.com>
16:49:30
well, or put it in our openshift guide:
<@nirik:matrix.scrye.com>
16:49:44
https://docs.fedoraproject.org/en-US/infra/developer_guide/openshift/#_openshift
<@nirik:matrix.scrye.com>
16:50:04
and I know a few apps use this pattern, but many don't.
<@nirik:matrix.scrye.com>
16:50:10
so we should make them all use it. ;)
<@nirik:matrix.scrye.com>
16:50:26
at least I think this will work for the need... well, we may need to add more tags?
<@smoliicek:fedora.im>
16:50:52
something like a redeploy tag could be useful?
<@nirik:matrix.scrye.com>
16:51:08
yeah, 'restart deployment'
<@nirik:matrix.scrye.com>
16:51:34
should I add this info to the ticket? or would someone else like to?
<@Zlopez:matrix.org>
16:51:39
It seems the DST caught up with me, I thought the meeting is starting in 10 minutes 😃
<@smoliicek:fedora.im>
16:51:55
should we create a new ticket to track where it is/isn't? (talking about the tags)
<@nirik:matrix.scrye.com>
16:52:16
Zlopez: yeah, we were all confused. ;(
<@nirik:matrix.scrye.com>
16:52:36
Vít Smolík: sure.
<@rahulks:fedora.im>
16:52:59
is there any meeting today ?
<@nirik:matrix.scrye.com>
16:53:04
pull requests for these might be a good thing for Rahul Singh to look at helping with? :)
<@nirik:matrix.scrye.com>
16:53:16
Bheda Rahul: you are in one right now.
<@smoliicek:fedora.im>
16:53:17
you are currently participating in an infrastructure meeting :)
<@rahulks:fedora.im>
16:53:50
is there any meeting link ?
<@smoliicek:fedora.im>
16:54:11
we use chat for meetings mostly
<@smoliicek:fedora.im>
16:54:11
this is the meeting
<@rahulks:fedora.im>
16:54:24
I understood
<@patrikp:matrix.org>
16:54:24
It's a text meeting.
<@smoliicek:fedora.im>
16:55:09
will you do it? or should I?
<@rahulks:fedora.im>
16:55:18
This group is for Infrastracture, I understood now. Thank you for helping. I will check all issues.
<@rahulks:fedora.im>
16:55:18
If you can give me any issues also ok for me.
<@nirik:matrix.scrye.com>
16:55:35
Vít Smolík: actually, lets just use the existing ticket.
<@nirik:matrix.scrye.com>
16:56:06
I added a note there. next ticket? well, we don't have much time left...
<@patrikp:matrix.org>
16:56:37
Let's do a couple of minutes of open floor then I'll stop it.
<@patrikp:matrix.org>
16:56:43
!topic Open Floor
<@nirik:matrix.scrye.com>
16:57:17
anyone have any questions/comments/ideas? :)
<@rahulks:fedora.im>
16:57:58
I am able to see the issues but where can i validate it to investigate and perform some tests
<@smoliicek:fedora.im>
16:58:22
i added the matrix outage to statusfpo, and will send out the email about it too
<@nirik:matrix.scrye.com>
16:59:41
rahulks: you can feel free to ask about issues in #admin:fedoraproject.org or #noc:fedoraproject.org and when someone can they will answer...
<@nirik:matrix.scrye.com>
16:59:48
Vít Smolík: thanks.
<@patrikp:matrix.org>
17:00:08
And we're at time
<@patrikp:matrix.org>
17:00:14
!info Thank you all for coming!
<@patrikp:matrix.org>
17:00:14
!endmeeting