2025-09-25 16:00:14 <@zlopez:fedora.im> !startmeeting Infrastructure (2025-09-25)
2025-09-25 16:00:15 <@meetbot:fedora.im> Meeting started at 2025-09-25 16:00:14 UTC
2025-09-25 16:00:15 <@meetbot:fedora.im> The Meeting name is 'Infrastructure (2025-09-25)'
2025-09-25 16:00:31 <@zlopez:fedora.im> !topic Hola y bienvenido
2025-09-25 16:00:31 <@zlopez:fedora.im> !meetingname infrastructure
2025-09-25 16:00:31 <@zlopez:fedora.im> !info Agenda is at: https://board.net/p/fedora-infra
2025-09-25 16:00:31 <@zlopez:fedora.im> !chair @nirik:matrix.scrye.com @zlopez:fedora.im @nb:fedora.im @dtometzki:fedora.im @jnsamyak:matrix.org  @james:fedora.im
2025-09-25 16:00:31 <@zlopez:fedora.im> !info About our team: https://docs.fedoraproject.org/en-US/cle/
2025-09-25 16:00:31 <@zlopez:fedora.im> !info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
2025-09-25 16:00:32 <@meetbot:fedora.im> The Meeting Name is now infrastructure
2025-09-25 16:00:43 <@zlopez:fedora.im> Hi everyone
2025-09-25 16:01:08 <@nirik:matrix.scrye.com> morning
2025-09-25 16:01:12 <@zlopez:fedora.im> I'm your host zlopez and this is Jacka... No, it's infrastructure weekly meeting
2025-09-25 16:03:56 <@nirik:matrix.scrye.com> lots of people out on pto I guess... or just busy
2025-09-25 16:04:11 <@zlopez:fedora.im> It's possible
2025-09-25 16:04:24 <@zlopez:fedora.im> I started clearing the chairs as well
2025-09-25 16:04:51 <@zlopez:fedora.im> Waiting for response from some folks
2025-09-25 16:05:36 <@zlopez:fedora.im> Let's continue with it
2025-09-25 16:05:46 <@zlopez:fedora.im> !info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
2025-09-25 16:05:46 <@zlopez:fedora.im> !info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
2025-09-25 16:05:46 <@zlopez:fedora.im> !topic New folks introductions
2025-09-25 16:05:53 <@zlopez:fedora.im> Is anybody new around?
2025-09-25 16:07:35 <@zlopez:fedora.im> It doesn't seem like we have
2025-09-25 16:07:52 <@zlopez:fedora.im> !info magic eight ball says:
2025-09-25 16:07:52 <@zlopez:fedora.im> !info chair 2025-10-02 - nirik
2025-09-25 16:07:52 <@zlopez:fedora.im> !info chair 2025-09-25 - zlopez
2025-09-25 16:07:52 <@zlopez:fedora.im> !topic Next chair
2025-09-25 16:07:52 <@zlopez:fedora.im> !info chair 2025-10-09 - ???
2025-09-25 16:07:52 <@zlopez:fedora.im> !info chair 2025-10-16 - ???
2025-09-25 16:07:52 <@zlopez:fedora.im> !info chair 2025-10-23 - ???
2025-09-25 16:07:52 <@zlopez:fedora.im> !info chair 2025-10-30 - ???
2025-09-25 16:08:05 <@zlopez:fedora.im> So we have a chair for next week
2025-09-25 16:08:34 <@nirik:matrix.scrye.com> yep. I can do it then I think
2025-09-25 16:08:44 <@zlopez:fedora.im> And I'm not here for the next one after, so do we want to leave it empty for now
2025-09-25 16:08:50 <@nirik:matrix.scrye.com> sure
2025-09-25 16:09:22 <@zlopez:fedora.im> In that case let's continue 
2025-09-25 16:09:24 <@zlopez:fedora.im> !info Started working on Fedora Infra application list in hackmd https://hackmd.io/@fnF231raRruGKZbPBWKsHQ/B1F1QhRseg [zlopez]
2025-09-25 16:09:24 <@zlopez:fedora.im> !info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0815 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2025-09-25 16:09:24 <@zlopez:fedora.im> topic announcements and information
2025-09-25 16:09:24 <@zlopez:fedora.im> !info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1900 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2025-09-25 16:09:45 <@zlopez:fedora.im> Oh, forgot ! in topic
2025-09-25 16:09:55 <@zlopez:fedora.im> !info https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team
2025-09-25 16:09:55 <@zlopez:fedora.im> !topic Oncall
2025-09-25 16:09:55 <@zlopez:fedora.im> !info on call from 2025-10-03 to 2025-10-09 - ???
2025-09-25 16:09:55 <@zlopez:fedora.im> !info on call from 2025-10-10 to 2025-10-16 - ???
2025-09-25 16:09:55 <@zlopez:fedora.im> !info on call from 2025-09-26 to 2025-10-02 - zlopez
2025-09-25 16:09:55 <@zlopez:fedora.im> !info on call from 2025-09-19 to 2025-09-25 - nirik
2025-09-25 16:10:36 <@zlopez:fedora.im> I will get to announcements again, I accidentally skipped
2025-09-25 16:10:49 <@nirik:matrix.scrye.com> no biggie
2025-09-25 16:10:53 <@zlopez:fedora.im> I'm probably tired today
2025-09-25 16:11:45 <@zlopez:fedora.im> !oncall
2025-09-25 16:11:46 <@zodbot:fedora.im> The following people are oncall:
2025-09-25 16:11:46 <@zodbot:fedora.im> ● @Zlopez:matrix.org (zlopez) Current Time for them: 18:11 (Europe/Prague)
2025-09-25 16:11:46 <@zodbot:fedora.im> If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)
2025-09-25 16:11:46 <@zodbot:fedora.im> 
2025-09-25 16:11:46 <@zodbot:fedora.im> ● @gwmngilfen:matrix.org (gwmngilfen) Current Time for them: 17:11 (Europe/London)
2025-09-25 16:12:06 <@nirik:matrix.scrye.com> huh.
2025-09-25 16:12:12 <@nirik:matrix.scrye.com> I could have sworn I changed it.
2025-09-25 16:12:19 <@zlopez:fedora.im> !oncall
2025-09-25 16:12:20 <@zodbot:fedora.im> The following people are oncall:
2025-09-25 16:12:20 <@zodbot:fedora.im> 
2025-09-25 16:12:20 <@zodbot:fedora.im> ● @Zlopez:matrix.org (zlopez) Current Time for them: 18:12 (Europe/Prague)
2025-09-25 16:12:20 <@zodbot:fedora.im> If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)
2025-09-25 16:12:44 <@zlopez:fedora.im> Set up, but it seems that Gwmngilfen had oncall this week
2025-09-25 16:12:56 <@gwmngilfen:fedora.im> i did?
2025-09-25 16:13:10 <@gwmngilfen:fedora.im> well, no pings anyway 🙂
2025-09-25 16:13:10 <@zlopez:fedora.im> It looks like that
2025-09-25 16:13:23 <@nirik:matrix.scrye.com> I was supposed to, but somehow I didn't set it... ;(
2025-09-25 16:13:42 <@gwmngilfen:fedora.im> no worries
2025-09-25 16:13:42 <@zlopez:fedora.im> Looking at the history, last change happened 11th September
2025-09-25 16:13:48 <@nirik:matrix.scrye.com> but, I wonder... is oncall really doing much for us? I mean, it seems like a nice idea, but it doesn't get used much anymore...
2025-09-25 16:14:14 <@zlopez:fedora.im> It's true that I didn't saw much oncall pings in the last few weeks
2025-09-25 16:14:23 <@gwmngilfen:fedora.im> i wonder if it would be easier to just add "mods" or "admins" or even "oncall" to our matrix notification wordlist
2025-09-25 16:14:26 <@nirik:matrix.scrye.com> it was used more on irc for some reason... not so much on matrix
2025-09-25 16:14:34 <@gwmngilfen:fedora.im> then if we're around we'll see it
2025-09-25 16:15:02 <@nirik:matrix.scrye.com> well, the idea was that the oncall person could take those inturrupts and everyone else wouldn't have to pay attention/be bothered by it.
2025-09-25 16:15:50 <@nirik:matrix.scrye.com> but given how little it's used, perhaps we should just change it to say 'please describe your issue in #admin:fedoraproject.org and if no response, file a ticket'
2025-09-25 16:16:12 <@gwmngilfen:fedora.im> seems reasonable
2025-09-25 16:16:15 <@zlopez:fedora.im> Maybe that seems like a good way forward
2025-09-25 16:16:16 <@nirik:matrix.scrye.com> (and we would need to update docs for that)
2025-09-25 16:16:28 <@nirik:matrix.scrye.com> (and the bot config)
2025-09-25 16:16:54 <@zlopez:fedora.im> Some default message for the bot
2025-09-25 16:17:16 <@nirik:matrix.scrye.com> or allow it to set a room as oncall instead of a user?
2025-09-25 16:17:17 <@gwmngilfen:fedora.im> one less meeting item is no bad thing either
2025-09-25 16:17:41 <@nirik:matrix.scrye.com> yeah. I can propose it on list/discussion and see if anyone objects?
2025-09-25 16:17:50 <@gwmngilfen:fedora.im> 👍️
2025-09-25 16:18:11 <@zlopez:fedora.im> That seems like a good idea
2025-09-25 16:18:30 <@nirik:matrix.scrye.com> added to todo. ✅
2025-09-25 16:18:43 <@zlopez:fedora.im> So let's contineu
2025-09-25 16:18:55 <@zlopez:fedora.im> !info Summary of last week: (from current oncall)
2025-09-25 16:19:02 <@zlopez:fedora.im> I didn't saw anything
2025-09-25 16:20:03 <@nirik:matrix.scrye.com> me either :)
2025-09-25 16:20:21 <@gwmngilfen:fedora.im> 🙈
2025-09-25 16:20:28 <@zlopez:fedora.im> Ok, so back to announcements
2025-09-25 16:20:29 <@zlopez:fedora.im> !info Started working on Fedora Infra application list in hackmd https://hackmd.io/@fnF231raRruGKZbPBWKsHQ/B1F1QhRseg [zlopez]
2025-09-25 16:20:29 <@zlopez:fedora.im> !topic announcements and information
2025-09-25 16:20:29 <@zlopez:fedora.im> !info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0815 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2025-09-25 16:20:29 <@zlopez:fedora.im> !info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1900 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2025-09-25 16:20:44 <@zlopez:fedora.im> Anything else to announce?
2025-09-25 16:21:02 <@nirik:matrix.scrye.com> !info anubis is deployed on many of our services, report any issues you see with it.
2025-09-25 16:21:16 <@nirik:matrix.scrye.com> !info mass update/reboot cycle next week
2025-09-25 16:21:28 <@gwmngilfen:fedora.im> i think zabbix will be on that list "soon" - but not this week
2025-09-25 16:21:33 <@nirik:matrix.scrye.com> !info f43 final freeze starting on oct 7th
2025-09-25 16:23:01 <@nirik:matrix.scrye.com> On zabbix... I wonder if the matrix alerts could be threaded (per host?) but thats a RFE not anything big
2025-09-25 16:23:31 <@gwmngilfen:fedora.im> i can investigate but we're using an upstream media type, so ...
2025-09-25 16:23:48 <@gwmngilfen:fedora.im> can always ask / fork, I guess
2025-09-25 16:23:58 <@nirik:matrix.scrye.com> no biggie, just a thought
2025-09-25 16:24:13 <@gwmngilfen:fedora.im> its a good one
2025-09-25 16:25:20 <@gwmngilfen:fedora.im> i cant stay today (will have to go in 5min or so) but if we can schedule ahead I can definitel do a learning sesion on Zabbix one of these weeks
2025-09-25 16:25:41 <@zlopez:fedora.im> Let's continue than
2025-09-25 16:25:42 <@zlopez:fedora.im> !info Go over existing items and fix them
2025-09-25 16:25:42 <@zlopez:fedora.im> !info https://nagios.fedoraproject.org/nagios 
2025-09-25 16:25:42 <@zlopez:fedora.im> !topic Monitoring discussion [nirik]
2025-09-25 16:25:57 <@zlopez:fedora.im> I will add that to upcoming learning topics
2025-09-25 16:26:11 <@gwmngilfen:fedora.im> can we make this section Nagios *and* Zabbix? 😛
2025-09-25 16:26:20 <@gwmngilfen:fedora.im> (maybe thats too soon)
2025-09-25 16:26:48 <@zlopez:fedora.im> You should be able to edit agenda as well 🙂
2025-09-25 16:27:01 <@nirik:matrix.scrye.com> so, Gwmngilfen fixed nagios earlier this week... we were not monitoring any of the cloud stuff. ;(
2025-09-25 16:27:10 <@nirik:matrix.scrye.com> and then I fixed up security groups in aws and firewalls.
2025-09-25 16:27:31 <@nirik:matrix.scrye.com> so, we have a few new things:
2025-09-25 16:27:33 <@zlopez:fedora.im> That was a lot of alerts to acknowledge 🙂
2025-09-25 16:27:53 <@nirik:matrix.scrye.com> yeah. ;(
2025-09-25 16:28:09 <@nirik:matrix.scrye.com> bvmhost-p10-01.mgmt.rdu3.fedoraproject.org still is alerting on http I know we fixed it once, but it did not seem to take
2025-09-25 16:28:47 <@nirik:matrix.scrye.com> datanommer01.fedorainfracloud.org is alerting, but I can't seem login to it. :( and actually, perhaps we should just nuke it and the RDS database in favor of the postrest plan?
2025-09-25 16:29:12 <@nirik:matrix.scrye.com> logdetective01.fedorainfracloud.org is alerting and there's a ticket for that team to fix up their firewall
2025-09-25 16:29:36 <@nirik:matrix.scrye.com> certgetter01.rdu3.fedoraproject.org is alerting on http (thats a long standing one we should fix)
2025-09-25 16:29:59 <@nirik:matrix.scrye.com> oci-registry01.rdu3.fedoraproject.org is still alerting on disk space. I guess I should add a few more TB to it. ;(
2025-09-25 16:30:32 <@nirik:matrix.scrye.com> proxy01.rdu3.fedoraproject.org is alerting on the fedoraproject wildcard cert. I plan to renew that this week. (along with several other certs we are not monitoing)
2025-09-25 16:31:21 <@nirik:matrix.scrye.com> Theres a rabbitmq queue alert that I thought I fixed, will make sure it's actually fixed.
2025-09-25 16:31:48 <@zlopez:fedora.im> That is the UNKNOWN one?
2025-09-25 16:32:03 <@nirik:matrix.scrye.com> yeah, that queue was removed...
2025-09-25 16:32:09 <@nirik:matrix.scrye.com> so we shouldn't be monitoring it anymore
2025-09-25 16:32:21 <@gwmngilfen:fedora.im> Zabbix side:
2025-09-25 16:32:21 <@gwmngilfen:fedora.im> - I pushed a bunch of threshold updates to quieten down some of the noisy hosts
2025-09-25 16:32:21 <@gwmngilfen:fedora.im> - i've fixed up a bunch of openvpn certs today that have restored agent connectivity for vmhost-x86-cc03/5/6 and ibiblio02
2025-09-25 16:32:54 <@gwmngilfen:fedora.im> theres some disk space issues in there too which need consideration (or new thresholds)
2025-09-25 16:33:37 <@gwmngilfen:fedora.im> log01 and ipa01 are both warning about / being over 80% full
2025-09-25 16:33:59 <@nirik:matrix.scrye.com> I guess nagios has a higher threshold there?
2025-09-25 16:34:09 <@gwmngilfen:fedora.im> probably. i can look and mirror it
2025-09-25 16:34:29 <@nirik:matrix.scrye.com> ipa01 is probibly something causing undue logs.
2025-09-25 16:34:34 <@nirik:matrix.scrye.com> log01 ditto. ;)
2025-09-25 16:34:42 <@gwmngilfen:fedora.im> these are new-ish alerts so something must be using disk
2025-09-25 16:35:35 <@nirik:matrix.scrye.com> log01 was almost full due to a crashlooping poddler... the compressed gigantic logs probibly are still taking up a lot of room
2025-09-25 16:35:48 <@nirik:matrix.scrye.com> it was around 100GB or so of logs
2025-09-25 16:35:53 <@gwmngilfen:fedora.im> ah, fair enough, let me ack that for now then
2025-09-25 16:35:53 <@nirik:matrix.scrye.com> (uncompressed)
2025-09-25 16:36:01 <@zlopez:fedora.im> That is fixed now
2025-09-25 16:36:25 <@gwmngilfen:fedora.im> oh an people01 is high-cpu again, looks like cgit is stuck again
2025-09-25 16:36:28 <@gwmngilfen:fedora.im> oh and people01 is high-cpu again, looks like cgit is stuck again
2025-09-25 16:36:35 <@nirik:matrix.scrye.com> yeah, but the old logs (now rotated and compressed) might be taking up space... will have to look
2025-09-25 16:37:15 <@gwmngilfen:fedora.im> staging Zabbix looks fine though 😛 ok, gotta run, I'll be on mobile if you need my input
2025-09-25 16:37:37 <@zlopez:fedora.im> On ipa01 I can see 4.1 GB on httpd logs
2025-09-25 16:37:59 <@zlopez:fedora.im> There are plenty of uncompressed ones
2025-09-25 16:38:40 <@nirik:matrix.scrye.com> anyhow, I think thats it for monitoring... we can file tickets/solve the issues out of meeting
2025-09-25 16:39:07 <@zlopez:fedora.im> Ok, what do we want to do next?
2025-09-25 16:39:27 <@nirik:matrix.scrye.com> I'd say backlog, but it's only 2 of us, so perhaps we just end early.
2025-09-25 16:40:12 <@zlopez:fedora.im> I'm +1 for ending early
2025-09-25 16:40:36 <@zlopez:fedora.im> !topic Open Floor
2025-09-25 16:41:06 <@zlopez:fedora.im> Let's switch to open floor for few minutes and I will end the meeting after that
2025-09-25 16:41:22 <@zlopez:fedora.im> So anybody who wants to say something have 5 minutes for now
2025-09-25 16:42:35 <@nirik:matrix.scrye.com> I'd tell a UDP joke here, but I am not sure you all would get it?
2025-09-25 16:47:13 <@zlopez:fedora.im> Thanks everyone for coming today
2025-09-25 16:47:16 <@zlopez:fedora.im> !endmeeting