2025-10-09 16:00:11 <@nirik:matrix.scrye.com> !startmeeting Infrastructure (2025-10-09)
2025-10-09 16:00:12 <@meetbot:fedora.im> Meeting started at 2025-10-09 16:00:11 UTC
2025-10-09 16:00:13 <@meetbot:fedora.im> The Meeting name is 'Infrastructure (2025-10-09)'
2025-10-09 16:00:16 <@james:fedora.im> Hey
2025-10-09 16:00:18 <@nirik:matrix.scrye.com> !info About our team: https://docs.fedoraproject.org/en-US/cle/
2025-10-09 16:00:18 <@nirik:matrix.scrye.com> !topic Hola y bienvenido
2025-10-09 16:00:18 <@nirik:matrix.scrye.com> !info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
2025-10-09 16:00:18 <@nirik:matrix.scrye.com> !info Agenda is at: https://board.net/p/fedora-infra
2025-10-09 16:00:18 <@nirik:matrix.scrye.com> !chair @nirik:matrix.scrye.com @zlopez:fedora.im @nb:fedora.im @dtometzki:fedora.im @jnsamyak:matrix.org  @james:fedora.im
2025-10-09 16:00:18 <@nirik:matrix.scrye.com> !meetingname infrastructure
2025-10-09 16:00:19 <@meetbot:fedora.im> The Meeting Name is now infrastructure
2025-10-09 16:00:30 <@nirik:matrix.scrye.com> Ahoy!
2025-10-09 16:01:58 <@nirik:matrix.scrye.com> Might be a small crowd today... several people are out/traveling...
2025-10-09 16:02:00 <@nirik:matrix.scrye.com> !info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
2025-10-09 16:02:00 <@nirik:matrix.scrye.com> !info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
2025-10-09 16:02:00 <@nirik:matrix.scrye.com> !topic New folks introductions
2025-10-09 16:02:05 <@nirik:matrix.scrye.com> anyone new around ?
2025-10-09 16:03:21 <@smooge:fedora.im> hi my name is Stephen Smoogen. I have not been system admining for 8 minutes now
2025-10-09 16:03:39 <@nirik:matrix.scrye.com> hello smooge!
2025-10-09 16:05:13 <@nirik:matrix.scrye.com> ok, then I guess lets move on
2025-10-09 16:05:16 <@nirik:matrix.scrye.com> !topic Next chair
2025-10-09 16:05:16 <@nirik:matrix.scrye.com> !info chair 2025-10-30 - ???
2025-10-09 16:05:16 <@nirik:matrix.scrye.com> !info chair 2025-10-23 - ???
2025-10-09 16:05:16 <@nirik:matrix.scrye.com> !info chair 2025-10-16 - zlopez
2025-10-09 16:05:16 <@nirik:matrix.scrye.com> !info magic eight ball says:
2025-10-09 16:05:26 <@nirik:matrix.scrye.com> anyone want to take the 23rd or 30th chair?
2025-10-09 16:05:33 <@nirik:matrix.scrye.com> or shall we just decide those next week?
2025-10-09 16:06:51 <@nirik:matrix.scrye.com> ok, next week it is.
2025-10-09 16:06:55 <@nirik:matrix.scrye.com> moving on to announcements
2025-10-09 16:07:00 <@nirik:matrix.scrye.com> !info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1900 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2025-10-09 16:07:00 <@nirik:matrix.scrye.com> !info f43 final freeze started on oct 7th
2025-10-09 16:07:00 <@nirik:matrix.scrye.com> !topic announcements and information
2025-10-09 16:07:00 <@nirik:matrix.scrye.com> !info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0815 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2025-10-09 16:07:00 <@nirik:matrix.scrye.com> !info Started working on Fedora Infra application list in hackmd https://hackmd.io/@fnF231raRruGKZbPBWKsHQ/B1F1QhRseg [zlopez]
2025-10-09 16:08:28 <@nirik:matrix.scrye.com> any other announcements?
2025-10-09 16:09:27 <@nirik:matrix.scrye.com> !topic Oncall
2025-10-09 16:09:27 <@nirik:matrix.scrye.com> !info on call from 2025-10-17 to 2025-10-23 - ???
2025-10-09 16:09:27 <@nirik:matrix.scrye.com> !info on call from 2025-10-10 to 2025-10-16 - ???
2025-10-09 16:09:27 <@nirik:matrix.scrye.com> !info on call from 2025-10-03 to 2025-10-09 - nirik
2025-10-09 16:09:27 <@nirik:matrix.scrye.com> !info https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team
2025-10-09 16:09:45 <@nirik:matrix.scrye.com> I saw no oncall pings, I still have a todo to start a discussion on what we want to do with oncall.
2025-10-09 16:09:56 <@nirik:matrix.scrye.com> I can just take this next week too and try and start that discussion.
2025-10-09 16:10:12 <@nirik:matrix.scrye.com> so, that takes us to monitoring!
2025-10-09 16:10:20 <@nirik:matrix.scrye.com> !topic Monitoring discussion [nirik]
2025-10-09 16:10:20 <@nirik:matrix.scrye.com> !info Go over existing items and fix them
2025-10-09 16:10:20 <@nirik:matrix.scrye.com> !info https://nagios.fedoraproject.org/nagios 
2025-10-09 16:11:02 <@nirik:matrix.scrye.com> basically the same as last week on the nagios side... some logdetective alerts and a cert alert this morning.
2025-10-09 16:11:35 <@nirik:matrix.scrye.com> we have been looking at some proxies alerting for disk space every day...
2025-10-09 16:11:39 <@nirik:matrix.scrye.com> it's a weird issue.
2025-10-09 16:13:11 <@nirik:matrix.scrye.com> zabbix alerts have been diminishing as we adjust thresholds.
2025-10-09 16:13:55 <@nirik:matrix.scrye.com> FYI, for anyone interested... prod zabbix alerts go to #zabbix:fedora.im and staging ones go to #noc-zabbix-testing:fedora.im
2025-10-09 16:14:10 <@nirik:matrix.scrye.com> (but you likely want to mute those rooms :)
2025-10-09 16:14:25 <@nirik:matrix.scrye.com> any monitoring questions?
2025-10-09 16:15:45 <@nirik:matrix.scrye.com> ok, given the low turnout today, I think perhaps we should head to open floor...
2025-10-09 16:15:49 <@nirik:matrix.scrye.com> !topic open floor
2025-10-09 16:16:58 <@gwmngilfen:fedora.im> !hi
2025-10-09 16:16:59 <@zodbot:fedora.im> Greg Sutcliffe (gwmngilfen) - he / him / his
2025-10-09 16:17:06 <@james:fedora.im> And then there was one more!
2025-10-09 16:17:11 <@gwmngilfen:fedora.im> sorry i'm late, family stuff. just showing my face before I go cook 😉
2025-10-09 16:18:06 <@gwmngilfen:fedora.im> staging alerts go to #fedora-zodbot:fedora.im actually, I think
2025-10-09 16:18:21 <@nirik:matrix.scrye.com> oh... right.
2025-10-09 16:18:29 <@nirik:matrix.scrye.com> we should standardize :)
2025-10-09 16:18:42 <@gwmngilfen:fedora.im> fyi I'll be doing a recording of some Zabbix show'n'tell on Monday, we'l try to get the vod out for that pronto
2025-10-09 16:18:56 <@gwmngilfen:fedora.im> we can maybe do a chat QnA afterwards?
2025-10-09 16:19:01 <@gwmngilfen:fedora.im> makes for a decent learning topic
2025-10-09 16:19:16 <@nirik:matrix.scrye.com> sure. sounds good!
2025-10-09 16:19:33 <@gwmngilfen:fedora.im> on the proxies - they *all* do the disk thing, but only some hit the alert
2025-10-09 16:20:15 <@nirik:matrix.scrye.com> yeah.
2025-10-09 16:20:19 <@gwmngilfen:fedora.im> stacked zabbix graphs ftw 😛
2025-10-09 16:20:34 <@gwmngilfen:fedora.im> i have no further insight, but I thought you might like to see that
2025-10-09 16:21:05 <@nirik:matrix.scrye.com> It's bound to be cron job triggered, but logrotate doesn't fully explain it. but we can see.
2025-10-09 16:21:09 <@james:fedora.im> Can you stack different days easily? Becuase the exact amount of data seems to change, but it's less obvious from collectd how much.
2025-10-09 16:21:19 <@gwmngilfen:fedora.im> sure
2025-10-09 16:21:24 <@james:fedora.im> Like maybe it's the same amount every monday, which would be interesting
2025-10-09 16:21:30 <@gwmngilfen:fedora.im> let me see if i can do a quick tutorial
2025-10-09 16:21:50 <@gwmngilfen:fedora.im> so head to zabbix.fpo > monitoring > latest data
2025-10-09 16:22:08 <@james:fedora.im> Wait ... I thought you were like a scottish AI assistant just giving me answers I mostly trust ;)
2025-10-09 16:22:23 <@gwmngilfen:fedora.im> hah
2025-10-09 16:22:30 <@gwmngilfen:fedora.im> then select "Proxies" as a host group in the filter
2025-10-09 16:22:48 <@gwmngilfen:fedora.im> and "FS [/]: Space: Used, in %" in the Name field (name of the item)
2025-10-09 16:23:01 <@gwmngilfen:fedora.im> if we're lucky this will work: https://zabbix.fedoraproject.org/zabbix.php?name=FS%20%5B%2F%5D%3A%20Space%3A%20Used%2C%20in%20%25&evaltype=0&tags%5B0%5D%5Btag%5D=&tags%5B0%5D%5Boperator%5D=0&tags%5B0%5D%5Bvalue%5D=&show_tags=3&tag_name_format=0&tag_priority=&state=-1&filter_name=&filter_show_counter=0&filter_custom_time=0&sort=host&sortorder=ASC&show_details=0&action=latest.view&groupids%5B%5D=38
2025-10-09 16:23:34 <@gwmngilfen:fedora.im> then select all and display graph - then you can adjust the timeframe of the graph
2025-10-09 16:23:50 <@gwmngilfen:fedora.im> fwiw it seems consistent - last 7 days:
2025-10-09 16:25:24 <@james:fedora.im> Yeh, psychedelic graphs ;)
2025-10-09 16:25:51 <@gwmngilfen:fedora.im> Zabbix is the best
2025-10-09 16:26:23 <@gwmngilfen:fedora.im> (i have no idea if /rainbow renders on other clients than Element, so that joke may be lost)
2025-10-09 16:27:10 <@james:fedora.im> Yeh, just checked and yesterday was slightly different to last Wed. 
2025-10-09 16:27:24 <@james:fedora.im> So who knows ... back to span camping, I guess.
2025-10-09 16:27:32 <@james:fedora.im> So who knows ... back to spawn camping, I guess.
2025-10-09 16:27:35 <@gwmngilfen:fedora.im> I'm sure it would be possible to create some dependant thing in Zabbix that logs what processes are running when it alerts ... but I haven't tried that yet
2025-10-09 16:28:55 <@gwmngilfen:fedora.im> otherwise, yes, spurious alerts are dropping, though there is more I think we can do there. we might well want to copy the list of nagios "services" like mail: true/false etc at some point and make use of it
2025-10-09 16:29:37 <@gwmngilfen:fedora.im> eg right now *everything* gets the postfix queue monitor, which is less useful on some hosts - but then again, still useful with the right threshold
2025-10-09 16:29:46 <@gwmngilfen:fedora.im> we will iterate ofc
2025-10-09 16:30:28 <@gwmngilfen:fedora.im> anyway! time too cook, flying visit 😛
2025-10-09 16:30:33 <@gwmngilfen:fedora.im> anyway! time to cook, flying visit 😛
2025-10-09 16:31:51 <@nirik:matrix.scrye.com> happy dinner
2025-10-09 16:32:05 <@nirik:matrix.scrye.com> looking at it logrotate takes ~30min.
2025-10-09 16:32:21 <@nirik:matrix.scrye.com> anyhow...
2025-10-09 16:32:38 <@nirik:matrix.scrye.com> if nothing else, lets end here and go back to it.
2025-10-09 16:32:59 <@james:fedora.im> Sure, sounds good.
2025-10-09 16:33:04 <@nirik:matrix.scrye.com> !endmeeting