#meeting-3:fedoraproject.org log

<@nirik:matrix.scrye.com>

16:00:11

!startmeeting Infrastructure (2025-10-09)

<@meetbot:fedora.im>

16:00:12

Meeting started at 2025-10-09 16:00:11 UTC

<@meetbot:fedora.im>

16:00:13

The Meeting name is 'Infrastructure (2025-10-09)'

<@james:fedora.im>

16:00:16

Hey

<@nirik:matrix.scrye.com>

16:00:18

!info About our team: https://docs.fedoraproject.org/en-US/cle/

<@nirik:matrix.scrye.com>

16:00:18

!topic Hola y bienvenido

<@nirik:matrix.scrye.com>

16:00:18

!info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra

<@nirik:matrix.scrye.com>

16:00:18

!info Agenda is at: https://board.net/p/fedora-infra

<@nirik:matrix.scrye.com>

16:00:18

!chair @nirik:matrix.scrye.com @zlopez:fedora.im @nb:fedora.im @dtometzki:fedora.im @jnsamyak:matrix.org @james:fedora.im

<@nirik:matrix.scrye.com>

16:00:18

!meetingname infrastructure

<@meetbot:fedora.im>

16:00:19

The Meeting Name is now infrastructure

<@nirik:matrix.scrye.com>

16:00:30

Ahoy!

<@nirik:matrix.scrye.com>

16:01:58

Might be a small crowd today... several people are out/traveling...

<@nirik:matrix.scrye.com>

16:02:00

!info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/

<@nirik:matrix.scrye.com>

16:02:00

!info This is a place where people who are interested in Fedora Infrastructure can introduce themselves

<@nirik:matrix.scrye.com>

16:02:00

!topic New folks introductions

<@nirik:matrix.scrye.com>

16:02:05

anyone new around ?

<@smooge:fedora.im>

16:03:21

hi my name is Stephen Smoogen. I have not been system admining for 8 minutes now

<@nirik:matrix.scrye.com>

16:03:39

hello smooge!

<@nirik:matrix.scrye.com>

16:05:13

ok, then I guess lets move on

<@nirik:matrix.scrye.com>

16:05:16

!topic Next chair

<@nirik:matrix.scrye.com>

16:05:16

!info chair 2025-10-30 - ???

<@nirik:matrix.scrye.com>

16:05:16

!info chair 2025-10-23 - ???

<@nirik:matrix.scrye.com>

16:05:16

!info chair 2025-10-16 - zlopez

<@nirik:matrix.scrye.com>

16:05:16

!info magic eight ball says:

<@nirik:matrix.scrye.com>

16:05:26

anyone want to take the 23rd or 30th chair?

<@nirik:matrix.scrye.com>

16:05:33

or shall we just decide those next week?

<@nirik:matrix.scrye.com>

16:06:51

ok, next week it is.

<@nirik:matrix.scrye.com>

16:06:55

moving on to announcements

<@nirik:matrix.scrye.com>

16:07:00

!info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1900 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org

<@nirik:matrix.scrye.com>

16:07:00

!info f43 final freeze started on oct 7th

<@nirik:matrix.scrye.com>

16:07:00

!topic announcements and information

<@nirik:matrix.scrye.com>

16:07:00

!info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0815 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org

<@nirik:matrix.scrye.com>

16:07:00

!info Started working on Fedora Infra application list in hackmd https://hackmd.io/@fnF231raRruGKZbPBWKsHQ/B1F1QhRseg [zlopez]

<@nirik:matrix.scrye.com>

16:08:28

any other announcements?

<@nirik:matrix.scrye.com>

16:09:27

!topic Oncall

<@nirik:matrix.scrye.com>

16:09:27

!info on call from 2025-10-17 to 2025-10-23 - ???

<@nirik:matrix.scrye.com>

16:09:27

!info on call from 2025-10-10 to 2025-10-16 - ???

<@nirik:matrix.scrye.com>

16:09:27

!info on call from 2025-10-03 to 2025-10-09 - nirik

<@nirik:matrix.scrye.com>

16:09:27

!info https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team

<@nirik:matrix.scrye.com>

16:09:45

I saw no oncall pings, I still have a todo to start a discussion on what we want to do with oncall.

<@nirik:matrix.scrye.com>

16:09:56

I can just take this next week too and try and start that discussion.

<@nirik:matrix.scrye.com>

16:10:12

so, that takes us to monitoring!

<@nirik:matrix.scrye.com>

16:10:20

!topic Monitoring discussion [nirik]

<@nirik:matrix.scrye.com>

16:10:20

!info Go over existing items and fix them

<@nirik:matrix.scrye.com>

16:10:20

!info https://nagios.fedoraproject.org/nagios

<@nirik:matrix.scrye.com>

16:11:02

basically the same as last week on the nagios side... some logdetective alerts and a cert alert this morning.

<@nirik:matrix.scrye.com>

16:11:35

we have been looking at some proxies alerting for disk space every day...

<@nirik:matrix.scrye.com>

16:11:39

it's a weird issue.

<@nirik:matrix.scrye.com>

16:13:11

zabbix alerts have been diminishing as we adjust thresholds.

<@nirik:matrix.scrye.com>

16:13:55

FYI, for anyone interested... prod zabbix alerts go to #zabbix:fedora.im and staging ones go to #noc-zabbix-testing:fedora.im

<@nirik:matrix.scrye.com>

16:14:10

(but you likely want to mute those rooms :)

<@nirik:matrix.scrye.com>

16:14:25

any monitoring questions?

<@nirik:matrix.scrye.com>

16:15:45

ok, given the low turnout today, I think perhaps we should head to open floor...

<@nirik:matrix.scrye.com>

16:15:49

!topic open floor

<@gwmngilfen:fedora.im>

16:16:58

!hi

<@zodbot:fedora.im>

16:16:59

Greg Sutcliffe (gwmngilfen) - he / him / his

<@james:fedora.im>

16:17:06

And then there was one more!

<@gwmngilfen:fedora.im>

16:17:11

sorry i'm late, family stuff. just showing my face before I go cook 😉

<@gwmngilfen:fedora.im>

16:18:06

staging alerts go to #fedora-zodbot:fedora.im actually, I think

<@nirik:matrix.scrye.com>

16:18:21

oh... right.

<@nirik:matrix.scrye.com>

16:18:29

we should standardize :)

<@gwmngilfen:fedora.im>

16:18:42

fyi I'll be doing a recording of some Zabbix show'n'tell on Monday, we'l try to get the vod out for that pronto

<@gwmngilfen:fedora.im>

16:18:56

we can maybe do a chat QnA afterwards?

<@gwmngilfen:fedora.im>

16:19:01

makes for a decent learning topic

<@nirik:matrix.scrye.com>

16:19:16

sure. sounds good!

<@gwmngilfen:fedora.im>

16:19:33

on the proxies - they *all* do the disk thing, but only some hit the alert

<@nirik:matrix.scrye.com>

16:20:15

yeah.

<@gwmngilfen:fedora.im>

16:20:19

stacked zabbix graphs ftw 😛

<@gwmngilfen:fedora.im>

16:20:34

i have no further insight, but I thought you might like to see that

<@nirik:matrix.scrye.com>

16:21:05

It's bound to be cron job triggered, but logrotate doesn't fully explain it. but we can see.

<@james:fedora.im>

16:21:09

Can you stack different days easily? Becuase the exact amount of data seems to change, but it's less obvious from collectd how much.

<@gwmngilfen:fedora.im>

16:21:19

sure

<@james:fedora.im>

16:21:24

Like maybe it's the same amount every monday, which would be interesting

<@gwmngilfen:fedora.im>

16:21:30

let me see if i can do a quick tutorial

<@gwmngilfen:fedora.im>

16:21:50

so head to zabbix.fpo > monitoring > latest data

<@james:fedora.im>

16:22:08

Wait ... I thought you were like a scottish AI assistant just giving me answers I mostly trust ;)

<@gwmngilfen:fedora.im>

16:22:23

hah

<@gwmngilfen:fedora.im>

16:22:30

then select "Proxies" as a host group in the filter

<@gwmngilfen:fedora.im>

16:22:48

and "FS [/]: Space: Used, in %" in the Name field (name of the item)

<@gwmngilfen:fedora.im>

16:23:01

if we're lucky this will work: https://zabbix.fedoraproject.org/zabbix.php?name=FS%20%5B%2F%5D%3A%20Space%3A%20Used%2C%20in%20%25&evaltype=0&tags%5B0%5D%5Btag%5D=&tags%5B0%5D%5Boperator%5D=0&tags%5B0%5D%5Bvalue%5D=&show_tags=3&tag_name_format=0&tag_priority=&state=-1&filter_name=&filter_show_counter=0&filter_custom_time=0&sort=host&sortorder=ASC&show_details=0&action=latest.view&groupids%5B%5D=38

<@gwmngilfen:fedora.im>

16:23:34

then select all and display graph - then you can adjust the timeframe of the graph

<@gwmngilfen:fedora.im>

16:23:50

fwiw it seems consistent - last 7 days:

<@james:fedora.im>

16:25:24

Yeh, psychedelic graphs ;)

<@gwmngilfen:fedora.im>

16:25:51

Zabbix is the best

<@gwmngilfen:fedora.im>

16:26:23

(i have no idea if /rainbow renders on other clients than Element, so that joke may be lost)

<@james:fedora.im>

16:27:10

Yeh, just checked and yesterday was slightly different to last Wed.

<@james:fedora.im>

16:27:24

So who knows ... back to span camping, I guess.

<@james:fedora.im>

16:27:32

So who knows ... back to spawn camping, I guess.

<@gwmngilfen:fedora.im>

16:27:35

I'm sure it would be possible to create some dependant thing in Zabbix that logs what processes are running when it alerts ... but I haven't tried that yet

<@gwmngilfen:fedora.im>

16:28:55

otherwise, yes, spurious alerts are dropping, though there is more I think we can do there. we might well want to copy the list of nagios "services" like mail: true/false etc at some point and make use of it

<@gwmngilfen:fedora.im>

16:29:37

eg right now *everything* gets the postfix queue monitor, which is less useful on some hosts - but then again, still useful with the right threshold

<@gwmngilfen:fedora.im>

16:29:46

we will iterate ofc

<@gwmngilfen:fedora.im>

16:30:28

anyway! time too cook, flying visit 😛

<@gwmngilfen:fedora.im>

16:30:33

anyway! time to cook, flying visit 😛

<@nirik:matrix.scrye.com>

16:31:51

happy dinner

<@nirik:matrix.scrye.com>

16:32:05

looking at it logrotate takes ~30min.

<@nirik:matrix.scrye.com>

16:32:21

anyhow...

<@nirik:matrix.scrye.com>

16:32:38

if nothing else, lets end here and go back to it.

<@james:fedora.im>

16:32:59

Sure, sounds good.

<@nirik:matrix.scrye.com>

16:33:04

!endmeeting