16:00:05 <jednorozec> #startmeeting Infrastructure (2023-11-02)
16:00:05 <zodbot> Meeting started Thu Nov  2 16:00:05 2023 UTC.
16:00:05 <zodbot> This meeting is logged and archived in a public location.
16:00:05 <zodbot> The chair is jednorozec. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:00:05 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:05 <zodbot> The meeting name has been set to 'infrastructure_(2023-11-02)'
16:00:06 <jednorozec> #meetingname infrastructure
16:00:06 <zodbot> The meeting name has been set to 'infrastructure'
16:00:06 <jednorozec> #chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
16:00:06 <zodbot> Current chairs: bodanel dtometzki jednorozec jnsamyak lenkaseg nb nirik patrikp zlopez
16:00:06 <jednorozec> #info Agenda is at: https://board.net/p/fedora-infra
16:00:06 <jednorozec> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:00:06 <jednorozec> #info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
16:00:08 <jednorozec> #topic namaste
16:00:42 <nirik> morning
16:02:05 <zlopez> .hello zlopez
16:02:07 <zodbot> zlopez: zlopez 'Michal Konecny' <michal.konecny@pacse.eu>
16:04:35 <jednorozec> hello everybody, hope you have a lovely day lets start with the meeting agenda
16:04:44 <jednorozec> #topic New folks introductions
16:04:44 <jednorozec> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:04:44 <jednorozec> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:04:45 <pcreech> .hello pcreech
16:04:46 <zodbot> pcreech: Sorry, but user 'pcreech' does not exist
16:04:50 <pcreech> .hello pcreech17
16:04:50 <zodbot> pcreech: pcreech17 'Patrick Creech' <pcreech@redhat.com>
16:05:02 <jednorozec> any new people?
16:10:05 <jednorozec> ok moving on
16:10:11 <jednorozec> #topic Next chair
16:10:12 <jednorozec> #info magic eight ball says:
16:10:12 <jednorozec> #info chair 2023-11-09 pcreech
16:10:12 <jednorozec> #info chair 2023-11-16 ???
16:10:12 <jednorozec> #info chair 2023-11-23 ???
16:10:35 <jednorozec> any volunteers for next meetings?
16:10:57 <nirik> I can take the 16th if no one else wants it.
16:11:15 <nirik> the 23rd is thanksgiving in the US... I'm actually going to be out that entire week. ;)
16:11:30 <jednorozec> 16th is yours nirik
16:11:43 <jednorozec> should we cancel 23rd? since all US will be offline?
16:11:44 <zlopez> I can take the 23rd
16:11:58 <zlopez> But we can cancel it as well
16:12:30 <jednorozec> we can decide next week, meanwhile I have add you both to 16th and 23rd
16:12:42 <zlopez> OK
16:13:07 <jednorozec> #topic announcements and information
16:13:07 <jednorozec> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0830 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
16:13:07 <jednorozec> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:13:07 <jednorozec> #info Fedora 39 final freeze in effect
16:13:07 <jednorozec> #info Fedora 39 GO/NO-GO is happening today
16:13:31 <nirik> #info Tomorrow is a day of learning at Red Hat
16:14:05 <zlopez> #info Fedora infra developer guide was updated https://docs.fedoraproject.org/en-US/infra/developer_guide/
16:14:42 <nirik> I've also (so slowly) been submitting docs pr's for onboading/mentoring changes.
16:15:52 <jednorozec> any other announcements?
16:16:38 <jednorozec> I am confused little bit by the meeting doc
16:17:09 <nirik> oh?
16:17:11 <jednorozec> the sections things we should discuss, has no formal start or end. Its just like that?
16:17:11 <zlopez> Next is usually oncall
16:17:22 <nirik> if there's nothing added there, we skip
16:17:23 <jednorozec> or we skip it at the end or how does that section works?
16:17:27 <jednorozec> ack
16:17:29 <zlopez> There is topic to discussion if any
16:17:42 <jednorozec> moving on
16:17:58 <nirik> (might add a note to that effect to the doc? :)
16:18:17 <jednorozec> the confusing part is that header is different
16:18:48 <jednorozec> but thats probably just that I didnt char this meeting for more than a year...
16:19:52 <nirik> probibly just historical...
16:19:53 <jednorozec> #topic Oncall
16:19:53 <jednorozec> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:19:53 <jednorozec> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
16:19:53 <jednorozec> ## .oncalltakeeu .oncalltakeus
16:19:53 <jednorozec> #info nirik is on call from 2023-10-27 to 2023-11-02
16:19:54 <jednorozec> #info jednorozec is on call from 2023-11-03 to 2023-11-09
16:19:54 <jednorozec> #info pcreech is on call from 2023-11-10 to 2023-11-16
16:19:58 <jednorozec> #info ??? is on call from 2023-11-16 to 2023-11-23
16:20:17 <jednorozec> anyone want to take 2023-11-16 to 2023-11-23
16:21:07 <nirik> I could if no one else...
16:21:19 <zlopez> I can as well
16:21:20 <nirik> or we could wait and see next week?
16:21:43 <jednorozec> well we are covered also for the week after next meeting
16:21:53 <jednorozec> so we can wait and see
16:22:19 <jednorozec> also nirik isn't that the week your are off?
16:22:34 <nirik> oh yeah, oops
16:22:35 <nirik> :)
16:22:43 <jednorozec> no work for you!
16:23:03 <jednorozec> If I was mean I would say no computers but that is just unfair :)
16:23:22 <nirik> ha. well, keep in mind that my idea of 'fun' is computers... so...
16:23:35 <nirik> but at least no meetings. :)
16:23:48 <jednorozec> yeah we have similar diagnosis, I can relate
16:24:20 <jednorozec> #info Summary of last week: (nirik)
16:24:33 <jednorozec> any pings? things on fire?
16:24:39 <nirik> so, there were 2 oncall pings (that I saw at least):
16:25:15 <nirik> 1. Our fedoraproject.org dns zone's dnssec signature expired... so everything stopped resolving for dnssec. ;( Luckily the fix was just to push a newly signed one out.
16:25:34 <nirik> 2. Some package building issues that weren't urgent and addressed in chat.
16:26:26 <nirik> otherwise just furious activity to try and get f39 out.
16:27:01 <nirik> EOF
16:27:52 <jednorozec> #topic Monitoring discussion [nirik]
16:27:52 <jednorozec> #info https://nagios.fedoraproject.org/nagios
16:27:52 <jednorozec> #info Go over existing out items and fix
16:28:19 <nirik> nothing really new here...
16:28:33 <jednorozec> I have one thing related to monitoring but not nagios
16:29:17 <jednorozec> After the freeze is over I would like to start monitoring compose cron jobs by zabbix
16:29:32 <nirik> well, after freeze we are planning on rolling zabbix out to prod...
16:29:40 <nirik> so this seems like just a part of that, but ok. ;)
16:29:58 <nirik> which cron jobs did you mean?
16:30:07 <jednorozec> yeah but its one thing to roll out the agents and doing the actuall monitoring
16:30:31 <jednorozec> so we dont have a "bell" for when compose gets stuc
16:30:36 <nirik> sure there's going to be a lot of migrating checks from nagios...
16:30:42 <jednorozec> when it fails we catch it when its ok we do as well
16:31:44 <jednorozec> but one have to check or set highlights on bot channel to catch the hanging...
16:31:54 <nirik> there's a nagios check for 'no bodhi push in 24h' but yeah, we can try and come up with zabbix checks... not sure how it would work, would have to think on it
16:32:15 <jednorozec> I already did with David he pointed me to some sources
16:32:50 <nirik> I suppose a generic one (which we want anyhow) that checks datagrepper could work...
16:33:13 <nirik> but those sorts of checks are difficult, because sometimes things just take longer and you don't want to always alert on that...
16:33:24 <nirik> but if you wait too long, its not catching the problem sooner.
16:34:05 <nirik> anyhow, +1 to checks we can always adjust them
16:34:18 <jednorozec> right, but from my perspective its better to allert 1-2h after the fact then to wait for a person to check and realize that something is wrong
16:34:42 <jednorozec> but yeah its not that easy when running time is variable...
16:35:40 <nirik> yeah. rawhide takes 3-5hours... so alerting after 6 or so seems like it would be a good first try
16:36:34 <nirik> also possibly alerting on DOOMed compose would be interesting... then we could ack it when investigating and it clears with the next INCOMPLETE/FINISHED
16:37:19 <jednorozec> I see we think in similar way :)
16:37:30 * nirik wishes we had newer zabbix tho
16:38:05 <jednorozec> oh I didnt check the version
16:38:21 <jednorozec> hopefully I didnt spend time reading docs for stuff that I cannot use :D
16:39:35 <jednorozec> but there is more to the meeting, so next topic should be me and 2nd part of pungi
16:40:04 <nirik> We have 6.0 I think (the current LTS), but 6.2 (out soon) has a bunch of good stuff.
16:41:29 <jednorozec> wanted to sync this with a blog post on what phases and how we use in fedora. But didn't managed to publish it yet.
16:42:19 <jednorozec> so should I just blast at you parts of pungi configs and docs pages or we can do backlog or brainstorming
16:43:51 <nirik> I'm fine either way. it's a small crowd today, so pungi might be good later if you want to prepare more...
16:44:55 <Southern_Gentlem> jednorozec,  as always i want to know how to respin the server iso
16:46:23 <jednorozec> Southern_Gentlem, by respin you mena the same respin as we do for live xfce and other images?
16:46:36 <jednorozec> so respin with latest and greatest from current release
16:50:32 <jednorozec> or you mean respin in context of fedora spins https://docs.fedoraproject.org/en-US/releases/spins/creating/
16:50:55 <nirik> I would think it would work ok with the right pungi.conf...
16:51:17 <jednorozec> just small portion of the config actually
16:52:30 * nirik goes to get more coffee before fesco/go-nogo
16:52:51 <jednorozec> #topic Open Floor
16:53:16 <aheath1992> Nothing from me
16:54:13 <zlopez> I have nothing for open floor
16:55:16 <jednorozec> Ok then, I will give you back 5min of your day
16:55:20 <jednorozec> enjoy!
16:55:29 <jednorozec> #endmeeting