16:00:48 <eddiejennings> #startmeeting Infrastructure (2023-01-12)
16:00:48 <zodbot> Meeting started Thu Jan 12 16:00:48 2023 UTC.
16:00:48 <zodbot> This meeting is logged and archived in a public location.
16:00:48 <zodbot> The chair is eddiejennings. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:00:48 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:48 <zodbot> The meeting name has been set to 'infrastructure_(2023-01-12)'
16:01:07 <eddiejennings> #meetingname infrastructure
16:01:07 <zodbot> The meeting name has been set to 'infrastructure'
16:01:21 <eddiejennings> #chair nirik zlopez nb bodanel dtometzki jnsamyak
16:01:21 <zodbot> Current chairs: bodanel dtometzki eddiejennings jnsamyak nb nirik zlopez
16:01:21 <eddiejennings> #info Agenda is at: https://board.net/p/fedora-infra
16:01:21 <eddiejennings> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:01:21 <eddiejennings> #info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
16:01:21 <eddiejennings> #topic greetings!
16:01:31 <eddiejennings> Happy new year everyone! :D
16:02:49 <nirik> morning everyone
16:02:53 <nirik> happy new year
16:03:18 <eddiejennings> My apologies for not doing the early day-of reminder :(  Hopefully more folk appear.
16:03:49 <darknao> .hi
16:03:49 <zodbot> darknao: darknao 'Francois Andrieu' <darknao@drkn.ninja>
16:03:57 <eddiejennings> Otherwise, we'll have a quick meeting ;)
16:04:00 <eddiejennings> .hi
16:04:01 <zodbot> eddiejennings: eddiejennings 'Eddie Jennings' <eddie@eddiejennings.net>
16:04:09 <mkonecny> .hello zlopez
16:04:10 <zodbot> mkonecny: zlopez 'Michal Konecny' <michal.konecny@pacse.eu>
16:04:27 <jnsamyak> .hi
16:04:28 <zodbot> jnsamyak: jnsamyak 'Samyak Jain' <samyak.jn11@gmail.com>
16:04:38 <eddiejennings> #topic New folks introductions
16:04:38 <eddiejennings> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:04:38 <eddiejennings> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:04:52 <eddiejennings> Do we have any new people with us this week?
16:05:03 <eddiejennings> If so, introduce yourself! :)  We don't bite!
16:06:34 <eddiejennings> Hearing none, let's move on
16:06:50 <eddiejennings> #topic Next chair
16:06:50 <eddiejennings> #info magic eight ball says:
16:06:50 <eddiejennings> #info chair 2023-01-12 - eddiejenningsjr
16:06:50 <eddiejennings> #info chair 2023-01-19 - dtometzki
16:06:51 <eddiejennings> #info chair 2023-01-26 - ?
16:06:51 <eddiejennings> #info chair 2023-02-02 - ?
16:07:10 <phsmoura> I can take 01-26
16:07:23 <eddiejennings> sold to phsmoura!
16:07:36 <eddiejennings> #info chair 2023-01-26 - phsmoura
16:08:08 <saibug[m]> .hi
16:08:09 <zodbot> saibug[m]: Sorry, but user 'saibug [m]' does not exist
16:08:23 <eddiejennings> Anyone interested in February 2nd?  Otherwise, we're fine for now with chairs
16:08:53 <mkonecny> I can take it
16:09:03 <eddiejennings> Welcome sailbug.  Try .hello your_FAS_name (unless nirik has a better suggestion)
16:09:09 <eddiejennings> sold to mkonecy!
16:09:10 <saibug[m]> .hello seddik
16:09:10 <zodbot> saibug[m]: seddik 'Seddik Alaoui Ismaili' <seddikalaouiismaili@gmail.com>
16:09:22 <mkonecny> I won!
16:09:29 <eddiejennings> #info chair 2023-02-02 - mkonecy
16:10:04 <eddiejennings> Let's move along!
16:10:22 <eddiejennings> #topic announcements and information
16:10:22 <eddiejennings> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:10:22 <eddiejennings> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:10:22 <eddiejennings> #info If your team wants support from the Fedora Program Management Team, file an isssue: https://pagure.io/fedora-pgm/pgm_team/issues?template=support_request
16:10:24 <eddiejennings> #info oncall should also handling #fedora-releng pings if possible
16:10:26 <eddiejennings> #info New design of Anitya available to try on https://stg.release-monitoring.org
16:10:28 <eddiejennings> #info F38 mass rebuild starts on 2023-01-18
16:10:30 <eddiejennings> #info F38 branching from rawhide on  2023-02-07
16:10:50 <eddiejennings> Any new announcements now that we're in 2023?
16:11:01 <mkonecny> #info SCM Requests are now processed automatically
16:11:33 <nirik> \o/
16:11:33 <eddiejennings> added
16:12:29 <smooge> big win
16:12:56 <eddiejennings> Any other announcements?
16:14:13 <eddiejennings> Sound like no more.  Let's keep on going.
16:14:41 <eddiejennings> #topic Oncall
16:14:41 <eddiejennings> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:14:41 <eddiejennings> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
16:14:41 <eddiejennings> ## .oncalltakeeu .oncalltakeus
16:14:41 <eddiejennings> #info eddiejenningsjr is on call from 2023-01-05 to 2023-01-12
16:14:42 <eddiejennings> #info ? is on call from 2023-01-13 to 2023-01-20
16:14:44 <eddiejennings> #info ? is on call from 2023-01-20 to 2023-01-27
16:14:58 <eddiejennings> The on-call from December though today finally comes to an end :D
16:15:09 <eddiejennings> Do we have a volunteer for this week?
16:16:01 <nirik> I can take this week
16:16:11 <dtometzki> .hi
16:16:13 <zodbot> dtometzki: dtometzki 'Damian Tometzki' <linux@tometzki.de>
16:16:26 <eddiejennings> sold to nirik
16:16:32 <nirik> .oncalltakeus
16:16:32 <zodbot> nirik: Kneel before zod!
16:16:34 <dtometzki> I can take from 01-20 -
16:16:40 <eddiejennings> #info nirik is on call from 2023-01-13 to 2023-01-20
16:17:00 <eddiejennings> sold to dtometzki
16:17:29 <eddiejennings> #info dtometzki is on call from 2023-01-20 to 2023-01-27
16:18:05 <eddiejennings> Thanks nirik and dtometzki for volunteering for on-call.
16:18:06 <dtometzki> thx
16:18:49 <nirik> no problem
16:19:19 <eddiejennings> For those interested in on-call, it's not a high-stress job.  You basically are a gate keeper for nirik and others.  You acknowledge a requests, if you can solve the problem that's great.  Otherwise, you'll direct folks to make a ticket.  The idea is to not have people ping folks directly.
16:19:41 <eddiejennings> Speaking of . . .
16:19:44 <eddiejennings> #info Summary of last week: (from current oncall )
16:20:25 <eddiejennings> The last few weeks have been fairly quiet.  Matthew Miller had a couple of pings, which I think nirik took care of (despite being on vacation)
16:21:17 <eddiejennings> Speaking of nirik. . .
16:21:23 <eddiejennings> #topic Monitoring discussion [nirik]
16:21:23 <eddiejennings> #info https://nagios.fedoraproject.org/nagios
16:21:23 <eddiejennings> #info Go over existing out items and fix
16:21:39 <nirik> so, lets see.
16:22:07 <nirik> there is currently some networking issues with some of the proxies, so thats causing nagios to not look nice right now
16:22:55 <nirik> There's some messaging checks that we need to clean up... I keep meaning to file a ticket on those.
16:23:33 <nirik> Otherwise the other day I fixed ipv6 on proxy09, which was alerting...
16:23:52 <mkonecny> Which messaging checks?
16:23:57 <nirik> I'll try and file a ticket or two soon on cleaning things up. Would be nice to have it fixed before freeze
16:24:11 <nirik> https://nagios.fedoraproject.org/nagios/cgi-bin//status.cgi?host=all&type=detail&servicestatustypes=8&serviceprops=4
16:24:45 <nirik> also some others.
16:25:16 <nirik> https://nagios.fedoraproject.org/nagios/cgi-bin//status.cgi?navbarsearch=1&host=busgateway
16:25:23 <nirik> these all on busgateway
16:25:39 <mkonecny> Oh, checks in nagios
16:26:09 <nirik> and this notifs-web01 one: HTTP CRITICAL: HTTP/1.1 308 PERMANENT REDIRECT - string 'Notifications' not found on 'http://localhost:80/notifications' - 579 bytes in 0.003 second response time
16:27:22 <nirik> Anyhow, any further questions on all that? or shall we move on?
16:27:38 <smooge> sounds good
16:27:52 <eddiejennings> None from me.
16:28:14 <eddiejennings> Before we move on to tickets.  I'd like to see if we can plan some learning topics.
16:28:39 <eddiejennings> Traditionally, we'd alternate weeks of going over the ticket queue and doing a learning topic.
16:28:54 <eddiejennings> Do we want to try to keep that pace, or maybe do one learning topic / month?
16:29:25 <nirik> I think it would be nice to alternate...
16:29:27 <nirik> if we can
16:29:41 <eddiejennings> I agree.
16:29:57 <eddiejennings> What say the rest of the gallery? :)
16:30:18 <dtometzki> I agree too
16:30:19 <mkonecny> I'm for it
16:30:50 <lenkaseg> +1
16:31:12 <eddiejennings> So it shall be.
16:31:41 <nirik> Im always happy to talk on a variety of topics... just need folks to let me know what they would like to hear about
16:32:05 <eddiejennings> That being said, that would have us doing a learning topic on 1/19.  And in February 2/2, and 2/16.
16:32:29 <eddiejennings> I've been testing btrbk on my laptop for btrfs snapshot backups, and I think I'll be ready to do a learning topic on it for 2/2
16:33:04 <mkonecny> I can probably talk about the new automation for SCM requests
16:33:06 <nirik> cool.
16:33:11 <nirik> mkonecny: +1
16:33:45 <eddiejennings> #info Using Btrbk for BTRFS snapshot backups [eddiejennings] on 2023-02-02
16:33:53 <lenkaseg> both great topics, cool!
16:34:23 <saibug[m]> yes both are great :0
16:34:25 <jnsamyak> +1
16:34:36 <eddiejennings> So for 1/19, what and who?
16:36:02 <mkonecny> I will probably not have time to talk about it next week?
16:36:20 <mkonecny> :-/
16:36:22 * nirik happy to talk about something he knows, but not sure what...
16:36:34 <smooge> feeding cats for fun and not profit
16:36:42 <eddiejennings> Do you want to talk about SCM requests on 1/19, mkonecny?
16:37:46 <mkonecny> I'm not sure if I will have time to talk about it
16:37:57 <mkonecny> Or to join this meeting
16:38:05 <nirik> I could talk about my new home server... but not sure how interesting that might be.
16:38:24 <eddiejennings> I'd be interested in that.  And compare to what I'm doing myself :D
16:38:39 <eddiejennings> mkonecy how about 2022-02-16?
16:40:34 <mkonecny> That's OK
16:40:35 <mkonecny> I recently reinstalled my own homeserver to CentOS 9 Stream :-)
16:41:22 <jednorozec> show your home lab ?
16:41:27 <eddiejennings> #info 2023-02-16 Automating SCM Requests [mkonecny]
16:41:39 <eddiejennings> And nirik for home server next week (2023-01-19)?
16:42:09 <mkonecny> This could even be a discussion
16:42:12 <nirik> yeah, how about a 'homeserver/lab roundtable? and we can all discuss setups?
16:42:17 <jednorozec> it might be a good idea to share homelab/server setups
16:42:27 <nirik> +1
16:42:52 <dtometzki> +1
16:42:57 <eddiejennings> +1 on the round table
16:43:39 <eddiejennings> #info 2023-01-19 Home Lab Setup Roundtable Discussion
16:44:03 <eddiejennings> Excellent.  Nice to have some of these in queue :)
16:44:08 <eddiejennings> Speaking of queues
16:44:28 <eddiejennings> #topic Revisit blocked tickets
16:44:28 <eddiejennings> #info Check if any blocked ticket is unblocked
16:44:28 <eddiejennings> #link https://pagure.io/fedora-infrastructure/issues?status=Open&tags=blocked&priority=0&close_status=
16:44:52 <eddiejennings> We won't bring up every ticket, but take a moment to look at the list, and if there is any news to discuss, mention the ticket
16:44:54 <nirik> nope, all still blocked
16:45:04 <eddiejennings> Ha!  That was quick :D
16:45:41 <eddiejennings> Moving along.
16:45:42 <eddiejennings> #topic Fedora Infra backlog refinement
16:45:42 <eddiejennings> #info Refine oldest tickets on Fedora Infra tracker
16:45:42 <eddiejennings> #link https://pagure.io/fedora-infrastructure/issues?status=Open&order_key=last_updated&order=asc
16:46:34 <saibug[m]> .ticket 10120
16:46:35 <zodbot> saibug[m]: Issue #10120: Investigate forwarding centralised logs to splunk - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10120
16:47:14 <saibug[m]> still waiting CA file ?
16:48:41 <nirik> yeah, I think aheath1992 said he would try and get back to this soon...
16:49:07 <aheath1992> Yep I can get the CA file next week
16:49:28 <eddiejennings> Excellent
16:49:35 <saibug[m]> .tocket 10671
16:49:45 <saibug[m]> * .ticket 10671
16:50:24 <saibug[m]> .ticket 10671
16:50:25 <zodbot> saibug[m]: Issue #10671: Alert admins of the openshift apps when the pod crashes - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10671
16:51:28 <eddiejennings> Coming from someone with zero openshift experience, and having a chat in Matrix, this seems like a good bit of building is involved
16:51:31 <mkonecny> My old friend :-)
16:51:31 <saibug[m]> i'm interested to work on it, but i think dkirwan is on it
16:51:57 <nirik> yeah, he may at least have a good idea on where to start...
16:52:07 <eddiejennings> I was thinking it would be something that we setup in the openshift console, that basically says if pod goes down, email these people
16:53:09 <eddiejennings> I'm sure dkirwan would appreciate the help / support.  It's too far outside my skill set for me to be of use right now :(
16:53:35 <nirik> I think we need to add alertmanager rules to do this, but I haven't looked into whats involved in those.
16:54:41 <saibug[m]> alertmanager rules are a requirements
16:54:41 <saibug[m]> yes
16:54:57 <eddiejennings> saibug[m], I'd reach out to dkirwan and you both work on it :)
16:56:27 <eddiejennings> In our last few minutes, let's move to
16:56:28 <eddiejennings> #topic Open Floor
16:56:51 <eddiejennings> The upcoming learning topics were put onto the board.net page already
16:57:35 <nirik> great.
16:58:03 <eddiejennings> I've now finally got the fedora-infra calendar on my various devices
16:58:24 <mkonecny> nirik: I noticed there is some spam being sent from mailer-daemon@fedoraproject.org, do you know about this?
16:58:32 <mkonecny> It even sends it to itself
16:58:48 <eddiejennings> Does anyone know what the COPR community person calendar event is?
16:59:13 <nirik> mkonecny: I've seen some of that, but mostly just dropped it... I don't think it's getting into lists or anything
16:59:49 <eddiejennings> Nevermind on my question.  Opening the event explained everything :)
17:00:10 <mkonecny> nirik: I was just surprised that it's sent from mailer-daemon address
17:00:12 <smooge> mkonecny, that seems to be automated spam from third parties using the mailer-daemon address as a 'maybe someone whitelisted it'
17:00:32 <mkonecny> Ok, I will ignore it then
17:00:51 <eddiejennings> And with that, we're at an hour.  Feel free to continue the discussion in #fedora-admin other the various other channels :)
17:00:54 <eddiejennings> #endmeeting