16:00:23 <aheath1992> #startmeeting Infrastructure (2023-03-16)
16:00:23 <zodbot> Meeting started Thu Mar 16 16:00:23 2023 UTC.
16:00:23 <zodbot> This meeting is logged and archived in a public location.
16:00:23 <zodbot> The chair is aheath1992. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:00:23 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:23 <zodbot> The meeting name has been set to 'infrastructure_(2023-03-16)'
16:00:37 <smooge> .hi
16:00:40 <aheath1992> #meetingname infrastructure
16:00:40 <zodbot> The meeting name has been set to 'infrastructure'
16:00:44 <phsmoura> .hi
16:00:53 <aheath1992> #chair nirik zlopez nb bodanel dtometzki jnsamyak
16:00:53 <zodbot> Current chairs: aheath1992 bodanel dtometzki jnsamyak nb nirik zlopez
16:02:42 <zodbot> smooge: smooge 'Stephen Smoogen' <smooge@redhat.com>
16:02:42 <aheath1992> #info Agenda is at: https://board.net/p/fedora-infra
16:02:42 <darknao> .hi
16:02:42 <aheath1992> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:02:42 <aheath1992> #info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
16:02:42 <aheath1992> #topic greetings!
16:02:42 <dtometzki> .hi
16:02:42 <aheath1992> Hello All!! Hows your day going
16:02:42 <nirik> morning
16:02:42 <eddiejenningsjr> .hello eddiejennings
16:02:42 <nirik> so far so good here.
16:02:42 <dtometzki> hi together
16:02:45 <smooge> zod's reloading fedora so hi/hello will work in a bit
16:02:45 <zodbot> phsmoura: phsmoura 'Pedro Moura' <pmoura@redhat.com>
16:02:46 <mkonecny> .hello zlopez
16:02:50 <zodbot> darknao: darknao 'Francois Andrieu' <darknao@drkn.ninja>
16:02:59 <smooge> there we go
16:03:09 <aheath1992> #topic New folks introductions
16:03:11 <eddiejenningsjr> Living the dream.  And realizing how expensive training will be as I look at moving beyond RHCE in the Red Hat cert world :(
16:03:19 <zodbot> dtometzki: dtometzki 'Damian Tometzki' <damian@riscv.tometzki.de>
16:03:22 <zodbot> eddiejenningsjr: eddiejennings 'Eddie Jennings' <eddie@eddiejennings.net>
16:03:25 <zodbot> mkonecny: zlopez 'Michal Konecny' <michal.konecny@pacse.eu>
16:03:40 <aheath1992> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:03:40 <aheath1992> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:04:06 <aheath1992> Do we have any new people among us? Please say hello.
16:05:21 <aheath1992> I guess we have no new arrivals this week. Moving on...
16:05:38 <aheath1992> #topic Next chair
16:05:54 <aheath1992> #info magic eight ball says:
16:05:54 <aheath1992> #info chair 2023-03-16 - aheath1992
16:05:54 <aheath1992> #info chair 2023-03-23 - Canceled due to F2F CPE meeting
16:05:55 <aheath1992> #info chair 2023-03-30 - nirik
16:05:55 <aheath1992> #info chair 2023-04-06 - mkonecny
16:05:55 <aheath1992> #info chair 2023-04-13 - eddiejenningsjr
16:06:12 <nirik> nice... pretty full list. ;)
16:06:43 <dtometzki> i can take 04-20 if you want add me
16:07:07 <aheath1992> Sold to dtometzki
16:07:31 <aheath1992> #info chair 2023-04-20 - dtometzki
16:08:10 <aheath1992> Reminder 2023-03-23 - Canceled due to F2F CPE meeting
16:08:27 <aheath1992> No meeting next week
16:08:58 <aheath1992> Moving Along
16:09:08 <aheath1992> #topic announcements and information
16:09:32 <aheath1992> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:09:32 <aheath1992> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:09:32 <aheath1992> #info If your team wants support from the Fedora Program Management Team, file an isssue: https://pagure.io/fedora-pgm/pgm_team/issues?template=support_request
16:09:32 <aheath1992> #info oncall should also handling #fedora-releng pings if possible
16:09:34 <aheath1992> #info Fedora 38 beta freeze is over
16:09:56 <aheath1992> Any more announcements?
16:10:23 <eddiejenningsjr> I love it.  F38 freeze is done, and I'm still trying to get approval to upgrade a client from F35 :P
16:10:25 <nirik> #info F38 Final freeze starts 023-04-04
16:10:28 <nirik> s/023/2023/
16:10:42 <eddiejenningsjr> well, beta freeze that is
16:11:26 <nirik> I'd like to schedule a mass update/reboot outage for the week of the 27th I think... will figure out details.
16:12:39 <aheath1992> Any other announcements?
16:13:05 <smooge> yes
16:13:42 <smooge> there will be some network switch reboots on 2023-03-22 from 11:00 -> 12:00 UTC.
16:14:03 <smooge> this may affect some services but downtime should be minimal
16:14:29 <aheath1992> Thanks for the info smooge!
16:14:30 <smooge> also things will 'pile up' in tickets next week as CPE staff will be in Face2Face meetings
16:15:17 <smooge> I will be helping on the network reboots. The tickets and pings in irc will have to handle themselves
16:15:50 <aheath1992> Understood
16:15:57 <nirik> Yeah, I will try and tirage things as they come, but yeah...
16:16:19 <nirik> I mean, what else is there to do on 10hour plane flights?
16:16:43 <aheath1992> Anything else before we move on?
16:17:12 <mkonecny> Try Fedora 38 Beta, it's good :-)
16:17:35 <nirik> +1
16:18:11 <aheath1992> #topic Oncall
16:18:11 <aheath1992> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:18:11 <aheath1992> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
16:18:11 <aheath1992> ## .oncalltakeeu .oncalltakeus
16:18:11 <aheath1992> #info darknao is on call from 2023-03-10 to 2023-03-16
16:18:12 <aheath1992> #info eddiejenningsjr is on call from 2023-03-17 to 2023-03-23
16:18:14 <aheath1992> #info ??? is on call from 2023-03-24 to 2023-03-30
16:18:57 <aheath1992> any volunteers for 12023-03-24 to 2023-03-30?
16:19:09 <eddiejenningsjr> .oncalltakeus
16:19:12 <zodbot> eddiejenningsjr: Kneel before zod!
16:20:14 <nirik> we could also just change it to say file a ticket for that week.
16:20:24 <phsmoura> +1
16:20:52 <mkonecny> I think most of the infra team will be still traveling back from F2F
16:21:32 <nirik> right.
16:22:12 <aheath1992> #info Summary of last week: (from current oncall darknao )
16:22:45 <darknao> nothing to report, not a single call this week
16:23:16 <aheath1992> darknao, always good to hear
16:23:33 <nirik> cool.
16:23:55 <aheath1992> #topic Monitoring discussion [nirik]
16:24:10 <aheath1992> #info https://nagios.fedoraproject.org/nagios
16:24:10 <aheath1992> #info Go over existing out items and fix
16:24:21 <aheath1992> nirik, anything to report?
16:25:28 <nirik> yeah, a few things.
16:25:58 <nirik> So, I got all the new servers we got racked up and installed. (Look for a document on this before too long)
16:26:25 <nirik> There's still a bunch of random little alerts we need to try and clean up.
16:26:35 <nirik> (There's a ticket describing those if anyone wants to give it a go)
16:27:03 <nirik> There's a few copr machines alerting, I'll ping them about those to try and sort those out.
16:27:11 <nirik> But overall it's looking better than it has in a while.
16:27:14 <nirik> EOF
16:28:07 <aheath1992> #topic Revisit blocked tickets
16:28:24 <aheath1992> #info Check if any blocked ticket is unblocked
16:28:24 <aheath1992> #link https://pagure.io/fedora-infrastructure/issues?status=Open&tags=blocked&priority=0&close_status=
16:28:58 <aheath1992> Only 4 tickets
16:29:23 <aheath1992> .ticket 10372
16:29:24 <zodbot> aheath1992: Issue #10372: [proposal] Move Pagure (pagure.io and src.fp.o) to use OIDC for web login - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10372
16:29:33 <nirik> me waits for pagure.
16:29:44 <nirik> none of them are unblocked yet...
16:29:55 <nirik> the mailman one is making some progress of late tho. ;)
16:30:18 <aheath1992> .ticket 8455
16:30:22 <zodbot> aheath1992: Issue #8455: Move mailman to newer release of Fedora or CentOS - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8455
16:30:51 <aheath1992> .ticket 8291
16:30:52 <zodbot> aheath1992: Issue #8291: Fedora messaging - Translation platform topics review - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8291
16:30:58 <aheath1992> .ticket 7361
16:30:59 <zodbot> aheath1992: Issue #7361: Signing infrastructure for aarch64 - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/7361
16:31:21 <nirik> Yeah, all blocked for various reasons. ;)
16:31:46 <aheath1992> Understood, moving along
16:31:53 <aheath1992> #topic Fedora Infra backlog refinement
16:32:13 <aheath1992> #info Refine oldest tickets on Fedora Infra tracker
16:32:13 <aheath1992> #link https://pagure.io/fedora-infrastructure/issues?status=Open&order_key=last_updated&order=asc
16:33:18 <aheath1992> .ticket 11013
16:33:19 <zodbot> aheath1992: Issue #11013: Redeploy openQA workers with consistent storage configuration - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11013
16:33:28 <nirik> BTW, I closed about 10 tickets yesterday. If there's anything I closed I shouldn't have, feel free to re-open it. ;)
16:33:53 <nirik> This has just been a matter of time... nothing is really blocking it aside from scheduling the time to do it.
16:34:09 <aheath1992> .ticket 10120
16:34:09 <nirik> Although we probibly need to decide what the config we want to use actually is.
16:34:10 <zodbot> aheath1992: Issue #10120: Investigate forwarding centralised logs to splunk - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10120
16:35:14 <aheath1992> nirik, went we waiting of freeze to end before we finish this?
16:35:21 <nirik> So, I think this is all pretty ready, but we need to update the PR with the certs we now have
16:35:26 <nirik> yep.
16:36:09 <aheath1992> ack, let me know when you do so I can watch Splunk to make sure all logs are coming in correctly
16:36:12 <nirik> I guess I could just push the pr and add another to tweak the certs stuff
16:36:38 <nirik> perhaps later this morning? if you will be around?
16:36:39 <aheath1992> However you want to do it.
16:36:52 <aheath1992> .ticket 10358
16:36:53 <zodbot> aheath1992: Issue #10358: IPA backup fails on ipa03 and ipa02.stg - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10358
16:37:52 <mkonecny> I see that we have a plan to move the machines to RHEL9 and see if the issue will be gone
16:38:10 <nirik> yeah, no idea if that will solve it, but worth a try.
16:38:33 <nirik> I could try stg after we go into final freeze?
16:38:34 <aheath1992> understood
16:38:56 <aheath1992> .ticket 11063
16:38:57 <zodbot> aheath1992: Issue #11063: Mini-initiative: feed Fedora Discussion user notifications into the new notifications system - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11063
16:39:00 * nirik adds to the todo list
16:40:14 <aheath1992> .ticket 10899
16:40:15 <zodbot> aheath1992: Issue #10899: Enhancement: priority signing in Robosignatory - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10899
16:40:41 <mkonecny> Ad 11063 This needs some investigation to know if we can just use some private queue in rabbitmq
16:40:54 <nirik> Hang on, I think there was more to this one...
16:41:55 <nirik> ah, right, it's in the ticket.
16:42:23 <nirik> I don't think a private queue will work... but I could be mistaken
16:42:36 <nirik> the priority stuff should do a lot.
16:43:09 <aheath1992> .ticket 11090
16:43:09 <zodbot> aheath1992: Issue #11090: nagios: busgateway01 and notifs nagios alerts - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11090
16:44:18 <nirik> thats the nagios one I mentioned eariler. ;)
16:44:28 <nirik> just needs someone to work on it.
16:44:39 <aheath1992> I have not had time to look at it or see if I have the access to work on it
16:45:25 <nirik> no hurry, just whoeever and whenever. It could be multiple people doing parts too.
16:45:37 <aheath1992> understood
16:45:51 <aheath1992> Moving along as we are running short on time
16:46:02 <nirik> I've been trying to update these tickets as we go... if I miss any, please let me know.
16:46:34 <aheath1992> Will do
16:46:38 <aheath1992> #topic Upcoming learning topics
16:46:49 <aheath1992> #info 2023-03-30 Learn how do contribute to our infra docs + live fixes [darknao, nirik]
16:47:06 <nirik> Hopefully I will be recovered from travel by then.
16:47:13 <aheath1992> We don't have any learning topics listed for the coming weeks. Any suggestions or offers?
16:48:13 <mkonecny> I'm just thinking what we didn't cover yet
16:48:18 <aheath1992> s/any/many/
16:49:31 <eddiejenningsjr> I've done some stuff with Wireguard and working with local repo mirrors in my home lab.  I could do a topic on either -- but they wouldn't be directly related to Fedora-Infra
16:49:51 <nirik> wireguard might be fun.
16:50:02 <nirik> I have considered moving us from openvpn in infra to wireguard...
16:50:06 <mkonecny> What about some topic of how to troubleshoot infra issues?
16:50:06 <eddiejenningsjr> I'm no Wireguard guru, but I'm happy to share what I've learend :)
16:50:51 <nirik> troubleshooting is always good...
16:52:08 <mkonecny> There is always something new to learn
16:52:26 <mkonecny> Maybe just sharing troubleshooting stories here would be nice :-)
16:52:43 <aheath1992> yep while we have a few mins left before the hour
16:52:46 <aheath1992> #topic Open Floor
16:52:55 <eddiejenningsjr> Could add the spin of troubleshooting infra issues as an apprentice:  Talk what can be done from the scope of what privileges an apprentice has.
16:53:15 <nirik> zlopez: yeah! a troubleshooting round table might be nice. I like the more interactive things.
16:53:16 <aheath1992> eddiejenningsjr, +1 to that
16:53:37 <nirik> yeah
16:53:42 <mkonecny> Eddie Jennings, Jr.: +1
16:54:15 <mkonecny> As member of sysadmin-main one sometimes forgot what the apprentice could do
16:54:33 <eddiejenningsjr> That may or may not be a way to try to push myself to carve time to get more involved outside of Thursday meetings :D
16:55:20 <nirik> I'd love to add some more things to the apprentice program if we can ever find the time.
16:55:30 <aheath1992> as a newer member that can help point me to areas I can help with
16:55:44 <nirik> right now it's pretty free form, but we could add things that people could specifically do...
16:56:34 <aheath1992> nirik, +1 that may help give new people guidance
16:57:19 <nirik> "debug a nagios alert" "look at logs for X and see errors" "propose a ansible PR doing XYZ" etc
16:57:46 <nirik> but I'll try and come up with a more well formed proposal when I ponder some more.
16:58:13 <nirik> "run a infra meeting" "be oncall"
16:58:19 <nirik> (all those could be badges too)
16:59:05 <aheath1992> anything else before the end of the meeting?
16:59:06 <nirik> anyhow, nothing more, and we are low on time. ;)
16:59:37 <mkonecny> Enjoy the rest of the week everyone
16:59:44 <aheath1992> #endmeeting