2024-07-25 16:00:08 <@nirik:matrix.scrye.com> !startmeeting Infrastructure (2024-07-25)
2024-07-25 16:00:10 <@meetbot:fedora.im> Meeting started at 2024-07-25 16:00:08 UTC
2024-07-25 16:00:10 <@meetbot:fedora.im> The Meeting name is 'Infrastructure (2024-07-25)'
2024-07-25 16:00:14 <@nirik:matrix.scrye.com> !chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
2024-07-25 16:00:14 <@nirik:matrix.scrye.com> !info Agenda is at: https://board.net/p/fedora-infra
2024-07-25 16:00:14 <@nirik:matrix.scrye.com> !info About our team: https://docs.fedoraproject.org/en-US/cpe/
2024-07-25 16:00:14 <@nirik:matrix.scrye.com> !info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
2024-07-25 16:00:14 <@nirik:matrix.scrye.com> !topic ahoy
2024-07-25 16:00:14 <@nirik:matrix.scrye.com> !meetingname infrastructure
2024-07-25 16:00:15 <@meetbot:fedora.im> The Meeting Name is now infrastructure
2024-07-25 16:00:50 <@nirik:matrix.scrye.com> welcome aboard everyone.
2024-07-25 16:00:56 <@seddik:fedora.im> .Hi 
2024-07-25 16:01:09 <@pcreech:matrix.org> !hello pcreech17
2024-07-25 16:01:11 <@zodbot:fedora.im> Patrick Creech (pcreech17)
2024-07-25 16:01:15 <@jnsamyak:matrix.org> !hi
2024-07-25 16:01:17 <@zodbot:fedora.im> Samyak Jain (jnsamyak) - he / him / his
2024-07-25 16:01:20 <@Zlopez:matrix.org> !hi
2024-07-25 16:01:23 <@seddik:fedora.im> !hi 
2024-07-25 16:01:24 <@zodbot:fedora.im> seddik alaouiismaili (seddik)
2024-07-25 16:01:28 <@zodbot:fedora.im> Michal Konecny (zlopez)
2024-07-25 16:02:42 <@nirik:matrix.scrye.com> lets wait a bell for more to come aboard
2024-07-25 16:03:33 <@nirik:matrix.scrye.com> (history aside: https://en.wikipedia.org/wiki/Ship's_bell )
2024-07-25 16:03:49 <@nirik:matrix.scrye.com> !info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
2024-07-25 16:03:49 <@nirik:matrix.scrye.com> !topic New folks introductions
2024-07-25 16:03:49 <@nirik:matrix.scrye.com> !info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
2024-07-25 16:04:01 <@nirik:matrix.scrye.com> any new crew to report in/
2024-07-25 16:04:04 <@nirik:matrix.scrye.com> any new crew to report in?
2024-07-25 16:04:55 <@nirik:matrix.scrye.com> seems not... lets sail on
2024-07-25 16:05:11 <@nirik:matrix.scrye.com> !topic Next chair
2024-07-25 16:05:11 <@nirik:matrix.scrye.com> !info chair 2024-08-15 - ???
2024-07-25 16:05:11 <@nirik:matrix.scrye.com> !info chair 2024-08-08 - cancelled (Flock)
2024-07-25 16:05:11 <@nirik:matrix.scrye.com> !info chair 2024-08-01 pcreech
2024-07-25 16:05:11 <@nirik:matrix.scrye.com> !info magic eight ball says:
2024-07-25 16:05:30 <@nirik:matrix.scrye.com> anyone want to take the captainship on the 15th?
2024-07-25 16:07:22 <@nirik:matrix.scrye.com> I guess we can decide next week...
2024-07-25 16:07:54 <@nirik:matrix.scrye.com> !info Remaining issues with new mailman3 deployment are being investigated
2024-07-25 16:07:54 <@nirik:matrix.scrye.com> !info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
2024-07-25 16:07:54 <@nirik:matrix.scrye.com> !info Fedora 41 Mass rebuild is done
2024-07-25 16:07:54 <@nirik:matrix.scrye.com> !info Fedora Infra weekly meeting is cancelled for 2024-08-08 because of Flock 2024
2024-07-25 16:07:54 <@nirik:matrix.scrye.com> !info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2024-07-25 16:07:54 <@nirik:matrix.scrye.com> !topic announcements and information
2024-07-25 16:07:54 <@nirik:matrix.scrye.com> !info Flock 2024 will happen from 7-11 Aug 2024. See you there!
2024-07-25 16:08:17 <@nirik:matrix.scrye.com> any other things to shout out from the top deck?
2024-07-25 16:09:31 <@nirik:matrix.scrye.com> !info https://fedoraproject.org/wiki/Infrastructure/Oncall
2024-07-25 16:09:31 <@nirik:matrix.scrye.com> !topic Oncall
2024-07-25 16:09:31 <@nirik:matrix.scrye.com> !info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
2024-07-25 16:09:34 <@nirik:matrix.scrye.com> !info ??? is on call from 2024-07-26 to 2024-08-01
2024-07-25 16:09:34 <@nirik:matrix.scrye.com> !info nirik is on call from 2024-07-19 to 2024-07-25
2024-07-25 16:09:47 <@nirik:matrix.scrye.com> anyone want oncall this coming week?
2024-07-25 16:09:57 <@Zlopez:matrix.org> I can take it
2024-07-25 16:10:27 <@nirik:matrix.scrye.com> great! thanks
2024-07-25 16:10:48 <@nirik:matrix.scrye.com> !info Summary of last week: (from current oncall)
2024-07-25 16:11:16 <@nirik:matrix.scrye.com> I saw one oncall about badges web being down... but it got handled before I got up
2024-07-25 16:11:59 <@nirik:matrix.scrye.com> thats all I can recall...
2024-07-25 16:12:09 <@nirik:matrix.scrye.com> !topic Monitoring discussion [nirik]
2024-07-25 16:12:09 <@nirik:matrix.scrye.com> !info Go over existing out items and fix
2024-07-25 16:12:09 <@nirik:matrix.scrye.com> !info https://nagios.fedoraproject.org/nagios 
2024-07-25 16:12:37 <@nirik:matrix.scrye.com> the usual ones that are down...
2024-07-25 16:12:51 <@nirik:matrix.scrye.com> a few small new ones just alerted... nothing major
2024-07-25 16:13:07 <@nirik:matrix.scrye.com> all looks pretty normal...
2024-07-25 16:13:17 <@Zlopez:matrix.org> I'm trying to fix the alerts as they show up :-)
2024-07-25 16:13:41 <@nirik:matrix.scrye.com> excellent, thats good
2024-07-25 16:14:30 <@nirik:matrix.scrye.com> ok, so shall we do a quick rhel7 checkin?
2024-07-25 16:14:36 <@nirik:matrix.scrye.com> I wanted to talk about flock as well if we have time
2024-07-25 16:14:49 <@Zlopez:matrix.org> We can do the rhel7 checkin
2024-07-25 16:15:07 <@Zlopez:matrix.org> There are 3 tasks that still are not finished
2024-07-25 16:15:07 <@nirik:matrix.scrye.com> !info Check the progress on tickets related to RHEL7 EOL (end of June)
2024-07-25 16:15:07 <@nirik:matrix.scrye.com> !topic RHEL7 EOL
2024-07-25 16:15:07 <@nirik:matrix.scrye.com> !ticket https://pagure.io/fedora-infrastructure/issue/11815
2024-07-25 16:15:32 <@nirik:matrix.scrye.com> so, pdc is super close now. We figured out that there shouldn't be anything using it anymore and I stopped httpd on it this morning...
2024-07-25 16:16:11 <@Zlopez:matrix.org> We will watch for errors if they show up somewhere
2024-07-25 16:16:38 <@nirik:matrix.scrye.com> yeah, I figure if nothing shows up by tomorrow I will retire it from ansible/vm's... will keep data around in case we need to bring it back.
2024-07-25 16:16:52 <@Zlopez:matrix.org> !ticket 11820
2024-07-25 16:16:53 <@zodbot:fedora.im> **fedora-infrastructure #11820** (https://pagure.io/fedora-infrastructure/issue/11820):**rhel7 - PDC**
2024-07-25 16:16:53 <@zodbot:fedora.im> ● **Opened:** 4 months ago by zlopez
2024-07-25 16:16:53 <@zodbot:fedora.im> 
2024-07-25 16:16:53 <@zodbot:fedora.im> ● **Assignee:** humaton
2024-07-25 16:16:53 <@zodbot:fedora.im> ● **Last Updated:** a week ago
2024-07-25 16:17:23 <@Zlopez:matrix.org> I should do the cleaning of toddlers as well
2024-07-25 16:17:31 <@naraiank:fedora.im> Can I join RHEL team as a testing engineer?
2024-07-25 16:17:46 <@naraiank:fedora.im> Sorry for coming in the middle.
2024-07-25 16:18:14 <@nirik:matrix.scrye.com> The Exorcist: well... we don't produce or test RHEL... :) Thats all internal to Red Hat. You can test CentOS stream if you like tho...
2024-07-25 16:18:55 <@Zlopez:matrix.org> The CentOS Stream will be definitely happy for more testers :-)
2024-07-25 16:19:16 <@naraiank:fedora.im> OK
2024-07-25 16:19:22 <@nirik:matrix.scrye.com> so on the 2 virthosts: 1) ibiblio05,they were going to check the mgmt connection, but I never heard back, so I pinged them again yesterday. 2) on rdu virthost the switch ports are set now, but no link, so I am asking on site people to check the cables... not sure when they will next be there for that. ;(
2024-07-25 16:19:42 <@naraiank:fedora.im> Do we have a channel to discuss RHEL things?
2024-07-25 16:20:08 <@nirik:matrix.scrye.com> Not that I know of...
2024-07-25 16:20:29 <@naraiank:fedora.im> OK
2024-07-25 16:21:22 <@Zlopez:matrix.org> You can find centos-stream on matrix as well
2024-07-25 16:21:57 <@Zlopez:matrix.org> ibiblio and rdu virthost are still running on RHEL7?
2024-07-25 16:21:58 <@naraiank:fedora.im> I will def'ly check
2024-07-25 16:22:04 <@nirik:matrix.scrye.com> on fedimg, I emailed david duncan asking for help with the cloud-image-uploader permissions. I think he may be traveling/on pto tho.
2024-07-25 16:22:08 <@smooge:fedora.im> there is? I only ever found the one on automotive and alt images
2024-07-25 16:22:28 <@Zlopez:matrix.org> !ticket 11803
2024-07-25 16:22:29 <@zodbot:fedora.im> 
2024-07-25 16:22:29 <@zodbot:fedora.im> ● **Assignee:** humaton
2024-07-25 16:22:29 <@zodbot:fedora.im> ● **Last Updated:** a month ago
2024-07-25 16:22:29 <@zodbot:fedora.im> ● **Opened:** 5 months ago by zlopez
2024-07-25 16:22:29 <@zodbot:fedora.im> **fedora-infrastructure #11803** (https://pagure.io/fedora-infrastructure/issue/11803):**rhel7 EOL - Fedimg**
2024-07-25 16:23:47 <@nirik:matrix.scrye.com> ibiblio05 and vmhost-x86-cc02 are yes. If this goes on too much longer I can power them off... ibibli05 has a proxy and ns02 that I could move off, but the other one has no guests currently I don't think.
2024-07-25 16:24:16 <@Zlopez:matrix.org> Stephen J Smoogen: Maybe they are still only on libera.chat, I saw few channels, but I don't see any centos-stream official
2024-07-25 16:25:34 <@Zlopez:matrix.org> nirik: It would be probably good to get rid of them
2024-07-25 16:25:44 <@Zlopez:matrix.org> If they couldn't be updated
2024-07-25 16:25:58 <@nirik:matrix.scrye.com> I know... I was hoping I could get to mgmt on them so I can reinstall them. ;)
2024-07-25 16:26:10 <@nirik:matrix.scrye.com> it's just been a slow process waiting for others.
2024-07-25 16:26:56 <@Zlopez:matrix.org> The Exorcist: Found this, that could be helpful https://docs.centos.org/en-US/stream-contrib/
2024-07-25 16:26:58 <@nirik:matrix.scrye.com> then that leavs fedmsg and github2fedmsg
2024-07-25 16:27:25 <@Zlopez:matrix.org> The fedmsg will go away when PDC, github2fedmsg and fedimg go away
2024-07-25 16:27:36 <@nirik:matrix.scrye.com> yep. the last one out the door.
2024-07-25 16:27:52 <@Zlopez:matrix.org> The github2fedmsg is moving forwards, hopefully there will be something on the staging soon
2024-07-25 16:28:14 <@Zlopez:matrix.org> There is a project created already, but nothing is deployed there yet
2024-07-25 16:28:37 <@Zlopez:matrix.org> I think that is all for RHEL7
2024-07-25 16:28:41 <@nirik:matrix.scrye.com> ok, yep...
2024-07-25 16:28:47 <@naraiank:fedora.im> +1, Thanks
2024-07-25 16:28:59 <@nirik:matrix.scrye.com> !topic flock 2024 planning
2024-07-25 16:29:19 <@nirik:matrix.scrye.com> So, I added to the discussion thread some suggested topics for our infra and releng workshop.
2024-07-25 16:29:36 <@Zlopez:matrix.org> I added few as well today
2024-07-25 16:29:44 <@nirik:matrix.scrye.com> Everyone do chime in on which things you think are important
2024-07-25 16:30:02 <@nirik:matrix.scrye.com> yeah, I saw... that in turn made me think of a few more too. ;) will reply with them later today
2024-07-25 16:30:20 <@nirik:matrix.scrye.com> we likely cannot get to everything. we will have to pick and choose and possibly timebox things.
2024-07-25 16:30:28 <@Zlopez:matrix.org> It would be probably best to share link to thread here :-)
2024-07-25 16:30:35 <@nirik:matrix.scrye.com> yeah, getting it
2024-07-25 16:30:49 <@nirik:matrix.scrye.com> https://discussion.fedoraproject.org/t/planning-for-infra-and-releng-hackfest-at-flock-2024/110244/13
2024-07-25 16:30:56 <@Zlopez:matrix.org> I will be happy to help with workshop itself
2024-07-25 16:31:35 <@nirik:matrix.scrye.com> !info lots of ideas for the workshop, please chime in in discussion if you have thoughts on them
2024-07-25 16:32:04 <@nirik:matrix.scrye.com> great. I think we probibly also need to get someone to take notes as we go... so we don't forget things / can share them later with those not there.
2024-07-25 16:32:38 <@nirik:matrix.scrye.com> possibly just a hackmd or something and get multiple people to help with the notes. ;)
2024-07-25 16:32:49 <@Zlopez:matrix.org> I was just wondering, the future of matrix server what is that about?
2024-07-25 16:33:21 <@nirik:matrix.scrye.com> well, it's all still being discussed... there was talk about moving from EMS hosting for us to us running the server...
2024-07-25 16:33:37 <@nirik:matrix.scrye.com> so I guess that could be dropped, since we don't really know what is going to happen
2024-07-25 16:33:41 <@Zlopez:matrix.org> Ok
2024-07-25 16:34:15 <@nirik:matrix.scrye.com> One addition on there is going to be the metrics server stuff that just was approved. Hopefully it all runs in openshift and isn't not too much work to setup...
2024-07-25 16:34:38 <@smooge:fedora.im> EMS hosting?
2024-07-25 16:34:45 <@Zlopez:matrix.org> That is the fedora metrics server?
2024-07-25 16:34:56 <@Zlopez:matrix.org> Or infra metrics?
2024-07-25 16:35:04 <@nirik:matrix.scrye.com> element matrix services. They host fedora.im and fedoraproject.org matrix servers for us. we (fedora) pay them to.
2024-07-25 16:35:25 <@smooge:fedora.im> The GNOME/Endless Metrics and Telemetry services
2024-07-25 16:35:34 <@smooge:fedora.im> nirik: thanks
2024-07-25 16:35:37 <@nirik:matrix.scrye.com> https://fedoraproject.org/wiki/Changes/Metrics
2024-07-25 16:35:58 <@nirik:matrix.scrye.com> The code / server already exists, but we would need to help them run a fedora one.
2024-07-25 16:36:28 <@Zlopez:matrix.org> That is probably better then write it from scratch
2024-07-25 16:36:56 <@nirik:matrix.scrye.com> https://github.com/endlessm/azafea
2024-07-25 16:37:22 <@nirik:matrix.scrye.com> https://github.com/endlessm/azafea-metrics-proxy
2024-07-25 16:37:56 <@nirik:matrix.scrye.com> anyhow, you can barely see that from the crows nest with a good spyglass. But we should start planning / prelim on it.
2024-07-25 16:38:40 <@nirik:matrix.scrye.com> !info Zlopez also has a state of fedora infra talk at flock.
2024-07-25 16:39:06 <@Zlopez:matrix.org> It will be some kind of summary what fedora infrastructure does :-)
2024-07-25 16:39:09 <@nirik:matrix.scrye.com> !info lenkaseg and David Kirwan have a communishift talk
2024-07-25 16:39:50 <@Zlopez:matrix.org> We still have some things missing regarding communishift
2024-07-25 16:39:54 <@nirik:matrix.scrye.com> !info jednorozec has a fedora-infra projects process talk
2024-07-25 16:40:04 <@nirik:matrix.scrye.com> Oh? whats missing?
2024-07-25 16:40:23 <@Zlopez:matrix.org> We don't have the script for cleaning the projects yet
2024-07-25 16:40:37 <@nirik:matrix.scrye.com> ah yes.
2024-07-25 16:40:38 <@Zlopez:matrix.org> But that is on our side
2024-07-25 16:41:09 <@nirik:matrix.scrye.com> but it should just be... 'oc delete project --whatever' right? well, and remove from ansible
2024-07-25 16:41:38 <@nirik:matrix.scrye.com> I guess it might need to delete the group/users too. I don't recall if that just syncs or not.
2024-07-25 16:42:04 <@Zlopez:matrix.org> Yeah, but we need to also document that and have some process to ask the owners of the project before this happens
2024-07-25 16:42:18 <@nirik:matrix.scrye.com> Which reminds me: we have 2 tickets for 'please don't delete this every cycle right now' and 1 ticket for 'we should make a cleanup script'.
2024-07-25 16:42:34 <@nirik:matrix.scrye.com> Can we just add those 2 projects as a note to the 'create cleanup script' ticket and close them?
2024-07-25 16:42:36 <@Zlopez:matrix.org> There is an operator for syncing the groups, but I'm not sure if it's removing anything
2024-07-25 16:43:00 <@Zlopez:matrix.org> nirik: I know about the tickets :-)
2024-07-25 16:43:21 <@Zlopez:matrix.org> Asked David Kirwan to work on the cleanup script
2024-07-25 16:43:39 <@nirik:matrix.scrye.com> the 2 'please don't delete this project' ones seem... like nothing that we can do anything about until there is a script... and the person working on the script would I hope read the ticket about that.
2024-07-25 16:44:37 <@Zlopez:matrix.org> He is already aware
2024-07-25 16:44:55 <@nirik:matrix.scrye.com> so, why do we have the tickets open taking up cognative space. ;)
2024-07-25 16:44:59 <@Zlopez:matrix.org> We will not delete anything without letting the project owners know about that
2024-07-25 16:45:31 <@Zlopez:matrix.org> We should probably just mention them in the cleanup script ticket, I will take care of that
2024-07-25 16:45:44 <@nirik:matrix.scrye.com> thanks.
2024-07-25 16:46:14 <@nirik:matrix.scrye.com> ok, any other flock discussion / ideas / notes? ;)
2024-07-25 16:46:46 <@nirik:matrix.scrye.com> shall we just move to open floor? or do we want to try and discuss anything else?
2024-07-25 16:49:43 <@Zlopez:matrix.org> Maybe we can add cleaning procedure for communishift as something that we can discuss on workshop
2024-07-25 16:50:08 <@nirik:matrix.scrye.com> sure. Might be nice to hash out the warning times/notifications/etc at least
2024-07-25 16:50:39 <@Zlopez:matrix.org> Yes, what would be best for time for tenants to know
2024-07-25 16:51:02 <@Zlopez:matrix.org> The cleaning will be best done at the EOL of previous Fedora release
2024-07-25 16:51:37 <@nirik:matrix.scrye.com> yeah, seems reasonable
2024-07-25 16:52:03 <@Zlopez:matrix.org> I think we can spend the last 10 minutes on open floor
2024-07-25 16:52:12 <@Zlopez:matrix.org> Hopefully we will not fall through it :-D
2024-07-25 16:52:15 <@nirik:matrix.scrye.com> !topic Open Floor
2024-07-25 16:52:57 <@nirik:matrix.scrye.com> Oh, also FYI, the next on site visit for IAD2 datacenter is the week of aug 5th. (of course the same week as flock)
2024-07-25 16:53:35 <@Zlopez:matrix.org> Oh
2024-07-25 16:53:51 <@Zlopez:matrix.org> I will be already traveling
2024-07-25 16:54:02 <@nirik:matrix.scrye.com> but we don't have too much pending. I will as usual write up a doc for them...
2024-07-25 16:54:56 <@nirik:matrix.scrye.com> also something to note is that our ticket queue has been seadily increasing for a while. ;(
2024-07-25 16:55:31 <@nirik:matrix.scrye.com> https://pagure.io/fedora-infrastructure/stats#issues
2024-07-25 16:55:40 <@Zlopez:matrix.org> I have few tickets open and trying to close them
2024-07-25 16:55:59 <@nirik:matrix.scrye.com> yeah. it's hard... always a pile coming in and sometimes they take a really long time to finish off.
2024-07-25 16:56:28 <@Zlopez:matrix.org> We should probably do a backlog refinement, as there could be probably few we can close
2024-07-25 16:56:50 <@nirik:matrix.scrye.com> yeah. next time perhaps?
2024-07-25 16:57:09 <@Zlopez:matrix.org> Next time sounds good :-)
2024-07-25 16:57:19 <@nirik:matrix.scrye.com> ok, if nothing else, we can ⛵ off into the 🌇
2024-07-25 16:57:49 <@nirik:matrix.scrye.com> Thanks for sailing with us today.
2024-07-25 16:57:52 <@nirik:matrix.scrye.com> !endmeeting