<@nirik:matrix.scrye.com>
16:00:08
!startmeeting Infrastructure (2024-07-25)
<@meetbot:fedora.im>
16:00:10
Meeting started at 2024-07-25 16:00:08 UTC
<@meetbot:fedora.im>
16:00:10
The Meeting name is 'Infrastructure (2024-07-25)'
<@nirik:matrix.scrye.com>
16:00:14
!chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
<@nirik:matrix.scrye.com>
16:00:14
!info Agenda is at: https://board.net/p/fedora-infra
<@nirik:matrix.scrye.com>
16:00:14
!info About our team: https://docs.fedoraproject.org/en-US/cpe/
<@nirik:matrix.scrye.com>
16:00:14
!info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
<@nirik:matrix.scrye.com>
16:00:14
!topic ahoy
<@nirik:matrix.scrye.com>
16:00:14
!meetingname infrastructure
<@meetbot:fedora.im>
16:00:15
The Meeting Name is now infrastructure
<@nirik:matrix.scrye.com>
16:00:50
welcome aboard everyone.
<@seddik:fedora.im>
16:00:56
.Hi
<@pcreech:matrix.org>
16:01:09
!hello pcreech17
<@zodbot:fedora.im>
16:01:11
Patrick Creech (pcreech17)
<@jnsamyak:matrix.org>
16:01:15
!hi
<@zodbot:fedora.im>
16:01:17
Samyak Jain (jnsamyak) - he / him / his
<@Zlopez:matrix.org>
16:01:20
!hi
<@seddik:fedora.im>
16:01:23
!hi
<@zodbot:fedora.im>
16:01:24
seddik alaouiismaili (seddik)
<@zodbot:fedora.im>
16:01:28
Michal Konecny (zlopez)
<@nirik:matrix.scrye.com>
16:02:42
lets wait a bell for more to come aboard
<@nirik:matrix.scrye.com>
16:03:33
(history aside: https://en.wikipedia.org/wiki/Ship's_bell )
<@nirik:matrix.scrye.com>
16:03:49
!info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
<@nirik:matrix.scrye.com>
16:03:49
!topic New folks introductions
<@nirik:matrix.scrye.com>
16:03:49
!info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
<@nirik:matrix.scrye.com>
16:04:01
any new crew to report in/
<@nirik:matrix.scrye.com>
16:04:04
any new crew to report in?
<@nirik:matrix.scrye.com>
16:04:55
seems not... lets sail on
<@nirik:matrix.scrye.com>
16:05:11
!topic Next chair
<@nirik:matrix.scrye.com>
16:05:11
!info chair 2024-08-15 - ???
<@nirik:matrix.scrye.com>
16:05:11
!info chair 2024-08-08 - cancelled (Flock)
<@nirik:matrix.scrye.com>
16:05:11
!info chair 2024-08-01 pcreech
<@nirik:matrix.scrye.com>
16:05:11
!info magic eight ball says:
<@nirik:matrix.scrye.com>
16:05:30
anyone want to take the captainship on the 15th?
<@nirik:matrix.scrye.com>
16:07:22
I guess we can decide next week...
<@nirik:matrix.scrye.com>
16:07:54
!info Remaining issues with new mailman3 deployment are being investigated
<@nirik:matrix.scrye.com>
16:07:54
!info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
<@nirik:matrix.scrye.com>
16:07:54
!info Fedora 41 Mass rebuild is done
<@nirik:matrix.scrye.com>
16:07:54
!info Fedora Infra weekly meeting is cancelled for 2024-08-08 because of Flock 2024
<@nirik:matrix.scrye.com>
16:07:54
!info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
<@nirik:matrix.scrye.com>
16:07:54
!topic announcements and information
<@nirik:matrix.scrye.com>
16:07:54
!info Flock 2024 will happen from 7-11 Aug 2024. See you there!
<@nirik:matrix.scrye.com>
16:08:17
any other things to shout out from the top deck?
<@nirik:matrix.scrye.com>
16:09:31
!info https://fedoraproject.org/wiki/Infrastructure/Oncall
<@nirik:matrix.scrye.com>
16:09:31
!topic Oncall
<@nirik:matrix.scrye.com>
16:09:31
!info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
<@nirik:matrix.scrye.com>
16:09:34
!info ??? is on call from 2024-07-26 to 2024-08-01
<@nirik:matrix.scrye.com>
16:09:34
!info nirik is on call from 2024-07-19 to 2024-07-25
<@nirik:matrix.scrye.com>
16:09:47
anyone want oncall this coming week?
<@Zlopez:matrix.org>
16:09:57
I can take it
<@nirik:matrix.scrye.com>
16:10:27
great! thanks
<@nirik:matrix.scrye.com>
16:10:48
!info Summary of last week: (from current oncall)
<@nirik:matrix.scrye.com>
16:11:16
I saw one oncall about badges web being down... but it got handled before I got up
<@nirik:matrix.scrye.com>
16:11:59
thats all I can recall...
<@nirik:matrix.scrye.com>
16:12:09
!topic Monitoring discussion [nirik]
<@nirik:matrix.scrye.com>
16:12:09
!info Go over existing out items and fix
<@nirik:matrix.scrye.com>
16:12:09
!info https://nagios.fedoraproject.org/nagios
<@nirik:matrix.scrye.com>
16:12:37
the usual ones that are down...
<@nirik:matrix.scrye.com>
16:12:51
a few small new ones just alerted... nothing major
<@nirik:matrix.scrye.com>
16:13:07
all looks pretty normal...
<@Zlopez:matrix.org>
16:13:17
I'm trying to fix the alerts as they show up :-)
<@nirik:matrix.scrye.com>
16:13:41
excellent, thats good
<@nirik:matrix.scrye.com>
16:14:30
ok, so shall we do a quick rhel7 checkin?
<@nirik:matrix.scrye.com>
16:14:36
I wanted to talk about flock as well if we have time
<@Zlopez:matrix.org>
16:14:49
We can do the rhel7 checkin
<@Zlopez:matrix.org>
16:15:07
There are 3 tasks that still are not finished
<@nirik:matrix.scrye.com>
16:15:07
!info Check the progress on tickets related to RHEL7 EOL (end of June)
<@nirik:matrix.scrye.com>
16:15:07
!topic RHEL7 EOL
<@nirik:matrix.scrye.com>
16:15:07
!ticket https://pagure.io/fedora-infrastructure/issue/11815
<@nirik:matrix.scrye.com>
16:15:32
so, pdc is super close now. We figured out that there shouldn't be anything using it anymore and I stopped httpd on it this morning...
<@Zlopez:matrix.org>
16:16:11
We will watch for errors if they show up somewhere
<@nirik:matrix.scrye.com>
16:16:38
yeah, I figure if nothing shows up by tomorrow I will retire it from ansible/vm's... will keep data around in case we need to bring it back.
<@Zlopez:matrix.org>
16:16:52
!ticket 11820
<@zodbot:fedora.im>
16:16:53
**fedora-infrastructure #11820** (https://pagure.io/fedora-infrastructure/issue/11820):**rhel7 - PDC**
<@zodbot:fedora.im>
16:16:53
● **Opened:** 4 months ago by zlopez
<@zodbot:fedora.im>
16:16:53
<@zodbot:fedora.im>
16:16:53
● **Assignee:** humaton
<@zodbot:fedora.im>
16:16:53
● **Last Updated:** a week ago
<@Zlopez:matrix.org>
16:17:23
I should do the cleaning of toddlers as well
<@naraiank:fedora.im>
16:17:31
Can I join RHEL team as a testing engineer?
<@naraiank:fedora.im>
16:17:46
Sorry for coming in the middle.
<@nirik:matrix.scrye.com>
16:18:14
The Exorcist: well... we don't produce or test RHEL... :) Thats all internal to Red Hat. You can test CentOS stream if you like tho...
<@Zlopez:matrix.org>
16:18:55
The CentOS Stream will be definitely happy for more testers :-)
<@naraiank:fedora.im>
16:19:16
OK
<@nirik:matrix.scrye.com>
16:19:22
so on the 2 virthosts: 1) ibiblio05,they were going to check the mgmt connection, but I never heard back, so I pinged them again yesterday. 2) on rdu virthost the switch ports are set now, but no link, so I am asking on site people to check the cables... not sure when they will next be there for that. ;(
<@naraiank:fedora.im>
16:19:42
Do we have a channel to discuss RHEL things?
<@nirik:matrix.scrye.com>
16:20:08
Not that I know of...
<@naraiank:fedora.im>
16:20:29
OK
<@Zlopez:matrix.org>
16:21:22
You can find centos-stream on matrix as well
<@Zlopez:matrix.org>
16:21:57
ibiblio and rdu virthost are still running on RHEL7?
<@naraiank:fedora.im>
16:21:58
I will def'ly check
<@nirik:matrix.scrye.com>
16:22:04
on fedimg, I emailed david duncan asking for help with the cloud-image-uploader permissions. I think he may be traveling/on pto tho.
<@smooge:fedora.im>
16:22:08
there is? I only ever found the one on automotive and alt images
<@Zlopez:matrix.org>
16:22:28
!ticket 11803
<@zodbot:fedora.im>
16:22:29
<@zodbot:fedora.im>
16:22:29
● **Assignee:** humaton
<@zodbot:fedora.im>
16:22:29
● **Last Updated:** a month ago
<@zodbot:fedora.im>
16:22:29
● **Opened:** 5 months ago by zlopez
<@zodbot:fedora.im>
16:22:29
**fedora-infrastructure #11803** (https://pagure.io/fedora-infrastructure/issue/11803):**rhel7 EOL - Fedimg**
<@nirik:matrix.scrye.com>
16:23:47
ibiblio05 and vmhost-x86-cc02 are yes. If this goes on too much longer I can power them off... ibibli05 has a proxy and ns02 that I could move off, but the other one has no guests currently I don't think.
<@Zlopez:matrix.org>
16:24:16
Stephen J Smoogen: Maybe they are still only on libera.chat, I saw few channels, but I don't see any centos-stream official
<@Zlopez:matrix.org>
16:25:34
nirik: It would be probably good to get rid of them
<@Zlopez:matrix.org>
16:25:44
If they couldn't be updated
<@nirik:matrix.scrye.com>
16:25:58
I know... I was hoping I could get to mgmt on them so I can reinstall them. ;)
<@nirik:matrix.scrye.com>
16:26:10
it's just been a slow process waiting for others.
<@Zlopez:matrix.org>
16:26:56
The Exorcist: Found this, that could be helpful https://docs.centos.org/en-US/stream-contrib/
<@nirik:matrix.scrye.com>
16:26:58
then that leavs fedmsg and github2fedmsg
<@Zlopez:matrix.org>
16:27:25
The fedmsg will go away when PDC, github2fedmsg and fedimg go away
<@nirik:matrix.scrye.com>
16:27:36
yep. the last one out the door.
<@Zlopez:matrix.org>
16:27:52
The github2fedmsg is moving forwards, hopefully there will be something on the staging soon
<@Zlopez:matrix.org>
16:28:14
There is a project created already, but nothing is deployed there yet
<@Zlopez:matrix.org>
16:28:37
I think that is all for RHEL7
<@nirik:matrix.scrye.com>
16:28:41
ok, yep...
<@naraiank:fedora.im>
16:28:47
+1, Thanks
<@nirik:matrix.scrye.com>
16:28:59
!topic flock 2024 planning
<@nirik:matrix.scrye.com>
16:29:19
So, I added to the discussion thread some suggested topics for our infra and releng workshop.
<@Zlopez:matrix.org>
16:29:36
I added few as well today
<@nirik:matrix.scrye.com>
16:29:44
Everyone do chime in on which things you think are important
<@nirik:matrix.scrye.com>
16:30:02
yeah, I saw... that in turn made me think of a few more too. ;) will reply with them later today
<@nirik:matrix.scrye.com>
16:30:20
we likely cannot get to everything. we will have to pick and choose and possibly timebox things.
<@Zlopez:matrix.org>
16:30:28
It would be probably best to share link to thread here :-)
<@nirik:matrix.scrye.com>
16:30:35
yeah, getting it
<@nirik:matrix.scrye.com>
16:30:49
https://discussion.fedoraproject.org/t/planning-for-infra-and-releng-hackfest-at-flock-2024/110244/13
<@Zlopez:matrix.org>
16:30:56
I will be happy to help with workshop itself
<@nirik:matrix.scrye.com>
16:31:35
!info lots of ideas for the workshop, please chime in in discussion if you have thoughts on them
<@nirik:matrix.scrye.com>
16:32:04
great. I think we probibly also need to get someone to take notes as we go... so we don't forget things / can share them later with those not there.
<@nirik:matrix.scrye.com>
16:32:38
possibly just a hackmd or something and get multiple people to help with the notes. ;)
<@Zlopez:matrix.org>
16:32:49
I was just wondering, the future of matrix server what is that about?
<@nirik:matrix.scrye.com>
16:33:21
well, it's all still being discussed... there was talk about moving from EMS hosting for us to us running the server...
<@nirik:matrix.scrye.com>
16:33:37
so I guess that could be dropped, since we don't really know what is going to happen
<@Zlopez:matrix.org>
16:33:41
Ok
<@nirik:matrix.scrye.com>
16:34:15
One addition on there is going to be the metrics server stuff that just was approved. Hopefully it all runs in openshift and isn't not too much work to setup...
<@smooge:fedora.im>
16:34:38
EMS hosting?
<@Zlopez:matrix.org>
16:34:45
That is the fedora metrics server?
<@Zlopez:matrix.org>
16:34:56
Or infra metrics?
<@nirik:matrix.scrye.com>
16:35:04
element matrix services. They host fedora.im and fedoraproject.org matrix servers for us. we (fedora) pay them to.
<@smooge:fedora.im>
16:35:25
The GNOME/Endless Metrics and Telemetry services
<@smooge:fedora.im>
16:35:34
nirik: thanks
<@nirik:matrix.scrye.com>
16:35:37
https://fedoraproject.org/wiki/Changes/Metrics
<@nirik:matrix.scrye.com>
16:35:58
The code / server already exists, but we would need to help them run a fedora one.
<@Zlopez:matrix.org>
16:36:28
That is probably better then write it from scratch
<@nirik:matrix.scrye.com>
16:36:56
https://github.com/endlessm/azafea
<@nirik:matrix.scrye.com>
16:37:22
https://github.com/endlessm/azafea-metrics-proxy
<@nirik:matrix.scrye.com>
16:37:56
anyhow, you can barely see that from the crows nest with a good spyglass. But we should start planning / prelim on it.
<@nirik:matrix.scrye.com>
16:38:40
!info Zlopez also has a state of fedora infra talk at flock.
<@Zlopez:matrix.org>
16:39:06
It will be some kind of summary what fedora infrastructure does :-)
<@nirik:matrix.scrye.com>
16:39:09
!info lenkaseg and David Kirwan have a communishift talk
<@Zlopez:matrix.org>
16:39:50
We still have some things missing regarding communishift
<@nirik:matrix.scrye.com>
16:39:54
!info jednorozec has a fedora-infra projects process talk
<@nirik:matrix.scrye.com>
16:40:04
Oh? whats missing?
<@Zlopez:matrix.org>
16:40:23
We don't have the script for cleaning the projects yet
<@nirik:matrix.scrye.com>
16:40:37
ah yes.
<@Zlopez:matrix.org>
16:40:38
But that is on our side
<@nirik:matrix.scrye.com>
16:41:09
but it should just be... 'oc delete project --whatever' right? well, and remove from ansible
<@nirik:matrix.scrye.com>
16:41:38
I guess it might need to delete the group/users too. I don't recall if that just syncs or not.
<@Zlopez:matrix.org>
16:42:04
Yeah, but we need to also document that and have some process to ask the owners of the project before this happens
<@nirik:matrix.scrye.com>
16:42:18
Which reminds me: we have 2 tickets for 'please don't delete this every cycle right now' and 1 ticket for 'we should make a cleanup script'.
<@nirik:matrix.scrye.com>
16:42:34
Can we just add those 2 projects as a note to the 'create cleanup script' ticket and close them?
<@Zlopez:matrix.org>
16:42:36
There is an operator for syncing the groups, but I'm not sure if it's removing anything
<@Zlopez:matrix.org>
16:43:00
nirik: I know about the tickets :-)
<@Zlopez:matrix.org>
16:43:21
Asked David Kirwan to work on the cleanup script
<@nirik:matrix.scrye.com>
16:43:39
the 2 'please don't delete this project' ones seem... like nothing that we can do anything about until there is a script... and the person working on the script would I hope read the ticket about that.
<@Zlopez:matrix.org>
16:44:37
He is already aware
<@nirik:matrix.scrye.com>
16:44:55
so, why do we have the tickets open taking up cognative space. ;)
<@Zlopez:matrix.org>
16:44:59
We will not delete anything without letting the project owners know about that
<@Zlopez:matrix.org>
16:45:31
We should probably just mention them in the cleanup script ticket, I will take care of that
<@nirik:matrix.scrye.com>
16:45:44
thanks.
<@nirik:matrix.scrye.com>
16:46:14
ok, any other flock discussion / ideas / notes? ;)
<@nirik:matrix.scrye.com>
16:46:46
shall we just move to open floor? or do we want to try and discuss anything else?
<@Zlopez:matrix.org>
16:49:43
Maybe we can add cleaning procedure for communishift as something that we can discuss on workshop
<@nirik:matrix.scrye.com>
16:50:08
sure. Might be nice to hash out the warning times/notifications/etc at least
<@Zlopez:matrix.org>
16:50:39
Yes, what would be best for time for tenants to know
<@Zlopez:matrix.org>
16:51:02
The cleaning will be best done at the EOL of previous Fedora release
<@nirik:matrix.scrye.com>
16:51:37
yeah, seems reasonable
<@Zlopez:matrix.org>
16:52:03
I think we can spend the last 10 minutes on open floor
<@Zlopez:matrix.org>
16:52:12
Hopefully we will not fall through it :-D
<@nirik:matrix.scrye.com>
16:52:15
!topic Open Floor
<@nirik:matrix.scrye.com>
16:52:57
Oh, also FYI, the next on site visit for IAD2 datacenter is the week of aug 5th. (of course the same week as flock)
<@Zlopez:matrix.org>
16:53:35
Oh
<@Zlopez:matrix.org>
16:53:51
I will be already traveling
<@nirik:matrix.scrye.com>
16:54:02
but we don't have too much pending. I will as usual write up a doc for them...
<@nirik:matrix.scrye.com>
16:54:56
also something to note is that our ticket queue has been seadily increasing for a while. ;(
<@nirik:matrix.scrye.com>
16:55:31
https://pagure.io/fedora-infrastructure/stats#issues
<@Zlopez:matrix.org>
16:55:40
I have few tickets open and trying to close them
<@nirik:matrix.scrye.com>
16:55:59
yeah. it's hard... always a pile coming in and sometimes they take a really long time to finish off.
<@Zlopez:matrix.org>
16:56:28
We should probably do a backlog refinement, as there could be probably few we can close
<@nirik:matrix.scrye.com>
16:56:50
yeah. next time perhaps?
<@Zlopez:matrix.org>
16:57:09
Next time sounds good :-)
<@nirik:matrix.scrye.com>
16:57:19
ok, if nothing else, we can ⛵ off into the 🌇
<@nirik:matrix.scrye.com>
16:57:49
Thanks for sailing with us today.
<@nirik:matrix.scrye.com>
16:57:52
!endmeeting