<@nirik:matrix.scrye.com>
16:00:55
!startmeeting Infrastructure (2024-06-27)
<@meetbot:fedora.im>
16:00:57
Meeting started at 2024-06-27 16:00:55 UTC
<@meetbot:fedora.im>
16:00:57
The Meeting name is 'Infrastructure (2024-06-27)'
<@smilner:fedora.im>
16:01:04
!hi
<@zodbot:fedora.im>
16:01:05
None (smilner)
<@nirik:matrix.scrye.com>
16:01:12
!info Agenda is at: https://board.net/p/fedora-infra
<@nirik:matrix.scrye.com>
16:01:12
!info About our team: https://docs.fedoraproject.org/en-US/cpe/
<@nirik:matrix.scrye.com>
16:01:12
!topic namaste
<@nirik:matrix.scrye.com>
16:01:12
!info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
<@nirik:matrix.scrye.com>
16:01:12
!meetingname infrastructure
<@nirik:matrix.scrye.com>
16:01:12
!chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
<@meetbot:fedora.im>
16:01:13
The Meeting Name is now infrastructure
<@nirik:matrix.scrye.com>
16:01:23
Morning everyone!
<@nirik:matrix.scrye.com>
16:02:07
lets wait a few for folks to wander in
<@Zlopez:matrix.org>
16:03:05
!hi
<@zodbot:fedora.im>
16:03:07
Michal Konecny (zlopez)
<@james:fedora.im>
16:04:25
!hi
<@zodbot:fedora.im>
16:04:26
James Antill (james)
<@nirik:matrix.scrye.com>
16:05:04
👋
<@nirik:matrix.scrye.com>
16:05:13
!info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
<@nirik:matrix.scrye.com>
16:05:13
!info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
<@nirik:matrix.scrye.com>
16:05:13
!topic New folks introductions
<@nirik:matrix.scrye.com>
16:05:19
any new folks here today?
<@nirik:matrix.scrye.com>
16:06:14
seems not...
<@nirik:matrix.scrye.com>
16:06:15
!info chair 2024-06-27 nirik
<@nirik:matrix.scrye.com>
16:06:15
!info chair 2024-07-18 ?
<@nirik:matrix.scrye.com>
16:06:15
!info chair 2024-07-11 pcreech
<@nirik:matrix.scrye.com>
16:06:15
!info chair 2024-07-04 cancelled due to holidays
<@nirik:matrix.scrye.com>
16:06:15
!info magic eight ball says:
<@nirik:matrix.scrye.com>
16:06:15
!topic Next chair
<@nirik:matrix.scrye.com>
16:06:41
anyone interested in the 18th chair? or shall we just wait and do it later?
<@Zlopez:matrix.org>
16:07:32
I can take it, I should be arround
<@nirik:matrix.scrye.com>
16:07:38
ok, lets just do it on the 11th. ;)
<@nirik:matrix.scrye.com>
16:07:48
ok, sold.
<@nirik:matrix.scrye.com>
16:08:02
thanks
<@nirik:matrix.scrye.com>
16:08:03
!info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
<@nirik:matrix.scrye.com>
16:08:03
!topic announcements and information
<@nirik:matrix.scrye.com>
16:08:03
!info Fedora infra meeting is cancelled for 2024-07-04 due to US Holidays
<@nirik:matrix.scrye.com>
16:08:03
!info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
<@nirik:matrix.scrye.com>
16:08:29
any other announcements?
<@nirik:matrix.scrye.com>
16:08:29
!info fedorapeople / fedoraplanet migration later today (21utc)
<@smooge:fedora.im>
16:09:04
!info mailman3 upgrade seems to be going well
<@nirik:matrix.scrye.com>
16:09:28
!info mailman is migrated to rhel9! hurray!
<@Zlopez:matrix.org>
16:10:39
Needs some polish, but it's finally done
<@nirik:matrix.scrye.com>
16:11:10
The new interface has some nice features too.
<@nirik:matrix.scrye.com>
16:11:16
Anyhow, no further announcements, lets move on
<@nirik:matrix.scrye.com>
16:11:35
!topic Oncall
<@nirik:matrix.scrye.com>
16:11:35
!info https://fedoraproject.org/wiki/Infrastructure/Oncall
<@nirik:matrix.scrye.com>
16:11:35
!info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
<@nirik:matrix.scrye.com>
16:11:39
!info ? is on call from 2024-07-05 to 2024-07-11
<@nirik:matrix.scrye.com>
16:11:39
!info nirik is on call from 2024-06-28 to 2024-07-04
<@nirik:matrix.scrye.com>
16:11:39
!info zlopez is on call from 2024-06-21 to 2024-06-27
<@nirik:matrix.scrye.com>
16:11:51
anyone want 5th to 11th?
<@nirik:matrix.scrye.com>
16:13:07
no biggie
<@nirik:matrix.scrye.com>
16:13:15
!info Summary of last week: (from current oncall)
<@nirik:matrix.scrye.com>
16:13:34
anything this last week Zlopez ?
<@Zlopez:matrix.org>
16:16:00
I didn't saw any ping
<@nirik:matrix.scrye.com>
16:16:16
ok, cool.
<@nirik:matrix.scrye.com>
16:16:28
!info https://nagios.fedoraproject.org/nagios
<@nirik:matrix.scrye.com>
16:16:28
!topic Monitoring discussion [nirik]
<@nirik:matrix.scrye.com>
16:16:48
So, we have buildhw-x86-16 still down, there's a buildvm thats down that I need to reinstall.
<@nirik:matrix.scrye.com>
16:17:01
There's some proxies in APAC alerting due to network packet loss.
<@nirik:matrix.scrye.com>
16:17:14
There's a few mailman alerts (api and no messages)
<@nirik:matrix.scrye.com>
16:17:24
and thats about it. Nothing really stands out otherwise.
<@nirik:matrix.scrye.com>
16:18:10
I acked the mailman ones, Zlopez will look at them tomorrow
<@nirik:matrix.scrye.com>
16:18:41
so... shall we do a quick rhel7 roundup? Or backlog? or end early?
<@Zlopez:matrix.org>
16:18:54
I saw some proxies instability
<@Zlopez:matrix.org>
16:19:14
I would go with rhel7
<@nirik:matrix.scrye.com>
16:19:19
sure
<@nirik:matrix.scrye.com>
16:19:26
!ticket https://pagure.io/fedora-infrastructure/issue/11815
<@nirik:matrix.scrye.com>
16:19:26
!info Check the progress on tickets related to RHEL7 EOL (end of June)
<@nirik:matrix.scrye.com>
16:19:26
!topic RHEL7 EOL
<@nirik:matrix.scrye.com>
16:19:55
so, lets see... if mailman is done and I will be doing people later today...
<@nirik:matrix.scrye.com>
16:20:21
we have left:
<@nirik:matrix.scrye.com>
16:21:06
pdc
<@nirik:matrix.scrye.com>
16:21:20
I think jednorozec was getting pagure-dist-git finished
<@Zlopez:matrix.org>
16:21:49
Yeah, he said he will be working on it today
<@nirik:matrix.scrye.com>
16:21:54
but there's a few smaller issues after that
<@humaton:fedora.im>
16:22:12
so the release is out there I am just deploying it into staging
<@nirik:matrix.scrye.com>
16:22:40
cool. so this removes the pdc check for git pushes?
<@nirik:matrix.scrye.com>
16:23:30
so that leaves fedfind and block_retired I guess.
<@humaton:fedora.im>
16:23:47
yes block retired is being done by Anton Medvedev
<@nirik:matrix.scrye.com>
16:24:08
Do we know the status there?
<@humaton:fedora.im>
16:24:22
and fedfind is another thing, I tried to extract the endpoints from PDC I am about 80% there
<@humaton:fedora.im>
16:24:35
not sure If I can make it before the end of month
<@nirik:matrix.scrye.com>
16:25:11
where would the endpoints live? and whats in them? ;) If you recall...
<@humaton:fedora.im>
16:25:15
he is going to schedule call for tomorrow to see if we can get it working by Monday
<@nirik:matrix.scrye.com>
16:25:32
ok
<@humaton:fedora.im>
16:25:43
Its the nvr<->releaseid<->release relation
<@humaton:fedora.im>
16:25:47
and few msaller ones
<@humaton:fedora.im>
16:26:20
I would like it to live in openshift since its django app extracted from PDC and updated to latest django
<@humaton:fedora.im>
16:26:36
let me create a public repository I have it just privately
<@nirik:matrix.scrye.com>
16:26:52
oof... another app? ;(
<@humaton:fedora.im>
16:27:28
well moving it into bodhi was one idea, but its get complicated because bodhi only knows some builds not all of them
<@nirik:matrix.scrye.com>
16:27:34
I guess we talked with adamw and he needed this and there was no other way? ;(
<@humaton:fedora.im>
16:28:02
this is the easiest, about 200 lines of python, I will make it nice
<@nirik:matrix.scrye.com>
16:28:31
yeah, but another app is ðŸ˜
<@humaton:fedora.im>
16:28:37
If we dont change the processes too much it should not require much attention and if so I will be the maintainer
<@humaton:fedora.im>
16:28:41
yeah
<@humaton:fedora.im>
16:28:42
I know
<@humaton:fedora.im>
16:29:16
I can move it to compose tracker, or some other compose related app we run
<@humaton:fedora.im>
16:29:33
not sure how nice that will be... but its doable
<@nirik:matrix.scrye.com>
16:29:57
yeah, don't know... might be worth talking to adam again to see if any new thoughts have come up
<@humaton:fedora.im>
16:30:24
ack
<@nirik:matrix.scrye.com>
16:30:36
anyhow, hopefully we can get there pretty soon on pdc
<@nirik:matrix.scrye.com>
16:30:46
next sundries...
<@nirik:matrix.scrye.com>
16:31:14
Zlopez: you got it all working in stg? or at least deploying?
<@nirik:matrix.scrye.com>
16:31:19
would you like me to roll it to prod tomorrow/later today? or you want to ? or ?
<@Zlopez:matrix.org>
16:31:41
I was able to package everything that was needed to run the playbook, but I didn't get to check if everything is working. I didn't saw any error reported by cronjobs hovewer
<@nirik:matrix.scrye.com>
16:32:25
I think mailman, pdc and fedorapeople are the big ones as they are externally facing... fedimg, sundries, and github2fedmsg are all more internal/not visible
<@Zlopez:matrix.org>
16:32:28
The playbook is now running through, I just disabled few things in playbook as they were no longer needed
<@nirik:matrix.scrye.com>
16:32:40
right, it's also hard to test because we point to 01 normally, so 02 likely doesn't get any traffic.
<@nirik:matrix.scrye.com>
16:33:01
we could redo 01 in stg, that should give more info
<@Zlopez:matrix.org>
16:33:16
That should be easy now
<@Zlopez:matrix.org>
16:33:39
I just don't have time right now to do it
<@nirik:matrix.scrye.com>
16:34:06
yeah, I am happy to after I finish up with fedorapeople...
<@Zlopez:matrix.org>
16:34:25
It would be nice to have another app done
<@nirik:matrix.scrye.com>
16:35:01
fedimg is being worked on. I need to poke at some aws perms ... planning on that after the meeting.
<@nirik:matrix.scrye.com>
16:35:20
github2fedmsg isn't done, likely will need a quick port
<@nirik:matrix.scrye.com>
16:35:58
Aurélien B said he or ryanlerch could likely do that.
<@nirik:matrix.scrye.com>
16:36:31
So, we are close. Not going to make everything in time, but all the external facing ones hopefully and the rest soon afer
<@nirik:matrix.scrye.com>
16:36:35
So, we are close. Not going to make everything in time, but all the external facing ones hopefully and the rest soon after
<@nirik:matrix.scrye.com>
16:37:18
any I missed or status to add?
<@Zlopez:matrix.org>
16:37:56
Mirrormanager ticket is still open
<@Zlopez:matrix.org>
16:38:11
Otherwise you went through everything
<@nirik:matrix.scrye.com>
16:38:47
yeah, the old rhel7 vm's are still running and it's still in ansible. I was waiting for the go-head to drop those.
<@Zlopez:matrix.org>
16:39:01
OK
<@nirik:matrix.scrye.com>
16:39:19
oh, there's also 2 virthosts still rhel7, but I am blocked on them by no access to mgmt... I have tickets in in both cases asking for things to get fixed.
<@Zlopez:matrix.org>
16:39:36
After this I plan to do some cleaning in IPA and mailman roles
<@nirik:matrix.scrye.com>
16:39:55
oh, we also have fedmsg still...
<@Zlopez:matrix.org>
16:40:24
After the RHEL7 EOL task finished we can get rid of fedmsg
<@nirik:matrix.scrye.com>
16:40:33
yeah, it shouldn't be urgent...
<@nirik:matrix.scrye.com>
16:40:46
Here's the full list of rhel7 vm's currently left:
<@nirik:matrix.scrye.com>
16:40:53
people02.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:40:53
busgateway01.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:40:53
busgateway01.stg.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:40:53
fedimg01.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:40:53
github2fedmsg01.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:40:53
github2fedmsg01.stg.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:40:53
ibiblio05.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:40:53
mm-backend01.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:40:53
mm-backend01.stg.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:40:53
mm-crawler01.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:40:53
mm-crawler01.stg.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:40:53
mm-crawler02.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:40:53
mm-frontend01.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:40:53
mm-frontend01.stg.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:40:53
mm-frontend-checkin01.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:40:53
pdc-web01.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:40:53
pdc-web01.stg.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:40:53
pdc-web02.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:40:53
sundries01.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:40:53
sundries01.stg.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:40:53
sundries02.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:40:53
virthost-cc-rdu02.fedoraproject.org 7.9
<@Zlopez:matrix.org>
16:41:27
Is ibiblio and virthost-cc-rdu02 in our power to migrate it?
<@nirik:matrix.scrye.com>
16:41:43
sundries02.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:41:43
sundries01.stg.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:41:43
sundries01.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:41:43
people02.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:41:43
pdc-web02.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:41:43
pdc-web01.stg.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:41:43
pdc-web01.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:41:43
mm-frontend-checkin01.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:41:43
virthost-cc-rdu02.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:41:43
``` busgateway01.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:41:43
busgateway01.stg.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:41:43
fedimg01.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:41:43
github2fedmsg01.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:41:43
github2fedmsg01.stg.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:41:43
ibiblio05.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:41:43
mm-backend01.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:41:43
mm-backend01.stg.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:41:43
mm-crawler01.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:41:43
mm-crawler01.stg.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:41:43
mm-crawler02.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:41:43
mm-frontend01.stg.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:41:43
mm-frontend01.iad2.fedoraproject.org 7.9
<@nirik:matrix.scrye.com>
16:42:03
Those are the 2 virthosts I mentioned above.
<@nirik:matrix.scrye.com>
16:42:06
I can't reach the mgmt on either currently, needs some onsite work
<@nirik:matrix.scrye.com>
16:43:40
Anyhow, anything else? shall we move to open floor?
<@Zlopez:matrix.org>
16:44:03
Nothing from my side
<@nirik:matrix.scrye.com>
16:45:32
!topic open floor
<@Zlopez:matrix.org>
16:45:42
I have to leave now I have a dog here that wants to urgently go out
<@Zlopez:matrix.org>
16:45:56
See you
<@nirik:matrix.scrye.com>
16:46:06
night!
<@nirik:matrix.scrye.com>
16:46:13
ok, if nothing else we can just close out...
<@nirik:matrix.scrye.com>
16:46:28
!endmeeting