2024-06-27 16:00:55 <@nirik:matrix.scrye.com> !startmeeting Infrastructure (2024-06-27)
2024-06-27 16:00:57 <@meetbot:fedora.im> Meeting started at 2024-06-27 16:00:55 UTC
2024-06-27 16:00:57 <@meetbot:fedora.im> The Meeting name is 'Infrastructure (2024-06-27)'
2024-06-27 16:01:04 <@smilner:fedora.im> !hi
2024-06-27 16:01:05 <@zodbot:fedora.im> None (smilner)
2024-06-27 16:01:12 <@nirik:matrix.scrye.com> !info Agenda is at: https://board.net/p/fedora-infra
2024-06-27 16:01:12 <@nirik:matrix.scrye.com> !info About our team: https://docs.fedoraproject.org/en-US/cpe/
2024-06-27 16:01:12 <@nirik:matrix.scrye.com> !topic namaste
2024-06-27 16:01:12 <@nirik:matrix.scrye.com> !info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
2024-06-27 16:01:12 <@nirik:matrix.scrye.com> !meetingname infrastructure
2024-06-27 16:01:12 <@nirik:matrix.scrye.com> !chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
2024-06-27 16:01:13 <@meetbot:fedora.im> The Meeting Name is now infrastructure
2024-06-27 16:01:23 <@nirik:matrix.scrye.com> Morning everyone!
2024-06-27 16:02:07 <@nirik:matrix.scrye.com> lets wait a few for folks to wander in
2024-06-27 16:03:05 <@Zlopez:matrix.org> !hi
2024-06-27 16:03:07 <@zodbot:fedora.im> Michal Konecny (zlopez)
2024-06-27 16:04:25 <@james:fedora.im> !hi
2024-06-27 16:04:26 <@zodbot:fedora.im> James Antill (james)
2024-06-27 16:05:04 <@nirik:matrix.scrye.com> 👋
2024-06-27 16:05:13 <@nirik:matrix.scrye.com> !info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
2024-06-27 16:05:13 <@nirik:matrix.scrye.com> !info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
2024-06-27 16:05:13 <@nirik:matrix.scrye.com> !topic New folks introductions
2024-06-27 16:05:19 <@nirik:matrix.scrye.com> any new folks here today?
2024-06-27 16:06:14 <@nirik:matrix.scrye.com> seems not...
2024-06-27 16:06:15 <@nirik:matrix.scrye.com> !info chair 2024-06-27 nirik
2024-06-27 16:06:15 <@nirik:matrix.scrye.com> !info chair 2024-07-18 ?
2024-06-27 16:06:15 <@nirik:matrix.scrye.com> !info chair 2024-07-11 pcreech
2024-06-27 16:06:15 <@nirik:matrix.scrye.com> !info chair 2024-07-04 cancelled due to holidays
2024-06-27 16:06:15 <@nirik:matrix.scrye.com> !info magic eight ball says:
2024-06-27 16:06:15 <@nirik:matrix.scrye.com> !topic Next chair
2024-06-27 16:06:41 <@nirik:matrix.scrye.com> anyone interested in the 18th chair? or shall we just wait and do it later?
2024-06-27 16:07:32 <@Zlopez:matrix.org> I can take it, I should be arround
2024-06-27 16:07:38 <@nirik:matrix.scrye.com> ok, lets just do it on the 11th. ;)
2024-06-27 16:07:48 <@nirik:matrix.scrye.com> ok, sold.
2024-06-27 16:08:02 <@nirik:matrix.scrye.com> thanks
2024-06-27 16:08:03 <@nirik:matrix.scrye.com> !info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2024-06-27 16:08:03 <@nirik:matrix.scrye.com> !topic announcements and information
2024-06-27 16:08:03 <@nirik:matrix.scrye.com> !info Fedora infra meeting is cancelled for 2024-07-04 due to US Holidays
2024-06-27 16:08:03 <@nirik:matrix.scrye.com> !info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
2024-06-27 16:08:29 <@nirik:matrix.scrye.com> any other announcements?
2024-06-27 16:08:29 <@nirik:matrix.scrye.com> !info fedorapeople / fedoraplanet migration later today (21utc)
2024-06-27 16:09:04 <@smooge:fedora.im> !info mailman3 upgrade seems to be going well
2024-06-27 16:09:28 <@nirik:matrix.scrye.com> !info mailman is migrated to rhel9! hurray!
2024-06-27 16:10:39 <@Zlopez:matrix.org> Needs some polish, but it's finally done
2024-06-27 16:11:10 <@nirik:matrix.scrye.com> The new interface has some nice features too.
2024-06-27 16:11:16 <@nirik:matrix.scrye.com> Anyhow, no further announcements, lets move on
2024-06-27 16:11:35 <@nirik:matrix.scrye.com> !topic Oncall
2024-06-27 16:11:35 <@nirik:matrix.scrye.com> !info https://fedoraproject.org/wiki/Infrastructure/Oncall
2024-06-27 16:11:35 <@nirik:matrix.scrye.com> !info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
2024-06-27 16:11:39 <@nirik:matrix.scrye.com> !info ? is on call from 2024-07-05 to 2024-07-11
2024-06-27 16:11:39 <@nirik:matrix.scrye.com> !info nirik is on call from 2024-06-28 to 2024-07-04
2024-06-27 16:11:39 <@nirik:matrix.scrye.com> !info zlopez is on call from 2024-06-21 to 2024-06-27
2024-06-27 16:11:51 <@nirik:matrix.scrye.com> anyone want 5th to 11th?
2024-06-27 16:13:07 <@nirik:matrix.scrye.com> no biggie
2024-06-27 16:13:15 <@nirik:matrix.scrye.com> !info Summary of last week: (from current oncall)
2024-06-27 16:13:34 <@nirik:matrix.scrye.com> anything this last week Zlopez ?
2024-06-27 16:16:00 <@Zlopez:matrix.org> I didn't saw any ping
2024-06-27 16:16:16 <@nirik:matrix.scrye.com> ok, cool.
2024-06-27 16:16:28 <@nirik:matrix.scrye.com> !info https://nagios.fedoraproject.org/nagios
2024-06-27 16:16:28 <@nirik:matrix.scrye.com> !topic Monitoring discussion [nirik]
2024-06-27 16:16:48 <@nirik:matrix.scrye.com> So, we have buildhw-x86-16 still down, there's a buildvm thats down that I need to reinstall.
2024-06-27 16:17:01 <@nirik:matrix.scrye.com> There's some proxies in APAC alerting due to network packet loss.
2024-06-27 16:17:14 <@nirik:matrix.scrye.com> There's a few mailman alerts (api and no messages)
2024-06-27 16:17:24 <@nirik:matrix.scrye.com> and thats about it. Nothing really stands out otherwise.
2024-06-27 16:18:10 <@nirik:matrix.scrye.com> I acked the mailman ones, Zlopez will look at them tomorrow
2024-06-27 16:18:41 <@nirik:matrix.scrye.com> so... shall we do a quick rhel7 roundup? Or backlog? or end early?
2024-06-27 16:18:54 <@Zlopez:matrix.org> I saw some proxies instability
2024-06-27 16:19:14 <@Zlopez:matrix.org> I would go with rhel7
2024-06-27 16:19:19 <@nirik:matrix.scrye.com> sure
2024-06-27 16:19:26 <@nirik:matrix.scrye.com> !ticket https://pagure.io/fedora-infrastructure/issue/11815
2024-06-27 16:19:26 <@nirik:matrix.scrye.com> !info Check the progress on tickets related to RHEL7 EOL (end of June)
2024-06-27 16:19:26 <@nirik:matrix.scrye.com> !topic RHEL7 EOL
2024-06-27 16:19:55 <@nirik:matrix.scrye.com> so, lets see... if mailman is done and I will be doing people later today...
2024-06-27 16:20:21 <@nirik:matrix.scrye.com> we have left:
2024-06-27 16:21:06 <@nirik:matrix.scrye.com> pdc
2024-06-27 16:21:20 <@nirik:matrix.scrye.com> I think jednorozec was getting pagure-dist-git finished
2024-06-27 16:21:49 <@Zlopez:matrix.org> Yeah, he said he will be working on it today
2024-06-27 16:21:54 <@nirik:matrix.scrye.com> but there's a few smaller issues after that
2024-06-27 16:22:12 <@humaton:fedora.im> so the release is out there I am just deploying it into staging
2024-06-27 16:22:40 <@nirik:matrix.scrye.com> cool. so this removes the pdc check for git pushes?
2024-06-27 16:23:30 <@nirik:matrix.scrye.com> so that leaves fedfind and block_retired I guess.
2024-06-27 16:23:47 <@humaton:fedora.im> yes block retired is being done by Anton Medvedev 
2024-06-27 16:24:08 <@nirik:matrix.scrye.com> Do we know the status there?
2024-06-27 16:24:22 <@humaton:fedora.im> and fedfind is another thing, I tried to extract the endpoints from PDC I am about 80% there
2024-06-27 16:24:35 <@humaton:fedora.im> not sure If I can make it before the end of month
2024-06-27 16:25:11 <@nirik:matrix.scrye.com> where would the endpoints live? and whats in them? ;) If you recall...
2024-06-27 16:25:15 <@humaton:fedora.im> he is going to schedule call for tomorrow to see if we can get it working by Monday
2024-06-27 16:25:32 <@nirik:matrix.scrye.com> ok
2024-06-27 16:25:43 <@humaton:fedora.im> Its the nvr<->releaseid<->release relation
2024-06-27 16:25:47 <@humaton:fedora.im> and few msaller ones
2024-06-27 16:26:20 <@humaton:fedora.im> I would like it to live in openshift since its django app extracted from PDC and updated to latest django
2024-06-27 16:26:36 <@humaton:fedora.im> let me create a public repository I have it just privately
2024-06-27 16:26:52 <@nirik:matrix.scrye.com> oof... another app? ;(
2024-06-27 16:27:28 <@humaton:fedora.im> well moving it into bodhi was one idea, but its get complicated because bodhi only knows some builds not all of them
2024-06-27 16:27:34 <@nirik:matrix.scrye.com> I guess we talked with adamw and he needed this and there was no other way? ;(
2024-06-27 16:28:02 <@humaton:fedora.im> this is the easiest, about 200 lines of python, I will make it nice
2024-06-27 16:28:31 <@nirik:matrix.scrye.com> yeah, but another app is 😭
2024-06-27 16:28:37 <@humaton:fedora.im> If we dont change the processes too much it should not require much attention and if so I will be the maintainer
2024-06-27 16:28:41 <@humaton:fedora.im> yeah
2024-06-27 16:28:42 <@humaton:fedora.im> I know
2024-06-27 16:29:16 <@humaton:fedora.im> I can move it to compose tracker, or some other compose related app we run
2024-06-27 16:29:33 <@humaton:fedora.im> not sure how nice that will be... but its doable
2024-06-27 16:29:57 <@nirik:matrix.scrye.com> yeah, don't know... might be worth talking to adam again to see if any new thoughts have come up
2024-06-27 16:30:24 <@humaton:fedora.im> ack
2024-06-27 16:30:36 <@nirik:matrix.scrye.com> anyhow, hopefully we can get there pretty soon on pdc
2024-06-27 16:30:46 <@nirik:matrix.scrye.com> next sundries...
2024-06-27 16:31:14 <@nirik:matrix.scrye.com> Zlopez: you got it all working in stg? or at least deploying?
2024-06-27 16:31:19 <@nirik:matrix.scrye.com> would you like me to roll it to prod tomorrow/later today? or you want to ? or ?
2024-06-27 16:31:41 <@Zlopez:matrix.org> I was able to package everything that was needed to run the playbook, but I didn't get to check if everything is working. I didn't saw any error reported by cronjobs hovewer
2024-06-27 16:32:25 <@nirik:matrix.scrye.com> I think mailman, pdc and fedorapeople are the big ones as they are externally facing... fedimg, sundries, and github2fedmsg are all more internal/not visible
2024-06-27 16:32:28 <@Zlopez:matrix.org> The playbook is now running through, I just disabled few things in playbook as they were no longer needed
2024-06-27 16:32:40 <@nirik:matrix.scrye.com> right, it's also hard to test because we point to 01 normally, so 02 likely doesn't get any traffic.
2024-06-27 16:33:01 <@nirik:matrix.scrye.com> we could redo 01 in stg, that should give more info
2024-06-27 16:33:16 <@Zlopez:matrix.org> That should be easy now
2024-06-27 16:33:39 <@Zlopez:matrix.org> I just don't have time right now to do it
2024-06-27 16:34:06 <@nirik:matrix.scrye.com> yeah, I am happy to after I finish up with fedorapeople...
2024-06-27 16:34:25 <@Zlopez:matrix.org> It would be nice to have another app done
2024-06-27 16:35:01 <@nirik:matrix.scrye.com> fedimg is being worked on. I need to poke at some aws perms ... planning on that after the meeting.
2024-06-27 16:35:20 <@nirik:matrix.scrye.com> github2fedmsg isn't done, likely will need a quick port
2024-06-27 16:35:58 <@nirik:matrix.scrye.com> Aurélien B said he or ryanlerch could likely do that.
2024-06-27 16:36:31 <@nirik:matrix.scrye.com> So, we are close. Not going to make everything in time, but all the external facing ones hopefully and the rest soon afer
2024-06-27 16:36:35 <@nirik:matrix.scrye.com> So, we are close. Not going to make everything in time, but all the external facing ones hopefully and the rest soon after
2024-06-27 16:37:18 <@nirik:matrix.scrye.com> any I missed or status to add?
2024-06-27 16:37:56 <@Zlopez:matrix.org> Mirrormanager ticket is still open
2024-06-27 16:38:11 <@Zlopez:matrix.org> Otherwise you went through everything
2024-06-27 16:38:47 <@nirik:matrix.scrye.com> yeah, the old rhel7 vm's are still running and it's still in ansible. I was waiting for the go-head to drop those.
2024-06-27 16:39:01 <@Zlopez:matrix.org> OK
2024-06-27 16:39:19 <@nirik:matrix.scrye.com> oh, there's also 2 virthosts still rhel7, but I am blocked on them by no access to mgmt... I have tickets in in both cases asking for things to get fixed.
2024-06-27 16:39:36 <@Zlopez:matrix.org> After this I plan to do some cleaning in IPA and mailman roles
2024-06-27 16:39:55 <@nirik:matrix.scrye.com> oh, we also have fedmsg still...
2024-06-27 16:40:24 <@Zlopez:matrix.org> After the RHEL7 EOL task finished we can get rid of fedmsg
2024-06-27 16:40:33 <@nirik:matrix.scrye.com> yeah, it shouldn't be urgent...
2024-06-27 16:40:46 <@nirik:matrix.scrye.com> Here's the full list of rhel7 vm's currently left:
2024-06-27 16:40:53 <@nirik:matrix.scrye.com> people02.fedoraproject.org  7.9
2024-06-27 16:40:53 <@nirik:matrix.scrye.com> busgateway01.iad2.fedoraproject.org  7.9
2024-06-27 16:40:53 <@nirik:matrix.scrye.com> busgateway01.stg.iad2.fedoraproject.org  7.9
2024-06-27 16:40:53 <@nirik:matrix.scrye.com> fedimg01.iad2.fedoraproject.org  7.9
2024-06-27 16:40:53 <@nirik:matrix.scrye.com> github2fedmsg01.iad2.fedoraproject.org  7.9
2024-06-27 16:40:53 <@nirik:matrix.scrye.com> github2fedmsg01.stg.iad2.fedoraproject.org  7.9
2024-06-27 16:40:53 <@nirik:matrix.scrye.com> ibiblio05.fedoraproject.org  7.9
2024-06-27 16:40:53 <@nirik:matrix.scrye.com> mm-backend01.iad2.fedoraproject.org  7.9
2024-06-27 16:40:53 <@nirik:matrix.scrye.com> mm-backend01.stg.iad2.fedoraproject.org  7.9
2024-06-27 16:40:53 <@nirik:matrix.scrye.com> mm-crawler01.iad2.fedoraproject.org  7.9
2024-06-27 16:40:53 <@nirik:matrix.scrye.com> mm-crawler01.stg.iad2.fedoraproject.org  7.9
2024-06-27 16:40:53 <@nirik:matrix.scrye.com> mm-crawler02.iad2.fedoraproject.org  7.9
2024-06-27 16:40:53 <@nirik:matrix.scrye.com> mm-frontend01.iad2.fedoraproject.org  7.9
2024-06-27 16:40:53 <@nirik:matrix.scrye.com> mm-frontend01.stg.iad2.fedoraproject.org  7.9
2024-06-27 16:40:53 <@nirik:matrix.scrye.com> mm-frontend-checkin01.iad2.fedoraproject.org  7.9
2024-06-27 16:40:53 <@nirik:matrix.scrye.com> pdc-web01.iad2.fedoraproject.org  7.9
2024-06-27 16:40:53 <@nirik:matrix.scrye.com> pdc-web01.stg.iad2.fedoraproject.org  7.9
2024-06-27 16:40:53 <@nirik:matrix.scrye.com> pdc-web02.iad2.fedoraproject.org  7.9
2024-06-27 16:40:53 <@nirik:matrix.scrye.com> sundries01.iad2.fedoraproject.org  7.9
2024-06-27 16:40:53 <@nirik:matrix.scrye.com> sundries01.stg.iad2.fedoraproject.org  7.9
2024-06-27 16:40:53 <@nirik:matrix.scrye.com> sundries02.iad2.fedoraproject.org  7.9
2024-06-27 16:40:53 <@nirik:matrix.scrye.com> virthost-cc-rdu02.fedoraproject.org  7.9
2024-06-27 16:41:27 <@Zlopez:matrix.org> Is ibiblio and virthost-cc-rdu02 in our power to migrate it?
2024-06-27 16:41:43 <@nirik:matrix.scrye.com> sundries02.iad2.fedoraproject.org  7.9
2024-06-27 16:41:43 <@nirik:matrix.scrye.com> sundries01.stg.iad2.fedoraproject.org  7.9
2024-06-27 16:41:43 <@nirik:matrix.scrye.com> sundries01.iad2.fedoraproject.org  7.9
2024-06-27 16:41:43 <@nirik:matrix.scrye.com> people02.fedoraproject.org  7.9
2024-06-27 16:41:43 <@nirik:matrix.scrye.com> pdc-web02.iad2.fedoraproject.org  7.9
2024-06-27 16:41:43 <@nirik:matrix.scrye.com> pdc-web01.stg.iad2.fedoraproject.org  7.9
2024-06-27 16:41:43 <@nirik:matrix.scrye.com> pdc-web01.iad2.fedoraproject.org  7.9
2024-06-27 16:41:43 <@nirik:matrix.scrye.com> mm-frontend-checkin01.iad2.fedoraproject.org  7.9
2024-06-27 16:41:43 <@nirik:matrix.scrye.com> virthost-cc-rdu02.fedoraproject.org  7.9
2024-06-27 16:41:43 <@nirik:matrix.scrye.com> ``` busgateway01.iad2.fedoraproject.org  7.9
2024-06-27 16:41:43 <@nirik:matrix.scrye.com> busgateway01.stg.iad2.fedoraproject.org  7.9
2024-06-27 16:41:43 <@nirik:matrix.scrye.com> fedimg01.iad2.fedoraproject.org  7.9
2024-06-27 16:41:43 <@nirik:matrix.scrye.com> github2fedmsg01.iad2.fedoraproject.org  7.9
2024-06-27 16:41:43 <@nirik:matrix.scrye.com> github2fedmsg01.stg.iad2.fedoraproject.org  7.9
2024-06-27 16:41:43 <@nirik:matrix.scrye.com> ibiblio05.fedoraproject.org  7.9
2024-06-27 16:41:43 <@nirik:matrix.scrye.com> mm-backend01.iad2.fedoraproject.org  7.9
2024-06-27 16:41:43 <@nirik:matrix.scrye.com> mm-backend01.stg.iad2.fedoraproject.org  7.9
2024-06-27 16:41:43 <@nirik:matrix.scrye.com> mm-crawler01.iad2.fedoraproject.org  7.9
2024-06-27 16:41:43 <@nirik:matrix.scrye.com> mm-crawler01.stg.iad2.fedoraproject.org  7.9
2024-06-27 16:41:43 <@nirik:matrix.scrye.com> mm-crawler02.iad2.fedoraproject.org  7.9
2024-06-27 16:41:43 <@nirik:matrix.scrye.com> mm-frontend01.stg.iad2.fedoraproject.org  7.9
2024-06-27 16:41:43 <@nirik:matrix.scrye.com> mm-frontend01.iad2.fedoraproject.org  7.9
2024-06-27 16:42:03 <@nirik:matrix.scrye.com> Those are the 2 virthosts I mentioned above.
2024-06-27 16:42:06 <@nirik:matrix.scrye.com> I can't reach the mgmt on either currently, needs some onsite work
2024-06-27 16:43:40 <@nirik:matrix.scrye.com> Anyhow, anything else? shall we move to open floor?
2024-06-27 16:44:03 <@Zlopez:matrix.org> Nothing from my side
2024-06-27 16:45:32 <@nirik:matrix.scrye.com> !topic open floor
2024-06-27 16:45:42 <@Zlopez:matrix.org> I have to leave now I have a dog here that wants to urgently go out
2024-06-27 16:45:56 <@Zlopez:matrix.org> See you
2024-06-27 16:46:06 <@nirik:matrix.scrye.com> night!
2024-06-27 16:46:13 <@nirik:matrix.scrye.com> ok, if nothing else we can just close out...
2024-06-27 16:46:28 <@nirik:matrix.scrye.com> !endmeeting