2025-11-10 20:02:42 <@abompard:matrix.org> !startmeeting Fedora Infrastructure Ops Daily Standup Meeting
2025-11-10 20:02:44 <@meetbot:fedora.im> Meeting started at 2025-11-10 20:02:42 UTC
2025-11-10 20:02:45 <@meetbot:fedora.im> The Meeting name is 'Fedora Infrastructure Ops Daily Standup Meeting'
2025-11-10 20:02:47 <@james:fedora.im> !next
2025-11-10 20:02:48 <@abompard:matrix.org> 
2025-11-10 20:02:48 <@abompard:matrix.org> !info Reminder: Speak up if you want to work on a ticket!
2025-11-10 20:02:48 <@abompard:matrix.org> 
2025-11-10 20:02:48 <@abompard:matrix.org> !chair @nirik:matrix.scrye.com, @phsmoura:fedora.im, @james:fedora.im
2025-11-10 20:02:48 <@abompard:matrix.org> 
2025-11-10 20:02:48 <@abompard:matrix.org> !info invitees @smilner:fedora.im
2025-11-10 20:02:48 <@abompard:matrix.org> 
2025-11-10 20:02:48 <@abompard:matrix.org> !meetingname fedora_infrastructure_ops_daily_standup_meeting
2025-11-10 20:02:48 <@abompard:matrix.org> 
2025-11-10 20:02:48 <@abompard:matrix.org> !info Meeting is 30 minutes MAX. At the end of 30, its stops
2025-11-10 20:02:48 <@abompard:matrix.org> !info Agenda is at https://board.net/p/fedora-infra-daily
2025-11-10 20:02:49 <@meetbot:fedora.im> The Meeting Name is now fedora_infrastructure_ops_daily_standup_meeting
2025-11-10 20:02:54 <@james:fedora.im> Hey
2025-11-10 20:02:55 <@phsmoura:fedora.im> hello
2025-11-10 20:03:33 <@james:fedora.im> !next
2025-11-10 20:03:33 <@abompard:matrix.org> 
2025-11-10 20:03:33 <@abompard:matrix.org> !topic Tickets needing review
2025-11-10 20:03:33 <@abompard:matrix.org> Run `!ticket #<number>` for each ticket, discuss and classify. DON'T DO THIS FOR PRIVATE TICKETS.
2025-11-10 20:03:33 <@abompard:matrix.org> !info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1
2025-11-10 20:03:33 <@abompard:matrix.org> 
2025-11-10 20:03:39 <@james:fedora.im> !ticket 12901
2025-11-10 20:03:41 <@zodbot:fedora.im> ● **Opened:** a day ago by rathann
2025-11-10 20:03:41 <@zodbot:fedora.im> **fedora-infrastructure #12901** (https://pagure.io/fedora-infrastructure/issue/12901):**update stuck in "Pending" state**
2025-11-10 20:03:41 <@zodbot:fedora.im> 
2025-11-10 20:03:41 <@zodbot:fedora.im> ● **Last Updated:** an hour ago
2025-11-10 20:03:41 <@zodbot:fedora.im> ● **Assignee:** Not Assigned
2025-11-10 20:03:54 <@nirik:matrix.scrye.com> morning
2025-11-10 20:04:09 <@nirik:matrix.scrye.com> I don't think this is the right place for this ticket. it's a qe/ci problem...
2025-11-10 20:04:50 <@nirik:matrix.scrye.com> yeah, I am not fully sure where to ask them to re-file it tho? fedora-qa?
2025-11-10 20:05:04 <@james:fedora.im> Seems like a good guess.
2025-11-10 20:05:13 <@james:fedora.im> !ticket 12904
2025-11-10 20:05:16 <@zodbot:fedora.im> ● **Opened:** an hour ago by lecris
2025-11-10 20:05:16 <@zodbot:fedora.im> **fedora-infrastructure #12904** (https://pagure.io/fedora-infrastructure/issue/12904):**State of sourcegraph indexing or equivalent service**
2025-11-10 20:05:16 <@zodbot:fedora.im> ● **Last Updated:** Never
2025-11-10 20:05:16 <@zodbot:fedora.im> 
2025-11-10 20:05:16 <@zodbot:fedora.im> ● **Assignee:** Not Assigned
2025-11-10 20:06:19 <@nirik:matrix.scrye.com> low low ?
2025-11-10 20:06:31 <@nirik:matrix.scrye.com> anubis is likely blocking this, but there's some way to allow it I recall reading.
2025-11-10 20:07:16 <@james:fedora.im> We don't run sourcegraph, right?
2025-11-10 20:07:31 <@nirik:matrix.scrye.com> no
2025-11-10 20:09:04 <@james:fedora.im> !ticket 12905
2025-11-10 20:09:04 <@zodbot:fedora.im> ● **Last Updated:** Never
2025-11-10 20:09:04 <@zodbot:fedora.im> 
2025-11-10 20:09:04 <@zodbot:fedora.im> ● **Assignee:** Not Assigned
2025-11-10 20:09:04 <@zodbot:fedora.im> ● **Opened:** 42 minutes ago by kevin
2025-11-10 20:09:04 <@zodbot:fedora.im> **fedora-infrastructure #12905** (https://pagure.io/fedora-infrastructure/issue/12905):**adjust to docker-distribution 3.0.0 update**
2025-11-10 20:09:56 <@james:fedora.im> I guess low / med-gain ... maybe even high-gain?
2025-11-10 20:09:58 <@nirik:matrix.scrye.com> just filed this, hit it over the weekend. ;(
2025-11-10 20:10:08 <@james:fedora.im> What is the downside to running the giant chown?
2025-11-10 20:10:10 <@nirik:matrix.scrye.com> There's a workaround in place right now, but we should really fix iut
2025-11-10 20:10:12 <@nirik:matrix.scrye.com> There's a workaround in place right now, but we should really fix it
2025-11-10 20:10:36 <@nirik:matrix.scrye.com> it's about 25TB of data
2025-11-10 20:10:53 <@nirik:matrix.scrye.com> it would take a long time and result in a massive amount of churn
2025-11-10 20:10:57 <@nirik:matrix.scrye.com> but we could do it
2025-11-10 20:11:15 <@james:fedora.im> Are netapp snapshots going to go insane? Do we rsync it?
2025-11-10 20:11:52 <@james:fedora.im> Can we just chown the directories?
2025-11-10 20:12:03 <@nirik:matrix.scrye.com> I'm not sure on snapshots. They might?
2025-11-10 20:12:12 <@nirik:matrix.scrye.com> we dont rsync it but it gets snapmirrored.
2025-11-10 20:12:30 <@nirik:matrix.scrye.com> and no, because oci registries write a bunch of link files. ;(
2025-11-10 20:13:16 <@james:fedora.im> Ahh ... I would guess that chown'ing all the files would still share the data in some clever copy on write way, but that's a guess.
2025-11-10 20:13:42 <@james:fedora.im> I can put it as low / low ... given we have a workaround.
2025-11-10 20:14:53 <@nirik:matrix.scrye.com> I am not sure either
2025-11-10 20:15:08 <@nirik:matrix.scrye.com> I'm sad that this was pushed as a stable f42 update, but oh well.
2025-11-10 20:16:00 <@james:fedora.im> yeh
2025-11-10 20:17:00 <@james:fedora.im> !next
2025-11-10 20:17:01 <@abompard:matrix.org> 
2025-11-10 20:17:01 <@abompard:matrix.org> The command is `!releng #<number>`
2025-11-10 20:17:01 <@abompard:matrix.org> 
2025-11-10 20:17:01 <@abompard:matrix.org> Do the same for releng issues, only issues that don't have tags:
2025-11-10 20:17:01 <@abompard:matrix.org> !info https://pagure.io/releng/issues?status=Open
2025-11-10 20:17:05 <@james:fedora.im> !next
2025-11-10 20:17:06 <@abompard:matrix.org> !topic Planning, Upcoming work and Open floor
2025-11-10 20:18:35 <@phsmoura:fedora.im> updating the proxies here
2025-11-10 20:20:10 <@nirik:matrix.scrye.com> phsmoura: leave the 01/10/101/110 for me... I'm still trying to sort things on them
2025-11-10 20:20:18 <@nirik:matrix.scrye.com> but 01.stg anytime for sure
2025-11-10 20:21:08 <@nirik:matrix.scrye.com> so, for me... try and keep my sanity. :) I'm trying to figure out why bvmhost-p10-01 blew up. I can't find anything wrong with it's configuration. ;(
2025-11-10 20:21:37 <@james:fedora.im> Blah, Fri/Today kind of sucked ... but hopefully push something useful in the next couple of days. Probably nftables cleanup, so it can go out during the updates next week.
2025-11-10 20:21:41 <@nirik:matrix.scrye.com> networking is likely going to update our firewall tomorrow (I hope). Possibly helping/fixing the tcp timeout issue. This will result in about a 10-20second outage of everything.
2025-11-10 20:22:08 <@phsmoura:fedora.im> ok, np.. those will be the last ones
2025-11-10 20:22:09 <@phsmoura:fedora.im> should do stg and prod??
2025-11-10 20:22:14 <@nirik:matrix.scrye.com> James: I had a thought on updates next week... but have been swamped and not moved it forward:
2025-11-10 20:22:43 <@nirik:matrix.scrye.com> phsmoura: per the ticket 02.stg should already be done (I did it a while back when debugging). But you can do the aws ones for sure one by one
2025-11-10 20:23:27 <@james:fedora.im> Kind of related to updates ... did we file anything upstream about the anubis downgrades we had to do?
2025-11-10 20:23:34 <@nirik:matrix.scrye.com> for updates: fabian mentioned that he setup a http proxy for the mgmt devices... I think we perhaps could do the same (noc01?) to allow them to reach out to get firmware updates. Then we can have them download those and have them ready when we reboot.
2025-11-10 20:24:31 <@nirik:matrix.scrye.com> James: not yet. I started to file one saturday, but then I realized all the tools ones have been fixed on the tools sides... the only left issue is cloudfront -> kojipkgs for ostree, and we need to try and figure out how to write that up/change it in config
2025-11-10 20:25:30 <@nirik:matrix.scrye.com> BTW, there is also a newer anubis release. They fixed some problems...
2025-11-10 20:27:26 <@nirik:matrix.scrye.com> we can, but... we don't have cloudfront or kojipkgs in stg... so...
2025-11-10 20:27:32 <@nirik:matrix.scrye.com> ¯\\\_(ツ)\_/¯ no good
2025-11-10 20:28:47 <@james:fedora.im> yeh, that's what I was worried about ... plus the usage in staging isn't at the same level.
2025-11-10 20:29:28 <@james:fedora.im> And on that note ... we have 1 minute left, anything anyone wants to talk about?
2025-11-10 20:32:27 <@james:fedora.im> !endmeeting