#meeting-3:fedoraproject.org log

<@abompard:matrix.org>

20:02:42

!startmeeting Fedora Infrastructure Ops Daily Standup Meeting

<@meetbot:fedora.im>

20:02:44

Meeting started at 2025-11-10 20:02:42 UTC

<@meetbot:fedora.im>

20:02:45

The Meeting name is 'Fedora Infrastructure Ops Daily Standup Meeting'

<@james:fedora.im>

20:02:47

!next

<@abompard:matrix.org>

20:02:48

<@abompard:matrix.org>

20:02:48

!info Reminder: Speak up if you want to work on a ticket!

<@abompard:matrix.org>

20:02:48

<@abompard:matrix.org>

20:02:48

!chair @nirik:matrix.scrye.com, @phsmoura:fedora.im, @james:fedora.im

<@abompard:matrix.org>

20:02:48

<@abompard:matrix.org>

20:02:48

!info invitees @smilner:fedora.im

<@abompard:matrix.org>

20:02:48

<@abompard:matrix.org>

20:02:48

!meetingname fedora_infrastructure_ops_daily_standup_meeting

<@abompard:matrix.org>

20:02:48

<@abompard:matrix.org>

20:02:48

!info Meeting is 30 minutes MAX. At the end of 30, its stops

<@abompard:matrix.org>

20:02:48

!info Agenda is at https://board.net/p/fedora-infra-daily

<@meetbot:fedora.im>

20:02:49

The Meeting Name is now fedora_infrastructure_ops_daily_standup_meeting

<@james:fedora.im>

20:02:54

Hey

<@phsmoura:fedora.im>

20:02:55

hello

<@james:fedora.im>

20:03:33

!next

<@abompard:matrix.org>

20:03:33

<@abompard:matrix.org>

20:03:33

!topic Tickets needing review

<@abompard:matrix.org>

20:03:33

Run `!ticket #<number>` for each ticket, discuss and classify. DON'T DO THIS FOR PRIVATE TICKETS.

<@abompard:matrix.org>

20:03:33

!info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1

<@abompard:matrix.org>

20:03:33

<@james:fedora.im>

20:03:39

!ticket 12901

<@zodbot:fedora.im>

20:03:41

● **Opened:** a day ago by rathann

<@zodbot:fedora.im>

20:03:41

**fedora-infrastructure #12901** (https://pagure.io/fedora-infrastructure/issue/12901):**update stuck in "Pending" state**

<@zodbot:fedora.im>

20:03:41

<@zodbot:fedora.im>

20:03:41

● **Last Updated:** an hour ago

<@zodbot:fedora.im>

20:03:41

● **Assignee:** Not Assigned

<@nirik:matrix.scrye.com>

20:03:54

morning

<@nirik:matrix.scrye.com>

20:04:09

I don't think this is the right place for this ticket. it's a qe/ci problem...

<@nirik:matrix.scrye.com>

20:04:50

yeah, I am not fully sure where to ask them to re-file it tho? fedora-qa?

<@james:fedora.im>

20:05:04

Seems like a good guess.

<@james:fedora.im>

20:05:13

!ticket 12904

<@zodbot:fedora.im>

20:05:16

● **Opened:** an hour ago by lecris

<@zodbot:fedora.im>

20:05:16

**fedora-infrastructure #12904** (https://pagure.io/fedora-infrastructure/issue/12904):**State of sourcegraph indexing or equivalent service**

<@zodbot:fedora.im>

20:05:16

● **Last Updated:** Never

<@zodbot:fedora.im>

20:05:16

<@zodbot:fedora.im>

20:05:16

● **Assignee:** Not Assigned

<@nirik:matrix.scrye.com>

20:06:19

low low ?

<@nirik:matrix.scrye.com>

20:06:31

anubis is likely blocking this, but there's some way to allow it I recall reading.

<@james:fedora.im>

20:07:16

We don't run sourcegraph, right?

<@nirik:matrix.scrye.com>

20:07:31

<@james:fedora.im>

20:09:04

!ticket 12905

<@zodbot:fedora.im>

20:09:04

● **Last Updated:** Never

<@zodbot:fedora.im>

20:09:04

<@zodbot:fedora.im>

20:09:04

● **Assignee:** Not Assigned

<@zodbot:fedora.im>

20:09:04

● **Opened:** 42 minutes ago by kevin

<@zodbot:fedora.im>

20:09:04

**fedora-infrastructure #12905** (https://pagure.io/fedora-infrastructure/issue/12905):**adjust to docker-distribution 3.0.0 update**

<@james:fedora.im>

20:09:56

I guess low / med-gain ... maybe even high-gain?

<@nirik:matrix.scrye.com>

20:09:58

just filed this, hit it over the weekend. ;(

<@james:fedora.im>

20:10:08

What is the downside to running the giant chown?

<@nirik:matrix.scrye.com>

20:10:10

There's a workaround in place right now, but we should really fix iut

<@nirik:matrix.scrye.com>

20:10:12

There's a workaround in place right now, but we should really fix it

<@nirik:matrix.scrye.com>

20:10:36

it's about 25TB of data

<@nirik:matrix.scrye.com>

20:10:53

it would take a long time and result in a massive amount of churn

<@nirik:matrix.scrye.com>

20:10:57

but we could do it

<@james:fedora.im>

20:11:15

Are netapp snapshots going to go insane? Do we rsync it?

<@james:fedora.im>

20:11:52

Can we just chown the directories?

<@nirik:matrix.scrye.com>

20:12:03

I'm not sure on snapshots. They might?

<@nirik:matrix.scrye.com>

20:12:12

we dont rsync it but it gets snapmirrored.

<@nirik:matrix.scrye.com>

20:12:30

and no, because oci registries write a bunch of link files. ;(

<@james:fedora.im>

20:13:16

Ahh ... I would guess that chown'ing all the files would still share the data in some clever copy on write way, but that's a guess.

<@james:fedora.im>

20:13:42

I can put it as low / low ... given we have a workaround.

<@nirik:matrix.scrye.com>

20:14:53

I am not sure either

<@nirik:matrix.scrye.com>

20:15:08

I'm sad that this was pushed as a stable f42 update, but oh well.

<@james:fedora.im>

20:16:00

yeh

<@james:fedora.im>

20:17:00

!next

<@abompard:matrix.org>

20:17:01

<@abompard:matrix.org>

20:17:01

The command is `!releng #<number>`

<@abompard:matrix.org>

20:17:01

<@abompard:matrix.org>

20:17:01

Do the same for releng issues, only issues that don't have tags:

<@abompard:matrix.org>

20:17:01

!info https://pagure.io/releng/issues?status=Open

<@james:fedora.im>

20:17:05

!next

<@abompard:matrix.org>

20:17:06

!topic Planning, Upcoming work and Open floor

<@phsmoura:fedora.im>

20:18:35

updating the proxies here

<@nirik:matrix.scrye.com>

20:20:10

phsmoura: leave the 01/10/101/110 for me... I'm still trying to sort things on them

<@nirik:matrix.scrye.com>

20:20:18

but 01.stg anytime for sure

<@nirik:matrix.scrye.com>

20:21:08

so, for me... try and keep my sanity. :) I'm trying to figure out why bvmhost-p10-01 blew up. I can't find anything wrong with it's configuration. ;(

<@james:fedora.im>

20:21:37

Blah, Fri/Today kind of sucked ... but hopefully push something useful in the next couple of days. Probably nftables cleanup, so it can go out during the updates next week.

<@nirik:matrix.scrye.com>

20:21:41

networking is likely going to update our firewall tomorrow (I hope). Possibly helping/fixing the tcp timeout issue. This will result in about a 10-20second outage of everything.

<@phsmoura:fedora.im>

20:22:08

ok, np.. those will be the last ones

<@phsmoura:fedora.im>

20:22:09

should do stg and prod??

<@nirik:matrix.scrye.com>

20:22:14

James: I had a thought on updates next week... but have been swamped and not moved it forward:

<@nirik:matrix.scrye.com>

20:22:43

phsmoura: per the ticket 02.stg should already be done (I did it a while back when debugging). But you can do the aws ones for sure one by one

<@james:fedora.im>

20:23:27

Kind of related to updates ... did we file anything upstream about the anubis downgrades we had to do?

<@nirik:matrix.scrye.com>

20:23:34

for updates: fabian mentioned that he setup a http proxy for the mgmt devices... I think we perhaps could do the same (noc01?) to allow them to reach out to get firmware updates. Then we can have them download those and have them ready when we reboot.

<@nirik:matrix.scrye.com>

20:24:31

James: not yet. I started to file one saturday, but then I realized all the tools ones have been fixed on the tools sides... the only left issue is cloudfront -> kojipkgs for ostree, and we need to try and figure out how to write that up/change it in config

<@nirik:matrix.scrye.com>

20:25:30

BTW, there is also a newer anubis release. They fixed some problems...

<@nirik:matrix.scrye.com>

20:27:26

we can, but... we don't have cloudfront or kojipkgs in stg... so...

<@nirik:matrix.scrye.com>

20:27:32

¯\\\_(ツ)\_/¯ no good

<@james:fedora.im>

20:28:47

yeh, that's what I was worried about ... plus the usage in staging isn't at the same level.

<@james:fedora.im>

20:29:28

And on that note ... we have 1 minute left, anything anyone wants to talk about?

<@james:fedora.im>

20:32:27

!endmeeting