#meeting-3:fedoraproject.org log

<@Zlopez:matrix.org>

17:01:14

!startmeeting Infrastructure (2025-07-10)

<@meetbot:fedora.im>

17:01:17

Meeting started at 2025-07-17 17:01:14 UTC

<@meetbot:fedora.im>

17:01:18

The Meeting name is 'Infrastructure (2025-07-10)'

<@Zlopez:matrix.org>

17:01:21

!info About our team: https://docs.fedoraproject.org/en-US/cle/

<@Zlopez:matrix.org>

17:01:21

!meetingname infrastructure

<@Zlopez:matrix.org>

17:01:21

!chair @nirik:matrix.scrye.com @zlopez:matrix.org @nb:fedora.im bodanel @dtometzki:fedora.im @jnsamyak:matrix.org @lenkaseg:fedora.im @patrikp:matrix.org

<@Zlopez:matrix.org>

17:01:21

!info Agenda is at: https://board.net/p/fedora-infra

<@Zlopez:matrix.org>

17:01:21

!topic Hola y bienvenido

<@Zlopez:matrix.org>

17:01:21

!info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra

<@meetbot:fedora.im>

17:01:22

The Meeting Name is now infrastructure

<@nirik:matrix.scrye.com>

17:01:25

morning

<@Zlopez:matrix.org>

17:01:30

!hi

<@zodbot:fedora.im>

17:01:32

Michal Konecny (zlopez)

<@Zlopez:matrix.org>

17:01:53

Welcome on the Fedora Infrastructure meeting :-)

<@Zlopez:matrix.org>

17:03:23

Let's start

<@Zlopez:matrix.org>

17:03:29

!topic New folks introductions

<@Zlopez:matrix.org>

17:03:29

!info This is a place where people who are interested in Fedora Infrastructure can introduce themselves

<@Zlopez:matrix.org>

17:03:29

!info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/

<@Zlopez:matrix.org>

17:03:41

Do we have anybody new on the meeting?

<@nirik:matrix.scrye.com>

17:03:57

Hopefully some others besides us two are here today...

<@nirik:matrix.scrye.com>

17:04:03

but it is still summer I guess.

<@james:fedora.im>

17:04:39

I am here, just ending my previous meeting too

<@Zlopez:matrix.org>

17:04:43

It is

<@Zlopez:matrix.org>

17:05:29

So let's go to next one

<@Zlopez:matrix.org>

17:05:59

!info magic eight ball says:

<@Zlopez:matrix.org>

17:05:59

!info char 2025-07-24 - ???

<@Zlopez:matrix.org>

17:05:59

!topic Next chair

<@Zlopez:matrix.org>

17:05:59

!info char 2025-07-17 - zlopez

<@nirik:matrix.scrye.com>

17:07:07

I guess I could take it...

<@Zlopez:matrix.org>

17:07:09

Anybody wants to take chair for next week?

<@Zlopez:matrix.org>

17:08:52

So it's yours nirik

<@Zlopez:matrix.org>

17:09:06

!info chair 2025-07-24 - nirik

<@Zlopez:matrix.org>

17:09:26

!info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1900 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org

<@Zlopez:matrix.org>

17:09:26

!info DC move is over, just cleanup and bringing up shipped machines left

<@Zlopez:matrix.org>

17:09:26

!topic announcements and information

<@Zlopez:matrix.org>

17:09:26

!info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0815 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org

<@Zlopez:matrix.org>

17:10:16

Anything else to announce?

<@nirik:matrix.scrye.com>

17:11:04

nothing off hand...

<@Zlopez:matrix.org>

17:11:40

!topic Oncall

<@Zlopez:matrix.org>

17:11:40

!info https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team

<@Zlopez:matrix.org>

17:11:40

!info on call from 2025-07-04 to 2025-07-10 - nirik

<@Zlopez:matrix.org>

17:11:40

!info on call from 2025-07-10 to 2025-07-17 - ?

<@Zlopez:matrix.org>

17:11:54

Hm, we didn't set any oncall last week

<@nirik:matrix.scrye.com>

17:12:00

ha, we didn't do this last time... yeah...

<@nirik:matrix.scrye.com>

17:12:12

I actually didn't notice any that I can recall.

<@Zlopez:matrix.org>

17:12:18

I can take it now

<@Zlopez:matrix.org>

17:12:36

There was a lot of post dc move things, so I think there wasn't much oncall

<@nirik:matrix.scrye.com>

17:13:05

yeah.

<@nirik:matrix.scrye.com>

17:13:20

folks were pretty nice about not pinging while we were moving / fixing things.

<@Zlopez:matrix.org>

17:13:22

!info on call from 2025-07-18 to 2025-07-24 - zlopez

<@Zlopez:matrix.org>

17:14:15

It seems that I was actually oncall :-D

<@Zlopez:matrix.org>

17:14:19

!oncall

<@zodbot:fedora.im>

17:14:20

The following people are oncall:

<@zodbot:fedora.im>

17:14:20

If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)

<@zodbot:fedora.im>

17:14:20

● @Zlopez:matrix.org (zlopez) Current Time for them: 19:14 (Europe/Prague)

<@zodbot:fedora.im>

17:14:20

<@nirik:matrix.scrye.com>

17:14:34

ha. whoops

<@nirik:matrix.scrye.com>

17:14:45

It may have been the bot wasn't working when I was supposed to be taking it

<@Zlopez:matrix.org>

17:14:53

That was set on 26th June :-D

<@james:fedora.im>

17:15:08

Oof

<@Zlopez:matrix.org>

17:15:48

I must say I didn't notice any pings during DC move

<@Zlopez:matrix.org>

17:16:16

So I can stay for another week

<@Zlopez:matrix.org>

17:16:49

!info on call from 2025-07-25 to 2025-07-31 - ???

<@nirik:matrix.scrye.com>

17:16:51

ok. Or I could take it to make up for not the last few weeks.

<@Zlopez:matrix.org>

17:16:57

Does anybody wants the next one

<@Zlopez:matrix.org>

17:18:32

Does anybody wants the next one?

<@nirik:matrix.scrye.com>

17:18:41

I can, but I will be away the 31st...

<@nirik:matrix.scrye.com>

17:18:55

(at flossy 31st,1st,2nd)

<@Zlopez:matrix.org>

17:19:38

In that case, let's decide on next week

<@Zlopez:matrix.org>

17:19:59

!info Summary of last week: (from current oncall)

<@Zlopez:matrix.org>

17:20:14

I didn't remember any call for the last few weeks

<@Zlopez:matrix.org>

17:20:34

I remember people complaining about things not working after DC move, but no oncall pings

<@Zlopez:matrix.org>

17:20:50

!info Go over existing items and fix them

<@Zlopez:matrix.org>

17:20:50

!topic Monitoring discussion [nirik]

<@Zlopez:matrix.org>

17:20:50

!info https://nagios.fedoraproject.org/nagios

<@nirik:matrix.scrye.com>

17:21:06

There's still a few items...

<@nirik:matrix.scrye.com>

17:21:56

3 listed as down. One is our friend logdetective02... not sure whats wrong with kernel02 (it should be up / working), compose-x86-riscv01 I never deployed yet. I can do that

<@nirik:matrix.scrye.com>

17:22:37

certgetter http... just needs something adjusted to return a 200 instead of a 403

<@Zlopez:matrix.org>

17:22:46

The kernel02 looks similar to what I was solving today with copr hypervisor

<@nirik:matrix.scrye.com>

17:22:52

db01.stg.rdu3.fedoraproject.org disk... probibly we should make the disk bigger there.

<@nirik:matrix.scrye.com>

17:23:17

registry disk. I guess I can just grow it again... we need to move to quay and avoid the problem someday

<@Zlopez:matrix.org>

17:23:22

It should be good to check if there isn't any excessive logging on db01

<@Zlopez:matrix.org>

17:23:32

I hit that on ipa01

<@nirik:matrix.scrye.com>

17:23:43

and the datanommer checks should be fixed sometime once packages are thru

<@nirik:matrix.scrye.com>

17:24:06

could be...

<@Zlopez:matrix.org>

17:24:33

Waiting for the epel9 compose right now :-)

<@nirik:matrix.scrye.com>

17:24:33

anyhow, if anyone wants to fix any of those, feel free... do announce that you are working on it tho, so others don't do so also

<@Zlopez:matrix.org>

17:24:59

I'm trying to fix what I can, but the amount of work is piling up

<@nirik:matrix.scrye.com>

17:25:42

sure, none of the nagios things are urgent

<@Zlopez:matrix.org>

17:25:57

And I found this one today for example https://pagure.io/fedora-infrastructure/issue/12620#comment-978914

<@nirik:matrix.scrye.com>

17:26:07

I plan to go thru the 'post dc move' ticket again, and refile things still outstanding as seperate tickets and close that one...

<@Zlopez:matrix.org>

17:26:26

That would be probably great

<@Zlopez:matrix.org>

17:26:44

I should probably file tickets for all the bigger things I'm working on

<@nirik:matrix.scrye.com>

17:26:51

the biggest thing I think... is the 500's from src.fedoraproject.org for some people.

<@nirik:matrix.scrye.com>

17:27:10

I thought it was related to the core dumps on pkgs01, but now I am not sure at all... could be something else.

<@Zlopez:matrix.org>

17:28:21

It doesn't seem like that, I hit it yesterday and it seems that this is cached somewhere as the requests weren't even in dist-git access log

<@Zlopez:matrix.org>

17:28:42

I will try to investigate more next time I will see it

<@nirik:matrix.scrye.com>

17:29:01

yeah, it's anoying a lot of users. ;(

<@nirik:matrix.scrye.com>

17:29:32

I wonder if the 500 is coming from the proxy for some reason

<@Zlopez:matrix.org>

17:29:48

I think so, but I need to check

<@Zlopez:matrix.org>

17:31:00

So what we want to do next? Give thumbs up to vote

<@Zlopez:matrix.org>

17:31:05

Backlog refinement

<@Zlopez:matrix.org>

17:31:18

Open floor

<@nirik:matrix.scrye.com>

17:32:07

lets backlog/see if we can close anything. ;)

<@Zlopez:matrix.org>

17:32:18

You got it

<@Zlopez:matrix.org>

17:32:33

!topic Revisit blocked tickets

<@Zlopez:matrix.org>

17:32:33

!info Check if any blocked ticket is unblocked

<@Zlopez:matrix.org>

17:32:33

!link https://pagure.io/fedora-infrastructure/issues?status=Open&tags=blocked&priority=0&close_status=

<@Zlopez:matrix.org>

17:32:57

!ticket 11377

<@zodbot:fedora.im>

17:33:01

<@zodbot:fedora.im>

17:33:01

● **Assignee:** Not Assigned

<@zodbot:fedora.im>

17:33:01

● **Opened:** 2 years ago by kevin

<@zodbot:fedora.im>

17:33:01

● **Last Updated:** 5 months ago

<@zodbot:fedora.im>

17:33:01

**fedora-infrastructure #11377** (https://pagure.io/fedora-infrastructure/issue/11377):**Install AWX for testing proof of concept**

<@Zlopez:matrix.org>

17:34:04

I don't think we can move this forward now

<@nirik:matrix.scrye.com>

17:34:50

I want to revisit this, but I am not sure I want to go with awx anymore.

<@Zlopez:matrix.org>

17:35:04

Do we have better option?

<@nirik:matrix.scrye.com>

17:35:06

perhaps we should just close this and open a new spike when we are ready to revisit it?

<@nirik:matrix.scrye.com>

17:35:13

There's some other options...

<@nirik:matrix.scrye.com>

17:35:52

semaphore ... and some other one I am not recalling

<@Zlopez:matrix.org>

17:36:09

I don't mind closing it

<@nirik:matrix.scrye.com>

17:36:44

we could also look at perhaps forgejo actions/ci stuff...

<@Zlopez:matrix.org>

17:37:38

Maybe doing ARC investigation for it?

<@nirik:matrix.scrye.com>

17:38:20

possibly.

<@nirik:matrix.scrye.com>

17:38:30

I know Gwmngilfen had some ideas too.

<@Zlopez:matrix.org>

17:38:39

Ok, let me close it

<@Zlopez:matrix.org>

17:39:59

!ticket 7361

<@zodbot:fedora.im>

17:40:01

● **Last Updated:** 4 months ago

<@zodbot:fedora.im>

17:40:01

● **Opened:** 6 years ago by pbrobinson

<@zodbot:fedora.im>

17:40:01

**fedora-infrastructure #7361** (https://pagure.io/fedora-infrastructure/issue/7361):**Signing infrastructure for aarch64**

<@zodbot:fedora.im>

17:40:01

<@zodbot:fedora.im>

17:40:01

● **Assignee:** Not Assigned

<@nirik:matrix.scrye.com>

17:40:46

getting so close... but not there yet.

<@nirik:matrix.scrye.com>

17:40:52

I can update it.

<@nirik:matrix.scrye.com>

17:41:20

basically we need to move to the new signing setup, get that all working (it should be close) and then we can look at it, but... there's some kernel issues

<@nirik:matrix.scrye.com>

17:41:26

and those need to get sorted out too.

<@Zlopez:matrix.org>

17:41:57

Sounds good

<@Zlopez:matrix.org>

17:43:10

That were all blocked tickets

<@Zlopez:matrix.org>

17:43:12

!info Refine oldest tickets on Fedora Infra tracker

<@Zlopez:matrix.org>

17:43:12

!topic Fedora Infra backlog refinement

<@Zlopez:matrix.org>

17:43:12

!link https://pagure.io/fedora-infrastructure/issues?status=Open&order_key=last_updated&order=asc

<@Zlopez:matrix.org>

17:43:27

!ticket 12246

<@zodbot:fedora.im>

17:43:28

**fedora-infrastructure #12246** (https://pagure.io/fedora-infrastructure/issue/12246):**deploy a Draupnir instance for our matrix servers**

<@zodbot:fedora.im>

17:43:28

<@zodbot:fedora.im>

17:43:28

● **Opened:** 9 months ago by kevin

<@zodbot:fedora.im>

17:43:28

● **Last Updated:** 9 months ago

<@zodbot:fedora.im>

17:43:28

● **Assignee:** Not Assigned

<@nirik:matrix.scrye.com>

17:43:59

we are using a community member run one now, so it's not super urgent

<@nirik:matrix.scrye.com>

17:44:12

we might want to wait until we move to our own server (there's another ticket on that)

<@Zlopez:matrix.org>

17:44:37

OK, do we want to mark this as blocked by another ticket?

<@nirik:matrix.scrye.com>

17:45:02

https://pagure.io/fedora-infrastructure/issue/12323

<@nirik:matrix.scrye.com>

17:45:07

yeah, seems reasonable

<@Zlopez:matrix.org>

17:46:56

Done

<@Zlopez:matrix.org>

17:47:09

!ticket 12323

<@zodbot:fedora.im>

17:47:10

<@zodbot:fedora.im>

17:47:10

● **Opened:** 7 months ago by kevin

<@zodbot:fedora.im>

17:47:10

● **Last Updated:** Never

<@zodbot:fedora.im>

17:47:10

● **Assignee:** kevin

<@zodbot:fedora.im>

17:47:10

**fedora-infrastructure #12323** (https://pagure.io/fedora-infrastructure/issue/12323):**Deploy Element Server Suite operator in staging**

<@Zlopez:matrix.org>

17:47:18

That is the next one :-D

<@nirik:matrix.scrye.com>

17:48:14

ha.

<@nirik:matrix.scrye.com>

17:48:32

So, talked with Gwmngilfen about this one the other day... I think he may want to work on it.

<@nirik:matrix.scrye.com>

17:48:54

We are kinda waiting for element to move us to MAS (matrix authentication server), which will make this easier to migrate.

<@nirik:matrix.scrye.com>

17:49:04

which they are supposed to do pretty soon

<@nirik:matrix.scrye.com>

17:49:13

so, I guess mention him in that ticket?

<@Zlopez:matrix.org>

17:49:29

Let me update the ticket

<@Zlopez:matrix.org>

17:51:04

Updated

<@Zlopez:matrix.org>

17:51:14

!ticket 10458

<@zodbot:fedora.im>

17:51:15

**fedora-infrastructure #10458** (https://pagure.io/fedora-infrastructure/issue/10458):**Help me move my discourse bots to production?**

<@zodbot:fedora.im>

17:51:15

● **Assignee:** ryanlerch

<@zodbot:fedora.im>

17:51:15

● **Last Updated:** 7 months ago

<@zodbot:fedora.im>

17:51:15

● **Opened:** 3 years ago by mattdm

<@zodbot:fedora.im>

17:51:15

<@Zlopez:matrix.org>

17:51:48

Last time I checked this was waiting for @mattdm

<@nirik:matrix.scrye.com>

17:52:02

yeah. I have a meetup with him in a few I can mention it. ;)

<@Zlopez:matrix.org>

17:53:35

We have 7 minutes left, so let's go to open floor and see if we have something to discuss there

<@Zlopez:matrix.org>

17:53:47

!topic Open Floor

<@Zlopez:matrix.org>

17:53:47

!endmeeting