#meeting-3:fedoraproject.org log

<@patrikp:matrix.org>

15:04:17

!startmeeting RELENG (2025-07-14)

<@meetbot:fedora.im>

15:04:18

Meeting started at 2025-07-14 15:04:17 UTC

<@meetbot:fedora.im>

15:04:19

The Meeting name is 'RELENG (2025-07-14)'

<@patrikp:matrix.org>

15:04:26

!info Meeting is 60 minutes MAX. At the end of 60, it STOPS.

<@patrikp:matrix.org>

15:04:26

!info Agenda is at https://hackmd.io/vm6biLBcTYKtkQUH5kQkmw.

<@patrikp:matrix.org>

15:04:26

!meetingname releng

<@patrikp:matrix.org>

15:04:26

!chair nirik jnsamyak patrikp amedvede

<@meetbot:fedora.im>

15:04:27

The Meeting Name is now releng

<@nirik:matrix.scrye.com>

15:04:37

morning. 👋

<@jnsamyak:matrix.org>

15:04:45

oh he just texted in my slack like 15 mins back

<@jnsamyak:matrix.org>

15:04:50

he couldntmak eit

<@patrikp:matrix.org>

15:05:04

I'll put him in for next week. 👍️ Init time?

<@patrikp:matrix.org>

15:05:10

!topic Init process.

<@patrikp:matrix.org>

15:05:10

Do we have anything for the init? Any blockers/tasks/issues/requests/features that need releng intervention?

<@nirik:matrix.scrye.com>

15:06:01

I have one thing to bring up (possibly here)?

<@jnsamyak:matrix.org>

15:06:06

Okay i have one thing regarding sigul

<@patrikp:matrix.org>

15:06:08

Go for it.

<@jnsamyak:matrix.org>

15:06:16

but nirik you beat me to it

<@jnsamyak:matrix.org>

15:06:18

so you go!

<@nirik:matrix.scrye.com>

15:06:31

There's a problem with updates composes ( https://pagure.io/releng/issue/12813 )

<@nirik:matrix.scrye.com>

15:06:51

I think it might be a tcp timeout thing... but haven't had a chance to try fixes on that

<@nirik:matrix.scrye.com>

15:07:01

there's currently 2 failed composes if someone wants to resume them

<@jnsamyak:matrix.org>

15:08:11

I looked it this morning, I was away since wednesday last week, but I guessed it was a timeout issue too but is it failing with every compose?

<@nirik:matrix.scrye.com>

15:08:26

it's most of them, not everyone...

<@nirik:matrix.scrye.com>

15:08:31

well, some, not most

<@jnsamyak:matrix.org>

15:08:31

What did you do to resume it?

<@jnsamyak:matrix.org>

15:08:46

I can take a look at it

<@jnsamyak:matrix.org>

15:08:52

and fix it after this call

<@nirik:matrix.scrye.com>

15:09:18

the usual thing... on bodhi-backend01: sudo -u apache bodhi-push --username (yourusername) --resume

<@patrikp:matrix.org>

15:09:22

Just what I was about to ask. Do we have any words written down about how the composes are done and where to look if they're failing (at least in which direction)? If I recall correctly it runs as a script as a cron job or something like that?

<@patrikp:matrix.org>

15:09:34

Or where is it explained how this is set up?

<@nirik:matrix.scrye.com>

15:10:04

https://docs.fedoraproject.org/en-US/infra/releng_misc_guide/sop_pushing_updates/

<@jnsamyak:matrix.org>

15:10:08

The compose setup for updates has been explained several times, is that what you are referrinfg to?

<@nirik:matrix.scrye.com>

15:10:12

but as usual, it needs some updates. ;)

<@jnsamyak:matrix.org>

15:10:12

it's also documented

<@jnsamyak:matrix.org>

15:10:32

ahhhh theseeee!

<@jnsamyak:matrix.org>

15:10:47

I thought about something else, leave this to me never mind

<@nirik:matrix.scrye.com>

15:11:32

anyhow, thats it on this. Just wanted to raise it up and be aware... we will need to resume updates every day until we fix it.

<@jnsamyak:matrix.org>

15:11:42

nirik: from the fixing it for future do you think increasing the timeout will help? Or is it flaky because of recent changes!

<@patrikp:matrix.org>

15:11:43

But out of curiosity where is it explained?

<@nirik:matrix.scrye.com>

15:13:47

jnsamyak: my current theory is that it's a tcp timeout like we hit with pkgs in the new datacenter... basically bodhi opens up some database connections, and then goes to run pungi and other things, and it takes > 30m... it comes back and tries to use one of it's db connections but the firewall between vlans there has killed the connection because it was idle. ;( At least thats my current theory. I think we can tell sqlalchemy to only have connections for a shorter time and restablish them... like what was done for pagure / pkgs.

<@jnsamyak:matrix.org>

15:14:43

I explained it to you myself, I think I also remember writing a document a year ago about this that i wrote for my understanding, and if you want i can host it too! But Anyways, I can explain it later once again in my 1:1

<@jnsamyak:matrix.org>

15:14:47

For the sigul

<@patrikp:matrix.org>

15:15:23

Unfortunately I don't have perfect memory so I have to consult documentation from time to time. Would it be possible to put in a doc that could be published perhaps?

<@jnsamyak:matrix.org>

15:15:25

list keys before signing to check if it was working but it just accepts the password and do nothibg about it

<@jnsamyak:matrix.org>

15:15:25

I have some questions - I tried running basic commands for it

<@patrikp:matrix.org>

15:15:42

But that's last comment on the topic, go ahead with the questions, sorry.

<@jnsamyak:matrix.org>

15:16:42

so nirik do I need to redo the setup again or the regular stuff should have just worked?

<@nirik:matrix.scrye.com>

15:19:48

grrrrrrrrrrrrrrrrrrrrrrrrrrrrrr. my main ups overloaded and took out all my servers and internet.

<@nirik:matrix.scrye.com>

15:20:24

I would have expected it to just work. It just hangs?

<@jnsamyak:matrix.org>

15:20:33

yeah

<@jnsamyak:matrix.org>

15:20:36

<@nirik:matrix.scrye.com>

15:21:11

we can debug in #releng:fedoraproject.org when you have some time?

<@jnsamyak:matrix.org>

15:21:20

yes

<@jnsamyak:matrix.org>

15:21:30

please continue

<@patrikp:matrix.org>

15:21:38

Anything else for the init?

<@patrikp:matrix.org>

15:21:53

!link https://fedorapeople.org/groups/schedule/f-42/f-42-releng-tasks.html

<@patrikp:matrix.org>

15:21:53

!topic Scheduled actions coming up in the next week.

<@patrikp:matrix.org>

15:21:53

!info Here we list/discuss anything about items that are due to be done in the next week.

<@patrikp:matrix.org>

15:21:53

!link https://fedorapeople.org/groups/schedule/f-43/f-43-releng-tasks.html

<@patrikp:matrix.org>

15:22:19

Mass rebuild soon.

<@patrikp:matrix.org>

15:22:33

!link https://docs.fedoraproject.org/en-US/infra/release_guide/mass_rebuild/

<@jnsamyak:matrix.org>

15:22:35

nothing for me

<@jnsamyak:matrix.org>

15:22:45

from*

<@nirik:matrix.scrye.com>

15:22:56

yeah, next week?

<@patrikp:matrix.org>

15:23:33

So, the plan is, for each step in the schedule we will have a corresponding SOP with the same title and some context (why it's important etc.), yes? By the end of the F43 release cycle.

<@jnsamyak:matrix.org>

15:23:54

I guess you would be working on it?

<@patrikp:matrix.org>

15:24:01

Aoife is the only person with access to the schedule but we can open tickets.

<@patrikp:matrix.org>

15:24:11

Yes but I will need a second and a third set of eyes and to ask questions.

<@jnsamyak:matrix.org>

15:24:12

Open tickets where?

<@patrikp:matrix.org>

15:24:40

https://pagure.io/fedora-pgm/schedule/issues

<@patrikp:matrix.org>

15:24:40

If we want to update anything about the schedule we can do so via opening a ticket here:

<@jnsamyak:matrix.org>

15:24:48

sure I keep explaining those process and what i do everytime on releng channel if you want you can take insipirations from there

<@patrikp:matrix.org>

15:24:49

Just for info.

<@patrikp:matrix.org>

15:25:31

Nothing apart from the mass rebuild coming up. Tickets?

<@jnsamyak:matrix.org>

15:25:41

not from me

<@nirik:matrix.scrye.com>

15:26:07

oh, I think I found the sigul problem. It was stuck... ;( try your commands anytime now...

<@jnsamyak:matrix.org>

15:26:22

oh thank you!

<@nirik:matrix.scrye.com>

15:26:36

weird error on the vault. ;( Hope it was a one off

<@patrikp:matrix.org>

15:27:33

Do you have any words of wisdom for the upcoming actions Kevin? How to prepare mentally and/or physically? Or we can go to tickets.

<@nirik:matrix.scrye.com>

15:28:19

nothing off hand. we should perhaps look over the SOP's for things upcoming before them and see if we see anything that needs adjustment...

<@jnsamyak:matrix.org>

15:28:32

<@nirik:matrix.scrye.com>

15:28:43

but otherwise, just go carefully along and get things done on time. ;)

<@patrikp:matrix.org>

15:29:04

!topic Tickets that need attention.

<@patrikp:matrix.org>

15:29:04

!link https://pagure.io/releng/issues

<@patrikp:matrix.org>

15:29:04

!info This is the time where you can bring up releng tickets to discuss if there are blockers, etc.

<@patrikp:matrix.org>

15:29:28

!releng 12825

<@zodbot:fedora.im>

15:29:29

● **Last Updated:** Never

<@zodbot:fedora.im>

15:29:29

● **Assignee:** Not Assigned

<@zodbot:fedora.im>

15:29:29

**releng #12825** (https://pagure.io/releng/issue/12825):**When processing stalled EPEL package requests, make the requester EPEL bugzilla override**

<@zodbot:fedora.im>

15:29:29

<@zodbot:fedora.im>

15:29:29

● **Opened:** 4 hours ago by churchyard

<@patrikp:matrix.org>

15:29:58

So I processed those, but I just set the collaborators in dist-git (we don't have a doc on processing those tickets). Sounds like something has to be manually set in Bugzilla?

<@nirik:matrix.scrye.com>

15:30:17

it's in src.fedoraproject.org...

<@jnsamyak:matrix.org>

15:30:45

What are the steps do you follow while doing stalled epels?

<@nirik:matrix.scrye.com>

15:30:48

https://src.fedoraproject.org/rpms/koji on the side there's a 'bugzilla asignee' section...

<@nirik:matrix.scrye.com>

15:31:07

but I am not sure how to set it via api/command line...

<@jnsamyak:matrix.org>

15:31:27

EXACTLY because you are missing these and you have been doing these stalled for longer time, are you missing those on every ticket>

<@jnsamyak:matrix.org>

15:31:34

EXACTLY because you are missing these and you have been doing these stalled for longer time, are you missing those on every ticket?

<@patrikp:matrix.org>

15:31:42

So all I do is go to src.fpo settings and add a user as collaborator for specific branches. But sounds like this is simple to do now that we know about it, it can be set in the web UI.

<@patrikp:matrix.org>

15:31:48

I guess so.

<@jnsamyak:matrix.org>

15:32:13

:) Can you go back to all your stalled package tickets and fix it please!

<@patrikp:matrix.org>

15:32:20

Let me assign to myself and take a look...

<@patrikp:matrix.org>

15:32:26

Next ticket?

<@jnsamyak:matrix.org>

15:33:45

Also, it would be good if you can try to write an automation around it, start simple, and then implement it as a toddler! Zlopez can we do that?

<@patrikp:matrix.org>

15:34:00

Automation for what?

<@jnsamyak:matrix.org>

15:34:02

I can help this groom with Zlopez and then we can see if I can put it to work

<@patrikp:matrix.org>

15:34:18

Nope, it's a matter of a couple of clicks. We don't need more tech debt.

<@jnsamyak:matrix.org>

15:34:35

For stalled packages so people can just go and request it from fedora-scm-requests

<@patrikp:matrix.org>

15:34:58

I don't know... This is something that should be done manually by a human.

<@jnsamyak:matrix.org>

15:35:02

it is not tbh, this saves a lot of time and human errors - it will not be more than a week work

<@jnsamyak:matrix.org>

15:35:11

may i know why?

<@jnsamyak:matrix.org>

15:35:42

nirik: it still stucks :(

<@jnsamyak:matrix.org>

15:35:52

but maybe my net is flaky

<@jnsamyak:matrix.org>

15:36:05

I have a hard stop in 9 mins

<@jnsamyak:matrix.org>

15:36:44

!releng 12782

<@zodbot:fedora.im>

15:36:47

<@zodbot:fedora.im>

15:36:47

**releng #12782** (https://pagure.io/releng/issue/12782):**Change proposal: Drop i686 support**

<@zodbot:fedora.im>

15:36:47

● **Opened:** 3 weeks ago by decathorpe

<@zodbot:fedora.im>

15:36:47

● **Assignee:** Not Assigned

<@zodbot:fedora.im>

15:36:47

● **Last Updated:** 2 weeks ago

<@jnsamyak:matrix.org>

15:36:51

This is something I want to discuss

<@patrikp:matrix.org>

15:36:54

You need to verify the Bugzilla ticket. It's not about clicking in the UI, it's about the stalled process overall. Clicking is just the final step.

<@jnsamyak:matrix.org>

15:37:26

We can always implement a logic there :)

<@patrikp:matrix.org>

15:37:41

Go ahead if you feel like it'd be worthwhile.

<@jnsamyak:matrix.org>

15:38:06

For that, I'll discuss it with jednorozec!

<@jnsamyak:matrix.org>

15:38:14

And see if our PO accepts :D

<@jnsamyak:matrix.org>

15:38:42

This is something I want to discuss becaure some of the composes should be affected by it, iirc

<@jnsamyak:matrix.org>

15:38:49

so it will involve releng

<@jnsamyak:matrix.org>

15:39:30

But it is for F44 so I want everyone to keep an eye and get involved in those discussion from releng too

<@jnsamyak:matrix.org>

15:40:17

!releng 12736

<@zodbot:fedora.im>

15:40:18

**releng #12736** (https://pagure.io/releng/issue/12736):**Fedora-KDE-42-1.1-x86_64-CHECKSUM has wrong ISO filename **

<@zodbot:fedora.im>

15:40:18

● **Assignee:** amedvede

<@zodbot:fedora.im>

15:40:18

● **Last Updated:** a day ago

<@zodbot:fedora.im>

15:40:18

● **Opened:** 2 months ago by ananas-comosus

<@zodbot:fedora.im>

15:40:18

<@jnsamyak:matrix.org>

15:40:27

Anton Medvedev: What's the status for this?

<@jnsamyak:matrix.org>

15:40:39

But i guess he is not here, so it is of no news

<@jnsamyak:matrix.org>

15:41:17

But that's all from me

<@patrikp:matrix.org>

15:41:51

All from me too.

<@patrikp:matrix.org>

15:41:57

!topic Choose next chair.

<@patrikp:matrix.org>

15:41:57

!info Next chair July 21st: Anton

<@patrikp:matrix.org>

15:42:04

!topic Discussion / Any other business / Open floor.

<@patrikp:matrix.org>

15:42:41

So, the OpenH264 SOP is (finally) published here.

<@patrikp:matrix.org>

15:42:41

https://docs.fedoraproject.org/en-US/infra/releng_misc_guide/sop_openh264/

<@nirik:matrix.scrye.com>

15:45:18

the i686 proposal was pulled... we can close that ticket. ;)

<@jnsamyak:matrix.org>

15:45:25

Thanks for this

<@nirik:matrix.scrye.com>

15:46:57

yeah, on that checksum thing. I tried to update the whitelist, but it didn't work...

<@nirik:matrix.scrye.com>

15:47:05

but I could be missing something there...

<@jnsamyak:matrix.org>

15:48:42

it was anton to figure it out did he ask for help?

<@patrikp:matrix.org>

15:48:57

This is the step where something went wrong?

<@patrikp:matrix.org>

15:48:57

https://docs.fedoraproject.org/en-US/infra/release_guide/torrentrelease/#_torrent_release

<@nirik:matrix.scrye.com>

15:49:01

jnsamyak: sigul is single threaded. it's trying to sign a large package right now, so yeah, things will hang until your request is at the top of the queue

<@nirik:matrix.scrye.com>

15:49:33

He was going to look, but... right now no one can download kde torrents, which is kinda bad. So, we should try and fix it as soon as we can.

<@jnsamyak:matrix.org>

15:50:07

Okay so we need to escalte it why it's not being handled by him if it's urgent if he can't accomodate it he should ask for help

<@nirik:matrix.scrye.com>

15:50:09

that sop strangely doesn't mention the whitelist. ;(

<@jnsamyak:matrix.org>

15:50:12

This is not good

<@jnsamyak:matrix.org>

15:50:43

but for the issue I'll take a look at it tomorrow morning it was not on my notice, sorry about it!

<@patrikp:matrix.org>

15:51:41

Anything else anybody would like to bring up? A good joke perhaps to end it on a light note?

<@patrikp:matrix.org>

15:51:51

I am a grim person so I don't know any.

<@nirik:matrix.scrye.com>

15:52:29

I never seem to remember jokes...

<@patrikp:matrix.org>

15:52:52

And on that terrible disappointment...

<@patrikp:matrix.org>

15:53:02

!endmeeting

<@patrikp:matrix.org>

15:53:02

!info Thank you all for coming!