#meeting-3:fedoraproject.org log

<@amedvede:fedora.im>

15:02:15

!startmeeting RELENG (2025-08-11)

<@meetbot:fedora.im>

15:02:16

Meeting started at 2025-08-11 15:02:15 UTC

<@meetbot:fedora.im>

15:02:16

The Meeting name is 'RELENG (2025-08-11)'

<@jnsamyak:matrix.org>

15:02:44

hello everyone!

<@amedvede:fedora.im>

15:02:45

!meetingname releng

<@meetbot:fedora.im>

15:02:45

The Meeting Name is now releng

<@nirik:matrix.scrye.com>

15:02:51

morning

<@jnsamyak:matrix.org>

15:02:56

How's monday and weekend treated you?

<@amedvede:fedora.im>

15:03:00

!chair nirik jnsamyak patrikp amedvede

<@amedvede:fedora.im>

15:03:08

!info Meeting is 60 minutes MAX. At the end of 60, it STOPS.

<@amedvede:fedora.im>

15:03:16

!info Agenda is at https://hackmd.io/vm6biLBcTYKtkQUH5kQkmw.

<@amedvede:fedora.im>

15:04:35

It was tough, cleaned whole house

<@nirik:matrix.scrye.com>

15:04:37

not too bad here...hot, but stayed inside mostly. ;)

<@jnsamyak:matrix.org>

15:05:21

ohh nice, it was raining here, but we had an indian festival on saturday, so there was a feast, and I'm still recovering from sweets and savour leftovers :D

<@amedvede:fedora.im>

15:07:07

ooh nice to hear, a lot of dancing I suppose

<@amedvede:fedora.im>

15:07:32

Let's start with Init

<@amedvede:fedora.im>

15:07:39

!topic Init process.

<@amedvede:fedora.im>

15:07:45

Do we have anything for the init? Any blockers/tasks/issues/requests/features that need releng intervention?

<@amedvede:fedora.im>

15:08:19

I believe the main thing now is mass branching

<@jnsamyak:matrix.org>

15:08:37

So this is branching week, this is gonna be hectic for me, but I want to cover it in the next items.

<@jnsamyak:matrix.org>

15:08:49

With updates/pr/questions etc

<@amedvede:fedora.im>

15:08:57

okay

<@jnsamyak:matrix.org>

15:09:08

if we have anything apart from that, init is your place :D

<@jnsamyak:matrix.org>

15:09:28

I have one

<@jnsamyak:matrix.org>

15:09:31

!releng 12869

<@zodbot:fedora.im>

15:09:32

<@zodbot:fedora.im>

15:09:32

● **Assignee:** Not Assigned

<@zodbot:fedora.im>

15:09:32

● **Last Updated:** Never

<@zodbot:fedora.im>

15:09:32

**releng #12869** (https://pagure.io/releng/issue/12869):**Test massive side-tag update in bodhi staging**

<@zodbot:fedora.im>

15:09:32

● **Opened:** 3 days ago by yselkowitz

<@nirik:matrix.scrye.com>

15:11:11

we could add mattia to cc and see if he has any time to work on it... but I think we would need to do the builds probibly. or find enough builds that exist but were never submitted as updates, which could be pretty tricky.

<@jnsamyak:matrix.org>

15:11:34

I'm not sure from releng povs, how can we test it? Is it more related to infra end or upstream perphaps, on how bodhi can take inn? because at the time of the mass rebuilds in fedora, we are direcrtly tagging the built packages to the tag right, there is no bodhio intervention in between

<@jnsamyak:matrix.org>

15:12:19

how do we want to label it? ops? investigation? med/med?

<@jnsamyak:matrix.org>

15:12:24

Also, how do we want to label it? ops? investigation? med/med?

<@nirik:matrix.scrye.com>

15:12:53

sure... and cc mattia

<@nirik:matrix.scrye.com>

15:13:18

there might be some way to test it outside infra... dunno if bodhi has a test infra to do that or not.

<@jnsamyak:matrix.org>

15:13:34

Okay as a step one on the ticket, I'll try to do that and see how far we can go with it

<@amedvede:fedora.im>

15:15:18

anything else for init?

<@jnsamyak:matrix.org>

15:15:50

!releng 12863

<@zodbot:fedora.im>

15:15:52

● **Assignee:** jnsamyak

<@zodbot:fedora.im>

15:15:52

**releng #12863** (https://pagure.io/releng/issue/12863):**Allow resigning f43 tagged packages with f44 key**

<@zodbot:fedora.im>

15:15:52

<@zodbot:fedora.im>

15:15:52

● **Opened:** 6 days ago by jnsamyak

<@zodbot:fedora.im>

15:15:52

● **Last Updated:** 5 days ago

<@nirik:matrix.scrye.com>

15:16:41

I think it's done. I poked a few stragglers this weekend... but you should check that everything is signed to make sure.

<@jnsamyak:matrix.org>

15:17:17

yep that's what i was typinh it's seems to have finished but i'll try to run the script

<@jnsamyak:matrix.org>

15:17:35

and see if there are any leftovers, otherwise we are good

<@jnsamyak:matrix.org>

15:17:50

That'

<@jnsamyak:matrix.org>

15:18:31

That's all I will cover branching bits in next topic, though this init is also for blockers, I'm assuming if there is not anything else to discuss there are no blockers too!

<@amedvede:fedora.im>

15:19:04

I have one

<@amedvede:fedora.im>

15:19:16

!releng 12810

<@zodbot:fedora.im>

15:19:17

● **Assignee:** amedvede

<@zodbot:fedora.im>

15:19:17

**releng #12810** (https://pagure.io/releng/issue/12810):**Packages containg a "+" do not have a f42 branch**

<@zodbot:fedora.im>

15:19:17

<@zodbot:fedora.im>

15:19:17

● **Opened:** a month ago by fesoj

<@zodbot:fedora.im>

15:19:17

● **Last Updated:** 30 minutes ago

<@nirik:matrix.scrye.com>

15:20:25

sounds like a possible bug in the mass branching script?

<@amedvede:fedora.im>

15:20:37

this is the issue, It applies on all packages containing +, at least checked 10 + packages

<@nirik:matrix.scrye.com>

15:20:38

(but +s in names are difficult... cause all kinds of issues)

<@jnsamyak:matrix.org>

15:20:52

+1 i already asked Anton Medvedev to look into it? did you check the script?

<@amedvede:fedora.im>

15:21:04

or https://pagure.io/releng/blob/main/f/scripts/branching/get_all_active_packages_branching.sh in this script

<@jnsamyak:matrix.org>

15:21:27

this is the one with the issue :)

<@nirik:matrix.scrye.com>

15:21:43

ah ha yeah...

<@nirik:matrix.scrye.com>

15:22:13

can we fix it up today? ;)

<@jnsamyak:matrix.org>

15:22:24

Anton Medvedev: ?

<@amedvede:fedora.im>

15:22:49

probably yeah

<@nirik:matrix.scrye.com>

15:23:06

I suppose we need to fix the f42 ones too. ;(

<@jnsamyak:matrix.org>

15:23:17

please this is on priority for tomorrow

<@amedvede:fedora.im>

15:23:54

I'm working on it right now, tomorrow morning will be ready

<@amedvede:fedora.im>

15:24:36

I mean my morning), what time we have mass branching planned?

<@jnsamyak:matrix.org>

15:25:45

please make sure to have a dry run, and trhe pr for it ready, the branching work will start at 14 UTC sharp and fetching the list of packages is something we do prior to it, so we can just run the branch script so ideally around 11-12 UTc, so if there is a bug, we can fix it

<@amedvede:fedora.im>

15:27:01

so right after standup?

<@nirik:matrix.scrye.com>

15:27:12

yeah, can test run it before hand...

<@amedvede:fedora.im>

15:28:08

Okay, that's it for Init

<@amedvede:fedora.im>

15:28:11

can we move on?

<@jnsamyak:matrix.org>

15:28:28

yes

<@amedvede:fedora.im>

15:28:42

!topic Scheduled actions coming up in the next week.

<@amedvede:fedora.im>

15:28:51

!info Here we list/discuss anything about items that are due to be done in the next week.

<@amedvede:fedora.im>

15:28:56

!link https://fedorapeople.org/groups/schedule/f-43/f-43-releng-tasks.html

<@jnsamyak:matrix.org>

15:29:23

Okay so regarding the MAss

<@jnsamyak:matrix.org>

15:29:32

Okay so regarding the Mass Branching updates

<@jnsamyak:matrix.org>

15:30:11

All the pre-preparatory steps has been handled with me, the tracker are there, the email has been sent.

<@jnsamyak:matrix.org>

15:30:22

!releng 12870

<@zodbot:fedora.im>

15:30:24

● **Last Updated:** 2 hours ago

<@zodbot:fedora.im>

15:30:24

● **Opened:** 11 hours ago by jnsamyak

<@zodbot:fedora.im>

15:30:24

**releng #12870** (https://pagure.io/releng/issue/12870):**Fedora 43 Mass Branching Tracker**

<@zodbot:fedora.im>

15:30:24

<@zodbot:fedora.im>

15:30:24

● **Assignee:** jnsamyak

<@nirik:matrix.scrye.com>

15:30:55

I'll go review stuff later this morning when I get time to.

<@jnsamyak:matrix.org>

15:31:00

I got the feedback from patrikp that the updates should be in the ticket, and I think I mostly try to maintain that!

<@jnsamyak:matrix.org>

15:31:24

I have decided to do the conversation in the releng channel as I have been doing these for last few releases

<@jnsamyak:matrix.org>

15:31:45

so people know what is going on, and how it is done, I hope it clears out the confusion.

<@jnsamyak:matrix.org>

15:32:36

- The koji blockers for external builders

<@jnsamyak:matrix.org>

15:32:36

- Anton to take a look at the get_all_active_package_script

<@jnsamyak:matrix.org>

15:32:36

Two things to figure out (out of one is already discussed)

<@nirik:matrix.scrye.com>

15:33:12

the second one should be pretty small... just some syntax...

<@jnsamyak:matrix.org>

15:33:12

Thanks to nirik he already is helping out with the second part of the issue

<@jnsamyak:matrix.org>

15:33:24

since the move from iptable -> nftable happened

<@jnsamyak:matrix.org>

15:34:59

But but one things as part of last branching retrospective happened, is we disabled it for external users, and then did the mass branching of packages in pkg machine and then merged the ansible pr, but then we re-ran the playbooks again out of one which was koji-hub playbook while running that reset the external ip blockers

<@jnsamyak:matrix.org>

15:35:37

we need to be either be quick and hope no one builds while we reset it offf, or find a solution to that issue :3

<@amedvede:fedora.im>

15:38:05

Be quick enough at 14 utc quite challenging

<@jnsamyak:matrix.org>

15:40:02

one solution I thought was disabling it as part of a commit in ansible, and then revertiing that commit, so even after we run the koji-hub playbook changes for rawhide and branched versions it stays disabled, thoughts?

<@amedvede:fedora.im>

15:44:08

It can work, I guess it should also be added as discribtion in announcement, so people know that during some time period branching will be blocked for them

<@jnsamyak:matrix.org>

15:44:50

that is already there in the email

<@jnsamyak:matrix.org>

15:44:59

that is already there in the email - the announcement

<@amedvede:fedora.im>

15:45:38

Oh I see

<@nirik:matrix.scrye.com>

15:46:05

yeah, we could do that indeed.

<@jnsamyak:matrix.org>

15:48:48

nirik: can you add it as part of the pr, or another commit? I'll follow it up and update the doc accordingly?

<@nirik:matrix.scrye.com>

15:49:04

I can look at it yeah.

<@jnsamyak:matrix.org>

15:49:14

Okay yeah I swear I had one more thing

<@jnsamyak:matrix.org>

15:49:24

I forgot but I remember it now

<@amedvede:fedora.im>

15:49:34

)

<@jnsamyak:matrix.org>

15:50:49

so we need to build the branched for fedora-repos and fedora-releases before we do the gating but after we turn of koji for external builders, and finally after running the playbooks we do need to build fedora-repos and fedora-release together for rawhide

<@jnsamyak:matrix.org>

15:51:20

but last time, we had to figure out a way on how to get this done, while having koji turned of from external machines

<@jnsamyak:matrix.org>

15:51:52

And, finally we build those in a side-tag in koji machine

<@nirik:matrix.scrye.com>

15:51:59

you should be able to do this from a compose machine, no?

<@jnsamyak:matrix.org>

15:52:20

it had an error in the compose machine :(

<@jnsamyak:matrix.org>

15:53:15

oh yeah i think from my user at compose machine the koji conf is not listed that's why? Because, I had an error of auth

<@nirik:matrix.scrye.com>

15:53:17

what was the error?

<@jnsamyak:matrix.org>

15:53:34

2025-02-04 18:17:26,163 [ERROR] koji: (gssapi auth failed: requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://koji.fedoraproject.org/kojihub/ssllogin)

<@nirik:matrix.scrye.com>

15:53:39

if you are doing it as you user, you will need to run fkinit to get a kerberos ticket

<@jnsamyak:matrix.org>

15:54:29

yeah I think i need to set it up I'll try that out today

<@jnsamyak:matrix.org>

15:54:42

but it worked from koji machine I remember

<@nirik:matrix.scrye.com>

15:55:08

it should work from compose-x86-01... thats kind what that machine is supposed to be for...

<@amedvede:fedora.im>

15:57:30

Okay, we are not rich with time, lets move on?

<@nirik:matrix.scrye.com>

15:57:46

anyhow, if you can test and confirm you can get koji/fedpkg working ok there today that would be great. Happy to help fix anything there

<@jnsamyak:matrix.org>

15:58:24

Also one last question we were discussing this last time, merging sidetags using koji move-build for fedora-repos and fedora-release is okay do I need to perform any additional manual steps to ensure the packages get properly signed and included in updates, OR does Bodhi handle everything automatically now?

<@jnsamyak:matrix.org>

15:58:41

yeah I'll do that before hopping into bed

<@nirik:matrix.scrye.com>

15:59:39

are we bypassing bodhi or using it? if using it, it should handle everything with the side tag fine...

<@jnsamyak:matrix.org>

16:00:39

yeah that's my concern, I think we should go with bodhi and just build it in side tags and then bodhi does it job, but it had created gating issues in the past no?

<@nirik:matrix.scrye.com>

16:01:31

yes, it will run gating tests on it...

<@nirik:matrix.scrye.com>

16:01:51

which might take a bit or be broken due to branching...

<@nirik:matrix.scrye.com>

16:01:59

but we could waive it...

<@nirik:matrix.scrye.com>

16:02:29

branching is such a dance, I don't recall which way we decided to do this last time. ;)

<@jnsamyak:matrix.org>

16:02:34

yeah but i remember we tagged the packages to f42-signing-pending

<@nirik:matrix.scrye.com>

16:03:38

if we are using bodhi that shouldn't be needed. Just build in sidetag, make a new bodhi update from the sidetag and it will handle it.

<@jnsamyak:matrix.org>

16:04:11

okay let's stick to that then

<@nirik:matrix.scrye.com>

16:04:23

enjoy dinner!

<@nirik:matrix.scrye.com>

16:04:52

oh, BTW, updates pushes all failed last night... seems to be some weird auth thing. I am resuming one to check if it's just a transitory thing.

<@jnsamyak:matrix.org>

16:04:57

I'll update the sop today for the same so we are clear for tomorrow, thanks for being part of branching discussion, this way I dont feel alone while doing it :D

<@jnsamyak:matrix.org>

16:05:14

Oh! okay

<@jnsamyak:matrix.org>

16:05:44

but i see no such blocker for branching tomorrow yet, maybe we will find something on issue tracker tomorrow :D

<@jnsamyak:matrix.org>

16:05:49

or no

<@jnsamyak:matrix.org>

16:05:52

or not :P

<@amedvede:fedora.im>

16:06:20

lets keep a positive mindset xD

<@amedvede:fedora.im>

16:06:50

okay, we are over time

<@amedvede:fedora.im>

16:07:08

need to decide one more thing

<@amedvede:fedora.im>

16:07:11

next chair

<@amedvede:fedora.im>

16:07:55

Lets choose that same way as this week

<@jnsamyak:matrix.org>

16:08:09

I'm off next week somedays, if branching goes fine :)

<@amedvede:fedora.im>

16:08:22

I'm not sure If I'll be here also

<@amedvede:fedora.im>

16:08:35

okay, lets decide later then

<@amedvede:fedora.im>

16:08:47

!info Thank you all for coming!

<@amedvede:fedora.im>

16:08:54

!endmeeting