#meeting-3:fedoraproject.org log

<@meetbot:fedora.im>

09:31:42

HTML Log: https://meetbot.fedoraproject.org/meeting-3_matrix_fedoraproject-org/2025-01-27/cpe-infra-releng-daily-standup.2025-01-27-09.00.log.html

<@meetbot:fedora.im>

09:31:42

Text Minutes: https://meetbot.fedoraproject.org/meeting-3_matrix_fedoraproject-org/2025-01-27/cpe-infra-releng-daily-standup.2025-01-27-09.00.txt

<@patrikp:matrix.org>

16:00:13

!startmeeting RELENG (2025-01-27)

<@meetbot:fedora.im>

16:00:16

Meeting started at 2025-01-27 16:00:13 UTC

<@meetbot:fedora.im>

16:00:16

The Meeting name is 'RELENG (2025-01-27)'

<@patrikp:matrix.org>

16:00:21

!info Agenda is at https://hackmd.io/vm6biLBcTYKtkQUH5kQkmw.

<@patrikp:matrix.org>

16:00:21

!meetingname releng

<@patrikp:matrix.org>

16:00:21

!chair nirik jnsamyak patrikp amedvede

<@patrikp:matrix.org>

16:00:21

!info Meeting is 60 minutes MAX. At the end of 60, it stops.

<@meetbot:fedora.im>

16:00:22

The Meeting Name is now releng

<@patrikp:matrix.org>

16:00:29

Hi.

<@nirik:matrix.scrye.com>

16:00:30

morning

<@nirik:matrix.scrye.com>

16:01:57

wonder if it's just us two today? :)

<@patrikp:matrix.org>

16:02:01

Not sure if Samyak is around, he rescheduled a 1:1 today because of noise so maybe he's travelling.

<@patrikp:matrix.org>

16:02:06

Could be yes.

<@nirik:matrix.scrye.com>

16:02:32

yeah, I think he said he might not be here today...

<@patrikp:matrix.org>

16:03:22

Let's freestyle it then. I'm looking at the tracker, there seems to be one new issue.

<@patrikp:matrix.org>

16:03:28

!releng 12547

<@zodbot:fedora.im>

16:03:29

● **Assignee:** Not Assigned

<@zodbot:fedora.im>

16:03:29

● **Last Updated:** 2 days ago

<@zodbot:fedora.im>

16:03:29

**releng #12547** (https://pagure.io/releng/issue/12547):**Updating flatpak-module-tools to 1.1**

<@zodbot:fedora.im>

16:03:29

● **Opened:** 3 days ago by yselkowitz

<@zodbot:fedora.im>

16:03:29

<@patrikp:matrix.org>

16:04:03

Updating the builders? That means running a playbook?

<@nirik:matrix.scrye.com>

16:04:31

well, I am planning a update/reboot cycle for wed... it can just be done then as part of that if it's not needed sooner.

<@nirik:matrix.scrye.com>

16:05:11

or we can do it sooner, but might be disruptive if kojid needs restarting

<@nirik:matrix.scrye.com>

16:05:27

so anyhow, how about I ask in ticket about that?

<@jnsamyak:matrix.org>

16:05:59

hello hello

<@patrikp:matrix.org>

16:06:06

It says before beta freeze which is well off. Says "preferably sooner" but sounds like Wednesday is OK...

<@nirik:matrix.scrye.com>

16:06:31

hey jnsamyak!

<@jnsamyak:matrix.org>

16:06:33

I'm here :3 trying to manage timings

<@patrikp:matrix.org>

16:06:41

Is there more information about the update/reboot cycle? An SOP somewhere I could take a look like?

<@patrikp:matrix.org>

16:07:43

https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/massupgrade/

<@patrikp:matrix.org>

16:07:45

This one?

<@jnsamyak:matrix.org>

16:08:48

I would just say to do it either this week - or next to next week before branching

<@jnsamyak:matrix.org>

16:08:53

or after

<@nirik:matrix.scrye.com>

16:09:12

yeah, I think thats the one...

<@nirik:matrix.scrye.com>

16:09:19

well, it will be done on wed.

<@nirik:matrix.scrye.com>

16:09:26

I am not going to exclude that one package. ;)

<@jnsamyak:matrix.org>

16:09:30

awesome 💃

<@patrikp:matrix.org>

16:10:19

Basically it's about running a bunch of playbooks and observing and making sure they don't fail?

<@nirik:matrix.scrye.com>

16:11:00

in the right order and making sure applications don't break after. :)

<@patrikp:matrix.org>

16:12:04

Excuse my ignorance but do you usually talk about it in the releng channel as you do those? I'd like to tag along/watch, looks like a practical thing to go through.

<@nirik:matrix.scrye.com>

16:12:23

In the #noc:fedoraproject.org channel usually.

<@nirik:matrix.scrye.com>

16:12:41

There's already an outage ticket and hackmd doc. ;)

<@patrikp:matrix.org>

16:13:20

Thanks, I'll take a look. I suppose this was the init? We kind of went into it freestyle but as Samyak is here I suppose let's move to the next section properly.

<@patrikp:matrix.org>

16:13:26

!info Here we list/discuss anything about items that are due to be done in the next week.

<@patrikp:matrix.org>

16:13:26

!link https://fedorapeople.org/groups/schedule/f-41/f-41-releng-tasks.html

<@patrikp:matrix.org>

16:13:26

!topic Scheduled actions coming up in the next week.

<@patrikp:matrix.org>

16:13:26

!link https://fedorapeople.org/groups/schedule/f-42/f-42-releng-tasks.html

<@patrikp:matrix.org>

16:13:49

Mass branching next week.

<@jnsamyak:matrix.org>

16:14:34

hehe I'm free for freestyling as well

<@nirik:matrix.scrye.com>

16:14:38

time goes by so fast. ;(

<@jnsamyak:matrix.org>

16:14:48

yeahhhhhhh ://///

<@jnsamyak:matrix.org>

16:15:01

I thought i would do something about it and it is here again

<@jnsamyak:matrix.org>

16:15:12

but nvm i had one quick thing about filing bugs

<@patrikp:matrix.org>

16:15:23

Sure, go ahead. 👍️

<@jnsamyak:matrix.org>

16:15:39

!releng 12545

<@zodbot:fedora.im>

16:15:40

<@zodbot:fedora.im>

16:15:40

● **Opened:** 4 days ago by churchyard

<@zodbot:fedora.im>

16:15:40

● **Assignee:** jnsamyak

<@zodbot:fedora.im>

16:15:40

● **Last Updated:** 4 days ago

<@zodbot:fedora.im>

16:15:40

**releng #12545** (https://pagure.io/releng/issue/12545):**300+ F42FTBFS bugzillas block the F41FTBFS tracker**

<@jnsamyak:matrix.org>

16:16:00

I feel I'm stuck at figuring bugzilla api

<@jnsamyak:matrix.org>

16:16:12

Last two comments, I have left my findings

<@nirik:matrix.scrye.com>

16:16:22

yeah, I am not sure who would know it better...

<@nirik:matrix.scrye.com>

16:17:07

how many bugs is it?

<@nirik:matrix.scrye.com>

16:17:18

ah, 334...

<@jnsamyak:matrix.org>

16:18:22

yeah, I could do it manually

<@jnsamyak:matrix.org>

16:18:27

but automation would be better

<@nirik:matrix.scrye.com>

16:18:28

can you just do a advanced query on the web interface and mass edit them there?

<@jnsamyak:matrix.org>

16:18:39

anyways I will figure something out

<@jnsamyak:matrix.org>

16:18:43

yeah selenium

<@jnsamyak:matrix.org>

16:18:46

is my last resort

<@nirik:matrix.scrye.com>

16:20:24

well, I meant I think there's some 'edit' thing for lists of bugs... can do it in one go... but I might be misremembering

<@jnsamyak:matrix.org>

16:20:41

I will check

<@jnsamyak:matrix.org>

16:20:54

thank you didnt know that

<@jnsamyak:matrix.org>

16:21:21

brb in 5 but that's all i have from my end on this topic

<@patrikp:matrix.org>

16:21:57

!link https://docs.fedoraproject.org/en-US/infra/release_guide/sop_mass_branching/

<@nirik:matrix.scrye.com>

16:22:38

there's also a checklist now I think? we should definitely use that

<@patrikp:matrix.org>

16:22:50

Ugh, already looking at the SOP...

<@nirik:matrix.scrye.com>

16:23:00

mass branching is easily the most complex part of the release cycle IMHO. So many things need changing

<@nirik:matrix.scrye.com>

16:23:19

https://docs.fedoraproject.org/en-US/infra/release_guide/mass_branching_checklist/

<@jnsamyak:matrix.org>

16:23:19

Yes there is, and I’ll open the tracker for that as well

<@nirik:matrix.scrye.com>

16:23:44

and... it would be great depending on time of course to actually start making pr's and getting everything lined up

<@jnsamyak:matrix.org>

16:25:07

I plan to start working on this week, so we can have all of then verified before next week

<@jnsamyak:matrix.org>

16:25:16

All helps are welcome 🤗

<@jnsamyak:matrix.org>

16:25:19

Hehe

<@patrikp:matrix.org>

16:25:34

Let's talk about this more about tomorrow's 1:1.

<@jnsamyak:matrix.org>

16:25:46

Sure thing

<@patrikp:matrix.org>

16:25:47

more during*

<@patrikp:matrix.org>

16:26:10

Alright. So that's it for upcoming actions and we went through the tickets... Let's see.

<@patrikp:matrix.org>

16:26:19

!topic Choose next chair.

<@patrikp:matrix.org>

16:26:35

Anybody want to host it? I can take it too, I don't mind it.

<@jnsamyak:matrix.org>

16:26:54

I can chair next week hopefullly if branching doesn’t bite my add

<@jnsamyak:matrix.org>

16:27:09

I can chair next week hopefullly if branching doesn’t bite my ss 😂

<@patrikp:matrix.org>

16:27:15

!info Next chair Feb 3rd: Samyak

<@patrikp:matrix.org>

16:27:39

!topic Discussion / Any other business.

<@patrikp:matrix.org>

16:27:49

We still have quite a bit of time. Anything else on your minds?

<@patrikp:matrix.org>

16:28:51

OK I have something then.

<@patrikp:matrix.org>

16:30:43

I'm playing with the OpenH composes, already talked to nirik about it, but now we can generate the repos, e.g.:

<@patrikp:matrix.org>

16:30:43

What exactly does it involve for it to be considered "multilib"? Wouldn't it only include adding 32 bit architectures? I'm missing the bigger picture.

<@patrikp:matrix.org>

16:30:43

<@patrikp:matrix.org>

16:30:43

https://kojipkgs.fedoraproject.org/repos-dist/f42-openh264/latest/

<@patrikp:matrix.org>

16:31:01

I made some attempts at the config file but unsuccessful.

<@nirik:matrix.scrye.com>

16:31:03

the koji dist-repo multilib is currently broken. ;(

<@patrikp:matrix.org>

16:31:15

But if it's just specifying arches in the config file anyway.....

<@nirik:matrix.scrye.com>

16:31:17

yeah, it's including i686 packages in the x86_64 repo

<@nirik:matrix.scrye.com>

16:31:30

(multilib that is)

<@patrikp:matrix.org>

16:31:59

So what does multilib mean in this context? I tried looking it up in a general sense but I'm not sure I get it.

<@nirik:matrix.scrye.com>

16:32:06

https://pagure.io/releng/issue/12473

<@nirik:matrix.scrye.com>

16:32:26

^ this is the request to do that (our current ones are not multilib)

<@nirik:matrix.scrye.com>

16:32:50

so, IMHO, we should not block on that... koji upstream is working on fixing it (I hope), so we can enable it later once thats fixed.

<@patrikp:matrix.org>

16:33:59

So we can ignore the multilib part? That makes it easier. Alright, in that case don't we have everything we need?

<@patrikp:matrix.org>

16:34:26

We have the rpms.

<@nirik:matrix.scrye.com>

16:34:28

yep. I think we do. as long as the repo looks/contains what the old one did (which it should).

<@nirik:matrix.scrye.com>

16:34:48

so, if the packages look right/good against the old one, I'd say we could send to cisco...

<@nirik:matrix.scrye.com>

16:35:06

at least for rawhide at first?

<@patrikp:matrix.org>

16:35:11

I will take a closer look, I have the older compose saved locally (odcs).

<@patrikp:matrix.org>

16:35:22

And write up how to actually get to this point...

<@patrikp:matrix.org>

16:35:39

Now the actual rpms should be available anywhere Koji is mounted?

<@nirik:matrix.scrye.com>

16:35:42

Yep. will be great to get it done/moving again

<@nirik:matrix.scrye.com>

16:35:50

yep.

<@jnsamyak:matrix.org>

16:36:02

+100

<@jnsamyak:matrix.org>

16:36:23

But I’m happy to see progress on this as well

<@patrikp:matrix.org>

16:36:51

I can see the rpms. 👍️

<@patrikp:matrix.org>

16:37:25

Again, for reference the SOP is here.

<@patrikp:matrix.org>

16:37:25

!link https://docs.fedoraproject.org/en-US/infra/releng_misc_guide/sop_generating_openh264_composes/

<@patrikp:matrix.org>

16:37:25

So hypothetical next steps once I confirm that we have everything and it can be shipped to Cisco...

<@patrikp:matrix.org>

16:37:59

And we are at the step where we would like to make the tarballs.

<@patrikp:matrix.org>

16:37:59

https://docs.fedoraproject.org/en-US/infra/releng_misc_guide/sop_generating_openh264_composes/#_make_the_changes_needed

<@jnsamyak:matrix.org>

16:38:05

Corrrect

<@patrikp:matrix.org>

16:38:50

And once Cisco publish them, what would be the next steps? Just sync to sundries which should be without issue and sync to mm-backend which now runs in OpenShift?

<@patrikp:matrix.org>

16:39:14

Or am I missing something?

<@jnsamyak:matrix.org>

16:40:17

No yes yes

<@jnsamyak:matrix.org>

16:40:52

Yes the last steps to be sync to sundries

<@patrikp:matrix.org>

16:41:32

The SOP is outdated, we ran into issues with mm being in OpenShift now but I'm pretty sure we overcame it so there should be a way to do it.

<@patrikp:matrix.org>

16:42:36

And dist-repo stores each individual compose so even a rollback shouldn't be difficult at all if something goes wrong, which was not the case before.

<@patrikp:matrix.org>

16:42:57

Overall these realizations make me very happy.

<@patrikp:matrix.org>

16:44:52

That's all I have on this topic.

<@patrikp:matrix.org>

16:49:59

I try to do it via this command:

<@patrikp:matrix.org>

16:49:59

I have one more thing, real quick. Ran into an issue doing package unretirements. When I try to push the unretirement commit I get:

<@patrikp:matrix.org>

16:49:59

`error: %patchN is obsolete, use %patch N (or %patch -P N): %patch0 -p1 -b .r

<@patrikp:matrix.org>

16:49:59

<@patrikp:matrix.org>

16:49:59

Could not execute pre_push_check: Failed to execute command.

<@patrikp:matrix.org>

16:49:59

error: failed to push some refs to 'ssh://pkgs.fedoraproject.org/rpms/tachyon'`

<@patrikp:matrix.org>

16:49:59

<@patrikp:matrix.org>

16:49:59

`GIT_SSH=/usr/local/bin/relengpush fedpkg --user releng push`

<@patrikp:matrix.org>

16:49:59

<@patrikp:matrix.org>

16:49:59

I thought it was a fedpkg thing but even if I try to push it directly via git it still fails.

<@patrikp:matrix.org>

16:49:59

`GIT_SSH=/usr/local/bin/relengpush git push origin`

<@patrikp:matrix.org>

16:49:59

<@patrikp:matrix.org>

16:49:59

What do I do? Should I fix the spec file and amend it to the unretirement commit? I don't want to mess with people's packages needlessly but it's a small enough change I guess.

<@nirik:matrix.scrye.com>

16:51:23

sorry, took longer than I thought it would... reading back

<@nirik:matrix.scrye.com>

16:52:03

pushing to sundries should be enough, mm now reads from that same volume

<@nirik:matrix.scrye.com>

16:52:37

On the unretire, the problem is not in your command, it's rpm changes... there might be a way to disable that precheck

<@nirik:matrix.scrye.com>

16:53:09

fedpkg has a --no-verify argument, passing that I think will bypass it.

<@patrikp:matrix.org>

16:55:02

When I grep for it (fedpkg --help | grep verify) I don't see such an option.

<@patrikp:matrix.org>

16:56:03

Maybe "pre-push-check" to NO?

<@nirik:matrix.scrye.com>

16:56:50

might depend on the fedpkg version? I was looking at rawhide here.

<@nirik:matrix.scrye.com>

16:57:16

oh, it's position dependent

<@nirik:matrix.scrye.com>

16:57:23

fedpkg push --help | grep verify

<@patrikp:matrix.org>

16:59:07

That did the trick!

<@patrikp:matrix.org>

16:59:10

Thanks!

<@patrikp:matrix.org>

16:59:16

And we are at time.

<@patrikp:matrix.org>

16:59:25

!info Thank you all for coming!

<@patrikp:matrix.org>

16:59:25

!endmeeting