#meeting-3:fedoraproject.org log

<@patrikp:matrix.org>

14:59:36

!startmeeting RELENG (2025-07-07)

<@meetbot:fedora.im>

14:59:38

Meeting started at 2025-07-07 14:59:36 UTC

<@meetbot:fedora.im>

14:59:38

The Meeting name is 'RELENG (2025-07-07)'

<@patrikp:matrix.org>

14:59:46

!info Meeting is 60 minutes MAX. At the end of 60, it stops.

<@patrikp:matrix.org>

14:59:46

!info Agenda is at https://hackmd.io/vm6biLBcTYKtkQUH5kQkmw.

<@patrikp:matrix.org>

14:59:46

!meetingname releng

<@patrikp:matrix.org>

14:59:46

!chair nirik jnsamyak patrikp amedvede

<@meetbot:fedora.im>

14:59:47

The Meeting Name is now releng

<@patrikp:matrix.org>

14:59:58

Good evening.

<@nirik:matrix.scrye.com>

15:01:10

morning

<@amedvede:fedora.im>

15:01:46

hey, monday warriors

<@patrikp:matrix.org>

15:02:07

Shall we go to init? I have something I'm hoping you can offer some advice on, Kevin.

<@jnsamyak:matrix.org>

15:02:08

hey hey

<@jnsamyak:matrix.org>

15:02:16

how's everyone?

<@patrikp:matrix.org>

15:02:28

All good. 👍️ Hello and welcome.

<@patrikp:matrix.org>

15:02:51

Should be a very easy one.

<@patrikp:matrix.org>

15:02:54

!topic Init process.

<@patrikp:matrix.org>

15:02:54

Do we have anything for the init? Any blockers/tasks/issues/requests/features that need releng intervention?

<@patrikp:matrix.org>

15:03:05

<@patrikp:matrix.org>

15:03:05

<@patrikp:matrix.org>

15:03:05

https://pagure.io/releng/issue/12779

<@patrikp:matrix.org>

15:03:05

<@patrikp:matrix.org>

15:03:05

I have looked through what documentation is available but I haven't been able to find anything relevant. What are we to do with such tickets? Could you clarify the process for me so that I may put it in an SOP, please?

<@patrikp:matrix.org>

15:03:05

<@patrikp:matrix.org>

15:03:05

1. Commit deletion policy

<@patrikp:matrix.org>

15:03:05

https://pagure.io/releng/issue/12793

<@patrikp:matrix.org>

15:03:05

<@patrikp:matrix.org>

15:03:05

I found a similar ticket from the past. It says that some of the builds were not signed and had to be re-tagged. What if there are many builds? Do they have to be checked and tagged individually? What are some handy Koji commands that could be useful here? Similar case, I would like to put it in an SOP.

<@patrikp:matrix.org>

15:03:05

<@patrikp:matrix.org>

15:03:05

https://pagure.io/releng/issue/11436

<@patrikp:matrix.org>

15:03:05

2. Stuck update

<@nirik:matrix.scrye.com>

15:04:58

so... lets focus on the first one first.

<@nirik:matrix.scrye.com>

15:05:05

Yes, there is a policy, let me try and find it.

<@nirik:matrix.scrye.com>

15:07:33

we basically have a script we can run that tells us if the commit can be deleted.

<@nirik:matrix.scrye.com>

15:08:26

oh, thats for branches, not commits tho

<@nirik:matrix.scrye.com>

15:08:40

for commits there's another thing

<@patrikp:matrix.org>

15:08:44

The releng scripts directory is a sight to behold.

<@patrikp:matrix.org>

15:09:42

I vaguely remember finding a reference to the COMMIT deletion somewhere in FESCo docs maybe. But couldn't find it again.

<@jnsamyak:matrix.org>

15:10:01

yeah but we dont delete a commit technically, unless it requires a legal thing to be removed we just revert a commit no?

<@patrikp:matrix.org>

15:10:25

I think the issue is that if you clone the repository it also clones the commit that was reverted?

<@nirik:matrix.scrye.com>

15:10:42

there's a clever way to move the commit to another branch thats not checked out by default.

<@nirik:matrix.scrye.com>

15:11:38

deleting commits normally is VERY BAD. because it re-writes history, so all the hashes change and any builds done before are no wrong and anyone who has it checked out is broken and has to reclone it.

<@patrikp:matrix.org>

15:12:43

The requester specifically mentioned nothing has been built yet. Does that factor in?

<@nirik:matrix.scrye.com>

15:12:54

<@patrikp:matrix.org>

15:13:15

So we have to do the different branch trick?

<@nirik:matrix.scrye.com>

15:14:29

https://pagure.io/releng/issue/7265

<@jnsamyak:matrix.org>

15:14:30

why can't we just revert it what's the problem?

<@amedvede:fedora.im>

15:14:46

whats wronge with this commit in first place?

<@nirik:matrix.scrye.com>

15:14:50

the problem is that they checked in a large binary, so forever after checking out the thing will be slow

<@jnsamyak:matrix.org>

15:15:11

it shows an empty file no?

<@patrikp:matrix.org>

15:15:28

If my understanding of git is correct then every commit is a snapshot. So it clones the tarball even if reverted.

<@nirik:matrix.scrye.com>

15:16:16

yeah, it's empty tho... so yeah... why not just revert it?

<@nirik:matrix.scrye.com>

15:16:38

The only case we should use the archive ref thing on is if someone checked in some big blob...

<@nirik:matrix.scrye.com>

15:16:55

in the past it was like the mono package had 100MB of tar.gzs in it.

<@patrikp:matrix.org>

15:16:55

In any case, it's not only about solving this one ticket, in time someone else will open another.

<@patrikp:matrix.org>

15:17:28

It has happened in the past and it'll happen again so it'd be good to have it written down.

<@jnsamyak:matrix.org>

15:17:28

exactly my point

<@nirik:matrix.scrye.com>

15:17:37

IMHO, it should be: 1) is it causing a problem? if it's just an empty file, it's not, just have them revert it.

<@nirik:matrix.scrye.com>

15:18:04

2) if it's a large binary file that will cause problems due to slow checkouts forever, then we can use this refs/archive trick

<@patrikp:matrix.org>

15:18:31

Is this something that could be brought to FESCo's attention? So they would give us guidance for how to handle it? Like a flowchart. Or it's up to us to decide?

<@nirik:matrix.scrye.com>

15:19:00

I think they already decided it a while back...

<@nirik:matrix.scrye.com>

15:19:12

AGREED: Move branches with huge commits to refs/archive/ and create new sanitized branches (+6, 0, 0)

<@nirik:matrix.scrye.com>

15:19:22

This is not a "huge commit"

<@patrikp:matrix.org>

15:19:35

Where did you get that from? That's what I've been looking for.

<@patrikp:matrix.org>

15:19:53

Ah, the issue linked above.

<@nirik:matrix.scrye.com>

15:19:58

I guess it's 475k

<@patrikp:matrix.org>

15:20:02

I'm blind, my apologies. So this one I consider completed.

<@patrikp:matrix.org>

15:20:26

I'll read through that and handle the ticket and write an SOP that gives more context about the policy.

<@nirik:matrix.scrye.com>

15:20:43

yes please. Also I guess "huge" is kinda subjective...

<@nirik:matrix.scrye.com>

15:20:56

is 475k huge? 1m? 100m?

<@patrikp:matrix.org>

15:22:25

That's actually a pretty important question, for this one I'll say "not huge" and put a disclaimer in the SOP about the vaguity.

<@amedvede:fedora.im>

15:22:27

right, lets put to the file what is "huge" commit for each of us, and average value will be considered as hure for SOP

<@patrikp:matrix.org>

15:23:53

And the stuck update one? I don't want to take up so much of the meeting, we're almost halfway through.

<@nirik:matrix.scrye.com>

15:24:03

oh yeah, lets see.

<@nirik:matrix.scrye.com>

15:24:36

yeah, thats in a weird state...

<@nirik:matrix.scrye.com>

15:25:56

so, yeah, we could try retagging those builds into f43-updates-testing-pending and see if bodhi wakes up... it might be also that it has to be tagged by autosign... not sure.

<@nirik:matrix.scrye.com>

15:26:21

I'd first try and just retag them and see if that works.

<@nirik:matrix.scrye.com>

15:26:42

it may be that bodhi didn't get the message that they were tagged/signed.

<@patrikp:matrix.org>

15:27:31

So you check the builds through the web UI to see what's wrong with the tags and then tag them manually, one by one, through Koji CLI? The workflow you did in the old ticket.

<@patrikp:matrix.org>

15:28:29

I don't understand exactly how it's clumped together. So one "update" can contain several "builds"? And is there some Koji CLI way to interact with "updates"?

<@nirik:matrix.scrye.com>

15:28:48

yeah, all the builds are in a single sigtag.

<@nirik:matrix.scrye.com>

15:29:43

So when the maintainer submits that as an update, bodhi finds all the builds there, tags them so autosign will sign them, autosign untags them, signs them and moves them to that updates-testing-pending tag and bodhi sees that so it knows they are signed and moves forward with it's process

<@patrikp:matrix.org>

15:30:39

And so it can be assumed that they are all tagged incorrectly as a whole? No need to inspect them further in Koji? I don't see more info about the individual builds in the web UI. Do you `koji list-history --build`?

<@nirik:matrix.scrye.com>

15:31:52

yeah, list-history, you can see it got to the part where autosign tagged them in, but bodhi didn't do anything after that like it should have.

<@nirik:matrix.scrye.com>

15:32:00

thats what makes me think the process broke down there.

<@patrikp:matrix.org>

15:32:44

OK, I think I got it, thanks a lot. That's all from me. Anything else for the init?

<@patrikp:matrix.org>

15:33:34

https://pagure.io/releng/issue/12807

<@patrikp:matrix.org>

15:33:34

Anton needs some permissions I think.

<@patrikp:matrix.org>

15:34:08

Is `packager` membership required for that?

<@amedvede:fedora.im>

15:34:28

yep, the reason of this that i cant proceed issues

<@nirik:matrix.scrye.com>

15:34:42

which sort of releng issues? there's different ones. ;)

<@patrikp:matrix.org>

15:35:03

Which step of unretirement needs `packager`?

<@amedvede:fedora.im>

15:35:47

1) i cant unblock tags on koji, because it requeres some rights

<@amedvede:fedora.im>

15:36:28

see koji list-permissions for some user

<@amedvede:fedora.im>

15:36:42

I'm not sure which group is given them

<@patrikp:matrix.org>

15:37:11

That's set in Koji itself if I recall correctly?

<@amedvede:fedora.im>

15:37:48

2) packager permition because I can't modify packages for example i can't add a person as a contributor for epel branches

<@jnsamyak:matrix.org>

15:38:30

okay i was away for 5, back on desk

<@patrikp:matrix.org>

15:39:10

That's not `packager` permission, that's `cvsadmin` in src.fedoraproject.org I think. Not sure.

<@jnsamyak:matrix.org>

15:39:12

so it's either the sysadmin / packagers for former i think this was required for looking at the src.fpo settings, and for the latter unretiring the package

<@jnsamyak:matrix.org>

15:40:06

cvsadmin for fedora scm processes if i'm not wrong i'll look into it in IPA for anton is in cvsadmin did you log in and log out Anton Medvedev ?

<@amedvede:fedora.im>

15:40:37

3) sysadmin permission, like I got torrentadmin permittion and it should be enough to make changes in torrent server, but it seems that torrent server doesn't recieve updates on my groups since friday, which is strange, usually it is faster, I'm not sure if its connected to the datacenter move

<@amedvede:fedora.im>

15:40:54

i did

<@amedvede:fedora.im>

15:41:21

like still nothing

<@jnsamyak:matrix.org>

15:41:21

maybe because of dc move it would be affected but can

<@jnsamyak:matrix.org>

15:41:40

maybe because of dc move it would be affected but can't assure you on that

<@jnsamyak:matrix.org>

15:42:24

i think patrik had the same issue when he started doing unretirements hence the packager group access was given

<@patrikp:matrix.org>

15:42:41

I don't think `packager` has anything to do with it.

<@amedvede:fedora.im>

15:42:41

tbh there is same issue that I can't log out, its just instantly relogin me

<@jnsamyak:matrix.org>

15:43:04

patrikp: otherwise do you own a package? Were you in the packager group before?

<@jnsamyak:matrix.org>

15:43:12

clear cache?

<@jnsamyak:matrix.org>

15:43:20

incognito :)

<@patrikp:matrix.org>

15:43:32

I got into `packager` the legitimate way as an intern but I don't think it's needed for anything. I just happen to be in it.

<@patrikp:matrix.org>

15:43:36

So for unretirements...

<@patrikp:matrix.org>

15:44:58

I think you need:

<@patrikp:matrix.org>

15:44:58

1) sysadmin-releng to be able to ssh to the compose machine

<@patrikp:matrix.org>

15:44:58

2) Koji permission to unblock the tags, this is set in Koji itself

<@patrikp:matrix.org>

15:44:58

3) `cvsadmin` group membership on src.fedoraproject.org in order to be able to access the settings of dist-git repos

<@nirik:matrix.scrye.com>

15:45:38

so, yeah, torrent may well be not updating. I noted remote machines need to point to the new dc for auth. I can try and fix that today/soon.

<@patrikp:matrix.org>

15:46:21

It's absolutely crazy how complicated all of this is for our infra.

<@amedvede:fedora.im>

15:46:22

1. right, but torrent machine is different for example to run sudo sysadmin-releng is not enough

<@patrikp:matrix.org>

15:46:39

I talked *specifically* about unretirements.

<@jnsamyak:matrix.org>

15:46:46

nirik: btw i gave anton cvsadmin today in my morning, for scm requests and for settings; just fying

<@nirik:matrix.scrye.com>

15:47:23

ok, yeah, packager shouldn't be needed. But packaging is fun if you want to go thru that process. ;)

<@patrikp:matrix.org>

15:47:55

Important question, take it one at a time? Let's figure out unretirements first then do the next?

<@amedvede:fedora.im>

15:48:08

oh I see, like it was strange, because i didn't see this machine listed in hackmd doc for datacenter move and i don't have datacenter location in its dns name so I didn't know how to get info

<@nirik:matrix.scrye.com>

15:48:23

yeah, one at a time sounds good.

<@nirik:matrix.scrye.com>

15:48:32

yes, it didn't move, but the authentication servers it uses did.

<@patrikp:matrix.org>

15:48:44

Does this sound right to you?

<@nirik:matrix.scrye.com>

15:48:54

yep

<@nirik:matrix.scrye.com>

15:49:18

I have a idea how to make koji permissions use our groups, but haven't had time to test it yet...

<@patrikp:matrix.org>

15:50:30

We only have 10 minutes to go. Do you want to go Kevin? Sorry, failure to moderate on my part.

<@nirik:matrix.scrye.com>

15:50:47

no worries... good discussions/info...

<@nirik:matrix.scrye.com>

15:51:27

So one thing I noticed... it seems we didn't complete the 42 cleanup yet? we were delayed by mirrormanager not updating, but that got fixed... I think there's a ticket, but I can't seem to find it?

<@jnsamyak:matrix.org>

15:51:49

yeah

<@amedvede:fedora.im>

15:51:58

it brings me the idea, that we may need to create a doc page where is writen which group what permissions give, we probably have it somewhere, but it's not trivial task to find

<@nirik:matrix.scrye.com>

15:52:15

good idea

<@jnsamyak:matrix.org>

15:52:17

I have an issue on removing modular tables on the db, with datacenter move i didnt do anything

<@jnsamyak:matrix.org>

15:52:32

this is where the move_to_archive playbook fails

<@patrikp:matrix.org>

15:52:37

!releng 12700

<@nirik:matrix.scrye.com>

15:52:41

this was development/42 , etc trees

<@zodbot:fedora.im>

15:52:42

● **Assignee:** amedvede

<@zodbot:fedora.im>

15:52:42

<@zodbot:fedora.im>

15:52:42

**releng #12700** (https://pagure.io/releng/issue/12700):**F42 Post Release Cleanup**

<@zodbot:fedora.im>

15:52:42

● **Closed: Fixed** a month ago by amedvede

<@zodbot:fedora.im>

15:52:42

● **Opened:** 3 months ago by jnsamyak

<@zodbot:fedora.im>

15:52:42

● **Last Updated:** a month ago

<@jnsamyak:matrix.org>

15:52:48

`Error: Unable to find a directory in [Fedora Archive] for pub/fedora/linux/updates/testing/40/Modular/source/tree`

<@nirik:matrix.scrye.com>

15:53:14

oh, it's an empty dir...

<@nirik:matrix.scrye.com>

15:53:17

https://dl.fedoraproject.org/pub/fedora/linux/development/42/

<@nirik:matrix.scrye.com>

15:53:41

and https://dl.fedoraproject.org/pub/fedora-secondary/development/42/

<@nirik:matrix.scrye.com>

15:54:04

so, yeah, sorry, it's almost all done. We should nuke those dirs. ;)

<@nirik:matrix.scrye.com>

15:54:29

oh yeah, the modular stuff should be removed.

<@jnsamyak:matrix.org>

15:54:33

Anton Medvedev: can you make a note of it please?

<@jnsamyak:matrix.org>

15:54:41

I remember you were doing these cleanups

<@amedvede:fedora.im>

15:55:14

yeah i was doing it

<@jnsamyak:matrix.org>

15:55:19

I needed to discuss the modular db issue but dont have time now, I need to jump in one 1:1 just after this, so I'll bring it to releng channel later

<@nirik:matrix.scrye.com>

15:55:32

sure, no hurry...

<@nirik:matrix.scrye.com>

15:56:38

so, one last final item: I resetup the sigul CA in the new datacenter... I could have just re-used the old one, but I didn't. ;( So, That means we need new sigul certs for folks who do signing tasks. I hope I can set that up right for you all, but your existing certs won't work

<@nirik:matrix.scrye.com>

15:57:02

I'll try and do that soon, but if you need to sign something, please poke me to get you setup

<@patrikp:matrix.org>

15:57:12

This *has to* be documented as you are doing it. It was extremely painful.

<@jnsamyak:matrix.org>

15:57:14

oh no :(

<@jnsamyak:matrix.org>

15:57:37

lol

<@nirik:matrix.scrye.com>

15:57:39

yes, I tore my hair out over it... but I now understand it better

<@patrikp:matrix.org>

15:57:42

The doc I saw suuuuuucks.

<@nirik:matrix.scrye.com>

15:57:46

so docs is 100% part of th eplan

<@nirik:matrix.scrye.com>

15:58:30

yes, the current docs are... not good

<@patrikp:matrix.org>

15:58:31

Did it also trigger a PTSD reaction for you?

<@jnsamyak:matrix.org>

15:58:43

aye aye

<@patrikp:matrix.org>

15:59:11

With that, any takers for next week's host?

<@amedvede:fedora.im>

15:59:43

I can take it

<@patrikp:matrix.org>

15:59:57

!topic Choose next chair.

<@patrikp:matrix.org>

15:59:57

!info Next chair July 14th: Anton

<@patrikp:matrix.org>

16:00:08

And we are at time.

<@patrikp:matrix.org>

16:00:14

!info Thank you all for coming!

<@patrikp:matrix.org>

16:00:14

!endmeeting