2025-07-07 14:59:36 <@patrikp:matrix.org> !startmeeting RELENG (2025-07-07)
2025-07-07 14:59:38 <@meetbot:fedora.im> Meeting started at 2025-07-07 14:59:36 UTC
2025-07-07 14:59:38 <@meetbot:fedora.im> The Meeting name is 'RELENG (2025-07-07)'
2025-07-07 14:59:46 <@patrikp:matrix.org> !info Meeting is 60 minutes MAX. At the end of 60, it stops.
2025-07-07 14:59:46 <@patrikp:matrix.org> !info Agenda is at https://hackmd.io/vm6biLBcTYKtkQUH5kQkmw.
2025-07-07 14:59:46 <@patrikp:matrix.org> !meetingname releng
2025-07-07 14:59:46 <@patrikp:matrix.org> !chair nirik jnsamyak patrikp amedvede
2025-07-07 14:59:47 <@meetbot:fedora.im> The Meeting Name is now releng
2025-07-07 14:59:58 <@patrikp:matrix.org> Good evening.
2025-07-07 15:01:10 <@nirik:matrix.scrye.com> morning
2025-07-07 15:01:46 <@amedvede:fedora.im> hey, monday warriors
2025-07-07 15:02:07 <@patrikp:matrix.org> Shall we go to init? I have something I'm hoping you can offer some advice on, Kevin.
2025-07-07 15:02:08 <@jnsamyak:matrix.org> hey hey
2025-07-07 15:02:16 <@jnsamyak:matrix.org> how's everyone?
2025-07-07 15:02:28 <@patrikp:matrix.org> All good. 👍️ Hello and welcome.
2025-07-07 15:02:51 <@patrikp:matrix.org> Should be a very easy one.
2025-07-07 15:02:54 <@patrikp:matrix.org> !topic Init process.
2025-07-07 15:02:54 <@patrikp:matrix.org> Do we have anything for the init? Any blockers/tasks/issues/requests/features that need releng intervention?
2025-07-07 15:03:05 <@patrikp:matrix.org>  
2025-07-07 15:03:05 <@patrikp:matrix.org>  
2025-07-07 15:03:05 <@patrikp:matrix.org>  https://pagure.io/releng/issue/12779
2025-07-07 15:03:05 <@patrikp:matrix.org>  
2025-07-07 15:03:05 <@patrikp:matrix.org>  I have looked through what documentation is available but I haven't been able to find anything relevant. What are we to do with such         tickets? Could you clarify the process for me so that I may put it in an SOP, please?
2025-07-07 15:03:05 <@patrikp:matrix.org>  
2025-07-07 15:03:05 <@patrikp:matrix.org> 1. Commit deletion policy
2025-07-07 15:03:05 <@patrikp:matrix.org>  https://pagure.io/releng/issue/12793
2025-07-07 15:03:05 <@patrikp:matrix.org>  
2025-07-07 15:03:05 <@patrikp:matrix.org>  I found a similar ticket from the past. It says that some of the builds were not signed and had to be re-tagged. What if there are many      builds? Do they have to be checked and tagged individually? What are some handy Koji commands that could be useful here? Similar case, I     would like to put it in an SOP.
2025-07-07 15:03:05 <@patrikp:matrix.org>  
2025-07-07 15:03:05 <@patrikp:matrix.org>  https://pagure.io/releng/issue/11436
2025-07-07 15:03:05 <@patrikp:matrix.org>  2. Stuck update
2025-07-07 15:04:58 <@nirik:matrix.scrye.com> so... lets focus on the first one first.
2025-07-07 15:05:05 <@nirik:matrix.scrye.com> Yes, there is a policy, let me try and find it.
2025-07-07 15:07:33 <@nirik:matrix.scrye.com> we basically have a script we can run that tells us if the commit can be deleted.
2025-07-07 15:08:26 <@nirik:matrix.scrye.com> oh, thats for branches, not commits tho
2025-07-07 15:08:40 <@nirik:matrix.scrye.com> for commits there's another thing
2025-07-07 15:08:44 <@patrikp:matrix.org> The releng scripts directory is a sight to behold.
2025-07-07 15:09:42 <@patrikp:matrix.org> I vaguely remember finding a reference to the COMMIT deletion somewhere in FESCo docs maybe. But couldn't find it again.
2025-07-07 15:10:01 <@jnsamyak:matrix.org> yeah but we dont delete a commit technically, unless it requires a legal thing to be removed we just revert a commit no?
2025-07-07 15:10:25 <@patrikp:matrix.org> I think the issue is that if you clone the repository it also clones the commit that was reverted?
2025-07-07 15:10:42 <@nirik:matrix.scrye.com> there's a clever way to move the commit to another branch thats not checked out by default.
2025-07-07 15:11:38 <@nirik:matrix.scrye.com> deleting commits normally is VERY BAD. because it re-writes history, so all the hashes change and any builds done before are no wrong and anyone who has it checked out is broken and has to reclone it.
2025-07-07 15:12:43 <@patrikp:matrix.org> The requester specifically mentioned nothing has been built yet. Does that factor in?
2025-07-07 15:12:54 <@nirik:matrix.scrye.com> no
2025-07-07 15:13:15 <@patrikp:matrix.org> So we have to do the different branch trick?
2025-07-07 15:14:29 <@nirik:matrix.scrye.com> https://pagure.io/releng/issue/7265
2025-07-07 15:14:30 <@jnsamyak:matrix.org> why can't we just revert it what's the problem?
2025-07-07 15:14:46 <@amedvede:fedora.im> whats wronge with this commit in first place?
2025-07-07 15:14:50 <@nirik:matrix.scrye.com> the problem is that they checked in a large binary, so forever after checking out the thing will be slow
2025-07-07 15:15:11 <@jnsamyak:matrix.org> it shows an empty file no?
2025-07-07 15:15:28 <@patrikp:matrix.org> If my understanding of git is correct then every commit is a snapshot. So it clones the tarball even if reverted.
2025-07-07 15:16:16 <@nirik:matrix.scrye.com> yeah, it's empty tho... so yeah... why not just revert it?
2025-07-07 15:16:38 <@nirik:matrix.scrye.com> The only case we should use the archive ref thing on is if someone checked in some big blob...
2025-07-07 15:16:55 <@nirik:matrix.scrye.com> in the past it was like the mono package had 100MB of tar.gzs in it.
2025-07-07 15:16:55 <@patrikp:matrix.org> In any case, it's not only about solving this one ticket, in time someone else will open another.
2025-07-07 15:17:28 <@patrikp:matrix.org> It has happened in the past and it'll happen again so it'd be good to have it written down.
2025-07-07 15:17:28 <@jnsamyak:matrix.org> exactly my point
2025-07-07 15:17:37 <@nirik:matrix.scrye.com> IMHO, it should be: 1) is it causing a problem? if it's just an empty file, it's not, just have them revert it.
2025-07-07 15:18:04 <@nirik:matrix.scrye.com> 2) if it's a large binary file that will cause problems due to slow checkouts forever, then we can use this refs/archive trick
2025-07-07 15:18:31 <@patrikp:matrix.org> Is this something that could be brought to FESCo's attention? So they would give us guidance for how to handle it? Like a flowchart. Or it's up to us to decide?
2025-07-07 15:19:00 <@nirik:matrix.scrye.com> I think they already decided it a while back...
2025-07-07 15:19:12 <@nirik:matrix.scrye.com> AGREED: Move branches with huge commits to refs/archive/ and create new sanitized branches (+6, 0, 0)
2025-07-07 15:19:22 <@nirik:matrix.scrye.com> This is not a "huge commit"
2025-07-07 15:19:35 <@patrikp:matrix.org> Where did you get that from? That's what I've been looking for.
2025-07-07 15:19:53 <@patrikp:matrix.org> Ah, the issue linked above.
2025-07-07 15:19:58 <@nirik:matrix.scrye.com> I guess it's 475k
2025-07-07 15:20:02 <@patrikp:matrix.org> I'm blind, my apologies. So this one I consider completed.
2025-07-07 15:20:26 <@patrikp:matrix.org> I'll read through that and handle the ticket and write an SOP that gives more context about the policy.
2025-07-07 15:20:43 <@nirik:matrix.scrye.com> yes please. Also I guess "huge" is kinda subjective...
2025-07-07 15:20:56 <@nirik:matrix.scrye.com> is 475k huge? 1m? 100m?
2025-07-07 15:22:25 <@patrikp:matrix.org> That's actually a pretty important question, for this one I'll say "not huge" and put a disclaimer in the SOP about the vaguity.
2025-07-07 15:22:27 <@amedvede:fedora.im> right, lets put to the file what is "huge" commit for each of us, and average value will be considered as hure for SOP
2025-07-07 15:23:53 <@patrikp:matrix.org> And the stuck update one? I don't want to take up so much of the meeting, we're almost halfway through.
2025-07-07 15:24:03 <@nirik:matrix.scrye.com> oh yeah, lets see.
2025-07-07 15:24:36 <@nirik:matrix.scrye.com> yeah, thats in a weird state...
2025-07-07 15:25:56 <@nirik:matrix.scrye.com> so, yeah, we could try retagging those builds into f43-updates-testing-pending and see if bodhi wakes up... it might be also that it has to be tagged by autosign... not sure.
2025-07-07 15:26:21 <@nirik:matrix.scrye.com> I'd first try and just retag them and see if that works.
2025-07-07 15:26:42 <@nirik:matrix.scrye.com> it may be that bodhi didn't get the message that they were tagged/signed.
2025-07-07 15:27:31 <@patrikp:matrix.org> So you check the builds through the web UI to see what's wrong with the tags and then tag them manually, one by one, through Koji CLI? The workflow you did in the old ticket.
2025-07-07 15:28:29 <@patrikp:matrix.org> I don't understand exactly how it's clumped together. So one "update" can contain several "builds"? And is there some Koji CLI way to interact with "updates"?
2025-07-07 15:28:48 <@nirik:matrix.scrye.com> yeah, all the builds are in a single sigtag.
2025-07-07 15:29:43 <@nirik:matrix.scrye.com> So when the maintainer submits that as an update, bodhi finds all the builds there, tags them so autosign will sign them, autosign untags them, signs them and moves them to that updates-testing-pending tag and bodhi sees that so it knows they are signed and moves forward with it's process
2025-07-07 15:30:39 <@patrikp:matrix.org> And so it can be assumed that they are all tagged incorrectly as a whole? No need to inspect them further in Koji? I don't see more info about the individual builds in the web UI. Do you `koji list-history --build`?
2025-07-07 15:31:52 <@nirik:matrix.scrye.com> yeah, list-history, you can see it got to the part where autosign tagged them in, but bodhi didn't do anything after that like it should have.
2025-07-07 15:32:00 <@nirik:matrix.scrye.com> thats what makes me think the process broke down there.
2025-07-07 15:32:44 <@patrikp:matrix.org> OK, I think I got it, thanks a lot. That's all from me. Anything else for the init?
2025-07-07 15:33:34 <@patrikp:matrix.org> https://pagure.io/releng/issue/12807
2025-07-07 15:33:34 <@patrikp:matrix.org> Anton needs some permissions I think.
2025-07-07 15:34:08 <@patrikp:matrix.org> Is `packager` membership required for that?
2025-07-07 15:34:28 <@amedvede:fedora.im> yep, the reason of this that i cant proceed issues
2025-07-07 15:34:42 <@nirik:matrix.scrye.com> which sort of releng issues? there's different ones. ;)
2025-07-07 15:35:03 <@patrikp:matrix.org> Which step of unretirement needs `packager`?
2025-07-07 15:35:47 <@amedvede:fedora.im> 1) i cant unblock tags on koji, because it requeres some rights
2025-07-07 15:36:28 <@amedvede:fedora.im> see koji list-permissions for some user
2025-07-07 15:36:42 <@amedvede:fedora.im> I'm not sure which group is given them
2025-07-07 15:37:11 <@patrikp:matrix.org> That's set in Koji itself if I recall correctly?
2025-07-07 15:37:48 <@amedvede:fedora.im> 2) packager permition because I can't modify packages for example i can't add a person as a contributor for epel branches
2025-07-07 15:38:30 <@jnsamyak:matrix.org> okay i was away for 5, back on desk
2025-07-07 15:39:10 <@patrikp:matrix.org> That's not `packager` permission, that's `cvsadmin` in src.fedoraproject.org I think. Not sure.
2025-07-07 15:39:12 <@jnsamyak:matrix.org> so it's either the sysadmin / packagers for former i think this was required for looking at the src.fpo settings, and for the latter unretiring the package
2025-07-07 15:40:06 <@jnsamyak:matrix.org> cvsadmin for fedora scm processes if i'm not wrong i'll look into it in IPA for anton is in cvsadmin did you log in and log out Anton Medvedev ?
2025-07-07 15:40:37 <@amedvede:fedora.im> 3) sysadmin permission, like I got torrentadmin permittion and it should be enough to make changes in torrent server, but it seems that torrent server doesn't recieve updates on my groups since friday, which is strange, usually it is faster, I'm not sure if its connected to the datacenter move
2025-07-07 15:40:54 <@amedvede:fedora.im> i did
2025-07-07 15:41:21 <@amedvede:fedora.im> like still nothing
2025-07-07 15:41:21 <@jnsamyak:matrix.org> maybe because of dc move it would be affected but can
2025-07-07 15:41:40 <@jnsamyak:matrix.org> maybe because of dc move it would be affected but can't assure you on that
2025-07-07 15:42:24 <@jnsamyak:matrix.org> i think patrik had the same issue when he started doing unretirements hence the packager group access was given 
2025-07-07 15:42:41 <@patrikp:matrix.org> I don't think `packager` has anything to do with it.
2025-07-07 15:42:41 <@amedvede:fedora.im> tbh there is same issue that I can't log out, its just instantly relogin me
2025-07-07 15:43:04 <@jnsamyak:matrix.org> patrikp: otherwise do you own a package? Were you in the packager group before?
2025-07-07 15:43:12 <@jnsamyak:matrix.org> clear cache?
2025-07-07 15:43:20 <@jnsamyak:matrix.org> incognito :)
2025-07-07 15:43:32 <@patrikp:matrix.org> I got into `packager` the legitimate way as an intern but I don't think it's needed for anything. I just happen to be in it.
2025-07-07 15:43:36 <@patrikp:matrix.org> So for unretirements...
2025-07-07 15:44:58 <@patrikp:matrix.org> I think you need:
2025-07-07 15:44:58 <@patrikp:matrix.org> 1) sysadmin-releng to be able to ssh to the compose machine
2025-07-07 15:44:58 <@patrikp:matrix.org> 2) Koji permission to unblock the tags, this is set in Koji itself
2025-07-07 15:44:58 <@patrikp:matrix.org> 3) `cvsadmin` group membership on src.fedoraproject.org in order to be able to access the settings of dist-git repos
2025-07-07 15:45:38 <@nirik:matrix.scrye.com> so, yeah, torrent may well be not updating. I noted remote machines need to point to the new dc for auth. I can try and fix that today/soon.
2025-07-07 15:46:21 <@patrikp:matrix.org> It's absolutely crazy how complicated all of this is for our infra.
2025-07-07 15:46:22 <@amedvede:fedora.im> 1. right, but torrent machine is different for example to run sudo sysadmin-releng is not enough
2025-07-07 15:46:39 <@patrikp:matrix.org> I talked *specifically* about unretirements.
2025-07-07 15:46:46 <@jnsamyak:matrix.org> nirik: btw i gave anton cvsadmin today in my morning, for scm requests and for settings; just fying
2025-07-07 15:47:23 <@nirik:matrix.scrye.com> ok, yeah, packager shouldn't be needed. But packaging is fun if you want to go thru that process. ;)
2025-07-07 15:47:55 <@patrikp:matrix.org> Important question, take it one at a time? Let's figure out unretirements first then do the next?
2025-07-07 15:48:08 <@amedvede:fedora.im> oh I see, like it was strange, because i didn't see this machine listed in hackmd doc for datacenter move and i don't have datacenter location in its dns name so I didn't know how to get info
2025-07-07 15:48:23 <@nirik:matrix.scrye.com> yeah, one at a time sounds good.
2025-07-07 15:48:32 <@nirik:matrix.scrye.com> yes, it didn't move, but the authentication servers it uses did.
2025-07-07 15:48:44 <@patrikp:matrix.org> Does this sound right to you?
2025-07-07 15:48:54 <@nirik:matrix.scrye.com> yep
2025-07-07 15:49:18 <@nirik:matrix.scrye.com> I have a idea how to make koji permissions use our groups, but haven't had time to test it yet...
2025-07-07 15:50:30 <@patrikp:matrix.org> We only have 10 minutes to go. Do you want to go Kevin? Sorry, failure to moderate on my part.
2025-07-07 15:50:47 <@nirik:matrix.scrye.com> no worries... good discussions/info...
2025-07-07 15:51:27 <@nirik:matrix.scrye.com> So one thing I noticed... it seems we didn't complete the 42 cleanup yet? we were delayed by mirrormanager not updating, but that got fixed... I think there's a ticket, but I can't seem to find it?
2025-07-07 15:51:49 <@jnsamyak:matrix.org> yeah 
2025-07-07 15:51:58 <@amedvede:fedora.im> it brings me the idea, that we may need to create a doc page where is writen which group what permissions give, we probably have it somewhere, but it's not trivial task to find
2025-07-07 15:52:15 <@nirik:matrix.scrye.com> good idea
2025-07-07 15:52:17 <@jnsamyak:matrix.org> I have an issue on removing modular tables on the db, with datacenter move i didnt do anything 
2025-07-07 15:52:32 <@jnsamyak:matrix.org> this is where the move_to_archive playbook fails
2025-07-07 15:52:37 <@patrikp:matrix.org> !releng 12700
2025-07-07 15:52:41 <@nirik:matrix.scrye.com> this was development/42 , etc trees
2025-07-07 15:52:42 <@zodbot:fedora.im> ● **Assignee:** amedvede
2025-07-07 15:52:42 <@zodbot:fedora.im> 
2025-07-07 15:52:42 <@zodbot:fedora.im> **releng #12700** (https://pagure.io/releng/issue/12700):**F42 Post Release Cleanup**
2025-07-07 15:52:42 <@zodbot:fedora.im> ● **Closed: Fixed** a month ago by amedvede
2025-07-07 15:52:42 <@zodbot:fedora.im> ● **Opened:** 3 months ago by jnsamyak
2025-07-07 15:52:42 <@zodbot:fedora.im> ● **Last Updated:** a month ago
2025-07-07 15:52:48 <@jnsamyak:matrix.org> `Error: Unable to find a directory in [Fedora Archive] for pub/fedora/linux/updates/testing/40/Modular/source/tree`
2025-07-07 15:53:14 <@nirik:matrix.scrye.com> oh, it's an empty dir...
2025-07-07 15:53:17 <@nirik:matrix.scrye.com> https://dl.fedoraproject.org/pub/fedora/linux/development/42/
2025-07-07 15:53:41 <@nirik:matrix.scrye.com> and https://dl.fedoraproject.org/pub/fedora-secondary/development/42/
2025-07-07 15:54:04 <@nirik:matrix.scrye.com> so, yeah, sorry, it's almost all done. We should nuke those dirs. ;)
2025-07-07 15:54:29 <@nirik:matrix.scrye.com> oh yeah, the modular stuff should be removed.
2025-07-07 15:54:33 <@jnsamyak:matrix.org> Anton Medvedev: can you make a note of it please?
2025-07-07 15:54:41 <@jnsamyak:matrix.org> I remember you were doing these cleanups
2025-07-07 15:55:14 <@amedvede:fedora.im> yeah i was doing it
2025-07-07 15:55:19 <@jnsamyak:matrix.org> I needed to discuss the modular db issue but dont have time now, I need to jump in one 1:1 just after this, so I'll bring it to releng channel later 
2025-07-07 15:55:32 <@nirik:matrix.scrye.com> sure, no hurry...
2025-07-07 15:56:38 <@nirik:matrix.scrye.com> so, one last final item: I resetup the sigul CA in the new datacenter... I could have just re-used the old one, but I didn't. ;( So, That means we need new sigul certs for folks who do signing tasks. I hope I can set that up right for you all, but your existing certs won't work
2025-07-07 15:57:02 <@nirik:matrix.scrye.com> I'll try and do that soon, but if you need to sign something, please poke me to get you setup
2025-07-07 15:57:12 <@patrikp:matrix.org> This *has to* be documented as you are doing it. It was extremely painful.
2025-07-07 15:57:14 <@jnsamyak:matrix.org> oh no :( 
2025-07-07 15:57:37 <@jnsamyak:matrix.org> lol
2025-07-07 15:57:39 <@nirik:matrix.scrye.com> yes, I tore my hair out over it... but I now understand it better
2025-07-07 15:57:42 <@patrikp:matrix.org> The doc I saw suuuuuucks.
2025-07-07 15:57:46 <@nirik:matrix.scrye.com> so docs is 100% part of th eplan
2025-07-07 15:58:30 <@nirik:matrix.scrye.com> yes, the current docs are... not good
2025-07-07 15:58:31 <@patrikp:matrix.org> Did it also trigger a PTSD reaction for you?
2025-07-07 15:58:43 <@jnsamyak:matrix.org> aye aye
2025-07-07 15:59:11 <@patrikp:matrix.org> With that, any takers for next week's host?
2025-07-07 15:59:43 <@amedvede:fedora.im> I can take it
2025-07-07 15:59:57 <@patrikp:matrix.org> !topic Choose next chair.
2025-07-07 15:59:57 <@patrikp:matrix.org> !info Next chair July 14th: Anton
2025-07-07 16:00:08 <@patrikp:matrix.org> And we are at time.
2025-07-07 16:00:14 <@patrikp:matrix.org> !info Thank you all for coming!
2025-07-07 16:00:14 <@patrikp:matrix.org> !endmeeting