2023-12-05 17:00:54 <@humaton:fedora.im> !startmeeting RELENG (2023-12-05) 2023-12-05 17:00:55 <@meetbot:fedora.im> Meeting started at 2023-12-05 17:00:54 UTC 2023-12-05 17:00:55 <@meetbot:fedora.im> The Meeting name is 'RELENG (2023-12-05)' 2023-12-05 17:01:02 <@humaton:fedora.im> !meetingname releng 2023-12-05 17:01:07 <@humaton:fedora.im> !chair nirik sharkcz pbrobinson phsmoura dustymabe jednorozec 2023-12-05 17:01:14 <@humaton:fedora.im> !topic init process 2023-12-05 17:02:51 <@humaton:fedora.im> Hello everybody 2023-12-05 17:03:02 <@patrikp:matrix.org> Good evening. 2023-12-05 17:03:06 <@nirik:matrix.scrye.com> !hi 2023-12-05 17:03:08 <@zodbot:fedora.im> Kevin Fenzi (kevin) - he / him / his 2023-12-05 17:03:54 <@humaton:fedora.im> !topic Fedora 37 is going EOL today 2023-12-05 17:04:05 <@humaton:fedora.im> so the ansible changes are in PR 2023-12-05 17:04:11 <@humaton:fedora.im> koji targets are removed 2023-12-05 17:05:19 <@humaton:fedora.im> so there is a bunch of updates in testing for the release https://bodhi.fedoraproject.org/releases/F37 2023-12-05 17:05:45 <@humaton:fedora.im> should we "force them" to stable or keep them there? 2023-12-05 17:06:25 <@nirik:matrix.scrye.com> nope, they didn't make it, too bad. 2023-12-05 17:06:57 <@humaton:fedora.im> ok one sec update for firefox made it after the stable push 2023-12-05 17:06:58 <@nirik:matrix.scrye.com> well, we should do one last stable push... 2023-12-05 17:07:07 <@nirik:matrix.scrye.com> ah, thats the one I saw. 2023-12-05 17:07:21 <@humaton:fedora.im> I am running one more for stable for the ff update and will move the bodhi release to "archived" 2023-12-05 17:07:48 <@humaton:fedora.im> Well I did the push about hour ago and the ff update made it to stable emanwhile... 2023-12-05 17:07:52 <@nirik:matrix.scrye.com> sounds good. We shouldn't mass push from testing, there may be good reasons none of those were requested stable... and maintainers had a while to make it happen. 2023-12-05 17:08:05 <@humaton:fedora.im> noted 2023-12-05 17:08:50 <@humaton:fedora.im> also the release is set to active:false in PDC 2023-12-05 17:09:02 <@humaton:fedora.im> all packages should have correct EOL for f37 branch there 2023-12-05 17:09:25 <@humaton:fedora.im> so what is missing is https://docs.fedoraproject.org/en-US/infra/release_guide/release_eol/#_bodhi_tasks and the following move to archive steps 2023-12-05 17:10:24 <@nirik:matrix.scrye.com> nd the announcement 2023-12-05 17:10:42 <@humaton:fedora.im> oh right the final one 2023-12-05 17:10:45 <@nirik:matrix.scrye.com> Oh, later today I will also take care of /mnt/koji/compose/f37/ since we don't need it anymore. 2023-12-05 17:11:07 <@humaton:fedora.im> cool 2023-12-05 17:13:00 <@nirik:matrix.scrye.com> oh, and updates-testing thing we should consider a solution to. 2023-12-05 17:13:06 <@humaton:fedora.im> go for it I will do the EOL announcement 2023-12-05 17:14:33 <@nirik:matrix.scrye.com> So, first koji storage. As you may know we have been nearing full for a long while and it's been a struggle to do much about it. 2023-12-05 17:14:56 <@nirik:matrix.scrye.com> I have a email thread/some tickets with the koji folks to try and get the move to archive script actually working. 2023-12-05 17:15:23 <@nirik:matrix.scrye.com> we have worked through several issues, but it's failed again a new way and it's on their side to see if they can find a fix. 2023-12-05 17:15:42 <@nirik:matrix.scrye.com> In the mean time I cleaned up our compose/updates dir (nuking all the old eol updates repos) 2023-12-05 17:15:52 <@nirik:matrix.scrye.com> and asked iot folks to clean up their old composes. 2023-12-05 17:16:05 <@nirik:matrix.scrye.com> so, now we are in somewhat better shape... but still not ideal. 2023-12-05 17:16:19 <@nirik:matrix.scrye.com> ntap-iad2-c02-fedora01-nfs01a:/fedora_koji 92T 90T 2.5T 98% /mnt/fedora_koji 2023-12-05 17:16:37 <@nirik:matrix.scrye.com> so we have 10T before the limit (and actually more since there's a large weekly snapshot again) 2023-12-05 17:17:14 <@nirik:matrix.scrye.com> there's about 4TB or so in snapshots. 2023-12-05 17:17:22 <@nirik:matrix.scrye.com> anyhow, I will keep plugging away at it. 2023-12-05 17:17:47 <@nirik:matrix.scrye.com> I'd also like to get you access to it jednorozec and possibly Zlopez also, so I am not a single point of failure. 2023-12-05 17:17:49 <@humaton:fedora.im> Just out of curiosity how much space one release takes? 2023-12-05 17:18:30 <@nirik:matrix.scrye.com> impossible to say. The way koji stores things is packages//// 2023-12-05 17:18:44 <@nirik:matrix.scrye.com> so all the releases are mixed in the same area. 2023-12-05 17:18:44 <@humaton:fedora.im> Sure give me more access and I will find more ways how to break it! :) 2023-12-05 17:18:53 <@humaton:fedora.im> I see 2023-12-05 17:19:00 <@humaton:fedora.im> a LOT of space :) 2023-12-05 17:19:21 <@nirik:matrix.scrye.com> archiving just tells koji to copy a build to another volume and it puts a link back in the main place... 2023-12-05 17:19:53 <@nirik:matrix.scrye.com> yeah, it's basically everything we ever released... pretty crazy. Newer releases are of course much bigger. 2023-12-05 17:20:13 <@nirik:matrix.scrye.com> I've archived up to about f30. all newer is on the main volume 2023-12-05 17:20:23 <@nirik:matrix.scrye.com> anyhow, I will keep poking at it. 2023-12-05 17:20:35 <@humaton:fedora.im> Its interesting how the number of builds grows from release to release, I just noticed while making some sidetags 2023-12-05 17:20:50 <@humaton:fedora.im> comparing f38 to f39 to f40 2023-12-05 17:21:16 <@nirik:matrix.scrye.com> yep. we add more packages, more artifacts, more images... 2023-12-05 17:21:22 <@nirik:matrix.scrye.com> we seldom drop anything 2023-12-05 17:21:53 <@nirik:matrix.scrye.com> So, if nothing more on that, next was the updates-testing thing... 2023-12-05 17:21:55 <@nirik:matrix.scrye.com> https://pagure.io/releng/issue/11775 2023-12-05 17:22:26 <@humaton:fedora.im> lets pu it nicely in to the minutes 2023-12-05 17:22:33 <@nirik:matrix.scrye.com> The problem is that after we enable updates-testing in a branched release, we don't untag things from it... so after a while there's a bunch of things in the base tag and updates-testing. 2023-12-05 17:22:40 <@humaton:fedora.im> !releng 11775 2023-12-05 17:22:40 <@zodbot:fedora.im> releng #11775 (https://pagure.io/releng/issue/11775): Fix issue where updates-testing builds are never untagged pre GA 2023-12-05 17:23:16 <@humaton:fedora.im> right 2023-12-05 17:23:33 <@humaton:fedora.im> there was a script to run after GA in releng repo 2023-12-05 17:23:39 <@nirik:matrix.scrye.com> I guess the most correct solution would be to add a script/call on to the branched compose process to check and untag any updates-testing builds in base tag _after_ a compose finishes ok. 2023-12-05 17:24:02 <@nirik:matrix.scrye.com> yeah, that script... has some problems. I need to submit a PR on it. 2023-12-05 17:24:12 <@nirik:matrix.scrye.com> but also, it doesn't do anything, just gives you the list. :) 2023-12-05 17:24:35 <@nirik:matrix.scrye.com> so, we could clean it up at GA, but we need to add to docs to do it. 2023-12-05 17:25:17 <@humaton:fedora.im> right but you can do a loop with those and untag them 2023-12-05 17:25:34 <@humaton:fedora.im> anyhow making it part of the branched compose makes sense 2023-12-05 17:25:51 <@nirik:matrix.scrye.com> yep, I did that for f39/f38... but we had forgotten to do it for a while. 2023-12-05 17:26:27 <@nirik:matrix.scrye.com> yeah, if we can make such a script to reliably do it that would be great. If not, we can do it manually. 2023-12-05 17:26:43 <@nirik:matrix.scrye.com> just FYI: 2023-12-05 17:26:45 <@nirik:matrix.scrye.com> 14G Fedora-39-updates-testing-20231204.0 2.9G Fedora-39-updates-testing-20231205.0 2023-12-05 17:27:00 <@nirik:matrix.scrye.com> it makes a pretty good amount of difference on compose time/syncing/bits 2023-12-05 17:28:18 <@humaton:fedora.im> ok, so lets put it in the docs now so it does not slip next release and write/update script to automate it with branched composes 2023-12-05 17:28:40 <@nirik:matrix.scrye.com> sounds good! 2023-12-05 17:30:37 <@nirik:matrix.scrye.com> The final thing I thought of was meetings. I am out starting next week and back 2nd week of jan... ;) I don't know what your schedule is, but we probibly want to cancel some meetings over the holidays. 2023-12-05 17:32:17 <@humaton:fedora.im> So I am out from 23.12 probably till 23.1 not sure yet maybe just 2 weeks in 2024 2023-12-05 17:33:03 <@humaton:fedora.im> I think samyak is back from PTO next week so I would probably keep that one 2023-12-05 17:33:06 <@nirik:matrix.scrye.com> ok. 19th and 26th are probibly clear to cancel... up to you if others 2023-12-05 17:33:21 <@humaton:fedora.im> yup 19th and 26th for sure 2023-12-05 17:33:23 <@nirik:matrix.scrye.com> yeah, can also decide next week (although I won't be here, thats fine) 2023-12-05 17:34:57 <@humaton:fedora.im> So EOL announcement email https://paste.centos.org/view/a1aaa6f6 2023-12-05 17:35:27 <@humaton:fedora.im> and created this just not to forget and update the docs properly https://pagure.io/infra-docs-fpo/pull-request/263# 2023-12-05 17:35:43 <@nirik:matrix.scrye.com> s/will go/has gone/ ? 2023-12-05 17:35:59 <@nirik:matrix.scrye.com> and link to the releng repo for template improvement ideas? otherwise looks good. 2023-12-05 17:38:23 <@humaton:fedora.im> updated https://paste.centos.org/view/c44f2619 2023-12-05 17:38:51 <@nirik:matrix.scrye.com> sure. looks fine. 2023-12-05 17:39:29 <@humaton:fedora.im> the last updates compose for 37 is finished going to archive the bodhi release 2023-12-05 17:40:40 <@humaton:fedora.im> ok I dont really have much for today just finishing the EOL work 2023-12-05 17:40:48 <@humaton:fedora.im> will run playbooks in a few 2023-12-05 17:41:11 <@humaton:fedora.im> oh one thing 2023-12-05 17:42:05 <@humaton:fedora.im> nirik: I was looking at making the epel keys, playing with sigul to see what we used as name and other params when creating epel9 2023-12-05 17:42:18 <@humaton:fedora.im> but sigul seems segfaulting when I use --help-command 2023-12-05 17:42:53 <@nirik:matrix.scrye.com> yeah, thats a bug in the older version... ;( 2023-12-05 17:43:07 <@humaton:fedora.im> Also does one needs any special ACLS to setup sigul? I would like to grant perms on f40+ and epel9+ to samyak 2023-12-05 17:43:18 <@nirik:matrix.scrye.com> There's a newer one, but I was going to deploy it when we get the new vault hw... 2023-12-05 17:43:27 <@humaton:fedora.im> I see 2023-12-05 17:43:47 <@nirik:matrix.scrye.com> nothing special. There should be a doc on onboarding new folks to it. 2023-12-05 17:44:53 <@nirik:matrix.scrye.com> and +1 to adding samyak. ;) 2023-12-05 17:44:53 <@humaton:fedora.im> ahh I have it 2023-12-05 17:45:57 <@humaton:fedora.im> EOL mail is out 2023-12-05 17:47:07 <@humaton:fedora.im> Should we end sooner today? I will be running the playbooks for a while and need to eat something... 2023-12-05 17:48:06 <@nirik:matrix.scrye.com> sure, works for me. 2023-12-05 17:49:53 <@humaton:fedora.im> !endmeeting