#meeting-3:fedoraproject.org log

<@humaton:fedora.im>

17:00:54

!startmeeting RELENG (2023-12-05)

<@meetbot:fedora.im>

17:00:55

Meeting started at 2023-12-05 17:00:54 UTC

<@meetbot:fedora.im>

17:00:55

The Meeting name is 'RELENG (2023-12-05)'

<@humaton:fedora.im>

17:01:02

!meetingname releng

<@humaton:fedora.im>

17:01:07

!chair nirik sharkcz pbrobinson phsmoura dustymabe jednorozec

<@humaton:fedora.im>

17:01:14

!topic init process

<@humaton:fedora.im>

17:02:51

Hello everybody

<@patrikp:matrix.org>

17:03:02

Good evening.

<@nirik:matrix.scrye.com>

17:03:06

!hi

<@zodbot:fedora.im>

17:03:08

Kevin Fenzi (kevin) - he / him / his

<@humaton:fedora.im>

17:03:54

!topic Fedora 37 is going EOL today

<@humaton:fedora.im>

17:04:05

so the ansible changes are in PR

<@humaton:fedora.im>

17:04:11

koji targets are removed

<@humaton:fedora.im>

17:05:19

so there is a bunch of updates in testing for the release https://bodhi.fedoraproject.org/releases/F37

<@humaton:fedora.im>

17:05:45

should we "force them" to stable or keep them there?

<@nirik:matrix.scrye.com>

17:06:25

nope, they didn't make it, too bad.

<@humaton:fedora.im>

17:06:57

ok one sec update for firefox made it after the stable push

<@nirik:matrix.scrye.com>

17:06:58

well, we should do one last stable push...

<@nirik:matrix.scrye.com>

17:07:07

ah, thats the one I saw.

<@humaton:fedora.im>

17:07:21

I am running one more for stable for the ff update and will move the bodhi release to "archived"

<@humaton:fedora.im>

17:07:48

Well I did the push about hour ago and the ff update made it to stable emanwhile...

<@nirik:matrix.scrye.com>

17:07:52

sounds good. We shouldn't mass push from testing, there may be good reasons none of those were requested stable... and maintainers had a while to make it happen.

<@humaton:fedora.im>

17:08:05

noted

<@humaton:fedora.im>

17:08:50

also the release is set to active:false in PDC

<@humaton:fedora.im>

17:09:02

all packages should have correct EOL for f37 branch there

<@humaton:fedora.im>

17:09:25

so what is missing is https://docs.fedoraproject.org/en-US/infra/release_guide/release_eol/#_bodhi_tasks and the following move to archive steps

<@nirik:matrix.scrye.com>

17:10:24

nd the announcement

<@humaton:fedora.im>

17:10:42

oh right the final one

<@nirik:matrix.scrye.com>

17:10:45

Oh, later today I will also take care of /mnt/koji/compose/f37/ since we don't need it anymore.

<@humaton:fedora.im>

17:11:07

cool

<@nirik:matrix.scrye.com>

17:13:00

oh, and updates-testing thing we should consider a solution to.

<@humaton:fedora.im>

17:13:06

go for it I will do the EOL announcement

<@nirik:matrix.scrye.com>

17:14:33

So, first koji storage. As you may know we have been nearing full for a long while and it's been a struggle to do much about it.

<@nirik:matrix.scrye.com>

17:14:56

I have a email thread/some tickets with the koji folks to try and get the move to archive script actually working.

<@nirik:matrix.scrye.com>

17:15:23

we have worked through several issues, but it's failed again a new way and it's on their side to see if they can find a fix.

<@nirik:matrix.scrye.com>

17:15:42

In the mean time I cleaned up our compose/updates dir (nuking all the old eol updates repos)

<@nirik:matrix.scrye.com>

17:15:52

and asked iot folks to clean up their old composes.

<@nirik:matrix.scrye.com>

17:16:05

so, now we are in somewhat better shape... but still not ideal.

<@nirik:matrix.scrye.com>

17:16:19

ntap-iad2-c02-fedora01-nfs01a:/fedora_koji 92T 90T 2.5T 98% /mnt/fedora_koji

<@nirik:matrix.scrye.com>

17:16:37

so we have 10T before the limit (and actually more since there's a large weekly snapshot again)

<@nirik:matrix.scrye.com>

17:17:14

there's about 4TB or so in snapshots.

<@nirik:matrix.scrye.com>

17:17:22

anyhow, I will keep plugging away at it.

<@nirik:matrix.scrye.com>

17:17:47

I'd also like to get you access to it jednorozec and possibly Zlopez also, so I am not a single point of failure.

<@humaton:fedora.im>

17:17:49

Just out of curiosity how much space one release takes?

<@nirik:matrix.scrye.com>

17:18:30

impossible to say. The way koji stores things is packages/<packagename>/<packageversion>/<packagerelease>/

<@nirik:matrix.scrye.com>

17:18:44

so all the releases are mixed in the same area.

<@humaton:fedora.im>

17:18:44

Sure give me more access and I will find more ways how to break it! :)

<@humaton:fedora.im>

17:18:53

I see

<@humaton:fedora.im>

17:19:00

a LOT of space :)

<@nirik:matrix.scrye.com>

17:19:21

archiving just tells koji to copy a build to another volume and it puts a link back in the main place...

<@nirik:matrix.scrye.com>

17:19:53

yeah, it's basically everything we ever released... pretty crazy. Newer releases are of course much bigger.

<@nirik:matrix.scrye.com>

17:20:13

I've archived up to about f30. all newer is on the main volume

<@nirik:matrix.scrye.com>

17:20:23

anyhow, I will keep poking at it.

<@humaton:fedora.im>

17:20:35

Its interesting how the number of builds grows from release to release, I just noticed while making some sidetags

<@humaton:fedora.im>

17:20:50

comparing f38 to f39 to f40

<@nirik:matrix.scrye.com>

17:21:16

yep. we add more packages, more artifacts, more images...

<@nirik:matrix.scrye.com>

17:21:22

we seldom drop anything

<@nirik:matrix.scrye.com>

17:21:53

So, if nothing more on that, next was the updates-testing thing...

<@nirik:matrix.scrye.com>

17:21:55

https://pagure.io/releng/issue/11775

<@humaton:fedora.im>

17:22:26

lets pu it nicely in to the minutes

<@nirik:matrix.scrye.com>

17:22:33

The problem is that after we enable updates-testing in a branched release, we don't untag things from it... so after a while there's a bunch of things in the base tag and updates-testing.

<@humaton:fedora.im>

17:22:40

!releng 11775

<@zodbot:fedora.im>

17:22:40

releng #11775 (https://pagure.io/releng/issue/11775): Fix issue where updates-testing builds are never untagged pre GA

<@humaton:fedora.im>

17:23:16

right

<@humaton:fedora.im>

17:23:33

there was a script to run after GA in releng repo

<@nirik:matrix.scrye.com>

17:23:39

I guess the most correct solution would be to add a script/call on to the branched compose process to check and untag any updates-testing builds in base tag _after_ a compose finishes ok.

<@nirik:matrix.scrye.com>

17:24:02

yeah, that script... has some problems. I need to submit a PR on it.

<@nirik:matrix.scrye.com>

17:24:12

but also, it doesn't do anything, just gives you the list. :)

<@nirik:matrix.scrye.com>

17:24:35

so, we could clean it up at GA, but we need to add to docs to do it.

<@humaton:fedora.im>

17:25:17

right but you can do a loop with those and untag them

<@humaton:fedora.im>

17:25:34

anyhow making it part of the branched compose makes sense

<@nirik:matrix.scrye.com>

17:25:51

yep, I did that for f39/f38... but we had forgotten to do it for a while.

<@nirik:matrix.scrye.com>

17:26:27

yeah, if we can make such a script to reliably do it that would be great. If not, we can do it manually.

<@nirik:matrix.scrye.com>

17:26:43

just FYI:

<@nirik:matrix.scrye.com>

17:26:45

14G Fedora-39-updates-testing-20231204.0 2.9G Fedora-39-updates-testing-20231205.0

<@nirik:matrix.scrye.com>

17:27:00

it makes a pretty good amount of difference on compose time/syncing/bits

<@humaton:fedora.im>

17:28:18

ok, so lets put it in the docs now so it does not slip next release and write/update script to automate it with branched composes

<@nirik:matrix.scrye.com>

17:28:40

sounds good!

<@nirik:matrix.scrye.com>

17:30:37

The final thing I thought of was meetings. I am out starting next week and back 2nd week of jan... ;) I don't know what your schedule is, but we probibly want to cancel some meetings over the holidays.

<@humaton:fedora.im>

17:32:17

So I am out from 23.12 probably till 23.1 not sure yet maybe just 2 weeks in 2024

<@humaton:fedora.im>

17:33:03

I think samyak is back from PTO next week so I would probably keep that one

<@nirik:matrix.scrye.com>

17:33:06

ok. 19th and 26th are probibly clear to cancel... up to you if others

<@humaton:fedora.im>

17:33:21

yup 19th and 26th for sure

<@nirik:matrix.scrye.com>

17:33:23

yeah, can also decide next week (although I won't be here, thats fine)

<@humaton:fedora.im>

17:34:57

So EOL announcement email https://paste.centos.org/view/a1aaa6f6

<@humaton:fedora.im>

17:35:27

and created this just not to forget and update the docs properly https://pagure.io/infra-docs-fpo/pull-request/263#

<@nirik:matrix.scrye.com>

17:35:43

s/will go/has gone/ ?

<@nirik:matrix.scrye.com>

17:35:59

and link to the releng repo for template improvement ideas? otherwise looks good.

<@humaton:fedora.im>

17:38:23

updated https://paste.centos.org/view/c44f2619

<@nirik:matrix.scrye.com>

17:38:51

sure. looks fine.

<@humaton:fedora.im>

17:39:29

the last updates compose for 37 is finished going to archive the bodhi release

<@humaton:fedora.im>

17:40:40

ok I dont really have much for today just finishing the EOL work

<@humaton:fedora.im>

17:40:48

will run playbooks in a few

<@humaton:fedora.im>

17:41:11

oh one thing

<@humaton:fedora.im>

17:42:05

nirik: I was looking at making the epel keys, playing with sigul to see what we used as name and other params when creating epel9

<@humaton:fedora.im>

17:42:18

but sigul seems segfaulting when I use --help-command

<@nirik:matrix.scrye.com>

17:42:53

yeah, thats a bug in the older version... ;(

<@humaton:fedora.im>

17:43:07

Also does one needs any special ACLS to setup sigul? I would like to grant perms on f40+ and epel9+ to samyak

<@nirik:matrix.scrye.com>

17:43:18

There's a newer one, but I was going to deploy it when we get the new vault hw...

<@humaton:fedora.im>

17:43:27

I see

<@nirik:matrix.scrye.com>

17:43:47

nothing special. There should be a doc on onboarding new folks to it.

<@nirik:matrix.scrye.com>

17:44:53

and +1 to adding samyak. ;)

<@humaton:fedora.im>

17:44:53

ahh I have it

<@humaton:fedora.im>

17:45:57

EOL mail is out

<@humaton:fedora.im>

17:47:07

Should we end sooner today? I will be running the playbooks for a while and need to eat something...

<@nirik:matrix.scrye.com>

17:48:06

sure, works for me.

<@humaton:fedora.im>

17:49:53

!endmeeting