<@jnsamyak:matrix.org>
17:03:06
!startmeeting RELENG (2024-01-09)
<@meetbot:fedora.im>
17:03:08
Meeting started at 2024-01-09 17:03:06 UTC
<@meetbot:fedora.im>
17:03:08
The Meeting name is 'RELENG (2024-01-09)'
<@jnsamyak:matrix.org>
17:03:21
!hello jnsamyak
<@zodbot:fedora.im>
17:03:25
Samyak Jain (jnsamyak) - he / him / his
<@jnsamyak:matrix.org>
17:03:39
Happy new year everyone, and welcome to first meeting of releng in 2024 :D
<@jnsamyak:matrix.org>
17:03:52
!meetingname releng
<@jnsamyak:matrix.org>
17:04:05
!chair nirik sharkcz pbrobinson phsmoura dustymabe jednorozec jnsamyak
<@jnsamyak:matrix.org>
17:04:18
!topic init process
<@nirik:matrix.scrye.com>
17:04:21
!hi
<@zodbot:fedora.im>
17:04:22
Kevin Fenzi (kevin) - he / him / his
<@jnsamyak:matrix.org>
17:05:41
nirik: you said you have some items on your list, do you wanna start? I have some change proposals to discuss!
<@patrikp:matrix.org>
17:06:23
!hi
<@zodbot:fedora.im>
17:06:25
None (patrikp)
<@nirik:matrix.scrye.com>
17:06:33
I have 3 items to bring up (but doesn't matter what order, whatever you like): ticket 11884 (epel8.6), discussion about builders before mass rebuild next week, and finally a meta discussion about releng and infrastructure and how we want them to be seperate or together.
<@jnsamyak:matrix.org>
17:07:35
interesting, I thought I might be needing coffee for this late meeting but I'm all awake now, go ahead any order okay to me nirik
<@nirik:matrix.scrye.com>
17:08:07
Don't mean to swap you or alarm. :)
<@nirik:matrix.scrye.com>
17:08:16
!releng 11884
<@zodbot:fedora.im>
17:08:17
**releng #11884** (https://pagure.io/releng/issue/11884):**epel/archive/8.6 branches missing**
● **Opened:** 23 hours ago by brianjmurrell
● **Last Updated:** 21 hours ago
● **Assignee:** Not Assigned
<@nirik:matrix.scrye.com>
17:08:48
This is a request to archive epel8.6 (epel as it was at the time rhel 8.6 came out), but... that was last year... and we don't have any composes from then.
<@nirik:matrix.scrye.com>
17:09:17
In theory we could try and figure out what builds were in epel8 at the time and do some kind of new compose with it... but some of those might even be gone now...
<@nirik:matrix.scrye.com>
17:09:32
so, I think we should just say... sorry, we can't do this, it was too long ago and we missed it.
<@nirik:matrix.scrye.com>
17:09:39
But open to other ideas.
<@jnsamyak:matrix.org>
17:10:32
makes sense, I also agree to smooge comment right there
<@jnsamyak:matrix.org>
17:11:05
this would be a lot of work, and the mass rebuild is also in a week a/c to the timeline
<@patrikp:matrix.org>
17:11:36
+1 on can't do this.
<@nirik:matrix.scrye.com>
17:12:01
ok, sounds good. If someone wants to close it, go ahead, or I can later.
<@patrikp:matrix.org>
17:12:10
Regarding what Smooge said, does that mean nobody is doing this anymore? Do we have an SOP somewhere?
<@nirik:matrix.scrye.com>
17:12:44
I did the last one and I thought we did have a sop. let me see...
<@nirik:matrix.scrye.com>
17:14:02
but I can't seem to find it off hand.
<@nirik:matrix.scrye.com>
17:14:48
I can look more later... it's pretty easy to do... just have to remember to do it.
<@nirik:matrix.scrye.com>
17:15:08
basically it's just rsyncing the epel/8 stuff to archive/epel/8.x/
<@jnsamyak:matrix.org>
17:15:40
archive stuff is mostly rsyncing from my experience just be careful from where to take and put things
<@nirik:matrix.scrye.com>
17:15:48
yep.
<@nirik:matrix.scrye.com>
17:15:55
ok, next thing... builders.
<@nirik:matrix.scrye.com>
17:15:59
!topic builders
<@nirik:matrix.scrye.com>
17:16:32
So, builders are still on f38 (normally we would have reinstalled or upgraded them to 39 by now... )
<@nirik:matrix.scrye.com>
17:16:51
we didn't because of createrepo_c changes. The one in f39 changes the default format
<@nirik:matrix.scrye.com>
17:17:17
So, most things would be fine, but... epel7/rhel7 cannot read the new format. So, all epel7 builds would break.
<@nirik:matrix.scrye.com>
17:17:53
Thinking about it over break though... I think perhaps we should just make a createrepo_c in infra tags thats the old version with a Epoch in it...
<@nirik:matrix.scrye.com>
17:18:16
but... now the mass rebuild starts next week.
<@nirik:matrix.scrye.com>
17:19:03
The buildvm-x86 builders probibly should be re-installed in any case...
<@nirik:matrix.scrye.com>
17:19:31
because there's a bug with 32bit builds (i686) where after a while the inodes on the builders are over 32bit and builds start to fail.
<@nirik:matrix.scrye.com>
17:20:01
So, we could: 1) just stick with f38 for now (at least until after mass rebuild), update them and reinstall x86 buildvms with f38
<@nirik:matrix.scrye.com>
17:20:24
2) do the createrepo_c local copy to downgrade it and reinstall/upgrade to f39
<@nirik:matrix.scrye.com>
17:20:37
that was supposed to be a 2... pesky matrix. ;)
<@nirik:matrix.scrye.com>
17:21:20
I guess those are the main options... time is running low tho before mass rebuild.
<@nirik:matrix.scrye.com>
17:21:35
and we want to have time before then to fix any issues...
<@nirik:matrix.scrye.com>
17:21:50
thoughts?
<@pcreech:matrix.org>
17:22:37
(sorry been lurking) Sounds like it's a tight window personally, so my brain thinks it would have to wait till after mass rebuild
<@pcreech:matrix.org>
17:23:00
is there any way to provide argumetns to createrepo through koji?
<@nirik:matrix.scrye.com>
17:23:05
just sticking with f38 is the more conservative option...
<@nirik:matrix.scrye.com>
17:23:20
There's a issue open to add that, and another for koji to use it...
<@jnsamyak:matrix.org>
17:23:25
I have a little idea about the koji builders so this might be silly question; from the 1 st option that is being suggested sure it will break lots of packages for epel7 but looking at the closer picture will the mass rebuild for f40 and 40 will be alright and we can do this fixing later on? OR did i got it all wrong^
<@nirik:matrix.scrye.com>
17:23:57
https://github.com/rpm-software-management/createrepo_c/issues/403
<@pcreech:matrix.org>
17:24:00
i'm assuming this is related to the new zstd or w/e it's called compression by default in the new createrepo_c?
<@nirik:matrix.scrye.com>
17:24:12
https://pagure.io/koji/issue/3945
<@nirik:matrix.scrye.com>
17:25:00
jnsamyak: well, both would keep epel7 fine...it's just a thing we need to consider/handle.
<@nirik:matrix.scrye.com>
17:25:11
yes, it's switching to zstd by default
<@nirik:matrix.scrye.com>
17:25:47
!ticket 11637
<@zodbot:fedora.im>
17:25:48
**fedora-infrastructure #11637** (https://pagure.io/fedora-infrastructure/issue/11637):**3rd part koji builders cannot parse fedora-rawhide repodata compression type**
● **Opened:** a month ago by kwizart
● **Last Updated:** 19 hours ago
● **Assignee:** Not Assigned
<@nirik:matrix.scrye.com>
17:26:58
so another datapoint, jednorozec did upgrade staging builders to f39 and they seemed ok there...
<@nirik:matrix.scrye.com>
17:27:04
but of course stg doesn't get as much use
<@pcreech:matrix.org>
17:28:34
would it be a silly thing to create a new channel with a builer or two in it just to regen repos for el7 builds?
<@nirik:matrix.scrye.com>
17:29:04
we could, but it adds complexity... means there's builders that are not the same, etc.
<@pcreech:matrix.org>
17:29:23
yup
<@jnsamyak:matrix.org>
17:31:02
from my little understanding, the first point that you suggested seems to be a good option; and once we are done with the mass rebuild and fixing it again; also since it has some releng impact should we create an issue on releng tracker as well so we can discuss it with releng pov when needed?
<@nirik:matrix.scrye.com>
17:31:37
sure, absolutely.
<@nirik:matrix.scrye.com>
17:32:02
I can pretty easily update them, reinstall just the x86 ones with f38 and then we can deal with f39 after mass rebuild...
<@nirik:matrix.scrye.com>
17:32:16
we do have a sop for the reinstalling.
<@jnsamyak:matrix.org>
17:32:29
!action - jnsamyak to put a summary and create an issue for koji builders on releng
<@nirik:matrix.scrye.com>
17:33:53
sounds good!
<@nirik:matrix.scrye.com>
17:34:21
Shall we go to the last one I had? or change proposals?
<@jnsamyak:matrix.org>
17:35:08
Let's hop into change proposals and in the rest of time we can come with your last topic in full beast mode? :P
<@jnsamyak:matrix.org>
17:35:36
!topic f40 change proposals
<@jnsamyak:matrix.org>
17:35:55
!link https://pagure.io/releng/issues?status=Open&tags=changes&tags=f40&close_status=
<@jnsamyak:matrix.org>
17:36:32
!releng 11865
<@zodbot:fedora.im>
17:36:33
**releng #11865** (https://pagure.io/releng/issue/11865):**F40 system-wide change: GNU Toolchain update for F40 https://fedoraproject.org/wiki/Changes/GNUToolchainF40**
● **Opened:** a week ago by codonell
● **Last Updated:** 6 days ago
● **Assignee:** Not Assigned
<@nirik:matrix.scrye.com>
17:37:03
we will need to ping them and confirm they will be ready by next week for mass rebuild.
<@jnsamyak:matrix.org>
17:37:40
This should be done on the tracker itself or we pinged them directly some where/channels?
<@nirik:matrix.scrye.com>
17:38:46
I would start on the ticket...
<@nirik:matrix.scrye.com>
17:39:12
There used to be a internal irc channel... but I'm sure thats no longer there... so we would have to go to email after I think?
<@jnsamyak:matrix.org>
17:39:30
yeah makes sense, I can follow up on the ticket if no one else wants to and will take it up from there
<@jnsamyak:matrix.org>
17:39:53
!action inform about mass rebuilt to GNU toolchain upstream
<@jnsamyak:matrix.org>
17:40:08
ok hoping on to next one
<@jnsamyak:matrix.org>
17:40:24
!releng 11864
<@zodbot:fedora.im>
17:40:26
**releng #11864** (https://pagure.io/releng/issue/11864):**F40 Change Optimized Binaries for the AMD64 Architecture**
● **Opened:** 2 weeks ago by zbyszek
● **Last Updated:** 6 days ago
● **Assignee:** Not Assigned
<@jnsamyak:matrix.org>
17:41:00
I think we can close this, I dont see any releng work here.
<@nirik:matrix.scrye.com>
17:41:07
I don't think this has any releng impact (and it's not approved yet anyhow)
<@jnsamyak:matrix.org>
17:41:35
same for this one, but wanted to recheck
<@jnsamyak:matrix.org>
17:41:40
!releng 11863
<@zodbot:fedora.im>
17:41:41
**releng #11863** (https://pagure.io/releng/issue/11863):**F40 Change Unify /usr/bin and /usr/sbin**
● **Opened:** 2 weeks ago by zbyszek
● **Last Updated:** a week ago
● **Assignee:** Not Assigned
<@nirik:matrix.scrye.com>
17:41:54
yep. No impact that I can think of off hand.
<@jnsamyak:matrix.org>
17:42:15
now to the interesting one
<@jnsamyak:matrix.org>
17:42:19
!releng 11859
<@zodbot:fedora.im>
17:42:20
**releng #11859** (https://pagure.io/releng/issue/11859):** F40 System-Wide Change: java-21-openjdk as system JDK in F40**
● **Opened:** 2 weeks ago by pmikova
● **Last Updated:** 2 weeks ago
● **Assignee:** Not Assigned
<@nirik:matrix.scrye.com>
17:43:03
oh this fun one
<@jnsamyak:matrix.org>
17:43:51
I see and have processed lots of side tags and other issues regarding this on the tracker for rebuilding openjdk, but nirik do you think from your god mind if from releng end we need to make any changes, i checked pungi configurations etc, couldn;t find anything in specific to ammend
<@nirik:matrix.scrye.com>
17:44:12
This says it needs mass rebuild... so we should check if they are ready.
<@nirik:matrix.scrye.com>
17:44:26
yeah, I am not sure it does need anything now that I think about it...
<@jnsamyak:matrix.org>
17:44:26
yep!
<@nirik:matrix.scrye.com>
17:44:37
openjdk is building... weird
<@nirik:matrix.scrye.com>
17:44:53
but I think that is all setup...
<@jnsamyak:matrix.org>
17:45:01
!action - inform and check with openjdk about mass rebuild releng#11859
<@jnsamyak:matrix.org>
17:45:29
makes sense, okay on to the next one
<@jnsamyak:matrix.org>
17:45:39
releng #11854
<@jnsamyak:matrix.org>
17:46:07
this is again interesting and complex at the same time i guess
<@nirik:matrix.scrye.com>
17:46:17
missing a ! there...
<@jnsamyak:matrix.org>
17:46:31
!releng 11854
<@zodbot:fedora.im>
17:46:32
**releng #11854** (https://pagure.io/releng/issue/11854):**F40 System-Wide Change: Build Fedora Cloud Edition Images Using Kiwi in Koji**
● **Opened:** 2 weeks ago by ngompa
● **Last Updated:** 8 hours ago
● **Assignee:** Not Assigned
<@nirik:matrix.scrye.com>
17:46:55
Yeah, so this one is also not yet approved...
<@jnsamyak:matrix.org>
17:47:04
I checked with arrfab today because it seems to be done at centos side of stuff
<@jnsamyak:matrix.org>
17:48:05
nirik: oh i missed thaat will check fesco ticket
<@nirik:matrix.scrye.com>
17:48:14
So, I am not thrilled about yet another stack and format for images... if we could get rid of ImageFactory I would like it more... but that would require moving container base image, base minimal and toolbox to kiwi....
<@nirik:matrix.scrye.com>
17:48:36
toolbox just moved last cycle, so I am not sure how excited they will be.
<@nirik:matrix.scrye.com>
17:49:20
The kiwi plugins are already upstream in koji, so it should not be too hard to enable... but then we have to deal with it's xml format and if it has any issues, etc.
<@jnsamyak:matrix.org>
17:49:21
+100 i have met with some toolbox follks in the asia summit conf and i can tell they will not be happy if this gets approved
<@nirik:matrix.scrye.com>
17:50:08
well, we will see what they say and how it goes. We could just end up with this net add and still have Imagefactory, which would be sad. ;(
<@jnsamyak:matrix.org>
17:50:25
yah! some more changes to empty the stack
<@jnsamyak:matrix.org>
17:50:42
!releng 11785
<@zodbot:fedora.im>
17:50:43
**releng #11785** (https://pagure.io/releng/issue/11785):**Impact check for SystemdSecurityHardening**
● **Opened:** a month ago by sundaram
● **Last Updated:** a month ago
● **Assignee:** Not Assigned
<@nirik:matrix.scrye.com>
17:50:57
no releng impact that I can think of...
<@jnsamyak:matrix.org>
17:51:31
!releng 11665
<@zodbot:fedora.im>
17:51:33
**releng #11665** (https://pagure.io/releng/issue/11665):**releng impact for Change: Drop DeltaRPMs**
● **Opened:** 4 months ago by decathorpe
● **Last Updated:** 3 weeks ago
● **Assignee:** Not Assigned
<@jnsamyak:matrix.org>
17:51:50
this is again a good one :)
<@nirik:matrix.scrye.com>
17:52:28
yeah, I think the work is all done now? we have it disabled... but we need to make sure we don't re-enable it at branching.
<@jnsamyak:matrix.org>
17:52:36
But there were relevent PRs that were open on pungi and ansible for this
<@jnsamyak:matrix.org>
17:52:59
wanted to make sure that what are the things we should look for at the time of branching
<@jnsamyak:matrix.org>
17:53:47
!info - at the time of mass rebuild make sure to not re-enable the deltarpms changes
<@nirik:matrix.scrye.com>
17:54:04
I think we are all set. We already disabled them in pungi-fedora long ago...
<@nirik:matrix.scrye.com>
17:54:22
and the ansible PR disabled them in bodhi for updates/updates-testing (on f40+)
<@jnsamyak:matrix.org>
17:54:45
yeah and i confirmed those changes are in action so we are good
<@jnsamyak:matrix.org>
17:54:50
ok last one
<@jnsamyak:matrix.org>
17:54:55
!releng 11845
<@zodbot:fedora.im>
17:54:57
**releng #11845** (https://pagure.io/releng/issue/11845):**F40 System-Wide Change: LLVM18**
● **Opened:** 3 weeks ago by tstellar
● **Last Updated:** 2 weeks ago
● **Assignee:** Not Assigned
<@jnsamyak:matrix.org>
17:55:45
i went through the proposal yesterday and there is lot of retirement request
<@jnsamyak:matrix.org>
17:55:47
llvm14
llvm15
llvm16
clang14
clang15
clang16
lld14
lld15
lld16
<@nirik:matrix.scrye.com>
17:55:48
I think there's no real impact here... all the retirements and rebuilds they can do themselves with the normal process.
<@nirik:matrix.scrye.com>
17:56:06
yeah... although some of those might end up staying...
<@nirik:matrix.scrye.com>
17:56:17
but they can 'fedpkg retire' them like any other package
<@jnsamyak:matrix.org>
17:57:10
should we ask them to open tickets for retirement or they are expecting us to move forward with these? I was confused 🫠 because they mentioned releng impact being involved now
<@nirik:matrix.scrye.com>
17:57:38
yeah, I don't think we need to do anything, they should be able to just use the normal process... unless I am missing something.
<@nirik:matrix.scrye.com>
17:57:41
we could ask in ticket
<@jnsamyak:matrix.org>
17:57:56
yeah will do
<@jnsamyak:matrix.org>
17:58:15
that's all hopefully thanks for discussing these with me nirik :D
<@jnsamyak:matrix.org>
17:58:19
nirik++
<@zodbot:fedora.im>
17:58:22
No Fedora Accounts users have the @jnsamyak:matrix.org Matrix Account defined
<@nirik:matrix.scrye.com>
17:58:36
no problem! Happy to discuss anything...
<@jnsamyak:matrix.org>
17:58:49
if you wanna bring up the last topic of the day go ahead! 🎉
<@nirik:matrix.scrye.com>
17:58:58
sure, just real quick...
<@nirik:matrix.scrye.com>
17:59:08
So, over the break I was working on docs some.
<@nirik:matrix.scrye.com>
17:59:46
Infrastructure and releng have always been very closely related teams... the line is blurry sometimes (is a builder down a infra problem or a releng one? etc).
<@nirik:matrix.scrye.com>
18:00:13
But they have been seperate teams (their own rooms, meetings, people (althought there is overlap))
<@nirik:matrix.scrye.com>
18:00:39
But... when I was working on docs, I found it a bit confusing, because the releng docs are mixed in the infra area...
<@nirik:matrix.scrye.com>
18:00:58
so I tried to say "Infra and Releng" but then it didn't make sense for some things and seemed confusing
<@nirik:matrix.scrye.com>
18:01:08
So, my question/thing to ponder:
<@nirik:matrix.scrye.com>
18:01:32
Should we move the releng docs to it's own repo/pages? Or should we try and make infra and releng more one team?
<@nirik:matrix.scrye.com>
18:01:58
(and we don't need to decide anything, I just wanted to bring it up to ponder on)
<@jnsamyak:matrix.org>
18:02:50
all valid points here and yes that's why we introduced the releng miscellaneous guide so we can move over non release task that were in infra section over to here;
My suggestion is instead of moving that on two different places have it at one place but with tags this is #infra, #releng or both
<@nirik:matrix.scrye.com>
18:03:44
well, the structure is odd then... if I have a joining area, do I mention releng and infra both seperately?
<@nirik:matrix.scrye.com>
18:04:10
ie, https://docs.fedoraproject.org/en-US/infra/
<@pcreech:matrix.org>
18:04:14
Don't have much xp in fedora on the duality of it, but in the teams i've been on, the same overlap does exist.
<@pcreech:matrix.org>
18:04:32
most infra people do releng as well and vice versa
<@nirik:matrix.scrye.com>
18:04:51
I mean if I say meetings, do I mention both and try and explain the difference? or goals/philosophy?
<@jnsamyak:matrix.org>
18:05:06
this is open for discussion this should start with Infra and Releng on the introductory page I guess that makes more sense because we see almost not everytime same set of folks dedicated to both work
<@nirik:matrix.scrye.com>
18:05:09
(yeah, it doesn't help that I do both and probibly confuse people) :)
<@nirik:matrix.scrye.com>
18:05:48
It's also unweildy to say "Infrastructure and Release Enginnering" everywhere.
<@nirik:matrix.scrye.com>
18:06:14
but I can start a discussion thread on it... and we can figure out what the path forward is. ;)
<@jnsamyak:matrix.org>
18:06:20
but again if we want a little structuring we can have under the same umbrella both the infra and releng and describe the goals of each area under the same hood if that makes sense?
<@jnsamyak:matrix.org>
18:06:42
that will be awesome action item
<@jnsamyak:matrix.org>
18:07:11
can i assign this action item to you nirik ? :P
<@nirik:matrix.scrye.com>
18:07:19
yep!
<@jnsamyak:matrix.org>
18:07:24
time check folks! :D
<@jnsamyak:matrix.org>
18:08:06
!action nirik to open an email discussing about docs future for Infra and Releng
<@jnsamyak:matrix.org>
18:08:45
That's all for today and we are over the time, thanks folks for participating! See ya all in the next one with more sturucted format :D
<@jnsamyak:matrix.org>
18:09:16
!endmeeting