<@sgallagh:fedora.im>
17:00:44
!startmeeting ELN (2024-01-12)
<@meetbot:fedora.im>
17:00:44
Meeting started at 2024-01-12 17:00:44 UTC
<@meetbot:fedora.im>
17:00:45
The Meeting name is 'ELN (2024-01-12)'
<@sgallagh:fedora.im>
17:00:50
!meetingname eln
<@michel:one.ems.host>
17:00:50
!hi
<@zodbot:fedora.im>
17:00:53
Michel Lind (salimma) - he / him / his
<@sgallagh:fedora.im>
17:01:15
@hellomynameis sgallagh
<@sgallagh:fedora.im>
17:01:27
!hellomynameis sgallagh
<@zodbot:fedora.im>
17:01:28
Stephen Gallagher (sgallagh) - he / him / his
<@michel:one.ems.host>
17:02:01
!hellomynameis michel-slm
<@zodbot:fedora.im>
17:02:02
Sorry, but Fedora Accounts user 'michel-slm' does not exist
<@michel:one.ems.host>
17:02:09
neat
<@davide:cavalca.name>
17:02:17
!hi
<@zodbot:fedora.im>
17:02:20
Davide Cavalca (dcavalca) - he / him / his
<@yselkowitz:fedora.im>
17:02:37
!hi
<@zodbot:fedora.im>
17:02:38
Yaakov Selkowitz (yselkowitz)
<@nhanlon:beeper.com>
17:03:17
!hi
<@zodbot:fedora.im>
17:03:30
Neil Hanlon (neil) - he / him / his
<@sgallagh:fedora.im>
17:04:15
Alright, I think we have enough folks here to get started.
<@sgallagh:fedora.im>
17:05:02
!topic Breaking inheritance for CentOS Stream 10 (Or: ELN Goes to Eleven)
<@nhanlon:beeper.com>
17:05:15
weeee
<@nhanlon:beeper.com>
17:05:36
Spinal^WSystemTap
<@salimma:fedora.im>
17:05:57
my thought exactly
<@sgallagh:fedora.im>
17:06:33
As previously discussed many times, CentOS Stream 10 will stop inheriting automatically from ELN as of the Fedora 40 Branch event
<@sgallagh:fedora.im>
17:07:32
I'm currently working on putting together a document identifying all of the pieces we need to have in place to make this happen.
<@conan_kudo:matrix.org>
17:08:37
!hi
<@zodbot:fedora.im>
17:08:40
Neal Gompa (ngompa) - he / him / his
<@sgallagh:fedora.im>
17:09:01
It's not ready for sharing yet, but it will be replacing https://docs.fedoraproject.org/en-US/eln/branching/ in the very near future. (Also, that document is flat-out incorrect in several places)
<@sgallagh:fedora.im>
17:09:39
So I'll start by listing the things I already have (at a high level) and then I'd like to ask you all to remind me of the things I forgot.
<@conan_kudo:matrix.org>
17:09:51
sounds good
<@nhanlon:beeper.com>
17:10:15
> the only problem is, I can't remember what it is I've forgotten
<@conan_kudo:matrix.org>
17:10:35
that's too bad Neville
<@sgallagh:fedora.im>
17:10:36
Thanks, Neville
<@nhanlon:beeper.com>
17:11:07
you know, if you remove some letters...
<@sgallagh:fedora.im>
17:12:31
* Pause ELNBuildSync as Fedora RelEng starts the branching process. This will finish out any ongoing rebuild batch, but not start any new ones
* Disable the DistroBuildSync (ELN->CentOS Stream 10) entirely
* Update rpm.macro.rhel (ELN Koji Tag) and rhel_dist_version (fedora-release)
<@sgallagh:fedora.im>
17:13:26
* Rebuild fedora-release
* Update the tag trigger for the ELNBuildSync configuration to f41
* Resume ELNBuildSync
<@sgallagh:fedora.im>
17:14:07
That's most of what I have now. As I was typing this out, yselkowitz also added "bump rpm.macro.eln up to 135", which I'd like to discuss.
<@yselkowitz:fedora.im>
17:14:43
left a comment in the doc
<@sgallagh:fedora.im>
17:15:16
I'm sure there are other pieces I'm forgetting at the moment, so I'd like to give you all a few minutes to shout them out.
<@yselkowitz:fedora.im>
17:15:48
based on some issues we had post f39-branching, whenever fedora bumps dist-git, eln also needs to, otherwise we can miss rebuilds.
<@conan_kudo:matrix.org>
17:15:54
What about the forking side on CentOS Stream?
<@sgallagh:fedora.im>
17:16:31
Conan Kudo: I'm not sure what you're asking. Could you rephrase?
<@yselkowitz:fedora.im>
17:16:51
iiuc this doc is just about actions on the fedora side
<@conan_kudo:matrix.org>
17:16:55
Yes
<@conan_kudo:matrix.org>
17:17:20
I would like both sides of it to be covered in the doc because it ensures we understand the relationship of tasks
<@sgallagh:fedora.im>
17:18:18
I'm not sure there's actually anything specific we need to do on the Stream side.
<@sgallagh:fedora.im>
17:18:45
Maybe bump the dist tag "+" value?
<@yselkowitz:fedora.im>
17:18:49
well, we'll retire those packages that have since been dropped from eln, and do a mass-rebuild iirc?
<@sgallagh:fedora.im>
17:19:01
yselkowitz: Ah, okay. That's fair.
<@yselkowitz:fedora.im>
17:19:21
not sure that *this* doc is the place for that though
<@yselkowitz:fedora.im>
17:19:31
wasn't this going into fedora docs?
<@conan_kudo:matrix.org>
17:19:32
I can think of at least three: redhat-rpm-config's version needs to be frozen, synced but dropped packages need to be retired, and the disttag needs to be locked
<@sgallagh:fedora.im>
17:20:03
Can you explain what you mean by the first and last of those?
<@sgallagh:fedora.im>
17:20:49
yselkowitz: From a pure perspective, you're probably right. From a practical one, I think I'd rather keep related topics in the same doc, even if it might not STRICTLY belong in Fedora Docs
<@conan_kudo:matrix.org>
17:21:01
redhat-rpm-config currently uses %baserelease on the Version field, making every rpm bump a new version, which is chaotic and makes it difficult to identify where things came from, the version at fork point needs to be frozen and %baserelease needs to move to the Release field
<@conan_kudo:matrix.org>
17:21:47
the last one is about getting rid of the `+N` suffix that's floating in the disttag on the CentOS Stream Koji setup
<@sgallagh:fedora.im>
17:22:30
The +N thing will probably persist for a bit longer, as it's a useful tool for us to get things lined up until the full toolchain we expect to ship with lands.
<@conan_kudo:matrix.org>
17:22:45
sure, but it's a step that should be documented regardless :)
<@sgallagh:fedora.im>
17:22:47
It's analagous to the .elnNNN field
<@conan_kudo:matrix.org>
17:23:14
yes, I understand it to be similar to the buildcounter field I wanted to have added to Fedora's disttag
<@sgallagh:fedora.im>
17:23:20
Agreed, but as it's not going to happen as part of the inheritance-break, this is probably the wrong place to do so.
<@sgallagh:fedora.im>
17:23:27
But I appreciate you calling it out
<@conan_kudo:matrix.org>
17:23:42
the first one needs to happen at inheritance break though :)
<@conan_kudo:matrix.org>
17:23:54
that said, I just wanted to make sure it wasn't forgotten this time like it was last time
<@sgallagh:fedora.im>
17:24:14
Which "it" do you mean in that sentence?
<@conan_kudo:matrix.org>
17:24:26
redhat-rpm-config version freeze
<@sgallagh:fedora.im>
17:24:26
(Sorry, my fault for carrying two conversations on at once)
<@sgallagh:fedora.im>
17:25:47
While having `%baserelease` in the Version field definitely feels weird, is there an actual problem caused by it?
<@yselkowitz:fedora.im>
17:25:58
I understand the request
<@yselkowitz:fedora.im>
17:26:34
he's saying it makes it harder to see where rhel redhat-rpm-config branched from fedora
<@conan_kudo:matrix.org>
17:26:38
yes
<@conan_kudo:matrix.org>
17:26:46
and as redhat-rpm-config upstream, this is my personal request
<@yselkowitz:fedora.im>
17:26:59
so if we were to branch today at 276-1
<@yselkowitz:fedora.im>
17:27:21
and changes were applied to both fedora and c10s
<@yselkowitz:fedora.im>
17:27:47
then (w/o this) both would become 277-1 but not necessarily identical (if the changes aren't the same)
<@conan_kudo:matrix.org>
17:27:55
exactly
<@conan_kudo:matrix.org>
17:27:59
I asked last time too, but it got forgotten in the craziness of doing it for the first time :P
<@sgallagh:fedora.im>
17:28:20
OK, that makes good sense. I'll add that to the SOP
<@conan_kudo:matrix.org>
17:28:30
thank you
<@conan_kudo:matrix.org>
17:28:55
I really appreciate it
<@sgallagh:fedora.im>
17:31:15
Any other thoughts on branch-time events?
<@conan_kudo:matrix.org>
17:31:36
I can't think of any others offhand?
<@conan_kudo:matrix.org>
17:31:51
I assume a mass build happens right after on both sides
<@conan_kudo:matrix.org>
17:31:58
so that cleans up things after that
<@sgallagh:fedora.im>
17:32:11
Not exactly
<@yselkowitz:fedora.im>
17:32:17
c10s yes, not eln
<@yselkowitz:fedora.im>
17:32:19
?
<@sgallagh:fedora.im>
17:32:38
For Fedora ELN, we won't do a mass-rebuild because we'll have just completed the F40 one
<@sgallagh:fedora.im>
17:32:52
I'm not sure what value there would be to doing another so soon
<@conan_kudo:matrix.org>
17:33:01
ticking over the RHEL 11 conditional
<@yselkowitz:fedora.im>
17:33:03
none, thanks for asking :-)
<@yselkowitz:fedora.im>
17:33:27
in rawhide mass rebuilds are before branching, not after, how is this different?
<@sgallagh:fedora.im>
17:33:29
Hmm, that's an interesting point.
<@conan_kudo:matrix.org>
17:33:40
we did that last time too, IIRC
<@conan_kudo:matrix.org>
17:33:56
because ELN needs to tick over to the next RHEL after branching
<@sgallagh:fedora.im>
17:33:57
yselkowitz: Basically, if we're bumping the `%rhel` value, then there are probably some conditionals out there that may function differently.
<@yselkowitz:fedora.im>
17:34:09
dnf5 yes
<@conan_kudo:matrix.org>
17:34:18
pipewire will too
<@conan_kudo:matrix.org>
17:34:22
as will a few others
<@sgallagh:fedora.im>
17:34:26
But I'm not sure we necessarily need to do the mass-build right away; we'll be plenty busy on the CS side for those next few weeks
<@yselkowitz:fedora.im>
17:34:43
but wouldn't there already have to be rebuilds in rawhide to trigger those anyway?
<@conan_kudo:matrix.org>
17:34:48
it needs to be scheduled, but not immediately
<@sgallagh:fedora.im>
17:35:11
yselkowitz: Sorry, I think I missed a JMP in there somewhere.
<@conan_kudo:matrix.org>
17:35:27
for example, the frame pointers flag will take effect for RHEL-ELN 11
<@conan_kudo:matrix.org>
17:35:37
it currently does not
<@sgallagh:fedora.im>
17:35:47
Ooh, that's a really good point.
<@yselkowitz:fedora.im>
17:35:51
for individual packages that have pre-set macros to change in f41
<@yselkowitz:fedora.im>
17:36:00
e.g. dnf5 (to obsolete dnf)
<@conan_kudo:matrix.org>
17:36:16
pipewire will re-enable libcamera support in RHEL-ELN 11
<@yselkowitz:fedora.im>
17:36:25
then there would have to anyways be a rebuild in rawhide to actually trigger that change, which will trigger an eln build
<@conan_kudo:matrix.org>
17:36:27
there is a list I keep track of these things for :P
<@sgallagh:fedora.im>
17:37:12
yselkowitz: I think he's listing packages/changes that specifically have `%if 0%{?rhel} > 10` or equivalent
<@yselkowitz:fedora.im>
17:37:35
how many of those don't also have `0%{?fedora} > 40` etc?
<@sgallagh:fedora.im>
17:37:54
The frame pointer one absolutely does need us to do a mass-rebuild at some point. While we *can* wait for a Fedora mass-rebuild to trigger it, I suspect the folks that want it around will want it faster.
<@yselkowitz:fedora.im>
17:38:14
it can wait
<@sgallagh:fedora.im>
17:38:30
yselkowitz: At least the FP one and it sounds like the pipewire one as well, though that's very targeted and wouldn't need a mass-rebuild
<@conan_kudo:matrix.org>
17:38:40
the point isn't whether or not to do it, the point is that it needs to scheduled
<@conan_kudo:matrix.org>
17:38:46
and part of the SOP
<@conan_kudo:matrix.org>
17:39:35
we already know that Fedora and RHEL differ at the low levels enough that we cannot trust Fedora rebuilds to imply ELN ones when ELN ticks over
<@davide:cavalca.name>
17:39:35
agreed, I think we definitely want a mass rebuild sooner than later for consistency
<@sgallagh:fedora.im>
17:39:49
There's also something to be said for doing that mass-rebuild fairly quickly, before the dependency chain spikes again :)
<@yselkowitz:fedora.im>
17:40:24
it's going to need to wait a bit
<@sgallagh:fedora.im>
17:40:27
I'm inclined to make it part of the branching process, but with the understanding that we may not get around to FIXING any issues revealed by it for some time
<@yselkowitz:fedora.im>
17:41:07
it's going to need to wait
<@davide:cavalca.name>
17:42:55
I think that's an acceptable tradeoff, it's understood folks will be busy, but I'd rather the state of the world reflect reality than not -- a pre-mass-rebuild state would be pretty misleading to use/test against
<@nhanlon:beeper.com>
17:43:51
I'm in agreement with Davide and Neal here. If it cannot be done, then I think we need to discuss _why_ it can't be done (or has to wait).
<@yselkowitz:fedora.im>
17:44:22
because we're not going to have the resources to deal with another mass rebuild immediately after branching
<@conan_kudo:matrix.org>
17:44:33
I'm fine with this. It also allows the community to assist in fixing issues.
<@conan_kudo:matrix.org>
17:44:59
what kind of resources? because compute resources we have
<@nhanlon:beeper.com>
17:45:14
thank you for explaining
<@conan_kudo:matrix.org>
17:45:33
and if we don't, then that's a problem that needs to be bubbled up
<@yselkowitz:fedora.im>
17:45:38
people resources, specifically from our team
<@sgallagh:fedora.im>
17:45:49
yselkowitz: I think what you're saying is that you expect the fallout to require a lot of attention and that you think it has to be you and I that provide it. I'm suggesting (and it seems that Conan Kudo is offering) that others can step up
<@yselkowitz:fedora.im>
17:46:35
any mass-rebuild requires a lot of attention
<@conan_kudo:matrix.org>
17:46:44
Yes.
<@davide:cavalca.name>
17:47:07
yeah, if you/your team are currently a SPOF for this that's something we should strive to improve
<@sgallagh:fedora.im>
17:47:08
Was that in reply to me, yselkowitz or both?
<@conan_kudo:matrix.org>
17:47:16
Yes.
<@davide:cavalca.name>
17:47:43
e.g. if there's specific things that aren't well documented yet we should focus on fixing that so others can pitch in
<@yselkowitz:fedora.im>
17:48:03
the way things are now, packages will start to be built with frame pointers post branching, and that will be a start.
<@conan_kudo:matrix.org>
17:48:03
Exactly. We fell into this trap last time, and we should endeavor to resolve it this time.
<@sgallagh:fedora.im>
17:48:05
There's one other hiccough here that may throw a spanner into the works.
<@sgallagh:fedora.im>
17:48:26
Fedora Release Engineering wants to decommission ODCS, which ELN is using for its composes.
<@yselkowitz:fedora.im>
17:48:34
e.g. most of the desktop will see updates post branching in prep for gnome 46 final and f40 ga
<@conan_kudo:matrix.org>
17:48:35
Oh.
<@sgallagh:fedora.im>
17:48:49
The effect of which is that we're going to end up with a changed location for pulling down composes as well
<@conan_kudo:matrix.org>
17:49:00
Wait, is ODCS not maintained?
<@sgallagh:fedora.im>
17:49:27
Apparently, Fedora ELN is the last user of it in Fedora-land
<@conan_kudo:matrix.org>
17:49:33
That's the only reason I've seen for decommissioning anything.
<@sgallagh:fedora.im>
17:49:45
And yeah, it's effectively unmaintained.
<@davide:cavalca.name>
17:49:48
what's the alternative to ODCS?
<@yselkowitz:fedora.im>
17:49:50
well it will be once osbs is decommissioned
<@sgallagh:fedora.im>
17:50:02
Davide Cavalca: We'll use the same compose infrastructure as Fedora
<@conan_kudo:matrix.org>
17:50:14
Where is that managed/configured?
<@conan_kudo:matrix.org>
17:50:55
Is it just pungi and comps now?
<@sgallagh:fedora.im>
17:50:55
Conan Kudo: Which part?
<@conan_kudo:matrix.org>
17:51:15
the distrobuildsync -> build artifacts -> publish pipeline
<@conan_kudo:matrix.org>
17:52:02
if ODCS goes away, I'm not sure what handles that stuff
<@sgallagh:fedora.im>
17:52:03
Right now it's basically a cron-job that asks ODCS to perform a compose based on the pungi-fedora `eln` branch
<@conan_kudo:matrix.org>
17:52:18
right
<@sgallagh:fedora.im>
17:52:42
I don't have all the details yet on what it's going to look like post-ODCS. Something about `nightly.sh` in pungi-fedora
<@conan_kudo:matrix.org>
17:53:02
well then keep us posted and try to document it somewhere :)
<@sgallagh:fedora.im>
17:53:08
Will do
<@sgallagh:fedora.im>
17:53:35
But mostly I bring it up because if composes are going to be in flux in this same timeframe, it may not be super-beneficial to rush to do the mass-rebuild
<@conan_kudo:matrix.org>
17:54:01
on the flip side, we may not get a compose for a while post-shutdown
<@conan_kudo:matrix.org>
17:54:09
which could be painful for other reasons
<@sgallagh:fedora.im>
17:56:27
For now, I've added "Schedule an ELN mass-rebuild to pick up any pending RHEL X+1 changes" to the checklist
<@sgallagh:fedora.im>
17:56:46
With the word "Schedule" doing a lot of work in that sentence :)
<@conan_kudo:matrix.org>
17:57:25
works for me
<@yselkowitz:fedora.im>
17:57:47
it would apply equally to non-cs fedora branching events too
<@sgallagh:fedora.im>
17:58:08
yselkowitz: Why?
<@sgallagh:fedora.im>
17:58:24
It should only be needed when the `%rhel` macro increases
<@yselkowitz:fedora.im>
17:58:41
oh I guess
<@yselkowitz:fedora.im>
17:59:05
doesn't change my opinion on this though, it will need to wait for a while post-branching
<@conan_kudo:matrix.org>
17:59:15
there will not be a fedora-driven mass build for months after the eln bump to 11, since our mass build fedora side is next week
<@conan_kudo:matrix.org>
18:00:36
the branching isn't until after FOSDEM
<@sgallagh:fedora.im>
18:00:57
Feb 6, specifically
<@conan_kudo:matrix.org>
18:01:08
I will be on a plane that day :D
<@nhanlon:beeper.com>
18:02:45
get the wifi to keep an eye on rebuild ;)
<@sgallagh:fedora.im>
18:03:04
Well, the actual process is likely to take more than a day
<@nhanlon:beeper.com>
18:03:32
yeah, probably better to enjoy the free wine/spirits ;)
<@sgallagh:fedora.im>
18:03:38
We're going to stop syncing immediately, but the remainder of the tasks (including waiting for an in-progress batch to finish and sync over) may take some time
<@sgallagh:fedora.im>
18:05:26
Alright, we're technically a few minutes over time, so does anyone have other items to bring up?
<@conan_kudo:matrix.org>
18:06:01
nothing from me
<@nhanlon:beeper.com>
18:06:12
not I
<@sgallagh:fedora.im>
18:06:41
OK, thanks. I'll get this doc cleaned up and I'll send an MR for Fedora Docs in the next working day or two.
<@conan_kudo:matrix.org>
18:07:28
awesome, thanks Stephen Gallagher!
<@sgallagh:fedora.im>
18:07:48
Thank you all for your input
<@nhanlon:beeper.com>
18:08:16
thank you!
<@yselkowitz:fedora.im>
18:10:19
endmeeting?
<@zodbot:fedora.im>
18:10:35
ngompa has already given cookies to sgallagh during the F39 timeframe
<@sgallagh:fedora.im>
18:11:10
!endmeeting