2024-01-12 17:00:44 <@sgallagh:fedora.im> !startmeeting ELN (2024-01-12)
2024-01-12 17:00:44 <@meetbot:fedora.im> Meeting started at 2024-01-12 17:00:44 UTC
2024-01-12 17:00:45 <@meetbot:fedora.im> The Meeting name is 'ELN (2024-01-12)'
2024-01-12 17:00:50 <@sgallagh:fedora.im> !meetingname eln
2024-01-12 17:00:50 <@michel:one.ems.host> !hi
2024-01-12 17:00:53 <@zodbot:fedora.im> Michel Lind (salimma) - he / him / his
2024-01-12 17:01:15 <@sgallagh:fedora.im> @hellomynameis sgallagh
2024-01-12 17:01:27 <@sgallagh:fedora.im> !hellomynameis sgallagh
2024-01-12 17:01:28 <@zodbot:fedora.im> Stephen Gallagher (sgallagh) - he / him / his
2024-01-12 17:02:01 <@michel:one.ems.host> !hellomynameis michel-slm
2024-01-12 17:02:02 <@zodbot:fedora.im> Sorry, but Fedora Accounts user 'michel-slm' does not exist
2024-01-12 17:02:09 <@michel:one.ems.host> neat
2024-01-12 17:02:17 <@davide:cavalca.name> !hi
2024-01-12 17:02:20 <@zodbot:fedora.im> Davide Cavalca (dcavalca) - he / him / his
2024-01-12 17:02:37 <@yselkowitz:fedora.im> !hi
2024-01-12 17:02:38 <@zodbot:fedora.im> Yaakov Selkowitz (yselkowitz)
2024-01-12 17:03:17 <@nhanlon:beeper.com> !hi 
2024-01-12 17:03:30 <@zodbot:fedora.im> Neil Hanlon (neil) - he / him / his
2024-01-12 17:04:15 <@sgallagh:fedora.im> Alright, I think we have enough folks here to get started.
2024-01-12 17:05:02 <@sgallagh:fedora.im> !topic Breaking inheritance for CentOS Stream 10 (Or: ELN Goes to Eleven)
2024-01-12 17:05:15 <@nhanlon:beeper.com> weeee
2024-01-12 17:05:36 <@nhanlon:beeper.com> Spinal^WSystemTap
2024-01-12 17:05:57 <@salimma:fedora.im> my thought exactly
2024-01-12 17:06:33 <@sgallagh:fedora.im> As previously discussed many times, CentOS Stream 10 will stop inheriting automatically from ELN as of the Fedora 40 Branch event
2024-01-12 17:07:32 <@sgallagh:fedora.im> I'm currently working on putting together a document identifying all of the pieces we need to have in place to make this happen.
2024-01-12 17:08:37 <@conan_kudo:matrix.org> !hi
2024-01-12 17:08:40 <@zodbot:fedora.im> Neal Gompa (ngompa) - he / him / his
2024-01-12 17:09:01 <@sgallagh:fedora.im> It's not ready for sharing yet, but it will be replacing https://docs.fedoraproject.org/en-US/eln/branching/ in the very near future. (Also, that document is flat-out incorrect in several places)
2024-01-12 17:09:39 <@sgallagh:fedora.im> So I'll start by listing the things I already have (at a high level) and then I'd like to ask you all to remind me of the things I forgot.
2024-01-12 17:09:51 <@conan_kudo:matrix.org> sounds good
2024-01-12 17:10:15 <@nhanlon:beeper.com> > the only problem is, I can't remember what it is I've forgotten
2024-01-12 17:10:35 <@conan_kudo:matrix.org> that's too bad Neville
2024-01-12 17:10:36 <@sgallagh:fedora.im> Thanks, Neville
2024-01-12 17:11:07 <@nhanlon:beeper.com> you know, if you remove some letters...
2024-01-12 17:12:31 <@sgallagh:fedora.im> * Pause ELNBuildSync as Fedora RelEng starts the branching process. This will finish out any ongoing rebuild batch, but not start any new ones
* Disable the DistroBuildSync (ELN->CentOS Stream 10) entirely
* Update rpm.macro.rhel (ELN Koji Tag) and rhel_dist_version (fedora-release)
2024-01-12 17:13:26 <@sgallagh:fedora.im> * Rebuild fedora-release
* Update the tag trigger for the ELNBuildSync configuration to f41
* Resume ELNBuildSync
2024-01-12 17:14:07 <@sgallagh:fedora.im> That's most of what I have now. As I was typing this out, yselkowitz also added "bump rpm.macro.eln up to 135", which I'd like to discuss.
2024-01-12 17:14:43 <@yselkowitz:fedora.im> left a comment in the doc
2024-01-12 17:15:16 <@sgallagh:fedora.im> I'm sure there are other pieces I'm forgetting at the moment, so I'd like to give you all a few minutes to shout them out.
2024-01-12 17:15:48 <@yselkowitz:fedora.im> based on some issues we had post f39-branching, whenever fedora bumps dist-git, eln also needs to, otherwise we can miss rebuilds.
2024-01-12 17:15:54 <@conan_kudo:matrix.org> What about the forking side on CentOS Stream?
2024-01-12 17:16:31 <@sgallagh:fedora.im> Conan Kudo: I'm not sure what you're asking. Could you rephrase?
2024-01-12 17:16:51 <@yselkowitz:fedora.im> iiuc this doc is just about actions on the fedora side
2024-01-12 17:16:55 <@conan_kudo:matrix.org> Yes
2024-01-12 17:17:20 <@conan_kudo:matrix.org> I would like both sides of it to be covered in the doc because it ensures we understand the relationship of tasks
2024-01-12 17:18:18 <@sgallagh:fedora.im> I'm not sure there's actually anything specific we need to do on the Stream side.
2024-01-12 17:18:45 <@sgallagh:fedora.im> Maybe bump the dist tag "+" value?
2024-01-12 17:18:49 <@yselkowitz:fedora.im> well, we'll retire those packages that have since been dropped from eln, and do a mass-rebuild iirc?
2024-01-12 17:19:01 <@sgallagh:fedora.im> yselkowitz: Ah, okay. That's fair.
2024-01-12 17:19:21 <@yselkowitz:fedora.im> not sure that *this* doc is the place for that though
2024-01-12 17:19:31 <@yselkowitz:fedora.im> wasn't this going into fedora docs?
2024-01-12 17:19:32 <@conan_kudo:matrix.org> I can think of at least three: redhat-rpm-config's version needs to be frozen, synced but dropped packages need to be retired, and the disttag needs to be locked
2024-01-12 17:20:03 <@sgallagh:fedora.im> Can you explain what you mean by the first and last of those?
2024-01-12 17:20:49 <@sgallagh:fedora.im> yselkowitz: From a pure perspective, you're probably right. From a practical one, I think I'd rather keep related topics in the same doc, even if it might not STRICTLY belong in Fedora Docs
2024-01-12 17:21:01 <@conan_kudo:matrix.org> redhat-rpm-config currently uses %baserelease on the Version field, making every rpm bump a new version, which is chaotic and makes it difficult to identify where things came from, the version at fork point needs to be frozen and %baserelease needs to move to the Release field
2024-01-12 17:21:47 <@conan_kudo:matrix.org> the last one is about getting rid of the `+N` suffix that's floating in the disttag on the CentOS Stream Koji setup
2024-01-12 17:22:30 <@sgallagh:fedora.im> The +N thing will probably persist for a bit longer, as it's a useful tool for us to get things lined up until the full toolchain we expect to ship with lands.
2024-01-12 17:22:45 <@conan_kudo:matrix.org> sure, but it's a step that should be documented regardless :)
2024-01-12 17:22:47 <@sgallagh:fedora.im> It's analagous to the .elnNNN field
2024-01-12 17:23:14 <@conan_kudo:matrix.org> yes, I understand it to be similar to the buildcounter field I wanted to have added to Fedora's disttag
2024-01-12 17:23:20 <@sgallagh:fedora.im> Agreed, but as it's not going to happen as part of the inheritance-break, this is probably the wrong place to do so.
2024-01-12 17:23:27 <@sgallagh:fedora.im> But I appreciate you calling it out
2024-01-12 17:23:42 <@conan_kudo:matrix.org> the first one needs to happen at inheritance break though :)
2024-01-12 17:23:54 <@conan_kudo:matrix.org> that said, I just wanted to make sure it wasn't forgotten this time like it was last time
2024-01-12 17:24:14 <@sgallagh:fedora.im> Which "it" do you mean in that sentence?
2024-01-12 17:24:26 <@conan_kudo:matrix.org> redhat-rpm-config version freeze
2024-01-12 17:24:26 <@sgallagh:fedora.im> (Sorry, my fault for carrying two conversations on at once)
2024-01-12 17:25:47 <@sgallagh:fedora.im> While having `%baserelease` in the Version field definitely feels weird, is there an actual problem caused by it?
2024-01-12 17:25:58 <@yselkowitz:fedora.im> I understand the request
2024-01-12 17:26:34 <@yselkowitz:fedora.im> he's saying it makes it harder to see where rhel redhat-rpm-config branched from fedora
2024-01-12 17:26:38 <@conan_kudo:matrix.org> yes
2024-01-12 17:26:46 <@conan_kudo:matrix.org> and as redhat-rpm-config upstream, this is my personal request
2024-01-12 17:26:59 <@yselkowitz:fedora.im> so if we were to branch today at 276-1
2024-01-12 17:27:21 <@yselkowitz:fedora.im> and changes were applied to both fedora and c10s
2024-01-12 17:27:47 <@yselkowitz:fedora.im> then (w/o this) both would become 277-1 but not necessarily identical (if the changes aren't the same)
2024-01-12 17:27:55 <@conan_kudo:matrix.org> exactly
2024-01-12 17:27:59 <@conan_kudo:matrix.org> I asked last time too, but it got forgotten in the craziness of doing it for the first time :P
2024-01-12 17:28:20 <@sgallagh:fedora.im> OK, that makes good sense. I'll add that to the SOP
2024-01-12 17:28:30 <@conan_kudo:matrix.org> thank you
2024-01-12 17:28:55 <@conan_kudo:matrix.org> I really appreciate it
2024-01-12 17:31:15 <@sgallagh:fedora.im> Any other thoughts on branch-time events?
2024-01-12 17:31:36 <@conan_kudo:matrix.org> I can't think of any others offhand?
2024-01-12 17:31:51 <@conan_kudo:matrix.org> I assume a mass build happens right after on both sides
2024-01-12 17:31:58 <@conan_kudo:matrix.org> so that cleans up things after that
2024-01-12 17:32:11 <@sgallagh:fedora.im> Not exactly
2024-01-12 17:32:17 <@yselkowitz:fedora.im> c10s yes, not eln
2024-01-12 17:32:19 <@yselkowitz:fedora.im> ?
2024-01-12 17:32:38 <@sgallagh:fedora.im> For Fedora ELN, we won't do a mass-rebuild because we'll have just completed the F40 one
2024-01-12 17:32:52 <@sgallagh:fedora.im> I'm not sure what value there would be to doing another so soon
2024-01-12 17:33:01 <@conan_kudo:matrix.org> ticking over the RHEL 11 conditional
2024-01-12 17:33:03 <@yselkowitz:fedora.im> none, thanks for asking :-)
2024-01-12 17:33:27 <@yselkowitz:fedora.im> in rawhide mass rebuilds are before branching, not after, how is this different?
2024-01-12 17:33:29 <@sgallagh:fedora.im> Hmm, that's an interesting point.
2024-01-12 17:33:40 <@conan_kudo:matrix.org> we did that last time too, IIRC
2024-01-12 17:33:56 <@conan_kudo:matrix.org> because ELN needs to tick over to the next RHEL after branching
2024-01-12 17:33:57 <@sgallagh:fedora.im> yselkowitz: Basically, if we're bumping the `%rhel` value, then there are probably some conditionals out there that may function differently.
2024-01-12 17:34:09 <@yselkowitz:fedora.im> dnf5 yes
2024-01-12 17:34:18 <@conan_kudo:matrix.org> pipewire will too
2024-01-12 17:34:22 <@conan_kudo:matrix.org> as will a few others
2024-01-12 17:34:26 <@sgallagh:fedora.im> But I'm not sure we necessarily need to do the mass-build right away; we'll be plenty busy on the CS side for those next few weeks
2024-01-12 17:34:43 <@yselkowitz:fedora.im> but wouldn't there already have to be rebuilds in rawhide to trigger those anyway?
2024-01-12 17:34:48 <@conan_kudo:matrix.org> it needs to be scheduled, but not immediately
2024-01-12 17:35:11 <@sgallagh:fedora.im> yselkowitz: Sorry, I think I missed a JMP in there somewhere.
2024-01-12 17:35:27 <@conan_kudo:matrix.org> for example, the frame pointers flag will take effect for RHEL-ELN 11
2024-01-12 17:35:37 <@conan_kudo:matrix.org> it currently does not
2024-01-12 17:35:47 <@sgallagh:fedora.im> Ooh, that's a really good point.
2024-01-12 17:35:51 <@yselkowitz:fedora.im> for individual packages that have pre-set macros to change in f41
2024-01-12 17:36:00 <@yselkowitz:fedora.im> e.g. dnf5 (to obsolete dnf)
2024-01-12 17:36:16 <@conan_kudo:matrix.org> pipewire will re-enable libcamera support in RHEL-ELN 11
2024-01-12 17:36:25 <@yselkowitz:fedora.im> then there would have to anyways be a rebuild in rawhide to actually trigger that change, which will trigger an eln build
2024-01-12 17:36:27 <@conan_kudo:matrix.org> there is a list I keep track of these things for :P
2024-01-12 17:37:12 <@sgallagh:fedora.im> yselkowitz: I think he's listing packages/changes that specifically have `%if 0%{?rhel} > 10` or equivalent
2024-01-12 17:37:35 <@yselkowitz:fedora.im> how many of those don't also have `0%{?fedora} > 40` etc?
2024-01-12 17:37:54 <@sgallagh:fedora.im> The frame pointer one absolutely does need us to do a mass-rebuild at some point. While we *can* wait for a Fedora mass-rebuild to trigger it, I suspect the folks that want it around will want it faster.
2024-01-12 17:38:14 <@yselkowitz:fedora.im> it can wait
2024-01-12 17:38:30 <@sgallagh:fedora.im> yselkowitz: At least the FP one and it sounds like the pipewire one as well, though that's very targeted and wouldn't need a mass-rebuild
2024-01-12 17:38:40 <@conan_kudo:matrix.org> the point isn't whether or not to do it, the point is that it needs to scheduled
2024-01-12 17:38:46 <@conan_kudo:matrix.org> and part of the SOP
2024-01-12 17:39:35 <@conan_kudo:matrix.org> we already know that Fedora and RHEL differ at the low levels enough that we cannot trust Fedora rebuilds to imply ELN ones when ELN ticks over
2024-01-12 17:39:35 <@davide:cavalca.name> agreed, I think we definitely want a mass rebuild sooner than later for consistency
2024-01-12 17:39:49 <@sgallagh:fedora.im> There's also something to be said for doing that mass-rebuild fairly quickly, before the dependency chain spikes again :)
2024-01-12 17:40:24 <@yselkowitz:fedora.im> it's going to need to wait a bit
2024-01-12 17:40:27 <@sgallagh:fedora.im> I'm inclined to make it part of the branching process, but with the understanding that we may not get around to FIXING any issues revealed by it for some time
2024-01-12 17:41:07 <@yselkowitz:fedora.im> it's going to need to wait
2024-01-12 17:42:55 <@davide:cavalca.name> I think that's an acceptable tradeoff, it's understood folks will be busy, but I'd rather the state of the world reflect reality than not -- a pre-mass-rebuild state would be pretty misleading to use/test against
2024-01-12 17:43:51 <@nhanlon:beeper.com> I'm in agreement with Davide and Neal here. If it cannot be done, then I think we need to discuss _why_ it can't be done (or has to wait).
2024-01-12 17:44:22 <@yselkowitz:fedora.im> because we're not going to have the resources to deal with another mass rebuild immediately after branching
2024-01-12 17:44:33 <@conan_kudo:matrix.org> I'm fine with this. It also allows the community to assist in fixing issues.
2024-01-12 17:44:59 <@conan_kudo:matrix.org> what kind of resources? because compute resources we have
2024-01-12 17:45:14 <@nhanlon:beeper.com> thank you for explaining
2024-01-12 17:45:33 <@conan_kudo:matrix.org> and if we don't, then that's a problem that needs to be bubbled up
2024-01-12 17:45:38 <@yselkowitz:fedora.im> people resources, specifically from our team
2024-01-12 17:45:49 <@sgallagh:fedora.im> yselkowitz: I think what you're saying is that you expect the fallout to require a lot of attention and that you think it has to be you and I that provide it. I'm suggesting (and it seems that Conan Kudo is offering) that others can step up
2024-01-12 17:46:35 <@yselkowitz:fedora.im> any mass-rebuild requires a lot of attention
2024-01-12 17:46:44 <@conan_kudo:matrix.org> Yes.
2024-01-12 17:47:07 <@davide:cavalca.name> yeah, if you/your team are currently a SPOF for this that's something we should strive to improve
2024-01-12 17:47:08 <@sgallagh:fedora.im> Was that in reply to me, yselkowitz or both?
2024-01-12 17:47:16 <@conan_kudo:matrix.org> Yes.
2024-01-12 17:47:43 <@davide:cavalca.name> e.g. if there's specific things that aren't well documented yet we should focus on fixing that so others can pitch in
2024-01-12 17:48:03 <@yselkowitz:fedora.im> the way things are now, packages will start to be built with frame pointers post branching, and that will be a start.
2024-01-12 17:48:03 <@conan_kudo:matrix.org> Exactly. We fell into this trap last time, and we should endeavor to resolve it this time.
2024-01-12 17:48:05 <@sgallagh:fedora.im> There's one other hiccough here that may throw a spanner into the works.
2024-01-12 17:48:26 <@sgallagh:fedora.im> Fedora Release Engineering wants to decommission ODCS, which ELN is using for its composes.
2024-01-12 17:48:34 <@yselkowitz:fedora.im> e.g. most of the desktop will see updates post branching in prep for gnome 46 final and f40 ga
2024-01-12 17:48:35 <@conan_kudo:matrix.org> Oh.
2024-01-12 17:48:49 <@sgallagh:fedora.im> The effect of which is that we're going to end up with a changed location for pulling down composes as well
2024-01-12 17:49:00 <@conan_kudo:matrix.org> Wait, is ODCS not maintained?
2024-01-12 17:49:27 <@sgallagh:fedora.im> Apparently, Fedora ELN is the last user of it in Fedora-land
2024-01-12 17:49:33 <@conan_kudo:matrix.org> That's the only reason I've seen for decommissioning anything.
2024-01-12 17:49:45 <@sgallagh:fedora.im> And yeah, it's effectively unmaintained.
2024-01-12 17:49:48 <@davide:cavalca.name> what's the alternative to ODCS?
2024-01-12 17:49:50 <@yselkowitz:fedora.im> well it will be once osbs is decommissioned
2024-01-12 17:50:02 <@sgallagh:fedora.im> Davide Cavalca: We'll use the same compose infrastructure as Fedora
2024-01-12 17:50:14 <@conan_kudo:matrix.org> Where is that managed/configured?
2024-01-12 17:50:55 <@conan_kudo:matrix.org> Is it just pungi and comps now?
2024-01-12 17:50:55 <@sgallagh:fedora.im> Conan Kudo: Which part?
2024-01-12 17:51:15 <@conan_kudo:matrix.org> the distrobuildsync -> build artifacts -> publish pipeline
2024-01-12 17:52:02 <@conan_kudo:matrix.org> if ODCS goes away, I'm not sure what handles that stuff
2024-01-12 17:52:03 <@sgallagh:fedora.im> Right now it's basically a cron-job that asks ODCS to perform a compose based on the pungi-fedora `eln` branch
2024-01-12 17:52:18 <@conan_kudo:matrix.org> right
2024-01-12 17:52:42 <@sgallagh:fedora.im> I don't have all the details yet on what it's going to look like post-ODCS. Something about `nightly.sh` in pungi-fedora
2024-01-12 17:53:02 <@conan_kudo:matrix.org> well then keep us posted and try to document it somewhere :)
2024-01-12 17:53:08 <@sgallagh:fedora.im> Will do
2024-01-12 17:53:35 <@sgallagh:fedora.im> But mostly I bring it up because if composes are going to be in flux in this same timeframe, it may not be super-beneficial to rush to do the mass-rebuild
2024-01-12 17:54:01 <@conan_kudo:matrix.org> on the flip side, we may not get a compose for a while post-shutdown
2024-01-12 17:54:09 <@conan_kudo:matrix.org> which could be painful for other reasons
2024-01-12 17:56:27 <@sgallagh:fedora.im> For now, I've added "Schedule an ELN mass-rebuild to pick up any pending RHEL X+1 changes" to the checklist
2024-01-12 17:56:46 <@sgallagh:fedora.im> With the word "Schedule" doing a lot of work in that sentence :)
2024-01-12 17:57:25 <@conan_kudo:matrix.org> works for me
2024-01-12 17:57:47 <@yselkowitz:fedora.im> it would apply equally to non-cs fedora branching events too
2024-01-12 17:58:08 <@sgallagh:fedora.im> yselkowitz: Why?
2024-01-12 17:58:24 <@sgallagh:fedora.im> It should only be needed when the `%rhel` macro increases
2024-01-12 17:58:41 <@yselkowitz:fedora.im> oh I guess
2024-01-12 17:59:05 <@yselkowitz:fedora.im> doesn't change my opinion on this though, it will need to wait for a while post-branching
2024-01-12 17:59:15 <@conan_kudo:matrix.org> there will not be a fedora-driven mass build for months after the eln bump to 11, since our mass build fedora side is next week
2024-01-12 18:00:36 <@conan_kudo:matrix.org> the branching isn't until after FOSDEM
2024-01-12 18:00:57 <@sgallagh:fedora.im> Feb 6, specifically
2024-01-12 18:01:08 <@conan_kudo:matrix.org> I will be on a plane that day :D
2024-01-12 18:02:45 <@nhanlon:beeper.com> get the wifi to keep an eye on rebuild ;) 
2024-01-12 18:03:04 <@sgallagh:fedora.im> Well, the actual process is likely to take more than a day
2024-01-12 18:03:32 <@nhanlon:beeper.com> yeah, probably better to enjoy the free wine/spirits ;)
2024-01-12 18:03:38 <@sgallagh:fedora.im> We're going to stop syncing immediately, but the remainder of the tasks (including waiting for an in-progress batch to finish and sync over) may take some time
2024-01-12 18:05:26 <@sgallagh:fedora.im> Alright, we're technically a few minutes over time, so does anyone have other items to bring up?
2024-01-12 18:06:01 <@conan_kudo:matrix.org> nothing from me
2024-01-12 18:06:12 <@nhanlon:beeper.com> not I
2024-01-12 18:06:41 <@sgallagh:fedora.im> OK, thanks. I'll get this doc cleaned up and I'll send an MR for Fedora Docs in the next working day or two.
2024-01-12 18:07:28 <@conan_kudo:matrix.org> awesome, thanks Stephen Gallagher!
2024-01-12 18:07:48 <@sgallagh:fedora.im> Thank you all for your input
2024-01-12 18:08:16 <@nhanlon:beeper.com> thank you!
2024-01-12 18:10:19 <@yselkowitz:fedora.im> endmeeting?
2024-01-12 18:10:35 <@zodbot:fedora.im> ngompa has already given cookies to sgallagh during the F39 timeframe
2024-01-12 18:11:10 <@sgallagh:fedora.im> !endmeeting