2024-07-26 16:00:10 <@sgallagh:fedora.im> !startmeeting ELN SIG (2024-07-26)
2024-07-26 16:00:12 <@meetbot:fedora.im> Meeting started at 2024-07-26 16:00:10 UTC
2024-07-26 16:00:12 <@meetbot:fedora.im> The Meeting name is 'ELN SIG (2024-07-26)'
2024-07-26 16:00:15 <@sgallagh:fedora.im> !meetingname eln
2024-07-26 16:00:15 <@meetbot:fedora.im> The Meeting Name is now eln
2024-07-26 16:00:28 <@sgallagh:fedora.im> !topic Init Process
2024-07-26 16:00:29 <@sgallagh:fedora.im> !hi
2024-07-26 16:00:30 <@zodbot:fedora.im> Stephen Gallagher (sgallagh) - he / him / his
2024-07-26 16:00:37 <@tdawson:fedora.im> !hi
2024-07-26 16:00:38 <@zodbot:fedora.im> Troy Dawson (tdawson)
2024-07-26 16:01:47 <@salimma:fedora.im> !hi
2024-07-26 16:01:48 <@zodbot:fedora.im> Michel Lind (salimma) - he / him / his
2024-07-26 16:02:12 <@yselkowitz:fedora.im> !hi
2024-07-26 16:02:14 <@zodbot:fedora.im> Yaakov Selkowitz (yselkowitz)
2024-07-26 16:03:11 <@sgallagh:fedora.im> I pinged Carl George to join as well, since we'll be discussing EPEL 11 a bit
2024-07-26 16:03:38 <@carlwgeorge:matrix.org> !hi
2024-07-26 16:03:39 <@zodbot:fedora.im> Carl George (carlwgeorge) - he / him / his
2024-07-26 16:03:47 <@sgallagh:fedora.im> !topic Agenda
2024-07-26 16:03:58 <@sgallagh:fedora.im> !info Agenda Item: Sunsetting ELN Extras
2024-07-26 16:04:12 <@sgallagh:fedora.im> I'd normally ask for other topics, but I expect this one to take the full time.
2024-07-26 16:04:42 <@sgallagh:fedora.im> !topic Sunsetting ELN Extras
2024-07-26 16:05:19 <@sgallagh:fedora.im> So, first I'll try to summarize the reasons why we're going to drop ELN Extras, then we can discuss the "how" of it.
2024-07-26 16:05:50 <@nhanlon:beeper.com> !hi 
2024-07-26 16:05:52 <@zodbot:fedora.im> Neil Hanlon (neil) - he / him / his
2024-07-26 16:06:21 <@sgallagh:fedora.im> The original goal of ELN Extras was to provide a way to bootstrap EPEL 10 early, so that we could have it ready to go when CentOS Stream 10 launched.
2024-07-26 16:07:02 <@davide:cavalca.name> !hi
2024-07-26 16:07:03 <@zodbot:fedora.im> Davide Cavalca (dcavalca) - he / him / his
2024-07-26 16:07:05 <@sgallagh:fedora.im> This failed, partly due to resource constraints, because we didn't branch ELN Extras to EPEL 10 at the same time we forked CentOS Stream 10 from ELN/Rawhide
2024-07-26 16:07:10 <@conan_kudo:matrix.org> !hi
2024-07-26 16:07:12 <@zodbot:fedora.im> Neal Gompa (ngompa) - he / him / his
2024-07-26 16:07:59 <@carlwgeorge:matrix.org> was that the original goal?  i only remember talk about it being used as a way to ensure spec files were ready macro wise, not for any actual bootstrapping.
2024-07-26 16:07:59 <@salimma:fedora.im> did it really fail though? we can still do those branches as soon as EPEL 10 is created in a couple of weeks or so
2024-07-26 16:08:02 <@sgallagh:fedora.im> So, ELN Extras has continued forwards, tracking Rawhide and is now in a state that we cannot comfortably assert is compatible with EL 10
2024-07-26 16:08:06 <@conan_kudo:matrix.org> Carl George: yes it was
2024-07-26 16:08:13 <@conan_kudo:matrix.org> that was a big part of why ELN Extras was created
2024-07-26 16:08:17 <@sgallagh:fedora.im> Carl George: Yes, that was our original pitch for it
2024-07-26 16:08:45 <@salimma:fedora.im> any downside to keeping it this way and retrying for 11?
2024-07-26 16:08:52 <@davide:cavalca.name> the other benefit of ELN Extras is ensuring that packages outside of core RHEL can be tested and used against ELN
2024-07-26 16:08:53 <@conan_kudo:matrix.org> not particularly
2024-07-26 16:08:56 <@salimma:fedora.im> after all, it still helps packages get ready macro wise as Carl noted
2024-07-26 16:09:19 <@davide:cavalca.name> with my Meta hat on, this is primarily what we use it for, and it's been pretty much instrumental in making it possible for us to put together an ELN testbed
2024-07-26 16:09:24 <@salimma:fedora.im> it would certainly make our ELN test fleet at Meta way harder to use if we don't have this
2024-07-26 16:09:26 <@salimma:fedora.im> ah, jinks
2024-07-26 16:09:39 <@davide:cavalca.name> as there are far too many packages not in the core distros that are needed in the real world for a deployment
2024-07-26 16:09:41 <@conan_kudo:matrix.org> indeed, one of the hopes here is that we'd have less trouble building these things in EPEL if they're being continually built and reported on for RHEL's content sizing
2024-07-26 16:09:58 <@davide:cavalca.name> so, while I'm not personally married to the current implemention, we do need something like this going forward
2024-07-26 16:10:17 <@salimma:fedora.im> and that package set is growing over time
2024-07-26 16:10:22 <@conan_kudo:matrix.org> Right. And we've needed it in Fedora KDE to ensure things aren't terribly broken for the stack too.
2024-07-26 16:10:29 <@tdawson:fedora.im> But one of the problems is that it ISN'T working.  Look at KDE, qt6, and ffmpeg ... none of those workloads work dues to really strange reason.
2024-07-26 16:10:43 <@sgallagh:fedora.im> I'm not finished listing the problems with it, but to jump ahead slightly, we want to replace it with a much earlier EPEL 11 project. I'll get to that in a few minutes.
2024-07-26 16:10:52 <@salimma:fedora.im> ah
2024-07-26 16:10:56 <@conan_kudo:matrix.org> one of the larger problems with ELN is that we don't do composes with images and run tests
2024-07-26 16:11:02 <@salimma:fedora.im> Stephen Gallagher: you could have led with that :D
2024-07-26 16:11:04 <@davide:cavalca.name> so, part of the problem with failing workloads is that there is no monitoring/alerting (that I know of at least)
2024-07-26 16:11:09 <@conan_kudo:matrix.org> this affects both ELN core and ELN extras
2024-07-26 16:11:26 <@conan_kudo:matrix.org> in fact, ELN is probably the least tested part of Fedora
2024-07-26 16:11:42 <@davide:cavalca.name> I generally spot issues with the Meta and/or workloads when they crop up internally, but if we had something that say emailed you whenever the workload failed then I suspect people would be more quick to act and fix them
2024-07-26 16:11:50 <@davide:cavalca.name> I generally spot issues with the Meta and/or HS workloads when they crop up internally, but if we had something that say emailed you whenever the workload failed then I suspect people would be more quick to act and fix them
2024-07-26 16:12:09 <@conan_kudo:matrix.org> one of the things I haven't been particularly happy about is treating ELN as a non-deliverable
2024-07-26 16:12:15 <@salimma:fedora.im> yeah, ISTR I've been fixing quite a few ELN build failures over time so... do issues really remain unfixed, or are we talking about other kinds of failures
2024-07-26 16:12:29 <@yselkowitz:fedora.im> I look at CR 6 days a week, if a workload starts failing I usually notice pretty quickly
2024-07-26 16:12:29 <@davide:cavalca.name> ok, let's let Stephen Gallagher finish and then we can continue the pile up :)
2024-07-26 16:13:53 <@sgallagh:fedora.im> Additional problems with ELN Extras is that it complicates our rebuilds and compose, at least in part because I didn't fully consider the ramifications of doing it all in the `eln` Koji tag.
2024-07-26 16:14:36 <@sgallagh:fedora.im> We have hack upon hack piled up in both ELNBuildSync and the compose configuration to pull in ELN Extras content.
2024-07-26 16:15:23 <@salimma:fedora.im> ah. so yeah making it more EPEL like is probably better
2024-07-26 16:15:31 <@conan_kudo:matrix.org> well maybe
2024-07-26 16:15:38 <@yselkowitz:fedora.im> and it makes it more difficult to filter unwanted packages from ELN in CR
2024-07-26 16:16:07 <@conan_kudo:matrix.org> some of this sounds like stuffing ELN in Fedora Koji was a mistake
2024-07-26 16:16:27 <@yselkowitz:fedora.im> no, putting ELN and ELN Extras together though was
2024-07-26 16:16:40 <@sgallagh:fedora.im> I was typing the same sentence, so ^^
2024-07-26 16:16:52 <@conan_kudo:matrix.org> but that's also solvable by splitting the tag
2024-07-26 16:17:09 <@conan_kudo:matrix.org> to put it bluntly, I don't want the EPEL workflow for ELN Extras packages
2024-07-26 16:17:12 <@sgallagh:fedora.im> Yes, which is one of the things I was going to propose.
2024-07-26 16:17:41 <@sgallagh:fedora.im> (splitting the tag)
2024-07-26 16:18:14 <@sgallagh:fedora.im> But by "I don't want the EPEL workflow" do you mean you don't want packagers to have to do their own commits and builds for ELN Extras?
2024-07-26 16:18:47 <@conan_kudo:matrix.org> yes
2024-07-26 16:20:29 <@sgallagh:fedora.im> One of the additional problems we've had with ELN Extras is that the automatic nature of rebuilds has led some people to assume that it means that "someone else" (Read: the people in this meeting) are responsible for fixing things when they go wrong.
2024-07-26 16:20:50 <@conan_kudo:matrix.org> do we file bugs when things fail?
2024-07-26 16:21:00 <@yselkowitz:fedora.im> which is pretty much the same for ELN itself fwiw...
2024-07-26 16:21:17 <@conan_kudo:matrix.org> because I know I am not getting bug reports for ELN failures
2024-07-26 16:21:23 <@yselkowitz:fedora.im> but every time I think I'm going to start having time to look at ELN Extras, something else comes up
2024-07-26 16:21:28 <@davide:cavalca.name> otoh, manual builds would be a lot of toil due to how much ELN changes, and doesn't seem especially practical
2024-07-26 16:21:41 <@sgallagh:fedora.im> Conan Kudo: We've been doing bugs for things in ELN proper
2024-07-26 16:21:51 <@sgallagh:fedora.im> But we literally do not have the cycles to handle ELN Extras
2024-07-26 16:22:05 <@salimma:fedora.im> right. I want it to be EPEL like but we don't want the package maintainers to have to do manual builds, the whole point is because they often don't bringup is slower :)
2024-07-26 16:22:12 <@conan_kudo:matrix.org> from my point of view, "ELN proper" is both ELN core and ELN extras
2024-07-26 16:22:21 <@yselkowitz:fedora.im> for us it's not
2024-07-26 16:22:34 <@salimma:fedora.im> there are two viewpoints. for prepping for the next EL, it's only core
2024-07-26 16:22:46 <@salimma:fedora.im> from the perspective of those who care about EPEL, both need to work
2024-07-26 16:22:55 <@sgallagh:fedora.im> ELN "core" is what Red Hat is paying us to do. ELN Extras is and always has been a volunteer effort
2024-07-26 16:23:02 <@conan_kudo:matrix.org> so here's the rub: if you don't consider it that way, everything for the people that contribute to supporting the EL ecosystem will always be broken
2024-07-26 16:23:07 <@sgallagh:fedora.im> I hate to break it down like that, but it's relevant here
2024-07-26 16:23:21 <@conan_kudo:matrix.org> and I hate to say it, but people don't use EL unless the extras are there
2024-07-26 16:23:40 <@davide:cavalca.name> that's fair, but in the end ELN Extras allows folks to actually test ELN in the real world and fix bugs/provide feedback/etc
2024-07-26 16:23:42 <@sgallagh:fedora.im> Conan Kudo: Which is why we now have people paid to work on EPEL :)
2024-07-26 16:23:42 <@conan_kudo:matrix.org> EL core continues to shrink release after release, forcing people to need more and more from extras for the same workloads
2024-07-26 16:23:42 <@tdawson:fedora.im> So, have YOU filed any bugs for eln extras?  Because your workloads have alot of them.
2024-07-26 16:24:07 <@conan_kudo:matrix.org> No, I go through and try to fix them as I work through them even in rawhide
2024-07-26 16:24:15 <@conan_kudo:matrix.org> what's the point in filing bugs against myself?
2024-07-26 16:24:35 <@conan_kudo:matrix.org> I fix ELN bugs in both core and extras when I spot them
2024-07-26 16:24:39 <@conan_kudo:matrix.org> I don't even tell people, I just do it
2024-07-26 16:24:53 <@sgallagh:fedora.im> No one is saying that the content we have in ELN Extras is unnecessary or useless. Just that we don't have the capability to handle it under the ELN umbrella.
2024-07-26 16:25:18 <@sgallagh:fedora.im> Which is why I am trying to get to the part where we work with EPEL to get it moved under that umbrella.
2024-07-26 16:25:29 <@yselkowitz:fedora.im> I wouldn't mind knowing when that happens, so that I can track fixes
2024-07-26 16:25:38 <@salimma:fedora.im> I think we're more in agreement than this debate seems to imply... can we skip ahead to the part about this "early EPEL 11" since I'm sure that will address some concerns?
2024-07-26 16:25:40 <@sgallagh:fedora.im> At least in part because the EPEL brand is much more established and people have demonstrated they're willing to work on it
2024-07-26 16:25:47 <@conan_kudo:matrix.org> There is no resources to work on it in EPEL and the EPEL workflows are painful for a churning distribution.
2024-07-26 16:25:55 <@davide:cavalca.name> fwiw I also generally fix bugs when I find them, but if there's value in tracking them more explicitly we can certainly look into doing that
2024-07-26 16:26:09 <@davide:cavalca.name> let's wait to actually hear Stephen Gallagher 's proposal here
2024-07-26 16:26:21 <@salimma:fedora.im> suggestion: should we have a standard way to write changelog entries so we can easily identify ELN fixes
2024-07-26 16:26:26 <@sgallagh:fedora.im> Conan Kudo: You are making assumptions. I'm trying to establish a starting point for a discussion.
2024-07-26 16:26:31 <@carlwgeorge:matrix.org> we can have different workflows for epel at different points in time
2024-07-26 16:27:05 <@conan_kudo:matrix.org> Fundamentally, I don't want spurious rebuild commits being checked for every time an ELN core rebuild happens
2024-07-26 16:27:30 <@conan_kudo:matrix.org> I don't want to have to manually massage every little thing
2024-07-26 16:27:37 <@conan_kudo:matrix.org> I don't have time for that
2024-07-26 16:27:38 <@yselkowitz:fedora.im> that's easily avoidable with incrementing dist-tags like we do in ELN
2024-07-26 16:27:52 <@sgallagh:fedora.im> Conan Kudo: If you'd let us at least *start* describing things, you might see we have some ideas around that.
2024-07-26 16:28:05 <@tdawson:fedora.im> Conan Kudo: Please let him speak.
2024-07-26 16:29:07 <@sgallagh:fedora.im> OK, I'm going to start explaining the high-level ideas I've had, and I'll EOM when I've gotten them all down.
2024-07-26 16:29:51 <@sgallagh:fedora.im> First of all, I agree that we want to have certain behaviors maintained from ELN Extras, such as automatic rebuilds of Rawhide content up to a certain point.
2024-07-26 16:30:38 <@sgallagh:fedora.im> I think that ELN as a "brand" has not gained the traction it needs for us to get greater participation in ELN Extras, despite it being valuable to the eventual release of EPEL N+1.
2024-07-26 16:31:02 <@sgallagh:fedora.im> We also described some of the build and compose issues related to having them share a tag infrastructure.
2024-07-26 16:31:40 <@sgallagh:fedora.im> What I propose is that we rebrand ELN Extras as an EPEL project and set up its tags accordingly.
2024-07-26 16:32:20 <@sgallagh:fedora.im> We should maintain automatic Rawhide rebuilds (including mass-rebuilds) up to the point where we branch c11s (I'm going to use 11 as an example here rather than keep saying N+1).
2024-07-26 16:33:18 <@sgallagh:fedora.im> Prior to that branch, our dist tags will be something like `.el~epel11_<buildrootnumber>` where `<buildrootnumber>` will be kept in sync with ELN for consistency.
2024-07-26 16:33:44 <@sgallagh:fedora.im> When we branch, we'll rebuild with `.el11` and start treating it with the traditional EPEL workflow.
2024-07-26 16:33:49 <@sgallagh:fedora.im> EOM
2024-07-26 16:33:51 <@salimma:fedora.im> this would certainly help with all those "when is EPEL X going to be ready" too, since now it will be ready even before cXs is created right?
2024-07-26 16:34:05 <@sgallagh:fedora.im> (That's as far as I've gotten so far, please poke at it)
2024-07-26 16:34:07 <@salimma:fedora.im> EPEL 11 tracks eln now, and tracks c11s once that is branched at which point we stop auto rebuilds
2024-07-26 16:34:23 <@salimma:fedora.im> I like this. no more begging people to please, please branch their packages 
2024-07-26 16:34:32 <@conan_kudo:matrix.org> There's still a problem.
2024-07-26 16:34:42 <@sgallagh:fedora.im> Oh, right, sorry. I forgot to mention: EPEL 11 will use ELN as its buildroot until the branch, at which point it will use CS 11
2024-07-26 16:34:48 <@conan_kudo:matrix.org> Nobody can use these packages easily, which makes development and testing hard.
2024-07-26 16:35:00 <@salimma:fedora.im> I don't understand. how?
2024-07-26 16:35:10 <@salimma:fedora.im> talking about the EPEL 11 side or the ELN Core side?
2024-07-26 16:35:10 <@davide:cavalca.name> won't these show up in a repo?
2024-07-26 16:35:12 <@conan_kudo:matrix.org> we don't publish images, we don't publish repos, there are no mirrors, and there is no website for it
2024-07-26 16:35:22 <@tdawson:fedora.im> I don't understand the last sentance "When we branch, we'll rebuild with .el11"  Are both "we's" in the context the ELN SIG?
2024-07-26 16:35:30 <@conan_kudo:matrix.org> that is 99% of the problem with ELN contributors
2024-07-26 16:35:37 <@davide:cavalca.name> afair there was talk of moving ELN to the mirror network, but that seems orthogonal to the topic at hand
2024-07-26 16:35:40 <@sgallagh:fedora.im> Conan Kudo: All of that is both untrue and in the process of being brought in line with Fedora right now
2024-07-26 16:35:48 <@davide:cavalca.name> and I'd assume if this is under EPEL it _would_ be on the mirror network to begin with
2024-07-26 16:35:50 <@yselkowitz:fedora.im> that's not the problem we're trying to solve here
2024-07-26 16:36:09 <@conan_kudo:matrix.org> the problem you're trying to solve is unburdening yourselves and scaling ELN Extras by making it an EPEL project
2024-07-26 16:36:11 <@salimma:fedora.im> yeah, ELN Core visibility is orthogonal to this topic. and ELN Extras will be more visible as part of EPEL, no? it will be mirrored by EPEL mirrors
2024-07-26 16:36:20 <@conan_kudo:matrix.org> therefore it needs to have some basic usability for people to actively work on it
2024-07-26 16:36:31 <@sgallagh:fedora.im> Let's not sidetrack this, but ELN is being moved to the common Fedora compose infrastructure and will be moving to mirrormanager. This is happening in the next month
2024-07-26 16:36:38 <@salimma:fedora.im> can we discuss that later? as Stephen said that's being worked on
2024-07-26 16:37:00 <@salimma:fedora.im> we're 36 mins in and we should talk more about how this EPEL 11 will work
2024-07-26 16:37:14 <@conan_kudo:matrix.org> seems like it's pretty set with how "EPEL 11" will work
2024-07-26 16:37:22 <@conan_kudo:matrix.org> he seems to have figured that out
2024-07-26 16:37:31 <@sgallagh:fedora.im> I made a proposal, I'm asking for feedback and corrections
2024-07-26 16:38:06 <@conan_kudo:matrix.org> my feedback is that it needs to be tied to some kind of public visibility of ELN
2024-07-26 16:38:11 <@davide:cavalca.name> is the plan to keep using an `eln` branch when we need a branch for this?
2024-07-26 16:38:20 <@conan_kudo:matrix.org> otherwise it will also fail due to lack of people able to work on it
2024-07-26 16:38:23 <@salimma:fedora.im> I'm in favor (and I guess the "tracking ELN" was my only question). any package that got transitively pulled in by an ELN Extras workload will get branched, right?
2024-07-26 16:38:33 <@sgallagh:fedora.im> Davide Cavalca: No, and that's a good question. I think it'll need to branch to `epel11` instead.
2024-07-26 16:38:37 <@salimma:fedora.im> right, the other question is if the branch is eln or epel11
2024-07-26 16:39:02 <@davide:cavalca.name> I'm ok with that; we just then need to be careful when ELN jumps and it goes `epel11` -> `epel12`
2024-07-26 16:39:11 <@yselkowitz:fedora.im> it's already failing for the same reason
2024-07-26 16:40:01 <@conan_kudo:matrix.org> yeah, but we don't change how we approach ELN as a whole, it will stay failing
2024-07-26 16:40:10 <@conan_kudo:matrix.org> I'm saying your approach isn't working
2024-07-26 16:40:29 <@yselkowitz:fedora.im> ELN is working very well for it's primary purpose, actually
2024-07-26 16:41:22 <@conan_kudo:matrix.org> probably not too much
2024-07-26 16:41:25 <@sgallagh:fedora.im> Davide Cavalca: Someone with time to do the work
2024-07-26 16:41:27 <@yselkowitz:fedora.im> or ELN for that matter...
2024-07-26 16:41:30 <@salimma:fedora.im> Stephen Gallagher: ^ last question, on automatic branching
2024-07-26 16:41:39 <@conan_kudo:matrix.org> the main issue is that we need something to read and track both core and extras tags
2024-07-26 16:41:44 <@carlwgeorge:matrix.org> i think it's possible to revamp eln extras into early epel11 and leave eln to continue making other improvements for the overall success of eln
2024-07-26 16:41:51 <@conan_kudo:matrix.org> which shouldn't be hard, the existing stuff can be changed to read multiple repos together
2024-07-26 16:41:55 <@carlwgeorge:matrix.org> we don't have to boil the ocean in one go
2024-07-26 16:41:55 <@davide:cavalca.name> oh ok, I'd assumed there was some fundamental technical issue making it hard
2024-07-26 16:42:03 <@sgallagh:fedora.im> Michel Lind 🎩: Yes, at the branch point of CS10, we'd have to auto-create `epel11` branches for content in EPEL 11.
2024-07-26 16:42:15 <@davide:cavalca.name> if it's just a matter of work I can probably throw some resources on it, if we can scope what actually needs doing
2024-07-26 16:42:20 <@davide:cavalca.name> (we don't need to do that here/now)
2024-07-26 16:42:22 <@conan_kudo:matrix.org> the thing that makes it hard for EPEL is that we can't read RHEL repos
2024-07-26 16:42:25 <@conan_kudo:matrix.org> for ELN, that's not a problem
2024-07-26 16:42:40 <@conan_kudo:matrix.org> but EPEL 10 will be different since that's CentOS Stream 10
2024-07-26 16:42:44 <@salimma:fedora.im> well, but EPEL is already slated to track CS until RHEL is released
2024-07-26 16:42:51 <@salimma:fedora.im> yeah. so that's not an issue for bring up anyway
2024-07-26 16:42:59 <@conan_kudo:matrix.org> well, it's why it wasn't fixed before
2024-07-26 16:44:33 <@davide:cavalca.name> one thing I like about this proposal is that is makes it clearer what ELN Extras is, which will hopefully help draw contributors from the EPEL pool
2024-07-26 16:44:43 <@davide:cavalca.name> one thing I like about this proposal is that is makes it clearer what ELN Extras is, which will hopefully help draw contributors from the EPEL maintainers pool
2024-07-26 16:45:07 <@carlwgeorge:matrix.org> i can imagine this working out to a few distinct phases:
2024-07-26 16:45:07 <@carlwgeorge:matrix.org> - phase 2: mass branching from rawhide to epel11 for a planned set of content, at the same time as the c11s branching.  no more automatic rebuilds, manual builds using c11s as a buildroot.  added in bodhi, with rawhide-style automatic creation of updates.
2024-07-26 16:45:07 <@carlwgeorge:matrix.org> - phase 3: official epel launch, disabling automatic creation of bodhi updates, enabling epel-testing repo
2024-07-26 16:45:07 <@carlwgeorge:matrix.org> - phase 1: no branches, no commits, just automatic rebuilds of rawhide packages using eln as a buildroot.  basically eln extras rebranded as epel11, implemented with epel11 koji tags.
2024-07-26 16:45:07 <@carlwgeorge:matrix.org> 
2024-07-26 16:45:43 <@davide:cavalca.name> Carl George: would this generally make your life easier or harder wrt the bringup of EPEL 11 ?
2024-07-26 16:45:43 <@sgallagh:fedora.im> Carl George: Exactly what I was thinking, yeah
2024-07-26 16:45:44 <@conan_kudo:matrix.org> as things currently work in Fedora, that's probably a workable way to do it
2024-07-26 16:47:00 <@carlwgeorge:matrix.org> ask me again after we have epel10 up and running, but i'm leaning towards slightly harder but for nice benefits
2024-07-26 16:47:38 <@carlwgeorge:matrix.org> at a minimum it's shifting work around to do some things easier, and just like with epel10 will require adjustments along the way as we break expectations
2024-07-26 16:47:42 <@davide:cavalca.name> the point I was getting at was, if there's things we _could_ be doing that would make life easier down the road, it might be worth thinking about that now, even if it means a bit more upfront work
2024-07-26 16:48:06 <@davide:cavalca.name> we're already reshuffling things, so it's an opportunity to tack on improvements if that's feasible
2024-07-26 16:50:14 <@sgallagh:fedora.im> The ELN team will help with getting the Content Resolver and auto-rebuild stuff working and help with writing SOPs for how various things work (like when we need to bump the buildroot number and how to do the major release branch)
2024-07-26 16:50:56 <@sgallagh:fedora.im> But the resources to maintain things may need some shuffling (or bartering) as we go along.
2024-07-26 16:52:03 <@carlwgeorge:matrix.org> sounds like after the epel10 hackfest at flock this year, next year we'll have an epel11 hackfest
2024-07-26 16:52:28 <@yselkowitz:fedora.im> I really hope it doesn't take another year...
2024-07-26 16:52:37 <@sgallagh:fedora.im> Carl George: I intend to try and cover some/much of this at the EPEL 10 hackfest, if I can :)
2024-07-26 16:53:03 <@sgallagh:fedora.im> At the very least, let's break out some time to figure out what we know vs. what needs investigation
2024-07-26 16:54:34 <@carlwgeorge:matrix.org> honestly i won't have brain cycles to spare for planning epel11 for a while.  everywhere i turn, i find something else that epel10's minor versions will break.  so this is very much a can i need to kick down the road a bit.
2024-07-26 16:55:04 <@carlwgeorge:matrix.org> we don't have to wait a full year to start, but i definitely don't have time for it in the next few months
2024-07-26 16:55:16 <@davide:cavalca.name> Stephen Gallagher: what I think could be useful is if you can break down the work/scope involved here, like Carl did with that EPEL 10 hackmd
2024-07-26 16:55:29 <@davide:cavalca.name> and then we can look if there's parts of it where folks outside Red Hat can pitch in
2024-07-26 16:55:48 <@carlwgeorge:matrix.org> said hackmd for anyone curious https://hackmd.io/@carlwgeorge/S1r2tzZsp
2024-07-26 16:56:24 <@salimma:fedora.im> even a year is still ample time before c11s branches :)
2024-07-26 16:57:07 <@salimma:fedora.im> on pitching in -- yeah, this is important enough, anything we can help with we can put into our H1 roadmap for next year
2024-07-26 16:57:43 <@yselkowitz:fedora.im> we really don't want ELN Extras hobbling along in its current (very broken) state for another year
2024-07-26 16:58:01 <@davide:cavalca.name> yeah, I expect I'll be busy with c10s for the next few months, but that also makes it somewhat easy to justify doing work to make things better/easier down the road as people can feel the pain close by
2024-07-26 16:58:18 <@sgallagh:fedora.im> I have to ask: is there any value to continuing to rebuild the ELN Extras content *right now*?
2024-07-26 16:58:45 <@sgallagh:fedora.im> Because if it's not helpful to EPEL 10 and EPEL 11 won't be useful for years... I'd kind of like to just drop it.
2024-07-26 16:59:02 <@davide:cavalca.name> I'd really rather we didn't until we have the new thing
2024-07-26 16:59:13 <@yselkowitz:fedora.im> imo not in its current state
2024-07-26 16:59:33 <@davide:cavalca.name> if we stop rebuilding now it'll break the CI on my end, among other things
2024-07-26 16:59:46 <@sgallagh:fedora.im> From a resourcing perspective, the choices are "drop it" vs. "let it keep going and ignore it"
2024-07-26 16:59:46 <@davide:cavalca.name> it's also not generally a great look to drop something on the floor without a replacement
2024-07-26 16:59:55 <@salimma:fedora.im> note... we don't need all the extras workload, right?
2024-07-26 17:00:13 <@tdawson:fedora.im> Although my stuff doesn't show up correctly in Content Resolver, I still like having the automatic rebuilds so I can see if things build or not.
2024-07-26 17:00:32 <@salimma:fedora.im> if Davide and I help out with the workloads we actually care about, and we ask for people to sign up to maintain the others and drop them otherwise, would that be enough to keep it limping along for a bit?
2024-07-26 17:00:48 <@yselkowitz:fedora.im> not necessarily
2024-07-26 17:00:52 <@davide:cavalca.name> for Meta specifically, we need the Meta workload and the Hyperscale one, but I would not be surprised to find out we're accidentally depending on something that's pulled by another workload
2024-07-26 17:00:57 <@carlwgeorge:matrix.org> yeah at a minimum the "keep the spec file ready" is a solid benefit.  if you don't care about the involved packages you can ignore, but people that care find it useful.
2024-07-26 17:01:08 <@sgallagh:fedora.im> Michel Lind 🎩: If you can volunteer to shepherd that, I think that could be "good enough" for now.
2024-07-26 17:01:19 <@salimma:fedora.im> Davide Cavalca: sure, but we can readd missing stuff if the workload we borrowed them from get decommissioned
2024-07-26 17:01:27 <@yselkowitz:fedora.im> one of the issues right now is that the filtering I'm using to keep ELN itself from spiking all the time is interfering with Extras
2024-07-26 17:01:37 <@davide:cavalca.name> and yeah, we can definitely help maintain our own (which, incidentally, I need to go and fix :p )
2024-07-26 17:01:38 <@salimma:fedora.im> so ... something like, "if it breaks for a week, without anyone saying they'll fix it, drop it"
2024-07-26 17:01:46 <@sgallagh:fedora.im> yselkowitz: Can you expand on that, please?
2024-07-26 17:02:35 <@sgallagh:fedora.im> (We're a little over time, but no one has this room after us, so if you can spare the time to continue, I think this is worthwhile)
2024-07-26 17:02:46 <@yselkowitz:fedora.im> because of the way CR is architected, and because we're using the same instance for both ELN and ELN Extras, they both use the same "repository" definition
2024-07-26 17:03:41 <@yselkowitz:fedora.im> as long as I've been involved with ELN, there have been certain culprit packages which keep creeping back into the ELN deptree, which can end up pulling in ~1k+ more dependencies
2024-07-26 17:04:07 <@yselkowitz:fedora.im> which CR then thinks are legitimately part of ELN, and tries rebuilding them, etc.
2024-07-26 17:04:20 <@tdawson:fedora.im> Would having a seperate CR instance for extras help?
2024-07-26 17:04:51 <@yselkowitz:fedora.im> so I instituted some filtering to keep those culprits out of ELN, and it has helped tremendously
2024-07-26 17:05:24 <@yselkowitz:fedora.im> but because of the single repo definition, it has interfered with ELN Extras (where some of those properly belong)
2024-07-26 17:05:31 <@salimma:fedora.im> this is something we'll need for the move to epel11 anyway right
2024-07-26 17:05:36 <@salimma:fedora.im> worth prioritizing
2024-07-26 17:05:41 <@sgallagh:fedora.im> Right, that's why I think we really need to switch tags/targets for Extras.
2024-07-26 17:05:54 <@yselkowitz:fedora.im> the next step was going to be to get CR to support separate repos for "addons" such as ELN Extras, but my first attempt didn't work
2024-07-26 17:05:57 <@sgallagh:fedora.im> But that's a project of its own.
2024-07-26 17:07:01 <@yselkowitz:fedora.im> then these discussions started, so I didn't want to spend a bunch of time "fixing" something that was going away anyway
2024-07-26 17:07:57 <@sgallagh:fedora.im> Yeah, I think that if we get the tags changed, that should work around those problems in CR
2024-07-26 17:08:06 <@yselkowitz:fedora.im> I'm not sure how CR will work in a separate instance like that either
2024-07-26 17:08:21 <@sgallagh:fedora.im> What do you mean by "separate instance"?
2024-07-26 17:08:32 <@tdawson:fedora.im> I mean a whole different machine.
2024-07-26 17:09:33 <@carlwgeorge:matrix.org> how would cr fit into this brave new epel11 world?  is the idea we'd still use it to define the content set?  is it necessary?  (forgive me if these are dumb questions, i don't fully grasp all the particulars of how things fit together currently)
2024-07-26 17:09:35 <@sgallagh:fedora.im> Well, we can simulate how things would work by running Content Resolver against CS 9 and EPEL 9, no? And see what issues came up?
2024-07-26 17:10:12 <@sgallagh:fedora.im> Carl George: Content Resolver lets us figure out the dependencies that need to be added in addition to the package you want to include.
2024-07-26 17:10:45 <@yselkowitz:fedora.im> I'm not sure that we have the resources on the current CR machine to add c9s back much less epel9
2024-07-26 17:10:50 <@sgallagh:fedora.im> Carl George: So if we want to add e.g. LibreOffice to EPEL 11, you add a workload and then you'll find out which other packages you'll need to maintain for it.
2024-07-26 17:11:05 <@sgallagh:fedora.im> yselkowitz: I'm talking about doing this on a private machine for testing one time
2024-07-26 17:11:10 <@sgallagh:fedora.im> Not continuous hosting
2024-07-26 17:12:22 <@yselkowitz:fedora.im> I still think CR will require some code work for this to work properly, but we need to know what model we're trying to support before we can figure out what to change
2024-07-26 17:12:28 <@carlwgeorge:matrix.org> right, and that makes sense to me for eln itself where the goal is to reduce the content set for rhel, but for epel11 would it be easier to just not have cr?  a maintainer could just add the packages they care about, and if they find out later that they're missing dependencies (either manually or by installability ci) they can add those packages to the content set too.
2024-07-26 17:13:00 <@carlwgeorge:matrix.org> basically eln and eln-extras/epel have opposing goals: reduce the content set or increase the content set
2024-07-26 17:13:10 <@yselkowitz:fedora.im> but doesn't EBS depends on CR's results?
2024-07-26 17:13:31 <@tdawson:fedora.im> Just so we know, we're over time ... sorry, my chair hat (which I'm not wearing here) let me know that.
2024-07-26 17:13:39 <@carlwgeorge:matrix.org> what is ebs in this context?
2024-07-26 17:13:48 <@yselkowitz:fedora.im> ELNBuildSync
2024-07-26 17:14:04 <@yselkowitz:fedora.im> it's what triggers builds of ELN/ELN Extras
2024-07-26 17:14:30 <@sgallagh:fedora.im> yselkowitz: I'm lost. Why are we talking about doing rebuilds just now?
2024-07-26 17:15:17 <@salimma:fedora.im> this might make sense, yeah
2024-07-26 17:15:36 <@salimma:fedora.im> as an EPEL maintainer, right now one pain point is it's not easy to see which packages get automatically pulled in by CR
2024-07-26 17:15:49 <@conan_kudo:matrix.org> content resolver is primarily useful to identify the chain to branch
2024-07-26 17:16:28 <@conan_kudo:matrix.org> this is essentially what I did when I helped with EPEL9 in the beginning, though I used OBS to do it
2024-07-26 17:16:47 <@conan_kudo:matrix.org> I cloned every package I cared about to OBS, then waited for the dependency errors, and filled them in until it worked
2024-07-26 17:16:53 <@salimma:fedora.im> can CR be run on a laptop? that would help :)
2024-07-26 17:16:55 <@conan_kudo:matrix.org> then worked backwards and built it again in EPEL
2024-07-26 17:17:07 <@sgallagh:fedora.im> Content Resolver's list is currently the only input to the autorebuild tool
2024-07-26 17:17:17 <@yselkowitz:fedora.im> depends on how many packages you're dealing with
2024-07-26 17:17:21 <@conan_kudo:matrix.org> now that sucks because you're doing things multiple times, but at least I don't have to be a human dependency resolver
2024-07-26 17:17:21 <@salimma:fedora.im> so basically you run it for something you want, and add the resulting set manually to your workload definition
2024-07-26 17:17:41 <@salimma:fedora.im> I use ebranch for my EPEL 9 bringup, but would not mind reusing another tool
2024-07-26 17:18:23 <@sgallagh:fedora.im> Michel Lind 🎩: https://github.com/minimization/content-resolver
2024-07-26 17:18:35 <@sgallagh:fedora.im> Help yourself :)
2024-07-26 17:18:49 <@salimma:fedora.im> will do :)
2024-07-26 17:19:43 <@carlwgeorge:matrix.org> i'm a bit lost.  current cr is low on resources, so adding more stuff is not appealing, but we have to add stuff to have automatic rebuilds at all?
2024-07-26 17:20:05 <@sgallagh:fedora.im> Carl George: Adding a whole new Distribution to scan would be unappealing.
2024-07-26 17:20:50 <@sgallagh:fedora.im> (My suggestion of using CS9/EPEL9 as a stand-in for what we want to do with CS11/EPEL11 was not meant to be part of the official CR instance, just a testing environment)
2024-07-26 17:21:26 <@sgallagh:fedora.im> But we have to upgrade the CR hosting sometime soon anyway. It's old and slow and we can probably do FAR better on a modern virtual host than the ancient one it's living on now
2024-07-26 17:22:50 <@yselkowitz:fedora.im> anyway, we're coming up on 90 minutes, what are the next steps?
2024-07-26 17:22:59 <@carlwgeorge:matrix.org> another wrinkle we should keep in mind: an epel maintainer adds a cr workload, it pulls in dependencies.  those dependencies automatically get an epel11 branch.  the epel maintainer has what they need in epel11 to do their work, but doesn't feel responsible for those dependencies, and the actual maintainer(s) of those dependencies don't know or care that their package is in epel11, and it goes unmaintained.
2024-07-26 17:23:18 <@yselkowitz:fedora.im> no they won't get a branch automatically
2024-07-26 17:23:36 <@sgallagh:fedora.im> Carl George: They won't get a branch until the CS11/EPEL11 branch date 
2024-07-26 17:23:39 <@yselkowitz:fedora.im> until branching time, that is
2024-07-26 17:23:50 <@carlwgeorge:matrix.org> yeah, that's what i'm talking about
2024-07-26 17:23:54 <@sgallagh:fedora.im> But well before that, CR maintains a list of who it thinks owns what.
2024-07-26 17:24:05 <@sgallagh:fedora.im> Based on the workload pulling it in
2024-07-26 17:24:12 <@carlwgeorge:matrix.org> the concern is for phase 2 and 3 of the phases i laid out
2024-07-26 17:24:46 <@sgallagh:fedora.im> Sure, what I think we need to do is have a checkpoint a month before branching where we require people to confirm their workloads and deps or they get dropped and not branched.
2024-07-26 17:25:02 <@carlwgeorge:matrix.org> on the eln side, i know there are conversations like "ok this package is in the content set, we have to find a rhel maintainer for it".  what does that look like on the epel side?  people will assume the fedora maintainer is it, but they may have no clue it's in epel now without their consent.
2024-07-26 17:25:52 <@sgallagh:fedora.im> Of course, this is touching on a feature I've wanted in CR for a long time, which is the ability to add a "test" workload that would show you what would change if a workload was added.
2024-07-26 17:25:55 <@conan_kudo:matrix.org> I would expect absent another maintainer stepping up, the workload maintainer would be assigned all those for the epel11 branch.
2024-07-26 17:26:19 <@conan_kudo:matrix.org> to some degree, I kind of expect them to sign up for enabling those things
2024-07-26 17:26:20 <@sgallagh:fedora.im> Conan Kudo: Right, the workload maintainer that pulls the package in gets to own it in EPEL.
2024-07-26 17:26:23 <@carlwgeorge:matrix.org> right, and that's something that has to intentionally happen
2024-07-26 17:26:33 <@sgallagh:fedora.im> If they don't want to do it, they need to find someone else to explicitly add it to a workload
2024-07-26 17:26:40 <@conan_kudo:matrix.org> yes
2024-07-26 17:26:42 <@carlwgeorge:matrix.org> if we aren't intentional about it, we'll just have unmaintained epel packages
2024-07-26 17:26:44 <@sgallagh:fedora.im> Or their workload doesn't branch.
2024-07-26 17:26:49 <@conan_kudo:matrix.org> workload documents are not really workloads: they are maintainer documents
2024-07-26 17:27:00 <@sgallagh:fedora.im> Carl George: Which has never happened before... :)
2024-07-26 17:27:00 <@conan_kudo:matrix.org> we probably should stop calling them workloads
2024-07-26 17:27:03 <@conan_kudo:matrix.org> because they aren't
2024-07-26 17:27:13 <@conan_kudo:matrix.org> they stopped being workloads a very long time ago
2024-07-26 17:27:17 <@sgallagh:fedora.im> That's a whole other discussion :)
2024-07-26 17:27:19 <@yselkowitz:fedora.im> ?
2024-07-26 17:27:59 <@sgallagh:fedora.im> The next steps are that I'll work on that requirements document and come back with something after Flock
2024-07-26 17:28:13 <@sgallagh:fedora.im> (Hopefully getting input on it *at* Flock)
2024-07-26 17:28:24 <@conan_kudo:matrix.org> I will not be able to make it to your talk
2024-07-26 17:28:33 <@conan_kudo:matrix.org> as mine is in paralle
2024-07-26 17:28:35 <@conan_kudo:matrix.org> as mine is in parallel
2024-07-26 17:29:37 <@sgallagh:fedora.im> Conan Kudo: Understood, but I'll try to take notes and revisit at the EPEL workshop as well
2024-07-26 17:29:49 <@sgallagh:fedora.im> Alright, we're out of time. Final remarks?
2024-07-26 17:29:53 <@conan_kudo:matrix.org> oh hay that's fixed now and it doesn't conflict anymore
2024-07-26 17:30:04 <@conan_kudo:matrix.org> they moved the talks around
2024-07-26 17:30:13 <@sgallagh:fedora.im> !action sgallagh to start preparing a work requirement document for ELN Extras -> EPEL 11. Will gather feedback at Flock.
2024-07-26 17:31:41 <@conan_kudo:matrix.org> hey sorry about earlier... I'm just really frustrated over a bunch of things going on right now
2024-07-26 17:32:21 <@sgallagh:fedora.im> I get it.
2024-07-26 17:32:29 <@sgallagh:fedora.im> !endmeeting