<@sgallagh:fedora.im>
16:00:10
!startmeeting ELN SIG (2024-07-26)
<@meetbot:fedora.im>
16:00:12
Meeting started at 2024-07-26 16:00:10 UTC
<@meetbot:fedora.im>
16:00:12
The Meeting name is 'ELN SIG (2024-07-26)'
<@sgallagh:fedora.im>
16:00:15
!meetingname eln
<@meetbot:fedora.im>
16:00:15
The Meeting Name is now eln
<@sgallagh:fedora.im>
16:00:28
!topic Init Process
<@sgallagh:fedora.im>
16:00:29
!hi
<@zodbot:fedora.im>
16:00:30
Stephen Gallagher (sgallagh) - he / him / his
<@tdawson:fedora.im>
16:00:37
!hi
<@zodbot:fedora.im>
16:00:38
Troy Dawson (tdawson)
<@salimma:fedora.im>
16:01:47
!hi
<@zodbot:fedora.im>
16:01:48
Michel Lind (salimma) - he / him / his
<@yselkowitz:fedora.im>
16:02:12
!hi
<@zodbot:fedora.im>
16:02:14
Yaakov Selkowitz (yselkowitz)
<@sgallagh:fedora.im>
16:03:11
I pinged Carl George to join as well, since we'll be discussing EPEL 11 a bit
<@carlwgeorge:matrix.org>
16:03:38
!hi
<@zodbot:fedora.im>
16:03:39
Carl George (carlwgeorge) - he / him / his
<@sgallagh:fedora.im>
16:03:47
!topic Agenda
<@sgallagh:fedora.im>
16:03:58
!info Agenda Item: Sunsetting ELN Extras
<@sgallagh:fedora.im>
16:04:12
I'd normally ask for other topics, but I expect this one to take the full time.
<@sgallagh:fedora.im>
16:04:42
!topic Sunsetting ELN Extras
<@sgallagh:fedora.im>
16:05:19
So, first I'll try to summarize the reasons why we're going to drop ELN Extras, then we can discuss the "how" of it.
<@nhanlon:beeper.com>
16:05:50
!hi
<@zodbot:fedora.im>
16:05:52
Neil Hanlon (neil) - he / him / his
<@sgallagh:fedora.im>
16:06:21
The original goal of ELN Extras was to provide a way to bootstrap EPEL 10 early, so that we could have it ready to go when CentOS Stream 10 launched.
<@davide:cavalca.name>
16:07:02
!hi
<@zodbot:fedora.im>
16:07:03
Davide Cavalca (dcavalca) - he / him / his
<@sgallagh:fedora.im>
16:07:05
This failed, partly due to resource constraints, because we didn't branch ELN Extras to EPEL 10 at the same time we forked CentOS Stream 10 from ELN/Rawhide
<@conan_kudo:matrix.org>
16:07:10
!hi
<@zodbot:fedora.im>
16:07:12
Neal Gompa (ngompa) - he / him / his
<@carlwgeorge:matrix.org>
16:07:59
was that the original goal? i only remember talk about it being used as a way to ensure spec files were ready macro wise, not for any actual bootstrapping.
<@salimma:fedora.im>
16:07:59
did it really fail though? we can still do those branches as soon as EPEL 10 is created in a couple of weeks or so
<@sgallagh:fedora.im>
16:08:02
So, ELN Extras has continued forwards, tracking Rawhide and is now in a state that we cannot comfortably assert is compatible with EL 10
<@conan_kudo:matrix.org>
16:08:06
Carl George: yes it was
<@conan_kudo:matrix.org>
16:08:13
that was a big part of why ELN Extras was created
<@sgallagh:fedora.im>
16:08:17
Carl George: Yes, that was our original pitch for it
<@salimma:fedora.im>
16:08:45
any downside to keeping it this way and retrying for 11?
<@davide:cavalca.name>
16:08:52
the other benefit of ELN Extras is ensuring that packages outside of core RHEL can be tested and used against ELN
<@conan_kudo:matrix.org>
16:08:53
not particularly
<@salimma:fedora.im>
16:08:56
after all, it still helps packages get ready macro wise as Carl noted
<@davide:cavalca.name>
16:09:19
with my Meta hat on, this is primarily what we use it for, and it's been pretty much instrumental in making it possible for us to put together an ELN testbed
<@salimma:fedora.im>
16:09:24
it would certainly make our ELN test fleet at Meta way harder to use if we don't have this
<@salimma:fedora.im>
16:09:26
ah, jinks
<@davide:cavalca.name>
16:09:39
as there are far too many packages not in the core distros that are needed in the real world for a deployment
<@conan_kudo:matrix.org>
16:09:41
indeed, one of the hopes here is that we'd have less trouble building these things in EPEL if they're being continually built and reported on for RHEL's content sizing
<@davide:cavalca.name>
16:09:58
so, while I'm not personally married to the current implemention, we do need something like this going forward
<@salimma:fedora.im>
16:10:17
and that package set is growing over time
<@conan_kudo:matrix.org>
16:10:22
Right. And we've needed it in Fedora KDE to ensure things aren't terribly broken for the stack too.
<@tdawson:fedora.im>
16:10:29
But one of the problems is that it ISN'T working. Look at KDE, qt6, and ffmpeg ... none of those workloads work dues to really strange reason.
<@sgallagh:fedora.im>
16:10:43
I'm not finished listing the problems with it, but to jump ahead slightly, we want to replace it with a much earlier EPEL 11 project. I'll get to that in a few minutes.
<@salimma:fedora.im>
16:10:52
ah
<@conan_kudo:matrix.org>
16:10:56
one of the larger problems with ELN is that we don't do composes with images and run tests
<@salimma:fedora.im>
16:11:02
Stephen Gallagher: you could have led with that :D
<@davide:cavalca.name>
16:11:04
so, part of the problem with failing workloads is that there is no monitoring/alerting (that I know of at least)
<@conan_kudo:matrix.org>
16:11:09
this affects both ELN core and ELN extras
<@conan_kudo:matrix.org>
16:11:26
in fact, ELN is probably the least tested part of Fedora
<@davide:cavalca.name>
16:11:42
I generally spot issues with the Meta and/or workloads when they crop up internally, but if we had something that say emailed you whenever the workload failed then I suspect people would be more quick to act and fix them
<@davide:cavalca.name>
16:11:50
I generally spot issues with the Meta and/or HS workloads when they crop up internally, but if we had something that say emailed you whenever the workload failed then I suspect people would be more quick to act and fix them
<@conan_kudo:matrix.org>
16:12:09
one of the things I haven't been particularly happy about is treating ELN as a non-deliverable
<@salimma:fedora.im>
16:12:15
yeah, ISTR I've been fixing quite a few ELN build failures over time so... do issues really remain unfixed, or are we talking about other kinds of failures
<@yselkowitz:fedora.im>
16:12:29
I look at CR 6 days a week, if a workload starts failing I usually notice pretty quickly
<@davide:cavalca.name>
16:12:29
ok, let's let Stephen Gallagher finish and then we can continue the pile up :)
<@sgallagh:fedora.im>
16:13:53
Additional problems with ELN Extras is that it complicates our rebuilds and compose, at least in part because I didn't fully consider the ramifications of doing it all in the `eln` Koji tag.
<@sgallagh:fedora.im>
16:14:36
We have hack upon hack piled up in both ELNBuildSync and the compose configuration to pull in ELN Extras content.
<@salimma:fedora.im>
16:15:23
ah. so yeah making it more EPEL like is probably better
<@conan_kudo:matrix.org>
16:15:31
well maybe
<@yselkowitz:fedora.im>
16:15:38
and it makes it more difficult to filter unwanted packages from ELN in CR
<@conan_kudo:matrix.org>
16:16:07
some of this sounds like stuffing ELN in Fedora Koji was a mistake
<@yselkowitz:fedora.im>
16:16:27
no, putting ELN and ELN Extras together though was
<@sgallagh:fedora.im>
16:16:40
I was typing the same sentence, so ^^
<@conan_kudo:matrix.org>
16:16:52
but that's also solvable by splitting the tag
<@conan_kudo:matrix.org>
16:17:09
to put it bluntly, I don't want the EPEL workflow for ELN Extras packages
<@sgallagh:fedora.im>
16:17:12
Yes, which is one of the things I was going to propose.
<@sgallagh:fedora.im>
16:17:41
(splitting the tag)
<@sgallagh:fedora.im>
16:18:14
But by "I don't want the EPEL workflow" do you mean you don't want packagers to have to do their own commits and builds for ELN Extras?
<@conan_kudo:matrix.org>
16:18:47
yes
<@sgallagh:fedora.im>
16:20:29
One of the additional problems we've had with ELN Extras is that the automatic nature of rebuilds has led some people to assume that it means that "someone else" (Read: the people in this meeting) are responsible for fixing things when they go wrong.
<@conan_kudo:matrix.org>
16:20:50
do we file bugs when things fail?
<@yselkowitz:fedora.im>
16:21:00
which is pretty much the same for ELN itself fwiw...
<@conan_kudo:matrix.org>
16:21:17
because I know I am not getting bug reports for ELN failures
<@yselkowitz:fedora.im>
16:21:23
but every time I think I'm going to start having time to look at ELN Extras, something else comes up
<@davide:cavalca.name>
16:21:28
otoh, manual builds would be a lot of toil due to how much ELN changes, and doesn't seem especially practical
<@sgallagh:fedora.im>
16:21:41
Conan Kudo: We've been doing bugs for things in ELN proper
<@sgallagh:fedora.im>
16:21:51
But we literally do not have the cycles to handle ELN Extras
<@salimma:fedora.im>
16:22:05
right. I want it to be EPEL like but we don't want the package maintainers to have to do manual builds, the whole point is because they often don't bringup is slower :)
<@conan_kudo:matrix.org>
16:22:12
from my point of view, "ELN proper" is both ELN core and ELN extras
<@yselkowitz:fedora.im>
16:22:21
for us it's not
<@salimma:fedora.im>
16:22:34
there are two viewpoints. for prepping for the next EL, it's only core
<@salimma:fedora.im>
16:22:46
from the perspective of those who care about EPEL, both need to work
<@sgallagh:fedora.im>
16:22:55
ELN "core" is what Red Hat is paying us to do. ELN Extras is and always has been a volunteer effort
<@conan_kudo:matrix.org>
16:23:02
so here's the rub: if you don't consider it that way, everything for the people that contribute to supporting the EL ecosystem will always be broken
<@sgallagh:fedora.im>
16:23:07
I hate to break it down like that, but it's relevant here
<@conan_kudo:matrix.org>
16:23:21
and I hate to say it, but people don't use EL unless the extras are there
<@davide:cavalca.name>
16:23:40
that's fair, but in the end ELN Extras allows folks to actually test ELN in the real world and fix bugs/provide feedback/etc
<@sgallagh:fedora.im>
16:23:42
Conan Kudo: Which is why we now have people paid to work on EPEL :)
<@conan_kudo:matrix.org>
16:23:42
EL core continues to shrink release after release, forcing people to need more and more from extras for the same workloads
<@tdawson:fedora.im>
16:23:42
So, have YOU filed any bugs for eln extras? Because your workloads have alot of them.
<@conan_kudo:matrix.org>
16:24:07
No, I go through and try to fix them as I work through them even in rawhide
<@conan_kudo:matrix.org>
16:24:15
what's the point in filing bugs against myself?
<@conan_kudo:matrix.org>
16:24:35
I fix ELN bugs in both core and extras when I spot them
<@conan_kudo:matrix.org>
16:24:39
I don't even tell people, I just do it
<@sgallagh:fedora.im>
16:24:53
No one is saying that the content we have in ELN Extras is unnecessary or useless. Just that we don't have the capability to handle it under the ELN umbrella.
<@sgallagh:fedora.im>
16:25:18
Which is why I am trying to get to the part where we work with EPEL to get it moved under that umbrella.
<@yselkowitz:fedora.im>
16:25:29
I wouldn't mind knowing when that happens, so that I can track fixes
<@salimma:fedora.im>
16:25:38
I think we're more in agreement than this debate seems to imply... can we skip ahead to the part about this "early EPEL 11" since I'm sure that will address some concerns?
<@sgallagh:fedora.im>
16:25:40
At least in part because the EPEL brand is much more established and people have demonstrated they're willing to work on it
<@conan_kudo:matrix.org>
16:25:47
There is no resources to work on it in EPEL and the EPEL workflows are painful for a churning distribution.
<@davide:cavalca.name>
16:25:55
fwiw I also generally fix bugs when I find them, but if there's value in tracking them more explicitly we can certainly look into doing that
<@davide:cavalca.name>
16:26:09
let's wait to actually hear Stephen Gallagher 's proposal here
<@salimma:fedora.im>
16:26:21
suggestion: should we have a standard way to write changelog entries so we can easily identify ELN fixes
<@sgallagh:fedora.im>
16:26:26
Conan Kudo: You are making assumptions. I'm trying to establish a starting point for a discussion.
<@carlwgeorge:matrix.org>
16:26:31
we can have different workflows for epel at different points in time
<@conan_kudo:matrix.org>
16:27:05
Fundamentally, I don't want spurious rebuild commits being checked for every time an ELN core rebuild happens
<@conan_kudo:matrix.org>
16:27:30
I don't want to have to manually massage every little thing
<@conan_kudo:matrix.org>
16:27:37
I don't have time for that
<@yselkowitz:fedora.im>
16:27:38
that's easily avoidable with incrementing dist-tags like we do in ELN
<@sgallagh:fedora.im>
16:27:52
Conan Kudo: If you'd let us at least *start* describing things, you might see we have some ideas around that.
<@tdawson:fedora.im>
16:28:05
Conan Kudo: Please let him speak.
<@sgallagh:fedora.im>
16:29:07
OK, I'm going to start explaining the high-level ideas I've had, and I'll EOM when I've gotten them all down.
<@sgallagh:fedora.im>
16:29:51
First of all, I agree that we want to have certain behaviors maintained from ELN Extras, such as automatic rebuilds of Rawhide content up to a certain point.
<@sgallagh:fedora.im>
16:30:38
I think that ELN as a "brand" has not gained the traction it needs for us to get greater participation in ELN Extras, despite it being valuable to the eventual release of EPEL N+1.
<@sgallagh:fedora.im>
16:31:02
We also described some of the build and compose issues related to having them share a tag infrastructure.
<@sgallagh:fedora.im>
16:31:40
What I propose is that we rebrand ELN Extras as an EPEL project and set up its tags accordingly.
<@sgallagh:fedora.im>
16:32:20
We should maintain automatic Rawhide rebuilds (including mass-rebuilds) up to the point where we branch c11s (I'm going to use 11 as an example here rather than keep saying N+1).
<@sgallagh:fedora.im>
16:33:18
Prior to that branch, our dist tags will be something like `.el~epel11_<buildrootnumber>` where `<buildrootnumber>` will be kept in sync with ELN for consistency.
<@sgallagh:fedora.im>
16:33:44
When we branch, we'll rebuild with `.el11` and start treating it with the traditional EPEL workflow.
<@sgallagh:fedora.im>
16:33:49
EOM
<@salimma:fedora.im>
16:33:51
this would certainly help with all those "when is EPEL X going to be ready" too, since now it will be ready even before cXs is created right?
<@sgallagh:fedora.im>
16:34:05
(That's as far as I've gotten so far, please poke at it)
<@salimma:fedora.im>
16:34:07
EPEL 11 tracks eln now, and tracks c11s once that is branched at which point we stop auto rebuilds
<@salimma:fedora.im>
16:34:23
I like this. no more begging people to please, please branch their packages
<@conan_kudo:matrix.org>
16:34:32
There's still a problem.
<@sgallagh:fedora.im>
16:34:42
Oh, right, sorry. I forgot to mention: EPEL 11 will use ELN as its buildroot until the branch, at which point it will use CS 11
<@conan_kudo:matrix.org>
16:34:48
Nobody can use these packages easily, which makes development and testing hard.
<@salimma:fedora.im>
16:35:00
I don't understand. how?
<@salimma:fedora.im>
16:35:10
talking about the EPEL 11 side or the ELN Core side?
<@davide:cavalca.name>
16:35:10
won't these show up in a repo?
<@conan_kudo:matrix.org>
16:35:12
we don't publish images, we don't publish repos, there are no mirrors, and there is no website for it
<@tdawson:fedora.im>
16:35:22
I don't understand the last sentance "When we branch, we'll rebuild with .el11" Are both "we's" in the context the ELN SIG?
<@conan_kudo:matrix.org>
16:35:30
that is 99% of the problem with ELN contributors
<@davide:cavalca.name>
16:35:37
afair there was talk of moving ELN to the mirror network, but that seems orthogonal to the topic at hand
<@sgallagh:fedora.im>
16:35:40
Conan Kudo: All of that is both untrue and in the process of being brought in line with Fedora right now
<@davide:cavalca.name>
16:35:48
and I'd assume if this is under EPEL it _would_ be on the mirror network to begin with
<@yselkowitz:fedora.im>
16:35:50
that's not the problem we're trying to solve here
<@conan_kudo:matrix.org>
16:36:09
the problem you're trying to solve is unburdening yourselves and scaling ELN Extras by making it an EPEL project
<@salimma:fedora.im>
16:36:11
yeah, ELN Core visibility is orthogonal to this topic. and ELN Extras will be more visible as part of EPEL, no? it will be mirrored by EPEL mirrors
<@conan_kudo:matrix.org>
16:36:20
therefore it needs to have some basic usability for people to actively work on it
<@sgallagh:fedora.im>
16:36:31
Let's not sidetrack this, but ELN is being moved to the common Fedora compose infrastructure and will be moving to mirrormanager. This is happening in the next month
<@salimma:fedora.im>
16:36:38
can we discuss that later? as Stephen said that's being worked on
<@salimma:fedora.im>
16:37:00
we're 36 mins in and we should talk more about how this EPEL 11 will work
<@conan_kudo:matrix.org>
16:37:14
seems like it's pretty set with how "EPEL 11" will work
<@conan_kudo:matrix.org>
16:37:22
he seems to have figured that out
<@sgallagh:fedora.im>
16:37:31
I made a proposal, I'm asking for feedback and corrections
<@conan_kudo:matrix.org>
16:38:06
my feedback is that it needs to be tied to some kind of public visibility of ELN
<@davide:cavalca.name>
16:38:11
is the plan to keep using an `eln` branch when we need a branch for this?
<@conan_kudo:matrix.org>
16:38:20
otherwise it will also fail due to lack of people able to work on it
<@salimma:fedora.im>
16:38:23
I'm in favor (and I guess the "tracking ELN" was my only question). any package that got transitively pulled in by an ELN Extras workload will get branched, right?
<@sgallagh:fedora.im>
16:38:33
Davide Cavalca: No, and that's a good question. I think it'll need to branch to `epel11` instead.
<@salimma:fedora.im>
16:38:37
right, the other question is if the branch is eln or epel11
<@davide:cavalca.name>
16:39:02
I'm ok with that; we just then need to be careful when ELN jumps and it goes `epel11` -> `epel12`
<@yselkowitz:fedora.im>
16:39:11
it's already failing for the same reason
<@conan_kudo:matrix.org>
16:40:01
yeah, but we don't change how we approach ELN as a whole, it will stay failing
<@conan_kudo:matrix.org>
16:40:10
I'm saying your approach isn't working
<@yselkowitz:fedora.im>
16:40:29
ELN is working very well for it's primary purpose, actually
<@conan_kudo:matrix.org>
16:41:22
probably not too much
<@sgallagh:fedora.im>
16:41:25
Davide Cavalca: Someone with time to do the work
<@yselkowitz:fedora.im>
16:41:27
or ELN for that matter...
<@salimma:fedora.im>
16:41:30
Stephen Gallagher: ^ last question, on automatic branching
<@conan_kudo:matrix.org>
16:41:39
the main issue is that we need something to read and track both core and extras tags
<@carlwgeorge:matrix.org>
16:41:44
i think it's possible to revamp eln extras into early epel11 and leave eln to continue making other improvements for the overall success of eln
<@conan_kudo:matrix.org>
16:41:51
which shouldn't be hard, the existing stuff can be changed to read multiple repos together
<@carlwgeorge:matrix.org>
16:41:55
we don't have to boil the ocean in one go
<@davide:cavalca.name>
16:41:55
oh ok, I'd assumed there was some fundamental technical issue making it hard
<@sgallagh:fedora.im>
16:42:03
Michel Lind 🎩: Yes, at the branch point of CS10, we'd have to auto-create `epel11` branches for content in EPEL 11.
<@davide:cavalca.name>
16:42:15
if it's just a matter of work I can probably throw some resources on it, if we can scope what actually needs doing
<@davide:cavalca.name>
16:42:20
(we don't need to do that here/now)
<@conan_kudo:matrix.org>
16:42:22
the thing that makes it hard for EPEL is that we can't read RHEL repos
<@conan_kudo:matrix.org>
16:42:25
for ELN, that's not a problem
<@conan_kudo:matrix.org>
16:42:40
but EPEL 10 will be different since that's CentOS Stream 10
<@salimma:fedora.im>
16:42:44
well, but EPEL is already slated to track CS until RHEL is released
<@salimma:fedora.im>
16:42:51
yeah. so that's not an issue for bring up anyway
<@conan_kudo:matrix.org>
16:42:59
well, it's why it wasn't fixed before
<@davide:cavalca.name>
16:44:33
one thing I like about this proposal is that is makes it clearer what ELN Extras is, which will hopefully help draw contributors from the EPEL pool
<@davide:cavalca.name>
16:44:43
one thing I like about this proposal is that is makes it clearer what ELN Extras is, which will hopefully help draw contributors from the EPEL maintainers pool
<@carlwgeorge:matrix.org>
16:45:07
i can imagine this working out to a few distinct phases:
<@carlwgeorge:matrix.org>
16:45:07
- phase 2: mass branching from rawhide to epel11 for a planned set of content, at the same time as the c11s branching. no more automatic rebuilds, manual builds using c11s as a buildroot. added in bodhi, with rawhide-style automatic creation of updates.
<@carlwgeorge:matrix.org>
16:45:07
- phase 3: official epel launch, disabling automatic creation of bodhi updates, enabling epel-testing repo
<@carlwgeorge:matrix.org>
16:45:07
- phase 1: no branches, no commits, just automatic rebuilds of rawhide packages using eln as a buildroot. basically eln extras rebranded as epel11, implemented with epel11 koji tags.
<@carlwgeorge:matrix.org>
16:45:07
<@davide:cavalca.name>
16:45:43
Carl George: would this generally make your life easier or harder wrt the bringup of EPEL 11 ?
<@sgallagh:fedora.im>
16:45:43
Carl George: Exactly what I was thinking, yeah
<@conan_kudo:matrix.org>
16:45:44
as things currently work in Fedora, that's probably a workable way to do it
<@carlwgeorge:matrix.org>
16:47:00
ask me again after we have epel10 up and running, but i'm leaning towards slightly harder but for nice benefits
<@carlwgeorge:matrix.org>
16:47:38
at a minimum it's shifting work around to do some things easier, and just like with epel10 will require adjustments along the way as we break expectations
<@davide:cavalca.name>
16:47:42
the point I was getting at was, if there's things we _could_ be doing that would make life easier down the road, it might be worth thinking about that now, even if it means a bit more upfront work
<@davide:cavalca.name>
16:48:06
we're already reshuffling things, so it's an opportunity to tack on improvements if that's feasible
<@sgallagh:fedora.im>
16:50:14
The ELN team will help with getting the Content Resolver and auto-rebuild stuff working and help with writing SOPs for how various things work (like when we need to bump the buildroot number and how to do the major release branch)
<@sgallagh:fedora.im>
16:50:56
But the resources to maintain things may need some shuffling (or bartering) as we go along.
<@carlwgeorge:matrix.org>
16:52:03
sounds like after the epel10 hackfest at flock this year, next year we'll have an epel11 hackfest
<@yselkowitz:fedora.im>
16:52:28
I really hope it doesn't take another year...
<@sgallagh:fedora.im>
16:52:37
Carl George: I intend to try and cover some/much of this at the EPEL 10 hackfest, if I can :)
<@sgallagh:fedora.im>
16:53:03
At the very least, let's break out some time to figure out what we know vs. what needs investigation
<@carlwgeorge:matrix.org>
16:54:34
honestly i won't have brain cycles to spare for planning epel11 for a while. everywhere i turn, i find something else that epel10's minor versions will break. so this is very much a can i need to kick down the road a bit.
<@carlwgeorge:matrix.org>
16:55:04
we don't have to wait a full year to start, but i definitely don't have time for it in the next few months
<@davide:cavalca.name>
16:55:16
Stephen Gallagher: what I think could be useful is if you can break down the work/scope involved here, like Carl did with that EPEL 10 hackmd
<@davide:cavalca.name>
16:55:29
and then we can look if there's parts of it where folks outside Red Hat can pitch in
<@carlwgeorge:matrix.org>
16:55:48
said hackmd for anyone curious https://hackmd.io/@carlwgeorge/S1r2tzZsp
<@salimma:fedora.im>
16:56:24
even a year is still ample time before c11s branches :)
<@salimma:fedora.im>
16:57:07
on pitching in -- yeah, this is important enough, anything we can help with we can put into our H1 roadmap for next year
<@yselkowitz:fedora.im>
16:57:43
we really don't want ELN Extras hobbling along in its current (very broken) state for another year
<@davide:cavalca.name>
16:58:01
yeah, I expect I'll be busy with c10s for the next few months, but that also makes it somewhat easy to justify doing work to make things better/easier down the road as people can feel the pain close by
<@sgallagh:fedora.im>
16:58:18
I have to ask: is there any value to continuing to rebuild the ELN Extras content *right now*?
<@sgallagh:fedora.im>
16:58:45
Because if it's not helpful to EPEL 10 and EPEL 11 won't be useful for years... I'd kind of like to just drop it.
<@davide:cavalca.name>
16:59:02
I'd really rather we didn't until we have the new thing
<@yselkowitz:fedora.im>
16:59:13
imo not in its current state
<@davide:cavalca.name>
16:59:33
if we stop rebuilding now it'll break the CI on my end, among other things
<@sgallagh:fedora.im>
16:59:46
From a resourcing perspective, the choices are "drop it" vs. "let it keep going and ignore it"
<@davide:cavalca.name>
16:59:46
it's also not generally a great look to drop something on the floor without a replacement
<@salimma:fedora.im>
16:59:55
note... we don't need all the extras workload, right?
<@tdawson:fedora.im>
17:00:13
Although my stuff doesn't show up correctly in Content Resolver, I still like having the automatic rebuilds so I can see if things build or not.
<@salimma:fedora.im>
17:00:32
if Davide and I help out with the workloads we actually care about, and we ask for people to sign up to maintain the others and drop them otherwise, would that be enough to keep it limping along for a bit?
<@yselkowitz:fedora.im>
17:00:48
not necessarily
<@davide:cavalca.name>
17:00:52
for Meta specifically, we need the Meta workload and the Hyperscale one, but I would not be surprised to find out we're accidentally depending on something that's pulled by another workload
<@carlwgeorge:matrix.org>
17:00:57
yeah at a minimum the "keep the spec file ready" is a solid benefit. if you don't care about the involved packages you can ignore, but people that care find it useful.
<@sgallagh:fedora.im>
17:01:08
Michel Lind 🎩: If you can volunteer to shepherd that, I think that could be "good enough" for now.
<@salimma:fedora.im>
17:01:19
Davide Cavalca: sure, but we can readd missing stuff if the workload we borrowed them from get decommissioned
<@yselkowitz:fedora.im>
17:01:27
one of the issues right now is that the filtering I'm using to keep ELN itself from spiking all the time is interfering with Extras
<@davide:cavalca.name>
17:01:37
and yeah, we can definitely help maintain our own (which, incidentally, I need to go and fix :p )
<@salimma:fedora.im>
17:01:38
so ... something like, "if it breaks for a week, without anyone saying they'll fix it, drop it"
<@sgallagh:fedora.im>
17:01:46
yselkowitz: Can you expand on that, please?
<@sgallagh:fedora.im>
17:02:35
(We're a little over time, but no one has this room after us, so if you can spare the time to continue, I think this is worthwhile)
<@yselkowitz:fedora.im>
17:02:46
because of the way CR is architected, and because we're using the same instance for both ELN and ELN Extras, they both use the same "repository" definition
<@yselkowitz:fedora.im>
17:03:41
as long as I've been involved with ELN, there have been certain culprit packages which keep creeping back into the ELN deptree, which can end up pulling in ~1k+ more dependencies
<@yselkowitz:fedora.im>
17:04:07
which CR then thinks are legitimately part of ELN, and tries rebuilding them, etc.
<@tdawson:fedora.im>
17:04:20
Would having a seperate CR instance for extras help?
<@yselkowitz:fedora.im>
17:04:51
so I instituted some filtering to keep those culprits out of ELN, and it has helped tremendously
<@yselkowitz:fedora.im>
17:05:24
but because of the single repo definition, it has interfered with ELN Extras (where some of those properly belong)
<@salimma:fedora.im>
17:05:31
this is something we'll need for the move to epel11 anyway right
<@salimma:fedora.im>
17:05:36
worth prioritizing
<@sgallagh:fedora.im>
17:05:41
Right, that's why I think we really need to switch tags/targets for Extras.
<@yselkowitz:fedora.im>
17:05:54
the next step was going to be to get CR to support separate repos for "addons" such as ELN Extras, but my first attempt didn't work
<@sgallagh:fedora.im>
17:05:57
But that's a project of its own.
<@yselkowitz:fedora.im>
17:07:01
then these discussions started, so I didn't want to spend a bunch of time "fixing" something that was going away anyway
<@sgallagh:fedora.im>
17:07:57
Yeah, I think that if we get the tags changed, that should work around those problems in CR
<@yselkowitz:fedora.im>
17:08:06
I'm not sure how CR will work in a separate instance like that either
<@sgallagh:fedora.im>
17:08:21
What do you mean by "separate instance"?
<@tdawson:fedora.im>
17:08:32
I mean a whole different machine.
<@carlwgeorge:matrix.org>
17:09:33
how would cr fit into this brave new epel11 world? is the idea we'd still use it to define the content set? is it necessary? (forgive me if these are dumb questions, i don't fully grasp all the particulars of how things fit together currently)
<@sgallagh:fedora.im>
17:09:35
Well, we can simulate how things would work by running Content Resolver against CS 9 and EPEL 9, no? And see what issues came up?
<@sgallagh:fedora.im>
17:10:12
Carl George: Content Resolver lets us figure out the dependencies that need to be added in addition to the package you want to include.
<@yselkowitz:fedora.im>
17:10:45
I'm not sure that we have the resources on the current CR machine to add c9s back much less epel9
<@sgallagh:fedora.im>
17:10:50
Carl George: So if we want to add e.g. LibreOffice to EPEL 11, you add a workload and then you'll find out which other packages you'll need to maintain for it.
<@sgallagh:fedora.im>
17:11:05
yselkowitz: I'm talking about doing this on a private machine for testing one time
<@sgallagh:fedora.im>
17:11:10
Not continuous hosting
<@yselkowitz:fedora.im>
17:12:22
I still think CR will require some code work for this to work properly, but we need to know what model we're trying to support before we can figure out what to change
<@carlwgeorge:matrix.org>
17:12:28
right, and that makes sense to me for eln itself where the goal is to reduce the content set for rhel, but for epel11 would it be easier to just not have cr? a maintainer could just add the packages they care about, and if they find out later that they're missing dependencies (either manually or by installability ci) they can add those packages to the content set too.
<@carlwgeorge:matrix.org>
17:13:00
basically eln and eln-extras/epel have opposing goals: reduce the content set or increase the content set
<@yselkowitz:fedora.im>
17:13:10
but doesn't EBS depends on CR's results?
<@tdawson:fedora.im>
17:13:31
Just so we know, we're over time ... sorry, my chair hat (which I'm not wearing here) let me know that.
<@carlwgeorge:matrix.org>
17:13:39
what is ebs in this context?
<@yselkowitz:fedora.im>
17:13:48
ELNBuildSync
<@yselkowitz:fedora.im>
17:14:04
it's what triggers builds of ELN/ELN Extras
<@sgallagh:fedora.im>
17:14:30
yselkowitz: I'm lost. Why are we talking about doing rebuilds just now?
<@salimma:fedora.im>
17:15:17
this might make sense, yeah
<@salimma:fedora.im>
17:15:36
as an EPEL maintainer, right now one pain point is it's not easy to see which packages get automatically pulled in by CR
<@conan_kudo:matrix.org>
17:15:49
content resolver is primarily useful to identify the chain to branch
<@conan_kudo:matrix.org>
17:16:28
this is essentially what I did when I helped with EPEL9 in the beginning, though I used OBS to do it
<@conan_kudo:matrix.org>
17:16:47
I cloned every package I cared about to OBS, then waited for the dependency errors, and filled them in until it worked
<@salimma:fedora.im>
17:16:53
can CR be run on a laptop? that would help :)
<@conan_kudo:matrix.org>
17:16:55
then worked backwards and built it again in EPEL
<@sgallagh:fedora.im>
17:17:07
Content Resolver's list is currently the only input to the autorebuild tool
<@yselkowitz:fedora.im>
17:17:17
depends on how many packages you're dealing with
<@conan_kudo:matrix.org>
17:17:21
now that sucks because you're doing things multiple times, but at least I don't have to be a human dependency resolver
<@salimma:fedora.im>
17:17:21
so basically you run it for something you want, and add the resulting set manually to your workload definition
<@salimma:fedora.im>
17:17:41
I use ebranch for my EPEL 9 bringup, but would not mind reusing another tool
<@sgallagh:fedora.im>
17:18:23
Michel Lind 🎩: https://github.com/minimization/content-resolver
<@sgallagh:fedora.im>
17:18:35
Help yourself :)
<@salimma:fedora.im>
17:18:49
will do :)
<@carlwgeorge:matrix.org>
17:19:43
i'm a bit lost. current cr is low on resources, so adding more stuff is not appealing, but we have to add stuff to have automatic rebuilds at all?
<@sgallagh:fedora.im>
17:20:05
Carl George: Adding a whole new Distribution to scan would be unappealing.
<@sgallagh:fedora.im>
17:20:50
(My suggestion of using CS9/EPEL9 as a stand-in for what we want to do with CS11/EPEL11 was not meant to be part of the official CR instance, just a testing environment)
<@sgallagh:fedora.im>
17:21:26
But we have to upgrade the CR hosting sometime soon anyway. It's old and slow and we can probably do FAR better on a modern virtual host than the ancient one it's living on now
<@yselkowitz:fedora.im>
17:22:50
anyway, we're coming up on 90 minutes, what are the next steps?
<@carlwgeorge:matrix.org>
17:22:59
another wrinkle we should keep in mind: an epel maintainer adds a cr workload, it pulls in dependencies. those dependencies automatically get an epel11 branch. the epel maintainer has what they need in epel11 to do their work, but doesn't feel responsible for those dependencies, and the actual maintainer(s) of those dependencies don't know or care that their package is in epel11, and it goes unmaintained.
<@yselkowitz:fedora.im>
17:23:18
no they won't get a branch automatically
<@sgallagh:fedora.im>
17:23:36
Carl George: They won't get a branch until the CS11/EPEL11 branch date
<@yselkowitz:fedora.im>
17:23:39
until branching time, that is
<@carlwgeorge:matrix.org>
17:23:50
yeah, that's what i'm talking about
<@sgallagh:fedora.im>
17:23:54
But well before that, CR maintains a list of who it thinks owns what.
<@sgallagh:fedora.im>
17:24:05
Based on the workload pulling it in
<@carlwgeorge:matrix.org>
17:24:12
the concern is for phase 2 and 3 of the phases i laid out
<@sgallagh:fedora.im>
17:24:46
Sure, what I think we need to do is have a checkpoint a month before branching where we require people to confirm their workloads and deps or they get dropped and not branched.
<@carlwgeorge:matrix.org>
17:25:02
on the eln side, i know there are conversations like "ok this package is in the content set, we have to find a rhel maintainer for it". what does that look like on the epel side? people will assume the fedora maintainer is it, but they may have no clue it's in epel now without their consent.
<@sgallagh:fedora.im>
17:25:52
Of course, this is touching on a feature I've wanted in CR for a long time, which is the ability to add a "test" workload that would show you what would change if a workload was added.
<@conan_kudo:matrix.org>
17:25:55
I would expect absent another maintainer stepping up, the workload maintainer would be assigned all those for the epel11 branch.
<@conan_kudo:matrix.org>
17:26:19
to some degree, I kind of expect them to sign up for enabling those things
<@sgallagh:fedora.im>
17:26:20
Conan Kudo: Right, the workload maintainer that pulls the package in gets to own it in EPEL.
<@carlwgeorge:matrix.org>
17:26:23
right, and that's something that has to intentionally happen
<@sgallagh:fedora.im>
17:26:33
If they don't want to do it, they need to find someone else to explicitly add it to a workload
<@conan_kudo:matrix.org>
17:26:40
yes
<@carlwgeorge:matrix.org>
17:26:42
if we aren't intentional about it, we'll just have unmaintained epel packages
<@sgallagh:fedora.im>
17:26:44
Or their workload doesn't branch.
<@conan_kudo:matrix.org>
17:26:49
workload documents are not really workloads: they are maintainer documents
<@sgallagh:fedora.im>
17:27:00
Carl George: Which has never happened before... :)
<@conan_kudo:matrix.org>
17:27:00
we probably should stop calling them workloads
<@conan_kudo:matrix.org>
17:27:03
because they aren't
<@conan_kudo:matrix.org>
17:27:13
they stopped being workloads a very long time ago
<@sgallagh:fedora.im>
17:27:17
That's a whole other discussion :)
<@yselkowitz:fedora.im>
17:27:19
?
<@sgallagh:fedora.im>
17:27:59
The next steps are that I'll work on that requirements document and come back with something after Flock
<@sgallagh:fedora.im>
17:28:13
(Hopefully getting input on it *at* Flock)
<@conan_kudo:matrix.org>
17:28:24
I will not be able to make it to your talk
<@conan_kudo:matrix.org>
17:28:33
as mine is in paralle
<@conan_kudo:matrix.org>
17:28:35
as mine is in parallel
<@sgallagh:fedora.im>
17:29:37
Conan Kudo: Understood, but I'll try to take notes and revisit at the EPEL workshop as well
<@sgallagh:fedora.im>
17:29:49
Alright, we're out of time. Final remarks?
<@conan_kudo:matrix.org>
17:29:53
oh hay that's fixed now and it doesn't conflict anymore
<@conan_kudo:matrix.org>
17:30:04
they moved the talks around
<@sgallagh:fedora.im>
17:30:13
!action sgallagh to start preparing a work requirement document for ELN Extras -> EPEL 11. Will gather feedback at Flock.
<@conan_kudo:matrix.org>
17:31:41
hey sorry about earlier... I'm just really frustrated over a bunch of things going on right now
<@sgallagh:fedora.im>
17:32:21
I get it.
<@sgallagh:fedora.im>
17:32:29
!endmeeting