16:00:17 <sgallagh> #startmeeting ELN (2023-10-06)
16:00:17 <zodbot> Meeting started Fri Oct  6 16:00:17 2023 UTC.
16:00:17 <zodbot> This meeting is logged and archived in a public location.
16:00:17 <zodbot> The chair is sgallagh. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:00:17 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:17 <zodbot> The meeting name has been set to 'eln_(2023-10-06)'
16:00:18 <sgallagh> #meetingname eln
16:00:18 <zodbot> The meeting name has been set to 'eln'
16:00:18 <sgallagh> #topic init process
16:00:19 <sgallagh> .hi
16:00:20 <zodbot> sgallagh: sgallagh 'Stephen Gallagher' <sgallagh@redhat.com>
16:00:52 <tdawson> greetings
16:01:02 <dcavalca> .hi
16:01:03 <zodbot> dcavalca: dcavalca 'Davide Cavalca' <davide@cavalca.name>
16:01:27 <michel-slm> .hello salimma
16:01:28 <zodbot> michel-slm: salimma 'Michel Lind' <michel@michel-slm.name>
16:01:32 * nirik is lurking at the back, have a topic for open floor/whatever.
16:02:17 * michel-slm also has a small topic
16:02:21 <sgallagh> #topic Agenda
16:02:26 <sgallagh> OK, let's set the agenda, then
16:02:43 <sgallagh> #info Agenda Item: Should ELN continue to be backed by Rawhide repos?
16:02:51 <sgallagh> nirik and michel-slm: What are your topics?
16:03:35 <michel-slm> I was going to ask if we should have our own fedocal entry, then I noticed we're in the SIGs listing https://calendar.fedoraproject.org/SIGs/
16:04:07 <michel-slm> relatedly, maybe this should be mentioned in our docs
16:04:56 <nirik> I was going to see if I can talk eln into not using odcs for composes. ;)
16:04:59 <tdawson> Oddly, I was looking for ELN in FEdora docs and ran into the same thing.  I don't see any top listings for FEdora SIGS either.
16:05:36 <nirik> most sigs use wiki pagres...
16:05:52 <sgallagh> #info Agenda Item: ELN and ODCS: Are they headed for a divorce?
16:06:23 <sgallagh> #info Agenda Item: ELN Docs need an overhaul
16:06:53 <michel-slm> eh, EPEL uses antora and that works fine
16:08:02 <sgallagh> Any other topics for today, or shall we begin?
16:08:12 <Son_Goku> .hello ngompa
16:08:13 <zodbot> Son_Goku: ngompa 'Neal Gompa' <ngompa13@gmail.com>
16:08:23 <Son_Goku> hey folks
16:08:27 * sgallagh waves
16:09:04 <sgallagh> #topic Should ELN continue to be backed by Rawhide repos?
16:09:23 <Son_Goku> oh dear
16:09:30 <Son_Goku> what caused this topic?
16:09:32 <dcavalca> what's the alternative?
16:09:51 <tdawson> https://src.fedoraproject.org/rpms/fedora-release/pull-request/284
16:10:05 <sgallagh> First, a bit of clarity. We're talking about the actual installed ELN system/container.
16:10:10 <tdawson> I think the subject sounds worse than it is.
16:10:29 <sgallagh> We definitely cannot avoid having Rawhide supplementing our buildroot
16:11:12 <sgallagh> Our original intent was for ELN to be custom-built in RHEL style, but to also have access to the wider Fedora package set via Rawhide repositories.
16:11:20 <sgallagh> Reality... has not played out as nicely as that.
16:12:23 <sgallagh> Mitigating this is that we have created ELN Extras in the time since the original decision about Rawhide was made.
16:13:26 <sgallagh> I don't have real numbers on this, but anecdotally I will say that *most* of Rawhide works just fine when installed onto ELN
16:13:35 <sgallagh> but when it does not... it REALLY does not.
16:14:20 <tdawson> I don't know about others, but I have accidentally updated my ELN machine to Fedora Rawhide ... twice.  So I am all for removing the fedora-rawhide repo being required.
16:14:21 <sgallagh> (Of particular note is anything built with LLVM/clang)
16:15:57 <sgallagh> Thoughts?
16:16:14 <sgallagh> tdawson: That's a useful data point. Thanks.
16:17:12 <michel-slm> we already prefer ELN packages over Rawhide if both are available right?
16:17:49 <michel-slm> so the failure cases... are these when Rawhide packages would install, but are unusable, or they are uninstallable / end up causing ELN packages being replaced by Rawhide ones?
16:17:56 <tdawson> Do we?  foo-1-1.fc40 > foo-1-1.eln
16:18:11 <michel-slm> ah the curse of the dist tag sorting lower
16:18:55 <michel-slm> hmm yeah. so what is the mechanism for an ELN install to ensure Rawhide is only used for missing packages?
16:19:05 <michel-slm> seems like even if we rebuild in time, the ELN rebuild will sort lower
16:20:27 <michel-slm> but for installed systems... as someone interested in ELN extras to keep track of what we need in EPEL, removing Rawhide means knowing when something is missing earlier rather than later
16:20:34 <sgallagh> michel-slm: The original idea was to have the ELN repo have a higher priority, but we never actually got around to doing that
16:20:42 <michel-slm> sgallagh: ahh, makes sense
16:21:08 <dcavalca> so fwiw I hadn't even realized one was supposed to have rawhide repos alongside the ELN ones on installed systems
16:21:14 <michel-slm> can we go halfway? have that priority be implemented, and the rawhide repo still shipped but enabled=0
16:21:22 <sgallagh> dcavalca: "Have to" is a strong term.
16:21:23 <dcavalca> for the ELN boxes I have at Meta, we're just using the ELN repos (we generate dnf.conf from Chef)
16:21:26 <tdawson> Please no ...
16:21:37 <tdawson> There is nothing stopping you from installing the rawhide repo if you want it.
16:21:41 <michel-slm> dcavalca: so... in practice, not having rawhide repo available is already working, right
16:21:49 <dcavalca> yep, that's been my experience so far
16:22:05 <michel-slm> tdawson: ok, fair point, no point shipping and maintaining a hacked repo config, just dnf install it as needed
16:22:09 <tdawson> But if you do an update, and Fedora release requires the rawhide-repo, then you get it whether you want it or not.
16:22:10 <sgallagh> The reason we ended up not implementing the priority is that it makes things hard for people doing their own custom stuff (like dcavalca's dnf.conf with config-mgmt tools)
16:22:29 <dcavalca> my only concern with https://src.fedoraproject.org/rpms/fedora-release/pull-request/284 is that it changes releasever, which I believe will make it mismatch with what's in os-release? as ELN systems still identify as rawhide there
16:22:43 <dcavalca> which might be fine, but I wanted to bring it up in case there's unforseen issues that might come out of it
16:23:45 <tdawson> I never noticed the releasever change ... well saw it and didn't think about it.
16:24:29 <sgallagh> Hmm
16:24:39 <neil> o/ sorry i'm late...
16:24:41 <sgallagh> There's definitely some thinking I need to do about that part
16:24:51 <michel-slm> we should also change os-release I suppose
16:25:06 <michel-slm> that ships from the same srpm, luckily
16:25:31 <dcavalca> to be clear, os-release currently identifies as rawhide with variant_id set to ELN
16:25:39 <dcavalca> that's how we tell ELN apart from Rawhide in Chef, for example
16:25:56 <sgallagh> Yeah... I don't think we want to change that
16:26:10 <michel-slm> ah right
16:26:18 <michel-slm> yeah, so ... what is gained from changing releasever then?
16:26:49 <sgallagh> I think I'm just going to request that we reject that part of the PR
16:27:06 <sgallagh> And consider that as a separate thing that needs justification
16:27:26 <sgallagh> So, to be clear: no one opposes dropping the default installation of Rawhide repos?
16:27:42 <sgallagh> default-and-mandatory, I should say
16:28:30 <tdawson> Correct ... install it if you want, but don't force it.
16:28:52 <Son_Goku> sgallagh: why aren't we using repo priorities?
16:29:10 <Son_Goku> sgallagh: I do not think dropping default install of rawhide repos is sufficiently thought out
16:29:58 <michel-slm> Son_Goku: sgallagh says it will make it harder for folks who write their own dnf.conf
16:30:02 <michel-slm> dcavalca: is that a concern for us?
16:30:14 <tdawson> Son_Goku: My biggest problem is that I (and I'm surprised I'm the only one saying this) keep getting updated to Fedora Rawhide.
16:30:18 <sgallagh> Changing repo priorities cannot be automatically done for anyone whose repos are modified or maintained in config management tools
16:30:18 <Son_Goku> michel-slm: that doesn't even make sense?
16:30:30 <Son_Goku> that has never been a concern of ours before
16:30:33 <sgallagh> So it's a significant documentation/education change that would have to be communicted.
16:30:34 <Son_Goku> and it shouldn't be now
16:30:49 <michel-slm> Son_Goku: I am just reiterating what was said
16:30:58 <dcavalca> we already generate our own dnf.conf and we don't put the rawhide repos in, I don't think this would impact Meta's usage of ELN at all
16:31:05 <michel-slm> I personally think those who deploy ELN themselves should know how to adopt this
16:31:14 <michel-slm> dcavalca: I mean if ELN adopts the use of priorities
16:31:17 <Son_Goku> sure, I'm also stating that using config management as an excuse is not a good justification
16:31:33 <michel-slm> because, if we drop rawhide by default, we still want to ensure it has a lower priority if it is later installed right
16:31:33 <sgallagh> Son_Goku: It's not an "excuse", it's a real issue.
16:31:58 <Son_Goku> no, it is, because we're avoiding using a feature to resolve a problem because people change their repo configs on their own with their own tools
16:32:03 <michel-slm> yes, I just want dcavalca and I to on the record, as one of such use cases, to say whether or not we think it's a blocker
16:32:10 <Son_Goku> that way lies madness and paralysis
16:32:22 <michel-slm> realistically, what's the worst thing that could happen?
16:32:50 <dcavalca> I still don't see the issue: if we use priorities, those would only apply to the repo configs we ship; why would that matter for users that don't use those and generate their own?
16:33:04 <tdawson> So, let's fliip this around, what benifit do we get by having the Fedora-rawhide repo installed by default ?
16:33:05 <michel-slm> right now we have a situation that Rawhide can win over ELN. if we set priority to stop that happening, and people who generate their own dnf.conf don't follow suit, worst case they are still affected by the issue
16:33:17 <sgallagh> Son_Goku: But the reality is that having Rawhide repos available by default has real and identifiable issues. Changing the priority fixes one of those (accidental upgrading away from ELN) but not the bigger one (Rawhide builds may not actually be compatible with ELN builds)
16:33:22 <michel-slm> it does not get extra broken. we should document how to set priority, sure
16:33:45 <Son_Goku> sgallagh: I am not sure the latter is a bug
16:33:58 <Son_Goku> err
16:34:00 <Son_Goku> not
16:34:00 <sgallagh> The example I noted earlier is that NOTHING built with Fedora's LLVM/clang will run successfully against libllvm from ELN
16:34:03 <michel-slm> to get some clarity -- Son_Goku wants priority and keeping rawhide installed, sgallagh prefers removing rawhide and not doing priority
16:34:10 <dcavalca> michel-slm: ok, so the scenario of concern is someone who is generating their own repo configs _and_ including both eln and rawhide repos in them
16:34:27 <Son_Goku> sgallagh: and why is that?
16:34:30 <sgallagh> Because they are built with different symbols. (Fedora includes a lot of experimental stuff)
16:34:53 <michel-slm> dcavalca: right... and in those cases we should just document how to set this up, right?
16:35:00 <michel-slm> not stay bug-to-bug reproducible with them
16:35:04 <sgallagh> You can argue that's a bug in LLVM (I have...) but it's not going to change
16:35:25 <Son_Goku> sgallagh: the correct answer is for us to resolve the difference between Fedora and RHEL then
16:35:29 <tdawson> So ... while we were talkikng I installed a fresh ELN install.  fedora-rawhide repo is installed, but I checked, and it is not enabled.
16:35:30 <dcavalca> michel-slm: yeah, I think this is one of those cases where we document what's going on, and people should know how to deal with it if they're deviating from stock installs like that
16:35:37 <michel-slm> this is where upcoming work on overriding repo configs in DNF would come in handy
16:35:46 <Son_Goku> upcoming? it's already done in dnf5
16:35:49 <michel-slm> we can have ELN ship an exclusion of packages we never want installed from rawhide
16:35:55 <tdawson> I'm scratching my head to figure out how I managed to update my ELN to rawhide ... twice.
16:36:12 <michel-slm> Son_Goku: right, but is ELN already using dnf5 by default?
16:36:15 <Son_Goku> yes
16:36:17 <Son_Goku> because rawhide is
16:36:21 <michel-slm> oh sweet
16:36:29 <Son_Goku> nobody ever undid that :/
16:36:42 <sgallagh> tdawson: I think Rawhide repos being disabled on ELN is a relatively recent change.
16:36:51 <michel-slm> so.. if we decide to punt again F40 branched will have to do yet another flip back to dnf4? ugh
16:36:59 <Son_Goku> yes
16:37:04 <sgallagh> tdawson: It's a long story
16:37:23 <tdawson> Ahh ... well if that's the case ... then ... well, I'm ok with how it is them.
16:38:22 <tdawson> Sorry, I didn't mean to do a complete 180 ... but if things have already changed, fixing my original issue ... then ... well ... this seems to be fixing it again in a different way.
16:38:32 <sgallagh> That disablement happening on ELN was an accidental side-effect, actually.
16:38:41 <michel-slm> happy accident
16:38:51 <sgallagh> But one that at least seems to suggest that few people are actually using Rawhide packages on ELN, since no one complained
16:39:16 <nirik> dnf5 default was reverted in rawhide... I am pretty sure.
16:39:29 <tdawson> I prefer not to use rawhide packages.  If something is breaking or not insallable on ELN, I want to know it.
16:39:46 <nirik> %  rpm -qf /usr/bin/dnf
16:39:46 <nirik> dnf-4.17.0-6.fc40.noarch
16:41:25 <tdawson> Don't mean to rush us, but there are two other agenda items.
16:42:57 <sgallagh> OK, with the reality being that Rawhide repos have been disabled for a while and no one has complained PLUS the fact that there are incompatibilities, I move that we just stop installing the disabled repos by default.
16:43:08 <sgallagh> Please vote for the record. I'm +1
16:43:15 <Son_Goku> in principle, I think that ELN not being backed by Rawhide is a bad idea until ELN Extras is something fully fleshed out and integrated as a community process
16:43:52 <Son_Goku> sgallagh: -1
16:44:10 <sgallagh> It remains backed by Rawhide in the buildroot, to be absolutely clear.
16:44:15 <Son_Goku> that doesn't count
16:44:20 <tdawson> I'm +1
16:44:39 <Son_Goku> RHEL and CentOS Stream have way more stuff in the buildroot nobody sees or can use at install time
16:44:42 <Son_Goku> it's effectively not there
16:44:56 <sgallagh> Your concerns are noted.
16:45:08 <Son_Goku> I intend to treat ELN as if it's RHEL, which means it needs effectively a functioning EPEL
16:45:15 <tdawson> It can be installed at will.  But right now, it CANNOT be uninstalled.
16:45:33 <michel-slm> I'm +1
16:45:38 <Son_Goku> my personal requirement for removing rawhide repos from ELN systems is making it so that there's an EPEL
16:45:40 <sgallagh> dcavalca, nirik?
16:45:46 <michel-slm> we want to flush out the missing ELN Extras packages
16:45:54 <michel-slm> and people can always install rawhide on top if they need it
16:45:54 <dcavalca> Son_Goku: fwiw I haven't had any trouble getting arbitrary packages added to ELN Extras, it just takes a bit for the PRs to get merged sometimes
16:46:00 <dcavalca> +0
16:46:13 <Son_Goku> dcavalca: we have no community process advocating and prioritizing them
16:46:21 <Son_Goku> that's my main beef with it
16:46:29 <nirik> +0 don't know enough about it to vote really...
16:46:40 * neil has been following along, but is +0
16:46:40 <tdawson> Son_Goku: But what about those that don't want that, right now they have no choice but to have it installed.  For your use case, you can still install it.  But for others use case, they cannot uninstall the rawhide repos.
16:46:42 <Son_Goku> (I'm also not entirely happy about the fact it's in a dusty corner in a repo controlled by Red Hat employees)
16:47:17 * Son_Goku sighs
16:47:22 <Son_Goku> tdawson: that's not the point I'm making
16:47:23 <dcavalca> docs are here for the record: https://docs.fedoraproject.org/en-US/eln/extras/
16:47:34 <dcavalca> definitely agree we could do more to promote this
16:47:42 <tdawson> Son_Goku: But that's the point I'm making, and the two points cannot live together right now.
16:48:09 * Son_Goku sighs
16:48:09 <tdawson> Son_Goku: As for only Red Hat employees being able to merge ELN-extra pull requests ... I agree with you there ...
16:48:19 <Son_Goku> at least in the LLVM example, LLVM is broken in Fedora too
16:48:26 <Son_Goku> because LLVM 17 broke everything
16:48:31 <sgallagh> #agreed We will remove the requirement for Rawhide repos to be installed alongside ELN repos. They remain available to install manually. (+3, 3, -1)
16:48:39 <sgallagh> We need to move on to the other topics now
16:48:49 <sgallagh> #topic ODCS and ELN
16:49:00 <sgallagh> nirik: You have a proposal for us to discuss?
16:49:49 <nirik> this isn't urgent, but I'd like to propose/float the idea that eln move to the same model we use for rawhide/branched composes... ie, have a compose-eln01 machine and a eln shell script that does whatever is needed/calls pungi... instead of odcs.
16:50:01 <nirik> odcs is still there, and has a bit of development, but not much...
16:50:18 <nirik> and eln and modularity is the only two uses for it in fedora land.
16:50:49 <sgallagh> If we were to move away from ODCS, I have two requirements it would need to meet.
16:50:56 <nirik> nothing needs decided super soon since I guess we need to keep it until f38 goes eol (unless I can convince them to drop it sooner)
16:51:28 <nirik> (the last module build for fedora was about a month ago FYI).
16:51:28 <sgallagh> 1) There needs to be a way for SIG members to trigger a compose when needed sooner than the normal schedule.
16:52:05 <Son_Goku> we don't have that now?
16:52:12 <Son_Goku> only you and tdawson do
16:52:17 <nirik> yeah, if it's a cron, you can adjust the cron to run whenever you like.
16:52:28 <sgallagh> 2) It (or a staging equivalent) needs to be able to carry a newer pungi than other Fedora composes. This is in preparation for some things we have coming down the pipeline in the next few months.
16:52:49 <Son_Goku> I don't know if that's a good requirement
16:52:57 <Son_Goku> that makes it too easy to run a fork
16:52:59 <sgallagh> Son_Goku: Technically anyone with sysadmin_apps membership does, I think?
16:53:03 <nirik> sure, since it's it's own vm, you can adjust the package set there as you like without breaking anything else
16:53:26 <nirik> of course we would want any such packages to be built in koji/signed, etc.
16:53:50 <Son_Goku> sgallagh: that's not the SIG
16:53:59 <sgallagh> Son_Goku: Nothing we're going to do there will remain forked; I'm in talks with folks doing bootable containers about using ELN to prototype some stuff that will be merged back into pungi
16:54:14 <Son_Goku> are you positive about that?
16:54:15 <sgallagh> Son_Goku: Fair enough, but we can get that adjusted
16:54:24 <Son_Goku> because I've heard this story before, and I'm skeptical
16:54:32 <sgallagh> Son_Goku: It will either get merged or canned, by my word.
16:54:36 <Son_Goku> okay
16:54:58 <Son_Goku> (for the peanut gallery, it took half a decade to remerge the forked pungi between RHEL and Fedora, and it started out this way)
16:55:09 <dcavalca> as long as we agree that deviations should be motivated and have a clear end stop I don't necessarily see a problem with this
16:55:37 <dcavalca> we do need to sort out the access though so that everyone in the SIG can help out with this as needed
16:56:09 <nirik> sure, we can adjust access to a sysadmin-eln group or whatever easily
16:56:28 <sgallagh> Thanks, I think that's a good idea.
16:56:39 <sgallagh> I'll formally recommend that Neal and Davide should have that access.
16:56:46 <Son_Goku> :)
16:57:01 <Son_Goku> I definitely want to avoid another big pungi merge thing again, because pungi4 was herculean
16:57:09 <Son_Goku> that's why I'm concerned
16:57:18 <sgallagh> It's absolutely valid.
16:57:21 <dcavalca> sgallagh: do you have a sense of when you'll know whether this will be merged or canned? :)
16:57:25 <Son_Goku> it's really really easy to get into that situation with the design of Fedora/RHEL infra
16:57:52 <sgallagh> dcavalca: If it's not merged by Fedora 40 GA, I'll can it on principle.
16:58:36 <sgallagh> OK, unfortunately we're out of time (which means, yet again, docs gets short-shrift)
16:58:47 <neil> as they deserve...
16:58:49 <neil> (joking)
16:58:49 <sgallagh> #topic ELN Docs
16:58:55 <sgallagh> .fire neil
16:58:55 <zodbot> adamw fires neil
16:59:06 * neil deserved that
16:59:11 <tdawson> *laughs*
16:59:20 <sgallagh> I've definitely not had the cycles to keep the docs as up to date as I'd like.
16:59:24 <tdawson> I think the ELN Docs can get discussed over on Matrix.
16:59:35 <sgallagh> I'd be REALLY happy if someone wanted to assume ownership of that space
17:00:20 <sgallagh> But we're out of time, so we'll continue over in the regular Matrix channel.
17:00:27 <sgallagh> Thank you for joining us, folks!
17:00:31 <sgallagh> #endmeeting