16:00:43 <sgallagh> #startmeeting ELN (2023-09-08)
16:00:43 <sgallagh> #meetingname eln
16:00:43 <zodbot> Meeting started Fri Sep  8 16:00:43 2023 UTC.
16:00:43 <zodbot> This meeting is logged and archived in a public location.
16:00:43 <zodbot> The chair is sgallagh. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:00:43 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:43 <zodbot> The meeting name has been set to 'eln_(2023-09-08)'
16:00:43 <zodbot> The meeting name has been set to 'eln'
16:00:43 <sgallagh> #topic Init Process
16:00:55 <michel-slm> .hello salimma
16:00:56 <zodbot> michel-slm: salimma 'Michel Lind' <michel@michel-slm.name>
16:01:00 <sgallagh> .hi
16:01:01 <zodbot> sgallagh: sgallagh 'Stephen Gallagher' <sgallagh@redhat.com>
16:01:16 <neil> .hi
16:01:18 <zodbot> neil: neil 'Neil Hanlon' <neil@shrug.pw>
16:01:24 <neil> (brb, finishing making my coffee)
16:01:57 <tdawson> Hello ... I made it this week.
16:02:16 <yselkowitz> .hi
16:02:17 <zodbot> yselkowitz: yselkowitz 'Yaakov Selkowitz' <yselkowi@redhat.com>
16:03:03 <dcavalca> .hi
16:03:04 <zodbot> dcavalca: dcavalca 'Davide Cavalca' <davide@cavalca.name>
16:04:22 <sgallagh> Thank you for joining us this week as we hop into our time machine and explore the deepest reaches of the late 1900s...
16:04:47 <sgallagh> IRC!
16:05:34 <sgallagh> #topic Agenda
16:05:50 <sgallagh> #info Agenda Item: Status Update
16:06:09 * neil plays ephemeral music for the time travel sequence
16:06:10 <sgallagh> Does anyone have any specific topics they'd like to add to the agenda this week?
16:06:14 <tdawson> Woaa ... the future was so bright I had to wear shades.
16:06:39 * neil would like to 'formally' join the ELN cabal, if such things would be allowed
16:07:10 <sgallagh> There's one very specific thing you need to do in order to join up.
16:07:21 <sgallagh> Attend the meeting on IRC... so welcome!
16:07:28 <neil> woo!
16:07:33 <neil> easiest requirements ever
16:07:58 <michel-slm> neil is one of the cool kids now
16:08:01 <dcavalca> I have a specific question about why grub2-efi-x64-modules is missing from recent composes, and a general one on how one would can figure that out by themselves
16:08:07 * neil borrows tdawson's shades
16:08:18 <dcavalca> I checked CR as I thought that's where this would be defined but didn't find any smoking gun
16:08:21 <sgallagh> #info Agenda Item: grub2-efi issues
16:08:46 <yselkowitz> is this a result of the changes to the compose?
16:09:20 <dcavalca> maybe? I can tell you it was definitely there a while ago, but I can't pinpoint exactly when it disappeared because my internal monitoring was broken at the same time...
16:09:20 <sgallagh> Alright, do we want to start with that topic?
16:09:27 * sgallagh takes that as a "yes"
16:09:36 <sgallagh> #topic grub2-efi issues
16:10:24 <sgallagh> dcavalca: Do you mean that it shows up in CR but isn't appearing in the compose?
16:10:50 <dcavalca> I meant I didn't see any reference to it in recent commits to CR
16:10:58 <dcavalca> I didn't think to actually check the output of CR but that's a good idea, looking now
16:11:13 <yselkowitz> it's not in ELN
16:11:26 <dcavalca> yeah I don't see it in CR either
16:11:31 <yselkowitz> looks like it should be in ELN extras openstack workload?
16:12:06 <sgallagh> https://tiny.distro.builders/view-rpm--view-eln-extras--grub2-efi-x64-modules.html
16:12:20 <dcavalca> you're right, yeah
16:12:32 <dcavalca> so I take it this was excluded from ELN on purpose at some point?
16:12:33 <sgallagh> Which means it *should* be appearing in the Extras repo
16:12:44 <yselkowitz> but I don't even see a package by that name in koji
16:12:51 <yselkowitz> was it renamed?
16:12:52 <dcavalca> grub2 is the source package
16:12:52 <sgallagh> yselkowitz it's a subpackage
16:13:07 <yselkowitz> yeah I got that, but look: https://koji.fedoraproject.org/koji/buildinfo?buildID=2282236
16:13:12 <dcavalca> and yeah I see it https://odcs.fedoraproject.org/composes/production/latest-Fedora-ELN/compose/Extras/x86_64/os/Packages/grub2-efi-x64-modules-2.06-97.eln130.noarch.rpm
16:13:45 <dcavalca> yselkowitz: it's this one right? https://koji.fedoraproject.org/koji/rpminfo?rpmID=35794554
16:14:04 <yselkowitz> oh its noarch
16:14:56 <sgallagh> dcavalca: So what is the actual issue you are seeing?
16:15:13 <yselkowitz> the grub2.spec is not exactly easy reading...
16:15:13 <sgallagh> Just that it's not part of the BaseOS/AppStream repos anymore?
16:15:21 <sgallagh> That may have happened when I cleaned it up recently.
16:15:33 <sgallagh> Dropping entries that are no longer in Content Resolver for ELN
16:16:04 <sgallagh> Ye[
16:16:05 <sgallagh> yep
16:16:10 <sgallagh> https://pagure.io/pungi-fedora/pull-request/1196#request_diff
16:16:12 <dcavalca> yeah, from my point of view, what I was trying to find out was whether this was intentional (i.e. this will be excluded from c10s too) or just a temporary glitch / ELN quirk
16:16:25 <dcavalca> ooh I forgot about pungi
16:16:27 <sgallagh> Looks intentional to me
16:16:31 <dcavalca> thanks, this explains it
16:16:41 <sgallagh> Well, Content Resolver is the official place
16:16:42 <yselkowitz> does it ship in rhel 9?
16:16:50 <sgallagh> I had to update Pungi to match CR
16:16:52 <neil> it does afaik
16:16:52 <dcavalca> yeah, it's in BaseOS
16:17:01 <dcavalca> well, on c9s, I don't have a RHEL 9 box to check
16:17:15 <neil> it's "required" if you want to be able to rebuild grub from source
16:17:24 <neil> (i mean, they're in the SRPM,  but)
16:17:45 <neil> perhaps systemd-boot and UKI has something to do with it (?)
16:18:00 <neil> dcavalca: yeah it's in BaseOS on RHEL proper, too
16:18:16 <yselkowitz> confirmed, it ships in 9
16:18:43 <sgallagh> We can probably look into why it's been removed from the Content Resolver input
16:19:02 <sgallagh> But it definitely looks intentional
16:19:40 <yselkowitz> are you sure, I don't see any mention of it in git log -p
16:20:15 <sgallagh> Neither do I. It's possible it's an oversight.
16:20:23 <sgallagh> Maybe something used to Requires: it and stopped?
16:20:51 <yselkowitz> possible, that spec file is complicated to put it nicely
16:21:16 <neil> yeah--and there _was_ stuff moved around due to systemd-boot, iirc
16:22:11 <sgallagh> It might be worth checking with the bootloader folks if that's an intentional change or an oversight.
16:22:28 <neil> looks like `tboot` is the only thing that Requires it on rl9
16:22:30 <neil> el9*
16:23:03 <dcavalca> fwiw I can't actually tell if we're using this for anything at Meta, but we _are_ explicitly installing it (which is how I noticed it went away)
16:23:08 <sgallagh> tboot in ELN doesn't have that dependency
16:23:12 <sgallagh> So that might explain it
16:23:48 <yselkowitz> https://gitlab.com/redhat/centos-stream/rpms/tboot/-/commit/26f7e15a839997e9daf0c2a85393fdfa12a7650b
16:24:00 <yselkowitz> "- Added a Requires line to install grub2-efi-x64-modules"
16:25:10 * yselkowitz used to work with tcamuso, could ask him what's up with that
16:25:20 <sgallagh> Please do
16:25:42 <dcavalca> thanks, yeah knowing one way or another would definitely be helpful here
16:25:44 <sgallagh> It looks like tboot in Fedora is maintained by Intel folks, not Red Hat folks.
16:26:12 <sgallagh> #info There's a disconnect between CentOS Stream and ELN; tboot depends on grub2-efi-x64-modules in CS9
16:26:35 <sgallagh> #action yselkowitz will contact the maintainers to find out what should happen in ELN
16:27:24 <sgallagh> #info The grub2-efi-x64-modules package was removed from ELN because it is not present in the Content Resolver output.
16:27:36 <sgallagh> Anything else to add here?
16:28:17 <dcavalca> I think that's it, thanks
16:28:39 <sgallagh> Actually, let me write out a little more detail
16:28:58 <sgallagh> I'll give a little history about what I changed in the pungi config last week,.
16:29:57 <sgallagh> As you know, RHEL maintains several repositories for packages. The main ones are BaseOS, AppStream and CRB. There are a few others around, but we'll focus on these.
16:31:09 <sgallagh> Content Resolver doesn't currently provide any mechanism for annotating which packages belong in which repo. We have a hack that we use to shunt Extras into a comps group, but that's "special".
16:31:56 <sgallagh> So we have to make use of a pungi config document called `prepopulate.json` to do the split, by listing every binary package produced by each SRPM we are including in each repo.
16:32:30 <sgallagh33> I got disconnected/reconnected: what was the last you saw from me?
16:32:42 <dcavalca> > <sgallagh> So we have to make use of a pungi config document called `prepopulate.json` to do the split, by listing every binary package produced by each SRPM we are including in each repo.
16:32:55 <sgallagh33> (Yes, this is as ugly as it sounds)
16:33:47 <dcavalca> ugh, yeah it sounds like teaching CR to map packages to repos would make this a lot simpler in the long term
16:34:02 <sgallagh33> What we did initially was just copy the `prepopulate.json` from CentOS Stream 9 into ELN and then hand-edit it whenever things changed in a major way.
16:34:19 <neil> we sorta did something like that with how we manage repositories in Peridot--though that's created _from_ preopulate.json lol
16:34:52 <sgallagh33> Last week, people started noticing that some packages that were explicitly removed from ELN were still showing up in the compose, so I ran a script to strip out everything from `prepopulate.json` that is not listed in Content Resolver
16:35:06 <sgallagh33> (That resulted in the MR I linked earlier)
16:35:10 <neil> (Link for the curious :P https://git.rockylinux.org/rocky/peridot-rocky/-/blob/r9/autocatalog.cfg?expanded=true&viewer=simple )
16:35:48 <sgallagh33> So that brings us to where we are today.
16:36:26 <sgallagh33> We have plans to rework the Content Resolver input format to allow us to annotate where things belong.
16:36:51 <dcavalca> sgallagh: you should consider writing a blog post or something explaining how the sausage is made
16:37:05 <sgallagh> bstinson is planning to present some design options to us in the next week or two
16:37:14 <dcavalca> there are a _lot_ of moving parts here, and it'd be useful to have a birds eye view of how everything fits together written down somewhere
16:37:29 <sgallagh> I will probably do that after we make this CR change
16:37:37 <dcavalca> (I meant you as global you, not necessarily you personally to be clear)
16:37:42 <sgallagh> Because it will change a lot of this information (and probably make it easier)
16:37:55 <dcavalca> makes sense, thanks
16:38:02 <neil> I think that'd be a really informational post, yeah
16:38:27 <sgallagh> Yeah, I need to do that. If only so that someone can take over if this lottery ticket I bought comes through...
16:39:03 <neil> i'd be happy to contribute (to the lottery or the post :) ). i am an okayish technical editor
16:40:01 <sgallagh> I will likely take you up on that.
16:40:05 <sgallagh> Thank you
16:40:57 <sgallagh> My ultimate goal is to have the Content Resolver input be the sole authority for what ends up in the compose.
16:41:23 <sgallagh> Right now, it's the list of what is *supposed* to be there. Which is close, but not quite the same thing
16:42:11 <sgallagh> (Which means that if you see a difference between CR and the compose, it is a bug)
16:42:38 <dcavalca> and ideally we can have automation to compare CR with the compose and file bugs when that happens
16:43:25 <sgallagh> I'd love to see that, but I can't promise anything in the near future. Too many moving parts and less than six months until CentOS Stream 10 kicks off.
16:43:39 <sgallagh> Contributions welcome, of course
16:44:19 <dcavalca> oh yeah, something for the long term for sure, and I wouldn't mind helping out if we get a chance
16:46:00 <sgallagh> Alright, I think we've covered this topic.
16:46:09 <sgallagh> #topic Status Update
16:46:13 <neil> same here. i think that'd be a welcome improvement :)
16:47:26 <sgallagh> I have a minor announcement: we're going to do the first pass of CentOS Stream 10 package import/sync early next week.
16:47:38 <dcavalca> yay!
16:47:48 <dcavalca> are we going to start making c10s composes as well?
16:47:51 <neil> woot! :)
16:48:04 <tdawson> dcavalca: Not yet
16:48:10 <sgallagh> This will involve a mini-mass-rebuild of ELN packages, so we'll be bumping the buildroot number to 132 and triggering a rebuild of most of the packages slated for the CS10 runtime.
16:48:31 <sgallagh> (There are some special cases that we need to exclude while we work out some issues with the kind folks at Gitlab)
16:49:17 <yselkowitz> my status: 3 packages currently FTBFS, still requires investigation. 5 packages still FTBFS in rawhide (down 1 from last time).  today's dependency spike should be squashed by this evening, staying approximately level with the last few weeks.
16:49:32 <sgallagh> Composes will come later, but before the end of 2023. We haven't nailed down a firm date yet, but we'll be starting the gears turning
16:49:39 <sgallagh> right after the import completes
16:50:19 <dcavalca> that's excellent
16:50:23 <sgallagh> #info First packages will be synced to CentOS Stream 10 next week. Composes to come later.
16:50:40 <dcavalca> as soon as we have signed composes I'll start getting this bootstrapped internally
16:50:55 <sgallagh> #info Direct CentOS Stream development is still not going to start; all work should continue to happen on ELN until further notice.
16:51:13 <sgallagh> (Where "further notice" means "until Fedora 40 Final Freeze")
16:52:06 <sgallagh> dcavalca Signed composes probably won't happen until the official launch of CS10, which will be once the ELN sync ends
16:52:47 <sgallagh> I mean, the composes might have signatures, but I don't think the public keys will be published before then.
16:53:53 <dcavalca> if we can get the signing in place sooner than later, that'd make my life easier
16:54:10 <sgallagh> We'll have to talk about that some more.
16:54:17 <neil> ditto - would like to get a head start on CS10 in OpenStack-Ansible
16:54:19 <dcavalca> we basically would like to start bootstrapping c10s internally as soon as possible so we can provide ongoing feedback early on
16:54:28 <sgallagh> I'm slightly of the opinion that we don't want to sign anything we're unwilling to stand behind :)
16:56:05 <sgallagh> ELN at least is very clearly a moving development target. CentOS Stream carries a stronger stability promise.
16:56:09 <tdawson> Although it's good to get things in place ... up until the F40 split, this is going to go through many updates/changes ... at least for  package version.
16:57:18 <dcavalca> yeah I think as long as we're clear on when Stream 10 is _actually_ released, having early composes out shouldn't be a problem
16:57:25 <dcavalca> we kinda did something similar with c9s and it worked pretty well IMO
16:57:48 <sgallagh> Yeah, I'm sure we can come up with something. Maybe we hand-deliver the public keys to people we know understand what they're getting into :)
16:58:20 <dcavalca> works for me :)
16:58:23 <sgallagh> I'll let folks like bstinson figure that one out. I just turn the crank
16:59:07 <sgallagh> Alright, we're pretty much at time. Any urgent last questions? If so, take them to #fedora-eln
16:59:21 <sgallagh> Thank you everyone for coming
16:59:28 <neil> 👍thanks sgallagh!
16:59:32 <neil> see y'all in a few weeks :)
16:59:47 <dcavalca> thanks!
16:59:50 <sgallagh> #endmeeting