16:01:12 <sgallagh> #startmeeting ELN (2022-10-07)
16:01:12 <zodbot> Meeting started Fri Oct  7 16:01:12 2022 UTC.
16:01:12 <zodbot> This meeting is logged and archived in a public location.
16:01:12 <zodbot> The chair is sgallagh. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:01:12 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:01:12 <zodbot> The meeting name has been set to 'eln_(2022-10-07)'
16:01:12 <sgallagh> #meetingname eln
16:01:12 <zodbot> The meeting name has been set to 'eln'
16:01:12 <sgallagh> #topic Init Process
16:01:17 <salimma> .hi
16:01:18 <zodbot> salimma: salimma 'Michel Alexandre Salim' <michel@michel-slm.name>
16:01:21 <sgallagh> .hi
16:01:24 <zodbot> sgallagh: sgallagh 'Stephen Gallagher' <sgallagh@redhat.com>
16:01:36 <davide> .hello dcavalca
16:01:37 <zodbot> davide: dcavalca 'Davide Cavalca' <dcavalca@fb.com>
16:02:12 <tdawson> Hello
16:02:42 <sgallagh> #chair dcavalca tdawson salimma
16:02:42 <zodbot> Current chairs: dcavalca salimma sgallagh tdawson
16:04:33 <sgallagh> OK, I guess that's everyone who is coming today
16:04:41 <sgallagh> #topic Compose Status
16:05:12 <sgallagh> #info The repo creation is working on all arches at the moment. We are still having issues with the container image generation on ppc64le and aarch64
16:05:40 <sgallagh> Those issues are probably kernel-related, but I haven't been able to get a solid reproducer in place to prove it yet
16:06:17 <sgallagh> Davide Cavalca: Do I remember correctly that you have access to some beefy aarch64 hardware?
16:06:51 <davide> I do, though most of the time I just aws graviton instances for aarch64 stuff
16:07:30 <sgallagh> I basically need to set up a KVM-capable aarch64 environment so I can run ImageFactory on it.
16:07:53 <sgallagh> But the only hardware I have easy access to is too slow to meaningfully work with (RPi 400)
16:08:26 <salimma> would an AWS instance work for you Stephen Gallagher
16:08:30 <sgallagh> I don't think AWS instances support KVM (virt-on-virt)
16:08:38 <davide> metal instances do
16:08:50 <davide> I haven't tried myself but it should work
16:09:08 <salimma> we can probably lend you one if that works
16:09:36 <smooge> hello
16:09:44 <salimma> hi smooge
16:09:53 <sgallagh> Hmm, I didn't realize bare-metal instances were available in AWS for aarch64
16:10:17 <davide> x2gd.metal is the type I generally use
16:10:20 <sgallagh> Though it's the storage that will likely kill me.
16:10:32 <davide> these also have a fast local nvme drive
16:10:45 <sgallagh> I meant capacity rather than speed
16:11:07 <sgallagh> ImageFactory needs to cache the entire RPM repo
16:11:23 <sgallagh> For ELN, not all of Fedora
16:11:41 <davide> iirc you got a few TBs, lemme check
16:12:23 <davide> yeah you get two nvme drives of 1900 GB each
16:12:32 <sgallagh> Hmm.. that may work then
16:12:33 <davide> plus whatever you want on EBS, but that's gonna be slower of course
16:14:14 <sgallagh> OK, time to see about getting budget for that...
16:14:50 <davide> if you just need an instance for a oneoff test, I can spin you one up on my account
16:15:21 <sgallagh> I think we (Fedora) have a deal with AWS. smooge do you know?
16:15:24 <davide> only caveat is that I can't leave the security group wide open, so I'd need a static IP of some kind to allowlist for ssh
16:16:15 <davide> I know that AWS has a pretty generous program for OSS project, but I dunno if Fedora is on that
16:16:15 <sgallagh> I appreciate the offer, but this could be days of work. I don't want to run up a huge bill on you
16:17:39 <sgallagh> I'll see what I can scrape up.
16:17:57 <davide> these aren't too bad, keeping one up for a week or two should be fine
16:18:40 <sgallagh> I'll see what I can find today and get back to you on Monday?
16:18:44 <davide> sounds good
16:18:54 <sgallagh> Thanks again
16:19:07 <sgallagh> #topic ELN Rebuild Status
16:19:28 <tdawson> Is that rpm rebuilds?
16:19:39 <sgallagh> tdawson has been working on cleaning up some of the build failures lately
16:19:44 <sgallagh> Yes
16:19:58 <tdawson> I've written a script to make is easier for me.
16:20:02 <sgallagh> Could you give us a brief summary of what you've found thus far?
16:20:06 <sgallagh> (Common errors, etc.)
16:20:06 <tdawson> https://github.com/tdawson/eln-fail-work
16:20:32 <tdawson> Sure
16:20:44 <tdawson> This file has my notes thus far - https://github.com/tdawson/eln-fail-work/blob/main/failed-builds.json
16:20:57 <tdawson> A very common build problem is with LLVM
16:21:13 <tdawson> Seems like something is missing in the RHEL build of LLVM-15
16:21:37 <tdawson> If you grep LLVM on that json, you'll see those errors.
16:22:03 <tdawson> Several of the gnome packages are failling on a gtk4 bug
16:22:46 <sgallagh> The LLVM issue occurs when the Fedora version of LLVM ends up in the buildroot
16:22:47 <tdawson> The container packages that run on golang need to update their specs to use the golang arches.  Their specs were written with RHEL 6 in mind, and they can be updated at this time.
16:22:57 <sgallagh> I recently removed it and clang from the auto-tag to avoid that.
16:22:57 <tdawson> sgallagh: Ahh ... ok
16:23:13 <tdawson> I'll give those another try.
16:23:24 <sgallagh> The reason is because LLVM built for RHEL/ELN disables a bunch of features.
16:24:08 <tdawson> There are several packages that fail on rawhide, those are lowest on my priority.
16:24:37 <sgallagh> #info Golang packages need to be updated to use %golang_arches which differs between Fedora and ELN
16:24:42 <tdawson> Some of them, like the kf5 things, just need to be built in the right order, I'm almost done with those.
16:25:43 <sgallagh> Having the Fedora version in the tag isn't sufficient to avoid the ordering problem?
16:25:44 <tdawson> And then there are they few that have their own errors, no trend.
16:25:55 <sgallagh> We should probably investigate that.
16:26:20 <tdawson> sgallagh: Ya, that's rather strange.
16:26:52 <tdawson> The thing is, they really shouldn't even be in ELN, they are getting pulled in by an odd depenency ... and I'd really rather get that fixed.
16:27:22 <sgallagh> Ah, good point.
16:28:05 <sgallagh> tdawson: Thanks for the update.
16:28:12 <sgallagh> tdawson++
16:28:26 <sgallagh> #topic ELN divergent branches
16:29:14 <sgallagh> It became apparent earlier this week that while we in this SIG agreed to allow `eln` branches in dist-git, we never actually designed a process for doing so.
16:29:47 <sgallagh> And, importantly, we never added this to the documentation.
16:30:47 <sgallagh> If I gave a quick summary of the plan (as it was agreed months ago), could I get a volunteer to turn it into documentation on https://github.com/fedora-eln/eln-docs/ ?
16:31:50 * tdawson notices everyone taking one step back.
16:32:09 <salimma> sure
16:32:12 <sgallagh> There are two basic things we need to cover: 1) How to disconnect from Rawhide and 2) How to reconnect again later.
16:32:15 <sgallagh> salimma++
16:32:27 <salimma> (whathave I signed up for :p)
16:33:31 <sgallagh> The process for the packagers will be fairly simple. We will ask them to file a ticket at https://github.com/fedora-eln/eln/issues requesting the creation of the `eln` branch.
16:34:04 <sgallagh> On our end, we will need to do two things: update the DistroBuildSync configuration to exclude that package and then request the branch via `fedpkg request-branch`
16:34:49 <sgallagh> Once that's complete, we will close the ticket. The documentation should make it clear that from then on, the maintainer is responsible for doing a `fedpkg build` from the `eln` branch.
16:35:32 <salimma> ok. so we should explicitly document on the user facing side that they shouldn't do fedpkg request-branch manually right
16:35:46 <salimma> what's the mechanism for limiting who can request a branch>?
16:36:02 <sgallagh> The process to reconnect to Rawhide is similar, but reversed. They open a ticket, we remove them from the DBS config exclusions and they can abandon the `eln` branch (until it's needed again)
16:36:23 <sgallagh> There really isn't one, but having the branch is useless without disconnecting the auto-rebuild
16:36:39 <sgallagh> So it's probably just easier for everyone if we handle the branch-request
16:36:54 <salimma> gotcha
16:37:15 <salimma> abandoning the eln branch is enough, they don't need to do 'fedpkg retire'?
16:37:24 <salimma> yeah I guess it's the ELN rebuild process that will just ignore that branch
16:37:34 <sgallagh> Right, the rebuild only pays attention to the rawhide branch
16:37:50 <salimma> can we point them a link to see which eln branches are currently active?
16:38:05 <sgallagh> Actually, we probably need one more section: how to revive an ELN branch after it was abandoned.
16:38:08 <salimma> and... what happens if they requested a branch, we added the exclusion, but the branch is  empty
16:38:13 <salimma> would anything depending on it temporarily fail?
16:38:27 <sgallagh> It's mostly the same as the first section, but should have a guide on how to use `git merge -X ours`
16:38:32 <salimma> (I'm sure these all were discussed months ago, my memory is flaky sorry)
16:39:14 <sgallagh> The branch won't be empty, we will branch from the current state of Rawhide
16:39:33 <salimma> sgallagh: aha of course
16:39:55 <sgallagh> (I forget if `fedpkg request-branch` does that automatically or if we will need to do a `git pull`
16:39:58 <salimma> most of my new branches are EPEL these days and those start empty. not an issue here
16:40:04 <sgallagh> That'll be part of our SoP
16:40:16 <salimma> oh wait
16:40:32 <salimma> yeah I forgot what happens when you request branch for say a new package. haven't had one for a while
16:41:02 <sgallagh> I think it will behave the same as EPEL, so the best thing to do is for us to manually make it equal to Rawhide.
16:41:06 <sgallagh> For simplicity's sake
16:41:06 <salimma> yeah
16:41:26 <salimma> so worst case scenario, what happens if either we forget, or a packager did the branch manually
16:41:40 <salimma> wait, ok, just in case we forget, since we establish the branch just gets ignored otherwise
16:41:58 <sgallagh> The key point is the DistroBuildSync config
16:42:20 <salimma> I can imagine either have distrobuildsync fallback to using rawhide, or just fail and we then know we have to fix something
16:42:26 <sgallagh> If that still points at Rawhide, then every new Rawhide build will build a new ELN build that will supersede any built from the branch
16:42:36 <sgallagh> DBS will not build from `eln`
16:42:42 <sgallagh> That will have to be done manually by the maintainer
16:43:02 <sgallagh> Since there's no long a direct link between Rawhide and ELN, we have nothing to trigger on
16:43:30 <salimma> right. but if DBS points to eln and we forgot to fill it, what happens
16:43:42 <sgallagh> DBS won't point to ELN
16:43:57 <sgallagh> It only has a list of packages to listen for Rawhide builds on
16:44:17 <sgallagh> If we remove it from that list, DBS does nothing with that package
16:44:36 <salimma> ok, sorry, my scenario is: package A is on the DBS list, but the eln branch is empty, and package B depends on package A
16:45:25 <salimma> (I'm probably overthinking it and we don't need to deal with this right now, so we can move on )
16:45:50 <sgallagh> Yeah, I think you're probably missing something.
16:46:19 <sgallagh> Building PackageB in your example would just use whatever the latest tagged into eln-build is
16:46:31 <sgallagh> eln-build inherits from eln and then from Rawhide
16:47:09 <salimma> ahh
16:47:17 <salimma> oh yeah I forgot about the rawhide fallback, thanks
16:47:25 <sgallagh> No problem.
16:48:03 <sgallagh> #action salimma to document the ELN branch process for maintainers and the SoP for the ELN SIG
16:48:08 <sgallagh> #topic Open Floor
16:48:17 <sgallagh> Any topics for Open Floor today?
16:48:43 <tdawson> Nothing from me
16:49:12 <salimma> just a small FYI that we (Meta) intend to have a tool to grab the list of our internal packages and spit out a YAML for ELN Extras
16:49:34 <salimma> wondering if something similar will be useful for more people, otherwise I'll just keep this internal
16:49:53 <sgallagh> I'm not sure I understand what YAML you mean
16:50:01 <davide> to elaborate, the usecase on our end is to attach some additional metadata to the packages (like, who cares about it and why)
16:50:04 <tdawson> For the content resolver
16:50:06 <salimma> but on the public facing side, I'll probably make ebranch be able to consume ELN Extras yaml lists and use that to mass-file branch requests
16:50:18 <salimma> ah, sorry, yes, for the content resolver
16:50:32 <davide> https://github.com/minimization/content-resolver-input/blob/main/configs/eln_extras_meta.yaml
16:50:53 <sgallagh> Ahh, got it
16:51:36 <sgallagh> It would probably be useful to have that script made public somewhere, yeah
16:51:58 <sgallagh> Our (Red Hat's) internal teams might want to use it to help generate their CR input also
16:52:41 <salimma> ah, ok. I'll probably split out anything that depends on our internal stack
16:53:15 <davide> yeah we can just make this take JSON as input, and then tie the input generation part to our internal config thing
16:53:16 <davide> this way it should be generally useful
16:53:41 <sgallagh> Thanks!
16:55:00 <salimma> still at design phase, so yeah let's see. with JSON it might just end up being... too similar to the YAML already
16:55:13 <salimma> do we have a validator for the YAML syntax? that might come in useful
16:55:20 <salimma> (to write, if it doesn't exist already)
16:55:32 <sgallagh> Especially since YAML is just a superset of JSON :)
16:55:33 <salimma> yup
16:55:44 <sgallagh> I don't actually know. tdawson might
16:56:13 <tdawson> I know Adam wrote one, so the SST's could validate their yamls ...
16:56:16 <tdawson> Looking ...
16:56:29 <salimma> oh nice. Williamson?
16:57:27 <tdawson> No, Samilak ... he wrote Content Resolver
16:57:32 <sgallagh> No, Adam Samalik
16:58:01 <salimma> ah, minimization Adam
16:58:43 <salimma> tdawson: if you find it later feel free to ping me in the ELN room
16:58:59 <sgallagh> OK, we are at the top of the hour.
16:59:01 <tdawson> I just found it, but it's in a different repo ..
16:59:24 <tdawson> https://github.com/minimization/content-resolver/blob/master/test_config_files.py
16:59:25 <sgallagh> Thanks very much for joining today.
16:59:31 <davide> thanks!
16:59:42 <sgallagh> Thanks, Troy!
17:00:27 <salimma> tdawson++
17:01:13 <sgallagh> #endmeeting