16:03:24 <sgallagh> #startmeeting ELN (2022-11-04)
16:03:25 <zodbot> Meeting started Fri Nov  4 16:03:24 2022 UTC.
16:03:25 <zodbot> This meeting is logged and archived in a public location.
16:03:25 <zodbot> The chair is sgallagh. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:03:25 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:03:25 <zodbot> The meeting name has been set to 'eln_(2022-11-04)'
16:03:25 <sgallagh> #meetingname eln
16:03:25 <zodbot> The meeting name has been set to 'eln'
16:03:25 <sgallagh> #topic init process
16:03:25 <sgallagh> .hi
16:03:26 <zodbot> sgallagh: sgallagh 'Stephen Gallagher' <sgallagh@redhat.com>
16:03:36 <jforbes> morning
16:04:15 <tdawson> Hello
16:05:47 <sgallagh> I'll wait just a minute longer to see if anyone else shows up
16:06:02 <salimma> .hi
16:06:03 <zodbot> salimma: salimma 'Michel Alexandre Salim' <michel@michel-slm.name>
16:06:47 <sgallagh> #topic Current status of aarch64 and ppc64le with debug kernels.
16:06:54 <sgallagh> Good news and bad news here.
16:07:33 <sgallagh> Good News: The debug config flag that triggers the OOPS has been narrowed down and reproduced by the kernel maintainers.
16:07:57 <sgallagh> .bug 2137809
16:07:59 <zodbot> sgallagh: 2137809 – kernel-6.1.0-0.rc1.20221019gitaae703b02f92.17.eln122 on aarch64: kernel BUG at mm/page_table_check.c:82! - https://bugzilla.redhat.com/2137809
16:08:27 <tdawson> Ya !!!   I hope that is the good news.
16:08:42 <davide> .hello dcavalca
16:08:43 <zodbot> davide: dcavalca 'Davide Cavalca' <dcavalca@fb.com>
16:09:02 <tdawson> Nevermind the last part of the sentance, it is the good news.
16:09:33 <sgallagh> Bad News: It seems to be a complicated issue and isn't fixed yet, so we'll continue to have broken aarch64 and ppc64le composes for a while longer
16:09:51 <jforbes> Monday's shoudl work though right?
16:10:08 <sgallagh> jforbes: I'm getting to that.
16:10:56 <sgallagh> Once a week, the Rawhide kernel is built with a non-debug configuration. The composes that occur while that build is active in the `eln` tag succeed completely.
16:11:39 <tdawson> Which day of the week is that?
16:12:32 <tdawson> I have a couple strange qt6 only failling on ppc64le on ELN that I'd like to try building with a non-debug kernel, just to see.
16:12:40 <sgallagh> So one option we could look at would be to mark the container images as blocking the compose. This would mean that only composes that reach FINISHED (vs. FINISHED_INCOMPLETE) would be promoted to "latest-production"
16:12:41 <sgallagh> tdawson: Kernel build is Monday, so usually the Tuesday Compose
16:13:14 <jforbes> sgallagh: since eln is triggered by hand for kernel, I can hold off on the next debug build a little bit here and there if needed
16:13:19 <sgallagh> The downside to this is it means that we'll essentially drop to a single compose per week
16:13:23 <tdawson> Cool, I've give it a try next week, see if it does anything.
16:14:04 <tdawson> These aren't main qt6 packages (like qt6-qtbase) ... so there is no real rush.
16:14:08 <sgallagh> Which means that the feedback loop between the compose and Content Resolver will get pretty long
16:14:49 <sgallagh> I see the following options:
16:16:11 <sgallagh> 1) Just wait and hope the bug is fixed quickly
16:16:11 <sgallagh> 2) Make the images blocking so we only push to latest-production on a complete success, slowing Content Resolver but ensuring bootability for users
16:16:32 <sgallagh> 3. Stop building debug kernels in ELN until the issue is reolved.
16:17:06 <michel-slm> 4. temporarily make aarch64 and ppc64le optional?
16:17:34 <sgallagh> michel-slm: The images are already optiona
16:17:39 <sgallagh> s/optiona/optional/
16:17:58 <jforbes> There are parts in the kernel QA pipeline which depend on working eln composes
16:18:00 <michel-slm> ah. in that case, making them blocking will probably affect more people right
16:18:02 <sgallagh> Only the repo and install trees are blocking, and those both complete successfully
16:19:25 <sgallagh> michel-slm: I *think* they currently only depend on the install trees
16:20:29 <sgallagh> jforbes: Your opinion on those options carries a lot of weight. What do you think?
16:22:15 <jforbes> Denys is food at finding the root cause, and this came in with the 6.1 merge window, so I do expect it will be fixed before too long. However, it is somewhat complex, so it might be a bit.
16:22:55 <sgallagh> "It will either be fixed quickly or take a while". Got it. ;-)
16:23:04 <jforbes> As cool a 1) sounds, 3) might be the more reasonable.  I would like to take it to the ark meeting on Monday though to get their opinion
16:23:22 <sgallagh> Sounds like a good idea.
16:23:37 <jforbes> Well, quickly vs a while in the context of  rc cycles, so not too long either way
16:24:06 <sgallagh> I suppose 3b would be "disable only the debug config item triggering the crash", if we wanted to retain SOME debug capability
16:24:24 <jforbes> But CKI/ark is really one of the biggest consumers of ELN for day to day process, so I would like to get the CKI ark input
16:25:42 <sgallagh> #action jforbes to ask the CKI/ARK team for their input on whether to disable some, all or none of the debugging until bz2137809 is fixed
16:26:12 <sgallagh> Thanks, jforbes
16:26:17 <jforbes> NP
16:26:26 <sgallagh> #topic ELN Process Documentation
16:26:51 <sgallagh> We had a recent email thread on devel@lists.fp.o that makes it clear that we need some better docs
16:27:23 <sgallagh> In this particular thread, a package maintainer mistakenly thought that they needed to build successfully on ELN for inclusion into Fedora.
16:27:54 <sgallagh> We need to make it more discoverable/understandable that ELN provides only a subset of Fedora packages
16:28:05 <salimma> ah, I meant to contribute a PR here (from two meetings ago) but didn't get the chance, sorry
16:28:10 <sgallagh> And probably provide some guidance on whether to consider ELN while doing a package review
16:28:26 <salimma> but this seems like related but different documentation, right?
16:29:00 <sgallagh> (The latter boiling down mainly to "is this package being split off of or a dependency for something already in ELN?")
16:29:40 <sgallagh> Michel Alexandre Salim 🎩: I'd say this is new content
16:31:50 <sgallagh> Well, I'll probably take this part on myself.
16:32:06 <sgallagh> #action sgallagh to write up some guidelines on when packagers should care about ELN
16:32:11 <sgallagh> #topic Open Floor
16:32:25 <sgallagh> Do we have any topics for the Open Floor today?
16:32:53 <tdawson> A minor topic ... about eln-extras
16:33:10 <tdawson> And I think this falls into documentation as well.
16:33:17 <sgallagh> Fire away
16:33:40 <tdawson> We need to let people know that just putting a package in eln-extras doesn't magically keep it buildable on RHEL.
16:34:04 <tdawson> They need to keep an eye on their packages and when they don't build in ELN, get them fixed.
16:34:52 <tdawson> Granted, if it's obvious, I'm going to probrubly fix it ... but if they aren't packages that I put in eln-extras ... I shouldn't be the one fixing them.
16:34:56 <sgallagh> Are you volunteering to write something up?
16:35:07 <sgallagh> Agreed
16:35:08 <tdawson> Yes
16:35:16 <sgallagh> Thank you
16:35:30 <sgallagh> #action tdawson to write up some ELN-Extras guidelines
16:35:36 <tdawson> It might take me a few weeks, because I want it to sound nice, not like a grumpy admin.
16:35:53 <sgallagh> Fair enough
16:36:10 <tdawson> And honestly, I love going through and fixing them ... but it takes my time away from other things.
16:37:00 <tdawson> That's all I have.
16:37:13 <sgallagh> Thank you
16:37:16 <sgallagh> Any other topics?
16:38:25 <sgallagh> Alright, thank you all for coming. See you in #fedora-eln :-)
16:38:29 <sgallagh> #endmeeting