17:02:52 <sgallagh> #startmeeting ELN (2023-02-10)
17:02:52 <zodbot> Meeting started Fri Feb 10 17:02:52 2023 UTC.
17:02:52 <zodbot> This meeting is logged and archived in a public location.
17:02:52 <zodbot> The chair is sgallagh. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
17:02:52 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
17:02:52 <zodbot> The meeting name has been set to 'eln_(2023-02-10)'
17:02:52 <sgallagh> #meetingname eln
17:02:52 <sgallagh> #topic init process
17:02:52 <zodbot> The meeting name has been set to 'eln'
17:02:52 <sgallagh> .hi
17:02:53 <zodbot> sgallagh: sgallagh 'Stephen Gallagher' <sgallagh@redhat.com>
17:03:00 <yselkowitz[m]> .hello yselkowitz
17:03:01 <zodbot> yselkowitz[m]: yselkowitz 'Yaakov Selkowitz' <yselkowi@redhat.com>
17:03:16 <tdawson> Hello
17:03:18 <sgallagh> .hi
17:03:21 <zodbot> sgallagh: sgallagh 'Stephen Gallagher' <sgallagh@redhat.com>
17:03:21 <davide> .hello dcavalca
17:03:24 <zodbot> davide: dcavalca 'Davide Cavalca' <davide@cavalca.name>
17:03:30 <sgallagh> Whoops, did that already
17:04:51 <sgallagh> OK, that's probably everyone who will be here today
17:04:59 <jforbes> I am here
17:05:06 <sgallagh> I stand corrected
17:05:12 <sgallagh> #topic Status Update
17:06:16 <sgallagh> We had a rough go of it since the F38 mass-rebuild, but composes have stabilized again. We've had functional repos for a week and now two straight days of FINISHED composes that include the container images.
17:06:23 <sgallagh> #info Composes have finally stabilized after the mass-rebuild
17:06:31 <davide> Yay
17:07:30 <jforbes> very nice
17:07:41 <tdawson> *cheers*
17:08:11 <sgallagh> The failed-build count continues to shrink, largely thanks to yselkowitz 's efforts
17:08:30 <sgallagh> #info Down to only 62 failed builds at the time of this meeting
17:08:44 <yselkowitz[m]> my view shows 50?
17:09:01 <sgallagh> Oops, I was including eln-extras\
17:09:08 <sgallagh> #undo
17:09:08 <zodbot> Removing item from minutes: INFO by sgallagh at 17:08:30 : Down to only 62 failed builds at the time of this meeting
17:09:28 <sgallagh> #info Down to only 50 failed builds at the time of this meeting!
17:09:29 <yselkowitz[m]> sorry haven't been watching eln-extras quite yet, focusing on eln proper
17:09:46 <sgallagh> Makes perfect sense.
17:09:50 <tdawson> I'll get the eln-extras, they are mainly KDE related.
17:10:06 <sgallagh> Thanks tdawson
17:10:06 <davide> I kept an eye on the stuff I added and it seems fine
17:10:07 <tdawson> Late time I looked there was only 3 failed eln-extras, so I thought we were good.
17:10:12 <yselkowitz[m]> and a good number of those 50 don't belong in ELN at all, trying to cut them out
17:10:22 <davide> But please let me know if you see any breakage from that
17:10:49 <sgallagh> #info Progress is being made on dropping unwanted packages from ELN
17:11:03 <sgallagh> Thank you, Davide
17:11:39 <sgallagh> Does anyone else have any ELN-related status to report?
17:11:56 <sgallagh> If not, I'll move to Open Floor in 120s
17:11:58 <davide> I'm making good progress on internal qualification
17:12:14 <davide> Hoping to talk about it at a conference later in the year
17:12:32 <davide> Possibly devconfcz if things line up right
17:13:08 <sgallagh> Can you describe for the log what "internal qualification" entails?
17:13:09 <davide> I've also filed a ticket on CentOS infra to investigate if we can build SIG content against ELN
17:13:44 <davide> Getting a small production deployment of ELN with rough feature parity with C9
17:14:06 <davide> With the goal of spotting issues early and doing less work by the time C10 drops
17:14:31 <davide> Right now it's single digit manually deployed hosts
17:14:46 <davide> Hoping to have automated provisioning later this month
17:14:57 <davide> And actual production workloads in March
17:15:28 <sgallagh> "production" as in actually handling real traffic?
17:15:34 <davide> Yep
17:15:40 <sgallagh> o.O
17:16:01 <sgallagh> That is both exciting and terrifying
17:16:16 <davide> That's the spirit :)
17:16:42 <sgallagh> Thank you for climbing that mountain
17:17:15 <sgallagh> I'll do what I can to supply belaying equipment 😀
17:17:26 <davide> Much appreciated
17:18:24 <sgallagh> #info Facebook is putting together a small production deployment of ELN to catch RHEL 10 issues early
17:18:45 <sgallagh> #topic Open Floor
17:20:33 <sgallagh> Anyone have anything they'd like to bring up today?
17:20:37 <yselkowitz[m]> yeah
17:20:46 <sgallagh> You have the floor
17:21:10 <yselkowitz[m]> wrt some of the remaining failures, a common thread has been test failures on ppc64le only in crypto-related packages
17:21:31 <sgallagh> Hmm
17:21:48 <yselkowitz[m]> I think we need to raise this up to someone that could look into that specifically
17:22:26 <Eighth_Doctor> hmm
17:22:35 <Eighth_Doctor> actually I have a question, related to the FB/Meta ELN 10 thing
17:22:52 <Eighth_Doctor> after yselkowitz :)
17:23:06 <Eighth_Doctor> (I'm just stating it now so I don't forget as I'm juggling three meetings)
17:23:18 <sgallagh> Thanks
17:23:35 <sgallagh> yselkowitz: So do you think we should bring it to the ppc64le folks or the crypto folks?
17:24:25 <yselkowitz[m]> ppc64le I guess to start with
17:25:06 <yselkowitz[m]> and maybe the tools team, since my guess is this is toolchain related somehow
17:25:23 <yselkowitz[m]> because we got a new gcc and compiler flag changes right before the mass rebuild, right?
17:25:36 <Eighth_Doctor> yes
17:25:52 <Eighth_Doctor> though usually those go wonky when they propagate to ELN
17:26:12 <Eighth_Doctor> at least most of the new flags were reverted for ELN, IIRC
17:26:15 <Eighth_Doctor> so we just have a new compiler
17:26:24 <sgallagh> OK, let's reach out to tstellar and see what he can do for us.
17:26:56 <yselkowitz[m]> ok thanks
17:27:50 <sgallagh> Conan Kudo: You're up
17:28:36 <Eighth_Doctor> so related to the thing Davide Cavalca brought up, I've wanted to see if we can do something about bringing in Hyperscale-related overlay content into ELN to validate
17:28:54 <Eighth_Doctor> I don't know how we'd do it, but I'd like to figure out some way to track it
17:29:18 <Eighth_Doctor> the most important one, to be honest, is the Hyperscale kernel
17:29:18 <Eighth_Doctor> which I'm the maintainer of
17:29:42 <davide> So my idea was to get ELN hooked up to CBS as another build target
17:30:04 <davide> Assuming that's feasible
17:30:22 <Eighth_Doctor> well the tricky problem is that the kernel is part of ARK
17:31:03 <Eighth_Doctor> and we have the "one-kernel-package" rule, and I haven't exactly gotten good feels for getting the Hyperscale config overlay added to the ARK sources
17:31:15 <Eighth_Doctor> so I don't really know how we're going to do it
17:31:22 <davide> Ah I see
17:31:27 <davide> Yeah that's tricky
17:32:49 <Eighth_Doctor> if I decide to ignore ELN, then I'd build some automation for periodically forking the ELN kernel package build, add our overlay, and build it in CBS in a separate tag
17:33:03 <Eighth_Doctor> but I'd actually rather have some way to continuously test it in ELN itself
17:33:40 <Eighth_Doctor> then we can have userspace + kernel stuff defined as an ELN workload
17:34:20 <Eighth_Doctor> I have a feeling this is unsolvable, but I wanted to bring it up if anyone has any ideas
17:36:25 * sgallagh waits for jforbes to say something :)
17:37:34 <jforbes> not sure what to say, the ark/eln kernel is what/where it is for a reason, and it heavily utilized
17:38:27 <Eighth_Doctor> I mean, at this point, I'm halfway to having scripts to automate regularly forking fedora/rhel kernel packages so if we can't do it in ELN, we'll just figure out some other way
17:39:03 <jforbes> why not just track it with a fork of kernel-ark?
17:39:22 <Eighth_Doctor> sure, I could, but I still need somewhere to build it :)
17:39:31 <jforbes> copr
17:39:47 * Eighth_Doctor shrugs
17:39:48 <Eighth_Doctor> I guess
17:40:13 <Eighth_Doctor> kernel package builds are really slow in copr compared to koji, but it's doable I guess
17:40:48 <jforbes> They aren't *that* bad in copr
17:41:20 <davide> Copr for prototyping seems reasonable
17:41:34 <davide> And if we hit a ceiling we can look again at alternatives
17:41:35 <Eighth_Doctor> arm is the only fast one now :)
17:41:41 <jforbes> The other option is koji scratch builds and set up a repository somewhere
17:42:03 <jforbes> Heh, arm is getting slower, we added debug builds for aarch64 meaning the build time is pretty much doubled
17:42:36 <Eighth_Doctor> yay :P
17:42:54 <jforbes> Which is why I flinched a bit when jlinton mentioned adding more arm variants.
17:43:01 <Eighth_Doctor> 🤣
17:43:11 <Eighth_Doctor> well, you know I want a 16k variant
17:43:17 <Eighth_Doctor> and apparently RHEL wants a 64k variant
17:43:34 <jforbes> and someone wants an RT variant once that is merged upstream
17:43:35 <Eighth_Doctor> and theoretically, we're going to see -rt variants for every variant coming real soon now(tm)
17:44:05 <Eighth_Doctor> the crystal ball seems to say 6.3~6.4, but I dunno if I believe it :)
17:44:11 <jforbes> And suddenly you have 8+ kernel builds for a single arch. Unless koji can learn to schedule those in parallel....
17:44:54 <jforbes> Oh, rt will not get a build per variant. Even if we did add other variants for page sizes, only one would get an RT
17:45:22 <jforbes> Well, with ELN maybe 2, but highly doubtful ELN is getting any sort of 16K variant any time soon
17:46:52 <Eighth_Doctor> sure, 16k would probably be Fedora-only
17:47:18 <Eighth_Doctor> but I saw the 64k variant merged in the CS9 kernel packaging and I'm just waiting for the MR to show up for ARK now
17:48:25 <jforbes> Even with that, it would be ELN and not Fedora at this point
17:49:32 <Eighth_Doctor> so then 16k for fedora and 64k for rhel :P
17:50:36 <jforbes> As I haven't even gotten around to buying a mac mini to use here, I am not highly motivated there either just yet
17:50:46 * Eighth_Doctor shrugs
17:53:05 <sgallagh> OK, we're coming to the end of the hour. It doesn't sound like this topic is moving forward today
17:53:09 <Eighth_Doctor> anyway, it just means yet another kernel tree for me
17:53:09 <tdawson> Since we are running short on time, I have just one thing.
17:53:22 <sgallagh> Go ahead, tdawson
17:53:35 <tdawson> Due to the s390x builder outage, I won't be getting to any of the alt-extra failed builds until next week.
17:53:59 <tdawson> That's it for me.
17:54:54 <sgallagh> not a problem
17:55:09 <sgallagh> OK, if that's everyone, I'll declare this meeting over.
17:55:15 <sgallagh> Thank you for coming!
17:55:22 <jforbes> thanks sgallagh
17:55:25 <davide> Thanks!
17:55:28 <sgallagh> #endmeeting