17:01:15 <sgallagh> #startmeeting eln
17:01:15 <zodbot> Meeting started Fri Jan 13 17:01:15 2023 UTC.
17:01:15 <zodbot> This meeting is logged and archived in a public location.
17:01:15 <zodbot> The chair is sgallagh. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
17:01:15 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
17:01:15 <zodbot> The meeting name has been set to 'eln'
17:01:37 <sgallagh> #meetingname eln
17:01:37 <zodbot> The meeting name has been set to 'eln'
17:01:50 <sgallagh> #topic Init Process
17:01:56 <sgallagh> .hi
17:01:57 <zodbot> sgallagh: sgallagh 'Stephen Gallagher' <sgallagh@redhat.com>
17:02:00 <tdawson> Hello, I made it this time.
17:02:09 <davide> .hello dcavalca
17:02:10 <zodbot> davide: dcavalca 'Davide Cavalca' <davide@cavalca.name>
17:02:29 <yselkowitz[m]> .hello yselkowitz
17:02:30 <zodbot> yselkowitz[m]: yselkowitz 'Yaakov Selkowitz' <yselkowi@redhat.com>
17:03:44 <sgallagh> Welcome to the first ELN meeting of 2023!
17:04:27 <sgallagh> I have one topic on the agenda for today, then Open Floor.
17:04:42 <sgallagh> #topic Frame Pointers in ELN/RHEL
17:05:34 <sgallagh> If anyone here has not heard, FESCo decided that Fedora would start compiling with -fno-omit-frame-pointers on x86_64 and aarch64 systems.
17:07:43 <sgallagh> I spoke with folks inside RHEL and was informed that the official stance is that Red Hat will not be enabling frame pointers at this time for RHEL (and RHEL 10 in particular)
17:08:14 <tdawson> So our compilers are going to diverge more from Fedora.
17:09:02 <sgallagh> This decision may be revisited if new information (including results from Fedora) comes to light, but as of now we can assume that RHEL 10 will not ship with them.
17:09:35 <davide> fwiw, as one of the Change owners I fully support the idea of looking at how this works out in Fedora first
17:09:45 <davide> as much as I'd personally like to eventually get this enabled for RHEL / CentOS Stream as well
17:09:57 <dtometzki> what are the reason for -fno-omit-frame-pointers ?
17:09:59 <sgallagh> tdawson: To be clear, our (ELN) compiler flags will diverge from Rawhide to stay closer to RHEL
17:10:16 <davide> dtometzki: see https://fedoraproject.org/wiki/Changes/fno-omit-frame-pointer
17:10:28 <sgallagh> dtometzki: Forgive me if I'm brief, as there are literally dozens of replies on Fedora Devel debating the use of them.
17:10:29 <tdawson> Yep ... understood.
17:11:36 <sgallagh> Short version:  They make code profiling easier/possible at the expense of an as-yet indeterminate performance impact. (Likely to be < 2% in most cases, but some specific places like Python have had outsized impact)
17:14:11 <sgallagh> That's pretty much all I had on that topic. Anyone have anything more here?
17:15:06 <davide> I saw this morning there was some fallout from the gating: https://bugzilla.redhat.com/show_bug.cgi?id=2160716
17:15:30 <sgallagh> Ah, yeah. That was fun :-/
17:15:35 <davide> but it looks like it's getting sorted out already: https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/239
17:15:46 <sgallagh> The implementation stumbled over some deep RPM esoterica
17:15:46 <davide> those macros are nasty
17:16:09 <sgallagh> But it looks like Miro has (once again) magicked us out of it
17:17:16 <sgallagh> #topic Open Floor
17:18:01 <yselkowitz[m]> there are currently 12 build failures (and 1 package waiting for automation), with PRs to handle 4 of those open
17:18:42 <yselkowitz[m]> mass rebuild starts on Wednesday, which will require close monitoring
17:19:20 <tdawson> Ya!!  mass rebuilds are always fun. :)
17:20:04 <sgallagh> Oh, related: I bumped the ELN buildroot version to 125 when we (re-)dropped the frame pointer flags.
17:20:52 <sgallagh> So everything built during the mass-rebuild (or since yesterday, really) should be identifiable by the .eln125 suffix
17:23:41 <sgallagh> One other topic came up a few minutes ago that we can discuss in the meeting
17:24:01 <sgallagh> #topic Content Resolver workload errors
17:24:39 <sgallagh> mhroncok: This probably will interest you.
17:25:04 <sgallagh> Miro filed https://github.com/fedora-eln/eln/issues/108#issuecomment-1382149185 today, which has me concerned.
17:25:18 * mhroncok reads up
17:25:35 <sgallagh> It seems that some packages are disappearing from the Content Resolver lists that should still be there
17:25:48 <sgallagh> mhroncok: There's no prior history; I invited you before discussing
17:26:31 <sgallagh> My current suspicion is that this is fallout from the fact that we have not been diligent in monitoring the failed workloads in CR
17:26:37 <sgallagh> #link https://tiny.distro.builders/view-errors--view-eln.html
17:26:41 <mhroncok> there were some nice things said about me even before that here :D
17:27:35 <mhroncok> sgallagh: I don't see the connection here, sorry
17:27:37 <tdawson> Are we still untagging packages that disapear from the Content Resolver list?
17:27:58 <mhroncok> the python workload seesm to be OK and some transitive builddeps are gone
17:27:58 <sgallagh> tdawson: That's Part Two of the problems.
17:27:59 <sgallagh> It seems that some of them have gotten missed.
17:28:02 <mhroncok> s/seesm/seems/
17:28:34 <mhroncok> tl;dr some package should be in ELN and were neither removed not rebuilt and are missing from tiny distro builders
17:28:51 <mhroncok> situation normal :D
17:28:55 <sgallagh> Right, and this is Bad™
17:29:21 <sgallagh> I'm going to dig into why they weren't properly untagged when they disappeared
17:29:32 <sgallagh> That's one of the DistroBuildSync functions
17:30:19 <sgallagh> tdawson, yselkowitz Could you investigate on the CR side why those deps fell out in the first place?
17:31:20 <yselkowitz[m]> might take me some time to catch up on the history there
17:31:44 <sgallagh> Right, I'm going to ask tdawson to mentor you at the same time.
17:31:58 <sgallagh> 🐦️🐦️ 🪨
17:32:19 <tdawson> Specifially (so we have some place to start) are we talking about python3-tox-current-env and python3-hatchling ?
17:33:17 <sgallagh> Yeah, those are a good place to start
17:34:46 <sgallagh> #action sgallagh to track down why DistroBuildSync isn't fully untagging dropped packages
17:34:46 <mhroncok> The --whatrequires python3-hatchling results are empty
17:34:55 <mhroncok> becasue there is no such package in the queried repos
17:35:07 <yselkowitz[m]> python-tox just had a major version bump, looks like hatchling is just a new dep
17:35:12 <mhroncok> but in fact there are packages that buildrequrie it trough a virtual provide
17:35:26 <mhroncok> it's not just required by tox
17:35:31 <sgallagh> #action tdawson and yselkowicz to figure out why Content Resolver is missing some required packages.
17:37:01 <yselkowitz[m]> the previous version shows no mention of hatchling in either SRPM or RPM: https://koji.fedoraproject.org/koji/buildinfo?buildID=2101750
17:37:39 <sgallagh> That's the EPEL version, FTR
17:37:47 <yselkowitz[m]> oops, but no difference: https://koji.fedoraproject.org/koji/buildinfo?buildID=2101748
17:39:09 <sgallagh> OK, we should probably take this to #eln:fedoraproject.org
17:39:16 <sgallagh> No need to keep it all in the meeting notes.
17:40:12 <sgallagh> OK, anyone have any other topics, or shall we end the meeting?
17:40:28 <davide> I have a couple of things
17:41:19 <davide> I'm going to spend some time this half integrating ELN in our internal provisioning flows, with the goal of deploying in on a small %age of production machines as way of doing ongoing qualification for the next CentOS Stream major release
17:41:38 <davide> this has already paid dividends after two days of work: https://github.com/rpm-software-management/rpm/issues/2351 :p
17:42:34 <sgallagh> Oh, nice
17:42:34 <davide> one thing that I wanted to discuss whether it would be possible to add systemd-extras to ELN Extras
17:42:46 <davide> as I need systemd-networkd for this to work, and we don't ship it in ELN
17:43:24 <sgallagh> Anyone can request a package added to ELN Extras, you just have to send an MR to Content Resolver adding it
17:43:38 <sgallagh> (Doing so makes you the maintainer in ELN, of course)
17:43:57 <davide> yeah, I was more wondering what the best way to maintain this would be
17:44:11 <davide> systemd-extras is an EPEL-only package, it doesn't have a rawhide branch
17:44:35 <davide> so I'm not entirely sure how it'd work with ELN, we'd effectively need a branch tracking rawhide dedicated for ELN
17:45:09 <davide> https://src.fedoraproject.org/rpms/systemd-extras for reference
17:45:14 <tdawson> Is there a reason we don't have it in rawhide?
17:45:33 <sgallagh> tdawson: Because in Fedora its subpackages are provided by the `systemd` RPM
17:45:42 <sgallagh> s/RPM/SRPM/
17:45:48 <davide> yeah, this package exists solely to provide missing subpackages in RHEL
17:46:00 <tdawson> Ahhh
17:46:57 <sgallagh> Davide Cavalca: The simplest answer is to just create an `eln` branch for `systemd-extras` and maintain it manually
17:47:14 <sgallagh> Similar to how we handle crypto-policies since they diverge so significantly
17:47:25 <jforbes> My experience only goes the other way, there are things which are separate packages in Fedora, but are all part of the kernel package in ELN. I handle the builds by hand on it all. I assume adding an ELN branch would work the same
17:47:44 <tdawson> Actually ... only 1 of the 3 sub-packages are in Fedora.  systemd-boot and systemd-timesyncd are not in Fedora.
17:47:58 <davide> ah excellent, a dedicated branch would for me, I just couldn't find an existing example
17:48:14 <davide> tdawson: systemd packaging is... complicated
17:48:27 <davide> those components are shipped in Fedora, but they're structured differently
17:48:34 <tdawson> Oh .. ok
17:49:02 <tdawson> Then I agree with sgallagh ... do an eln branch.
17:49:40 <davide> I will give this a try and report back in case there's any issues, thanks folks
17:49:51 <tdawson> The eln branches aren't auto-built though ... just want to emphasise that sgallagh said "maintain it manually"
17:50:48 <sgallagh> I guess in theory I could make it trigger off the EPEL9 build, but I don't think that's a good precedent to set
17:51:06 <sgallagh> Especially since they will likely diverge
17:51:16 <sgallagh> So yeah, pretend I didn't suggest that :)
17:51:41 <davide> yeah, if anything this should probably be hooked to updates in Rawhide to systemd proper
17:52:17 <davide> is it just a matter of doing a build with fedpkg from the eln branch?
17:52:25 <sgallagh> Yep
17:52:25 <davide> I assume we're doing submitting updates for ELN
17:52:44 <sgallagh> Bodhi handles ELN identically to how it handles Rawhide
17:52:44 <davide> * we're not submitting, i.e. it works like Rawhide
17:52:52 <davide> perfect, thanks for clarifying
17:53:09 <davide> ok, I will coordinate with the maintainer then
17:53:38 <sgallagh> Oh, while you're doing so, can you try to figure out why the systemd package is FTBFS on ELN right now? :)
17:53:56 <sgallagh> https://koji.fedoraproject.org/koji/buildinfo?buildID=2109438
17:54:14 <davide> yeah sure, I can try and take a look
17:54:28 <sgallagh> Thanks, I've been asking Zbigniew, but I think he's too swamped at the moment.
17:55:31 <davide> that's all I had for today
17:57:53 <tdawson> Nothing else from me
17:58:46 <sgallagh> #endmeeting