2024-02-09 17:01:06 <@sgallagh:fedora.im> !startmeeting Fedora ELN (2024-02-09)
2024-02-09 17:01:07 <@meetbot:fedora.im> Meeting started at 2024-02-09 17:01:06 UTC
2024-02-09 17:01:07 <@meetbot:fedora.im> The Meeting name is 'Fedora ELN (2024-02-09)'
2024-02-09 17:01:09 <@sgallagh:fedora.im> !meetingname eln
2024-02-09 17:01:48 <@sgallagh:fedora.im> !topic Init Process
2024-02-09 17:01:50 <@sgallagh:fedora.im> !hi
2024-02-09 17:01:51 <@zodbot:fedora.im> Stephen Gallagher (sgallagh) - he / him / his
2024-02-09 17:03:24 <@smooge:fedora.im> hello
2024-02-09 17:03:30 <@tdawson:fedora.im> Howdy
2024-02-09 17:04:19 <@nhanlon:beeper.com> !hi
2024-02-09 17:04:21 <@zodbot:fedora.im> Neil Hanlon (neil) - he / him / his
2024-02-09 17:04:34 <@nhanlon:beeper.com> g'day folks
2024-02-09 17:05:11 <@sgallagh:fedora.im> Hello my friends!
2024-02-09 17:05:21 <@jonathanspw:fedora.im> !hi
2024-02-09 17:05:22 <@yselkowitz:fedora.im> !hi
2024-02-09 17:05:22 <@zodbot:fedora.im> Jonathan Wright (jonathanspw)
2024-02-09 17:05:23 <@zodbot:fedora.im> Yaakov Selkowitz (yselkowitz)
2024-02-09 17:05:23 <@sgallagh:fedora.im> And welcome to another edition of "What could possibly go wrong?"
2024-02-09 17:05:59 <@sgallagh:fedora.im> So, I'll jump right into our main topic for today
2024-02-09 17:06:54 <@sgallagh:fedora.im> !topic Forking EL 10
2024-02-09 17:07:23 <@sgallagh:fedora.im> (For those of you who watched "The Good Place"... that phrasing was intentional.)
2024-02-09 17:08:29 <@sgallagh:fedora.im> !info As previously announced, Tuesday February 13th marks the end of Fedora ELN syncing to CentOS Stream 10
2024-02-09 17:09:19 <@sgallagh:fedora.im> From the ELN side, we have a number of steps that will be taken to move on towards RHEL 11
2024-02-09 17:09:48 <@sgallagh:fedora.im> !link https://docs.fedoraproject.org/en-US/eln/branching/
2024-02-09 17:10:40 <@sgallagh:fedora.im> We are currently in the process of cleaning up the last remaining fallout from the F40 mass-rebuild.
2024-02-09 17:11:14 <@sgallagh:fedora.im> There are two main tickets for tracking these issues:
2024-02-09 17:11:20 <@sgallagh:fedora.im> !link https://github.com/fedora-eln/eln/issues/180
2024-02-09 17:11:36 <@sgallagh:fedora.im> !link https://github.com/fedora-eln/eln/issues/179
2024-02-09 17:11:54 <@sgallagh:fedora.im> Sorry, three:
2024-02-09 17:11:57 <@sgallagh:fedora.im> !link https://github.com/fedora-eln/eln/issues/176
2024-02-09 17:12:29 <@sgallagh:fedora.im> Those are "Packages in the ELN set that are FTBFS in Rawhide", "Packages in the ELN set that are FTBFS in CentOS Stream 10" and "Packages in the ELN set that are FTBFS in ELN",
2024-02-09 17:12:34 <@sgallagh:fedora.im> (respectively)
2024-02-09 17:13:05 <@sgallagh:fedora.im> In an ideal world, tickets 176 and 180 will be finished by Monday.
2024-02-09 17:13:15 <@sgallagh:fedora.im> In our world... we need help.
2024-02-09 17:15:57 <@nhanlon:beeper.com> i volunteer Troy
2024-02-09 17:16:03 <@sgallagh:fedora.im> yselkowitz: Could you run us through the non-leaf remainder?
2024-02-09 17:16:16 <@yselkowitz:fedora.im> from #176:
2024-02-09 17:16:33 <@yselkowitz:fedora.im> asio: waiting on a gcc update now in testing
2024-02-09 17:16:55 <@yselkowitz:fedora.im> atlas: is a mess with the new gcc, but we're looking to make it a leaf
2024-02-09 17:17:04 <@yselkowitz:fedora.im> gtk-doc: needs help
2024-02-09 17:17:16 <@yselkowitz:fedora.im> fortunately that's not gcc related :-)
2024-02-09 17:17:27 <@yselkowitz:fedora.im> https://gitlab.gnome.org/GNOME/gtk-doc/-/issues/150
2024-02-09 17:17:45 <@yselkowitz:fedora.im> openblas: https://github.com/OpenMathLib/OpenBLAS/issues/4475
2024-02-09 17:18:38 <@yselkowitz:fedora.im> perl-Tk: also a mess with the new gcc, and upstream is dead
2024-02-09 17:19:07 <@yselkowitz:fedora.im> squid is a leaf, so worst case we skip it
2024-02-09 17:19:24 <@yselkowitz:fedora.im> zziplib: https://bugzilla.redhat.com/show_bug.cgi?id=2256917 with links to github inside
2024-02-09 17:20:11 <@yselkowitz:fedora.im> from #180
2024-02-09 17:20:37 <@yselkowitz:fedora.im> jackson-*: something weird going on with (x)mvn there
2024-02-09 17:20:53 <@sgallagh:fedora.im> Do we have anyone in attendance who will admit to understanding maven?
2024-02-09 17:21:06 <@sgallagh:fedora.im> Because it's black magic as far as I'm concerned
2024-02-09 17:21:18 <@yselkowitz:fedora.im> jss has a PR, and the rrdtool issue is explained in the ticket, I won't rehash here
2024-02-09 17:21:36 <@nhanlon:beeper.com> I can try and poke at gtk-doc. it sounds like my brand of self flagellation. 
2024-02-09 17:21:52 <@yselkowitz:fedora.im> from #179:
2024-02-09 17:22:03 <@yselkowitz:fedora.im> thanks Neil Hanlon
2024-02-09 17:22:09 <@yselkowitz:fedora.im> from #179:
2024-02-09 17:22:15 <@sgallagh:fedora.im> !action Neil Hanlon to look into gtk-doc
2024-02-09 17:22:34 <@yselkowitz:fedora.im> gmp, libkcapi, and nettle have very odd fips-related errors only in c10s, hasn't been an issue in eln
2024-02-09 17:22:55 <@yselkowitz:fedora.im> libecpg and tftp appear to be race conditions in the makefiles
2024-02-09 17:23:07 <@sgallagh:fedora.im> Our suspicion on those is that it may be related to the kernel running on the build hosts being RHEL 8 rather than recent Fedora kernels
2024-02-09 17:23:20 <@sgallagh:fedora.im> those being gmp, libkcapi and nettle
2024-02-09 17:23:56 <@sgallagh:fedora.im> I'll come back around to that later; I have some thoughts for ELN11
2024-02-09 17:23:58 <@yselkowitz:fedora.im> most of the rest are test failures due to infrastructure pecularities (e.g. rhel 8 kernel) or flakiness, although retries haven't helped yet
2024-02-09 17:24:25 <@yselkowitz:fedora.im> tracker-miners needs a meson configure patch: https://gitlab.gnome.org/GNOME/tracker-miners/-/issues/300
2024-02-09 17:24:49 <@yselkowitz:fedora.im> I think that pretty much covers it, happy to answer any questions
2024-02-09 17:27:11 <@sgallagh:fedora.im> If anyone has cycles they can dig out of the couch cushions to help push on any of these, that would be immensely helpful
2024-02-09 17:30:19 <@sgallagh:fedora.im> So it goes...
2024-02-09 17:30:47 <@sgallagh:fedora.im> OK, I have one other minor topic that's mostly directed at the folks here who know something about Fedora Infrastructure
2024-02-09 17:31:01 <@sgallagh:fedora.im> !topic RHEL-like Koji Builders
2024-02-09 17:31:42 <@sgallagh:fedora.im> What would it take to add some Koji builder instances that share a configuration with CentOS Stream/RHEL for ELN?
2024-02-09 17:32:27 <@sgallagh:fedora.im> In the interest of "shifting left" and identifying failures earlier (such as those where tests might fail if the kernel on the builder is too old)
2024-02-09 17:33:02 <@sgallagh:fedora.im> I'm kind  of directing this question at Stephen J Smoogen, who is no longer on that team but who I hope still remembers a few things about it :)
2024-02-09 17:33:44 <@smooge:fedora.im> ugh. I am in a meeting but can help in a bit
2024-02-09 17:34:19 <@sgallagh:fedora.im> It's not urgent, just something I want to explore after we fork and can start looking forward to RHEL 11
2024-02-09 17:35:30 <@sgallagh:fedora.im> !topic Open Floor
2024-02-09 17:36:26 <@smooge:fedora.im> so the main issues would be needed to work through is that the RHEL builders are running RHEL in kernel land but the Fedora builders are running the Fedora kernel.
2024-02-09 17:37:28 <@smooge:fedora.im> if we skip a 1:1 kernel issue, what would be needed to put into koji configs outside of what is done already?
2024-02-09 17:38:13 <@sgallagh:fedora.im> Sorry, I don't understand "if we skip a 1:1 kernel issue".
2024-02-09 17:38:29 <@smooge:fedora.im> so I may have misread by what you meant as a kernel too old
2024-02-09 17:39:28 <@sgallagh:fedora.im> Our issue is that some packages have test suites (or `%configure` suites) that require kernel features that work on Fedora, but once we sync to CentOS Stream they fail
2024-02-09 17:39:39 <@smooge:fedora.im> are you meaning you would want specific builders which were RHEL only and a chain of builds (like koschei) done on them?
2024-02-09 17:40:33 <@sgallagh:fedora.im> We'd like to be able to catch those issues earlier (in Fedora ELN) since we won't turn on the sync for CentOS Stream 11 for at least a year.
2024-02-09 17:42:08 <@sgallagh:fedora.im> Also the feedback loop is much tigher when things fail in ELN vs. after the syncing to CS
2024-02-09 17:42:08 <@smooge:fedora.im> So I think you would need to work out if Fedora has the resources for builders which are not 'the same' for whatever arches you needed this built on. And i mean in physical hardware and mental space for kevin :)
2024-02-09 17:42:36 <@nhanlon:beeper.com> surely we can download more ram for kevin? :) 
2024-02-09 17:43:27 <@sgallagh:fedora.im> That's not how this works. That's not how _any_ of this works!
2024-02-09 17:44:54 <@sgallagh:fedora.im> OK, as previously noted, this is just an exploratory discussion right now.
2024-02-09 17:45:03 <@smooge:fedora.im> understood.
2024-02-09 17:45:53 <@sgallagh:fedora.im> And maybe the answer ends up being "turn on the CS sync earlier".
2024-02-09 17:46:04 <@smooge:fedora.im> I would say as an explatory discussion.. why use koji and why use Fedora space but instead use something in the cloud which rebuilds all the items in ELN for maybe one or two arches in a continuous method.
2024-02-09 17:47:03 <@smooge:fedora.im> a dedicated COPR or something similar tied with something like Koshei might get a proof of concept going 
2024-02-09 17:47:37 <@sgallagh:fedora.im> I don't think we need a continuous rebuild. That seems like overkill (and a lot of electricity)
2024-02-09 17:47:44 <@smooge:fedora.im> longer term something better in koji could be worked out as resources are made available
2024-02-09 17:47:52 <@smooge:fedora.im> well even a once a week build
2024-02-09 17:49:15 <@smooge:fedora.im> you can then set things up in the 'hardware' for what you want really tested as in 'my build system's libs needs work'
2024-02-09 17:49:31 <@sgallagh:fedora.im> I do have one more thing to bring up before we run out of time.
2024-02-09 17:49:39 <@sgallagh:fedora.im> !topic OpenSSL 3.2
2024-02-09 17:50:15 <@sgallagh:fedora.im> As some of you may have seen, OpenSSL 3.2 hit Fedora Rawhide yesterday. We already have at least two reports of breakage there.
2024-02-09 17:51:10 <@sgallagh:fedora.im> I'd like to recruit the members of this SIG to try and test as many things as possible in Rawhide that rely on OpenSSL today.
2024-02-09 17:52:04 <@sgallagh:fedora.im> Currently, we've blocked it from landing in Fedora ELN out of an abundance of caution right before the fork, but if we can get to a high level of confidence in it by Monday, we might consider letting it in
2024-02-09 17:53:54 <@sgallagh:fedora.im> Can each of you spare 30 minutes or more today to poke at things in Rawhide?
2024-02-09 17:55:41 <@sgallagh:fedora.im> OK, last item of business: next meeting.
2024-02-09 17:55:53 <@sgallagh:fedora.im> !topic Next Meeting
2024-02-09 17:56:48 <@sgallagh:fedora.im> I will be unable to attend the next meeting (two weeks from today), but given the big happenings over that period, I'd appreciate if someone else could step up and hold it in my absence.
2024-02-09 17:57:13 <@tdawson:fedora.im> The next meeting is Feb. 23, correct?
2024-02-09 17:57:22 <@sgallagh:fedora.im> Correct
2024-02-09 17:57:27 <@tdawson:fedora.im> That is a Red Hat Recharge day
2024-02-09 17:57:54 <@yselkowitz:fedora.im> oh that's funny timing
2024-02-09 17:58:08 <@smooge:fedora.im> I would say do the next week
2024-02-09 17:58:10 <@tdawson:fedora.im> I'm not saying we need to cancel it ... I just noticed it last night.
2024-02-09 17:58:20 <@smooge:fedora.im> I think this is a definite 'cancel'
2024-02-09 17:58:33 <@smooge:fedora.im> but I have the luxury of being able to say that :)
2024-02-09 17:59:47 <@sgallagh:fedora.im> Hmm, I need to fix my PTO; I must have put it in before they picked that date because it's costing my a PTO day...
2024-02-09 18:00:09 <@sgallagh:fedora.im> Anyway, I guess we'll skip and consider having one the following week based on interest
2024-02-09 18:00:49 <@sgallagh:fedora.im> !info There will be no meeting on Feb. 23rd. We may have an out-of-schedule meeting on Mar. 1 if interest is high.
2024-02-09 18:00:58 <@sgallagh:fedora.im> Alright, we're at time. Thanks for coming, folks!
2024-02-09 18:01:03 <@sgallagh:fedora.im> !endmeeting