<@sgallagh:fedora.im>
17:00:25
!startmeeting Fedora ELN (2024-01-26)
!meetingname eln
!topic Init Process
<@meetbot:fedora.im>
17:00:26
Meeting started at 2024-01-26 17:00:25 UTC
<@meetbot:fedora.im>
17:00:26
The Meeting name is 'Fedora ELN (2024-01-26) !meetingname eln !topic Init Process'
<@sgallagh:fedora.im>
17:00:40
!hi
<@zodbot:fedora.im>
17:00:41
Stephen Gallagher (sgallagh) - he / him / his
<@tdawson:fedora.im>
17:00:53
Hello
<@salimma:fedora.im>
17:00:54
!hi
<@zodbot:fedora.im>
17:00:55
Michel Lind (salimma) - he / him / his
<@davide:cavalca.name>
17:01:21
!hi
<@zodbot:fedora.im>
17:01:23
Davide Cavalca (dcavalca) - he / him / his
<@nhanlon:beeper.com>
17:02:24
!hi
<@zodbot:fedora.im>
17:02:25
Neil Hanlon (neil) - he / him / his
<@nhanlon:beeper.com>
17:02:27
g'day folks
<@nhanlon:beeper.com>
17:02:33
happy friday, etc etc
<@sgallagh:fedora.im>
17:02:39
Greetings my friends!
<@sgallagh:fedora.im>
17:02:47
Greetings, my friends!
<@yselkowitz:fedora.im>
17:02:50
!hi
<@zodbot:fedora.im>
17:02:51
Yaakov Selkowitz (yselkowitz)
<@sgallagh:fedora.im>
17:03:31
Alright, I think that's everyone that I expected to see here today, so let's get started!
<@sgallagh:fedora.im>
17:03:38
!topic Mass Rebuild Status
<@sgallagh:fedora.im>
17:04:03
So, this cycle's mass-rebuild has been... difficult
<@sgallagh:fedora.im>
17:04:54
You have probably seen the announcement that Fedora is delaying the start of the F40 Branch and all other dev milestones up to and including the start of the Beta Freeze
<@sgallagh:fedora.im>
17:05:29
We (ELN) were among those that requested this delay, because our mass-rebuild only triggers when the Fedora mass-rebuild is tagged back into Rawhide.
<@salimma:fedora.im>
17:05:33
is it mostly binutils/gcc or is there other difficulties?
<@salimma:fedora.im>
17:05:43
(not python like last cycle I hope)
<@sgallagh:fedora.im>
17:06:03
Our estimates were that there was not going to be sufficient time before the branching date to clean things up in Fedora before EL10 breaks inheritance from Rawhide.
<@tdawson:fedora.im>
17:06:23
There was some authentication issues
<@salimma:fedora.im>
17:06:31
yeah the auth lost us over 2 days :(
<@sgallagh:fedora.im>
17:06:37
There were issues with the GNU Toolchain as well as FAS
<@salimma:fedora.im>
17:06:43
do we have a list of packages we need to prioritize fixing in Fedora/ELN?
<@yselkowitz:fedora.im>
17:06:52
yes, https://github.com/fedora-eln/eln/issues/176
<@salimma:fedora.im>
17:07:00
awesome
<@sgallagh:fedora.im>
17:07:15
Michel Lind🎩: https://tiny.distro.builders/view--view-eln.html ? 😛
<@salimma:fedora.im>
17:07:17
the provenpackagers should be able to chip in. I'll try to prioritize this when fixing
<@salimma:fedora.im>
17:07:45
when is the deadline? I'll be free-er after CentOS Connect next Friday
<@sgallagh:fedora.im>
17:07:50
It would be fantastic to have some help with the cleanups, but we probably want to figure out how to coordinate so we don't end up doubling-up
<@salimma:fedora.im>
17:08:17
load the broken list on a spreadsheet, people put their names to the ones they are fixing?
<@sgallagh:fedora.im>
17:08:18
Michel Lind🎩: Fedora 40 Branching, rescheduled for Feb. 13
<@salimma:fedora.im>
17:08:26
feb 13 should work
<@sgallagh:fedora.im>
17:08:31
So really, Feb 12, since the Branch could occur at 00:00 Feb 13
<@salimma:fedora.im>
17:08:49
with fosdem travel it's probably easier to use the spare time to fix packages than doing focus work anyway
<@salimma:fedora.im>
17:08:53
ack, 12
<@yselkowitz:fedora.im>
17:08:58
which means a fix build through CI gating in rawhide before then
<@tdawson:fedora.im>
17:08:58
Note that the mass rebuild is not finished. There are still 6500 packages to go ... and at the rate they are going ... that might take until Monday.
<@sgallagh:fedora.im>
17:09:25
Troy Dawson: I've heard rumblings that the failures may be resubmitted as well.
<@salimma:fedora.im>
17:09:41
yeah. I've been prioritizing fixing my packages that I know will not succeed in the mass rebuild this week, to help get them out of the way
<@sgallagh:fedora.im>
17:09:44
Since some internal compiler error issues have been found and fixed since.
<@salimma:fedora.im>
17:09:50
I've definitely seen some that got submitted twice
<@tdawson:fedora.im>
17:09:58
It seems they are doing them in batches of 40 or so ... possibly 50, but I've never seen the count that high.
<@sgallagh:fedora.im>
17:10:21
Michel Lind🎩: If they're early in the alphabet, they may have been resubmitted because of the binutils issue early on
<@tdawson:fedora.im>
17:10:40
Usually they just slam koji and let koji work it's way through them.
<@salimma:fedora.im>
17:10:45
Stephen Gallagher: in my case it's the folly stack. which has tight coupling between the packages :)
<@smooge:fedora.im>
17:10:45
that sounds right.. doing more than that will just end up with stuff really piling up due to the lack of equal build speeds
<@sgallagh:fedora.im>
17:11:19
Stephen J Smoogen: Did they change the approach this cycle?
<@smooge:fedora.im>
17:12:02
I think I read something in one of the channels but since my matrix doesn't search well and I don't have copies of on my system... assume my memory is bad
<@smooge:fedora.im>
17:12:26
I think they altered the batch size to try and deal with an issue but I am not sure.
<@tdawson:fedora.im>
17:12:57
Well ... whatever they did, they changed the time from 2 days to 7.
<@smooge:fedora.im>
17:13:15
well there are some other issues which have slowed things down
<@nhanlon:beeper.com>
17:13:26
I'm down to help with cleanup
<@tdawson:fedora.im>
17:13:32
On the flip side, as someone said above, it gives people a chance to fix they failed builds, as long as they are early in the alphabet.
<@smooge:fedora.im>
17:13:52
there are things which need to be done in a build which talk to bodhi and other parts of the build system. Anything which affects user lookups etc is going to cause slowdowns
<@sgallagh:fedora.im>
17:14:48
Is FAS still running off a single FreeIPA node, or did the replication get fixed?
<@nhanlon:beeper.com>
17:15:11
replication is fixed aiui
<@salimma:fedora.im>
17:16:21
Bodhi's new update page has been timing out all morning for me today (funnily viewing updates is working fine)
<@sgallagh:fedora.im>
17:16:22
OK, let's proceed assuming that the mass-rebuild will be finished sometime next week.
<@salimma:fedora.im>
17:16:40
yeah, how do we want to coordinate this? I think that's the main issue now
<@sgallagh:fedora.im>
17:16:48
I was just about to say that
<@sgallagh:fedora.im>
17:17:16
Trying to work from a shared spreadsheet is one approach, but it might just be better to work in the ticket
<@nhanlon:beeper.com>
17:17:34
https://framacalc.org/abc/en/ if we go the spreadsheet route
<@salimma:fedora.im>
17:18:06
<3 Frama and I have not tried this one yet
<@sgallagh:fedora.im>
17:18:08
As yselkowitz has been leading this effort, I'll defer to him for what he wants to do
<@salimma:fedora.im>
17:18:11
but ticket is fine
<@yselkowitz:fedora.im>
17:18:34
I don't want to have everyone need to sign up for a new service just to track this
<@yselkowitz:fedora.im>
17:18:50
a google spreadsheet works for me, or we use the github ticket
<@salimma:fedora.im>
17:19:23
there's nothing private about this so yeah either works
<@salimma:fedora.im>
17:19:29
I'm sure Google already knows what I do ;)
<@sgallagh:fedora.im>
17:19:29
Michel Lind🎩: It won't be what they say so much as *how* they say it. Probably delivered by arrow.
<@salimma:fedora.im>
17:19:39
with a poisoned tip
<@sgallagh:fedora.im>
17:21:21
Any strong preferences?
<@nhanlon:beeper.com>
17:21:32
nope, whatever is least resistance
<@salimma:fedora.im>
17:22:25
weak preference for spreadsheet
<@sgallagh:fedora.im>
17:22:33
OK, a spreadsheet might indeed be easiest, as I think on it. That way the name of who is working on it is closely connected to the package.
<@salimma:fedora.im>
17:22:33
scales better with a lot of packages
<@sgallagh:fedora.im>
17:22:38
Vs. comments in the ticket
<@salimma:fedora.im>
17:22:55
yeah and maybe have a status column so we can track how close it is (if someone is stuck I'm sure they could use help)
<@sgallagh:fedora.im>
17:23:17
One request: if you start working on something, but then need to abandon it, PLEASE make that statement loudly.
<@nhanlon:beeper.com>
17:23:18
framacalc fwiw doesn't seem to require registration, sorta just like an Etherpad or so
<@nhanlon:beeper.com>
17:23:31
and COLORS!
<@sgallagh:fedora.im>
17:24:07
Let's not try to get too fancy. I don't want people spending more than 15s updating the spreadsheet. There's plenty of better things to be doing.
<@nhanlon:beeper.com>
17:24:37
agreed, only mostly joking about the colors thing :P
<@yselkowitz:fedora.im>
17:25:25
well if it doesn't require signup then no objections to trying framacalc
<@sgallagh:fedora.im>
17:26:31
yselkowitz: Can I count on you to populate the list of packages requiring attention?
<@yselkowitz:fedora.im>
17:26:49
yes I already have that in csv form too
<@sgallagh:fedora.im>
17:27:06
yselkowitz++
<@zodbot:fedora.im>
17:27:08
sgallagh gave a cookie to yselkowitz. They now have 8 cookies, 2 of which were obtained in the Fedora 39 release cycle
<@sgallagh:fedora.im>
17:27:38
!action yselkowitz will populate a framacalc spreadsheet with the packages requiring attention post-mass-rebuild
<@sgallagh:fedora.im>
17:28:20
!info Any interested person should feel welcome to jump in and help, instructions on how will be posted to https://github.com/fedora-eln/eln/issues/176
<@sgallagh:fedora.im>
17:29:12
Let's move on to the branching discussion
<@sgallagh:fedora.im>
17:29:21
!topic EL10 Branching
<@sgallagh:fedora.im>
17:29:29
First, a quick status update.
<@sgallagh:fedora.im>
17:30:08
!info sgallagh has updated https://docs.fedoraproject.org/en-US/eln/branching/ with the operations required at branch-time
<@sgallagh:fedora.im>
17:30:37
This now covers both normal Fedora branching and the special events that take place when we branch a new EL release
<@sgallagh:fedora.im>
17:31:37
It covers the parts needed on the Fedora side. The CentOS Stream side will need its own document, but right now that's mostly internal at RH because all of the infrastructure access is behind the RH firewall
<@salimma:fedora.im>
17:31:44
well I'm happy to add color and validation if nobody else is doing it ;)
<@sgallagh:fedora.im>
17:31:45
That doc will be forthcoming
<@sgallagh:fedora.im>
17:32:16
Michel Lind🎩: The only validation we need is telling each other we're doing a great job :)
<@sgallagh:fedora.im>
17:32:58
Now, one thing I realized yesterday that we haven't talked about or, indeed, given enough thought to around EL10: EPEL
<@sgallagh:fedora.im>
17:33:43
We've had 35,000 foot plans in place to fork ELN-Extras into EPEL for RHEL 10 for a while, but it occurred to me that we haven't actually identified the process for how to do that.
<@tdawson:fedora.im>
17:34:18
Oh ... I thoguht that was something that EPEL would take care of.
<@sgallagh:fedora.im>
17:34:23
Obviously, what we decide will need to be coordinated with the EPEL SIG, so I'm going to politely ask Troy Dawson to be our liaison here :)
<@tdawson:fedora.im>
17:34:46
Right now we don't have epel10 branching ability ...
<@sgallagh:fedora.im>
17:34:57
I have no objections to the EPEL folks taking point on this, but I want to make sure we know what's happening in that space
<@zodbot:fedora.im>
17:35:12
salimma has already given cookies to sgallagh during the F39 timeframe
<@tdawson:fedora.im>
17:35:17
Meaning fedpkg branch-release epel10 will not work .... I think
<@salimma:fedora.im>
17:35:58
we have been talking about potentially allowing certain packages to be pre-branched for epel10 but haven't come to a consensus I think
<@tdawson:fedora.im>
17:35:59
Carl has a presentation on it at CentOS Connect ... we can ask him if he's got that in place yet.
<@salimma:fedora.im>
17:36:12
but yeah maybe limiting it to pacakges already in ELN Extras is a nice compromise
<@salimma:fedora.im>
17:36:29
the one we talked about is more wide ranging - if it's in EPEL X we branch it for X+1
<@tdawson:fedora.im>
17:36:43
Ya, I believe we said something along that lines. If you want it pre-branched, but it in eln-extras
<@salimma:fedora.im>
17:36:50
thing is.. we don't need just the packages directly listed right? we need the transitive dependencies too that are not in the base EL
<@sgallagh:fedora.im>
17:36:51
Michel Lind🎩: This was always the stated intent of ELN-Extras.
<@sgallagh:fedora.im>
17:36:59
That it's basically proto-EPEL
<@salimma:fedora.im>
17:37:02
yeah
<@salimma:fedora.im>
17:37:26
as you said, it's been 30k feet high and we have not previously discussed the specific details of how to tie those two :)
<@sgallagh:fedora.im>
17:37:26
Right, and Content Resolver helps with that, fortunately.
<@tdawson:fedora.im>
17:38:03
The problem is that ELN branches before epel10 even has a buildroot to test with ... well ... maybe not, but I'm pretty sure it's not setup yet.
<@salimma:fedora.im>
17:38:05
does someone have a good writeup for that? e.g. 'given this workload definition, where can I see the packages it pulls in too'
<@sgallagh:fedora.im>
17:38:58
Troy Dawson: Ideally, we want EPEL to branch from ELN-Extras at the same time as CS10 from ELN
<@sgallagh:fedora.im>
17:39:17
Because otherwise, they may start picking up changes intended for F41
<@yselkowitz:fedora.im>
17:39:50
or we wait a bit and branch off of f40
<@yselkowitz:fedora.im>
17:40:12
since presumably it will take some time before epel10 can start being built?
<@tdawson:fedora.im>
17:40:36
Ya ... I'm pretty sure it's not going to be setup in two weeks ... but I do like yselkowitz idea of branching from f40
<@sgallagh:fedora.im>
17:40:56
Actually, yeah. I hadn't thought about that, but you're right
<@sgallagh:fedora.im>
17:41:10
That's probably our best move
<@salimma:fedora.im>
17:42:04
when I bring in a new package - unless I'm sure it's safe I tend to branch to epel from the latest branched release, yeah
<@sgallagh:fedora.im>
17:43:52
OK, so for packages that don't have a separate `eln` branch, we can branch from F40
<@tdawson:fedora.im>
17:44:11
Sounds good to me.
<@tdawson:fedora.im>
17:44:27
I would have that nothing in -extras has an ELN branch.
<@sgallagh:fedora.im>
17:44:29
Those few that do have an `eln` branch might require some additional attention, but we can probably branch from there (or some point in its history)
<@sgallagh:fedora.im>
17:44:38
"There" being the `eln` branch
<@tdawson:fedora.im>
17:44:40
I would hope that nothing in -extras has an ELN branch.
<@sgallagh:fedora.im>
17:44:56
Oh, right. They wouldn't would they.
<@sgallagh:fedora.im>
17:45:04
Oh, right. They wouldn't, would they.
<@sgallagh:fedora.im>
17:45:10
Oh, right. They wouldn't, would they?
<@sgallagh:fedora.im>
17:45:34
Anyone opposed to that plan?
<@salimma:fedora.im>
17:45:40
seems solid
<@tdawson:fedora.im>
17:46:20
I'm ok with it. I'll bring it up in the EPEL meetings.
<@sgallagh:fedora.im>
17:46:55
!agreed Packages in ELN-Extras will branch for EPEL 10 from the `f40` branch at some point after we branch EL10.
<@sgallagh:fedora.im>
17:47:10
Thank you, Troy Dawson
<@zodbot:fedora.im>
17:47:18
sgallagh gave a cookie to tdawson. They now have 62 cookies, 6 of which were obtained in the Fedora 39 release cycle
<@sgallagh:fedora.im>
17:47:42
Does anyone else have a topic around the Branching?
<@sgallagh:fedora.im>
17:48:51
Alright, I'll take that as a no
<@sgallagh:fedora.im>
17:48:56
!topic Open Floor
<@sgallagh:fedora.im>
17:49:07
For the first time in a while, we have time for Open Floor!
<@yselkowitz:fedora.im>
17:49:14
F40FTBFS spreadsheet is ready: https://lite.framacalc.org/tgyaghuoob-a5pt
<@salimma:fedora.im>
17:49:33
woot
<@salimma:fedora.im>
17:49:43
I like how it defaults to Frencn
<@salimma:fedora.im>
17:49:47
I like how it defaults to French
<@yselkowitz:fedora.im>
17:49:49
I'm not entirely impressed with this framacalc, but we'll give it a try
<@salimma:fedora.im>
17:49:58
Allons enfants de la patriee.....
<@yselkowitz:fedora.im>
17:49:59
my French is too rusty unfortunately
<@salimma:fedora.im>
17:50:05
is it open to public? interesting
<@salimma:fedora.im>
17:50:37
bodhi is down today so might as well work on ELN :P
<@salimma:fedora.im>
17:51:05
I was just reciting the first line of the national anthem :D
<@sgallagh:fedora.im>
17:51:08
!link https://lite.framacalc.org/tgyaghuoob-a5pt
<@sgallagh:fedora.im>
17:53:52
I'll look into the libmodulemd issue with Petr.
<@sgallagh:fedora.im>
17:54:14
I haven't had a chance to review the patches in detail yet, but I've been following the discussion
<@yselkowitz:fedora.im>
17:54:29
was the nodejs20 ICE the same or different?
<@sgallagh:fedora.im>
17:54:47
Unclear yet; I'm going to try building it against the latest gcc and see if it just works
<@sgallagh:fedora.im>
17:55:04
Immediately after this meeting
<@sgallagh:fedora.im>
17:55:41
OK, five minutes left: any more topics?
<@tdawson:fedora.im>
17:55:53
Nothing from me
<@salimma:fedora.im>
17:56:37
not from me
<@sgallagh:fedora.im>
17:57:11
Alright, I'll close the meeting then. Thanks for coming!
<@sgallagh:fedora.im>
17:57:14
!endmeeting