16:00:10 <sgallagh[m]> #startmeeting Fedora ELN SIG (2021-07-02)
16:00:10 <zodbot> Meeting started Fri Jul  2 16:00:10 2021 UTC.
16:00:10 <zodbot> This meeting is logged and archived in a public location.
16:00:10 <zodbot> The chair is sgallagh[m]. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:10 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:10 <zodbot> The meeting name has been set to 'fedora_eln_sig_(2021-07-02)'
16:00:10 <sgallagh[m]> #meetingname eln
16:00:10 <zodbot> The meeting name has been set to 'eln'
16:00:10 <sgallagh[m]> #chair sgallagh bookwar dcavalca
16:00:10 <sgallagh[m]> #topic init process
16:00:10 <zodbot> Current chairs: bookwar dcavalca sgallagh sgallagh[m]
16:00:28 <sgallagh[m]> .hello sgallagh
16:00:29 <zodbot> sgallagh[m]: sgallagh 'Stephen Gallagher' <sgallagh@redhat.com>
16:00:38 <jforbes> .hello2
16:00:38 <zodbot> jforbes: jforbes 'Justin M. Forbes' <jforbes@redhat.com>
16:00:39 <mhroncok> .hello churchyard
16:00:42 <zodbot> mhroncok: churchyard 'Miro Hrončok' <mhroncok@redhat.com>
16:01:02 * mhroncok decided to show in case we'll discuss the build order proposal
16:01:12 <mhroncok> *to show up
16:01:24 <sgallagh[m]> It's on the agenda, so yes we will discuss it :)
16:01:55 <King_InuYasha> .hello ngompa
16:01:56 <zodbot> King_InuYasha: ngompa 'Neal Gompa' <ngompa13@gmail.com>
16:02:31 <sgallagh[m]> Greetings, your majesty.
16:03:38 <sgallagh[m]> OK, let's get rolling
16:03:58 <sgallagh[m]> #topic ELN package rebuild strategy
16:04:06 <sgallagh[m]> #link https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/VYTDBTSCWUNS732XXVVWXL7S235AEZKG/
16:04:11 <sgallagh[m]> argh
16:04:14 <sgallagh[m]> #undo
16:04:14 <zodbot> Removing item from minutes: <MeetBot.items.Link object at 0x7f3388c3bf10>
16:04:18 <King_InuYasha> lol
16:04:22 <King_InuYasha> you did it *again* :P
16:04:29 <sgallagh[m]> #link https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/EKBW5DRRYGMJ5KOAHBGSXMOKGVSA3NPE/
16:04:33 <King_InuYasha> you really love nodejs :P
16:04:37 <sgallagh[m]> I copied it from my email :-P
16:04:48 <jforbes> nobody loves nodejs
16:05:34 <King_InuYasha> :D
16:05:43 <sgallagh[m]> This proposal got a bit of chatter on the mailing list which seems to have died down.
16:06:19 <sgallagh[m]> Does anyone here want to raise any concerns about it?
16:06:39 <sgallagh[m]> mhroncok: I know you'd come up with an edge-case or two, if you want to get those on the record.
16:06:40 <jforbes> So, I am generally okay with this, it will do a lot for stability of ELN.  My only concern (and this is happening now), is that ELN can pull in buildreqs from rawhide which should not/do not exist in ELN at all.  That will become a problem later when things branch to CentOS stream or whatever
16:07:01 <King_InuYasha> yeah, I'm personally a bit concerned about that as well
16:07:30 <King_InuYasha> I know that was a goal to be able to use Rawhide as an outlay and the seed, but I wouldn't expect it to do that on an ongoing basis
16:07:42 <sgallagh[m]> "Should not" and "do not" are two different situations.
16:07:53 <jforbes> While that happens now, this proposal only makes it a bit worse in that major version bumps are the most likely to bring in new deps which may or may not be ELN targeted
16:08:19 <mhroncok> jforbes: I don't unerstand how does thsi proposal make it worse
16:09:04 <jforbes> FWIW, I only noticed this because we had a line in the wrong place in the kernel, so it tried to bring in bpftools as a buildreq for i686, which doesn't exist in eln. It only failed because the rawhide version was built against a newer glibc than ELN had available
16:09:23 <sgallagh[m]> jforbes: I'm probably missing something, but I don't see the issue with this, other than that it means trimming will have to happen later if a dep really isn't desired in CentOS Stream
16:09:54 <mhroncok> I can only see that it makes this works becasue things are more likely to build
16:10:00 <mhroncok> and when things don't build, they don't change
16:10:01 <jforbes> mhroncok: it doesn't make it a lot worse.  It is already hard to detect because a package build succeeds, but people analyze sidetags more than general builds
16:10:09 <mhroncok> *makes this worse
16:10:30 <King_InuYasha> it's too bad we can't just track side tags and make JIT clones of them
16:10:43 <jforbes> When the expectation is that rawhide buildreqs are normal, it just buries the problem a bit more
16:11:23 <King_InuYasha> basically, as side-tags are being populated, the ELN bot could do the same
16:11:23 <sgallagh[m]> King_InuYasha: We *can*, but it turns out to be an incredible amount of effort for minimal gain over the proposed strategy.
16:11:41 <sgallagh[m]> And there are unsolvable race-conditions
16:12:03 <King_InuYasha> of creating a side tag and populating it as one is pushed in for rawhide?
16:12:08 <mhroncok> jforbes: using a rawhide build of a package as a BR in eln is normal... but "the machinery" should also pull that package into eln and rebuild it there, so any later rebuild happesn against the eln package
16:12:10 <jforbes> Overall, I am all for this proposal.  Longer term, I would like to maybe see something that compares the depsolver to the package list in ELN and won't install from rawhide if the package is in eln already
16:12:21 <King_InuYasha> that is, for each package submitted into a side-tag, another would go into the ELN clone side tag
16:12:29 <King_InuYasha> and when the rawhide side tag is merged, the ELN one would be too
16:12:31 <mhroncok> jforbes: and if that specific package is not desired to be in ELN, it should not be BRed from an ELn package
16:12:48 <sgallagh[m]> King_InuYasha: I can walk you through the details outside of the meeting, but it's a lot harder than it sounds.
16:12:54 <King_InuYasha> sgallagh[m]: sure
16:13:13 <mhroncok> King_InuYasha: the entire reason for this proposal is that nobody was able to figure that thing you describe out yet
16:13:19 <jforbes> mhroncok: unless the package is already in eln, and the subpackage does not exist on a specific arch
16:13:31 <King_InuYasha> mhroncok: ehh?
16:13:45 <King_InuYasha> it doesn't seem that difficult to do, but we can talk afterward about it
16:13:50 <King_InuYasha> the thing as-is is fine
16:13:55 <mhroncok> King_InuYasha: a request to do clone ELn side tags exists since the beginnign of ELN
16:14:03 <sgallagh[m]> As mhroncok says, that was how we were originally trying to approach the problem, but solving all the difficult edge-cases is proving impossible.
16:14:07 <jforbes> Maybe this is  just a really weird corner case I happened to hit and it won't be something anyone else seems to hit. If that is the case, I am not too concerned, I already know to look for it now
16:14:23 <sgallagh[m]> Vs. the current proposal which is far less work and covers ~99% of the need.
16:14:50 <jforbes> sgallagh[m]: yes, in all instances, I think this solves a whole lot more problems than it creates
16:14:53 <sgallagh[m]> jforbes: I think that corner case is detectable via Content Resolver, so I'm not overly concerned.
16:15:04 <sgallagh[m]> CR will tell you what workload(s) pulls it in
16:15:35 <mhroncok> well if a package has %if-eln ExcludeArch, this will indeed be problematic
16:16:01 <mhroncok> becasue whether or not dependent packages can build will be basically a coin flip
16:16:02 <jforbes> My main concern, is we never would have known we had the bug or that this was happening had the eln buildroot not been broken for a day and the new glibc not been available in the depsolver
16:17:03 <jforbes> Anyway, minor concern for the record. Still in favor of the proposal
16:17:09 <mhroncok> right. when the eln buildroot is outdated, it can help discover problems
16:17:30 <sgallagh[m]> True, but it causes more than it discovers.
16:17:36 <King_InuYasha> let's do it then
16:17:46 <King_InuYasha> I'm all for more problems in ELN :P
16:17:47 <mhroncok> my edge case that I told sgallagh[m] about earlier over chat is...
16:17:52 <sgallagh[m]> (The continuous kernel integration team uses ELN and had been hounding me to get the composes fixed)
16:18:11 <mhroncok> let's get a real example
16:18:22 <sgallagh[m]> King_InuYasha: I am too... more problems in ELN will lead to fewer in CentOS Stream/RHEL :-)
16:18:24 <mhroncok> python3-docs does not build with python3-sphinx (yet)
16:18:57 <mhroncok> when we update Python, I want to update the python3-docs package, but I cannot, as we are already at python3-sphinx 4
16:19:16 <mhroncok> correction: python3-docs does not build with python3-sphinx *4* (yet)
16:19:29 <mhroncok> so what I would normally do is to...
16:19:32 <mhroncok> 1) create a side tag
16:19:40 <mhroncok> 2) tag pytohn3-sphinx 3 into it
16:19:45 <mhroncok> 3) build python3-docs
16:19:52 <mhroncok> 4) untag pytohn3-sphinx 3
16:20:08 <mhroncok> 5) create a rawhide bodhi update  from the side tag with just python3-docs
16:20:31 <mhroncok> the proposed solution would tag that build of python3-docs into ELN
16:20:37 <mhroncok> and attempted a rebuild of it
16:20:55 <mhroncok> but it would still fail, becasue it has no information about builds tagged to the buildroot temporarily
16:21:03 <sgallagh[m]> The rebuild would fail, but the Rawhide version would still be there
16:21:08 <mhroncok> correct
16:21:26 <mhroncok> but no amount of automated rebuilds would ever make it suceed as a true ELN build
16:21:50 <sgallagh[m]> My response to mhroncok about this elsewhere was that I think this is probably something we can document as a known limitation and deal with on an individual basis.
16:22:02 <mhroncok> a human operator would need to figure out what happens (e.g. by aksing me "how the hell did you manage to get it built in rawhide?")
16:22:04 <sgallagh[m]> Rather than attempt to engineer a solution that can cover it perfectly.
16:22:21 <sgallagh[m]> (Lots of alcohol)
16:22:35 <mhroncok> and I agreed with sgallagh[m] abotu this, i just wanted to make sure it's a known thing
16:22:41 <jforbes> Yup, occasional (hopefully rare) corner cases are going to happen with anything we do
16:23:18 <sgallagh[m]> Thanks for summarizing, mhroncok
16:23:41 * mhroncok hopes that when this happens, the maintainers won't see 1000 failed ELN rebuild attempts a week, but would rather be approached by a person
16:24:21 <sgallagh[m]> Yeah, we'll make sure that doesn't happen
16:25:04 <sgallagh[m]> One thing I suggested elsewhere was that we should probably have a daily report from the compose which (among other things) lists any packages with a `.fcNN` release tag in it.
16:25:30 <sgallagh[m]> Because those will generally be things requiring investigation (though I hope most will just turn out to be test-flakes)
16:26:52 <mhroncok> https://docs.fedoraproject.org/en-US/eln/status/
16:27:02 <mhroncok> this would basically change to...
16:27:30 <mhroncok> "true ELN build" vs "inherited Fedora build"
16:27:48 <sgallagh[m]> Ack
16:28:04 <sgallagh[m]> I keep forgetting that page exists and I really need to make it part of my morning routine to read it
16:29:01 <sgallagh[m]> So yeah, we'll just tweak that a bit to make it easy to filter on that case.
16:29:46 <sgallagh[m]> Alright, to get the formalities out of the way:
16:29:46 <sgallagh[m]> Proposal: ELN rebuilds will adopt the mechanism proposed in https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/EKBW5DRRYGMJ5KOAHBGSXMOKGVSA3NPE/
16:29:49 <sgallagh[m]> +1
16:30:45 <jforbes> +1
16:30:49 <King_InuYasha> +1
16:31:53 <sgallagh[m]> mhroncok was +1 upthread, so:
16:32:00 * mhroncok doesn't vote
16:32:05 <mhroncok> not a sig member
16:33:00 <sgallagh[m]> #agreed ELN rebuilds will adopt the mechanism proposed in https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/EKBW5DRRYGMJ5KOAHBGSXMOKGVSA3NPE/ (+3, 0, -0)
16:33:05 <sgallagh[m]> mhroncok: Do you want to be? :)
16:33:22 <sgallagh[m]> #topic Open Floor
16:33:34 <mhroncok> sgallagh[m]: I have too many things to worry about already, thanks
16:33:38 <sgallagh[m]> That was the only topic I had on the agenda, but if anyone else wants to bring something up...
16:34:39 <jforbes> nothing else here
16:34:51 <King_InuYasha> nothing here
16:35:03 <sgallagh[m]> Then have 25 minutes back
16:35:07 <sgallagh[m]> Thank you all for coming.
16:35:27 <sgallagh[m]> #endmeeting