17:00:37 <tstellar> #startmeeting FESCO (2023-10-26)
17:00:37 <zodbot> Meeting started Thu Oct 26 17:00:37 2023 UTC.
17:00:37 <zodbot> This meeting is logged and archived in a public location.
17:00:37 <zodbot> The chair is tstellar. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
17:00:37 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
17:00:37 <zodbot> The meeting name has been set to 'fesco_(2023-10-26)'
17:00:42 <zbyszek> .hello2
17:00:42 <zodbot> zbyszek: zbyszek 'Zbigniew Jędrzejewski-Szmek' <zbyszek@in.waw.pl>
17:00:43 <nirik> morning
17:00:43 <tstellar> #meetingname fesco
17:00:43 <zodbot> The meeting name has been set to 'fesco'
17:00:51 <tstellar> #chair nirik, decathorpe, zbyszek, sgallagh, mhroncok, dcantrell, mhayden, Conan_Kudo, Pharaoh_Atem, Son_Goku, King_InuYasha, Sir_Gallantmon, Eighth_Doctor, tstellar
17:00:51 <zodbot> Current chairs: Conan_Kudo Eighth_Doctor King_InuYasha Pharaoh_Atem Sir_Gallantmon Son_Goku dcantrell decathorpe mhayden mhroncok nirik sgallagh tstellar zbyszek
17:00:56 <tstellar> #topic init process
17:00:57 <sgallagh_> .hi
17:01:02 <zodbot> sgallagh_: Sorry, but user 'sgallagh_' does not exist
17:01:04 <mhayden> .hello2
17:01:05 <zodbot> mhayden: mhayden 'Major Hayden' <mhayden@redhat.com>
17:01:06 <sgallagh_> .hello sgallagh
17:01:08 <zodbot> sgallagh_: sgallagh 'Stephen Gallagher' <sgallagh@redhat.com>
17:01:08 <tstellar> .hello2
17:01:11 <zodbot> tstellar: tstellar 'Tom Stellard' <tstellar@redhat.com>
17:01:19 <michel-slm> .hello salimma
17:01:22 <zodbot> michel-slm: salimma 'Michel Lind' <michel@michel-slm.name>
17:02:05 <decathorpe> .hi
17:02:05 <zodbot> decathorpe: decathorpe 'Fabio Valentini' <decathorpe@gmail.com>
17:02:20 <dcantrell> .hello2
17:02:21 <zodbot> dcantrell: dcantrell 'David Cantrell' <dcantrell@redhat.com>
17:03:16 <dcavalca> .hi
17:03:17 <zodbot> dcavalca: dcavalca 'Davide Cavalca' <davide@cavalca.name>
17:03:20 <Son_Goku> .hello ngompa
17:03:21 <zodbot> Son_Goku: ngompa 'Neal Gompa' <ngompa13@gmail.com>
17:03:38 <tstellar> OK, looks like we have quorum.
17:04:10 * rwmjones is here to answer questions about frame pointers if you have any
17:04:16 <tstellar> #topic #3084 Re-evaluate -fno-omit-frame-pointer compile flag for F40
17:04:25 <tstellar> .fesco 3084
17:04:29 <zodbot> tstellar: Issue #3084: Re-evaluate -fno-omit-frame-pointer compile flag for F40 - fesco - Pagure.io - https://pagure.io/fesco/issue/3084
17:05:21 <nirik> Do we want/need to gather any more info here? or should we just vote or ?
17:05:28 <sgallagh_> So, the question before us is whether frame pointers have caused a slowdown?
17:05:48 <sgallagh_> (Or, a sufficient slowdown to justify removal)
17:05:57 <Son_Goku> Based on Phoronix data, I don't think so
17:06:05 <dcavalca> I haven't seen any report of meaningful slowdowns
17:06:09 <decathorpe> as far as I can tell, the "sufficient" part is ... just not the case
17:06:13 <sgallagh_> So far, the only qualitative data I've seen suggests that, no: there's been no meaningful slowdown
17:06:35 <sgallagh_> sorry, s/qualitative/quantitative/
17:06:39 <dcavalca> I'll also note that while packages had the option to opt out, less than 30 did so according so sourcegraph
17:06:41 <tstellar> I'm not really sure how to scientifically measure that.
17:06:43 <sgallagh_> Used *exactly* the wrong word there.
17:06:43 <zbyszek> Yep, the feature works as predicted during the endless discussions when it was being approved.
17:06:47 <rwmjones> I measured qemu TCG emulation, and gcc compiles, and in both cases it was about 1%
17:06:56 <dcavalca> With most of that composes by the llvm stack
17:07:12 <dcavalca> * composed (sorry, on the phone)
17:07:39 * nirik nods
17:07:40 <dcavalca> Yeah about 1pct or less is in line with what we'd observed in benchmarks as well
17:07:59 <Son_Goku> I think the only thing that suffered was Python 3.11 and older, and those opt out for that reason
17:08:06 <Son_Goku> (that is, of things in Fedora)
17:08:21 <dcavalca> Yup, and the opt out has been removed in 3.12
17:08:47 <tstellar> I don't think a lot of maintainers did benchmarks to check for slowdowns.
17:08:59 <sgallagh_> Does anyone want to speak in favor of reverting it? (By which I mean: provide a compelling argument with supportable data)
17:09:16 <mhayden> 🦗
17:09:21 <tstellar> I'm personally in favor of reverting it.
17:09:54 <dcavalca> If folks observe slowdowns in specific packages we're happy to take a look at those
17:10:12 <dcavalca> I was planning to go over all the current opt outs and evaluate them anyways
17:10:20 <sgallagh_> tstellar: Could you elaborate?
17:11:09 * nirik wonders what exactly we said we would do here... just check it after a while? some threshold for revert?
17:11:31 <zbyszek> nirik: We didn't. We just said we will "evaluate".
17:11:35 <sgallagh_> nirik: We didn't set a threshold, just an agreement to revisit and see if things went downhill
17:11:50 <dcavalca> When the change was approved there were concerns this would lead to meaningful slowdowns and bad press
17:11:58 <dcavalca> That just... didn't happen
17:11:59 <nirik> yeah, I can't find anything about revisit in the change, but ok.
17:12:16 <tstellar> I just think this could be done using COPR and it has a negative impact on most packages with no benefit.
17:12:17 <decathorpe> what about ... we wait for Fedora 39 release coverage, and the inevitable performance comparisons with ubuntu 23.10?
17:12:32 <tstellar> I mean no benefit for most packages not no benifit overall.
17:12:45 <tstellar> I mean there are clear benefits, I just think they don't outweigh the costs.
17:12:53 <Son_Goku> well, we already had that with F38 and Ubuntu 23.04
17:12:56 <rwmjones> it makes performance much easier to measure, it's really impossible otherwise
17:13:02 <dcavalca> A copr doesn't help with full system profiling. This was discussed back in the original change, but the point is being able to observe the system as it's running
17:13:20 <dcavalca> If one has to replace all the packages that's not viable in practice
17:13:21 <Son_Goku> hell, the gnome people started actually tackling perf issues for the first time in years because of it being in Fedora
17:13:22 <tstellar> dcavalca: Why not?
17:13:29 <zbyszek> tstellar: when you say "this could be done with copr" it's like if you didn't actually read the arguments behind the change.
17:13:58 <zbyszek> The stuff that dcavalca is talking about were covered over and over and over during the discussino.
17:13:59 <Son_Goku> and the kde folks are interested in doing the same too, since everyone's perf tooling depends on frame pointers
17:14:14 * nirik pictures 'copr-shadow' rebuilding all of fedora as it happens so users could 'dnf reinstall *' from it. oof.
17:14:17 <dcavalca> Because by the time you've done so the issue won't be reproducible anymore
17:14:33 <Son_Goku> right, and you can't do fancy things like record and send to a developer
17:14:53 <Son_Goku> essentially we have a level of instrumentation that we've never really had before
17:14:55 <dcavalca> The whole point is making continuous system profiling available to everyone anytime
17:14:55 <rwmjones> in future there will be alternatives to frame pointers (key here being "in future"), which we should keep an eye on: https://lwn.net/Articles/940686/
17:14:59 <tstellar> So the main reason why COPR can't be used is too many packages?
17:15:20 <rwmjones> tstellar: to use frame pointers, everything on the system has to be compiled with them
17:15:20 <Son_Goku> tstellar: the problem with COPR is that nobody will be running those packages by default
17:15:21 <sgallagh_> Can I put a pause on the discussion, for the moment? We don't need to relitigate the *current* state.
17:15:26 <rwmjones> because you're interested in whole system perforance
17:15:41 <sgallagh_> We have to decide if changing that is necessary, desired or unwanted.
17:16:27 <tstellar> Son_Goku: But don't just the developers need the -fno-omit-frame-pointer packages?
17:16:31 <sgallagh_> So we should treat any proposal to revert as a new event: one that requires evidence and a justification to proceed.
17:16:44 <Son_Goku> tstellar: no, it has to go all the way down and up
17:16:55 <Son_Goku> from the tip of the application to the bottom of the dependency chain
17:17:10 <tstellar> sgallagh_: True, I don't wnat to re-litigate everything.  And really if a majority of FESCO want to keep this then you don't really need to spend time trying to convinve me to change my mind.
17:17:11 <Son_Goku> otherwise you'll have huge gaps when you try to analyze it
17:17:20 <decathorpe> Son_Goku: I see only phoronix article comparing Fedora 37 and 38 beta, but none vs. ubuntu 23.04. I remember this surprised me half a year ago
17:17:30 <dcantrell> we have debuginfo and debugsource, let's create fpinfo packages too  :)
17:17:37 * Son_Goku dies
17:18:01 <Son_Goku> that would require changing ELF... and I don't feel like that's going to go well :P
17:18:38 <dcavalca> There's a bunch of potential options in the far future, but frame pointers is the only one that works now
17:18:41 <Son_Goku> well, also I don't think that'd work because the compiler compiles to code differently too
17:18:50 <Son_Goku> s/to/the/
17:19:07 <dcavalca> More than happy to revisit this when things like sframes become a reality
17:19:08 * Son_Goku ruminates about FatELF...
17:19:25 <dcavalca> But for the time being we have to work with what we have
17:19:49 <tstellar> One question I had was is anyone doing performance analayis on s390x or aarch64?  Is it just x86_64 so far?
17:19:58 <rwmjones> just x86-64 over here
17:20:04 <dcavalca> We've done aarch64 as part of the original change
17:20:15 <sgallagh_> Son_Goku: I think we'd have to call it SANTA :-P
17:20:15 <dcavalca> Daan is working on s390x right now
17:20:15 <Son_Goku> x86_64, aarch64, and riscv64 were done
17:20:41 <Son_Goku> ppc64le was already in place before
17:20:43 <dcavalca> Note that s390x uses backchain which is a different thing and might require some additional support work
17:20:43 <Son_Goku> iirc
17:20:51 <decathorpe> I did some perf work where I compared x86_64, aarch64, and s390x ... but nothing spectacular
17:21:07 <decathorpe> (did uncover a glibc perf regression though)
17:21:10 <dcavalca> We'll do a separate change for that when we have more clarity, likely for the F40 cycle
17:21:22 <dcavalca> And yeah ppc64le Is already covered
17:21:53 <Son_Goku> those mainframe people will be able to get to one million and one containers with the extra perf optimizing juice :P
17:22:03 <tstellar> We're at the 15 minute mark.  Do we need to keep discussing?  Is there anything to vote on or is our position that some needs to submit a change to revert if they want this reverted?
17:22:17 * Son_Goku remembers when someone told him that mainframes were cool because they could run one million containers without breaking a sweat
17:22:49 <sgallagh_> I think my view is that if someone wants to formally propose reverting, we can vote and see where that goes.
17:22:56 <sgallagh_> But no action == status quo
17:23:19 <decathorpe> +1 (to the procedure, not the actual vote)
17:23:22 <Son_Goku> Proposal: FESCo indicates that there is nothing that indicates we need to do anything for -fno-omit-frame-pointer change.
17:23:52 <mhayden> +1 to Son_Goku's proposal
17:24:04 <decathorpe> +1
17:24:08 <dcantrell> +1
17:24:10 <sgallagh_> I haven't seen anything to convince me otherwise. +1
17:24:23 <zbyszek> Hmm, so Son_Goku's proposal contradicts sgallagh's procedure… But anyway: +1 to the proposal
17:24:43 <tstellar> -1
17:24:52 <Son_Goku> I'm doing it so we can close the ticket and end the endless discussion
17:25:10 <Son_Goku> +1 to my own proposal
17:25:10 <tstellar> nirik: ?
17:25:25 <sgallagh_> Well, any proposal is acceptable, but it's basically a proposal to not do anything... that seems extraneous, but not harmful.
17:25:30 <nirik> +1 to keeping frame pointer
17:26:35 <Son_Goku> I think that's everyone, so...
17:26:41 <tstellar> #agree APPROVED (+7, 0, -1)
17:27:02 <sgallagh_> tstellar: Could you edit that to include what was approved?
17:27:05 <zbyszek> I think it's just 6
17:27:07 <Son_Goku> #undo
17:27:07 <zodbot> Removing item from minutes: AGREED by tstellar at 17:26:41 : APPROVED (+7, 0, -1)
17:27:15 <Son_Goku> #agreed FESCo indicates that there is nothing that indicates we need to do anything for -fno-omit-frame-pointer change. (+7, 0, -1)
17:27:18 <tstellar> Do I need pound proposal first.
17:27:22 <sgallagh_> no
17:27:22 <tstellar> Ah OK, thanks.
17:27:30 <zbyszek> No, 7, sorry.
17:27:36 <Son_Goku> it's 7, because we need to count me too :)
17:27:50 <tstellar> #topic Next week's chair
17:27:52 * decathorpe waits for a certain somebody to come complaining about how we're all in $BIG_CORP's pocket
17:27:58 * Son_Goku sighs
17:28:19 <Son_Goku> I expect that someone to jump at us again soon anyway
17:28:30 <decathorpe> Plasma 6? :)
17:28:33 <Son_Goku> yup
17:28:52 * zbyszek is at a conference next week, might miss the meeting.
17:28:53 <decathorpe> 🥳️
17:29:09 <Son_Goku> I'm excited about Plasma 6, there are so many goodies coming our way
17:29:17 <Son_Goku> and literally years of improvements coming all at once
17:30:18 <dcantrell> how different is it from KDE 3?  that was the last time I used KDE
17:30:39 <decathorpe> (next week is a holiday here, not sure where I will be)
17:30:40 <Son_Goku> well... it can be similar or different
17:30:44 <tstellar> I can chair next week.  I was the backup chair this week, so didn't do much.
17:31:03 <Son_Goku> it's been a long time since I've used KDE 3 though 😉
17:31:34 <tstellar> Seeing no objections...
17:31:45 <tstellar> #action tstellar will chair next meeting
17:31:53 <decathorpe> tstellar++  - thanks!
17:31:55 <zodbot> decathorpe: Karma for tstellar changed to 2 (for the release cycle f39):  https://badges.fedoraproject.org/tags/cookie/any
17:31:58 <tstellar> #topic Open Floor
17:32:01 <Son_Goku> tstellar++
17:32:02 <zodbot> Son_Goku: Karma for tstellar changed to 3 (for the release cycle f39):  https://badges.fedoraproject.org/tags/cookie/any
17:32:02 <zbyszek> tstellar++, thanks
17:32:07 <mhayden> tstellar++
17:32:33 <dcantrell> tstellar++
17:32:33 <zodbot> dcantrell: Karma for tstellar changed to 4 (for the release cycle f39):  https://badges.fedoraproject.org/tags/cookie/any
17:34:28 <Son_Goku> so the go/no-go meeting is going on in #fedora-meeting
17:34:39 <tstellar> Re the disucssion about using COPR for -fno-omit-frame-pointers.  I'm happy to discuss this more outside of the meeting.  I use COPR for mini rebuilds of Fedora and have been trying to improve redhat-rpm-config to make this easier.  I would like to know more about what the gaps are.  Maybe find me in #fedora-buildsys if you want to discuss.
17:34:41 <sgallagh_> Yeah, my attention has been split
17:35:51 <tstellar> Any Open Floor topics ?  I'll give it a few more minutes.
17:36:00 <Son_Goku> nothing from me
17:36:11 <zbyszek> tstellar: the problem is that to e.g. do profiling of a meson C project build inside of a kgx running under gnome-shell you'd effectively need to recompile couple hundred packages. It is "possible", but nobody is ever going to do it.
17:37:12 <dcavalca> even if a "fedora with frame pointers remix" existed and costed nothing to maintain, nobody would use it
17:37:38 <tstellar> #endmeeting