17:00:34 <zbyszek> #startmeeting FESCO (2022-07-05)
17:00:34 <zodbot> Meeting started Tue Jul  5 17:00:34 2022 UTC.
17:00:34 <zodbot> This meeting is logged and archived in a public location.
17:00:34 <zodbot> The chair is zbyszek. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
17:00:34 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
17:00:34 <zodbot> The meeting name has been set to 'fesco_(2022-07-05)'
17:00:34 <zbyszek> #meetingname fesco
17:00:34 <zodbot> The meeting name has been set to 'fesco'
17:00:34 <zbyszek> #chair nirik, decathorpe, zbyszek, sgallagh, mhroncok, dcantrell, music, mhayden, Conan_Kudo, Pharaoh_Atem, Son_Goku, King_InuYasha, Sir_Gallantmon, Eighth_Doctor
17:00:34 <zodbot> Current chairs: Conan_Kudo Eighth_Doctor King_InuYasha Pharaoh_Atem Sir_Gallantmon Son_Goku dcantrell decathorpe mhayden mhroncok music nirik sgallagh zbyszek
17:00:37 <zbyszek> #topic init process
17:00:40 <nirik> morning all
17:00:44 <zbyszek> .hello2
17:00:45 <zodbot> zbyszek: zbyszek 'Zbigniew Jędrzejewski-Szmek' <zbyszek@in.waw.pl>
17:00:49 <mhayden> .hello2
17:00:50 <zodbot> mhayden: mhayden 'Major Hayden' <mhayden@redhat.com>
17:00:51 <Eighth_Doctor> .hello ngompa
17:00:57 <zodbot> Eighth_Doctor: ngompa 'Neal Gompa' <ngompa13@gmail.com>
17:01:13 <zbyszek> Miro said that he'll not be here because of the state holiday
17:01:16 <dcantrell> .hello2
17:01:17 <zodbot> dcantrell: dcantrell 'David Cantrell' <dcantrell@redhat.com>
17:01:29 <zbyszek> We have quorum.
17:01:41 <zbyszek> I'll a minute more to see if anyone shows up.
17:01:45 <bcotton> .hello2
17:01:46 <zodbot> bcotton: bcotton 'Ben Cotton' <bcotton@redhat.com>
17:01:55 <dustymabe> .hi
17:01:56 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
17:01:58 <mhayden> happy to be here for the first time with all of you 🫂
17:02:12 <codonell> :-)
17:02:18 <DaanDeMeyer[m]> I'm joining for the first time as well, hi everyone
17:02:49 <zbyszek> Hi Ben, Hi Dusty, Hi Mayor, Hi Daan
17:03:05 <davide> . hello dcavalca
17:03:06 <zodbot> davide: dcavalca 'Davide Cavalca' <dcavalca@fb.com>
17:03:10 <davide> .hello dcavalca
17:03:11 <zodbot> davide: dcavalca 'Davide Cavalca' <dcavalca@fb.com>
17:03:14 <zbyszek> Hi Davide
17:03:19 <zbyszek> #topic #2759 Proposal: periodic check on packagers reachability
17:03:20 <zbyszek> .fesco 2759
17:03:22 <zodbot> zbyszek: Issue #2759: Proposal: periodic check on packagers reachability - fesco - Pagure.io - https://pagure.io/fesco/issue/2759
17:03:51 <decathorpe> hello! sorry, I'm late
17:04:02 <zbyszek> We're just starging
17:04:05 <zbyszek> *starting
17:04:14 <decathorpe> phew
17:05:31 <zbyszek> So... I'd propose the following workflow: we enumerate things that should be fixed, and then vote a proposal "approve, with the following items to adjust before the policy is merged". WDYT?
17:05:33 <nirik> so, are we voting on this? or ? I feel like there is still tweaks being made to it.
17:05:52 <decathorpe> yeah, not sure if this is ready yet.
17:05:56 <Eighth_Doctor> likewise
17:06:08 <nirik> zbyszek: well, is there a hurry?
17:06:24 <zbyszek> No, no hurry.
17:06:42 <zbyszek> So… should we make comments in the pull request and ask Mattia for another round?
17:06:48 <dcantrell> I think so
17:07:02 <nirik> I think bcotton's last note on the pr is something we should add/adjust... but also, we should run this at times in our cycle when it's less disruptive...
17:07:15 <dcantrell> I think the idea is valid, but we need a definition around what is "inactive or unreachable maintainer"
17:07:32 <nirik> ie, we shouldn't mark a bunch of people inactive right before a final freeze, causing a scramble on packages that may need fixing.
17:08:01 <dcantrell> (afk for a few minutes...)
17:08:15 <zbyszek> dcantrell: but the PR goes to great lengths to describe what checks are made. Those checks define this.
17:08:38 <nirik> dcantrell: well, according to what we have so far: no activty in X fedora things in 12 months and not answering a ticket about it.
17:09:05 <zbyszek> nirik: OK, can you add a comment on the PR with some time in the cycle where it'd be good to do this?
17:09:13 <bcotton> agreed. id like to avoid doing this between the start of beta freeze and the final release
17:09:38 <zbyszek> I guess some time soon after release would be good.
17:09:46 <zbyszek> New cycle, less people or something like that.
17:10:09 <zbyszek> Also many people contribute during the pre-release bugfixing, so that'll reduce our false positive rate.
17:10:29 <decathorpe> so, maybe around the time of Fedora N-2 EOL?
17:10:38 <nirik> hum...
17:10:54 <nirik> does it make sense to coordinate this with the orphaned packages/retirement cycle?
17:11:46 <decathorpe> retirement of long-term FTBFS / FTI packages happens sometime between beta freeze and final freeze IIRC? so probably not ...
17:12:09 <bcotton> I was thinking maybe 1 week after final release and 1 week before beta freeze as a starting point. that gives us 4x a year somewhat spaced out and FTI retirement is a week before the start of beta freeze
17:12:33 <zbyszek> bcotton: +1
17:12:41 <nirik> yeah, sounds reasonable
17:14:04 <dcantrell> (back now)
17:14:09 <decathorpe> any reason why 4 times a year and not just 2?
17:14:56 <bcotton> the proposal as written is 6, so that seemed like a fair compromise. i'd be fine with 2
17:15:08 <decathorpe> ah, ok. I missed that from the ticket.
17:15:10 <bcotton> we do the inactive provenpackager check 2x a year already
17:15:31 <zbyszek> Yeah, I think 2 is fine. It doesn't need to be frequent.
17:15:33 <decathorpe> yeah, most stuff happens twice a year, well, because, release cycle
17:16:04 <codonell> Because glibc releases twice a year ;-)
17:16:18 <nirik> 2 sounds fine to me. Especially since we are going from 0 now. ;)
17:16:20 <decathorpe> because Python releases ... oph, oops
17:16:34 <bcotton> heh
17:16:42 <zbyszek> #action mattia so submit another version taking into account comments in the ticket and on the pull request
17:16:49 <bcotton> so yeah, i'd be happy to have it happen alongside the provenpackager check
17:17:33 <zbyszek> #topic #2804 Change proposal: Fallback Hostname
17:17:33 <zbyszek> .fesco 2804
17:17:34 <zodbot> zbyszek: Issue #2804: Change proposal: Fallback Hostname - fesco - Pagure.io - https://pagure.io/fesco/issue/2804
17:17:36 <bcotton> that happens at branch day, effectively https://fedorapeople.org/groups/schedule/f-37/f-37-pm-tasks.html
17:18:35 <zbyszek> We have +4,0,0 votes in the ticket
17:18:43 <decathorpe> This one is only tagged with meeting because of procedural -1, if I see this correctly?
17:18:57 <zbyszek> No, that -1 was withdrawn / superseded already.
17:18:58 <dcantrell> +1 from me too
17:19:11 * nirik is also +1, not sure why I missed the ticket for voting there.
17:19:12 <Eighth_Doctor> yeah, so I don't know why we're discussing this?
17:19:13 <decathorpe> Yeah, but the meeting tag was added after the -1 vote.
17:19:16 <mhayden> kudos for dustymabe for digging through that problem and suggesting a solution
17:19:39 <zbyszek> I think we *could* have a better solution for this, but I don't want to block the proposal. I'll be: 0
17:19:42 <decathorpe> FWIW +1
17:19:42 * dustymabe accepts cookies
17:20:11 <zbyszek> #agreed APPROVED(+6, 0, 0)
17:20:21 <decathorpe> dustymabe++ essential, functional, and advertisement cookies
17:20:21 <zodbot> decathorpe: Karma for dustymabe changed to 2 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:20:22 <zbyszek> (I'm counting the votes in the ticket and here together.)
17:20:38 <dustymabe> thank you zbyszek - thanks all
17:20:42 <zbyszek> #undo
17:20:42 <zodbot> Removing item from minutes: AGREED by zbyszek at 17:20:11 : APPROVED(+6, 0, 0)
17:20:46 <zbyszek> #agreed APPROVED(+7, 0, 0)
17:21:03 <zbyszek> #undo
17:21:03 <zodbot> Removing item from minutes: AGREED by zbyszek at 17:20:46 : APPROVED(+7, 0, 0)
17:21:04 <zbyszek> #agreed APPROVED(+7, 1, 0)
17:21:14 <zbyszek> #topic #2817 Change proposal: Add -fno-omit-frame-pointer to default compilation flags
17:21:17 <zbyszek> .fesco 2817
17:21:18 <zodbot> zbyszek: Issue #2817: Change proposal: Add -fno-omit-frame-pointer to default compilation flags - fesco - Pagure.io - https://pagure.io/fesco/issue/2817
17:22:04 <nirik> I'm -1 on this so far...
17:22:07 <mhayden> this one feels like a tradeoff: benefits a small number of debugging/profiling situations used by a relatively small number of users versus a performance impact that affects all users
17:22:22 <nirik> mhayden: yeah.
17:22:24 <mhayden> i've been on both ends of this issue and neither are fun
17:22:40 <zbyszek> I think we need some better benchmarking.
17:22:44 * Eighth_Doctor shrugs
17:22:48 <nirik> but the small number of users profiling at least know how to get out of it... the end users don't know/can't
17:23:09 <codonell> The debugging and profiling tooling could be improved. While you can never gain back the lost register usage.
17:23:12 <DaanDeMeyer[m]> I got distracted by reproducing the phoronix results, I haven't gotten to a proper distro rebuild benchmark yet
17:23:24 <mhayden> i gotta follow Spock on this one with a -1  ("Logic clearly dictates that the needs of the many outweigh the needs of the few.")
17:23:36 <zbyszek> I think people underestimate how useful this could be.
17:23:51 <davide> zbyszek @zbyszek:libera.chat: are there any specific benchmarks you'd want to see done?
17:23:56 <decathorpe> I think the problem is that the current benchmarks show the best-case impact where only leaf applications are rebuilt with those flags ... worst-case (the whole world is rebuilt) is likely much worse than just a few % slower
17:23:58 <nirik> I'm open to more data...
17:24:03 <zbyszek> Right now profiling on Linux isn't too great, and this proposal would make it better.
17:24:24 <DaanDeMeyer[m]> We're also brainstorming ways to make this more useful for everyone, e.g. by adding a profiling daemon that can do on demand or trigger based profiling of the entire system
17:24:31 <DaanDeMeyer[m]> Of course that doesn't exist yet
17:24:32 <zbyszek> The argument that "people who need this can rebuild" is weak, because you can't really rebuild a swath of the distro, at least not easily.
17:24:56 <mhayden> agreed with nirik -- more data on performance impact here would be good, especially if we knew "workloads that do X would be most affected"
17:25:04 <tstellar> zbyszek: COPR makes this pretty easy in my opinion.
17:25:06 <mhayden> (meant X as a variable, not as X11) 😉
17:25:30 <fweimer> zbyszek: We should improve the profiling tools.
17:25:37 <zbyszek> davide: sorry, I don't know.
17:25:49 <Eighth_Doctor> COPR does not make rebuilding the distro easy
17:25:50 <davide> the problem with rebuilding is that you don't necessarily know in advance what you'd need to rebuild
17:25:52 <tstellar> Also, this proposal should help make it easier to adjust compiler flags when rebuilding packages: https://fedoraproject.org/wiki/Changes/RPMMacrosForBuildFlags
17:25:55 <Eighth_Doctor> it's extremely hard to rebuild the distribution
17:26:04 <Eighth_Doctor> and Fedora's tooling is horrible for doing this
17:26:16 <fweimer> And it's not just LBR, SHSTK also provides accurate backtraces without complicated traversal.
17:26:28 <Eighth_Doctor> I've actually attempted to do this before, and it's pretty awful
17:26:34 <codonell> I agree with Florian, I don't think we need to rebuild the distribution. This proposal suggests the wrong solution.
17:26:38 <music[m]> Hi, I’m sorry to have missed the first part of the meeting. I’m here now.
17:27:01 <DaanDeMeyer[m]> fweimer: Any links to stack unwinding tools are appreciated, I tried to list most of them in the proposal, but SHSTK is an acronym I haven't come across yet
17:27:10 <tstellar> Eighth_Doctor: It's worked out well for me when I've tried.  Maybe we are doing different things.
17:27:15 <codonell> SHSTK = Shadow Stack
17:27:26 <fweimer> DaanDeMeyer[m]: It's the part of Intel CET that's also supported by AMD (starting with Zen 3).
17:27:28 <davide> I also don't necessarily want to make it easy / attractive for companies to fork the distro - imo we should encourage upstream use as much as possible
17:27:40 <tstellar> e.g. https://copr.fedorainfracloud.org/coprs/g/fedora-llvm-team/clang-built-f36/
17:27:40 <davide> (sorry element keeps crashing on me today, catching up)
17:27:46 <fweimer> DaanDeMeyer[m]: Sadly, kernel support is *still* missing.
17:28:27 <DaanDeMeyer[m]> kernel not supporting anything aside from frame pointers is the reason this proposal is here in the first place :(
17:28:52 <fweimer> davide: A colleague of mine is working on mass rebuild tooling for situations like this because Fedora (ELN) rejected *our* flag change requests.
17:28:58 <davide> The other angle here (and I'm not saying this is necessary a good argument) is that making profiling more accessible leads to optimizations which down the road can lead to user benefits
17:30:05 <fweimer> DaanDeMeyer[m]: I thought this proposal was about userspace?
17:30:06 <fche> (yes, but no need to presume that Slower but working profiling a la perf/stap/whatever are insufficient to lasso those future benefits)
17:30:20 <Eighth_Doctor> tstellar: you're not trying to make a usable system, you're just compiling everything
17:30:23 <Eighth_Doctor> there's a huge difference
17:30:41 <Eighth_Doctor> build orders, bootstraps, feature things, dynamic BRs, etc.
17:30:46 <Eighth_Doctor> and verification
17:30:49 <DaanDeMeyer[m]> fweimer: We need frame pointers in userspace so we can unwind stacks in BPF
17:30:56 <Eighth_Doctor> and that's not even compiling everything
17:31:00 <polacek> the proposal would mean offsetting a lot of performance improvements we've made for unclear gains, as pointed out elsewhere
17:31:08 <decathorpe> I just don't think making Fedora run less efficiently for all our users is worth it. What's next? Compiling all packages with ASAN / TSAN by default? :)
17:31:26 <decathorpe> The same arguments would be valid there ...
17:31:30 <fweimer> DaanDeMeyer[m]: That's the first time I hear this requirement.
17:31:40 <tstellar> Eighth_Doctor: Right, but this is more like what someone wanted to do a rebuild to test -fno-omit-frame-pointer would do.
17:31:41 <fweimer> We have a kernel space DWARF-based unwinder, can't you use that instead?
17:32:14 <fweimer> DaanDeMeyer[m]: It's a bit unfair not to put key details like that in your proposal.
17:32:40 <DaanDeMeyer[m]> From my understanding, we don't have a DWARF unwinder in the kernel, wasn't it explicitly rejected?
17:32:47 <DaanDeMeyer[m]> Apologies, that certainly wasn't my intention
17:32:48 <Eighth_Doctor> fweimer: the proposal implies there's no kernel DWARF unwinder
17:33:24 <Eighth_Doctor> and nothing I can find indicates that's changed
17:33:27 <fche> indeed; stap does dwarf unwinding in-kernel but using its own runtime
17:33:49 <fweimer> Eighth_Doctor: Yes, but as fche set on the list, there is a kernel-space DWARF unwinder you can use.
17:34:17 <fche> fweimer, I don't think bpf has enough liberty to call into a piece of custom code like that
17:34:42 <fche> DaanDeMeyer[m], has bpf upstream been consulted about improving their unwinder, future paths etc.?
17:35:14 <DaanDeMeyer[m]> This proposal is partially motivated by the request of Andrii, libbpf's maintainer actually
17:37:27 <zbyszek> Hmm. So I think we have two avanues of research needed: 1. do a partial rebuild and some more comprehensive benchmarks to see what happens with non-leaf packages.
17:37:29 <DaanDeMeyer[m]> Aside from a proposal to copy the stack to userspace and do unwinding there that I found somewhere, there haven't been any concrete proposals, they rely very heavily on having frame pointers available for the related tooling to work
17:37:30 <fche> this context is missing from the change, no?  the word bpf doesn't appear in there
17:37:48 <DaanDeMeyer[m]> Indeed, I will add a paragraph about this specific usecase
17:37:55 <zbyszek> 2. add more discussion of alternative approaches to the proposal
17:38:27 <zbyszek> DaanDeMeyer[m]: thanks
17:38:44 <zbyszek> #action DaanDeMeyer to add more information to the proposal
17:38:48 <DaanDeMeyer[m]> Sounds good, I'll look into doing a more comprehensive rebuild
17:38:53 <zbyszek> Do we want to continue the discussion here?
17:39:08 <zbyszek> (now)
17:39:11 <polacek> the proposal also doesn't mention -mno-omit-leaf-frame-pointer IIRC
17:39:14 <DaanDeMeyer[m]> (Any ideas for workloads to benchmark are appreciated as well)
17:39:17 <polacek> which it should
17:39:20 <fche> (also would appreciate spelling out why a fedora change would be desirable if meta/facebook already recompile their distro)
17:39:33 <polacek> good point
17:39:38 <DaanDeMeyer[m]> We don't actually
17:39:41 <Eighth_Doctor> fche: they don't
17:39:46 <Eighth_Doctor> that's the whole point
17:39:48 <fche> ok, just meta/google then?
17:39:50 <Eighth_Doctor> and as Davide Cavalca said, they don't want to
17:40:00 <Eighth_Doctor> only google does it
17:40:03 <Eighth_Doctor> meta doesn't
17:40:06 <zbyszek> Also, people want to profile Fedora itself too.
17:40:12 <mjw> Random pointer on unwinding speed from the valgrind author. TLDR, it can be pretty fast: https://blog.mozilla.org/jseward/2013/08/29/how-fast-can-cfiexidx-based-stack-unwinding-be/
17:41:24 <zbyszek> OK, let's return to this when we have more data.
17:41:39 <zbyszek> Thanks everyone for very useful comments and links and discussion ;)
17:41:40 <zbyszek> #topic Next week's chair
17:41:47 <zbyszek> Vlntr?
17:41:49 <zbyszek> Vlntrs?
17:42:32 <decathorpe> I can be fallback host(name) in case of no volunteers
17:42:51 <mhayden> i'm on vacation during the next meeting 🏖
17:43:12 <zbyszek> #action decathorpe will (host)chair next meeting
17:43:19 <zbyszek> #topic Open Floor
17:43:35 <zbyszek> Anyone?
17:44:13 <zbyszek> If not, I'll wrap things up in a minute.
17:45:02 <zbyszek> See you next week.
17:45:02 <zbyszek> #endmeeting