15:06:32 <decathorpe> #startmeeting FESCo (2020-02-03)
15:06:32 <zodbot> Meeting started Mon Feb  3 15:06:32 2020 UTC.
15:06:32 <zodbot> This meeting is logged and archived in a public location.
15:06:32 <zodbot> The chair is decathorpe. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:06:32 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:06:32 <zodbot> The meeting name has been set to 'fesco_(2020-02-03)'
15:06:37 <decathorpe> #meetingname fesco
15:06:37 <zodbot> The meeting name has been set to 'fesco'
15:06:44 <decathorpe> #chair nirik, ignatenkobrain, decathorpe, zbyszek, bookwar, sgallagh, contyk, mhroncok, dcantrel
15:06:44 <zodbot> Current chairs: bookwar contyk dcantrel decathorpe ignatenkobrain mhroncok nirik sgallagh zbyszek
15:06:54 <dcantrell> .hello dcantrell
15:06:58 <zodbot> dcantrell: Sorry, but you don't exist
15:07:00 <dcantrell> .hello dcantrel
15:07:01 <zodbot> dcantrell: dcantrel 'David Cantrell' <dcantrell@redhat.com>
15:07:09 <dcantrell> (I always mistype that)
15:07:09 <decathorpe> sorry for the delay.
15:07:12 <zbyszek> .hello2
15:07:13 <zodbot> zbyszek: zbyszek 'Zbigniew Jędrzejewski-Szmek' <zbyszek@in.waw.pl>
15:07:16 <nirik> morning
15:07:17 <zbyszek> Sorry, I forgot the time.
15:07:23 <sgallagh> #topic Init Process
15:07:28 <sgallagh> .hello2
15:07:28 <zodbot> sgallagh: sgallagh 'Stephen Gallagher' <sgallagh@redhat.com>
15:07:37 <zbyszek> .hello2
15:07:38 <zodbot> zbyszek: zbyszek 'Zbigniew Jędrzejewski-Szmek' <zbyszek@in.waw.pl>
15:08:32 <decathorpe> alright, I think there's only two topics for today, everything else is either not ready yet or was already voted on in ticket
15:09:01 <zbyszek> That's my impression too.
15:09:19 <decathorpe> #topic #2320 F32 System-Wide Change: Enable EarlyOOM
15:09:29 <decathorpe> .fesco 2320
15:09:30 <zodbot> decathorpe: Issue #2320: F32 System-Wide Change: Enable EarlyOOM - fesco - Pagure.io - https://pagure.io/fesco/issue/2320
15:10:01 <decathorpe> according to the last comments, the Workstation WG has decided to enable EarlyOOM by default, and we don't have to decide anything.
15:10:22 * nirik is ok with that.
15:10:39 <zbyszek> Yeah. Proposal: we just acknowledge the WG decision.
15:10:39 <decathorpe> does anybody want to overrule their decision? :)
15:10:42 <sgallagh> WFM
15:10:51 <dcantrell> I'm fine with that, I just have one question
15:11:00 <sgallagh> I don't think we need to vote unless we propose to overrule them
15:11:03 <dcantrell> what's the process to disable it if someone wants to disable it?  Is it trivial?
15:11:20 <cmurf> 'systemctl disable earlyoom'
15:11:21 <sgallagh> dcantrell: Sounds like `systemctl disable earlyoom.service` I think
15:11:24 <dcantrell> fair
15:11:28 <dcantrell> thanks
15:11:29 <decathorpe> yeah that should do it
15:12:00 <decathorpe> everybody OK with ACK'ing the WG decision? I don't think we need to vote on that.
15:12:13 <zbyszek> Yes, and this will be "remembered", in the sense that the service will not be re-enabled on package upgrades.
15:12:34 <cmurf> disabling/reverting to previous behavior is in the change proposal, under release notes
15:12:43 <nirik> or removing it. ;)
15:13:33 <decathorpe> #info FESCo acknowledges the Workstation WG's decision and does not want to overrule
15:13:54 <decathorpe> #topic #2323  F32 System-Wide Change: Reduce installation media size by improving the compression ratio of SquashFS filesystem
15:14:00 <decathorpe> .fesco 2323
15:14:03 <zodbot> decathorpe: Issue #2323: F32 System-Wide Change: Reduce installation media size by improving the compression ratio of SquashFS filesystem - fesco - Pagure.io - https://pagure.io/fesco/issue/2323
15:14:23 <decathorpe> looks like there's only lack of opinion here
15:14:32 <sgallagh> I haven't been able to follow the discussion due to being all over the place the last month.
15:15:00 <sgallagh> I realize there are alternative proposals out, but I want to know if there is any known downside to the Change as filed.
15:15:00 <decathorpe> yeah, some alternatives were brought up, including replacing rsync with unsquashfs, or using zstd instead of xz
15:15:39 <zbyszek> sgallagh: the downside is more CPU usage during installation.
15:15:42 <nirik> I think zstd was a winner
15:16:03 <sgallagh> zbyszek: That doesn't seem like much of an issue.
15:16:06 <zbyszek> Kamil Paral makes very good arguments why we don't want to go in that direction.
15:16:22 <decathorpe> increased FS latency is also not nice for live systems
15:16:43 <nirik> zbyszek: which direction?
15:16:44 <zbyszek> If we follow the change, we save a bit in image size, but pay quite a bit in latency and CPU time.
15:17:27 <sgallagh> zbyszek: This applies only to Live images?
15:17:34 <cmurf> my oversimplificiation is the proposal improves compression a bit, decreases CPU somewhat (if sticking with rsync, perhaps more with unsquashfs)
15:17:39 <sgallagh> Or are the install DVDs/netinst.iso also affected?
15:17:47 <cmurf> but leaves a lot on the table, CPU and latency wise
15:18:00 <cmurf> I expect all ISO's produced by Anaconda are affected
15:18:21 <zbyszek> I don't want to say that we shouldn't optimize the size, but the arguments to switch away from the current scheme and towards zstd are pretty compelling. And then we shouldn't ramp up xz settings.
15:18:22 <cmurf> all contain a "LiveOS" using a squashfs image
15:18:29 <dcantrell> installation is already heavily CPU bound with depsolving and running the rpm transaction, but adding any increase in runtime to the installation process impacts users first impressions of a release
15:18:57 <sgallagh> zbyszek: Arguments are nice, but is anyone actually working on that?
15:19:04 <nirik> I think we can get faster install + smaller/as small with zstd
15:19:14 * sgallagh doesn't like rejecting a proposal on the grounds that someone *might* do it better... someday
15:19:19 <decathorpe> nirik: that was my take-away as well
15:19:35 <dcantrell> is there anything we can remove from the squashfs image?
15:19:56 <dcantrell> having played the installer image game for a long time, this was sort of a constant game of chase of how to make the image small enough for all platforms and carry the required tools
15:19:59 <cmurf> the proposal suggests dropping the nested ext4 file system, and using plain squashfs
15:20:03 <dcantrell> (worst offender always: ppc64)
15:20:26 <sgallagh> dcantrell: Does that remain true for ppc64le?
15:20:27 <cmurf> *suggests* it doesn't state it outright, but it's been part of the discussion
15:20:40 <dcantrell> sgallagh: sadly, yes.  they just doubled the limit
15:20:40 <decathorpe> cmurf: there are several options listed, but all of them need to be implemented in pungi first
15:20:41 <dcantrell> :/
15:20:54 <sgallagh> yay
15:21:10 <cmurf> decathorpe: and I suppose in koji if the idea is to give releng choices
15:21:21 <decathorpe> yeah. I assume so
15:21:27 <dcantrell> but honestly, this is the kind of problem that just comes back around from time to time
15:21:51 <cmurf> but the work and testing on the plain squashfs + zstd testing has been done in lorax and anaconda
15:21:53 <dcantrell> as written, I don't have a problem with it.  I would like to know if anything could be removed from the squashfs image.  And maybe some metrics on increased CPU load during install
15:22:06 <cmurf> whereas there's still work to be done for unsquashfs
15:22:06 <nirik> perhaps we could broadly say what we want to optimize for?
15:22:07 <sgallagh> The current proposal just means tweaking some config options, yes? Not making real changes to the compose?
15:22:23 <dcantrell> sgallagh: that is my interpretation
15:22:35 <cmurf> dcantrell: what do you mean by "if anything could be removed from"?
15:22:36 * nirik is -1 for the current proposal as written. I think we can do better. ;)
15:22:44 <decathorpe> sgallagh: all of the choices are listed here ... https://fedoraproject.org/wiki/Category:Changes/OptimizeSquashFS#User_Experience
15:22:53 <sgallagh> Then I'm inclined to approve it conditionally through Beta, removing it for GA at FESCo's discretion
15:23:11 <sgallagh> If there are better options using zstd, let's get those for F33
15:23:24 <dcantrell> cmurf: when we dealt with installer image size constraints on the anaconda team, there were two approaches we could take.  inventory what's in the image and look for things to remove and/or jack the compression ratio to try to squeeze it down more
15:23:50 <nirik> I don't want a bunch of churn... we should decide and do it once. ;)
15:23:58 <sgallagh> Because we can guess all we like, but actual data from Beta would be the best option.
15:24:23 <sgallagh> nirik: I disagree; we should try it and see what works and adjust as we go
15:24:27 <nirik> the change author was optimizing for space savings...
15:24:31 <cmurf> dcantrell: ahh gotcha, well on the Workstation WG side, if we get flagged that the image is over 2G, then we go look what caused the increase, that's about it
15:24:33 <decathorpe> well, we'd need an actual thing to compare it to.
15:24:51 <nirik> I think install time would be most important, followed by size, and only 3rd creation time
15:25:09 <dcantrell> cmurf: understood.  yeah, at some point you're out of options.  but if it hadn't been done, it's worth bringing up.
15:25:18 <decathorpe> I don't suppose pungi/koji can produce the same image twice with different options?
15:25:19 <cmurf> it's not yet clear if this change requires or expects Anaconda to use unsquashfs instead of rsync; not clear if Anaconda folks are agreeable to that change
15:25:48 <sgallagh> decathorpe: It can, but it would mean a huge increase in compose time
15:25:52 <cmurf> if that doesn't happen, installation performance doesn't improve much at all
15:26:16 <decathorpe> sgallagh: well that would be what's needed for an actual comparison ...
15:26:21 <nirik> and storage space and mirrors and...
15:26:31 <cmurf> koji lacks a switch to ask for different compression options; and a separate switch to ask for plain squashfs instead of the existing ext4 nested squashfs
15:26:40 <zbyszek> For me, the interesting benchmark is the second in the Change page: "installation time". With the proposed change, installation time increases noticeably, with zstd is cut by half.
15:26:45 <sgallagh> decathorpe: I was thinking more in terms of whether it becomes qualitatively worse
15:26:56 <nirik> zbyszek: +1 (same for me)
15:26:59 <sgallagh> As in, do we get feedback from people saying "hey, this is really slow"?
15:27:01 <decathorpe> worse than what? :)
15:27:06 <sgallagh> F31
15:27:28 <decathorpe> hoping that not too much has changed between F31 and F32 ... wll
15:27:30 <sgallagh> Because if there's a small slowdown but no one notices... who cares?
15:28:03 <cmurf> haha well i noticed, well before this change, nirik can attest to my releng emails, and bcl can attest to my anaconda emails
15:28:06 <sgallagh> Vs. if they can download it using less bandwidth (particularly for those with caps), that's a real benefitr
15:28:11 <zbyszek> Well, if we are making changes, and we can do a change which cuts installation time by 40%, then I don't see the point in doing a change which increases the installation time by 2% to save the image size a bit.
15:28:36 <sgallagh> zbyszek: And I ask again: who is committed to doing that work?
15:28:41 <dcantrell> zbyszek: +1
15:28:44 <sgallagh> I haven't heard anyone say they're stepping up
15:29:11 <nirik> this change? adjusted to used zstd?
15:29:25 <zbyszek> Dunno, maybe OPs can be convinced to go in that direction?
15:29:26 <sgallagh> This Change has commitment from the proposers
15:29:36 <sgallagh> Who is committing to make the other changes?
15:29:44 <cmurf> well that's sorta the disappointing part, is the change owner wants to do xz optimization, not zstd; the emphasis is strictly on ISO size
15:30:12 <nirik> right, I am saying we tell them we would prefer to optimize for install time, size, then creation time
15:30:18 <nirik> and ask them to do that.
15:30:19 <cmurf> zbyszek: i've been trying :D
15:30:21 <sgallagh> Sure, and I understand there may be better options. But are we specifically discouraging them from doing this work?
15:30:21 <mhroncok> hey. I'm late, sorry about that
15:30:38 <sgallagh> Because we can't say "it should be done this way" and waddle off with no one working on it
15:30:41 <decathorpe> mhroncok: o/
15:31:17 <dcantrell> no but we can ask about the prioritization of outcomes.  like install time vs. iso size
15:31:24 <zbyszek> sgallagh: I think we're talking past one another. The change as proposed makes a (small) step in the wrong direction. So even apart from any other proposals, I'll be voting against.
15:31:57 <sgallagh> zbyszek: OK, that's a reasonable answer.
15:32:11 <decathorpe> do we want to vote on the current proposal (jacking up xz compression)?
15:32:16 <sgallagh> It just sounds to me like most folks here are discussing how they *wish* it would be done instead of discussing how it's currently proposed.
15:32:29 <nirik> I don't think it would hurt to ask them... at worst they say they can't work on it.
15:32:43 <dcantrell> sgallagh: we are the ENGINEERING committee, we like implementation details
15:33:12 <zbyszek> decathorpe: I think we should do the vote, and then (if the outcome is negative), discuss what we propose instead.
15:33:15 <dcantrell> I do question the reason/value of saving space in an iso image.  Like where is that still a problem that we care about?
15:33:29 <decathorpe> zbyszek: alright then
15:33:38 <sgallagh> dcantrell: More often than you'd expect, unfortunately.
15:33:50 <nirik> it's all a balancing act...
15:33:56 <dcantrell> sgallagh: but weighted against install time.  yes, it's a balancing act
15:34:02 <decathorpe> Vote on current proposal (modyfying xz compression parameters for smaller ISO):
15:34:03 <nirik> but I think we can ask that it be balanced in a better way
15:34:39 <zbyszek> -1
15:34:42 <nirik> -1 on current change, hope for zstd targeting install time first, then space, then creation time
15:34:51 <decathorpe> -1 for the propsal as-is
15:35:05 <sgallagh> -1 I agree that the current proposal doesn't sound like a net improvement.
15:35:13 <dcantrell> -1 after this discussion I feel we need more info and prioritizations
15:35:28 <decathorpe> mhroncok: vote?
15:35:39 <mhroncok> 0 for now, sorry for missing most of the discussion here
15:35:57 <decathorpe> no problem
15:36:03 <decathorpe> I think -5 is enough votes, right?
15:36:09 * mhroncok also has opinions about how to possibly handle this
15:36:19 <sgallagh> decathorpe: It's a majority, so yes
15:36:30 <nirik> so, should we continue on list? or ?
15:36:37 <sgallagh> Opinions are easy. Code is hard.
15:37:02 <decathorpe> #agree REJECTED (-5, 1, -0) proposal in its current form
15:37:12 <sgallagh> #undo
15:37:12 <zodbot> Removing item from minutes: AGREED by decathorpe at 15:37:02 : REJECTED (-5, 1, -0) proposal in its current form
15:37:29 <sgallagh> #agree REJECTED (+0, 1, -5) proposal in its current form
15:37:30 <mhroncok> (+0, 1, -5)
15:37:36 <mhroncok> right
15:37:41 <decathorpe> oops. thanks
15:37:46 <sgallagh> np
15:38:00 <cmurf> one advantage of the current proposal is wiring up the plain squashfs image type in pungi (and maybe, hopefully, koji)
15:38:01 <decathorpe> guess I'll need something to lower my fever again -.-
15:38:26 <zbyszek> But let's make sure that we communicate to the change owners that we hope they can work on an alternative approach using zstd.
15:38:27 <sgallagh> decathorpe: Just walk downstairs.
15:38:51 <decathorpe> I suggest we continue discussion on devel and ask proposal owners about alternative approaches
15:39:02 <zbyszek> Do we have the required work scoped somewhere (pungi, koji)?
15:39:19 <sgallagh> I think nirik and dcantrell have the right of it, though
15:39:50 <sgallagh> We really should decide which of those three optimizations we care most about.
15:40:14 <zbyszek> I too agree with the priority order nirik listed.
15:40:25 * cmurf would like to hear mhroncok's option(s) also
15:40:25 <dcantrell> yes, agreed.  the conversation should be able change priorities and we gain/lose for each, and avoid the implementation details
15:40:27 <sgallagh> Mind restating it?
15:40:34 <dcantrell> *should be about
15:40:41 <dcantrell> sgallagh: in the ticket?
15:40:56 <sgallagh> Maybe open a new thread on Devel?
15:41:11 <sgallagh> We should agree on the problem we're solving before continuing to discuss solutions
15:41:11 <dcantrell> yeah, I can do that
15:41:42 <decathorpe> sgallagh: I agree, without knowing what we want to achieve it's hard to agree on a way to do it
15:41:51 <mhroncok> cmurf: my very vague idea was that the change goes "do things differently, for all", while we might be able to go gradually: in Fedora 32, we offer additional iso for workstation etc. with uses this, as a tech preview
15:41:55 <mhroncok> something like that
15:42:55 * nirik is not a fan or more media... :(
15:42:58 <zbyszek> mhroncok: I disagree. The space savings are not enough to justify having at least two isos, i.e. ~doubling storage size.
15:43:01 <sgallagh> I'm with nirik
15:43:26 <sgallagh> But let's start by framing the problem.
15:43:46 <dcantrell> I'm posting on devel to continue the conversation
15:43:58 <decathorpe> dcantrel++
15:44:01 <mhroncok> right, this is not very well thought
15:44:09 <mhroncok> (what i have said)
15:44:37 <mhroncok> starting by framing the problem is a good idea
15:44:41 <sgallagh> I can try to make up a Google Survey to poll people about the relative value of  space vs. installation size vs compose time
15:44:47 <decathorpe> let's continue the discussion asynchronously
15:44:50 <decathorpe> sgallagh: good idea.
15:45:03 <nirik> hard to do right, but might be worth trying...
15:45:23 <decathorpe> anything else? we're stuck on this topic for 30 minutes already
15:45:24 <sgallagh> It doesn't necessarily need to be scientifically sound.
15:45:53 <mhroncok> move on
15:45:59 <zbyszek> Thanks dcantrell and sgallagh, both routes sound useful.
15:46:02 <decathorpe> #topic Open Floor
15:46:32 <decathorpe> #undo
15:46:32 <zodbot> Removing item from minutes: <MeetBot.items.Topic object at 0x7fdff22bfb50>
15:46:46 <decathorpe> #topic Next Week's Chair
15:46:54 <decathorpe> any volunteers?
15:47:09 <zbyszek> I might miss the next two meetings...
15:47:10 <sgallagh> I haven't done it in quite some time. I suppose I'm due.
15:47:27 <dcantrell> I'd like to volunteer for the meeting after next
15:47:40 <decathorpe> #action sgallagh will chair next week's meeting
15:47:52 <decathorpe> #topic Open Floor
15:49:52 <zbyszek> decathorpe: let's move on
15:50:25 <decathorpe> you all can have 10 minutes of your day back :)
15:50:33 <decathorpe> #endmeeting