15:06:32 #startmeeting FESCo (2020-02-03) 15:06:32 Meeting started Mon Feb 3 15:06:32 2020 UTC. 15:06:32 This meeting is logged and archived in a public location. 15:06:32 The chair is decathorpe. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:06:32 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:06:32 The meeting name has been set to 'fesco_(2020-02-03)' 15:06:37 #meetingname fesco 15:06:37 The meeting name has been set to 'fesco' 15:06:44 #chair nirik, ignatenkobrain, decathorpe, zbyszek, bookwar, sgallagh, contyk, mhroncok, dcantrel 15:06:44 Current chairs: bookwar contyk dcantrel decathorpe ignatenkobrain mhroncok nirik sgallagh zbyszek 15:06:54 .hello dcantrell 15:06:58 dcantrell: Sorry, but you don't exist 15:07:00 .hello dcantrel 15:07:01 dcantrell: dcantrel 'David Cantrell' 15:07:09 (I always mistype that) 15:07:09 sorry for the delay. 15:07:12 .hello2 15:07:13 zbyszek: zbyszek 'Zbigniew Jędrzejewski-Szmek' 15:07:16 morning 15:07:17 Sorry, I forgot the time. 15:07:23 #topic Init Process 15:07:28 .hello2 15:07:28 sgallagh: sgallagh 'Stephen Gallagher' 15:07:37 .hello2 15:07:38 zbyszek: zbyszek 'Zbigniew Jędrzejewski-Szmek' 15:08:32 alright, I think there's only two topics for today, everything else is either not ready yet or was already voted on in ticket 15:09:01 That's my impression too. 15:09:19 #topic #2320 F32 System-Wide Change: Enable EarlyOOM 15:09:29 .fesco 2320 15:09:30 decathorpe: Issue #2320: F32 System-Wide Change: Enable EarlyOOM - fesco - Pagure.io - https://pagure.io/fesco/issue/2320 15:10:01 according to the last comments, the Workstation WG has decided to enable EarlyOOM by default, and we don't have to decide anything. 15:10:22 * nirik is ok with that. 15:10:39 Yeah. Proposal: we just acknowledge the WG decision. 15:10:39 does anybody want to overrule their decision? :) 15:10:42 WFM 15:10:51 I'm fine with that, I just have one question 15:11:00 I don't think we need to vote unless we propose to overrule them 15:11:03 what's the process to disable it if someone wants to disable it? Is it trivial? 15:11:20 'systemctl disable earlyoom' 15:11:21 dcantrell: Sounds like `systemctl disable earlyoom.service` I think 15:11:24 fair 15:11:28 thanks 15:11:29 yeah that should do it 15:12:00 everybody OK with ACK'ing the WG decision? I don't think we need to vote on that. 15:12:13 Yes, and this will be "remembered", in the sense that the service will not be re-enabled on package upgrades. 15:12:34 disabling/reverting to previous behavior is in the change proposal, under release notes 15:12:43 or removing it. ;) 15:13:33 #info FESCo acknowledges the Workstation WG's decision and does not want to overrule 15:13:54 #topic #2323 F32 System-Wide Change: Reduce installation media size by improving the compression ratio of SquashFS filesystem 15:14:00 .fesco 2323 15:14:03 decathorpe: Issue #2323: F32 System-Wide Change: Reduce installation media size by improving the compression ratio of SquashFS filesystem - fesco - Pagure.io - https://pagure.io/fesco/issue/2323 15:14:23 looks like there's only lack of opinion here 15:14:32 I haven't been able to follow the discussion due to being all over the place the last month. 15:15:00 I realize there are alternative proposals out, but I want to know if there is any known downside to the Change as filed. 15:15:00 yeah, some alternatives were brought up, including replacing rsync with unsquashfs, or using zstd instead of xz 15:15:39 sgallagh: the downside is more CPU usage during installation. 15:15:42 I think zstd was a winner 15:16:03 zbyszek: That doesn't seem like much of an issue. 15:16:06 Kamil Paral makes very good arguments why we don't want to go in that direction. 15:16:22 increased FS latency is also not nice for live systems 15:16:43 zbyszek: which direction? 15:16:44 If we follow the change, we save a bit in image size, but pay quite a bit in latency and CPU time. 15:17:27 zbyszek: This applies only to Live images? 15:17:34 my oversimplificiation is the proposal improves compression a bit, decreases CPU somewhat (if sticking with rsync, perhaps more with unsquashfs) 15:17:39 Or are the install DVDs/netinst.iso also affected? 15:17:47 but leaves a lot on the table, CPU and latency wise 15:18:00 I expect all ISO's produced by Anaconda are affected 15:18:21 I don't want to say that we shouldn't optimize the size, but the arguments to switch away from the current scheme and towards zstd are pretty compelling. And then we shouldn't ramp up xz settings. 15:18:22 all contain a "LiveOS" using a squashfs image 15:18:29 installation is already heavily CPU bound with depsolving and running the rpm transaction, but adding any increase in runtime to the installation process impacts users first impressions of a release 15:18:57 zbyszek: Arguments are nice, but is anyone actually working on that? 15:19:04 I think we can get faster install + smaller/as small with zstd 15:19:14 * sgallagh doesn't like rejecting a proposal on the grounds that someone *might* do it better... someday 15:19:19 nirik: that was my take-away as well 15:19:35 is there anything we can remove from the squashfs image? 15:19:56 having played the installer image game for a long time, this was sort of a constant game of chase of how to make the image small enough for all platforms and carry the required tools 15:19:59 the proposal suggests dropping the nested ext4 file system, and using plain squashfs 15:20:03 (worst offender always: ppc64) 15:20:26 dcantrell: Does that remain true for ppc64le? 15:20:27 *suggests* it doesn't state it outright, but it's been part of the discussion 15:20:40 sgallagh: sadly, yes. they just doubled the limit 15:20:40 cmurf: there are several options listed, but all of them need to be implemented in pungi first 15:20:41 :/ 15:20:54 yay 15:21:10 decathorpe: and I suppose in koji if the idea is to give releng choices 15:21:21 yeah. I assume so 15:21:27 but honestly, this is the kind of problem that just comes back around from time to time 15:21:51 but the work and testing on the plain squashfs + zstd testing has been done in lorax and anaconda 15:21:53 as written, I don't have a problem with it. I would like to know if anything could be removed from the squashfs image. And maybe some metrics on increased CPU load during install 15:22:06 whereas there's still work to be done for unsquashfs 15:22:06 perhaps we could broadly say what we want to optimize for? 15:22:07 The current proposal just means tweaking some config options, yes? Not making real changes to the compose? 15:22:23 sgallagh: that is my interpretation 15:22:35 dcantrell: what do you mean by "if anything could be removed from"? 15:22:36 * nirik is -1 for the current proposal as written. I think we can do better. ;) 15:22:44 sgallagh: all of the choices are listed here ... https://fedoraproject.org/wiki/Category:Changes/OptimizeSquashFS#User_Experience 15:22:53 Then I'm inclined to approve it conditionally through Beta, removing it for GA at FESCo's discretion 15:23:11 If there are better options using zstd, let's get those for F33 15:23:24 cmurf: when we dealt with installer image size constraints on the anaconda team, there were two approaches we could take. inventory what's in the image and look for things to remove and/or jack the compression ratio to try to squeeze it down more 15:23:50 I don't want a bunch of churn... we should decide and do it once. ;) 15:23:58 Because we can guess all we like, but actual data from Beta would be the best option. 15:24:23 nirik: I disagree; we should try it and see what works and adjust as we go 15:24:27 the change author was optimizing for space savings... 15:24:31 dcantrell: ahh gotcha, well on the Workstation WG side, if we get flagged that the image is over 2G, then we go look what caused the increase, that's about it 15:24:33 well, we'd need an actual thing to compare it to. 15:24:51 I think install time would be most important, followed by size, and only 3rd creation time 15:25:09 cmurf: understood. yeah, at some point you're out of options. but if it hadn't been done, it's worth bringing up. 15:25:18 I don't suppose pungi/koji can produce the same image twice with different options? 15:25:19 it's not yet clear if this change requires or expects Anaconda to use unsquashfs instead of rsync; not clear if Anaconda folks are agreeable to that change 15:25:48 decathorpe: It can, but it would mean a huge increase in compose time 15:25:52 if that doesn't happen, installation performance doesn't improve much at all 15:26:16 sgallagh: well that would be what's needed for an actual comparison ... 15:26:21 and storage space and mirrors and... 15:26:31 koji lacks a switch to ask for different compression options; and a separate switch to ask for plain squashfs instead of the existing ext4 nested squashfs 15:26:40 For me, the interesting benchmark is the second in the Change page: "installation time". With the proposed change, installation time increases noticeably, with zstd is cut by half. 15:26:45 decathorpe: I was thinking more in terms of whether it becomes qualitatively worse 15:26:56 zbyszek: +1 (same for me) 15:26:59 As in, do we get feedback from people saying "hey, this is really slow"? 15:27:01 worse than what? :) 15:27:06 F31 15:27:28 hoping that not too much has changed between F31 and F32 ... wll 15:27:30 Because if there's a small slowdown but no one notices... who cares? 15:28:03 haha well i noticed, well before this change, nirik can attest to my releng emails, and bcl can attest to my anaconda emails 15:28:06 Vs. if they can download it using less bandwidth (particularly for those with caps), that's a real benefitr 15:28:11 Well, if we are making changes, and we can do a change which cuts installation time by 40%, then I don't see the point in doing a change which increases the installation time by 2% to save the image size a bit. 15:28:36 zbyszek: And I ask again: who is committed to doing that work? 15:28:41 zbyszek: +1 15:28:44 I haven't heard anyone say they're stepping up 15:29:11 this change? adjusted to used zstd? 15:29:25 Dunno, maybe OPs can be convinced to go in that direction? 15:29:26 This Change has commitment from the proposers 15:29:36 Who is committing to make the other changes? 15:29:44 well that's sorta the disappointing part, is the change owner wants to do xz optimization, not zstd; the emphasis is strictly on ISO size 15:30:12 right, I am saying we tell them we would prefer to optimize for install time, size, then creation time 15:30:18 and ask them to do that. 15:30:19 zbyszek: i've been trying :D 15:30:21 Sure, and I understand there may be better options. But are we specifically discouraging them from doing this work? 15:30:21 hey. I'm late, sorry about that 15:30:38 Because we can't say "it should be done this way" and waddle off with no one working on it 15:30:41 mhroncok: o/ 15:31:17 no but we can ask about the prioritization of outcomes. like install time vs. iso size 15:31:24 sgallagh: I think we're talking past one another. The change as proposed makes a (small) step in the wrong direction. So even apart from any other proposals, I'll be voting against. 15:31:57 zbyszek: OK, that's a reasonable answer. 15:32:11 do we want to vote on the current proposal (jacking up xz compression)? 15:32:16 It just sounds to me like most folks here are discussing how they *wish* it would be done instead of discussing how it's currently proposed. 15:32:29 I don't think it would hurt to ask them... at worst they say they can't work on it. 15:32:43 sgallagh: we are the ENGINEERING committee, we like implementation details 15:33:12 decathorpe: I think we should do the vote, and then (if the outcome is negative), discuss what we propose instead. 15:33:15 I do question the reason/value of saving space in an iso image. Like where is that still a problem that we care about? 15:33:29 zbyszek: alright then 15:33:38 dcantrell: More often than you'd expect, unfortunately. 15:33:50 it's all a balancing act... 15:33:56 sgallagh: but weighted against install time. yes, it's a balancing act 15:34:02 Vote on current proposal (modyfying xz compression parameters for smaller ISO): 15:34:03 but I think we can ask that it be balanced in a better way 15:34:39 -1 15:34:42 -1 on current change, hope for zstd targeting install time first, then space, then creation time 15:34:51 -1 for the propsal as-is 15:35:05 -1 I agree that the current proposal doesn't sound like a net improvement. 15:35:13 -1 after this discussion I feel we need more info and prioritizations 15:35:28 mhroncok: vote? 15:35:39 0 for now, sorry for missing most of the discussion here 15:35:57 no problem 15:36:03 I think -5 is enough votes, right? 15:36:09 * mhroncok also has opinions about how to possibly handle this 15:36:19 decathorpe: It's a majority, so yes 15:36:30 so, should we continue on list? or ? 15:36:37 Opinions are easy. Code is hard. 15:37:02 #agree REJECTED (-5, 1, -0) proposal in its current form 15:37:12 #undo 15:37:12 Removing item from minutes: AGREED by decathorpe at 15:37:02 : REJECTED (-5, 1, -0) proposal in its current form 15:37:29 #agree REJECTED (+0, 1, -5) proposal in its current form 15:37:30 (+0, 1, -5) 15:37:36 right 15:37:41 oops. thanks 15:37:46 np 15:38:00 one advantage of the current proposal is wiring up the plain squashfs image type in pungi (and maybe, hopefully, koji) 15:38:01 guess I'll need something to lower my fever again -.- 15:38:26 But let's make sure that we communicate to the change owners that we hope they can work on an alternative approach using zstd. 15:38:27 decathorpe: Just walk downstairs. 15:38:51 I suggest we continue discussion on devel and ask proposal owners about alternative approaches 15:39:02 Do we have the required work scoped somewhere (pungi, koji)? 15:39:19 I think nirik and dcantrell have the right of it, though 15:39:50 We really should decide which of those three optimizations we care most about. 15:40:14 I too agree with the priority order nirik listed. 15:40:25 * cmurf would like to hear mhroncok's option(s) also 15:40:25 yes, agreed. the conversation should be able change priorities and we gain/lose for each, and avoid the implementation details 15:40:27 Mind restating it? 15:40:34 *should be about 15:40:41 sgallagh: in the ticket? 15:40:56 Maybe open a new thread on Devel? 15:41:11 We should agree on the problem we're solving before continuing to discuss solutions 15:41:11 yeah, I can do that 15:41:42 sgallagh: I agree, without knowing what we want to achieve it's hard to agree on a way to do it 15:41:51 cmurf: my very vague idea was that the change goes "do things differently, for all", while we might be able to go gradually: in Fedora 32, we offer additional iso for workstation etc. with uses this, as a tech preview 15:41:55 something like that 15:42:55 * nirik is not a fan or more media... :( 15:42:58 mhroncok: I disagree. The space savings are not enough to justify having at least two isos, i.e. ~doubling storage size. 15:43:01 I'm with nirik 15:43:26 But let's start by framing the problem. 15:43:46 I'm posting on devel to continue the conversation 15:43:58 dcantrel++ 15:44:01 right, this is not very well thought 15:44:09 (what i have said) 15:44:37 starting by framing the problem is a good idea 15:44:41 I can try to make up a Google Survey to poll people about the relative value of space vs. installation size vs compose time 15:44:47 let's continue the discussion asynchronously 15:44:50 sgallagh: good idea. 15:45:03 hard to do right, but might be worth trying... 15:45:23 anything else? we're stuck on this topic for 30 minutes already 15:45:24 It doesn't necessarily need to be scientifically sound. 15:45:53 move on 15:45:59 Thanks dcantrell and sgallagh, both routes sound useful. 15:46:02 #topic Open Floor 15:46:32 #undo 15:46:32 Removing item from minutes: 15:46:46 #topic Next Week's Chair 15:46:54 any volunteers? 15:47:09 I might miss the next two meetings... 15:47:10 I haven't done it in quite some time. I suppose I'm due. 15:47:27 I'd like to volunteer for the meeting after next 15:47:40 #action sgallagh will chair next week's meeting 15:47:52 #topic Open Floor 15:49:52 decathorpe: let's move on 15:50:25 you all can have 10 minutes of your day back :) 15:50:33 #endmeeting