15:00:04 <contyk> #startmeeting FESCO (2019-06-14)
15:00:04 <zodbot> Meeting started Fri Jun 14 15:00:04 2019 UTC.
15:00:04 <zodbot> This meeting is logged and archived in a public location.
15:00:04 <zodbot> The chair is contyk. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:04 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:04 <zodbot> The meeting name has been set to 'fesco_(2019-06-14)'
15:00:06 <contyk> #meetingname fesco
15:00:06 <zodbot> The meeting name has been set to 'fesco'
15:00:08 <contyk> #chair nirik, bowlofeggs, jforbes, zbyszek, bookwar, sgallagh, contyk, mhroncok, otaylor
15:00:08 <zodbot> Current chairs: bookwar bowlofeggs contyk jforbes mhroncok nirik otaylor sgallagh zbyszek
15:00:09 <zbyszek> .hello2
15:00:10 <zodbot> zbyszek: zbyszek 'Zbigniew Jędrzejewski-Szmek' <zbyszek@in.waw.pl>
15:00:10 <contyk> #topic init process
15:00:12 <contyk> .hello psabata
15:00:13 <zodbot> contyk: psabata 'Petr Šabata' <psabata@redhat.com>
15:00:17 <sgallagh> .hello2
15:00:19 <bcotton> .hello2
15:00:22 <zodbot> sgallagh: sgallagh 'Stephen Gallagher' <sgallagh@redhat.com>
15:00:24 <zodbot> bcotton: bcotton 'Ben Cotton' <bcotton@redhat.com>
15:00:32 <sgallagh> I have a hard stop in an hour, but it looks like a fairly short agenda at least
15:00:52 <contyk> if we even get quorum
15:01:18 <zbyszek> mhroncok has a talk in 20 minutes, he'll not be here.
15:01:31 <contyk> yeah, and he'd probably want to comment on the Taiga one
15:01:33 <jforbes> .hello2
15:01:34 <zodbot> jforbes: jforbes 'Justin M. Forbes' <jforbes@redhat.com>
15:03:00 <contyk> need one more
15:03:09 <bowlofeggs> .hello2
15:03:09 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <rbarlow@redhat.com>
15:03:11 <bcotton> i can pretend to be miro
15:03:23 <contyk> yay
15:03:42 <bowlofeggs> bcotton: the election ends before the next fesco meeting, right?
15:03:54 <contyk> it does
15:04:05 <bowlofeggs> ah, then this is my last fesco meeting
15:04:13 <bcotton> bowlofeggs: yes. it ends at 23:59 UTC on 20 June
15:04:15 <bowlofeggs> i'll miss you all, it was an honor to serve with you
15:04:15 <bcotton> RIP bowlofeggs
15:04:18 <bcotton> bowlofeggs++
15:04:19 <zodbot> bcotton: Karma for bowlofeggs changed to 15 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:04:19 <bowlofeggs> hahaha
15:04:27 <contyk> might be my last one, too
15:04:39 <jforbes> thanks for serving
15:04:42 <contyk> okay, let's begin with the follow-up
15:04:48 <contyk> #topic #2136 F32 Self-Contained Change: Track Changes in Taiga
15:04:49 <contyk> .fesco 2136
15:04:51 <contyk> https://pagure.io/fesco/issue/2136
15:04:52 <zbyszek> thanks for serving
15:04:54 <zodbot> contyk: Issue #2136: F32 Self-Contained Change: Track Changes in Taiga - fesco - Pagure.io - https://pagure.io/fesco/issue/2136
15:05:01 <zbyszek> bowlofeggs++
15:05:01 <zodbot> zbyszek: Karma for bowlofeggs changed to 16 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:05:04 <zbyszek> contyk++
15:05:05 <zodbot> zbyszek: Karma for psabata changed to 7 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:05:11 <contyk> :]
15:05:12 <bowlofeggs> cookies!
15:05:21 <bowlofeggs> i am +1 to taiga
15:05:26 <contyk> me too, still
15:05:31 <contyk> bcotton: anything new?
15:06:21 <jforbes> I am still +1.
15:06:22 * zbyszek was +1 in the ticket to mhroncok's proposal
15:06:25 <bcotton> so as FESCo requested, i contacted vondruch to test. he expressed to miro that he as no interest in trying it
15:07:20 <zbyszek> So I think we can treat that part of discussion as resolved.
15:07:54 <contyk> I actually wouldn't limit the testing to f32
15:08:08 <bcotton> yeah. i'm certainly open to community feedback, but if someone's not willing to look at it, then there's not much i can do
15:08:10 <contyk> I'd just go with it and if it proves to be deficient and bad, we can discuss switching back or to something else
15:08:32 <zbyszek> Right.
15:09:00 <contyk> if we just limit that to the next release, we'll need to have the discussion again for f33
15:09:56 <zbyszek> contyk: no, I don't think that was the intent.
15:10:01 <jforbes> Well, that was the discussion from last week, and we had enough +1s.  Anyway, I am not opposed to the proposal, but I don't think it is necessary either
15:10:17 <contyk> I know it wasn't the intent
15:10:36 <bowlofeggs> i think we can just approve it
15:10:37 <sgallagh> Yeah, I'm also -1 to Miro's restriction
15:10:37 <contyk> but I'd just revisit the topic only once/if there's a need for it
15:10:41 <jforbes> zbyszek: yes, the way I read the proposal was it is approved, but we have to review before F33 to make it either permanent or revert
15:10:47 <sgallagh> Let's just move ahead and back it out if it becomes a problem
15:10:55 <bowlofeggs> if it isn't great someone can write a new change to switch to something else
15:11:09 <bowlofeggs> i dont' think we need to have a trial period and vote on it again later
15:11:13 <bowlofeggs> that's just more work for everyone
15:11:16 <sgallagh> bowlofeggs: But what if it's so bad that they can't file a corrective Change? :-P
15:11:24 <bowlofeggs> and the result is the same - we can always switch anyway
15:11:27 <bcotton> sgallagh: don't give away my secret plans
15:11:28 <bowlofeggs> sgallagh: hahaha
15:11:29 <contyk> ;)
15:11:35 <zbyszek> OK, you are right. Let's approve it.
15:12:08 <contyk> proposed #agreed The Taiga switch is approved without any specific release restrictions. We can revisit in the future if there is a need for it.
15:12:14 <sgallagh> ack
15:12:33 <jforbes> +1
15:12:35 <zbyszek> +1
15:13:02 <contyk> bowlofeggs?
15:13:15 <bowlofeggs> +1
15:13:21 <contyk> #agreed The Taiga switch is approved without any specific release restrictions. We can revisit in the future if there is a need for it.
15:13:33 <contyk> okay
15:13:36 <contyk> #topic #2144 F31 System-Wide Change: Switch RPMs to zstd compression
15:13:37 <contyk> .fesco 2144
15:13:39 <contyk> https://pagure.io/fesco/issue/2144
15:13:39 <zodbot> contyk: Issue #2144: F31 System-Wide Change: Switch RPMs to zstd compression - fesco - Pagure.io - https://pagure.io/fesco/issue/2144
15:13:41 <contyk> #chair dmach
15:13:41 <zodbot> Current chairs: bookwar bowlofeggs contyk dmach jforbes mhroncok nirik otaylor sgallagh zbyszek
15:14:14 <bowlofeggs> this one seems pretty controversial
15:14:31 <jforbes> Right, and part of the fuss is RHEL and such are not using it.
15:15:00 <sgallagh> jforbes: Don't have the support compiled in, specifically
15:15:06 <contyk> I'm in favor of progress as long as we have a plan to make it work
15:15:09 <jforbes> But, RHEL is unlikely to get it until Fedora has used it, so this seems a chicken or egg problem
15:15:38 <dmach> jforbes, +1
15:15:43 <sgallagh> I'm frankly starting to come down on the side of "the hoped-for benefits aren't sufficiently guaranteed to justify the risks and incompatibilities that will fall out of it".
15:16:01 <bowlofeggs> i'm undecided on this one
15:16:11 <dmach> there are concerns about potential problems in the infrastructure; I'd like to know more about the possible issues so we can address that
15:16:23 <sgallagh> From my reading of the thread, the space and decompression savings aren't as clear-cut as the original proposal suggested.
15:16:55 <jforbes> So, what about the path forward of compiled in support in Fedora which could then perhaps make RHEL updates, as an intermediary step?
15:17:19 <sgallagh> jforbes: It's been in Fedora for some time
15:17:27 <zbyszek> sgallagh: I'd say quite the opposite: the decompression savings have been shown pretty clearly, and the potential problems are unclear.
15:17:31 <sgallagh> It was available for RHEL 8.0, but RHT opted not to flip that flag on
15:17:40 <sgallagh> (presumably to avoid needing to carry the lib)
15:17:49 <jforbes> Ahh
15:18:27 <zbyszek> The part I'm unclear on is on the question of multithreading
15:19:16 <dmach> zbyszek, could you be more specific?
15:19:19 <zbyszek> I.e. whether we can safely enable multithreaded compression and decompression of xz and zstd.
15:19:46 <bowlofeggs> yeah i'm convinced that the savings are established; for me it's the question of how that balances against the potential compatibility issues
15:20:01 <bowlofeggs> also, i was surprised to hear that so many people use fedora RPMs on EL?
15:20:09 <bowlofeggs> that seems kinda odd to me
15:20:18 <bowlofeggs> but i guess it's not odd because it sounds common?
15:20:33 <bcotton> bowlofeggs: i don't think it's *use* so much as *interact with*, which maybe is less of a problem than was first thought https://pagure.io/fesco/issue/2144#comment-575856
15:20:35 <jforbes> Well, I could see a lot more people using fedora srpms on RHEL
15:20:36 <contyk> I guess that's because EPEL is lagging behind or doesn't ship the content at all
15:20:59 <bowlofeggs> jforbes: ah and SRPMs would also be compressed the new way i suppose
15:21:39 <dmach> zbyszek, multi-threaded zstd compression produces the same result as single-threaded; AFAIK multi-threaded xz compression is not reproducible (but I may have missed something)
15:22:05 <jforbes> I haven't looked at the rpm code in years, but I would guess there isn't a flag to use different compression on src vs binary
15:22:17 <zbyszek> dmach: thanks. So this makes the gap between xz and zstd even bigger, IIUC.
15:23:00 <dmach> jforbes, srpm compression change is not part of the proposal; IIRC it's gz and makes no sense to change it; it compresses an already compressed tarball anyway
15:23:24 <jforbes> aha, so srpms would still work
15:23:28 <dmach> yes
15:23:29 <contyk> that's good
15:23:30 <bowlofeggs> oh that's interesting
15:23:45 <bowlofeggs> and it sounds like teh headers would also be readable from RHEL to help things like satellite still work
15:23:54 <bowlofeggs> just not the payload
15:23:57 <bowlofeggs> according to igor
15:24:14 <dmach> correct, headers are uncompressed
15:24:58 <bowlofeggs> i would suggest that it's reasonable to say that installing Fedora RPMs on not-Fedora isn't a stated supported use case for the Fedora project
15:25:20 <bowlofeggs> we can acknowledge that people do that, but i don't think the Fedora project has documented that we support doing that
15:25:24 <jforbes> And F29+ can all handle the compression without issue?
15:25:56 <dmach> yes, the change landed in f28
15:26:08 <ignatenkobrain> yes
15:26:27 <contyk> does this affect non-Fedora koji builders or composers?
15:26:44 <jforbes> Though I suppose if someone were running F27 still, they would be unable to upgrade without going into an intermediate
15:27:04 <sgallagh> jforbes: We officially support N-2 only
15:27:28 <sgallagh> So they could only expect F27->F29 to work anyhow
15:27:36 <bowlofeggs> mizdebsk: are you around? if so, do you know if the zstd compression would affect our koji builders?
15:27:59 <dmach> contyk, it depends on which rpm version they use and if it has zstd enabled; for example mock on el7 won't work with Fedora RPMs without a patched rpm
15:28:10 <jforbes> sgallagh: If someone is running 1 year+ unsupported, I don't think an extra step on upgrade would be too much to ask
15:28:30 <sgallagh> jforbes: I think we're agreeing here. It's a non-issue
15:28:32 <jforbes> And I know what we support, but I also see what people actually do, or attempt to do
15:28:43 <contyk> indeed, I'm mostly worried about rhel/epel7 builders if we still have any
15:28:49 <contyk> and sounds like that would be an issue
15:28:53 <kwizart> most builders are at least fc29 based IIRC
15:29:00 <ignatenkobrain> all Fedora builders are now on f29
15:29:13 <kwizart> even armhfp ?
15:29:27 <ignatenkobrain> yes
15:30:02 <contyk> cool then
15:30:09 <sgallagh> Doesn't mock do a two-layer thing now where it uses the target system's RPM and dnf to install software?
15:30:10 <bowlofeggs> i think i lean towards approving the proposal, in the spirit of "First" and "leading edgeness"
15:30:14 <sgallagh> I thought I remembered that being the case
15:30:16 <contyk> any other services that could be affected, like koschei?
15:30:52 <bowlofeggs> contyk: mizdebsk could answer that
15:30:54 <ignatenkobrain> sgallagh: no, we don't
15:31:04 <ignatenkobrain> contyk: koschei is not affected
15:31:05 <bowlofeggs> maybe we should get a sign off from some stakeholders before voting
15:31:08 <sgallagh> ignatenkobrain: Well get on that :)
15:31:10 <contyk> and even then you'd still need to unpack the first package set
15:31:16 <bowlofeggs> like infra, mizdebsk, copr, maybe others?
15:31:24 <contyk> bowlofeggs: +1
15:31:36 <ignatenkobrain> koschei is reading just a headers from koji, nothing else
15:31:41 <bowlofeggs> not sure who else we should ask
15:31:44 <bowlofeggs> releng
15:32:33 <bowlofeggs> i think i'm ok with fedora RPMs not being installable on EL, but i would like to know that our infra and other systems are going to work ok
15:32:39 <zbyszek> But doesn't a system-wide proposal already require releng ask?
15:32:47 <bowlofeggs> yeah i think it does
15:32:56 <contyk> not anymore, I think
15:33:02 <bcotton> system-wide still does
15:33:22 <ignatenkobrain> https://pagure.io/releng/issue/8395
15:33:39 <contyk> okay
15:33:43 <zbyszek> OK, so that's done.
15:33:45 <bowlofeggs> https://pagure.io/releng/issue/8395
15:33:50 <bowlofeggs> heh, ignatenkobrain beat me to it
15:33:54 <contyk> I'd still give it a week to gather feedback from other potential stakeholders
15:33:56 <bcotton> it's not clear how closely mboddu looked at it, though
15:33:57 <sgallagh> So mboddu acknowledged that it needs a mass-rebuild, but I'm not sure that means he reviewed its potential impact
15:33:59 <contyk> and vote next Friday
15:34:06 <bowlofeggs> ok so we have a releng sign off
15:34:24 <bowlofeggs> sgallagh: yeah that's true
15:34:44 * mboddu reading back
15:35:03 <zbyszek> I don't think delaying the vote one week would change much.
15:35:07 <bowlofeggs> proposal: Request feedback from infra, koschei, releng, and copr about whether their systems will work with zstd compression
15:35:07 <sgallagh> mboddu: Switching RPMs to zstd compression: what will it break in the release process?
15:35:18 <sgallagh> Anything obvious?
15:35:41 <sgallagh> In particular: what parts of the process might require the RPMs to be handled by EL7 systems?
15:35:59 <bcotton> or EL8
15:35:59 <zbyszek> Instead, we should make sure to announce this change extra-widely so that non-Fedora stakeholders are notified sufficiently early to make necessary adjustments.
15:36:06 <mboddu> sgallagh: Oh right, I haven't thought about EPEL
15:36:07 <sgallagh> or EL8
15:36:20 <sgallagh> mboddu: We're not making this change for EPEL-built RPMs
15:36:23 <zbyszek> So something like Fedora Magazine article.
15:37:26 <mboddu> sgallagh: Then I guess, I wouldn't expect any obvious breakages
15:37:28 <sgallagh> mboddu: I'm concerned about any place that the Fedora 31 build-and-release pipeline might still be running RHEL systems rather than Fedora systems
15:37:37 <sgallagh> That's what I wanted to hear :)
15:38:01 <bowlofeggs> can we find a way to insert some extra scriptlets with this change to slow it back down to its current speed and give wwoods some more work to do?
15:38:07 <sgallagh> OK, I'm willing to vote +1 today
15:38:22 <sgallagh> .fire bowlofeggs
15:38:22 <zodbot> adamw fires bowlofeggs
15:38:24 <bowlofeggs> haha
15:38:42 <zbyszek> bowlofeggs: there's a systemd bug open that might do just that :(
15:38:47 <bowlofeggs> i think i'd prefer to delay voting to hear from at least infra (nirik), but that also means i won't be voting at all since it's my last fesco ☺
15:38:55 <bowlofeggs> zbyszek: hahah
15:39:49 <bowlofeggs> i do lean +1 though, it's just that i'd like to know that infra formally thinks this will work
15:39:56 <bowlofeggs> i could do a conditional +1 though
15:40:03 <contyk> we could also move the next meeting so that it happens before the elections finish ;)
15:40:07 <bowlofeggs> haha yeah
15:40:14 <sgallagh> I'm good with a "+1 from FESCo, please figure out how to make it work before activating it"
15:40:25 <bowlofeggs> sgallagh: yeah i'd +1 that
15:40:30 <zbyszek> sgallagh: +1
15:40:38 <mboddu> I would suggest waiting on infra +1 as well
15:40:45 <jforbes> nirik in thread seemed to think it would work, but would require some changes to infra.
15:40:51 <contyk> I'd rather get the feedback first because some of those things might be complicated if they are affected
15:40:53 <ignatenkobrain> bowlofeggs: then I'll +1 it on behalf of you on next FESCo after voting (if I get here)
15:41:06 <contyk> but if everything is fine, I'm also +1
15:41:10 <bowlofeggs> ignatenkobrain: ☺
15:41:53 <sgallagh> I mean, this is a Change Proposal. Our job is "do we want this Change to happen?".
15:42:03 <contyk> ok, so, vote now and deal with the fires if required?
15:42:09 <sgallagh> It sounds like we agree that it does, and what's left is figuring out how to avoid breakage.
15:42:16 <sgallagh> Which doesn't need to happen at our level.
15:42:20 <jforbes> Right, and I am +1 to the change itself
15:42:23 <sgallagh> That's why we have Contingency Plans
15:42:44 * zbyszek would love to vote and move on to other things
15:42:53 <contyk> ok, so proposed #agreed F31 System-Wide Change: Switch RPMs to zstd compression is approved.
15:42:59 <sgallagh> +1
15:43:11 <zbyszek> +1
15:43:13 <bowlofeggs> sgallagh: yeah that makes sense
15:43:18 <bowlofeggs> +1
15:43:22 <jforbes> +1
15:43:34 <contyk> #agreed F31 System-Wide Change: Switch RPMs to zstd compression is approved. (+5, 0, -0)
15:43:54 <contyk> #topic Next week's chair
15:44:04 <contyk> preferrably someone who will still be on FESCo
15:44:08 <contyk> but bowlofeggs could also do it, I guess
15:44:14 <jforbes> I can do it
15:44:14 <sgallagh> I'll be on PTO next week (and possibly not on FESCo, if the voting doesn't go my way)
15:44:30 <contyk> jforbes++
15:44:41 <contyk> #action jforbes will chair the next meeting
15:44:43 <sgallagh> jforbes++
15:44:43 <zodbot> sgallagh: Karma for jforbes changed to 4 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:44:47 <sgallagh> contyk++
15:44:49 <contyk> #topic Open Floor
15:44:59 <bowlofeggs> haha
15:45:12 <bowlofeggs> well everyone, i've really enjoyed being on FESCo the past two years
15:45:28 <bowlofeggs> it's been fun and challenging and at times a real brain teaser
15:45:40 <ignatenkobrain> You probably want to discuss contyk ticket
15:45:43 <bowlofeggs> in particular, i did enjoy being forced to think abotu parts of the project i wouldn't know about otherwise
15:45:46 <ignatenkobrain> About modularity
15:45:49 <sgallagh> Yeah, nothing like FESCo to remind you that you're a small speck in the Open Source Universe :)
15:45:58 <bowlofeggs> it's a great way to expand your view
15:46:02 <bowlofeggs> haha yes
15:46:18 <contyk> I'd keep the discussion on the list for now
15:46:25 <bowlofeggs> i think i will run again in the future, and i wish those who are running now good luck!
15:46:31 <contyk> we need to figure out how to proceed with everything
15:46:49 <ignatenkobrain> contyk: fine with me
15:47:10 <bowlofeggs> the vote was hard for a good reason too - all the candidates are good
15:47:32 <bowlofeggs> so we'll have a good FESCo next Friday no matter who is chosen
15:48:17 <contyk> will close in two minutes if nothing else pops up
15:49:20 <zbyszek> ignatenkobrain, contyk: I think it's better to hold the discussion on fedora-devel. More poeople can participate there.
15:49:30 <contyk> yep
15:50:01 <contyk> alright
15:50:04 <contyk> #endmeeting