14:00:29 #startmeeting FESCO (2020-10-07) 14:00:29 Meeting started Wed Oct 7 14:00:29 2020 UTC. 14:00:29 This meeting is logged and archived in a public location. 14:00:29 The chair is sgallagh. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:29 Useful Commands: #action #agreed #halp #info #idea #link #topic. 14:00:29 The meeting name has been set to 'fesco_(2020-10-07)' 14:00:29 #meetingname fesco 14:00:29 The meeting name has been set to 'fesco' 14:00:29 #chair nirik, ignatenkobrain, decathorpe, zbyszek, sgallagh, mhroncok, dcantrell, cverna, Conan_Kudo, Pharaoh_Atem, Son_Goku, King_InuYasha, Sir_Gallantmon, Eighth_Doctor 14:00:29 Current chairs: Conan_Kudo Eighth_Doctor King_InuYasha Pharaoh_Atem Sir_Gallantmon Son_Goku cverna dcantrell decathorpe ignatenkobrain mhroncok nirik sgallagh zbyszek 14:00:29 #topic init process 14:00:37 .hello2 14:00:38 zbyszek: zbyszek 'Zbigniew Jędrzejewski-Szmek' 14:00:39 Sorry for the late agenda, but it's on the list now. 14:00:49 hello 14:00:58 morning 14:00:58 09:59:16 – perfect timing. 14:01:37 * sgallagh preens 14:01:43 .hello ngompa 14:01:44 King_InuYasha: ngompa 'Neal Gompa' 14:02:12 .hello2 14:02:13 dcantrell: dcantrell 'David Cantrell' 14:02:16 .hello2 14:02:18 bcotton: bcotton 'Ben Cotton' 14:02:30 .hello2 14:02:31 decathorpe: decathorpe 'Fabio Valentini' 14:03:10 OK, I think we can get started 14:03:23 #topic #2441 F34 System-Wide Change: RPM-level auto release and changelog bumping 14:03:23 .fesco 2441 14:03:23 sgallagh: Issue #2441: F34 System-Wide Change: RPM-level auto release and changelog bumping - fesco - Pagure.io - https://pagure.io/fesco/issue/2441 14:03:53 This was deferred until 2440 was resolved. 14:04:02 Which it has been. 14:04:03 * nirik wonders if nim is ok, haven't seen them on irc or mailing lists recently. ;( 14:04:35 yeah :( 14:04:41 2440 has been resolved, but not implemented. 14:05:29 I think we should wait until nim is back. I don't see any reason to hurry. 14:05:55 I agree 14:06:07 same here 14:06:11 OK, that's fine with me. I just wanted to try to get it back on the agenda. 14:06:24 But if nim is unreachable, there's nothing to do here. 14:06:56 proposed #agreed We need more information from the Change owner who is out of contact. We will wait until they return. 14:07:06 +1 14:07:09 hey, sorry for being late 14:07:20 mhroncok: No worries. You didn't miss anything. 14:07:21 +1 14:07:28 sgallagh: what chnage is this? 14:07:36 .fesco 2441 14:07:37 sgallagh: Issue #2441: F34 System-Wide Change: RPM-level auto release and changelog bumping - fesco - Pagure.io - https://pagure.io/fesco/issue/2441 14:08:01 +1 14:08:45 +1 to sgallagh proposal 14:08:58 +1 to sgallagh's proposal to wait until nim returns 14:09:10 #agreed We need more information from the Change owner who is out of contact. We will wait until they return. (+6, 0, -0) 14:09:36 #topic #2474 F34 System-Wide Change: Rust Crate Packages For Release Branches 14:09:36 .fesco 2474 14:09:37 sgallagh: Issue #2474: F34 System-Wide Change: Rust Crate Packages For Release Branches - fesco - Pagure.io - https://pagure.io/fesco/issue/2474 14:09:52 This is ignatenkobrain's change, but he's not here today. 14:09:53 we were hoping to have igor chime in here... but I don't know that he did? 14:09:59 sgallagh: it's decathorpe's change 14:10:05 No. 14:10:07 Oh my mistake 14:10:24 I haven't gotten feedback from him for the past few weeks :( 14:10:27 No. → I meant that ignatenkobrain hasn't responded. 14:10:56 I assume we should just proceed with counting the votes 14:10:58 Has anyone heard from him recently? 14:11:06 not on IRC 14:11:24 Via any communication mechanism? 14:11:28 he did briefly talk to me on Sunday, but he's otherwise been incommunicado 14:11:50 OK, so he's not in a hospital somewhere at least. 14:11:57 at least, I hope not 14:12:49 * nirik is fine waiting or just collecting votes now. 14:13:02 I'm going to suggest we re-vote based on a slightly modified proposal: 14:14:25 proposed #agreed FESCo approves the new Rust packaging approach and grants a general exception to skip re-review for compat packages and to allow the backwards-incompatible -devel change. 14:14:39 sgallagh: +1 14:14:44 one sec 14:14:51 I'm 0 on this Change 14:14:54 sgallagh: compat packages are already exempt from review 14:15:01 I think we don't require re-review for compat packages. 14:15:02 yes 14:15:02 Exactly. 14:15:02 Are they? I missed that 14:15:09 * sgallagh revises 14:15:10 yeah, that was changed a couple of years ago 14:15:17 * nirik nods 14:15:22 I think it was LLVM that triggered that change 14:15:34 proposed #agreed FESCo approves the new Rust packaging approach and grants an exception to allow the backwards-incompatible -devel change. 14:15:35 it was a pain to review new LLVM versions every time the graphics stack required updates 14:15:45 sgallagh: +1 14:15:57 yeah, other than that, +1 to the proposal 14:16:04 +1 14:16:07 (obviously :)) 14:16:09 +1 14:16:21 +1 14:16:21 And what is "backwards-incompatible -devel change"? 14:16:39 zbyszek: semver major bumps in crate -devel packages 14:16:55 exactly, but those are not installed by users in any case 14:17:15 OK. +1 14:17:25 +1 14:17:26 decathorpe: well unless they install them in which case they are installed :P 14:18:12 well, they are useless outside of RPM builds, so updating them is no issue either 14:18:46 #agreed FESCo approves the new Rust packaging approach and grants an exception to allow the backwards-incompatible -devel change. (+7, 0, -0) 14:19:03 #topic #2479 F33 UEFI Secure Boot signing keys 14:19:03 .fesco 2479 14:19:04 sgallagh: Issue #2479: F33 UEFI Secure Boot signing keys - fesco - Pagure.io - https://pagure.io/fesco/issue/2479 14:19:33 we still need more info here, probibly from pjones 14:19:41 I asked pjones to join and I think he's here 14:20:10 yeah, pjones seems to be here 14:20:11 I *think* I read that what we thought was this issue being hit in the wild was a red herring. 14:20:17 it would be bad if in a few months we would be in a situation that f33 images won't boot ... 14:20:20 other than that :) 14:20:28 respinning all media would be a ginormous pain, possible, but a pain... and brings up tons of questions on it's own. ;( 14:20:43 also, shim doesn't build 14:20:50 right 14:21:10 mhroncok: /building/ is a problem I can solve in minutes 14:21:10 how long was the "temporary exception from FTBFS policy" expected to last anyway? 14:21:20 the question is if doing a build is the right thing to do 14:21:21 pjones: oh, please do 14:21:30 I'm not sure we should ever permit exceptions like that :/ 14:21:42 decathorpe: until upstream release 16 14:21:47 decathorpe: whoch ha snot happened yet 14:21:54 and may not happen for a long time :( 14:21:55 *which has not 14:21:59 having something built but not signed/used is not very useful. 14:22:16 nirik: it's more useful than a situation where we can't build it at all when we *need* to 14:22:25 yeah, so right now I'm trying to avoid getting another one signed unless we have to 14:22:31 King_InuYasha: if we had not permitted the exeption, tha package would have been retired 14:22:45 I'm not sure how 14:23:19 nirik: the actual signed shim doesn't build anything 14:23:29 having a shim-unsigned package that can't compile is *bad* 14:23:45 because of the situation we have *right now* 14:23:48 so I'm more interested in #2479 right now 14:24:03 dcantrell: good point 14:24:31 sgallagh mentioned the analysis currently points to this being a red herring 14:24:33 yeah, lets stop arguing this... 14:24:39 pjones, do you have any updates on #2479? 14:24:44 not as of 17 hours ago 14:24:58 there's a comment that indicates that there might actually be a real revocation happening 14:25:56 so it' 14:26:14 it's looking possible that ubuntu is applying an update by default which there had been agreement to not yet deploy 14:26:26 :( 14:26:27 and I'm trying to chase that down with them, because they shouldn't be doing that 14:26:42 if I can't get them to pull it, then we'll have to do a build and get it signed 14:26:58 they pushed out Ubuntu 20.04.1 with that sb update 14:27:03 so it's on the media basically forever 14:27:48 yeah, I'm not going to comment on what their update processes are and whether that's a solvable problem. that's a question for them. 14:28:07 it may be that they can, and soon, it may not. 14:28:09 pjones: so, it would then be helpfull for us to actually make this a blocker... so we can check if we need to fix it on our end or not before our release? 14:29:05 nirik: I would very much rather solve it on their end, and making it a blocker seems like it encourages solving it on our end? 14:29:31 pjones: well, if they solve it we can say 'no longer a blocker, go on' 14:29:38 pjones: not necessarily 14:29:44 but at least we have that to make sure we solve it one way or another 14:29:51 +1 14:29:57 * King_InuYasha shrugs 14:29:58 I have to admit I'm confused about why Ubuntu's change affects us. Is this a case of dual-booting rendering us unbootable? 14:29:59 the real question is how many people we actually expect to see the problem; if we can't fix the dbx deployment, does it actually matter that it's in this release vs next? 14:30:09 sgallagh: they have a secureboot-db package that auto-updates firmware keys 14:30:13 sgallagh: yes 14:30:20 OK, thanks for the clarification 14:30:30 sgallagh: where "dual boot" also includes "serial unrelated installations" in this case 14:30:35 I'm somewhat surprised that _we don't_ have such a thing 14:30:41 Ugh 14:30:42 but I guess we expect fwupd to do that 14:30:50 OK, that's... pretty terrible. 14:30:51 I think those of us here are likely the type crowd most likely to see this problem 14:31:01 no, we have exactly the same thing in dbxtool, we just haven't deployed the new data in it yet 14:31:08 ah 14:31:10 good to know 14:31:25 so ubuntu pushed data early and that may force our hand 14:31:30 yup 14:31:33 I really don't like the idea of a bug existing that prevents people from switching from Ubuntu to us. 14:31:55 knowing what I know about their processes, I see it as unlikely that we can get them to fix it 14:32:00 pjones: is this ubuntu thing exposing the problem originally described in the ticket sooner than expected, or is the ubuntu thing an additional problem? 14:32:13 let's file an anti-trust suit :) 14:32:14 let's file an anti-trust suit :) 14:32:18 hah 14:32:54 sgallagh: it's even more unfortunate when you consider the number of laptops that ship with Ubuntu or a derivative of it preinstalled and people often come to Fedora from that 14:32:58 mhroncok: well the problem in the ticket is a problem /because/ it's sooner than expected 14:33:37 pjones: if this have not happened in ubuntu now, the pronbelm in the ticket would still be a problem later, no? 14:34:11 mhroncok: if we can make them not deploy it for several months, we will be at a point where I am no longer trying to avoid doing another signed build. 14:34:19 mhroncok: As I understand it, there wouldn't have been a problem if the distros coordinated the updates. 14:34:37 pjones: but we would eb at a point where we need to respin the images with our new signed build 14:35:06 mhroncok: well we would hopefully have another release pending 14:35:16 right 14:35:28 if we can make this an F34 problem, it's not a problem. 14:35:28 * nirik doesn't know how long the ramp is... if we update in f34 does f33 stop working? or is it longer than our support cycle? 14:36:00 nirik: well at that point yes we would have unusable f33 images for a relatively small set of machines, but f34 would work. 14:36:07 Is it possible to detect this specific issue and display a message to the user advising them that they should install without secure boot or wait until F34 and provide a link to a detailed explanation? 14:36:17 would we want to respin f33 media with an update? Maybe, I could see it, but maybe not. 14:36:32 sgallagh: I think that's best handled as a known bug/release note type thing 14:36:35 no, we can't "display" anything, we don't have any code running 14:36:57 respinning is possible, but a big pain. ;) 14:36:58 dcantrell: Well, I was curious if we could make it more immediately visible, but yeah. 14:36:59 are there still respin SIGs that respin install media for stable releases with all the updates vacuumed up? 14:37:04 pjones: Ack 14:37:06 sgallagh: fair 14:37:13 dcantrell: yes, there is and they do. ;) 14:37:14 and if we ship the signed build now, we would fix the ubuntu problem and the 33/34 problem as well? 14:37:15 dcantrell: Yes 14:37:32 pjones: what's your motivation for trying to avoid a new signed build? 14:37:33 we do not respin install media 14:37:39 we only respin live media 14:37:39 ok, so if this is left as an F34 issue with an update dropped in to F33 then for the affected machines wanting F33 they can use a respin iso? 14:37:47 there is no way to respin install media 14:37:55 mhroncok: signing a build can take an indeterminate amount of time 14:38:00 well that's just not right, but either way you can install from live media 14:38:06 sure there is. you have to basically make another release. ;) 14:38:07 Since it goes off to Microsoft and they respond whenever they get around to it. 14:38:16 mhroncok: I would prefer not to answer that in a public, logged forum 14:38:39 :( 14:39:41 sgallagh: it isn't anything near that dire 14:39:50 so anyhow, how about we make that bug a final blocker so we have a place to track this and wait for more info? Or I suppose we could just propose it as a blocker and leave it in that state until we know more 14:40:01 It's already proposed as a blocker 14:40:12 I think I'd be fine with just ignoring the issue for now. 14:40:12 I already proposed it as a blocker as soon as I discovered the breakage 14:40:23 ah, it already is. yeah 14:40:31 I'm not sure it needs to be a blocker 14:40:35 zbyszek: yeah. 14:40:45 I'd rtaher have it as a blcoker 14:40:50 me too 14:40:59 with "ubuntu managed to fix this somehow on thair end" as and acceptable fix 14:41:01 I would also prefer that pjones go and start the process to get a new dual-signed shim 14:41:41 there's upstream work we'd very much prefer we finish before I do that 14:41:42 if the time is "indeterminate" better sooner than later 14:41:49 * nirik trusts pjones to know what he is doing. :) 14:42:00 What nirik said 14:42:03 same 14:42:15 proposal: close 2479 and let the bug go thru the normal blocker process 14:42:20 I trust pjones too, but that doesn't excuse this mess at all 14:42:57 nirik: -1 14:42:58 nirik: -1 I think that we need to block on having a mitigation plan in place at the least 14:43:17 sgallagh++ 14:43:20 nirik: -1 14:43:43 so, you all want to make it a fesco blocker? 14:43:51 I do 14:43:57 * nirik is -1 to that. 14:44:06 the problem I have with this being a blocker is that because it requires the signing from microsoft, we risk impacting the schedule because we don't have a guaranteed delivery back from them on the signed shim. we've already stated this impacts a small subset of systems, so I'm not sure it's a huge risk 14:44:09 proposal: FESCo considers this serious enough to warrant being a blocker, though the resolution does not necessarily have to be technical. 14:44:28 nirik: yes 14:44:40 sgallagh: +1 14:44:50 sgallagh: -1 14:45:10 dcantrell: the problem is that it's an increasing subset as updates and installations roll out 14:45:29 and to be blunt, Fedora 33 will fail on a lot of reviewer's computers because of this 14:46:00 maybe that's a small subset, but that has a disproportionate impact 14:46:06 that's a fair point 14:46:29 it looks like it already has a lot of support in the normal process to be a blocker... 14:46:30 King_InuYasha: Indeed, that is definitely a risk 14:46:30 I don't want Fedora 33 to be known as a flop of a release because of this mess 14:46:31 there is no real way to have that spin positive 14:46:57 King_InuYasha: we want it to flop for other reasons :) 14:47:06 * King_InuYasha snorts 14:47:07 .fire dcantrell 14:47:07 adamw fires dcantrell 14:47:12 haha 14:47:58 well, if you wall want to make it a blocker don't let me stop you. ;) other votes? 14:48:04 I figured that the part of my proposal after the comma was sufficiently weaselly for this. 14:48:11 ok, so I would say if we make this a blocker then we have to delegate decision making at that point to pjones and trust the judgement of the boot loader team 14:48:15 But feel free to counter-propose if someone has a better phrasing 14:48:57 I'm fine with the existing phrasing, but I suspect it will wind up being that we'll need to ship an updated shim 14:49:20 ¯\_(ツ)_/¯ 14:50:02 to be clear, blockers are based on behavior, not implementation. so there's always an implicit trust of the people working on the problem to solve it in the best way possible given the state of the universe 14:50:12 ack 14:50:23 0, I don't understand the operational constrains enough to vote meaningfully. 14:50:49 0 as well, I delegate boot loader decisions to the people who understand them :) 14:50:53 are we voting on sgallagh's proposal "fesco blcoker, not neccessarily technical" ? 14:50:58 mhroncok: Yes 14:51:09 what is a fesco blocker vs a blocker again? 14:51:25 nobody votes in blockerbot 14:51:26 dcantrell: we approve a blocker outside of the normal process and rules. 14:51:28 dcantrell: a blocker needs to vialate a speciffied crietrion 14:51:28 can we not have meetings until a new signed shim is available? 14:51:29 a fesco blocker is one fesco decides is a blocker... it must be fixed/closed 14:51:33 dcantrell: It means that FESCo considers it a blocking issue even if existing blocker criteria isn't violated 14:51:36 dcantrell: fesco blockers can do anything 14:51:45 yeah, alright 14:52:30 also, not sure if it can be waved w/out fesco's ack 14:52:38 afaik, no 14:52:46 that would seem to be the important distinction 14:52:56 waiving the blocker requires fesco approval too, afaik 14:53:08 right 14:53:17 (fwiw I've been talking to the canonical people during this meeting, and I've got a meeting with MS people to discuss options later today) 14:53:18 Well, waiving only happens at Go/No-Go and FESCo is required to have a representative there. 14:53:28 Normally there are several, though rarely a quorum 14:53:36 pjones++ 14:53:36 sgallagh: Karma for pjones changed to 4 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 14:54:04 I think we can make this a blocker and if pjones manages to do magic instead of signing a new build, it's fixed 14:54:10 I think it best to just go vote in the normal blocker process and make it a blocker there. :) 14:54:10 pjones++ 14:54:10 Conan_Kudo: Karma for pjones changed to 5 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 14:55:00 nirik: I am afraid that "Fedora boots after Ubuntu" is not listed in the blocker rules 14:55:14 and hence I'd rather fesco-block it 14:55:38 Yeah, with my blocker-reviewer hat on, I'd vote -1 on those grounds, but I'm +1 with my FESCo hat on. :-/ 14:55:43 +1 to sgallagh's proposal since the concerns for reviewer systems and initial experiences is real 14:55:49 I'm okay with it being a blocker, and I don't think we have to have a rule if everyone is agreed on it. 14:56:08 we probably need to revise criteria for F34+ to add multiboot criteria 14:56:09 (but I'm also going to work to alleviate that status, whichever way we have to do it) 14:56:24 Thanks, pjones 14:56:34 pjones++ 14:56:34 mhroncok: Karma for pjones changed to 6 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 14:56:49 https://fedoraproject.org/wiki/Basic_Release_Criteria#Release-blocking_images_must_boot / 14:56:49 ? 14:57:18 nirik: Conditional violations often get waived at Final 14:57:31 pjones++ 14:57:31 decathorpe: Karma for pjones changed to 7 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 14:57:34 They could sure... but not willy nilly... 14:57:57 Anyway, I think we've talked through it, can we have a formal revote on my proposal? 14:58:01 proposal: FESCo considers this serious enough to warrant being a blocker, though the resolution does not necessarily have to be technical. 14:58:04 sgallagh: +1 14:58:08 +1 14:58:21 +1 to sgallagh's proposal since the concerns for reviewer systems and initial experiences are real 14:58:22 +1 14:58:33 +1 14:58:55 I suppose +1... but if this is the last bug and we want to waive it at go/no-go... 14:59:19 I'll be opposed to waiving it at Go/No-Go 14:59:21 +1, and +1 to what nirik said 15:00:01 #agreed FESCo considers this serious enough to warrant being a blocker, though the resolution does not necessarily have to be technical. (+7, 0, -0) 15:00:42 #topic Next week's chair 15:00:50 Who wants it? 15:00:50 sgallagh: let me know if you cannot attend so i can be opposed on your behalf :) 15:01:40 mhroncok: I have a suggestion about that which I'll discuss in the Open Floor. 15:01:57 decathorpe: haven't you said you would chair the next meeting? 15:02:08 * nirik is glad fellow fesco members have time machines. Can you tell me eactly what the go-nogo will look like? and winning lotto numbers? 15:02:20 I can 15:02:35 nirik: Not without altering the timeline, I'm afraid. 15:02:54 nirik: the winning lotto numbers will be mostly of a hindic design that gets called "arabic" because of path dependency of how they made their way west 15:04:30 #action decathorpe will chair next meeting 15:04:36 #topic Open Floor 15:04:53 I have a proposal on how to deal with FESCo blockers at Go/No-Go: 15:05:29 In the event that fewer than a quorum of FESCo members is present at Go/No-Go, a FESCo blocker may only be waived by a unanimous vote of those members present. 15:05:40 If a quorum is present, normal voting rules would apply. 15:05:43 pjones: good to know! 15:05:48 sgallagh: and if nobody is present? 15:05:56 mhroncok: It's automatically No-Go 15:06:06 because FESCo is one of the required signatories 15:06:13 sure, thats reasonable 15:06:20 sound reasonbale 15:06:21 sgallagh: makes sense to me 15:06:26 *sounds 15:07:40 Any opposition? 15:08:28 not from me 15:08:33 resistance is futile 15:08:49 OK, I'm counting that as +5 15:09:17 #agreed In the event that fewer than a quorum of FESCo members is present at Go/No-Go, a FESCo blocker may only be waived by a unanimous vote of those members present. (+5, 0, -0) 15:09:45 Anything else for Open Floor? 15:10:32 * sgallagh lights the fuse 15:10:43 * bcotton ducks under the table 15:10:52 * sgallagh rolls the grenade under the table 15:11:06 * King_InuYasha runs 15:11:13 #endmeeting