17:02:14 #startmeeting F37 Final Go/No-Go meeting 17:02:14 Meeting started Thu Nov 10 17:02:14 2022 UTC. 17:02:14 This meeting is logged and archived in a public location. 17:02:14 The chair is bcotton. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions. 17:02:14 Useful Commands: #action #agreed #halp #info #idea #link #topic. 17:02:14 The meeting name has been set to 'f37_final_go/no-go_meeting' 17:02:20 #meetingname F37-Final-Go_No_Go-meeting 17:02:20 The meeting name has been set to 'f37-final-go_no_go-meeting' 17:02:28 yay, the bridge works 17:02:31 #topic Roll call 17:02:34 for now. ;) 17:02:36 morning 17:02:40 * bittin is here 17:02:41 nirik-- 17:02:41 evening 17:03:32 Evening where I am 17:03:38 Hello! 17:03:39 in LDN 17:03:43 hello 17:03:48 same in SE 17:04:06 * coremodule is here 17:04:13 .hello geraldosimiao 17:04:14 geraldosimiao: geraldosimiao 'Geraldo S. Simião Kutz' 17:04:21 .hello bittin 17:04:24 bittin: bittin 'Luna Jernberg' 17:04:39 #topic Purpose of this meeting 17:04:54 #info Purpose of this meeting is to check whether or not F37 is ready for shipment, according to the release criteria. 17:04:54 #info This is determined in a few ways: 17:04:55 #info 1. No remaining blocker bugs 17:04:55 #info 2. Release candidate compose is available 17:04:55 #info 3. Test matrices for Final are fully completed 17:05:11 #topic Current status - blockers 17:05:11 #link https://qa.fedoraproject.org/blockerbugs/milestone/37/final/buglist 17:05:11 #chair adamw 17:05:11 Current chairs: adamw bcotton 17:05:17 #info 2 Proposed Blockers 17:05:17 #info 0 Accepted Blockers 17:05:25 #topic (2141237) ABRT on KDE can't store any passwords: Secret Service is not available 17:05:26 #link https://bugzilla.redhat.com/show_bug.cgi?id=2141237 17:05:26 #link https://pagure.io/fedora-qa/blocker-review/issue/1010 17:05:26 #info Proposed Blocker, kf5-kwallet, NEW 17:05:26 #info Ticket vote: FinalBlocker (+4,0,-11) (+bcotton, +catanzaro, +kparal, +nixuser, -frantisekz, -geraldosimiao, -jbwillia, -nb, -sumantrom, -sgallagh, -zbyszek, -pboy, -ararezaee, -mindless-canary, -lruzicka) 17:05:36 and 1 Proposed FE 17:05:48 so, this one's really close for me, but i'm either +1/waive or -1. 17:05:51 okay, before we discuss this one, i want everyone to know that if your reasoning for -1 includes "because we're already late enough" or "can be fixed in an update" that I am ignoring oyu 17:05:54 have no comments on this one as i don't use KDE so let the people using KDE decide 17:05:59 because that's not how this works 17:06:09 bittin: proposed FEs don't matter here 17:06:14 bcotton: ah ok 17:06:20 * nirik reads up on this bug 17:06:38 ah yeah, that one. -1 blocker 17:06:47 https://bugzilla.redhat.com/show_bug.cgi?id=2141237 17:06:50 https://bugs.kde.org/show_bug.cgi?id=458341 17:08:08 imo, it's significantly broken from the "normal user" perspective to warrant being a blocker. but, especially since there's a workaround, i'm comfortable waiving it. so i'm going to stick with my +1 17:08:26 -1 FB 17:08:43 yeah, i think on the whole i have to say a weak +1 17:08:57 it's very very tight, but it does just about count as basic functionality for me 17:09:24 I'm not sure I agree people will generate new api keys... if you generate one and save it you can re-use it as many times as you like, you just have to fill it in 17:09:54 nirik: it's hard to predict whether people will treat it as "normal" to save the api key somewhere or not 17:10:05 it's reasonable to kinda 'expect' the app to save it for you, i think... 17:10:12 and it doesn't break actually filing the bugs... 17:10:14 i'll note that four of the -1s in the ticket fall into "that's not how this works bucket", but with the additonal votes in meeting, we're at +5,0,-9, i if i can math 17:10:14 still, generating a new one is hardly the most onerous thing 17:10:20 sure, I agree it's a bug and not ideal. 17:10:34 and once you hit this bug once you're presumably gonna know what to do the next time. i'm not saying the impact is horrible, just it does barely squeak past the threshold for me 17:10:46 * adamw trusts the math 17:10:54 I think it's just barely on the other side for me. ;) 17:11:02 I'm just here for the waiver 17:11:22 -1 aa 17:12:02 so, what do we do here? we have more votes for reject... 17:12:08 proposed #agreed 2141237 - RejectedBlocker (Final) - The majority does not consider this to violate basic functionality and those who do would be willing to waive it, so in the interests of time, we are taking this as a consensus to reject. 17:12:15 ack 17:12:16 ack 17:12:20 ack 17:12:21 ack 17:12:25 ack 17:12:33 ack 17:13:10 #agreed 2141237 - RejectedBlocker (Final) - The majority does not consider this to violate basic functionality and those who do would be willing to waive it, so in the interests of time, we are taking this as a consensus to reject. 17:13:18 and onto the shim one 17:13:23 #topic (2113005) Live image made with BOOTX64.EFI from latest shim-x64-15.6-2 fails to boot on some boards 17:13:23 #link https://bugzilla.redhat.com/show_bug.cgi?id=2113005 17:13:23 #link https://pagure.io/fedora-qa/blocker-review/issue/1012 17:13:23 #info Proposed Blocker, shim, POST 17:13:23 #info Ticket vote: FinalBlocker (+1,0,-0) (+bittin) 17:13:38 * Eighth_Doctor waves 17:13:42 .hello ngompa 17:13:43 Eighth_Doctor: ngompa 'Neal Gompa' 17:13:44 * bittin voted in the ticket before the meeting 17:13:47 hey Neil 17:13:56 so, uh, this is totally my fault 17:14:05 hey Eighth_Doctor 17:14:16 i've known about this for months but for whatever reason went from not thinking it was a huge deal to thinking there's nothing much we can do about it 17:14:23 adamw: do we have any idea how widespread this might be? 17:14:29 but the questions kparal asked made me think, oh wait, this kind of is a big deal isn't it? 17:14:33 adamw: can you fix it before Tuesday next week ;) 17:15:26 well, possibly, that's one thing to talk about. but anyhoo 17:15:32 i hope it's not terribly widespread 17:16:02 as i understand it, we don't have a signed shim or a timeline for a signed shim yet, is that right? 17:16:34 so far the affected boards are a "Kontron VX3040" which seems to be something fairly obscure, and two Asus boards of a specific vintage 17:16:37 there's no way we're going to get a fix for this 17:16:45 it takes too long and there's no levers we can do 17:16:51 there is a comment with different HW 17:16:57 s/do/use/ 17:17:01 Asus Z87-PLUS motherboard 17:17:08 that is OLD intel board 17:17:34 this is one of the longest beta periods we've had, so i feel pretty comfortable that it's not widespread at this point (famous last works) 17:17:37 last words, too 17:17:42 Intel 4th generation CPUs aren't that old 17:17:44 my h97m-e is also affected 17:17:48 i believe they're of similar vintages 17:17:49 so, we hope this is not very widespread, even if it was, we don't have any way to get a signed shim fast... 17:18:06 we could recompose by overriding with an older shim binary 17:18:09 hm I have one h77 and h110 but didnt test it 17:18:17 well 17:18:23 for me, the worst affected case is upgrades 17:18:27 can two shims be shipped? or would that cause other problems 17:18:33 i hadn't thought about it till kamil asked me the question, but i suspect they will be affected 17:18:36 i am running a test of that now 17:18:38 can we kick out the newer shim? 17:18:43 adamw: ah 17:18:45 so the case where you upgrade your system and it stops booting is, uh. bad 17:18:46 as in, untag it entirely? 17:18:56 not without re-composing 17:19:04 but what i was wondering is whether we can ship the older shim as an update 17:19:10 * dustymabe is sorry he is late - time change 17:19:19 new package build, but same old files 17:19:22 Turns out I have a Z87 motherboard at home. I can quickly test at least booting the latest Live 17:19:31 i don't know if that invalidates the signature but i'm hoping not 17:19:55 i pinged robbie on rh's internal chat to see if he could join the meeting but i guess he didn't see it 17:20:14 the signed shim package is just binary blobs checked into lookaside 17:20:20 the other question with shipping the older shim is, of course, what are the dangers of that? does the newer shim fix some important things? 17:20:21 afaik, we don't sign it in fedora 17:20:45 no, microsoft signs it, that's the point, but i guess my question is do they sign the files or the package 17:20:52 the files 17:20:54 the package is us 17:20:54 i'm assuming/hoping it's the files but i don't know for sure. 17:21:00 yeah, it's files. 17:21:04 then we can probably do a new build of the package with the old files. it's a possibility anyhow 17:21:14 I wonder if the newer shim is needed for the newer dbx... 17:21:19 we can play whatever games we want with the shim RPM as long as the files are untouched 17:21:21 The answer to does it fix important things is almost certainly "yes" 17:21:23 nirik: yeah, questions like that concern me. 17:21:42 Exactly 17:21:49 My Gigabyte Z87 board boots OK with the latest KDE Live RC 17:22:11 This has been a long cold winter^Wfreeze, and I agree with bcotton that if this was widespread we would have had a lot more reports. 17:22:36 good point 17:23:37 kparal: yeah, i suspect maybe only asus is affected. or i guess asus and this weirdo manufacturer i hadn't heard of before 17:23:51 I'm at the point of marking it as a commonbug and saying we're going to have a live respin in the future with it fixed 17:23:54 nirik: the only thing about that that concerns me is, maybe other people didn't find this bug report 17:23:57 so I would be inclined to say it's not a blocker because it's not widespread enough, or say it is and waive it based on not being able to fix it in time... but of course more data would be good. 17:24:03 but yeah, i haven't seen any 'lying around the place' 17:24:28 i'm ok with -1 blocker *to the fresh f37 install case* on the basis of "not very widespread" 17:24:31 I have one ASUS board, but it wasn't affected (probably because it's even older and its UEFI implementation is far too simple to be useful) 17:24:37 -1 blocker for me 17:24:41 i just feel a bit less comfortable with that for the upgrade case because it's just a horrible experience 17:24:55 "yay i'm going to have a shiny new fedora! ...oh noes now my system does not boot what the hell do I do?" 17:24:56 +1 blocker for me (as i wrote in the ticket) 17:25:07 i mean, you're literally stuck with the rescue shell or live boot at that point 17:25:08 -1 blocker, +1 commonbug, maybe we should have a warning notice or something? 17:25:43 i'm currently doing a test to see if upgrades are affected, but i'd be surprised if they aren't 17:26:12 both of my intel boards booted latest live 17:26:16 so -1 blocker 17:26:35 There is some hope, quoting: "Strangely, once booted with the workaround above and installed to disk, the disk boots successfully even with this /boot/efi/EFI/BOOT/BOOTX64.EFI from shim-x64-15.6-2.x86_64" 17:26:52 O.o 17:27:19 I see some reports of this same thing against rockylinux 8.5 :) 17:27:22 We have a _lot_ of upgraded systems at this point. 17:27:48 kparal: huh, that's interesting 17:27:52 guess we'll see how my test turns out 17:27:56 I feel like this might have more noise if it were very widespread 17:27:58 huh: https://ask.fedoraproject.org/t/f37-invalid-image-error-while-booting/26632 17:28:07 i did also verify that if you replace the files on an f37 live and install, the installed system works (that's what i expected as it should copy over the same files) 17:28:41 make a manual intervention post as a common bug? or is that mostly an Arch thing? :p 17:28:43 nirik: yeah, that's the same problem. the "solution' given there will only work with secure boot disabled, because it essentially bypasses shim. 17:28:45 so one report there with beta. 17:28:49 right 17:29:54 so i think i count +1,0,-4. anyone else want to weigh in? 17:30:20 I'm -1 blocker to be clear if you didn't count me yet 17:30:25 -1 blocker, +1 commonbug 17:30:38 this is another really close one for me :| i can buy that it's not very common, it's just so nasty for those it does hit. 17:30:52 kinda wanna see how the upgrade test turns out, but apparently nobody else is waiting on that. :P 17:31:01 (i'm currently updating before upgrading) 17:31:11 I'm fine waiting... more data is good. 17:31:18 you have a board thats affected there? 17:31:36 i'm happy to stall for a few minutes. i feel like we know what the answer will be, but computers have surprised me before 17:31:36 it doesn't change much for me, because we can't fix it 17:31:37 yes, i do. just realized i didn't actually mention that in the bug. :D 17:31:42 but I'd like to be pleasantly surprised 17:32:05 Conan Kudo: well, we can do the 'shim an older shim as an update' thing. at least theoertically 17:32:07 ok on wait a little more 17:32:26 it'd be nice to have robbie here to kick around the pros and cons though :| 17:34:13 Robbie recommended in the BZ to downgrade shim 17:34:31 so we could do the 15.6+really15.4 version hack 17:34:32 as a workaround for this specific hardware, yes 17:34:41 obviously if you have an affected system, that's the best option 17:35:06 but whether it's a good idea to downgrade for everyone is a different question 17:35:24 my update is at 28%... 17:35:44 35%...:D 17:35:50 "i did also verify that if you..." <- You mean building your own Live, or replacing how? 17:36:22 kparal: you can just mount the ESP on the USB stick and overwrite the files 17:36:36 Ah, easy 17:36:54 the ESP isn't part of the complex image stuff, it's a straightforward partition, once you've written the stick 17:37:03 yay isohybrid 17:37:05 it'd be harder if you wanted to write a DVD, but we don't support that any more anyway :P 17:37:12 90% now 17:37:16 stuff a random fat32 partition alongside udf :D 17:37:21 adamw: thats good to know 17:37:35 so we can document that workaround for fresh installs 17:37:36 adamw: Once you installed, have you tried updating shim package and rebooting? 17:37:49 well, there's no update on a fresh install 17:38:00 the system will still think the 'current' shim is installed, though it wouldn't pass rpm -V 17:38:05 i didn't test, but that's what i'd expect 17:38:06 Reinstalling shim then 17:38:24 i'd expect that'd bork it. it'd be the same case as the upgrade test, really 17:38:25 dnf rei shim :) 17:38:31 It would be faster to check, though 😄 17:38:47 it would, except i already moved on to the f36 upgrade test 17:38:51 i only have one affected system :P 17:39:59 ok, starting the upgrade now 17:40:01 clearly an oversight. 17:41:08 heh 17:41:27 so, while we're waiting on this 17:41:33 let's say my test confirms the upgrade breaks the system 17:41:39 does that make anyone vote +1 ? 17:41:48 apart from me 17:42:16 * bittin 17:42:31 alas, no 17:42:34 I'm really torn. That's brown-paper-bag level catastrophe if it happens to a meaningful number of people. 17:42:40 I already assumed it'd break on upgrades 17:42:40 That does make it more of a pain... but still not more widespread. 17:42:48 my -1 baked in the assumption that upgrades would be broken,too. what would move me to +1 is if it were more widespread 17:43:11 well, to be more accurate, knowing it was more widespread 17:43:20 * nirik nods 17:43:24 if we could easily kick out the new shim and roll back everything, then I'd switch to +1 17:43:48 untagging new shim, tag in old shim, recompose 17:43:52 otherwise, -1 and bite the bullet 17:44:12 we could technically do that. but we'd need the evaluation of whether it's likley to break anything else. 17:45:16 humm, hold on a tick here 17:45:37 just noticed something. shim builds are not fedora release versioned...and 15.6-2 is already in f35-updates and f36-updates , as well as f37 17:45:41 https://koji.fedoraproject.org/koji/buildinfo?buildID=1997554 17:45:46 wut 17:45:56 right. 17:45:57 so...if we already sent it out as an update to f35 and f36...we already broke anyone it would've broken, i guess? 17:46:01 then this affects everyone anyway 17:46:09 -1 and commonbug then 17:46:13 and apparently "oops?" 17:46:28 if we downgrade we loose CVE-2022-28737 and aarch64 bug #2101248 17:46:34 yeah, and also 17:46:44 my f36 test install actually got the newer shim on the update (before the upgrade) 17:46:46 and it booted fine 17:46:54 so...does this for some reason only cause problems when booting live, maybe? 17:47:07 hum... thats a possibility. 17:47:11 in that case, i'm a lot more comfortable with -1 17:47:39 kparal had a affected machine too? 17:47:53 No I don't 17:47:55 no, i think he said his is ok 17:48:00 it's not an assus 17:48:12 ah, ok. 17:48:40 so, okay, i guess on current info i'm -1 17:48:40 sorry for the detour, everyone 17:48:46 adamw: well someone has too test 17:48:53 when was it sent to f36? 17:49:01 proposed #agreed 2113005 - RejectedBlocker (Final) - This bug appears to be limited to specific hardware and may only affect booting live images, so it does not rise to the level of a release blocker 17:49:21 ack 17:49:25 ack 17:49:26 ack 17:49:29 adamw: what's the upgrade progress? 17:49:32 ack 17:49:37 Southern_Gentlem: Mon Jul 18 10:23:05 17:49:43 ack 17:49:47 ack 17:50:04 nirik, in that case we have several months of updated lives with no issue 17:50:06 Southern_Gentlem: i dunno, but it's there 17:50:09 https://dl.fedoraproject.org/pub/fedora/linux/updates/36/Everything/x86_64/Packages/s/ 17:50:24 vs. https://dl.fedoraproject.org/pub/fedora/linux/releases/36/Everything/x86_64/os/Packages/s/ 17:50:44 ack 17:50:49 ack 17:50:52 * Eighth_Doctor really wishes we promoted live-respins more 17:50:53 #agreed 2113005 - RejectedBlocker (Final) - This bug appears to be limited to specific hardware and may only affect booting live images, so it does not rise to the level of a release blocker 17:50:58 Eighth_Doctor: +1 17:51:03 #info We have zero accepted blockers 17:51:13 **happy dance** 17:51:16 does that mean we did it? 17:51:23 hopefully 17:51:24 finally? 17:51:24 not yet 17:51:25 :-) 17:51:26 #topic Current status - test matrices 17:51:26 #link https://fedoraproject.org/wiki/Category:Fedora_37_Test_Results 17:51:35 adam, tell us how this is the most-tested release ever 17:51:47 kparal: so the workaround is basically: get https://dl.fedoraproject.org/pub/fedora/linux/releases/36/Everything/x86_64/os/Packages/s/shim-x64-15.4-5.x86_64.rpm , extract BOOTX64.EFI from it, write the F37 image to USB , mount the EFI partition from the USB stick , copy the old BOOTX64.EFI over the one on the stick 17:51:50 if you want to document that 17:51:57 Ben Cotton (he/him): well funny you should say that! because it basically is 17:52:08 * bittin always forgets to report but all RCs of Workstation has worked to install in Virtualbox for me so far 17:52:13 adamw: or just downgrading shim in rescue ? 17:52:13 kparal: you should be able to try all that for yourself even on an unaffected system, just so you can write it down accurately 17:52:21 sorry, i'd do it, but i'm not gonna have time 17:52:31 I'll document it 17:52:36 thanks 17:53:07 we have juuuuuust about everything covered 17:53:16 nice 17:53:27 Alright 17:53:50 we're missing FMW on Windows 11, dualboot with macOS, and one active directory test (but that's one we normally miss and assume it's working OK if the others work, and the freeipa kickstart test works) 17:53:51 we don't even need to wait for the AD testing. ;) 17:54:16 we don't have cloud testing on EC2 Xen or openstack, but meh. 17:54:33 on the whole i'd say coverage is excellent 17:54:42 * dustymabe has to go pick kids up - bbiab 17:54:42 Yes! 17:54:55 #info Test coverage is excellent 17:55:06 * mattdm wonders when it will be okay to drop Xen ... 17:55:22 probably never 17:55:23 NCC ShipIT 17:55:37 when its not in the kernel anylonger 17:55:40 🚢 17:56:04 anything else on test coverage? 17:57:14 that's EC2 Xen for cloud, not general-purpose xen 17:57:20 i.e. "does it work on xen EC2 instance types" 17:57:28 Soas is working fine 17:57:33 actually, i did the ec2 testing and i kinda assumed the instance type i used was kvm, but i didn't check 17:57:40 so, let's say, we tested on one ec2 instance type. :P 17:57:43 Just test it 17:57:45 yeah, my musing was about how much longer that is going to be a thing 17:58:02 anyway i will stop distracting 17:58:04 #topic Current status - RC 17:58:05 EC2 defaults to Xen except on specific types 17:58:22 We have RC 1.7 17:58:25 Tell us about it, RelEng! 17:58:51 on my ec2 instance fed37 runs 17:58:52 it's a lovely little rc, and it can be yours on tuesday for a GO today! 17:59:10 and fluffy!! 17:59:16 it FINISHED I think, so we have everything 17:59:30 https://kojipkgs.fedoraproject.org/compose/37/Fedora-37-20221105.0/STATUS 17:59:30 GO!!!!!! 17:59:33 woohoo 17:59:36 🛳️🛳️🛳️🛳️🛳️🛳️🛳️🛳️🛳️🛳️🛳️🛳️ 17:59:43 NCC ShipIT 17:59:44 #info RC 1.7 is FINISHED (no missing artifacts) 17:59:57 #topic Go/No-Go decision 17:59:57 I will poll each team. Please reply “go” or “no-go” 18:00:00 FESCo? 18:00:05 go go go 18:00:10 GO 18:00:11 (fesco hat on): GO! 18:00:14 #info FESCo is GO 18:00:18 RelEng? 18:00:23 GO 18:00:24 also go 18:00:29 #info RelEng is GO 18:00:34 QA? 18:01:01 GO 18:01:02 GO 18:01:07 #info QA is GO 18:01:46 i'd just like the record to reflect that this is, to the best of my memory, the first time adam has said "go" and not "well since we don't have any blockers we know about, I guess we have no choice but to be go" (my paraphrase) 18:01:49 #agreed Fedora Linux 37 is GO 18:01:49 #info Fedora Linux 37 will release on target date #3 (2022-11-15) 18:01:54 #action bcotton to announce decision 18:01:56 Ben Cotton (he/him): i've got to get to the airport 18:02:01 yay 18:02:16 #info adam is GOing to the airport 18:02:18 great 18:02:30 i usually say that because we technically set up a policy in QA that our decision is based on objective requirements. this was back in the days when viking-ice was keeping us all honest 18:02:42 he didn't want it to be a subjective call 18:02:47 #topic Open floor 18:02:47 Anything else we need to discuss before closing? 18:03:07 nooope 18:03:24 as a bit of a process man myself, i take great delight in the careful incantation you give us, adam 18:03:24 Thank you everyone!!! 18:03:27 my f36->f37 upgraded test system boots okay 18:03:30 so that is weird, but good news 18:03:32 i'll update the bug 18:03:33 nope 18:04:01 adamw: nice, thanks for testing 18:04:16 Everything is fine 18:04:16 QA/releng -- what's your feeling at this point about the "extra" week delay? 18:04:20 🎉 18:04:43 always good to test things as much as they must be tested 18:05:27 mattdm: sounds like a question for when adam's not rushing to the airport :-) 18:05:34 last call! 18:05:38 the number of RC we did helped us to refine the process of syncing RCs to RH QE 18:05:43 fine :) 18:06:07 thanks everyone! 18:06:08 #endmeeting