17:01:16 #startmeeting F37 Final Go/No-Go meeting 17:01:16 Meeting started Thu Oct 27 17:01:16 2022 UTC. 17:01:16 This meeting is logged and archived in a public location. 17:01:16 The chair is bcotton. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions. 17:01:16 Useful Commands: #action #agreed #halp #info #idea #link #topic. 17:01:16 The meeting name has been set to 'f37_final_go/no-go_meeting' 17:01:16 #meetingname F37-Final-Go_No_Go-meeting 17:01:16 The meeting name has been set to 'f37-final-go_no_go-meeting' 17:01:21 #topic Roll Call 17:01:31 \o 17:01:34 morning 17:01:37 looks like I picked the wrong week to quit sniffing glue 17:02:07 :) 17:02:16 there's no good week. 17:02:27 pong 17:02:56 .hi 17:02:57 sgallagh: sgallagh 'Stephen Gallagher' 17:02:59 .hello siosm 17:03:00 travier: siosm 'Timothée Ravier' 17:03:41 .hi 17:03:42 bytehackr: bytehackr 'Sandipan Roy' 17:03:47 .hello2 17:03:47 coremodule: coremodule 'Geoffrey Marr' 17:04:32 * nirik wonders if anyone put out the adamw signal. 17:04:55 he 17:05:04 he's explaining why i am the way that i am to his cat 17:05:04 get out your magic 8 balls folks (i said it earlier, but want it in the minutes) 17:05:24 :D 17:05:37 just used my SeaGL magic 8 ball and it said "review logs". i'm not sure how to interpret that 17:05:48 .hello gui1ty 17:05:49 Penguinpee: gui1ty 'Sandro .' 17:05:50 but! let's start the process 17:05:57 .hello2 17:05:58 cmurf: cmurf 'Chris Murphy' 17:06:05 #topic Purpose of this meeting 17:06:05 #info Purpose of this meeting is to check whether or not F37 is ready for shipment, according to the release criteria. 17:06:05 #info This is determined in a few ways: 17:06:12 #info 1. No remaining blocker bugs 17:06:12 #info 2. Release candidate compose is available 17:06:12 #info 3. Test matrices for Final are fully completed 17:06:19 #topic Current status - blockers 17:06:19 #link https://qa.fedoraproject.org/blockerbugs/milestone/37/final/buglist 17:06:33 #info 1 Proposed Blockers 17:06:33 #info 2 Accepted Blockers 17:06:35 SO! 17:06:45 #topic (2137661) upcoming critical openssl vulnerability 17:06:45 #link https://bugzilla.redhat.com/show_bug.cgi?id=2137661 17:06:45 #link https://pagure.io/fedora-qa/blocker-review/issue/987 17:06:51 .hi 17:06:52 dustymabe: dustymabe 'Dusty Mabe' 17:06:52 #info Proposed Blocker, openssl, NEW 17:06:52 #info Ticket vote: FinalBlocker (+11,0,-3) (+bytehackr, +mattdm, +bcotton, +ngompa, +geraldosimiao, +chrismurphy, +nixuser, +copperi, +sumantrom, +frantisekz, +augenauf, -sgallagh, -catanzaro, -gui1ty) 17:07:08 quite a lot of +1s 17:07:42 .hello2 17:07:45 frantisekz: frantisekz 'František Zatloukal' 17:07:48 indeed 17:07:48 most of them chosen ones 17:08:15 the main hesitation, which i understand, is that we don't know what the vulnerability actually is 17:08:17 I'm not sure what critera is being used here...I don't think we have much that fits this 17:08:28 .hello copperi 17:08:29 copperi[m]: copperi 'Jan Kuparinen' 17:08:46 ""The release must contain no known security bugs of 'important' or higher impact according to the Red Hat severity classification scale which cannot be satisfactorily resolved by a package update (e.g. issues during installation)."" 17:08:52 #info Criterion: The release must contain no known security bugs of 'important' or higher impact according to the Red Hat severity classification scale which cannot be satisfactorily resolved by a package update (e.g. issues during installation). 17:08:52 .hello ngompa 17:08:53 Eighth_Doctor: ngompa 'Neal Gompa' 17:08:57 #link https://fedoraproject.org/wiki/Fedora_37_Final_Release_Criteria#Security_bugs 17:09:11 right, but we don't know if it could be resolved by an update or not... 17:09:16 fwiw, I'm not a "chosen one" for being able to review the bug 17:09:27 I have no idea and I'm going off what the "chosen ones" say 17:09:37 yeah, i don't have access to the actual bug either 17:09:41 My argument is that there currently isn't a known bug that fits that description 17:09:46 All we have is supposition. 17:09:49 riight and as per pre-announce security bulletin of red hat its a critical: https://access.redhat.com/security/vulnerabilities/RHSB-2022-004 17:09:56 could someone wearing a red head dress at least give some indication s to the severity and scope. 17:10:11 sorry, lost track of time 17:10:12 Penguinpee: We don't have access either. Only the Product Security Team 17:10:13 .hello adamwill 17:10:14 adamw: adamwill 'Adam Williamson' 17:10:27 and I am not sure they can really say much of anything... 17:10:33 ah, I see. 17:10:33 (and some of the QA people have access too...) 17:10:38 Penguinpee: anyone who know details of the bug really is not allowed to say anything beyond what's publicly disclosed already 17:10:41 my preference would actually be to not block on this, but I'm super wary of ignoring RH security 17:10:41 So we have to work from the public information 17:10:52 There is nothing in the bug of any substance 17:11:15 * Penguinpee divides time between cooking dinner and IRC 17:11:31 well unfortunately RH security isn't saying whether Fedora 37 as it is, should be released or delay 17:11:41 even if it's just informational advise 17:11:54 so I guess the big thing is if it affects common things like curl / web browsers... as people often use live media to run those things for testing/whateevr. 17:11:54 they recommended to mattdm we delay 17:11:54 cmurf: yes, they did. 17:12:20 nirik: it doesn't affect Firefox, since that uses NSS 17:12:23 Eighth_Doctor: they did? i didn't see that, ok that's sorta changes the calculation I suppose 17:12:24 https://pagure.io/fedora-qa/blocker-review/issue/987#comment-822932 17:12:27 but it does affect everything else 17:12:32 "RH product security tells me that they advise making sure our installation media and various images are updated." 17:12:51 yeah interesting 17:12:51 got it 17:12:52 it could affect netinst too I guess? 17:12:56 technically we could satisfy this if we could do regular image composes, but we can't 17:13:04 FWIW in Fedora CoreOS we have a bit more control over our users update schedule (automated updates) so we're considering delaying our rebase to F37 for our `testing` stream until after the SSL CVE fix has been rolled out. 17:13:07 adamw: but that begs the question sgallagh posed in the ticket 17:13:23 Eighth_Doctor: right. 17:13:31 yes, that is a reasonable concern 17:13:37 anyhow, I guess I'm +1 also to blocking on it. ;( 17:13:40 the underlying problem is we have no process to re-compose media and roll that out 17:13:47 adamw: Which is? 17:13:49 we have respins sig for x86 media, but that's it 17:13:55 Eighth_Doctor: agreed 17:13:58 the fact that we're largely stuck with this for 6 months (and entirely stuck on anythign that the Respins SIG doesn't Respin) makes the thought of waiting a week a lot more palatable 17:14:02 knowing we have that time constraint, would it be feasible to push go/no-go next week to friday? or can we not do that? 17:14:04 Conan Kudo: Other than the Respins? 17:14:18 example 92 why we need easier re-compose ability 17:14:25 adamw: which time constraint? 17:14:25 Stephen Gallagher: respins is technically not a re-compose process for all deliverables 17:14:29 Conan Kudo: we have no process because we don't want the burden of doing it. the process isn't the hard part. 17:14:43 finding time for releng and qa to go through a whole spin-and-test cycle multiple times during a release is the hard part. 17:14:51 i've been told in the past that we cannot rely on CPE to do any of the releng work over the weekend, so a friday go/no-go seems unpossible 17:15:00 nirik: the constraint that the openssl disclosure/release is planned for next tuesday 17:15:12 which puts us on a tight timeframe to try and build, test and sign off next thursday 17:15:15 now, could we do a go/no-go the following monday or tuesday and then release on thursday/friday? maybe 17:15:21 as sgallagh pointed out at https://pagure.io/fedora-qa/blocker-review/issue/987#comment-823010 17:15:33 it could be tight, but I think it's possible... 17:15:45 afaik tuesday releases are mostly 1. tradition and 2. press timing 17:16:11 and 0. timing for all the things that need to be done for a release 17:16:15 we could release on Thursday as a special case if we want to push it out next week 17:16:51 i feel like changing the release date is a bigger and more difficult change than squeezing the go/no-go, but i'm not the one who does the work between thursday and tuesday, i guess 17:16:54 i'd have to defer to nirik on that 17:17:09 and yes, getting the fix on tuesday and signing off on thursday is possible, it's just tight 17:17:23 the openssl update will trigger OpenQA runs at least, so we can get some reasonable data on how much it breaks 17:17:23 we need the disclosure to happen, then we need the openssl build to run, then we need the compose to run 17:17:28 then we have whatever time is left to validate 17:17:36 adamw: I think it's too tight, since openssl drives so much of the distro 17:17:39 i'm not sure we can do sufficient regression testing between tuesday and thursday 17:17:45 we're getting a little ahead of ourselves at this point 17:17:46 I don't think it's practical to release on thursday... say thing comes out tuesday, we make a rc, we need to test it, get sign off, sync it to mirrors, etc. 17:17:50 I wouldn't be comfortable carrying over the results of earlier composes 17:17:53 we don't know when tuesday the fix will appear, or how much longer it'll take to get a compose 17:17:57 first we have to decide if this is a blocker 17:18:11 then we can figure out what to do about it 17:18:15 blocker +1 17:18:26 my preference would to be to not block on it, but everyone else suggests we should so... 17:18:34 because F36 is equally impacted, as is RHEL 9 17:18:36 everyone is toast 17:18:54 +1 blocker here. 17:18:56 given that i'm not a security expert, the details aren't public, and the advice from rh's security experts is to patch it, i'm still +1 17:19:04 My point is that if we had shipped on the original schedule, we wouldn't have ground the gears to a halt after Go/No-Go with so little info to go on 17:19:16 sgallagh good point 17:19:38 Conan Kudo: that's outside our area of competency. but if rh prodsec is saying that to us, i would assume they'll also be telling RH that a new RHEL 9.x is needed asap 17:19:40 sgallagh: true but knowledge and our position in the timeline changes things 17:19:45 I'd actually be more confident about not blocking if we could update the images :( 17:19:46 RHEL does do 'respins', in that sense, of course. 17:20:06 If I remember correctly, we did an official re-compose at least once in the past 17:20:09 adamw: it's probably going to slip right into RHEL 9.1 releasing next month, but who knows 17:20:21 * mattdm is here. sorry, family thing 17:20:25 nb: Followed immediately by a general agreement never to do so again 17:20:28 nb: we did it forever ago, when the compose process was different and we had a lot fewer things, iirc. 17:20:41 I am not in favor of doing a release, then doing another release right after when we could wait and do it once. ;) 17:20:44 since then we've done kinda unofficial single image fixups, but never a full rebuild 17:20:45 we also don't generally have a "make images" button 17:21:19 have we heard from anaconda folks? 17:21:34 .hello churchyard 17:21:35 that may not be the only thing impacted on the media but seems like the most important thing potentially impacted 17:21:35 mhroncok: churchyard 'Miro Hrončok' 17:21:36 I think I am a weak +1 to block. If Product Security Team says they recommend it, then we might should listen to them 17:21:48 as much as I want to not block 17:21:49 so i'm seeing roughly +12/-4, which is a pretty strong outcome 17:22:02 So RH is mostly concerned about vulnerable images getting out into the wild? But that's still true for all current images. 17:22:02 I'm maintaining my +1 even though I really want to do -1 17:22:18 i keep saying, don't focus overly on the installer 17:22:27 we also have to consider anything people might plausibly do on live imaegs 17:22:29 adamw: I'm more worried about arm images 17:22:41 live images don't bother me at all 17:22:46 Penguinpee: yes, all existing images are vulnerable, but if we don't wait, we don't have one that is not 17:22:49 Penguinpee: yes, but if we slip a week and include the fix in f37, we have non-vulnerable images out for the public in a week and a half 17:22:52 adamw: Which is bullcrap, IMHO. Since all of our existing images are riddled with holes 17:22:57 I'm not factoring them into my decision, since those are respun regularly 17:22:57 if we don't slip, we do not have non-vulnerable images out for another six months 17:23:13 Conan Kudo: not all of them and not on non-x86_64 17:23:13 I'm not too worried about live images - respins sig makes them every 2 weeks 17:23:21 but ARM and installer and stuff don't get respun 17:23:34 Conan Kudo: since the respins are not official, and not that widely publicized, that's not really a safe assumption. 17:23:38 adamw: why is that. there are respins of almost everything. 17:23:43 Ben Cotton (he/him): yes, my point is x86 live images are the only things I am not using as a factor for my decision 17:23:45 There's also things like server kvm image 17:23:56 adamw: which is a mistake in itself that we should seriously fix 17:23:58 none of the labs get respun, either, as far as i can tell 17:24:02 I wish I had data on cloud images -- how many people actually update, or ever use later versions than the first. 17:24:05 But it's guesswork. 17:24:09 Penguinpee: there's no validation. we don't really know if the respins...work. and i'm not having QA go through a full validation cycle every two weeks. 17:24:12 so respins are absolutely not the answer here 17:24:12 quite a lot of the +12 is predicated on a presumption that there's something on the media affected by a zero day exploit, if that's not true or has more limited impact, the vote might swing in the exact opposite direction 17:24:13 If someone is installing the KVM image and not immediately doing a DNF update, they deserve what they get 👍️ 17:24:13 I wish I had more data period 17:24:21 but all we have is public information, which is almost nothing 17:24:22 The problem is not so much about what folks might do with live image but more how bad it can impact netinstalls. If I can compromise a netinstall then it's really bad 17:24:24 mattdm: I know people _try_ to use updated cloud images, but that build process breaks so often lately... 17:24:29 and the discoverability is super-low 17:24:31 i think anaconda uses its own ssl unless passing a flag to use openssl? 17:24:34 adamw the respins get tested, maybe not to the extent of what QA normally does, but respins does have their own testnig process 17:24:37 we get questions about it in the cloud-sig all the time 17:24:38 what if the process of updating itself (TLS is everywhere) could compromise the system 17:24:58 then all those other distros are going to be looking to re-spin their install media too 17:25:12 cmurf: we have to consider that possibility based on the public information (a critical vuln in openssl). it's a much safer thing to do to assume that might be the case, than assume it *isn't* the case. 17:25:14 We have the same issue for FCOS, even though we don't use that much openssl: if you can compromise the first boot of FCOS then that's really bad 17:25:14 dustymabe: What if openssl turns out it's been a trojan horse all along? 17:25:21 We can make up whatever boogeymen we want. 17:25:26 it feels a bit like meeting a secret agent: trust me (and then all hell breaks loose) 17:25:36 * Eighth_Doctor shrugs 17:25:37 better safe now than sorry later 17:25:39 sgallagh: or maybe they are trying to zip through a trojan horse with this update - rememember remember the 1st of november 17:25:49 this is the downside of centralizing your crypto, even if it doesn't happen too often, it's painful when it does 17:25:51 Ha 17:26:01 sgallagh: (I am not quite sure I'm comfortable with that as a statement. But that's probably another discussion.) 17:26:28 okay, well we can argue all day, but i don't see any minds changing one way or another so 17:26:30 proposed #agreed 2137661 - AcceptedBlocker(Final) - Given the limited public information, we determine this violates"The release must contain no known security bugs of 'important' or higher impact according to the Red Hat severity classification scale which cannot be satisfactorily resolved by a package update" 17:26:37 ack 17:26:41 ack 17:26:42 That was overstated, but honestly: if you don't patch your systems, we can't do much for you 17:26:46 Ben Cotton (he/him): ack 😭 17:26:48 as usual, Stephen Gallagher retains his right to "i told you so" 17:26:51 ack 17:26:53 Stephen Gallagher: there's also the publicity angle to consider, of course. i do agree to some extent with a lot of your points, but it kinda looks pretty bad if we ship a release on the day this openssl news comes out and people say "so that's fixed, right?" and we say "no"... 17:27:03 * Penguinpee is not changing his mind (based on too little available info) 17:27:05 ack 17:27:06 ack 17:27:12 i'll also point out that we could still choose to waive this blocker ;-) 17:27:20 ack 17:27:24 * sgallagh slugs Ben Cotton (he/him) 17:27:24 i would probably word it to be less definitive about "this violates", but whatever 17:27:25 ack 17:27:36 patch 17:27:38 well, it's difficult-to-fix given the code doesn't exist 17:27:38 and yes, we can continue arguing about it if someone wants to propose waiving it. ;) 17:27:45 nack 17:27:48 Stephen Gallagher: patch away 17:27:49 Penguinpee: Let me clarify that a little bit. I wish I could give even more information but I don't have much to go on. Since we have so little, I asked some people with more access for their opinions, and they shared it. 17:27:50 so you could waive under that condition :P 17:27:50 Conan Kudo: sure it exists. it's just not public yet. 17:28:01 adamw: doesn't exist to me 17:28:04 or anyone else 17:28:10 we actually don't even know if a patch is ready either 17:28:15 proposed #agreed 2137661 - AcceptedBlocker(Final) - Given the limited public information, we are unable to definitively determine whether this violates"The release must contain no known security bugs of 'important' or higher impact according to the Red Hat severity classification scale which cannot be satisfactorily resolved by a package update". We therefore are blocking out of an abundance of caution. 17:28:28 we know a disclosure is happening on Nov 1 with a release, we don't know if they're working right up to the deadline 17:28:29 ack 17:28:32 ack 17:28:33 ack 17:28:37 sure, ack 17:28:44 ack 17:28:44 ack 17:28:46 +1 17:28:47 Stephen Gallagher: ack 😭 17:28:50 ack 17:28:50 +1 17:28:57 yeah, i like that more, ack 17:28:59 ick err ack 17:29:00 Conan Kudo: Generally speaking, we can actually make that assumption. 17:29:07 mattdm: thanks. yet I'm not comfortable with making a decision on little to no info. So, maybe I should vote FinalBlocker 0 17:29:21 #agreed 2137661 - AcceptedBlocker(Final) - Given the limited public information, we are unable to definitively determine whether this violates"The release must contain no known security bugs of 'important' or higher impact according to the Red Hat severity classification scale which cannot be satisfactorily resolved by a package update". We therefore are blocking out of an abundance of caution. 17:29:31 Stephen Gallagher: having been involved in a few of these before, I don't make that assumption anymore 17:29:31 Historically, an embargo lift date is never given until a fix is identified (or the embargo has been lifted by other means, such as an accidental disclosure) 17:29:40 Conan Kudo: sure we do. they've declared a release date. 17:29:51 sgallagh: that is a rather bold statement 17:29:52 .hello geraldosimiao 17:29:53 geraldosimiao: geraldosimiao 'Geraldo S. Simião Kutz' 17:29:54 Penguinpee: the way I look at is it if my good friend told me to not go outside today, and to trust them. I'd stay inside 17:29:54 okay, moving on 17:29:56 Conan Kudo: they didn't announce that they're (only) disclosing an issue, they announced the release of 3.0.7, containing a fix for a CRITICAL issue. 17:30:05 let's review the already-accepted blockers 17:30:06 #topic (2135772) Editing the recurring event freezes Calendar. 17:30:06 #link https://bugzilla.redhat.com/show_bug.cgi?id=2135772 17:30:06 #link https://pagure.io/fedora-qa/blocker-review/issue/977 17:30:06 #info Accepted Blocker, gnome-calendar, NEW 17:30:07 s/never/rarely/ then? 17:30:27 Stephen Gallagher: we'll go with that 17:30:27 so, if we didn't have anything else, i'd be in favor of waiving this 17:30:36 but, now we have something else. 17:30:36 dustymabe: https://xkcd.com/1170/ 17:30:36 sgallagh: frequently embargo lift dates are set before work has even begun on a fix 17:30:41 So this is the one that upstream said "wow, this is totally bad" 17:30:59 I would propose to waive 2135772 17:31:16 That tends to happen more with cross-project flaws than ones in a specific library. 17:31:19 i don't see any point in waiving it now if we're gonna slip for openssl 17:31:34 in fact we kinda only intended the waiver process to be used in order to sign off a release 17:31:38 Like if an algorithm is discovered as vulnerable in multiple implementations. 17:31:39 dustymabe: i'd be with you if this were about a good friend. i don't feel like that regarding RH (no offense) 17:31:44 #info Upstream says "I'm pretty positive nothing about recurrent events has ever worked properly. There's a lot more that needs to be done to make it work reliably, and I'm considering how to do that, but yeah, it's just stupidly broken." 17:31:48 adamw well, true 17:31:51 if we agree to waive this but we're still no-go, it puts it in a kind of weird twilight zone 17:32:06 waive them both 17:32:11 so, is anyone going to propose that we waive the openssl bug? 17:32:14 waivers are only valid until the next go/no-go, imo, whether that's the same release or the next milestone 17:32:19 difficult to fix and late proposed blockers 17:32:43 let's get through reviewing all of the blockers and then we'll discuss waiving 17:32:57 fair 17:33:06 anything else to add on 2135772? 17:33:06 is this one likely to get a fix? 17:33:22 sounds like it's ambiguous whether it gets a fix 17:33:40 nirik: i wouldn't bet on it getting a fix for f37 at this point 17:33:45 I don't think this is getting a fix 17:33:46 * nirik wonders if we could disable recurrent events until they are fixed 17:33:55 it's not a data loss or corrupting bug, it just doesn't change the recurring event and the program crashes? 17:33:58 unless the fix is hiding editing recurring events 17:34:20 cmurf: Can't be deleted, either 17:34:27 it's a bad bug but i'd say it's going to get fixed on its own time scheduled regardless of what we have to say about it 17:34:33 that sounds like a question for the Workstation WG to decide 17:34:40 if it's made a blocker, Workstation WG will just remove it from the media 17:34:50 so let's look at our last accepted blocker 17:34:50 we've discussed it and that's what would happen 17:34:56 #topic (2137600) Plasma on Wayland hangs at startup when booting with nomodeset ("basic graphics" mode) on UEFI 17:34:57 #link https://bugzilla.redhat.com/show_bug.cgi?id=2137600 17:34:57 #link https://pagure.io/fedora-qa/blocker-review/issue/986 17:34:57 #info Accepted Blocker, kwin, NEW 17:35:04 cmurf: It's already declared a blockr 17:35:15 Ohh, that 17:35:16 *blocker 17:35:36 (Blockr is the dating app for blocker bug meeting participants...) 17:35:37 ... 17:35:42 Stephen Gallagher: wow, worst dating app *ever* 17:36:04 so, yeah, we basically know what's going on here and the proposed kernel patches would probably fix it 17:36:07 🤦‍♂️ 17:36:25 yeah, it also looks like we're going to add basic graphics mode testing too 17:36:42 hopefully we can stop this trend of discovering bugs with basic graphics mode at the very last second 17:36:43 well, we already had it, but we've extended it to make it clearer what combinations need testing 17:36:56 which is a sad chart of sadness, but hey. 17:37:02 yeeeeeah 17:37:11 Reading the bug I was wondering who would be doing that but now I get it 🍎 17:37:25 yuuuup 17:37:40 blame fancy computers 17:37:51 Will we officially support M1's with F37? 17:37:57 NO 17:38:12 so we should probably not block then 17:38:52 travier: the bug we have does not affect M1s 17:38:56 in fact they're likely the only thing that works 17:39:01 oh 17:39:06 (well, them and any other platform that uses a native 10-bit framebuffer) 17:39:11 M1s for everyone! 17:39:12 no, it broke non-M1 17:39:19 that's why the bug 17:39:28 M1 is the only type that works right now 🤣 17:39:38 Oh, so it's for NVIDIA setups then? 17:39:45 yup 17:39:48 earlier on i was suggesting we could go with a patch that fixes other things but breaks M1s, but that's not a concern any more, the proposed kernel patches ought to make things work for everyone. 17:40:00 so it sounds like there are at least short-term patches in the works that we could incorporate relatively quickly 17:40:01 ok, so it makes more sense to block on it then 17:40:10 travier: it's already a blocker 17:40:11 perhaps even Actual Fixes™? 17:40:20 we're at the point in the meeting where we discuss existing blockers and see what's going on with them 17:40:24 .👍 17:40:24 yeah, we can probably ship it into the kernel nowish if drm folks are tentatively okay with it 17:40:31 so, assuming we're not still trying to ship this week, this should be fine. 17:41:03 wait, what? I thought this was a kwin bug, not kernel 17:41:05 we'll let the drm folks fight about it a bit then jforbes can include...whatever they decide on. or something we decide on if they can't agree. 17:41:12 jforbes: it's both 17:41:19 jforbes: it's...sort of both, but we have proposed kernel patches to fix it 17:41:28 lovely 17:41:44 jforbes: see https://bugzilla.redhat.com/show_bug.cgi?id=2137600#c37 17:41:44 should we re-assign this bz to the kernel component? 17:41:51 kwin is working on a fix too, but it might not be backportable to 5.26 17:42:00 apparently v1 of that patch incited a hot controversy on "IRC", i've no idea what IRC channel or what the controversy was. 17:42:03 definitely don't expect a 5.25 backport, since the 5.25 series just ended 17:42:09 Ben Cotton (he/him): i think it's okay, everyone relevant is on it. 17:42:15 roger 17:42:28 #info kernel patches exist to address this issue 17:42:28 * jforbes notes that 6.0.5 kernels are in updates-testing now. 17:42:53 Which, given that 5.19 is EOL, and 5.19.17 is known broken, isn't such a bad thing, but... 17:42:56 * Eighth_Doctor notes plasma 5.26 is in updates-testing now 17:43:20 yeah, i don't know if we're having that debate in this meeting... 17:43:44 kernel and plasma are in the same boat at the same time :/ 17:44:20 okay, anything else on this one? 17:44:44 adamw: I know, and given the workflow now, I can always go back, but F35 and 36 will be moving forward either way, not going to hold them back 17:44:47 adamw: could we add basic graphics mode testing to OpenQA? 17:45:07 even to just suss out super-basic stuff like this would be a huge win 17:45:15 I could maybe be talked into a one-week slip if we have small, targeted fixes for the blockers. But if we're bumping major components, I'm going to push for a two-week slip. 17:45:23 it doesn't encounter this issue in vms 17:45:36 it's reproducible in UEFI VMs 17:45:54 okay, sounds like we've said all that we need to say on this particular bug for now. so 17:45:57 Eighth_Doctor: Yes 17:46:11 #topic Current status - blockers 17:46:11 #info 3 Accepted Blockers 17:46:34 does anyone want to propose that we waive all three accepted blockers? 17:47:32 two of the three blockers have fixes we could release nowish 17:47:37 Conan Kudo: we could, yeah. i didn't do it in the past because it's not considered 'valid' for release validation, but yeah, might be worth it to catch some problems. the concern might be that it would run into some bug that nobody wants to fix because it only affects VMs and there's no reason to use nomodeset in a VM 17:47:57 Conan Kudo: if we waive them, they don't get released 17:47:57 Conan Kudo: er...two? 17:48:12 there's a patch for calendar to hide editing recurring events, afaik 17:48:27 and see aforementioned dri-devel posts for kernel fixes 17:49:06 oh, okay. not sure if hiding recurring events is really the answer, but meh. i would say, one way or another, i don't want to slip for the calendar bug alone. whether that means waiving it or removing calendar or hiding recurring events i don't care 17:49:21 the KDE nomodeset one i was willing to waive for being discovered late and having a workaround (use the X11 session) 17:49:33 sure 17:49:33 but the openssl one feels like...we really should block on it 17:49:42 -1 to waiving openssl 17:49:47 it doesn't make much sense to say "we're accepting it as a blocker out of caution" and then say "...but we're not actually blocking on it" 17:49:56 Right. :) 17:50:04 I agree. We need to block for openssl at this point. 17:50:20 Decision was made, let's stick to it. 17:50:24 I'm not saying we should waive anything 17:50:32 we already put ourselves into this state, we should stick to it 17:50:43 I'm just pointing out that the non-openssl blockers have potential fixes 17:50:44 i'm not seeing any proposals to waive, so let's move on 17:50:57 So as long as we're not waiving at least one, waiving the others is irrelevant 17:51:04 For the record, I am _not_ 100% sure that come Tuesday, we won't all be like "what, really, that's what that was all about?" 17:51:05 exactly! it's all or nothing 17:51:15 catch the waive 17:51:25 But, I'm, like, 70% sure? Which is pretty high in terms of confidence about the future in Fedora in general :) 17:51:26 🏄‍♂️ 17:51:35 mattdm: sure, me either. but we've gotta be responsible, i guess 17:51:46 🏄‍♂️ 17:51:55 we will accept only a moderate amount of 'told ya so' from sgallagh 17:52:16 i dunno i think i'd waive all security bugs until we have a secure initrd :\ 17:52:16 #topic Current status - test matrices 17:52:16 #link https://fedoraproject.org/wiki/Category:Fedora_37_Test_Results 17:52:38 I'm not going to play that card. I respect the decision, I just think it would have been a different story a week ago. 17:52:40 with the understanding that adam is considering invalidating all of our current tests ;-) how are things looking? 17:53:06 seems a bit pointless to go over this now 17:53:18 but briefly, if we were going to sign off on this candidate, coverage is great. :P 17:53:43 preemptive invalidation 😛 17:54:24 #info Coverage is great 17:54:25 #topic Current status - Release Candidate 17:54:25 #info We have one. It has blockers 17:54:30 #topic Go/No-Go decision 17:54:44 pretty clearly no go today. 17:54:53 Nyet 17:54:58 #info By policy, F37 Final is NO-GO due to outstanding blockers 17:55:15 no-go, indeed. 17:55:21 #info The next F37 Go/No-Go meeting will be Thursday, 2022-11-03 at 1700 UTC 17:55:21 😭 17:55:21 #info F37 shifts to target date #3: Tuesday 2022-11-08 17:55:29 I think we need to discuss the slip, though 17:55:32 #info The Release Party will happen prior to the release 17:55:53 Or rather, whether we're going to accept big changes 17:56:00 oh gosh 17:56:01 #link https://fedoramagazine.org/youre-invited-to-the-fedora-linux-37-release-party/ 17:56:01 #topic Now what? 17:56:09 well, we could always turn it into an upgrade party? 17:56:20 people can technically upgrade to 37 ahead of time now 17:56:22 As I said above, I think if we are going to pull in a new kernel, new KDE and new Firefox... we should slip two weeks. 17:56:34 sgallagh: Aka "I hope Adam was joking with that comment in the blocker vote ticket?" 17:56:41 which kinda ties in with the 'we might need two weeks anyway given the openssl schedule' thing 17:56:43 Because that's an unreasonable amount of burden to put on the QA folks. 17:56:44 #info There's a rumination on the table to do additional updates 17:56:46 #link https://pagure.io/fedora-qa/blocker-review/issue/987#comment-823063 17:56:56 i didn't call it a proposal on purpose, but 17:56:56 i wasn't really joking. i do think there's a case for it 17:57:01 as jforbes says, kernel 5.19 is EOL now 17:57:03 I don't think we need to slip extra... if there's bugs it won't get a go 17:57:08 and so is Plasma 5.25 17:57:10 Well, No one has agreed to pull in a new kernel, but if we were to do so, we did have a full 6.0 kernel test week just last week. 17:57:21 And F36 and F35 will be on 6.0 kernels 17:57:27 I mean, there's always new stuff coming out just around the corner. 17:57:41 mattdm: yes, but not literally sitting in updates-testing 17:57:53 that's a lot of churn in a small window 17:57:56 if jforbes is on board with 6.0 in f37, I will quickly get on board with that 17:57:57 it just feels like, if we're going to be pretty late with the release, it's a better story to say "look, we baked you a nice shiny release with kernel 6.0 and GNOME 43.1 and Plasma 5.26" than "look, we baked you all this stuff that's old now" 17:57:59 Conan Kudo: Sometimes! "Zero day" 17:58:16 it's a hilarious set of tradeoffs, sure 17:58:19 just pick one 17:58:19 and Plasma 5.26 has been sitting in testing for a while now 17:58:23 And from a press standpoint, shipping with a 5.19 kernel when 6.0 is out, is one thing, but after 5.19 is EOL kinda sucks. Plus we get a good bit of hardware support 17:58:25 if we pulled in gnome, kernel, and plasma today and did an RC to test so that we're just waiting for openssl, i'd be okay with that, i think 17:58:27 it certainly is more risky from a QA perspective, but this is me with my Fedora on, not my QA hat. :D 17:58:42 bcotton: +1 17:58:48 Ben Cotton (he/him): That's a reasonable approach. 17:58:49 adamw: not confidence inspiring! 17:58:50 the GNOME update isn't built yet, i'd have to talk to workstation folks about the schedule for that 17:58:57 Plasma and kernel are in u-t already though 17:58:59 But I did not force 6.0 for a reason, so I think it is entirely QAs call as to whether we go with 6.0 or not 17:59:01 and firefox 17:59:03 i wouldn't want to wait and then try to do the whole matrix in ~36 hours 17:59:27 with my QA hat on, i am more relaxed about stuff like this than i would have been in years past, since we do have openqa now, and good communication with upstreams, and so on 17:59:38 Plasma updates run through OpenQA now 17:59:47 so they actually do get significant automated testing 17:59:52 pulling in new stuff isn't a complete leap into the unknown like it used to be 17:59:55 Can we validate an RC with just the minimal openssl fix and also the other thing in parallel or close to parallel? 18:00:05 even if we don't do another rc, we can test nightlys... 18:00:14 Ben Cotton (he/him): oh, i wouldn't want that either. this is kinda presuming we slip *two* weeks 18:00:40 the gnome update shouldn't be too disruptive and we just did a kernel 6.0 test week as jforbes noted 18:00:40 mattdm: we could, yes. the tricky thing there might be the KDE nomodeset fix 18:00:48 since the whole internet runs on openssl, i'm pretty sure cloud, server, and iot folks will want to see thorough regression testing lest we make our lives more miserable than shipping with a 0day 18:00:49 nirik: i suspect we get more testing if we call it an RC, but that might be wrong 18:00:52 we are gonna need a kernel and/or kwin update to fix that 18:01:00 cmurf++ 18:01:00 sgallagh: Karma for chrismurphy changed to 2 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 18:01:08 Workstation may not be as susceptible to openssl regressions but that's just a guest 18:01:09 guess 18:01:20 bcotton: well, if we just push those things stable, nightly's will be testable... but of course we can't go back 18:01:23 we can talk to kde upstream about a 5.26 kwin fix, but I'm much more confident on jforbes pulling the simpledrm patch 18:01:32 Two weeks is based on the assumption that we won't have a trustable fix in time on Tuesday to have a compose we feel good about on Thursday? 18:01:36 Hugl hasn't started working on a fix yet for kwin 18:01:50 i suppose we could have one RC with only the openssl fix, and releasing that would require waiving the nomodeset bug; and another RC with all the new stuff, including a fix for the nomodeset bug... 18:02:06 mattdm: yes. 18:02:10 mattdm: It's based on the fact that OpenSSL is *everywhere* and I at least wouldn't be comfortable with a 36-hour testing marathon before Go/No-Go 18:02:18 thats a lot more work sadly. 18:02:34 Because I'd want to see a thorough retest with that change, particularly if it happens to be some core functionality. 18:02:38 nirik: I would definitely feel better with a "go back" option. 18:02:51 even if the fix appears at 00:01 tuesday in europe, or something, we still need the openssl package build to happen and someone to file a compose request (could be done earlier if it's not me, i guess) and the compose to happen before we can start testing it 18:02:55 we could also create nightlies with u-t content, no? 18:03:01 sgallagh: and we won't know until late tuesday 18:03:04 like that's a thing that should be technically possible 18:03:09 afternoon EDT 18:03:15 Let's just drop openssl... 18:03:35 too late for that, we just made systemd depend on it :( 18:03:44 So realistically, if we're blocked for openssl, I think we really should give ourselves two weeks, possibly lifting the Freeze for the remainder of this week? 18:04:06 jforbes: more practical to drop initrd's 😆 18:04:08 Stephen Gallagher: I think that's sensible 18:04:11 heh, was a joke, a whole lot of things depend on it, and it would take us months to unwind 18:04:23 doing 2 rc's would allow for a 'go back' but I think would also result in less testing... only those who specially tested that rc would have those things. If we landed them in nightlys everyone running f37 would get them 18:04:31 cmurf: there is a proposal for that of sorts. 18:04:32 Nah, we can just slot in libressl, right? /me runs 18:04:43 Stephen Gallagher: ugh, sigh 18:04:44 -1 on lifting freeze 18:04:51 or perhaps -100 18:04:52 oh, the announcement has a window 18:04:53 "This release will be made available on Tuesday 1st November 2022 between 18:04:53 1300-1700 UTC." 18:04:54 sgallagh: i think 2 weeks is reasonable to just assume at this point, but i don't know about everything being released from freeze... 18:05:03 that could be a big can of worms 18:05:15 yeah I'm not a big fan of opening the flood gates again 18:05:17 I donj't think we should lift the freeze. Allowing a few shiny updates through, maybe, but not wide open 18:05:26 practically speaking, FESCo would have to approve lifting the freeze and by the time we got enough votes to do it, it'd close again anyway 18:05:28 openssl builds take ~40 minutes 18:05:38 Maybe we can get Python 3.11 final in 18:05:40 so at absolute best, really, we're gonna be kicking off the compose at 1400 UTC 18:05:49 I'm definitely envisioning an impending disaster with lifting the freeze. 18:05:50 OK, I withdraw the Freeze-lift suggestion 18:05:57 which means it'd be finished about...2000 UTC? 18:06:00 > <@adamwill:fedora.im> "This release will be made available on Tuesday 1st November 2022 between 18:06:00 > 1300-1700 UTC." 18:06:00 alright so roughly by noon on tuesday, EDT 18:06:02 -1 to lifting freeze, +1 to cherry-picking some updates 18:06:18 mhroncok: is py3.11 final in u-t yet? 18:06:28 yeah, i wouldn't propose lifting the freeze, we're not that late. i'd rather want to target shiny things, as i said - kernel, gnome, plasma, firefox 18:06:41 Eighth_Doctor: Bodhi says so https://bodhi.fedoraproject.org/updates/FEDORA-2022-a9a4c48d06 -- may not be on the mirros yet 18:06:43 So, I do want a little bit of time for feedback to pass on the kernel fix, but the plan is to put that fix into 6.0.5 and do another build? 18:07:10 jforbes: basically, yeah. we'd defer to your sense for the upstream process there, but of course the earlier we can get a build to test the better 18:07:15 s/mirros/mirrors/ 18:07:15 Even with that, I'm a little worried. The only thing that balances that is ... those things are coming to people anyway and this gives them more testing... 18:07:27 Or 6.0.6 will release on Saturday 18:07:58 mattdm: yeah, if we don't pull these things in, people get them as a 0-day update anyway 18:08:08 Additional question: who decides which things get cherry-picked in? 18:08:17 which, i dunno. who likes giant 0-days? i feel like it makes people think 'why isn't this stuff in the release?' 18:08:25 Stephen Gallagher: we use FEs, i guess. that's what they're for. 18:08:36 Ah, right. That makes sense. 18:08:47 if there seems to be general support for the concept, i will file FE bugs after this meeting, and we can vote on 'em. 18:08:54 in the end QE decides which FE's are pulled in 18:08:58 (Suffering from a head-cold today, so not braining entirely well) 18:09:12 jforbes: saturday would be fine, i guess, if we're assuming a two week slip. 18:09:18 okay, so how's this for an omnibus summary of The Plan? 18:09:31 Is there any way we could do an openssl-minimal-fix compose and snapshot that state if everything turns out to be ... pear-shaped... two weeks from now? 18:09:40 proposed #agreed Target date #3 shifts to 15 November in order to allow time for thorough verification of Tuesday's openssl update. Kernel 6.0 will be allowed to fix RHBZ 2137600. We will vote for FEs for GNOME and Plasma updates through the usual process 18:09:50 Proposal: Slip two weeks for a new target date of Nov. 8, allow certain important packages to request FEs 18:10:05 Sorry, Nov 15 18:10:05 err 18:10:08 mattdm: we *could*. that would more or less require us to wait until tuesday to start including anything *else*, though 18:10:22 or, well. hmm. at least, we couldn't push anything else stable till we get that minimal compose done. 18:10:32 s/8/15/ 18:10:33 nirik: wdyt? 18:10:34 my proposal is probably wrong, but we can fix it! 18:10:37 and like I said, the stuff would get less testing 18:10:57 Ben Cotton (he/him): patch; i'd rather just send kernel through the FE process too 18:11:16 i'm anticipating people will vote for it, but if they don't, we can do a sort of rollback and do a 5.19 kernel with the drm fix. it's just kind of a pain for jforbes. 18:11:16 ^Same 18:11:35 proposed #agreed Target date #3 shifts to 15 November in order to allow time for thorough verification of Tuesday's openssl update. We will vote for FEs for kernel, GNOME, and Plasma updates through the usual process 18:11:41 dang. that's what i had originally :-) 18:11:41 if we made a rc1.5 with kernel/gnome/plasma/firefox/whatever it would only get testing from openqa + users specifically testing it. If we push them stable, everyone running f37 will get them 18:11:48 patch 18:11:48 ack 18:12:06 * bcotton awaits another patch 18:12:22 * nirik wonders if we shouldn't leave open the chance of one week... 18:12:24 proposed #agreed Target date #3 shifts to 15 November in order to allow time for thorough verification of Tuesday's openssl update. We will vote for FEs through the usual process to allow some important late features to land. 18:12:26 but otherwise ok 18:12:47 i can live with that. i specifically called out those three to show intent, but the end result is the same 18:13:12 nirik: it's reaaaaaally borderline. best case we have a compose for testing at 2000 UTC on tuesday, before go/no-go at 1700 UTC (right?) thursday 18:13:13 Right, but since we've already had at least a fourth (python 3.11) proposed, I figured it was better to be generic 18:13:25 2000 UTC tuesday is pretty much after brno is done testing for the night, so realistically they get one day on it 18:13:27 nirik: The difference would be tomorrow, the weekend, monday, and probably tuesday, right? 18:13:32 me and the other NA folks get a day and a half 18:13:43 Stephen Gallagher: yeah, also firefox 18:14:03 I am planning to propose FE for mozjs102/gjs combo too 18:14:13 mattdm: yes... 18:14:20 frantisekz: would that not be part of gnome 43.1? 18:14:27 Propose all you want; doesn't guarantee acceptance ;-) 18:14:53 heh, should be, I shall ping somebody from Workstation WG to just edit my patch or pull those builds from it to GNOME mega 18:14:53 I'm -1 to this proposal without having a minimal just-the-openssl (waive the other current blockers) fallback 18:15:35 I guess we could do that as RC 1.5? 18:15:43 and RC 1.6 mega thingy? 18:16:00 other way? 18:16:03 mattdm: With respect, those other issues *are* blockers. We were going to waive them via the Late Blocker exception, but now that time is available they should really get included if at all possible 18:16:22 I am open to bringing back release names for "Fedora Linux 37 (Mega Thingy)" 18:16:33 heh :D 18:16:39 That... might not go so well in newsfeeds. 18:16:46 "Fedora Linux 37 (In a Row?)" 18:16:52 "Fedora Linux 37 Creepy Spam" 18:17:18 * nb proposes "Fedora Linux 37 (The Return of Beefy Miracle)"!! 18:17:25 mattdm: only if you can get a logo designed to go with it 18:17:37 Isn't that just a synonym for mattdm's? 18:17:45 lol 18:17:50 Stephen Gallagher: Yes, that's the same exception I'd suggest waiving them under. That compose would be as if we are trying to get a release out as fast as possible. 18:18:30 mattdm: As noted above, that still limits us severely, since we can't (realistically) start on the testing of those fixes prior to getting the OpenSSL patch. 18:19:08 And if we start work on them, we need to finish it. Backing them out will be harder than fixing them.. 18:19:55 > * <@nb:libera.chat> proposes "Fedora Linux 37 (The Return of Beefy Miracle)"!! 18:19:55 needs more unicode 18:19:59 we can be somewhat cautious landing things... I think we could do some test compose and run it thru openqa at least? and//or tell users 'install these from updates-testing' and test? 18:20:06 I mean... the openssl stuff needs to be tested anyway 18:20:32 mattdm: It needs to be regression tested. 18:20:32 adamw: Variation in needs more cowbell? 18:20:33 but it would be good to test the rest now while we have time instead of waiting until we have openssl in hand... 18:20:37 Bug verification is different 18:20:43 alright, so the current proposal on the table is 18:20:43 proposed #agreed Target date #3 shifts to 15 November in order to allow time for thorough verification of Tuesday's openssl update. We will vote for FEs through the usual process to allow some important late features to land. 18:20:45 we know what mattdm thinks of it. where does everyone else stand? 18:20:45 * Variation on needs more cowbell? 18:20:45 yes, unless i'm missing something, we could do a candidate compose with other stuff in it before the openssl-only compose, if we wanted 18:20:54 as we can add things to and remove things from the side tag as we like 18:20:57 ack 18:21:09 ack 18:21:13 Yes: reminder... I am the FPL, not the Fedora Pope :) 18:21:21 ack 18:21:28 i am on board with the plan, i'm willing to work with releng to accommodate matt's "get a minimal openssl-fix-only compose on tuesday" plan if it's generally desired, we can work out the details as we go 18:21:36 let's land all the stuff ahead of openssl 18:22:03 we can't "land" it, in the sense of push it stable, before openssl, if we want to do the minimal compose 18:22:04 Eighth_Doctor: that would be my preference too, then we only have openssl to deal with 18:22:24 but we can do a candidate compose with kernel 6.0, plasma 5.26, firefox 106, and whatever else we want. if we like. we could do that now. 18:22:27 right, having a openssl only compose makes it harder 18:22:28 nirik, Conan Kudo same here 18:22:46 adamw: I like. Do it. 18:23:31 +1 for adding things early to give time to test 18:23:44 then the openssl fix will be the cherry 18:23:51 Just so I'm understanding: the candidate compose would pull those things in but they wouldn't be moved to stable generally? 18:23:54 right now we don't have the simpledrm fix in the kernel, and we don't have gnome 43.1. but we could do a build with other things, i guess. 18:24:19 mattdm: right. 18:24:22 adamw: Well, 6.0.5 is in update's testing, so if you are going to do one, push that in 18:24:25 mattdm: that's an option, yes. nirik is arguing (AIUI) for pushing them stable earlier so we get more testing (from folks who don't have u-t enabled). the cost of that option is we could not do the minimal compose. 18:24:29 we have a f37-compose tag we can put things in 18:24:43 right 18:24:59 It doesn't have the particular fix, but the specific fix is easy to verify 18:25:19 (this is, of course, assuming we accept FEs) 18:25:33 jforbes: yeah. i can build a scratch kernel to test the fix, i guess 18:25:42 okay, so it sounds like we have a general consensus 18:25:47 adamw: I can build one with it 18:25:59 Ben Cotton (he/him): well, we are still arguing the 'do a minimal compose or not' point, but i guess we can continue that outside the meeting 18:26:11 #agreed Target date #3 shifts to 15 November in order to allow time for thorough verification of Tuesday's openssl update. We will vote for FEs through the usual process to allow some important late features to land. 18:26:12 jforbes: if you could that'd be great 18:26:14 Just wanted to see a little feedback when we have 2 different patches pushed out in hours with heated discussion 18:26:21 both today 18:26:31 jforbes: i'll leave it up to you whether to go with v1 or v2...the drm-devel list does seem to have the discussion about what some people didn't like in v1 18:26:46 er, dri-devel 18:27:02 right 18:27:05 https://lists.freedesktop.org/archives/dri-devel/2022-October/377407.html 18:27:25 The "additional testing" would be from people who have updated already and might notice if we break something horribly? 18:27:37 (Sorry I'm slow at typing and thinking both at once today.) 18:27:57 mattdm: yes. we've already disabled u-t by default for f37 now 18:28:03 so a typical f37 install does not get the stuff from u-t 18:28:11 only people who actively enable it, like for stable releases 18:28:12 (I noticed: I had to turn it back on!) 18:28:31 if we push things stable, anyone who's running f37 but hasn't turned on u-t will get it, and hopefully yell if it breaks stuff 18:29:05 What practical benefit would that give vs. doing it 5-6 days from now? Gonna break either way. 18:29:32 Or rather, if we do it later but have the composes you mentioned, hopefully we'll catch it before then. 18:30:10 less time to fix things we do find... 18:30:11 what is u-t ? 18:30:17 updates-testing repo 18:30:53 nirik: Assuming that they're _only_ found by people we shipped broken stuff to (on beta, to be fair) 18:31:15 the practical benefit would be we'd find out sooner and have more time to fix whatever was borked. 18:32:05 Yeah, I get the more time thing -- but that's assuming that the wider push will find things we wouldn't with u-t + validation testing, right? 18:32:16 yes. 18:32:39 The number of users of f37 is much higher than the number who test rc's... :) at least I think so 18:33:11 i think we can work out this part of the plan outside of the meeting 18:33:21 anything else before we close out? 18:33:22 I guess I'm setting the "big important hard to fix thing found in that way" weight as low in my thinking. 18:33:47 fedora-37 0.1766 % 66767 18:34:05 Ben Cotton (he/him): Okay. I've said my bit. I think the risk mitigation is worth it. 18:34:22 now you're speaking my language 18:34:30 (ie, 67k mirror hits for f37 today) 18:34:47 nirik: one sec, looking at countme 18:36:41 For this week, 33220 for updates-released and 9367 for updates-testing. 18:37:52 And generally the former should be a superset of the latter. So about 28% of current F37 systems have the testing repo enabled 18:38:19 right, but thats not what I am compairing. ;) That number to... say 10 qe people? 18:38:43 https://bugzilla.redhat.com/show_bug.cgi?id=2138247 - kernel 6.0 FE proposal 18:38:50 anyhow, I'll stop we can do the fallback compose if you want 18:39:48 mattdm: how many of those with testing still enabled are because packit is checking, but they are not updating at all 18:40:12 meaning, they aren't actually testing. I have such a laptop here 18:40:47 jforbes: Definitely some! I was just trying to think of how I can quantify that 18:41:03 last call for meeting-relevant #content 18:41:11 lol sorry 18:41:23 right, but it could be some, it could be a majority, we have no way of knowing since the default change was in an update itself 18:42:14 #endmeeting