17:05:46 #startmeeting F38 Final Go/No-Go meeting 17:05:46 Meeting started Thu Apr 13 17:05:46 2023 UTC. 17:05:46 This meeting is logged and archived in a public location. 17:05:46 The chair is amoloney. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions. 17:05:46 Useful Commands: #action #agreed #halp #info #idea #link #topic. 17:05:46 The meeting name has been set to 'f38_final_go/no-go_meeting' 17:05:46 Old school 17:05:53 #topic Roll Call 17:05:57 .hello lruzicka 17:05:58 rockin. 17:05:58 lruzicka: lruzicka 'Lukáš Růžička' 17:05:58 .hello nphilipp 17:06:00 nils: nphilipp 'Nils Philippsen' 17:06:01 .hello bittin 17:06:01 morning everyone 17:06:03 luna-: bittin 'Luna Jernberg' 17:06:05 Hi all! 17:06:12 .hello geraldosimiao 17:06:13 geraldosimiao: geraldosimiao 'Geraldo S. Simião Kutz' 17:06:28 #topic Purpose of this meeting 17:06:28 #info Purpose of this meeting is to check whether or not F38 Final is ready for shipment, according to the release criteria. 17:06:42 #info This is determined in a few ways: 17:06:49 #info 1. No remaining blocker bugs 17:06:57 #info 2. Release candidate compose is available 17:07:07 #info 3. Test matrices are fully completed 17:07:34 #topic Current status - blockers 17:07:34 #link https://qa.fedoraproject.org/blockerbugs/milestone/38/final/buglist 17:07:34 #chair adamw 17:07:34 Current chairs: adamw amoloney 17:08:01 .hello2 17:08:02 frantisekz: frantisekz 'František Zatloukal' 17:08:22 * SumantroMukherje is here 17:08:41 .hello farribeiro 17:08:41 ahoyhoy 17:08:41 farribeiro: farribeiro 'Fábio Ribeiro' 17:08:44 so, for blockers... 17:09:13 #info https://bugzilla.redhat.com/show_bug.cgi?id=2185827 is confirmed fixed in both RC-1.4 and RC-1.5 17:09:34 let's talk about the accepted blocker 17:09:40 #topic (2113005) Live image made with BOOTX64.EFI from latest shim-x64-15.6-2 fails to boot on some boards 17:09:41 grrr 17:09:44 there goes the rich text thing 17:09:52 #topic (2113005) Live image made with BOOTX64.EFI from latest shim-x64-15.6-2 fails to boot on some boards 17:09:58 #link https://bugzilla.redhat.com/show_bug.cgi?id=2113005 17:10:01 #link https://pagure.io/fedora-qa/blocker-review/issue/1087 17:10:06 #info Accepted Blocker, shim, NEW 17:10:16 hmm, not sure making me a chair worked 17:10:21 rats 17:10:24 anyone know what my nick is on the irc side? 17:10:33 adamw ? 17:10:34 let me try do it again 17:10:39 #chair adamw 17:10:39 Current chairs: adamw amoloney 17:10:43 thats normal 17:10:47 it never says anything for link or info 17:10:54 i thought it echoed topic, though 17:11:02 but i guess not 17:11:05 okay! proceeding 17:11:15 i'm gonna propose we formally waive this one, which has been the plan all along 17:11:18 Ctrl-shift-V work to paste as plain text? Sometimes is kind of a convention, almost... 17:11:24 not on the matrix side sadly. It does on irc. 17:11:26 it sucks, but we are basically blocked on fixing this. 17:11:33 waive +1 17:11:43 we can't get a new shim signed unless we have nx support in the kernel, there's no clear timeframe for that getting merged 17:11:44 waive +1 17:11:46 +1 to waive. We don't know when it will be fixed. It only affects some subset of machines. 17:11:58 i am +1 to waiving 17:12:00 +1 waive and hope people have the right machine 17:12:09 +1 waive 17:12:14 CommonBugs? 17:12:18 it's already there 17:12:22 Perfect 17:12:22 +1 waive 17:12:24 kparal has updated it to cover f37 and f38, thanks kparal 17:12:44 +1 waive 17:12:45 kparal++ 17:13:15 #info https://bugzilla.redhat.com/show_bug.cgi?id=2113005 has been waived and is added to the CommonBugs page for reference 17:13:25 this would be under the 'difficult to fix' clause? 17:13:50 proposed #agreed 2113005 - waived to F39 Beta blocker - we agreed to waive this blocker as a "difficult to fix" blocker per https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process#Exceptional_cases (external constraints make it impractical to address this in any reasonable timeframe). for now it becomes an f39 beta blocker. 17:13:50 yes 17:14:01 ack 17:14:01 ack 17:14:08 ack 17:14:13 amoloney: sorry, i'm following the blocker review bureaucracy :P 17:14:15 ack 17:14:16 ack 17:14:32 I was warned about that :-D sorry! went too soon 17:14:34 #agreed 2113005 - waived to F39 Beta blocker - we agreed to waive this blocker as a "difficult to fix" blocker per https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process#Exceptional_cases (external constraints make it impractical to address this in any reasonable timeframe). for now it becomes an f39 beta blocker. 17:14:35 ack 17:14:35 this is the way 17:14:40 okay, that was the easy one 17:14:44 let's do the hard one! 17:14:54 this is a proposed blocker: 17:14:59 #topic (2186358) Missing package warning when installing to NVMe device with F38 DVD and no network 17:15:02 #link https://bugzilla.redhat.com/show_bug.cgi?id=2186358 17:15:06 #link https://pagure.io/fedora-qa/blocker-review/issue/1153 17:15:11 #info Proposed Blocker, distribution, VERIFIED 17:15:21 #info Ticket vote: FinalBlocker (+5,0,-0) (+tflink, +geraldosimiao, +sumantrom, +augenauf, +kparal) 17:15:30 please note, a lot of those votes came in while this was assumed to be a fatal error. 17:15:46 * luna- has not had time to look at it, any tldr? 17:16:05 so, here's the sitch: this was proposed late (yesterday). we did a very quick fix and built RC-1.5 with the fix. the fix is confirmed in RC-1.5, but it has obviously had very little other testing (only smoke testing). 17:16:28 install w/o network repos on nvme disk produces error about missing package. you can continue with the install and the system works but the prompt does halt the isntallation 17:16:36 adding to the hilarity, a lot of images failed in the RC-1.5 build due to some kinda transient network problem or something. so we probably don't want to just ship RC-1.5 as-is. 17:16:52 fix in RC 1.6 ;) 17:16:53 so, if we ignore the warning and proceed, the installation goes correct? 17:16:59 yes 17:17:00 yes 17:17:20 so I think this is no more a blocker 17:17:21 I would be -1 blocker and common bugs 17:17:22 or rather with :p 17:17:43 it kind of looks pretty bad 17:17:51 the last question has been around ks installs - the ks installs I've done halt for input at the same point 17:17:55 our choices here are, as I see them: 1a) reject this as a blocker, ship RC-1.4. 1b) accept as a blocker but waive as a late blocker, ship RC-1.4. 2a) accept as a blocker, ship RC-1.5 (probably not a good choice). 2b) accept as a blocker, ship some kind of RC-1.4/RC-1.5 hybrid (probably RC-1.4 but with the server tree or image dir from RC-1.5). 3) accept as a blocker, slip for another compose and try to get everything to build this time. 17:18:03 nirik, the bad look could be improved with common bugs 17:18:06 but not being fatal makes it harder to decide 17:18:10 tflink: If that's the case, then I have to vote +1 blocker 17:18:20 Because that is effectively a fatal error. 17:18:36 if you need the install to be unattended, yeah, it's an issue 17:18:51 * mattdm is considering ughness of this 17:18:53 If you're using a kickstart, the assumption has to be that it's unattended. 17:19:02 ^^ 17:19:06 I think 2a is out, we are missing a number of labs/spins. 17:19:12 * luna- votes or 3 17:19:17 *for 17:19:26 I want to do one more ks install with the non-everything repo and no updates. the ks install I've tested uses the cdrom as an install source which I can't imagine anyone actually does 17:19:29 Do people do no-network kickstart installs? 17:19:30 seconded, +1 for 3) 17:19:57 mattdm: For certain definitions of "people", yes 17:20:00 for the record, i would argue the appropriate criterion here is "The installer must be able to complete an installation using any supported locally connected storage interface" with a side order of "Any installation method or process designed to run unattended must do so. There should be no prompts requiring user intervention." 17:20:12 Stephen Gallagher: are those *really* people, though 17:20:28 It's not uncommon for gov't entities to have a secure network with no access to the mirrors. 17:20:41 nirik (@nirik:libera.chat): how awful would 2b) be? given that we already have to hack up the compose anyway 17:20:51 is it a lot more work/mess to hack the Server bits from 1.5 into 1.4 ? 17:20:54 I think it would be somewhat horrible. 17:20:57 I am in favor of 2b 17:20:57 I'd lean harder on the Go button here but we are at the early target 17:20:59 (I'd personally go for 1b) 17:21:09 the metadata will be wrong... and websites is using metadata. 17:21:15 1b for me 17:21:18 The images would have 1.4/1.5 and confuse people 17:21:27 No finding more bugs next week though, yeah? 17:21:29 nirik (@nirik:libera.chat): websites uses fedfind, i think 17:21:36 so it'd be on me to make sure fedfind handled the mess 17:21:37 it upsets my releng brain a lot 17:21:50 which...i can probably make it do. i think we did this kinda thing once before, actually? 17:21:57 I'm thinking 1b) or 3) 17:21:57 not entirely I am 100% sure. ;) 17:22:00 Frankly, I'm in favor of slipping for this. 17:22:04 from view of server WG I would ask for 3 17:22:11 (And you know how much I hate saying that) 17:22:13 adamw, did you decide it warranted a repeat performance? :D 17:22:13 I removed old f37 composes and broke f37 websites last week. 17:22:18 so I am sure they are using the metadata 17:22:30 nils: i dunno, i repress memories of release cycles the day after they finish, usually ;) 17:22:42 We put some missed spins up separately at least once 17:22:46 adamw++, good coping mechanism 17:22:48 well, we do have some FE material that didn't get to the last 1.5 RC so I thinh doing a new RC is agood idea 17:22:54 I forget, does the default source for a network install include more than just the DVD contents? 17:23:04 tflink (@tflink:fedora.im): yes 17:23:15 geraldosimiao: I'd actually be opposed to shoehorning any new changes in if we slip 17:23:27 nirik (@nirik:libera.chat): well, we don't *publish* the metadata for the actual release tree because it's a lie 17:23:29 tflink: Depends on which netinst it is. 17:23:31 so i don't see how they could be? 17:23:36 spins are different from server... server would need it's own tree and images and pxe images. 17:23:53 I mean this here, form xfce https://bodhi.fedoraproject.org/updates/FEDORA-2023-76b7e94b44 17:23:55 they were using the staged rc 17:23:58 *for 17:24:07 ie, /pub/alt/stage/37/.... 17:24:14 ah. 17:24:20 thats what I deleted. since I didn't think we needed it anymore 17:24:28 how bad could it be to recompose for 1.6 without any further changes? 17:24:51 It could be fine, but not sure we could test it really in time to ship next week. 17:24:51 also... 17:24:51 .hello ngompa 17:24:53 Eighth_Doctor: ngompa 'Neal Gompa' 17:25:13 I do have some ideas about the network/weirdness. No concrete fix, but possibly some things to try. 17:25:22 nirik: What was your "it could be fine" message in response to? 17:25:38 doing a 1.6 compose without any changes. 17:25:44 Conan Kudo: in theory not at all. we thought 1.5 would be easy. :P 17:26:07 we (websites) can use any metadata you provide, we can even point each edition to a separate set of metadata if we need to (but I would prefer not to) 17:26:08 but it went badly due to "transient network errors" right? 17:26:24 we hope they were transient :P 17:26:35 if we do, I'd like to try and push some changes in infra (namely to make internal stuff use less loaded proxies and disable h2 on the registry at least) 17:27:06 so, um 17:27:18 the failures were "timed out" which doesn't really tell us what happened. There was also a http2 error I found on one. 17:27:24 it kinda sounds like there's a pretty strong consensus to make this +1 blocker, whatever we do about it? 17:27:32 let me see if we have the votes 17:28:24 +1 blocker 17:28:39 +1 blocker 17:28:52 yeah, sadly I think this is a blocker... +1 blocker. 17:28:52 +1 blocker, +1 waive 17:28:54 +1 blocker 17:28:58 yeah, It think I'm still +1 blocker due to the issues with unattended install unless the usual networked install ks methods work and this is only an issue with a cdrom source 17:28:59 +1 blocker 17:29:15 +1 blocker but I'm serious about not finding more bugs next week. :) 17:29:22 +1 blocker +1 waive 17:29:24 I keep my former +1 blocker 17:29:31 okay 17:29:38 let's do this in stages, then 17:30:58 proposed #agreed 2186358 - AcceptedBlocker (Final) - this is accepted as a violation of "The installer must be able to complete an installation using any supported locally connected storage interface" combined with "Any installation method or process designed to run unattended must do so. There should be no prompts requiring user intervention", in the case of an attempted unattended install of Server DVD to NVMe storage 17:31:07 ack 17:31:08 ack 17:31:10 ack 17:31:12 ack 17:31:13 ack 17:31:15 was that short enough for IRC side? 17:31:17 ack 17:31:20 yep 17:31:30 #agreed 2186358 - AcceptedBlocker (Final) - this is accepted as a violation of "The installer must be able to complete an installation using any supported locally connected storage interface" combined with "Any installation method or process designed to run unattended must do so. There should be no prompts requiring user intervention", in the case of an attempted unattended install of Server DVD to NVMe storage 17:31:31 ack 17:31:33 ack 17:31:35 ack 17:31:41 okay. so. votes on waiving this and shipping RC-1.4? 17:31:43 ack 17:31:48 +1 waive 17:31:53 -1 for waiving 17:32:00 -1 17:32:01 +1 waive 17:32:06 +1 waive 17:32:17 -1 17:32:43 + waive - I think it's an obscure case we can document 17:32:53 i am going to invent, as a principle, that unless we have a strong consensus for waiving (along the lines of the approximate +3 we expect for a blocker/fe vote), the default is to *not* waive 17:33:05 I assume under the 'last minute' clause? I'm a bit torn... 17:33:10 this isn't in the policy anywhere, if anyone thinks i'm wrong, we can argue about that too. :D 17:33:16 I'm in favor of that principle 17:33:30 yep, sounds alright adamw 17:33:39 yeah, this would be as a "Last minute blocker bug" - it clearly doesn't qualify as "Difficult to fix", it took me about twenty seconds. :D 17:33:49 I guess I am a weak +1 to waive... but will bow to others if we don't. ;) 17:34:16 may i please be heard? 17:34:31 oh thats another reason 1.4/1.5 hybrid will be difficult. Different comps in server tree vs everything. People who make dvds from everything would still hit the bug. 17:34:37 Given that we're still at the Early Release point, I'm inclined to block on this. 17:34:49 linuxrulez: sure, everyone's heard here 17:34:56 we are on early fina target still 17:34:56 waive -1 17:35:02 pboy: do you have a vote here? 17:35:05 that bug is not a new one, it's several weeks old. 17:35:14 con we test for 24 hours and reconven for a GO/No GO tomorrow? 17:35:17 i would like to have it fixed! 17:35:24 pboy: it's newly proposed as a blocker, which is the criterion 17:35:31 OK 17:35:32 s/reconven/reconvene 17:35:40 "bugs proposed as blockers 5 days or fewer before the scheduled Go_No_Go_Meeting for a milestone release (Beta or Final) can be considered under this policy" 17:36:01 waive -1 17:36:05 linuxrulez: i think we've decided previously that we're not going to do that. it's also problematic because we don't have a compose to ship rn 17:36:05 we have a week to do a new, good RC with all right. 17:36:07 a great quesiton... not usually since it means we don't have enough time to stage the release, mirror it everywhere, get things in place, etc. 17:36:09 if 1.5 was a good compose we'd have an easier choice 17:36:14 note 17:36:24 if we vote not to waive this, i am still gonna ask for votes/thoughts on shipping 1.5 or a 1.4/1.5 hybrid 17:36:39 that's what i meant by doing this in stages 17:36:40 i'm trying to make it clear by considering one thing at a time 17:36:49 * sgallagh nods 17:37:09 linuxrulez: the big problem is that our compose process is a long and slow one 17:37:17 pboy: so i'm counting your vote as -1 waive, right? 17:37:18 so getting one out isn't as simple or quick as we'd like 17:37:43 well, the compose is 4ish hours... but then we have to test it, sync it around, etc. 17:37:47 adamw. yes. -wiaive 17:38:29 so as things stand, we're at -6, +3.5 17:38:35 (counting nirik as +0.5) 17:38:37 nirik: least it's not 9 hours 17:38:48 i think it's reasonable to consider that as rejecting the proposal to waive 17:38:52 iirc, last it was discussed at flock it was that long 17:38:55 yeah, it's better than it was at the worst for sure. 17:39:00 we did some work on it. ;) 17:39:16 (also that was seven years ago) 17:39:28 much sad 17:39:44 #agreed we took a vote on waiving 2186358 as a late blocker (thus clearing RC-1.4), the vote was +3.5/-6, so the proposal does *not* succeed, the bug is not waived and stands as a blocker in RC-1.4 17:39:48 😢 17:39:51 ack 17:39:57 ack 17:39:57 ack 17:39:58 ack 17:40:01 ack 17:40:01 Adding a reason: the bug would affect many new devices of all things, very bad. 17:40:06 ack 17:40:13 ack 17:40:21 okay, so: the resolved situation is, there is an accepted blocker for RC-1.4. there is no accepted blocker for RC-1.5, but it is missing a lot of images and has reduced testing coverage 17:40:26 ack 17:40:37 i'm gonna hand the meeting back to amoloney at this point as the blocker review portion is done 17:40:43 we can talk about what to do from here next 17:40:43 adamw: is there missing anything that couldn't be taken from 1.4? 17:40:48 roger, roger 17:40:49 I assume none of the missing images are blocking images? 17:41:01 Stephen Gallagher: technically no, but there's a lot of important stuff missing 17:41:05 like...silverblue x86_64 17:41:10 eek 17:41:20 Stephen Gallagher: right. there's 10 missing artifacts. 17:41:30 the full list is at https://pagure.io/releng/failed-composes/issue/4855 17:41:55 also Sway is missing, which has some buzz on the socials right now, it'd be a shame not to have that 17:42:02 s390x too, some missing 17:42:15 at least Budgie didn't fail 17:42:24 yeah, sway is only at rc 1.4 17:42:38 budgie is good to go 17:42:40 frantisekz: between rc-1.4 and rc-1.5 we have almost a complete compose, but combining them in any way is difficult, as nirik said 17:42:53 and to be clear, all these failed for infrastructure reasons, not any bug in their contents or whatever. 17:43:14 could we hybridize the results and create a new number to inject for metadata? 17:43:22 #info bug 2186358 is not waived and blocks RC-1.4 17:43:26 Can we reconvene in four hours after running a 1.6? 17:43:38 should we change topic to discuss plan going forward? 17:43:38 adamw: yeah, that was meant as for " has reduced testing coverage" 17:43:41 Conan Kudo: we *could* do anything. the problem is the metadata is quite big and complicated and tied to the directory layout on the mirror, and it's already generated 17:43:50 blech 17:43:56 trying to "fix it up" would involve a lot of hand-editing things and moving things around and hoping we didn't get anything wrong 17:44:02 or move to discussing other accepted blockers? 17:44:12 In other words, it might be harder to fix all of it up than to just run another compose 17:44:13 i think this is the only accepted blocker as things stand? 17:44:17 it has checksums and all kinds of things. ;) 17:44:25 adamw, rolling a new compose with reduced testing sounds safer than that 17:44:29 the next topic would be test coverage, right? which plays into the discussion 17:44:32 It is L:) 17:44:44 Yes test matrices are next 17:44:48 changing topic now 17:44:55 Stephen Gallagher: it would definitely be easier to just run another compose from that perspective, yes 17:45:12 #topic Current status - test matrices 17:45:30 amoloney: we are kinda running ahead of the discussion, which tends to happen in this meeting, but it's fine to just plod through the topics and get everything down 'formally' with info tags and everything 17:45:31 #link https://fedoraproject.org/wiki/Category:Fedora_38_Test_Results 17:45:37 if we want to pull a rabbit out of our hat, if we think we can make another compose today and get some testing in to verify nothing's changed, we could postpone as suggested 17:45:39 it also gives people time to reconsider things 17:46:00 ack 17:46:05 ack 17:46:09 it's not like I have to dig into sddm into the wee hours of the morning again :) 17:46:10 +1 for that 17:46:12 RC-1.5 test results: https://fedoraproject.org/wiki/Test_Results:Fedora_38_RC_1.5_Summary 17:46:25 RC-1.4 test results: https://fedoraproject.org/wiki/Test_Results:Fedora_38_RC_1.4_Summary 17:46:57 RC-1.4 coverage is good. only somewhat significant missing things i see are some windows and macos dual boot tests. 17:47:02 The sole code change between them was this blocker issue, yes? 17:47:17 however, those were done with 1.5 17:47:18 +1 for postpone, not for difg into sddm 17:47:21 yes. it's not even a code change 17:47:24 it's just a comps change 17:48:01 the effect of the change is that nvme-cli is pulled into the server DVD package repo, and also included on all live images 17:48:01 ack 17:48:01 ack 17:48:01 +1 to postpone and have a RC 1.6 with no bugs and Sway and Silverblue 17:48:01 it did not drive any blocking image oversize or we'd have a bug 17:48:30 well, if we do that, is it worth trying to pull a rabbit out of our hats? 17:48:51 like, can we carry forward testing info from RC 1.4 and RC 1.5 we have now? 17:48:51 #info combined test coverage for RC-1.4 and RC-1.5 is very good, and the difference between them is small; we should have confidence in the quality of either (aside from the accepted blocker in 1.4) or any combination of the two 17:49:01 since none of the content is changing for 1.6 17:49:05 Conan Kudo: we can, but we need to at least do smoke testing 17:49:13 what do we do if we delay a decision, wait for an rc1.6, and it too is missing images...? 17:49:17 it is always a possibility that some bitflip or something makes an image go bad 17:49:23 coremodule: We slip. 17:49:23 fair 17:49:39 so we have been pretty resistant to trying to hero things in the last few cycles 17:49:40 but i mean, i'm ok with it if everyone is 17:49:41 yeah, if it screws up again, we'll just delay by a week and beat the crap about the network :) 17:49:48 it depends if pushing the decision to later today or tomorrow is too tight for other teams 17:49:58 I expect this fix get into xfce iso too https://pagure.io/fedora-qa/blocker-review/issue/1149 17:50:05 Yeah. Not worth hero work for an early target. 17:50:18 geraldosimiao: if we're trying to hero things, we should not incldue *any* other changes 17:50:23 if we slip a week we can maybe pull in high-value FEs 17:50:24 adamw: +1000 17:50:43 anyhow 17:50:47 the official topic rn is test coverage 17:50:53 anyone have anything to add to my #info ? 17:50:55 right, so... I can think of 2 small infra changes that might make things more reliable... 17:50:57 that was part of 1.5 17:51:00 wasn't that part of 1.5 already? 17:51:00 Eighth_Doctor: no, it dint get 17:51:04 meh 17:51:08 * nirik waits for the next topic 17:51:23 we did not pull anything into 1.5 compared to 1.4 except the comps change to fix the blocker. that was by design. 17:51:37 okay 17:52:05 well, I care slightly less in this case since unofficial respins (that need to become official somehow) would pick this up anyway 17:52:56 yeah, ok. but xfce people will point this: we do not trust our own installer... 17:53:13 nirik: Could you fire off a 1.6 compose while we debate this? 17:53:14 as the livesys-scripts change is only for xfce, if adamw wants to include it, it's net neutral for everyone else 17:53:18 We can always ignore it :) 17:53:26 sgallagh: we need to fix infra first 17:53:33 ok 17:54:08 I'm okay with or without including the livesys-scripts update for 1.6, leaning slightly toward doing so since I had to expend effort for this release :P 17:54:09 I am looking at the failures from 1.5. ;) 17:54:29 no workstation s390x for rc1.5 17:54:54 geraldosimiao: I'm sure all zero of our s390x users will be upset by that 17:55:03 ok 17:55:08 😁 17:55:22 i do all my mainframe tasks from GNOME, what do you mean 17:55:26 I didn't realize we even built workstation for s390x. TIL 17:55:27 so whats the next topic? shall we move along? or are we are deciding 1.6 hero or not? 17:55:38 we should move along, i think 17:55:40 amoloney? 17:55:47 happy to! 17:55:54 we do must release in a proper manner budgie and sway, since this are new spins 17:56:01 I think we need to know how long it might be before a 1.6 could be kicked off. 17:56:14 If we think the answer is "two hours", maybe we give it a go. 17:56:18 you've info'd on the test matrices already @adamw, anything else needed here to close this off? 17:56:22 If the answer is 8+ hours, I say we slip 17:56:41 (Debate open if it's between those extremes) 17:56:44 next up is the release candidate topic which I will set now if were good 17:57:09 i'm good on the coverage topic 17:57:12 tflink[m], the s390 builds of gimp uncovered quite some security issues way back when 😁 17:57:18 #topic Current status - RC 17:58:44 #info we have an RC-1.4 which is disqualified by the accepted blocker (and missing s390x cloud base) but otherwise good 17:59:25 #info we have RC-1.5 with the fix for the blocker, but missing many other images, including ones we consider important even if they're not technically blocking, like Silverblue x86_64 and the new Sway spin 18:00:04 #info combining RC-1.4 and RC-1.5 in some way is theoretically possible but practically difficult and likely to result in some kind of unforeseen mess 18:00:53 i guess it's kinda unprecedented to block the release because "non-blocking" images are missing, but it feels like a no-brainer. we'd be dummies to ship with no Silverblue installer. 18:01:09 +1 agree 18:01:11 Under a strict reading of our rules, we *should* be shipping RC-1.5... but I'm of the opinion that "a sufficiently large number of technically non-blocking issues" constitutes a blocker. 18:01:26 +1 18:01:29 at this point we're not really on the blocker bug process, we're on the go/no-go process... 18:02:44 are we good to that topic then? 18:03:24 * nirik nods 18:03:25 yeah, we can move to the decision topic 18:03:32 excellent 18:03:34 #topic Go/No-Go decision 18:03:44 I will poll each team. Please reply “go” or “no-go” 18:03:53 well 18:03:59 FESCo? 18:04:00 i think this is where we can have the more freeflowing discussion about hero options etc. 18:04:01 Do we have five FESCo members present? We could probably dig back out of this precedent if we get a FESCo blocker for the images. 18:04:17 well, we're here 18:04:20 because if we want to hero it, the technical thing to do (I think) is to adjourn this meeting 18:04:21 that's two 18:04:32 nirik makes three 18:04:40 Stephen Gallagher: tbh, there isn't a solid rule here that i can find 18:04:53 adamw: or keep open and reconvene later 18:05:05 all we have is https://docs.fedoraproject.org/en-US/program_management/pgm_guide/sop/go-nogo/ and https://docs.fedoraproject.org/en-US/program_management/pgm_guide/release_process/#_gono_go_meeting 18:05:08 I have a bit more data to offer: 18:05:32 the older https://fedoraproject.org/wiki/Go_No_Go_Meeting says that *QA's* vote is based on fixed principles that would require QA to vote "go" at this point, but it doesn't say anything about any other stakeholders' vote 18:06:18 Let's hear nirik's data before we go too far down that road. 18:06:24 the blocker process is very wordy and rules-y, because i wrote most of it. :P but we're outside of the blocker process at this point, into territory which is much less rules-y and wordy because neither me nor anyone like me has got to it yet. :D 18:06:27 Of the 10 artifacts that failed in 1.5, 9 of them failed with some timeout or 503 error. I think this might be solved by a simple dns change. smooge and I are discussing it in the noc channel. The last one failed with a http/2 error, but it might also be related to high traffic on proxies and be solved by the dns thing, so I don't know that diabling http/2 on there is worth doing right now. 18:06:51 so, I think it's reasonable to update dns, then fire a 1.6... if folks want to try doing that. 18:07:04 its always the dns ;) 18:07:13 If QA is good with that, I'm in favor. 18:07:18 there's no harm in firing one 18:07:40 right, but then when and how do we decide what to do with it? ;) 18:07:44 how long would you all like to give it? 18:07:50 we can talk about that while it runs. :P 18:08:03 if we talk long enough, maybe it'll finish! 18:08:08 When it completes and we smoke-test it, we can make a decision on whether to ship it or delay a week 18:08:11 :D 18:08:23 i don't think this meeting has an anti-filibuster provision...:P 18:08:37 ha. 18:08:43 so i would like to tell the channel my life story, starting at the age of one day 18:08:46 continue tommorow with a smoke tested RC 1.6 :D ;) 18:08:57 thats a 3 hour job? or something like 8 hour? building all of them? 18:09:09 geraldosimiao: 4 hours i think someone said 18:09:10 * sgallagh starts quoting that Simpson's episode about wearing an onion on his belt... 18:09:12 im infoing this now so if people are happy to wait while infra makes some changes 18:09:16 would it make sense to consider including the livesys-scripts update into there for xfce? 18:09:22 ok 18:09:31 about 4.5 hours per https://kojipkgs.fedoraproject.org/compose/38/Fedora-38-20230412.0/logs/global/pungi.global.log 18:09:31 it's a pretty concentrated change 18:09:38 Conan Kudo: no. 18:09:39 ah 18:09:41 we're not touching nothing. 18:09:47 ... 18:09:52 if we slip a week, we'll do it 18:09:56 for a potential hero run, nope 18:10:01 okay 18:10:03 fine with me 18:10:04 Don't be telling me no double negatives! 18:10:12 🤣 18:10:15 we can run a 1.7, we're not gonna run out of integers :D 18:10:27 sgallagh, triple it is then! 18:10:39 Have we ever actually gotten to RC 1.double-digits? 18:10:43 I wonder what that would break... :) 18:10:46 1.7, integer, make your choice :P 18:10:47 #info the meeting is in a waiting period while an RC-1.6 runs with some infra changes before reconvening to vote to Go/No Go 18:10:56 nils: 7 is the integer ;) 18:11:00 rc-1.Ӛ 18:11:06 it's not technically a decimal or a fraction, the dot is just a separator 18:11:23 so, uh, yeah 18:11:32 i feel like bcotton would usually be the one arguing against trying to fudge this 18:11:39 let me see if i can find some old logs to channel him 18:11:45 he left it to me....should I argue? 18:11:47 It's a good thing he's not here then! 18:11:48 I can 18:11:54 adamw, maybe you can fill me in where the 1 comes from, I didn’t see anything else there, and by the time we’re done the compose is, too? *crosses fingers* 18:12:17 nils: hah :D um, they have names in, uh, i think it's productmd ? 18:12:43 adamw, wouldn’t the way du jour be to let some AI channel Ben? 😂 18:12:44 nils: I think that number has to exist for RHEL reasons, so we just always keep it at 1 18:12:57 Actually Ben told me we (royal we) dont ever rerun the compose... 18:13:11 * tflink[m] goes to ask ChatGPT whether we should slip ... 18:13:32 that makes me think... what could go wrong with rhel, if we tried to hcange the 1 to 2, or emoji? who's for that to find out? 18:13:43 you 18:13:51 yeah, I tried to kill off the 1. a while back... there's some reason for it... 18:13:53 tflink: I already did, but I found its answer of "Don't worry about it, puny humans" to be somewhat concerning. 18:13:55 or Stephen Gallagher 18:13:57 oh, no, it is just meant to be a major/minor thing, hum. i thought they were actually separate attributes somehow. i guess i misremembered. 18:14:03 sgallagh, lol 18:14:09 nirik (@nirik:libera.chat): i think the reason is internal composes of something use it 18:14:39 also productmd is very tied to both the major and the minor being there, it'd be sad without them - https://github.com/release-engineering/productmd/blob/master/productmd/composeinfo.py#L106 etc. 18:15:09 frantisekz: a 2.n should work fine. emoji.n maybe not. :D 18:15:38 * nils wonders why it isn’t 38 then by this point 18:16:13 nils: Do not taunt happy fun productmd 18:16:37 ok, dns change made. 18:16:40 nils: the label (that's what that thing is) didn't exist until, uh...when did we start using pungi 4? somewhere around f20? 18:16:42 thanks smooge 18:16:44 frantisekz: Fedora-KDE-Live-x86_64-38-😄.5.iso 18:17:06 sgallagh, is that “taunt (happy fun productmd)” or “(taunt happy fun) productmd”? 😁 18:17:20 smooge++ 18:17:20 nils: Karma for smooge changed to 6 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 18:17:25 geraldosimiao: that's only *marginally* sillier than that time we thought it'd be a great idea to put both a space and a non-ASCII character in the release name for the first time 18:17:26 geraldosimiao yeah :D 18:17:31 that went well 18:17:52 yes, it uncovered bugs 18:18:02 The 🐈 is alive! 18:18:12 …should be the official line of a QA person 18:18:58 meanwhile, finding old go/no-go meeting logs is harder than it should be 18:19:17 anyhow. I am ready to fire 1.6... shall I ? 18:19:23 do it 18:19:43 nirik, +1 18:19:44 nirik: I thought you'd meant 🚀.6 though... 18:19:54 * Eighth_Doctor feels tempted to propose emojis as version codenames 18:19:59 he is waiting for adamw to say do so 18:20:20 FIRE! 18:20:26 Conan Kudo: considering all software handles non-ascii unicode without any issues, I'm sure that would go well :) 18:20:47 tflink: that's sarcasm, right? :P 18:20:50 🔥 18:20:53 here's something from an f37 final go/nogo: "17:14:51 i've been told in the past that we cannot rely on CPE to do any of the releng work over the weekend, so a friday go/no-go seems unpossible" 18:21:01 https://meetbot-raw.fedoraproject.org/teams/f37-final-go_no_go-meeting/f37-final-go_no_go-meeting.2022-10-27-17.01.log.txt 18:21:02 Conan Kudo: Sure, but only if every release codename is a unicode emoji new to ICU as of that release 18:21:08 oooh 18:21:12 I like where you're thinking 18:21:12 nirik (@nirik:libera.chat): do it 18:21:16 i'll retrospectively file a request 18:21:39 1.6: STARTED 18:21:45 nirik++ 18:21:50 Conan Kudo: 18:21:51 ok then :) 18:21:53 the best case is that it blows up right now 18:21:54 https://pagure.io/releng/issue/11378#comment-851434 18:21:55 nirik: can you keep posting live buildlog here now? 18:21:56 who won 18:21:56 nirik++ 18:21:56 geraldosimiao: Karma for kevin changed to 13 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 18:21:58 :D 18:22:02 ann poor sould behind a matrix bridge… 18:22:06 nirik++ smooge++ 18:22:06 *souls 18:22:21 nirik++ 18:22:21 sgallagh: Karma for kevin changed to 14 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 18:22:21 smooge++ 18:22:23 you'd think I'd remember how to do cookies 18:22:24 sgallagh: Karma for smooge changed to 7 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 18:22:26 nirik++ 18:22:27 smooge++ 18:22:31 nirik++ 18:22:35 smooge++ 18:22:39 frantisekz: just tail https://kojipkgs.fedoraproject.org/compose/38/Fedora-38-20230413.1/logs/global/pungi.global.log 18:22:50 thanks :) 18:23:08 I can setup redirect from tail to my hexchat window... 18:23:13 but seriously, nirik - do you foresee problems getting necessary releng work done if we try to ship this thing? 18:23:14 we can all watch together here... :D 18:23:14 so... we wait? do we close this and do a new meeting? 18:23:15 sgallagh, seriously, I think codepoints new to ICU wouldn’t matter a bit 18:23:21 smooge++ 18:23:21 geraldosimiao: Karma for smooge changed to 8 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 18:23:52 nils: We'd find out quickly if you're right 😈 18:24:06 adamw: well, I think we don't usually work weekends, but thats not because no one wants to (although that might be true too), but because we use the weekend to sync to mirrors. We stage on friday and mirrors get a lot of syncing done over the weekend. 18:24:10 sgallagh, so let’s do it! 😂 18:24:15 how do i tail an URL? *n00b* 18:24:23 if we made a new release on say sunday, that wouldn't give enough time until tuesday. 18:24:24 nirik (@nirik:matrix.scrye.com): it's been long enough since we tried to fudge/hero that i don't remember which we do. we probably can't clog up #fedora-meeting for hours, other people must have meetings planned 18:24:38 yes it generally takes 3 days for mirrors to get a full release from us 18:24:59 .nextmeetings 18:24:59 sgallagh: One moment, please... Looking up the channel list. 18:25:02 sgallagh: b'In #fedora-meeting is Fedora UK Ambassadors (starting in 34 minutes)' 18:25:05 sgallagh: b'In #fedora-social is "Late" Fedora Social Hour (starting in 4 hours)' 18:25:05 adamw: the last one I remember was for F34, and I'm pretty sure that was us closing and opening the meeting a couple of times 18:25:08 sgallagh: b'In #fedora-meeting is Flatpak SIG (starting in 3 days)' 18:25:09 I think we kept it open before... but... I don't care, we should do what makes sense. Do note that in 4 hours amoloney might be not around. ;) 18:25:11 sgallagh: b'In #fedora-meeting is Fedora QA Meeting (starting in 3 days)' 18:25:14 nirik (@nirik:matrix.scrye.com): so if this compose is done in five hours and we can do sufficient smoke testing to sign it off in, uh, an hour or two, can we stage it in seven hours? 18:25:14 sgallagh: b'In #fedora-blocker-review is Fedora 38 Blocker Review Meeting (starting in 3 days)' 18:25:30 i'm chaired, i can take over, i guess 18:25:30 so if you want to release on Tuesday you have to be ready by mid Friday 18:25:46 smooge: how "mid" are we talking 18:25:50 I can set an alarm :) 18:25:59 or maybe tailing the URL was a joke? 18:26:01 yeah, I would think so. if 1.6 is not suitable, then we should just punt a week. or if it takes more than a few hours to tell. 18:26:20 luna-: it was mostly a joke, but there's probably a way :D 18:26:49 adamw: I would go with 1400 UTC to have it staged in place on /pub/fedora /pub/fedora-secondary 18:26:51 adamw: stupid autist that takes things literal, worked good in a browser tough :p 18:26:54 you can just reload it from time to time if you want... but note that it gets pretty big. ;) 18:26:59 well, by the time it finishes it'll be around midnight czech, so it'll be mostly on me, tim and coremodule to smoke test it i guess 18:27:17 smooge: alright 18:27:23 adamw: we can use the fedora blocker review room for this? 18:27:25 so it needs to be in a place releng can do that 4 hours of work which means 1000 UTC ( 0300 PT) 18:27:30 in Sweden too and don't really have energy to smoke test midnight on a Friday 18:27:33 if its a problem here 18:27:33 so i trust in you guys 18:27:38 luna-: sorry, i joke way more than is responsible. :P 18:27:43 hopefully jednorozec can do the actual staging (before I am awake :) 18:27:45 yay 18:28:04 yeah luna, please don't stay up! we got it 18:28:04 adamw: its fine, just that i somethings take stuff a bit to literal atleast my system did not break 18:28:12 oh, so one other catch... 18:28:14 May I ask...is the chance of 1.6 being good enough to use good enough to delay ~4 hours 18:28:18 and then see 18:28:26 amoloney: i think so, yeah 18:28:30 I'm in Brazil, so, plenty of light still 18:28:45 oh great geraldo, you can help too then 18:28:52 we usually want a final branched with jusr exactly the final content... so that will need to run before we open updates, but that shouln't be too bad 18:29:07 Ok then, if its worth the gamble then this hopefully might be the extent of the heroics for this one! :) 18:29:07 yeah, we can probably do a stable push now in fact 18:29:44 so there isn't another meeting on the official schedule at least for three days 18:29:50 so we can probably just leave the meeting open 18:30:28 oh, you mean 'specific stable push' not a 'open the floodgates' one right? 18:30:43 👀 18:31:04 #info F38 Go/No Go Meeting is in an OPEN state while a new RC is generated (RC-1.6). The meeting will reconvene in ~4 hours to check the status and test coverage of the new compose 18:31:41 nirik (@nirik:matrix.scrye.com): yeah, of course - i just mean we can sync up what's in 1.4/1.5/1.6 18:31:54 actually quick question - will there be reps from FESCo, QA & RelEng here to vote on the RC later to Go/No Go? 18:31:58 the only tricky thing is https://bodhi.fedoraproject.org/updates/FEDORA-2023-1ef83c76e4 , which needs to get pushed even though it's obsoleted 18:32:04 qa, yes, i'll be here 18:32:30 amoloney: I can be here for FESCo 18:32:48 amoloney: nirik never sleeps 18:32:52 so releng is covered 18:32:58 Beautiful :) 18:33:28 I'll be around :) 18:33:28 I do have to drop but Ill be back online to help out with the admin stuff 18:33:40 amoloney++ 18:33:40 smooge: Karma for amoloney changed to 1 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 18:33:46 amoloney++ 18:33:50 and thanks for all the karma from others 18:34:07 luna I don't think zodbot recognizes your nick so its not giving karma 18:34:07 You all estimate to be able to discuss again at about 2300 Ireland Time? 18:34:12 amoloney++ 18:34:12 nirik: Karma for amoloney changed to 2 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 18:34:26 I really have no idea how to do the UTC conversions, sorry :-/ 18:34:39 date -u tells you the current time in UTC 18:34:56 it is 18:34 18:35:00 or *you* can tell me :-p 18:35:06 someone should add a conversion command to zodbot 18:35:17 tell a person the UTC time, they know the UTC time once. teach a person to find the UTC time... 18:35:24 hahahaha 18:35:47 UTC is the most widely used unknown timezone :) 18:35:54 i also find https://www.timeanddate.com/worldclock/ super handy 18:35:56 adamw, what if I told you that the Irish are on their standard time now? 18:36:02 I operate on my own time 18:36:14 nils: I'd tell you we have a bug for that 18:36:14 nils: i would say what's the alternative? 18:36:24 amoloney, ok, everybody except you 😜 18:36:30 which is about 5-25 mins late for everything :-D 18:36:31 amoloney: I have empirical evidence that you operate on *my* time. 18:36:36 adamw, Irish Winter Time or somesuch 18:36:37 hahahaha 18:36:39 https://www.jdieter.net/posts/2021/03/28/the-paddys-day-bug/ 18:37:15 its 19:36 here for me so Ill be back on around 2300 my time...is 3.5 hrs enough time to give the RC testing and such? 18:37:21 this bug has cropped up several times now in other places :) 18:37:44 amoloney, not sure, the compose builds for another 4:15 I guess? 18:37:50 amoloney: That's about when the compose will finish, so no testing yet 18:37:51 amoloney: i'm looking at more like 6-7hrs 18:38:10 amoloney++ 18:38:10 geraldosimiao: Karma for amoloney changed to 3 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 18:38:20 we'll want to at least boot the blocking images 18:38:23 I swore when I read 6-7 hours 18:38:30 coremodule: will you be around to test arm images? 18:39:31 amoloney: Yeah, but you also swore when your read "the", so... 18:39:32 right am off now, fingers crossed RC-1.6 does the business and will see you all in an undetermined amount of hours thats more than 4 and less than 7 18:39:52 anyone else need to be added as chair at this time? 18:40:15 and do I need to set the topic or info as anything else than I have already done? 18:40:40 Are we holding this open or closing it? 18:40:55 meeting stays open I believe? 18:41:48 yeah, it doesn't seem like anyone else is going to need the channel 18:42:00 amoloney: i think we can just leave it like this 18:42:08 ack 18:42:12 ciao! 18:43:35 * nirik went to get coffee. was there a question for me? 18:44:55 probably! 18:45:32 nirik: Does the day end in "y"? 18:46:03 yep... 'the day' does end in y. ;) 18:46:22 Thanks! 18:46:38 What is the flight speed of an unladen swallow? 18:46:59 * nils practices the Wilhelm scream 18:47:17 African, or European? 18:47:27 Both! Either! 18:47:44 monty python feelings 18:48:08 😄 18:48:22 .meetings 18:48:25 .nextmeeting 18:48:25 luna-: (nextmeeting ) -- Return the next meeting scheduled for a particular channel. 18:48:32 .nextmeeting #fedora-meeting 18:48:33 luna-: b'In #fedora-meeting is Fedora UK Ambassadors (starting in 11 minutes)' 18:48:36 luna-: b'In #fedora-meeting is Flatpak SIG (starting in 3 days)' 18:48:39 luna-: b'In #fedora-meeting is Fedora QA Meeting (starting in 3 days)' 18:48:42 luna-: - https://calendar.fedoraproject.org/location/fedora-meeting%40irc.libera.chat/ 18:49:13 yeah, this UK Ambassadors... is gona happen here? 18:49:20 oh wait 18:49:22 dang 18:49:27 i misread the list... 18:49:31 we can send away the britts, nah maybe i should be nice :D 18:49:40 LOL 18:49:59 we'd better end meeting for now and reconvene later, then 18:50:09 i'll do it since amoloney left 18:50:09 ack 18:50:20 won't join then a bit too late will read the email later tommorow 18:50:27 on what happend :) ;) 18:50:44 Things will have happened, that’s for sure! 18:50:48 if it said boom in a good or bad way 18:50:51 #agreed we will adjourn the meeting while RC-1.6 composes and undergoes smoke testing. we will reconvene at 0000 UTC 18:51:06 ack 18:51:14 ack 18:51:15 see y'all then 18:51:25 #endmeeting