16:59:56 <bcotton> #startmeeting F32 Final Go/No-Go meeting
16:59:56 <zodbot> Meeting started Thu Apr 16 16:59:56 2020 UTC.
16:59:56 <zodbot> This meeting is logged and archived in a public location.
16:59:56 <zodbot> The chair is bcotton. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:59:56 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:59:56 <zodbot> The meeting name has been set to 'f32_final_go/no-go_meeting'
16:59:58 <bcotton> #meetingname F32-Final-Go_No_Go-meeting
16:59:58 <zodbot> The meeting name has been set to 'f32-final-go_no_go-meeting'
17:00:08 <bcotton> oh no, i started 4 seconds early!
17:00:19 <sgallagh> .hello2
17:00:20 <zodbot> sgallagh: sgallagh 'Stephen Gallagher' <sgallagh@redhat.com>
17:00:39 <mboddu> .hello mohanboddu
17:00:40 <zodbot> mboddu: mohanboddu 'Mohan Boddu' <mboddu@bhujji.com>
17:00:44 <frantisekz_> .hello2
17:00:46 <zodbot> frantisekz_: Sorry, but you don't exist
17:00:53 <nirik> morning
17:00:55 <bcotton> pwnt
17:00:56 <frantisekz_> .hello frantisekz
17:00:57 <zodbot> frantisekz_: frantisekz 'František Zatloukal' <fzatlouk@redhat.com>
17:01:21 <pbokoc> .hello2
17:01:21 <nils> .hello nphilipp
17:01:21 <zodbot> pbokoc: pbokoc 'None' <pbokoc@redhat.com>
17:01:24 <zodbot> nils: nphilipp 'Nils Philippsen' <nphilipp@redhat.com>
17:01:26 <pbokoc> yeah that's me
17:02:04 <bcotton> #topic Roll Call
17:02:11 * pwhalen is here
17:02:17 <bcotton> (not that we aren't already doing this, but...)
17:02:21 <mboddu> So, we have to do it again? :P
17:02:30 <mboddu> .hello mohanboddu
17:02:31 <zodbot> mboddu: mohanboddu 'Mohan Boddu' <mboddu@bhujji.com>
17:02:35 <bcotton> mboddu: :-p
17:02:43 <sgallagh> .hello2
17:02:44 <zodbot> sgallagh: sgallagh 'Stephen Gallagher' <sgallagh@redhat.com>
17:02:53 <bcotton> (you really don't, it's okay :-) )
17:02:58 <nirik> I don't think we need to do it again. ;)
17:03:03 <mhroncok> hello
17:03:04 <nirik> lets not get stuck in a loop. ;)
17:03:06 <mboddu> Haha, I was just kidding
17:03:21 <bcotton> .helloisitmeyourelookingfor
17:04:39 <bcotton> hokie dokie, let's get started
17:04:48 <bcotton> #topic Purpose of this meeting
17:04:49 <bcotton> #info Purpose of this meeting is to check whether or not F32 Final is ready for shipment, according to the release criteria.
17:04:51 <bcotton> #info This is determined in a few ways:
17:04:54 <bcotton> #info 1. No remaining blocker bugs
17:04:56 <bcotton> #info 2. Release candidate compose is available
17:04:57 <bcotton> #info 3. Test matrices for Final are fully completed
17:05:08 <bcotton> soooooo, here comes the fun
17:05:25 <bcotton> #topic Current status - blockers
17:05:27 <bcotton> #link https://qa.fedoraproject.org/blockerbugs/milestone/32/final/buglist
17:05:41 <bcotton> #info 0 Accepted Blockers
17:05:50 <bcotton> #info 2 Proposed Blockers
17:05:55 * bcotton plays the dramatic music
17:06:04 <bcotton> #topic (1824418) Rescue Mode doesn't recognize LVM -Partitions
17:06:06 <bcotton> #link https://bugzilla.redhat.com/show_bug.cgi?id=1824418
17:06:07 <bcotton> #info Proposed Blocker, anaconda, NEW
17:06:19 <bcotton> so i know folks are actively looking into this As We Speak[tm]
17:06:27 <nirik> this challenger just stepped into the ring a bit ago. ;)
17:06:40 <lruzicka> .hello2
17:06:41 <zodbot> lruzicka: lruzicka 'Lukáš Růžička' <lruzicka@redhat.com>
17:06:50 <bcotton> my understanding is that this is reproducible, but only on some bare metal
17:07:03 <lruzicka> I would say, correct.
17:07:18 <sgallagh> bcotton: I'm not sure if it's only "some" yet.
17:08:01 <lruzicka> adamw could not reproduce it on bare metal, I think ... so it affects some, however the "some" can be a vast number.
17:08:12 <adamw> ahoyhoy
17:08:15 <adamw> sorry, still looking at this
17:08:31 <adamw> sgallagh: it's not reproducing on my test box so far
17:08:49 <adamw> my best bet so far is it has something to do with the disk characteristics
17:08:55 <adamw> (filesystem, whatever)
17:09:18 <bcotton> so the question is: given what we know right now, is this a blocker. and if it is, can we invoke the late blocker exception
17:09:35 <bcotton> should we debate the first part or would a few more minutes of testing make this more clear?
17:09:45 * kparal tests quickly
17:09:47 <adamw> i think more testing and data  will help
17:09:52 <bcotton> okay
17:10:08 * jlanda waves
17:10:10 <bcotton> #info we are still testing and will come back to this one
17:10:25 <bcotton> #topic (1824216) final f32-backgrounds version isn't in stable repo
17:10:26 <bcotton> #link https://bugzilla.redhat.com/show_bug.cgi?id=1824216
17:10:28 <bcotton> #info Proposed Blocker, f32-backgrounds, MODIFIED
17:10:49 <nirik> I agree with sgallagh.
17:10:56 <adamw> my vote remains as it was in the bug: +1 blocker, +1 invoke late blocker exception and waive it
17:11:15 <frantisekz_> adamw, agreed, +1 blocker, waive
17:11:20 <lruzicka> adamw, to which one? the backgrounds?
17:11:21 <mboddu> I agree with sgallagh as well
17:11:23 * nirik also agrees with adamw. I guess I am just agreeable today
17:11:27 <adamw> also, i guess, +1 FE so if we slip for the rescue mode thing we pull it in
17:11:32 <adamw> lruzicka: backgrounds. yes.
17:11:36 <lruzicka> I am lost in who agrees with whom for which bug.
17:11:38 <adamw> since that is the topic now.
17:11:44 <adamw> lruzicka: look at #topic. :)
17:12:07 <mboddu> I am +1 FE but I dont think we should consider it as a blocker, if we do, then +1 to late blocker exception
17:12:07 <lruzicka> Ok, so my vote for background is +1 blocker definitely
17:12:14 <nirik> +1 blocker, +1 invoke 'too late blocker exception', +1 FE, and if we release with it, common bugs and perhaps a mention in the announcement.
17:12:18 <bcotton> i cound +4/-2 blocker from the ticket
17:12:28 <adamw> i'm really struggling for how this is -1 blocker tbh
17:12:36 <adamw> the criterion says the 'final artwork' must be included
17:12:36 <jlanda> +1b +1 waived b +1fe (is the fe really needed for a waived accepted blocker?)
17:12:39 <lruzicka> but how do you imagine to waive it? does it mean Fedora is not going to get a new background?
17:12:43 <adamw> this is clearly the final artwork and it is clearly different to what's in there right now
17:12:52 <bcotton> adamw: i think it's the unwritten "ugh, seriously?" exception :-)
17:12:52 <adamw> jlanda: yeah it is
17:13:08 <adamw> lruzicka: no, it means we get the one that's there right now. the one from beta. that's kinda MINTY FRESH color
17:13:23 <adamw> not the newer version that's a bit bluer. and has an animated version that doesn't get used by default.
17:13:36 <lruzicka> ok, in that case, I am ok with waving it.
17:13:38 <pwhalen> +1 blocker, waive
17:13:39 <jlanda> actually we should not discuss this, no final background is an autoblocker :D
17:13:49 <adamw> also petition for the unofficial release name for F32 to be MINTY FRESH
17:13:58 <sgallagh> adamw++
17:14:01 <mhroncok> +1 blocker, no idea what a waive is
17:14:03 <nirik> ha. yes!
17:14:16 <mboddu> +1 to MINTY FRESH :)
17:14:20 <adamw> jlanda: no it isn't
17:14:25 <adamw> "The default desktop background in a release-blocking desktop being the same as one of the last two stable releases" is an automatic blocker, but this isn't that.
17:14:43 <bcotton> okay so it seems like the consensus is that this is a blocker. anyone want to make a compelling argument that it is not? (setting aside the question of if we invoke the late blocker exception or not)
17:16:02 <bcotton> proposed #agreed  AcceptedFinalBlocker - BZ #1824216 - per Final criterion "The proposed final Fedora artwork must be included and used as the background on release-blocking desktops."
17:16:24 <nirik> ack
17:16:27 <jlanda> ack
17:16:27 <mboddu> ack
17:16:47 <sgallagh> mhroncok: We have a special rule we can invoke that a blocker that comes in too late (<24 hours before Go/No-Go can be waived by this meeting)
17:16:49 <lruzicka> ack
17:17:04 <adamw> ack (well, i usually write it as "#agreed 1824216 - AcceptedBlocker (Final) - explanation" but hey)
17:17:28 <mhroncok> sgallagh: thanks. lruzicka was already briefing me on this via DM
17:17:41 <bcotton> #agreed 1824216 - AcceptedBlocker (Final) - per Final criterion "The proposed final Fedora artwork must be included and used as the background on release-blocking desktops."
17:17:52 <bcotton> adamw: thanks. i can never remember what the style is :-)
17:17:57 <adamw> sgallagh: it's actually "bugs proposed as blockers 5 days or fewer before the scheduled Go_No_Go_Meeting"
17:18:04 <mhroncok> whatever the decision on waiving this will be, i strongly suggest we don't change the color of the backgroun on zero day update
17:18:06 <sgallagh> Even better
17:18:10 <mhroncok> or any update after GA
17:18:15 <adamw> mhroncok: we'll burn that bridge when we get to it
17:18:20 <mhroncok> ack
17:18:24 * sgallagh looks around for his torch
17:18:36 <lruzicka> sgallagh, maybe a box of matches is enough
17:18:36 <bcotton> okay, so the next question, do we invoke the exception that allows us to waive the blocker because it was reported < 5 days before go/no-go
17:18:44 <nirik> +1
17:18:45 <bcotton> in the ticket, i counted +3/-1
17:19:03 <lruzicka> do we know a reason why the final background is not there?
17:19:04 <sgallagh> I suggest we wait on this
17:19:10 <adamw> yes! yes we do. as argued in the bug, it got proposed like a day before the meeting. that's way too late.
17:19:22 <sgallagh> lruzicka: The maintainer was late in delivering it
17:19:34 <adamw> sgallagh: even if we slip i honestly would like to waive this just to establish the principle that the frickin background needs to get done on time
17:19:35 <sgallagh> I suggest we defer this question though
17:19:38 <mboddu> Lets wait on the rescue bug and if thats a blocker, then we can include this
17:19:48 <sgallagh> adamw: That's... fair
17:19:51 <adamw> for the purposes of getting it into a slipped respin, +1 FE is enough
17:20:01 <bcotton> if we defer, then it's no longer a late bug :-)
17:20:28 <adamw> actually, hm, i dunno if we decided on a process for the case where we waive a bug under this policy but then the release slips for some other reason
17:20:33 <bcotton> i'd argue that the waived status is not permanent, but applies at each go/no-go
17:20:56 <nirik> bcotton: are you a lawyer? :) But I like it...
17:20:59 <adamw> i mean, that's not how it's written
17:21:00 <mhroncok> what bcotton says makes perfect sense
17:21:12 <adamw> the policy says when we waive it we accept it as a blocker for the next milestone
17:21:35 <adamw> so it's kinda implicit that at that point it's no longer a proposed or accepted blocker for this milestone
17:21:39 <sgallagh> Which would mean it becomes a blocker for a slipped release, I guess
17:21:46 <adamw> no, it means this should block f33 beta
17:21:48 <bcotton> adamw: i'd argue that's a failure of imagination on our (my?) part and the intent is that it's not permanent
17:21:51 <adamw> (which is obviously silly for another reason)
17:21:57 <sgallagh> I think it's poorly phrased, but sure
17:22:01 <adamw> bcotton: i honestly would incline the other way
17:22:10 <adamw> in *this* case it's just a case of pulling in a package, okay
17:22:18 <adamw> but what if we for instance waived the 'kde user switch' bug or something
17:22:23 <adamw> something which requires significant upstream development
17:22:31 <adamw> we then have to keep re-waiving it if we slip? ehhh
17:22:38 <adamw> oh well. it can work either way.
17:23:01 <adamw> the practical element is also icky for me if we don't waive it "permanently" because then we would just have to leave it on the proposed blocker list
17:23:17 <bcotton> well no, it's an accepted blocker now
17:23:32 <adamw> right now it's an accepted F32 Final blocker
17:23:42 <bcotton> because as you've pointed out in teh past, it has to be accepted before it can be waived
17:23:46 <adamw> if we waive it, the current process says we make it an accepted F33 Beta blocker instead
17:23:47 <mboddu> I think we should just wait and see what happens with the rescue bug and then we can vote on this as definite blocker or waive it with the exception
17:24:07 <bcotton> mboddu: but what if i'm stalling to get an answer to the rescue bug :-)
17:24:14 <lruzicka> for me it makes the following sense: if the late blocker is so serious that it would harm the system badly, it should not be waived anytime, so I think if we wave a bug that is a blocker but not really a serious one, than it could go as a next release beta blocker as adamw is saying.
17:24:16 <adamw> bcotton: shhhhh we were supposed to keep that as subtext!
17:24:16 * nirik is +1 to wave it... the FE is enought to bring it in if we need it
17:24:37 <adamw> that's the part you aren't supposed to read out loud
17:24:50 <mboddu> ^^ or what nirik said
17:25:16 <mboddu> +1 waive and +1 FE
17:25:36 <bcotton> #action bcotton to propose edits to the late blocker exception to make waiver status more clear
17:26:05 <mhroncok> can we decide here to only waive it in a way that bcotton was reading the policy?
17:26:05 <adamw> i think mboddu's idea may be the best
17:26:28 <adamw> that we only consider waives once we are sure we have no non-waiveable blockers
17:26:48 <adamw> i.e. it's only worth considering applying the 'late blocker' stuff if there's a chance we'll be go
17:26:52 <bcotton> are we ready to come back to the other bug?
17:27:04 <kparal> no, argue for 10 more minutes please :)
17:27:09 * kparal debugging
17:27:22 <kparal> (just kidding)
17:27:51 <adamw> let's just topic back to the rescue mode one yeah
17:28:00 <adamw> we have some more data now at least. and it's not good data!
17:28:08 <bcotton> #topic (1824418) Rescue Mode doesn't recognize LVM -Partitions
17:28:10 <bcotton> #link https://bugzilla.redhat.com/show_bug.cgi?id=1824418
17:28:11 <bcotton> #info Proposed Blocker, anaconda, NEW
17:28:12 <nirik> I know we kid, but if we need more time to debug a bug, we should just explicitly wait, so we don't keep everyone sitting in the meeting listening to elevator version of 'run to the hills' :)
17:28:55 <bcotton> so based on the information available, i'm +1 blocker here and also I'm +1 late blocker exception because it's not universal and also people can rescue from F31 media so there's a workaround
17:28:56 <kparal> so, I can reproduce it on both bare metal (KDE, default part) and VM (minimal inst from Everything netinst, default part)
17:28:58 <adamw> how about the tori amos version of raining blood
17:29:09 <kparal> F31 DVD can rescue to system just fine, F32 DVD can't
17:29:22 <frantisekz_> +1 blocker, +1 late/waive
17:29:28 <adamw> ehhh
17:29:34 <adamw> i don't know if i'd want to waive this one...
17:29:39 <adamw> it's more...substantial
17:29:47 * kparal agrees with adamw
17:29:49 <mboddu> Yeah, I am not sure either
17:29:50 <frantisekz_> come to the dark side....we have cookies adamw :) :) :D
17:30:03 <adamw> f31 as a workaround is fine unless you took your f32 dvd to the offline location, or whatever...
17:30:07 <nirik> question: does the 'rescue' option on boot on the installed system work?
17:30:11 <lruzicka> I will ask, is there a specific criterion that the rescue mode must work?
17:30:14 <adamw> lruzicka: yeah
17:30:30 <adamw> https://fedoraproject.org/wiki/Fedora_32_Beta_Release_Criteria#Rescue_mode
17:30:35 <adamw> "The rescue mode of the installer must start successfully and be able to detect and mount any installation performed according to the applicable criteria, and provide a shell with access to utilities capable of performing typical recovery operations."
17:30:40 <adamw> pretty clear
17:30:54 <kparal> nirik: do you mean the rescue option in grub? That just uses full initramfs with all drivers, no?
17:30:59 <adamw> this is a conditional violation as it doesn't happen to everyone, but it's clearly hitting a lot of cases
17:31:02 <nirik> kparal: yes. that one
17:31:07 <lruzicka> adamw, that is pretty clear
17:31:07 <adamw> they're kinda...different rescues
17:31:26 <adamw> you can certainly get into a situation where you need the media rescue and the rescue kernel won't work
17:31:31 <nirik> sure.
17:31:36 <sgallagh> And neither have ever worked in the history of computing :-P
17:31:41 <nirik> but there is some overlap.
17:31:42 <adamw> (also that whole rescue kernel area seems hella janky since BLS showed up and i'm not sure i trust it any more)
17:31:54 <kparal> but that's a completely different "rescue"
17:32:08 <bcotton> as some anecdata, i can't remember the last time i used either rescue mode, if ever
17:32:19 <bcotton> and clearly my use case is totally representative
17:32:21 <kparal> ok, I can reproduce the same thing in VM with default Workstation Live install
17:32:29 <sgallagh> I've attempted to use the install media one on several occasions. It has never worked for me.
17:32:36 <kparal> so this is clearly either about VM configuration or disk size or something
17:32:49 <sgallagh> kparal: Really big VM disk?
17:32:49 <frantisekz_> I didn't know there is such rescue mode... also, I didn't see it in Workstation Live/UEFI at all
17:32:51 <nirik> is it uefi vs bios at all?
17:32:52 <mboddu> I never used rescue mode myself either, but thats just me
17:32:54 <kparal> sgallagh: 15GB
17:32:58 <lruzicka> seems so, because my VM was just fine.
17:32:58 <adamw> nirik: don't think so, openqa tests both
17:33:00 <kparal> nirik: BIOS in my case
17:33:01 * sgallagh facepalms
17:33:01 <adamw> openqa test passed on both
17:33:14 <kparal> I use SCSI disks in VM
17:33:18 <adamw> frantisekz_: this rescue mode doesn't exist on lives, only traditional installer images
17:33:20 <kparal> but the bare metal has SATA
17:33:42 <frantisekz_> (adamw, now, I am feeling for even stronger late waive)
17:33:46 <lruzicka> my VM was also a 15Gib, bios based
17:33:59 <bcotton> okay, let's start with the first part
17:34:04 <adamw> are you guys booting your VMs with a virtual optical disc or a virtual USB?
17:34:20 <adamw> can someone post the anaconda logs from a failure case?
17:34:23 <nirik> could be somehow scsi vs virtio vs ?
17:34:24 <bcotton> proposed #agreed 1824418 - AcceptedBlocker (Final) - The rescue mode of the installer must start successfully and be able to detect and mount any installation performed according to the applicable criteria, and provide a shell with access to utilities capable of performing typical recovery operations.
17:34:32 <lruzicka> but I am using a VirtiO drive for disks
17:34:44 <adamw> i do not have a blocker vote yet
17:34:46 <sgallagh> my VM was 20GiB, BIOS with virtual CDROM
17:35:00 <sgallagh> virtio disks
17:35:01 <adamw> i'd really like to see if we can figure out what the trigger for the failure is
17:35:25 <lruzicka> sata and scsi disk?
17:35:28 <sgallagh> Maybe SCSI is indeed the problem
17:35:36 <sgallagh> lruzicka: both use the same codepath, I think
17:35:43 * sgallagh tries
17:35:44 <adamw> my working test box is SATA
17:35:49 <lruzicka> because my failed laptop is sata
17:35:51 <kparal> ahh, we're getting somewhere!
17:36:01 <adamw> <adamw> my working test box is SATA
17:36:01 <adamw> <lruzicka> because my failed laptop is sata
17:36:03 <kparal> I just created a fresh VM, used the disk from the "broken" system, and rescue works
17:36:04 <adamw> well great
17:36:23 <kparal> so, give me 5 minutes to test differences
17:36:25 <lruzicka> adamw, do you have magnetic disks or ssd?
17:36:32 <adamw> spinning rust
17:36:38 <nirik> huh... race condtion perhaps
17:36:41 <adamw> could be
17:36:43 <sgallagh> my failed sata bare metal is an SSD
17:36:47 <lruzicka> my laptop is ssd
17:36:51 <lruzicka> and it is failed
17:37:00 <sgallagh> HMM
17:37:15 <sgallagh> My VM failed when I switched the drive method from VirtIO to SCSI
17:37:23 <sgallagh> Rebooting to see if that's consistent
17:37:56 <kparal> yes, virtio disks work, scsi disks don't
17:38:01 * kparal testing sata
17:38:47 <nirik> so this code is in anaconda? or ?
17:39:17 <kparal> sata doesn't work
17:39:34 <adamw> nirik: yeah, this is all just a special mode of anaconda
17:39:46 <adamw> can someone post anaconda logs from the failure case? if they're accessible?
17:39:47 <kparal> I think it only works with virtio disks. That's why people often don't see it in VMs, but some do. And why it always fails on bare metal
17:39:48 <sgallagh> Likely blivet
17:39:57 <adamw> kparal: it doesn't always fail on bare metal
17:40:02 <adamw> kparal: it works fine on my bare metal test box
17:40:06 <adamw> and its disks are not VirtIO :P
17:40:10 <sgallagh> adamw: What interface?
17:40:11 <kparal> aha
17:40:13 <adamw> SATA
17:40:16 <kparal> interesting
17:40:19 <sgallagh> SATA or SATA in Parallel compat mode?
17:40:25 <kparal> well, I see a clear patterns in VMs, though
17:40:29 <adamw> what the heck is parallel compat mode
17:40:32 <adamw> is that a thing
17:40:38 <sgallagh> It is, yeah
17:40:44 <kparal> you can use IDE in BIOS I think? even for SATA drives
17:40:47 <adamw> *fry from futurama gif*
17:40:52 <sgallagh> If it's an early SATA system, the BIOS could report it as IDE
17:41:03 <adamw> then no, i don't think so.
17:41:18 <adamw> the motherboard is from, like, the last decade.
17:41:19 * kparal working on getting the logs
17:41:22 <sgallagh> adamw: Actually, my newer system still has it
17:41:42 <sgallagh> SATA Operation: ATA vs AHCI
17:43:05 <sgallagh> For now, I think we have enough evidence to suggest that it probably won't work on a fairly large subset of physical systems.
17:43:18 <sgallagh> So the question is "how concerned about this are we?"
17:43:19 <bcotton> agreed
17:43:42 <adamw> oh, well, if you'd just said controller mode...
17:43:43 <nirik> ♫♫ run to the hills... run for your life ♫♫
17:43:45 <adamw> i think it's on AHCI.
17:44:01 <kparal> adamw: https://kparal.fedorapeople.org/tmp/rescue-bug/
17:44:05 <adamw> thanks
17:44:05 <kparal> I'll upload it to bugzilla later
17:44:08 <nirik> It's hard to say without knowing the scope more.
17:44:38 <nirik> DEBUG:blivet:IGNORED:                                             blivet.safe_dbus.DBusCallError: Failed to call GetFirmwareInitiatorName method on /org/freedesktop/UDisks2/Manager with None arguments: GDBus.Error:org.freedesktop.UDisks2.Error.ISCSI.NoFirmware: No firmware found
17:44:41 <nirik> looks weird
17:44:52 <nirik> oh, thats iscsi. nevermind
17:45:16 <adamw> yeah, that's normal
17:45:22 <adamw> kparal: https://kparal.fedorapeople.org/tmp/rescue-bug/lvm.log is FORBIDDEN
17:45:24 <adamw> can you fix perms?
17:46:07 <kparal> adamw: should be fixed
17:46:30 <sgallagh> storage.log shows it finding the /dev/mapper/fedora_localhost stuff
17:46:57 <kparal> this is a default Workstation Live install
17:47:06 <sgallagh> Right, that's my point
17:47:16 <adamw> well
17:47:16 <sgallagh> It's detecting and starting the device mapper
17:47:18 <adamw> it shows it finding it
17:47:21 <adamw> then tearing it donw
17:47:27 <adamw> then INFO:anaconda.threading:Running Thread: AnaTaskThread-FindExistingSystemsTask-1 (140437230679808) AFTER that
17:47:31 <sgallagh> Right, it starts it in read-only mode first
17:47:44 <adamw> and then between that and INFO:anaconda.threading:Thread Done: AnaTaskThread-FindExistingSystemsTask-1 (140437230679808) , no more LVM discovery
17:47:58 <adamw> on my working case, i see LVM discovery stuff between those two FindExistingSystemsTask messages
17:49:31 <kparal> btw, my "broken" bare metal is on AHCI
17:49:34 <bcotton> okay, those of you who are not ready to vote: what more information do you need to reach a decision?
17:50:02 <sgallagh> bcotton: Ideally, confirmation that a fix is findable in a week
17:50:36 <nirik> It's still not clear to me whats affected...
17:50:39 <adamw> https://paste.centos.org/view/raw/a18337f0 is the log from my success case
17:50:43 <nirik> it seems like a lot, but not all.
17:51:02 <adamw> bcotton: i'd just really like to find out what the trigger for the bug is
17:51:20 <adamw> it's still possible it's something fairly rare but we just happen to be good at hitting it. feedback so far is from qa people, who tend to have odd systems. :P
17:51:33 <nirik> how about we keep this meeting going, and adjorn for 40min for debugging? and resume at 18:30?
17:51:39 <bcotton> i'd argue whether or not it can be fixed in a week is irrelevant to whether or not its a blocker (in part because it was submitted late)
17:51:51 <sgallagh> Also me with a fairly mainstream Dell
17:52:11 <frantisekz_> imo, if somebody is able to download non-standard iso to boot rescue mode not available on live media, that user should be able to look into common bugs and download F31 iso to proceed with rescue
17:52:30 <kparal> that's a stretch
17:52:35 <bcotton> okay, let's move along to test coverage and rc status and if we need more time, we'll put this on the shelf
17:52:42 <kparal> they will be able, but they won't know about it
17:52:55 <bcotton> #topic Current status - RC
17:52:56 <frantisekz_> so they'd google and find out...
17:52:56 <nirik> by the same token it could be possible to fix this with a update image...
17:53:15 <bcotton> #info RC 1.3 is the current release candidate
17:53:23 <kparal> frantisekz_: unlikely they they'd find it somewhere in the top results
17:53:25 <bcotton> mboddu: how are we in terms of missing deliverables?
17:53:39 <frantisekz_> I'd say likely kparal... I think we won't agree on this
17:53:56 <mboddu> bcotton: Not bad, couple of labs, some armhfp stuff and ppc64le silverblue is missing due to a network glitch
17:54:06 <kparal> I'll try to rescue my own system, be back in a few minutes
17:54:17 <mhroncok> kparal: or not
17:54:47 <bcotton> mboddu: did the Comp Neuro lab make it? we're including it in release announcements
17:55:22 <mboddu> bcotton: Yes, its in there
17:55:26 <bcotton> hooray!
17:55:41 <mboddu> bcotton: https://pagure.io/releng/failed-composes/issue/1300 - failed images list
17:55:51 <bcotton> #info A couple of labs, some armhfp stuff and ppc64le silverblue is missing
17:56:00 <bcotton> #info Comp Neuro Lab is in the RC
17:56:13 <bcotton> #link https://pagure.io/releng/failed-composes/issue/1300
17:56:19 <bcotton> any other questions on the RC?
17:56:48 <nirik> does it bring joy?
17:56:58 <adamw> i guess we can note it still doesn't have IoT in it
17:57:03 <mboddu> nirik: Nope, it brings MINTY FRESH :)
17:57:05 <adamw> the whole "so, IoT, huh" thing was never really...dealt with
17:57:42 <bcotton> #info IoT is still not in the main compose
17:59:13 <bcotton> okay, then let's move on to test matricies
17:59:28 <bcotton> #topic Current status - test matricies
17:59:30 <bcotton> #link https://fedoraproject.org/wiki/Category:Fedora_32_Test_Results
17:59:42 <lruzicka> I tried to rescue my main work laptop, T580 and it was possible.
17:59:47 <lruzicka> UEFI mode
18:00:06 <kparal> I could rescue my UEFI SATA desktop and my UEFI NVME encrypted laptop
18:00:18 <frantisekz_> lruzicka with or without CSM? I think that can make a difference?
18:00:24 <bcotton> #link https://fedoraproject.org/wiki/Test_Results:Fedora_32_RC_1.3_Summary
18:00:28 <kparal> no CSM in my case
18:00:50 <kparal> sorry, I'm off topic it seems
18:01:12 <kparal> I don't know what the decision was, my logs are cut
18:01:45 <mboddu> kparal: Postponed and going through test matrices and stuff
18:01:47 <bcotton> kparal: no decision, we moved on so y'all can continue testing
18:01:50 <kparal> thanks
18:02:00 <bcotton> i see lots of green checkmarks
18:02:14 <adamw> coverage looks pretty great to me
18:02:22 <bcotton> i see xen got tested, even :-)
18:02:23 <frantisekz_> even the mac os x is green, which I've been fighting with for like... 3 hours
18:02:24 <lruzicka> frantisekz_, what is a CSM
18:02:40 <kparal> we're big fans of the green color
18:02:41 <frantisekz_> compatibility mode for UEFI boot... roughly
18:02:46 <bcotton> kparal++
18:02:52 <lruzicka> frantisekz_, how do I know?
18:02:53 <frantisekz_> you can switch it on/off in uefi lruzicka
18:03:01 <lruzicka> frantisekz_, let me look
18:03:05 <jlanda> A big headache nowadays lruzicka
18:03:09 <bcotton> #info "Covereage looks pretty great" - adamw
18:03:12 <bcotton> #undo
18:03:13 <zodbot> Removing item from minutes: INFO by bcotton at 18:03:09 : "Covereage looks pretty great" - adamw
18:03:17 <bcotton> #info "Coverage looks pretty great" - adamw
18:03:59 <bcotton> rescue mode tests are green so clearly this bug we've been looking at doesn't exist ;-)
18:04:10 <lruzicka> so, my laptop is CSM support: Yes
18:04:12 <frantisekz_> disabling csm should prohibit you from booting in bios mode... that's good indication you have it either enabled/disabled
18:04:18 <frantisekz_> try to disable it lruzicka?
18:04:22 <bcotton> any other questions or comments about the test coverage?
18:04:42 <lruzicka> My boot is setup as UEFI first and I seeing the UEFI style screens, bot bios, so I am not in bios mode.
18:04:58 <lruzicka> mind my typos, sorry
18:05:05 <frantisekz_> you're in uefi with csm, can you try uefi without csm?
18:05:34 <bcotton> i'm going to take that as a no
18:06:26 <lruzicka> I am trying with UEFI without CSM, give me a minute
18:06:36 <bcotton> are we ready to tackle the rescue bug or should we stand in recess until 1830 UTC?
18:06:47 <adamw> still debugging a bit
18:06:56 <adamw> we've found an interesting diff between the failure and success cases...
18:06:59 <bcotton> #topic Recess
18:07:09 <bcotton> #undo
18:07:09 <zodbot> Removing item from minutes: <MeetBot.items.Topic object at 0x7fdff4376dd0>
18:07:13 <bcotton> #topic Recess until 1830 UTC
18:07:36 <bcotton> #info We are in recess until 1830 UTC to allow for further debugging of the rescue bug
18:09:26 <lruzicka> frantisekz_, no change .. same behaviour
18:09:38 <frantisekz_> okay, good to know, thanks!
18:14:22 <mboddu> .time
18:14:22 <zodbot> mboddu: 06:14 PM, April 16, 2020
18:30:08 <bcotton> #topic (1824418) Rescue Mode doesn't recognize LVM -Partitions
18:30:09 <bcotton> #link https://bugzilla.redhat.com/show_bug.cgi?id=1824418
18:30:11 <bcotton> #info Proposed Blocker, anaconda, NEW
18:30:17 <bcotton> welcome back, everyone
18:30:52 <bcotton> so we've reached the point where we need to decide if this is a blocker or not
18:31:03 <bcotton> any updates from the last 20 minutes of testing?
18:31:22 <kparal> we found out that it also affects the installer when doing custom/blivet partitioning
18:31:31 <kparal> i.e. it violates another criterion, let me find it
18:31:37 <bcotton> why are you trying to make me sad?
18:31:40 <kparal> here's a screenshot: https://i.imgur.com/XLZkPLW.png
18:32:12 <kparal> I guess this one:
18:32:14 <kparal> "The installer must be able to create and install to any workable partition layout using any file system and/or container format combination offered in a default installer configuration. "
18:32:18 <kparal> https://fedoraproject.org/wiki/Fedora_32_Final_Release_Criteria#Disk_layouts
18:32:33 <kparal> but currently it can't use LVMs (on certain setups)
18:32:38 <kparal> existing LVMs
18:32:56 <kparal> adamw: is that the right criterion?
18:33:14 <adamw> er
18:33:16 <adamw> i think there's a better one
18:33:47 <kparal> this one? https://fedoraproject.org/wiki/Fedora_32_Beta_Release_Criteria#Custom_partitioning
18:33:51 <adamw> yeah
18:33:53 <kparal> When using both the installer-native and the blivet-gui-based custom partitioning flow, the installer must be able to:
18:33:53 <kparal> Correctly interpret, and modify as described below, any disk with a valid ms-dos or gpt disk label and partition table containing ext4 partitions, LVM and/or btrfs volumes, and/or software RAID arrays at RAID levels 0, 1 and 5 containing ext4 partitions
18:33:58 <kparal> etc
18:34:06 <adamw> "Correctly interpret, and modify as described below, any disk with a valid ms-dos or gpt disk label and partition table containing ext4 partitions, LVM and/or btrfs volumes, and/or software RAID arrays at RAID levels 0, 1 and 5 containing ext4 partitions"..."Remove existing storage volumes"..."Assign mount points to existing storage volumes"
18:34:17 <bcotton> okay, so this seems like it's definitely a blocker
18:34:42 * nirik nods sadly
18:34:46 <adamw> yeah, i'm more blocker on this than i was before
18:34:54 <mboddu> Yeah, Seems like a blocker
18:35:14 <adamw> we are still trying to work out the exact cause, but blivet is definitely having trouble interpreting perfectly valid existing LVM setups and it's violating criteria
18:35:17 <sgallagh> I agree, it's a blocker.
18:35:23 <sgallagh> But... a *late* blocker?
18:35:42 <adamw> i think i'm still +1 for that
18:35:54 <adamw> i.e., i don't think we should waive it
18:36:12 * mboddu agrees with adamw
18:36:19 <bcotton> proposed #agreed 1824418 - AcceptedBlocker (Final) - This violates both the custom partitioning and rescue mode criteria
18:36:21 * nirik is on the fence...
18:36:22 <sgallagh> Yeah, I think I agree. If it was *only* rescue mode...
18:36:42 <adamw> ack
18:36:43 <kparal> ack
18:36:46 <nirik> ack
18:36:48 <mboddu> ack
18:36:51 <frantisekz_> ack
18:36:57 <bcotton> #agreed 1824418 - AcceptedBlocker (Final) - This violates both the custom partitioning and rescue mode criteria
18:36:58 <adamw> we did say when we added the 'late blocker' thing we only intended to use it rarely
18:37:01 * nirik wonders... does that make it a double blocker? :)
18:37:09 <adamw> in really clear situations where the bug was something not *really* too bad
18:37:18 <adamw> to me, 'oh noes we forgot the default background' is a reasonable case of that
18:37:23 <adamw> 'the installer can't read LVM' is...not
18:37:31 <nirik> yeah
18:37:36 <bcotton> for the rescue mode, i'm totally on board with the late blocker exception, but with the custom partioning....ehhhhhh
18:37:37 <sgallagh> yeah
18:37:47 <mboddu> I would consider the backgrounds thing a late blocker since it wont cause any functionality issues
18:37:57 <mboddu> But this is, definitely blocker for me
18:38:16 <mhroncok> I agree with that
18:38:30 <sgallagh> (sidenote: Can we leave the mention of the wallpaper out of the delay announcement? That's going to be embarrassing on the news sites...)
18:38:30 <bcotton> let me put it this way: does anyone want to make a case that the anaconda blocker should be waived
18:39:03 <bcotton> sgallagh: i'm just going to say that the number of blocker bugs is > 0
18:39:22 <frantisekz_> I'll stand for mine... "if somebody is able to download non-standard iso to boot rescue mode not available on live media, that user should be able to look into common bugs and download F31 iso to proceed with rescue" (reposting for the record)
18:40:00 <kparal> frantisekz_: this is no longer just about rescue
18:40:07 <nirik> frantisekz_: but also they can't re-install over an existing install... thats larger in scope
18:40:16 <frantisekz_> indeed it is kparal
18:40:25 <sgallagh> nirik: Well, not with the custom storage GUI at least
18:40:26 <frantisekz_> yep
18:40:33 <frantisekz_> acked blocker, don't worry guys :)
18:40:33 <kparal> frantisekz_: I don't follow
18:40:46 <kparal> custom/blivet partitioning is broken
18:40:49 <kparal> not just rescue
18:40:50 <frantisekz_> also, need to run, so see you... next week I guess
18:40:57 <nirik> sure, but that might be a pretty common case... 'wipe all my old install, but leave /home and /data alone'
18:41:08 <lruzicka> .hello2
18:41:09 <zodbot> lruzicka: lruzicka 'Lukáš Růžička' <lruzicka@redhat.com>
18:41:18 <sgallagh> I don't disagree. Just noting the scope was smaller than it sounds at first
18:41:29 <lruzicka> Sorry about that, but got into relabeling issues when trying to rescue my home equipment
18:41:33 <lruzicka> where are we?
18:41:46 <kparal> lruzicka: still broken
18:42:02 <sgallagh> We've learned it's worse than we thought
18:42:05 <kparal> lruzicka: breaks custom partitioning too
18:42:08 <kparal> acked as a blocker
18:42:12 <bcotton> okay, well i'm not hearing anyone say "this is fine"
18:42:14 <lruzicka> out of three tested machines at my place, one is broken, two are fine
18:42:20 <kparal> nobody but frantisekz_ wants to waive it
18:42:24 <kparal> it seems
18:42:59 <lruzicka> hmm
18:43:08 <mboddu> But frantisekz_ also ack'ed it to be a blocker
18:43:14 <mhroncok> and he needs to run
18:43:20 <mhroncok> so we can ignore him :D
18:43:26 <lruzicka> because he thought that we might waive it
18:43:44 <bcotton> okay, i guess we should make this formal
18:44:08 <bcotton> #topic Go/No-Go decision
18:44:12 <bcotton> I will poll each team. Please reply “go” or “no-go”
18:44:16 <bcotton> FESCo?
18:44:17 <mboddu> No-Go it is :(
18:44:20 <sgallagh> No-Go
18:44:25 <mhroncok> No-Go
18:44:28 <bcotton> #info FESCo is no-go
18:44:31 * mboddu is not FESCo
18:44:32 <bcotton> Releng?
18:44:36 <mboddu> No-Go
18:44:43 <bcotton> #info releng is no-go
18:44:47 <bcotton> QA?
18:44:51 <bcotton> #info QA is no-go
18:44:53 <bcotton> ;-)
18:45:03 <mhroncok> everybody is no go
18:45:07 * kparal nods
18:45:08 <bcotton> #agreed Fedora 32 Final is NO-GO
18:45:10 <bcotton> #info The next F32 Final Go/No-Go meeting will be Thursday, 2020-04-23 at 1700 UTC
18:45:10 <pbokoc> it's a no-go party!
18:45:19 <bcotton> #action bcotton to announce decision
18:45:24 <mboddu> pbokoc: Can we attend it virtually? :P
18:45:28 <bcotton> #topic Open floor
18:45:30 <bcotton> Anything else we need to discuss before closing?
18:45:30 <lruzicka> which means, we are not going to waive neither of the blockers, right?
18:45:41 <nirik> we already waved the first one I thought?
18:45:55 <mhroncok> not eally
18:45:57 <lruzicka> so we should unwaive it, but afaik we wanted to wait
18:45:58 <nirik> but I guess not.
18:46:03 <sgallagh> I think it'll be moot, since we are slipping anyway
18:46:04 <nirik> so sure, just keep it
18:46:05 <bcotton> nirik: we punted to see if the anaconda bug was a thing
18:46:09 <sgallagh> the +1 FE is going to handle it.
18:46:12 <adamw> sorry i disappeared, still debugging
18:46:26 <mboddu> sgallagh: But we considered it as a blocker already
18:46:29 <adamw> i'd go with mboddu's approach here
18:46:32 <adamw> it is accepted as a blocker
18:46:32 <nirik> sgallagh: nope... it's still a blocker .
18:46:36 <lruzicka> adamw, its a no go in your absence
18:46:38 <sgallagh> My mistake
18:46:39 <adamw> and we do not need to consider waiving it any more since we're slipping
18:46:45 <adamw> so we just treat it as a regular accepted blocker
18:46:54 <nirik> so the magic waive rule is still unused. ;)
18:46:58 <lruzicka> adamw, what about the background blocker?
18:46:59 <sgallagh> adamw: And consider waiving it next week if Luya forgets to push it? :)
18:47:03 <adamw> lruzicka: i'm tlaking about that one.
18:47:06 <lruzicka> ok
18:47:25 <adamw> since we're slipping for the blivet bug, we don't actually need to consider applying the late blocker policy to the backgrounds bug.
18:47:35 <bcotton> the release readiness meeting is in ~13 minutes in this very channel
18:47:51 <nirik> exciting!
18:47:51 <mhroncok> bcotton: nothing is ready
18:48:01 <mhroncok> bcotton: cancel the meeting :D
18:48:39 <bcotton> thanks everyone. we'll see you next week and do our best not to double the criteria violation of bz 1824418 again :-D
18:48:50 <bcotton> #endmeeting