16:18:20 #startmeeting F33-blocker-review 16:18:20 Meeting started Mon Jul 20 16:18:20 2020 UTC. 16:18:20 This meeting is logged and archived in a public location. 16:18:20 The chair is adamw. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:18:20 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:18:20 The meeting name has been set to 'f33-blocker-review' 16:18:20 #meetingname F33-blocker-review 16:18:20 #topic Roll Call 16:18:20 The meeting name has been set to 'f33-blocker-review' 16:18:27 .hello ngompa 16:18:27 .hello2 16:18:27 King_InuYasha: ngompa 'Neal Gompa' 16:18:30 .hello2 16:18:31 bcotton: bcotton 'Ben Cotton' 16:18:33 #chair cmurf coremodule 16:18:33 Current chairs: adamw cmurf coremodule 16:18:33 pwhalen: pwhalen 'Paul Whalen' 16:19:15 impending boilerplate alert 16:19:22 #topic Introduction 16:19:23 Why are we here? 16:19:23 #info Our purpose in this meeting is to review proposed blocker and nice-to-have bugs and decide whether to accept them, and to monitor the progress of fixing existing accepted blocker and nice-to-have bugs. 16:19:23 #info We'll be following the process outlined at: 16:19:24 #link https://fedoraproject.org/wiki/QA:SOP_Blocker_Bug_Meeting 16:19:24 #info The bugs up for review today are available at: 16:19:26 #link http://qa.fedoraproject.org/blockerbugs/current 16:19:28 #info The criteria for release blocking bugs can be found at: 16:19:30 #link https://fedoraproject.org/wiki/Basic_Release_Criteria 16:19:32 #link https://fedoraproject.org/wiki/Fedora_33_Beta_Release_Criteria 16:19:34 #link https://fedoraproject.org/wiki/Fedora_33_Final_Release_Criteria 16:19:44 #info for Beta, we have: 16:19:44 #info 5 Proposed Blockers 16:19:49 #info 1 Proposed Freeze Exceptions 16:19:58 #info for Final, we have: 16:20:07 #info 2 Proposed Blockers 16:20:17 #topic Proposed Beta blockers 16:20:30 oh, who wants to secretarialize? 16:20:41 i'll oblige that one 16:20:54 #info coremodule will secretarialize 16:21:04 #topic (1856496) Anaconda cannot reclaim space from swap partition 16:21:05 #link https://bugzilla.redhat.com/show_bug.cgi?id=1856496 16:21:05 #info Proposed Blocker, anaconda, NEW 16:23:08 this does seem covered by Beta requirement "When using the guided partitioning flow, the installer must be able to: ... Remove existing storage volumes to free up space, at the user's direction" 16:23:09 i haven't had a chance to reproduce, but the reporter/tester did a good job i think 16:23:17 so it's convincing 16:23:18 i haven't either 16:23:29 +1 blocker 16:23:39 it'd be good if someone could, just to confirm there's no other unexpected user-specific factor here 16:23:43 yeah 16:23:50 but as described probably +1 for me 16:23:52 +1 blocker as I see it 16:23:52 +1 blocker 16:23:59 +1 blocker 16:24:08 +1 16:25:25 proposed #agreed 1856496 - AcceptedBlocker (Beta) - it would be good to have independent confirmation of this, but as described we accept it as a blocker as a violation of Beta requirement "When using the guided partitioning flow, the installer must be able to: ... Remove existing storage volumes to free up space, at the user's direction" 16:25:44 ack 16:25:45 ack 16:25:47 ack 16:25:48 ack 16:26:01 #agreed 1856496 - AcceptedBlocker (Beta) - it would be good to have independent confirmation of this, but as described we accept it as a blocker as a violation of Beta requirement "When using the guided partitioning flow, the installer must be able to: ... Remove existing storage volumes to free up space, at the user's direction" 16:26:09 #topic (1855174) btrfs installation crashes with multiple disks, when one of them is too small for the OS 16:26:09 #link https://bugzilla.redhat.com/show_bug.cgi?id=1855174 16:26:09 #info Proposed Blocker, btrfs-progs, NEW 16:26:11 the first blocker of F33, mah feels 16:26:23 i think we may have had an automatic blocker before 16:26:38 good, then i feel nothing 16:26:40 this definitely seems to be an auto-blocker 16:26:42 heh 16:26:48 coremodule, yeah, the eels 16:26:48 the reason this bug happens is kparal 16:26:50 haha 16:26:50 * adamw has acquired an extremely large furry wrist rest (not called bach), please excuse any typing slowness... 16:27:08 kparal came up with the perfect test, two small devices, bizarrely uneveningly sized 16:27:16 classic kparal 16:27:19 it's an awesome case too :D 16:27:34 and it exposes a long standing bad mkfs.btrfs default of using data raid0 in the multiple device case 16:27:40 so it's gonna get fixed :P 16:27:47 https://github.com/kdave/btrfs-progs/issues/270 16:28:56 for Kamil his surname is his omen 16:29:06 unevenly 16:29:16 uneveningly is not a word haha 16:29:17 so it must be the way it is 16:29:31 +1 blocker 16:29:32 seems like +1 blocker to me, anyway, under "When using the guided partitioning flow, the installer must be able to: ... Complete an installation using any combination of disk configuration options it allows the user to select" 16:29:45 and it can't so it's a blocker, +1 16:29:50 ok it's slightly 'conditional' in that you need a fairly specific set of disks to trigger it, but i'm still +1 16:30:07 +1 blocker 16:30:09 likewise, +1 16:30:10 yes, +1 blocker. If one disk is enough to host the system, the size of the second disk is irrelevant 16:30:19 +1 blocker 16:30:20 +1 blocker 16:30:21 it is conditional, but lvm+ext4 default in the same situation succeeds 16:30:35 and the change to use data single by btrfs makes it behave that way too 16:30:51 proposed #agreed 1855174 - AcceptedBlocker (Beta) - accepted as a violation of "When using the guided partitioning flow, the installer must be able to: ... Complete an installation using any combination of disk configuration options it allows the user to select" 16:31:09 ack 16:31:11 ack 16:31:17 ack 16:31:18 ack 16:31:24 ack 16:31:30 but your data on the surviving drive can be saved, unlike the lvm_ext4 case (esoteroic info) 16:31:47 cmurf: doesn't matter if it can't install so that you can _have_ data to save 16:32:04 #agreed 1855174 - AcceptedBlocker (Beta) - accepted as a violation of "When using the guided partitioning flow, the installer must be able to: ... Complete an installation using any combination of disk configuration options it allows the user to select" 16:32:13 #topic (1830343) Network manager started stuck when I tries connecting to VPN (openconnect) after upgrade gnome-shell to 3.37.1-1.fc33 version 16:32:14 #link https://bugzilla.redhat.com/show_bug.cgi?id=1830343 16:32:14 #info Proposed Blocker, gnome-shell, NEW 16:32:20 yeah i mean once it's fixed 16:33:00 so this is basically 'networkmanager-openconnect is broken', aiui 16:33:11 it's been hanging around for a while 16:34:04 yikes 16:34:12 it does seem pretty blockery 16:34:54 I'd say so... but do we have a good way to test these things to verify fixes? 16:35:06 i can think of workarounds - connecting from console with nmcli for e.g. 16:35:09 now that's a good question 16:35:16 we don't have specific VPN requirements 16:35:30 so this becomes a sort of conditional violation of other criteria that require VPN connection, i guess... 16:35:33 soo there's other issues here too, including if you have a VPN profile that disables ipv6, NetworkManager ignores it, ipv6 is still enabled when your VPN is enabled 16:36:15 i think i'd struggle to +1 this under the criteria as they stand, but if someone proposed VPN criteria that'd be an interesting discussion... 16:36:15 I *personally* would like to block on this (especially as $DAYJOB is switching to VPN tech that requires this), but I don't know how we'd validate this reasonably 16:36:24 do we have any idea how common openconnect is? 16:36:27 very 16:36:56 it's the only way to use Cisco AnyConnect and Palo Alto GlobalProtect based corporate VPNs 16:37:08 yeah a clear criterion for VPN is reasonable: also rolls in related systemd-resolved change 16:37:13 $DAYJOB is moving from the former to the latter, for example 16:37:42 I do not know if there's an open source server side thing that can pretend to be an AC or GP VPN gateway, though 16:37:49 for testing and such 16:38:17 testing part is a hurdle but maybe not insurmountable 16:38:22 adamw: I think I would at least give it a freeze exception 16:38:49 King_InuYasha: i don't tend to worry about that till we're near a freeze 16:38:49 ah 16:38:50 month + 5 days away 16:38:54 :D 16:39:04 also, btw, seems this might not be as simple as 'openconnect broken' 16:39:19 it might be broken only in a certain configuration 16:39:20 the ticket mentions 2FA causing it to get stuck 16:39:29 per https://bugzilla.redhat.com/show_bug.cgi?id=1830343#c4 16:39:37 yeah, iiuc, the expected workflow is you send username + password 16:39:42 and the remote end sort of challenges back for a 2FA code 16:39:51 it sounds like GNOME is dropping the ball at that point 16:40:02 so if your openconnect VPN isn't configured that way, it might be OK 16:40:14 are the logs sufficiently revealing? 16:40:33 so mine would be screwed then, since it does a 2FA flow 16:40:38 though it's weird and forces a browser prompt 16:40:48 there could be unexpected interaction with systemd-resolved 16:40:51 cmurf: the one from #c1? i don't think so, that reads like a 'warning you are doing something bad but it'll keep working for now' to me 16:41:02 uh hmmm 16:41:11 could ask for full logs i guess 16:41:14 yeah 16:41:27 cmurf: i think this was filed before systemd-resolved change? 16:41:34 2020-05-01 16:41:37 I'm not confident that I could trigger this bug, given that the 2FA method the reporter is using is different from mine 16:41:40 oh well that could fix it too for that matter 16:41:54 King_InuYasha: wouldn't hurt to try i gues 16:41:55 s 16:42:01 yeah punt 16:42:05 adamw: I can and will certainly try 16:42:38 see if it's still a problem with current Rawhide, and if so please attach 'journalctl -b -o short-monotonic' 16:43:01 needinfo 16:44:42 i mean, if we punt, what are we punting *for*? 16:44:56 what more information are we expecting to make a decision? 16:45:00 i like to have that clear when we punt 16:45:18 logs from the journal specifically about what's happening with nm, nm-openconnect, and openconnect itself 16:45:50 it tends to be pretty verbose in the journal iirc 16:45:52 is this still reproducible since systemd-resolved landed, and if so please attach logs 16:46:18 did resolved land yet? 16:46:30 it could be better or worse or just different 16:47:03 yes it did land 16:47:10 running on rawhide now 16:47:20 do we have test media to link to the poster to install and try with? 16:48:12 yeah what's the current recommended rawhide? 16:49:03 20200719 worked for me (except with io='io_uring' enabled in qemu) both VM and baremetal 16:50:05 King_InuYasha: and when we get the logs is it going to change our opinion on whether it's a blocker or not? as opposed to helping us fix the bug? 16:50:40 good point 16:51:47 back to the 'we probably need a VPN criterion' discussion 16:52:35 i'm fine punting to discuss a criterion 16:53:51 arguably we could stretch the Final criterion "All applications that can be launched using the standard graphical mechanism after a default installation of Fedora Workstation on the x86_64 architecture must start successfully and withstand a basic functionality test. " 16:54:21 but i think it'd be better to have an explicit vpn criterion (particularly with an eye toward what is out of scope, because this could get edge-casey quickly) 16:54:34 bcotton, then we arrive to "what is the basic functionality" 16:55:06 yeah 16:55:22 lruzicka: yeah, that's the stretch :-) 16:55:40 is it basic when VPN companies who say "we don't collect logs ever ever ever" get caught collecting logs in a huge leak of their not ever collected logs 16:55:59 haha, no, because that's not in Fedora :-) 16:56:13 :D 16:56:15 i know 16:56:22 ok, this is taking too long :) 16:56:22 but it's complicated is the point, not basic 16:57:06 proposed #agreed 1830343 - punt (delay decision) - we see this as a bad problem and on its face potentially a blocker, but existing criteria don't really cover it, and we're not sure exactly how broken it is yet. punting for more research on how common the bug is, and also a potential proposal of VPN-specific release criteria 16:57:14 ack 16:57:17 ack 16:57:18 ack 16:57:21 ack 16:57:24 for that matter, we don't really have *network* criteria, that might be good to write out... 16:57:24 ack 16:57:29 #agreed 1830343 - punt (delay decision) - we see this as a bad problem and on its face potentially a blocker, but existing criteria don't really cover it, and we're not sure exactly how broken it is yet. punting for more research on how common the bug is, and also a potential proposal of VPN-specific release criteria 16:57:37 #topic (1856514) In default boot with 2GB of RAM or less, network install images crash with "loading repo 'rawhide' failure: write_ext(0) has failed: -1" since systemd-246~rc1-1.fc33 16:57:37 #link https://bugzilla.redhat.com/show_bug.cgi?id=1856514 16:57:37 #info Proposed Blocker, systemd, NEW 16:58:00 so when i proposed this as a blocker we hadn't figured out the "only with 2GB RAM or less" constraint so it was more clearly a blocker 16:58:10 but even with that i think it's a contender 16:58:16 install has worked with 2GB of RAM for a loooong time 16:58:30 most of my Fedora VMs are 2GB of RAM 16:58:32 does it happen on server, too? 16:58:35 I'd be pretty sad if this doesn't work 16:58:39 and the release notes state the minimum as 1GB 16:58:40 https://docs.fedoraproject.org/en-US/fedora/rawhide/release-notes/welcome/Hardware_Overview/ 16:58:45 lruzicka: it'll happen with any netinst, i think 16:58:55 it doesn't happen with Server DVD because it doesn't have to populate remote repos there 16:59:00 here, it is then ... +1 blocker 16:59:01 I'd solidly put it in the blocker camp 16:59:03 +1 16:59:09 +1 blocker 16:59:31 +1 blocker 16:59:41 yeah, I think I'm definitely +1 blocker 16:59:44 there are other systemd-246 blockery realm bugs 16:59:54 +1 blocker 17:00:32 on arm 1GB hasnt worked in a long time, we need to nfs mount the squashfs to do network installs on systems with 1GB . Now that fails too 17:01:32 what's using all that memory? 17:01:40 separate discussion i guess... 17:01:55 libsolv cache generation in RAM 17:02:35 oh 17:03:19 i could see us waiving this in future if someone says 'welp there's nothing we can do' 17:03:21 but for now i guess +1 is good 17:03:44 adamw: thats what happened when I filed the issue on 1GB failing 17:04:10 thankfully the nfs mount work around worked.. until now 17:04:15 proposed #agreed 1856514 - AcceptedBlocker (Beta) - this is accepted as a violation of Basic criterion "The installer must run when launched normally from the release-blocking images" when booting release-blocking netinst images with 2GB of RAM or less. We note that 2GB is a fairly common config for VMs, and the official docs state 1GB is the minimum requirement 17:04:29 ack 17:04:33 ack 17:04:37 ack 17:04:41 ack 17:05:17 ack 17:05:19 ack 17:05:54 i think there's some sort of leak or other problem because not only 2G should be enough, all these environments now have swaponzram 17:06:06 where before that didn't get activated in every case until anaconda launched 17:06:13 well, swaponzram just landed, right? 17:06:15 i'm gonna look at it 17:06:16 yes 17:06:27 the @core change hasn't 17:06:48 but it's on most all other media now 17:07:12 #agreed 1856514 - AcceptedBlocker (Beta) - this is accepted as a violation of Basic criterion "The installer must run when launched normally from the release-blocking images" when booting release-blocking netinst images with 2GB of RAM or less. We note that 2GB is a fairly common config for VMs, and the official docs state 1GB is the minimum requirement 17:08:33 oic what's up 17:08:36 cgroups related limits 17:08:52 mayyybe what's needed is a more aggressive zram value for he install environments... 17:09:15 go with 100% 17:09:39 anyhoo 17:09:43 continuing on! 17:09:48 #topic (1857043) FreeIPA server deployment fails in Fedora-Rawhide-20200714.n.0 due to pki-tomcat failing to run with "java.lang.ClassNotFoundException: org.apache.tomcat.util.modeler.modules.MbeansDescriptorsIntrospectionSource" 17:09:48 #link https://bugzilla.redhat.com/show_bug.cgi?id=1857043 17:09:48 #info Proposed Blocker, tomcat, NEW 17:09:53 this one's pretty slam dunk, i think 17:10:01 freeipa deployment don't work, it's required to work 17:10:02 +1 17:10:06 +1 17:10:08 haha 17:10:16 +1 17:10:20 +1 17:10:29 +1 17:10:40 (I like freeipa, so I especially want it to work!) 17:12:28 +1 17:13:49 proposed #agreed 1857043 - AcceptedBlocker (Beta) - accepted as a clear violation of Basic criterion "It must be possible to configure a Fedora Server system installed according to the above criteria as a FreeIPA domain controller, using the official deployment tools provided in the distribution FreeIPA packages." 17:14:14 ack 17:14:19 ack 17:14:21 ack 17:14:55 ack 17:15:00 #agreed 1857043 - AcceptedBlocker (Beta) - accepted as a clear violation of Basic criterion "It must be possible to configure a Fedora Server system installed according to the above criteria as a FreeIPA domain controller, using the official deployment tools provided in the distribution FreeIPA packages." 17:15:34 #info that's all the proposed Beta blockers, and the only proposed Beta FE is the same openconnect bug we punted, so let's move on to... 17:15:37 #topic Proposed Final blockers 17:15:43 #topic (1855292) btrfs is not resizable in anaconda 17:15:43 #link https://bugzilla.redhat.com/show_bug.cgi?id=1855292 17:15:43 #info Proposed Blocker, anaconda, NEW 17:15:54 oh dear 17:16:46 I'm conflicted here 17:17:10 on one hand, btrfs *does* support shrink, but I don't know if we can get this implemented and exposed in Anaconda right now 17:18:02 i'm -1 17:18:09 the criterion means "if a mechanism is there it must work" 17:18:13 it does not require that any mechanisms be there 17:18:28 hmm that's fair 17:18:46 we don't have a mechanism supported in anaconda right now, so I would go with -1 too 17:18:58 -1 for the same reason adam said (but it'd be nice if this were available) 17:19:11 i mean, if fesco wants to require that we grow btrfs resizing as a condition of the 'btrfs by default' change that'd be one conversation 17:19:26 but to me it's clear the criteria don't flat out require us to be able to resize all fs types 17:19:34 yeah 17:19:37 also i think we can't shrink xfs, right? and that's already the server default 17:19:42 we already have this with LVM+XFS 17:19:46 yeah, no shrinking there 17:19:51 so it's not like we don't have precedent for not being able to shrink our own default fses :) 17:19:58 wow, that was a lot of negatives! 17:20:00 I am not sure what this means. Is it that a previously created btrfs partition cannot be shrunk in anaconda to save some space for another installation? 17:20:04 lruzicka: yes 17:20:19 lruzicka: it's just not a thing blivet currently supports 17:20:27 so the installer doesn't let you try it 17:20:35 yep 17:20:42 but I would be able to shrink ext4, right? 17:20:45 yes 17:20:55 btrfs itself supports shrinking the filesystem, but blivet does not expose this capability 17:21:01 I somehow think this is a step down 17:21:48 but if we accept the this cost to introduce btrfs over ext4, then I agree, we do not push it now 17:21:56 -1 blocker, (+1 future enhancement ) 17:21:57 I mean creation of such functionality 17:22:05 it's probably worth a conversation to anaconda on the difficulty of fixing this as an enhancement 17:22:26 personally, I'd like it available in the safe subsets where anaconda doesn't have to make it complicated to shrink 17:22:50 (e.g. weird multi-disk setups) 17:23:06 err e.g. only single disk setups, and not weird multi-disk ones 17:23:45 -1 blocker 17:24:01 -1 blocker 17:24:04 nice to have but we don't have it for the current default (lvm+ext4) 17:24:17 cmurf: we could record this as an RFE, right? 17:24:30 -1 blocker for me too 17:24:34 btrfs can do it, it's just getting udisks and libblockdev sorted out, and constrained to single device probably 17:24:48 yeah i think this might actually be a dup 17:25:06 i'm pretty sure i have an old RFE for this very think lurking in RHBZ 17:25:09 proposed #agreed 1855292 - RejectedBlocker (Final) - it would be nice to have this feature, but we agree the criteria are not intended to require it, and we have substantial precedent for not being able to shrink our own default filesystem layouts 17:25:11 s/think/thing/ 17:25:42 ack 17:25:44 ack 17:25:59 I must go now, sorry about that. Have a nice time. 17:26:03 ack 17:26:08 lruzicka: bye 17:26:10 ack 17:26:50 fwiw i've started the conversation with anaconda+libblockdev+udisks folks to support it 17:26:54 needed in Disks anyway 17:26:57 #agreed 1855292 - RejectedBlocker (Final) - it would be nice to have this feature, but we agree the criteria are not intended to require it, and we have substantial precedent for not being able to shrink our own default filesystem layouts 17:26:59 thanks lruzicka! 17:27:22 cmurf: awesome 17:27:27 #topic (1829079) Second session fails to start correctly when user switching 17:27:28 #link https://bugzilla.redhat.com/show_bug.cgi?id=1829079 17:27:28 #info Proposed Blocker, gnome-shell, POST 17:27:35 so a PR for this showed up over the weekend, I need to test it today 17:27:46 also gparted can do it but seriously overestimates how much it can shrink (filed multiple bugs for that) 17:27:58 did we agree the new criterion yet? i forget 17:28:06 we did 17:28:11 oh we did 17:28:11 https://fedoraproject.org/wiki/Fedora_33_Final_Release_Criteria#User_switching 17:28:21 so, pretty straightforward +1 there. 17:28:37 yeah it should work 17:28:40 +1 blocker 17:29:10 +1 blocker 17:29:38 +1 blocker 17:29:40 proposed #agreed 1829079 - AcceptedBlocker (Final) - accepted as a clear violation of "User switching must work using the mechanisms offered (if any) by all release-blocking desktops in their default configuration." 17:29:42 +1 17:29:45 ack 17:29:52 ack 17:29:57 ack 17:29:57 ack 17:30:50 ack 17:31:08 ack 17:31:09 #agreed 1829079 - AcceptedBlocker (Final) - accepted as a clear violation of "User switching must work using the mechanisms offered (if any) by all release-blocking desktops in their default configuration." 17:31:12 okely dokely 17:31:16 that's all the proposals! 17:31:19 #topic Open floor 17:31:22 any other business, folks? 17:31:52 as usual, I am going to have lunch, then I'll update each bug with the secretarialization 17:31:57 so give me an hour or so 17:34:17 rgr 17:36:06 I need food too... 17:36:14 but this has gone well :) 17:36:40 * bcotton ate during the meeting 17:37:55 hmm i'm not having much luck with this systemd 2G RAM VM bug... 17:38:16 as in, it's not using more than the available swap 17:42:42 interesting, the bug report is netinstaller 17:42:55 that should need even less ram 17:45:11 not that anyone is waiting for this channel, but we can probably #endmeeting at this point 17:45:35 #endmeeting