16:02:42 #startmeeting F31-blocker-review 16:02:42 Meeting started Mon Jul 29 16:02:42 2019 UTC. 16:02:42 This meeting is logged and archived in a public location. 16:02:42 The chair is adamw. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:02:42 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:02:42 The meeting name has been set to 'f31-blocker-review' 16:02:43 #meetingname F31-blocker-review 16:02:43 #topic Roll Call 16:02:43 The meeting name has been set to 'f31-blocker-review' 16:02:46 .hello2 16:02:47 frantisekz: frantisekz 'František Zatloukal' 16:02:47 now do your hellos again :P 16:02:53 .hello2 16:02:54 bcotton: bcotton 'Ben Cotton' 16:02:57 * kparal is here 16:02:57 .hello2 16:02:58 coremodule: coremodule 'Geoffrey Marr' 16:03:02 .hello2 16:03:03 * bcotton is slightly distracted for the next hour 16:03:04 tablepc: tablepc 'Pat Kelly' 16:03:15 * coremodule is ready, willing, and able to act as secretary for the meeting. 16:04:12 thanks for that 16:04:27 #chair frantisekz lruzicka 16:04:27 Current chairs: adamw frantisekz lruzicka 16:04:41 impending boilerplate alert! 16:04:43 #topic Introduction 16:04:43 Why are we here? 16:04:43 #info Our purpose in this meeting is to review proposed blocker and nice-to-have bugs and decide whether to accept them, and to monitor the progress of fixing existing accepted blocker and nice-to-have bugs. 16:04:43 #info We'll be following the process outlined at: 16:04:45 #link https://fedoraproject.org/wiki/QA:SOP_Blocker_Bug_Meeting 16:04:45 #info The bugs up for review today are available at: 16:04:47 #link http://qa.fedoraproject.org/blockerbugs/current 16:04:49 #info The criteria for release blocking bugs can be found at: 16:04:53 #link https://fedoraproject.org/wiki/Basic_Release_Criteria 16:04:55 #link https://fedoraproject.org/wiki/Fedora_31_Beta_Release_Criteria 16:04:57 #link https://fedoraproject.org/wiki/Fedora_31_Final_Release_Criteria 16:05:03 #info for Beta, we have: 16:05:05 #info 3 Proposed Blockers 16:05:07 #info 2 Accepted Blockers 16:05:51 .hello2 16:05:52 lbrabec: lbrabec 'Lukas Brabec' 16:06:45 too late, you're not allowed 16:06:47 =) 16:07:00 .fire lbrabec 16:07:00 adamw fires lbrabec 16:07:04 that's right 16:07:13 alrighty, let's start with proposed Beta blockers 16:07:14 #topic (1731415) Network spoke doesn't work on ARM initial-setup 16:07:14 #link https://bugzilla.redhat.com/show_bug.cgi?id=1731415 16:07:14 #info Proposed Blocker, initial-setup, NEW 16:08:11 hmm, do we have a criteria for that? I am assuming you can go on with installation with skipping network spoke 16:08:21 this looks more like a problem with a specific network adapter than a problem on 'arm' exactly 16:08:38 * pwhalen is here 16:08:38 Seems straightforward if the installation can't continue, but... Can't you skip setting up network? 16:08:57 hey pwhalen 16:09:01 got any more useful info on this? 16:09:14 I dont, but I'll look now 16:10:02 * coremodule is testing this now... 16:10:29 i think frantisekz and coremodule are right that this shouldn't be a showstopper though 16:10:39 of course, if the network does not work after i-s either, *that* could be a showstopper 16:14:23 can you also do a network installation for arm devices, or is it a dd-over approach always? 16:14:26 Its going to take me a few minutes to verify, but if i-s gets stuck after selecting network, sounds like a blocker to me 16:14:35 Image is writing currently, we could return to this one if you're waiting for me... 16:14:55 kparal, you can do a network install as well 16:15:23 in that case this could violate a number of criteria 16:15:45 like "When using a release-blocking dedicated installer image, the installer must be able to use either HTTP or FTP repositories (or both) as package sources. Release-blocking network install images must default to a valid publicly-accessible package source. " 16:16:34 according to https://fedoraproject.org/wiki/Releases/30/ReleaseBlocking , no netinst image is blocking for arm 16:16:37 only the appliance images 16:16:42 (31 page doesn't exist yet) 16:16:49 booting now 16:17:27 network installs work fine on the rpi3 16:17:32 adamw: haven't thought about that, thanks 16:18:22 seems like -1 bug for me 16:19:32 do all rpi3s have the same network adapter? 16:19:36 hmmm, I too am getting flooded with kernel messages as in comment 5, but it does make it to i-s 16:20:21 coremodule, debug kernel 16:20:48 ah 16:21:18 well, it seemed to configure the network okay, i just configured a user and it is to a login prompt... 16:21:26 works fine for me, minus the slew of debug messages 16:21:36 coremodule, did you select the network spoke? 16:21:58 used a wired ethernet connection, selected to spoke, refreshed the network spoke page, then hit continue 16:22:21 okay 16:22:27 hrm, sounds ok then. I'll double check here as well 16:22:30 so this is sounding like not-a-blocker on current info 16:23:03 agreed, I'll update the bug if I can reproduce 16:23:16 based on my findings I am -1 blocker 16:23:21 -1 here too 16:23:28 proposed #agreed 1731415 - RejectedBlocker (Beta) - the network spoke not working in itself is not a blocker per the criteria (so long as user and root spokes work). testing during the meeting by multiple testers did not reproduce the bug or find any criteria-violating cases in other scenarios 16:23:38 ack 16:23:49 ack 16:24:04 ack 16:24:07 ack. do we want to give it an FE preemptively? 16:24:13 ack 16:24:15 ack 16:24:21 ack 16:25:00 bcotton: not until we know more, i think 16:25:03 #agreed 1731415 - RejectedBlocker (Beta) - the network spoke not working in itself is not a blocker per the criteria (so long as user and root spokes work). testing during the meeting by multiple testers did not reproduce the bug or find any criteria-violating cases in other scenarios 16:25:37 #topic (1733388) Circular locking often causes btrfs installs to hang with kernel-5.3.0-0.rc0.git7.1.fc31 and later 16:25:37 #link https://bugzilla.redhat.com/show_bug.cgi?id=1733388 16:25:37 #info Proposed Blocker, kernel, NEW 16:26:01 so...i'm actually not totally sure the circular locking thing is the problem here 16:26:11 but btrfs installs definitely started breaking quite often around 0720 16:26:35 there's a bunch of later call traces that show the problem, and it's already been fixed upstream 16:27:12 i matched up those call traces in the openqa log to the upstream patchset and confirmed it with upstream devs so it should land in rc2 16:27:41 well, linus merged it, so it will be in rc2 16:28:22 ah, k 16:28:27 rc2 is building right now 16:28:30 if you want I can change the title to reflect the actual problem? 16:28:34 if you like 16:28:42 so the other question here is, should btrfs block the release 16:29:20 after reading the comments, I think we should suggest to anaconda team to demote btrfs visibility in the installer 16:29:33 add a warning, hide it behind a cmdline option, something like that 16:29:47 because this will bite us again in the future 16:30:19 +1 demoting btrfs 16:30:38 that argument is logical but i think the conversation needs to happen on devel@ because it involves more than one team 16:30:45 only once btrfs is deemed stable and supported in the installer team, it should be prominently offered to users. or we need to change the criteria in some way 16:30:48 the anconda team put in a metric f ton of work in btrfs UI 16:31:01 cmurf: sure, sounds reasonable 16:31:20 definitely seems like a discussion we should have 16:31:34 it might also be complicated by the blivet-gui path... 16:31:48 btrfs is no less prominent than RAID or any other thing that could blow up on us and we don't have md experts on the kernel team either 16:31:51 there would need to be a way for anaconda to signal to blivet-gui not to display btrfs 16:32:17 i mean, quite a lot of things we could plausibly block on, that could blow up during very very early kernel release candidates are things the Fedora dev team doesn't want to touch 16:32:30 they depend on upstream taking many many bugs seriously and urgently 16:32:44 and this particular bug was fixed before I reported it by btrfs upstream 16:33:44 i think we're better off carving out some criterion exception than asking anaconda folks to change the installer 16:33:50 it should probably be a wider discussion than just btrfs 16:34:01 and that too 16:34:14 what if squashfs blew up for some reason? lol 16:34:32 and we could also let anaconda install just a single solution without any options and make everything be hidden behing cli options, just like windows. 16:34:34 there isn't really even an upstream maintainer (because it's been such a successful project) 16:34:43 * lruzicka was kidding. 16:34:57 lruzicka: well actually :P 16:35:09 this has been on my wish list since forever but i'm the crazy person 16:35:18 i'm a point and shoot installer fan 16:35:27 cmurf, :) 16:35:29 so 16:35:30 let's see 16:36:31 by current policy, this ought to be a blocker 16:36:36 but anaconda team thinks current policy is garbage 16:36:37 what to do 16:36:45 the only thing we can do, pinky: delay! 16:37:11 well you can punt and just ignore that this is strictly speaking a blocker, by obviating it as a blocker when rc2 fixes it 16:37:52 proposed #agreed 1733388 - punt (delay decision) - this seems fairly likely to be a blocker under current policy, but anaconda team believes current policy casts too wide a net. we will start a(nother) discussion of storage criteria on the mailing lists and see where that goes 16:37:57 and hope to our dear sweet and fluffy lord it's another 5 years before there's a btrfs bug that does this in an early rc kernel let alone a late one 16:38:00 policy says blocker, we should block, when fixed this will go away ... or vote on policy change 16:38:01 ack 16:38:10 ahem i will have you know my lord is not fluffy 16:38:15 ack 16:38:19 ack 16:38:21 spikey 16:38:22 ack 16:38:55 ack 16:39:09 ack 16:39:14 ack 16:39:28 btw there have been nasty storage bugs in the kernel worse than this in early rc's 16:40:29 #agreed 1733388 - punt (delay decision) - this seems fairly likely to be a blocker under current policy, but anaconda team believes current policy casts too wide a net. we will start a(nother) discussion of storage criteria on the mailing lists and see where that goes 16:40:42 #topic (1715699) LiveOS boot, journalctl is missing many early messages 16:40:42 #link https://bugzilla.redhat.com/show_bug.cgi?id=1715699 16:40:42 #info Proposed Blocker, systemd, POST 16:40:48 oh good this one again 16:41:23 so, we rejected this last time 16:41:44 what's the justification for the re-proposal, cmurf? 16:42:10 it happens out of the box by default 16:42:16 i don't have to do anything to trigger it 16:42:49 i don't think we were assuming you did 16:42:58 we just decided the problem wasn't big enough to constitute the logging system "not working" 16:42:59 I guess I need to pay more attention to the Live boot 16:43:23 at this point it becomes a very subjective discussion, i guess - what constitutes "working" 16:43:33 we could make the criteria 650 pages long and define every term like that, but let's not 16:43:45 missing 25s of early boot is not acceptable 16:43:55 i cannot get the information i need any other way 16:44:08 early debug shell is not available 16:44:22 there are all these early boot assembly things in dracut that *only* exist in the journal 16:44:45 the instant they are deleted by systemd-journald due to this bug, there is no possible way to understand what's happening in early boot 16:44:56 i've got 4-5 issues i can't understand because of this bug 16:45:18 this is just re-litigating the discussion from last week, though, that was what we discussed then 16:45:30 (or two weeks ago or whatever it was) 16:45:46 given that there's no really new information here i'm voting -1 consistent with previous decision 16:46:05 that's not logical 16:46:12 * kparal is unhappy 16:46:17 the criteria is clear, we need logging for a reason 16:46:30 having the journal intact for early boot it the MOST critical time we need it 16:46:40 for your purposes, sure 16:46:48 but not necessarily for everyone's purposes 16:47:01 that is myopic 16:47:04 especially not just when booting live images 16:47:13 it's just, like, my opinion, man. 16:47:19 the bug isn't triggered in that case 16:47:21 but the criteria are for the whole project 16:47:25 cmurf: my question is "what changed since the last vote?" i'm sympathetic to your argument, but if nothing's changed then the decision has been made, whether it was right or not :-) 16:47:36 bcotton: we can revisit the decision. that's valid. 16:47:48 but my vote is that i'm not. :P 16:47:48 what has changed is my understanding of the bug which i added to the bug 16:48:02 oh sure, but to your point, relitigating isn't necessarily very helpful 16:48:05 cmurf: perhaps you could propose a new criterion related to the ability of the system to be debugged for failures? 16:48:13 kparal: that sounds...vague 16:48:22 sure 16:48:30 perhaps we can make it better 16:48:37 this is incredible 16:49:02 how in the world can you say logging is working when 90% of startup is MISSING because systemd-journald deleted the log!? 16:49:08 that's just bizarre and not rational 16:49:15 I think that releasing a product that works but can't be inspected for failures is a very short-sighted decision 16:49:17 there is a criterion directly violated by the bug 16:49:18 cannot logs be sent elsewhere? 16:49:24 the question is of the right balance, of course 16:49:30 but such a criterion would make sense to me 16:49:42 lruzicka: forwarding does not forward everything, that is a valid problem that i've also asked systemd devs about 16:49:57 cmurf: the criterion you cite is for post-install systems. this is only Live, right? 16:50:10 LiveOS yes 16:50:17 i cited a criterion for post install? 16:50:23 then the criterion doesn't apply in the strict sense 16:50:34 "A system logging infrastructure must be available, enabled by default, and working." 16:50:38 comment 6 16:50:53 https://fedoraproject.org/wiki/Basic_Release_Criteria#System_logging 16:51:10 oh fuck 16:51:10 it's in the section "Post-install requirements" 16:51:14 yep 16:51:21 i missed 2.5 entirely 16:51:26 i went only to 2.5.4 16:51:45 well *that's* why i'm being stubborn you guys! 16:51:46 you can argue that's an oversight, though 16:51:53 no one ever said that before! haha! 16:52:00 cmurf++ 16:52:00 bcotton: Karma for chrismurphy changed to 1 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 16:52:00 i'd probably want to block on no log messages ever showing up in live boots at all 16:52:06 as written it's definitely not a blocker 16:52:10 it is available, enabled by default and working, just not for first n seconds (I am not taking any stance here, just mentioning and being a troll :) ) 16:52:14 i am just not convinced that losing the first few seconds of messages on live boots is bad enough not to ship the release 16:52:16 but this bug is fucked you have no idea how much it's pissing me off 16:52:25 frantisekz: as i said, it comes down to how you define 'working' 16:52:40 cmurf: "it really annoys me" is not a good grounds for a blocker decision, though :) 16:52:46 i can't even verify startup storage assembly *feature changes* are doing the correct thing because of this bug 16:52:53 adamw: that is not my point 16:53:32 really fucking annoying explains my attitude and obsintance, and apparently totally skipping over the post-install location of the criterion 16:53:43 it's obviously not a blocker 16:53:58 now that this important detail has been pointed out to me of course 16:54:27 now we understand each other better :) 16:54:31 exactly! 16:54:41 well, but 16:54:53 as i said, i'm not sure the criteria are right there 16:55:04 i don't know what that means 16:55:07 would we be happy if the bug was "no journal messages ever appear on live boot"? 16:55:17 gotcha 16:55:19 and just say, oh, sure, not a blocker cos the criterion only applies post-install? 16:55:30 right or even netinstalls 16:55:45 what if we had no journal for any of the installer images 16:55:55 i hadn't actually clocked the post-install thing till kparal pointed it out, that's not why i voted -1 16:56:00 But we also want LiveOS to be an advertisement of Fedora for indecisive people ... what if they realize there are no early logs and install Ubuntu instead? 16:56:06 i was assuming the criterion *did* cover the live image case 16:56:19 lruzicka: i mean...would they? 16:56:26 adamw: but you think it's OK for 90% of the journal to be missing? huh? 16:56:43 cmurf: that seems like an unfair characterization 16:56:46 it's not 90% 16:56:47 up to gdm appearing, most of the journal is deleted 16:56:57 it's 90% (or something) *at the point the system comes up* 16:57:09 after you've been running it for two hours, it's probably like 5% 16:57:15 so what's the number? is it 90% or 5%? 16:57:20 in real world, it can make e.g. GPU bugs hard to debug 16:57:25 I do not know, I am just saying that it can happen. I understand that for most people it does not matter, but apparently, it does for some, like cmurf 16:57:25 sure it can 16:57:30 i'm not debating that it's a bug we should fix 16:57:35 the question is, do we block the release on it. 16:58:07 we could, of course, be cowards and just punt it and make sure the fix lands by next week.. 16:58:19 is the fix complicated, or is it just increase some settings? 16:58:24 you cannot make it a blocker as the criterion is written 16:58:26 I guess we will keep the existing decision. It would be good to move the logging criterion to somewhere where it affects everything, though 16:58:51 but yes I definitely think this is a blocker worthy bug 16:58:55 lruzicka: just a defaults tweak essentially 16:59:34 anyone else have votes? 16:59:51 so far we have definite +1 from cmurf, -1 from me, kparal appears to be -1, not sure if i missed anyone 16:59:55 nope 16:59:57 i'm a -1 17:00:03 the criterion is the criterion 17:00:19 -1 on provision that someone pokes the fix 17:00:44 i disagree with its location in post-install but that is not a reason to subjectively ignore it and just get what i want anyway 17:01:31 cmurf: process-wise, if we all agree the criterion should apply to live environments, we can agree that the criterion should change in that way, and vote as if it had already 17:01:33 we've done that before 17:01:49 it's not like we are following our own rules every time... 17:01:51 (so long as there are enough people at the meeting and the agreement on the criteria change is strong) 17:01:55 -1 on the criteria as defined (and maybe regardless) 17:02:20 i'm a -1 based on the criterion as written and didn't realize 17:02:31 -1 17:02:32 +1 if the crition applies to netinstall and LiveOS 17:02:42 +1 on changing the criteria to cover loggin in all systems 17:03:19 okay, let me try and sum that up :P 17:04:20 proposed #agreed 1715699 - RejectedBlocker (Beta) - as the criteria are currently written, this is not a blocker as the logging requirement applies only post-install. Opinions on whether this bug should be a blocker if the criteria were to apply to live environments were split, so we may revisit the bug once more if the criteria are changed and the bug is not fixed by then 17:04:40 yeah i can live with that 17:04:45 ack 17:04:53 when are we to change the criteria? 17:04:58 implicitly, please go ahead and submit a criteria change proposal to the list if you think we should change it 17:05:05 ack 17:05:09 ack 17:05:09 ack 17:05:12 lruzicka: let's have someone send a draft to the list and we can vote on it 17:05:16 ack 17:05:22 er, discuss it ,sorry 17:05:41 anyone want an action item for that? blocker review action items are a bit academic as no-one ever follows up on them, but hey 17:05:52 ack 17:06:15 #agreed 1715699 - RejectedBlocker (Beta) - as the criteria are currently written, this is not a blocker as the logging requirement applies only post-install. Opinions on whether this bug should be a blocker if the criteria were to apply to live environments were split, so we may revisit the bug once more if the criteria are changed and the bug is not fixed by then 17:06:26 I will make a proposal for the change 17:06:35 tablepc, thanks 17:06:53 tablepc: btw, in future, can you please send proposals as plain text, not document attachments? it makes it much easier for people to see 17:07:06 you can write it in libreoffice then just copy and paste into the email, it should work fine 17:07:34 sure 17:07:40 #action tablepc to draft a criteria amendment (others are also welcome to draft their own versions if they like) 17:07:44 probably should cc devel@ in case some other parties who need early boot/startup logging stuff want to opine 17:08:05 you know, adamw, if the policies were on docs.fpo.o, proposals could be submitted as a PR 17:08:07 * bcotton ducks 17:08:34 Do I need to sign up for the @devel mail list to sent it to them? 17:08:35 bunch of NetworkManager messages are missing, dracut, udev, device discovery 17:08:38 bcotton: sure they could 17:08:46 i'm not sure i'd find that an improvement? but it's a choice! 17:08:52 :-) 17:08:55 tablepc: just sending to test@ is fine i think 17:09:10 someone else can forward to devel@ 17:09:20 * cmurf is someone else 17:09:23 i think you can actually send it to devel@ if you're not subscribed, though 17:09:29 it'll get held for moderation, but it should get approved 17:09:41 or use hyperkitty interface 17:09:44 right 17:10:10 lists.fedoraproject.org and login via fas and use the web interface to post to anylist 17:10:55 #topic Accepted Beta blockers 17:11:02 so, let's take a quick look at these, as there are only two 17:11:08 #topic (1728419) bes not installable: needs rebuild, probably update to 3.20.5 17:11:08 #link https://bugzilla.redhat.com/show_bug.cgi?id=1728419 17:11:08 #info Accepted Blocker, bes, NEW 17:11:25 this is a blocker because it's on the server DVD and packages on server DVD not being installable is against the rules 17:11:52 most obvious solution is to take it off the server DVD, that discussion appears to be in sgallagh's court atm 17:12:30 Is that still on the DVD? It should have disappeared with the changes I made on Wednesday 17:12:50 * sgallagh is not really here, just happened to hit WiFi at the wrong moment. 17:14:23 sgallagh: oh, i didn't re-check it specifically 17:14:30 sgallagh: i just figured you'd update the bug if you changed it 17:14:36 * adamw goes and looks at the test result 17:14:39 * sgallagh dropped 1GiB of content with that change 17:14:58 oh indeed it's gone now 17:14:59 I forgot to update the bug and went away to Niagara. Currently sitting in Tim Hortons. 17:15:04 the test is still failing, but now on something else 17:15:20 sgallagh: why would you do that? go to starbucks, it's better and they don't hire temporary foreign workers and pay them naff all 17:15:21 :P 17:15:47 #info this appears to be fixed now, we'll close it 17:15:48 I’ve never been to a Tim Hortons before and the person giving me a ride is meeting me here. 17:15:56 that's a good reason. 17:16:49 #topic (1727343) [abrt] PackageKit: libdnf::Repo::Impl::detachLibsolvRepo(): packagekitd killed by SIGSEGV 17:16:49 #link https://bugzilla.redhat.com/show_bug.cgi?id=1727343 17:16:49 #info Accepted Blocker, libdnf, ON_QA 17:16:58 the fix for this already hit rawhide, so we should probably drop the blocker tags 17:17:10 the bug is filed against f30, which is why it's not closed yet. 17:18:11 #info this is fixed in Rawhide, so we'll drop the blocker metadata. bug is filed against F30 so it can't be closed yet 17:18:16 #topic Open floor 17:18:20 alrighty, that's all the bugs 17:18:26 any other discussion/comedy routines? 17:19:16 What do cows like to do while drinking their chocolate milk? 17:19:42 Watch a Mooo vie 17:20:25 .fire tablepc 17:20:25 adamw fires tablepc 17:20:33 Have a Great rest of your day! 17:21:37 =) 17:22:46 * cmurf is gonna bug adamw on #anaconda in 3... 2... 1... 17:23:09 * adamw leaves 17:26:06 sounds like we're done here! 17:26:10 thanks for coming, folks 17:26:11 ooh 17:26:13 also, for the record: 17:26:22 #info coremodule secretarialized, but i forgot to mention it earlier 17:26:24 #endmeeting