17:00:22 #startmeeting F25 Beta Go/No-Go meeting 17:00:22 Meeting started Thu Oct 6 17:00:22 2016 UTC. The chair is jkurik. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:00:22 Useful Commands: #action #agreed #halp #info #idea #link #topic. 17:00:22 The meeting name has been set to 'f25_beta_go/no-go_meeting' 17:00:26 #meetingname F25-Beta-Go_No_Go-meeting 17:00:26 The meeting name has been set to 'f25-beta-go_no_go-meeting' 17:00:35 #topic Roll Call 17:00:41 .hello jkurik 17:00:42 jkurik: jkurik 'Jan Kurik' 17:00:46 .hello coremodule 17:00:47 coremodule: coremodule 'Geoffrey Marr' 17:00:48 morning 17:00:59 .hello mohanboddu 17:01:00 mboddu: mohanboddu 'Mohan Boddu' 17:01:05 .hello nmilosev 17:01:08 nmilosev: nmilosev 'Nemanja Milosevic' 17:01:36 * kparal lurks 17:01:40 Hi everybody 17:01:47 Morning! 17:02:11 hi 17:02:54 #chair dgilmore adamw nirik mboddu 17:02:54 Current chairs: adamw dgilmore jkurik mboddu nirik 17:02:59 .hello sgallagh 17:03:00 sgallagh: sgallagh 'Stephen Gallagher' 17:03:08 #chair sgallagh 17:03:08 Current chairs: adamw dgilmore jkurik mboddu nirik sgallagh 17:03:15 .hello adamwill 17:03:15 adamw: adamwill 'Adam Williamson' 17:03:43 ok, so looks like we have the minimal set of people to start 17:03:46 #topic Purpose of this meeting 17:03:54 #info Purpose of this meeting is to check whether or not F25 Beta is ready for shipment, according to the release criteria. 17:04:06 #info This is determined in a few ways: 17:04:07 #info No remaining blocker bugs 17:04:08 #info Test matrices are fully completed 17:04:10 #info Release candidate compose is available 17:04:17 #topic Current status 17:04:33 #info The F25 Beta RC is available: 17:04:35 #link https://fedorahosted.org/rel-eng/ticket/6484 17:04:36 #link http://dl.fedoraproject.org/pub/alt/stage/25_Beta-1.1/ 17:04:47 #info Test results for F25 Beta are available: 17:05:00 #link https://fedoraproject.org/wiki/Test_Results:Fedora_25_Beta_1.1_Summary 17:05:02 #link https://fedoraproject.org/wiki/Test_Results:Fedora_25_Beta_1.1_Installation 17:05:03 #link https://fedoraproject.org/wiki/Test_Results:Fedora_25_Beta_1.1_Base 17:05:05 #link https://fedoraproject.org/wiki/Test_Results:Fedora_25_Beta_1.1_Server 17:05:06 #link https://fedoraproject.org/wiki/Test_Results:Fedora_25_Beta_1.1_Cloud 17:05:08 #link https://fedoraproject.org/wiki/Test_Results:Fedora_25_Beta_1.1_Desktop 17:05:09 #link https://fedoraproject.org/wiki/Test_Results:Fedora_25_Beta_1.1_Security_Lab 17:05:25 We have 5 accepted blockers and 2 proposed blockers. I guess all of the fixes for the 5 blockers are already in the compose. Am I correct ? 17:05:39 adamw, mboddu, dgilmore ^^ ? 17:06:02 * nirik nods. should be yes. 17:06:14 ok, thanks for the confirmation 17:06:17 yes. 17:06:24 Hi 17:06:33 .fas lailah 17:06:33 Kohane|2: lailah 'Sylvia Sánchez' 17:06:47 #info We have 5 accepted blockers and 2 proposed blockers. The fixes for the 5 accepted blockers are already in the compose. 17:07:05 Let's start with Mini-blocker review 17:07:16 adamw: may I ask you please to chair the mini-blocker review ? 17:09:02 sure 17:09:54 I'm a little wary that those are all ON_QA rather than VERIFIED. 17:10:02 Is that just reporting lagging reality? 17:10:43 sgallagh: well, we haven't technically verified each one with Beta-1.1 17:10:48 though they've all been verified some other way 17:10:57 i'll go through and check them as we go along with the meeting 17:10:58 #topic (1381996) TypeError: SynchronizedABCMeta object argument after ** must be a mapping, not NoneType (Intel firmware RAID discovery fails when RAID set is 'migrating') 17:10:59 #link https://bugzilla.redhat.com/show_bug.cgi?id=1381996 17:10:59 #info Proposed Blocker, anaconda, NEW 17:11:01 ack 17:11:25 so the upshot of this one is that anaconda will choke on mdraid sets that are in a 'migrating' state 17:11:44 various things are considered to be 'migrating' states, including the initial resync of a freshly-created set 17:12:04 so if you create a brand new RAID>0 set then try and install to it immediately, anaconda will fall over 17:12:10 -1 blocker based on the weird state the array was in 17:12:12 but if you wait for the initial resync to be done it'll work OK 17:12:17 * nirik is -1 blocker for this one. Corner case. Documentable 17:12:21 dgilmore: it's not actually a weird state, it's fairly normal 17:12:30 why does it sync anything when the raid is completely empty? 17:12:34 but i'm OK with -1 for Beta. we should probably fix it for Final though, in fact I'm +1 Final blocker. 17:12:39 I agree to have this in CommonBugs, as already proposed. I am -1 to block on this 17:12:44 kparal: i dunno, something about how RAID works. but it's apparently standard. 17:12:50 adamw: well its weird to install in raid0 then reinstall after changing the raid type 17:13:01 dgilmore: that's not actually necessary, that was an early guess 17:13:08 -1 Beta blocker, +1 final blocker 17:13:09 at least I think its not a typical use case 17:13:13 dgilmore: just 'create brand new RAID-1 set, immediately try to install' is a sufficient reproducer 17:13:14 -1 17:13:21 I mean... 17:13:36 -1 Beta blocker, +1 Final blocker 17:13:41 I guess a fix for some of it would be good for final. That case at least 17:14:01 nirik: we can fix it, it's not too difficult, just requires teaching libblockdev to understand the mdadm output for migrating sets. 17:14:42 * adamw counts 17:14:45 -1 Beta Blocker 17:14:50 adamw: in some of those cases you probibly don't want to install tho? guess it can be handed in bug since you asked that very thing 17:14:55 pschindl will be available in 5 minutes, if you need any details from him 17:15:15 nirik: yeah, waiting to hear on that. but at least the initial resync i think we should just silently accept, it sounds like. 17:15:23 yep 17:15:30 kparal: i'm *pretty* sure we understand this one...next one we might need him 17:15:39 anyone opposed to Final blocker for this? 17:15:56 +1 final here. 17:15:59 I guess not. 17:16:12 Apparently.... no... 17:17:33 -1 beta, and I'm not sure about final, depends on what anaconda/dmraid people say, but anaconda could at least say "raid initialization in progress, try later" 17:17:36 proposed #agreed 1381996 - RejectedBlocker (Beta) AcceptedBlocker (Final) - this affects one fairly common case (immediate install to a freshly-created RAID>0 set), but we believe a commonbugs note is sufficient for Beta. For Final the 'initial sync' case at least should be properly handled 17:17:50 ack 17:17:52 kparal: i think everyone's pretty much on board with fixing it for Final. 17:17:56 ack 17:18:04 ack 17:18:16 ack 17:18:23 #agreed 1381996 - RejectedBlocker (Beta) AcceptedBlocker (Final) - this affects one fairly common case (immediate install to a freshly-created RAID>0 set), but we believe a commonbugs note is sufficient for Beta. For Final the 'initial sync' case at least should be properly handled 17:18:28 ack 17:18:34 #topic (1382274) gi.overrides.BlockDev.DMError: Failed to group_set 17:18:35 #link https://bugzilla.redhat.com/show_bug.cgi?id=1382274 17:18:35 #info Proposed Blocker, dmraid, NEW 17:18:57 so this is another firmware RAID case, in this case Promise fwraid rather than Intel. it uses dmraid, which is a path we had not tested at all for F25 (I don't think) until today. 17:19:13 I actually saw this one, the board is Asus M5A97 Pro I believe, AMD board 17:19:16 however, the data in the bug kinda suggests this may be some kind of hardware issue... 17:19:17 * nirik wishes we knew how common these things are anymore. 17:19:23 adamw: not nvidia raid 17:19:27 kparal: ah, k. 17:19:50 just for the record, non-Intel firmware RAID controllers are mostly supported by dmraid not mdraid. 17:20:12 +1 17:20:14 so dlehman says it looks like dmraid doesn't assemble the set on boot at all, which of course means anaconda can't find it to install to 17:20:31 we don't know why that is, yet; it may be a dmraid bug or it may be failing hardware or something 17:21:07 if pschindl's coming back soon, we can maybe try to figure out why 17:21:25 We've so far only seen this on a single piece of hardware? 17:21:40 sgallagh: yes 17:21:50 * pschindl is here. 17:21:51 and it also happens on f24 there FWIW right? 17:21:57 sgallagh: yeah. if anyone else has a non-Intel firmware RAID controller and can do an install to it, now would be a great time to try 17:21:58 nirik: yes 17:22:15 yeah, which is the other suspicious thing; like pschindl, I vaguely recall that he tested F24 on this same hw at release time and it worked 17:22:21 but...I wouldn't swear to it in a court of law :) 17:22:29 LOL 17:22:43 If it's also an issue on F24, I'm inclined to vote -1 17:22:59 kparal: will pschindl have the affected box handy when he comes back? 17:23:08 Simply because if it was affecting a significant portion of users, we'd have had a report before now 17:23:10 adamw: see above, he's here now 17:23:16 and no, we're not in the office 17:23:18 Hi all. 17:23:24 ah, okay. 17:23:29 so can't do much to diagnose it at this point 17:23:30 I will be back in the office on Monday 17:23:43 okay, so we're gonna have to basically guess at this one. 17:23:48 unless anyone else has the hardware to test. 17:23:51 but if someone will be in office tomorrow he can test it 17:23:59 I haven't, sorry. 17:24:04 it's just about booting to anaconda. 17:24:18 if the dmraid was tested on F24 and it worked on other hardware, and the F24 fails on this HW, then I am -1 to block on this 17:24:44 Agree. I'm -1 Beta blocker. 17:24:48 if it is just one hardware and also an issue in F24, I am inclined towards -1 17:24:59 pschindl: It was asserted that you also tried running F24 on this same hardware and it failed there, is that true? 17:25:05 /me just wants the confirmation 17:25:11 sgallagh: yes 17:25:26 sgallagh: yes, it's noted in the bug 17:25:35 Sorry, missed that in the BZ 17:25:40 OK, then I'm -1 blocker 17:25:50 (for the reasons I cited earlier) 17:26:02 Yes, I tried to boot F24 installer and it failed today on the same computer. 17:26:14 i can go with -1 on the grounds that we at least *suspect* this is something up with the hardware, and we tested this kinda too late anyway. if it turns out that all non-intel fwraid is really broken we should take that as a final blocker of course. 17:26:31 sure 17:26:41 i'm just gonna dig around my archives for a minute to see if i actually can confirm whether pschindl tested this same box for f24 release 17:26:54 I seem to vaguely remember that 17:27:04 maybe it's again somehow related to raid initialization? 17:27:09 I'm -1 right now too. If we find what's the problem with that machine I'll repropose it (for final). 17:27:15 * nirik is -1 also 17:28:34 adamw: We have two machines with firmware raid. This one is next to Kamil, so I don't test on it so often. I have another next to me. But I think that I tested it on both. 17:29:33 kparal: well, if it is it'd just be a coincidence, as the codepaths are totally different (obviously we're not parsing mdadm output for dmraid sets) 17:29:44 Strange thing about this problem is that there isn't any md* device even though the fw says that raid was created. 17:30:06 welp, that's enough -1s, anyhow 17:30:29 pschindl: well, that's basically what dlehman said too, it looks like dmraid never actually constructs the array, so of course anaconda can't find it 17:30:59 further debugging of this probibly needs dmraid commands run on the affected machine to figure out what it thinks is going on 17:31:56 proposed #agreed 1382274 - RejectedBlocker (Beta) - this was reported very late and without sufficient information to be sure if it's a real bug or some kind of hardware/setup issue. It's rejected for Beta, but we will investigate further and may accept it for Final if it turns out to be a genuine and serious bug. 17:32:12 ack 17:32:14 ack 17:32:19 ack 17:32:24 pschindl: for the future - Intel firmware RAID and Promise firmware RAID hit *very* different codepaths, so it's always good to test both, and test them early enough that we can fix bugs 17:32:25 ack 17:32:30 ack 17:32:34 pschindl: one passing doesn't give a very strong indication at all that the other will also pass 17:32:56 ack 17:32:57 pschindl: i can test intel fwraid too (as you know) but i don't have any non-intel fwraid hardware, you're our only hope for that :) 17:33:05 #agreed 1382274 - RejectedBlocker (Beta) - this was reported very late and without sufficient information to be sure if it's a real bug or some kind of hardware/setup issue. It's rejected for Beta, but we will investigate further and may accept it for Final if it turns out to be a genuine and serious bug. 17:35:01 that's all the proposed blockers, i believe 17:35:09 does anyone have any last-minute proposals? speak now or forever hold your peace 17:35:42 ... looks like we can move on :) 17:36:24 No, I have nothing to propose. 17:36:35 Yes, we can move on 17:39:29 * nirik is good to move on. jkurik ? 17:40:22 ah, sorry 17:40:42 #topic Test Matrices coverage 17:40:59 #link https://fedoraproject.org/wiki/Test_Results:Fedora_25_Beta_1.1_Summary 17:41:09 coverage is pretty strong 17:41:37 the only significant omissions are a few of the real media default_boot_and_install tests, and cloud on ec2 or openstack (cloud tests have only been run locally) 17:42:21 for those who don't know - coconut's results in the 'Default boot and install' table are kind of a lie, as those tests are really supposed to be run with real shiny optical discs on a bare metal system, while openQA tests in vms 17:42:34 i'm gonna propose some changes to the wiki layout there after Beta 17:42:44 I could do a openstack test real quick if people wanted to wait... 17:42:51 so only the human results in that table can be considered fully valid 17:42:51 but if it worked locally it's probibly ok 17:42:53 nirik: sure 17:42:59 adamw: ok, thanks. 17:42:59 if you can at least just boot it up and make sure 17:43:21 i think for beta we don't need to be super strict about testing literally every single image in every single way, though. we have covered each major image type. 17:43:32 for final i'd like to see a full set of human results in that table. 17:43:51 so lets wait for nirik to run the test 17:44:05 otherwise I am fine with the coverage 17:44:23 QE team: good job 17:44:24 adamw: to be clear, we've never done that 17:44:29 adamw: Maybe as a stopgap you should make coconut not report successes there, only failures? 17:45:04 I know I mostly just scan pages for empty spots; not sure who else does that, but coconut's results make it look skippable 17:45:06 kparal: whenever i put a human result in that table it's always with a real disc on bare metal 17:45:14 sgallagh: we don't report failures with openqa at all into wiki, too many false negatives 17:45:25 sgallagh: nah, what i'd like to do is have a table with a lot more columns, covering virt, optical media, and usb (and kill the separate usb table) 17:45:30 that's gonna be my proposal anyway 17:45:31 adamw: ok, I mostly use usb for that table 17:45:31 ok 17:45:37 kparal: zoiks, ok, that's bad :P 17:45:43 clearly we need a better table 17:46:28 kparal: i'm actually thinking about setting up reporting of failures from openqa, but that's a side track... 17:48:15 * adamw wonders if it's still possible to buy CD-Rs 17:48:37 adamw: you might need to add ostree testing results to wiki as well 17:48:53 ok, came up fine. 17:48:56 $ uname -a 17:48:56 Linux kevintest1.novalocal 4.8.0-0.rc7.git0.1.fc25.x86_64 #1 SMP Mon Sep 19 15:24:06 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux 17:48:56 mboddu: yes, it's planned. 17:49:00 nirik: yaay 17:49:14 nirik: thanks 17:49:15 nirik: did resize work? (if that applies to openstack, /me doesn't know the clouds) 17:49:24 adamw: amazon says yes :) 17:49:36 yes 17:49:40 [ 25.944074] EXT4-fs (vda1): resizing filesystem from 786176 to 10485504 blocks 17:49:40 [ 32.236772] EXT4-fs (vda1): resized filesystem to 10485504 17:49:48 awesome 17:49:52 and it's got the full 40G for / 17:49:57 so, i'm gonna say we're good for coverage, anyone disagree? 17:50:00 /dev/vda1 40G 529M 38G 2% / 17:50:12 adamw, yes, it is possible to buy CD-R 17:50:17 proposed #info Test Matrices coverage for the Beta release is pretty strong and sufficient 17:50:34 people still use them for burning music cds sometimes 17:50:54 nb: yeah, I still do so :) 17:51:16 oh good, cos i'm about to run out 17:51:21 ack 17:51:22 * moto-timo is surprised that my new car still has cd player 17:51:25 * nirik isnt sure he has a cdrom drive anymore. ;) 17:51:34 #info Test Matrices coverage for the Beta release is pretty strong and sufficient 17:51:36 #topic Go/No-Go decision 17:52:06 so we need QE, RelEng and FESCo representatives to vote whether we are Go or No-Go 17:52:15 I am personaly Go 17:52:26 per qa policy, we vote go - coverage is sufficient and there are no unaddressed blockers 17:52:35 go for me (fesco, releng, whatever I am) 17:52:44 * adamw just trying to verify the remaining accepted blockers 17:52:53 adamw: I have 100 cd-r you can have :D 17:53:02 releng is go 17:53:37 adamw: all of them should be verified already 17:53:43 /me lost connection; we're voting on go/no-go? 17:53:52 sgallagh: yes, we do 17:53:58 Go! 17:54:06 kparal: they haven't all been verified *with Beta-1.1* though 17:54:15 kparal: they've been verified with updates or nightlies or whatever 17:54:20 hm 17:54:24 i just like to double check they're OK in the release 17:54:28 i think they're all fine though. 17:54:30 ok, thanks 17:54:44 adamw: shall we wait ? 17:55:00 eh, nah 17:55:04 i'm sure it'll be fine 17:55:08 if it's not i'll just pretend it is 17:55:14 :) 17:55:15 ok 17:55:27 #agreed The status of the F25 Beta 1.1 compose is GO 17:55:37 #action jkurik to publish the Go/No-Go result 17:55:47 #topic Open floor 17:55:57 anything else to discuss today on this meeting ? 17:56:23 are we gonna be trying to make FMW the 'default download' for Workstation for Beta? 17:56:25 or is that final stuff 17:57:09 default download? 17:57:25 mboddu: The thing listed on getfedora.org 17:57:30 aiui the idea is that for F25 when you go to getfedora and click 'Download' you'll get FMW, not a .iso image 17:57:33 Rather than sending people directly to the ISO 17:58:09 sgallagh: I know about FMW but I dont know we are having plans to make it default 17:58:33 * nirik doesn't know the timeline for that 17:58:34 one further question is whether we should keep workstation dvd in the tree, when we know it's untested and also something else than people would expect (ostree image) 17:58:39 mboddu: We have been for about 18 months 17:58:40 AFAIK the websites for Beta were reworked to support FMW 17:59:01 kparal: Is it working at all yet? It wasn't yesterday. 17:59:04 sgallagh: I always used dl.fp.org 17:59:23 sgallagh: I didn't try it, I just read you and dgilmore reporting it's broken 17:59:25 I've been told by walters that the nightlies work but they aren't paying attention to Fedora... 17:59:49 from the conversation I had yesterday with sticker: jkurik: Also, just in case robyduck isn't able to attend... I can't speak for him as lead, but I can say that one of the websites items, to get mediawriter web content available for Beta, is in good shape thanks to his excellent work 18:00:18 huh. ok. 18:00:25 so I believe the FMW is the "default download" already for Beta 18:01:01 https://stg.getfedora.org/en/workstation/prerelease/ 18:01:11 well, as always depends on what you mean by 'default' ;) 18:01:58 nirik: the download button points to the Alpha image 18:02:21 kparal: we know the beta version does not work 18:02:30 kparal: and we do not have a good way to remove it 18:02:33 well, yes, how could it point to beta when we haven't released it yet. ;) but sure we need to make sure thats right before release day 18:02:42 we would have to munge a lot of metadata and data in pdc 18:02:57 dgilmore: how about just chmod 000 it 18:03:05 I will check with robyduck during the readiness meeting later today 18:03:09 nirik: sure 18:03:22 dgilmore: don't you strip the metadata from the release tree already? 18:03:34 dgilmore: since it gets split and some of the images listed in it aren't there in the final location 18:03:50 but PDC, yeahg. 18:04:12 still, the metadata in PDC is already kind of a lie, for releases that get split into the 'main' chunk and the 'alt' chunk and sent to the mirrors. 18:04:36 adamw: it is in pdc 18:05:39 ok, anything else ? 18:05:41 for the record, future ostree installer images will have -ostree- in their filename rather than -dvd- (after https://pagure.io/pungi/pull-request/420 is landed and starts being used for Fedora composes) 18:07:00 that includes the 'Atomic host' installer image as well as the Workstation one 18:07:20 adamw: nice :) 18:08:06 adamw: it is merged, just needs built and deployed 18:08:42 well, it wasn't until like 30 seconds ago :P 18:09:09 thanks folks for the meeting and see most of you on the readiness meeting in less then one hour 18:09:31 * moto-timo waves 18:09:47 #endmeeting