17:00:07 #startmeeting F19 Alpha Go/No-Go meeting 17:00:07 Meeting started Thu Apr 18 17:00:07 2013 UTC. The chair is jreznik. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:00:07 Useful Commands: #action #agreed #halp #info #idea #link #topic. 17:00:22 #meetingname F19 Alpha Go/No-Go meeting 17:00:22 The meeting name has been set to 'f19_alpha_go/no-go_meeting' 17:00:34 #topic Roll Call 17:00:43 * satellit listening 17:00:56 hey! who's around? 17:02:39 * nirik is here. 17:02:49 may need to do a quick reboot here in a sec tho. 17:03:13 let's wait a moment for more people to come, I moved the meeting to -2 channel, not to overflow into board meeting in case we would need it 17:03:46 * nirik will be back in a min 17:03:49 #chair nirik tflink adamw rbergeron 17:03:49 Current chairs: adamw jreznik nirik rbergeron tflink 17:03:53 * tflink is here 17:05:47 ok, let's start - and let's hope nirik make it before we start real "work" here 17:05:59 #topic Purpose of this meeting 17:06:08 #info Purpose of this meeting is to see whether or not F19 Alpha is ready for shipment, according to the release criteria. 17:06:12 #info This is determined in a few ways: 17:06:17 #info No remaining blocker bugs 17:06:23 #info Test matrices for Alpha are fully completed 17:06:29 #link http://qa.fedoraproject.org/blockerbugs/milestone/19/alpha/buglist 17:06:35 #link https://fedoraproject.org/wiki/Test_Results:Fedora_19_Alpha_RC4_Install 17:06:42 #link https://fedoraproject.org/wiki/Test_Results:Fedora_19_Alpha_RC4_Base 17:06:49 #link https://fedoraproject.org/wiki/Test_Results:Fedora_19_Alpha_RC4_Desktop 17:06:54 #link https://fedoraproject.org/wiki/Fedora_19_Alpha_Release_Criteria 17:07:33 did you ping dgilmore? 17:08:51 thanks for ping, and I hope rbergeron will show up too (chair should did it) 17:09:31 * rbergeron is here, albeit via phone irc, as the internets apparently aren't available at cloud conferences, or at marriotts in rooms where i sleep. 17:09:56 rbergeron: cloud does not need internet! 17:10:56 * satellit afk 17:11:02 for accepted blocker bugs - there's no unresolved bug 17:11:28 we have one proposed bug - so let's start with mini blocker review as alway, tflink, may I ask you? 17:11:38 sure 17:12:03 do we want to go through the accepted blockers as well? 17:12:17 "for accepted blocker bugs - there's no unresolved bug" 17:12:33 and I think we covered it pretty well yesterday 17:12:38 just making sure 17:12:54 #info right now, we have 1 proposed blocker for F19 alpha 17:13:03 #topic (953447) EFI boot on Mac halts after GRUB 17:13:03 #link https://bugzilla.redhat.com/show_bug.cgi?id=953447 17:13:03 #info Proposed Blocker, lorax, NEW 17:14:42 so, two macs are busted 17:14:50 * adamw attempts to engage his caring circuits 17:14:51 it sounds like this is a variant of the failing-to-boot-post-install bug 17:15:00 i don't think so, i think satellit is muddying the waters 17:15:19 all we know is that USB EFI boot of an F19 RC4 image failed on two macs. 17:15:30 * nirik is back 17:15:56 i'd like to get data from a few more macs really, but we don't really have time. mjg59 is on the other side of the country from his, and I haven't seen cmurf this cycle. 17:16:01 anyone here have a mac? 17:16:26 * tflink has an older one 17:16:39 I think it's new enough to work, though 17:16:40 were we still excluding macs due to their 'interesting' efi? 17:16:43 (as a blocker) 17:16:43 I can test on the current uefi macs 17:16:46 * spot has them 17:16:50 spot: ooh, can you? thanks 17:16:55 shouldn't take long 17:17:03 shall we wait for tflink and spot to test? 17:17:23 nirik: that's a good question, seems like other uefi systems are not affected by this one 17:17:25 i think all that's needed is to write dvd or netinst to a usb stick and try to do an EFI boot from it. 17:18:01 we took the 'mac exception' out of the criteria kinda on purpose, but we have some wiggle room in how they're currently written: this just comes down to the old 'how widespread is the bug' judgement call 17:18:12 we can spin it 'ahh! all macs are broken!' or we can spin it 'eh, there aren't that many macs'. 17:19:23 * nirik nods. ok 17:19:40 * jreznik is ok with waiting for guys to retest and also with what we already did on efi side - few more macs are probably not a big issue at this time, just my thinking 17:20:45 something odd is going on, trying again with more debug 17:21:18 * rbergeron suspects we have far lower usage as a total coming from macs during alpha/beta vs. final 17:22:55 spot: are you trying it too? 17:23:45 jreznik: yes 17:23:53 making the usb key off the netinst iso of tc6 now 17:24:15 we need rc3 at least 17:24:30 whoops. :/ 17:24:48 looking on https://bugzilla.redhat.com/show_bug.cgi?id=949761#c29 17:25:04 spot: rc4 netinst should be fine 17:25:07 my fault, I parsed right to the end of the file list 17:25:29 just a few more minutes. 17:25:35 it seems to be something really early in the boot process - removing quiet and adding rd.debug does nothing 17:26:53 is there anything I can add to the cmdline to get more debug output? 17:27:09 tflink: is yours failing at the same place as the reporters? 17:27:13 right after grub? 17:27:14 yeah 17:27:30 okay, writing out to usb now 17:27:33 so, that's another confirm 17:27:34 it seems to load the initrd and just stop in its tracks 17:27:47 i guess pjones/mjg59 would be the ones who could possibly debug it 17:27:55 speaking of... 17:27:58 adamw: I just pinged him 17:28:29 pjones: [19:25] it seems to be something really early in the boot process - removing quiet and adding rd.debug does nothing 17:28:32 pjones: so now we have three confirmations that Macs just apparently die right after grub when booting USB-written RC3/RC4 images. 17:28:34 [19:26] is there anything I can add to the cmdline to get more debug output? 17:28:38 cool. 17:29:02 tflink: perhaps add initcall_debug ? 17:29:14 I just did another uefi install with the same usb drive, so I'm reasonably sure that the media is OK 17:30:03 the macbook pro stops after "Secure boot not enabled" 17:30:30 so confirmed too 17:30:49 spot: so you've actually got one in front of you? 17:30:58 pjones: yes 17:31:00 pjones: I do as well 17:31:06 tflink: spot: the original reporter isn't very clear - do you actually see grub at all? 17:31:13 yes 17:31:15 nirik: no dice, that didn't give me more output 17:31:23 grub loads fine, kernel starts to boot 17:31:32 alright, kernel issue. 17:31:37 you need jwb, not me ;) 17:32:00 something starts to boot, it pulls stuff from the disk and then just stops 17:32:01 still, i'm not sure we really need to live debug in the meeting 17:32:10 true 17:32:12 we're not gonna be able to fix it, respin and re-test in the next six hours 17:32:15 so we either ship it or we don't 17:32:26 mac has always been a sort of "best effort" thing in the past. 17:32:28 i dont think macs are an alpha blocker. beta, sure. 17:32:33 I really don't think it should be an alpha blocker. 17:32:40 beta would be... awfully nice of us. 17:32:48 jwb: hi 17:32:50 hello 17:32:53 yo 17:32:56 join the Mac party 17:33:04 also, you should probably drag jforbes in here. he's on f19 duty 17:33:52 eh, I think there's no realistic way we could fix this, respin and retest in 6 hours (like adamw said before) 17:34:09 right, we can just do the debugging in #fedora-kernel or whatever 17:34:12 just so you're aware, i ahve no idea what you're talking about 17:34:16 sorry 17:34:31 the bug in #topic - looks like EFI boot on all Macs is busted in RC3/RC4 17:34:36 the question becomes whether we're OK with releasing alpha without mac support 17:34:38 jwb: see topics, seems it passes grub and then... 17:34:51 has anyone tried with an actual silver disc yet, btw? 17:34:52 I thought that dd prepared media was supposed to still work 17:35:04 * tflink has not 17:35:11 oh, did you all use litd? 17:35:23 we should check dd, litd, actual disc, to see exactly what the status is 17:35:31 by 'we', of course, i mean 'you' 17:35:37 that's "the royal we" 17:35:53 adamw: from the bug: 1. Create USB stick with RC3/4 using either litd or dd 17:35:58 mac images have some kind of weird, perverted el torito thing done to them in order to boot, don't they? 17:36:01 ah, so satellit tried both 17:36:10 * rbergeron isn't sure how many macs even have drives for that these days? 17:36:11 jwb: i think that's for *BIOS* boot on macs 17:36:25 it's still oversized :-/ won't fit on a DVD 17:36:47 jwb: they do, but the bootloader is working, so we're seeing the file system fine 17:37:12 I doubt if it's a disk formatting issue 17:37:35 it's getting far enough to print "Secure boot not enabled" 17:37:50 btw. there are not many changes between tc6 and rc3 accepted (and even new lorax wasn't used for rc3) 17:37:53 spot, the kernel doesn't print that 17:38:04 jwb: the shim does, right? 17:38:12 spot, yes, which runs before grub 17:38:18 jwb: spot and tflink say that after they select a boot option it loads from the disk for a while and then stops - my assumption would be that it's simply faulting before fbcon comes up 17:38:23 uefi->shim->grub->kernel 17:38:42 that is the only thing printed on the fb. 17:38:47 the bug says "mac mini" 17:38:51 is that the 32-bit mac mini? 17:38:55 because just no 17:39:02 * spot can reproduce it on the current macbook pro 17:39:13 ok 17:39:32 if i change the grub config options and F10 boot it, it prints "Booting a command list" 17:39:47 but aside from the shim print, thats it 17:40:27 right, that's being printed when grub calls shim's verify routine to verify the kernel is signed properly 17:40:33 adamw: i can burn netinst to cd, one sec 17:40:33 someone said adding GRUB_TERMINAL_OUTPUT=console makes it boot? 17:40:39 which means we know a) we're able to read the kernel, and b) it works 17:40:43 jwb: ignore that, it's satellit muddying the waters 17:40:54 jwb: I... kind of suspect that he's simply commented on the wrong damned bug 17:41:14 pjones: no, but he wrote a very confusing comment. the real takeaway from that is that it worked with TC6 but it doesn't with RC3/RC4 17:41:19 (which seems odd in itself, but hey) 17:41:27 if spot/tflink want to double check that at some point it'd be helpful i guess 17:41:35 so essentially what we're left at is that grub loads the kernel, and then that's it 17:41:40 does it load the initramfs? 17:42:13 so what somebody needs to do is to get F18 on one of the machines and try updating each of grub-efi and kernel individually and together on the installed machine. 17:43:19 If one of them fails, we have a culprit. If they work fine together, then it really is the media, but since shim is validating kernel correctly, I doubt that. 17:43:25 so the most likely candidate that changed between TC6 and RC3 - assuming that satellit is correct that TC6 worked - is kernel 17:43:26 Take an F19 live USB image. Mount the HFS+ filesystem on it. Copy over grub.efi from F18. 17:43:26 spot: could you retry it with that tc6? so it will have use too... 17:43:28 on my one UEFI machine, which is not a Mac, grub will print out 'Loading initial ramdisk...' before the kernel is started 17:43:49 jwb: whether or not you see that depends on particulars of the uefi implementation, sadly. 17:43:56 TC6 had kernel-3.9.0-0.rc6.git2.1.fc19 , RC3/RC4 have kernel-3.9.0-0.rc6.git2.3.fc19 17:43:59 pjones, ah. i see. unfortunate 17:44:14 jwb: not sure, it loads something from the usb disk but I'm not sure what 17:44:33 the only other things that changed were lorax (which could _feasibly_ have something to do with it, but it seems unlikely), anaconda, python-blivet and gnome-session 17:44:40 anyway, somebody with a box needs to do what either I or mjg59 has said. 17:44:44 ok 17:44:46 and then we'll know which thing is busted. 17:44:50 or both 17:44:58 sure. 17:44:58 adamw: well, according to the comment, new lorax wasn't used for rc3 17:45:01 oh right 17:45:12 so again, assuming TC6 vs. RC3 is correct, the only plausible suspect is the kernel 17:45:32 i do not have a mac, and i don't think anyone else on the kernel team does. if we did, we would have been testing this for a while now given how troublesome they are 17:45:32 jwb: the bug was reported on a newer mac mini - 64bit afaik 17:45:41 so pls guys, try tc6 17:45:55 Ok 17:45:58 So that thing that I said to try? 17:45:59 Try it. 17:46:08 Because that actually provides useful information 17:46:08 tflink: you heard the man 17:46:25 did I miss something? 17:46:27 * tflink looks 17:47:30 i think they're talking about: 17:47:31 so what somebody needs to do is to get F18 on one of the machines and try updating each of grub-efi and kernel individually and together on the installed machine. 17:47:34 If one of them fails, we have a culprit. If they work fine together, then it really is the media, but since shim is validating kernel correctly, I doubt that. 17:47:37 Take an F19 live USB image. Mount the HFS+ filesystem on it. Copy over grub.efi from F18. 17:47:48 i'm not sure if there was a *second* step to mjg59's idea :) 17:48:00 yeah, seeing that now 17:48:12 Well, his version is half of my version. 17:48:22 I assume if grub works he'll tell you to copy a kernel in. 17:48:24 odd, the shiny disk shows up with 3 bootloaders titled 'windows', and 2x 'EFI boot' 17:48:36 * tflink is starting to modify the live image 17:48:41 yeah, cmurf and satellit have mentioned that before. there's some reason for it. i don't recall what. 17:48:48 murphy 17:48:56 If one of them doesn't say "Fedora" then something's broken 17:49:22 so, let's give this the old college try once more: 17:49:30 can we do the debugging in #fedora-kernel and do the blocker discussion here? 17:49:51 i mean, it seems pretty clear now that this affects all (or at least a lot of) macs, but doesn't affect anything else. 17:50:09 it is 17:50:38 yeah, sounds like a plan 17:50:39 and it's probably too late to do anything with it now 17:51:04 so it sounds like we're mostly -1 on this 17:51:12 oh, well there's T.C. Hollingsworth's bug that sounds kinda similar - https://bugzilla.redhat.com/show_bug.cgi?id=951761 - but I don't think it's the same. 17:51:13 if not unanimously? 17:51:25 the dev take seems to be generally -1 17:51:31 i wouldn't really fight a -1 17:51:51 though it might be nice if we could ninja out an image that works for mac folks a week later or something. 17:52:00 and looking on what we already did with uefi, I'm -1 too 17:52:17 yes, we're unanimously against mac blockers. 17:52:20 (dev, that is) 17:52:21 adamw: or beta TCs will follow soon 17:52:45 rbergeron: are you with jreznik? 17:52:47 adamw: i think that seems reasonable 17:52:56 adamw: physically? 17:52:57 proposed #agreed 953447 - RejectedBlocker - While this does seem to affect all apple hardware, that isn't enough to block F19 alpha over. Thus, this is rejected as a blocker for F19 alpha. 17:52:59 okay, so dev and management are -1 and qa is pretty meh. sounds like -1 to me 17:53:01 or in spirit/mind 17:53:08 rbergeron: like, spiritually, maaaan 17:53:10 opinion :) yes. 17:53:22 ack 17:53:32 ack/nak/patch? 17:53:34 tflink: maybe patch it with adamw's idea to spin mac version later? 17:53:41 eh, let's keep that unofficial./ 17:53:48 if we had more folks reporting; i honestly don't know that we have a ton of mac usage until post-release. clearly not here for rc's though. (sadly) 17:53:48 ok 17:53:50 yeah, agreed 17:53:57 ack, and agree iwth adam re: unofficial hope 17:54:16 ack, not promising something we don't know we can fulfil 17:54:16 rbergeron: satellit and cmurf are two people who reliably test on macs. satellit's reports can be frustrating to interpret, though. 17:54:25 if you suspect this is kernel, then you guys should really CC the kernel team because this is the first i've heard of this bug 17:54:30 for future reference 17:54:30 adamw: or jskladan 17:54:39 jwb: the reporter guessed at lorax, but it sounds more kernel-y, so i'll re-assign 17:54:44 jwb: it was filed less than 24 hours ago 17:54:56 adamw: i can test on a mac 17:55:16 do we have enough acks now? 17:56:12 ah, missed one 17:56:20 #agreed 953447 - RejectedBlocker - While this does seem to affect all apple hardware, that isn't enough to block F19 alpha over. Thus, this is rejected as a blocker for F19 alpha. 17:56:23 adamw: yeah, just saying the overall % of folks i think are affected is low. worst case: we bring irritated people out of the woodworks with the alpha and have a slightly better guestimation :) 17:56:31 that would be all of the proposed blockers 17:56:45 unless there's been one proposed since we started the meeting 17:57:07 don't see any, at least picked up by the app 17:57:39 i have one 17:57:44 not officially proposed, but one to bring up 17:57:56 https://bugzilla.redhat.com/show_bug.cgi?id=953329 17:58:07 i hit that a couple of times last night trying to install to very blank disks 17:58:29 i kinda owe bcl some data and a clear reproducer, but wanted to bring it up just in case it makes anyone poop bricks 17:58:41 the good-ish news is that both times I hit it, rebooting and trying again worked, so there's an 'obvious' workaround 17:59:00 commonbugs? 17:59:05 yeah 17:59:43 adamw: document but if it works second time its a reasanoblish workaround for alpha 18:01:11 has anyone else tried an install to a very 'blank' disk, just out of interest? 18:01:38 one time I hit it on a disk I had wiped all partitions from and created a new disk label using 'fdisk', the other time was a disk that had been part of a RAID array that I destroyed 18:01:46 aren't virt installs usually blank disks? 18:03:09 jreznik: it kinda depends on your testing method. not for me - I just re-use a couple of VMs over and over. 18:03:16 I did a new virt install yesterday with RC3, it went fine 18:03:37 just-created lv as a backing store 18:03:37 good news. 18:04:35 adamw: do you want this one to go formally though the review or just mark it as commonbugs? as it seems tflink did blank install 18:04:35 sounds like no-one's too worried about this one. or we just all want to release the alpha and go drink. :) 18:04:41 commonbugs is fine 18:04:42 already marked 18:04:51 ok 18:05:08 so anything else for mini blocker review? 18:05:28 nothing I'm aware of 18:06:20 #info after mini blocker review, there are no accepted & unresolved blocker bugs 18:06:55 #topic Test matrices coverage 18:07:12 (see links above) 18:08:10 looks fine 18:08:22 KDE tests are from RC3, but that shouldn't be a problem, no KDE-related changes in RC4. 18:08:33 install and base are covered afaics. except the perennial SCSI. 18:09:09 * nirik poked at the Xfce spin the other day FWIW, and it seemed to work fine for me. 18:09:37 LXDE is known broken right adamw ? 18:10:02 yeah 18:10:08 anyone to do a really quick - does kde boots test? my virt-manager is broken for some reason, asked kde guys to take a look but... I expect everything is ok as the changes between rc3/rc4 were low 18:10:22 sure, i can do that. 18:12:10 KDE live x64 boots fine, starting an install. 18:12:30 ok, thanks 18:13:05 install's running fine. looks a-ok. 18:13:36 well, I think we can move on 18:14:22 #info test matrices coverage looks good 18:15:04 #info KDE tests are copied from RC3, with quick RC4 test up to installation, no KDE-related changes in RC4 18:15:25 #info install and base are covered except the perennial SCSI 18:15:38 #topic Go/No-Go decision 18:16:13 we are here for Alpha! the Go/No-Go decision 18:16:26 releng is +1 to go 18:16:31 adamw: may I ask you for QA? 18:16:45 #info releng is +1 go 18:16:53 rbergeron: you as FPL? 18:17:36 * nirik is +1 for go. 18:17:57 #info fedora program manager is +1 to go 18:18:25 yes, go, based on things as i see the scrollback of slightly 18:18:26 :) 18:18:27 +1 18:18:32 * jsmith is +1 for go as a nobody/ex-FPL, for what little it's worth 18:18:52 QA is go per our requirements 18:19:36 "QA approves the release if all validation tests appropriate to the release phase (Alpha, Beta or Final) have been performed and there are no accepted blocker bugs that are unaddressed in the candidate compose." 18:19:39 #info FPL is +1 to go (and ex-FPL too) 18:20:05 #info QA is +1 go per requirements 18:20:25 #info QA approves the release if all validation tests appropriate to the release phase (Alpha, Beta or Final) have been performed and there are no accepted blocker bugs that are unaddressed in the candidate compose. 18:20:44 ok, I think we are all set 18:21:25 proposed #agreed Fedora 19 Alpha is +1 to go by QA, FPL, releng, FPGM and developers 18:22:18 any acks/nacks/patch 18:22:37 ack 18:23:32 ack 18:23:39 tflink: your 'proposed #agreed' meme is spreading :) 18:23:45 ack 18:23:56 ack 18:23:57 adamw: another qa patent? 18:24:11 #agreed Fedora 19 Alpha is +1 to go by QA, FPL, releng, FPGM and developers 18:24:12 yes. you owe us ONE MEEEEEELLION DOLLARS 18:24:26 assuming it works better than the non-deterministic fuse :) 18:24:27 adamw: ill buy you a beer 18:25:11 ok guys, thank you for coming, thanks for a great job on alpha! 18:25:29 whee! 18:25:53 without that uefi mess, we would even beat ourselves and release on time :) 18:25:56 yay. let's drink 18:26:03 proposed #agreed UEFI stinks 18:26:11 ack 18:26:12 ack 18:26:24 i imagine there isn't an 'ack' big enough for mjg59 18:26:45 #info jreznik to announce result of Go/No-Go meeting 18:27:28 otherwise I think infra is ready to mirror bits, websites guys would need checksums for final isos 18:27:54 adamw: I expect you will try to put commonbugs page together, right? 18:29:44 nirik: ^^^ (one line off) 18:29:47 sure 18:30:05 yep. should be all set. 18:30:11 thanks 18:30:32 anything else to bring up? /me hopes not and setting the fuse (deterministic one) 18:30:53 3... 18:33:13 2... 18:33:13 * adamw taps on fuse 18:33:32 1... 18:33:41 and adamw taps fuse 18:33:44 thanks again! 18:33:48 #endmeeting