17:00:54 #startmeeting F16-Alpha Blocker Review 17:00:54 Meeting started Fri Jul 15 17:00:54 2011 UTC. The chair is jlaska. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:00:54 Useful Commands: #action #agreed #halp #info #idea #link #topic. 17:01:01 #meetingname f16-alpha-blocker 17:01:01 The meeting name has been set to 'f16-alpha-blocker' 17:01:06 #topic Roll Call 17:01:12 * brunowolff is here 17:01:22 okay, what lucky souls do we have for our journey today? 17:01:28 * jlaska tips hat to brunowolff 17:01:39 yo 17:01:39 * tflink is wearing his blocker review party hat 17:01:53 alright tflink! :) 17:01:55 hey adamw 17:01:59 * Viking-Ice sits and sharpens his battleaxe.. 17:02:00 Viking-Ice: ping 17:02:03 oh there he is 17:02:06 Viking-Ice: joined bef ... hey Viking-Ice 17:02:51 * j_dulaney waves 17:02:54 rbergeron: mentioned she won't make the meeting today 17:02:56 Hey j_dulaney! 17:03:03 wow, that's a lot of bugs. even if most of them are related to eachother 17:03:04 Yo 17:03:09 The cloud SIG meeting is occurring at the same time. 17:03:20 * rbergeron has broken glasses and is waiting 17:03:23 huh? 17:03:34 cloud meeting in 2 hrs 17:03:43 tflink: we're probably going to be dealing with almost all of them together 17:03:49 brunowolff: Most of 'em seem to be incomplete features 17:03:51 we can move if needed 17:03:52 I hope so! 17:03:53 tflink, I can throw in couple of dozen or so ;) 17:03:57 I must have misread the time. I thought I saw 17, but maybe it was 19. 17:03:57 okay, let's get this party started ... 17:03:58 more that is 17:04:09 #topic Why are we here? 17:04:12 oh. it should be 19 17:04:15 Some quick links an reminders ... 17:04:26 Viking-Ice: the more the merrier? 17:04:34 #info We'll be walking through the proposed, accepted blockers and NTH bugs listed at http://fedoraproject.org/wiki/Current_Release_Blockers 17:04:35 hehe 17:04:46 #link http://fedoraproject.org/wiki/Fedora_16_Alpha_Release_Criteria 17:05:04 If you like these meetings so much, you want to roll your own ... 17:05:05 #link http://fedoraproject.org/wiki/QA:SOP_Blocker_Bug_Meeting 17:05:13 jlaska: propose we start with 713562 - i.e. all the systemd bugs 17:05:16 I think that about covers it 17:05:26 adamw: I agree ... let's cover as many of those as possible at once 17:05:43 who wants to secretize? 17:05:44 i think we can just take 'em as a lump 17:05:50 agreed 17:05:51 * adamw doesn't mind doing it 17:05:53 I liked the proposal to not consider them QA blockers. 17:06:06 #topic All proposed systemd Alpha blockers 17:06:19 right - shall i reiterate that for the record? 17:06:29 * jlaska grabbing your mail link 17:06:30 please 17:06:52 the idea behind these being blockers is fesco wants at least part of the sysv-to-systemd service conversion process complete before alpha goes out, as part of the feature process 17:07:04 #link http://lists.fedoraproject.org/pipermail/test/2011-July/101332.html 17:07:25 however, afaik none of these bugs actually materially affects the quality standards for an alpha release, so i don't think it's really appropriate to handle them under this process 17:07:34 agreed 17:07:44 +1 17:07:47 thanks to viking for putting them all under a meta-blocker so we can easily change the alpha-blocking status =) 17:07:57 which leaves 714478 ,722466 the only ones to deal with today 17:08:07 so if we agree we can leave this process to be handled by fesco, we can just make not block f16alpha and we're done 17:08:12 *make 713562 not block... 17:08:15 yeah, agreed ... nice to identify these and resolve this *early* 17:08:18 +1 to not blocker 17:08:26 +1 to not blocker 17:08:32 +1 to not blocker 17:08:48 #chair adamw 17:08:48 Current chairs: adamw jlaska 17:08:49 of course, if for some reason a bug in this group *does* impact the quality standards - release criteria - it could be proposed as a release blocker directly 17:08:52 cool 17:08:55 Note blocked by QA anyway 17:09:05 Viking-Ice: right. 17:09:39 #agreed all the sysv-to-systemd conversion bugs should not be handled by the release blocker review process but the feature process, so 713562 is rejected as a blocker 17:09:42 Viking-Ice: I thought I saw one more true blocker 17:09:45 adamw: thanks ... beat me to it 17:10:19 this actually does kinda not fall under that feature process either 17:10:27 718722 17:10:32 is another non-systemd blocker 17:10:39 but anyway let's get the actual potential blockers done 17:10:50 let's get the non-blockers marked in the minutes with #info's 17:10:51 720034 17:11:10 At least it isn't a feature 17:11:53 722466 also isn't systemd. 17:12:03 That was stated 17:12:16 along with 714478 17:12:26 so I have the following non-systemd bugs ... 714478 722466 718722 720034 17:12:41 So, 714478, 722466, 720034, and 718722 17:13:05 okay 17:13:14 adamw: are you mass updating the systemd bugs with this decision? 17:13:39 just remove it from the alpha tracker but keep the tracker bug open 17:13:40 jlaska: no need to, only the tracker needs updating 17:13:46 great 17:13:50 Viking-Ice: yup, that's what i did. see https://bugzilla.redhat.com/show_bug.cgi?id=713562#c3 17:13:51 Bug 713562: unspecified, unspecified, ---, notting, NEW, Tracker bug for SYSV to systemd conversion 17:13:53 okay, ready to move on? 17:13:56 yup 17:14:13 we can work from https://bugzilla.redhat.com/showdependencytree.cgi?id=713560&hide_resolved=1 17:14:38 #topic https://bugzilla.redhat.com/show_bug.cgi?id=714478 17:14:39 and viking / j_dulaney, if any of the systemd bugs should be proposed blockers, please go ahead and mark 'em as blocking f16alpha directly 17:14:39 Bug 714478: unspecified, unspecified, ---, kernel-maint, NEW, CPU lockup during boot 17:14:54 #info CPU lockup during boot 17:15:14 This affects 32 bit SMP systems, but is a race so not all are affected equally. 17:15:25 adamw: I don't see any, unless something were to actually fail a release criterion 17:15:35 j_dulaney: cool. 17:15:44 brunowolff: so I guess the question is how many systems are affected? 17:15:45 I mine, I hit it 100%, but I got the impression it didn't always occur on ARM. 17:16:24 It's hard to say. Upstream the reports were being misfiled against RCU instead of the scheduler. 17:16:48 you tested the patch was good, right? 17:17:01 * jlaska reading adamw's "[...] guide to kernel patch testing" 17:17:12 #info The fix has been posted to lkml, but is not yet in Linus' tree. 17:17:14 * j_dulaney liked that 17:17:22 I had both an athlon MP and a dual processor Xeon see 100% boot failure on about a dozen boots. 17:17:50 Yes, the patch appears to solve the issue. I am running 3.0 kernels on both affected machines now. 17:18:14 so it sounds like we have enough to get this issue out for some kernel-devel feedback 17:18:26 any takers on if the frequency alone is enough to mark this as a blocker? 17:18:51 +1 17:18:51 if anything, I think this might hit our failsafe clause ... 17:19:09 the lkml thread is http://lkml.org/lkml/2011/7/12/150 17:19:25 it was an IBMer who said "It appears that there have been several, but they were blaming RCU instead. ;-)" 17:19:44 McKenny is THE RCU guy. 17:19:45 given the tenor of the upstream conversation and the severity of the bug i'd be +1 blocker 17:20:00 I'm in for blocker 17:20:08 yeah, that comment you linked to makes it seem like more people should be hitting this 17:20:08 +1 blocker. 17:20:18 and we may just not have a lot of rawhide users yet, let alone i686 rawhide 17:20:29 i686 SMP. 17:20:29 we've got 3 for blocker so far 17:20:33 even better 17:20:38 +1 17:20:48 adamw: which clause would yo ufile this under? ... our get out of jail escape clause? 17:20:51 * j_dulaney has grabbed a nightly, but he hasn't actually burned it 17:21:12 I think we enough votes for blocker 17:21:25 * Viking-Ice sneaks in have .... 17:21:27 jlaska: it prevents you booting the system 17:21:32 right 17:21:38 criteria 4 17:21:53 so, i mean, pick one 17:21:54 it's the number of potential systems affected that pulls this one in 17:21:57 heh 17:22:00 I pick 1-14 17:22:05 4, 14, whichever 17:22:23 #agreed 714478 - AcceptedBlocker - causes boot failures for i686 kernel 17:22:47 #topic https://bugzilla.redhat.com/show_bug.cgi?id=718722 17:22:48 Bug 718722: unspecified, unspecified, ---, pjones, NEW, Mismatched or corrupt version of stage1/stage2 17:22:54 #info Mismatched or corrupt version of stage1/stage2 17:23:02 #2 for bruno 17:23:05 I didn't actually test installs with this. 17:23:25 I ran into it when repartitioning my system without doing a fresh install. 17:23:29 I hit this with rawhide install + updates 17:23:39 Later the same problem showed up in an F15 update. 17:24:04 Looks like a GCC issue 17:24:07 I think the problem was triggered by a change outside of grub, though poor grub coding may still be the root cause. 17:24:11 hum that probably was another bug this is grub <1 not 1> 17:24:33 do we think fresh installs would also hit this? 17:24:43 Perhaps 17:24:44 hard to say since we've not been able to get through any just yet 17:24:58 I would think so, but I just haven't tested that. 17:25:07 probably be a good idea to check. 17:25:20 leave it 'open' until we do a rats run or whatever? 17:25:39 yeah, we *should* have results by next mid-week 17:25:41 6 : Mismatched or corrupt version of stage1/stage2 17:25:41 This error is returned if the install command points to incompatible or corrupt versions of 17:25:41 the stage1 or stage2. It can’t detect corruption in general, but this is a sanity c heck on 17:25:41 the version numbers, which should be correct. 17:25:45 adamw +1 17:26:00 adamw, +1 17:26:07 It would be nice to ask one of the grub maintainers to look at it promptly though as this may impact other testing. 17:26:13 proposed #agreed 718722 - leave on the list pending rawhide acceptance install results. Will reevaluate next week 17:26:21 I think we'd need to poke pjones for this 17:26:24 * jlaska thought we were moving to grub2 in F16 17:26:26 ack 17:26:31 jlaska, we are afaik 17:26:41 hence I'm not so sure how relevant this is 17:26:57 yeah 17:27:13 sounds like we have enough votes for leaving this unchanged, and revisiting pending install results next week 17:27:30 any nak/s? speak now, or forever hold thy peace 17:27:57 #link http://fedoraproject.org/wiki/Features/Grub2 17:28:01 thx 17:28:10 #agreed 718722 - leave on the list pending rawhide acceptance install results. Will reevaluate next week 17:28:20 #topic https://bugzilla.redhat.com/show_bug.cgi?id=720034 17:28:21 Bug 720034: medium, medium, ---, schwab, NEW, Error: unsupported locale setting 17:28:23 The grub2 feature isn't very far along. I don't think relying on it is a safe way to proceed. 17:28:31 #info unsupported locale setting 17:28:54 brunowolff: yeah, let's keep it on the list for next week ... and if someone wants to volunteer to check-in w/ pjones for feedback 17:29:01 if not ... I'll ping him 17:29:22 #action jlaska - check-in w/ pjones for grub assistance on 718722 17:29:42 okay, this bug was filed before, but found during the limited testing twu was able to perform on the first RATs run 17:30:24 +1 it's a blocker 17:30:34 +1 17:30:36 yeah, killing live installs ... 17:30:52 #info impacts Alpha criteria - The installer must boot (if appropriate) and run on all primary architectures from default live image, DVD, and boot.iso install media 17:30:54 +1 blocker 17:31:01 +1 17:31:15 yup, seems straightforward 17:31:20 #agreed 720034 - AcceptedBlocker - appears to prevent all live installs 17:31:35 okay, moving on ... 17:31:38 #topic https://bugzilla.redhat.com/show_bug.cgi?id=722466 17:31:39 Bug 722466: unspecified, unspecified, ---, mgracik, NEW, creating 32 bit isos fails when there is 2 kernels 17:31:44 #info creating 32 bit isos fails when there is 2 kernels 17:31:56 Our good friend dgilmore filed this earlier 17:32:15 it's something he discovered while trying to build install images for the RATs milestone 17:32:28 basically ... kernel-PAE isn't included right now 17:32:48 I'm fuzzy on where that kernel is needed still 17:33:09 but wanted to raise it for discussion here since I would consider the install images incomplete without this 17:33:10 Hum I dont think this is an blocker ( other than it blocks releng from creating the live cd ) 17:33:17 so I'm a -1 on this one 17:33:29 Viking-Ice: how did you get live cd? 17:33:36 You want the PAE kernel if you have more the a GB of memory. 17:33:45 Indeed 17:33:53 more than 4GB. 17:33:56 jlaska, long time ago which did not boot btw 17:34:01 (well, more than 3GB, really.) 17:34:03 I am not sure this is an alpha blocker. 17:34:13 liveuser gdm problems 17:34:16 what's the impact of this - we don't get live images, or we get live images but only with the regular kernel? 17:34:19 I'm +1 17:34:22 no no ... 17:34:25 Inefficiencies hit well be fore 3 GB. 17:34:29 I thought that this was just the installer image 17:34:35 it would mean we have install ISO media that doesn't contain kernel-PAE (package or pxe images) 17:34:43 yup -1 17:34:47 to a blocker 17:35:09 do systems that need PAE function when using the vanilla kernel? 17:35:11 I think it comes down to how important it is to have PAE 17:35:16 agreed 17:35:19 yeah, -1 for me; you can install kernel-PAE after installing the system, and not having a PAE kernel isn't going to prevent anyone installing and booting 17:35:23 jlaska: yes. 17:35:38 what PAE does is allow a 32-bit system to address memory beyond the 4GB range 17:35:59 that's what I was unclear on ... whether the lack of this installer kernel would prevent Alpha install 17:36:08 using a non-PAE 32-bit kernel on a system with more than 3GB or so of memory will lead to less than the full amount of memory being available to the kernel, but that's all, it has no other impact 17:36:11 I don't know that we'd want final to go out without the PAE kernel. You can't do fire and forget updates to fix this. 17:36:13 no, it should not. 17:36:25 You need to manually install the PAE kernels. 17:36:33 brunowolff: for final it's a more interesting question, but for alpha it seems pretty obviously not a blocker to me. 17:36:59 definitely for final it would be a blocker but for alpha nope 17:37:11 from my pov 17:37:11 -1 alpha blocker 17:37:20 should we re-propose this for another milestone then? 17:37:22 -1 alpha blocker 17:37:27 * jlaska works up the #agreed line 17:37:40 Ok, -1 for Alpha, +1 for Final 17:37:41 I think it should be proposed as a final blocker. 17:37:45 if anyone thinks it should be a final blocker they can re-propose it for final 17:37:51 i can do that when i update it 17:38:04 revisit at beta 17:38:08 #agreed 722466 RejectedBlocker - Does not prevent Alpha installs and can be manually installed after anaconda. May consider as a F16Blocker 17:38:31 Yay 17:38:33 anything else to cover on this bug? 17:39:08 No more bugs, and it took only forty minutes 17:39:23 we finished the adamw list of bugs ... lemme double check my updated wiki too ... 17:39:48 That's a good start. But I doubt they'll stay this short as testing ramps up. 17:39:55 :) 17:40:05 * tflink has a general question before we end 17:40:06 right, we'll probably get more once we have testable install images 17:40:17 it's almost certain 17:40:27 and remember too, we haven't frozen yte 17:40:32 so all this could change tomorrow 17:40:42 * j_dulaney shudders 17:40:43 queue ominous thunder 17:40:57 #topic Open Discussion - 17:41:00 open floor time 17:41:15 * jlaska looks better now ... http://fedoraproject.org/wiki/Current_Release_Blockers 17:41:34 has anyone had a chance to look at my proposal to keep these meetings from going to 4 hours? https://fedorahosted.org/fedora-qa/ticket/221 17:42:12 * jlaska still processing 17:42:17 adamw has, obviously but I'm wondering if there are thoughts on whether this would actually work and be worth doing 17:42:41 I'm thinking that the total amount of work needed wno't change ... it's where do we want to move that work (into another process etc..).) 17:43:02 lemme ask ... 17:43:08 yeah, pretty much 17:43:09 what's the ideal outcome for this entire process? 17:43:37 #topic discussion on reducing blocker meeting length 17:43:41 Still cover each bug, but reduce time spent on the bugs -inmeeting 17:43:42 for this part? to discourage 4 hour review meetings and reduce the total number of person-hours spent on gathering initial info (people * time) 17:43:49 #link https://fedorahosted.org/fedora-qa/ticket/221 17:44:00 tflink: take it further ... 17:44:16 * j_dulaney thinks dropping into the bug reports exactly what criteria are blocked 17:44:18 the meeting feels like the last line of defense 17:44:37 so what ideal behavior are we hoping for to reduce meeting time? 17:44:39 if we broke up the initial information gathering process, we wouldn't have to sit here on IRC for every bug asking about impact, workarounds etc. 17:45:39 its possible that we could extend the process to have a pestering-bot on bz to ask questions on every new proposed blocker/nth 17:46:03 but I wasn't planning on that for right now 17:46:21 I like trying to ensure that bugs answer some basic questions ahead of time ... we sometimes get that now (not always though) 17:46:44 I think you or adamw raised concern about missing important bugs because they didn't have their "paperwork" complete 17:47:02 not at all what you're trying to convey I know ... I'd just be worried it come off that way 17:47:05 yeah, the SOP does actually ask that you note what criterion a bug impacts when you nominate it as a blocker, but obviously that doesn't always happen 17:47:05 yeah, that is the difficulty of trying to review only the "processed" buts at the meeting 17:47:18 an easy fix to that is we always discuss every bug at the meeting 17:47:27 so we would still have to go over all the bugs at the meeting 17:47:27 the advance processing is entirely optional, but if we do it, it helps out the meeting 17:47:52 but if the process is successful, we would have less need to delegate pestering during the review meetings 17:47:53 adamw +1 17:47:54 incent the correct behavior? 17:48:09 you get imaginary awesome points for filing out the checklist ahead of time 17:48:17 If we want people to attend the meetings, we need to keep the time needed to attend reasonable or we will just end up with a few die hards attending. 17:48:58 Being able to quickly make decisions on bugs will help speed things up. 17:49:58 brunowolff: I don't recall huge droves, ever. 17:50:09 hmm,m this is a tough nut to crack 17:50:27 I don't imagine we'll solve it right now ... but I think it's good that tflink raised it to get ideas flowing 17:50:27 certainly however our meeting time is always tied to the amount of bugs we have to process and that we can not change thus if the meeting takes 4+ then it takes 4+ hours 17:50:47 the only option there would be to have multiple weekly meetings 17:50:51 well, i think we can overplay how badly things work now. i actually think it works pretty well. the numbers we have are quite nice, really. 17:50:53 same total time, less time permeeting 17:51:04 it'd get pretty unwieldy if we had 10+ people showing up every time... 17:51:19 in terms of too many people on IRC talking at once? 17:51:27 adamw: I think that the process works fine. I just don't like the 4 hour meetings :) 17:51:46 tflink, comes with the territory ... 17:51:49 #info Still searching for ways to reduce blocker meeting length 17:51:59 another hope of mine is that there could be more async conversation in bz if more were processed 17:52:07 YES! 17:52:14 ie get them solved outside the meeting 17:52:23 perhaps we should talk to a subset of active developers for what they'd like? 17:52:27 yup that's the only way to solve that problem 17:52:30 do they like that we pull them into this meeting 17:52:40 perfect questions in bz, or private pings/emails etc... 17:52:55 keep the process open 17:52:57 that's what this is all about right, trying to prioritize bugs for developer attn 17:53:14 tflink: one simple idea is just: go ahead and start doing it 17:53:20 heh, that's truee 17:53:30 adamw: wait, you mean do work? :-D 17:53:34 there's absolutely nothing to stop us doing things in blocker bug comments outside of meetings; right now we just don't do it much 17:53:47 I figured I could get it started but thought that I would ask for thoughts before I did 17:53:49 so...go ahead and start posting questions and things in blocker bug comments, and we'll see how it goes 17:54:08 no need to use any 'structured' stuff like keywords / whiteboard for now, keep it simple and if it works we can start formalizing it 17:54:19 another thought ... if the reporter/tester and the developer all agree something is a blocker ... there's no need to discuss it 17:54:24 just mark it blocker and move on, right? 17:54:24 good point on the formalization 17:54:40 adamw: wait we do this all the time so let's first set the criteria on how we can measure if we are actually improving things 17:55:10 do we want to provide a avg time per bug measurement at the end of each meeting? 17:55:13 Viking-Ice: mean time/bug during meetings? 17:55:28 shouldn't be too hard to parse that from the meetbot logs 17:55:47 would be nice for meeting to just spit out ... "Average time per #topic" 17:55:48 jlaska: hum, i'm not entirely sure that holds: i think there've been cases where the reporter and developer haven't entirely understood the blocker definitions 17:55:56 adamw: so? 17:56:12 some baseline of measuring we are very good at just starting doing stuff without actually having something to compare them with if they are doing good or bad 17:56:13 jlaska: so, they could both think a bug is a blocker but it might not actually be one 17:56:15 I mean, that's not exactly correct ... but do we care in cases like that 17:56:40 possibly, yeah. what if the dev doesn't manage to fix it in time? then we'd slip for a bug that shouldn't be a blocker... 17:56:44 adamw: the concern would be that fixing that "blocker", might introduce other problems? 17:56:51 no no 17:56:52 that too 17:56:59 we always monitor approved bugs for progress 17:57:04 I don't think there's a way around that 17:57:24 how would that be an improvement over what we already have? 17:57:33 wouldn't we still end up reviewing the validity of the bug? 17:57:34 but do we need to inspect *every* blocker decision where the developer + tester/reporter agree already 17:57:43 just doing it as an accepted blocker instead of proposed? 17:57:49 Viking-Ice: i think in this case it should be pretty obvious to measure: if tim starts posting questions in bugs and no one replies, that's obviously fail. if we all start enthusiastically discussing things in comment threads and blocker meetings suddenly get faster, that's success. =) 17:58:08 * tflink starts sharpening his poking stick 17:58:35 tflink: yeah, it does move it to the accepted list, but only involves time in this meeting should the bug not be resolved in time 17:58:43 then we would need to reevaluate the deciion 17:58:55 good point, I hadn't thought about that 17:59:00 * jlaska thinking outloud ... it's always been slightly annoying to second guess *every* proposed bug 17:59:12 I think we need to push some trust out 17:59:15 if its fixed before the release, it wouldn't matter so much if it were 100% valid 17:59:36 getting past code freeze could be an issue, but I'm not sure that risk is enough not to try 17:59:41 hmm, yeah 17:59:47 jlaska: I don't think that it should be too hard to see if it fails a release criterion 18:00:15 agreed 18:00:22 is this a new activity for the bug zappers community ... monitoring blocker submissions? 18:00:35 * jlaska stops for now ... otherwise I'll contribute to a long meeting again! 18:01:03 either way, I'll start montoring new proposed blockers and we'll see what happens 18:01:14 do it 18:01:33 to it 18:01:42 tflink: thanks for raising this here 18:01:46 * j_dulaney must go, time for noms 18:01:51 okay gang ... any other items to discuss? 18:01:57 * jlaska sets fuse for *1* minute 18:01:59 Actually, time to catch the bus to get noms 18:02:01 Peace 18:02:05 thanks, cya j_dulaney 18:02:11 #topic Open Discussion - last call 18:02:35 30 seconds .. 18:02:41 * jlaska hums jeopardy theme 18:02:59 okay, thanks for your time everyone! 18:03:04 it's almost as great as the hold music. 18:03:05 Really. 18:03:11 let's find some more blockers between now and next Friday 18:03:16 I wonder if alex feels that way when he hears the jeopardy theme :) 18:03:20 and of course, escalate them with care *and* criteria :D 18:03:30 rbergeron: love that hold musak :) 18:03:32 #endmeeting