18:59:18 #startmeeting F-13-Alpha engineering readiness meeting 18:59:18 Meeting started Thu Feb 25 18:59:18 2010 UTC. The chair is jlaska. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:59:19 Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:59:40 #topic gathering devel, rel-eng and qa reps 19:00:13 hello! 19:00:19 Oxf13: ping 19:00:37 who do we have on the devel side ... notting around? 19:01:00 * stickster not on devel side but here 19:01:01 * Oxf13 sets down the bottle of booze 19:01:09 Oxf13: pick that sucker back up! 19:01:14 oh, i wouldn't let that get too far away if i were you 19:01:15 and pass it around too 19:01:35 Alcohol: Proud Partner of the Fedora Development Process 19:02:02 notting isn't available for the meeting ... anyone else want to represent devel 19:02:15 notting lurks 19:02:35 The notting knows.... 19:02:43 I could dual role 19:02:53 but maybe we could drag clumens in here 19:02:53 * nirik also lurks. 19:03:08 nirik: howdy 19:04:02 welcome dlehman 19:04:26 sometimes I regret turning off join/part 19:04:31 perhaps someone from desktop would like to represent devel folks? as they are the critical path? 19:04:32 and Mr. Lumens 19:05:52 i guess i count as desktop 19:06:16 alrighty, let's get started then 19:06:24 #topic intro 19:06:43 I'll keep this brief ... but why are we here? 19:07:04 Going by https://fedoraproject.org/wiki/Engineering_Readiness_Meetings .. 19:07:38 #info The purpose is to decide whether the alpha has met the release criteria 19:07:55 since we'll probably reference this link quite a bit ... 19:07:56 https://fedoraproject.org/wiki/Fedora_13_Alpha_Release_Criteria 19:08:12 mclasen would be a better representative but he's offline atm. might be on in a bit. 19:08:33 alright ... so we've got folks from devel (notting, ajax, clumens, dlehman), rel-eng (Oxf13) and qa (adamw, jlaska) 19:08:45 now the fun part ... 19:08:54 desktop is here too, ajax, dcbw 19:09:02 #topic Go or No Go 19:10:03 So where do we stand with the Alpha release criteria 19:10:14 so lets go over what works, what doesn't work, and what's unknown 19:10:17 we've got OPEN blocker bugs -- https://bugzilla.redhat.com/showdependencytree.cgi?id=538273&hide_resolved=1 19:10:27 Oxf13: yeah, we'll hit that 19:10:29 on the actual 'official' build we have, rc3, we have multiple blocker bugs 19:11:02 traditional x86-64 install is broken, live install is broken, installing updates is broken 19:11:24 we have somewhat-tested fixes for all three issues, but we do not have an anaconda build which incorporates the first two yet 19:11:39 #info traditional x86-64 install is broken (bug#568235) 19:11:41 and we certainly don't have a full set of candidate images which incorporate the fixes, never mind have run the full test matrix on such images 19:11:44 #info live install is broken (bug#565840) 19:12:12 #info installing updates is broken (bug#568193 bug#567346) 19:12:54 that would imply that the 'go/no go' decision is obvious. 19:13:00 *nod 19:13:03 For those watching at home ... the current RC3 test results are located at https://fedoraproject.org/wiki/Test_Results:Fedora_13_Alpha_RC3_Install 19:13:27 and https://fedoraproject.org/wiki/Test_Results:Fedora_13_Alpha_RC3_Desktop 19:13:37 notting: it'd be more obvious if we didn't have fixes in hand for all known issues 19:13:38 adamw: ah, thx 19:14:47 Oxf13: I don't want to slip either ... but we've already delayed this meeting once, and the purpose of this meeting was to be the "Where are we now" check point 19:14:55 so we don't keep waiting, and waiting 19:15:16 right, I'm just adding input 19:15:26 Oxf13: right on 19:16:00 Oxf13: and adamw: have been closer to verifying those fixes 19:16:01 i'm not comfortable with just getting an anaconda build and spinning up an rc4 and saying 'oh it's probably fine' and shipping it...and i can't see any other way we could get in under the wire 19:16:10 so unfortunately i can't see much choice but slipping at this point :/ 19:16:47 we touched /sbin/loader ... I think we should enact "/sbin/loader rule#3 - don't do touch it again" 19:17:17 tell me more about the loader touch. Was it to fix another bug, or was it something else? 19:17:33 it was to address a bug filed during earlier rawhide acceptance testing 19:17:50 bug#563009 19:18:54 ok, so that wasn't a blocker bug 19:19:08 no, I don't believe so 19:19:45 .bug 563009 19:19:47 stickster: Bug 563009 No logging type for "loader" messages - https://bugzilla.redhat.com/show_bug.cgi?id=563009 19:20:05 stickster: ah, so that's the new zodbot incantation 19:20:18 ok, good to know. 19:20:30 alright, so I think we have a no go from QA (adamw and jlaska) 19:20:52 Oxf13: any thoughts/concerns/comment from rel-eng? 19:20:57 well. 19:21:42 I'm really reluctant to say nogo, because we're so close. And because it feels like a big fat failure after all the adjustments and work we put in for earlier feature freeze, NFR, etc.. 19:22:04 I agree that we don't have much data to provide confidence in the fixes at this point 19:22:47 and I think what will likely happen is we'll make RC4 today, have it pass testing and sit there for 10~ days 19:23:11 i agree that's the most likely case, but the chance of something else happening is substantially non-zero... 19:23:43 right. 19:23:55 I'd even offer not spinning RC4 until we have confirmation on these remaining issues 19:23:57 while it sits there, updates-testing and branched will move on 19:24:09 Oxf13: so there's something we couldn't offer in previous releases, right? 19:24:16 jlaska: we've got conformation on everything but x86_64 booting 19:24:18 * stickster feels Oxf13's pain, and moreover, that the release criteria and these meetings are a great step forward in removing doubt and making it so we can move on in an objective way without blame 19:24:23 updates-testing and rawhide will go on 19:25:03 * stickster wants to recognize the superhuman effort of rel-eng and QA to try and get us over the line in time 19:25:04 It also sucks because technically we have enough time to spin the isos, validate the wiki and post the isos for download 19:25:26 but we've moved our decision meeting back earlier than the amount of time it takes to complete those tasks 19:25:59 Oxf13: If I understand though, after this meeting, there are dependent tasks that make it difficult to reverse the decision should testing yield alpha failures? 19:26:30 jlaska: I believe that was the reasoning at least for final release. I don't know for certain if that's the case for Alpha/Beta releases 19:26:42 It's a decision point. Hurts when you know things could change, but that doesn't remove the need for having one. 19:27:02 "If we only had..." :-) 19:27:06 i think the question is whether the decision point may be too early for alpha/beta releases 19:27:07 given that this is the go/nogo point, releng would have to vote nogo, even though it hurts. badly 19:27:17 * stickster pats Oxf13 on the back 19:27:22 * adamw hands oxf13 the bottle back 19:27:27 and even though we could see all green by tomorrow 19:27:31 adamw: hey, you broke the circle 19:27:39 Who's bogarting the bottle 19:27:51 jlaska: his need is greater than ours 19:27:57 indeed 19:28:22 alright ... anyone from devel care to weigh in? 19:28:39 it's hard to test if installation doesn't work welll 19:28:42 hopefully one person, but we have multiple people representing ... clumens: dlehman: ajax: notting: dcbw 19:29:09 i have nothing good to say. 19:29:18 Oxf13: hand the bottle to clumens next please 19:29:18 jlaska: from the standpoint of the current data/matrix, i don't see how we can say go 19:29:28 from a desktop point, #568193 not being especially tested yet makes me side with no go. 19:29:37 .bug 568193 19:29:38 stickster: Bug 568193 updating fails with a fatal error - https://bugzilla.redhat.com/show_bug.cgi?id=568193 19:29:51 dcbw: that's why you're supposed to isntall from the last known good point and yum update 19:30:04 ajax: oxf13 and I have positive results with latest packagekit, but we don't have an official image with it yet. 19:30:07 so that we remove the mental block of "installer doesn't work, abandon all hope" 19:30:24 jlaska: Should we #info the various votes to make the minutes sensible? 19:30:24 Oxf13: right, that can happen in parallel with installer testing of course 19:30:31 stickster: you beat me too it 19:30:49 Oxf13: but in the end, if we can't make installation work well, we can't ship it until we have a week at least of full-on testing *after* the installer works well 19:30:55 * stickster just playing back seat driver as a frequent Reader O' Minutes 19:31:01 since tons of people install with the livecd 19:31:42 #info QA votes 'no go' - citing additional testing against remaining F13Alpha blockers and walking the Alpha test matrix 19:32:19 i guess i don't see why, if we have such confidence in the pending fixes, we'd wait until 2 weeks from today 19:32:28 #info Rel-Eng votes 'no go' - reluctant considering how close we are and that the next RC will likely be the final 19:32:36 dcbw: I think we were more looking for data on userland stuff after the install 19:32:43 ajax: we're tied to Tuesday releases 19:32:52 ajax: and thus if we're no go, we slip a week 19:33:03 which means not this tusday as originally planned, but next tuesday 19:33:19 k 19:33:44 ajax: if you notice we haven't done much x86-64 installer matrix testing at all 19:33:52 ajax: since one of the bugs that's only jsut been resolved broke x86-64 install entirely 19:34:03 so we have no verification that all those tests pass on x86-64, even with the *initial* fiox 19:34:07 clumens: dlehman: You guys are machines. Thank you for putting so much sweat into trying to get things in working order. 19:34:13 I'd ask what x86_64 testing happened prior to RC3 19:34:16 yeah, big thanks to anaconda guys 19:34:26 since it was RC2->RC3 that broke x86_64 19:34:26 #info Devel votes 'no go' - notting cited the current data and matrix results, dcbw cited importance of installing 19:34:47 Oxf13: the only way I wouldn't retest x86_64 against RC4 ... is if we shipped RC2 19:34:49 agreed w/ no-go 19:35:04 i have a 64-bit machine running f13 (updated from f12); as far as post-install goes, it seems pretty solid. 19:35:18 jlaska: that's not what I asked 19:35:21 Oxf13: but I agree that having previous test data for that architecture puts us in better shape than being in the dark 19:36:03 slipping sucks, but if we slip we can get the x86_64 matrix completed, no? 19:36:18 dlehman: correct 19:36:42 Oxf13: RC2 install testing - https://fedoraproject.org/wiki/Test_Results:Fedora_13_Alpha_RC2_Install#Test_Matrix 19:36:47 it's one week. may not even need to slip beta. 19:36:47 dlehman: we should be able to get that matrix complete tonight/tomorrow 19:36:59 Oxf13: tomorrow 19:37:11 at best 19:37:26 Oxf13: RC1 install testing - https://fedoraproject.org/wiki/Test_Results:Fedora_13_Alpha_RC1_Install#Test_Matrix 19:37:48 dlehman: I would think so 19:37:56 Oxf13: the test composes and acceptance runs are there too .. https://fedoraproject.org/wiki/Category:Fedora_13_Test_Results 19:38:11 ok, looking at the RC2 red x86_64 results, those are issues which are confirmed fixed on RC3 i386 19:38:17 and they weren't x86-64 specific 19:38:39 Oxf13: what are you suggesting? 19:38:53 dlehman: The procedure from here is that the release slips by a week, IIRC: https://fedoraproject.org/wiki/Engineering_Readiness_Meetings#Contingency_Plan 19:38:58 I'm suggesting that it's extremely unlikely we'll find an x86_64 specific issue, once we get past the booting 19:39:11 ah, ok 19:39:16 Oxf13: oh yeah, I agree with that 19:39:18 stickster: that contingency plan was made before we had no frozen rawhide 19:39:26 stickster: in which rawhide would stay frozen if we slipped 19:39:32 stickster: which is not the case any more 19:40:22 Oxf13: Was that discussed at the NFR conference? I don't remember 19:40:33 this particular wrinkle wasn't 19:40:46 well ... we're slipping ... so should we move on to contigency plan? 19:40:47 mostly because I forgot that this would be a benefit of NFR 19:40:57 jlaska: sure 19:41:08 Oxf13: sounds like you might have other options/ideas there ... 19:41:15 #topic contingency plan 19:41:35 #info https://fedoraproject.org/wiki/Engineering_Readiness_Meetings#Contingency_Plan indicates a 1 week slip as a contingency plan 19:41:47 Oxf13: take it away 19:41:50 traditionally we've struggled with the decision to slip all further milestones or not if we silp a release 19:42:28 often we've sided on slipping everything, because slipping a alpha or beta increased the amount of time trees were frozen and inhibited further bugfixing and testing 19:42:33 limiting developer time 19:42:44 however, with NFR, that's no longer the case 19:42:50 rawhide itself is still wide open 19:43:01 F-13 builds can be published to updates-testing 19:43:09 and tested there 19:43:11 but ... the milestones do offer a line in the sand that draws people out to test 19:43:23 where they might not come download isos before then 19:43:49 but NFR does seem to open the door for more test options than we had before 19:43:49 so by slipping alpha, we're not actually taking away any development time from the developers 19:44:07 aiui oxf13 is just pointing out that now there's no particular need to slip beta and final 19:44:11 just because we're slipping alpha 19:44:33 yes, I'm proposing that we do not adjust beta/final dates. 19:44:41 Because content hasn't stopped dead while we wait to get things composed and then unfrozen? 19:44:48 stickster: that is correct 19:45:02 Thanks, "I'm just a simple caveman" to quote a friend :-) 19:45:14 there are not only one, but two pressure release valves that have been left open while we get Alpha out the door 19:45:43 Right, crazy crap -> Rawhide, controlled, careful... uh, stuff -> F-13-updates-testing 19:46:09 even more controlled/tested content F-13-base ? 19:46:27 yes, we can and will still approve things in bodhi into the branched repo 19:46:33 things not of crit-path nature will just make it in 19:46:43 things of crit-path will not make it in unless releng/qa says so 19:47:03 Which, supposedly, people will be able to grab as part of a routine first-session update run in the Alpha 19:47:12 stickster: yes 19:47:18 * stickster wondering how that affects Alpha test matrix? 19:47:21 stickster: or even during alpha install 19:47:29 I guess no different than previous teset releases 19:47:31 *test 19:47:34 right 19:47:41 there has always been a flood of updates for milestone installs 19:47:47 only this time the flood is more controlled and tested 19:48:06 Kinder, gentler firehose ;-) 19:48:45 Go on, my caveman questions are answered 19:49:07 I feel like there is still something to the milestone announcement that draws people out to download/test where they may not have with nightly images. I'd be worried if we had multiple slips and how that might impact the feedback we want. 19:49:25 I'm not sure I follow 19:49:26 not that I'm arguing we slip the whole thing, since Oxf13 you point out how NFR helps us out now 19:49:29 we're still going to have milestones 19:49:33 right right 19:49:36 we're still going to have plenty of change between the milestones 19:49:50 and one milestone will be a week later than originally intended 19:50:11 right, we reduce the test time leading up to the next milestone 19:50:28 for those subsets of users who only test the milestones 19:50:30 only that I think people wait for the bits to go out ... before they dive in 19:50:56 Oxf13: right, we have a lot of folks who more closely follow things are provide feedback near realtime 19:51:21 so just my worry ... if we also slipped the Beta .. or had to slip the Alpha again ... I'd want to revisit this 19:51:25 adamw: what's your take? 19:51:39 a week doesn't hugely worry me 19:51:49 * jlaska info's ... 19:52:06 jlaska: So what you're saying, if I understand you... 19:52:23 is it #proposal or #idea for doing a vote? 19:52:25 is that the contingency plan should always have the option of slipping everything a week, but it's not an automatic thing 19:52:32 #info Oxf13 notes that with No Frozen Rawhide (NFR), we no longer have the pressure valves from previous releases. Developing (rawhide) and testing (f-13-updates-testing) can continue 19:52:40 Oxf13: ah yes! 19:52:56 Oxf13: I think it's #idea? 19:53:26 #idea Do not slip Beta/Final milestone dates, due to NFR not shorting developers on development time. 19:53:31 I'm +1 obviously 19:53:38 thanks, you type faster ;) 19:54:13 +1 19:54:30 I agree w/ adamw, I'm up for seeing how that goes. But hope we can revisit should additional slips happen 19:55:11 any other takers? 19:55:49 Yeah I'd like to hear from the developers on this one 19:56:07 notting clumens dlehman ajax dcbw what say you? 19:56:46 Oxf13: sounds reasonable at a first glance 19:56:56 I'd like to note that more than a week here puts pressure on docs, translation, marketing, and other groups too. It's not just about bits getting out the door 19:57:19 So it's vital that we not consider this "It's OK that Alpha and Beta both slip up to one week" 19:57:56 I would almost want to err on the side of, "We can absorb the first one, all others come at a cost." 19:58:03 seems sensible, at least for this round. 19:58:06 +1 19:58:19 stickster: yes, that's what we've usually done, with grumbling from various crowds about shorting the developers time 19:58:30 #agreed decided to absorb the slip and not move the F-13 release out by one week (future slips may incur a cost) 19:58:57 alright folks ... anything else to decide here? 19:59:14 we have to assign some action items 19:59:50 okay, help me out ... what are the tasks? 19:59:50 #action Oxf13 will announce the slip to (devel-)announce 19:59:59 ah, gotcha 20:00:02 somebody has to update the schedule pages 20:00:07 Oxf13: I'll take that 20:00:09 which poelcat usually does 20:00:14 (and logistics, plz, to let folks know there isn't a readiness meeting today) 20:00:17 I can talk to poelcat (unless stickster wants it) 20:00:18 because he owns the taskjuggler generation 20:00:34 I can deal with the wiki, but not so sure about TJ 20:00:42 #action stickster will update schedule pages and poke poelcat about TJ 20:00:43 #action jlaska or stickster - talk to poelcat for task juggler schedule update 20:00:55 heh, an #undo will remove one of those 20:01:02 (the latest) 20:01:04 we need somebody to re-schedule the readiness meeting 20:01:06 #undo 20:01:07 neat 20:01:07 Removing item from minutes: 20:01:13 Oxf13: I can handle that too, I told poelcat I would 20:01:25 #action stickster will re-schedule release readiness meeting 20:01:25 +1 on not slipping beta 20:01:57 what else remains? 20:02:01 We should also communicate to RHEL engineering, in case they were using our Alpha as a milestone of sorts 20:02:16 stickster: want to take that one? (off the record as it were)? 20:02:23 Sure 20:02:36 Oxf13: I'll end up relaying your announcement, so it depends on your action 20:02:40 ok. I can't think of anything else. Just getting RC4 created and ran through 20:02:56 Oxf13: we'll obviously rinse and repeat on that 20:03:50 jlaska: I sincerely hope there isn't a rinse/repeat for RC5 20:04:04 #action QA - continue testing RC4 when available and verifying remaining F13Alpha bugs 20:04:12 Oxf13: here here 20:04:32 okay, shall we call it a meeting? 20:04:53 please 20:05:03 thanks everyone 20:05:04 my liver is late for an appointment 20:05:07 heh 20:05:09 #endmeeting