18:08:33 #startmeeting Fedora Release Engineering 18:08:41 #topic roll call 18:08:56 * stickster 18:08:56 here 18:09:02 ping: notting jeremy jwb lmacken wwoods warren dgilmore poelcat rdieter 18:09:03 * jlaska 18:09:10 * poelcat 18:09:17 also ping those people who are here for the go/nogo 18:09:20 * notting is here 18:09:28 releng.. hmm. 18:09:30 * jwb is not here today, sorry 18:09:50 * onekopaka will be here. 18:10:12 0xf13, jeremy is on vacation 18:10:27 yeah, that's what i'm here for 18:12:02 brb, have to carry a box out side for my wife. 18:12:58 * warren here 18:13:02 but sick 18:14:45 sorry about that. 18:14:50 #topic old business 18:15:00 lets get through some of our old business before we hit the go / no go stuff 18:15:31 #orphans (old business) 18:15:33 er 18:15:37 #topic orphans (old business) 18:16:10 there are only two orphans left with deps on them, cryptix and tomoe. I keep getting told that tomoe will be picked up by the translation team, but it hasn't happened yet. WIll send nag amil 18:16:22 cryptix is about to be recursively removed. 18:16:24 I'll warn. 18:16:30 i saw some tomoe changes in koji recently i think? 18:16:43 #action Oxf13 will send warning about recursively removing cryptix today 18:16:54 adamw: there may have been koji action, but pkgdb still has it listed as orphaned 18:17:11 #topic critical path (old business) 18:17:19 There were 2 action items here. 18:17:28 one was for me to package offtrac and get the tag-request target working. 18:17:34 * dgilmore is here 18:17:46 offtrac has been packaged, and I just submitted a tag request for rawhide. I have changes pending to makefile.common to add the target 18:18:07 and there is a build of fedora-packager pending I do believe that adds the 'fedora-hosted' tool which is used with offtrac to talk to our Trac 18:18:22 dgilmore: is there a build pending for fedora-packager? (just waiting for offtrac to land I'd imagine)? 18:18:38 Oxf13: it should be in dist-f12 now 18:18:52 ok, we'll have to make sure it gets tagged for f12-alpha 18:18:54 and in updates-testing for the rest 18:19:07 fedora-packager-0.3.7-1.fc12 is in alpha right now 18:19:30 http://koji.fedoraproject.org/koji/buildinfo?buildID=126049 18:19:36 * daMaestro lurks 18:19:40 0.3.8 is what we want 18:19:44 ok. 18:20:01 #action Oxf13 will make sure the latest fedora-packager gets tagged for Alpha so that the tag-request make target can be used. 18:20:13 the other action item here was for lmacken to report on any bodhi changes for critical path. 18:20:32 I don't see him here so lets move on. 18:20:46 #action lmacken still needs to report on critical path bodhi updates 18:21:15 #topic rpm changes (old business) 18:21:24 i suck still 18:21:25 there was an action item here for dgilmore to draft an SOP on rpm changes 18:21:27 ok. 18:21:42 #action dgilmore will draft an SOP (with help) for managing RPM changes in rawhide 18:21:43 ill do it this arvo 18:21:56 #topic no frozen rawhide (old business) 18:22:15 There were 2 actions here. 1 a ticket in bodhi for changes, which lmacken filed. 18:22:35 two was reporting to FESCo that the changes were punted to F13. I have reported as such. 18:23:01 #topic signing (old business) 18:23:19 smooge attempted to get sign-vault1 rebuilt for us, but ran into issues with the hardware raid. 18:23:34 we decided ot leave it as is and use it for signing during alpha and will attempt to rebuild later 18:23:36 yes. 18:23:43 I can rebuild next week 18:23:48 if thats ok 18:23:58 it would seem that it needs a local console 18:24:03 I still owe a lot of wiki writeups for the signing server, but those are delayed until after alpha gets done. 18:24:22 #topic Fedora 12 Alpha 18:24:35 and now the reason we're all gathered here today 18:24:40 18:24:56 i'm getting an award? 18:25:00 heh 18:25:07 adamw: a pwnie 18:25:08 adamw: yes, you are the best #7 doom player :) 18:25:10 adamw: you get extra work 18:25:10 so We were supposed to compose a release candidate last Thursday 18:25:32 unfortunately the F12Alpha blocker list was not clear, and attempts to clear it led to even more blockers on the list 18:25:52 it still isn't clear today, therefor it will not be possible to release per our scheduled release date 18:26:23 I'm recommending a week slip to finish fixing the remaining blockers and finish testing to discover any other potential blockers 18:27:07 *sigh* 18:27:11 * poelcat also adds recommendation to push slip through the entire schedule 18:27:14 Does anybody disagree with that recommendation? 18:27:19 and move GA by one week out 18:27:22 do we have confidence that the existing blockers can be fixed in the next three days? 18:27:33 notting: perhaps a denise question? 18:27:45 Oxf13: no critical path progress has been made in bodhi 18:27:48 poelcat: i'd tend to agree with the compressed timeframe already 18:27:54 qa agrees with the slip, for the record 18:28:05 there are only 3 bugs not in modified state 18:28:16 one is a traceback at end of install, and there is a patch up for review on that. 18:28:26 noone understands the mouse one yet 18:28:29 one is a missing X cursor and we've had a couple failed attempts at fixing this 18:28:33 and it will need more attention 18:28:39 there is an extremely bad work around possible though 18:29:06 the last is another anaconda issue, with partitioning 18:29:17 bugzilla is really slow for me 18:29:24 we do have patches for 516304 18:29:53 although 499854 (the partitioning one) may actually be fixed 18:29:59 looks like the tester was using old stage2 18:30:38 i just ask, because even though we say 'we slip a week', we're not adding a week from today to do fixes, are we? 18:30:42 all the modified ones could stand retest, and I'm not sure what more lurks - reviewing for likely blockers but slow 18:30:49 right 18:31:00 retesting is blocked on the other issues preventing a newer anaconda build from hitting rawhide 18:31:15 denise: we are planning to go back and validate MODIFIED bugs once we have something to test with 18:31:21 notting: yeah, I understand. 18:31:48 do we have a reasonable expectation we'll have a usable anaconda with the fixes in it landing in rawhide in the next few days? 18:31:55 hrm, actually we're in a bad place with 499854. Looks like joel had an updates.img that fixed the problem, but he went AFK before posting the patch for review 18:32:05 adamw: I do. 18:32:12 Oxf13, blocked by anaconda problems? 18:32:25 denise: yes. 18:32:28 Oxf13: can we have a newer anaconda tagged by end of day ... even if test blockers remain? 18:32:41 jlaska: if we get one that doesn't traceback at end of install, sure 18:33:00 Oxf13, clumens thinks he has a patch for that one 18:33:00 who is speaking for the anaconda team? 18:33:06 Oxf13: well, we aren't going to go to Alpha with anaconda-12.7 right, and having something newer would help us test MODIFIED bugs 18:33:07 poelcat: denise is 18:33:17 Oxf13: oh, i thought you were :) 18:33:45 denise: Yeah, somebody has to review it though, and then do a build, and then I have ot test that build to make sure no other regressions snuck in (again) 18:33:49 frankly even one that traces back at the end of install would be useful 18:33:55 at least we can test all the bugs that happen _before_ that point 18:33:58 adamw: agreed 18:34:07 I'd like a newer anaconda so we can have more eyes on it if possible 18:34:21 to find other bugs sooner, and try and work down the increasing list of MODIFIED bugs 18:34:29 I realize I'm asking for an anaconda that has known bugs 18:34:50 that's fine. 18:35:01 I'm just worried about what's lurking since last Tuesday's anaconda 18:35:09 #agreed A newer anaconda will be tagged today, even if it's known broken, to help with verifying other bug fixes. 18:35:20 if people complain ... feel free to blame me 18:35:39 jlaska: sure. The anaconda we got on Thursday was a non-starter. And by the time I got testing going on Friday, it was rather late 18:36:37 Oxf13: I hear ya, I appreciate trying to keep a stable anaconda landing for us. My spidey sense is starting to kick in, I'd like to try and unleash some testers on something newer too 18:38:06 jlaska: they could have been pointed at the boot.iso I posted on my people page 18:38:16 (and in fact, many were) 18:38:25 we explicitly avoid home grown boot media in our test plan 18:38:34 organic 18:38:42 free range 18:38:53 it's good for one off stuff, but we've been burned on that before 18:39:06 just like we've been burned verifying known broken things (: 18:39:14 because the fixes for the brokenness can often re-break something else 18:39:25 well, that's why we have QA 18:39:49 anyway, it's a fairly moot argument if we don't have anything new to test 18:40:29 denise: It'd be really good to get some pressure on fixing or attempting to fix or calling for help on the remaining blocker issues, so that we can have a build to test before too late tonight 18:41:18 Oxf13, 7 developers, 3 concurrent releases, and 5.4 in flames all last week - yes, they are trying, but pressure can be counter-productive 18:41:38 Oxf13, but they are definitely focused on blockers 18:41:47 denise: then I need a judgement call. Can we reasonably expect any work to happen in the next 3 days? 18:41:53 Oxf13, and i am looking to be sure we're not missing more 18:41:55 if not, how long do we need to slip for? 18:42:14 Oxf13, yes, f12 is top priority 18:42:30 Oxf13, 5.4 firedrill appears over 18:42:38 knock wood fast 18:43:27 ok, going back to my original suggestion 18:43:30 so IMHO, chances are good that 2 blockers will find fixes - unclear on mouse one, not yet understood 18:43:47 do we all agree that a 1 week slip in the Alpha schedule is called for here? 18:44:03 +1 18:44:12 I think at least 4 day slip 18:44:13 poelcat: I"d rather delay any full schedule changes until after we have a working Alpha, in case we have to slip again. 18:44:18 +1 18:44:24 jlaska: we previously agreed to do slips in 1 week segments 18:44:41 +1 to one week slip, +1 to discussing further schedule implications in a week 18:44:47 not sure I get a vote but +1 if so 18:44:48 Oxf13: i disagree... better to take the hit and reflect reality today 18:44:50 Oxf13: we always ship on a tuesday right 18:45:00 dgilmore: right. 18:45:17 * stickster agrees with poelcat, what's the big cost in updating schedule again later? 18:45:24 5 minutes of my time :) 18:45:33 poelcat: I'd rather discuss remaining schedule when we know how much time Alpha will actually take. 18:45:58 Oxf13: there is no reason to do that... we've already hammered out what we thought was a good schedule 18:45:59 What if we cascade that slip into the main schedule, and revisit if additional slips are needed? 18:46:07 especially since that needs to be a discussion with more people than what's in this channel. 18:46:50 the easiest and best way to create a predictable schedule we can all trust in is to adjust as we go... absorb and pray hasn't historically worked well for us 18:47:03 I'm not saying absorb and prey 18:47:16 I'm saying wait until we know if we have to slip ALpha again 18:47:28 all future dates are dependent upon when alpha goes out 18:47:38 so before we adjust everything, lets' see when alpha actually goes out 18:47:55 then revisit to see if the adjustments we have to make land cleanly and don't disrupt anything else for any schedule holders 18:48:16 * poelcat still doesn't see what advantage that gives 18:48:25 poelcat: 'only slipping final once'? 18:48:54 poelcat: A) we don't have the right people here for a full schedule shifting move. B) getting those people together is scheduling fun. C) doing it more than once is even more fun. 18:49:03 Oxf13: i agree with doing it that way 18:49:04 notting: no reason we couldn't commit to the slip now, and actually jiggle the schedule later. 18:49:15 Oxf13: who else do we need? 18:49:27 poelcat: the release readiness team? 18:49:27 poelcat: RHEL for one 18:49:38 Oxf13: RHEL???? 18:49:59 poelcat: we need to be sure that any adjustment in our schedule doesn't conflict with any of their schedules 18:50:17 ditto mirrors 18:50:33 we need to make sure any adjustments we make don't wind up landing multiple distro releases in the same timeframe 18:50:45 * poelcat notes RHEL needing to sign off as "new critieria" 18:51:05 poelcat: it's not about RHEL signing off, it's about us not scheduling conflicts 18:51:06 Oxf13: is this the infrastructure push issue? 18:51:31 notting: more like mirror push back. they get really really pissy if we try to do distro releases at the same time 18:51:38 and it deeply effects our ability to stage the release 18:52:11 poelcat: basically what I'm saying is that full schedule shifts aren't something that can just be decided on a whim, and we don't have the proper people/planning in place to vote on such things. 18:52:36 We can agree in principle to adjust the end schedule to match what we change at Alpha, but I don't think it's right to set those dates now. 18:52:42 Oxf13: I think this is all the more reason to keep mirrors and anyone else involved informed early and often 18:53:17 stickster: sure, we can inform them that we're moving alpha, and will likely move final release a week, pending a scheduling meeting. 18:55:11 that's a positive take-away ... this meeting is to determine go/no_go. We've determined no_go already. So now either in this meeting, or in a future meeting we ensure representation (Rel-Eng, Program Mgmt, QA, Devel etc...) and to discuss impact? 18:56:46 who needs to be involved and who needs to be informed of the outcome from that meeting? 18:57:10 jlaska: right. I think it's fair and reasonable to discuss how long we slip Alpha at this meeting, and leave full schedule impact to a future meeting (preferably after we have a GOLD Alpha) 18:57:41 jlaska: i think the groups in poelstra's release readiness meeting certainly need to be at least informed, if not directly involved 18:58:07 well "the world" needs to be informed 18:58:17 Is there a place where we document who needs to be involved to decide overall schedule slips? 18:58:31 * poelcat is confused on our approach... traditionally releng made a recommednation to FESCo 18:59:03 poelcat: I might not be taking into account how this has been handled in the past 18:59:04 i like the idea of involving the other groups where it affects them 18:59:11 but in this case idon't see how it does 18:59:31 poelcat: you don't see how changing the remaining schedule dates wouldn't effect other groups? 18:59:33 really? 18:59:49 Oxf13: all it does is give them more time. where is the risk? 18:59:56 Most of the groups in the release readiness meeting are only affected in that they have an extra week to apply somewhere in their schedule. For the most part they base that schedule off the final date. 19:00:13 * stickster has been in the majority of those schedule meetings so feels pretty confident in saying that. 19:00:32 poelcat: a lot of those groups have deadlines and schedule items. Any change in that should be vetted by them, to make sure it doesn't land on times where key people are unavailable 19:00:55 not to mention that we need time to vette the new dates of things landing, as mentioned before, for conflicts with other distros (including RHEL) 19:01:19 or for any events that may be taking significant members of our development team away 19:02:36 * poelcat thinks better to s/RHEL/resources|upstream since many have been flamed in the past for saying that RHEL == Fedora 19:02:54 maybe not RHEL == Fedora, butyou get the idea 19:03:53 Proposal: Slip alpha 1 week. Intend to slip final one week. Discuss schedule consequences at the planned release readiness meeting this Wednesday? 19:04:22 if we're slipping by a week, wouldn't we also slip the release readyness meeting? 19:04:30 as we can't possibly know if we're ready to release by Wed. 19:04:35 notting: +1 19:04:36 notting: hmmm...i was plannig to slip the meeting out by a week 19:05:06 I otherwise agree with notting's proposal, just strip the "this Wednesday" off. 19:05:40 the date's not a huge concern for me, but i think that's the right group of people 19:05:50 nod 19:06:18 unless we want to delegate discussion to FESCo, with notes to the groups to make sure they're around for that meeting 19:06:28 i think a quick mail to logistics list could solve problem of moving schedule 19:06:49 good point, that's the intended purpose of that list iirc 19:06:58 (or one of them) 19:07:07 +1 19:07:38 * jlaska tosses .5 ... adamw has the other .5 19:08:03 consider it tossed 19:08:11 (as the bishop said to the nun) 19:08:12 I'm +1 as well. 19:08:35 adamw: or the drunk fratboy to the midget 19:09:12 don't be ridiculous. drunk fratboys are never capable of speech. 19:09:12 +1 to notting w/ mail to logistics list asking for feedback/impact of slip 19:10:08 poelcat: what are your experiences, how hard is it to have a draft updated schedule available for review? 19:10:12 #agreed Slip alpha 1 week. Intend to slip final one week. Discuss schedule consequences at the planned release readiness meeting while using logistics list to gather feedback. 19:10:58 I would like to request that someone on the rel-eng team document how the slip decisions are made for future reference. 19:11:23 * poelcat volunteers to document 19:11:26 stickster: I guess you don't like "halfhazard" as a document... 19:11:49 It makes the raptors happy, that's for sure ;-) 19:11:58 The short answer "Blocker bug not clear by go / no go meeting" 19:12:26 incl. a list of RACI folks for good measure 19:13:10 stickster: wow, you really are in mgmt training! 19:13:11 yes, raci please :) 19:13:26 another point to note ... starting late last week I think we knew this was coming 19:13:27 skvidal: I have to have something to justify that yacht 19:13:52 we missed 2 interim Alpha milestones for QA ... I think that's a good indicator that "storms a'comin'" 19:13:56 jlaska: we could have had a miracle of a weekend and gotten an anaconda build over the weekend or early this morning that would have put all the bugs in MODIFIED state 19:14:13 jlaska: at which point we would not necessarily have been needing to slip, depending on the testing of an RC made today 19:14:35 so while we suspected it, it wasn't a foregone conclusion 19:14:37 Oxf13: if we had the RC today, that's not a lot of time for our team to get feedback before syncing 19:15:02 jlaska: 4 days? 19:15:12 * poelcat thinks we should move on 19:15:31 Oxf13: sorry, yeah past experience here contaminating my answer 19:15:33 jlaska: you do recall having the hidden second RC date 19:15:41 but yeah we can move on 19:16:05 * poelcat sees nothing "hidden" on the schedule :) 19:16:57 well, duh ;) 19:17:04 that's why it's hidden 19:17:16 i thought we were all about transparency :) 19:17:47 we're also all about not scheduling things which shouldn't be necessary 19:17:54 nor counted on or depended on 19:18:02 * jlaska confused 19:18:35 what are we talking about at this point? :) 19:18:47 Oxf13: oh oh, I think we had 2 candidate compose dates in there originally .. but removed one since folks don't like 2 candidates 19:18:52 Oxf13: is that what you meant? 19:19:10 jlaska: yes, that's what I meant. 19:19:13 gotcha 19:19:21 the date of the second RC would have fallen either today or tomorrow 19:19:41 #topic open floor 19:19:54 notting: whatever you want to talk about. 19:19:58 heh 19:20:02 wait, I think I missed some action items 19:20:11 was there anything anybody needed to /do/ based on that last decision? 19:20:18 we need to announce the slip 19:20:37 I guess I can do that. 19:20:44 #action Oxf13 will announce Alpha slip 19:20:57 and take the discussion to logistics@ 19:20:59 We need to post to logistics list to discuss schedule change. 19:21:03 1. Announce Alpha slip; 2. Announce intention to slip final; 3. Poll on logistics list for impact 19:21:06 poelcat: can you take that? 19:21:11 2 & 3 may be combined 19:21:24 Oxf13: yes, np 19:21:47 #action poelcat will announce intention to slip final and start discussion on logistics list to discuss impact of such slip. 19:21:49 in the past FESCo approved schedule changes... does that apply/no here? 19:22:05 poelcat: we may have reached the point where we don't need FESCo to rubber stamp our schedule changes. 19:22:31 well, we can take a vote of the fesco members that are here (whee, cheating!) 19:22:41 +1 :) 19:22:48 +1 19:23:01 +1 19:23:13 +1, and we should look at not having to rubberstamp moving foward, unless there is a reason for it. 19:23:18 nod 19:23:29 * notting looks at jwb/dgilmore 19:23:33 i did think we delegated in the F11 timeframe 19:23:35 ok, #agreed no need to vette the slip decision via FESCo 19:23:36 nirik: yes 19:23:37 wait, dgilmore already voted for the slip :) 19:23:47 not sure if the bot picked that up. 19:23:51 #agreed no need to vette the slip decision via FESCo 19:23:52 what? 19:24:01 oh, +1 for slip 19:24:11 aka "whatever Oxf13 said" 19:24:16 * jwb goes back to his hole 19:24:18 ok, we have a quorum of fesco 19:24:20 I'm not sure if there is any case where rel-eng/qa decide to slip that FESCo would ever override them... at least I am having a hard time thinking of such a case. 19:24:23 * jds2001 goes back to his hole 19:24:42 nirik: yeah, it makes no sense. 19:25:08 i can only think of fesco perhaps saying "you should slip more" 19:25:29 and that would be a really bizarre circumstance 19:25:33 * poelcat can add this to the document about scheduling too 19:25:36 jds2001: FESCo could use whips and say make it happen without :) 19:25:47 lol 19:26:06 i can think of one, but i'm not about to offer it in public 19:26:10 i change my vote! Oxf13 you didnt need that slip anyway! :) 19:26:29 alright, anybody else got anything for this meeting? 19:26:46 Oxf13: signing the alpha. yes, no, maybe? 19:26:52 notting: maybe 19:27:14 my sigul run sortof crashed part way through and skipped over to requesting written out rpms 19:27:23 and so a bunch got written out, but a bunch are still not even signed yet 19:27:28 so I restarted it, and it's still signing things 19:27:42 and I have no idea how far through it is 19:27:53 notting: I need to give you the key ID to put into mash configs 19:28:29 #action Oxf13 to deliver F12 key ID to notting for mash configs 19:28:30 ok 19:28:45 well, 'config'. we only package rawhide. 19:28:54 might need to deliver to lmacken for bodhi too 19:28:54 notting: 57bbccba 19:29:01 there that was easy 19:29:36 alright, if there are no other topics, I really need some lunch 19:29:56 Oxf13: just let me know when we have a final yes/no decision on signing, so we can frob deltas appropriately 19:30:31 notting: yes if they get written in time, no if not. Either way I think we should sign the repodata 19:30:47 Oxf13, uh, what? 19:31:02 i'd rather not do that until we figure out how we're going to do that for updates too 19:31:16 jwb: it's a compromise to not having signed rpms for alpha 19:31:26 (and a test case for doing it in updates) 19:31:28 Oxf13, i wasn't talking about RPMs 19:31:37 I know 19:31:40 and i don't see how it's a testcase at all 19:32:02 jwb: just like us turning on deltas in rawhide first, then turning it on for updates 19:32:23 we're going to do a one-off for alpha, see how that goes, then talk about automating it 19:32:35 see how it goes in what terms? 19:32:40 inside yum? 19:32:46 when configs are turned on to use it, does shit go boom 19:32:54 yum, packagekit, etc.. 19:33:13 so we've already tested that on a smaller scale right? 19:33:18 smaller yes 19:33:35 damn. late for Real Life meeting 19:33:44 late for Real Life Lunch 19:33:45 i slight disagree still but haven't time to debate 19:33:53 jwb: ok, we can debate later 19:33:58 * jwb falls back to 'whatever Oxf13 says' 19:35:17 #endmeeting