17:01:01 <Oxf13> #startmeeting Fedora Release Engineering
17:01:01 <zodbot> Meeting started Fri Jul 16 17:01:01 2010 UTC.  The chair is Oxf13. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:01:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
17:01:10 <Oxf13> #meetingname fedora-releng
17:01:10 <zodbot> The meeting name has been set to 'fedora-releng'
17:01:16 <Oxf13> #topic roll call
17:01:35 * brunowolff is here for GA and spins discussion
17:01:44 * jsmith lurks
17:01:49 <Oxf13> ping: notting poelcat rdieter lmacken spot wwoods brunowolff dgilmore
17:02:03 * poelcat 
17:02:05 <jsmith> dgilmore is speaking at FUDCon
17:02:55 * nirik is hanging around in the cheap seats.
17:03:32 <Oxf13> #info present are brunowolff poelcat notting jsmith nirik
17:03:47 <Oxf13> lets get to brunowolff's topic right off
17:03:54 <Oxf13> #topic Spins
17:04:02 <Oxf13> brunowolff: what have you got for us?
17:04:30 <brunowolff> Spins SIG hasn't done a very good job in the past for working with releng to get spins built for the GA release.
17:04:40 <brunowolff> Jesse usually ends up cleaning up after us.
17:04:54 <zodbot> Announcement from my owner (stickster): Fedora Board open IRC meeting in #fedora-board-meeting at 1800 UTC today (~1 hour from now).
17:05:12 <brunowolff> I'd like to talk about two issues, one is communication for the GA and the other is testing requirements for spins.
17:05:16 <rdieter> here now, hi.
17:05:42 <Oxf13> #info rdieter also present
17:05:48 <brunowolff> Maybe it's best to talking about testing requirements first since that has an impact on how we want to communicate.
17:05:58 <Oxf13> brunowolff: make sense
17:06:16 <brunowolff> The proposal is that spin owners or their delegates be required to do some testing of the Spin RCs.
17:06:53 <brunowolff> nirik and I were OK with that requirement in spins, and people in QA said about time.
17:07:22 <brunowolff> QA has some test plans that we can use, but no resources for doing the testing.
17:07:36 <nirik> ie, a positive 'I, or my delegate tested this [ x ]'
17:07:45 <brunowolff> If we do this we need to communicate to the spin owners when and where to get the RC spin versions,
17:07:59 <brunowolff> when and where to report back.
17:08:15 <Oxf13> yeah, the RCs can move fast and furious
17:08:21 <brunowolff> We need to know what to do if a spin's owner is MIA or if a spin fails the tests.
17:08:43 <Oxf13> given the amount of stuff we produce for a release, it is not uncommon for us to start spinning an RC, get some content posted, then find out it's busted and have to start over, long before we get to composing the spin content
17:09:04 <Oxf13> which is unfortunate for the spins as an RC for them doesn't get posted
17:09:06 <brunowolff> I don't want to require too much testing for this go around; even so I think there is a good chance that some spin owner will end up being MIA.
17:10:06 <brunowolff> based on what I have seen for RCs, things need to move pretty fast.
17:10:59 * nirik just wants to avoid a 'oops. I never tested this and it's broken and we already released it' thing again.
17:11:07 <rdieter> sounds like if testing doesn't occur or if spin owner is MIA, then spin needs to get dropped.
17:11:18 <brunowolff> If custom spins are going to need changes and still be posted with the rest of the release stuff the owners will need to be very active.
17:11:40 <brunowolff> Yes, dropped. But what does that mean for things like the release web pages?
17:12:02 <Oxf13> so I think the best thing to do re the testing, is to test the nightly image that is made from branched that day
17:12:09 <brunowolff> I suspect there is more than just not publishing the ISO.
17:12:21 <Oxf13> and if in creating an RC, I have to pull any extra builds in for the RC, I need to communicate that with the spin folks to see if that package effects their spin
17:12:27 <poelcat> what about building final spins at the time of final RC so there is time to figure things out
17:12:33 <poelcat> vs. after testing the final RC
17:12:54 <brunowolff> I think using nightlies make sense. The odds that other changes made that day to fix other things will break a spin seems small.
17:13:03 <poelcat> brunowolff: okay
17:13:41 <nirik> I could rebuild nightlies to pull in specific packages if asked I suppose... just need a local repo with the newer packages added in.
17:13:44 <brunowolff> It is certainly better than what we've done in the past.
17:13:53 <Oxf13> poelcat: I'm not sure I understand your suggestion
17:14:08 <notting> how do we determine that the owner is MIA, other than just the absence of testing feedback?
17:14:36 <poelcat> Oxf13: i understood that "final spins" get created after we declare the RC gold
17:14:36 <nirik> notting: I was hoping we would have a positive ack... ie, if no ack, they are assumed mia.
17:14:49 <brunowolff> I think someone (probably the wrangler) needs to tell them when to test and when to report back from tests.
17:14:49 <poelcat> Oxf13: so i was suggesting start earlier
17:15:02 <Oxf13> poelcat: I don't think that's right
17:15:13 <Oxf13> poelcat: if I have time while creating the RC, I create the spins
17:15:18 <poelcat> i understood the final spin signoff to be between the thurs we sync to mirrors and GA day
17:15:22 <notting> nirik: and therefore, it's dropped?
17:15:32 <Oxf13> but if we discover a problem with the RC before I get the spins created, then I stop composing, and work on the next RC
17:15:42 <brunowolff> They should already know about the nightlies, so pointing them there for testing is simpler.
17:15:49 <nirik> notting: yeah, might be too harsh tho. I'm a bit sad at how little involvement spins owners seem to have. ;(
17:15:54 <poelcat> brunowolff: do we have a formal "spins sign off date on the schedule?"
17:15:57 <Oxf13> so due to the nature of the sheer amount of things we have to compose and move around, the likelihood of getting the entire set of things built for each RC is pretty low
17:16:01 <brunowolff> Not yet.
17:16:25 <brunowolff> This is new, and after this meeting there should be information about what the requirements of the owners are.
17:16:26 <Oxf13> I would prefer it be at the same time as the RC signoff
17:16:39 <Oxf13> in case the spin requires some package update that would be on the DVD set
17:16:52 <brunowolff> I as the interim spins wrangler will communicate that requirement and make sure the or else is prominent.
17:18:25 <brunowolff> I envision that I will be communicating with the Spin onwers and that I would also be providing summaries or issues to releng.
17:19:11 <poelcat> brunowolff: best to just keep it on a wiki page where everyone can see what's in or out
17:19:19 <poelcat> including Oxf13
17:20:31 <brunowolff> Wiki for status is OK, though I expect to try to be available by IRC as well. I can probably take a day or two off work to help with this.
17:21:27 <brunowolff> If we use the nightlies for spin testing, which day's nightlies are we going to be able to use?
17:21:47 <brunowolff> Is it when we start RCs or after a final RC has been accepted?
17:21:54 <brunowolff> Or something else?
17:21:56 <Oxf13> when we start RCs
17:22:04 <brunowolff> Ok.
17:22:15 <Oxf13> basically I'll announce that we're in RC phase, and not taking any changes into the branched repo unless they are to fix issues found in the RC
17:22:16 <brunowolff> What happens if something is found to be broken?
17:22:31 <Oxf13> a blocker bug filed, and brought to the attention of releng
17:22:53 <Oxf13> we'll have to do some manual tagging to get the fixed package into branched instead of having it go out as a 0-day update
17:23:05 <brunowolff> What happens if the owner is MIA or a bug cannot be fixed before GA?
17:23:30 <notting> i would suspect it has to go out as an update , then
17:23:38 <notting> if it's really broken and the owner is MIA, we wouldn't ship the spin?
17:23:39 <brunowolff> I am guessing we wouldn't slip the release for just one spin (other than the main ones).
17:23:50 <Oxf13> yeah, I think that's the line we have to draw
17:24:04 <Oxf13> if it can't be fixed by the time the rest of the release is ready to go, it gets dropped
17:24:30 <brunowolff> OK. If for whatever reason we don't ship a spin, what else needs to happen?
17:24:41 <brunowolff> Does the design team need to be notified?
17:24:42 <Oxf13> website updates
17:24:44 <Oxf13> yea
17:24:50 <notting> and it depends on the level of the bug, of course. if it's 'random app on the spin crashes', we ship the spin and just do a zero-day update
17:25:45 <brunowolff> Oh, I forgot another timing issue, when do the spin owners have to acknowledge by? (Or be treated as MIA.)
17:26:48 <Oxf13> brunowolff: by the go / no go date I'd say
17:26:53 <brunowolff> OK.
17:27:01 <brunowolff> How do slips affect this?
17:27:03 <notting> i'd like to think we have an initial checkin at some point around beta or after, and then another one at RC phase
17:27:04 <Oxf13> no response would be treated as "no go" for the spin
17:27:19 <Oxf13> brunowolff: slips will push that go / no go date out
17:27:20 <brunowolff> Do a new set of tests need to be made? Do MIAs get an extension?
17:27:35 <Oxf13> brunowolff: and if we're slipping, that means we're pulling in some other packages, and will likely need to restart the test and confirm process
17:27:47 <Oxf13> yes, I'd say MIAs get an extension
17:27:56 <brunowolff> So that will be retest and you get an extension.
17:28:10 <Oxf13> if the package changed isn't on  a spin, I think we can re-use the previous "go" vote.
17:28:36 <Oxf13> and potentially re-use the previous iso set.
17:28:39 <brunowolff> So retest may not be needed. Only if the spin was broken.
17:28:51 <Oxf13> only if the package set changed and caused a need to re-spin the spin
17:29:38 <brunowolff> I think that covers the first part well enough and I'll be able to write that up.
17:29:59 <brunowolff> The second part is what do you guys think the minimum test requirements are?
17:30:29 <brunowolff> Which test plans must be executed and do both i686 and x86_64 have to have those tests done?
17:30:57 <Oxf13> I think that falls more on the QA side of the house
17:31:15 <Oxf13> I'm just happy if somebody boots both arches and makes sure the login screen comes up and people can log in
17:31:17 <rdieter> that should probably largely be up to the spin owners to help decide
17:31:37 <Oxf13> beyond that I think it is a matter of QA and a matter of the spin owner deciding what is critical functionality for the spin
17:31:42 <brunowolff> OK. I'll propose something to Woods and co.
17:31:45 <Oxf13> let me break it down this way
17:31:50 <Oxf13> releng requires that testing be done.
17:32:10 <Oxf13> QA should require a bare minimum acceptable testing in general for a spin
17:32:25 <Oxf13> the spin owners can add on top of that what they consider the critical functionality testing for the spin in question
17:32:57 <brunowolff> I already have pointers to test plans and I'll look those over and propose some minimum for this release and see what they say.
17:34:58 <brunowolff> I think that answers my questions for today. I'll need to write some stuff up and run it by people.
17:36:27 <Oxf13> ok, thanks for being patient with me and meetings brunowolff
17:36:58 <Oxf13> anything else from anybody on spins?
17:37:44 <Oxf13> #info Spin owners must check in with testing before the go / no go date or risk having spin dropped
17:37:49 <brunowolff> No problem.
17:38:05 <Oxf13> #info A slip of the release will reset the spin go / no go date as well and give MIA spin owners an extension
17:38:25 <Oxf13> #info In case of a slip, if the updated packages are not on a spin, a re-test will not be needed
17:38:49 <Oxf13> #info All releng requires is that testing be done.  The amount of testing is to be determined by QA and the spin owners
17:39:08 * dgilmore is here
17:39:26 <Oxf13> #info releng will communicate to spin owners when RC phase starts, and when new packages are needed beyond nightly compose for future RCs
17:39:39 <Oxf13> I think that captures most of what we covered today
17:40:44 <Oxf13> alright moving on
17:40:48 <Oxf13> #topic dist-git
17:40:54 <Oxf13> Lots of progress here
17:41:15 <Oxf13> #info Users can now use dist-git and fedpkg for cloning, committing, pushing, and building in koji.stg
17:41:36 <Oxf13> #info oxf13 feels that we are pretty close to required functionality for initial roll out
17:41:42 <dgilmore> there has been lots of bug reports which is good
17:41:50 * dgilmore agrees
17:41:59 <Oxf13> #info new package creation and branch scripts have been modified to work with dist-git, as has the lookaside cgi
17:42:30 <nirik> Oxf13: so thats just a modifcation to process-cvs-requests.py ?
17:42:31 <nirik> or ?
17:42:57 <Oxf13> process-cvs-requests.py doesn't call anything cvs specific
17:43:03 <notting> do we have an idea from the bug reports, checkins, or koji logs, just how many packagers are testing it?
17:43:04 <nirik> oh true.
17:43:05 <Oxf13> I think it gives you output to feed into pkgdb2branch.py
17:43:22 <Oxf13> so I've modified pkgdb2branch.py and setup-git-package to work with dist-git
17:43:30 * nirik played with it yesterday... everything worked pretty well for causal testing.
17:43:35 <Oxf13> notting: I could get some idea.  Koji db got re-set
17:43:47 <Oxf13> I could get some idea from the gitolite logs that is
17:43:59 <dgilmore> Oxf13: all the cvs stuff is handled outside of process-cvs-requests.py afaik
17:44:00 <Oxf13> as far as forward progress there are a couple things I want to work on
17:44:37 <Oxf13> I want to attempt using subdir branching, so that we have a top level F-13/ directory with a HEAD within, and subsequent branches of F-13 woudl live within that directory, like F-13/newversionfoo
17:44:56 <Oxf13> which would make detection of what base top level branch to send builds to easier and wouldn't require a "branch" file within the repo
17:45:00 <dgilmore> notting: id guess we have had more than a dozen users. how much more not sure
17:45:28 <Oxf13> it would make checking out the base F-13 branch slightly more complicated, git checkout -b F-13 --track origin/F-13/HEAD  or some such, but I can bury that in fedpkg
17:45:49 <dgilmore> Oxf13: sounds reasonable
17:45:51 <Oxf13> the other thing I need to work on is discovering which packages don't come over well with parsecvs
17:46:16 <Oxf13> kernel I know of, log4j I think is another one, I just need to run a script across all the repos to find the broken ones, and make sure they can use git cvs-import
17:46:22 <dgilmore> Oxf13: i wonder if we need to try clone them all.
17:46:24 <Oxf13> git cvs-import seemed to work for kernel, it was just a /lot/ slower
17:46:38 <dgilmore> or if we can work out a way to detect it on the git server
17:46:40 <Oxf13> dgilmore: I think a "git status" would be enough on each repo dir on pkgs.stg
17:48:02 <Oxf13> there is only 165 entries in the gitolite log, so that's not a whole lot of testing
17:48:03 <dgilmore> Oxf13: konversation fails to checkout
17:48:07 <Oxf13> but that's to be expected of our community
17:48:15 <dgilmore> if you want a example of bad to look at
17:48:39 <dgilmore> Oxf13: half of that is probably me
17:48:46 <Oxf13> [jkeating@pkgs1.stg konversation.git]$ git log || echo "broken"
17:48:47 <Oxf13> fatal: bad default revision 'HEAD'
17:48:47 <Oxf13> broken
17:49:15 <Oxf13> so yes, looks like we'll be able to use the repos directly on the git server to detect the problem ones.
17:49:25 <Oxf13> when is the next FESCo meeting?
17:49:29 <dgilmore> that will make it simpler
17:49:36 <dgilmore> tuesday?
17:49:41 <nirik> tuesday - 2010-07-20 at 19:30 UTC
17:49:41 <Oxf13> #info Need to detect broken conversions and attempt with git cvs-import
17:50:07 <Oxf13> #info will try directory based upstream top level branches for target discovery (eg F-13/HEAD F-13/coolbranch)
17:51:00 <Oxf13> #info We need to draft a roll out plan and present it to FESCo, complete with rollback plan
17:51:23 <Oxf13> thankfully it appears to be very easy to roll back if things go south
17:51:52 <Oxf13> #action We need people to start gathering a list of wiki pages that will need modifications for dist-git
17:53:25 <dgilmore> Oxf13: whats our plan for git.fedoraproject.org or whatever the git server will be called?
17:53:34 <dgilmore> reuse the cvs server?
17:53:35 <Oxf13> it'll be pkgs.fedoraproject.org
17:53:56 <Oxf13> for the sake of roll back, I'd rather we used a brand new system so that we can easily swap back to cvs.fedoraproject.org
17:54:04 <Oxf13> and we can keep cvs.fedoraproject.org around in read-only mode
17:54:05 <dgilmore> ok
17:54:30 <dgilmore> lets make sure we have a resource for it
17:55:21 <Oxf13> #action Oxf13 to talk to fedora admins about resources for pkgs.fedoraproject.org production dist-git server
17:56:44 <Oxf13> Ok, I think that's all I have on dist-git.  Any questions?
17:57:11 <dgilmore> nope
17:59:23 <Oxf13> alright, that's all I have for the meeting then.
17:59:30 <Oxf13> #topic open floor
17:59:58 <dgilmore> i have nothing right now
18:01:58 <Oxf13> alright I'll close up shop, thanks for coming all!
18:02:01 <Oxf13> #endmeeting