17:01:01 #startmeeting Fedora Release Engineering 17:01:01 Meeting started Fri Jul 16 17:01:01 2010 UTC. The chair is Oxf13. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:01:01 Useful Commands: #action #agreed #halp #info #idea #link #topic. 17:01:10 #meetingname fedora-releng 17:01:10 The meeting name has been set to 'fedora-releng' 17:01:16 #topic roll call 17:01:35 * brunowolff is here for GA and spins discussion 17:01:44 * jsmith lurks 17:01:49 ping: notting poelcat rdieter lmacken spot wwoods brunowolff dgilmore 17:02:03 * poelcat 17:02:05 dgilmore is speaking at FUDCon 17:02:55 * nirik is hanging around in the cheap seats. 17:03:32 #info present are brunowolff poelcat notting jsmith nirik 17:03:47 lets get to brunowolff's topic right off 17:03:54 #topic Spins 17:04:02 brunowolff: what have you got for us? 17:04:30 Spins SIG hasn't done a very good job in the past for working with releng to get spins built for the GA release. 17:04:40 Jesse usually ends up cleaning up after us. 17:04:54 Announcement from my owner (stickster): Fedora Board open IRC meeting in #fedora-board-meeting at 1800 UTC today (~1 hour from now). 17:05:12 I'd like to talk about two issues, one is communication for the GA and the other is testing requirements for spins. 17:05:16 here now, hi. 17:05:42 #info rdieter also present 17:05:48 Maybe it's best to talking about testing requirements first since that has an impact on how we want to communicate. 17:05:58 brunowolff: make sense 17:06:16 The proposal is that spin owners or their delegates be required to do some testing of the Spin RCs. 17:06:53 nirik and I were OK with that requirement in spins, and people in QA said about time. 17:07:22 QA has some test plans that we can use, but no resources for doing the testing. 17:07:36 ie, a positive 'I, or my delegate tested this [ x ]' 17:07:45 If we do this we need to communicate to the spin owners when and where to get the RC spin versions, 17:07:59 when and where to report back. 17:08:15 yeah, the RCs can move fast and furious 17:08:21 We need to know what to do if a spin's owner is MIA or if a spin fails the tests. 17:08:43 given the amount of stuff we produce for a release, it is not uncommon for us to start spinning an RC, get some content posted, then find out it's busted and have to start over, long before we get to composing the spin content 17:09:04 which is unfortunate for the spins as an RC for them doesn't get posted 17:09:06 I don't want to require too much testing for this go around; even so I think there is a good chance that some spin owner will end up being MIA. 17:10:06 based on what I have seen for RCs, things need to move pretty fast. 17:10:59 * nirik just wants to avoid a 'oops. I never tested this and it's broken and we already released it' thing again. 17:11:07 sounds like if testing doesn't occur or if spin owner is MIA, then spin needs to get dropped. 17:11:18 If custom spins are going to need changes and still be posted with the rest of the release stuff the owners will need to be very active. 17:11:40 Yes, dropped. But what does that mean for things like the release web pages? 17:12:02 so I think the best thing to do re the testing, is to test the nightly image that is made from branched that day 17:12:09 I suspect there is more than just not publishing the ISO. 17:12:21 and if in creating an RC, I have to pull any extra builds in for the RC, I need to communicate that with the spin folks to see if that package effects their spin 17:12:27 what about building final spins at the time of final RC so there is time to figure things out 17:12:33 vs. after testing the final RC 17:12:54 I think using nightlies make sense. The odds that other changes made that day to fix other things will break a spin seems small. 17:13:03 brunowolff: okay 17:13:41 I could rebuild nightlies to pull in specific packages if asked I suppose... just need a local repo with the newer packages added in. 17:13:44 It is certainly better than what we've done in the past. 17:13:53 poelcat: I'm not sure I understand your suggestion 17:14:08 how do we determine that the owner is MIA, other than just the absence of testing feedback? 17:14:36 Oxf13: i understood that "final spins" get created after we declare the RC gold 17:14:36 notting: I was hoping we would have a positive ack... ie, if no ack, they are assumed mia. 17:14:49 I think someone (probably the wrangler) needs to tell them when to test and when to report back from tests. 17:14:49 Oxf13: so i was suggesting start earlier 17:15:02 poelcat: I don't think that's right 17:15:13 poelcat: if I have time while creating the RC, I create the spins 17:15:18 i understood the final spin signoff to be between the thurs we sync to mirrors and GA day 17:15:22 nirik: and therefore, it's dropped? 17:15:32 but if we discover a problem with the RC before I get the spins created, then I stop composing, and work on the next RC 17:15:42 They should already know about the nightlies, so pointing them there for testing is simpler. 17:15:49 notting: yeah, might be too harsh tho. I'm a bit sad at how little involvement spins owners seem to have. ;( 17:15:54 brunowolff: do we have a formal "spins sign off date on the schedule?" 17:15:57 so due to the nature of the sheer amount of things we have to compose and move around, the likelihood of getting the entire set of things built for each RC is pretty low 17:16:01 Not yet. 17:16:25 This is new, and after this meeting there should be information about what the requirements of the owners are. 17:16:26 I would prefer it be at the same time as the RC signoff 17:16:39 in case the spin requires some package update that would be on the DVD set 17:16:52 I as the interim spins wrangler will communicate that requirement and make sure the or else is prominent. 17:18:25 I envision that I will be communicating with the Spin onwers and that I would also be providing summaries or issues to releng. 17:19:11 brunowolff: best to just keep it on a wiki page where everyone can see what's in or out 17:19:19 including Oxf13 17:20:31 Wiki for status is OK, though I expect to try to be available by IRC as well. I can probably take a day or two off work to help with this. 17:21:27 If we use the nightlies for spin testing, which day's nightlies are we going to be able to use? 17:21:47 Is it when we start RCs or after a final RC has been accepted? 17:21:54 Or something else? 17:21:56 when we start RCs 17:22:04 Ok. 17:22:15 basically I'll announce that we're in RC phase, and not taking any changes into the branched repo unless they are to fix issues found in the RC 17:22:16 What happens if something is found to be broken? 17:22:31 a blocker bug filed, and brought to the attention of releng 17:22:53 we'll have to do some manual tagging to get the fixed package into branched instead of having it go out as a 0-day update 17:23:05 What happens if the owner is MIA or a bug cannot be fixed before GA? 17:23:30 i would suspect it has to go out as an update , then 17:23:38 if it's really broken and the owner is MIA, we wouldn't ship the spin? 17:23:39 I am guessing we wouldn't slip the release for just one spin (other than the main ones). 17:23:50 yeah, I think that's the line we have to draw 17:24:04 if it can't be fixed by the time the rest of the release is ready to go, it gets dropped 17:24:30 OK. If for whatever reason we don't ship a spin, what else needs to happen? 17:24:41 Does the design team need to be notified? 17:24:42 website updates 17:24:44 yea 17:24:50 and it depends on the level of the bug, of course. if it's 'random app on the spin crashes', we ship the spin and just do a zero-day update 17:25:45 Oh, I forgot another timing issue, when do the spin owners have to acknowledge by? (Or be treated as MIA.) 17:26:48 brunowolff: by the go / no go date I'd say 17:26:53 OK. 17:27:01 How do slips affect this? 17:27:03 i'd like to think we have an initial checkin at some point around beta or after, and then another one at RC phase 17:27:04 no response would be treated as "no go" for the spin 17:27:19 brunowolff: slips will push that go / no go date out 17:27:20 Do a new set of tests need to be made? Do MIAs get an extension? 17:27:35 brunowolff: and if we're slipping, that means we're pulling in some other packages, and will likely need to restart the test and confirm process 17:27:47 yes, I'd say MIAs get an extension 17:27:56 So that will be retest and you get an extension. 17:28:10 if the package changed isn't on a spin, I think we can re-use the previous "go" vote. 17:28:36 and potentially re-use the previous iso set. 17:28:39 So retest may not be needed. Only if the spin was broken. 17:28:51 only if the package set changed and caused a need to re-spin the spin 17:29:38 I think that covers the first part well enough and I'll be able to write that up. 17:29:59 The second part is what do you guys think the minimum test requirements are? 17:30:29 Which test plans must be executed and do both i686 and x86_64 have to have those tests done? 17:30:57 I think that falls more on the QA side of the house 17:31:15 I'm just happy if somebody boots both arches and makes sure the login screen comes up and people can log in 17:31:17 that should probably largely be up to the spin owners to help decide 17:31:37 beyond that I think it is a matter of QA and a matter of the spin owner deciding what is critical functionality for the spin 17:31:42 OK. I'll propose something to Woods and co. 17:31:45 let me break it down this way 17:31:50 releng requires that testing be done. 17:32:10 QA should require a bare minimum acceptable testing in general for a spin 17:32:25 the spin owners can add on top of that what they consider the critical functionality testing for the spin in question 17:32:57 I already have pointers to test plans and I'll look those over and propose some minimum for this release and see what they say. 17:34:58 I think that answers my questions for today. I'll need to write some stuff up and run it by people. 17:36:27 ok, thanks for being patient with me and meetings brunowolff 17:36:58 anything else from anybody on spins? 17:37:44 #info Spin owners must check in with testing before the go / no go date or risk having spin dropped 17:37:49 No problem. 17:38:05 #info A slip of the release will reset the spin go / no go date as well and give MIA spin owners an extension 17:38:25 #info In case of a slip, if the updated packages are not on a spin, a re-test will not be needed 17:38:49 #info All releng requires is that testing be done. The amount of testing is to be determined by QA and the spin owners 17:39:08 * dgilmore is here 17:39:26 #info releng will communicate to spin owners when RC phase starts, and when new packages are needed beyond nightly compose for future RCs 17:39:39 I think that captures most of what we covered today 17:40:44 alright moving on 17:40:48 #topic dist-git 17:40:54 Lots of progress here 17:41:15 #info Users can now use dist-git and fedpkg for cloning, committing, pushing, and building in koji.stg 17:41:36 #info oxf13 feels that we are pretty close to required functionality for initial roll out 17:41:42 there has been lots of bug reports which is good 17:41:50 * dgilmore agrees 17:41:59 #info new package creation and branch scripts have been modified to work with dist-git, as has the lookaside cgi 17:42:30 Oxf13: so thats just a modifcation to process-cvs-requests.py ? 17:42:31 or ? 17:42:57 process-cvs-requests.py doesn't call anything cvs specific 17:43:03 do we have an idea from the bug reports, checkins, or koji logs, just how many packagers are testing it? 17:43:04 oh true. 17:43:05 I think it gives you output to feed into pkgdb2branch.py 17:43:22 so I've modified pkgdb2branch.py and setup-git-package to work with dist-git 17:43:30 * nirik played with it yesterday... everything worked pretty well for causal testing. 17:43:35 notting: I could get some idea. Koji db got re-set 17:43:47 I could get some idea from the gitolite logs that is 17:43:59 Oxf13: all the cvs stuff is handled outside of process-cvs-requests.py afaik 17:44:00 as far as forward progress there are a couple things I want to work on 17:44:37 I want to attempt using subdir branching, so that we have a top level F-13/ directory with a HEAD within, and subsequent branches of F-13 woudl live within that directory, like F-13/newversionfoo 17:44:56 which would make detection of what base top level branch to send builds to easier and wouldn't require a "branch" file within the repo 17:45:00 notting: id guess we have had more than a dozen users. how much more not sure 17:45:28 it would make checking out the base F-13 branch slightly more complicated, git checkout -b F-13 --track origin/F-13/HEAD or some such, but I can bury that in fedpkg 17:45:49 Oxf13: sounds reasonable 17:45:51 the other thing I need to work on is discovering which packages don't come over well with parsecvs 17:46:16 kernel I know of, log4j I think is another one, I just need to run a script across all the repos to find the broken ones, and make sure they can use git cvs-import 17:46:22 Oxf13: i wonder if we need to try clone them all. 17:46:24 git cvs-import seemed to work for kernel, it was just a /lot/ slower 17:46:38 or if we can work out a way to detect it on the git server 17:46:40 dgilmore: I think a "git status" would be enough on each repo dir on pkgs.stg 17:48:02 there is only 165 entries in the gitolite log, so that's not a whole lot of testing 17:48:03 Oxf13: konversation fails to checkout 17:48:07 but that's to be expected of our community 17:48:15 if you want a example of bad to look at 17:48:39 Oxf13: half of that is probably me 17:48:46 [jkeating@pkgs1.stg konversation.git]$ git log || echo "broken" 17:48:47 fatal: bad default revision 'HEAD' 17:48:47 broken 17:49:15 so yes, looks like we'll be able to use the repos directly on the git server to detect the problem ones. 17:49:25 when is the next FESCo meeting? 17:49:29 that will make it simpler 17:49:36 tuesday? 17:49:41 tuesday - 2010-07-20 at 19:30 UTC 17:49:41 #info Need to detect broken conversions and attempt with git cvs-import 17:50:07 #info will try directory based upstream top level branches for target discovery (eg F-13/HEAD F-13/coolbranch) 17:51:00 #info We need to draft a roll out plan and present it to FESCo, complete with rollback plan 17:51:23 thankfully it appears to be very easy to roll back if things go south 17:51:52 #action We need people to start gathering a list of wiki pages that will need modifications for dist-git 17:53:25 Oxf13: whats our plan for git.fedoraproject.org or whatever the git server will be called? 17:53:34 reuse the cvs server? 17:53:35 it'll be pkgs.fedoraproject.org 17:53:56 for the sake of roll back, I'd rather we used a brand new system so that we can easily swap back to cvs.fedoraproject.org 17:54:04 and we can keep cvs.fedoraproject.org around in read-only mode 17:54:05 ok 17:54:30 lets make sure we have a resource for it 17:55:21 #action Oxf13 to talk to fedora admins about resources for pkgs.fedoraproject.org production dist-git server 17:56:44 Ok, I think that's all I have on dist-git. Any questions? 17:57:11 nope 17:59:23 alright, that's all I have for the meeting then. 17:59:30 #topic open floor 17:59:58 i have nothing right now 18:01:58 alright I'll close up shop, thanks for coming all! 18:02:01 #endmeeting