18:18:30 #startmeeting Fedora Release Engineering 18:18:42 #topic Roll Call 18:18:49 * dgilmore is present 18:18:50 * notting is here 18:19:16 ping: notting jwb rdeiter dgilmore poelcat warren spot lmacken 18:19:23 * spot is around 18:19:29 * jwb is 1/2 here 18:19:39 * warren here 18:21:06 alright lets get started. 18:21:09 #topic old business 18:21:22 Reviewing the logs from last week there were a couple action items. 18:21:36 however I didn't see poelcat respond 18:21:53 the first action was for poelcat to add bodhi dates to the F12 schedule 18:24:53 Does not look like this has happened yet. 18:25:03 move on for now? 18:25:18 #action poelcat still needs to add bodhi dates to F12 schedule. 18:25:30 #nick poelcat 18:25:31 #action poelcat still needs to add bodhi dates to F12 schedule. 18:25:33 there we go 18:25:48 Next up was orphan clean up 18:26:00 action item was for me to file a releng ticket about purging the orphans. 18:26:17 I've done so, and sent the first round of lists to review to the community 18:26:25 a bunch have been picked up, I need to send the updated list out 18:26:51 #action f13 to send an updated list of orphans to be purged 18:27:07 Next was FAD follow up, which I had an action to review. 18:27:19 I failed, so I still need to do this 18:27:32 #action f13 will /really/ spend time on FAD this week. 18:28:05 There were topics about updates pushess, which I had an action item to file a ticket to back down gpg sig cleaning, which I forgot to do so that's still an open action item. 18:28:23 #action f13 still needs to file ticket about backing down how aggressive the gpg signed copy cleanup is in koji 18:28:35 and the last action was to talk about critical path this week. 18:28:47 that's the end of old business, lets start right in at critical path. 18:28:50 #topic Critical Path 18:28:58 skvidal: wwoods: this is your show now, are you around? 18:29:13 * skvidal reads up 18:29:29 okay 18:29:58 gist is that there are a set of pkgs which will require an extra step if the owner wants to update them 18:30:06 and/or change them in a time where we would normally be frozen 18:30:48 we've got the list of pkgs now - and once I add the solve out deps problem I can get the list of owners and we'll notify them 18:31:01 f13: is there another component of this which is special for releng? 18:31:20 well, the "extra step" would involve something with bodhi right? 18:31:20 skvidal: "is bodhi ready"? 18:31:38 I do not know of any bodhi changes - lmacken? 18:32:16 i thought it required some magic 'tag but no push' bodhi workflow 18:32:33 I believe it requires some 18:32:40 but I do not know the status of them 18:34:05 right, it would require bodhi work, and then releng/qa work on the side of twiddling whatever bit in the update request necessary to make it "go" 18:35:21 ok 18:38:00 lmacken: don't suppose you're around? 18:38:17 didn't think so 18:38:52 so maybe next week we can get a report from luke on what if any changes have gone into bodhi to support critical path packages. 18:39:00 #nick lmacken 18:39:15 #action lmacken to report next week on bodhi changes for Critical Path Packages 18:39:27 f13: http://poelstra.fedorapeople.org/schedules/f-12/f-12-releng-tasks.html #66 was supposed to address bodhi 18:39:43 so without bodhi support, can we still do some of the proposal, give extra special care to things in the critical path when freeze requests come through? 18:39:46 * poelcat in a conflicting meeting 18:40:00 poelcat: right you are, I'll note that later. 18:40:10 f13 anything else missing? 18:40:40 sorry for crashing current topic 18:40:44 I don't think so, that was just a follow up from last week 18:41:55 f13: yeah, I'm kind of around 18:42:43 so lets assume for a moment, that nothing else changes in our development cycle, and we're still using trac to handle freeze requests 18:43:09 no changes have gone into bodhi for the critical path stuff... is there a ticket somewhere that defines what the workflow needs to look like in bodhi? 18:43:13 how would we make good on Critical Path in this scenario? We'd need to know if the package freeze break requested is a critical path package 18:43:30 and then we'd need to get an indication in the ticket that it has had a second look 18:43:44 lmacken: I don't think a ticket exists, nothing outside of the feature page itself 18:44:44 f13: you would need some way to know its a critical package 18:44:58 and soem way to sigify that its been triple checked 18:45:22 isn't that what I just said? 18:45:41 so, someone (tm) would have to do the check against crit path packages, and then just do manual acks in the ticket 18:46:18 here is what I'm thinking 18:46:47 if we're still using Trac for tag requests, I'd like to spend some effort and get "make tag-request' a target in the make system, that will do the trac filing for people 18:46:58 as part of that, it could check to see if the package in question is a critical path package so we'd know it in the ticket 18:47:18 f13: thats doable 18:47:19 that would require getting offtrack packaged and installed on people's systems, as well as having some quick utility to check if a package is crit path or not 18:48:28 we'd also have to use said utility for the tickets that are manually filed 18:48:38 so auto filed tickets should have some signature that it was filed from the make system 18:49:28 I should probably work on offtrack, since the likelyhood of getting no frozen rawhide approved in the next couple weeks seems pretty low. 18:49:44 f13: we could add offtrac as a requirement for bodhi-client or fedora-packager 18:50:05 #action f13 will work on getting offtrac packaged up in Fedora and a 'make tag-request' target added to the make system. Also tied into checking if critical path package or not. 18:57:54 hrm, brb, gotta take care of a diaper 19:03:24 back. 19:03:35 wwoods: skvidal: anything else to discuss on Critical Path? 19:04:35 f13: I'd like to discuss you feeding your child to goats - but that's not critpath related, really. 19:04:47 LOL 19:05:46 http://www.flickr.com/photos/iamjessekeating/3736843791/ <--EVIDENCE 19:06:04 there were goat pellets in his hands 19:06:23 moving along. 19:06:36 sure 19:06:37 #topic No Frozen Rawhide 19:06:58 This is the big FAD proposal, with the most to change. 19:07:13 It seems to have generally good feedback from those I've talked to about it, and from the mailing list, although not much there 19:08:18 so lets look at where we are with our needs. 19:08:27 compose tools, those were supposed to get a lot faster to handle multiple composes in a day 19:08:37 we got better, but are we to the point were we can do multiple in a day? 19:09:21 last night's rawhide mash started at 02:17 19:09:53 Finished up at 08:57 19:10:00 nearly 7 hours 19:10:26 to do another one would put it at 14 hours 19:10:42 you could gamble on 3 a day 19:10:49 but 2 would be doable 19:11:00 * nirik somewhat jokingly wonders if we really need drpms for ppc64 (since it's not really a tree that should be used by end users much) 19:11:03 f13: though with smaller changes how much quicker would it be? 19:11:08 yeah, but then we're not leaving very much time left for doing updates pushes and epel pushes 19:11:26 nirik: that's actually one request that's been filed, the ability to configure which arches get drpms 19:11:27 nirik: there's a bug on that 19:11:33 f13: epel pushes are independent of the other two but its all load on /mnt/koji 19:11:49 dgilmore: yeah, it's the load on /mnt/koji I wonder about 19:11:56 yeah, I don't know how much it would help, but they seem not too required. 19:12:07 we were going to try running two composes at once on releng2 and releng-stage but I think our mirror issues have killed that effort 19:12:12 jwb: do you know where we are with that? 19:12:26 f13: i think we have done updates and epel at the same time 19:13:00 updates right now are taking an artificially long time to finish due to restarts 19:13:06 * jwb reads scrollback 19:13:27 f13, two composes at once? 19:13:51 jwb: yeah, we were going to test to see if we would get benefit of splitting composes across mutliple boxes 19:13:59 or if /mnt/koji would be too much of a single bottleneck 19:14:12 f13, oh. we never tried that because we're still fixing fallout from drpms 19:14:17 among other things 19:14:50 right 19:14:54 f13, also, releng1.stg can't talk to koji. so that's sort of useless 19:14:59 at least at the moment 19:15:06 oh wait 19:15:08 which makes me worried that we wouldn't be able to produce multiple trees in a single day 19:15:13 you said composes... 19:15:18 What needs to happen on koji1.stg to make releng1.stg not useless? 19:15:24 did you mean update pushes, or tree composes? 19:15:56 jwb: well, that kind of depends 19:16:08 jwb: we'd be doing one cron compose to produce rawhide based on the dist-rawhide tag 19:16:40 then we'd have to produce 2 other trees from tags managed by bodhi 19:16:50 and then we'd have to do the normal bodhi pushes 19:17:51 thinking it through, and thinking about where things land for the content to be the release 19:18:21 we might have to do a full Everything compose at Alpha, and that stays static, while we compose an updates-testing and updates like repo set to complement it. So the content being composed is smaller each night 19:18:38 or we have to do a rawhide like everything tree each night, and then one addon repo that is the updates-testing 19:20:46 either way we'd be doing something close to 2x cron ran rawhide composes, and then whatever other bodhi pushes and epel pushes are necessary 19:22:03 we'd also need changes in bodhi for how to handle update requests for a pending release rather for something already released 19:22:11 and the push tools around that to know what to do (tag, but don't mash) 19:22:27 we have to fine tune the mirror layout 19:22:30 (see above) 19:22:44 Those are the real hard parts 19:22:53 and need to be figured out before we take it to FESCo 19:23:02 so who would like to work with me on these things this week? 19:23:36 #action f13 will try to get folks together to define missing parts of the no frozen rawhide proposal this week 19:24:19 anything else on this topic? 19:26:16 alright 19:26:25 #topic Fedora 12 Alpha 19:26:31 Couple of dates of interest 19:26:54 This Friday there is a bug blocker review day 19:27:15 next Tuesday is Feature Freeze, which is also the orphan blocking day 19:27:35 next Wed. we need to do a test compose and deliver it to QA for an initial round of testing 19:28:08 another blocker bug day that Friday 19:28:14 the following Tuesday is the alpha freeze 19:28:41 and the Thursday after that is when we're supposed to compose the Alpha release candidate, assuming that the blocker bugs are clear 19:29:02 I've got tickets filed in trac for most of these things 19:31:36 guess there is nothing left on those. 19:31:45 #topic open floor 19:31:56 if there is no new topic I'll close the meeting in 5 minutes 19:32:07 so, XZ in rpm, and x86 support -> mass rebuilds 19:32:13 x86 support changes are ready to go 19:32:38 XZ in rpm is not quite ready (need a new rpm build for infrastructure - working on it) 19:33:04 oh geez, mass rebuilds, forgot that 19:33:08 #topic Mass Rebuild 19:33:18 i should say, x86 support changes are live in koji now 19:33:20 even less time now to do 19:33:26 since friday 19:33:30 ok 19:33:37 what's the ETA on the XZ support? 19:34:26 getting the raw bits in... today? then it would be a matter of flipping the default in redhat-rpm-config &* sending out a large warning to the lists 19:34:47 f13: we need builder support first 19:34:50 what's going to happen during the time where that config is enabled but we haven't rebuilt everything? 19:35:10 notting, so... i'm assuming f11 RPM will not be able to cope with f12 RPMs after that flip 19:35:14 jwb: correct 19:35:19 I'm looking at https://fedoraproject.org/wiki/Fedora_11_Mass_Rebuild for how we did it for 11, we should duplicate that for 12 as close as possible. 19:35:26 notting, does that sound a bit bad? 19:35:50 f13: i'm not sure what you mean. XZ support in rpm requires macros to enable it; if we don't do that, nothing happens unless some wacko sets them by hand in their own .spec file 19:35:52 I'd hope there would be a planned F10/F11 update to rpm to handle the F12 content 19:35:58 f13, right 19:36:04 yes. but i don't control that 19:36:28 of course EL-5 users will be just as hosed as they are today 19:36:31 notting: what's going to happen when we enable it, but we haven't actually done the mass rebuild yet, so we have a few days worth of builds that use the new rpm but many that don't? 19:36:33 anything bad? 19:36:58 f13: they'll fall into the boat that jwb mentioned. aside from that, nothing too bad 19:37:00 dgilmore: it's easy enough to tell rpm to ignore file md5 or to generate srpms with the md5 file checksum. Harder to ignore the compression 19:37:02 f13: id hope rpm deals with each rpm just fine 19:37:06 nod 19:37:18 as noted on the feature page , xz does not break deltarpm, although it renders the deltas suboptimal 19:37:50 So if the stuff gets in today, and we start rebuilds on Friday 19:38:00 stuff will still be rebuilding by the 28th 19:38:04 which is the feature freeze 19:38:12 do we need to wait until friday? 19:38:41 last time we started on the 23rd after about 6 days warning 19:39:25 4 days later the automated builds were done 19:39:34 with 500~ builds needed to be done manually 19:40:10 notting: we don't have to wait, but it just gives maintainers less time to opt-out, and it will cut into time that maintainers are scrambling to finish things up by feature freeze 19:40:35 plus our compose tools are going to go nuts when 2K packages change each day. drpms will go through the roof 19:40:46 turn them off 19:40:48 see above :) 19:41:08 ok, we could do that 19:41:18 at some point we'll turn them back on though right? 19:41:33 yeah, after the mass rebuilds are done 19:41:44 guess that won't be too bad 'cause then it'll just be things changing after the mass rebuild. 19:41:47 but there's not a lot of sense in delta-ing across payload compression differences 19:41:49 so yeah, I like that idea. 19:42:13 mmcgrath: ping; did the CVS filesystem overhaul complete? 19:42:32 f13: yeah 19:42:33 mmcgrath: CVS locking was one of the biggest bottlenecks in last mass rebuild, is it better now do you think? 19:42:45 f13: I'd think so but it's worth testing for real 19:42:49 k 19:43:20 so if we started Thursday, we could potentially be done by Monday 19:43:32 of course, mirrors are just now trying to settle down from the outage 19:43:38 this isn't going to help 19:44:52 f13: so, i can clone the F11 page and do some adjustments 19:46:01 show of hands, how many people think we can/should do this? 19:46:23 notting: haha, I just erased a line asking you to do that, first I wanted to get some sort of consensus that we should push forward on it 19:46:45 * jwb raises 1/2 a hand 19:46:55 mmcgrath: I'd be interested in your opinion too. How well do you think our infrastructure could handle a mass rebuild this weekend? 19:46:56 * notting raises a hand 19:48:13 f13: I believe it'd handle it well. 19:48:23 as for publishing to the mirrors that's different. 19:48:33 yeah, I'm considering that part of our infrastructure 19:48:43 Though we're better positioned now then we were this time last week. 19:49:37 f13: i think we are in a fair state to do it this weeken 19:49:37 d 19:49:56 * ricky would rather not have another factor to add to the mirror madness analysis, but ah well :-/ 19:49:58 ok. 19:50:05 f13: do you have an estimate of how many files would change? 19:50:17 mmcgrath: all of them in rawhide 19:50:24 what dgilmore said 19:50:26 mmcgrath: rough estimate would be every file under development/ 19:51:20 #agreed We should attempt a mass rebuild driven by releng for arch and XZ support. 19:51:37 #action notting will copy the F11 mass rebuild wiki page for F12 19:52:12 Do folks feel that Thursday is a good starting day? 19:52:28 leaves a little time for folks to opt-out, and for mirrors to settle down, and for us to finish before the feature freeze 19:53:17 mirrors settling down is a lie 19:53:23 mmcgrath: That's currently 106377 files, for what that's worth. About 1/6th or so of fedora-enchilada 19:53:26 but the opt-out thing is fair 19:53:39 ricky: thanks 19:53:50 ricky, minus drpms. those won't get touched for now 19:53:58 jwb: you don't think mirrors will be in a better place by Thursday? 19:54:06 f13: so, i'm saying we'll have automated rebuilds done by July 28th, and any cleanups done by the 4th? 19:54:31 f13, better, perhaps. settled, no. i have a push i'm going to start today hopefully 19:54:54 jwb: Oh, I didn't know we had drpms for development 19:55:06 ricky, we had them there first even :) 19:55:09 notting: that's a fair target. I don't know that we've /ever/ had cleanups from mass rebuilds fully done 19:55:10 Ah, nice 19:55:33 The new number is 90662 then 19:55:57 #agreed Mass rebuilds will start Thursday the 23rd, to finish by Tuesday the 28th. Cleanups will target completion by tuesday the 4th 19:56:21 please read over https://fedoraproject.org/wiki/Fedora_12_Mass_Rebuild 19:57:25 the XZ rpm payloads link is a dead link 19:57:33 reload! 19:57:41 bingo 19:57:58 i'll update the releng scripts with s/f11/f12/ 19:58:16 Timeline looks odd 19:58:37 looks like it references the F11 stuff 19:58:42 cleared; that appeared to be a post-mortem section 20:00:42 notting: ok, looks good 20:00:46 ready for announcement 20:00:51 ok. i'll send mail to f-d-a 20:01:33 #action notting will announce the mass rebuild to fedora-devel-announce 20:01:39 anything else regarding this topic? 20:01:46 it's going to be a busy week next week (: 20:04:48 Ok, I think I'm going to call meeting. Thanks for hanging in there, it was a long one. 20:04:58 #endmeeting