18:03:29 #startmeeting Fedora Release Engineering Meeting 18:03:33 #topic roll call 18:03:56 ping: notting, jeremy, jwb, wwoods, lmacken, spot, warren, rdieter, dgilmore, poelcat 18:04:00 pong 18:04:02 * notting is here 18:06:32 * dgilmore is here 18:06:47 hi 18:06:49 * wwoods 18:07:51 well that's a good set. Lets get going 18:07:57 #topic old business 18:08:31 #topic orphans (old business) 18:08:49 I sent out an updated list of orphans and write a script to depsolve those orphans if they were removed. 18:09:00 I need to send it out again with current status. 18:09:22 #topic FAD (old business) 18:09:39 I spent some time last week polishing up the no frozen rawhide proposal for FESCo submission, and it passed! 18:09:50 We'll talk more about what that means to us later in the meeting. 18:10:08 #topic gpg purging in koji (old business) 18:10:27 * poelcat logging 18:10:30 I did not get this ticket filed, however I did look at the script and noticed that it doesn't even have our current keys listed, so this script needs even more work. 18:10:48 so that's still an open action item. 18:11:09 #action f13 will file a ticket about koji-gc signature script needing work to be current. 18:11:28 that was the left over stuff from last meeting's old business. 18:11:32 lets review last week's topics. 18:11:41 #topic Critical Path (old business) 18:11:52 There were 2 action items here. 18:12:03 one was lmacken reporting on bodhi changes for critical path. 18:12:09 I haven't seen lmacken surface yet this meeting. 18:12:14 I'm kind of here 18:12:21 ah 18:12:25 do you have anything to report? 18:12:27 but I haven't looked into the critical path + bodhi stuff yet :( 18:12:34 ok. 18:12:41 it "shouldn't be too hard" 18:12:45 #action lmacken to report next week on bodhi changes for Critical Path Packages 18:12:50 will do :) 18:13:17 the next action item was for me to get offtrack packaged so that 'make tag-request' would work in case we don't get no frozen rawhide done in time and would be using bodhi for update submissions 18:13:26 I did not get this done, so it continues to be an action item. 18:13:50 #action f13 will get offtrack packaged and make tag-request functional. 18:13:57 is there a link for status / proposed process changes for bodhi/critpath stuff? 18:14:12 that's a good question. 18:14:24 one would hope that it was linked from the proposal page 18:15:26 lmacken: is there a ticket filed already? 18:15:37 I don't believe so 18:17:22 as the people running with the proposal, wwoods skvidal can one of you file a ticket for what we want to see changed? 18:17:29 Here's the thing: there's two parts to the proposal, right? 1) here's the package list, and 2) here's some new rules regarding updates to packages on the list 18:17:33 * skvidal reads back 18:17:36 yes 18:17:45 and what changes we want to bodhi to make 2 happen 18:17:48 from the QA POV, I only care about #1 18:17:56 because I'm doing automated testing on packages in that list 18:18:07 okay 18:18:14 wwoods: you care about 2, because you'll have to interact with bodhi somehow to record your results. 18:18:16 f13: open a releng or a bodhi ticket? 18:18:19 bodhi 18:18:23 f13: okay 18:19:01 I'll do that 18:19:07 #action skvidal will file a bodhi ticket to define the changes we want to see for critical path packages 18:19:07 f13: right, but I've not been further involved in drafting that part of the proposal 18:19:26 which is why I'm asking - I don't know what's reasonable to ask of the maintainers 18:19:26 anything else on critical path? 18:19:29 wwoods: if you'd like to be - I'm all ears - but I can file a ticket 18:19:42 I'd like to stay informed at least, but yeah, do that 18:19:45 wwoods: and I'll pass it to you for your critique if that's helpful 18:19:48 wwoods: maintaines shouldn't see much different 18:19:57 probably modeled after the security team clearance for security updates 18:21:00 Ok, if there is nothing else, we'll move on 18:24:04 #topic mass rebuild 18:24:09 there we go 18:24:28 Ok, bill completed his task of creating the f12 rebuild page 18:24:34 and announcing the rebuild. 18:24:48 I further updated the scripts and we kicked off the rebuild. 18:25:11 ricky and I fixed another performance issue with cvs, and that enabled me to just do a single thread submission script 18:25:18 and more than overwhelmed the buildsystem with submissions 18:25:37 although there was some issues with my use of ssh sockets and some stalling of hte script, all the builds had been submitted by last night 18:25:46 and the buildsystem had about 3K pending this morning 18:25:52 woo 18:26:00 However, we've had to pause the rebuilds in order to install a security fix for python 18:26:07 oops. 18:26:11 which should happen soon as dgilmore noted. 18:26:19 just waiting on rhn sync 18:26:36 so I doubt we'll be done with all the builds by Tuesday, but Wed looks pretty good 18:26:43 We did start a day late waiting on glibc/gcc changes 18:27:09 so once the scripted builds end, I'll be looking at the failures and generating the lists of things that still need to build, as well as doing the mass tagging. 18:27:24 #action once rebuilds finish, lists will be generated of failed/needed rebuilds 18:27:31 #action f13 once rebuilds finish, lists will be generated of failed/needed rebuilds 18:27:44 where are we with updates to f10/f11 rpm? 18:27:53 f11 is in updates-testing 18:28:06 are we caring about f10? 18:28:15 panu said he was going to look at F-10 next, but he didn't appear to get to it before he left on friday 18:28:21 ok 18:28:29 (and he's on vacation now) 18:28:41 oh boy 18:28:52 I talked with the QA folks about this a bit earlier today 18:29:03 so, we can poke ffesti/jnovy to pick that up 18:29:15 there is a problem where the rawhide rpm in the rebuild requires the new compression support to install 18:29:19 it also requires new glibc 18:29:34 so we can't just drop a old formatted new rpm build, as it would require the new glibc which requires the new compression 18:29:42 so some rawhide users may get caught in the lurch 18:30:09 * jwb can see the "gah! broken!" vs. "rawhide eats babies" threads now 18:30:10 I do believe they are going to get the rawhide blog page updated with some of this information 18:30:11 f13: gahh 18:30:13 eww, sounds familiar 18:30:25 yeah, i'ts like the md5sum thing, only worse 18:30:57 so that's twice now we've hit a situation like this. is there something we should be doing to not have this happen? 18:31:07 maybe need a special koji target with an older/compat glibc/rpm ? 18:31:11 at least this time it's happening before Alpha, so we won't be leading a bunch of people into a trap 18:31:11 jwb: don't build rpm in rawhide? 18:31:18 jwb: like going back to no updates to rpm 18:31:31 might need a SOP for RPM format changes that break compat 18:31:35 there are reasons why rpm was stagnant for very long times 18:31:41 wwoods, right 18:31:54 if you stage all changes in $prior-updates, it would sort of work 18:31:56 how long did we have an rpm in rawhide that supported the new compression format? 18:31:56 but be rather odd 18:32:10 wwoods: we do because it also effects buildsystem, and lots of other things 18:32:24 f13: a week 18:32:26 f13: rpm landed on monday 18:32:33 maybe a bit longer but not much 18:32:43 would we want longer than that? 18:33:01 actually, lets see if we can get a volunteer to own a SOP on rpm changes 18:33:18 I've got a bit much to take that on, is there anybody here that would be willing? 18:33:22 for *this specific* case, we could build a bumped f-11 version, and tag that into f-12 18:33:25 (if not, we can pitch it to the -devel list) 18:33:39 f13: i will since it effects buildsysem among others 18:33:39 notting: why not just tag the f11-testing one ? 18:33:51 What about the general case? Is it acceptable to make changes that will result in rawhide being unable to be upgraded without manual intervention? 18:33:58 #action dgilmore will draft (with help) an SOP for managing rpm changes in rawhide. 18:33:58 f13: that would cause 4.7.1-1.fc11 to be newest over 4.7.1-1.fc12 18:34:00 If so - what's the required minimum update window? 18:34:16 like "If you don't upgrade rawhide at least once every week/two weeks/month stuff may break" 18:34:34 notting: for people that already have 4.7.1-1.fc12 that's fine. For people who don't, they're going to be a lower n-v-r and update to the .f11 one 18:34:49 I don't think F12 would /ship/ like this, just be a crutch until after Alpha 18:35:09 wwoods: one week maybe 2 seems reasonable to me 18:35:25 f13: id think 2 weeks 18:35:36 f13: *shrug*. wfm. 18:35:50 f13: we'd be breaking our own rawhide-never-goes-backwards rule 18:36:11 so I'd assume that the SOP should require double that (4 weeks) between the compatibility package hitting the repo (i.e. RPM with xz support) and the rebuild landing 18:36:14 notting: ah, good point. 18:36:32 wwoods: that's really long 18:36:41 er, wait, maybe not double. maybe 3 weeks. or two weeks and a day. or something. 18:36:44 and going to make feature freeze a lie 18:36:56 yeah obviously this case is grandfathered in 18:36:58 or we're going to have to start doing our rebuilds /after/ feature freeze 18:37:13 and adjusting schedules to cope with that 18:37:28 does a two-week window work, then? 18:38:17 dunno. 18:38:20 f13: id say 2 weeks + 1 day 18:38:34 I'd say work with dgilmore to put together an SOP and present it to the community for feedback 18:38:36 for when the rebuild content would land 18:38:40 so next tuesday 18:38:55 it's too late to change anything for this rebuild 18:38:57 wwoods: lets talk afte the meeting 18:38:59 sure 18:39:03 because packages are already landing that require the new rpm bits 18:39:42 f13: right but the rpm that needs new glibc that needs new rpm has not yet landed right? 18:39:50 f13: its in the rebuild chain? 18:40:08 in this case we might just need to muddle through without the SOP to avoid blowing the schedule 18:40:16 but surely we can learn from this 18:40:32 dgilmore: maybe? 18:41:00 dgilmore: no 18:41:07 f13: i think we need to carefully land things with circular deps 18:41:09 yeah, latest rpm build doesn't require the new glibc 18:41:12 dgilmore: panu did a 4.7.1 rebuild in the middle/late week 18:41:20 f13: it does on x86. not on x86_64. (not sure why) 18:41:26 dgilmore: if by carefully you man "all at once". 18:41:34 notting: ergh. 18:41:40 notting: delay of build start 18:41:42 notting: thats odd 18:41:54 notting: koji makes sure that the same buildroot is used on all arches 18:41:57 the x86 build could have started well before the x86_64 build 18:42:00 oh 18:42:06 f13: doesnt matter 18:42:07 x86 could have different symbol versions than x86-64 18:42:11 gotcha 18:42:24 koji 1.3 uses repoid so they all use the same repo 18:42:53 we're getting off in the weeds here though 18:43:03 We've got an action item, I'd like to see some work happen outside the meeting. 18:43:09 right 18:44:23 ok, lets move on to the next topic. 18:45:05 #topic No Frozen Rawhide 18:45:15 so yay, this got passed by FESCo 18:45:24 https://fedoraproject.org/wiki/No_Frozen_Rawhide_Proposal for the logs 18:45:50 I think the biggest blocker here is bodhi 18:46:07 which means I need to file a ticket in bodhi for the changes. 18:46:18 #action f13 to file a ticket with bodhi to get changes in to support no frozen rawhide 18:46:34 once the tickets exist, also make sure it's on my radar, and that I know it's a high priority :) 18:47:18 There is hope we can get it done by alpha freeze, but if we don't get it ready by alpha freeze, we punt to F13 18:47:22 lmacken: noted 18:47:47 If we do get it done in time, we'll have to adjust our schedules a bit too, so we'll need to send poelcat a 6-back of $beverage 18:49:19 other than that it'll just be some more compose tool work with cron jobs and such, and letting the mirrors know that changes are coming to layouts 18:49:23 and lots of wiki page changes. 18:49:58 anything else on this before we move on? 18:51:05 #link https://fedoraproject.org/wiki/No_Frozen_Rawhide_Proposal 18:51:05 f13: who has the action to do magic tag-fu on the f11 rpm? 18:51:11 f13: and are we bumping the release or not? 18:51:34 notting: good question. 18:51:46 #topic Rawhide fixup for rpm deps 18:52:00 so, the options are: 18:52:14 1) tag existing F-11 package (and break 'rawhide never goes backwards' rule) 18:52:21 2) build a bumped F-11 package, and tag that 18:52:57 we probably should do 2 18:52:59 so I'm +1 on 2 18:54:23 i can do that. just build into updates-candidate, tag for f12, but not push in bodhi for f-11? 18:54:36 sounds like a plan 18:55:31 ok, i'll do so. i'll e-mail the rpm team so i don't utterly confuse them 18:57:05 #agreed We will build a bumped .f11 package and tag it directly into rawhide (but not push it as an update in F11) 18:57:18 #action notting will build, tag, and inform the rpm team about the rpm changes. 18:57:45 alright, couple more topics. 18:57:50 #topic Fedora 12 Alpha 18:58:01 Feature freeze is tomorrow, but we're still not done with rebuilds. 18:58:18 and something is making compose images not show up, due to package mismatch?! I've got to investigate that more after this meeting/lunch 18:58:27 #action f13 will investigate why we aren't getting installer images in rawhide 18:58:39 need to get that working before our test compose is due later this week 18:58:53 and our last topic 18:58:58 #topic Package Signing 18:59:10 I made great progress with mitr over the last week on sigul, our new rpm signing software. 18:59:45 I've got a test deployment of it here at my house in virt and have been able to successfully generate a new key and import an existing key, and sign + import + write out multiple builds with each key using sigul 19:00:17 I also re-write sign_unsigned as sigulsign_unsigned that is far more simple, far less code, and capable of taking in a list of builds or a tag and running sigul in batch mode 19:00:37 smooge and I are working on getting our new signing hardware in place and bridge guest prepared for deployment 19:00:50 It's my goal that we'll be able to use sigul and sigulsign_unsigned by the end of this week. 19:01:02 yes.. I am trying to get the box to do what its supposed to do. 19:01:02 Then we can look at automating it in some ways 19:03:35 #open floor 19:03:37 whoops 19:03:40 #topic open floor 19:03:54 If there are no other topics to discuss, I'll close the meeting in 5 19:05:19 are we signing alpha? 19:06:06 yes 19:06:29 I can't guarantee signed packages /after/ alpha though 19:07:17 Alright, thanks all! Until next week! 19:07:18 #endmeeting