17:03:25 #startmeeting Fedora Release Engineering Meeting 17:03:25 Meeting started Fri Oct 29 17:03:25 2010 UTC. The chair is Oxf13. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:03:25 Useful Commands: #action #agreed #halp #info #idea #link #topic. 17:03:30 #meetingname fedora-releng 17:03:30 The meeting name has been set to 'fedora-releng' 17:03:38 #topic Roll Call 17:03:47 * nirik is lurking around. 17:04:03 * dgilmore is here 17:04:04 ping: notting dgilmore nirik rdieter spot lmacken poelcat wwoods 17:04:06 here 17:04:52 * skvidal lurks b/c he has a question for later 17:05:44 well hopefully notting will join us soon. 17:05:58 Oxf13: just assign tasks to him until he shows up 17:06:12 #info present are nirik dgilmore rdieter skvidal Oxf13 17:06:20 #topic Fedora 14 Release 17:06:36 We all know that F14 is eminent 17:06:49 * nirik cheers. 17:06:56 from a releng perspective i think things have been smooth 17:07:00 The content has already been staged for mirrors, and the content exists on the torrent server. I just have to re-arrange it and create torrent configs 17:07:07 yes, things went very smooth for us 17:07:12 sorry, ridiculous line @ lunch 17:07:14 #info notting also present 17:07:14 back now 17:07:37 let me know when .torrents are up and I will seed. 17:07:41 I only had a couple of late nights making sure all the builds were in the right place in bodhi for the blocker bugs. 17:08:14 We did a number of unofficial test composes leading up to the one release candidate, but that one candidate was on time, and good enough, which is pretty awesome. 17:08:21 is that a first? 17:08:23 awesome^2 17:08:30 notting: AFAIK yes 17:08:38 notting: without actually fact checking, I believe it's a first. 17:09:13 at least I don't recall ever having only one RC for final, but I wasn't around for Fedora Core 1~4 17:09:39 #info smooth from releng 17:09:46 #info only needed one RC, and it was delivered on time 17:09:54 #info almost ready to go, just need to create torrent configs 17:10:04 #action Oxf13 to finish creating torrent configs and contact pre-seeders 17:10:45 My plan is to unlock the regular mirrors and spin mirrors my Monday night before I go to bed, so somewhere around 11pm Pacific. This should give loads of time for mirrors to sync up the open bit. 17:11:06 I plan to turn on the torrents at that time too 17:11:09 Oxf13: that worked well for beta and alpha 17:11:41 That /should/ leave no actions left for releng on release morning 17:12:03 infrastructure will take care of turning on the websites, and jared smith will send out the release announcement. 17:12:10 the only downside to our current process is that it pretty much gets rid of excitement around the release time. ;) But oh well. 17:12:11 I may even sleep in that day 17:12:14 Oxf13: mm redirect ? 17:12:23 notting: that can actually remain as is for a day 17:12:37 notting: that can be done after the fact 17:12:50 there are likely to be more mirrors with a populated development/14/ than an unlocked releases/14/ 17:13:16 * poelcat here 17:14:01 #info poelcat is also present 17:14:14 #info Plan to unlock mirrors and torrents the evening before release 17:14:37 #info releng tasks would be complete for release morning. Release afternoon/evening will do mirrormanager redirect change. 17:15:00 What we should also do is try to make sure that F14 stable updates are in fact in a stable state without broken deps, at least for release day 17:15:20 they're there now. dunno about state. 17:15:36 nod 17:17:04 i've been tweaking the ticket list as things come up 17:17:06 Does anybody have anything else to discuss regarding the release of F14? 17:17:09 notting: thanks 17:17:18 i dont 17:17:23 I do have an updated compose+stage SOP notes I've been working on 17:18:35 alright lets move on 17:18:45 #topic Fedora Release Engineering Future 17:18:59 I've hinted around this in various places before, but it's becoming more official 17:19:14 I'm going to take a year~ break from Fedora release engineering tasks 17:20:02 woah. Still going to be working on fedora? or have some other plans? 17:20:03 I'm going to be working internally at RHT to migrate their source control much like I did for Fedora. However RHTs setup is more complex from workflow and integrated systems, rather than sheer size and contributor base 17:20:20 * dgilmore is going to be taking a much more active role in Fedora Releng 17:20:29 gosh. thanks for your work, Oxf13. 17:20:36 as such, I figure it'll take a good 9 months, but a year for padding to complete the work 17:21:06 as dgilmore said, he is going to take a much more active role in Fedora releng, essentially taking over for me while I work internally 17:21:41 Oxf13: thanks for all your work... I hope you will still have time to be active in other places in fedora time permitting. 17:21:43 I'm certainly going to remain available for contact, and keeping some eye on what's going on, but I won't be following thing closely, or monitoring ticket queues and the like 17:22:03 Oxf13: thanks for all your work. 17:22:15 indeed thanks Oxf13 17:22:22 the work I do within RHT will have some benefit to Fedora as well, as it should make it easier to move changes back/forth between Fedora and RHEL, which has been a source of problems 17:22:41 and I do plan to return to Fedora releng duties when this task is done 17:22:49 * Oxf13 tries to capture this for meeting notes. 17:23:17 #info Oxf13 (Jesse Keating) to step down as Fedora Release Engineering lead for a years time to work on internal Red Hat source migration 17:23:37 #info dgilmore (Dennis Gilmore) to take over as Release Engineering Lead 17:23:56 #info Transition to happen in 1~ month's time after dgilmore returns from vacation 17:24:32 #info Oxf13 to remain available for contact and knowledge base throughout this time 17:24:44 still coming to fudcon? 17:24:51 I will also be at FUDCon in AZ 17:24:58 * dgilmore booked fudcon airfare lastnight 17:25:12 as for later fudcons, I cannot say. Most likely not, so that budget can go to dgilmore to go 17:25:20 * nirik hopes to make it. I guess I should book stuff sometime. 17:25:57 nirik: soon -and you'll need to call to get the group rate 17:26:03 their website appears to be... dumb 17:26:10 phone? how quaint. 17:26:46 nirik: it'll get you further 17:26:50 people are flexible 17:26:52 machines suck 17:27:47 heh 17:27:54 Anything further on the current topic? 17:28:04 will we continue to have regular meetings? 17:28:09 on fridays? 17:28:13 * dgilmore just notes he is looking forward to business as usual 17:28:19 * poelcat thinks that is something we should get back to 17:28:24 continue? :) 17:28:41 poelcat: we will have meeting. we can re-evaluate when if need be 17:28:44 I think regular meetings are a good thing, provided there are things to discuss/note. 17:29:40 Yes, I've been far far too slack in having meetings 17:29:46 nirik: right, 17:29:55 I hope dgilmore can improve upon this, because I've set a really low bar :) 17:30:08 i know there's been other things going on, i just think if we want to grow the team having a consistent meetup helps 17:30:12 Oxf13: :) i hope so 17:30:43 anyways 17:32:31 #action dgilmore to take over Fedora releng meetings, potentially arranging a new time 17:32:40 (again, once he gets back from vacation) 17:33:00 Oxf13: i guess we should go over things for the next 3 weeks 17:33:52 do we have any real tasks other than the updates train? 17:33:59 notting: just updates 17:34:14 and working on things for F-15 17:34:38 we need to ratify the schedule too 17:34:50 * nirik is happy to help pushing updates. 17:34:55 poelcat: maybe have that as the next item for this meeting after this? 17:35:41 afaik, the next month should just have schedule ratification and updates pushes, and general maintenance of development systems 17:35:55 oh yeah, I'll still be working on fedpkg too in the mean time 17:35:57 i can help occasionally with updates, but i'm traveling/at-a-conference all next week 17:36:06 even during my year off, since fedpkg ill be the basis of the internal tool. 17:36:22 #info Oxf13 to continue working on fedpkg. 17:36:43 nirik: are you ok to be point on updates for the next few weeks? 17:36:59 sure. 17:37:25 I have not yet done fedora updates pushes, but I should be all set for them and have instructions, etc. 17:37:28 ok. feel free to pester us or lmacken if things explode 17:37:41 I plan to update http://fedoraproject.org/wiki/Pushing_fedora_updates with all I have seen soon too. 17:37:50 nirik: its basicaly the same as epel4 and 5 17:38:15 yeah, except for the sigul part, but thats working fine with epel6 for me. 17:38:28 yep 17:38:43 cool 17:38:44 Im confident that you will be fine 17:38:54 #action nirik to be on point for updates for the next few weeks 17:39:24 and tagging tickets should be relatively well handled already 17:39:52 * nirik hasn't done any fedora tagging requests, but can keep handling the epel ones. 17:40:15 there is abunch of people who handle fedora ones 17:41:06 * nirik always sees rdieter doing them. ;) 17:41:19 yeah he does alo as does notting 17:41:24 Oxf13: and i do some 17:41:38 and there is a guy in Brno who I brought up to speed on them 17:41:56 Nick Petrov 17:42:07 that's nice to have more timezone coverage 17:42:25 till does some also 17:44:35 Ok, ready to move on? 17:44:48 yep 17:45:22 #topic Fedora 15 Schedule 17:45:31 I admit, I haven't looked at the proposed schedule yet 17:45:35 poelcat: share us a link? 17:46:04 https://fedorahosted.org/rel-eng/ticket/4233 17:47:03 one thing looking at it i think would be nice to change is the feature freeze 17:47:14 maybe have it 2 weeks after fudcon 17:47:39 though that likely means extending everything 2 weeks 17:48:15 otherwise features comming out from the fudcon will have to be F-16 features 17:48:21 +1 dgilmore 17:49:38 couple of data points I'd be interested in. 17:50:00 more details in the upstream project timelines, and a comparison to Ubuntu / SuSE release schedule 17:50:07 of course im assuming that the work done on features can be quickly done 17:50:23 https://fedorahosted.org/fesco/ticket/467 17:50:28 may also play into this. 17:50:49 April 28 2011 is the next projected Ubuntu release 17:51:03 which is just 2 days after the proposed release date 17:51:11 mirrors would not like that 17:51:31 yeah 17:51:37 that would be hell on them 17:52:19 * nirik notes the last ubuntu release caused pretty much no mirror load here, but it was on a sunday and not sure we were in their main mirror rotation. 17:52:27 # Thu, Mar 10 2011, openSUSE 11.4 Public Release 17:52:36 opensuse is due in march 17:53:20 may-nov seem to work the best for us 17:54:19 and you have the holidays in december that seems to put a hold on everything 17:54:41 KDE 4.6 is due at the end of january 17:54:43 so the proposal to kick the feature freeze out 2 weeks to take advantage of FUDCon, and pushing everything else out 2 weeks would clear us of Ubuntu 17:56:27 gnome 3 has a release date of april 6 17:56:40 Oxf13: yeah it would 17:56:48 xfce 4.8 has a release date of... sometime. ;) 17:57:02 nirik: when its done right :) 17:57:26 #proposal Move feature freeze and all later dates 2 weeks later to make way for FUDCon and Ubuntu release 17:57:26 they are making progress, but have no firm release date set anymore. 17:57:35 I'm +1 obviously 17:57:38 im +1 17:57:46 sounds fine to me. 17:58:12 that does add a lot of pressure on us not to slip 17:58:20 yes it does 17:58:29 notting: why does it add pressure? 17:58:46 well, if we push two weeks out, and then slip 3 more weeks... we just get really late 17:58:51 oh 17:58:58 late as in "past may 1st" ? 17:59:06 june 17:59:36 poelcat: I thought the target was "May Day" eg may 5th :) 17:59:43 may day and halloween 18:00:02 or am I confusing holidays and may day is actually may 1? 18:00:13 I'm an idiot. 18:00:15 may day is the first 18:00:17 yes, may day = may 1st :) 18:00:28 with the 2 weeks we are at may 10 for GA 18:00:32 so 2 extra weeks would put us at 2011-05-10 18:00:37 dgilmore: :) 18:00:42 perhaps one too many margaritas on cinco de mayo 18:00:57 it all makes sense to me 18:01:09 geven that F14 only slipped one week over all, I think we're on the right track for on-time delivery 18:01:10 i don't see us having any pressure if we say that is our GA date from the start 18:01:44 and it sounds like we have solid reasons for adding two weeks past our normal start date 18:02:01 the only thing to (minorly) take into account is it takes time away from Fedora 16 18:02:12 right 18:02:12 * poelcat realizes that technically that isn't true 18:02:23 but people do tend to think in single release mode 18:02:26 well, if we slip a lot, that makes f16 spill into holidays right? 18:02:33 it means we branch two weeks later than normal 18:02:42 * dgilmore is assuming that having feature freeze after fudcon means we get some extra coolness into F-15 18:02:51 nirik: no, because we always target the second release of the year at Oct 31~ 18:02:53 or at least potentially. 18:03:09 that will give fedora 15 a longer development window than usual 18:03:10 it just makes "time for development" a bit shorter 18:03:19 fwiw 18:03:20 eg, time when rawhide == F16 before we branch F16 18:03:21 right 18:03:27 i'm +1, it's good enough of a reason to aim later 18:03:36 I think that's enough votes to pass 18:03:49 #agreed Move feature freeze and all later dates 2 weeks later to make way for FUDCon and Ubuntu release 18:03:54 * poelcat wants to double check with QA to make sure they didn't need any different spacing 18:03:55 with systemd and gnome-shell, it will be good to have more time to make sure all is well. 18:04:00 Can we take that as an implicit acceptance of the schedule? 18:04:20 Oxf13: i think so 18:04:27 yes, contingent on QA's feedback would be my suggestion 18:04:49 they have a meeting on monday i think 18:05:08 yeah, I meant from a releng POV 18:05:29 #agreed Release Engineering accepts amended F15 schedule 18:05:49 thanks poelcat 18:05:50 * poelcat will add note to ticket when updated dates are posted 18:06:01 thank you, great constructive discussion 18:06:01 skvidal: ping, you had a topic? 18:07:17 yes 18:07:24 pkg signing and desires of input 18:07:28 ok 18:07:40 #topic Package Signing 18:07:40 there is movement afoot to change pkg signing kinda a lot 18:07:46 yes 18:07:51 specifically, do away with pkgs being signed by gpg keys 18:07:55 just to start off, what I had planned for package signing 18:07:57 and do away with attached signatures 18:08:08 We already have an approved proposal to reduce our signing down to one key for all packages 18:08:27 strange things are afoot at the rpm -K? 18:08:34 and to automatically sign any "official" build that comes through koji (eg a non-scratch build done from source control) 18:09:06 #info releng plans on reducing to one key to sign all rpms and automatically signing every official build that comes through koji with it. 18:09:07 notting: there will be a bunch of changes to things like that, yes - lots of changes 18:10:01 #info Changes coming in package signing, desire to move to detached signatures, and no longer using gpg for the signatures 18:10:23 who is the driver of these changes, or these desires? 18:10:24 skvidal: x509 certs for signing? 18:10:30 dgilmore: thats the plan 18:10:38 Oxf13: multiple people, actually 18:10:48 mjcox and bress are among the leaders 18:10:48 I know long ago jeremy katz had talked about x509, as a way to get "layered" signatures, but it never really went anywhere 18:11:00 panu is interested in killing gpg in rpm 18:11:03 #info effort led by RHT security team 18:11:24 all of the discussion started with the gpg ca-key requirement for a rhel release 18:11:41 and in implementing that I discovered we'd be re-inventing all of CRLs for gpg keys by doing it 18:11:48 and the question became - why are we doing this? 18:11:55 skvidal: so, rpm would only keep md5/sha1 'file was not eaten by a grue during download' sigs? 18:12:23 notting: there are many details to work out - but it is likely we'd be storing the detached sig information on the system - but probably not in the rpmdb 18:13:36 I am curious what the motivation is 18:13:46 other than "hey I don't have to maintain code for that anymore!" 18:13:54 skvidal: and what kind of timeframe? 18:14:06 dgilmore: well I think the discussion is for this for rhel7ish timeframe 18:14:13 Oxf13: x509 web of trust vs. gpg web of trust? 18:14:16 skvidal: ok 18:14:36 Oxf13: and the problem is not just 'I don't havve to maintain this code naymore' 18:14:39 it is also 18:14:43 notting: I guess that's a thing, somebody would have to explain in small words why that's a "better" thing. 18:14:52 "hey, I don't have to maintain this path of dev that NO ONE ELSE IS FOLLOWING" 18:15:05 Oxf13: b/c no one in the world gives a crap about gpg keys 18:15:09 but EVERYONE cares about x509 18:15:15 Oxf13: gpg, all you have is distributed keyservers that have to be uploaded to, revocation is a pain? 18:15:18 so no REAL work is being done to check out gpg keys 18:15:28 fair enough 18:15:29 with x509, we have established CAs (for better or worse), etc. 18:15:38 and the CRLs for x509 18:15:45 I was able to grasp gpg fairly easily. Every time I look at x509 my brain melts 18:15:46 have an established interface 18:15:53 Oxf13: do me a favor 18:16:04 Oxf13: go read the trust-establishing code for gpg 18:16:08 hah 18:16:12 x509 will look like a dainty walk in the park 18:16:15 I'm not debating that x509 is better 18:16:25 I'm just stating that I've had a hard time coming up to speed on it 18:16:32 skvidal: question from me is do we get any benefit from using an established ca, or can we publish our own 18:16:41 we can publish our own 18:16:49 at least for fedora 18:16:52 cacert! :) 18:17:03 and while rpm might be able to get rid of some code, koji et al are going to have to change a bunch of code 18:17:17 except 18:17:19 that code gets EASIER 18:17:32 b/c they don't; have to magically assemble an rpm + header 18:17:34 it's just an rpm 18:17:38 and a detached sig 18:17:43 sure, easier code, but different code :) 18:17:43 that means sigul goes faster? 18:17:51 (well the detached sig may be slightly more complicated) 18:17:51 notting: it goes different 18:18:00 notting: getting more mitr time to make it go different may be a challenge 18:18:06 we store a detatched sig today 18:18:17 it does mean we get to store less on /mnt/koji 18:18:18 ooof. this becomes pain for things like mash, etc. 18:18:22 that im happy about 18:18:45 notting: why? 18:18:53 skvidal: it has to shuffle around more files, no? 18:19:02 1 more file per pkg 18:19:14 that's one more than it does now ;) 18:19:15 which, I suspect, will have a very similar name 18:19:17 1 more file per sig, if we don't go a single sig 18:19:32 and that would be another advantage of going to a detached sig 18:19:36 layering signatures 18:19:44 for remixes, for example. 18:19:59 and before anyone mentions --addsign 18:20:04 I'd like to suggest someone TRY --addsign 18:21:16 yeah, doesn't add 18:21:17 it replaces 18:21:20 obvs. we just move to v6 payload, which includes the normal rpm and an arbitrary number of detached signatures attached 18:21:39 .... 18:21:52 I really do like the idea of /not/ fucking with the payload to change a signature 18:22:05 makes deltas and downstreams a lot easier 18:22:11 if we're going to add a ton more files, we need to consider splitting the tree up by alphabet 18:22:15 and if we ever (oh please say no) have to resign and resync something 18:22:24 mdomsch: :) 18:22:25 it was a joke 18:22:44 it goes with the XML headers 18:23:01 actually 18:23:08 if we go to detached sigs 18:23:22 one of the big reasons for going to a single sig for every rpm out of koji goes away 18:23:40 which is keeping hardlinks across the releases and reducing mirror shuffle at signing days 18:24:09 i suppose it boils down to: 18:24:16 1) does rel-eng have a choice in the matter? 18:24:24 2) is this proposal finalizing anytime soon? 18:24:35 3) is there going to be a F2F (fudcon?) about this? 18:25:04 * dgilmore suspects 1) no) 2) hopefully 3) best if we did 18:25:22 okay 18:25:27 so here's what I'm thinking right now 18:25:49 I have to finish 2 features (maybe today) and then I want to work on this in earnest 18:25:54 (and apologies, i have a stop at 2:30) 18:26:07 specifically - getting the code in nss+python to create the x509 sigs and check them, sanely 18:26:47 I'd like to be in a place to be able to MAKE the sigs and have them functional and available in yum around f15 - but not necessarily for inclusion 18:26:53 and then probably f16 make it a feature 18:27:08 Ok 18:27:23 i can deal with that 18:27:27 #info hopeful timeline is to be able to make x509 sigs by F15, but not necessarily include them 18:27:27 taking time to ramp up a new and potentially destablizing feature? shocking. ;) Sounds good 18:27:32 does this also include re-doing the repository metadata signing that very few (if anyone) use? 18:27:41 can we plan for a F2F on this issue at fudcon? 18:27:47 notting: ideally 18:28:00 notting: it will mean drop kicking ALL of the gpg handing in yum 18:28:05 notting: :) 18:28:07 Oxf13: sounds round to me 18:28:15 I'll put it up as something I'm going to talk about 18:28:26 skvidal: hrm. don't think you can expect everyone to transition overniht 18:28:33 notting: I can WISH! 18:28:58 notting: I agree, of course, but it would mean making the repo-signing capability also support x509 sigs 18:29:20 not that anyone should be signing repos for f-14. oops. 18:29:34 notting: sadly, true b/c of pygpgme :( 18:29:34 :-) 18:30:25 #info skvidal is planning to do a FUDCon session about this. 18:30:38 * skvidal makes a note 18:30:51 anything that lets me stop caring about pygpgme is good in my book 18:31:38 no kidding 18:31:43 Ok, anything else on this topic? 18:31:46 well 18:31:47 maybe 18:31:48 so 18:31:53 how do y'all feel about api breaks? 18:32:02 meh, that's what rawhide is for 18:32:11 b/c a subject that has come up is since rhel6 is coming out sometime 18:32:17 that soon is a good time 18:32:20 we're already getting an "api" break of sorts due to xz and rpm and deltarpm 18:32:23 to kill all the stuff in yum 18:32:27 that we want to get rid of 18:32:31 effectively breaking everyone's code 18:32:41 what do I care, I'm gone for a year :) 18:32:46 skvidal: f-15 is a good time to disrupt the world 18:32:49 har har har 18:32:50 (note, I expect I'll be the one to do the pungi work to fix it) 18:33:00 dgilmore: yah - not sure what the timeline is, yet 18:33:05 but it's something back there in my mind 18:33:13 okay 18:33:15 that's all I had really 18:33:19 thanks for listening 18:33:30 thanks for talking 18:34:25 #info yum api break may be happening soon 18:34:38 #topic Rawhide and deltas 18:35:05 #info xz had api change, which results in an odd deltarpm situation 18:35:25 so, new xz landed, which is different from the xz in F14 18:35:30 the situation is thus 18:35:42 if a package built in F14 or earlier is available in rawhide 18:35:45 and there is a delta for it 18:35:58 a rawhide system will attempt to rebuild the rpm from the delta 18:36:03 but it will use the client xz 18:36:12 and the end result will be something that doesn't match the original rpm 18:36:30 the first step at working around this is thus. 18:36:49 We will tag every package that is currently being inherited into dist-f15 explicitly with dist-f15 18:36:59 we will then break the inheritance between dist-f15 and anything earlier 18:37:23 This will require building in rawhide anything that will show up in rawhide 18:37:37 and ensuring that when a rawhide client rebuilds from a delta, they get something that matches the original 18:37:56 a side note, is that delta generation is currently broken and turned off in rawhide 18:39:13 #info delta generation currently broken in rawhide, will attempt to re-enable after inheritance fixes are done. 18:39:35 #action Oxf13 to break inheritance between F15 and anything earlier, and force tag all currently inherited packages into dist-f15 18:39:48 #info developers who want a new build to show up in rawhide will have to explicitly build for rawhide. 18:39:55 any questions on this? 18:40:03 none 18:40:08 Oxf13: so there's a mass rebuild coming that'll do this? 18:40:08 here 18:40:30 mdomsch: there shouldn't need to be a mass rebuild 18:40:34 k 18:40:46 there will be a lack of deltas for the things we inherited 18:40:57 but we never promise complete delta coverage anyway 18:41:45 alright moving on 18:41:54 #topic Open Floor 18:42:00 any last minute items? 18:42:34 * dgilmore has nothing 18:42:55 * dgilmore wishes everyone well for the release and will be enjoying time off 18:43:03 * nirik has nothing. 18:43:11 dgilmore: yes, do enjoy the sun and the timtams 18:44:06 Ok, that's a wrap then. Thanks everybody. 18:44:13 dgilmore: would you mind posting the minutes? I have to run to another meeting 18:44:16 #endmeeting