15:30:46 <dgilmore> #startmeeting
15:30:46 <zodbot> Meeting started Mon Mar 31 15:30:46 2014 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:30:46 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:30:54 <dgilmore> #meetingname releng
15:30:54 <zodbot> The meeting name has been set to 'releng'
15:31:29 <dgilmore> #chair dwa tyll_ masta nirik sharkcz
15:31:29 <zodbot> Current chairs: dgilmore dwa masta nirik sharkcz tyll_
15:31:42 <dgilmore> #topic roll call
15:31:47 <dgilmore> who is here?
15:32:03 <bochecha> I am
15:32:16 <dgilmore> hey bochecha
15:32:38 <nirik> morning
15:33:17 * masta is here
15:33:50 <dgilmore> not seen peep from dwa today
15:35:26 <dgilmore> #topic tickets https://fedorahosted.org/rel-eng/report/10
15:35:51 <dgilmore> https://fedorahosted.org/rel-eng/ticket/5854
15:35:56 <dgilmore> masta: any update?
15:36:34 <masta> dgilmore: honestly I completely forgot about that ticket. I can just knock it out, it's very easy
15:36:57 <masta> sry for the bad but honest answer
15:37:05 <dgilmore> masta: no problem
15:37:25 <dgilmore> https://fedorahosted.org/rel-eng/ticket/5362
15:37:33 <dgilmore> I have no update on this one either
15:37:42 <nirik> it would be nice to get it done. ;)
15:37:59 <dgilmore> yeah, need to sit down and sort it out this week
15:38:08 <dgilmore> at least to see what it will give up
15:38:10 <dgilmore> us
15:38:15 <dgilmore> https://fedorahosted.org/rel-eng/ticket/5870
15:38:34 <dgilmore> nirik: so this is likely a long time project
15:38:52 <nirik> yeah.
15:39:01 <nirik> I didn't think we would solve it in 5minutes. ;)
15:39:13 <tyll_> Hi all
15:39:18 <dgilmore> hey tyll_
15:40:06 <nirik> ideally, we would get sigul to grow support for unattended signing (that it could lock down via ip at least and possibly some other means)
15:40:14 <dgilmore> yeah
15:40:23 <dgilmore> maybe a ssl key for auth
15:40:30 <nirik> failing that we would run another thing that just does these types of signing.
15:40:42 <nirik> for koji and copr
15:40:44 <bochecha> as I mentioned in the ticket, that's what we've done at $dayjob (a signing server simpler than sigul though, but ip-locked)
15:41:11 <dgilmore> if we auto sign the rpms, we likely will need to sign manually repodata
15:41:17 <masta> I'd say let them be signed with an uncontrolled key... not to imply a lack of security.... but something we automate
15:41:21 <nirik> I definitely don't want the private key or passphrase on the hub.
15:41:27 <dgilmore> which will put a manual step at the end of compsoses and psuhes
15:42:05 <dgilmore> nirik: maybe a fedmsg listener, that does it elsewhere
15:42:07 <nirik> obs-signd may be an ok option. I can ask security folks to look at it...
15:42:20 <dgilmore> i know nothing about it
15:42:30 <nirik> dgilmore: yeah, thats an option too
15:42:45 <masta> i like the fedmsg idea
15:43:26 <tyll_> #topic ticket 5870 rawhide signing
15:43:32 <dgilmore> nirik: that way we can at least do it on a locked down box that is not publicly available
15:43:37 <tyll_> #info don't want the private key or passphrase on the hub
15:43:40 <nirik> dgilmore: true.
15:43:49 <nirik> there would be a race then tho
15:43:56 <dgilmore> indeed
15:44:02 <nirik> build something right before rawhide compose, might not get signed in time
15:44:29 <dgilmore> two ways to deal with it
15:44:46 <dgilmore> we culd have buildrawhide make sure everything is signed
15:45:01 <dgilmore> or we add a slight delay
15:45:10 <dgilmore> and tag into a holding tag
15:45:13 <nirik> so if we signed all rawhide builds with a rawhide key, what happens at branched? we resign with FN key and each package has 2 signatures/rpms written out?
15:45:17 <dgilmore> then move when its signed
15:45:34 <masta> +! slight delay
15:45:48 <nirik> we have talked about a pending tag before... would allow some place to gate.
15:46:11 <dgilmore> nirik: well we could resign,or we just sign with the next fedora key
15:46:25 <dgilmore> we will make a f22 key at f21 branching
15:46:51 <dgilmore> rawhide going forward at that point could be signed with the f22 key
15:47:16 <nirik> ok, then in that model there isn't a rawhide key, we just sign it with the key for the next fedora when it was built?
15:47:34 <dgilmore> yep
15:47:38 <nirik> that may not work on new installs, wouldn't import the older keys?
15:48:00 <dgilmore> we would need to make sure that it can have both
15:48:20 <nirik> well, there may be even older at some point...
15:48:28 <dgilmore> other option is to sign everything in rawhide at branching with the next fedora key
15:48:35 <nirik> for things that faill mass rebuild that we keep shipping
15:49:32 <nirik> I guess I prefer that, but I'm not sure fully why. ;)
15:49:47 <tyll_> we could address the old rpms at branching by resigning them
15:49:58 <nirik> it might make it more clear when/fromwhat you installed something.
15:50:06 <dgilmore> at branching we will need to make sure everything is signed with the release key
15:50:26 <tyll_> #info  at branching we will need to make sure everything is signed with the release key
15:50:42 <dgilmore> making sure that rawhide has access to and can import 2 keys should be doable
15:51:02 <dgilmore> for instance right now f20 and f21 keys
15:51:33 <nirik> yeah, but once we go to f22 there may be stuff that fails mass rebuild and is still f20, so for f22 we still might need the f20 key, etc...
15:51:42 <dgilmore> nirik: no
15:52:00 <tyll_> nirik: this would be fixed by re-signing everything when branching
15:52:03 <dgilmore> nirik: we would need to make sure at branching everything is signed with the f21 key like we do today
15:52:07 <nirik> ah I see. ok.
15:52:15 <tyll_> #info < dgilmore> making sure that rawhide has access to and can import 2 keys should be doable
15:52:32 <nirik> so, at branch, f21 gets all f21 signed and rawhide gets all f21 signed from there, but f22 after that?
15:53:08 <nirik> or rawhide always has a rawhide key?
15:53:32 <dgilmore> right
15:53:48 <dgilmore> I do not think we ever have a rawhide key
15:53:52 <tyll_> signing rawhide with the F22 key is less effort, because not every RPM needs to be re-signed
15:54:09 <dgilmore> just have a key for the next release
15:55:01 <masta> so if we can automate signing rawhide with the next branch key, can we do the same for non-rawhide builds?
15:55:10 <nirik> but then that requires sigul be used right?
15:55:24 <tyll_> #info at branch, f21 gets all f21 signed and rawhide gets all f21 signed from there, but f22 after that
15:55:53 <nirik> ie, if sigul has the f21 key in it, it has to be used to sign you can't use signd or something else...
15:57:19 <dgilmore> nirik: right
15:57:25 <nirik> so, if we wanted to use another service (signd or whatever) we would need to have a seperate key
15:58:13 <dgilmore> not necessarily
15:58:23 <dgilmore> you can import keys into sigul
15:58:40 <dgilmore> but that does kinda defeat part of its purpose
15:58:45 <nirik> yeah.
15:59:35 <nirik> so, let me do this:
15:59:55 <nirik> I'll talk with threebean about fedmsg and triggering signing from it, see if there's concerns there.
16:00:10 <dgilmore> okay
16:00:22 <nirik> I'll also ping mitr and see if there's any chance for sigul work this year... and how hard it would be to add a way to allow non passphrase/unattended.
16:00:29 <dgilmore> at the least I think doing that we can secure the signing process
16:00:32 <tyll_> #action nirik will talk with threebean about fedmsg and triggering signing from it, see if there's concerns there.
16:00:47 <tyll_> #action nirik will ping mitr and see if there's any chance for sigul work this year... and how hard it would be to add a way to allow non passphrase/unattended
16:00:56 * nirik nods.
16:01:00 <dgilmore> may be some other way for us to get build notifications
16:01:24 <dgilmore> at the least we can look into feasability
16:02:53 <dgilmore> anything anyone else want to add here>
16:02:54 <dgilmore> ?
16:03:30 <tyll_> Did you notice that I mentioned in the channel that there are GPG smart cards that can hold RSA 4096 bit keys?
16:03:48 <tyll_> This might be something cheap to get HSM like security
16:04:01 <nirik> yeah, thats an option... we already use them for secure boot signing.
16:04:08 <dgilmore> tyll_: we use some for signing the binaries for secure boot
16:04:53 <tyll_> ah, that's great
16:06:49 <dgilmore> nirik: ill leave the meeting keyword on the ticket, we can follow up next week
16:06:54 <nirik> sounds good.
16:07:01 <nirik> talking to mitr in devel now too. ;)
16:07:10 <dgilmore> #topic Secondary Arches: s390
16:07:19 <dgilmore> no sharkcz
16:07:26 <dgilmore> #topic Secondary Arches: ppc
16:07:36 <dgilmore> dwa: ping, status update
16:07:52 <dgilmore> he may not be here
16:09:02 <dgilmore> #topic Secondary Arches: aarch64
16:09:10 <dgilmore> aarch64 is getting there
16:09:15 <dgilmore> its following rawhide
16:09:37 <dgilmore> need to look at doing some nightly installer composes
16:09:38 <masta> the usual challenges, getting patched upstream
16:10:19 <masta> dgilmore: that would be nice, right now it wouldn't work but maybe a few more key builds it could
16:10:38 <dgilmore> masta: would give us a list of things to fix
16:11:42 <masta> sure, I'll see what I can find out.
16:11:50 <dgilmore> we are looking good to get a f21 release
16:11:55 <dgilmore> okay
16:12:03 <dgilmore> #topic mass rebuild
16:12:38 <dgilmore> with the f21 schedule set and the gcc 4.9 change just proposed we need to look at when we will do a mass rebuild
16:13:15 <nirik> there's still changes rolling in...
16:13:18 <nirik> but yeah
16:13:39 <dgilmore> https://fedoraproject.org/wiki/Releases/21/Schedule
16:14:07 <dgilmore> https://fedoraproject.org/wiki/Changes/GCC49
16:14:19 <dgilmore> there is already the extra security change in
16:15:02 <jreznik> dgilmore: do you want to create ticket to track potential mass rebuilds?
16:15:11 <dgilmore> jreznik: we should
16:15:18 <dgilmore> we have branching at no earlier than 2014-07-08
16:15:33 <dgilmore> #info branching is no earlier than 2014-07-08
16:15:34 <jreznik> I was thinking about it when announcing that change... I'll do it
16:16:29 <dgilmore> so we really want to make sure that the  mass rebuild is done by June 9th at the latest
16:17:13 <dgilmore> so we need to start in the next 7-8 weeks
16:17:34 <tyll_> #info we really want to make sure that the  mass rebuild is done by June 9th at the latest
16:17:36 <masta> I want to get in on the mass rebuild action when it happens
16:17:50 <jreznik> gcc 49 release is the first half of april
16:17:53 <masta> maybe even fire it off, would be good trainning
16:18:00 <dgilmore> masta: well the script that does it can only be run by one person
16:18:25 <masta> dgilmore: ah, ok then... as I expected =(
16:19:03 <dgilmore> jreznik: im thinking that it would be good to get the mass rebuild done early may
16:19:09 <dgilmore> nice and early
16:19:25 <masta> would give time to address the fall-out
16:20:09 <masta> aarch64 folks probably want to rush to get a few patches in prior to the mass rebuild
16:20:28 <dgilmore> masta: well done my June 9th gives us 4 weeks for cleanup
16:20:28 <jreznik> seems like that new gcc should not be a big issue from what's written in change, so doable
16:21:55 <dgilmore> right
16:22:13 <dgilmore> need to make sure ppc64le and aarch64 are in good shape
16:22:49 <dgilmore> and work out if there are any other changes needing a mass rebuild
16:22:56 <dgilmore> then work out some scheduling
16:23:14 <tyll_> for the java change it was mentioned but not required
16:23:29 <jreznik> how much does sec mass rebuilds affects primary (except the stuff that has to be in?)
16:23:36 <tyll_> https://fedoraproject.org/wiki/Changes/Java8
16:24:18 <dgilmore> mass rebuilds do not really accomodate specs needing changes other than nvr bumps
16:24:39 <dgilmore> jreznik: just want to make sure that they have things they need in
16:26:02 <dgilmore> anyway, we need a ticket and to track whats needed and when to do it
16:26:08 <dgilmore> #topic open floor
16:26:18 <dgilmore> does anyone have anything for open floorr?
16:26:19 <jreznik> dgilmore: https://fedorahosted.org/rel-eng/ticket/5877
16:26:29 <dgilmore> jreznik: cheers
16:27:10 <masta> nada
16:27:21 <jreznik> dgilmore: obvious open floor question is next - today I was asked by Christian, how they can officially produce initial WS composes snapshots
16:27:44 <dgilmore> jreznik: they cant
16:28:02 <bochecha> about the lookaside hashing, last meeting we decided to talk about it on the list, but there hasn't been much talk on it :-/
16:28:06 <dgilmore> jreznik: we do need to work out exactly how we will compose things
16:28:13 <jreznik> see my email... I think it's a bit premature, as I don't see much content being even set for products but would be nice to be at least somehow ready by that time
16:28:20 <tyll_> #info next meeting will be 2014-04-07 15:30 UTC
16:28:35 <jreznik> dgilmore: yeah, that's that "I don't see much content being even set for products"
16:28:53 <dgilmore> jreznik: i have setup kickstarts for each product
16:29:44 <dgilmore> I need to send the Workstation folks an email about their deliverables
16:30:38 <dgilmore> its still not clear to me what all the deliverables will be
16:31:13 <jreznik> and I'll ping Christian to really reply to it
16:32:35 <dgilmore> cheers
16:32:56 <dgilmore> bochecha: we do need to make a plan for it
16:33:53 <bochecha> dgilmore, a plan for the migration?
16:33:59 <dgilmore> bochecha: yes
16:34:11 <bochecha> the thing is, last week there wasn't even a consensus on which hash(es) to use, how many,...
16:34:31 <dgilmore> bochecha: well sha256 or sha512
16:34:41 <bochecha> that seems like it would need to be discussed before a plan
16:35:09 <dgilmore> weither we make hardlinks for other hashes I dont care to much, there is some but very minimal use cases for them
16:35:22 <dgilmore> id prefer sha512
16:35:49 <bochecha> me too, as that's what I had sent patches for (i.e less work :P)
16:36:39 <bochecha> for the migration plan, would it help if I sent something to the list about how I saw it when I made the patches, and so the path that the patches allow?
16:37:36 <tyll_> bochecha: having a proposal helps getting it discussed
16:39:01 <bochecha> alright, I'll try and do that this week then :)
16:39:01 <dgilmore> i agree with tyll_ here
16:40:25 <dgilmore> #action bochecha to send in a proposal for lookaside cache migration
16:40:32 <dgilmore> anything else?
16:41:00 <bochecha> nothing from me
16:41:11 <tyll_> nothing here
16:41:18 <dgilmore> #endmeeting