16:51:20 <dgilmore> #startmeeting RELENG (2015-03-23)
16:51:21 <zodbot> Meeting started Mon Mar 23 16:51:20 2015 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:51:21 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:51:27 <dgilmore> #meetingname releng
16:51:27 <zodbot> The meeting name has been set to 'releng'
16:51:27 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou
16:51:27 <zodbot> Current chairs: bochecha dgilmore masta nirik pbrobinson pingou sharkcz tyll
16:51:28 <dgilmore> #topic init process
16:51:46 <tyll> Hi
16:51:48 * nirik is sort of here, but drowning in emails too. ;)
16:51:58 * jreznik is watching
16:52:08 <Corey84> not only me this morning, nice
16:52:21 <Corey84> .fas corey84
16:52:22 <zodbot> Corey84: corey84 'Corey84' <sheldon.corey@gmail.com>
16:52:28 <dgilmore> hey Corey84 :)
16:52:34 <Corey84> dgilmore,  morn
16:53:01 <Corey84> that luks.py crap is REALLY getting annoying
16:53:28 <dgilmore> #topic #5931 [Proposal] Move new branch and unretire requests to pkgdb2
16:53:35 <dgilmore> https://fedorahosted.org/rel-eng/ticket/5931
16:53:45 <dgilmore> lets get moving quickly since we have a late start
16:53:54 <dgilmore> where is pingou
16:54:20 * Corey84 calls pingou  to the party
16:54:34 <pingou> hi  :)
16:55:09 <pingou> oh
16:55:10 <Corey84> pingou,  requested in -releng   by dgilmore
16:55:22 * pingou just saw /topic
16:55:23 <pingou> Corey84: no I was asked to come here ;-)
16:55:35 <Corey84> nvm  :)
16:55:42 <pingou> so on the pkgdb2 front, it's running on stg for few days, I have heard no report
16:55:47 <dgilmore> pingou: okay
16:55:48 <Corey84> too many channels / discussions atm
16:55:56 <dgilmore> pingou: so when will we go live?
16:55:57 <pingou> I guess we could move to prod which gives us a week or so before freeze
16:56:04 <pingou> dgilmore: today if we like
16:56:09 <dgilmore> pingou: lets
16:56:25 <dgilmore> I am sure people will complain if they do not like it
16:56:33 <pingou> the thing being that people can still use the old process anyway
16:56:47 <dgilmore> sure
16:56:49 * Corey84 will be in -blocker-review as well if i don't reply
16:56:58 <pingou> and we will likely have to support both for a little time until doc and people catch up :)
16:56:58 <dgilmore> we should work out a time to retire that
16:57:03 <pingou> +1
16:57:11 * nirik nods. sounds all reasonable.
16:57:19 <pingou> I guess it'll depend how smooth things are
16:57:20 <dgilmore> :) cool
16:57:28 <dgilmore> pingou: so lets make it live today
16:57:36 <pingou> ok, so I'm finishing one quick thing on anitya and I'll push 1.24 to prod :)
16:57:50 <pingou> then I'll run away, very fast, so you might see me passing by your house
16:57:50 <dgilmore> then we can evaluate in a couple of weeks when to shut down the old method
16:57:58 <pingou> (be nice hand me some water when I pass by)
16:58:08 <dgilmore> pingou: I will give you iced coffee
16:58:12 <pingou> \ó/
16:58:13 <dgilmore> more energy :P
17:00:02 <dgilmore> #topic #6047 handle packages that are retired only in Branched but not Rawhide
17:00:11 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6047
17:01:14 <dgilmore> I am not really sure what is the right thing here
17:02:00 <dgilmore> does anyone have any opinion?
17:02:13 <tyll> We decided a while ago that we unblock pkgs in Rawhide if it is only retired in branched
17:02:13 <nirik> I guess I'd say we should plan for the most common case...
17:02:19 <pingou> I'd go for #2
17:02:23 <nirik> and make the other case exception.
17:02:33 <dgilmore> tyll: okay. guess we did not document that
17:02:46 <tyll> I started working on this on the blocking script, but it needs more rewriting since it was not planned on the beginning
17:02:49 <dgilmore> tyll: do we do that now?
17:02:51 <nirik> sure thats fine with me
17:03:10 <tyll> not yet, but I also did not yet have to retire pkgs in Branched
17:03:21 <tyll> it came up when we cleaned up broken deps before final freeze
17:03:26 <dgilmore> okay
17:03:59 <Corey84> not sure why we would retire in branched and not rh tho (but  not a big rh user atm )
17:04:37 <pingou> Corey84: so that it can be fixed in rawhide and brought back in n+1
17:04:52 <Corey84> makes sense then
17:05:09 <dgilmore> tyll: I just went to update the ticket and you had :)
17:05:18 <tyll> dgilmore: sorry
17:05:49 <dgilmore> tyll: it is fine :) just trying to make sure I update as we go so things do not get missed
17:06:03 <dgilmore> #topic #5886 need method for distributing urgent fixes... urgently
17:06:11 <dgilmore> https://fedorahosted.org/rel-eng/ticket/5886
17:06:18 <dgilmore> I have thought about this more
17:06:24 <nirik> there was another thread on the infra list about this last week...
17:06:37 <nirik> I think we just need to decide how we want to do it, and propose that
17:06:38 <dgilmore> I almost think it needs managed in bodhi so we get updateinfo into the repo
17:07:09 <dgilmore> we need to be able to run the security push even if another push is going on
17:07:16 <dgilmore> so treat it like a seperate product
17:07:17 <nirik> either that, or have a stand alone way to inject that
17:07:26 <dgilmore> like we do with epel and fedora
17:07:31 <dgilmore> yeah
17:07:36 <dgilmore> lmacken: ping
17:07:53 <dgilmore> since lmacken did all the updateinfo stuff
17:07:58 <dgilmore> and bodhi
17:08:12 <dgilmore> it needs to be light weight and quick
17:08:13 <nirik> I think we might be able to do bodhi2, but mm is going to make it a lot slower
17:08:35 <dgilmore> we should be able to push out a security update like for heartbleed or shell shock in under 1 hour
17:08:59 <dgilmore> nirik: 2 thoughts
17:09:03 <dgilmore> 1 we use mm
17:09:20 <dgilmore> 2 we use baseurl and point at https://dl.fp.o
17:09:24 <nirik> right.
17:09:55 <nirik> mm means we sync it, it needs to crawl it to find it, needs to update all the mirrorlists, it just takes a while...
17:09:57 <dgilmore> if we use baseurl we lose the metadata checksum that mm gives
17:10:04 <dgilmore> but we have quicker turn around
17:10:11 <pbrobinson> I thought the whole point of the idea for the quick security updates is to not use MM to get it available quick while MM was dealing with getting it out to the masses
17:10:11 <nirik> perhaps we can make mm2 handle things better.
17:10:13 <dgilmore> and using https likely mitigates it anyway
17:10:22 * masta is here
17:10:38 <dgilmore> nirik: well its likely going to be synced anyway
17:10:57 <nirik> I dislike the idea of making any changes in bodhi1/mm1 for this... we need to get bodhi2/mm2 delpoyed and should make any changes there.
17:11:19 <dgilmore> how far away is mm2?
17:11:22 <lmacken> dgilmore: pong
17:11:23 <dgilmore> and bodhi 2
17:11:24 * lmacken reads backlog
17:11:42 <nirik> all mirrorlists are running mm2 now. We still need to sort things in the crawler/backend/frontend...
17:11:44 <dgilmore> lmacken: do we have a standalone script to inject updateinfo into a repo?
17:11:52 <nirik> so, I am now shooting for after beta.
17:11:58 <lmacken> right, so I just ported all of the updateinfo.xml handling code in bodhi2 over to createrepo_c APIs
17:12:08 <lmacken> it should hopefully be much faster, and flexible.
17:12:13 <dgilmore> lmacken: and would it be hard to setup a seperate product that we could use to push security updates
17:12:23 <lmacken> so if we wanted to use it outside of bodhi, it's totally possible with only createrepo_c.... parsing & generating the updateinfo.xml
17:12:30 <masta> oh cool, yeah... I hate to block urgent updates due to an ongoing push.
17:12:45 <lmacken> dgilmore: a seperate out-of-band masher tool? should be feasible
17:12:57 <lmacken> or would it not require mashing?
17:13:13 <dgilmore> lmacken: well i was thinking say Fedora-Security instread of  Fedora or EPEL
17:13:27 <dgilmore> gahh I can not type
17:13:32 <lmacken> ah, a seperate product within bodhi.
17:13:36 <lmacken> yeah, we can make that happen.
17:13:37 <dgilmore> lmacken: right
17:13:45 <dgilmore> as an option to do it
17:14:11 <dgilmore> somehow we would need to tag builds into a tag say fedora-22-security
17:14:14 <dgilmore> mash that tag
17:14:30 <dgilmore> inject updateinfo so the secuity plugin works
17:14:34 <dgilmore> and make it public
17:14:51 <dgilmore> it is not something we would do often
17:14:56 <lmacken> yep, as long as there is a koji tag for it, integrating it into the bodhi flow should be fairly simple.
17:15:01 <masta> ok so let me see if I get this, there will be a fast path repo called fedora-security (or whatever) that is an enabled repo for the community, and the update lands there first... then later it drops in the regular push?
17:15:09 <dgilmore> there is probably 3 or 4 times in history we would have used it
17:15:26 <dgilmore> masta: right
17:15:29 <pbrobinson> masta: yes
17:16:00 <pbrobinson> dgilmore: I bet if it was there we'd likely use it more esp for things like some of the firefox updates etc
17:16:09 <masta> I love that idea, as recently all the important CVEs have been needing to push on my shift, and we had to wait. it was a major pity.
17:16:50 <dgilmore> pbrobinson: maybe but the point is to get things like heartbleed and shellshock updates out quickly
17:17:25 <nirik> I think we need a clear critera for what it could be used for... I think I even proposed something...
17:17:49 <nirik> or someone did...
17:17:58 <nirik> CRITICAL rating on the security scale or whatever
17:18:01 <masta> well, the CVE should be critical or even 'important'
17:18:34 <dgilmore> nirik: right, it is basiclly the world is falling down lets get it fixed asap
17:18:50 <dgilmore> not oh hey there is a bunch of things that I want out now
17:18:53 <masta> well even important updates could drop in there
17:19:24 * nirik would say critical only
17:19:27 <masta> but I'd say the community contract will be to focus on critical CVes
17:19:39 <pbrobinson> dgilmore: right, I understand that, my point is but there's some stuff for end users that in some cases is potentially equally as bad for pawning browsers etc which is probably just as important to get out quickly that might not affect servers, FREAK was one example here, it was client side SSL that was the problem
17:19:59 <pbrobinson> and so it's likely if it was there we'd be aware of more use cases
17:20:54 <dgilmore> pbrobinson: sure something like FREAK would be included
17:21:01 <lmacken> so with bodhi2, if a 'push everything with a request' happens that contains security updates, it will only do those first, and won't start on the others until they hit the mirror. https://github.com/fedora-infra/bodhi/blob/pyramid/bodhi/masher.py#L115-L162
17:21:08 <dgilmore> pbrobinson: but not the run of the mill firefox update
17:21:21 <dgilmore> even though it likely has a bunch of CVE's
17:21:26 <dgilmore> same for openjdk
17:21:49 <dgilmore> lmacken: that is potentially a very bad idea
17:22:41 <pbrobinson> dgilmore: was more thinking the pwn2own style firefox releases as an example, some of those end up making FREAK look like a walk in the park, I didn't mean for all of them.... it was an example ;-)
17:22:45 <dgilmore> lmacken: say I update firefox with CVE fixes, it relies on nspr that doesn't contain security fixes, and I put them as two updates
17:22:54 <dgilmore> you can not get the updated firefox
17:23:07 <dgilmore> sure they should be batched together, people do not always do that
17:23:21 <dgilmore> pbrobinson: :) yeah
17:23:25 <lmacken> dgilmore: right, the critical fixes will need to be bundled at the moment until it's smart enough to look at the buildroot and auto-batch
17:23:37 <masta> So what about the product name? Fedora-security or Fedora-critical, or what?
17:24:34 <lmacken> dgilmore: oh, I may have mis-conveyed what it does. It'll do the whole stable push first, even with non-security updates.
17:24:41 <pbrobinson> masta: it's not a product
17:25:07 <dgilmore> lmacken: okay, but will do testing after?
17:25:10 <pbrobinson> masta: and it's more at the moment about implementation as opposed to the colour of the bike shed I think.
17:25:16 <lmacken> dgilmore: yes, but stable takes priority at the moment
17:25:28 <dgilmore> lmacken: that is not so bad then
17:25:33 <pbrobinson> dgilmore: how do we deal with atomic/docker etc with the super speed repo channel?
17:25:34 <masta> pbrobinson: hehe...
17:25:53 <dgilmore> pbrobinson: we will have to make and push a atomic tree at the same time
17:26:17 <dgilmore> pbrobinson: we may need to also make new livecds, arm disk images, cloud images, etc also
17:27:01 <dgilmore> I think in this case though we get the yum and atomic repos pushed first and deal with the rest after
17:27:12 <dgilmore> just to expidiate the process
17:27:33 <nirik> and of course the worry with any process like this: we may push out something broken. :(
17:27:40 <pbrobinson> and that would just pull directly from koji/local non mirrored standard repos given we'd be tagging it straight to the -updates so there'd be no MM and hence we don't need to change that bit of the process?
17:27:44 <dgilmore> nirik: indeed
17:28:05 <dgilmore> pbrobinson: it would have the security repo enabled
17:28:24 <nirik> pbrobinson: we are taking about a new repo enabled for users... fedora-criticial or whatever
17:28:29 <dgilmore> pbrobinson: we would point it at the newly mashed repo
17:28:58 <dgilmore> pbrobinson: we would add a new repo to fedora-repos that all users have enabled by default
17:29:38 <dgilmore> maybe we can mark the repo as a security repo for the yum/dnf security plugins and skip updateinfo
17:31:11 <dgilmore> anyway more questions that need resolved still
17:31:28 <lmacken> with a small repo, the updateinfo should be quick to generate. It has to query the build rpms from koji for each one build, which is why it takes forever.
17:31:29 <dgilmore> we do need to try get it sorted so that it is rolled out for f22 when we ship
17:31:48 <dgilmore> lmacken: right, the repo should always be very small
17:31:54 <dgilmore> so mashing should be quick
17:32:01 <dgilmore> making the atomic repo should be quick
17:32:15 <dgilmore> rsyncing to the master mirror should be quick
17:32:49 <dgilmore> lets move on for now
17:32:53 <dgilmore> #topic #5870 rawhide signing
17:32:59 <dgilmore> https://fedorahosted.org/rel-eng/ticket/5870
17:33:10 <nirik> lets take this off the agenda until we are not blocked on sigul?
17:33:13 <dgilmore> so we have quite a few GSOC people interested in working on sigul
17:33:21 <nirik> hopefully someone can fix it. ;)
17:33:29 <dgilmore> just wanted to give that update and will take it off the agenda
17:33:31 <nirik> I have manually been signing rawhide, so it's mostly signed
17:33:40 <nirik> but of course not 100%
17:34:13 <dgilmore> :)
17:34:24 <masta> thanks nirik
17:34:29 <dgilmore> #topic #6110 allow sudo for sysadmin-releng on sign bridges
17:34:35 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6110
17:34:39 <dgilmore> this just needs doing
17:34:46 * nirik is fine with this, sure.
17:34:54 <dgilmore> #action change sudoers on sigul-bridge boxes
17:35:09 <dgilmore> #topic Secondary Architectures updates
17:35:10 <dgilmore> #topic Secondary Architectures update - ppc
17:35:22 <dgilmore> pbrobinson: how is ppc this week?
17:35:31 <pbrobinson> the next RC is in process
17:35:37 <pbrobinson> I'm hoping this will be the last
17:35:47 <pbrobinson> packages looking a lot better now :)
17:35:57 <dgilmore> awesome
17:36:06 * pbrobinson is getting sick of signing packages ;-)
17:36:09 <dgilmore> #info in much better shape, Aplha RC's happening
17:36:18 <dgilmore> #topic Secondary Architectures update - s390
17:36:27 <dgilmore> no sharkcz
17:36:34 <dgilmore> #topic Secondary Architectures update - arm
17:36:43 <dgilmore> pbrobinson: how is arm?
17:36:49 <pbrobinson> there will be another RC RSN
17:36:57 <pbrobinson> I'm just awaiting rc5 kernel to land
17:37:20 <pbrobinson> ghc should be fixed this week which is basically our main packages blocker there
17:37:32 <dgilmore> wesome
17:37:35 <dgilmore> awesome
17:37:54 <dgilmore> #info RC's happening, ghc looks to be fixed soon
17:37:58 <pbrobinson> so user space is looking good, kernel is the main concern there but the big lock up bug fix landed in rc4
17:38:03 <jwb> fyi, rc5 is broken on at least one of my machines
17:38:25 <pbrobinson> jwb: I'll ping you on fedora-kernel for more details
17:38:32 <pbrobinson> and thanks
17:38:36 <jwb> i915 issue
17:38:50 <pbrobinson> jwb: ah :-) doesn't affect ARM ;-)
17:39:09 <jwb> it does in that i'm unlikely to build it on primary today
17:39:58 <pbrobinson> jwb: yes, I figured that much
17:40:02 <dgilmore> jwb: thanks
17:40:15 <jwb> np
17:40:20 <dgilmore> pbrobinson: anything more for arm?
17:41:08 <dgilmore> I will take that as no
17:41:10 <dgilmore> #topic Open Floor
17:41:21 <dgilmore> just wanted to bring up one thing
17:41:59 <Corey84> pbrobinson,  not seeing it koji is it in the rh compose still?
17:42:27 <dgilmore> looks like a kernel change has changed behaviour for procfs which with builders updated to latest f21 bits not having a entry for  / in mtab breaking at least building ruby
17:42:44 <dgilmore> Corey84: ? nots eeing what?
17:42:49 <dgilmore> seeing
17:43:49 <Corey84> the rc5 kernel
17:44:07 <Corey84> Linux server.linuxnet 4.0.0-0.rc4.git2.1.fc23.x86_64 #1 SMP Fri Mar 20 16:10:24 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
17:44:25 <dgilmore> Corey84: jwb said its not yet built and he wont due to a bug
17:44:37 <Corey84> missed that
17:44:47 <pbrobinson> Corey84: rc5 hasn't been pushed to git let alone koji yet
17:45:11 * Corey84 seeing that now  ---too many meetings atm
17:45:57 <dgilmore> okay, if there is nothing else for the open floor I will close up
17:46:52 <dgilmore> #endmeeting