16:33:13 <dgilmore> #startmeeting RELENG (2016-01-25)
16:33:13 <zodbot> Meeting started Mon Feb 22 16:33:13 2016 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:33:13 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:33:13 <zodbot> The meeting name has been set to 'releng_(2016-01-25)'
16:33:13 <dgilmore> #meetingname releng
16:33:13 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion
16:33:13 <zodbot> The meeting name has been set to 'releng'
16:33:13 <zodbot> Current chairs: bochecha dgilmore masta maxamillion nirik pbrobinson pingou sharkcz tyll
16:33:16 <dgilmore> #topic init process
16:33:32 <pingou> o/
16:33:33 * nirik is sort of here, but also trying to fight monday morning fires.
16:33:35 <tyll> Hi there
16:33:39 * sharkcz is here
16:33:41 <maxamillion> .hello maxamillion
16:33:42 <zodbot> maxamillion: maxamillion 'Adam Miller' <maxamillion@gmail.com>
16:33:55 * pbrobinson is here.... sort of!
16:35:19 <dgilmore> lets get started
16:35:33 <dgilmore> there is no tickets for meeting
16:35:40 <dgilmore> #topic branching
16:36:01 <dgilmore> #info tomorrow is f24 branching
16:36:03 * pingou has something for this
16:36:11 <dgilmore> so we need to deliver composes
16:36:15 <dgilmore> pingou: shoot
16:36:26 <pingou> so few ideas: do we want to turn off fedmsg before branching to avoid spamming everyone?
16:36:56 <pingou> by default the script that adjusts the branch on repo takes the first commit in the repo, that's probably not something we want for branching
16:36:58 <nirik> we didn't the last few times and it took a while to catch up. ;)
16:37:21 <pingou> (iirc it's something we ran into last time, the branch pointed to the wrong commit)
16:37:50 <dgilmore> pingou: we can not turn off fedmsg
16:38:18 <dgilmore> not without locking everyone out from git
16:38:30 <nirik> I suppose we want the master/head commit... which might be ahead of what actually built in rawhide, but should be close.
16:38:51 <dgilmore> for mass branching we want what is HEAD for master
16:39:10 <dgilmore> but for regular branches where there is no built content we want to first commit
16:39:27 <pingou> dgilmore: we could turn off fedmsg during the branching, run the script afterward to sync the git repos  (manually) and restart fedmsg
16:39:55 <dgilmore> pingou: we can not
16:40:14 <dgilmore> pingou: not unless we turn off all access to git during branching
16:40:20 <pingou> dgilmore: why?
16:40:23 <dgilmore> which we used to do with CVS
16:40:28 <pingou> git doesn't rely on fedmsg
16:40:36 <dgilmore> pingou: because other people can commit
16:40:38 <nirik> yeah, true.
16:40:41 <eseyman> because you don't want people not gettign notifications of commits
16:40:43 <dgilmore> and those notifications have to get out
16:40:53 <pingou> ah ok for this aspect
16:41:02 <pingou> true there
16:41:25 <pingou> so we'll spam everyone for the branching
16:41:26 <dgilmore> wqew either lock it down and send no notifications, or leave it open and send all
16:41:31 <dgilmore> yeah
16:41:46 <pingou> but we still probably want to turn off fedmsg-hubs on pkgs01 during the branching
16:41:55 <pingou> to avoid creating the branch at the wrong commit
16:42:24 <dgilmore> pingou: okay
16:42:44 <dgilmore> we do want the branch to have the rigth commit
16:42:59 <pingou> I can take care of doing this part if people are ok with it
16:43:11 <pingou> (branching in pkgdb and then adjust the git repos)
16:43:16 <dgilmore> pingou: we will need to coordinate it but sure
16:43:25 <pingou> cool
16:43:50 <pingou> just fmy, when is it supposed to start? 15UTC?
16:44:02 <dgilmore> pingou: no set time
16:44:05 <pingou> k
16:44:10 <dgilmore> pingou: we do it when everything is ready
16:44:27 <pingou> I can practice in stg in the morning so hopefully ducks are in line when you come online
16:44:45 <dgilmore> pingou: we are likely going to do it late to try get the golang rebuild in first
16:45:28 <pingou> ok
16:45:31 <tyll> #info branching will start late to let golang rebuild finish
16:45:39 <pingou> then I might not make it
16:45:56 <dgilmore> pingou: understandable
16:45:59 <pingou> if so, I'll document the procedure and send it around
16:46:07 <pingou> (from what I did in stg)
16:46:23 <tyll> #info fedmsg for branching cannot be turned off because notifications of other commits need to be sent
16:46:32 <pbrobinson> looking at the golang build now, I'm not sure we'll be too late, but I'm not convinced the list given me is correct, will know more about it in a few hours
16:47:01 <dgilmore> I have a compose running right now. I am hoping taht it has everything we need, except atomic that needs to be bolted on
16:47:05 <pbrobinson> but it doesn't look more than around 100 odd packages (which is why I'm not convinced it's correct)
16:48:19 <dgilmore> if it does not work out, it does not work out
16:48:28 <dgilmore> but if we can make less work we should do it
16:49:19 <dgilmore> we also need to actually test a secondary arch compose today
16:49:35 <pbrobinson> yes, so it only looks to be a few 100 packages
16:49:52 <dgilmore> #info golang is ~100 packages
16:50:09 <dgilmore> #topic Secondary Architectures updates
16:50:11 <dgilmore> #topic Secondary Architectures update - ppc
16:50:18 <dgilmore> lets get the secondaries done
16:50:22 <dgilmore> how is ppc?
16:51:13 <pbrobinson> we're around 6K behind on mass rebuilds, so a little to go, getting there, not as fast as I would like
16:51:27 <dgilmore> okay
16:51:29 <pbrobinson> should be done this week one way or the other (or I might go insane)
16:51:48 <pbrobinson> got some changes to infra to do this week
16:52:05 <pbrobinson> should be ready to compose by end of the week
16:52:47 <dgilmore> cool
16:53:12 <dgilmore> ppc hub is not yet moved to ansible, right?
16:54:06 <pbrobinson> nope, should be happening this week with a move of IPs etc
16:54:11 <dgilmore> cool
16:54:22 <nirik> pbrobinson: if I can help with redoing ppc8-01 or ppc hub,etc, let me know
16:54:26 <dgilmore> #info ppc koji hub to be ansibilised soon, likely this week
16:54:28 * nirik would love to get that done too.
16:54:34 <pbrobinson> nirik: you will be ;-)
16:54:59 <pbrobinson> nirik: plan is to get initial infra ticket for vlan move done tonight, just trying to get other stuff out of the way
16:55:32 <nirik> ok.
16:55:35 <pbrobinson> nirik: and move the compose devices too so we can get rid of the last of the P7 stuff
16:55:47 * pbrobinson needs more hours in the week!
16:55:56 <dgilmore> pbrobinson: that or more clones
16:55:59 <nirik> cool. We actually should be able to just add the new vlan on another interface and use it...
16:56:04 <pbrobinson> :-)
16:56:20 * nirik wonders if those p8's for cloud have shown up yet... I couldn't figure out how to track them
16:56:36 <pbrobinson> nirik: I'm not sure we have enough patched in, we're already using 3, will look later this evening
16:56:42 <dgilmore> nirik: its teh cloud, its magic
16:57:00 <nirik> I am pretty sure theres at least one more connected, but also if not, we could ask DC folks to connect them for us.
16:57:06 <maxamillion> "git push to the motha fuckin' cloud" - threebean
16:57:16 <maxamillion> that's basically my favorite quote like ... ever
16:57:16 <dgilmore> threebean++
16:57:17 <zodbot> dgilmore: Karma for ralph changed to 25 (for the f23 release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:57:45 <dgilmore> anything ppc wise we need to discuss?
16:57:55 <pbrobinson> nope
16:58:00 <dgilmore> #topic Secondary Architectures update - s390
16:58:09 <dgilmore> sharkcz: how is teh world acording to s390
16:58:22 <pbrobinson> dead?
16:58:26 * pbrobinson hides
16:58:29 <sharkcz> mass rebuild in progress, first gcc6 s390 specific bug appeared
16:58:39 * dgilmore notes that he cleaned up eol releases and moved them to archive
16:58:49 <dgilmore> which freed up lots of disk for s390
16:58:59 <sharkcz> yep
16:59:04 <dgilmore> sharkcz: s390 bugs sound fun
16:59:18 <sharkcz> I think there will ppc ones too ...
16:59:27 <dgilmore> no doubt
17:00:35 <sharkcz> remaining issue is how to mount /mnt/koji for the composer
17:01:27 <dgilmore> sharkcz: yeah. I am really not sure how we are going to do that
17:01:43 <dgilmore> sharkcz: pungi does have a yum repo mode
17:01:56 <dgilmore> but I am not sure how we could make it work
17:02:21 <dgilmore> sharkcz: we will have to work with nirik, I wonder if we can put a s390 node on the fedora vpn
17:02:28 <dgilmore> and do nfs over vpn
17:02:40 <dgilmore> pretty ugly
17:02:44 <nirik> I don't think that will help any.
17:02:52 <pbrobinson> gluster replica?
17:03:00 <dgilmore> nirik: why not?
17:03:04 <nirik> I think the only way to make it work is to do work, copy, do work back
17:03:28 <nirik> networking folks are dead set against vpn over the wan.
17:03:34 <nirik> sorry nfs
17:03:47 <dgilmore> which is why I said over fedora vpn
17:03:53 <dgilmore> then they do not have to know
17:03:59 <dgilmore> but it is not great
17:03:59 * nirik frowns
17:04:20 <dgilmore> I get their concerns
17:04:32 <dgilmore> we will have to look at all options
17:04:41 <dgilmore> and likely s390 will be some special child
17:04:42 <nirik> so it has to be run on that arch?
17:04:43 <pbrobinson> not sure we can fix it in this meeting
17:04:49 <dgilmore> nirik: some of it does
17:05:18 <dgilmore> same as some of it has to  be run on x86_64, i386, armhfp etc
17:05:30 <dgilmore> I doubt we can fix it now
17:05:43 <dgilmore> nirik: maybe we should all have a meeting to look at it later
17:05:51 <nirik> yeah, all I can think of is splitting it and doing a sync over and back... but thats not great either
17:06:06 <nirik> anything over the wan is going to be dirt slow
17:06:11 <dgilmore> yeah
17:06:28 <nirik> qemu s390 in phx2? ;)
17:06:31 * nirik runs
17:06:39 <dgilmore> the making of teh boot.iso and install repo  and the dvd are what needs s390x
17:06:49 <dgilmore> nirik: it may be fast enough :P
17:06:57 <dgilmore> run it on ppc64be
17:07:21 <dgilmore> it could be an option
17:07:48 <sharkcz> +1, especially if the user-space emulation would work, not the full system
17:07:50 <dgilmore> #info we need to figure out some s390 compose issues
17:08:16 <dgilmore> anything else anyone wants to discuss in s390 land?
17:08:31 <sharkcz> not from me
17:08:33 <dgilmore> #topic Secondary Architectures update - arm
17:08:37 <dgilmore> pbrobinson: how is arm?
17:08:49 <pbrobinson> about 5K packages to go
17:08:51 <dgilmore> the hub moved to being ansibilised correct?
17:08:53 <pbrobinson> mostly looking OK
17:08:59 <pbrobinson> the hub is now ansible
17:09:04 <dgilmore> awesome
17:09:12 <pbrobinson> in fact 100% of the arm.koji infra is now ansible
17:09:14 <dgilmore> #info arm hub managed by infra ansible
17:09:34 <dgilmore> #info arm koji infra 100% managed by infra ansible
17:09:56 <maxamillion> that's awesome
17:09:59 <pbrobinson> we have a couple of minor issues around redirects that have issues with the hub post rebuild, I need to work with nirik to work out the last of the issue and get them fixed up
17:10:25 <nirik> sure
17:10:40 <dgilmore> cool
17:10:52 <pbrobinson> then next up once I've dealt with the PPC stuff is to get the moonshot commissioned
17:11:22 * nirik nods.
17:11:23 <pbrobinson> there's still some kernel work I need to chase on that one to work out the details
17:11:31 <pbrobinson> but we should be OK RSN
17:11:39 <dgilmore> cool
17:12:22 <pbrobinson> basically next up on the list there is pungi4/images and friends
17:12:28 <dgilmore> okay
17:12:38 <dgilmore> we should dig into that today if you have time
17:12:46 <pbrobinson> I don't have time today
17:12:49 <dgilmore> okay
17:13:18 <dgilmore> I am planning to setup branched to use the new process
17:13:19 <pbrobinson> I have some key bits I need today before branching and have a house inspection tomorrow
17:13:20 <nirik> is the plan to land pungi4 today before branching?
17:13:35 <dgilmore> nirik: that is my goal
17:13:39 <nirik> cool.
17:13:57 <nirik> dgilmore: does that mean we need to reinstall branched-composer and rawhide-composer as f23?
17:14:05 <dgilmore> at the least setup branched with it
17:14:09 <pbrobinson> I'll ensure I get to pungi4 mid week but I don't have enough hours for that in secondaries today
17:14:15 <dgilmore> and have rawhide follow soon
17:14:25 <nirik> ok, IMHO it would be nice ot move them both at once.
17:15:49 <dgilmore> it would
17:16:00 <dgilmore> and thats what I am going to aim to do
17:16:18 <dgilmore> anything aarch64 related?
17:17:21 <dgilmore> #topic Open Floor
17:17:21 <pbrobinson> not from me
17:17:29 <maxamillion> I have one thing for open floor
17:17:33 <dgilmore> so to continue on
17:17:44 <dgilmore> #info compose process is aiming to change today
17:17:52 <dgilmore> maxamillion: go for it
17:18:00 <maxamillion> I'm going to be deploying the koji-containerbuild plugin to stage today so I *might* break stage koji intermittently through out the day, my appologies in advance
17:18:32 <dgilmore> maxamillion: :D no problem
17:18:38 <maxamillion> :)
17:18:40 <nirik> oh, I had 2 quick things.
17:18:47 <dgilmore> nirik: go for it
17:18:51 <sharkcz> have 1 item too
17:19:25 <nirik> 1) I've gotten most of the builders reinstalled. Hoping to do buildhw's today... arm socs seem a bit unstable of late, but I am not sure why yet.
17:19:34 <dgilmore> I have one also
17:19:41 <decause> #info it was great seeing many of you folks during the DevConf.cz trip and Brno meetings :)
17:19:42 <dgilmore> nirik: cool, thanks
17:19:54 <maxamillion> decause++
17:19:54 <zodbot> maxamillion: Karma for decause changed to 20 (for the f23 release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:20:05 <nirik> 2) I am planning on reinstalling all of staging in the next few days. Will try and ping everyone that might be using it before I do... but if you have unfinished junk on any staging machine, make sure you save it somewhere else too
17:21:11 <dgilmore> nirik: cool. thanks. I think I have everything moved to prod now
17:21:26 <dgilmore> nirik: but maxamillion is messing around in stage
17:21:30 <nirik> ok. I will make sure I don't break maxamillion's stuff. ;)
17:21:39 <dgilmore> and lsedlar has been trying to get composes working in stage
17:21:55 <maxamillion> nirik: it'll be broken already, don't worry :)
17:22:13 <maxamillion> nirik: it'll all be in ansible though so if you re-install everything, it should just all magically work
17:22:21 <maxamillion> *should*
17:22:23 <dgilmore> maxamillion: :D
17:22:32 <dgilmore> famous last words
17:22:42 <maxamillion> nirik: or at the very least, be equally as broken as it previously was
17:22:46 <nirik> right. I plan to nuke everything (except db hosts) and ansible reinstall it all
17:22:57 <dgilmore> cool
17:23:17 <dgilmore> anything else?
17:23:27 <sharkcz> pull requests in dist-git, there was a discussion on fedora devel list and the outcome was they would help in number of situations, how to make it an official RFE?
17:23:53 <dgilmore> sharkcz: there was a discussion on teh infra list also
17:24:16 <dgilmore> sharkcz: someone would need to work with pingou and get it developed and setup in stg
17:24:19 <sharkcz> ah, seems I'm not subscribed there
17:24:36 <dgilmore> once we were sure of it working we could promote to prof
17:24:37 <dgilmore> prod
17:24:56 <sharkcz> is there already a ticket?
17:25:15 <dgilmore> nirik: ^
17:25:19 <dgilmore> not sure if tehre is
17:25:28 <nirik> there's been a lot of discussion... not sure if there's a ticket.
17:25:48 <nirik> we would need to work out a bunch of things first, so I would prefer we finish discussing before we file a ticket
17:25:54 <dgilmore> I do not remeber a ticket
17:26:03 <dgilmore> nirik++
17:26:21 <nirik> there was more discussion about it in pingou's recent gitolite stuff
17:26:24 <dgilmore> I think it is something we shoudl work towards
17:26:32 <nirik> I think it would be nice, yeah
17:26:53 <dgilmore> but until we have a concrete plan we should hold off assigning work
17:27:35 <nirik> being able to do PR's could be a nice way to find comaintainers and also make sure people submitting patches get credit, and bugs get fixed faster.
17:27:51 <sharkcz> nirik: +1000
17:28:13 <dgilmore> nirik: indeed
17:28:29 <dgilmore> small package patches in bugzilla do not work well
17:29:06 <bochecha> dgilmore: it's not too bad with git-bz... but pull requests still would work much better :)
17:29:31 <dgilmore> bochecha: I have seen patches sit forever in bz
17:29:42 <dgilmore> pull requests may not fix that
17:29:47 <bochecha> sure, I don't think that's a problem with bugzilla itself though
17:29:49 <dgilmore> but it should make it more visible
17:29:55 <dgilmore> then something could be done
17:30:02 <bochecha> agreed
17:30:40 <dgilmore> #info everyone agrees we should move to enabling pull requests for dist-git
17:30:51 <dgilmore> anything else?
17:31:07 <dgilmore> https://lists.fedoraproject.org/archives/list/rel-eng@lists.fedoraproject.org/thread/LMCW5NVNCCCGQTB6ONRTCB2QP2IOBIAJ/
17:31:26 <dgilmore> just wanted to make sure everyone saw that thread
17:31:36 <dgilmore> and chimied in if the had concerns or ideas?
17:32:19 <sharkcz> haven't much time yet to read it carefully
17:32:51 <dgilmore> I still have not fully caught up from Fosdem and devconf
17:33:10 <dgilmore> but jwboyer pinged me and I made sure I followed up on it
17:33:21 <dgilmore> it is a big change in how we do things
17:34:20 <nirik> it should probibly get comment from devel list sometime before we do it too.
17:34:27 <dgilmore> nirik: indeed
17:34:41 <dgilmore> I had suggested starting on releng tor efine it a bit
17:34:46 <dgilmore> then take to devel
17:35:11 <dgilmore> we will not make any change before f25
17:35:21 <dgilmore> but it may not be until f26
17:35:30 <dgilmore> I would like to see things happen a bit sooner
17:38:08 <dgilmore> anyway
17:38:15 <dgilmore> if nothing else I will end teh meeting
17:38:32 <dgilmore> #endmeeting