16:33:13 <dgilmore> #startmeeting RELENG (2016-01-25) 16:33:13 <zodbot> Meeting started Mon Feb 22 16:33:13 2016 UTC. The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:33:13 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:33:13 <zodbot> The meeting name has been set to 'releng_(2016-01-25)' 16:33:13 <dgilmore> #meetingname releng 16:33:13 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion 16:33:13 <zodbot> The meeting name has been set to 'releng' 16:33:13 <zodbot> Current chairs: bochecha dgilmore masta maxamillion nirik pbrobinson pingou sharkcz tyll 16:33:16 <dgilmore> #topic init process 16:33:32 <pingou> o/ 16:33:33 * nirik is sort of here, but also trying to fight monday morning fires. 16:33:35 <tyll> Hi there 16:33:39 * sharkcz is here 16:33:41 <maxamillion> .hello maxamillion 16:33:42 <zodbot> maxamillion: maxamillion 'Adam Miller' <maxamillion@gmail.com> 16:33:55 * pbrobinson is here.... sort of! 16:35:19 <dgilmore> lets get started 16:35:33 <dgilmore> there is no tickets for meeting 16:35:40 <dgilmore> #topic branching 16:36:01 <dgilmore> #info tomorrow is f24 branching 16:36:03 * pingou has something for this 16:36:11 <dgilmore> so we need to deliver composes 16:36:15 <dgilmore> pingou: shoot 16:36:26 <pingou> so few ideas: do we want to turn off fedmsg before branching to avoid spamming everyone? 16:36:56 <pingou> by default the script that adjusts the branch on repo takes the first commit in the repo, that's probably not something we want for branching 16:36:58 <nirik> we didn't the last few times and it took a while to catch up. ;) 16:37:21 <pingou> (iirc it's something we ran into last time, the branch pointed to the wrong commit) 16:37:50 <dgilmore> pingou: we can not turn off fedmsg 16:38:18 <dgilmore> not without locking everyone out from git 16:38:30 <nirik> I suppose we want the master/head commit... which might be ahead of what actually built in rawhide, but should be close. 16:38:51 <dgilmore> for mass branching we want what is HEAD for master 16:39:10 <dgilmore> but for regular branches where there is no built content we want to first commit 16:39:27 <pingou> dgilmore: we could turn off fedmsg during the branching, run the script afterward to sync the git repos (manually) and restart fedmsg 16:39:55 <dgilmore> pingou: we can not 16:40:14 <dgilmore> pingou: not unless we turn off all access to git during branching 16:40:20 <pingou> dgilmore: why? 16:40:23 <dgilmore> which we used to do with CVS 16:40:28 <pingou> git doesn't rely on fedmsg 16:40:36 <dgilmore> pingou: because other people can commit 16:40:38 <nirik> yeah, true. 16:40:41 <eseyman> because you don't want people not gettign notifications of commits 16:40:43 <dgilmore> and those notifications have to get out 16:40:53 <pingou> ah ok for this aspect 16:41:02 <pingou> true there 16:41:25 <pingou> so we'll spam everyone for the branching 16:41:26 <dgilmore> wqew either lock it down and send no notifications, or leave it open and send all 16:41:31 <dgilmore> yeah 16:41:46 <pingou> but we still probably want to turn off fedmsg-hubs on pkgs01 during the branching 16:41:55 <pingou> to avoid creating the branch at the wrong commit 16:42:24 <dgilmore> pingou: okay 16:42:44 <dgilmore> we do want the branch to have the rigth commit 16:42:59 <pingou> I can take care of doing this part if people are ok with it 16:43:11 <pingou> (branching in pkgdb and then adjust the git repos) 16:43:16 <dgilmore> pingou: we will need to coordinate it but sure 16:43:25 <pingou> cool 16:43:50 <pingou> just fmy, when is it supposed to start? 15UTC? 16:44:02 <dgilmore> pingou: no set time 16:44:05 <pingou> k 16:44:10 <dgilmore> pingou: we do it when everything is ready 16:44:27 <pingou> I can practice in stg in the morning so hopefully ducks are in line when you come online 16:44:45 <dgilmore> pingou: we are likely going to do it late to try get the golang rebuild in first 16:45:28 <pingou> ok 16:45:31 <tyll> #info branching will start late to let golang rebuild finish 16:45:39 <pingou> then I might not make it 16:45:56 <dgilmore> pingou: understandable 16:45:59 <pingou> if so, I'll document the procedure and send it around 16:46:07 <pingou> (from what I did in stg) 16:46:23 <tyll> #info fedmsg for branching cannot be turned off because notifications of other commits need to be sent 16:46:32 <pbrobinson> looking at the golang build now, I'm not sure we'll be too late, but I'm not convinced the list given me is correct, will know more about it in a few hours 16:47:01 <dgilmore> I have a compose running right now. I am hoping taht it has everything we need, except atomic that needs to be bolted on 16:47:05 <pbrobinson> but it doesn't look more than around 100 odd packages (which is why I'm not convinced it's correct) 16:48:19 <dgilmore> if it does not work out, it does not work out 16:48:28 <dgilmore> but if we can make less work we should do it 16:49:19 <dgilmore> we also need to actually test a secondary arch compose today 16:49:35 <pbrobinson> yes, so it only looks to be a few 100 packages 16:49:52 <dgilmore> #info golang is ~100 packages 16:50:09 <dgilmore> #topic Secondary Architectures updates 16:50:11 <dgilmore> #topic Secondary Architectures update - ppc 16:50:18 <dgilmore> lets get the secondaries done 16:50:22 <dgilmore> how is ppc? 16:51:13 <pbrobinson> we're around 6K behind on mass rebuilds, so a little to go, getting there, not as fast as I would like 16:51:27 <dgilmore> okay 16:51:29 <pbrobinson> should be done this week one way or the other (or I might go insane) 16:51:48 <pbrobinson> got some changes to infra to do this week 16:52:05 <pbrobinson> should be ready to compose by end of the week 16:52:47 <dgilmore> cool 16:53:12 <dgilmore> ppc hub is not yet moved to ansible, right? 16:54:06 <pbrobinson> nope, should be happening this week with a move of IPs etc 16:54:11 <dgilmore> cool 16:54:22 <nirik> pbrobinson: if I can help with redoing ppc8-01 or ppc hub,etc, let me know 16:54:26 <dgilmore> #info ppc koji hub to be ansibilised soon, likely this week 16:54:28 * nirik would love to get that done too. 16:54:34 <pbrobinson> nirik: you will be ;-) 16:54:59 <pbrobinson> nirik: plan is to get initial infra ticket for vlan move done tonight, just trying to get other stuff out of the way 16:55:32 <nirik> ok. 16:55:35 <pbrobinson> nirik: and move the compose devices too so we can get rid of the last of the P7 stuff 16:55:47 * pbrobinson needs more hours in the week! 16:55:56 <dgilmore> pbrobinson: that or more clones 16:55:59 <nirik> cool. We actually should be able to just add the new vlan on another interface and use it... 16:56:04 <pbrobinson> :-) 16:56:20 * nirik wonders if those p8's for cloud have shown up yet... I couldn't figure out how to track them 16:56:36 <pbrobinson> nirik: I'm not sure we have enough patched in, we're already using 3, will look later this evening 16:56:42 <dgilmore> nirik: its teh cloud, its magic 16:57:00 <nirik> I am pretty sure theres at least one more connected, but also if not, we could ask DC folks to connect them for us. 16:57:06 <maxamillion> "git push to the motha fuckin' cloud" - threebean 16:57:16 <maxamillion> that's basically my favorite quote like ... ever 16:57:16 <dgilmore> threebean++ 16:57:17 <zodbot> dgilmore: Karma for ralph changed to 25 (for the f23 release cycle): https://badges.fedoraproject.org/tags/cookie/any 16:57:45 <dgilmore> anything ppc wise we need to discuss? 16:57:55 <pbrobinson> nope 16:58:00 <dgilmore> #topic Secondary Architectures update - s390 16:58:09 <dgilmore> sharkcz: how is teh world acording to s390 16:58:22 <pbrobinson> dead? 16:58:26 * pbrobinson hides 16:58:29 <sharkcz> mass rebuild in progress, first gcc6 s390 specific bug appeared 16:58:39 * dgilmore notes that he cleaned up eol releases and moved them to archive 16:58:49 <dgilmore> which freed up lots of disk for s390 16:58:59 <sharkcz> yep 16:59:04 <dgilmore> sharkcz: s390 bugs sound fun 16:59:18 <sharkcz> I think there will ppc ones too ... 16:59:27 <dgilmore> no doubt 17:00:35 <sharkcz> remaining issue is how to mount /mnt/koji for the composer 17:01:27 <dgilmore> sharkcz: yeah. I am really not sure how we are going to do that 17:01:43 <dgilmore> sharkcz: pungi does have a yum repo mode 17:01:56 <dgilmore> but I am not sure how we could make it work 17:02:21 <dgilmore> sharkcz: we will have to work with nirik, I wonder if we can put a s390 node on the fedora vpn 17:02:28 <dgilmore> and do nfs over vpn 17:02:40 <dgilmore> pretty ugly 17:02:44 <nirik> I don't think that will help any. 17:02:52 <pbrobinson> gluster replica? 17:03:00 <dgilmore> nirik: why not? 17:03:04 <nirik> I think the only way to make it work is to do work, copy, do work back 17:03:28 <nirik> networking folks are dead set against vpn over the wan. 17:03:34 <nirik> sorry nfs 17:03:47 <dgilmore> which is why I said over fedora vpn 17:03:53 <dgilmore> then they do not have to know 17:03:59 <dgilmore> but it is not great 17:03:59 * nirik frowns 17:04:20 <dgilmore> I get their concerns 17:04:32 <dgilmore> we will have to look at all options 17:04:41 <dgilmore> and likely s390 will be some special child 17:04:42 <nirik> so it has to be run on that arch? 17:04:43 <pbrobinson> not sure we can fix it in this meeting 17:04:49 <dgilmore> nirik: some of it does 17:05:18 <dgilmore> same as some of it has to be run on x86_64, i386, armhfp etc 17:05:30 <dgilmore> I doubt we can fix it now 17:05:43 <dgilmore> nirik: maybe we should all have a meeting to look at it later 17:05:51 <nirik> yeah, all I can think of is splitting it and doing a sync over and back... but thats not great either 17:06:06 <nirik> anything over the wan is going to be dirt slow 17:06:11 <dgilmore> yeah 17:06:28 <nirik> qemu s390 in phx2? ;) 17:06:31 * nirik runs 17:06:39 <dgilmore> the making of teh boot.iso and install repo and the dvd are what needs s390x 17:06:49 <dgilmore> nirik: it may be fast enough :P 17:06:57 <dgilmore> run it on ppc64be 17:07:21 <dgilmore> it could be an option 17:07:48 <sharkcz> +1, especially if the user-space emulation would work, not the full system 17:07:50 <dgilmore> #info we need to figure out some s390 compose issues 17:08:16 <dgilmore> anything else anyone wants to discuss in s390 land? 17:08:31 <sharkcz> not from me 17:08:33 <dgilmore> #topic Secondary Architectures update - arm 17:08:37 <dgilmore> pbrobinson: how is arm? 17:08:49 <pbrobinson> about 5K packages to go 17:08:51 <dgilmore> the hub moved to being ansibilised correct? 17:08:53 <pbrobinson> mostly looking OK 17:08:59 <pbrobinson> the hub is now ansible 17:09:04 <dgilmore> awesome 17:09:12 <pbrobinson> in fact 100% of the arm.koji infra is now ansible 17:09:14 <dgilmore> #info arm hub managed by infra ansible 17:09:34 <dgilmore> #info arm koji infra 100% managed by infra ansible 17:09:56 <maxamillion> that's awesome 17:09:59 <pbrobinson> we have a couple of minor issues around redirects that have issues with the hub post rebuild, I need to work with nirik to work out the last of the issue and get them fixed up 17:10:25 <nirik> sure 17:10:40 <dgilmore> cool 17:10:52 <pbrobinson> then next up once I've dealt with the PPC stuff is to get the moonshot commissioned 17:11:22 * nirik nods. 17:11:23 <pbrobinson> there's still some kernel work I need to chase on that one to work out the details 17:11:31 <pbrobinson> but we should be OK RSN 17:11:39 <dgilmore> cool 17:12:22 <pbrobinson> basically next up on the list there is pungi4/images and friends 17:12:28 <dgilmore> okay 17:12:38 <dgilmore> we should dig into that today if you have time 17:12:46 <pbrobinson> I don't have time today 17:12:49 <dgilmore> okay 17:13:18 <dgilmore> I am planning to setup branched to use the new process 17:13:19 <pbrobinson> I have some key bits I need today before branching and have a house inspection tomorrow 17:13:20 <nirik> is the plan to land pungi4 today before branching? 17:13:35 <dgilmore> nirik: that is my goal 17:13:39 <nirik> cool. 17:13:57 <nirik> dgilmore: does that mean we need to reinstall branched-composer and rawhide-composer as f23? 17:14:05 <dgilmore> at the least setup branched with it 17:14:09 <pbrobinson> I'll ensure I get to pungi4 mid week but I don't have enough hours for that in secondaries today 17:14:15 <dgilmore> and have rawhide follow soon 17:14:25 <nirik> ok, IMHO it would be nice ot move them both at once. 17:15:49 <dgilmore> it would 17:16:00 <dgilmore> and thats what I am going to aim to do 17:16:18 <dgilmore> anything aarch64 related? 17:17:21 <dgilmore> #topic Open Floor 17:17:21 <pbrobinson> not from me 17:17:29 <maxamillion> I have one thing for open floor 17:17:33 <dgilmore> so to continue on 17:17:44 <dgilmore> #info compose process is aiming to change today 17:17:52 <dgilmore> maxamillion: go for it 17:18:00 <maxamillion> I'm going to be deploying the koji-containerbuild plugin to stage today so I *might* break stage koji intermittently through out the day, my appologies in advance 17:18:32 <dgilmore> maxamillion: :D no problem 17:18:38 <maxamillion> :) 17:18:40 <nirik> oh, I had 2 quick things. 17:18:47 <dgilmore> nirik: go for it 17:18:51 <sharkcz> have 1 item too 17:19:25 <nirik> 1) I've gotten most of the builders reinstalled. Hoping to do buildhw's today... arm socs seem a bit unstable of late, but I am not sure why yet. 17:19:34 <dgilmore> I have one also 17:19:41 <decause> #info it was great seeing many of you folks during the DevConf.cz trip and Brno meetings :) 17:19:42 <dgilmore> nirik: cool, thanks 17:19:54 <maxamillion> decause++ 17:19:54 <zodbot> maxamillion: Karma for decause changed to 20 (for the f23 release cycle): https://badges.fedoraproject.org/tags/cookie/any 17:20:05 <nirik> 2) I am planning on reinstalling all of staging in the next few days. Will try and ping everyone that might be using it before I do... but if you have unfinished junk on any staging machine, make sure you save it somewhere else too 17:21:11 <dgilmore> nirik: cool. thanks. I think I have everything moved to prod now 17:21:26 <dgilmore> nirik: but maxamillion is messing around in stage 17:21:30 <nirik> ok. I will make sure I don't break maxamillion's stuff. ;) 17:21:39 <dgilmore> and lsedlar has been trying to get composes working in stage 17:21:55 <maxamillion> nirik: it'll be broken already, don't worry :) 17:22:13 <maxamillion> nirik: it'll all be in ansible though so if you re-install everything, it should just all magically work 17:22:21 <maxamillion> *should* 17:22:23 <dgilmore> maxamillion: :D 17:22:32 <dgilmore> famous last words 17:22:42 <maxamillion> nirik: or at the very least, be equally as broken as it previously was 17:22:46 <nirik> right. I plan to nuke everything (except db hosts) and ansible reinstall it all 17:22:57 <dgilmore> cool 17:23:17 <dgilmore> anything else? 17:23:27 <sharkcz> pull requests in dist-git, there was a discussion on fedora devel list and the outcome was they would help in number of situations, how to make it an official RFE? 17:23:53 <dgilmore> sharkcz: there was a discussion on teh infra list also 17:24:16 <dgilmore> sharkcz: someone would need to work with pingou and get it developed and setup in stg 17:24:19 <sharkcz> ah, seems I'm not subscribed there 17:24:36 <dgilmore> once we were sure of it working we could promote to prof 17:24:37 <dgilmore> prod 17:24:56 <sharkcz> is there already a ticket? 17:25:15 <dgilmore> nirik: ^ 17:25:19 <dgilmore> not sure if tehre is 17:25:28 <nirik> there's been a lot of discussion... not sure if there's a ticket. 17:25:48 <nirik> we would need to work out a bunch of things first, so I would prefer we finish discussing before we file a ticket 17:25:54 <dgilmore> I do not remeber a ticket 17:26:03 <dgilmore> nirik++ 17:26:21 <nirik> there was more discussion about it in pingou's recent gitolite stuff 17:26:24 <dgilmore> I think it is something we shoudl work towards 17:26:32 <nirik> I think it would be nice, yeah 17:26:53 <dgilmore> but until we have a concrete plan we should hold off assigning work 17:27:35 <nirik> being able to do PR's could be a nice way to find comaintainers and also make sure people submitting patches get credit, and bugs get fixed faster. 17:27:51 <sharkcz> nirik: +1000 17:28:13 <dgilmore> nirik: indeed 17:28:29 <dgilmore> small package patches in bugzilla do not work well 17:29:06 <bochecha> dgilmore: it's not too bad with git-bz... but pull requests still would work much better :) 17:29:31 <dgilmore> bochecha: I have seen patches sit forever in bz 17:29:42 <dgilmore> pull requests may not fix that 17:29:47 <bochecha> sure, I don't think that's a problem with bugzilla itself though 17:29:49 <dgilmore> but it should make it more visible 17:29:55 <dgilmore> then something could be done 17:30:02 <bochecha> agreed 17:30:40 <dgilmore> #info everyone agrees we should move to enabling pull requests for dist-git 17:30:51 <dgilmore> anything else? 17:31:07 <dgilmore> https://lists.fedoraproject.org/archives/list/rel-eng@lists.fedoraproject.org/thread/LMCW5NVNCCCGQTB6ONRTCB2QP2IOBIAJ/ 17:31:26 <dgilmore> just wanted to make sure everyone saw that thread 17:31:36 <dgilmore> and chimied in if the had concerns or ideas? 17:32:19 <sharkcz> haven't much time yet to read it carefully 17:32:51 <dgilmore> I still have not fully caught up from Fosdem and devconf 17:33:10 <dgilmore> but jwboyer pinged me and I made sure I followed up on it 17:33:21 <dgilmore> it is a big change in how we do things 17:34:20 <nirik> it should probibly get comment from devel list sometime before we do it too. 17:34:27 <dgilmore> nirik: indeed 17:34:41 <dgilmore> I had suggested starting on releng tor efine it a bit 17:34:46 <dgilmore> then take to devel 17:35:11 <dgilmore> we will not make any change before f25 17:35:21 <dgilmore> but it may not be until f26 17:35:30 <dgilmore> I would like to see things happen a bit sooner 17:38:08 <dgilmore> anyway 17:38:15 <dgilmore> if nothing else I will end teh meeting 17:38:32 <dgilmore> #endmeeting