15:32:38 <dgilmore> #startmeeting RELENG (2015-07-13) 15:32:39 <zodbot> Meeting started Mon Jul 13 15:32:38 2015 UTC. The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:32:39 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:32:48 <dgilmore> #meetingname releng 15:32:48 <zodbot> The meeting name has been set to 'releng' 15:32:48 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion 15:32:48 <zodbot> Current chairs: bochecha dgilmore masta maxamillion nirik pbrobinson pingou sharkcz tyll 15:32:51 <dgilmore> #topic init process 15:32:51 <nirik> morning all. 15:32:58 <bochecha__> hi there 15:32:59 * nirik is going to run to get coffee, back in a min 15:33:00 * maxamillion is here 15:33:03 * sharkcz is here 15:34:12 <dgilmore> lmacken: here? 15:37:28 <dgilmore> I guess not 15:37:45 <dgilmore> #topic #6164 bodhi2 status update requested 15:37:53 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6164 15:38:05 <dgilmore> nirik: any idea of any changes here this week? 15:38:25 <nirik> I know lmacken was working on it. ;) stg is looking much more functional from a web side. 15:38:45 <dgilmore> okay 15:38:48 <nirik> I think we should push ahead and as soon as we can land it in prod and work through any problems 15:38:56 <nirik> because I really want it to land. ;) 15:39:10 <dgilmore> nirik: I really want it to land also 15:39:18 <dgilmore> but afaik it can not yet even mash a repo 15:39:48 <dgilmore> there is many basic pieces of functionality still missing 15:40:10 <nirik> I don't know the mashing status. 15:40:38 <dgilmore> what is the url for it? 15:40:48 <nirik> https://admin.stg.fedoraproject.org/updates/ 15:40:55 <nirik> is the web in stg 15:41:16 <dgilmore> I thougt it was on a different url 15:41:39 <nirik> it was, but we decided to move it so it would match what it would be in prod 15:41:58 <dgilmore> might be a bad idea 15:42:07 <nirik> how so? 15:42:10 <dgilmore> because I do not think its close to ready from what I know 15:42:28 <dgilmore> if we need to get some bug fix tested on the existing code can we? 15:42:38 <pbrobinson> well lets test it and see 15:43:05 <pbrobinson> got to start somewhere in getting it ready for prod and I suppose the url is a start :) 15:43:11 <nirik> dgilmore: the bodhi1 instances are still there. we could switch it back... 15:43:21 <nirik> but really it's never been completely functional in stg... 15:43:25 <lmacken> sorry, I was attending to puppy doodies^Wduties 15:43:27 <nirik> because koji wasn't and other parts 15:43:35 <dgilmore> nirik: okay 15:43:49 <dgilmore> lmacken: so where does bodhi2 stand? 15:43:55 <nirik> I dunno if things are good enough now we could test mashing in stg or not. 15:43:57 <dgilmore> what works and what does not work? 15:44:04 <lmacken> lots of bodhi2 changes have gone in, and I have a bunch more queued up... it's getting close and today I'm hoping we'll be able to submit new updates to bodhi.stg 15:44:14 <lmacken> then from there I'm going to test full pushes 15:44:53 <dgilmore> lmacken: okay, how do we plan to mash repos and make atomic trees? 15:45:06 <lmacken> had to refactor a bunch of path issues... since in bodhi1 it's mounted to /updates, but bodhi2 is designed so it would be /updates/{updates,releases,etc} 15:45:15 <lmacken> dgilmore: the code is the same as it is in bodhi1 at the moment 15:45:22 <lmacken> Ideally i'd love to get the koji plugins written 15:45:34 <lmacken> well, the mash one was written by bochecha__, so we just need to deploy & test 15:45:45 <lmacken> then fork and tweak for atomic 15:45:57 <bochecha__> lmacken: I don't think we'll use my mash plugin 15:46:00 <lmacken> but doing that in koji is not a blocker from my perspective 15:46:00 <nirik> we have run_root now... 15:46:04 <bochecha__> lmacken: afaik we're going with the runroot one 15:46:07 <lmacken> bochecha__: dgilmore said it's probably the way to go at the FAD 15:46:11 <lmacken> (yours) 15:46:15 <bochecha__> lmacken: that was a long time ago :) 15:46:19 <lmacken> no, the recent fad 15:46:22 <bochecha__> oh 15:46:23 <dgilmore> bochecha__: I am not really sure what is teh best way to do it 15:46:40 <bochecha__> well, my plugin is still at the same place, if it's needed :) 15:46:51 <dgilmore> bochecha__: a plugin that does one task and does it well and is isolated to that is likely a bit safer 15:47:17 <dgilmore> in that if we get some argument wrong we do not risk desroying things 15:47:24 <bochecha__> "does it well", with the limited testing I did... almost 2 years ago ;) 15:47:28 <bochecha__> but sure 15:48:09 <bochecha__> the mash plugin still needs to mount /mnt/koji inside the chroot as rw, so it still has potential for destroying things (for sure, less than a plugin that runs anything as root with anything mounted inside the chroot) 15:48:19 <dgilmore> bochecha__: I guess we need to come up with a plan. do we set up everything to run as runroot tasks, or have different plugins to do teh different tasks 15:49:05 * nirik doesn't have too much opinion. 15:49:31 <nirik> I guess runroot is there and working, so I would lean toward just using it for now and if we want/need specific plugins later we could add them? 15:50:05 <dgilmore> this is probably an area we should engage with the wider releng world, talk with CentOS and Red Hat internal Releng guys and all agree on a path forward 15:50:24 <bochecha__> I also don't have much opinion on either 15:50:31 <dgilmore> we could do as runroot first and look at other options later 15:50:38 <nirik> sure, but to avoid stalling on it we can use runroot now. ;) 15:52:48 <dgilmore> sure 15:53:07 <dgilmore> just need to get teh mock bug fixed that is blocking using runroot 15:53:22 <nirik> oh yeah. ;( 15:53:31 <dgilmore> https://bugzilla.redhat.com/show_bug.cgi?id=1208092 15:54:19 * nirik should cc on it. does 15:54:49 <dgilmore> lmacken: so at what point can we move to the next step of testing in stg? 15:54:55 <dgilmore> lmacken: what needs to be done? 15:55:01 <dgilmore> lmacken: what can we do to help? 15:55:29 <lmacken> dgilmore: fixing a few bugs which I'm working on now... mostly path related, but I am very close to being able to submit new updates in stg 15:55:49 <lmacken> figuring out the mash/atomic compose in koji situation would help 15:56:10 <lmacken> but for now I'm just going to leave the push code as-is 15:56:25 <lmacken> and focus on standing up the web frontend, masher, client tools and python api 15:56:26 <dgilmore> lmacken: okay, I think going as is is best for now 15:56:36 <dgilmore> as it lets us test with a known working base 15:56:39 <lmacken> yeah 15:56:50 <dgilmore> once we have the frontend ready we can work on the backened changes 15:57:27 <dgilmore> lmacken: cool. feel free to ping me if you need anything from me 15:57:29 * nirik nods 15:57:35 <lmacken> will do 15:57:35 * dgilmore wants to get this moving forward 15:57:36 <nirik> yeah, same here. Happy to help 15:57:51 <lmacken> there are plenty of github issues to help with too :) 15:58:10 <tflink> lmacken: I can point taskotron staging at bodhi2 stg if that would help 15:58:14 <dgilmore> that would mean making a github account 15:58:38 <lmacken> tflink: it interacts with it via python-fedora, right? probably not quite ready yet 15:58:55 <lmacken> dgilmore: okay, I'll plan on migrating to pagure at some point then... 15:58:55 <tflink> lmacken: yeah, we use the normal python interfaces 15:59:05 <lmacken> tflink: okay yeah, I'm working on those currently 15:59:33 <lmacken> if anyone wants to help migrate github issues over to pagure for bodhi, that would help get more eyes on things apparently 16:00:31 <lmacken> EOF on that 16:00:52 <dgilmore> thanks lmacken 16:01:04 <dgilmore> #topic Secondary Architectures updates 16:01:05 <dgilmore> #topic Secondary Architectures update - ppc 16:01:15 <dgilmore> pbrobinson: how is ppc? 16:01:42 <pbrobinson> moving forward, there's an issue with kernels, nightly compose is running, working on getting cloud images automated 16:02:08 <sharkcz> build-wise should be few days behind primary 16:03:56 <dgilmore> cool 16:07:06 <dgilmore> #topic Secondary Architectures update - s390 16:07:11 <dgilmore> sharkcz: how is s390? 16:08:20 <sharkcz> a bit better :-) we are hours behind primary, mainly with the exception of the ghc-* stack where the compiler changed the ABI hash during f22 cycle and there are old builds using the old hash 16:09:17 <dgilmore> ghc seems to be special 16:09:19 <pbrobinson> sharkcz: you only have to look at ghc in marginally the wrong way and it changes it's abi hash! 16:09:39 <sharkcz> pbrobinson: really something like that ;-) 16:10:20 <sharkcz> don't understand how the -j value in parallel make can make the difference 16:10:27 <pbrobinson> me either! 16:12:15 <dgilmore> #topic Secondary Architectures update - arm 16:12:25 <dgilmore> #undo 16:12:25 <zodbot> Removing item from minutes: <MeetBot.items.Topic object at 0x3b936a50> 16:12:40 <dgilmore> sharkcz: nirik: where are we with the s390 hub migration 16:13:03 <nirik> I've not done anything further on it... 16:13:03 <sharkcz> in the middle, I plan to focus on it this week 16:13:29 <dgilmore> sharkcz: okay, what is missing? 16:13:57 <sharkcz> the setup for the users dealing with shadow configs 16:14:30 <dgilmore> okay, can we move the hub, then sort that out? 16:14:41 <pbrobinson> sharkcz: let's you and I sync on that this week and come to an agreement 16:15:20 <sharkcz> pbrobinson: ack 16:15:44 <dgilmore> #info sharkcz and pbrobinson to meet and agree on how to do koji-shadow in ansible 16:15:45 <sharkcz> dgilmore: nirik would like to see it in ansible, so we should agree on the setup first 16:15:54 * nirik is happy to help however he can 16:16:07 <dgilmore> sharkcz: it has to be in ansible 16:16:17 <dgilmore> sharkcz: agreeing on how to do it should be first 16:16:29 <dgilmore> sharkcz: but does that really prevent us migrating the hub 16:16:31 <maxamillion> oh man, koji-shadow setup/deployment in ansible would be awesome 16:16:41 <dgilmore> and using the new ansibled one 16:17:06 <sharkcz> dgilmore: dunno much about ansible, can it incrementally add things? 16:17:14 <pbrobinson> maxamillion: there's really not that much to the setup 16:17:17 <dgilmore> sharkcz: we should still be able to run the setup from the old box, just point it at the new hub 16:17:34 <dgilmore> sharkcz: you can 16:17:36 <pbrobinson> sharkcz: I know enough, I think we can sort that out OOB from the meeting 16:17:47 <nirik> sure, we can have a base config and add things to it... 16:17:48 <sharkcz> ok 16:18:23 <dgilmore> #topic Secondary Architectures update - arm 16:18:32 <dgilmore> pbrobinson: okay lets really do arm now 16:18:34 <maxamillion> pbrobinson: yeah, people say that but then there's no documentation for it ... sooo .... >.> 16:18:35 <pbrobinson> so we're getting there, the main packaging blockers have unblocked for the time being and we're slowly closing the gap to rawhide 16:18:47 <pbrobinson> we'll be losing capacity from around the 16th for 10 days or so because for some reason in the US it takes 10 days to get something from BOS to PHX 16:19:15 <pbrobinson> that's 80% capacity 16:19:35 <pbrobinson> so we'll be running with 2 builders 16:19:38 <nirik> 10 days!? 16:19:59 <pbrobinson> nirik: Yep, I'm sure it's because we're taking the push bike delivery option or some shit 16:20:11 <nirik> thats crazy. Should be 1-2 tops... ;( 16:20:25 <tibbs|w> It can be done in six hours if you're willing to drive the stuff to and from an airport. 16:21:14 <pbrobinson> nirk: apparently somebody is being a tight arse and doing cheap arse road freight option and won't let me ship some other boxes doing nothing and swap them back later 16:21:25 <dgilmore> pbrobinson: how will that effect alpha readiness? 16:21:49 <nirik> pbrobinson: odd. perhaps out of meeting we can poke them to use a better/faster method... 16:22:18 <nirik> perhaps stickster could talk with them 16:22:19 <pbrobinson> dgilmore: fuck knows, it will likely be massively delayed, I've told people that and they seem to think I'm happy to work 24*7 so if it's delayed fuck them all!!!!! 16:23:12 <dgilmore> #info Alpha likely to be massively delayed due to time to get hardware mmoved 16:23:26 <dgilmore> anything else for arm? 16:23:34 <pbrobinson> not from my side 16:24:56 <dgilmore> #topic f23 status 16:25:03 <dgilmore> so we have branching tomorrow 16:25:14 <dgilmore> we have pungi kinda working 16:25:18 <pbrobinson> wow, that's come up fast :) 16:25:31 <dgilmore> I pinged clark and he is going to fix teh mock bug today 16:25:33 <nirik> and tc1 is also due tomorrow 16:25:42 <nirik> (but of course makes no sense until branching is done) 16:25:49 <maxamillion> pbrobinson: +1 16:25:52 <dgilmore> nirik: yeah. 16:26:08 <dgilmore> so we have a week or so to get tooling lined up 16:26:51 <pbrobinson> dgilmore: got a BZ on the mock bug? 16:27:27 <sharkcz> pbrobinson: https://bugzilla.redhat.com/show_bug.cgi?id=1208092 16:28:22 <pbrobinson> ah... look, I'm already on it (not shocking!) 16:28:45 <dgilmore> pbrobinson: linked it earlier 16:29:00 <dgilmore> so this next week will be super hectic 16:29:19 <pbrobinson> dgilmore: yea, but you know I generally just ignore you ;-) 16:29:47 <dgilmore> pbrobinson: ;) hey now 16:29:53 <pbrobinson> :-D 16:30:13 <tyll> Are we going to retire FTBFS builds before branching as outlined on the schedule? Currently it is only about one week since I sent the initial reminder e-mail and recently there were still a lot pkgs affected 16:30:18 <dgilmore> #info branching tomorrow 16:30:35 <dgilmore> tyll: well branching is tomorrow 16:30:45 <dgilmore> we would have to do it first thing 16:31:21 <pbrobinson> tyll dgilmore: personally I'd be flexible and give people some more time, there's movement there and I don't think it hurts so much 16:31:35 <dgilmore> pbrobinson: I agree 16:31:58 <sharkcz> +1 16:32:29 <pbrobinson> when I looked there's less than 30 packages older than F-21 that are broken, I fixed a few up, and retired some others 16:33:43 <dgilmore> does anyone have a strong opinion that it has to happen before branching? 16:33:53 <dgilmore> Branching tomorrow has to happen 16:34:10 <dgilmore> so to retire them before branching would mean we have to rush it through 16:34:37 <dgilmore> we can retire the f23 and master branches in a week or two as fallout settles down 16:35:35 <dgilmore> #info FTBFS cleanup will happen post branching and f23 and master branches will be retired 16:35:49 <tyll> shall we use the alpha freeze as deadline? 16:35:56 <tyll> it would be two weeks 16:35:58 <dgilmore> tyll: yeah 16:36:25 <dgilmore> #info Alpha freeze will be the deadline to have FTBFS cleanups done 16:36:27 <tyll> I will announce it in the next e-mail 16:36:44 <dgilmore> #topic open floor 16:36:55 <dgilmore> does anyone have anything they want to raise? 16:37:00 * tyll was some things 16:37:07 <dgilmore> tyll: go ahead 16:37:49 <tyll> I was wondering if there is a tool to get notifications based on the Fedora schedule e.g. for the FTBFS cleanup 16:38:12 <dgilmore> tyll: there is a calendar for it 16:38:24 <dgilmore> but afaik it does not add notifications 16:38:49 <tyll> ok, I see 16:39:07 <dgilmore> https://fedorapeople.org/groups/schedule/f-23/f-23-releng-tasks.html 16:39:11 <bochecha__> dgilmore: after we branch, will it be too late to make progress on the sha512 migration of the lookaside cache? 16:39:13 <dgilmore> there is a ics file for it 16:39:21 <dgilmore> bochecha__: no 16:39:34 <dgilmore> bochecha__: but we will have a limited window 16:39:51 <dgilmore> bochecha__: I will try look at your script after my morning of meeting hell 16:40:11 <bochecha__> dgilmore: cool, thanks, let me know how I can help (with the script, I can't do anything about all those meetings :P) 16:40:34 <tyll> also I noticed that the ppc koji seems to be different that the other kojis as it seems to also show this bug: https://fedorahosted.org/koji/ticket/310 - e.g. signing rawhide for it does not work currently 16:40:54 <tyll> but I do not know if there is the same server-side exception 16:41:00 <dgilmore> tyll: https://fedorapeople.org/groups/schedule/f-23/f-23-releng.ics 16:41:05 <dgilmore> that is the ocs file 16:41:29 <pbrobinson> dgilmore: do you know if anyone has had a chance to look at live media creator integration with koji? 16:41:32 <dgilmore> tyll: ppc was updated to a newer koji code base last week 16:41:37 <dgilmore> we need to update the others 16:41:45 <dgilmore> so we will likely hit it everywhere 16:41:54 <dgilmore> pbrobinson: no one has :( 16:41:59 <pbrobinson> dgilmore: we need to look at the signing issue against the newer ppc koji instance 16:42:26 <dgilmore> pbrobinson: indeed 16:43:04 <dgilmore> does anyone have anything else? 16:43:16 <pbrobinson> not that I can think of 16:43:23 <dgilmore> I would like to grab a bite to eat before the council meeting in 17 minutes 16:43:55 <tyll> I am done 16:44:16 <maxamillion> nothing I can think of 16:44:21 <dgilmore> #endmeeting