15:32:38 <dgilmore> #startmeeting RELENG (2015-07-13)
15:32:39 <zodbot> Meeting started Mon Jul 13 15:32:38 2015 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:32:39 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:32:48 <dgilmore> #meetingname releng
15:32:48 <zodbot> The meeting name has been set to 'releng'
15:32:48 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion
15:32:48 <zodbot> Current chairs: bochecha dgilmore masta maxamillion nirik pbrobinson pingou sharkcz tyll
15:32:51 <dgilmore> #topic init process
15:32:51 <nirik> morning all.
15:32:58 <bochecha__> hi there
15:32:59 * nirik is going to run to get coffee, back in a min
15:33:00 * maxamillion is here
15:33:03 * sharkcz is here
15:34:12 <dgilmore> lmacken: here?
15:37:28 <dgilmore> I guess not
15:37:45 <dgilmore> #topic #6164 bodhi2 status update requested
15:37:53 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6164
15:38:05 <dgilmore> nirik: any idea of any changes here this week?
15:38:25 <nirik> I know lmacken was working on it. ;) stg is looking much more functional from a web side.
15:38:45 <dgilmore> okay
15:38:48 <nirik> I think we should push ahead and as soon as we can land it in prod and work through any problems
15:38:56 <nirik> because I really want it to land. ;)
15:39:10 <dgilmore> nirik: I really want it to land also
15:39:18 <dgilmore> but afaik it can not yet even mash a repo
15:39:48 <dgilmore> there is many basic pieces of functionality still missing
15:40:10 <nirik> I don't know the mashing status.
15:40:38 <dgilmore> what is the url for it?
15:40:48 <nirik> https://admin.stg.fedoraproject.org/updates/
15:40:55 <nirik> is the web in stg
15:41:16 <dgilmore> I thougt it was on a different url
15:41:39 <nirik> it was, but we decided to move it so it would match what it would be in prod
15:41:58 <dgilmore> might be a bad idea
15:42:07 <nirik> how so?
15:42:10 <dgilmore> because I do not think its close to ready from what I know
15:42:28 <dgilmore> if we need to get some bug fix tested on the existing code can we?
15:42:38 <pbrobinson> well lets test it and see
15:43:05 <pbrobinson> got to start somewhere in getting it ready for prod and I suppose the url is a start :)
15:43:11 <nirik> dgilmore: the bodhi1 instances are still there. we could switch it back...
15:43:21 <nirik> but really it's never been completely functional in stg...
15:43:25 <lmacken> sorry, I was attending to puppy doodies^Wduties
15:43:27 <nirik> because koji wasn't and other parts
15:43:35 <dgilmore> nirik: okay
15:43:49 <dgilmore> lmacken: so where does bodhi2 stand?
15:43:55 <nirik> I dunno if things are good enough now we could test mashing in stg or not.
15:43:57 <dgilmore> what works and what does not work?
15:44:04 <lmacken> lots of bodhi2 changes have gone in, and I have a bunch more queued up... it's getting close and today I'm hoping we'll be able to submit new updates to bodhi.stg
15:44:14 <lmacken> then from there I'm going to test full pushes
15:44:53 <dgilmore> lmacken: okay, how do we plan to mash repos and make atomic trees?
15:45:06 <lmacken> had to refactor a bunch of path issues... since in bodhi1 it's mounted to /updates, but bodhi2 is designed so it would be /updates/{updates,releases,etc}
15:45:15 <lmacken> dgilmore: the code is the same as it is in bodhi1 at the moment
15:45:22 <lmacken> Ideally i'd love to get the koji plugins written
15:45:34 <lmacken> well, the mash one was written by bochecha__, so we just need to deploy & test
15:45:45 <lmacken> then fork and tweak for atomic
15:45:57 <bochecha__> lmacken: I don't think we'll use my mash plugin
15:46:00 <lmacken> but doing that in koji is not a blocker from my perspective
15:46:00 <nirik> we have run_root now...
15:46:04 <bochecha__> lmacken: afaik we're going with the runroot one
15:46:07 <lmacken> bochecha__: dgilmore said it's probably the way to go at the FAD
15:46:11 <lmacken> (yours)
15:46:15 <bochecha__> lmacken: that was a long time ago :)
15:46:19 <lmacken> no, the recent fad
15:46:22 <bochecha__> oh
15:46:23 <dgilmore> bochecha__: I am not really sure what is teh best way to do it
15:46:40 <bochecha__> well, my plugin is still at the same place, if it's needed :)
15:46:51 <dgilmore> bochecha__: a plugin that does one task and does it well and is isolated to that is likely a bit safer
15:47:17 <dgilmore> in that if we get some argument wrong we do not risk desroying things
15:47:24 <bochecha__> "does it well", with the limited testing I did... almost 2 years ago ;)
15:47:28 <bochecha__> but sure
15:48:09 <bochecha__> the mash plugin still needs to mount /mnt/koji inside the chroot as rw, so it still has potential for destroying things (for sure, less than a plugin that runs anything as root with anything mounted inside the chroot)
15:48:19 <dgilmore> bochecha__: I guess we need to come up with a plan. do we set up everything to run as runroot tasks, or have different plugins to do teh different tasks
15:49:05 * nirik doesn't have too much opinion.
15:49:31 <nirik> I guess runroot is there and working, so I would lean toward just using it for now and if we want/need specific plugins later we could add them?
15:50:05 <dgilmore> this is probably an area we should engage with the wider releng world, talk with CentOS and Red Hat internal Releng guys and all agree on a path forward
15:50:24 <bochecha__> I also don't have much opinion on either
15:50:31 <dgilmore> we could do as runroot first and look at other options later
15:50:38 <nirik> sure, but to avoid stalling on it we can use runroot now. ;)
15:52:48 <dgilmore> sure
15:53:07 <dgilmore> just need to get teh mock bug fixed that is blocking using runroot
15:53:22 <nirik> oh yeah. ;(
15:53:31 <dgilmore> https://bugzilla.redhat.com/show_bug.cgi?id=1208092
15:54:19 * nirik should cc on it. does
15:54:49 <dgilmore> lmacken: so at what point can we move to the next step of testing in stg?
15:54:55 <dgilmore> lmacken: what needs to be done?
15:55:01 <dgilmore> lmacken: what can we do to help?
15:55:29 <lmacken> dgilmore: fixing a few bugs which I'm working on now... mostly path related, but I am very close to being able to submit new updates in stg
15:55:49 <lmacken> figuring out the mash/atomic compose in koji situation would help
15:56:10 <lmacken> but for now I'm just going to leave the push code as-is
15:56:25 <lmacken> and focus on standing up the web frontend, masher, client tools and python api
15:56:26 <dgilmore> lmacken: okay, I think going as is is best for now
15:56:36 <dgilmore> as it lets us test with a known working base
15:56:39 <lmacken> yeah
15:56:50 <dgilmore> once we have the frontend ready we can work on the backened changes
15:57:27 <dgilmore> lmacken: cool. feel free to ping me if you need anything from me
15:57:29 * nirik nods
15:57:35 <lmacken> will do
15:57:35 * dgilmore wants to get this moving forward
15:57:36 <nirik> yeah, same here. Happy to help
15:57:51 <lmacken> there are plenty of github issues to help with too :)
15:58:10 <tflink> lmacken: I can point taskotron staging at bodhi2 stg if that would help
15:58:14 <dgilmore> that would mean making a github account
15:58:38 <lmacken> tflink: it interacts with it via python-fedora, right? probably not quite ready yet
15:58:55 <lmacken> dgilmore: okay, I'll plan on migrating to pagure at some point then...
15:58:55 <tflink> lmacken: yeah, we use the normal python interfaces
15:59:05 <lmacken> tflink: okay yeah, I'm working on those currently
15:59:33 <lmacken> if anyone wants to help migrate github issues over to pagure for bodhi, that would help get more eyes on things apparently
16:00:31 <lmacken> EOF on that
16:00:52 <dgilmore> thanks lmacken
16:01:04 <dgilmore> #topic Secondary Architectures updates
16:01:05 <dgilmore> #topic Secondary Architectures update - ppc
16:01:15 <dgilmore> pbrobinson: how is ppc?
16:01:42 <pbrobinson> moving forward, there's an issue with kernels, nightly compose is running, working on getting cloud images automated
16:02:08 <sharkcz> build-wise should be few days behind primary
16:03:56 <dgilmore> cool
16:07:06 <dgilmore> #topic Secondary Architectures update - s390
16:07:11 <dgilmore> sharkcz: how is s390?
16:08:20 <sharkcz> a bit better :-) we are hours behind primary, mainly with the exception of the ghc-* stack where the compiler changed the ABI hash during f22 cycle and there are old builds using the old hash
16:09:17 <dgilmore> ghc seems to be special
16:09:19 <pbrobinson> sharkcz: you only have to look at ghc in marginally the wrong way and it changes it's abi hash!
16:09:39 <sharkcz> pbrobinson: really something like that ;-)
16:10:20 <sharkcz> don't understand how the -j value in parallel make can make the difference
16:10:27 <pbrobinson> me either!
16:12:15 <dgilmore> #topic Secondary Architectures update - arm
16:12:25 <dgilmore> #undo
16:12:25 <zodbot> Removing item from minutes: <MeetBot.items.Topic object at 0x3b936a50>
16:12:40 <dgilmore> sharkcz: nirik: where are we with the s390 hub migration
16:13:03 <nirik> I've not done anything further on it...
16:13:03 <sharkcz> in the middle, I plan to focus on it this week
16:13:29 <dgilmore> sharkcz: okay, what is missing?
16:13:57 <sharkcz> the setup for the users dealing with shadow configs
16:14:30 <dgilmore> okay, can we move the hub, then sort that out?
16:14:41 <pbrobinson> sharkcz: let's you and I sync on that this week and come to an agreement
16:15:20 <sharkcz> pbrobinson: ack
16:15:44 <dgilmore> #info sharkcz and pbrobinson to meet and agree on how to do koji-shadow in ansible
16:15:45 <sharkcz> dgilmore: nirik would like to see it in ansible, so we should agree on the setup first
16:15:54 * nirik is happy to help however he can
16:16:07 <dgilmore> sharkcz: it has to be in ansible
16:16:17 <dgilmore> sharkcz: agreeing on how to do it should be first
16:16:29 <dgilmore> sharkcz: but does that really prevent us migrating the hub
16:16:31 <maxamillion> oh man, koji-shadow setup/deployment in ansible would be awesome
16:16:41 <dgilmore> and using the new ansibled one
16:17:06 <sharkcz> dgilmore: dunno much about ansible, can it incrementally add things?
16:17:14 <pbrobinson> maxamillion: there's really not that much to the setup
16:17:17 <dgilmore> sharkcz: we should still be able to run the setup from the old box, just point it at the new hub
16:17:34 <dgilmore> sharkcz: you can
16:17:36 <pbrobinson> sharkcz: I know enough, I think we can sort that out OOB from the meeting
16:17:47 <nirik> sure, we can have a base config and add things to it...
16:17:48 <sharkcz> ok
16:18:23 <dgilmore> #topic Secondary Architectures update - arm
16:18:32 <dgilmore> pbrobinson: okay lets really do arm now
16:18:34 <maxamillion> pbrobinson: yeah, people say that but then there's no documentation for it ... sooo .... >.>
16:18:35 <pbrobinson> so we're getting there, the main packaging blockers have unblocked for the time being and we're slowly closing the gap to rawhide
16:18:47 <pbrobinson> we'll be losing capacity from around the 16th for 10 days or so because for some reason in the US it takes 10 days to get something from BOS to PHX
16:19:15 <pbrobinson> that's 80% capacity
16:19:35 <pbrobinson> so we'll be running with 2 builders
16:19:38 <nirik> 10 days!?
16:19:59 <pbrobinson> nirik: Yep, I'm sure it's because we're taking the push bike delivery option or some shit
16:20:11 <nirik> thats crazy. Should be 1-2 tops... ;(
16:20:25 <tibbs|w> It can be done in six hours if you're willing to drive the stuff to and from an airport.
16:21:14 <pbrobinson> nirk: apparently somebody is being a tight arse and doing cheap arse road freight option and won't let me ship some other boxes doing nothing and swap them back later
16:21:25 <dgilmore> pbrobinson: how will that effect alpha readiness?
16:21:49 <nirik> pbrobinson: odd. perhaps out of meeting we can poke them to use a better/faster method...
16:22:18 <nirik> perhaps stickster could talk with them
16:22:19 <pbrobinson> dgilmore: fuck knows, it will likely be massively delayed, I've told people that and they seem to think I'm happy to work 24*7 so if it's delayed fuck them all!!!!!
16:23:12 <dgilmore> #info Alpha likely to be massively delayed due to time to get hardware mmoved
16:23:26 <dgilmore> anything else for arm?
16:23:34 <pbrobinson> not from my side
16:24:56 <dgilmore> #topic f23 status
16:25:03 <dgilmore> so we have branching tomorrow
16:25:14 <dgilmore> we have pungi kinda working
16:25:18 <pbrobinson> wow, that's come up fast :)
16:25:31 <dgilmore> I pinged clark and he is going to fix teh mock bug today
16:25:33 <nirik> and tc1 is also due tomorrow
16:25:42 <nirik> (but of course makes no sense until branching is done)
16:25:49 <maxamillion> pbrobinson: +1
16:25:52 <dgilmore> nirik: yeah.
16:26:08 <dgilmore> so we have a week or so to get tooling lined up
16:26:51 <pbrobinson> dgilmore: got a BZ on the mock bug?
16:27:27 <sharkcz> pbrobinson: https://bugzilla.redhat.com/show_bug.cgi?id=1208092
16:28:22 <pbrobinson> ah... look, I'm already on it (not shocking!)
16:28:45 <dgilmore> pbrobinson: linked it earlier
16:29:00 <dgilmore> so this next week will be super hectic
16:29:19 <pbrobinson> dgilmore: yea, but you know I generally just ignore you ;-)
16:29:47 <dgilmore> pbrobinson: ;) hey now
16:29:53 <pbrobinson> :-D
16:30:13 <tyll> Are we going to retire FTBFS builds before branching as outlined on the schedule? Currently it is only about one week since I sent the initial reminder e-mail and recently there were still a lot pkgs affected
16:30:18 <dgilmore> #info branching tomorrow
16:30:35 <dgilmore> tyll: well branching is tomorrow
16:30:45 <dgilmore> we would have to do it first thing
16:31:21 <pbrobinson> tyll dgilmore: personally I'd be flexible and give people some more time, there's movement there and I don't think it hurts so much
16:31:35 <dgilmore> pbrobinson: I agree
16:31:58 <sharkcz> +1
16:32:29 <pbrobinson> when I looked there's less than 30 packages older than F-21 that are broken, I fixed a few up, and retired some others
16:33:43 <dgilmore> does anyone have a strong opinion that it has to happen before branching?
16:33:53 <dgilmore> Branching tomorrow has to happen
16:34:10 <dgilmore> so to retire them before branching would mean we have to rush it through
16:34:37 <dgilmore> we can retire the f23 and master branches in a week or two as fallout settles down
16:35:35 <dgilmore> #info FTBFS cleanup will happen post branching and f23 and master branches will be retired
16:35:49 <tyll> shall we use the alpha freeze as deadline?
16:35:56 <tyll> it would be two weeks
16:35:58 <dgilmore> tyll: yeah
16:36:25 <dgilmore> #info Alpha freeze will be the deadline to have FTBFS cleanups done
16:36:27 <tyll> I will announce it in the next e-mail
16:36:44 <dgilmore> #topic open floor
16:36:55 <dgilmore> does anyone have anything they want to raise?
16:37:00 * tyll was some things
16:37:07 <dgilmore> tyll: go ahead
16:37:49 <tyll> I was wondering if there is a tool to get notifications based on the Fedora schedule e.g. for the FTBFS cleanup
16:38:12 <dgilmore> tyll: there is a calendar for it
16:38:24 <dgilmore> but afaik it does not add notifications
16:38:49 <tyll> ok, I see
16:39:07 <dgilmore> https://fedorapeople.org/groups/schedule/f-23/f-23-releng-tasks.html
16:39:11 <bochecha__> dgilmore: after we branch, will it be too late to make progress on the sha512 migration of the lookaside cache?
16:39:13 <dgilmore> there is a ics file for it
16:39:21 <dgilmore> bochecha__: no
16:39:34 <dgilmore> bochecha__: but we will have a limited window
16:39:51 <dgilmore> bochecha__: I will try look at your script after my morning of meeting hell
16:40:11 <bochecha__> dgilmore: cool, thanks, let me know how I can help (with the script, I can't do anything about all those meetings :P)
16:40:34 <tyll> also I noticed that the ppc koji seems to be different that the other kojis as it seems to also show this bug: https://fedorahosted.org/koji/ticket/310 - e.g. signing rawhide for it does not work currently
16:40:54 <tyll> but I do not know if there is the same server-side exception
16:41:00 <dgilmore> tyll: https://fedorapeople.org/groups/schedule/f-23/f-23-releng.ics
16:41:05 <dgilmore> that is the ocs file
16:41:29 <pbrobinson> dgilmore: do you know if anyone has had a chance to look at live media creator integration with koji?
16:41:32 <dgilmore> tyll: ppc was updated to a newer koji code base last week
16:41:37 <dgilmore> we need to update the others
16:41:45 <dgilmore> so we will likely hit it everywhere
16:41:54 <dgilmore> pbrobinson: no one has :(
16:41:59 <pbrobinson> dgilmore: we need to look at the signing issue against the newer ppc koji instance
16:42:26 <dgilmore> pbrobinson: indeed
16:43:04 <dgilmore> does anyone have anything else?
16:43:16 <pbrobinson> not that I can think of
16:43:23 <dgilmore> I would like to grab a bite to eat before the council meeting in 17 minutes
16:43:55 <tyll> I am done
16:44:16 <maxamillion> nothing I can think of
16:44:21 <dgilmore> #endmeeting