16:56:09 <dgilmore> #startmeeting RELENG (2015-12-14)
16:56:09 <zodbot> Meeting started Mon Dec 14 16:56:09 2015 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:56:09 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:56:09 <zodbot> The meeting name has been set to 'releng_(2015-12-14)'
16:56:21 <dgilmore> #meetingname releng
16:56:21 <zodbot> The meeting name has been set to 'releng'
16:56:21 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion
16:56:21 <zodbot> Current chairs: bochecha dgilmore masta maxamillion nirik pbrobinson pingou sharkcz tyll
16:56:24 <dgilmore> #topic init process
16:56:26 * masta is here
16:56:51 <maxamillion> .hello maxamillion
16:56:52 <zodbot> maxamillion: maxamillion 'Adam Miller' <maxamillion@gmail.com>
16:57:20 <nirik> morning
16:59:43 <dgilmore> so we have no tickets for the meeting
16:59:59 <dgilmore> epel7 is now building ppc64le by default
17:00:19 <masta> nice
17:00:29 <dgilmore> there is bodhi work that needs to happen. but outside of updating the mash configs I am not sure what we need to do
17:01:25 <masta> yeah, could be some whitelist of arches to make sure ppc64le is there
17:01:57 <dgilmore> may be some issues with mm
17:02:10 <dgilmore> but not sure. hopefully it just picks it up and runs with it
17:02:27 * nirik nods
17:03:03 <maxamillion> is there going to be a mass rebuild for ppc64le or has it been shadowing to pick up all the epel stuff already?
17:03:44 <dgilmore> maxamillion: it was all scratch built with the same nvrs and imported
17:04:27 <maxamillion> rgr
17:04:47 <maxamillion> dgilmore: can those just be tagged/imported so that it goes out to the repos?
17:04:55 <dgilmore> maxamillion: done already
17:05:16 <dgilmore> we just need to add ppc64le to the mash configs, make sure they are all signed and do a push
17:05:19 <maxamillion> ahhh ok
17:05:24 <maxamillion> cool cool
17:05:58 <dgilmore> we may not need to change anything in bodhi
17:06:03 <dgilmore> its a bit up in teh air
17:06:37 <maxamillion> is there somewhere we should file a ticket? or someone we should follow up with to make sure?
17:06:44 * sharkcz is somewhat here
17:07:08 <dgilmore> maxamillion: I need to get with lmacken and see
17:07:13 <dgilmore> which reminds me
17:07:18 <maxamillion> dgilmore: rgr
17:07:29 <dgilmore> we need to make sure we have docs from lmacken on how to do everything with bodhi
17:08:05 <masta> hopefully there are not too many exclusive/exclude-arch packages in epel7 that drop LE. Would be nice to have good parity.
17:08:19 <maxamillion> "do everything" ... what specifically? (legitimately curious, I'd like to try and start writing down RelEng processes because I know there's plenty I'm still not familiar with and would love to have references for my use as well as for potential newcomers into releng space)
17:08:40 <dgilmore> maxamillion: adding releases, doing teh eol process,
17:08:48 <maxamillion> dgilmore: gotchya
17:08:55 <dgilmore> maxamillion: basically everything that we need to do
17:09:37 <maxamillion> dgilmore: right, but if someone asked me "what is everything that Rel Eng needs to do in bodhi?" I wouldn't know how to answer that question properly, and I'd like to have us as a group have those kinds of things written down in the releng docs
17:10:10 <dgilmore> maxamillion: honestly I am not 100% sure
17:10:16 <maxamillion> dgilmore: ah, rgr
17:10:19 <dgilmore> which is paert of the problem
17:10:22 <maxamillion> well we should sort that out :D
17:10:29 <dgilmore> and why we need docs
17:10:36 <lmacken> dgilmore: what bodhi work needs to happen to add another arch?
17:10:45 <dgilmore> lmacken: we do not know
17:11:01 <dgilmore> lmacken: we know we need to update the mash configs
17:11:03 <lmacken> I am pretty sure the answer is: none
17:11:11 <dgilmore> outside of that we have no clue
17:11:13 <lmacken> yeah, bodhi doesn't care about arches
17:11:34 <lmacken> only koji tags, so I'm almost positive it's just a matter of mash config changes
17:11:34 <masta> lmacken: that is great news then =)
17:11:46 <maxamillion> lmacken: rocking +1
17:11:49 <dgilmore> lmacken: okay. hopefully it does not cause things to puke by adding an arch
17:12:13 <lmacken> we can test it with updates-testing any time
17:12:21 <dgilmore> lmacken: epel-testing
17:12:23 <dgilmore> :P
17:15:17 <masta> so I guess this week Peter is on push duty, maybe we can get LE in the configs before he starts pushing (presumably) later today?
17:15:49 <masta> or perhaps we plan it out more after scoping?
17:15:50 <dgilmore> masta: I will work with him on it
17:15:55 <nirik> he already pushed stuff this morning. ;)
17:15:58 <dgilmore> not sure where the signing stands
17:16:29 <dgilmore> but it seems we just need to update the mash configs
17:16:46 <dgilmore> lmacken: but we would really appreciate docs on how to do all the admin tasks
17:16:57 <dgilmore> release management etc
17:17:05 <lmacken> dgilmore: yeah, I have a half-modified SOP from the last EOL that I need to wrap up
17:17:12 * lmacken makes a note to finish that this week
17:17:41 <dgilmore> lmacken: thanks
17:17:57 <dgilmore> #action lmacken to finish updating the sops for bodhi
17:19:46 <dgilmore> #topic updates pushing
17:19:55 <dgilmore> so pbrobinson is on duty this week
17:19:59 * nirik nods.
17:20:03 <dgilmore> I am on duty next week
17:20:11 <nirik> I'm going to be around for the holidays, happy to take over for anyone or whatever
17:20:13 <dgilmore> but we have thursday and Friday off work
17:20:32 <dgilmore> I will probably push on thusday but not Friday
17:20:48 <dgilmore> not going to do anything on Chirstmas day
17:20:58 <nirik> except open presents? ;)
17:21:00 <dgilmore> we then have shutdown at Red Hat
17:21:09 <dgilmore> nirik: well that and drink eggnog
17:21:20 <dgilmore> masta: is on duty for the shutdown week
17:21:22 <nirik> mmmmm... need to get some eggnog
17:21:52 <nirik> as noted, I should be around, so I am happy to take over for anyone... masta: you want me to do that week?
17:22:03 * nirik doesn't expect there will be many updates really tho
17:22:21 <masta> yes, I plan to push during shutdown as best effort. That means my family would have to massively guilt me into not pushing, which I expect to resist.
17:22:26 <dgilmore> I am wondering if for shutdown we do a day each, or just say no updates?
17:22:30 <dgilmore> or something else
17:23:14 <nirik> how about we look and if there's not that many we just defer to the next day (or no security updates or something)
17:23:36 <masta> from experience, the pushes are minimal during shutdown. So a few minutes of work signing mostly.
17:23:38 <nirik> note also, that second week in jan I will be out at phx2, so likely will not want to be doing any pushing that week if thats when I am scheduled
17:24:14 <masta> nirik: I can cover you if needed, just let me know.
17:24:23 <dgilmore> nirik: you ate the week of the 4th
17:24:29 <nirik> ok.
17:24:35 <dgilmore> are even
17:24:38 <nirik> that week I should be around.
17:25:25 <dgilmore> nirik: cool
17:27:34 <dgilmore> #info nirik and masta will do best effort pushing during shutdown
17:27:52 <dgilmore> #topic Open Floor
17:28:02 <dgilmore> lets skip secondaries this week.
17:28:20 <dgilmore> pbrobinson: is not here and sharkcz said he mostly wasn't here
17:28:33 <dgilmore> does anyone have anything they want to discuss
17:28:34 <nirik> I'm hoping to coordinate with pbrobinson this week and migrate arm hub/db... but if not, then we can do it in jan
17:28:47 <dgilmore> nirik: :) would be nice
17:28:55 <nirik> that replacement energycard should be arriving tomorrow...
17:29:03 <dgilmore> That will be very nice
17:29:12 <masta> oh neat, then I guess after that we can buile epel7 for rhelsa =)
17:29:26 <dgilmore> masta: maybe
17:30:18 <nirik> I cannot currently powercyle arm04 due to crappy apc devices.
17:30:33 <nirik> do we want to adjust what hosts are in what channels any?
17:30:39 <nirik> or I guess we can do that out of meeting
17:32:23 <dgilmore> nirik: we can do it out of meeting
17:32:38 <dgilmore> but it would be nice to get a couple of arm systems with ssd
17:32:58 <dgilmore> there is a pretty significant slow down on spinning disk
17:34:23 <dgilmore> #info we will have a meeting next week, but not the week after, and probably not on the 4th unless we have something that needs discussing
17:34:28 <nirik> we are getting 15more ssds, but not sure when
17:35:41 <dgilmore> that will be nice
17:36:15 <dgilmore> nirik: reminds me. I had a request last week for docker base images for arm
17:36:35 <dgilmore> outside of the meeting I would like to discuss how we can do it short term
17:36:50 <dgilmore> all options pretty much suck
17:37:28 <dgilmore> does anyone have anything else?
17:37:48 <nirik> sure.
17:39:15 <masta> I'm trying to help with old tickets. #4084 is really old and I'd like to get rid of it.
17:39:42 <masta> took me a while to fully appreciate the problem statement.
17:40:00 <masta> might take a bit more thought on how to solve.
17:41:07 <masta> I'm hoping it would not be too expensive to add a routine to mash to scope-out what was done in release repos for a multilib, and do the right thing for the scenario durring updates.
17:41:39 <masta> that's it for me, eof
17:42:30 <dgilmore> masta: thats a really difficult problem to solve
17:42:57 <dgilmore> pretty much you are going to have to feed it a second repo to use for resolving deps
17:43:04 <masta> dgilmore: I know, that's way it's been open forever and neglected.
17:43:18 <dgilmore> but that does remind me
17:43:40 <dgilmore> I am working on getting together a koji update with a bunch of bug fixes that we need for infra
17:44:01 <dgilmore> i will be pushing out updates to all supported releases also
17:44:06 <dgilmore> hopfully this week
17:44:26 <masta> nice
17:45:35 <nirik> excellent
17:47:17 <dgilmore> I hit bugs when I tried to do a real image build rather than a scratch one
17:47:34 <nirik> oh yeah, did you get stg working?
17:47:53 <nirik> also, is it ok to run ansible on koji/build* in stg? or do you have changes there?
17:49:49 <dgilmore> nirik: did not get stg working
17:49:57 <dgilmore> imag building there just does not work
17:50:13 <nirik> humf. we will have to visit that when we get time/energy then
17:50:21 <dgilmore> nirik: I think i have changes, but go ahead
17:52:10 <dgilmore> nirik: I did have to change the redirects for kojipkgs to not use https
17:52:27 <dgilmore> oz/imagefactory apparently do not do ssl
17:52:34 <nirik> yeah, thats just sad.
17:52:40 <nirik> but ok
17:52:50 <dgilmore> apparently never tested
17:52:54 <dgilmore> and it does not work
17:52:55 <nirik> I am trying to get the delta for ansible changes as close to 0 as I can.
17:53:07 <dgilmore> okay
17:53:39 <dgilmore> nirik: we use http in prod for all teh koji communications
17:53:50 <dgilmore> at teh least we do not redirect it to https
17:53:57 <dgilmore> the image tasks use http
17:54:28 <nirik> that may be some stg difference...
17:55:59 <dgilmore> yeah
17:56:16 <dgilmore> #info planning a koji update this week
17:56:31 <dgilmore> need to be sure to get a fix for https://fedorahosted.org/rel-eng/ticket/6274 in
17:56:32 <maxamillion> dgilmore: bold move right before the winter break
17:56:37 <dgilmore> maxamillion: :P
17:56:42 <maxamillion> +1 DO IT LIVE
17:56:57 <dgilmore> maxamillion: I won't be here to fix it ;)
17:57:13 <maxamillion> me either ... LET IT BURN
17:57:14 <maxamillion> >.>
17:57:18 <dgilmore> I suspect we will have a couple of koji updates in teh new year also
17:57:23 <maxamillion> likely so
17:57:36 <dgilmore> maxamillion: where is osbs at?
17:58:19 <maxamillion> dgilmore: nowhere official yet, it will be in osbs.fedoracloud.org when I can get things working and request that DNS pointer, then I'll install the koji plugin in stg koji that will farm off builds to it
17:58:48 <maxamillion> dgilmore: right now I have a cloud instance in fedora infra cloud that I'm fighting with making work, then I need to make sure I can duplicate making it work via ansible
17:58:49 <dgilmore> maxamillion: hrrrm okay
17:59:05 <dgilmore> maxamillion: one of the changes I am bringing into koji is content generators
17:59:15 <maxamillion> dgilmore: cool cool
17:59:22 <dgilmore> which is what they are going to use internally and not the plugin afaik
17:59:37 <dgilmore> maxamillion: we need to make sure we line things up the same
17:59:46 <nirik> maxamillion: I can setup a base playbook for you and dns, etc... just let me know the name and size...
17:59:49 <maxamillion> dgilmore: I was under the impression that the plugin can do a meta-data import with content generators
17:59:59 <maxamillion> nirik: that would be awesome, thanks
18:00:12 <nirik> then you can just add to that
18:00:22 <dgilmore> maxamillion: okay, talking to mikem he said the plugin did not use content generators
18:00:26 <maxamillion> nirik: let me look at the size, but I'd like to name to be something like osbs.dev.fedoracloud.org or similar (I'm open to suggestion)
18:00:40 <dgilmore> could be some confusion there about things as they are so much in flight
18:00:48 <nirik> our convention on the others is "name-dev.fedorainfracloud.org'
18:00:50 <maxamillion> dgilmore: right, I knew it could work without but I thought it was also supporting them ... :/
18:01:09 <maxamillion> nirik: ah, +1 osbs-dev.fedorainfracloud.org would be perfect
18:02:24 <dgilmore> maxamillion: guess we need to be sure of what we need
18:02:46 <dgilmore> I know I need some pykickstart fixes
18:02:46 <maxamillion> nirik: I'm currently using a ms1.medium and considering the limited scope of the dev osbs, I think that should be sufficient
18:02:47 <nirik> maxamillion: ok, can talk outside meeting on sizes, etc.
18:02:51 <nirik> ok.
18:02:58 <nirik> I can get that setup later today hopefully
18:03:02 <dgilmore> and some other fixes for ssl issues and image building issues
18:03:05 <maxamillion> nirik: awesome, greatly appreciated
18:03:35 <dgilmore> okay lets be done for today
18:03:38 <maxamillion> +1
18:03:42 <dgilmore> #endmeeting