16:30:57 <dgilmore> #startmeeting RELENG (2015-11-02)
16:30:57 <zodbot> Meeting started Mon Nov  2 16:30:57 2015 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:57 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:31:08 <dgilmore> #meetingname releng
16:31:08 <zodbot> The meeting name has been set to 'releng'
16:31:08 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion
16:31:08 <zodbot> Current chairs: bochecha dgilmore masta maxamillion nirik pbrobinson pingou sharkcz tyll
16:31:11 <dgilmore> #topic init process
16:31:15 <maxamillion> .hello maxamillion
16:31:16 <zodbot> maxamillion: maxamillion 'Adam Miller' <maxamillion@gmail.com>
16:31:25 <dgilmore> .gday ausil
16:31:32 * nirik is around, but as usual on a monday morning putting out fires and really busy
16:31:32 <dgilmore> we should make that a thing
16:31:43 * sharkcz is not here
16:33:14 <maxamillion> dgilmore: who owns zodbot? I imagine it'd just be a simple alias entry
16:33:57 <dgilmore> maxamillion: infra does
16:34:05 <dgilmore> sharkcz: thanks :)
16:34:28 <dgilmore> we have no tickets with meeting keyword
16:34:35 <dgilmore> #topic meeting time
16:34:43 <maxamillion> dgilmore: rgr
16:34:45 * pbrobinson is here
16:35:00 <dgilmore> some people almost never turn up
16:35:20 <dgilmore> and others that probably should be here are not and I am not sure they have it in their agenda
16:35:27 * linuxmodder hanging in back (busy but  here)
16:35:35 <dgilmore> perhaps we need to try find a better time
16:35:38 <pbrobinson> dgilmore: who do you think should be here that isn't?
16:36:00 <dgilmore> pbrobinson: kushal, lmacken at the least
16:36:33 <dgilmore> though maybe not kushal
16:36:34 <pbrobinson> do they know they should be expected?
16:36:48 <dgilmore> pbrobinson: no idea, but lmacken at the least should
16:36:49 <pbrobinson> I mean lmacken knows
16:36:52 <kushal> I am here now
16:37:28 <dgilmore> kushal: :) thanks
16:37:45 <kushal> I had it in my calendar, but sadly it is not on UTC :(
16:37:59 <kushal> I mean the meeting time changed for me :(
16:38:03 <dgilmore> kushal: the meeting is not its on US time
16:38:10 <kushal> dgilmore, Yes.
16:38:16 <kushal> dgilmore, I will fix my calendar.
16:38:38 <dgilmore> it is 10:30am US Central time year round
16:38:54 * lmacken here
16:38:58 <dgilmore> but maybe we need to change the time. I am not sure monday morning works
16:39:02 <nirik> monday kinda sucks for me, as I get pulled into fires monday morning when I get in
16:39:33 <pbrobinson> I'm relatively flexible, just earlier the better ;-)
16:40:59 <dgilmore> sharkcz almost never turns up, and I think he needs to always be here
16:41:01 * masta looks in
16:41:05 <masta> howdy
16:41:40 <dgilmore> I guess I will send an email to the list to start a discussion on meeting time
16:42:05 <dgilmore> #topic bodhi issues
16:42:19 <dgilmore> we have hit a lot of issues with bodhi in the last few weeks
16:42:26 <dgilmore> we need to make sure we file bugs
16:42:45 <dgilmore> lmacken: we need you to update the bodhi infrastructure sop
16:42:52 <lmacken> the big ones have been nfs & ostree related... the rpm-ostree one has been fixed.
16:42:59 <lmacken> dgilmore: yes, I already started working on it
16:43:36 <lmacken> Bodhi should be good to go for F23 at the moment
16:44:00 <masta> lmacken: I'm on push duty this week, I'll make sure to mention anything weird.
16:44:06 <lmacken> masta: thanks...
16:44:34 <lmacken> for those pushing, please file issues or ping me when weird things explode. restarting it and trying again can make things worse :P
16:44:43 <dgilmore> masta: all release can be pushed
16:44:46 <dgilmore> releases
16:44:50 <pbrobinson> did the days required in testing get upped from 3 -> 7?
16:44:59 <lmacken> pbrobinson: yeah, it should be back to the defaults
16:46:12 <dgilmore> #info report all issues with pushing
16:46:20 <dgilmore> lmacken: where shoudl issues be reported?
16:46:30 <lmacken> dgilmore: github gets the most attention
16:46:45 <dgilmore> lmacken: for those that doe not have github accounts?
16:46:47 <masta> dgilmore: ok thanks, I was going to only do f23-testing
16:46:49 <lmacken> or bz or pagure for those who refuse to use it
16:47:07 <masta> dgilmore: so I'll just do them all in one push
16:47:15 <dgilmore> masta: yes
16:47:52 <dgilmore> https://pagure.io/bodhi/issues
16:48:12 <dgilmore> #info Please file any RFE's and bug reports
16:48:37 <dgilmore> lets move on
16:48:51 <dgilmore> #topic Fedora 23 retrospective
16:49:12 <dgilmore> should we have a meeting to go over f23 and see what went well and what needs improvement?
16:49:57 <pbrobinson> might be a good idea
16:50:06 <pbrobinson> or start the discussion on the ML?
16:50:20 <dgilmore> either could be good
16:50:35 <nirik> sure. overall I think things went pretty well...
16:50:41 <dgilmore> if we are going to have a meeting it will need to be wednesday
16:51:07 <nirik> for f24 it might be nice to rotate who is doing composes so it doesn't all fall on dgilmore and more people know what to do... :)
16:51:14 <maxamillion> +1
16:51:28 <maxamillion> I'd like to participate and do at least one compose for F24
16:51:39 <dgilmore> I am planning to make major changes to how we compose f24
16:51:41 <pbrobinson> I did some of them in the Beta cycle
16:52:10 <dgilmore> an issue we hit a lot is that the compose requests come in late US time
16:52:19 <dgilmore> we have zero APAC help
16:52:58 <pbrobinson> well I could in on APAC time if needed.... but then there's no EMEA coverage ;-)
16:53:06 <dgilmore> unless pbrobinson or myself happen to be in APAC
16:53:32 <kushal> dgilmore, If I get you a volunteer, can you (and others) help him/her to get trained?
16:54:02 <kushal> for APAC time :)
16:54:08 <maxamillion> dgilmore: as a first step towards getting APAC help, lets get your changes to the process in for F24, get the new process well documented and then we can potentially get others to participate in different TZs
16:54:17 <pbrobinson> I'm not sure we should be doing that, there needs to be access to signing and all sorts of other core infra
16:54:21 <dgilmore> kushal: yes,  but it is something that requires someone build up trust
16:54:32 <kushal> dgilmore, yup, no doubt in that part :D
16:54:44 <dgilmore> which is the hardest bit
16:55:19 <dgilmore> pbrobinson: right
16:55:33 <dgilmore> if we get to my ideal world it will not matter
16:55:42 <dgilmore> as composes will be hands off
16:56:02 <masta> yeah, I'd like to try a compose too.
16:56:53 <nirik> yeah, that would be nice.
16:57:20 <pbrobinson> we're getting there, pungi4 going live would be the next step towards that
16:57:35 <dgilmore> #ACTION dgilmore to figure out how to best to a retrospective
16:57:47 <dgilmore> #undo
16:57:47 <zodbot> Removing item from minutes: ACTION by dgilmore at 16:57:35 : dgilmore to figure out how to best to a retrospective
16:57:56 <dgilmore> #ACTION dgilmore to figure out how to best do a retrospective
16:58:16 <dgilmore> #topic next two weeks
16:58:54 <dgilmore> I will be on PTO from Thursday November 5th to Friday November 13th
17:00:06 <pbrobinson> dgilmore: OK, so is there anything we need to be aware of in that window, I think it should all be just standard process, poke rawhide etc as needed?
17:00:24 <dgilmore> pbrobinson: should just be business as normal
17:00:59 <dgilmore> I will be available if needed but I plan to just work on getting a bunch of things done on the house and take a few days to relax
17:01:12 <pbrobinson> I'm sure we can survive ;-)
17:01:31 <pbrobinson> we'll try not to miss you too much ;-)
17:01:35 <dgilmore> I have no doubt
17:01:49 <dgilmore> pbrobinson: I am sure you will not miss me at all ;)
17:01:54 <kushal> :)
17:02:13 <pbrobinson> dgilmore: I don't know, I speak to you most days one way or the other :-P
17:03:40 <dgilmore> #topic Secondary Architectures updates
17:03:44 <dgilmore> pbrobinson: indeed
17:04:05 <pbrobinson> so aarch64 and ppc64 have been signed off
17:04:06 <dgilmore> I made a suggestion last week we merge the secondary arch hubs into one
17:04:11 <dgilmore> I am not sold on it
17:04:18 <pbrobinson> i'm not either
17:04:23 <dgilmore> pbrobinson: excellent
17:04:55 <dgilmore> I would like us to sit down and figure out the long term plan for secondary arches and the tooling from the releng perspective
17:05:21 <pbrobinson> I would like to see what we discussed at flock happen
17:05:26 <dgilmore> I strongly believe that long term the definition is just where we put the bits
17:05:33 <pbrobinson> IE separate the term of secondary from the koji instances
17:05:35 <dgilmore> and the compose tools will do the work
17:05:42 <dgilmore> but how do we get there
17:06:05 <dgilmore> pbrobinson: I would like to see that be a reality also
17:06:20 <dgilmore> we just need to figure out how to get there
17:06:45 <pbrobinson> I think this meeting is probably not the best spot to discuss it in the short term, likely too big a topic for today
17:06:55 <dgilmore> pbrobinson: likely yes
17:07:07 <dgilmore> we likely will need a meeting just for it
17:07:15 <dgilmore> but I would like folks to be thinking about it
17:07:22 <pbrobinson> I'm slowly working through what I think would be a proposal in my head to write up and send out for further discussion
17:07:29 <dgilmore> I think I will try organise a meeting when I am back
17:08:41 <dgilmore> #topic Secondary Architectures update - ppc
17:08:47 <dgilmore> pbrobinson: ppc is ready?
17:08:49 <maxamillion> just for posterity, we're talking about removing the need for shadow koji instances and allowing secondary arches to join the main koji environment, but just not as something that is a blocker or gates anything, yes?
17:08:54 <maxamillion> oh ... nvm ... too late
17:09:04 <dgilmore> #undo
17:09:04 <zodbot> Removing item from minutes: <MeetBot.items.Topic object at 0x38685ad0>
17:09:09 <pbrobinson> dgilmore: yes, syncing out to the mirrors as this meeting runs
17:09:14 <dgilmore> #topic Secondary Architectures update - ppc
17:09:20 <dgilmore> maxamillion: kinda
17:09:24 <maxamillion> dgilmore: alright
17:09:27 <dgilmore> pbrobinson: okay
17:09:37 <dgilmore> #info ppc is on its way to the mirrors
17:09:57 <pbrobinson> all tags/targets locked down and adjusted
17:09:58 <dgilmore> pbrobinson: anything else ppc related?
17:10:03 <pbrobinson> not really
17:10:15 <dgilmore> pbrobinson: one quick topic
17:10:17 <dgilmore> epel
17:10:22 <dgilmore> adding ppc64le
17:10:24 <pbrobinson> will be rebuilding EPEL builders this week
17:10:41 <maxamillion> oh, that's kind of awesome
17:10:42 <pbrobinson> and sending out a plan for the ppc64le build and import later in the week
17:10:52 <maxamillion> I know some folks who will be really happy for ppc64le in EPEL
17:11:00 <pbrobinson> I wanted to get F23 out of the way first
17:11:03 <maxamillion> +1
17:11:12 <pbrobinson> maxamillion: I know dozens
17:11:12 <masta> yeah that will be great
17:11:17 <dgilmore> pbrobinson: :) infra is still frozen until wednesday
17:11:19 <maxamillion> pbrobinson: I know like ... 4
17:11:28 <maxamillion> pbrobinson: but still, people! stuff and things!
17:11:29 <pbrobinson> dgilmore: I'm aware
17:11:54 <dgilmore> :)
17:11:56 <dgilmore> okay
17:11:57 <dgilmore> #topic Secondary Architectures update - s390
17:12:02 <dgilmore> sharkcz: is not here
17:12:10 <dgilmore> pbrobinson: any idea where it stands?
17:12:11 <pbrobinson> basically the plan is to have epel for LE ready around the 7.2 GA timeframe
17:12:33 <pbrobinson> I think s390 is looking OK, I suspect he'll release next week
17:12:40 <dgilmore> okay
17:12:49 <dgilmore> #topic Secondary Architectures update - arm
17:13:02 <pbrobinson> arm is the same state as ppc
17:13:09 <pbrobinson> f23 is done
17:13:19 <pbrobinson> tags/targets locked down etc
17:13:22 <dgilmore> #info f23 arm is done
17:14:15 <nirik> I'd like to start moving forward with arm hub migration again after the freeze.
17:14:29 <dgilmore> nirik: that woudl be good
17:14:46 <pbrobinson> nirik: you're reading my mind
17:15:16 <pbrobinson> nirik: post release I'm ready to do both arm and ppc, I'll be rebuilding all the compose boxes too to standard
17:15:25 <nirik> great.
17:15:32 <nirik> happy to work with you on all that
17:15:34 <pbrobinson> nirik: and once that is done I think arm/ppc will be 100% ansible
17:15:44 <nirik> yep. Will be lovely. :)
17:15:48 <maxamillion> "arm hub" ?
17:16:08 <dgilmore> maxamillion: arm.koji.fp.o
17:16:10 <maxamillion> ah
17:16:21 <maxamillion> where is that migrating to?
17:16:40 <pbrobinson> nirik: I'm got some plans to move the koji-shadow stuff to a dedicated VM for some automation, will discuss later and file RFR etc a bit later this week probably
17:16:46 <dgilmore> maxamillion: from a manually setup box with no fas, to something that has fas and is managed in ansible
17:17:01 <maxamillion> dgilmore: ah, +1
17:17:04 <pbrobinson> nirik: that should mean there doesn't need to be any isms of the secondary hubs in terms of accesss
17:17:04 <nirik> pbrobinson: ok, yeah, dgilmore mentioned that a while back...
17:17:35 <pbrobinson> nirik: this is a bit different, some issues on PPC from a week or so ago
17:17:43 <nirik> ok
17:18:23 <dgilmore> anything else arm related?
17:18:37 <pbrobinson> not at the moment
17:18:51 <dgilmore> #topic Open Floor
17:19:11 <dgilmore> does anyone have anything they want to bring up?
17:20:17 <tyll> What is the purpose/meaning of this: https://fedoraproject.org/wiki/ReleaseEngineering/PriorityPipeline ?
17:21:01 <dgilmore> tyll: it is to prioritise the development work for releng
17:21:25 <dgilmore> to show people what we are working on and what our priorities are
17:22:49 <dgilmore> tyll: do you have questions about it?
17:23:06 <tyll> How is it decided what to add there and what is the priority of something?
17:23:41 <dgilmore> tyll: though meetings with stakeholders as they are identified
17:23:46 <tyll> I was wondering what would be needed to get using a proper CA for koji added to this
17:24:15 <dgilmore> tyll: talk to acarter
17:24:39 <dgilmore> we are still getting together the full list of things
17:24:56 <dgilmore> we do have plans to switch the ca used for koji
17:25:11 <dgilmore> we have yet to work out a plan for the fedora CA
17:26:24 <tyll> So the idea is for other groups to meet and decide what rel-eng's priorities should be? AFAICS rel-eng is not part of the stakeholders
17:26:38 <dgilmore> tyll: no
17:26:54 <dgilmore> the idea is to get with the people that need us to deliver things
17:27:07 <dgilmore> then work out what things are the highest priority
17:27:15 <dgilmore> including thinsg releng wanst to do for releng
17:27:39 <dgilmore> its to ensure we are not a silo, and that people have visibility into what we are doing
17:28:17 <dgilmore> if they want something from us they need to convince us that we should do it
17:28:25 <dgilmore> or provide resources to get it done
17:29:04 <tyll> I think I understand it a little now - what is acarter's involvement? Is she coordinating all groups?
17:29:24 <dgilmore> she is helping to keep us on task and organised
17:29:51 <dgilmore> she is helping with the administrative side of things
17:30:08 <tflink> and coordinating some - we've started talking about where QA and releng overlap and if there are places we can help eachother
17:30:22 <dgilmore> right
17:30:46 <tyll> ok, thank you
17:31:40 <tyll> another topic I have is rawhide signing
17:31:53 <dgilmore> one of the goals I am trying to work towards at the moment is unifying the tools and process used to make Fedora and RHEL
17:32:10 <maxamillion> and CentOS where applicable
17:32:15 <maxamillion> basically to not duplicate efforts
17:32:16 <dgilmore> and CentOS
17:32:28 <dgilmore> CentOS is more a down the road target
17:33:51 <tyll> it seems to me that sigul would be currently stable enough to sign rawhide, if batch-size 1 is used
17:33:53 <dgilmore> a big reason I am trying to do that is so that when we need extra development resources to deliver something new, we can pull in extra people from inside of Red Hat to help get things done
17:34:13 <dgilmore> and to help make sure that Fedora is truely upstream for REHL and CentOS
17:34:25 <tyll> I see
17:34:38 <dgilmore> tyll: that uses a different code path that has always been stable
17:34:41 <dgilmore> but is slower
17:35:36 <tyll> btw. is http://people.redhat.com/jgregusk/not-piracy/build-plans-13-paths.svg a URL that is only available within Red Hat?
17:36:00 <tyll> dgilmore: it seems to be fast enough for a daily Rawhide compose
17:36:01 <dgilmore> tyll: people.redhat.com afaik is public
17:37:22 <tyll> dgilmore: ah, ok, maybe I had different problems when trying to acces it recently
17:38:27 <nirik> note that I restarted the vault a few times recently... might be related to rawhide signing
17:38:28 <tyll> regarding the autosigning: I had a simple while loop just singing all unsinged Rawhide packgages running for a while and mostly everything was singed, only when something was built close to the compose start it was missed
17:39:28 <dgilmore> tyll: we have gotten commitment from internal security engineering to fix some of the sigul issues
17:39:48 <dgilmore> so e can try fix the vault and bridge lockups
17:39:55 <dgilmore> and the client hangs
17:41:45 <tyll> awesome :-)
17:41:51 <dgilmore> we need to figure out how to reproduce the issues and get the necessary debugging info
17:42:42 <tyll> iirc sigul already has a lot of debug output if enabled
17:43:52 <dgilmore> not really
17:44:00 <dgilmore> its not very useful?
17:45:25 <tyll> afaik it shows all the data that is exchanged but it is not easy to read manually
17:47:28 <dgilmore> afaik it is only useful for some things
17:47:32 <tyll> since all requests have IDs, it should be at least possible to see at which point the bridge hangs processing requests
17:47:43 <dgilmore> the issues we are hitting are not one of them
17:48:53 <dgilmore> anyway lets finish the meeting if there is nothing else
17:49:01 <tyll> ok
17:49:02 <dgilmore> we can continue this in #fedora-releng
17:49:57 <dgilmore> #endmeeting