15:33:02 <dgilmore> #startmeeting RELENG (2015-05-04)
15:33:02 <zodbot> Meeting started Mon May  4 15:33:02 2015 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:33:02 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:33:09 <dgilmore> #meetingname releng
15:33:09 <zodbot> The meeting name has been set to 'releng'
15:33:10 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion
15:33:10 <zodbot> Current chairs: bochecha dgilmore masta maxamillion nirik pbrobinson pingou sharkcz tyll
15:33:12 <dgilmore> #topic init process
15:33:15 <dgilmore> who all is here
15:33:18 * lmacken 
15:33:22 * threebean 
15:33:22 <nirik> morning
15:33:23 * maxamillion is here
15:33:26 * jkurik is here
15:33:27 <maxamillion> actually, brb
15:34:01 <dgilmore> #topic #6164 bodhi2 status update requested
15:34:03 * pbrobinson is here
15:34:06 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6164
15:34:10 <dgilmore> lets get started
15:34:13 <lmacken> So yeah, regarding Bodhi2 :)
15:34:17 * sharkcz is here
15:34:19 <dgilmore> lmacken: so where are we with bodhi 2
15:34:22 <lmacken> We wrapped up the entire masher port during PyCon recently
15:34:38 <lmacken> and I'm confident that the majority of the push issues have been resolved, as well as the prioritization of "important updates"
15:34:46 <lmacken> there is still a lot we can do in that regard, but it'll definitely help
15:35:01 <dgilmore> okay
15:35:02 <lmacken> At the moment, we're currently blocking on EL7 packages for python-{cryptography,cornice}
15:35:06 <lmacken> https://github.com/fedora-infra/bodhi/issues/142
15:35:17 <lmacken> but once those are built, we should be able to start kicking off staging pushes shortly after
15:35:33 <lmacken> then we can start looking at some of the low-hanging fruit within things like mash
15:35:37 <dgilmore> lmacken: so what is the plan to go live?
15:35:46 <lmacken> like testing out/benchmarking the multilib patch toshio sent out last week
15:36:06 <lmacken> dgilmore: it's hard to say when it'll be in production, but once it's in staging we'll have a much better idea.
15:36:18 <lmacken> I'll be ignoring the frontend stuff until the backend masher is rolling as well
15:36:45 <dgilmore> okay
15:36:53 * nirik sighs at python-six.
15:36:59 <lmacken> but things are looking good, I did a lot of work on the masher and fixing various race conditions recently and the push process should be *much* better :)
15:37:02 <nirik> perhaps we should look at a forward compat version in epel?
15:37:08 <lmacken> yeah we may need to...
15:37:33 <lmacken> so yeah, hopefully we'll be pushing in staging by next week's meeting
15:37:34 <dgilmore> lmacken: does bodhi2 queue up pending things when in teh middle of a push?
15:37:43 <lmacken> dgilmore: yeah
15:37:52 <lmacken> and after the push it goes and looks to see what met the testing requirements during the push
15:37:55 <lmacken> and queues those up for the next
15:37:56 <dgilmore> bodhi1 not doing so causes significant issues
15:38:33 <lmacken> it also has taskotron gating, as well as locking updates in certain states
15:39:06 <lmacken> we also hit a regression in the latest SQLAlchemy that was fixed upstream, but we'll probably need to work around it for EL7
15:39:26 <lmacken> so yeah, more next week :)
15:39:26 <dgilmore> okay
15:39:26 <lmacken> EOF
15:39:43 <dgilmore> okay thanks. Please be sure to keep us in the loop as things move along
15:39:46 <lmacken> will do.
15:40:05 <dgilmore> #topic #6158 Request to discuss Rel-Eng Project Planning Proposal
15:40:07 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6158
15:40:40 <dgilmore> so I am okay with doing this
15:40:51 <dgilmore> we just need to decide what is the best way to do so
15:41:44 * nirik is happy to leave it to the people doing the work. I can adapt to whatever you want.
15:41:58 <dgilmore> I am kinda in the same boat
15:42:15 <pbrobinson> and like everything no doubt it'll evolve as we go
15:42:31 <dgilmore> I would like to have whatever we use be used by more than just releng
15:42:55 <dgilmore> I can see the same thing being very useful for infra, and probably qA also
15:42:58 <dgilmore> QA
15:43:17 <nirik> right.
15:43:20 <threebean> i bet qa would like to not move off of phabricator.  it seems like they put a good bit of work into it.
15:43:23 <dgilmore> so we probably should get a meeting setup with the potential users and find something suitable for all
15:43:31 <dgilmore> threebean: right
15:43:46 <threebean> for infra/apps, I'd be fine moving to whatever.  we'll just be a little slower to get there as we need to rebuild one or two tools to go with the move.
15:43:50 <dgilmore> threebean: but maybe we can move them to a common instance
15:44:14 * nirik prefers pagure to phab... but I can see why they might want to stick with what they have
15:44:23 <dgilmore> maybe we do use something else and leave QA to themselves
15:44:24 <tflink> yeah, it would take quite a bit to get us to move away from phab
15:44:35 <threebean> tflink: understandable.
15:45:02 <dgilmore> nirik: I think for the git side im inclined to go with pagure, and see how it suits us
15:45:07 <dgilmore> but that is the next topic
15:45:31 <dgilmore> so maybe QA does one thing and releng and infra share something
15:46:16 <dgilmore> does anyone have anything specific that they want to bring up?
15:46:20 <dgilmore> maxamillion: ^
15:46:43 <jkurik> do we have any idea where to start with this topic ?
15:47:43 <dgilmore> jkurik: well whatever we use has to be open
15:47:56 <dgilmore> so a start would be to find the open options.
15:48:06 <dgilmore> which we seem to have done mostly
15:48:11 <dgilmore> at least gotten a few options
15:48:15 <jkurik> I mean whether selection of a tool is the task we would like to start with
15:48:56 <lmacken> workflow, then tool, as opposed to tool then workflow?
15:49:13 <dgilmore> jkurik: well we either select a tool and make sure we can live with its workflow, or we pick a workflow and fine the tools that implement it
15:49:27 <dgilmore> s/fine/find/
15:49:37 <jkurik> :)
15:49:52 <dgilmore> to date the discussion has been around a small number of tools
15:50:01 <maxamillion> dgilmore: sorry, was afk ... reading backscroll
15:50:02 <dgilmore> each that has a different workflow
15:52:42 * lmacken is a fan of the git-flow, and pagure having seperate git repos for issues and the wiki gives people the ability to never leave the cli
15:52:57 <maxamillion> there wasn't anything specific I wanted to bring up, no ... I agree that meeting with the other groups and trying to come to a consensus cross-team would be good
15:53:50 <dgilmore> lmacken: not sure that today it really supports the concept of project planning and task management though
15:54:02 <dgilmore> lmacken: but maybe we can extend it to do so
15:54:16 <maxamillion> I would like to vote for pagure though as my top choice, phab isn't a distant second though (for whatever my vote in that space is worth)
15:54:21 <lmacken> not sure... seems like issues+tags = milestones?
15:54:51 <dgilmore> maybe it just needs some thought on how we want to work and manage things
15:55:03 <dgilmore> and will not take too much to make it workable
15:55:15 <dgilmore> one thing I really want to avoid is a huge management overhead
15:55:47 <dgilmore> I feel if we make it too hard to keep track of projects and the workflow management we will just ignore it
15:55:51 <jkurik> the idea of using kanban-like workflow has quite low management overhead
15:56:48 <dgilmore> yeah
15:57:32 <maxamillion> jkurik: +1
15:57:50 <dgilmore> if we can do it from the cli I know I would be happy
15:58:17 <dgilmore> anyway
15:58:22 <dgilmore> lets move on
15:58:26 <maxamillion> that was one thing I really liked about the kanban workflow on my old team is it was really low overhead compared to other stuff we'd done in the past
15:58:42 <dgilmore> #action setup a meeting with interested parties to try come up with a common tooling
15:59:00 <pbrobinson> I like the idea of low overhead
15:59:00 <dgilmore> #topic #6159 Request to discuss Git workflow for rel-eng tooling
15:59:06 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6159
15:59:40 <dgilmore> so I would like to make some changes to how we commit to the releng git repos
15:59:49 <dgilmore> I would like to have all patches signed off
16:00:00 <dgilmore> which is just add -s when commiting
16:00:51 <dgilmore> #info all pacthes need to be signed off by adding -s when commiting, or --signoff when doing git-send-email
16:01:07 <threebean> so, I'm confused as to what value that adds. can someone explain?
16:01:07 <dgilmore> I would like to have all patches reviewed by someone
16:01:39 <pingou> I understand the added value for projects with outside contributors, but in our case we have the FPCA
16:01:51 <dgilmore> threebean: it just says that you actually wrote the code
16:01:51 <pingou> patch review++ :)
16:02:18 <pingou> dgilmore: if we were gpg signing the commits that would
16:02:22 <threebean> ok.  I guess I don't see the difference from the 'author' field.
16:02:22 <dgilmore> pingou: we do have contributions from outside though
16:02:34 <pingou> dgilmore: otherwise anyone can fake it just like for the author field
16:02:57 <dgilmore> pingou: not saying it is perfect
16:03:04 <pingou> ok :)
16:03:14 <maxamillion> can that process be documented somewhere as a SOP? I'm bad and don't have any experience with git-send-email ... most of my exposure to git is a side effect of github
16:03:21 <dgilmore> pingou: some of the upstream projects I work on require it, and I do not think it hurts
16:03:31 <masta> threebean: it opens the door to a process where there are signed off (written by) and reviewed-by lines in the git record. Not everything committed/pushed into a git tree was authored by the person doing the committing/pushing.
16:04:04 <pingou> dgilmore: agreed on the doesn't hurt
16:04:43 <threebean> masta: ack.  (there's already a distinction between committer and author in the metadata.. but not for reviewer.  so, ok..)
16:04:49 <dgilmore> pingou: It may not add a huge amount of value but some
16:05:10 * nirik would love to someday sign commits.
16:05:30 <threebean> nirik: you mean cryptographically?
16:05:35 <masta> nirik: that would be neat, but not everyone uses (or wants) pgp keys
16:05:36 <dgilmore> #action dgilmore to write up a page like http://www.denx.de/wiki/U-Boot/Patches
16:05:37 <tyll> can we start with this requirement for just the core scripts in the releng git repo? It sounds to me like a lot of extra effort for some small scripts that are not used for mission-critical tasks,
16:05:47 <nirik> threebean: yes. gpg
16:05:48 <tyll> e.g. the check for orphan packages
16:06:02 <dgilmore> tyll: no. its not hard to do
16:06:39 <dgilmore> tyll: it also extends to pungi, mash, and the other tools we manage
16:06:54 <pbrobinson> I don't see that adding -s to a command is a lot of extra effort and it should really be all or nothing
16:06:55 <pingou> note: it can be automated on a per repo basis via the .git/config
16:07:21 <dgilmore> code review can only help
16:07:31 <dgilmore> there is quite a few of us that can do the review
16:08:07 <pingou> format.signoff
16:08:25 <pingou> (for format-patch)
16:09:49 <dgilmore> as to the git repo hosting. I think I would like to move the git repos from fedorahosted to being hosted in a pagure.  ideally under a fedoraproject.org domain name
16:10:22 <dgilmore> but if it is run by fedora but under pagure.org or pagure.io I would not object
16:10:34 <dgilmore> it would also allow for easy code review
16:10:46 <masta> dgilmore: you must really like pagure. Mind giving a brief rational for the switch?
16:10:54 <dgilmore> it would be a pretty big change in the current git workflows
16:11:06 <dgilmore> http://pagure.dev.fedoraproject.org/
16:11:13 <dgilmore> is the dev instance of pagure
16:11:56 <dgilmore> there is two git repos per project.  one is for docs and one for code
16:12:06 <lmacken> and issues
16:12:09 <pingou> (there are 4)
16:12:16 <pingou> code, docs, tickets, pull-requests
16:12:19 <dgilmore> so hopefully having docs ina  git repo that is easily changeable we do better at keeping docs up to date
16:12:27 <nirik> I think it has a lot of promise, and additionally we know who works on it, so we can get them to prioritize things we need. :)
16:12:37 <dgilmore> while I have not ever used the pull request model, a lot of people seem to really like it
16:12:42 <dgilmore> it allows for code review
16:12:53 <tyll_> Sorry, I lost my Internet link - what are the semantics of "-s" now - does it need to be added by the one doing the patch review before commiting or by the patch submitter before submitting? Is directly commiting still allowed?
16:13:06 <dgilmore> pingou: ahh cool
16:13:19 <dgilmore> tyll_: by the person writing the code
16:13:49 <dgilmore> tyll_: direct commiting right now is still allowed. but as we move to having things reviewed. we will not allow direct commiting
16:14:08 <dgilmore> but we need to be on top of and make sure code is quickly reviewed
16:14:32 <dgilmore> no direct commiting will apply to everyone. including me
16:15:09 <tyll_> and the "-s" just means that one is the author of the patch?
16:15:57 <dgilmore> masta: so my main reasons is for wanting pagure for git repo hosting comes down, to it supports a workflow a lot of people like. the developer has our needs in mind. it allows us to do code review.
16:16:03 <dgilmore> tyll_: yes
16:16:30 <dgilmore> tyll_: http://www.denx.de/wiki/view/U-Boot/Patches#Review_Process_Git_Tags
16:16:39 <dgilmore> I plan to write a page like that
16:16:44 <dgilmore> that explains things
16:17:07 <maxamillion> +1
16:17:12 <maxamillion> that would be awesome
16:17:35 <masta> dgilmore: +1 sign-off
16:17:44 <masta> dgilmore: +1 pagure
16:18:01 <dgilmore> we may need to add some support to pagure to doing things like adding Reviewed-by and other flags
16:18:38 <maxamillion> +1 pagure, but I'm also open to discussing phab with QA ... if we can all agree on a single toolset I think that has it's own set of advantages
16:19:15 <tyll_> dgilmore: thx, I always assumed the signed-off lines were added by patch reviewers
16:19:47 <dgilmore> tyll_: np
16:20:30 <dgilmore> #action dgilmore to investigate and look at a migration path and plan to pagure
16:20:45 <dgilmore> will discuss things with qa also
16:20:59 <dgilmore> #topic Secondary Architectures updates
16:21:00 <dgilmore> #topic Secondary Architectures update - ppc
16:21:07 <pbrobinson> Beta is out
16:21:08 <dgilmore> pbrobinson: how is ppc?
16:21:11 <dgilmore> woohoo
16:21:16 <pbrobinson> rest it looking pretty good actually
16:21:24 <dgilmore> that is good
16:21:37 <dgilmore> did you get far with getting images etc working?
16:21:38 <pbrobinson> spent some time yesterday cleaning up some packages across both ppc/aarch64 (and ARMv7 actually)
16:21:49 <pbrobinson> well....
16:21:53 <pbrobinson> we're getting there
16:22:03 <pbrobinson> it's crashing still but slowly getting there
16:22:09 <dgilmore> that is something
16:22:10 <pbrobinson> will be debugging it some more shortly
16:22:27 <dgilmore> cool
16:22:28 <sharkcz> also pbrobinson is making progress on the infrastructure
16:22:34 <dgilmore> anything you want to add on ppc?
16:22:42 <pbrobinson> not from me
16:22:47 <dgilmore> #topic Secondary Architectures update - s390
16:22:52 <dgilmore> sharkcz: how is s390?
16:22:57 <sharkcz> beta is out :-)
16:23:08 <dgilmore> w00t
16:23:21 <dgilmore> when did it go out?
16:23:26 <sharkcz> and I have submitted productization patches for lorax fopr s390, new lorax is in f22 so GA will be productized
16:23:34 <sharkcz> I've sent it today
16:23:48 <sharkcz> but wa son mirrors sicne Thu
16:23:56 <sharkcz> was on, since
16:24:00 <dgilmore> sharkcz: we can only release Tuesday through Thursday
16:24:14 <dgilmore> sharkcz: we have a policy on that
16:24:20 <dgilmore> we as in Fedora
16:24:54 <sharkcz> ok, will wait next time
16:24:58 <dgilmore> https://fedorahosted.org/fesco/ticket/974
16:25:25 <dgilmore> FESCo has said they can only be Tuesday, Wednesday or Thursday.  before that it was Tuesday only
16:25:50 <dgilmore> Secondary arches need to be careful to follow policy
16:26:10 <dgilmore> any thing else for s390?
16:26:32 <sharkcz> nope, it looks good overall
16:27:04 <dgilmore> okay
16:27:06 <dgilmore> #topic Secondary Architectures update - arm
16:27:19 <dgilmore> pbrobinson: pwhalen: hows aarch64?
16:27:20 <pbrobinson> Beta out
16:27:27 <pbrobinson> basically the same status as PPC
16:27:31 <dgilmore> cool
16:27:39 <pbrobinson> same issues with the cloud images and a similar stastus
16:27:40 <dgilmore> ghc settled down?
16:27:41 <pbrobinson> status
16:27:45 <pbrobinson> HAHAHAHA
16:27:48 <pbrobinson> nope!
16:28:07 <dgilmore> fooey
16:28:13 <pbrobinson> I've given up on it for F-22 and I'm working with the maintainer to ensure a better experience for all in F-23
16:28:21 <dgilmore> cool
16:28:29 <pbrobinson> the basics are there so it vaguely apparently works
16:28:33 <maxamillion> pbrobinson: +1
16:28:49 <dgilmore> anything else?
16:28:50 <pbrobinson> but I personally can't bring myself to care about it anymore, I've wasted way too much time already
16:28:55 <pbrobinson> nope
16:29:00 <dgilmore> #topic Open Floor
16:29:04 <pbrobinson> all looking reasonable there other than ghc
16:29:11 <dgilmore> anyone have anything they want to bring up?
16:29:28 <nirik> just a FYI, there may well be an outage tomorrow night...
16:29:38 <dgilmore> okay
16:29:44 <nirik> we might be moving our storage finally. This would affect everything that has a netapp nfs mount.
16:29:47 <pbrobinson> nirik: care to outline any more details?
16:29:51 <pbrobinson> AH!!! NICE :)
16:30:11 <nirik> yeah, hopefully. I will file a outage ticket/send announcement when I pin down a time.
16:30:31 <nirik> hopefully it shouldn't take too long... but the nfs host will change.
16:30:46 <nirik> so, unmount, wait for them to do a final mirror, then remount with new host
16:30:55 <dgilmore> nirik: cool. please be sure to put an update in #fedora-releng
16:31:01 <nirik> will do.
16:31:03 <sharkcz> nirik: this is primary arches only change now?
16:31:12 <nirik> sharkcz: its everything, so secondary too.
16:31:16 <sharkcz> ah, ok
16:31:21 <nirik> unless you need us to do that different time?
16:31:33 <pbrobinson> nirik: OK, so you'll co-ordinate with all the appropriate people?
16:31:35 <sharkcz> no problem from me, just asking
16:31:43 <nirik> pbrobinson: will try. :)
16:31:57 <nirik> once this is done we should have more space too...
16:32:02 <nirik> so we can grow some volumes.
16:33:06 <pbrobinson> nirik: was more offering to help where I can on secondary stuff
16:33:28 <nirik> oh, thanks. :) yes, that would be appreciated.
16:33:32 <dgilmore> I have one thing to bring up. seems mash is not hardlinking things at the moment.  I plan to spend some time today and try figure it out
16:33:55 <dgilmore> nirik: will need to make sure koji-shadow is stopped beforehand
16:34:06 <dgilmore> nirik: likely need to disable builders also
16:34:20 <dgilmore> make sure there is no builds going on
16:34:56 <nirik> yep.
16:35:13 <nirik> they wanted us to try and make sure things were as 'quiet' as we could before the cutover so the final mirroring was quick
16:35:22 <dgilmore> okay
16:35:45 <dgilmore> any idea how performance will be on the new netapp?
16:36:09 <nirik> no idea. Its a newer 'c-mode' instance...
16:36:17 <dgilmore> okay
16:36:24 <dgilmore> guess we will find out
16:36:30 <nirik> and we should have access to all the disk we actually 'own'
16:36:43 <dgilmore> that would be nice
16:37:06 <nirik> I've asked them a bunch of questions about what we can and can't do config wise on it, but they haven't really answered.
16:37:25 <dgilmore> :( okay
16:37:51 <dgilmore> seems that what we have today does not always perform well
16:38:02 <dgilmore> but that could be due to how we use it
16:38:33 <dgilmore> it would be nice if we can monitor the performance and see if there are things we can adjust to make performance better
16:38:41 <nirik> agreed.
16:38:54 <nirik> they have promised me docs, so I will fwd those out as I get them
16:39:00 <dgilmore> no idea how possible that is
16:39:06 <dgilmore> okay cool
16:39:13 <dgilmore> anything else from anyone?
16:39:18 <dgilmore> we are 9 minutes over
16:40:37 <dgilmore> ill take that as a no
16:40:42 <dgilmore> #endmeeting