15:01:48 <kshlm> #startmeeting Community Meeting 2017-02-15
15:01:48 <zodbot> Meeting started Wed Feb 15 15:01:48 2017 UTC.  The chair is kshlm. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:48 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:01:48 <zodbot> The meeting name has been set to 'community_meeting_2017-02-15'
15:02:10 <kshlm> I'll wait for 3 more minutes for people to show up.
15:03:51 <ndevos> oh, is this the roll call topic already>
15:03:53 <ndevos> ?
15:04:03 * ndevos <- is present
15:04:15 <kshlm> In the meantime, add anything you want to discuss to https://bit.ly/gluster-community-meetings
15:04:48 <kshlm> ndevos, I'd not called for it yet :)
15:05:36 * shyam is here
15:05:42 * amye is here
15:06:04 <nigelb> I'm vaguely here.
15:06:06 <kshlm> Okay, roll call time
15:06:09 <kshlm> #topic Roll call
15:06:29 <ndevos> hi, I'm here!
15:06:31 <nigelb> o~
15:06:44 <kshlm> Welcome to todays meeting everyone, please chime in if you're participating today.
15:07:02 <kshlm> Hi ndevos and nigelb!
15:07:10 <ndevos> hello kshlm _o/
15:07:13 <kshlm> And shyam and amye!
15:07:28 <amye> howdy
15:07:29 * shyam is here (ok just did that again ;) )
15:08:09 <kshlm> Okay let's start
15:08:19 <kshlm> We have 2 topics for discussion.
15:08:45 <kshlm> #topic GlusterFS-3.10 reminder - last week before release
15:08:50 <amye> sweet!
15:09:02 <kshlm> shyam, You're up.
15:09:10 <shyam> Ah ok...
15:09:20 <shyam> So some general reminders (mails going out as well)
15:09:41 <shyam> Please test and report testing feedback on the created github issues
15:10:10 <shyam> Please mark any bugs that should block 3.10 against its tracker (https://bugzilla.redhat.com/show_bug.cgi?id=1416031 )
15:10:25 <shyam> ndevos has been good at reminding people about this, thanks!
15:10:59 <shyam> Please review the release notes and make any chahnges to them (whoever has features delivered this release)
15:11:12 <kshlm> shyam, How has the testing feedback been till now?
15:11:20 <ndevos> shyam: do you know of a github-url that shows all the issues where I am assigned?
15:11:28 <shyam> I have some from bitrot and glusterd
15:11:40 <shyam> ndevos: give me a few minutes and I will get that for you
15:12:01 <shyam> I am doing our some perf tests and upgrade tests, results out this weekend
15:12:13 <shyam> Other than that, nothing more ATM
15:12:22 <kshlm> ndevos, https://github.com/gluster/glusterfs/issues/assigned/nixpanic
15:12:34 <ndevos> shyam: thanks, I'd like to check and probably re-assign some of them to others :)
15:13:02 <kshlm> shyam, Cool. Do you require any more help?
15:13:07 <shyam> RC1 will be tagged when we have the current blockers fixed (hopefully this weekend again), reminder mail going out on pending issues today!
15:13:53 <shyam> Other things look good, fstat is quiet about 3.10, git cherry reports for backports in 3.9/8 to 3.10 looks healthy (finishing up last minute weekly checks on that)
15:14:15 <shyam> Nothing more for 3.10 kshlm TEST! TEST! TEST! is all that is needed :)
15:14:41 <amye> shyam, want me to start firing things out on the social medias about this too?
15:14:47 <kshlm> shyam, I'll do my bit for GlusterD.
15:14:55 <shyam> ndevos: Link to testing issues: http://bit.ly/2kDCR8M (need to add the assigned filter to that)
15:15:14 <kshlm> 3.10 has to be our most methodical and documented releases ever.
15:15:28 <shyam> amye: yes we need to polish the messaging based on the release notes and start the media stuff as well
15:15:37 <kshlm> shyam++ for doing all this
15:15:38 <zodbot> kshlm: Karma for shyam changed to 1 (for the f25 release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:15:57 <amye> shyam, yep. Let us connect offline on that, but if we're at TEST TEST TEST all the things, then it's time.
15:16:10 * kshlm wishes we could track karma per gluster release like zodbot just did for fedora
15:16:11 <shyam> yes, agree
15:16:29 <shyam> Oops, yes agree amye
15:16:32 <amye> np
15:16:36 <ndevos> shyam: btw, how do we give feedback? Should we add a label like 'VERIFIED' or something?
15:16:59 <shyam> Add comments to the issue, on what was tested etc. and then close the issue if you are done with testing the same
15:17:20 <ndevos> ok, just closing works too
15:17:31 <kshlm> And about blockers?
15:17:43 <shyam> Here is a good example: https://github.com/gluster/glusterfs/issues/90
15:17:48 <ndevos> blockers are bugs right, not github issues
15:18:03 <shyam> yes, blockers are bugs, tracked against the BZ tracker
15:18:17 <shyam> (for this release, and hopefully for the last time ;) )
15:18:19 <kshlm> Cool!
15:18:55 <ndevos> (yes, we'll need to have the tools and scripts updated to work with github too)
15:18:56 <shyam> Any more github issues queries, would be good to address those, so that there is little confusion there...
15:20:22 <shyam> ok I am done with 3.10 related call to actions... thanks
15:20:22 <ndevos> yes, add those to the bug triage guidelines/process doc - uhm, and probably revisit the whole process for github?
15:20:43 <kshlm> shyam, Thanks.
15:20:45 <kshlm> Moving on.
15:20:55 <kshlm> #topic GlusterFS-3.8.9 packaging problems
15:21:05 <kshlm> ndevos, I guess this was you.
15:21:09 <ndevos> yeah :)
15:21:30 <ndevos> kkeithle is on holidays and has instructed multiple volunteers for packaging on several occasions
15:21:47 <ndevos> unfortunately, nobody picks up the packaging work when he is out
15:22:21 <ndevos> 3.8.9 was tagged on monday, and I've done the CentOS Storage SIG builds, others are still not there yet
15:22:37 <kshlm> Is the process documented somewhere? If so someone could give it a try.
15:22:51 <kshlm> (if they can get the right access to the builders)
15:22:58 <ndevos> I think so (no idea where), but you also need the right permissions
15:23:30 <kshlm> nigelb, You've spoken to kkeithle about this haven't you.
15:23:32 <ndevos> I can do the Fedora builds part of the distribution, and will probably do them tomorrow
15:23:41 <kshlm> Do you any suggestions?
15:23:43 <ndevos> teh others for download.gluster.org need to be done by others
15:24:12 <nigelb> I have. I don't have any suggestions at the moment.
15:24:28 <shyam> Just saying: we need this automated, I faced similar when RC0 for 3.10 was released
15:24:36 <nigelb> I have ideas, but they're not in a stage where it will be useful for the current problem.
15:24:42 <kshlm> nigelb, Do you know of any documentation for this?
15:24:49 <kshlm> That will be helpful.
15:25:17 <ndevos> unfortunately not everything can be automated, most distributions require a person to do the actual commits to their builders
15:25:23 <kshlm> Also, you could probably provide the permissions required for the volunteer.
15:25:45 <nigelb> Nope, sorry.
15:26:05 <nigelb> The permissions are not at our level.
15:26:08 <nigelb> They're at the distro level.
15:26:12 <ndevos> all volunteers should be subscribed to the packaging@gluster.org list, but there are also others subscribed...
15:26:50 <nigelb> And the gpg signing key for gluster, is rightfully private (even I don't actually have access per se).
15:26:52 <kshlm> For download.gluster.org?
15:27:12 <nigelb> Ah, download.gluster.org, I'm happy to coordinate access. The problem is signing key.
15:27:18 <kshlm> The distro level stuff that kkeithle did was for fedora, ubuntu ppas and opensuse build service.
15:27:42 <nigelb> You know what, I'll happily take an action item to document the process.
15:27:51 <nigelb> I want to automate this, so it's useful for me to document this anyway.
15:28:06 <kshlm> nigelb, as you wish :)
15:28:37 <kshlm> #action nigelb will document kkeithleys build process for glusterfs packages
15:28:48 <ndevos> that sounds like a very useful step :)
15:28:51 <amye> That works. It doesn't sound like that's a transparent process these days.
15:29:07 <nigelb> Part of it needs to stay private. Part of it will be public.
15:29:18 <ndevos> but, I wont wait for it before announcing glusterfs-3.8.9, and users of non CentOS distributions will have to wait a little longer
15:29:21 <kshlm> Now that we've established we're basically useless without kkeithle around to build packages, what do we do with 3.8.9?
15:29:36 * amye facepalms
15:30:04 <amye> When's kkeithle back?
15:30:21 <shyam> Coming Monday (if I remember right)
15:30:51 <nigelb> ^
15:30:59 <nigelb> I suggest waiting.
15:31:00 <amye> What I'm hearing is that we're blocked until Monday on this, we should document that this is a blocker and we should work to assign responsibility for 'backup packager person'.
15:31:01 <ndevos> and by then there shouls also a 3.9 update :)
15:31:13 <nigelb> it's only a few days.
15:31:27 <nigelb> But also +1 to backup packaging person.
15:31:27 <ndevos> ...hmm, I think my keyboard is dropping letters and words
15:31:30 <amye> Asking people clearly isn't working, so we should work to fix this more directly by March's release cycle.
15:32:21 <kshlm> +1 to get a better fix for this problem
15:32:34 <ndevos> we should probably figure out who signed up for Kalebs packaging sessions, and get those engineers to commit to working on it
15:32:39 <kshlm> So we all agree to wait till kkeithle returns?
15:32:44 <nigelb> that was me and misc, ndevos
15:32:47 <amye> I'll work with nigelb to get the documentation together and we'll put out a call on Gluster-infra to make this more transparent.
15:32:58 <kshlm> ndevos, I signed up as well, but never attended any.
15:32:59 <ndevos> nigelb: not only, kaleb gave multiple sessions
15:33:11 <nigelb> My long term solution: Having d.g.o process fully automated.
15:33:15 <ndevos> kshlm: well, s/signed up for/attended/
15:33:39 <kshlm> I should have some emails on the process, may be I can dig it up.
15:33:41 <amye> Short term solution - let's have 3-4 people who are available for packaging that can rotate through.
15:33:45 <nigelb> Part of this work is the nightly pipeline that I was working on. My time has been taken over by glusto lately.
15:34:00 <ndevos> of course, the automation should be done at one point, and we can even do it for the CentOS Storage SIG packages with a special bot account if we want to
15:34:04 <nigelb> Eventually, I want d.g.o process to be one click.
15:34:49 <nigelb> That's less manual work for Kaleb.
15:35:24 <ndevos> nigelb: yes, every single distribution build would already reduce work, its just difficult to get bot accounts everywhere
15:36:06 <ndevos> unless we want to host our own builders, but I guess that will be waaaay to much work to maintain in our infra
15:36:18 <amye> Other ridiculous idea: the release managers also get trained on packaging?
15:36:32 <amye> If you are a release manager, you can package for other versions and maybe your own?
15:36:39 <misc> if the end goal is to have the package in the distribution, our own builder do not help
15:37:19 <ndevos> misc: that is the end goal, but we provide different supported glusterfs versions, most distributions allow a single version only
15:37:34 <kshlm> amye, ndevos is already partially there.
15:37:49 <amye> kshlm, that's what made me think of it.
15:38:14 <kshlm> But that would be a lot of work for a single release-maintainer to do all the packaging stuff.
15:38:17 <ndevos> amye: doing it for all distributions needs quite some knowledge about all of their packaging systems :-(
15:38:17 <misc> ndevos: yup, because having too much supported version is too complicated and increase workload
15:38:32 <amye> ndevos, and it means more documentation work as well :)
15:38:48 <nigelb> Don't we just support 3 distros? Centos/Fedora/Ubuntu?
15:38:52 <ndevos> misc: we have a list of which versions are in which distributions - http://gluster.readthedocs.io/en/latest/Install-Guide/Community_Packages/
15:38:58 <nigelb> *support with packages
15:39:08 <ndevos> nigelb: and suse
15:39:19 <ndevos> nigelb: and maybe the real debian too
15:39:23 <nigelb> Ah, SuSE.
15:39:23 <ivan_rossi> nigelb: and debian
15:39:31 <kshlm> Before I forget
15:39:37 <nigelb> Noted.
15:39:42 <kshlm> #agreed delay 3.8.9 till kkeithle returns on monday
15:39:59 <amye> With our apologies to kkeithle.
15:40:00 <misc> ndevos: my eyes bleed at that ascii table
15:40:14 <misc> (hopefully, I have a red tshirt, so that's not gonna be too weird)
15:40:24 <ndevos> kshlm: do you mean delaying the release announcement of 3.8.9, or only the packaging for 'kalebs' distributions?
15:40:57 <ndevos> misc: well, yes, it took me some time to make sense of the table too...
15:41:09 <kshlm> ndevos, The announcement. I assume that's what everyone agreed to.
15:41:44 <ndevos> kshlm: oh, I would have done the announcement later today... the CentOS packages have been community tested with emails on gluster-users
15:42:07 <shyam> ndevos: So for 3.10 as well the CentOS packages will get community tested mails?
15:42:10 <ndevos> so, users are aware that the release is available, somewhat
15:42:12 <kshlm> That's fine by me as well.
15:42:55 <ndevos> shyam: I've requested the build environment for 3.10 at the centos rel-eng team, I think I Need to poke them now they are back from travelling
15:42:57 <kshlm> So we're gonna announce 3.8.9 today, and announce the rest of the packages when the get built next week?
15:43:27 <misc> ndevos: in the end, I think that's not exactly what the target demographic want to know (if I can speak myself as part of the said demographic)
15:43:28 <ndevos> shyam: it is up to the community members to start testing them, many are only interested in the version they actively use
15:43:28 <shyam> ndevos: thanks!
15:43:42 <misc> there is no explanation on what is ltm/stm on that page
15:44:02 <misc> that list where to find the package, and sometime the distribution
15:44:27 <ndevos> misc: we're *bad* at writing text, LTM/STM is documented on https://www.gluster.org/community/release-schedule/ and that should have been linked
15:44:40 <misc> (ie, if centos storage sig is the source of the package, Fedora is the target distro)
15:45:05 <misc> and there is no link to instruction on how to use them :/
15:45:30 <misc> ndevos: I still think we should make a hack day on the doc or something
15:45:44 <misc> I am sure if we can do that in Malta, people would be more motivated
15:46:14 <ndevos> oh, I've never been to Malta, happy to go an write docs there
15:46:53 <amye> Malta++
15:47:05 <ndevos> kshlm: I think we're done with the 3.8.9 topic ;-)
15:47:17 <amye> if we did want to do a hack day, when is the best time?
15:47:30 <misc> summer time :) ?
15:47:33 <nigelb> you know that time when we have outage schduled?
15:47:39 <nigelb> that's a good time for a docs hackday ;)
15:48:15 <amye> nigelb, I think you just stole my thunder. That was what I was thinking.
15:48:35 <ndevos> I think the best time differs for many of us, we need a list of items that need addressing
15:49:11 <ndevos> and outage time is not suitable for misc and nigelb, they will be busy elsewhere
15:49:12 <kshlm> I see one more topic.
15:49:25 <amye> Let's table this as we don't have a ton of time left and I'm sure.. yes that. Another item.
15:49:26 <kshlm> Let's give some time for it.
15:49:36 * ndevos also needs to catch his train in 10 minutes
15:49:45 <misc> ndevos: depend, I am gonna just wait on people moving stuff around
15:49:45 <kshlm> #topic Infra outage in March
15:50:05 <nigelb> The community cage is undergoing an outage in March.
15:50:14 <amye> are we still on March 14th?
15:50:14 <misc> (unless we push it)
15:50:23 <misc> amye: I suspect, yes
15:50:27 <misc> at least for the network
15:50:39 <nigelb> The entire rack of servers are being moved from one DC to another.
15:51:08 <nigelb> The outage window is for 72h, but we're hoping that it gets fixed in less than 24h.
15:51:43 <misc> well, there is 10 racks of equipement to move and to plug, so 12h should be enough, depend how many people are working on it
15:51:56 <amye> Is there anything planned for the week of March 14th from our side that we haven't already discussed?
15:52:06 <kshlm> nigelb, misc, what all would we be losing in that time?
15:52:15 <nigelb> We'll lose gerrit and jenkins.
15:52:22 <shyam> 3.10 *should* be out by then ...
15:52:40 <shyam> It may come in between our bugzilla to github move... but that should be ok
15:53:03 <kshlm> amye, there would be a 3.8 release happening around the time
15:53:06 <nigelb> (Centos CI will also be down)
15:53:13 <kshlm> nigelb, What about d.g.o?
15:53:13 <vbellur> I think we should announce this on both -users and -devel so that everybody can plan around the outage
15:53:19 <nigelb> d.g.o will be up.
15:53:23 <misc> kshlm: d.g.o is ok
15:53:26 <amye> kshlm, that's what I was afraid of
15:53:32 <nigelb> That's hosted on rackspace (no plans to move it at the current time)
15:53:42 <nigelb> We will once we have final confirmation.
15:53:53 <misc> nigelb: well, one da, I will have it in HA in rackspace and the cage
15:54:10 <misc> but the list of servers/VM in the cage: https://github.com/gluster/gluster.org_ansible_configuration/blob/master/hosts#L27
15:54:45 <ndevos> I should be able to get the 3.8.10 release out before the outage... (but will be travelling, and that may be causing difficulties)
15:55:32 <nigelb> That's pretty much it about the outage.
15:55:50 <kshlm> How long do you think it'll before we have a confirmed time?
15:55:59 <amye> We've got a cage meeting tomorrow
15:56:08 <misc> well, the outage was supposed to be done in 2 part
15:56:21 <misc> then during planning, it was proposed to do all at once
15:56:25 <nigelb> we should know tomorrow at the meeting.
15:56:41 <misc> now, i sent a email to say "pls stick to the plan kthxbye" :)
15:57:14 * ndevos drops off to catch a train, will be back online later
15:57:18 <kshlm> Cool. In any case, the earliest will be 14 March. We should have enough time to plan for things.
15:57:23 <amye> Correct.
15:57:32 <kshlm> Alright.
15:57:36 <misc> in a ideal world, we would had a redundant gerrit setup
15:57:45 <misc> and redundant jenkins too
15:57:53 <misc> but that's not doable in 1 month
15:58:47 <kshlm> That's it for today.
15:58:54 <kshlm> I'm ending the meeting now.
15:59:12 <amye> Thank you!
15:59:12 <kshlm> Thanks for attending everyone. I'll be sending out the meeting notes tomorrow.
15:59:22 <kshlm> Good night!
15:59:28 <kshlm> #endmeeting