12:01:09 <hagarth> #startmeeting
12:01:09 <zodbot> Meeting started Wed May  6 12:01:09 2015 UTC.  The chair is hagarth. Information about MeetBot at http://wiki.debian.org/MeetBot.
12:01:09 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
12:01:18 <hagarth> Let's get rolling. :)
12:01:22 <hagarth> #topic Roll Call
12:01:26 * jimjag is here
12:01:28 <hagarth> who do we have here today?
12:01:31 * spot is here
12:02:00 <hagarth> is everybody else too busy with 3.7.0? :)
12:02:03 * ndevos is here _o/
12:02:08 <bene2> here
12:02:31 <hagarth> looks like that is the case .. let us get started nevertheless
12:02:34 * JustinCl1ft waves
12:02:43 <hagarth> #link agenda - https://public.pad.fsfe.org/p/gluster-community-meetings
12:02:45 * overclk is here
12:02:45 * soumya is here
12:02:46 * krishnan_p _o/
12:02:59 <hagarth> #topic Action items from last meeting
12:03:01 <partner> hep
12:03:01 * raghu is here
12:03:02 * kkeithley1 is here
12:03:14 <hagarth> raghu to create new 3.6.4 tracker bug today
12:03:17 <hagarth> raghu: is this done?
12:03:22 <raghu> yeah. Donew
12:03:38 <raghu> https://bugzilla.redhat.com/show_bug.cgi?id=1216965
12:03:47 <hagarth> raghu: cool, you can also set an alias for that.
12:03:59 <raghu> sure
12:04:05 <hagarth> next AI - raghu to release 3.6.4beta1 by the end of next week
12:04:20 <hagarth> raghu: on track?
12:04:33 <raghu> Yeah. Its on track.
12:04:52 <hagarth> raghu: great, thanks.
12:04:59 <hagarth> next AI - ndevos to do release announcement for 3.5.4beta1 when the packages are ready
12:05:08 <raghu> I would have done that already. But there was a crash for which a patch has been backported by jdarcy. I am waiting for that to get merged
12:05:08 <hagarth> ndevos: ready with this?
12:05:13 <hagarth> raghu: ok
12:05:34 <ndevos> oh, I think the packages have been tested, but hchiramm or lala was going to get back to me about the
12:05:36 <ndevos> m
12:05:47 <hagarth> hchiramm: any updates here?
12:05:55 <ndevos> maybe I've missed their email, or I'm waiting for one
12:06:06 <hchiramm> hagarth, yes
12:06:24 <hchiramm> ndevos, hagarth the packages are available at download.g.org
12:06:36 <hagarth> hchiramm: cool!
12:06:48 <hagarth> ndevos: maybe we should just send out an announce note then?
12:06:58 <ndevos> hagarth: yes, I'll get that done
12:07:09 <hagarth> ndevos: great, thanks.
12:07:22 <hagarth> next AI - overclk will review the Overlay feature page by next meeting or Gluster Summit
12:07:34 <hagarth> overclk: sounds like next week - is that right?
12:07:53 <overclk> yeh
12:08:00 <hagarth> overclk: ok
12:08:03 <hagarth> next AI - JustinClift to have basic forge v2 operational by Gluster Summit
12:08:06 <overclk> during summit :)
12:08:11 <hagarth> overclk: ok :)
12:08:18 <overclk> hagarth, not only review but will talk about it :)
12:08:26 <hagarth> overclk: that would be fantastic!
12:08:58 <hagarth> JustinCl1ft: any updates here?
12:09:01 <JustinCl1ft> hagarth: Working on it ;)
12:09:17 <JustinCl1ft> Some repos have been mirrored to GitHub, and stats collector updated
12:09:18 <hagarth> JustinCl1ft: cool, is there a drop dead date for us to move off forge?
12:09:20 <JustinCl1ft> But, that's about it.
12:09:27 <JustinCl1ft> No, there isn't.  :)
12:09:48 <hagarth> JustinCl1ft: maybe we should have a deadline for this :)
12:09:55 <JustinCl1ft> Meh
12:10:07 * tigert is in
12:10:17 <hagarth> one of the issues that kshlm noticed was gerrit's async mirroring to both github & forge was causing load spikes
12:10:23 <hagarth> but more on that in the infra part of the meeting
12:10:31 <hagarth> next AI - JustinClift to prod the alternate weekly meeting times discussion back to life
12:10:31 <JustinCl1ft> Interesting
12:10:36 <JustinCl1ft> Haven't done it
12:10:43 <JustinCl1ft> Anyone else want to pick it up?
12:10:46 <JustinCl1ft> overclk ?
12:11:01 <hagarth> JustinCl1ft: ok, carrying forward this AI
12:11:19 <hagarth> JustinCl1ft: maybe have a discussion on this next week at BCN?
12:11:28 <JustinCl1ft> BCN ?
12:11:28 * kshlm is late
12:11:33 <hagarth> Barcelona
12:11:55 * JustinCl1ft nods
12:12:14 <hagarth> #action JustinCl1ft to hold a discussion on alternate weekly meeting times in Barcelona
12:12:18 <overclk> JustinCl1ft, I'll take a pass on that. anyway we'll discuss in BCN..
12:12:22 <hagarth> next AI - JustinClift to reach out to likely testers when 3.7.0beta1 RPMs are available
12:12:36 <hagarth> JustinCl1ft: any luck here?
12:12:43 <JustinCl1ft> Haven't done that.  The main tester I was goingt o reach out to had issued with 3.6.3 upgrade
12:12:49 <JustinCl1ft> (yesterday)
12:12:55 <JustinCl1ft> So, I didn't want to push it ;)
12:13:03 <hagarth> JustinCl1ft: yes, looks like we will not get much testing pre 3.7.0 GA
12:13:15 <JustinCl1ft> ???
12:13:30 <JustinCl1ft> Why the hard cut-off-date for GA, instead of doing it after testing?
12:13:33 <hagarth> JustinCl1ft: much (non-developer) testing before GA
12:13:41 <JustinCl1ft> What happened to "when it's ready" ?
12:13:56 <hagarth> JustinCl1ft: that has not been our strategy
12:14:01 <bene2> perf is testing it, Ben Turner is testing it
12:14:14 <JustinCl1ft> hagarth: I know.  See the state of the regression tests?  That's the result
12:14:15 <hagarth> we decided to move to a time based schedule, rather than a content based one a few releases back.
12:14:32 <JustinCl1ft> Can we change this?  Please.
12:14:44 <hagarth> JustinCl1ft: the regression tests are a lot better atm.
12:14:53 <hagarth> JustinCl1ft: another sound topic for Barcelona/post 3.7.0.
12:15:01 * JustinCl1ft grumbles
12:15:14 <ndevos> timebased is good, features that are not ready do not delay other features, and will have to wait for the next release :)
12:15:17 <hagarth> JustinCl1ft: we need to figure out how we can get more community testing before a release
12:15:25 <JustinCl1ft> Btw "we decided to move to a time based schedule, rather than a content based one a few releases back."
12:15:41 <JustinCl1ft> ^^^ We didn't communicate that to our Community did we?
12:15:43 <krishnan_p> We should move to merge window based strategy in the future (4.0?), hopefully improving our quality ahead of release.
12:15:48 <hagarth> JustinCl1ft: we did that
12:15:53 <bene2> do regression tests do anything that would stress Gluster in the slightest?
12:15:59 <ndevos> JustinCl1ft: I think we discussed that mainly in this meeting?
12:16:09 <JustinCl1ft> ndevos: Hmmmm.
12:16:17 <hagarth> JustinCl1ft: I can look up the archives but maybe for later.
12:16:20 <JustinCl1ft> k
12:16:42 <hagarth> bene2: regression tests are not very intensive as far as I can see.
12:17:20 <hagarth> bene2: however regression tests open up interesting latent races which we have been fixing in the recent past.
12:17:26 <hagarth> so it aide the cause of stability.
12:17:33 <jdarcy> bene2: They're not supposed to be *stress* tests.  They exercise a wide range of *functionality* but not all at once.
12:17:41 <hagarth> next AI - jdarcy will post a GlusterFS 4.0 update to the list(s)
12:17:52 <jdarcy> I guess I should do that, huh?
12:18:01 <hagarth> jdarcy: +1 :)
12:18:23 <hagarth> next AI - tigert to announce new Gluster website layout demo to the mailing lists
12:19:04 <tigert> did that, not much response though
12:19:09 <hagarth> tigert: right
12:19:10 <tigert> though we lack content most
12:19:25 <hagarth> tigert: we can discuss more in the website section of the meeting
12:19:35 <ndevos> tigert: what about annoucing planet.gluster.org too?
12:19:46 <tigert> let me take that as an action item
12:19:52 <ndevos> and include the rss urls for people that like to add it to their readers?
12:20:10 <tigert> yeah
12:20:16 <bene2> jdarcy, I'm not talking wide range of functionality.  Just more than 1 thread, 1 client or 1 server at a time, simple tests.
12:20:33 <ndevos> #action tigert will announce planet.gluster.org and include the rss urls too
12:20:37 <tigert> I need to dig it up, I don't do much rss myself (though won't the readers be smart and figure those out themselves?)
12:20:39 <jdarcy> bene2: Patches welcome.
12:21:10 <tigert> http://glusternew-tigert.rhcloud.com/ has our documentation linked now btw
12:21:12 <hagarth> bene2: a few tests run on multiple servers .. in addition we are looking to overhaul our upstream testing to have more categories of tests.
12:21:24 <JustinCl1ft> bene2: Some of the tests start up several nodes on the same host for testing interaction stuff.  It's for bug testing more than anything though.
12:21:29 <ndevos> tigert: I dont know, maybe checking if there is an rss feed it enough
12:21:38 <hagarth> next AI - spot to bring up the Branding? question for it in response
12:21:40 <tigert> ndevos: yeah I think it has, I'll check
12:21:50 <hagarth> spot: I believe this is done?
12:22:05 <spot> hagarth: should be, i'll double check to make sure.
12:22:12 * spot has been juggling a lot lately
12:22:18 <hagarth> spot: sure
12:22:33 <hagarth> btw I like software {re}defined storage too :)
12:22:43 <jdarcy> bene2: There's also a question of how much we want to slow down our *pre commit* check.  Obviously we should do real stress/longevity tests before a *release*, but before every commit?
12:22:57 <bene2> agreed, don't want to do this every commit
12:23:38 <hagarth> jdarcy: our nightlies do cover multiple servers and clients .. but we don't have coverage for all possibilities with gluster today.
12:23:38 <bene2> but better to do it upstream in an automated fashion than downstream when it's too late
12:24:07 <hagarth> that ends our list of AIs from the previous meeting
12:24:15 <hagarth> moving on
12:24:17 <ndevos> bene2, jdarcy: it would be amazing if we have an easy to run automated test, we can schedule that in Jenkins and include qemu/samba/ganesha things there too
12:24:18 <jdarcy> bene2: Absolutely.  We should define that pre-release testing process, preferably (IMO) in the form of a script.
12:24:28 <hagarth> #topic Gluster 3.7
12:24:30 <hagarth> jdarcy: +1
12:24:48 <JustinCl1ft> Sounds like we need a "testing" topic for the open floor
12:24:50 <hagarth> 3.7.0 is all set to be released tomorrow
12:24:56 <hagarth> JustinCl1ft: please add that
12:25:08 <hagarth> hchiramm, raghu et al have been helping with release notes
12:25:09 <kkeithley> yes, +1 for testing during open floor
12:25:13 <jdarcy> JustinCl1ft: Sorry for helping to sidetrack.
12:25:14 <JustinCl1ft> Done
12:25:18 <ndevos> hagarth: no beta2 for added patches?
12:25:32 <Joe_f> hagarth: tiering got couple of patches to backport .. we are doing it ... will send a list of them later for merging ;)
12:25:32 <kkeithley> don't derail the juggernaut. ;-)
12:25:33 <hagarth> ndevos: no, since we are not getting any feedback
12:25:48 <hagarth> Joe_f: sure
12:25:54 <ndevos> hagarth: well, we got some for packaging issues...
12:26:12 <hagarth> ndevos: yes, let us discuss that in the openfloor?
12:26:17 <JustinCl1ft> hagarth: Have we killed all of the regression failures?
12:26:23 <JustinCl1ft> Like, we have 0 now?
12:26:37 <jdarcy> Not even close.
12:26:47 <JustinCl1ft> We agreed to do that.
12:26:48 <hagarth> JustinCl1ft: the regular offenders have been killed .. lot of regression tests are passing now
12:27:07 <JustinCl1ft> We agreed to fix themm all.  Not just "some of them".
12:27:12 <hagarth> JustinCl1ft: yes, but infra issues are not helping
12:27:14 <Nagaprasad> question on 3.7.0.  Is the GA planned on 7th or 8th?
12:27:24 <JustinCl1ft> That's not an excuse
12:27:25 <bene2> is there some sort of report that summarizes?
12:27:29 <jdarcy> Some of the worst offenders are still being skipped in run-tests.sh, and there's a whole rogues' gallery of others that fail less often individually but quite a bit as a group.
12:27:30 <ndevos> hagarth: well, I think there are still some patches that need to get in before 3.7.0 GA...
12:27:43 <hagarth> JustinCl1ft: no, but we can't keep moving the release forever. that's the point I am trying to make.
12:27:57 <hagarth> ndevos: certainly
12:28:00 <JustinCl1ft> No-one is suggesting we move it forever
12:28:12 <JustinCl1ft> I'm saying we stick with what we said we'd do
12:28:32 <ndevos> hagarth: should we not cleanup the tracker before deciding when to GA?
12:28:45 <ndevos> https://bugzilla.redhat.com/showdependencytree.cgi?maxdepth=2&hide_resolved=1&id=glusterfs-3.7.0 lists *many* bugs still in POST :-/
12:28:45 <hagarth> JustinCl1ft: sure, I don't have a problem with that.
12:28:53 <JustinCl1ft> k.
12:29:00 <hagarth> ndevos: yes, unfortunately it is beyond my limited bz skills :-/
12:29:12 <hagarth> and I really need help from bz owners to clean them up.
12:29:27 <ndevos> hagarth: yeah, I understand its not easy to do... but at the moment we have no idea whats pending :-/
12:29:52 <kkeithley> Where's the etherpad with what's pending?
12:29:57 <hagarth> ndevos: I will send one more reminder on the list about that .. there are quite a few assigned to bugs@gluster.org
12:30:19 <kkeithley> Or is the etherpad really a fair representation?
12:30:21 <hagarth> kkeithley: the URL that I sent in a note to gluster-devel 2 days back has it
12:30:24 <ndevos> hagarth: yes, an other reminder would help
12:30:41 <hagarth> kkeithley: https://goo.gl/j8PzVk (new & assigned list)
12:30:41 <ndevos> kkeithley: we have bugzilla for tracking the status of bugs, dont etherpads get oiut of sync?
12:30:42 * kkeithley was hoping someone had the link handy
12:30:49 <hagarth> we can similarly evolve a POST list
12:31:12 <soumya> kkeithley, https://public.pad.fsfe.org/p/pending_gluster_3.7_patches
12:31:15 <kkeithley> yeah, nothing is perfect
12:31:21 <hagarth> so here's my take on regression tests:
12:31:42 <hagarth> 1. we clean up whatever is affecting us before 3.7.0 GA
12:32:09 <hagarth> 2. if a test is listed in the spurious failures etherpad but doesn't happen anymore easily, we don't hold GA for such tests.
12:32:16 <hagarth> does this seem reasonable?
12:32:51 <krishnan_p> what happens to 3.7.0 tracker  bugs?
12:32:57 <hagarth> soumya: at this point I am also inclined to push out patches to 3.7.1 if they are not absolutely essential for 3.7.0
12:33:11 <ndevos> hagarth: once a test failed, other tests are not run, we should not 'overrule' those failed regressions
12:33:18 <hagarth> krishnan_p: move all non-URGENT bugs to 3.7.1 after one more round of cleanup by individual developers
12:33:30 <hagarth> ndevos: noted, haven't been doing that this week.
12:33:39 <ndevos> ok :)
12:33:40 <krishnan_p> hagarth, sounds reasonable. In fact, bugs that are left unattended should be moved out in bulk
12:33:53 <ndevos> hagarth: is there a 3.7.1 tracker already?
12:33:56 <hagarth> krishnan_p: right
12:34:01 <hagarth> ndevos: will need to create one
12:34:02 <jdarcy> Why does the 3.7 tracker depend *directly* on bugs on master?
12:34:19 <bene2> all bugs left unattended?  That seems like the wrong criterion
12:34:29 <ndevos> jdarcy: probably from before the branching of 3.7 from master
12:34:40 <ndevos> s/probably/HOPEFULLY/
12:34:40 <hagarth> jdarcy: we also do not have a 3.7pre tag
12:34:41 <soumya> hagarth, where do we specify the patches which are must for 3.7.0  (example rpc-protocol changes)?
12:35:04 <hagarth> soumya: please add them in the etherpad  :)
12:35:17 <ndevos> soumya: they should be blockers of the 3.7.0 tracker
12:35:28 <krishnan_p> bene2, hmm. What would be the alternative?
12:35:39 <soumya> hagarth, ndevos ..thanks :) already done..i thought they may get pushed out even then to 3.7.1
12:35:40 <hagarth> bene2: we are trying to get better at bug triaging .. not something that we can fix in a short order.
12:36:12 <hagarth> soumya: add a MUST tag to those patches which you absolutely want to see in 3.7.0
12:36:20 <soumya> hagarth, sure
12:36:42 <hagarth> want to touch a bit upon release maintenance of 3.7.x
12:36:57 <bene2> krishnan_p, a bug can be fatal but unattended, that's my point.  The solution here is to prioritize and attend to the bugs that matter most, such as data integrity.
12:37:00 <hagarth> we've had a single release maintainer for our previous release trains
12:37:33 <hagarth> since 3.7.x is a feature packed release train, I propose that all maintainers manage 3.7.x too.
12:37:35 <krishnan_p> bene2, perfect. Its the maintainer/developer responsibilty. Not release maintainer's
12:37:55 <kkeithley> sounds good to me
12:38:22 <krishnan_p> hagarth, hmm. Are we dividing all the responsibilities of a release maintainer to all component maintainers?
12:38:26 <ndevos> krishnan_p: in my opinion the release maintainer should keep an eye on the open bugs for the release he/she maintains
12:38:32 <hagarth> krishnan_p: yes
12:39:00 <krishnan_p> hagarth, It make sense to divide the patch maintainence. But GA coordination and decision making? Too many cooks?
12:39:03 <hagarth> krishnan_p: that always has been the case - if you look up the responsibilites for maintainers
12:39:10 <kkeithley> release maintainer can alway revert changes he or she doesn't approve of
12:39:22 <krishnan_p> hagarth, I am more than happy to chip in. I don't want a dining philosophers problem :)
12:39:29 <kkeithley> if the component maintainer oversteps his or her bounds
12:39:57 <hagarth> krishnan_p: we will announce a schedule and get an ack from all maintainers as per the schedule (i.e. before we release).
12:39:59 <ndevos> oh, but currently component maintainers are not supposed to merge the changes into release-* branches, only review+approve them
12:40:57 <hagarth> for 3.7.x, component maintainers will be merging changes into release-3.7.
12:40:59 * kkeithley thinks, hence this discssion
12:41:06 <krishnan_p> hagarth, by that the 'we' is responsible for coordinating. Isn't that what current release maintainers do, over and above patch maintenance
12:41:10 <kkeithley> *discussion*
12:41:13 <ndevos> is it really much work for a release maintainer to click the "submit" button on completely reviewed/tested patches?
12:41:46 <hagarth> ndevos: I foresee the volume of patches to be a lot more for release-3.7
12:42:17 * krishnan_p wonders if we should try merge windows approach for 3.7.x.
12:42:23 <ndevos> hagarth: I do not see a bottleneck in the release maintainer if component maintainers review and +2 patches in a timely matter
12:42:54 <hagarth> ndevos: I have felt that in the run upto 3.7.0 itself .. hence this change.
12:43:11 <hagarth> krishnan_p: yes, it almost boils down to announcing a schedule for all 3.7.x releases
12:43:27 <hagarth> and getting all patches into release-3.7 during the merge window
12:43:39 <hagarth> anything beyond that gets moved to the next 3.7.x update.
12:43:41 <ndevos> hagarth: yes, I understand, but I think many component maintainers could be more responsove, and review with +2 instead of +1
12:43:50 <ndevos> *responsive even
12:44:01 <hagarth> ndevos: agree there.
12:44:24 <hagarth> ndevos: why don't we try this for release-3.7 and see how this fares? we can always do a course correction if necessary.
12:44:26 <kkeithley> yeah, I don't understand this propensity to give multiple +1s
12:44:28 <krishnan_p> hagarth, I will seek more clarification in the mail that you are plannng to send
12:44:48 <ndevos> hagarth: sure, lets try it out
12:44:50 <krishnan_p> hagarth, my biggest concern is coordination. Probably a detailed plan for your proposal would shed more light
12:45:06 <ndevos> kkeithley: well, multiple +1's would be for a patch that hits different components
12:45:13 <hagarth> krishnan_p: I may not be able to put everything in black & white, but I'll try :).
12:45:29 <ndevos> or, someone that is not a component maintainer should not +2 it, maybe
12:45:32 <krishnan_p> hagarth, It could evolve in the ML
12:45:45 <hagarth> krishnan_p: I will help with larger co-ordination needs.
12:45:49 <kkeithley> ndevos: that would be reasonable, but I see it a lot on patches that only touch a single component.
12:45:50 <hagarth> krishnan_p: sure, let us do that.
12:46:00 <krishnan_p> kkeithley, ndevos all the +1/+2 behaviour is mostly cultural
12:46:09 <hagarth> Nagaprasad: release date contingent on cleaning up our regression test sanity :)
12:46:18 <hagarth> anything more on 3.7.x ?
12:46:41 <Nagaprasad> thanks
12:46:41 <kkeithley> krishnan_p: understood. But it's something we should work on. ;-)
12:46:56 <krishnan_p> kkeithley, I think so too. Old habits die hard :)
12:47:01 <hagarth> figure not, moving on
12:47:06 <hagarth> #topic Gluster 3.6
12:47:06 <ndevos> krishnan_p: sure, and there is a difference between +1 and +2, but component maintainers should be happy to +2 some changes :)
12:47:43 <krishnan_p> ndevos, agree.
12:47:44 <hagarth> raghu: updates here?
12:47:48 <raghu> Not much on release-3.6. I have merged some patches. Found that there was a crash. jdarcy has backported the patch. Its going through regressions. Once it passes, I will make a beta1 of 3.6.4
12:48:36 <hagarth> raghu: I will also send the CLI backport (in response to the issue reported by corvidtech yesterday).
12:49:03 <bene2> bye
12:49:04 <raghu> hagarth: sure. Will wait for it.
12:49:11 <hagarth> #action raghu to release 3.6.4beta1 this week
12:49:18 <hagarth> moving on to 3.5
12:49:22 <hagarth> #topic Gluster 3.5
12:49:40 <ndevos> I'll announce 3.5.4beta1 later today
12:49:49 <hagarth> ndevos: cool
12:50:05 <ndevos> there are some changes that I would like to include as well, so I'll do a beta2 later this week
12:50:16 <hagarth> ndevos: right, ok
12:50:18 <ndevos> or early next week, depending on how busy I am with other tasks
12:50:48 * ndevos does not have anyting else to sat
12:50:50 <ndevos> *say
12:50:51 <hagarth> ndevos: sounds good to me, we can probably do beta2 in Barcelona :)
12:51:01 <hagarth> #topic Gluster 4.0
12:51:08 <hagarth> Jeff - any updates here?
12:52:09 <jdarcy> There was a very productive meeting in BLR last week.
12:52:33 <jdarcy> Losts of ideas about 4.0 features, which we'll surely be discussing in Barcelona next week.
12:52:46 <jdarcy> People seem pretty excited to get started.
12:53:08 <hagarth> jdarcy: yay!
12:53:27 <kshlm> jdarcy, could you share the notes you took down?
12:53:49 <jdarcy> kshlm: Sure.  I'll send to the ML.
12:53:57 <kshlm> I missed the meeting last week, and would like to read up before BCN.
12:54:03 <kshlm> Thanks.
12:54:14 <hagarth> anything more on 4.0?
12:54:21 <jdarcy> Nope.
12:54:35 <hagarth> ok, thanks. Moving on
12:54:42 <hagarth> #topic Open Floor
12:54:54 <hagarth> first one - Infra issues (gerrit etc.)
12:55:08 <hagarth> gerrit has been a laggard .. partly it could be related to iWeb
12:55:20 <hagarth> JustinCl1ft: can we raise a ticket with iWeb about packet losses?
12:55:56 <hagarth> kshlm: any thoughts that you would like to share from your debugging in r.g.o?
12:56:26 <kshlm> There is a problem with the network in their datacenter itself.
12:56:37 <kshlm> Lots of packet loss within their network.
12:56:38 <kkeithley> I've seen one regression test that passed, but the ssh command to set +1Verified had a connection timeout. Related? Am I the only one that's seen this?
12:56:51 <hagarth> kkeithley: nope, I have seen those too.
12:56:51 <JustinCl1ft> hagarth: I'll try.
12:57:19 <JustinCl1ft> hagarth: Last time we had problems they couldn't authenticate me.  But they could see the firewall was having issues, so they rebooted it anyway
12:57:19 <hagarth> JustinCl1ft: thanks, kshlm sent a note on gluster-infra about packet loss yesterday.
12:57:34 <msvbhat> BTW are we planning test day/week this time? Before GA?
12:57:37 <hagarth> JustinCl1ft: I see .. maybe Johnmark can help too.
12:57:39 <msvbhat> hagarth: ^^
12:58:03 <hagarth> msvbhat: no, since we did not have a lot of traction previously, we decided to not have one this time around.
12:58:34 <msvbhat> hagarth: Ah
12:58:41 <JustinCl1ft> kshlm: Just to ask.. are you measuring that from multiple source locations?
12:58:57 <JustinCl1ft> kshlm: Just in case it's something to do with an interaction from your source location
12:59:29 <kshlm> I did.
12:59:34 <JustinCl1ft> Hmmm.... although we have backups for Gerrit happening nightly, I don't think we ever for the Jenkins nightly backup happening
12:59:48 <JustinCl1ft> Might be a good idea to get that happening pretty pronto.  Just in case.
12:59:50 <kshlm> Tested from newgerrit, RH-bengaluru, and my home network
12:59:54 <JustinCl1ft> kshlm: Cool
13:00:03 <kshlm> I also pinged build.g.o and r.g.o
13:00:23 <kshlm> from each other, lots of packet loss even though they're in the same DC.
13:00:33 <JustinCl1ft> hagarth: We can setup Gerrit in Rackspace fairly easily
13:00:35 <hagarth> JustinCl1ft: since the number of patches being sent and users are also more at this time, it seems to cause problems too.
13:00:47 <hagarth> JustinCl1ft: we probably need to move r.g.o to a more beefy server.
13:01:06 <JustinCl1ft> We have new server gear... it has no public ips though
13:01:16 <JustinCl1ft> So, it's pretty much dead in the water until misc gets back
13:01:30 <hagarth> JustinCl1ft: let us discuss this over in BCN too .. far too many things already for there :)
13:01:32 <kshlm> The newer gerrit site seems to keep a lot more connections open to the server. The gerrit server isn't able to repond to all of the requests quickly.
13:01:42 <hagarth> we probably need more days for the summit :D
13:01:43 <kshlm> The network issues on top don't help.
13:01:54 <JustinCl1ft> k
13:01:57 <JustinCl1ft> Yeah
13:02:07 <hagarth> next topic - Doc update
13:02:17 <hagarth> hchiramm: want to provide a quick update here?
13:02:34 <hchiramm> hagarth, yep
13:02:57 <hchiramm> the documentation is getting rendered in readthedocs
13:03:01 <hchiramm> as u can see here
13:03:20 <hchiramm> http://gluster.readthedocs.org/en/latest/
13:03:28 <hchiramm> the plan is to keep only developer docs inside
13:03:33 <hchiramm> gluster source
13:03:43 <hchiramm> and move everything to a seperate github repository
13:03:49 <hchiramm> for easy contribution
13:03:57 <aravindavk> hchiramm: Markdown format or RST?
13:03:59 <ndevos> yeah, sounds like a good approach
13:04:01 <hchiramm> I will send out an email soon about the proposed changes
13:04:07 <hchiramm> aravindavk, markdown
13:04:18 <hagarth> hchiramm: thanks, sounds good to me!
13:04:33 <hagarth> next topic - Summit agenda
13:04:41 <hagarth> spot: anything about $TOPIC?
13:04:45 <spot> Summit agenda is up on the wiki
13:04:57 <spot> http://www.gluster.org/community/documentation/index.php/GlusterSummit2015
13:05:27 <hagarth> spot: cool, thanks!
13:05:44 <spot> All of the speakers have been notified as to their sessions and should be prepared to present!
13:05:50 <hagarth> spot: will there be any room for smaller BoF like discussions?
13:05:59 <hagarth> spot: I don't have any slides ready yet ;)
13:06:02 <spot> We have a second room for that
13:06:07 <hagarth> spot: ok, cool.
13:06:43 <hagarth> moving on..
13:06:57 <hagarth> next topic - https://glusternew-tigert.rhcloud.com
13:07:15 <hagarth> tigert: would you like to provide an update here?
13:07:59 <hagarth> tigert seems to be MIA .. moving on
13:08:14 <hagarth> next topic - Testing
13:08:34 <hagarth> JustinCl1ft, kkeithley: thoughts here?
13:08:38 <ndevos> do we really need to test?
13:08:46 <kkeithley> how about switching the focus from regression to unit tests?
13:08:57 <kshlm> kkeithley, +1
13:09:05 <ndevos> oh, YES, cmocka tests should be required
13:09:06 <hagarth> kkeithley: define unit tests ?
13:09:14 <kkeithley> cmocka-based tests
13:09:28 <hagarth> I wouldn't want to miss regression tests even if we add unit tests.
13:09:44 <kkeithley> no, just a focus shift, not a complete sea change
13:09:50 <ndevos> indeed, regression tests are a must, and unittests should be a must too
13:10:05 <hagarth> ndevos, kkeithley: I am fine with that
13:10:05 <kshlm> Could we atleast reduce the set of regression tests run for every change submitted to gerrit?
13:10:21 <ndevos> add a new function, add a unittest
13:10:32 <hagarth> kshlm: if we can determine what are a good subset of tests to be run, we can do that.
13:10:39 <JustinCl1ft> Stop
13:10:45 * hagarth stops
13:10:51 <ndevos> kshlm: we could run more in ... <stops>
13:10:56 <JustinCl1ft> kshlm: I think you're meaning "can we make regression tests run faster?"
13:11:03 <kshlm> Running the full test suite and waiting for a minimum of 2 hours to get the results for every change is just a waste of time IMO.
13:11:11 <JustinCl1ft> That's really what you're after yeah?
13:11:28 <hagarth> kshlm: add a non-durable mode to glusterd and we probably should get that back in 60 minutes ;)
13:11:31 <ndevos> kshlm: you do not need to watch the console log for the test, you may switch to an other task ;-)
13:11:45 <JustinCl1ft> Guys, hold on
13:11:59 <JustinCl1ft> kshlm is right, in that the wait time affects our iteration speed
13:12:08 <JustinCl1ft> If we can get that sped up, we iterate faster
13:12:12 <JustinCl1ft> However...
13:12:18 <kshlm> JustinCl1ft, thanks for getting the iterate word.
13:12:28 <kshlm> That's what I was looking for.
13:12:43 <JustinCl1ft> We might be able to do the full regressions tests "differently" and still get the same coverage... faster
13:12:44 <ndevos> sure, we already started to move tests into seperate directories, that enables use to run tests in parallel - only needs the Jenkins support
13:12:53 <JustinCl1ft> Yep
13:12:57 <JustinCl1ft> That's the idea
13:13:20 <JustinCl1ft> In the same kind of way we have the "multiple things voting", we could do "multiple parts voting"
13:13:37 <JustinCl1ft> In theory, we might be able to get the whole thing done in under 20 mins
13:13:43 <ndevos> indeed, I would love to see a "passed the NFS tests" somewhere :)
13:13:52 <hagarth> ndevos: indeed!
13:14:04 <JustinCl1ft> So, I don't think cutting out test is the right approach
13:14:15 <hagarth> JustinCl1ft: I think we also need more scale-out on demand
13:14:16 <JustinCl1ft> As that reduces our coverage of finding edge case bugs
13:14:23 <JustinCl1ft> hagarth: And yeah.  We need that too.
13:14:23 <kshlm> Well, we do not need to run every test for every change.
13:14:37 <kshlm> What use is running io-tests for glusterd only changes?
13:14:54 <hagarth> I would also like to discuss a broader testing strategy soon.
13:14:54 <JustinCl1ft> kshlm: Finding unexpected bugs
13:15:03 <ndevos> kshlm: but how would the test framework know when not to run certain tests?
13:15:26 <jdarcy> As long as neither the number of tests nor the number of machines changes, the number of complete tests per hour won't either.  We'll still be backlogged during "rush hour" every day.
13:15:40 <JustinCl1ft> We should probably schedule another meeting for "GlusterFS Testing"
13:15:49 <kshlm> JustinCl1ft, We could do periodic full regressions.
13:15:55 <jdarcy> Parallelization will only improve latency for tests run on an otherwise-idle infrastructure.
13:16:00 * JustinCl1ft just noticed we're 15 minutes over... and I was supposed to join another meeting at the start of the hour
13:16:02 <hagarth> JustinCl1ft: sure
13:16:12 <kshlm> ndevos, Don't know. Probably pick tests based on the commit title.
13:16:16 <jdarcy> In fact, it might make things worse, as test set A continues to run even as test set B has already failed on another machine.
13:16:39 <hagarth> how about a discussion next week during one of the lightning talks?
13:16:49 <ndevos> kshlm: oh, thats dangerous!
13:16:54 <JustinCl1ft> jdarcy: Yeah, that's why we need to scale up the testing infra
13:17:03 <hagarth> we should use containers for infinite parallelism :D
13:17:22 <ndevos> kshlm: what if glusterd gets a change for a volume option, likely some other tests should get run then
13:17:32 <JustinCl1ft> jdarcy: This particular change gets the individual test results back quicker, it doesn't increase overall test volume throughput ;)
13:18:06 <jdarcy> Also, re: unit tests.  Great idea, but who's volunteering to teach people how to write unit tests that check *behavior* without overly constraining *implementation*?  I've already seen problems with that in the few unit tests we have.
13:18:21 <JustinCl1ft> That sounded like you volunteering. :D
13:18:24 <ndevos> JustinCl1ft: and queues will build up more, but also get reduced faster...
13:18:56 <kshlm> ndevos, in such a case I'd expect the title to contain the other component as well, so we'd run the correct tests.
13:19:16 <ndevos> jdarcy: lpabon started with the unit tests, he also has some presentations about it
13:19:19 <JustinCl1ft> ndevos: Yeah.  It's a change in the priority of our testing to recognise iteration speed is important enough that we want to put more focus on it
13:19:20 <jdarcy> JustinCl1ft: I think the people expressing the greatest insistence and enthusiasm should be the ones to volunteer.
13:19:24 <kshlm> In any case, I was just thinking out lout.
13:19:50 <hagarth> kshlm: good thoughts, we need to engage all of us in a discussion on our testing strategy soon.
13:19:57 <jdarcy> ndevos: Yes, and his unit tests *copied the implementation* to an extent that it made necessary fixes more difficult.  Bad example.
13:20:01 <ndevos> kshlm: yeah, thinking out loud here too :)
13:20:13 <hagarth> there are tonnes of areas to improve as far as our testing goes
13:20:19 <ndevos> jdarcy: I know, I had to fix some of them several times already :-/
13:20:20 <JustinCl1ft> Really, any time we want to change our coding practises, our failure point thus far has been in enforcement
13:20:23 <jdarcy> If that's what "we must have unit tests for every function means" count me out . . . of the project.
13:20:51 <JustinCl1ft> Every time we go hard-arse about enforcing a change, it seems to work.  Failure when we don't (because it's hard) ;)
13:21:24 <ndevos> I do not think a rule like that helps, common sense for adding functions that get re-used enough, and adding tests for those should be a way to start
13:21:30 <hagarth> how many other projects written in C have 100% or close to that number in terms of unit test coverage?
13:21:52 <jdarcy> hagarth: SQLite might come closest.
13:22:01 <hagarth> jdarcy: right..
13:22:06 <jdarcy> Or libcurl.
13:22:08 <ndevos> I think Samba has a lot of tests, no idea how much % they reach
13:22:13 <JustinCl1ft> SQLite has a shitload of test coverage
13:22:21 <Joe_f> hagarth: sqlite rock !
13:22:43 <ndevos> thank you for your contribution, Joe_f :D
13:22:54 <jdarcy> But mostly, C projects are weaker w.r.t. test coverage than e.g. Java or Python projects.
13:22:58 <JustinCl1ft> Hmmm, I haven't looked at the PG test coverage in years.  I don't even remember how it's done any more.
13:23:00 <hagarth> jdarcy: yeah
13:23:33 <ndevos> well, C has compiler tests already, Python does not, so I hope they test that more
13:23:44 <hagarth> I think we can derive best practices from other projects but I would hate to enforce anything unless the goals and implementation patterns are clear to everybody
13:23:45 <jdarcy> I guess that could be interpreted as a reason to migrate at least some parts of our code to a better language.
13:23:57 <JustinCl1ft> Bash!
13:24:00 <JustinCl1ft> ;P
13:24:10 <jdarcy> JustinCl1ft: OK.  <bashes JustinCl1ft>
13:24:14 <JustinCl1ft> :)
13:24:16 <hagarth> lol
13:24:32 <hagarth> how do we take this discussion further? mailing list or BCN?
13:24:33 <kkeithley> Go? Swift?
13:24:53 <kshlm> Swift?
13:24:58 <hagarth> Apple's swift?
13:25:00 <ndevos> S3?
13:25:10 <jdarcy> kkeithley: Go seems like the crowd favorite.  Rust or D are perhaps more task-appropriate.
13:25:11 <kkeithley> I was joking, but yes, Apple's Swift. ;-)
13:25:28 <jdarcy> Or Nim, for that matter.
13:25:34 <kshlm> I'd love to do Go.
13:25:52 <JustinCl1ft> Go has a lot of people intested in it, even from our team.  There is difficuly in debugging and core collection from external people's prod deployments from memory though.  kshlm would have a better idea, as I think he looked into it.
13:25:57 <hagarth> let us see.. I propose that we discuss some of this in BCN next week and have a more detailed conversation on the ML
13:26:00 <kkeithley> Whatever is new and hawt.
13:26:05 <jdarcy> We should have a hackfest, rewrite volgen in Rust/D/Nim/Go and compare the results.
13:26:10 <ndevos> so, do the Go developers write unittests?
13:26:19 <hagarth> ndevos: Go, figure that out :)
13:26:24 <ndevos> :P
13:26:34 <JustinCl1ft> jdarcy: It's a decent idea
13:26:48 <jdarcy> ndevos: I don't know, haven't looked at too many Go projects.
13:26:50 <JustinCl1ft> jdarcy: To that list, "figure out how to support it" for each of the written results
13:26:56 <hagarth> alright everyone, hope to see most of you in Barcelona next week! safe travels for those who are going to be around there!
13:27:05 <JustinCl1ft> Good point
13:27:15 * JustinCl1ft waves
13:27:17 <hagarth> and we will cancel the online community meeting next week
13:27:18 <kshlm> See you all in Barcelona.
13:27:28 <jdarcy> Bar-ce-LO-na!
13:27:30 <kkeithley> vaya con dios
13:27:32 * kshlm will hopefully get his visa soon
13:27:36 <hagarth> thanks, bye for now!
13:27:40 <hagarth> #endmeeting