15:00:25 <hagarth> #startmeeting
15:00:25 <zodbot> Meeting started Wed Dec 18 15:00:25 2013 UTC.  The chair is hagarth. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:25 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:26 <purpleidea> new code words: . means ack - means nack (to avoid typing)
15:00:26 <glustermeetbot> Meeting started Wed Dec 18 15:08:18 2013 UTC.  The chair is hagarth. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:26 <glustermeetbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
15:00:32 <hagarth> hi all
15:00:37 <purpleidea> hi
15:00:40 <ndevos> hi
15:00:49 <dbruhn> Hola
15:00:55 <hagarth> #topic Rollcall & AI follow up
15:01:09 <hagarth> who do we have here today?
15:01:19 <purpleidea> purpleidea johnmark ndevos
15:01:29 <hagarth> and dbruhn too
15:01:48 <hagarth> let us get started reviewing AIs from last week
15:02:06 <hagarth> ndevos: any update on our joint AI? :)
15:03:08 <ndevos> hagarth: purpleidea made some progress, assuming you mean the vm images?
15:03:15 <hagarth> ndevos: right
15:03:28 <hagarth> I think we should use that for our beta
15:03:51 <purpleidea> oh yeah, johnmark got me storage (thanks) and ndevos is building normal images, and i'm working on vagrant + puppet-gluster (for general testing)
15:03:55 <ndevos> definitely, hosting has been arranged, thanks johnmark!
15:04:03 <johnmark> ndevos: purpleidea: you're welcome :)
15:04:06 <hagarth> on the topic of 3.5 beta, we are still awaiting some geo-replication fixes that are blocking its release.
15:04:20 <johnmark> hagarth: yeah, saw the update. thanks
15:04:21 <purpleidea> hagarth: my one question is i assume we want to use any rpm from bits.g.o, is that right?
15:04:47 <pk> hagarth: We also need to fix entry self-heal problems reported by Venkatesh
15:04:56 <pk> hagarth: for 3.5 I mean
15:05:07 <hagarth> purpleidea: that is right, we plan to move everything to download.g.o. right now, all 3.5 rpms are on bits.g.o
15:05:20 <hagarth> pk: yes, have noted that.
15:05:44 <purpleidea> hagarth: okay. currently i used the dir structure on download.gluster.org will that be the same?
15:05:45 <johnmark> pk: thanks
15:05:54 <ndevos> purpleidea: yes, we need to make sure that the packages are on download.gluster.org (is bits.g.o a mirror or the same system)
15:05:56 <johnmark> purpleidea: that will be the same
15:06:00 <hagarth> purpleidea: yes, that will be the same
15:06:13 <johnmark> ndevos: bits.g.o is a different system
15:06:18 <purpleidea> hagarth: FWIW, the dir structure on bits is quite different than download.g.o
15:06:19 <hagarth> there is also another issue which was reported by Raghavendra Bhat on gluster-devel around the client_t implementation, we would need to fix that too.
15:06:30 <johnmark> frankly, we could just as easily copy over RPMS until we get the autobuilds fixed
15:06:46 <hagarth> purpleidea: yes, that is owing to the release script which uploads packages onto bits.g.o. We should be able to fix that.
15:07:15 <johnmark> hagarth: shoudl we create a tracking bug for outstanding BZs?
15:07:20 <hagarth> johnmark: sounds like a plan, we could also include links to packages on download.g.o in the email that we send out.
15:07:28 <johnmark> hagarth: ok
15:07:35 <hagarth> johnmark: I think that would be very useful. Let me take that AI on me.
15:07:44 <hagarth> #action hagarth to create a tracking bug for 3.5
15:08:00 <hagarth> ok, moving on to 3.5.0
15:08:03 <hagarth> #topic 3.5.0
15:08:10 <pk> hagarth: Avati is on PTO according to the automated replies. He told he is working on a fix for client_t + locks bug. I forgot the release data for 3.5 :-(
15:08:25 <pk> hagarth: date*
15:08:31 <hagarth> the current status is this - we are blocked on geo-replication
15:08:32 <pk> hagarth: He is on PTO till 23rd
15:08:36 <johnmark> hagarth: thanks
15:08:59 <hagarth> geo-replication doesn't work really well in 3.5.0, that needs to be fixed for 3.5.0 beta to happen.
15:09:13 <pk> hagarth: Do we have time till 23rd? If not I will look into that bug.
15:09:30 <hagarth> venky shankar and others are looking into the geo-rep problems, once that is sorted out we can do a beta.
15:09:43 <hagarth> pk: don't think we can wait till 23rd
15:10:02 <pk> hagarth: Ok I will look into it then
15:10:05 <hagarth> pk: can you also sync up with Raghavendra Bhat? He has some ideas on a lightweight patch for that.
15:10:16 <pk> hagarth: Sure sir
15:10:21 <johnmark> heh
15:10:23 <purpleidea> hagarth: for the testing, let me know if the format changes from: https://github.com/purpleidea/puppet-gluster/blob/master/manifests/repo.pp#L26 and when qa is added, and i'll add the relevant code to make it automatic for testers.
15:10:44 <hagarth> purpleidea: will do that, thanks.
15:10:52 * johnmark is going to write a bot to rewrite github links
15:10:54 <pk> hagarth: Anyone looking into the race issue in 3.5 reported by Emmanuel today?
15:11:02 <hagarth> #action hagarth to look into format changes from https://github.com/purpleidea/puppet-gluster/blob/master/manifests/repo.pp#L26
15:11:28 <hagarth> pk: not that I am aware of, we should translate that to a bug and add it is a dependency for 3.5
15:11:37 <hagarth> coming to the schedule for 3.5.0
15:11:41 <johnmark> hagarth: is it realistic to say that outstanding patches + fixes could be merged by early next week?
15:11:55 <hagarth> here's the latest update on the schedule:
15:11:59 <johnmark> ok
15:12:07 <hagarth> #link http://www.gluster.org/community/documentation/index.php/Planning35#GlusterFS_3.5_Release_Planning
15:12:20 <hagarth> I am open to suggestions on the schedule
15:12:39 <hagarth> beta has slipped from the original scheduled date of Dec 9th
15:12:56 <hagarth> so has the community test phase 2
15:13:09 <hagarth> and GA is contingent on these two happening
15:13:26 <johnmark> hagarth: right. the earliest we could have a beta would be 12/23
15:13:31 <johnmark> from my calculations
15:13:45 <johnmark> and we need at least two weeks afterwards for "baking in"
15:13:53 <hagarth> johnmark: seems like that
15:14:01 <johnmark> hagarth: so I'm betting on mid-january
15:14:08 <pk> hagarth: I am not sure if this is the right forum to ask this question. What all kinds of tests are going to happen on test day?
15:14:25 <johnmark> pk: wahtever we can pull together :)
15:14:33 <pk> johnmark: cool :-)
15:14:45 <hagarth> pk: you can take a look into the test weekend page that was created earlier
15:14:56 <pk> hagarth: sure
15:15:03 <hagarth> we provide some recommendations on what needs to be tested
15:15:10 <johnmark> ben england sent me this: https://github.com/bengland2/smallfile
15:15:10 <purpleidea> i briefly looked at integrating the ben england smallfiles stuff with puppet-gluster + vagrant, maybe we want to dump out automatic performance graphs alongside smoke tests?
15:15:16 <johnmark> as a potential testing tool
15:15:21 <hagarth> johnmark: yes, mid-january seems more like it
15:15:24 <purpleidea> johnmark: yeah
15:15:28 <johnmark> purpleidea: oooh.. I like how you think
15:15:38 <hagarth> johnmark: yes, we can use that for our performance tests.
15:15:51 <johnmark> hagarth: ok, thanks. Ben is wondering how to socialize that
15:15:57 <pk> hagarth: Are these performance tests going to be performed on VMs?
15:15:58 <hagarth> purpleidea, johnmark: we need to see how long the tests take
15:15:59 <johnmark> hagarth: so we need to help him with that
15:16:14 <purpleidea> johnmark: thanks. if someone wants to sit and get me up to speed, i could probably make that go automatically everytime a set of rpms are generated (and the machines to run it on)
15:16:16 <johnmark> hagarth: ok
15:16:18 <hagarth> if the tests don't take significant time, we can have it as part of smoke.
15:16:31 <hagarth> else we could have it on a nightly build
15:16:47 <hagarth> pk: excellent question, we would need to run these tests on physical machines
15:16:48 <purpleidea> hagarth: i think it makes sense to run on every qa rpm build
15:16:59 <purpleidea> hagarth: ben england used vm's fwiw
15:17:23 <hagarth> purpleidea: interesting .. but the VMs have to be in a very controlled enviornment
15:17:31 <johnmark> I'll make sure Ben England sees the invite to next week's meeting
15:17:39 <johnmark> so he cna weigh in himself
15:17:40 <hagarth> else the environment flakiness can affect the overall performance results
15:17:51 <pk> hagarth: Exactly!
15:17:55 <purpleidea> hagarth: if you like. i think it's useful not for data, but for catching big differences, if you find something way off, you run it yourself
15:18:53 <hagarth> purpleidea: would you mind taking up integrating smallfiles in Jenkins?
15:19:17 <hagarth> we can also potentially consider Avati's perf-test.sh
15:19:22 <purpleidea> hagarth: not sure what's required. i'm more interested in the puppet / vagrant side really
15:19:46 <purpleidea> ben england is probably the right guy if it's a straight up jenkins issue
15:19:53 <hagarth> purpleidea: ok. I can start a thread on that in gluster-devel.
15:20:01 <johnmark> hagarth: +1
15:20:06 <pk> hagarth: +1
15:20:25 <hagarth> improving Jenkins/CI will help in getting better quality.
15:20:50 <hagarth> #info 3.5.0 GA moved to mid January
15:21:01 <hagarth> I will update the planning35 page.
15:21:09 <hagarth> any more questions on 3.5?
15:21:29 <hagarth> guess not, let us move on.
15:21:40 <hagarth> #topic 3.4.2
15:21:51 <hagarth> 3.4.2 made a lot of progress over the last week
15:21:57 <johnmark> hagarth: woohooo
15:22:15 <hagarth> we got the dht problem sorted out, Lukas updated saying that he no longer observes the problem with qa4.
15:22:28 <hagarth> there are minor CLI fixes that we need to get into 3.4.2
15:22:28 <johnmark> hagarth: great news
15:22:56 <hagarth> the CLI fixes are to warn users about rdma and replace-brick
15:23:29 <hagarth> since both don't work well and have been a continuous source of complaints in the community, it is better to warn when anybody attempts these commands.
15:23:56 <hagarth> though we have documented the behavior, nobody reads the fine manuals :)
15:24:06 <johnmark> hagarth: agreed.
15:24:30 <hagarth> once that is done, I think we are good to release 3.4.2
15:24:50 <hagarth> am aiming for this Friday, unless we run into other problems
15:25:14 <hagarth> any other problems that we would want to fix for 3.4.2?
15:25:32 <hagarth> there was a backport request on NUFA but it does seem like a lot of work
15:25:42 <hagarth> I intend deferring that to 3.4.3
15:25:49 <johnmark> hagarth: I think that is acceptable
15:26:15 <hagarth> #info 3.4.2 tentatively scheduled to be released on 12/20
15:26:17 <johnmark> hagarth: if it were a heavily requested backport, I could see doing it
15:26:27 <johnmark> hagarth: could we give maybe a couple of days for beta?
15:26:39 <hagarth> johnmark: agree, should we push it to 23rd then?
15:26:52 <johnmark> hagarth: sure. xmas gift to teh community :)
15:27:23 <hagarth> johnmark: yes, I already saw tweets about our iscsi tgt support being an xmas gift to the community :)
15:27:31 <johnmark> hagarth: heh heh
15:27:34 <hagarth> so we seem to be rolling out a lot of goodies ;)
15:27:35 <purpleidea> tgt?
15:27:37 <johnmark> our gifts overfloweth
15:27:38 <pk> hagarth: I see the patch which might have exposed softlink heal problem in 3.4 branch
15:27:47 <johnmark> purpleidea: dude, don't you watch the gluster blog?
15:27:50 <johnmark> *sigh*
15:27:53 <purpleidea> johnmark: it's broken!
15:27:54 <johnmark> heh
15:27:59 <johnmark> purpleidea: ? how so?
15:28:06 <johnmark> purpleidea: could you please report these things?
15:28:12 <hagarth> pk: I see, do we know the issue?
15:28:15 <purpleidea> johnmark: oh, i thought someone did
15:28:27 <johnmark> purpleidea: nope.
15:28:41 <pk> hagarth: I will need to sit with Venkatesh and figure it out. Will update you tomorrow
15:29:05 <hagarth> pk: thanks, if it is a relatively simple fix, we can consider pulling that into 3.4.2 as well.
15:29:16 <pk> hagarth: Sure.
15:29:24 <hagarth> #info 3.4.2 tentatively scheduled to be released on 12/23
15:29:31 <hagarth> anything more on 3.4.2?
15:29:42 <purpleidea> oh, iscsi target.. cool
15:30:04 <hagarth> ok, moving on
15:30:07 <hagarth> #topic Backward compatibility
15:30:31 <hagarth> this is a continuation of the discussion that we briefly alluded to in last week's meeting
15:30:46 <pk> hagarth: yes
15:30:54 <hagarth> our definition of backward compatibility is this:
15:31:03 <johnmark> hagarth: wait... is there a workaround for the softlink issue?
15:31:10 <hagarth> johnmark: ok
15:31:24 <hagarth> johnmark: pk should be able to update us tomorrow on that
15:31:29 <pk> hagarth: Not that I know of
15:31:40 <pk> hagarth: Will update the thread tomorrow
15:31:46 <hagarth> pk: thanks.
15:32:15 <johnmark> pk: thanks
15:32:22 <johnmark> hagarth: ok, sorry. carry on :)
15:32:25 <hagarth> johnmark: I think pk, Venkatesh and Raghavendra Bhat deserve some swag for the critical problems identified.
15:32:32 <johnmark> hagarth: +1 :)
15:32:46 <hagarth> ok, coming back to backward compat
15:33:10 <hagarth> 1. old servers and new clients (fuse/libgfapi) should be able to interoperate
15:33:40 <hagarth> 2. state/metadata maintained by GlusterFS should be usable across versions
15:33:49 <pk> hagarth: Isn't it old clients and new servers?
15:34:03 <hagarth> pk: I missed the vice-versa in 1. , thanks for pointing out
15:34:36 <hagarth> 3. We should be able to support a heterogeneous (old + new) trusted storage pool
15:34:57 <purpleidea> #3 cool ^^^
15:34:58 <hagarth> by 3., we refer to servers in a cluster running different versions.
15:35:25 <hagarth> the question that we need to sort out is, what should be our guarantee around these 3 aspects?
15:35:43 <johnmark> hagarth: wow re: #3
15:36:02 <hagarth> right now, our state is - 3.3 and 3.4 are compatible
15:36:09 <ira> hagarth: #3 is a requirement for rolling upgrades?
15:36:54 <ira> Or do we expect to support a cluster running like that for 6 months?
15:36:57 <hagarth> ira: primarily for that. In large clusters, operators sometimes find it difficult to upgrade everything in a maintenance window.
15:37:14 <kdhananjay> hagarth: One question with regard to #3. Do we need to bump up op-versions across minor releases, since they primarily contain bug fixes and no new features?
15:37:36 <johnmark> kdhananjay: but sometimes there are new features
15:37:42 <hagarth> ira: we would not want anybody to be running a heterogeneous cluster for a long time.
15:37:57 <ira> hagarth: Then we should be clear about that.  ;)
15:38:10 <hagarth> would it be fair to say that we support #3 for rolling upgrades only?
15:38:21 <johnmark> hagarth: I think that's fair
15:38:23 <ira> hagarth: That sounds very reasonable here.
15:38:23 <purpleidea> kdhananjay: if you do, i'd appreciate hearing about it for: https://github.com/purpleidea/puppet-gluster/blob/master/manifests/host.pp#L86
15:38:53 <ira> hagarth: It also helps with bugs that really do require full cluster upgrades to fix.   :/
15:38:54 <kdhananjay> purpleidea: Sure.
15:39:09 <hagarth> kdhananjay: we would not want to bump up op-versions across minor releases ideally. Are there features that we are getting into minor releases?
15:39:12 <johnmark> otherwise we get into a hairy state of determining least common denominator, feature-wise
15:39:13 <purpleidea> kdhananjay: much appreciated! if there are currently any missing values, let me know
15:39:41 <hagarth> ira: absolutely, #3 is mostly for convenience, but not for steady operation state.
15:39:52 <kdhananjay> hagarth: Checking the code.
15:40:56 <hagarth> kdhananjay: ok, we need to be vigilant about op-versions. For providing the ability to introduce new features in minor releases, we would be better off in having a range for op-versions for a release.
15:41:28 <kdhananjay> hagarth: Agreed.
15:41:55 <hagarth> in that case, we could bump up all op-versions in 3.5 :)
15:42:01 <ira> And on #1, do we expect to run in that state long term?
15:43:13 <hagarth> ira: yes, that can happen. Normally the client installations are managed by compute administrators and the servers are managed by a different set. Co-ordinating upgrades/maintenance windows is an operational problem across these two entities.
15:44:18 <hagarth> also if an application is compiled with libgfapi and happens to be running, we would not want to disrupt it normally.
15:44:41 <ira> hagarth: Agreed... that makes sense.  How big a gap do we support?
15:45:12 <pk> ira: In the previous meet we decided it is 2 releases.
15:45:22 <johnmark> pk: 2 major releases?
15:45:28 <ira> So 3.3 -> 3.4 -> 3.5?
15:45:40 <hagarth> ira: yeah, possibly that
15:45:46 <ira> (I realize 3.5 is a special case... but assume it is not for that discussion)
15:46:20 <hagarth> is everybody fine with 2 major releases?
15:46:38 <ira> Major defined, as revving the minor version number ;).
15:46:38 <hagarth> anyway, we are going to break quite a bit when we move to 4.0 :)
15:46:57 <johnmark> hagarth: looking forward to that!
15:47:06 <johnmark> ira: correct :D
15:47:12 <hagarth> right
15:47:26 <hagarth> johnmark: yeah, we can schedule some discussions in January around 4.0.
15:47:35 <johnmark> hagarth: looking forward to it
15:47:42 <hagarth> ok, anything more on backward comaptibility - we seem to have consensus here
15:48:01 <hagarth> s/map/mpa/
15:48:16 <pk> hagarth: We probably need a bit more discussion around op-versions then send a summary mail on gluster-devel and take it forward.
15:48:17 <hagarth> let us move on
15:48:35 <hagarth> pk: sounds good, will you and kdhananjay be able to take the lead on that?
15:48:42 <pk> hagarth: Sure
15:48:49 <kdhananjay> hagarth: Sure.
15:48:56 <hagarth> #action pk and kdhananjay to send out a mail on gluster-devel around op-versions.
15:49:05 <hagarth> #topic 3.6 planning
15:49:30 <hagarth> now that we are getting into the last phase of 3.5.0, it is time to start thinking about 3.6
15:49:51 <johnmark> like, is it going to be 3.6? or 4.0?
15:50:02 <hagarth> first question - do we all like the short release cycle that happened with 3.5?
15:50:19 <hagarth> shorter i would say, considering the time we took to release 3.4 ;)
15:50:29 <hagarth> johnmark: 3.6 will be an interim release before 4.0
15:50:51 <hagarth> some of the work around volume snapshots and nsr are probably going to appear in 3.6
15:51:09 <hagarth> 4.0 will take some time to be out
15:51:17 <pk> hagarth: I am thinking changing afr heal commands to gfapi just like we did for "heal info" command
15:51:34 <hagarth> pk: that would be nice
15:51:34 <ira> hagarth: Are we branching 3.6/4.0?
15:51:47 <pk> hagarth: Not sure how much effort index changes will be. Need to thing about that before committing
15:52:06 <pk> hagarth: think*
15:52:11 <hagarth> ira: not yet, we will continue to operate on master for 3.6
15:52:30 <hagarth> 4.0 will probably happen in a different branch/repo and appear on master after we branch 3.6
15:52:35 <hagarth> pk: sure
15:52:45 <hagarth> my mental model is this
15:52:49 <johnmark> hagarth: ok
15:52:52 <hagarth> have a short release cycle for 3.6
15:53:01 <hagarth> and then look at 4.0
15:53:09 <johnmark> hagarth: how short?
15:53:15 <hagarth> 3.6 would be the vehicle for a lot of features that are in the pipeline
15:53:28 <purpleidea> hagarth: reflink support ?
15:53:32 <hagarth> johnmark: we can throw the planning window open now
15:53:43 <johnmark> hagarth: I think we need to stick to our time-based releases
15:53:50 <hagarth> and have a release planning meeting towards end of January
15:54:03 <johnmark> if we're going out saying "our releases will be 6 months apart" then we should stick to that
15:54:04 <hagarth> we could look at 3-4 months from then for releasing 3.6
15:54:30 <hagarth> we can tentatively look at having 3.6 out in April/May.
15:54:34 <johnmark> hagarth: ok
15:54:55 <hagarth> I will post a tentative schedule this week
15:55:01 <johnmark> hagarth: thanks
15:55:23 <hagarth> purpleidea: yes, we can consider that for 3.6
15:55:32 <purpleidea> +10
15:55:56 <hagarth> i am really interested in getting volume snaps, erasure coding, nsr, reflink, xlators for btrfs etc. out in 3.6
15:56:18 <hagarth> I will also try to publish a backlog so that we can identify and vote what is good for 3.6
15:56:22 <pk> hagarth: by erasure coding you mean the one implemented by xavih?
15:56:46 <hagarth> pk: yes, intel also released a library for erasure coding.. need to check if we can write a plugin for that too
15:56:57 <pk> hagarth: nice!
15:57:05 <hagarth> any more questions on 3.6?
15:57:19 <johnmark> hagarth: yeah, which reminds me, we really need to make sure t.gohad from Intel is participating in this next release cycle
15:57:31 <hagarth> #action hagarth to send out a tentative schedule for 3.6
15:57:52 <hagarth> johnmark: absolutely, we could start reaching out to potential contributors after the holidays
15:58:07 <johnmark> hagarth: +1
15:58:11 <hagarth> #topic open discussion
15:58:29 <hagarth> I plan to cancel this meeting for the next 2 weeks.
15:58:33 <purpleidea> okay, review.gluster needs proper signed https so does download.gluster.o
15:58:36 <johnmark> hagarth: good idea :)
15:58:54 <hagarth> so that everybody comes back with fresh ideas for 3.6 in January :)
15:58:56 <johnmark> purpleidea: yeah, I have the cert. Will install it
15:59:02 <purpleidea> johnmark: sweet
15:59:03 <johnmark> hagarth: thanks!
15:59:12 <hagarth> johnmark: great
15:59:15 <purpleidea> also in case you missing the mailing list announce: http://gluster.org/pipermail/gluster-users/2013-December/038329.html
15:59:27 <purpleidea> there is now JMWbot ! you can get it to bug johnmark for you!
15:59:34 <johnmark> purpleidea: :)
15:59:35 <purpleidea> available in #gluster
15:59:38 <hagarth> purpleidea: +10
15:59:46 <hagarth> :)
15:59:46 <pk> purpleidea: +10
15:59:52 <purpleidea> #link http://gluster.org/pipermail/gluster-users/2013-December/038329.html
15:59:59 <johnmark> purpleidea: as soon as you hand out the source code, we cna extend it to be a general purpose reminder bot :)
16:00:13 <purpleidea> so far it works, i'll post the source shortly when it's not a weird hour of the day
16:00:19 <johnmark> purpleidea: becuase right now, I have no way to remind you of that ;)
16:00:20 <ndevos> purpleidea: do you know if it works? are the reminders actioned?
16:00:21 <hagarth> ok, it has been a good meeting - thanks everybody and here's wishing you all a very happy holiday season
16:00:31 <johnmark> hagarth: thank you!
16:00:36 <hagarth> see you all in 2014!
16:00:40 <hagarth> #endmeeting