17:00:19 <dustymabe> #startmeeting atomic_wg
17:00:19 <zodbot> Meeting started Wed Mar 29 17:00:19 2017 UTC.  The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:19 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
17:00:19 <zodbot> The meeting name has been set to 'atomic_wg'
17:01:01 <trishnag> hi
17:01:06 <jbrooks> .fas jasonbrooks
17:01:07 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <JBROOKS@REDHAT.COM>
17:01:10 <dustymabe> #topic roll call
17:01:14 <trishnag> .hello trishnag
17:01:15 <zodbot> trishnag: trishnag 'Trishna Guha' <trishnaguha17@gmail.com>
17:01:16 <dustymabe> .hellomynameis dustymabe
17:01:18 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dustymabe@redhat.com>
17:01:21 <yzhang> .hello yzhang
17:01:22 <zodbot> yzhang: yzhang 'Yu Qi Zhang' <jzehrarnyg@gmail.com>
17:01:39 <bowlofeggs> .hello bowlofeggs
17:01:40 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <randy@electronsweatshop.com>
17:02:15 <maxamillion> .hello maxamillion
17:02:15 <zodbot> maxamillion: maxamillion 'Adam Miller' <maxamillion@gmail.com>
17:02:22 <sayan> .hello sayanchowdhury
17:02:23 <zodbot> sayan: sayanchowdhury 'Sayan Chowdhury' <sayan.chowdhury2012@gmail.com>
17:02:28 <dustymabe> sayan: o/
17:02:34 <walters> .hello walters
17:02:35 <zodbot> walters: walters 'Colin Walters' <walters@redhat.com>
17:02:49 <dustymabe> #chairs walters sayan maxamillion bowlofeggs yzhang trishnag jbrooks
17:02:58 <dustymabe> #chair walters sayan maxamillion bowlofeggs yzhang trishnag jbrooks
17:02:58 <zodbot> Current chairs: bowlofeggs dustymabe jbrooks maxamillion sayan trishnag walters yzhang
17:03:02 <dustymabe> there we go :)
17:03:06 <dustymabe> like riding a bike
17:03:18 <dustymabe> #topic agenda
17:03:31 <dustymabe> ok I'm going to run through previous meeting action items
17:03:45 <dustymabe> then go over a few discussion points/FYI items
17:03:48 <dustymabe> then we'll get to tickets
17:04:02 <dustymabe> #topic previous meeting action items
17:04:08 <dustymabe> AKA the maxamillion show
17:04:14 <maxamillion> o hai
17:04:19 <dustymabe> maxamillion to verify if we can change case in koji etc., or fix it if we can't
17:04:21 <dustymabe> maxamillion to follow-up on fixing ami permissions, links with dgilmore
17:04:23 <dustymabe> maxamillion to verify if we can change case in koji etc., or fix it if we can't
17:04:25 <dustymabe> maxamillion to follow-up on fixing ami permissions, links with dgilmore
17:04:27 <dustymabe> yzhang to follow up on atomic CLI task in #253
17:04:29 <dustymabe> , dustymabe to open tracker bug for f26 issues that will be used throughout f26 release
17:04:31 <dustymabe> maxamillion: :)
17:04:55 <miabbott> .hello miabbott
17:04:55 <zodbot> miabbott: miabbott 'Micah Abbott' <miabbott@redhat.com>
17:04:58 <maxamillion> We *can* change case, but with a catch .... what's the ticket # on that? I'll update with info
17:05:15 <yzhang> I think #253 was the original discussion?
17:05:20 <dustymabe> yzhang: indeed
17:05:21 <yzhang> https://pagure.io/atomic-wg/issue/253
17:05:22 <maxamillion> and I've followed-up with dgilmore, but last I checked it was still in-flight
17:05:25 <maxamillion> yzhang: thanks
17:05:35 <dustymabe> maxamillion: i really don't understand the 'in-flight' part of that
17:05:42 <dustymabe> that's been the status for two weeks
17:05:45 <roshi> .hello roshi
17:05:46 <zodbot> roshi: roshi 'Mike Ruckman' <mruckman@redhat.com>
17:05:52 * roshi has coffee now and is ready
17:06:00 <maxamillion> dustymabe: last I heard, whatever needs doing is still in the process of being done ... someone is waiting on someone to provide credentials
17:06:02 <dustymabe> not trying to be difficult, but there is something holding it up. most likely TIME
17:06:49 <yzhang> So just to clarify, all lowercase does not work in koji atm?
17:06:55 <dustymabe> maxamillion: which should take very little time unless there is a bigger issue, correct?
17:07:45 <maxamillion> dustymabe: I would have thought, yes
17:08:07 <dustymabe> maxamillion: i'll start an email thread with dgilmore, sounds like someone just needs to be poked
17:08:13 <dustymabe> and/or this needs to be escalated
17:08:14 <maxamillion> dustymabe: +1
17:08:48 <dustymabe> ok maxamillion yzhang had a question for you
17:09:30 <yzhang> More that I wanted to make sure what exactly is the issue, and whether i can rebuild some containers in the registry with lowercase or not
17:09:32 <yzhang> maxamillion ^
17:09:40 <maxamillion> yzhang: it does work, but there's a catch ... we have to change one of the fields but I'm not sure we want to
17:09:45 <maxamillion> just a sec
17:10:16 <maxamillion> https://github.com/release-engineering/koji-containerbuild/blob/master/koji_containerbuild/plugins/builder_containerbuild.py#L65
17:10:20 <maxamillion> yzhang: this is what we can use ^^^^
17:10:56 <dustymabe> maxamillion: thanks for the update.
17:11:00 <maxamillion> yzhang: so if we don't do BZComponent, we'd need to use com.redhat.component but I'd rather do org.fedoraproject.component .... however our BZ instance is actually bugzilla.redhat.com so I don't know for sure
17:11:04 <maxamillion> which is right*
17:11:11 <yzhang> maxamillion: I see
17:11:14 <maxamillion> I'm going to comment on the ticket with all this also
17:11:18 <maxamillion> for posterity
17:11:29 <yzhang> Ok, thanks!
17:11:29 <dustymabe> as for my action item i created the tracker ticket for f26 atomic blocker bugs
17:11:42 <dustymabe> https://pagure.io/atomic-wg/issue/261
17:11:56 <maxamillion> we can patch that however we want an bend it to our will *however* there's implications beyond just adding the new values to the LABEL_NAME_MAP through out osbs-client and atomic-reactor
17:11:58 <dustymabe> we'll use this to track issues we want to get fixed
17:12:17 <dustymabe> yzhang: you had one action item, care to update us?
17:12:31 <yzhang> Yep, so its very similar to the above discussion
17:12:37 <yzhang> I opened an issue: https://github.com/projectatomic/atomic/issues/948
17:12:48 <yzhang> dwalsh was very concise in his answer
17:13:19 <dustymabe> yzhang: yes he was.. want to summarize?
17:13:21 <yzhang> Basically to use lowercase, notably following: https://github.com/projectatomic/ContainerApplicationGenericLabels
17:13:46 <dustymabe> so he wants us to use lowercase?
17:13:50 <yzhang> Yep
17:13:58 <dustymabe> perfect, that's what we wanted anyway right?
17:14:04 <roshi> think so
17:14:07 <roshi> that makes it easy
17:14:24 <yzhang> Basically, except that there is no 'com.redhat.component' or 'BZComponent'
17:15:00 <yzhang> or well I should have said
17:15:02 <miabbott> maybe we should propose such a field
17:15:14 <yzhang> that it falls under the reverse DNS notation
17:15:47 <yzhang> Which goes back to what maxamillion was saying about org.fedoraproject.component
17:15:48 <dustymabe> yzhang: i thought we were just deciding on upper vs lowercase
17:15:50 <maxamillion> yzhang: https://github.com/projectatomic/ContainerApplicationGenericLabels/blob/master/vendor/redhat/labels.md
17:15:54 <yzhang> Right
17:15:57 <dustymabe> and trying to make sure lowercase didn't break the atomic CLI
17:15:59 <dustymabe> right?
17:16:00 <maxamillion> yzhang: there are lowercase, it's just not in the overview doc
17:16:13 <yzhang> Right yes, sorry
17:16:22 <yzhang> Re-summary: We should use lowercase, does not break atomic CLO
17:16:24 <maxamillion> so the question then becomes, do we make our own vendor labels?
17:16:24 <yzhang> CLI*
17:16:40 <dustymabe> so in the interest of not running off the rails to much
17:16:48 <dustymabe> maybe we take that question into another ticket?
17:17:02 <dustymabe> can we safely close this ticket with the summary?
17:17:16 <dustymabe> #253, that is
17:17:21 <yzhang> I think so yes
17:17:28 <roshi> makes sense to me
17:17:33 <dustymabe> maxamillion: ^^ ?
17:17:50 <dustymabe> that is one ticket assigned to you that we can close!!
17:18:00 <dustymabe> assuming you agree
17:18:19 <maxamillion> I don't know if the ticket should be closed or not
17:18:30 <maxamillion> because we can't just switch, the build will fail
17:18:50 <dustymabe> maxamillion: is that the "caveat" ?
17:19:00 <maxamillion> yes, I explained it before
17:19:01 <maxamillion> *something* needs to be done first, we either need to be alright with com.redhat.component or patch code
17:19:27 <maxamillion> either is fine, the patching would not be difficult, but it's also not just a simple couple line change
17:19:57 <dustymabe> so the current summary is: we have decided to go with all lowercase, but there is a blocker before we can get there. lay out steps to resolve blocker issue, etc
17:20:02 <maxamillion> if we want something like org.fedoraproject.component then we should make our own upstream vendor labels doc and patch code
17:20:15 <maxamillion> dustymabe: +1
17:20:24 <trishnag> dustymabe yes.
17:20:40 <maxamillion> any of these options are fine, I just want to make sure expectations are appropriately set
17:20:43 <dustymabe> ok maxamillion do you mind adding that to your summary of the "caveat" when you comment on the bug
17:20:52 <maxamillion> dustymabe: will do
17:21:13 <dustymabe> I would actually be pretty ok with com.redhat.component
17:21:18 <dustymabe> at least for now
17:21:23 <yzhang> Just curious since I'm new to all of this, for a (relative) big decision like this of "whether we should be our own upstream vendor"
17:21:29 <yzhang> Whats the decision process
17:21:31 <dustymabe> if we wanted to change it later we could, but if we never did then it would still work
17:22:16 <dustymabe> yzhang: not sure if I can answer your whole question
17:22:43 <dustymabe> but usually the decision process is: we discuss, come up with a proposed solution, a decent amount of people agree, hopefully no one opposes
17:22:51 <dustymabe> if we do have opposition we would then vote
17:23:15 <dustymabe> but we haven't really got there too often TBH
17:23:19 <yzhang> dustymabe: I see, thanks!
17:23:23 <yzhang> Sorry for the sidetrack
17:23:33 <dustymabe> np
17:23:39 <dustymabe> ok on to announcements
17:23:42 <dustymabe> #topic announcements
17:23:53 <dustymabe> We had a new F25AH release yesterday!
17:23:59 <dustymabe> #info We had a new F25AH release yesterday!
17:24:04 <yzhang> \o/
17:24:14 <dustymabe> #link http://www.projectatomic.io/blog/2017/03/fedora_atomic_mar28/
17:24:20 <maxamillion> container releases will go out today
17:24:30 <dustymabe> maxamillion: cool!
17:25:04 <dustymabe> #info Fedora 26 is getting more stable
17:25:18 <dustymabe> follow along for known bugs that we want to get fixed in the tracker bug
17:25:33 <dustymabe> #link https://pagure.io/atomic-wg/issue/261
17:25:52 <dustymabe> ok now for some discussion
17:26:04 <dustymabe> #topic containerized kubernetes on FAH
17:26:17 <dustymabe> so jbrooks has done some more work on containerized kube on atomic host
17:26:25 <dustymabe> jbrooks: do you have the link to your blog post
17:26:38 <dustymabe> #link https://jebpages.com/2017/03/17/test-containerized-kube-and-system-container-based-flannel-and-etcd/
17:26:41 <dustymabe> found it :)
17:26:44 <jbrooks> beat me
17:26:55 <dustymabe> jbrooks: your domain name is pretty searchable :)
17:27:00 <jbrooks> :)
17:27:02 * roshi will have to try this
17:27:18 <dustymabe> so basically we are starting to bump up against f26
17:27:27 * roshi futzed with containerized openshift yesterday, it was, rough
17:27:38 <dustymabe> i'd like to discuss a few aspects of containerized kubernetes
17:28:11 <dustymabe> well containerized kube and/or possible other solutions to the same problem
17:28:46 <dustymabe> where $problem is: how do we not ship the container orchestration layer baked in atomic and let users choose their own
17:29:35 <dustymabe> so one thing that does concern me about kubernetes in Fedora in general is that we don't have a lot of involvement in Fedora for kubernetes.
17:29:47 <jbrooks> When the kube rpms are baked in the only thing you're prevented from doing is layering on other kube rpms
17:30:03 <jbrooks> Most of the run-in-a-container options don't care if kube rpms are installed
17:30:11 <dustymabe> jbrooks: agreed
17:30:13 <jbrooks> So it's a savings of space
17:30:50 <dustymabe> so Red Hat as a whole does have a lot of involvement in kubernetes upstream and in openshift origin upstream
17:31:23 <dustymabe> for openshift origin they produce their own "container images for origin" and we consume those from docker hub
17:31:32 <dustymabe> when we run, for example, the openshift ansible installer
17:31:52 <jbrooks> Mmm, I tried to install origin-clients the other day w/ layering and kubernetes-client conflicted w/ that
17:32:12 <jbrooks> for oc cluster up
17:32:17 <dustymabe> jbrooks: right, which wouldn't be a problem if kube was not there
17:32:22 <jbrooks> yeah
17:33:00 <dustymabe> so - concern - it seems like the origin community produces its own artifacts and doesn't use or test what we put in fedora
17:33:21 <dustymabe> the origin community also participates in kubernetes upstream
17:33:35 <dustymabe> but not necessarily in the kube that we have in Fedora
17:34:07 <dustymabe> this is a roundabout way of saying, are we concerned that basically the people using kube in Fedora seem to be limited to this WG
17:34:35 <jbrooks> That no one outside of this WG uses it?
17:34:36 <dustymabe> i.e. this WG and the users of atomic host, not the people who are developing on it
17:34:51 <dustymabe> jbrooks: no, sorry. meant to say this WG and the users of this WG
17:35:02 <jbrooks> Well, fedora's kube is straight from upstream
17:35:26 <jbrooks> but I do get the impression that it's more common to just grab binaries built by the upstream and use those
17:35:29 <dustymabe> jbrooks: right, but the problem there is that when we have issues with kube we don't have much expertise that we can rely on
17:36:00 <dustymabe> basically we just repackage kube, but we don't have anyone with development expertise in kube that is participating in Fedora
17:36:13 <maxamillion> as a point of reference ... I've actually never used "Vanilla kube", I only use origin (to be fair, it's because I'm lazy and I like the openshift-ansible installer and the added features of openshift)
17:36:24 <dustymabe> so when the etcd high cpu load bug comes to us, we don't really have anyone to lean on
17:36:32 <jbrooks> Similar to docker, the kernel, etcd, and so on
17:36:39 <maxamillion> I don't have much interest is plain kube, OpenShift adds the things I find useful about it
17:37:04 <jbrooks> Abstractly, I would be fine just caring about openshift
17:37:08 <dustymabe> jbrooks: we do have docker expertise
17:37:19 <jbrooks> in the wg?
17:37:30 <dustymabe> dwalsh and runcom and lsm5 all participate there and they put the packages into fedora
17:37:43 <jbrooks> But I see keeping the link between openshift and kube vital as strategically important
17:37:44 <dustymabe> jbrooks: not exactly in the WG, but outside the WG, yes
17:37:52 <walters> i'd vote to try removing it again for f26 now that package layering works for it
17:38:00 <jbrooks> We do have people we can talk to about kube, for sure
17:38:09 <dustymabe> jbrooks: we (red hat) do
17:38:15 <miabbott> walters +1
17:38:18 <jbrooks> Yeah, just like w/ docker
17:38:19 <walters> well, except the upstream kube rpms going in /opt...
17:38:20 <dustymabe> the problem is that they don't care as much because that kube isn't something they deliver
17:38:30 <jbrooks> Right
17:38:34 <dustymabe> they don't use it
17:38:43 <maxamillion> walters: +1
17:38:49 <miabbott> rhel technically still ships kube rpms
17:38:58 <dustymabe> and asking them to pull off of what they are working on to test this version of kube in fedora that may or may not be "old" to them
17:39:16 <dustymabe> ehh - i just thought I would bring it up and see what everyone thought
17:39:24 <jbrooks> Fedora's kube is always ahead of openshift now
17:39:35 <jbrooks> I'm fine w/ taking it out of the image
17:39:43 <jbrooks> You can do layering, you can do containers
17:39:45 <dustymabe> ok so that was my concern
17:39:48 <maxamillion> miabbott: oh? I thought they were removed in 7.3 for some reason
17:39:50 <jbrooks> Both work as drop in replacement
17:39:59 <jbrooks> No, it's still in rhelk
17:40:01 <jbrooks> rhel
17:40:10 <jbrooks> Only supported for single node
17:40:14 <dustymabe> another thing I wanted to bring up is rpm-ostree livefs layering that is a new feature coming
17:40:14 <miabbott> maxamillion: in rhelah, we still ship the kube client
17:40:26 <miabbott> all the other pieces are in containers
17:40:28 <dustymabe> theoretically this would allow us to install kube on system start
17:40:30 <maxamillion> miabbott: ahhh ok
17:40:34 <dustymabe> and not have to reboot
17:40:39 <jbrooks> The kubelet and proxy are also built in
17:40:42 <jbrooks> in rhelah
17:40:48 <roshi> I think perhaps part of the reason only WG gmembers use them is because we don't really have good docs on how to or why you'd want to
17:40:49 <maxamillion> dustymabe: swank
17:40:56 <miabbott> what jbrooks said  :)
17:40:57 <maxamillion> roshi: +!
17:40:59 <maxamillion> +1 *
17:41:08 <roshi> so only people with the relevant base knowledge will even *try*
17:41:12 <dustymabe> walters: can you confirm what I said?
17:41:20 <roshi> which is me, lacking the base knowledge but powering through anyways
17:41:24 <jbrooks> When is the live layering coming?
17:41:36 <jbrooks> And the containerized route, that doesn't require a reboot
17:41:41 <jbrooks> currently
17:41:55 <jbrooks> You could drop in the unit files w/ cloud-init
17:42:08 <jlebon> jbrooks: WIP is here: https://github.com/projectatomic/rpm-ostree/pull/652
17:42:24 <dustymabe> jlebon: do we know how far that is from being a real thing?
17:43:05 <jlebon> that's a question for walters -- a lot of the deps have been merged in already
17:43:26 <walters> on the order of a few weeks at most to be available as a flagged-experimental feature
17:44:03 <dustymabe> walters: jlebon as you develop on this. if you could try installing kube on f26 and see if you can get it to work "as a live layered package"
17:44:16 <dustymabe> that would help us when we finally get the livefs feature
17:44:36 <dustymabe> because that would probably be the first thing we try
17:45:29 <walters> man downloading from kojipkgs is slow here =(
17:45:39 <walters> we should stick this stuff in s3
17:46:20 <dustymabe> ok - summary: some concern about being a bit of an island for 'fedora delivered' kube and openshift, livefs package layering might help us move away from baked kube, containerized kube is coming along (ready for f26??)
17:46:38 <dustymabe> jbrooks: i know you are carrying quite a fork of kubernetes ansible repo
17:46:53 <dustymabe> jbrooks: what are your thoughts on "how ready" containerized kube is for f26 ?
17:47:20 <dustymabe> #info containerized kube summary: some concern about being a bit of an island for 'fedora delivered' kube and openshift, livefs package layering might help us move away from baked kube, containerized kube is coming along (ready for f26??)
17:47:31 <jbrooks> dustymabe, for just the kube part, it's super simple, if you can handle dropping in the unit files yourself, you don't need to mod the ansible at all
17:47:38 <jbrooks> the etcd and flannel bits are trickier
17:48:03 <dustymabe> jbrooks: so do you propose we keep etcd and flannel in the image?
17:48:23 <jbrooks> dustymabe, I think that'd be best
17:48:48 <jbrooks> But
17:49:03 <jbrooks> If we're ok w/ requiring a reboot to use package layering...
17:49:16 <jbrooks> That's a pretty easy route
17:49:33 <dustymabe> jbrooks: or not requiring one at all with livefs package layering ?
17:49:38 <jbrooks> It's like step one: rpm-ostree install foo bar -r step two, proceed as before
17:49:50 <jbrooks> Down the road, that'll be even better
17:50:07 <dustymabe> jbrooks: since walters and jlebon are awesome i think we'll have it for f26 :)
17:50:11 <jbrooks> The issue that prevented kube-apiserver from being layer-able is fixed
17:51:41 <dustymabe> yzhang: are you interested in taking part of this effort
17:51:42 <dustymabe> ?
17:52:12 <roshi> dumb question, but does this play into the FOSP at all?
17:52:29 <dustymabe> especially since you have done some work eith etcd and flannel system containers, might be good to review some of the work jbrooks has done in his blog post
17:52:36 <yzhang> Sure
17:52:55 <yzhang> Just that I'm not very familiar with containerized kube
17:53:02 <yzhang> or I guess kube in general
17:53:02 <jbrooks> yzhang, I have some thoughts on how it might be made more seamless I'd like to bounce off you
17:53:14 <jbrooks> We can ignore that part ;)
17:53:16 <dustymabe> #action yzhang to look at jbrooks' work on containerized kube and help polish it https://jebpages.com/2017/03/17/test-containerized-kube-and-system-container-based-flannel-and-etcd/
17:53:26 <yzhang> Ok, ping me anytime jbrooks
17:53:33 <dustymabe> did I spell polish right?
17:53:39 <yzhang> Will take a more in depth look at that post
17:53:41 <yzhang> yes
17:53:45 <dustymabe> or is that polish like poland
17:53:47 <walters> roshi, to the best of my knowledge, the FOSP work is not using Atomic Host
17:53:48 <jbrooks> like the sausage
17:53:49 <yzhang> same word
17:53:54 <yzhang> Different caps
17:54:00 <yzhang> much like labels
17:54:06 <dustymabe> walters: my understanding is that it would use atomic host to run openshift on it
17:54:18 <dustymabe> after all the Atomic WG is the ones that came up with FOSP
17:54:30 <dustymabe> wouldn't make sense if it didn't, right?
17:54:57 <dustymabe> ok thanks for taking part in that discussion everyone
17:55:02 <dustymabe> i'm not going to run through tickets today
17:55:04 <walters> one would think so, but again my understanding is that the previous work didn't
17:55:19 <dustymabe> but if there was something you wanted to discuss there is open floor
17:55:24 <dustymabe> #topic open floor
17:55:37 <dustymabe> take it away!
17:55:53 <walters> anyways i forgot, we already don't ship kube in f26 today, and `rpm-ostree install origin-clients` works fine
17:56:27 <walters> the origin-clients -> git -> perl dependency is a bit unfortunate
17:56:32 <dustymabe> walters: right - we have to firm up our stance on kube in tree or out of tree for f26 soon
17:57:03 <jbrooks> dustymabe, I would have been just fine leaving it out if layering had worked for kube-master at the time
17:57:11 <dustymabe> if livefs comes soon then we can really offer users either containerized kube OR just install the rpms like they were in the past
17:57:28 <jbrooks> And we can do that now + one reboot, which isn't the end of the world
17:57:44 <jlebon> but containerized kube should be the recommended approach right?
17:57:49 <dustymabe> jbrooks: yeah, but the people in the cloud really don't want that one reboot :) - i'm one of them
17:58:05 <dustymabe> jlebon: sure - i don't disagree with that
17:58:50 <dustymabe> jlebon: it's a lot easier to sell a change though, if you offer people to "stay where they were in the past" with just a little modification
17:59:05 <dustymabe> ok, any other open floor items?
17:59:12 <jbrooks> I actually could turn these kube containers into system containers, and we'd lose the req to drop in a unit file
17:59:50 <dustymabe> jbrooks: could you run them as a system container or as a regular container ?
17:59:59 <jbrooks> I think so
18:00:05 <dustymabe> jbrooks: yzhang can help with that
18:00:08 <jlebon> dustymabe: what i mean is, if the current recommended workflow to use the RPMS, then let's stick with that until we're ok recommending  the containerized approach
18:00:19 <jlebon> rather  than changing people's approach twice
18:00:19 <yzhang> can do
18:00:39 <dustymabe> jlebon: ehh, i prefer to give people options and let them give us feedback
18:01:12 <dustymabe> i could be wrong?
18:01:33 <dustymabe> ok time is up
18:01:38 <dustymabe> going to wind this down
18:01:41 <dustymabe> 3...
18:01:47 <dustymabe> 2...
18:01:50 <dustymabe> 1...
18:01:54 <dustymabe> #endmeeting