15:00:48 #startmeeting RDO meeting - 2016-09-14 15:00:48 Meeting started Wed Sep 14 15:00:48 2016 UTC. The chair is chandankumar. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:48 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:00:48 The meeting name has been set to 'rdo_meeting_-_2016-09-14' 15:00:48 Meeting started Wed Sep 14 15:00:48 2016 UTC and is due to finish in 60 minutes. The chair is chandankumar. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:49 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:50 yes 15:00:51 The meeting name has been set to 'rdo_meeting___2016_09_14' 15:00:57 #topic roll call 15:00:59 \o/ 15:01:09 bot clash! 15:01:12 o/ 15:01:20 o/ 15:01:21 \o 15:01:26 o/ 15:01:28 o/ 15:01:36 o/ 15:01:37 damn, I'm not an op currently 15:01:50 #chair apevec jpena amoralej dmsimard DrBacchus flepied hrybacki number80 15:01:50 Current chairs: DrBacchus amoralej apevec chandankumar dmsimard flepied hrybacki jpena number80 15:01:57 o/ 15:02:02 #chair trown 15:02:02 Current chairs: DrBacchus amoralej apevec chandankumar dmsimard flepied hrybacki jpena number80 trown 15:02:35 jruzicka: meeting time 15:02:39 o/ 15:02:54 nan mais ça reshi 15:03:01 #chair jruzicka 15:03:01 Current chairs: DrBacchus amoralej apevec chandankumar dmsimard flepied hrybacki jpena jruzicka number80 trown 15:03:07 argh 15:03:08 hi 15:03:12 o/ 15:03:18 #chair jrist 15:03:18 Current chairs: DrBacchus amoralej apevec chandankumar dmsimard flepied hrybacki jpena jrist jruzicka number80 trown 15:03:26 so starting with first topic 15:03:37 #topic Infrastructure concerns 15:03:44 Duck: please go ahead. 15:04:00 quack 15:04:02 #chair Duck 15:04:03 Current chairs: DrBacchus Duck amoralej apevec chandankumar dmsimard flepied hrybacki jpena jrist jruzicka number80 trown 15:04:06 o/ 15:04:13 #chair coolsvap 15:04:13 Current chairs: DrBacchus Duck amoralej apevec chandankumar coolsvap dmsimard flepied hrybacki jpena jrist jruzicka number80 trown 15:04:51 o/ 15:04:56 #chair jschlueter 15:04:56 Current chairs: DrBacchus Duck amoralej apevec chandankumar coolsvap dmsimard flepied hrybacki jpena jrist jruzicka jschlueter number80 trown 15:05:49 Duck has some infrastructure concerns for for seperate mailing lists and dedicated meeting. 15:06:30 Merged openstack/UcsSdk-distgit: Use pypi.io for Source0 URL http://review.rdoproject.org/r/2284 15:06:43 well, more generally, on how to help on the infra 15:06:54 [infra] tag on rdo-list would not be good enough? 15:06:59 and having a proper channel to discuss would be nice 15:07:14 apevec: it is not used, as I know of 15:07:15 let's keep all on #rdo 15:07:28 I think this was brought up before, I think it's appropriate to use rdo-list (with [infra] tag perhaps) and it's fine to keep discussions in #rdo as far as I'm concerned. Don't be shy :) 15:07:31 yeah, let's start using it to help easier filtering 15:07:44 when the certificate expiration happened, twice, there was nothing on the ML 15:07:59 there sensu checks 15:08:11 I'm fine with it but it does NOT work 15:08:12 you mean would should send email alerts from those? 15:08:19 Duck: there actually was an email on the ML :) 15:08:25 also this meeting never has any infra topics 15:08:36 https://www.redhat.com/archives/rdo-list/2016-August/msg00237.html 15:08:41 which would mean everything is perfect but… I tend to not believe it 15:08:44 we have one today about CBS outage :) 15:08:52 It didn't use the specified tag. 15:09:10 dmsimard: without the tag => filtered out 15:09:17 Duck: the meeting tends to be regarding topics of discussions, not updates on the status of the infrastructure 15:09:22 so there is de facto no mail to wrn 15:09:23 The difficulty being that Duck and misc are responsible for infra across many projects, and are unlikely to see these messages unless they are appropriately flagged. 15:09:27 shall we document list of "official" tags ? 15:09:52 apevec: it's hard to enforce, even on openstack-dev there are untagged emails :) 15:10:06 yeah, but we can try 15:10:08 the concern is also that infrastructure is all over the place 15:10:19 and we didn't followed up on the rdo-infra plan :/ 15:10:40 misc: did you mention you were coming to Barcelona ? 15:10:54 It could be a great opportunity to have work sessions 15:10:54 dmsimard: yep 15:11:01 dmsimard: and yep 15:11:12 While it's understood that people will spin up stuff as needed, we want to ensure that those services are supported long-term, and that requires access, documentation, and automation. 15:11:15 so with rbowen we began to list the resources, where they are, and who is responsible for 15:11:18 but I think duck and I have different concerns :) 15:11:27 if folks arrive early on Monday, let's plan rdo infra meetup then? 15:11:31 would be nice to ensure it is complete 15:11:43 and there is no SPOF 15:11:46 Duck, DrBacchus: I believe jpena has an inventory somewhere 15:11:50 That doc is here: https://docs.google.com/spreadsheets/u/1/d/1cpTHMixEiaqHA-qR-ol2uO76USYna08WrTvUJPCxHSI/edit#gid=0 15:11:51 of the hosts 15:11:52 and a todolist of things so we could pick some 15:11:55 etc etc 15:11:57 Oh, ok, well, we should merge those things, then. 15:12:03 DrBacchus: that document isn't public 15:12:33 Yeah, at the moment it's invite-only but I can move that to somewhere public. 15:12:43 DrBacchus: It's ok I'll request access 15:12:51 There's also https://www.rdoproject.org/infra/ which is probably not up to date either. 15:12:55 another concern is also that we have no idea on who do what :) 15:13:24 let's document that all in a single doc 15:13:25 misc: it's a fair assumption that jpena and I are responsable/accountable for most of the infrastructure 15:13:30 so there's a few things OSAS is involved into, like web stuff 15:13:44 the goal is to have more things public and more people involved too 15:13:59 so no take over, no blocking people, etc 15:14:14 hguemar created openstack/neutron-lib-distgit: Sync requirements http://review.rdoproject.org/r/2292 15:14:59 ok, problem statement is clear, let's get actions 15:15:21 hguemar created openstack/neutron-lib-distgit: Create rpm-mitaka branch http://review.rdoproject.org/r/2293 15:15:22 feel free to add actions against yourself. :-) 15:15:27 So, action items are 1) update https://www.rdoproject.org/infra/ with all the services, and who is responsible for each 15:15:39 apevec: I'm +1 for a work session on monday in BCN 15:15:51 +1 15:16:00 #chair rbowen 15:16:00 Current chairs: DrBacchus Duck amoralej apevec chandankumar coolsvap dmsimard flepied hrybacki jpena jrist jruzicka jschlueter number80 rbowen trown 15:16:06 It should be productive and put us all on the same page 15:16:07 That seems a reasonable place to start, as that is the biggest hole at the moment, it seems? 15:16:12 BCN? 15:16:17 barcelona 15:16:18 barcelona 15:16:18 Barcelona IATA code 15:16:22 ho 15:16:34 (cause RH use IATA code for whatever historical reason) 15:16:34 so misc you'll be my representative then 15:16:39 rbowen: will i add that action item to you? 15:16:44 +1 to work session, but I'm arriving to BCN around 11am 15:16:47 we meet at the airport ? ;-) 15:16:51 misc: with all my concerns too 15:17:05 Duck: sure 15:17:06 chandankumar: Sure, I can add the info that I have. Also need jpena to crosscheck and update. 15:17:07 rbowen, Duck, misc: Are we comfortable with starting with that ? BCN is in a month. 15:17:10 flepied: just redhatter internal slang :) 15:17:14 dmsimard: I am ok 15:17:39 #action rbowen to update https://www.rdoproject.org/infra/ with services and responsibilities, then hand off to jpena for crosscheck. 15:17:46 we can work on documents in the meantime 15:18:13 rbowen: ack 15:18:13 next topic? 15:18:16 dmsimard: well, probably better to have some live to work better. May be able to join via Blue Jeans. 15:18:21 o/ 15:18:26 yes, let's clarify the doc 15:18:28 #chair imcsk8 15:18:28 Current chairs: DrBacchus Duck amoralej apevec chandankumar coolsvap dmsimard flepied hrybacki imcsk8 jpena jrist jruzicka jschlueter number80 rbowen trown 15:18:34 Duck: yeah sure, we can have a hangouts or bluejeans 15:19:04 dmsimard: could you do an invitation when exact timing is determined? 15:19:06 #action Schedule a work session for RDO Infrastructure sometime, somewhere monday afternoon 15:19:06 dmsimard: we likely need to go to the office, no ? 15:19:16 cause I do not trust network to work :) 15:19:19 misc: ah, there is an office in bcn? We could do it there 15:19:27 how far is it from the conference center ? 15:19:29 dmsimard: yep, there's an office 15:19:40 a brand new one :) 15:19:42 misc: you need to see, battlle with, and drink with people live 15:19:43 #undo 15:19:43 Removing item from minutes: ACTION by dmsimard at 15:19:06 : Schedule a work session for RDO Infrastructure sometime, somewhere monday afternoon 15:19:45 misc: ack for office, but we need to take care of unbadged people who wants to join us 15:19:50 There may also be space/time at the event venue. I need to check with Jmadriaga on that. Unless we prefer the office. 15:20:00 #action Schedule a work session for RDO Infrastructure sometime at the BCN Red Hat office 15:20:07 rbowen, I'd prefer conf 15:20:13 bah 15:20:14 dmsimard: not too far 15:20:16 rbowen: that'd be fantastic if we get space at conference site 15:20:20 #undo 15:20:20 Removing item from minutes: ACTION by dmsimard at 15:20:00 : Schedule a work session for RDO Infrastructure sometime at the BCN Red Hat office 15:20:32 ok, I'll ask and see what there is. Should be easier Monday than actually during the event. 15:20:32 #action Schedule a work session for RDO Infrastructure somewhere monday afternoon 15:20:44 3rd time is a charm 15:21:08 move on? 15:21:15 #topic phase out rpm-xxx branches 15:21:24 number80: you? 15:22:12 rpm-RELEASE to be precise 15:22:14 rbowen: thanks 15:22:15 rpm-master stays :) 15:22:39 yes 15:22:57 related card is https://trello.com/c/URAtrhLU/86-automate-stable-packages-releases 15:23:06 as soon as I'll finish branching newton, I'll submit a review to drop rpm-RELEASE branches 15:23:11 we're starting this with Newton 15:23:17 Yes 15:23:24 no rpm-newton 15:23:37 are we removing rpm-mitaka and earlier? 15:23:43 ideally yes 15:23:49 do you see issues with migration? 15:23:59 jpena: it will need testing to avoid storming DLRN 15:24:03 apevec: ^ 15:24:09 hash are differents 15:24:15 that requires DLRN/puppet-dlrn support, just need to know in advance to update DLRN workers and fix any errors 15:24:41 actually if keeping rpm-mitaka is preferred, that's an option 15:24:43 we have a bit of backlog 15:24:45 number80, can't avoid that, we'll have to let full rebuild 15:25:12 but yeah if we don't have DLRN resource to process that quickly better leave mitaka 15:25:19 #info do not merge any rdoinfo review removing rpm-RELEASE branches before checking w/ jpena 15:25:30 please :) 15:25:51 trown created rdoinfo: Add DIB and python-tripleoclient to tripleo UC exceptions http://review.rdoproject.org/r/2294 15:25:52 number80: my plan is to update DLRN/puppet-dlrn on the instance when we create the Ocata workers 15:26:07 that's next Friday or Monday next week 15:26:14 (I guess) 15:26:21 jpena: ack, I'll submit review and will let you merge it 15:26:21 yes, RC1 is branching point 15:26:45 yeah newton branching status 15:26:51 clients are all done 15:26:53 #link https://trello.com/c/zT4BghsV/366-branching-newton-rdo 15:27:18 services are starting today, I identified few ones that could be done in // by someone else ^ 15:27:28 (I'll add more later) 15:27:29 i'd like to help on that 15:27:36 \o/ 15:27:37 yes, me and amoralej will join 15:27:48 but i'll need some additional permissions in review.r.o 15:28:01 just need to split the checklist among us 15:28:09 amoralej, I'll look at that after meeting 15:28:21 ok apevec, let's sync after the meeting 15:28:27 #action apevec to check branching permissions for amoralej 15:28:43 as for mitaka, I'm checking that in parallel of newton branching 15:29:12 so as follow-up, I have some fedora bundled updates to do + mass-tagging 15:29:25 mass-tagging will happen after running weirdo generic jobs 15:29:37 apevec, amoralej i would like to help with branchin too 15:29:59 imcsk8, ack - let's first see how initial split goes 15:30:12 we might have so other followup tasks besides branching 15:30:19 yes 15:30:43 number80: btw, I live in Mitaka :-) 15:30:52 number80: Mitaka-shi 15:30:54 apevec: ok, i'll be standing by 15:30:55 #info number80 is checking Mitaka in parallel of newton branching 15:31:06 next topic? 15:31:10 yes 15:31:16 #topic python-shade packaging 15:31:30 Merged openstack/glance_store-distgit: Use tarballs.o.o for Source0 URL http://review.rdoproject.org/r/2277 15:31:54 I think it goes under release independet repo rdo-tools 15:32:07 yes, it was proposed to put it in EPEL 15:32:24 but I sugget to develop it first in new rdo-tools (coming soon) repo 15:32:44 which was discussed on rdo-list and one of the previous meetings 15:32:51 apevec: yes, 15:33:05 up to what time new rdo-tools repo is coming? 15:33:08 apevec: can you take care of repo creation w/ CentOS? 15:33:13 chandankumar: asap 15:33:17 it's lower prio due to newton wrapup 15:33:25 but I'll start the process w/ tags in CBS 15:33:38 #action apevec to request CBS tags for rdo-tools repo 15:33:58 related, I've already requested Ocata CBS tags 15:34:20 so we'll start with ocata-testing for deps on Trunk master 15:34:24 jpena, ^ FYI 15:34:38 apevec: ack, the puppet-dlrn update for that is already updated 15:34:59 Anything more on this, if not then next topic? 15:35:07 next 15:35:14 #topic [Ci-users] [Infra] - Major outage for several services 15:35:22 thank 15:35:24 https://lists.centos.org/pipermail/ci-users/2016-September/000392.html 15:35:29 * Arrfab is here 15:35:31 please read it 15:35:41 Arrfab: dmsimard please go ahead 15:35:44 so D-Day is Monday Oct 10 15:36:03 plan is to have Newton GA repo pushed before that 15:36:14 #chair Arrfab 15:36:14 Current chairs: Arrfab DrBacchus Duck amoralej apevec chandankumar coolsvap dmsimard flepied hrybacki imcsk8 jpena jrist jruzicka jschlueter number80 rbowen trown 15:36:16 What about cycle-trailing ? 15:36:17 Arrfab, just to confirm buildlogs.c.o and mirror.c.o are not affected ? 15:36:40 Outage for: CBS, ci.centos.org, internal DLRN build server 15:36:40 dmsimard, trown is optimistic 3o will be ready Oct 6 15:36:43 apevec: they aren't 15:36:59 EmilienM: is ^ totally off base? 15:37:02 that is quite optimistic :) 15:37:18 considering puppet-openstack typically doesn't release the same day as GA 15:37:22 apevec: so last chance to see pkgs landing on either buildlogs or mirror.centos.or is monday 10th in the morning (CET time) 15:37:27 remember that number80's goal from Austin meetup is to release RDO minutes after upstream announcement :) 15:37:39 Arrfab, ack 15:37:56 dmsimard: true that you mentioned some "sub-components" that will probalby only appear in the days after GA 15:37:57 dmsimard: we do 15:38:03 dmsimard: nothing new about this ? 15:38:11 we released Mitaka on GA and will release newton on GA 15:38:12 ack on not available Oct 10 15:38:17 EmilienM: you do ? I guess I'm thinking of something else 15:38:18 only TripleO might slip a bit 15:38:24 I understand the internal DLRN build server is affected? 15:38:27 jpena: yes 15:38:28 we have max 2 weeks to release final 15:38:43 so no builds during these 2 days unless we have a fallback 15:38:49 jpena: it doesn't last two days 15:39:01 jpena: We're looking at an optimistic 8 hours of downtime on the 10th 15:39:02 Yeah, I like M-I style goals :D 15:39:13 dmsimard: I thought it was 48 15:39:21 trown: you probably read 4-8 15:39:25 right quaid ^ ? 15:39:30 dmsimard: I wouldn't count on such a small downtime though (even if we want to minimize that) 15:39:41 The mail says "the announced downtime will be probably ~48h." 15:39:43 it's 48h 15:39:57 trown: correct : that's the downtime announced by RH IT to move and power back the racks 15:40:04 Arrfab: hrm, I think I saw quaid mention 4-8 hours. Must have misread that :( 15:40:21 so, should we kick a temp build server just in case we have an emergency for centos-newton? 15:40:37 jpena: either that or expedite the nodepool setup 15:41:10 jpena: if that's critical for you/rdo, wondering if we want to postpone the migration (but all the RH people will be angry about this, again) 15:41:19 dmsimard, jpena yeah, good opportunity to move to new arch :) 15:41:33 Arrfab, no, let do it 15:41:37 Arrfab, no need to postpone 15:41:43 there's never good time 15:41:55 Ocata is already starting 15:42:01 ok, as said, we'll work hard with all the involved people to get cbs.centos.org back online as first service 15:42:01 jpena: I think if we make the nodepool idea a priority it's realistic to have it live before oct 10th, I don't believe there is a lot of work involved -- most of the pre-reqs are already in place ? 15:42:02 CD is 24x7x365 15:42:15 and CI being the second one (in parallel) 15:42:34 Arrfab, ack for that prio list 15:42:36 trown proposed rdoinfo: Add DIB to tripleo UC exceptions http://review.rdoproject.org/r/2294 15:42:49 dmsimard: just curious : what's that "nodepool" thing ? 15:43:08 Arrfab: we want to decentralize the package building to help us scale and build things faster 15:43:32 dmsimard: I was saying what I'd wanted not what we had, sorry for the confusion 15:43:35 dmsimard: my only concern is where we can start with it (and how we can connect the nodepool instances with the "central" DLRN server), but yes, let's go for it 15:43:50 Arrfab: as we package more and more projects, and there are more and more commits (and we package every commit), one single build server is no longer enough 15:43:56 dmsimard: oh, ok, but so those other pkgs will never appear on the centos infra either ... 15:44:15 Arrfab: nodepool is what we use in review.rdoproject.org to dispatch builds onto ephemeral VMs (same thing as upstream openstack-infra) 15:44:45 dmsimard: so a kind of "pre-build" before submitting a pkg to cbs.centos.org ? 15:44:49 Arrfab, those are Trunk rpms which we push to http://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-master-tested/ 15:45:03 it's build on each upstream commit 15:45:14 push to buildlogs is after passing CI 15:45:28 apevec: yeah, but that I mean is that if you built on something else (external builders), that will not appear on buildlogs either 15:45:41 apevec: *ack* 15:45:52 that's a challenge that jpena and dmsimard are solving 15:46:23 action items here? 15:46:32 jpena, dmsimard : let's discuss that when you have a minute 15:46:50 #action dmsimard and jpena to start working towards moving dlrn package building to review.rdoproject.org (nodepool) on post-commit upstream pipeline 15:46:57 from my side, I so consider that we keep the announced date for HW migration 15:47:06 yes 15:47:31 next topic? 15:47:32 worst case, if the package building is not ready when we're close to the deadline, we can spawn a temp builder 15:48:12 yes, have a planB 15:48:28 jpena: well, as said, up2you, but don't expect those pkgs to land on centos infra then (as long as everybody is aware, that's fine) 15:48:58 Arrfab: understood. After the DLRN instance is back, we can just copy stuff back to it 15:50:30 let's move on, we have few more topic 15:50:35 #topic exceptions for upper-constraints based repo for tripleo gated projects 15:50:40 trown: please go ahead 15:50:41 this is mine 15:50:53 please do not use master-head ! 15:50:54 this is pretty much resolved in the etherpad 15:51:12 I did not realize there was a list of exceptions for tripleo to upper-constraints 15:51:14 we just need a list which projects should be unpinned 15:51:17 put up a patch to add DIB http://review.rdoproject.org/r/2294 15:51:22 EOL 15:51:32 trown: so fast! 15:51:32 cool, +2 15:51:52 moving to next item if no action item? 15:51:58 yep 15:52:12 #topic What should we improve for test days? 15:52:17 rbowen: please go ahead 15:52:35 some suggestions from etherpad: 15:52:37 Well, I didn't create this topic, but, yeah, it's something that we need to do better. 15:52:46 There are some good suggestions in there. 15:52:49 * Test setup matrix cleanup 15:53:00 * More detailed, beginner-friendly instructions for testing. (Test day as product preview for potential users.) 15:53:08 I would like test day to bringing potential new users to us, and be a way for people to see what's coming in the future. 15:53:09 color says it was flepied 15:53:11 * ask other teams to provide use cases they want to validate 15:53:18 That's hard at the moment, because the instructions assume you already know a lot. 15:53:27 * only on single etherpad with instructions and registering people to use cases 15:53:28 yes it was me as a feedback from doing the last ones 15:53:36 This is perhaps something that pkovar and the other docs folks can help with? Or QE? 15:53:50 chandankumar: +1 for registering people 15:53:51 I think people show up, but they're not sure how to get started. 15:54:01 we also kind of need new folks to take on these actions, as just getting stuff testable consumes much of existing pre-testday effort 15:54:03 would test scripts help? 15:54:15 Test scripts could be a useful thing, sure. 15:54:24 imcsk8: +1 for different test setup 15:54:41 trown: Yeah, folks spend most of test day just standing it up, and then we're out of time. 15:54:54 what about including tempest run on each of the setup? 15:55:47 i think test days are more about user experience 15:56:30 Any action items from here? 15:56:34 imcsk8: agreed. we should target to reproduce what a typical user will experiment when installing RDO 15:56:40 there could be more promotion for test days amond the community 15:56:56 * chandankumar reminds we have 4 mins left. 15:57:06 Perhaps we can take this back to the mailing list, then. 15:57:11 rbowen: +1 15:57:16 but then, it'd be useful to let users provide real use cases instead of follow some of them provided by us 15:57:16 +1 15:57:18 +1 15:57:18 We have a lot of actions, but unclear who will actually do them.' 15:57:20 hguemar proposed openstack/neutron-distgit: Add %upstream_version fallback macros http://review.rdoproject.org/r/2291 15:57:21 hguemar created openstack/neutron-distgit: Sync requirements http://review.rdoproject.org/r/2295 15:57:40 rbowen: will you do that? 15:57:45 Yes, I will. 15:57:57 #action rbowen to raise topic of improving test day experience on rdo-list 15:58:05 #topic Announcements 15:58:20 #info Demos/Q&A at RDO booth at OpenStack Summit Barcelona - signup at https://etherpad.openstack.org/p/rdo-barcelona-summit-booth 15:58:34 #info We are planning an RDO Community Meetup, in conjunction with the Ceph community, at OpenStack Summit. Tuesday night, October 25th, 5pm-8pm. Agenda is at https://etherpad.openstack.org/p/rdo-barcelona-meetup-schedule 15:58:45 #info Newton RC1 test day September 22, 23 - Details coming soon to https://www.rdoproject.org/testday/ 15:59:16 #info New tempest rpms are available now here https://trunk.rdoproject.org/centos7-master/consistent/ Feel free to try out 15:59:37 #info openstack-sahara-tests is available in RDO now https://review.rdoproject.org/r/#/q/topic:add-sahara-tests 15:59:51 #topic chair for next meeting 16:00:01 i can do it 16:00:01 Anyone up for that? 16:00:13 #action imcsk8 will chair for next meeting 16:00:19 #topic open floor 16:00:41 question about promotion job in CI 16:00:44 apevec: please have a look on python-heat-tests-tempest package https://review.rdoproject.org/r/#/c/1980/ 16:01:07 scheduling is now every 12 hours instead of 4 hours 16:01:18 i have tested it, works fine. 16:01:20 amoralej: only for stable pipelines 16:01:33 it's 12 hours for master now 16:01:33 * chandankumar reminds we are in last minute. 16:01:39 if i checked it right 16:02:01 could we put it back to 4 hours? 16:02:14 yeah, please 16:02:20 amoralej: hmm you are right... not sure when that changed... it should definitely be 4 16:02:34 amoralej: will i end the meeting, we are running out of time? 16:02:45 ok, thanks trown 16:02:46 chandankumar: ya go for it, we can discuss after if needed 16:02:50 #endmeeting