15:00:41 <jruzicka> #startmeeting RDO meeting - 2016-08-03
15:00:41 <zodbot> Meeting started Wed Aug  3 15:00:41 2016 UTC.  The chair is jruzicka. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:41 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:41 <zodbot> The meeting name has been set to 'rdo_meeting_-_2016-08-03'
15:00:42 <openstack> Meeting started Wed Aug  3 15:00:41 2016 UTC and is due to finish in 60 minutes.  The chair is jruzicka. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:43 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:46 <openstack> The meeting name has been set to 'rdo_meeting___2016_08_03'
15:00:48 <galiral> hey jrist
15:00:53 * jrist waves
15:00:54 <galiral> usually on the controller
15:01:02 <galiral> what service are installed
15:01:12 <jruzicka> double meetbot for double victory
15:01:13 <tosky> hi
15:01:24 <number80> o/
15:01:27 <tosky> the agenda item is me :)
15:01:31 <imcsk8> o/
15:01:32 <social> o/
15:01:42 <jpena> o/
15:01:50 <rbowen> o/
15:02:01 <jruzicka> #chair tosky number80 imcsk8 social jpena rbowen
15:02:01 <zodbot> Current chairs: imcsk8 jpena jruzicka number80 rbowen social tosky
15:02:23 <jruzicka> #topic jar and exceptions for openstack-sahara-tests (https://bugzilla.redhat.com/show_bug.cgi?id=1318765)
15:02:30 <coolsvap> o/
15:02:39 <galiral> CONFIG_NOVA_INSTALL=
15:02:56 <galiral> let me get it right, it is referring to the nova apis or the compute node?
15:02:57 <jruzicka> galiral, please wait until the meeting is over which is soon :)
15:02:59 <dmsimard> \o
15:03:02 <galiral> sorry
15:03:12 <jruzicka> #chair coolsvap dmsimard
15:03:12 <zodbot> Current chairs: coolsvap dmsimard imcsk8 jpena jruzicka number80 rbowen social tosky
15:03:45 <jruzicka> hmm, bot doesn't comment on the topic... well go on anyway, tosky ;)
15:03:49 <tosky> so, the topic was partially discussed in the previous meetings but I could not attend
15:04:24 <tosky> basically: there are some jar (yeah, I know) which used to live in sahara (and some of them are still there) and now in sahara-tests (used for testing)
15:04:52 <tosky> if I'm not mistaken we shipped it in the past, I know all the issues about shipping binaries and so on, so my point here is:
15:05:16 <tosky> - I would like to ask an exception to have the binaries shipped as they are (they are mostly examples) in sahara-tests
15:05:43 <tosky> - I would work upstream for fixing this (the sources are mostly in sahara-extras) but most likely there is no time for N, more likely for O
15:05:49 <tosky> and I would need guidance on that
15:06:00 <tosky> how to organize things in the way that they could be accepted
15:06:22 <tosky> so I'm asking for an exception so that openstack-sahara-tests could go still potentially in mitaka
15:06:28 <tosky> </eom>
15:06:41 * social does nog like bundled jars
15:06:44 <social> *not
15:06:52 <jruzicka> noone does ;)
15:07:00 <number80> I'm ok with it on the principle as long as it IS DOCUMENTED IN THE SPEC
15:07:31 <jpena> also, if the source location is known, we could add at least a pointer to them somewhere in the spec
15:07:39 <number80> apevec was doing the review, so I'd like to get his opinion before we vote
15:07:49 <number80> jpena: good point
15:07:55 <tosky> I don't like them either, but it's not so simple to untangle, also because... java
15:08:27 <number80> true, one of the things I'd do is create a trello card to follow that with an actual deadline
15:08:59 <tosky> what I'd need, as I said, are best example on how to deal with such cases (I guess we have other mixed packages in java)
15:09:06 <tosky> or packages with java parts
15:09:42 <number80> tosky: well, it will depend on a lot of factors, if we get MEAD in CBS, it may get simpler
15:09:56 <number80> (MEAD is not universal solution but it's one of them)
15:10:36 <number80> I suggest since apevec is likely on a call, to do informal vote
15:10:38 <tosky> so, on the upstream side, just tell me what I should provide so that, for any possible solution implemented in packaging (I will ask zigo too), things don't become too complicated
15:11:10 * apevec out of call, reads back
15:11:12 <number80> proposal: grant sahara-tests bundling exceptions and track progress on unbundling jars
15:11:19 <number80> good
15:11:37 <number80> at least, reviewer should have a say before final decision :)
15:12:00 <apevec> yes, that's what I wanted to do, collect all source then ask for exception, but didn't get to it yet
15:12:19 <apevec> on upstream side, it would help to provide at least READM next to each binary jar
15:12:26 <apevec> to document how was it built
15:12:42 <number80> that should be an action, do you take it tosky?
15:13:32 <tosky> number80: add a README? Yes, does it need to be really in the same directory or could it be with the global documentation?
15:13:55 <tosky> apevec: ^^
15:14:01 <number80> tosky: preferably same dir
15:14:06 <number80> (IMHO)
15:14:30 <apevec> tosky, either way, if upstream prefers to keep developer docs in one place, that's also fine
15:14:53 <tosky> ack
15:15:06 <number80> ok
15:15:23 <number80> #action tosky ask upstream to document how sahara-tests jars are built
15:15:32 <number80> so let's formally vote the proposal ^
15:15:37 <number80> +1
15:15:47 <jruzicka> +1
15:15:48 <tosky> it's more "tosky send a review to document how sahara-tests jars are built"
15:15:55 <number80> #undo
15:15:55 <zodbot> Removing item from minutes: ACTION by number80 at 15:15:23 : tosky ask upstream to document how sahara-tests jars are built
15:16:10 <number80> #action tosky send a review to document how sahara-tests jars are built
15:16:25 <number80> tosky: I'm fine with you doing the work :)
15:17:05 <apevec> let's info  the exact proposal
15:17:38 <apevec> #info  proposal: grant sahara-tests bundling exceptions and track progress on unbundling jars
15:17:50 <apevec> +1 from me
15:18:04 <number80> +1 (for minutes)
15:18:06 <jruzicka> once there is an exception, there will be no pressure to fix but OK
15:18:12 <jruzicka> +1
15:18:27 <number80> jruzicka: next action would be creating a tracking card in trello :)
15:18:29 <apevec> jruzicka, exception comes with docs
15:18:40 <social> +1
15:18:47 <jruzicka> with great docs comes great exception? :)
15:18:57 <jpena> +1
15:19:03 <number80> no, that's the exception of that quote :)
15:19:25 <number80> #agreed grant sahara-tests bundling exceptions and track progress on unbundling jars
15:19:55 <number80> #action number80 create follow-up card in trello
15:20:12 <jruzicka> #topic open floor
15:20:18 <apevec> #undo
15:20:22 <apevec> check agenda :)
15:20:27 <jruzicka> oh, new agenda :D
15:20:32 <jruzicka> never too late :-p
15:20:46 <apevec> meh, I can't undo
15:20:51 <jruzicka> #undo
15:20:51 <zodbot> Removing item from minutes: <MeetBot.items.Topic object at 0x3cdb2310>
15:21:00 <jruzicka> #topic temp CI pipeline for http://trunk.rdoproject.org/centos7-new/
15:21:13 <jruzicka> #chair apevec
15:21:13 <zodbot> Current chairs: apevec coolsvap dmsimard imcsk8 jpena jruzicka number80 rbowen social tosky
15:21:19 <tosky> jruzicka: the pressure is you coming to my cubicle
15:21:35 <apevec> that's me, I've been asking on IRC about that but no definitive answer yet
15:21:47 <jruzicka> tosky, I'm not that motivated to unbundle the jars :-p
15:22:06 <number80> jruzicka: that should be a motivation to encourage tosky doing it ;)
15:22:06 <apevec> flepied, jpena, weshay, ^ can we have one temp full CI pipeline run against that temp repo?
15:22:17 <apevec> we need that pass before switching centos7-master
15:22:39 <apevec> dmsimard suggested generic weirdo jobs, which is fine but we also need oooq
15:22:42 <apevec> at least minimal
15:22:54 <jpena> I'm not very familiar with creating CI pipelines, expected dmsimard to help 0:)
15:22:56 <apevec> weshay, other option is to run internally on RHEL
15:23:08 <dmsimard> I can look
15:23:26 <apevec> jpena, that's why I'm asking, who can take this, it's critical before we do the switch
15:23:34 <apevec> I mean, we could also just switch
15:23:48 <apevec> and related what would be good timing for the switch?
15:23:49 <apevec> flepied, ^
15:23:55 <flepied> apevec: I'm in favor of switching
15:24:03 <jpena> what would happen to the CIs that use hashed repos? They won't be available under the new url
15:24:05 <apevec> we should get pass on old centos7-master today
15:24:23 <apevec> we're already 8d behind
15:24:42 <apevec> jpena, yes, that's the price of the progress
15:24:47 <chandankumar> \o/
15:24:53 <apevec> jpena, we have it archived on buildlogs
15:25:12 <flepied> jpena: who is using a hash outside of Puppet ?
15:25:14 <apevec> jpena, but good point, we'd break puppet-ci and tripleo-ci
15:25:16 <weshay> apevec, ah.. probably best to create a temp pipeline for it
15:25:26 <apevec> weshay, that's what I proposed :)
15:25:35 <apevec> but not sure who can do it
15:25:52 <number80> flepied: maybe kolla does
15:25:53 <apevec> jpena, flepied - so we'll need to sync tripleo and p-o-i pins
15:25:53 <weshay> apevec, I'll do it
15:26:26 <flepied> apevec: agreed
15:26:27 <weshay> apevec, can you create a card in trello w/ the details
15:26:36 <jpena> apevec: ok, let's hope they're not the same as any existing hash in newton-uc
15:26:50 <apevec> jpena, hmm, good point
15:27:01 <apevec> dmsimard, ^ re. kola, do you know if they use passed-ci symlink  or exact hash like puppet ci?
15:27:01 <jpena> let me check
15:27:16 <apevec> weshay, ok, I'll fork from https://trello.com/c/guK9Ag12/157-dlrn-builds-must-reflect-what-is-tested-upstream#
15:27:16 <dmsimard> apevec: I haven't checked in a while, let me look
15:27:36 <apevec> jpena, poi has pin bump proposed
15:27:49 <dmsimard> Kolla uses the current-passed-ci cdn
15:27:51 <dmsimard> https://github.com/openstack/kolla/blob/56178a58dc1ea3f9117f4c03893ebafcf0e1f57c/kolla/common/config.py#L29à
15:27:53 <dmsimard> https://github.com/openstack/kolla/blob/56178a58dc1ea3f9117f4c03893ebafcf0e1f57c/kolla/common/config.py#L29
15:28:30 <apevec> ok, then kolla is good
15:28:38 <apevec> jpena, bump in poi is https://review.openstack.org/349155
15:28:39 <jpena> we're lucky, neither the current or proposed hashes in https://review.openstack.org/#/c/349155/11/manifests/repos.pp are in use in newton-uc
15:28:55 <apevec> cool
15:29:21 <apevec> poi bump is blocked on https://review.openstack.org/349765
15:29:33 <apevec> failed in gate :(
15:29:37 <apevec> ok, actions
15:30:00 <apevec> #action weshay to create temp CI pipeline for http://trunk.rdoproject.org/centos7-new/
15:30:13 <number80> dmsimard: thanks
15:30:43 <apevec> #action jpena to copy puppet-openstack-integration and tripleo-ci odl hashed repos to centos7-newton
15:30:47 <apevec> anything else?
15:30:59 <apevec> let's sync tomorrow and then decide when to switch
15:31:17 <jpena> apevec: that's after the temp CI pipeline gives us the ok, right?
15:31:18 <apevec> hopefully we'll have another promotion with current centos-master by then
15:31:24 <apevec> jpena, right
15:31:40 <apevec> once that passes, we switch centos7-master to centos-newton
15:31:43 <jpena> btw, for kolla we need to sync as well, they download http://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-master-tested/delorean.repo
15:31:49 <jpena> which points to a hashed repo
15:32:07 <apevec> ah ok, then we need old current-passed too
15:32:22 <apevec> which we need anyway until we get promotion with the new repo
15:32:29 <jpena> yep
15:32:40 <jpena> where's the tripleo CI repo configured?
15:33:11 <jpena> I'd assume it uses current-tripleo, but just in case
15:33:23 <weshay> jpena, apevec this may take more than a day to get a temp pipeline
15:34:00 <sdake> heard a beep on kolla - anything you need from us apevec?
15:34:17 <apevec> sdake, not for now, we'll try to NOT break you :)
15:34:32 <apevec> jpena, it is that symlink
15:34:38 <jpena> ack
15:34:46 <sdake> apevec cool - heads up - we will soon be mirroring rdo repos inside openstack infrastructure
15:34:50 <sdake> i hae a patch in the works
15:35:14 <sdake> if you break something (links/etc) we can easily repair that
15:35:26 <jruzicka> nice!
15:35:48 <sdake> but if there are link changes would be nice to know ahead of time so i can get the mirroring done correctly
15:36:06 <apevec> sdake, can you give us link to the review?
15:37:00 <apevec> we might want to rethink delorean.repo b/c it links back to trunk.rdo
15:37:16 <sdake> apevec sitting on my ssd atm, will publish later today - i'll send a link to rdo mailer when done
15:37:23 <apevec> thanks!
15:37:36 <apevec> I'm good with the topic, anything else?
15:37:38 <sdake> ya something that would break kolla is removing trunk builds all-together
15:38:01 <sdake> could you expand on what you have in mind related to rethinking delorean.repo?
15:38:29 <apevec> I'd prefer to point external users to the mirror we have on buildlogs.centos.org
15:38:41 <jpena> we have our passed-ci stuff in the centos CDN, however the delorean.repo file points back to trunk.rdoproject.org (non-CDN)
15:38:47 <apevec> http://trunk.rdoproject.org/centos7-master/current-passed-ci/ already redirects to http://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-master-tested//
15:39:03 <apevec> but delorean.repo inside still points to specific hashed repo on trunk.rdo
15:39:17 <apevec> defeating the purpose of the buildlogs CDN :)
15:39:29 <apevec> what jpena  said :)
15:39:40 <apevec> alright, but that's off-topic
15:39:48 <apevec> just related
15:39:54 <dmsimard> apevec: it's a highly available delorean.repo !
15:40:01 <apevec> lol
15:40:09 <apevec> yeah, "great success"
15:40:23 <apevec> (that's my 11yr old talk)
15:40:29 <number80> next topic?
15:40:30 <jruzicka> astounding success indeed
15:40:31 <apevec> next
15:40:32 <jruzicka> #topic Ideas to improve DLRN instance performance
15:40:35 <jruzicka> jpena, ^
15:40:56 <jpena> apevec mentioned today that there were some talks last week about how to improve the current worker's performance
15:41:28 <apevec> if someone has doubts this isn't needed check https://trunk.rdoproject.org/centos7-master/report.html
15:41:35 <jpena> I started doing some brainstorming today, and most of the time I ended up redesigning koji
15:41:40 <apevec> build time 15:33 for commit 11:01
15:41:59 <apevec> still 4h  behind after yesterday's 3h pause...
15:42:25 <apevec> jpena, that's good, I heard number80 is doing that at flock in a hallways :)
15:42:28 <jpena> I had another idea, which is to run parallel builds using some python multiprocessing support. It might work, but I'll have to test it
15:42:56 <number80> only client-side :)
15:43:08 <apevec> that's where you start :)
15:43:29 <apevec> jpena, so you think distributed builds on post-commit won't fly?
15:44:28 <apevec> with 3rd party setup against review.o.o
15:44:51 <sdake> apevec understood re cdn - that was a hue help to kolla's cis - however only have the repos are in the cdn
15:44:51 <jpena> apevec: I came up with a separation between api thread (receiving input from post-commit) and DB thread (downloading rpms from api builders, creating repos, etc)
15:44:54 <sdake> the deps repos are not
15:45:13 <sdake> have/half
15:45:25 <dmsimard> apevec: jpena wasn't in the meeting re: post commit. Did anyone fill him in ?
15:45:36 <jpena> and then we have to think how to authenticate, sync for multiple commits for the same project...
15:45:41 <jpena> add retrigger logic
15:45:48 <sdake> apevec if you see harm in a oo mirror of rdo let me know
15:46:02 <sdake> and i'll consider rethinking my patch
15:46:18 <apevec> sdake, I don't but wanted to double-check what exactly would be mirrored
15:46:31 <sdake> delorean and delorean-deps
15:46:33 <apevec> dmsimard, I didn't keep notes
15:46:38 <sdake> (iirc)
15:46:59 <sdake> as well as the stable branches - down the road
15:47:19 <number80> dmsimard: that's different efforts fixing the same problem, we're //-izing solution completions
15:47:28 <sdake> so basically mirroring everything for everyone in OO to use
15:47:31 <number80> we still need to improve DLRN the time being
15:47:39 <sdake> should be goodness for RDO but I could be wrong
15:47:42 <apevec> jpena, yeah, we'd have to serialize per project
15:48:10 <sdake> i'd like to make oo.org a consumer of rdo in addition to kolla
15:48:14 <number80> jpena: plan to //-ize mock builds?
15:48:49 <apevec> number80, yeah, we're single threaded per DLRN instance
15:48:49 <jpena> number80: yes, I think we could do it. We have a beefy server (8 cores), so we could run 3-4 mocks in parallel for the current master, which takes most of the work
15:48:52 <number80> (it'd just mean hacking mock profiles to use different path for chrooted root)
15:49:00 <apevec> also there's 2min delay after each build
15:49:06 <jpena> number80: exactly that's what I was thinking
15:49:19 <number80> good plan overall
15:49:34 <number80> IMHO, authentication and retriggering is lower prio
15:49:38 <number80> but good
15:49:44 <apevec> ok, let's try multithreading on a single machine first
15:49:53 <apevec> b/c machine itself is not very loaded, afaict
15:50:10 <number80> with a beefy server, I think we can make it reasonably working
15:50:29 <apevec> jpena, what would be the action, do you want to post proposal on rdo-list?
15:50:40 <number80> we don't have C extensions of stuff that are CPU-intensive
15:50:48 <number80> s/of/or/
15:51:07 <jpena> apevec: if there are no other ideas, action would be for me to try a patch
15:51:09 <apevec> number80, most intensive is sphinx build
15:51:27 <jruzicka> #action jpena to try parallel mock builds
15:51:51 <apevec> yep, one patch says more than thousands of specs
15:52:11 <jpena> I even thought of making build plugin-aware, so we could offload to koji. But that's another story :)
15:52:45 <apevec> and then you'd DoS Koji instance :)
15:52:51 <jpena> mwahaha :D
15:53:02 <jruzicka> haha, that's fun.
15:53:10 <apevec> let's keep CBS Koji for stable branch builds
15:53:30 <number80> apevec:
15:53:43 <jruzicka> we should wrap up
15:53:46 <number80> yes
15:53:49 <apevec> yes
15:53:56 <number80> but jpena++
15:54:20 <jruzicka> #topic chair for next meeting
15:54:53 <number80> I can take it
15:55:22 <jruzicka> #info number80 to chair next meeting
15:55:33 <jruzicka> #topic open floor
15:55:59 <chandankumar> I have written a blog on how to use fedora-review for rdo packages https://github.com/redhat-openstack/website/pull/669
15:56:10 <chandankumar> feel free to try and have your comments there
15:56:10 <rbowen> Vote for OpenStack Summit presentations open for a few more days: https://www.openstack.org/summit/barcelona-2016/vote-for-speakers/
15:56:28 <dmsimard> rbowen: oh thanks for that, reminds me I need to advertise my presentations a bit
15:56:38 <rbowen> Well, there aren't direct links to presentations any more.
15:56:46 <dmsimard> Yeah, I know that :)
15:56:49 <rbowen> Due to the megathread on the community mailing list a couple of months back.
15:56:59 * number80 thinks that is just tradition that serves no purpose
15:57:07 <dmsimard> I'm okay with that, there's definitely less spam in my various feeds
15:57:11 <tosky> yeah, there is a selection anyway later
15:57:11 <rbowen> Yes, that seemed to be the consensus of the thread.
15:57:11 <apevec> partisan voting? never!
15:57:27 * number80 haven't even looked
15:57:29 <rbowen> But, I thought I'd mention it anyway. :-)
15:58:01 <jruzicka> #endmeeting