15:01:35 <mjturek> #startmeeting RDO meeting - 2018-06-13
15:01:35 <zodbot> Meeting started Wed Jun 13 15:01:35 2018 UTC.
15:01:35 <zodbot> This meeting is logged and archived in a public location.
15:01:35 <zodbot> The chair is mjturek. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:35 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:01:35 <zodbot> The meeting name has been set to 'rdo_meeting_-_2018-06-13'
15:01:35 <number80> ack
15:01:35 <openstack> Meeting started Wed Jun 13 15:01:35 2018 UTC and is due to finish in 60 minutes.  The chair is mjturek. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:36 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:38 <openstack> The meeting name has been set to 'rdo_meeting___2018_06_13'
15:01:46 <mjturek> #topic roll call
15:01:51 <mjturek> o/
15:01:52 <leanderthal> o/
15:01:57 <mjturek> #chair mjturek leanderthal
15:01:57 <zodbot> Current chairs: leanderthal mjturek
15:01:58 <openstack> Current chairs: leanderthal mjturek
15:02:09 <PagliaccisCloud> o/
15:02:23 <jpena> o/
15:02:39 <mjturek> #chair PagliaccisCloud jpena
15:02:39 <zodbot> Current chairs: PagliaccisCloud jpena leanderthal mjturek
15:02:40 <openstack> Current chairs: PagliaccisCloud jpena leanderthal mjturek
15:03:08 <mjturek> cool anyone else?
15:03:16 <rdogerrit> Merged rdo-jobs master: Add set-artifact-path-for-children role  https://review.rdoproject.org/r/14218
15:04:10 <mjturek> alright I guess we can move on to the agenda! Which as usual can be found here https://etherpad.openstack.org/p/RDO-Meeting
15:04:27 <number80> o/
15:04:27 <mjturek> #topic Test Day
15:04:39 <mjturek> #chair number80
15:04:39 <zodbot> Current chairs: PagliaccisCloud jpena leanderthal mjturek number80
15:04:40 <openstack> Current chairs: PagliaccisCloud jpena leanderthal mjturek number80
15:05:02 <mjturek> #link https://dashboards.rdoproject.org/rdo-dev
15:05:13 <mjturek> anyone have anything to discuss here?
15:05:30 <amoralej> o/
15:05:54 <leanderthal> is this going to be green for tomorrow?
15:06:03 <amoralej> there is still a blocker
15:06:14 <amoralej> ykarel, do you know the status of promotion blockers in master?
15:07:38 <ykarel> amoralej, all are clear currently, promotion job is running with the fixe
15:07:41 <ykarel> fixes
15:07:46 <amoralej> ok,
15:07:50 <amoralej> let's cross fingers, then
15:08:07 <ykarel> o/
15:09:16 <mjturek> amoralej: what's the blocker?
15:09:27 <rdogerrit> pabelanger proposed rdo-jobs master: Update for DLRN jobs  https://review.rdoproject.org/r/14202
15:09:28 <number80> We don't need latest promotion to run the test days
15:09:34 <number80> just one recent enough
15:09:41 <amoralej> mjturek, no more blokers according with what ykarel said
15:10:10 <leanderthal> have we run a test days without the latest promotion?
15:10:16 <leanderthal> does anyone know what that was like?
15:10:18 <amoralej> number80, i'd say m2 releases were done after last promotion for some services
15:10:23 <amoralej> but we are not too bad
15:10:31 <number80> leanderthal: we used to do that before
15:10:36 <leanderthal> ah, okay.
15:10:50 <leanderthal> do we write up something in the workarounds or something? what's it like?
15:11:04 <number80> amoralej: yes, just wanted to stress that it's not bad enough to cancel if the net promotion doesn't pass
15:11:17 <amoralej> yes, i agree
15:11:24 <leanderthal> cool cool
15:12:10 <mjturek> alright looks like we're winding down on this topic
15:12:16 <mjturek> everyone good to move on?
15:12:37 <leanderthal> to the test day environment?
15:12:55 <number80> leanderthal:for rocky-1 => https://etherpad.openstack.org/p/rdo-test-days-rocky1-workarounds
15:13:11 <number80> nothing drastic :)
15:13:23 <leanderthal> number80, fair, but promotion was green
15:13:43 <mjturek> ahhh right - apevec anything you'd like to say about the triplo allinone?
15:13:50 <mjturek> #link http://lists.openstack.org/pipermail/openstack-dev/2018-June/131135.html
15:14:30 <leanderthal> it's more in aschultz and EmilienM 's hands now for set up
15:14:31 <number80> mjturek: it won't be ready for tomorrow, but we'll try for the next milestone
15:14:47 <leanderthal> dmsimard and jpena have purchased the resources
15:14:59 <leanderthal> number80, ah, okay - you've got the official word, then?
15:15:08 <mjturek> #info tripleo-allinone not ready for test days but try to target next milestone
15:15:14 <EmilienM> we're still working on it
15:15:21 <leanderthal> oh cool!
15:15:24 <Pharaoh_Atem> there's a triplo-allinone? :D
15:15:25 <EmilienM> you can use it to deploy services like Keystone, and more
15:15:30 <leanderthal> #YESSSSSSSS
15:15:38 <EmilienM> we deployed all-in-one today and spawned a VM but it requires some patches I thikn
15:15:39 <leanderthal> EmilienM, i'd love love love love it to be ready for tomorrow
15:15:54 <EmilienM> Pharaoh_Atem: https://docs.openstack.org/tripleo-docs/latest/install/containers_deployment/standalone.html
15:16:28 <leanderthal> EmilienM, you're working with jpena and dmsimard to get it available for test days?
15:16:35 <dmsimard> have a power outage out of nowhere.. gonna need to shut down my stuff before my ups goes out (including my bouncer), be back later
15:16:40 <EmilienM> I'm working with Alex ( mwhahaha )
15:16:46 <leanderthal> NICE
15:16:52 <Pharaoh_Atem> OOOOH
15:16:54 <PagliaccisCloud> does the all-in-one deployment always launch containers in kvm?
15:17:17 <PagliaccisCloud> that might be a dumb question. lemme read through the link
15:18:55 <EmilienM> PagliaccisCloud: read the links, test it and ask questions on #tripleo :-)
15:19:16 <PagliaccisCloud> got it. doing The Thing EmilienM :D
15:19:19 * number80 thinks we should consider a webcast when it'll be ready :)
15:19:26 <mjturek> alirght cool so let's move along!
15:19:28 <Pharaoh_Atem> damn, I'm in too many channels :(
15:19:34 <PagliaccisCloud> +1
15:19:38 <EmilienM> number80: we have a demo already
15:19:44 <EmilienM> https://asciinema.org/a/185533
15:20:04 <EmilienM> it's on the email (linked here): http://lists.openstack.org/pipermail/openstack-dev/2018-June/131135.html
15:20:09 <mjturek> oh cool
15:20:17 <EmilienM> right, read the links and then ask questions :D
15:21:14 <mjturek> #link https://asciinema.org/a/185533
15:21:30 <leanderthal> EmilienM, do people need to create accounts or anything? will you update https://etherpad.openstack.org/p/rdo-rocky-m2-cloud as needed for tomorrow, please?
15:21:33 <mjturek> #info tripleo-allinone demo available
15:22:22 <EmilienM> leanderthal: create account on what?
15:22:23 <mjturek> alright in the interest of time let
15:22:34 <EmilienM> what you need is a fresh centos7 vm and follow the doc :D
15:22:48 <mjturek> s cap this topic in 5 minutes
15:23:19 <EmilienM> again the All-In-One (standalone) is experimental, I'm not sure we want to rely 100% on that for testing days
15:23:28 <leanderthal> EmilienM, we set up a cloud on internap during test days
15:23:34 <EmilienM> you can use it to deploy Keystone, we know it works. Today we're working on next steps
15:24:05 <leanderthal> today we need to decide if we're using tripleo or packstack for the internap cloud
15:25:18 <leanderthal> it's not the main event of test days, but it is the available cloud if people don't have their own hardware available to deploy
15:25:23 <EmilienM> Alex and I are working (among X other things) on it today
15:25:29 <EmilienM> but if deadline is tomorrow I suggest to go with packstack
15:25:42 <number80> ack
15:25:49 <leanderthal> EmilienM, fair. let's go with packstack for tomorrow, then.
15:25:58 <leanderthal> EmilienM, i look forward to seeing it at the august test days!
15:26:10 <EmilienM> if people are looking for something using TripleO, give the All-In-One a try
15:26:36 <leanderthal> i'll specifically add a scenario to test tripleo all in one, too.
15:26:39 <EmilienM> and Alex and I are on IRC to provide support, so I encourage people to give it a try
15:26:44 <leanderthal> perfect
15:26:53 <EmilienM> ping us directly if you want to deploy more than Keystone
15:27:01 <leanderthal> cool cool
15:27:03 <mjturek> alright thanks all let's move along to the next topic
15:27:04 <EmilienM> I think we'll have a patch up today, when we're done with $meetings
15:27:11 * EmilienM disappears
15:27:16 <leanderthal> jpena, are you available to set up the cloud environment for tomorrow?
15:27:46 <mjturek> #topic BOF planning at DevConf.in 2018
15:27:47 <leanderthal> mjturek, sorry - i need this finalized for tomorrow's test days
15:28:05 <mjturek> whoops! sorry leanderthal - feel free to spill over to this for a minute
15:28:07 <jpena> leanderthal: ok, I'll try to get it ready between the evening and tomorrow morning (Europe time)
15:28:16 <leanderthal> jpena, thank you so so so much
15:28:47 <mjturek> #info jpena will setup cloud environment for tomorrow
15:29:04 <leanderthal> #action jpena will setup cloud environment for tomorrow
15:29:08 <leanderthal> ;-)
15:29:10 <mjturek> :) thanks
15:29:12 <leanderthal> mjturek++
15:29:24 <mjturek> chandankumar: your topic!
15:29:37 <mjturek> who is going to be at DevConf 2018?
15:30:46 <leanderthal> DevConf.IN ^
15:31:19 <chandankumar> mjturek: I am here
15:31:33 <mjturek> hey!
15:31:53 <chandankumar> So we have a bof at DevConf.In related to RDO OpenStack Ansible and OpenShift
15:32:03 <leanderthal> chandankumar, thx so much for arranging a BoF at DefConf.IN!! !
15:32:07 <chandankumar> we just have one hour slot
15:32:22 <chandankumar> I need some ideas to plan it in a better way
15:33:09 <rdogerrit> pabelanger created config master: Add legacy-DLRN-rpmbuild as trusted job  https://review.rdoproject.org/r/14219
15:33:54 <chandankumar> Generally in BOF, people will be coming and shotting questions, but I need a better way to drive it
15:34:39 <mjturek> chandankumar: have you posted to the ML about it as well?
15:34:47 <chandankumar> mjturek: nope
15:35:05 <chandankumar> mjturek: it is confirmed yesterday, so i wanted to use the meeting for ideas :-)
15:35:27 <mjturek> chandankumar: I would definitely post to rdo-dev advertising it later on!
15:35:41 <chandankumar> mjturek: sure
15:36:03 <mjturek> fair enough - so looking for suggestions on the structure of it?
15:36:27 <number80> chandankumar: it'd be nice to create etherpad and coordinate shifts there
15:37:03 <chandankumar> number80: for BOF, it is just one hour, I hope most of openstack flocks will be there, we need to drive the discussion and get out of it
15:37:35 <mjturek> still, an etherpad would be a good place for people to propose topics
15:37:45 <chandankumar> mjturek: sure I will do it right now
15:38:12 <mjturek> cool! can you paste the link here as well?
15:38:41 <leanderthal> chandankumar, test days are august 2-3, the BoF could be a mini recap of what happened based on the etherpads if you have someone local willing to process the info
15:38:41 <chandankumar> mjturek: https://review.rdoproject.org/etherpad/p/devconfin-openstackbof
15:39:01 <chandankumar> i will put the ideas there
15:39:04 <mjturek> #link https://review.rdoproject.org/etherpad/p/devconfin-openstackbof
15:39:34 <mjturek> #action please post any ideas for devconfin bof topics in above etherpad
15:39:45 <mjturek> alright let's move along
15:39:50 <number80> oh
15:39:52 <rdogerrit> pabelanger proposed rdo-jobs master: Update for DLRN jobs  https://review.rdoproject.org/r/14202
15:40:11 <mjturek> what's up number80
15:41:21 <mjturek> #topic zuulv3 migration
15:41:51 <mjturek> so first point here is discussing secrets
15:41:51 <chandankumar> leanderthal: dumped your idea there, thanks :-)
15:42:01 <leanderthal> poifect
15:42:08 <pabelanger> So, just a heads up, I am sure people have started to notice some jobs are now running as zuulv3 and ansible.
15:43:09 <pabelanger> we've been using zuul-migrate to handle that, however one potential issue. It looks like some jobs use secrets in check pipelines, CBS builds I think. This isn't going to be safe in zuulv3, as somebody would leak the secret easy.
15:43:30 <pabelanger> so, questions are, does it need to be cbs or can it be something else?
15:43:51 <pabelanger> If needs to be, then we can make a trusted job, in config project, but somebody need to help do the work for that
15:44:20 <pabelanger> or maybe we make them non-voting to start in zuulv3, for now to continue with zuul-migrate.
15:44:34 <pabelanger> but right now, they are not going to work, as they are written today
15:44:51 <apevec> amoralej, cbs builds in check are scratch?
15:44:59 <amoralej> yes
15:45:17 <apevec> that _might_ be approximated by local mock w/ the same CBS buildroot repos
15:45:39 <amoralej> but anyway we need cbs in gate pipeline
15:45:52 <apevec> is it in gate or post?
15:45:54 <amoralej> gate is not a problem?
15:45:56 <amoralej> gate
15:45:57 <pabelanger> gate will have the same issue, credentials could leak
15:46:02 <amoralej> in post we apply tags
15:46:06 <amoralej> but builds are done in gate
15:46:07 <pabelanger> but post is usually fine
15:46:20 <apevec> how could it leak?
15:46:31 <amoralej> pabelanger, credentials in this case is a tgz with certs
15:46:37 <apevec> isn't that CVE ?
15:46:42 <amoralej> what could be a good alternative?
15:46:46 <pabelanger> upload patch, $echo secret, +W patch in untrusted project
15:46:53 <pabelanger> leak secret
15:47:29 <apevec> hmm, that feels like security issue by design?
15:47:39 <pabelanger> amoralej: we can create a promote pipeline, which happens after gate and before post, to build things. This is what we do upstream
15:48:01 <number80> pabelanger: scratch builds also requires certs
15:48:03 <mjturek> not sure if it's helpful but internally we've used hiera-eyaml when setting up public facing jobs https://github.com/voxpupuli/hiera-eyaml
15:48:14 <number80> so not an option
15:48:17 <amoralej> apevec, would it be possible to create a user that can only do scrath builds?
15:48:19 <pabelanger> apevec: secrets are disable by default, zuul operate needs to manually enable them, and understand the consequences
15:48:30 <apevec> amoralej, not in CBS
15:48:30 <pabelanger> upstream, we only allow secrets for untrusted jobs in post
15:48:38 <pabelanger> check and gate, we do not allow
15:49:01 <apevec> amoralej, but see above, gate is also an issue, so limiting scratch would not help
15:49:19 <pabelanger> https://zuul-ci.org/docs/zuul/user/config.html?highlight=post%20review#attr-pipeline.post-review
15:49:22 <amoralej> apevec, for actual builds we could create othere pipeline
15:49:29 <number80> I hate myself but setuid?
15:49:49 <pabelanger> yes, we should create a promote pipeline for builds, which happens after code has merged
15:49:50 <amoralej> pabelanger, what are trusted jobs?, the ones in config repo?
15:49:55 <pabelanger> amoralej: yes
15:50:05 <number80> Or use selinux to allow only cbs binary to read certificates
15:50:10 <amoralej> couldn't make those jobs trusted?
15:50:23 <pabelanger> centrally managed, small team to review changes and ensure echo $secret isn't approved
15:50:32 <pabelanger> amoralej: yes, somebody needs to step up and help write them
15:50:48 <number80> amoralej: it can be sneaked inside the spec itself
15:50:59 <number80> spec file is glorified shell script
15:51:31 <amoralej> number80, as is today
15:51:36 <pabelanger> which can happen before or after zuul-migrate, but I am feeling we'll need to force migrate this to get off jenkins and split zuul sooner then later, which means they might be broken / non-voting until somebody does the work
15:52:30 <number80> amoralej: yep
15:53:17 <amoralej> number80, actually not
15:53:27 <amoralej> the rpmbuild is done in CBS
15:53:30 <number80> I start to believe that a selinux policy is the best option
15:53:38 <number80> ?
15:53:38 <amoralej> it can't access certs in the job instance
15:54:07 <number80> how the src.rpm is built? mock, right?
15:54:07 <amoralej> i mean, when executing pre, post, etc... in the spec
15:54:11 <pabelanger> number80: what is the thoughts on selinux?
15:54:28 <number80> pabelanger: use selinux to limit access to the cert to a specific binary
15:54:35 <number80> so anyone else will be denied
15:54:37 <amoralej> number80, not in mock, but when building the srpm, not steps are executed
15:54:46 <amoralej> right?
15:55:09 <pabelanger> number80: don't think that will work, if I am untrusted and root, I can still just disable selinux
15:55:18 <pabelanger> then access secret
15:55:20 <number80> amoralej: in practice yes (there's a way but it's tricky)
15:55:49 <number80> pabelanger: you can't completely disable it, so it'll still be logged in permissive (so we'll know)
15:56:11 <amoralej> number80, i still think we are safe
15:56:15 <number80> yup
15:56:20 <pabelanger> but, we won't solve this in the next 5mins, but I'd like people to be aware, current cbs jobs migrated to zuulv3 don't work. So we need to come up with a plan, ASAP to help finish zuul-migrate
15:56:22 <mjturek> we're getting to the last couple minutes of the meeting so may need to pick this discussion back up afterwards
15:56:23 <amoralej> from spec side
15:56:38 * number80 suggests to continue on the list?
15:56:43 <mjturek> +1
15:56:45 <amoralej> +1
15:56:48 <leanderthal> +1
15:57:00 <mjturek> #info zuulv3 migration discussions to continue on ML
15:57:12 <mjturek> #topic next week's chair
15:57:38 <mjturek> anyone?
15:57:39 <amoralej> i can take it
15:57:42 <mjturek> \o/
15:57:51 <leanderthal> NICE
15:57:52 <mjturek> #action amoralej to chair next week's meeting
15:57:56 <mjturek> thanks amoralej
15:58:11 <mjturek> so not much time left but let's move to open floor
15:58:16 <mjturek> #topic open floor
15:58:30 <mjturek> anyone have anything they'd like to bring up?
15:59:04 <mjturek> alright, well it's been a busy meeting!
15:59:14 <number80> Thanks mjturek for chairing :)
15:59:18 <mjturek> np!
15:59:19 <leanderthal> thx mjturek
15:59:31 <mjturek> thanks everyone for joining!
15:59:35 <mjturek> #endmeeting