15:01:18 <amoralej> #startmeeting RDO meeting - 2019-03-20
15:01:18 <zodbot> Meeting started Wed Mar 20 15:01:18 2019 UTC.
15:01:18 <zodbot> This meeting is logged and archived in a public location.
15:01:18 <zodbot> The chair is amoralej. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:18 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:01:18 <zodbot> The meeting name has been set to 'rdo_meeting_-_2019-03-20'
15:01:19 <openstack> Meeting started Wed Mar 20 15:01:18 2019 UTC and is due to finish in 60 minutes.  The chair is amoralej. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:20 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:22 <openstack> The meeting name has been set to 'rdo_meeting___2019_03_20'
15:01:38 <fultonj> o/
15:01:40 <mjturek> o/
15:01:47 <ykarel> o/
15:01:55 <baha> o/
15:02:04 <rdogerrit> Merged puppet/puppet-collectd-distgit stein-rdo: Update to post 10.1.1-rc0 (d7b6b8c493e808534bdd4b1e500065ce46e40434)  https://review.rdoproject.org/r/19550
15:02:12 <amoralej> #chair fultonj mjturek ykarel baha
15:02:12 <zodbot> Current chairs: amoralej baha fultonj mjturek ykarel
15:02:13 <openstack> Current chairs: amoralej baha fultonj mjturek ykarel
15:02:38 <jpena> o/
15:02:52 <amoralej> #chair jpena
15:02:52 <zodbot> Current chairs: amoralej baha fultonj jpena mjturek ykarel
15:02:53 <openstack> Current chairs: amoralej baha fultonj jpena mjturek ykarel
15:04:01 <fultonj> does anyone have the agenda link handy?
15:04:09 <mjturek> https://etherpad.openstack.org/p/RDO-Meeting
15:04:12 <amoralej> https://etherpad.openstack.org/p/RDO-Meeting
15:04:20 <fultonj> tks
15:04:23 <amoralej> please add your topics
15:04:26 <amoralej> we are starting
15:04:38 <amoralej> in 3
15:04:40 <amoralej> 2
15:04:42 <amoralej> 1
15:04:44 <amoralej> go!
15:04:59 <amoralej> #topic default maintainers are gone in rdoinfo: deps - https://review.rdoproject.org/r/19389
15:05:15 <amoralej> so, this was from  apevec, but i think he's not around
15:05:23 <rdogerrit> Merged puppet/puppet-tomcat-distgit stein-rdo: Update to post 2.5.0 (5366908b1b86e9ba6d112b1dcc6eaf08118b4d7d)  https://review.rdoproject.org/r/19553
15:05:39 <amoralej> #info dependencies without explicit maintainers have been reset to nobody
15:06:05 <amoralej> so now, we need for maintainers to sign up
15:06:41 <amoralej> for dependencies specific to a project or very limited set, they should be maintained by that project team
15:06:44 * Duck o/
15:06:53 <amoralej> #chair Duck
15:06:53 <zodbot> Current chairs: Duck amoralej baha fultonj jpena mjturek ykarel
15:06:54 <openstack> Current chairs: Duck amoralej baha fultonj jpena mjturek ykarel
15:08:02 <amoralej> #actions maintainers to check deps in https://github.com/redhat-openstack/rdoinfo/blob/master/deps.yml and add themselves where applicable
15:08:40 <amoralej> #info the processes to manage deps are in https://www.rdoproject.org/documentation/requirements
15:08:42 <Duck> always better to see reality than expecting people who have moved to other things to care :-)
15:09:08 <weshay> guys is the rdo kibana working?
15:09:40 * ykarel doubt it's workin
15:10:00 <amoralej> i see an error "no results found" in https://review.rdoproject.org/app/kibana
15:10:42 <amoralej> ok, so i think that's it for the first topic
15:10:45 <amoralej> let's move on
15:10:58 <amoralej> we have quite long agenda today
15:11:07 <rdogerrit> Rafael Folco created config master: Make containers-build-push-base job branchless  https://review.rdoproject.org/r/19572
15:11:07 <rdogerrit> Rafael Folco created config master: Use new containers-build-push-base branchless job  https://review.rdoproject.org/r/19573
15:11:18 <amoralej> #topic finally drop *-patches repos (openstack/$PROJECT mirrors) in review.rdo
15:12:00 <amoralej> so i guess apevec only put this to start action of actually removing it
15:12:16 <amoralej> as this was discussed and agreed in the past
15:12:46 <amoralej> jpena, can we remove repos using resources directory in SF?
15:13:07 <jpena> amoralej: I *think* so, but we'll have to check
15:13:13 <amoralej> ok
15:13:19 <amoralej> i'll add you an action jpena
15:13:26 <jpena> ack
15:13:39 <amoralej> #action jpena to check how to remove repos using resources in SoftwareFactory
15:13:53 <amoralej> onec we know how to do it we'll get them out
15:14:07 <amoralej> all of them except the ones with patches applied
15:14:57 <amoralej> #topic RDO phase1 tripleo jobs running in ci.centos will be moved to some other system, weshay is looking into it
15:15:01 <amoralej> ykarel, this is yours
15:15:28 <ykarel> so CI Team is planning to run some baremetal scenarios, and ci.centos resulted some bottlenack in past
15:15:39 <weshay> we'll be looking into it..
15:15:42 <ykarel> so weshay is looking for other systems
15:16:10 <weshay> I wouldn't say any decisions have been made, I'd love to know if we think there are any adverse implications from moving away from ci.centos
15:17:00 <amoralej> baremetal means *real baremetal*
15:17:01 <amoralej> ?
15:17:12 <weshay> we have baremetal hooked into internal sf.. where we could run the same jobs against and vote on rdo promotions
15:17:14 <weshay> via dlrn
15:17:16 <amoralej> or using vms with libvirt as done in ci.centos.org?
15:17:31 <weshay> so.. can be either, or both
15:18:07 <amoralej> so, i think we can keep weirdo ones in ci.centos.org, i'd say it has not been an issue so far
15:18:08 <weshay> there wouldn't be much value in rerunning real bm env on rdo if we did in tripleo
15:18:16 <weshay> but it's certainly possible
15:18:38 <weshay> right.. tripleo we have pacemaker disabled because the hardware in ci.centos can't support it
15:18:53 <weshay> so.. it's forcing unsupported settings essentially
15:18:58 <weshay> in tripleo
15:19:04 <amoralej> i'd say making jobs to vote in dlrn-api should not be a problem
15:19:07 <weshay> would rather run w/ clearly supported settings
15:19:10 <weshay> aye
15:19:27 <amoralej> it'd be good if we could maintain public logs for the reporting jobs
15:19:36 <PagliaccisCloud> \o
15:19:45 <amoralej> we used to do it in the past, i'm not sure what's the current status
15:19:50 <weshay> amoralej I believe the logs are public
15:19:51 <amoralej> #chair weshay PagliaccisCloud
15:19:51 <zodbot> Current chairs: Duck PagliaccisCloud amoralej baha fultonj jpena mjturek weshay ykarel
15:19:52 <openstack> Current chairs: Duck PagliaccisCloud amoralej baha fultonj jpena mjturek weshay ykarel
15:20:03 <weshay> wait.. they are not
15:20:12 <weshay> so ya.. we can log to the thirdparty log server
15:20:16 <weshay> as we did w/ rdo phase 2
15:20:22 <amoralej> yeap, exactly
15:20:28 <weshay> rock on
15:20:38 <amoralej> other than that, i'd say it's ok
15:20:48 <weshay> great..
15:20:52 <ykarel> weshay, any ETA for this movement? or atleast a card/bug where this is tracked, so we can follow up there
15:21:14 <amoralej> so, the idea would be to move the oooq jobs running in ci.centos replaced by other jobs running in other more appropiate ci environment
15:21:25 <weshay> nothing scheduled, but I'll add it as an epic in taiga
15:21:34 <amoralej> keep reporting to dlrn-api, keep them as promotion criteria
15:21:47 <amoralej> but running in non public hardware
15:21:48 <ykarel> weshay, ack
15:21:50 <weshay> amoralej for promotion.. yes.. the check jobs we have in ci.centos will probably be killed
15:21:59 <amoralej> did i understand it right?
15:22:17 <weshay> ya.. non-public hardware but logging publically
15:22:20 <amoralej> yes
15:22:33 <amoralej> check jobs, you mean the ones voting upstream check gates?
15:22:58 <weshay> we have non-voting check jobs running from ci.centos on tq/tqe changes
15:23:10 <weshay> since ocata they have not been reliable
15:23:19 <amoralej> yeap
15:23:22 <amoralej> ack
15:23:26 <weshay> I may leave standalone jobs there
15:23:39 <weshay> probably will
15:24:10 <amoralej> ok
15:24:48 <amoralej> i'd like to keep puppet promotion jobs in ci.centos.org, i understand non-oooq are out of the scope for this change
15:24:50 <amoralej> right?
15:25:33 <ykarel> yes topic was that only
15:25:44 <ykarel> move TripleO jobs
15:25:52 <amoralej> ok
15:25:57 <amoralej> good
15:26:16 <amoralej> so i thing we are done with that topic
15:26:21 <amoralej> anything to add?
15:26:28 <ykarel> let's create an action to create epic
15:27:29 <amoralej> #action weshay to create an epic in taiga to move oooq jobs out of ci.centos.org
15:27:35 <amoralej> weshay, ^ are you ok?
15:28:03 <weshay> that's it
15:28:09 <amoralej> ok
15:28:20 <amoralej> #topic ppc64le container builds update
15:28:27 <mjturek> hey
15:28:34 <amoralej> #info     logging uploads are working https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/417/logs/logs/
15:28:41 <amoralej> congrats mjturek ! :)
15:28:51 <ykarel> cool
15:28:53 <mjturek> and baha and dmsimard ^ :)
15:29:08 <mjturek> looks like the base container started failing recently though
15:29:18 <mjturek> we merged a patch to kolla to fix
15:29:29 <mjturek> and are wondering when it would hit RDO?
15:29:43 <baha> (The patch just merged today, if that helps)
15:30:22 <amoralej> it will be probably in next dlrn run
15:30:30 <jpena> baha: what's the patch number? It's probably built in RDO Trunk alreayd
15:30:32 <amoralej> as kolla follows master trunk
15:30:37 <amoralej> https://review.openstack.org/#/c/644389/
15:30:41 <amoralej> not built yet
15:30:52 <mjturek> oh awesome!
15:31:04 <amoralej> mjturek, you will see it coming in https://trunk.rdoproject.org/centos7-master/report.html
15:31:05 <baha> I was hoping that was how that worked, but we figured we'd check to make sure =)
15:31:05 <amoralej> soon
15:31:20 <amoralej> baha, what repo do you use?
15:31:22 <ykarel> amoralej, are they consuming kolla build from not promoted repo?
15:31:24 <rdogerrit> Kashyap Chamarthy proposed openstack/nova-distgit rpm-master: Use granular libvirt and QEMU RPMs for RHEL-8 Nova  https://review.rdoproject.org/r/19556
15:31:24 <amoralej> consistent or promoted?
15:31:37 <amoralej> i assumed consisntent, but i'm not sure
15:32:24 <mjturek> I think we use current
15:32:46 <amoralej> it'd be better consistent, probably
15:32:46 <mjturek> should we be using consistent?
15:32:56 <mjturek> okay we can switch over to that I think
15:33:04 <amoralej> consistent is like current but before the last FTBFS
15:33:14 <mjturek> FTBFS?
15:33:24 <amoralej> Failed to Build from Source
15:33:29 <amoralej> means some package failed
15:33:31 <mjturek> got it! thanks amoralej
15:33:32 <amoralej> to build
15:33:46 <amoralej> we always rely on consistent repos to promote
15:33:52 <baha> We push master branch over to the container build script so yeah, I think we would be using whatever is latest
15:33:58 <amoralej> the ones with failed builds are considred "inconsistent"
15:34:18 <mjturek> I see, and may be part of current
15:35:20 <mjturek> anyway, we're good otherwise, just didn't know how to find out when patches hit rdo
15:35:26 <mjturek> thanks amoralej and jpena!
15:35:36 <baha> Yes, thank you!
15:35:46 <amoralej> mjturek, a new dlrn run just started
15:35:51 <amoralej> it should build that patch
15:35:53 <mjturek> awesome!
15:36:16 <amoralej> so, we can move on to next topic?
15:36:21 <mjturek> yep!
15:36:27 <amoralej> ok, thanks mjturek
15:36:31 <amoralej> #topic Status of Stein preparation
15:36:52 <amoralej> so, as you know projects are creating rc1 tags upstream
15:37:17 <amoralej> so we are currently building more and more packages in CBS for stein tags
15:37:33 <amoralej> we are in the process of branching disgtgits
15:38:14 <amoralej> #action maintainers should send required updates to package distgits for stein asap as new branches stein-rdo are being cut
15:38:35 <amoralej> we have frozen non-openstack puppet modules too
15:38:51 <amoralej> to the commits used in the last promotion 4 days ago
15:39:09 <amoralej> #info non-OpenStack puppe modules have been frozen for Stein
15:39:38 <amoralej> next steps will be to keep building rc tags for all available packages
15:39:51 <amoralej> and start creating jobs to test them in rdoinfo gate
15:40:18 <amoralej> also, synchronization from stein-testing to https://buildlogs.centos.org/centos/7/cloud/x86_64/openstack-stein/ is in place
15:40:30 <amoralej> i think that's the summary
15:40:37 <amoralej> ykarel, am i missing something important?
15:40:56 <ykarel> amoralej, no, all done
15:41:02 <amoralej> ok
15:41:11 <ykarel> we can link etherpad here
15:41:19 <amoralej> good
15:41:24 <amoralej> or to the card
15:41:34 <amoralej> #info https://trello.com/c/nkadXFWF/702-stein-release-preparation
15:41:35 <ykarel> yes that also works
15:42:13 <amoralej> jpena, btw, is the namespace for stein created in the rdo registry?
15:42:21 <jpena> amoralej: yes, it is
15:42:32 <amoralej> ok, i'll close that too then
15:42:50 <ykarel> amoralej, we also need to create current-tripleo and previous-current-tripleo symlinks
15:43:12 <amoralej> to bootstrap the ci jobs, right?
15:43:15 <ykarel> CI Team started creating jobs
15:43:25 <ykarel> yes those jobs will need those symlinks
15:44:00 <amoralej> ok, we'll find out what repos to use for that links
15:44:07 <amoralej> but ykarel
15:44:16 <ykarel> okk, jobs preparation:- https://review.rdoproject.org/r/#/q/topic:oooq/branching
15:44:22 <amoralej> if we start running only promotion periodic jobs
15:44:32 <amoralej> do those need current-tripleo?
15:44:48 <ykarel> amoralej, updates job need that afair
15:45:19 <amoralej> we can create it if needed
15:45:30 <amoralej> just i'd prefer not to do if we can avoid it
15:45:31 <ykarel> yes, we have some time
15:45:37 <amoralej> but we will not block
15:46:05 <ykarel> yes
15:46:14 <rdogerrit> wes hayutin created rdo-infra/ci-config master: update master criteria for promotion  https://review.rdoproject.org/r/19574
15:46:48 <amoralej> jpena, wrt packstack, is there anything pending?
15:46:52 <amoralej> before creating rc
15:47:02 <jpena> let me check
15:47:41 <jpena> not a lot, we could add https://review.openstack.org/639075 and https://review.openstack.org/634910
15:47:42 <apevec> hmm rabbitmq update is failing CI
15:47:48 <rdogerrit> Ronelle Landy proposed config master: Report containers-build-push to DLRN  https://review.rdoproject.org/r/19030
15:48:37 <amoralej> jpena, ack, i'll review those
15:48:46 <amoralej> ok, let's move on
15:49:06 <amoralej> #topic  Nautilus update
15:49:16 <amoralej> fultonj, stage is yours
15:49:20 <fultonj> thanks
15:49:27 <amoralej> #info     Ceph Nautilus was released yesterday: https://ceph.com/releases/v14-2-0-nautilus-released
15:49:30 <amoralej> \o/
15:49:38 <fultonj> nautilus!
15:49:44 <fultonj> https://review.rdoproject.org/r/#/c/18721
15:49:48 <fultonj> we basically need to keep working on it
15:49:53 <amoralej> #info want to get Nautilus into Stein so we can use Ansible 2.7 as per https://review.rdoproject.org/r/#/c/18721
15:49:59 <fultonj> #action fultonj to investigate why cinder not working and give update in https://review.rdoproject.org/r/#/c/18721
15:50:21 <amoralej> #info stein GA should be released with nautilus ceph
15:50:31 <amoralej> also, some changes are required in puppet-ceph
15:50:40 <fultonj> puppet-ceph?
15:50:44 <amoralej> yes
15:50:48 <fultonj> we use ceph-ansible to deploy nautilus
15:50:58 <fultonj> what's using puppet-ceph?
15:51:06 <amoralej> there is a puppet module wich is still maintained by puppet-openstack team
15:51:17 <amoralej> and tested with puppet-openstack-integration scenarios
15:51:39 * ykarel need to leave
15:51:52 <rdogerrit> Merged rdo-infra/ci-config master: update master criteria for promotion  https://review.rdoproject.org/r/19574
15:51:54 <fultonj> tripleo has deployed ceph without puppet-ceph since pike
15:52:27 <fultonj> still tested in the puppet-openstack-integration scenarios... is that using tripleo?
15:52:35 <amoralej> fultonj, that's the failure in legacy-weirdo-validate-buildsys-tags job in https://review.rdoproject.org/r/#/c/18721/
15:52:53 <amoralej> fultonj, there are rdo users not using tripleo :)
15:52:59 <rdogerrit> Merged config master: Report containers-build-push to DLRN  https://review.rdoproject.org/r/19030
15:53:01 <fultonj> amoralej: that's cool
15:53:33 <fultonj> hmmm
15:53:40 <amoralej> if needed we will request to remove ceph deployment temporarily in puppet-openstack-integration
15:53:45 <amoralej> until puppet-ceph is ready
15:53:59 <fultonj> amoralej: is the cinder failure in legacy-weirdo-validate-buildsys-tags ?
15:54:04 <amoralej> nop
15:54:15 <fultonj> ok
15:54:16 <amoralej> that failure is because puppet-ceph uses ceph-disk
15:54:25 <amoralej> which does not longer exist
15:54:27 <fultonj> ceph-disk removed
15:54:30 <fultonj> in nautilus
15:54:31 <fultonj> yep
15:54:32 <amoralej> it needs to be updated to use ceph-volume iirc
15:54:37 <fultonj> amoralej: exactly
15:55:34 <amoralej> so a new branch stable/mimic was cut with the last commit in master
15:55:44 <amoralej> and master will be fixed to work with nautilus
15:55:47 <amoralej> that's the plan
15:55:55 <amoralej> i thing tobias-urdin is working on it
15:56:31 <amoralej> it's a bad time for these changes as we are late in the cycle, but puppet modules are cycle-trailing
15:56:41 <amoralej> so i thing we still have time to get it done in stein
15:57:15 <amoralej> apart from that, fultonj i'll keep waiting on your debug of the cinder issue in ooo jobs
15:57:21 <amoralej> let me know if i can help somehow
15:57:24 <fultonj> amoralej: thanks
15:57:48 <fultonj> i guess that's all on that topic for now
15:57:49 <amoralej> i'm also concerned about how to push all those changes together and not break the gates
15:58:15 <amoralej> i mean, update ceph, ceph-ansible, ansible and ceph containers tags
15:58:18 <amoralej> coordinately
15:58:57 <amoralej> if we don't coordinate all those changes, we will break ooo gates
15:59:06 <amoralej> and maybe others, as puppet
15:59:17 <fultonj> gfidente fmount ^ fyi
15:59:23 <fultonj> amoralej: noted
15:59:39 <amoralej> but, let's figure out first how to get it working in the check jobs
15:59:50 <amoralej> we are almost out of time
15:59:57 <fultonj> ack
16:00:04 <amoralej> thanks fultonj
16:00:10 <amoralej> #topic open floor
16:00:19 <amoralej> any topic you want to arise?
16:00:34 <amoralej> well
16:00:40 <amoralej> chair for next meeting
16:00:44 <amoralej> i almost forget
16:00:48 <amoralej> any volunteer?
16:01:41 <jpena> I'll take it
16:01:51 <amoralej> thanks jpena
16:02:02 <amoralej> #action jpena to chair next meeting
16:02:15 <amoralej> i think we can close the meeting
16:02:24 <amoralej> last call for any other out of the agenda topic?
16:02:47 <amoralej> #endmeeting