15:01:18 <amoralej> #startmeeting RDO meeting - 2019-03-20 15:01:18 <zodbot> Meeting started Wed Mar 20 15:01:18 2019 UTC. 15:01:18 <zodbot> This meeting is logged and archived in a public location. 15:01:18 <zodbot> The chair is amoralej. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:18 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:01:18 <zodbot> The meeting name has been set to 'rdo_meeting_-_2019-03-20' 15:01:19 <openstack> Meeting started Wed Mar 20 15:01:18 2019 UTC and is due to finish in 60 minutes. The chair is amoralej. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:20 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:01:22 <openstack> The meeting name has been set to 'rdo_meeting___2019_03_20' 15:01:38 <fultonj> o/ 15:01:40 <mjturek> o/ 15:01:47 <ykarel> o/ 15:01:55 <baha> o/ 15:02:04 <rdogerrit> Merged puppet/puppet-collectd-distgit stein-rdo: Update to post 10.1.1-rc0 (d7b6b8c493e808534bdd4b1e500065ce46e40434) https://review.rdoproject.org/r/19550 15:02:12 <amoralej> #chair fultonj mjturek ykarel baha 15:02:12 <zodbot> Current chairs: amoralej baha fultonj mjturek ykarel 15:02:13 <openstack> Current chairs: amoralej baha fultonj mjturek ykarel 15:02:38 <jpena> o/ 15:02:52 <amoralej> #chair jpena 15:02:52 <zodbot> Current chairs: amoralej baha fultonj jpena mjturek ykarel 15:02:53 <openstack> Current chairs: amoralej baha fultonj jpena mjturek ykarel 15:04:01 <fultonj> does anyone have the agenda link handy? 15:04:09 <mjturek> https://etherpad.openstack.org/p/RDO-Meeting 15:04:12 <amoralej> https://etherpad.openstack.org/p/RDO-Meeting 15:04:20 <fultonj> tks 15:04:23 <amoralej> please add your topics 15:04:26 <amoralej> we are starting 15:04:38 <amoralej> in 3 15:04:40 <amoralej> 2 15:04:42 <amoralej> 1 15:04:44 <amoralej> go! 15:04:59 <amoralej> #topic default maintainers are gone in rdoinfo: deps - https://review.rdoproject.org/r/19389 15:05:15 <amoralej> so, this was from apevec, but i think he's not around 15:05:23 <rdogerrit> Merged puppet/puppet-tomcat-distgit stein-rdo: Update to post 2.5.0 (5366908b1b86e9ba6d112b1dcc6eaf08118b4d7d) https://review.rdoproject.org/r/19553 15:05:39 <amoralej> #info dependencies without explicit maintainers have been reset to nobody 15:06:05 <amoralej> so now, we need for maintainers to sign up 15:06:41 <amoralej> for dependencies specific to a project or very limited set, they should be maintained by that project team 15:06:44 * Duck o/ 15:06:53 <amoralej> #chair Duck 15:06:53 <zodbot> Current chairs: Duck amoralej baha fultonj jpena mjturek ykarel 15:06:54 <openstack> Current chairs: Duck amoralej baha fultonj jpena mjturek ykarel 15:08:02 <amoralej> #actions maintainers to check deps in https://github.com/redhat-openstack/rdoinfo/blob/master/deps.yml and add themselves where applicable 15:08:40 <amoralej> #info the processes to manage deps are in https://www.rdoproject.org/documentation/requirements 15:08:42 <Duck> always better to see reality than expecting people who have moved to other things to care :-) 15:09:08 <weshay> guys is the rdo kibana working? 15:09:40 * ykarel doubt it's workin 15:10:00 <amoralej> i see an error "no results found" in https://review.rdoproject.org/app/kibana 15:10:42 <amoralej> ok, so i think that's it for the first topic 15:10:45 <amoralej> let's move on 15:10:58 <amoralej> we have quite long agenda today 15:11:07 <rdogerrit> Rafael Folco created config master: Make containers-build-push-base job branchless https://review.rdoproject.org/r/19572 15:11:07 <rdogerrit> Rafael Folco created config master: Use new containers-build-push-base branchless job https://review.rdoproject.org/r/19573 15:11:18 <amoralej> #topic finally drop *-patches repos (openstack/$PROJECT mirrors) in review.rdo 15:12:00 <amoralej> so i guess apevec only put this to start action of actually removing it 15:12:16 <amoralej> as this was discussed and agreed in the past 15:12:46 <amoralej> jpena, can we remove repos using resources directory in SF? 15:13:07 <jpena> amoralej: I *think* so, but we'll have to check 15:13:13 <amoralej> ok 15:13:19 <amoralej> i'll add you an action jpena 15:13:26 <jpena> ack 15:13:39 <amoralej> #action jpena to check how to remove repos using resources in SoftwareFactory 15:13:53 <amoralej> onec we know how to do it we'll get them out 15:14:07 <amoralej> all of them except the ones with patches applied 15:14:57 <amoralej> #topic RDO phase1 tripleo jobs running in ci.centos will be moved to some other system, weshay is looking into it 15:15:01 <amoralej> ykarel, this is yours 15:15:28 <ykarel> so CI Team is planning to run some baremetal scenarios, and ci.centos resulted some bottlenack in past 15:15:39 <weshay> we'll be looking into it.. 15:15:42 <ykarel> so weshay is looking for other systems 15:16:10 <weshay> I wouldn't say any decisions have been made, I'd love to know if we think there are any adverse implications from moving away from ci.centos 15:17:00 <amoralej> baremetal means *real baremetal* 15:17:01 <amoralej> ? 15:17:12 <weshay> we have baremetal hooked into internal sf.. where we could run the same jobs against and vote on rdo promotions 15:17:14 <weshay> via dlrn 15:17:16 <amoralej> or using vms with libvirt as done in ci.centos.org? 15:17:31 <weshay> so.. can be either, or both 15:18:07 <amoralej> so, i think we can keep weirdo ones in ci.centos.org, i'd say it has not been an issue so far 15:18:08 <weshay> there wouldn't be much value in rerunning real bm env on rdo if we did in tripleo 15:18:16 <weshay> but it's certainly possible 15:18:38 <weshay> right.. tripleo we have pacemaker disabled because the hardware in ci.centos can't support it 15:18:53 <weshay> so.. it's forcing unsupported settings essentially 15:18:58 <weshay> in tripleo 15:19:04 <amoralej> i'd say making jobs to vote in dlrn-api should not be a problem 15:19:07 <weshay> would rather run w/ clearly supported settings 15:19:10 <weshay> aye 15:19:27 <amoralej> it'd be good if we could maintain public logs for the reporting jobs 15:19:36 <PagliaccisCloud> \o 15:19:45 <amoralej> we used to do it in the past, i'm not sure what's the current status 15:19:50 <weshay> amoralej I believe the logs are public 15:19:51 <amoralej> #chair weshay PagliaccisCloud 15:19:51 <zodbot> Current chairs: Duck PagliaccisCloud amoralej baha fultonj jpena mjturek weshay ykarel 15:19:52 <openstack> Current chairs: Duck PagliaccisCloud amoralej baha fultonj jpena mjturek weshay ykarel 15:20:03 <weshay> wait.. they are not 15:20:12 <weshay> so ya.. we can log to the thirdparty log server 15:20:16 <weshay> as we did w/ rdo phase 2 15:20:22 <amoralej> yeap, exactly 15:20:28 <weshay> rock on 15:20:38 <amoralej> other than that, i'd say it's ok 15:20:48 <weshay> great.. 15:20:52 <ykarel> weshay, any ETA for this movement? or atleast a card/bug where this is tracked, so we can follow up there 15:21:14 <amoralej> so, the idea would be to move the oooq jobs running in ci.centos replaced by other jobs running in other more appropiate ci environment 15:21:25 <weshay> nothing scheduled, but I'll add it as an epic in taiga 15:21:34 <amoralej> keep reporting to dlrn-api, keep them as promotion criteria 15:21:47 <amoralej> but running in non public hardware 15:21:48 <ykarel> weshay, ack 15:21:50 <weshay> amoralej for promotion.. yes.. the check jobs we have in ci.centos will probably be killed 15:21:59 <amoralej> did i understand it right? 15:22:17 <weshay> ya.. non-public hardware but logging publically 15:22:20 <amoralej> yes 15:22:33 <amoralej> check jobs, you mean the ones voting upstream check gates? 15:22:58 <weshay> we have non-voting check jobs running from ci.centos on tq/tqe changes 15:23:10 <weshay> since ocata they have not been reliable 15:23:19 <amoralej> yeap 15:23:22 <amoralej> ack 15:23:26 <weshay> I may leave standalone jobs there 15:23:39 <weshay> probably will 15:24:10 <amoralej> ok 15:24:48 <amoralej> i'd like to keep puppet promotion jobs in ci.centos.org, i understand non-oooq are out of the scope for this change 15:24:50 <amoralej> right? 15:25:33 <ykarel> yes topic was that only 15:25:44 <ykarel> move TripleO jobs 15:25:52 <amoralej> ok 15:25:57 <amoralej> good 15:26:16 <amoralej> so i thing we are done with that topic 15:26:21 <amoralej> anything to add? 15:26:28 <ykarel> let's create an action to create epic 15:27:29 <amoralej> #action weshay to create an epic in taiga to move oooq jobs out of ci.centos.org 15:27:35 <amoralej> weshay, ^ are you ok? 15:28:03 <weshay> that's it 15:28:09 <amoralej> ok 15:28:20 <amoralej> #topic ppc64le container builds update 15:28:27 <mjturek> hey 15:28:34 <amoralej> #info logging uploads are working https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/417/logs/logs/ 15:28:41 <amoralej> congrats mjturek ! :) 15:28:51 <ykarel> cool 15:28:53 <mjturek> and baha and dmsimard ^ :) 15:29:08 <mjturek> looks like the base container started failing recently though 15:29:18 <mjturek> we merged a patch to kolla to fix 15:29:29 <mjturek> and are wondering when it would hit RDO? 15:29:43 <baha> (The patch just merged today, if that helps) 15:30:22 <amoralej> it will be probably in next dlrn run 15:30:30 <jpena> baha: what's the patch number? It's probably built in RDO Trunk alreayd 15:30:32 <amoralej> as kolla follows master trunk 15:30:37 <amoralej> https://review.openstack.org/#/c/644389/ 15:30:41 <amoralej> not built yet 15:30:52 <mjturek> oh awesome! 15:31:04 <amoralej> mjturek, you will see it coming in https://trunk.rdoproject.org/centos7-master/report.html 15:31:05 <baha> I was hoping that was how that worked, but we figured we'd check to make sure =) 15:31:05 <amoralej> soon 15:31:20 <amoralej> baha, what repo do you use? 15:31:22 <ykarel> amoralej, are they consuming kolla build from not promoted repo? 15:31:24 <rdogerrit> Kashyap Chamarthy proposed openstack/nova-distgit rpm-master: Use granular libvirt and QEMU RPMs for RHEL-8 Nova https://review.rdoproject.org/r/19556 15:31:24 <amoralej> consistent or promoted? 15:31:37 <amoralej> i assumed consisntent, but i'm not sure 15:32:24 <mjturek> I think we use current 15:32:46 <amoralej> it'd be better consistent, probably 15:32:46 <mjturek> should we be using consistent? 15:32:56 <mjturek> okay we can switch over to that I think 15:33:04 <amoralej> consistent is like current but before the last FTBFS 15:33:14 <mjturek> FTBFS? 15:33:24 <amoralej> Failed to Build from Source 15:33:29 <amoralej> means some package failed 15:33:31 <mjturek> got it! thanks amoralej 15:33:32 <amoralej> to build 15:33:46 <amoralej> we always rely on consistent repos to promote 15:33:52 <baha> We push master branch over to the container build script so yeah, I think we would be using whatever is latest 15:33:58 <amoralej> the ones with failed builds are considred "inconsistent" 15:34:18 <mjturek> I see, and may be part of current 15:35:20 <mjturek> anyway, we're good otherwise, just didn't know how to find out when patches hit rdo 15:35:26 <mjturek> thanks amoralej and jpena! 15:35:36 <baha> Yes, thank you! 15:35:46 <amoralej> mjturek, a new dlrn run just started 15:35:51 <amoralej> it should build that patch 15:35:53 <mjturek> awesome! 15:36:16 <amoralej> so, we can move on to next topic? 15:36:21 <mjturek> yep! 15:36:27 <amoralej> ok, thanks mjturek 15:36:31 <amoralej> #topic Status of Stein preparation 15:36:52 <amoralej> so, as you know projects are creating rc1 tags upstream 15:37:17 <amoralej> so we are currently building more and more packages in CBS for stein tags 15:37:33 <amoralej> we are in the process of branching disgtgits 15:38:14 <amoralej> #action maintainers should send required updates to package distgits for stein asap as new branches stein-rdo are being cut 15:38:35 <amoralej> we have frozen non-openstack puppet modules too 15:38:51 <amoralej> to the commits used in the last promotion 4 days ago 15:39:09 <amoralej> #info non-OpenStack puppe modules have been frozen for Stein 15:39:38 <amoralej> next steps will be to keep building rc tags for all available packages 15:39:51 <amoralej> and start creating jobs to test them in rdoinfo gate 15:40:18 <amoralej> also, synchronization from stein-testing to https://buildlogs.centos.org/centos/7/cloud/x86_64/openstack-stein/ is in place 15:40:30 <amoralej> i think that's the summary 15:40:37 <amoralej> ykarel, am i missing something important? 15:40:56 <ykarel> amoralej, no, all done 15:41:02 <amoralej> ok 15:41:11 <ykarel> we can link etherpad here 15:41:19 <amoralej> good 15:41:24 <amoralej> or to the card 15:41:34 <amoralej> #info https://trello.com/c/nkadXFWF/702-stein-release-preparation 15:41:35 <ykarel> yes that also works 15:42:13 <amoralej> jpena, btw, is the namespace for stein created in the rdo registry? 15:42:21 <jpena> amoralej: yes, it is 15:42:32 <amoralej> ok, i'll close that too then 15:42:50 <ykarel> amoralej, we also need to create current-tripleo and previous-current-tripleo symlinks 15:43:12 <amoralej> to bootstrap the ci jobs, right? 15:43:15 <ykarel> CI Team started creating jobs 15:43:25 <ykarel> yes those jobs will need those symlinks 15:44:00 <amoralej> ok, we'll find out what repos to use for that links 15:44:07 <amoralej> but ykarel 15:44:16 <ykarel> okk, jobs preparation:- https://review.rdoproject.org/r/#/q/topic:oooq/branching 15:44:22 <amoralej> if we start running only promotion periodic jobs 15:44:32 <amoralej> do those need current-tripleo? 15:44:48 <ykarel> amoralej, updates job need that afair 15:45:19 <amoralej> we can create it if needed 15:45:30 <amoralej> just i'd prefer not to do if we can avoid it 15:45:31 <ykarel> yes, we have some time 15:45:37 <amoralej> but we will not block 15:46:05 <ykarel> yes 15:46:14 <rdogerrit> wes hayutin created rdo-infra/ci-config master: update master criteria for promotion https://review.rdoproject.org/r/19574 15:46:48 <amoralej> jpena, wrt packstack, is there anything pending? 15:46:52 <amoralej> before creating rc 15:47:02 <jpena> let me check 15:47:41 <jpena> not a lot, we could add https://review.openstack.org/639075 and https://review.openstack.org/634910 15:47:42 <apevec> hmm rabbitmq update is failing CI 15:47:48 <rdogerrit> Ronelle Landy proposed config master: Report containers-build-push to DLRN https://review.rdoproject.org/r/19030 15:48:37 <amoralej> jpena, ack, i'll review those 15:48:46 <amoralej> ok, let's move on 15:49:06 <amoralej> #topic Nautilus update 15:49:16 <amoralej> fultonj, stage is yours 15:49:20 <fultonj> thanks 15:49:27 <amoralej> #info Ceph Nautilus was released yesterday: https://ceph.com/releases/v14-2-0-nautilus-released 15:49:30 <amoralej> \o/ 15:49:38 <fultonj> nautilus! 15:49:44 <fultonj> https://review.rdoproject.org/r/#/c/18721 15:49:48 <fultonj> we basically need to keep working on it 15:49:53 <amoralej> #info want to get Nautilus into Stein so we can use Ansible 2.7 as per https://review.rdoproject.org/r/#/c/18721 15:49:59 <fultonj> #action fultonj to investigate why cinder not working and give update in https://review.rdoproject.org/r/#/c/18721 15:50:21 <amoralej> #info stein GA should be released with nautilus ceph 15:50:31 <amoralej> also, some changes are required in puppet-ceph 15:50:40 <fultonj> puppet-ceph? 15:50:44 <amoralej> yes 15:50:48 <fultonj> we use ceph-ansible to deploy nautilus 15:50:58 <fultonj> what's using puppet-ceph? 15:51:06 <amoralej> there is a puppet module wich is still maintained by puppet-openstack team 15:51:17 <amoralej> and tested with puppet-openstack-integration scenarios 15:51:39 * ykarel need to leave 15:51:52 <rdogerrit> Merged rdo-infra/ci-config master: update master criteria for promotion https://review.rdoproject.org/r/19574 15:51:54 <fultonj> tripleo has deployed ceph without puppet-ceph since pike 15:52:27 <fultonj> still tested in the puppet-openstack-integration scenarios... is that using tripleo? 15:52:35 <amoralej> fultonj, that's the failure in legacy-weirdo-validate-buildsys-tags job in https://review.rdoproject.org/r/#/c/18721/ 15:52:53 <amoralej> fultonj, there are rdo users not using tripleo :) 15:52:59 <rdogerrit> Merged config master: Report containers-build-push to DLRN https://review.rdoproject.org/r/19030 15:53:01 <fultonj> amoralej: that's cool 15:53:33 <fultonj> hmmm 15:53:40 <amoralej> if needed we will request to remove ceph deployment temporarily in puppet-openstack-integration 15:53:45 <amoralej> until puppet-ceph is ready 15:53:59 <fultonj> amoralej: is the cinder failure in legacy-weirdo-validate-buildsys-tags ? 15:54:04 <amoralej> nop 15:54:15 <fultonj> ok 15:54:16 <amoralej> that failure is because puppet-ceph uses ceph-disk 15:54:25 <amoralej> which does not longer exist 15:54:27 <fultonj> ceph-disk removed 15:54:30 <fultonj> in nautilus 15:54:31 <fultonj> yep 15:54:32 <amoralej> it needs to be updated to use ceph-volume iirc 15:54:37 <fultonj> amoralej: exactly 15:55:34 <amoralej> so a new branch stable/mimic was cut with the last commit in master 15:55:44 <amoralej> and master will be fixed to work with nautilus 15:55:47 <amoralej> that's the plan 15:55:55 <amoralej> i thing tobias-urdin is working on it 15:56:31 <amoralej> it's a bad time for these changes as we are late in the cycle, but puppet modules are cycle-trailing 15:56:41 <amoralej> so i thing we still have time to get it done in stein 15:57:15 <amoralej> apart from that, fultonj i'll keep waiting on your debug of the cinder issue in ooo jobs 15:57:21 <amoralej> let me know if i can help somehow 15:57:24 <fultonj> amoralej: thanks 15:57:48 <fultonj> i guess that's all on that topic for now 15:57:49 <amoralej> i'm also concerned about how to push all those changes together and not break the gates 15:58:15 <amoralej> i mean, update ceph, ceph-ansible, ansible and ceph containers tags 15:58:18 <amoralej> coordinately 15:58:57 <amoralej> if we don't coordinate all those changes, we will break ooo gates 15:59:06 <amoralej> and maybe others, as puppet 15:59:17 <fultonj> gfidente fmount ^ fyi 15:59:23 <fultonj> amoralej: noted 15:59:39 <amoralej> but, let's figure out first how to get it working in the check jobs 15:59:50 <amoralej> we are almost out of time 15:59:57 <fultonj> ack 16:00:04 <amoralej> thanks fultonj 16:00:10 <amoralej> #topic open floor 16:00:19 <amoralej> any topic you want to arise? 16:00:34 <amoralej> well 16:00:40 <amoralej> chair for next meeting 16:00:44 <amoralej> i almost forget 16:00:48 <amoralej> any volunteer? 16:01:41 <jpena> I'll take it 16:01:51 <amoralej> thanks jpena 16:02:02 <amoralej> #action jpena to chair next meeting 16:02:15 <amoralej> i think we can close the meeting 16:02:24 <amoralej> last call for any other out of the agenda topic? 16:02:47 <amoralej> #endmeeting