15:00:37 <amoralej> #startmeeting RDO meeting - 2017-08-09
15:00:37 <zodbot> Meeting started Wed Aug  9 15:00:37 2017 UTC.  The chair is amoralej. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:37 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:37 <zodbot> The meeting name has been set to 'rdo_meeting_-_2017-08-09'
15:00:38 <openstack> Meeting started Wed Aug  9 15:00:37 2017 UTC and is due to finish in 60 minutes.  The chair is amoralej. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:39 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:41 <openstack> The meeting name has been set to 'rdo_meeting___2017_08_09'
15:00:51 <amoralej> #topic roll call
15:01:13 <PagliaccisCloud> yo o/
15:01:29 <dmsimard> \o\
15:01:31 <dmsimard> \o/
15:01:33 <dmsimard> /o/
15:01:49 <amoralej> :)
15:01:55 <chandankumar> o\
15:02:05 <amoralej> #chair dmsimard PagliaccisCloud chandankumar
15:02:05 <zodbot> Current chairs: PagliaccisCloud amoralej chandankumar dmsimard
15:02:06 <openstack> Current chairs: PagliaccisCloud amoralej chandankumar dmsimard
15:02:22 <rdogerrit> Yatin Karel proposed openstack/os-brick-distgit rpm-master: switch from oslosphinx to openstackdocstheme  https://review.rdoproject.org/r/8338
15:02:26 <number80> o/
15:02:31 <PagliaccisCloud> ᕕ( ᐕ )ᕗ ᕠ( ᐛ )و  ᕕ( ᐕ )ᕗ
15:02:32 <ykarel> o/
15:02:59 <dmsimard> ok PagliaccisCloud has got me there
15:03:39 <amoralej> #chair number80 ykarel
15:03:39 <zodbot> Current chairs: PagliaccisCloud amoralej chandankumar dmsimard number80 ykarel
15:03:39 <openstack> Current chairs: PagliaccisCloud amoralej chandankumar dmsimard number80 ykarel
15:04:38 <weshay> trown, amoralej fyi we have a problem here.. http://logs.openstack.org/33/481233/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq-pike/2094c73/console.html
15:04:48 <weshay> http://logs.openstack.org/33/481233/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq-pike/2094c73/console.html#_2017-08-09_13_02_37_199321
15:04:57 <amoralej> ok, let's start with the first topic
15:05:29 <weshay> oh sorry
15:05:30 <amoralej> weshay, we are in a meeting, i'll take a look later
15:05:32 <amoralej> no prob
15:05:56 <amoralej> #topic Bundling oslosphinx with openstackdocstheme for projects not landing migration in time for Pike upper-constraints (but failing master-head)
15:06:01 <amoralej> dmsimard, that's yours
15:06:06 <dmsimard> oh hai
15:06:24 <jschlueter> o/
15:06:25 <dmsimard> I haven't really had time to check how many projects were still in this situation, if any
15:06:44 <dmsimard> #link https://review.rdoproject.org/r/#/q/topic:openstackdocstheme
15:07:20 <dmsimard> The gist of the issue is that some projects are not going to land the openstackdocstheme patch in the first release of Pike due to feature freeze and because they have not tagged something in time
15:07:25 <amoralej> #chair jschlueter
15:07:25 <zodbot> Current chairs: PagliaccisCloud amoralej chandankumar dmsimard jschlueter number80 ykarel
15:07:25 <openstack> Current chairs: PagliaccisCloud amoralej chandankumar dmsimard jschlueter number80 ykarel
15:07:52 <amoralej> dmsimard, we had to apply the proposed solution of adding both oslo-sphinx and openstackdocsthem in a couple of packages
15:08:04 <dmsimard> This causes the problem that master-head is FTBFS and master-uc is building -- but if you submit a patch to master-uc it fails to build because DLRN-rpmbuild (CI) builds with master-head, not master-uc.
15:08:45 <dmsimard> Right, so I just wanted to have consensus that we /will/ be adding oslosphinx and docsthemes to a few projects in order to unblock this (there should be a clear TODO in the spec for removing it)
15:09:02 <number80> +1 from me
15:09:05 <amoralej> +1
15:09:24 <ykarel> +1
15:09:45 <dmsimard> jschlueter: you ok with ^ ?
15:09:50 <dmsimard> I know you hate sphinx :)
15:10:11 <jschlueter> dmsimard: I hate sphinx runtime dep, BuildDep ok
15:10:18 <dmsimard> ok, thanks
15:10:25 <jschlueter> +1
15:11:06 <dmsimard> #agreed projects blocked in transit between oslosphinx and openstackdocstheme will carry both packages as build requirements to unblock until there is a proper tagged release with only openstackdocstheme in it.
15:11:11 <jschlueter> dmsimard: maybe stage a review for removing it with a workflow -1 and a well identified upstream commit or condition to watch for
15:11:47 <dmsimard> jschlueter: it's something that I would like to be picked up automatically in the future
15:11:47 <amoralej> jschlueter, i added FIXME notes but yeah, it makes sense to create the review with W -1
15:11:57 <dmsimard> jschlueter: as part of syncing requirements between reqs.txt and spec
15:12:09 <dmsimard> jschlueter: i.e, tooling sees that oslosphinx is no longer there, drops it
15:12:23 <dmsimard> but yes, there should be a TODO/FIXME
15:12:29 <jschlueter> dmsimard: that's a big leap from where we are currently
15:12:30 <rdogerrit> Yatin Karel proposed openstack/os-brick-distgit rpm-master: switch from oslosphinx to openstackdocstheme  https://review.rdoproject.org/r/8338
15:12:46 <dmsimard> jschlueter: it's something I wrote about two years ago :)
15:12:59 <dmsimard> jschlueter: https://trello.com/c/woaK75me/132-create-a-tool-to-match-requirements-between-reqstxt-and-specfiles
15:13:15 <dmsimard> ok, if there isn't anything else, we can move on to next topic
15:13:43 <amoralej> ok
15:13:48 <amoralej> #topic Any particular plan around shipping Pike almost simultaneously with CentOS 7.4 ?
15:13:59 <dmsimard> that's mine too
15:14:04 <amoralej> #info https://seven.centos.org/2017/08/centos-linux-7-1708-based-on-rhel-7-4-source-code/
15:14:24 <amoralej> #info As reference, the issues found in 7.3 release - https://review.rdoproject.org/etherpad/p/release-centos-7.3
15:14:29 <dmsimard> I just wanted to gently remind everyone that we're essentially shipping 7.4 and pike side by side
15:14:58 <amoralej> yeah, more fun :)
15:15:13 <dmsimard> like, we're almost in RC and 7.4 is going to be rolling out in continous release stream
15:15:23 <PagliaccisCloud> thisisfine.gif
15:15:46 <dmsimard> I'll basically ask a crazy question just for the sake of discussion and ideas
15:16:16 <dmsimard> If 7.4 is not out by Pike, are we delaying the release of Pike ?
15:16:25 <dmsimard> Or the other way around
15:16:33 <dmsimard> well, not that we can delay 7.4
15:17:16 <rdogerrit> Matthias Runge proposed config master: Add four more packages to centos-opstools  https://review.rdoproject.org/r/8348
15:17:17 <dmsimard> I guess what I'm trying to convey (badly) is that there could be things in 7.4 that makes it hard to support both 7.3 and 7.4
15:17:18 <number80> dmsimard: no delay for our release
15:17:26 <amoralej> so, definition of done is that the repos pass ci pipeline
15:17:27 <number80> we'll fix RHEL 7.4 issues when they arise
15:17:39 <dmsimard> myoung: are you there ?
15:17:50 <number80> myoung and weshay will be testing trunk on RHEL 7.4
15:17:57 <myoung> dmsimard: ping
15:17:57 <zodbot> myoung: Ping with data, please: https://fedoraproject.org/wiki/No_naked_pings
15:18:11 <dmsimard> myoung: have you been testing RDO on RHEL 7.4 since like, last week (or even before that?)
15:18:48 <myoung> dmsimard: RDO on RHEL 7.3, earlier this week we talked about testing on 7.4.  I can likely give that a try later on today or tomorrow
15:18:51 <amoralej> some packages in rhel are not provided in centos base but in sig repos, specially kvm, that tends to be a problem
15:19:11 <amoralej> and ceph
15:19:14 <myoung> RDO Phase 2 tests the images created (centos) in RDO Phase 1, as well as images I build that are RDO (master, ocata, newton) + RHEL 7.3
15:19:26 <rdogerrit> Gabriele Cerami proposed config master: tripleo-upsteam: calling script from the cloned project dir  https://review.rdoproject.org/r/8356
15:19:30 <dmsimard> myoung: the faster we can test 7.4 to identify any potential issue the better
15:19:38 <weshay> ++
15:19:41 <dmsimard> because we still don't have 7.4 in centos and we won't for a while
15:19:47 <weshay> ++
15:20:14 <dmsimard> we will only have it on ci.centos, however we could enable centos CR repo in our DIB images so that we can see problems on jobs running from review.rdo
15:20:17 <amoralej> temorarily we need to support both 7.3 and 7.4, as jobs in rdo-ci will run with 7.4 from CR but jobs in rdo-cloud and in jobs in upstream gates
15:20:23 <myoung> weshay, dmsimard: ack, I'll put them in non-voting however, as I don't want to turn everything red until we shake out the issues, presently we're on 7.3 to match RDO (centos 7.3)
15:20:24 <number80> dmsimard: it was decided already to have such pipeline internally
15:20:40 <dmsimard> myoung: sure, they don't need to vote.
15:20:50 <dmsimard> number80: what pipeline ?
15:20:53 <myoung> oh...
15:20:54 <myoung> o/
15:20:57 <amoralej> +1 to enable CR in images for jobs running from ci.centos.org
15:20:59 <weshay> rdo phase2
15:21:02 <number80> RDO testing on 7.4
15:21:04 <amoralej> #chair myoung weshay
15:21:04 <zodbot> Current chairs: PagliaccisCloud amoralej chandankumar dmsimard jschlueter myoung number80 weshay ykarel
15:21:05 <openstack> Current chairs: PagliaccisCloud amoralej chandankumar dmsimard jschlueter myoung number80 weshay ykarel
15:21:06 <dmsimard> number80: oh ok
15:21:24 <dmsimard> amoralej: you would only enable CR on the weirdo ci.centos.org jobs ?
15:21:41 <amoralej> dmsimard, yes, we should use the same repos in all jobs in a pipeline
15:21:53 <amoralej> we shouldn't run oooq jobs with CR and weirdo without it
15:21:54 <amoralej> and
15:21:58 <amoralej> the sooner the better
15:22:07 <amoralej> it'd be nice if we could have both images
15:22:07 <dmsimard> amoralej: ok, we can do that -- it'll also be an opportunity to update that image to the same we are using in review.rdo
15:22:20 <dmsimard> amoralej: well, not exactly the same, but more in line
15:22:47 <amoralej> dmsimard, ok
15:23:14 <dmsimard> #action dmsimard to update the weirdo dib image for ci.centos.org jobs to enable centos CR
15:23:57 <dmsimard> #action myoung to start testing RDO on RHEL 7.4
15:24:00 <dmsimard> myoung: ^ fair ?
15:24:04 <amoralej> just we need to be careful if we need to push something to fix jobs in 7.4 to not break 7.3
15:24:40 <dmsimard> amoralej: the tricky part is that fixes can end up being anywhere.. in weirdo, in puppet, p-o-i, in packaging..
15:24:47 <dmsimard> we'll have to do on case by case basis
15:24:52 <amoralej> yes
15:24:57 <myoung> dmsimard: ack, fair.  Earlier this week I brought online 7.4 slaves so in theory things should be good to go.
15:25:12 <amoralej> last time we created a temporary repo with fixes
15:25:32 <amoralej> that was enabled only when using 7.4
15:25:38 <amoralej> 7.3, i mean
15:25:40 <dmsimard> I remember that
15:26:01 <dmsimard> ok, anything else we need to think about ?
15:26:07 <myoung> dmsimard, amoralej: yes, RDO on RHEL took some experimentation, and there were a number of odd packaging / dep issues to resolve.  It's going to be iterative but I think it's more understood than when we started doing this last fall.
15:26:08 <dmsimard> WWAS ? (What would Alan say)
15:26:54 <amoralej> :)
15:27:16 <number80> dmsimard: we support RHEL so we ought to find a solution
15:27:29 <dmsimard> number80: oh, I remember something
15:27:32 <number80> likely a temporary repo to host fixed deps
15:27:48 <dmsimard> number80: I saw in the CentOS 1708 blog post that the ppc build is likely not going to land simultaneously
15:27:54 <dmsimard> is that an issue ?
15:27:55 <number80> my fear is that copr do not use EPEL buildroot which are based on RHEL
15:28:14 <number80> dmsimard: ppc64le is not primary arch so acceptable
15:28:30 <number80> we haven't finished bootstrap nor have a full CI pipeline
15:28:58 <dmsimard> number80: ok, just thought I would point that out
15:29:09 <number80> ack
15:29:10 <dmsimard> number80: are we shipping a stable ppc64le release ?
15:29:22 <number80> dmsimard: we're trying too
15:29:26 <number80> -o
15:29:37 <dmsimard> ok
15:30:03 <dmsimard> I think we can move on
15:30:07 <amoralej> ok
15:30:20 <amoralej> #topic DLRN builder instance migration to RDO Cloud
15:30:30 <amoralej> that's from jpena, but i'll introduce it
15:30:50 <amoralej> #info Plan: https://review.rdoproject.org/etherpad/p/DLRN-builder-migration
15:30:59 <amoralej> #info Proposal: do it during the week of Aug 14-18
15:31:17 <amoralej> so, we are moving DLRN builder instance to RDO cloud
15:31:56 <amoralej> one of the drivers is that we have the database in rdo-cloud and having dlrn close to it should help to alleviate db access issues
15:32:10 <dmsimard> not just that but also improve build performance in general
15:32:16 <amoralej> yes
15:32:21 <dmsimard> amoralej: we're doing it builder by builder, right ? not all at once ?
15:32:38 <amoralej> yes, builder by builder
15:32:47 <amoralej> repos will be always available at trunk.r.o
15:33:15 <amoralej> but we need to disable build process while syncing content between old and new server
15:33:34 <dmsimard> yeah, I think the downtime should not be /too/ bad for each individual builder, depends on how long the rsync takes
15:33:43 <dmsimard> We're still purging on a regular basis, yes ?
15:33:47 <amoralej> yes
15:33:59 <amoralej> we may even run a head-only if the lag is too big for master-uc
15:34:31 <dmsimard> yeah ironically master-uc is the one that is going to take the longest and has the most builds
15:34:50 <amoralej> one of the impact is that we should change jobs in ci.centos to use public instance instead of internal
15:35:02 <dmsimard> We're looking at ~185GB of data to transfer across all builders, it's not so bad.
15:35:25 <dmsimard> amoralej: We can also consider leaving the fedora and newton builders in ci.centos to die
15:35:36 <dmsimard> amoralej: considering we need to move to f26 and newton will EOL soon
15:36:04 <amoralej> mmmm, that sounds good dmsimard
15:36:17 <dmsimard> amoralej: I think we only need to move trunk-primary.rdoproject.org DNS for the jobs to start using the new place
15:36:17 <amoralej> when is newton eol?
15:36:36 <dmsimard> expected 2017-10-11
15:36:49 <amoralej> dmsimard, but we want them to keep using the builder or should be moved to the server?
15:36:54 <amoralej> to public, i mean
15:37:12 <dmsimard> I don't understand the question
15:37:28 <amoralej> you mean, move trunk-primary to the new server, right?
15:37:48 <dmsimard> yeah, it's ok to keep the concept of trunk-primary and trunk-backup
15:38:07 <amoralej> we have some jobs which use repositories from trunk-primary, mainly taking advantage of low latency as it was close
15:38:09 <dmsimard> but I think jobs only use it to retrieve the hash and promote unless mistaken
15:38:16 <amoralej> ah, ok
15:38:17 <dmsimard> they actually use trunk.rdo for repos
15:38:23 * dmsimard looks
15:38:27 <amoralej> i thought they use it for everything
15:38:51 <dmsimard> I know weirdo doesn't
15:38:59 <dmsimard> it just recovers the hash and uses trunk.rdo
15:39:04 <amoralej> no, only jobs in ci.centos, oooq
15:39:45 <amoralej> ok, then we are safe
15:39:55 <dmsimard> https://github.com/rdo-infra/ci-config/blob/master/jenkins/jobs/scripts/centos-master-current-tripleo.sh
15:40:01 <dmsimard> https://github.com/rdo-infra/ci-config/blob/master/jenkins/jobs/scripts/promote-repo-promote.sh
15:40:25 <dmsimard> trown, weshay: OOOQ doesn't actually use the "trunk-primary.rdoproject.org" DNS right ? That's just to know where to promote ?
15:40:41 <dmsimard> like, it uses the DLRN hash and appends it to trunk.rdo ?
15:41:25 <weshay> dmsimard, it only uses it in ci
15:41:37 <dmsimard> weshay: define "in ci"
15:41:54 <dmsimard> weshay: because we're essentially moving "trunk-primary" and we want to know what to pay attention to
15:42:58 <weshay> ya.. I'm checking the configs now
15:43:03 <amoralej> dmsimard, weshay i think it uses trunk.r.o https://github.com/openstack/tripleo-quickstart/blob/master/config/release/centosci/pike-current-tripleo.yml#L10
15:43:07 <weshay> tq/config/release/centosci
15:43:54 <amoralej> i think it should be ok
15:44:11 <weshay> ya.. looks like I'm out of date
15:44:22 <weshay> https://github.com/openstack/tripleo-quickstart/blob/master/config/release/centosci/master.yml
15:44:49 <amoralej> ok,
15:45:11 <amoralej> so the plan is to start doing the migration on 16-aug
15:45:33 <amoralej> unless there is any issue with that date
15:45:57 <dmsimard> weshay: cool, thanks
15:47:03 <dmsimard> amoralej: I added some notes to the plan etherpad
15:47:07 <amoralej> #action jpena to send a mail to rdo-cloud about the migration planned date
15:47:14 <dmsimard> s/rdo-cloud/rdo-list/
15:47:15 <amoralej> work for jpena ^ :)
15:48:52 <amoralej> ok, let's move on
15:49:10 <amoralej> #topic We still haven't done any work towards shipping stable release of containers through CentOS registry
15:49:39 <amoralej> dmsimard, ^
15:49:44 <dmsimard> yeah
15:50:08 <dmsimard> weshay, Slower, EmilienM, jschlueter ^
15:50:26 <dmsimard> Pike release is near and we have zero work done towards releasing what we would call stable container images
15:50:34 <amoralej> here "stable release" means cloudsig repos, right?
15:51:02 <dmsimard> yes, and unless we change our mind, it means getting the containers build out of the CentOS registry pipeline
15:51:11 <dmsimard> it's unfortunate that apevec isn't here to discuss this..
15:51:47 <dmsimard> considering the container images are tripleo specific, I wonder if it's still relevant to publish them under the "cloug sig" brand
15:51:56 <amoralej> dmsimard, we can add it to next week meeting also
15:52:05 <dmsimard> or if we want to just publish them on dockerhub or something
15:52:37 <dmsimard> building the container images out of the centos registry pipeline would be an interesting experience because it is somewhat close to the downstream approach but it might turn out to be a lot of work
15:52:58 <dmsimard> amoralej: apevec is back next week ?
15:53:01 <amoralej> i don't have context about that discussion but using centos registry for that sounds as a good idea
15:53:16 <amoralej> ups, i don't think so
15:54:10 <amoralej> dmsimard, using centos registry requires to do anything special to build images?
15:54:16 <amoralej> or we could use kolla-build
15:54:19 <dmsimard> amoralej: yeah, it's like downstream
15:54:31 <dmsimard> amoralej: we need to provide pseudo distgit repos with Dockerfiles
15:54:32 <number80> registry is just registry
15:54:48 <dmsimard> amoralej: and then they have jenkins automation to build those Dockerfiles and push them to the registry
15:54:52 <amoralej> so, it's not only at push time, but at build time
15:55:23 <amoralej> it sounds as quite different to what we have :(
15:55:35 <dmsimard> amoralej: same thing as downstream with brew afaict
15:55:39 <amoralej> can kolla only create dockerfiles instead of doing the images?
15:56:07 <amoralej> but kolla has some idea about parent/child images, right?
15:56:28 <amoralej> that needs to be followed when building the images, i mean
15:57:35 <amoralej> dmsimard, so, what'd be the action about that?
15:58:06 <dmsimard> I have absolutely no idea, from the different discussions I've had this can go in any direction
15:58:18 <dmsimard> It doesn't sound like TripleO was interested in having RDO as their upstream for containers
15:58:41 <amoralej> i'm thinking in users, not in upstream for tripleo, tbh
15:58:49 <dmsimard> so in that sense, it would mean that TripleO would be responsible for building and making their stable containers available
15:59:06 <dmsimard> amoralej: kolla can generate just the dockerfiles, yes
15:59:21 <dmsimard> amoralej: and yes, there is the notion of relationship between images
15:59:31 <dmsimard> amoralej: jschlueter has some script to build the things in the right order
15:59:51 <amoralej> so, tripleo can build the images and provide a registry from undercloud, right?
16:00:15 <amoralej> that could be an option to users until there is some public registry with images
16:00:33 <dmsimard> that gets a bit too in-depth for me to answer, I know there is a registry somewhere but I don't know if users are expected to build new images themselves
16:01:23 <jschlueter> dmsimard, amoralej: I do have some of that logic.
16:01:39 <number80> dmsimard: no, we are supposed to ship them at some point but not necessarily for Pike
16:01:59 <dmsimard> number80: who we, and what them
16:02:03 <dmsimard> number80: it's an important nuance
16:02:14 <number80> we == RDO
16:02:29 <dmsimard> RDO is accountable for building and shipping tripleo containers ?
16:02:36 <number80> them == containers usable by tripleo
16:03:03 <dmsimard> I would see RDO accountable for shipping vanilla containers
16:03:07 <number80> AFAIK, providing artefact usable by tripleo is part of RDO Definition of Done
16:03:33 * number80 is not pushing for this version of DoD but until it's changed, this is the only one we have
16:03:46 <dmsimard> ok, we'll have to discuss with the folks involved in tripleo
16:04:04 <dmsimard> we're over time for now
16:04:11 <dmsimard> let's make it a topic next week too.
16:04:17 <dmsimard> although I won't be there, /me on PTO next week
16:04:55 <number80> dmsimard: that's a discussion for the mailing, it will be a long long one :)
16:05:09 <amoralej> yeah, i think we need to involve other folks
16:05:12 <dmsimard> number80: it's an important distinction to make
16:05:21 <dmsimard> number80: RDO provides packaging and the distribution of that packaging
16:05:40 <dmsimard> number80: those are containers specific to the tripleo project
16:06:00 <dmsimard> we have to be careful about what we do and why we do it
16:06:27 <dmsimard> I'll fight for RDO to stay as neutral and agnostic as possible even if that means I'll come across as the bad guy :)
16:06:47 <number80> Yes, but we made that definition of DoD before containers, we are accountable for these artefacts until we change it
16:07:17 * number80 playing Devil Advocate here
16:07:47 <dmsimard> sure
16:08:14 <number80> I'm not found of the idea of providing external stuff either, but we have to be consistent and not have RDO rely on *random* third-party containers
16:08:39 <number80> (let's close the meeting and someone chairing the next one)
16:08:45 <number80> *have
16:08:47 <amoralej> yeah
16:09:14 <amoralej> #topic volunteers to chair next meeting?
16:10:13 <amoralej> no volunteers?
16:10:41 <amoralej> ok, i'll take it again
16:10:50 <amoralej> #action amoralej to chair next meeting
16:11:02 <amoralej> we are out of time, but there is anything for open floor?
16:11:07 <amoralej> #topic open floor
16:11:49 <amoralej> ok, i'll close the meeting
16:11:56 <amoralej> thanks everyone for joining!
16:12:03 <amoralej> #endmeeting