15:01:50 <amoralej> #startmeeting RDO meeting - 2019-05-29
15:01:50 <zodbot> Meeting started Wed May 29 15:01:50 2019 UTC.
15:01:50 <zodbot> This meeting is logged and archived in a public location.
15:01:50 <zodbot> The chair is amoralej. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:50 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:01:50 <zodbot> The meeting name has been set to 'rdo_meeting_-_2019-05-29'
15:01:51 <openstack> Meeting started Wed May 29 15:01:49 2019 UTC and is due to finish in 60 minutes.  The chair is amoralej. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:52 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:54 <openstack> The meeting name has been set to 'rdo_meeting___2019_05_29'
15:02:00 <amoralej> #topic roll call
15:02:02 <fultonj> o/
15:02:04 <ykarel> o/
15:02:06 <baha> o/
15:02:21 <jpena> o/
15:02:36 <amoralej> #chair fultonj ykarel baha jpena
15:02:36 <zodbot> Current chairs: amoralej baha fultonj jpena ykarel
15:02:37 <openstack> Current chairs: amoralej baha fultonj jpena ykarel
15:04:09 <mjturek> o/
15:04:17 <amoralej> #chair mjturek
15:04:17 <zodbot> Current chairs: amoralej baha fultonj jpena mjturek ykarel
15:04:19 <openstack> Current chairs: amoralej baha fultonj jpena mjturek ykarel
15:04:21 <amoralej> let's start
15:04:37 <amoralej> #topic  Planning for moving RDO Train to CentOS8 has started
15:04:45 <amoralej> #info https://review.rdoproject.org/etherpad/p/moving-rdo-to-centos8
15:05:17 <amoralej> CentOS has announced their work on CentOS8 https://wiki.centos.org/About/Building_8
15:06:09 <amoralej> we'd like to release RDO Train in CentOS8 so we've started planning all tasks to get it done
15:06:50 <amoralej> as there is no ETA for CentOS8 GA, it's difficult to define timelines
15:07:14 <amoralej> but at least, we have a draft on the work to do, order and how this will affect others
15:07:18 <amoralej> as tripleo-ci, etc...
15:07:22 <amoralej> weshay, ^ FYI
15:08:03 <amoralej> feel free to provide any feedback, questions and doubts
15:08:34 <amoralej> the initial approach is to provid RDO Train CloudSIG official repos *only* for CentOS8
15:08:51 <amoralej> although we may keep a RDO Trunk repo (DLRN) for CentOS7 too
15:09:45 <amoralej> so that's it from me, this was just for awareness
15:09:57 <amoralej> my main concern is for operators using RDO, to prepare for upgrade
15:10:22 <ykarel> hmm, i think there is some work going on in TripleO to do upgrades
15:10:29 <amoralej> yes
15:10:32 <amoralej> correct
15:10:37 <ykarel> where host is upgraded fresh with centos8 something like it
15:10:52 <amoralej> i'm not sure how will that go, but good to check with them
15:11:04 <jpena> btw, no matter what, Train will be the last release with Python 2 support, see https://governance.openstack.org/tc/resolutions/20180529-python2-deprecation-timeline.html
15:11:06 <ykarel> may be jistr can put some light and references on it &&
15:11:07 <amoralej> also, users not using TripleO should get prepared
15:11:13 <amoralej> jpena, yeap, i saw that
15:12:26 <amoralej> anyway, as soon as we have CentOS8 and can start working on it, i'll send a mail to RDO MLs explaining it
15:12:49 <ykarel> hmm good to send on CentOS as well
15:12:57 <ykarel> as i think we need involvment from other sigs
15:12:59 <amoralej> yeah, good point
15:13:25 <amoralej> correct, we need to check plans with storagesig, virtsig (not totally sure) and opstools
15:13:38 <amoralej> we depend on them somehow
15:14:18 <amoralej> i'll also share our plans in next centos buildsys meeting
15:14:27 <jistr> just a brief update: we're looking into upgrades indeed. Reprovisioning was an option but was sorta NACKed because it would put a lot of stress on storage services like Swift and Ceph. So we're looking into in-place upgrades only right now.
15:14:31 <ykarel> yes good
15:14:44 <jistr> currently CentOS 8 doesn't exist so we're not doing any work on that front yet
15:15:08 <jistr> when it gets released we'll see what are the upgrade options
15:15:10 <amoralej> jistr, your plan is to also in-place upgrades on CentOS/RDO?
15:15:14 <amoralej> jistr, ok
15:15:28 <amoralej> we'll check by then
15:15:35 <ykarel> jistr, Thanks for the updates
15:15:37 <jistr> amoralej: it's possible it will not happen, but i don't know. It depends what will be in-place upgrade options of CentOS
15:15:50 <amoralej> but i presume you'd like to do some sort of CI upstream if possible
15:16:08 <amoralej> if CentOS provides it
15:16:11 <jistr> amoralej: we do basic CI without a real OS upgrade. We would have no chance fitting into wall time otherwise.
15:16:26 <jistr> so we run upgrade tasks in "no-op" mode mostly
15:16:39 <amoralej> #action amoralej to check upgrades plans with jistr after CentOS8 is in available
15:16:41 <jistr> mainly validating the workflows that trigger the tasks, but not so much the tasks themselves
15:16:47 <ykarel> jistr, is there some place, etherpad etc where we can follow plan/status etc
15:17:24 <ykarel> may be it's being discussed in TripleO meetings
15:17:51 <amoralej> yeah, this is a tripleo topic mainly, but good to know to inform users
15:17:54 <jistr> ykarel: there's an approved spec where a lot of the problematic has been discussed. Other than that we don't do much tracking presently but if there's interest we can think of something.
15:18:31 <jistr> much of the current WIP code is here https://review.opendev.org/#/q/topic:leapp
15:18:33 <ykarel> jistr, yes good to have some visibility
15:19:11 <fultonj> jistr: as per my understanding all of the above is true, but i also learned at the ptg that controller nodes would be reprovisioned (which shouldn't be an issue as they're not storing as much data)
15:20:52 <jistr> fultonj: re controllers reprovisioned -- must have been some misunderstanding somewhere i think. We're currently testing in-place upgrade of controllers. If we have to make in-place upgrades work anyway, there's probably not much benefit in trying to mix in reprovisioning too...
15:21:40 <amoralej> it seems there is some leapp version in copr, not sure how useful it will be for CentOS https://leapp-to.github.io/gettingstarted
15:21:47 <amoralej> i guess we need to wait for CentOS8
15:22:31 <fultonj> https://etherpad.openstack.org/p/upgrades_denver_ptg
15:22:59 <fultonj> line 31
15:23:31 <fultonj> jistr: ^ it's quite possible i misunderstood :)
15:23:31 <jistr> fultonj: ok but that's for fast forward between Queens and Train
15:24:18 <fultonj> so it doesn't affect Rocky to Stein
15:24:21 <jistr> fultonj: because FFWD brings another set of trouble for upgrade (mainly around compute nodes and needed live migration because of OS upgrade, but live migration unsupported across 3 version difference)
15:25:10 <fultonj> jistr: ok, thanks i think i was confusing the U with the FFU.
15:25:30 <jistr> fultonj: yea it doesn't affect R->S. We're looking into other options for FFWD which we previously thought of as a no-go for R->S. The Q->T upgrade is much more constrained so it forced a complete re-evaluation of possibilities.
15:26:36 <fultonj> jistr: ack
15:27:52 <amoralej> jistr, note the major os upgrade in centos will be s-t
15:27:56 <amoralej> no r->s
15:28:06 <amoralej> or *->t
15:28:19 <amoralej> ok, let's move on to the next topic
15:28:21 <fultonj> amoralej: oh yeah, diff(upstream,downstream)
15:28:29 <amoralej> yeap :)
15:28:35 <jistr> amoralej: yea, that's not good for us, but kinda expected... :)
15:28:59 <amoralej> #topic RDO Status mail being sent weekly
15:29:32 <amoralej> last week some of us discussed about sending a weekly mail with info about RDO
15:29:52 <amoralej> to keep maintainers and users more in the loop about what's happening
15:30:12 <amoralej> we'll start sending it this week
15:30:42 <amoralej> in friday afternoon, CET time
15:30:51 <amoralej> or would it be better on monday?
15:31:04 <jpena> I'd say Monday
15:32:07 <amoralej> ok, i think it makes sense
15:32:30 <ykarel> +1 to Monday
15:32:38 <amoralej> #action amoralej to send RDO weekly status mail on monday
15:32:42 <amoralej> i'll start sending it
15:32:52 <amoralej> we can figure out how to rotate it
15:33:31 <amoralej> so, from now on, expect a new mail in your inbox every monday :)
15:34:06 <amoralej> #undo
15:34:06 <zodbot> Removing item from minutes: ACTION by amoralej at 15:32:38 : amoralej to send RDO weekly status mail on monday
15:34:07 <openstack> Removing item from minutes: #action amoralej to send RDO weekly status mail on monday
15:34:20 <amoralej> #action amoralej to send RDO weekly status mail every monday morning CET time
15:34:38 <amoralej> and final topic
15:34:59 <amoralej> #OpenStack clients update in Fedora status
15:35:24 <amoralej> #info all OpenStack clients have been updated in Fedora Rawhide to Stein releases
15:35:28 <amoralej> update is done
15:35:50 <amoralej> all packages are updated in Fedora rawhide repo and will be in F32
15:36:55 <amoralej> also some packages have been removed
15:37:02 <jpena> congrats, that was a lot of work
15:37:12 <amoralej> #info some packages have been removed from Fedora https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/DDVGRKS2EQMVD2XAZ7GK5ADIJC7AKXVK/
15:37:26 <ykarel> nice
15:37:42 <amoralej> i expect next ones to be a bit easier, without the python2/python3 mess and with the help of some scripts i used this time
15:38:29 <amoralej> time for open floor
15:38:33 <amoralej> #topic open floor
15:38:44 <amoralej> anything you'd like to bring to the meeting?
15:39:09 <baha> mjturek and I have a topic!
15:39:52 <baha> We've spoken to weshay and tonyb about the manifest list situation, and they seem fairly certain we ultimately won't need them in the RDO registry, which is good.
15:40:24 <baha> The proposal is to instead tag the images as ppc64le and leave them in the same namespace as the x86 images on the RDO registry end of things
15:40:40 <jpena> that sounds great
15:40:48 <baha> It's definitely easier :)
15:40:51 <amoralej> yeap
15:41:12 <amoralej> you will use manifest in docker registry? or follow the same approach?
15:41:25 <baha> That just means we need to figure out container uploads. The job is red right now because centos has been having some node outages, but it's otherwise ready to start pushing containers
15:41:34 <baha> We will use manifests in the docker registry yes!
15:41:50 <amoralej> ok
15:41:50 <baha> The idea is to use a promotion job as an in-between step to group all of them together via manifests
15:43:02 <amoralej> makes sense
15:43:54 <mjturek> we were having authentication issues with uploading previously
15:44:02 <mjturek> we'll probably need to rope you guys in to help with that
15:44:33 <mjturek> if you know how we can get access to the key that allows uploads to rdoproject registry, let us know!
15:45:28 <jpena> mjturek: I guess you'll be using the same user as the tripleo ci jobs, right?
15:45:47 <mjturek> jpena we're using root on cico
15:46:12 <jpena> mjturek: I mean from the rdo registry side. You need to authenticate with docker login -u xxx -p xxx
15:46:22 <mjturek> ahhh okay
15:46:49 <mjturek> so we'll need to get access to those creds?
15:47:30 <jpena> mjturek: yep. I need to check how to make them available to ci.centos
15:47:36 * jpena adds to to-do
15:47:47 <mjturek> awesome thank you!
15:48:18 <ykarel> ok i have one query, any idea when RDO zuul will be upgraded to move to ansible >= 2.7
15:48:27 <ykarel> i know there is already a plan, but any idea about the timeline
15:48:38 <jpena> ykarel: https://lists.rdoproject.org/pipermail/dev/2019-May/009087.html June 5th
15:48:48 <amoralej> ykarel, last plan i saw was for 2.7 not 2.8
15:49:26 <ykarel> jpena, Thanks
15:49:34 <ykarel> amoralej, hmm /me also aware about 2.7
15:49:54 <ykarel> amoralej, actually /me looking because 2.5 has a issue while running meta: reset_connection
15:50:13 <ykarel> moving to 2.7 will fix it
15:50:42 <amoralej> oh, i thought you were thinking in the issue with the 2.8 and containers-build
15:51:01 <ykarel> amoralej, it's related to that only
15:51:13 <ykarel> amoralej, more context https://review.opendev.org/#/c/660247/
15:51:42 <ykarel> amoralej, so i trying to move away from system ansible and rely on zuul's ansible to setup docker registry
15:51:56 <ykarel> but docker registry has a meta task: reset_connection
15:52:08 <ykarel> which doesn't work with conditions and ansible 2.5
15:52:19 <amoralej> that's probably the reason it was implemented in that way initially
15:52:44 <ykarel> hmm possible
15:53:02 <amoralej> that's what i didn't understand well in first place
15:54:01 <amoralej> so, after june 5th we probably can move to remove the embedded playbook and push ansible 2.8?
15:54:19 <ykarel> but with 5th june it will be 2.7
15:54:56 <ykarel> amoralej, i also pushed backup plan https://review.opendev.org/#/c/662021/
15:54:57 <amoralej> but iiuc that willfix the reset_connection?
15:55:25 <amoralej> ah, yeah, that should also be fine
15:55:26 <ykarel> amoralej, yes reset_connection will be fixed by it
15:56:06 <amoralej> then once we have 2.7 we can merge your initial fix and remove the nested playboot totally, including system ansible installation
15:56:06 <ykarel> yes it's good to move ansible-2.8 in Train, followed by stein, and currently docker_login issue is the only blocker
15:56:14 <ykarel> amoralej, yes
15:56:22 <amoralej> ok, it's clear then
15:56:49 <amoralej> any volunteer to chair next week?
15:57:00 * ykarel we have holiday here next week
15:57:09 <jpena> I can do it
15:57:26 <amoralej> #action jpena to chair next week meeting
15:58:01 <amoralej> anything else?
15:58:13 <amoralej> otherwise i will close the meeting?
15:59:11 <amoralej> thank you for joining!
15:59:20 <amoralej> #endmeeting