15:03:36 <amoralej> #startmeeting RDO meeting - 2019-04-24
15:03:36 <zodbot> Meeting started Wed Apr 24 15:03:36 2019 UTC.
15:03:36 <zodbot> This meeting is logged and archived in a public location.
15:03:36 <zodbot> The chair is amoralej. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:03:36 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:03:36 <zodbot> The meeting name has been set to 'rdo_meeting_-_2019-04-24'
15:03:37 <openstack> Meeting started Wed Apr 24 15:03:36 2019 UTC and is due to finish in 60 minutes.  The chair is amoralej. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:03:38 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:03:40 <openstack> The meeting name has been set to 'rdo_meeting___2019_04_24'
15:03:48 <amoralej> #chair ykarel
15:03:48 <zodbot> Current chairs: amoralej ykarel
15:03:49 <openstack> Current chairs: amoralej ykarel
15:03:52 <mjturek> o/
15:03:54 <baha> o/
15:04:02 <fultonj> o/
15:04:45 <jpena> o/
15:05:24 <amoralej> #chair mjturek baha fultonj jpena
15:05:24 <zodbot> Current chairs: amoralej baha fultonj jpena mjturek ykarel
15:05:25 <openstack> Current chairs: amoralej baha fultonj jpena mjturek ykarel
15:06:41 <amoralej> let's start
15:06:45 <amoralej> #topic Move to tripleo-ci playbooks for ppc64le container builds
15:07:01 <mjturek> that's me and baha!
15:07:10 <amoralej> yeap, go on
15:07:17 <mjturek> so we have migrated to the tripleo-ci  playbooks
15:07:24 <mjturek> thanks for everyone who reviewed
15:07:31 <mjturek> we have a follow up in flight to  fix a small issue
15:08:06 <baha> (over at https://review.rdoproject.org/r/#/c/20425/)
15:08:08 <mjturek> but the standing issue is that amoralej has some concerns about how we configure repos
15:08:17 <mjturek> #link https://review.rdoproject.org/r/#/c/20425/
15:08:33 <mjturek> amoralej: could you detail what you might think is wrong?
15:08:35 <ykarel> dlrn repos vs CBS repos?
15:08:58 <amoralej> mjturek, i remember having that discussion with you, but can't remember my concern tbh
15:09:11 <mjturek> it was at  baha actually!
15:09:18 <mjturek> if that stokes your memory
15:09:21 <amoralej> oh, about deps?
15:09:44 <baha> [13:17:46]  amoralej: baha, could i see logs of container-build jobs in ppc64le nodes?, i have a doubt about how you configure repos
15:09:54 <baha> That was both the beginning and end of the discussion, because I was out at that moment =)
15:10:07 <baha> I did send you an email with logs and the script that we run, though.
15:10:51 <amoralej> #action amoralej to check repo configuration of build-container jobs for ppc64le
15:11:00 <mjturek> works for me!
15:11:02 <amoralej> i'll check if everything is fine anyway
15:11:12 <amoralej> and will read irc logs
15:11:18 <mjturek> cool thank you amoralej !
15:11:19 <amoralej> do you use promoted hash?
15:11:31 <mjturek> I'm  not positive
15:11:38 <mjturek> the hash is determined by the playbooks
15:11:52 <mjturek> all we indicate is master and periodic
15:12:13 <amoralej> i think that was my doubt
15:12:32 <amoralej> do you collect the full server logs somewhere?
15:12:45 <amoralej> i mean files in /etc/yum.repos.d/
15:12:52 <amoralej> or other config files
15:13:03 <mjturek> ahhh we can grab the repo files and email them to you if that helps
15:13:09 <leanderthal> o/
15:13:16 <mjturek> we have a build that we can red
15:13:17 <leanderthal> sorry for lateness
15:13:19 <mjturek> ref*
15:13:40 <ykarel> amoralej, by default delorean repos are setup by those playbooks
15:14:03 <ykarel> delorean current + delorean current-tripleo + delorean deps
15:14:04 <amoralej> ykarel, current-tripleo?
15:14:12 <ykarel> amoralej, ^^
15:14:24 <amoralej> ykarel, ack
15:14:35 <amoralej> i'll check it anyway
15:15:06 <ykarel> amoralej, https://github.com/openstack/tripleo-ci/blob/master/roles/tripleo-repos/tasks/main.yml#L73-L79
15:15:09 <ykarel> amoralej, ack
15:15:25 <rdogerrit> Merged rdo-infra/ci-config master: Update dependencies for ppc64le upstream containers job  https://review.rdoproject.org/r/20425
15:15:41 <amoralej> mjturek, so https://review.rdoproject.org/r/#/c/20425/ should be enough to get it working?
15:15:44 <mjturek> thanks ykarel :)
15:15:56 <amoralej> at least the only known issues, right?
15:15:59 <mjturek> that is the hope yes
15:16:02 <amoralej> ykarel, ack
15:16:07 <mjturek> correct the only known issue
15:16:15 <baha> Any known issues, yes! We haven't been able to test the full script upstream yet though, due to the dependency issues
15:16:16 <amoralej> thanks mjturek
15:16:33 <amoralej> hopefully we'll have ppc64le containers soon :)
15:16:36 <amoralej> let's move on
15:16:50 <mjturek> :)
15:16:57 <amoralej> #topic Nautilus and Stein
15:17:17 <fultonj> o/
15:17:21 <amoralej> #info start merging required reviews
15:17:34 <amoralej> fultonj, we have kind of everything we need pending to merge?
15:17:55 <weshay|rover> hey
15:18:14 <amoralej> #chair fultonj weshay|rover
15:18:14 <zodbot> Current chairs: amoralej baha fultonj jpena mjturek weshay|rover ykarel
15:18:15 <openstack> Current chairs: amoralej baha fultonj jpena mjturek weshay|rover ykarel
15:18:17 <weshay|rover> aye.. so the review mentions a migration process
15:18:30 <weshay|rover> links or info to that are where.. how long.. etc
15:18:45 <weshay|rover> are they queued up on top of this patch?
15:18:48 <fultonj> https://etherpad.openstack.org/p/tripleo-integration-squad-status
15:19:12 <fultonj> ^ weshay|rover yes, the patchs linked there
15:19:17 <weshay|rover> fultonj probably should be in the review
15:19:30 <weshay|rover> in the future :)
15:20:09 <fultonj> weshay|rover: sure
15:20:29 <weshay|rover> mwhahaha ^ are we going to hold merging while ceph updates are going on?
15:20:48 <fultonj> https://etherpad.openstack.org/p/RDO-Meeting
15:21:00 <mwhahaha> no i thought we were disabling the voting of the ceph stuff
15:21:18 <fultonj> yes we are disaboing the voting for ceph 001/004
15:21:27 <fultonj> disabling*
15:21:29 <weshay|rover> right.. do we want to merge anything while those jobs are non-voting?
15:21:30 <fultonj> temporarily
15:21:46 <amoralej> https://review.opendev.org/#/c/655220/
15:22:23 <leanderthal> #chair leanderthal
15:22:47 <fultonj> i thought making them non-voting would keep tripleo ci from breaking and then we could make them voting again when 001/004 are working
15:22:59 <fultonj> we keep them from voting so people can still merge
15:23:04 <fultonj> (i assumed)
15:23:26 <fultonj> i guess it depends on priority
15:23:52 <weshay|rover> fultonj this should be handled much like an IT change window imho
15:23:59 <fultonj> do we need to merge more changes faster or do we need to be more certain nothing breaks 001/004
15:24:27 <weshay|rover> where you move gating jobs to non-voting, having everything needed for the migration queued up and ready to merge while NOTHING else is merged.... once you are set w/ your patches...
15:24:37 <weshay|rover> bring the jobs back to voting.. and change window is closed
15:24:46 <fultonj> all jobs queued up
15:24:53 <fultonj> s/jobs/reviews
15:24:53 <weshay|rover> anyone disagree w/ that?
15:26:15 <fultonj> i assume a revert of https://review.opendev.org/#/c/655220 is sufficeint to bring it back to voting and so are handling things like you describe weshay|rover
15:26:23 <weshay|rover> aye
15:26:32 <fultonj> the only difference is that i didn't ask anyone to non +w while we're in that state
15:26:57 <fultonj> if i was going to, one way to implement it would be to keep them voting though
15:27:29 <fultonj> i guess we want the middle state we can merge if we have to, but we ask people to take it easy on the +w until they're voting again
15:27:48 <fultonj> weshay|rover: should i email openstack-discuss w/ [tripleo] for this?
15:28:02 <fultonj> i'm happy to do so
15:28:16 <fultonj> fyi: fmount: ^ gfidente ^
15:29:12 <fultonj> (silence is golden, i'll send an fyi to the list)
15:29:23 * gfidente not against the email but hopefully the timespan will be short
15:29:34 <weshay|rover> fultonj we were just discussing in our retrospective, that it would VERY helpful if when moving any job to non-voting emails are sent to the list and we have at least one ack from ci
15:30:03 <ykarel> since zuul is already overloaded, merging all those changes will take some time
15:30:10 <ykarel> http://zuul.openstack.org/status
15:30:18 <gfidente> weshay|rover fultonj maybe also mention we have all changes queued so it's not supposed to last long
15:30:30 <weshay|rover> gfidente that's awesome and thank you
15:30:43 <fultonj> weshay|rover: ok, i'll be happy to send the email
15:30:56 <weshay|rover> the potential to break things with these kinds of change.. either directly or indirectly is fairly high
15:30:56 <gfidente> weshay|rover thanks
15:31:03 <gfidente> weshay|rover yeah
15:31:05 <fultonj> i had mentioned it during tripleo meeting (not yesterday) but previous week and rdo meeting but i did't say non-voting
15:31:33 * fultonj has room to improve
15:31:40 <weshay|rover> and should be done carefully and with full communication.. meaning..  the folks who have to keep things running in CI..  have responded back to you
15:31:55 <gfidente> fultonj ack me too, thanks weshay|rover
15:32:14 <weshay|rover> fultonj right.. unfortunately we all know sometimes you have to say things many times
15:32:36 <fultonj> weshay|rover: yes, i will eamil today and always going forward
15:33:28 <fultonj> ykarel: you mentioned that promotions would be affected
15:33:38 <weshay|rover> the negative outcome I'm trying to prevent is of course more failing jobs, and going w/o promotions for weeks at a time
15:33:45 <weshay|rover> which I don't think anyone wants here
15:34:06 <ykarel> fultonj, yes after first two patches in patch merges jobs in promotion pipleline will fail
15:34:08 <ykarel> ceph jobs
15:34:17 <mwhahaha> if we switch the gate to non-voting we need to kick those jobs out of the promotion pipeline as well
15:34:19 <ykarel> first two patches in chain
15:34:23 <mwhahaha> so it shouldn't block promotion
15:34:56 <ykarel> yes can be done if window is long
15:35:00 <amoralej> fultonj, first round of reviews will switch only master or stein too?
15:35:09 <mwhahaha> should just be done, doesn't matter the window length
15:35:15 <ykarel> okk
15:35:18 <mwhahaha> that should be our operating procedure for these types of changes
15:35:20 <amoralej> we realized we need to backport at least t-h-t to stein
15:35:23 <mwhahaha> if we know it's going to break, disable in all the places
15:35:31 <ykarel> amoralej, tripleo-common also
15:35:49 <weshay|rover> gfidente fultonj coordinating w/ our ruck/rovers would be a minimum imho
15:36:26 <fultonj> amoralej: master first
15:36:39 <fultonj> should we backport non-merged change to stein now to queue it up?
15:36:43 <amoralej> fultonj, my doubt is if any of those changes will break stein
15:37:09 <amoralej> as some ci projects are branchless
15:37:35 <weshay|rover> tripleo-ci, tq/tqe are branchless
15:37:41 <amoralej> i.e. https://review.opendev.org/#/c/651286/ is updating stein
15:37:43 <ykarel> yes stein will break too also after tq and tqe patches are merged
15:37:43 <amoralej> too
15:38:34 <weshay|rover> rfolco|ruck ^
15:38:39 <weshay|rover> please read through the above
15:38:58 <amoralej> i think proposing cherry-picks for tht and tripleo-common may be good
15:39:17 <fultonj> amoralej: you mean in advance?
15:39:24 <amoralej> yes
15:39:26 <rfolco|ruck> weshay|rover, ack
15:39:27 <amoralej> to minimize impact
15:39:36 <fultonj> amoralej: ok
15:39:40 * fmount reading all
15:39:48 <amoralej> we can wait to give +W until it's merged in master
15:39:52 <amoralej> but proposing it should be fine
15:41:36 <rdogerrit> Gorka Eguileor proposed openstack/cinder-distgit rpm-master: WIP: Create python-cinder-common subpackage  https://review.rdoproject.org/r/17910
15:42:01 <fultonj> ok, i'll update https://etherpad.openstack.org/p/RDO-Meeting with a link to the stein cherry-picks
15:42:04 <amoralej> fultonj, can you create action item?
15:42:36 <ykarel> fultonj, can you also check my comment on tht patch, if valid good to fix before cherry-picks
15:43:00 <ykarel> https://review.opendev.org/#/c/651239/
15:43:21 <weshay|rover> fultonj do all these need to merge? https://review.opendev.org/#/q/topic:ceph-nautilus
15:44:25 <fultonj> #action fmount to cherry pick relevant stein nautilus changes and update https://etherpad.openstack.org/p/tripleo-integration-squad-status
15:44:50 <fmount> fultonj: ack ^
15:45:11 <fultonj> weshay|rover: we don't need all of topic:ceph-nautilus to merge
15:45:33 <weshay|rover> k.. so just what's in https://etherpad.openstack.org/p/tripleo-integration-squad-status lines 9-17
15:45:51 <fultonj> weshay|rover: yes
15:45:58 <fultonj> amoralej ykarel agree ^ ?
15:46:30 <amoralej> correct
15:46:35 <amoralej> + stein cherry-picks
15:46:45 <ykarel> yes
15:47:06 <ykarel> ORDER of merge to check: non-voting patch 655220 -> 652959 -> 651281 -> 651286 -> 651287 -> 651239
15:47:10 <fmount> weshay|rover: line 19 contains the order of merge
15:47:51 <weshay|rover> fultonj gfidente so you are starting to today..  how long do you expect we'll need? 1 day?  I see the patches have +2's
15:48:12 <fultonj> weshay|rover: hopefully 1 day
15:48:16 <fultonj> it depends
15:48:37 <weshay|rover> aye.. I know .. it can :)
15:48:40 <fultonj> we have stuff ready to go in so i think the machines, not people, would be the bottleneck
15:49:03 <weshay|rover> fultonj gfidente note.. that you are starting this process while 3rd party is in a degraded state
15:49:10 <weshay|rover> you are NOT getting full feedback
15:50:55 <fultonj> weshay|rover: you're referring to "RDO Third Party CI check" right?
15:50:56 <weshay|rover> hoping 3rd party is back online after https://review.opendev.org/#/c/655389/
15:51:03 <weshay|rover> right
15:51:18 <ykarel> weshay|rover, we need one more after ^^
15:51:31 <ykarel> https://review.opendev.org/#/c/655450/1
15:52:27 <weshay|rover> seems like a pretty bad time to start imho
15:53:19 <fultonj> weshay|rover: so you propose we start after 655450 + 655389 land?
15:53:22 <amoralej> weshay|rover, any ETA about when third party jobs are recovered?
15:53:47 <weshay|rover> fultonj I wouldn't start until 3rd party jobs are green..  amoralej should be today afaict
15:54:20 <amoralej> today would be great
15:54:21 <fultonj> only first move has been the +w of https://review.opendev.org/#/c/655220
15:54:52 <weshay|rover> fultonj and that's the most damaging patch
15:55:54 <fultonj> weshay|rover: added a -1 to workflow
15:56:04 <fultonj> i added
15:56:44 <weshay|rover> fultonj it's too late for that.. you have to abandon and restore
15:56:51 <weshay|rover> it's still in queue
15:57:09 <ykarel> i think it will not be merged even if all jobs pass
15:57:24 <ykarel> if there is no +W in the patch
15:57:41 <amoralej> that's my understanding too
15:57:46 <weshay|rover> k.. it's out of the queue now
15:57:55 <fultonj> weshay|rover: Abandoned
15:58:21 <ykarel> ok then restore only after third party is good
15:58:23 <weshay|rover> fultonj k.. you can restore it..  and get it queued up for a workflow
15:58:34 <weshay|rover> ykarel++
15:58:37 <amoralej> #agreed to wait for third party oooq jobs to be working before merging ceph-nautilus patches
15:58:52 <amoralej> is this correct ^ ?
15:59:04 <fultonj> amoralej: yes
15:59:10 <weshay|rover> I think that is smart
15:59:27 <weshay|rover> a little patience will keep us out of cix meetings :)
15:59:33 <ykarel> :)
15:59:42 <amoralej> :)
15:59:45 <amoralej> hopefully :)
15:59:55 <amoralej> btw, for the puppet side https://review.opendev.org/#/q/topic:ceph-nautilus+status:open+owner:%22Alfredo+Moralejo+%253Camoralej%2540redhat.com%253E%22
15:59:59 <weshay|rover> hopefully
16:00:01 <fultonj> thanks weshay|rover
16:00:07 <amoralej> if anyone can help
16:00:17 <weshay|rover> fultonj gfidente thanks to you guys!!!
16:00:27 <amoralej> it's quite late
16:00:31 <amoralej> let's move on
16:00:33 <amoralej> to the last topic
16:00:49 <amoralej> #topic Prepare announcement mail for RDO Stein - https://review.rdoproject.org/etherpad/p/stein-release-announcement
16:01:31 <amoralej> once we have everything working with nautilus we will send announcement of rdo stein
16:01:40 <amoralej> we need to start preparing announcement mail
16:01:46 <fultonj> thanks amoralej
16:02:21 <amoralej> i've created an etherpad to get new features or things that are worthy to include
16:02:26 <amoralej> as ceph nautilus :)
16:03:05 <amoralej> please, feel free to add it
16:03:36 <fmount> thanks amoralej and ykarel
16:04:01 <fultonj> #action fultonj to add ceph content to https://review.rdoproject.org/etherpad/p/stein-release-announcement
16:04:08 <jpena> I'll check the contributor list
16:04:11 <amoralej> #action all add content to the announcement mail for RDO stein
16:04:27 <amoralej> thanks jpena, i was about to ask leanderthal
16:04:32 <amoralej> but seems is not here
16:04:40 <jpena> we can get it using repoxplorer
16:04:51 <jpena> just need to remember the url for the server :)
16:04:51 <amoralej> ok, i wasn't sure about the tool we use
16:04:55 <amoralej> :)
16:05:07 <amoralej> i'll check what new packages were added
16:05:31 <ykarel> can use RDO-stein tracker
16:05:35 <ykarel> or rdoinfo
16:05:39 <amoralej> it'd be great if we could get it released before next meeting but i have some doubts we can get everything merged and released
16:05:49 <amoralej> yeah ykarel i use rdo_projects
16:05:53 <amoralej> to get the list
16:06:02 <ykarel> amoralej, yeah
16:06:03 <ykarel> good
16:06:25 <amoralej> let's move to open floor
16:06:29 <amoralej> #topic open floor
16:06:44 <amoralej> we are late but still have time in case someone has something to add
16:07:35 <amoralej> last call...
16:07:56 <amoralej> ok, i'll close the meeting
16:07:58 <amoralej> #endmeeting