15:01:26 #startmeeting RDO meeting - 2017-12-13 15:01:26 Meeting started Wed Dec 13 15:01:26 2017 UTC. The chair is chandankumar. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:26 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:01:26 The meeting name has been set to 'rdo_meeting_-_2017-12-13' 15:01:27 Meeting started Wed Dec 13 15:01:25 2017 UTC and is due to finish in 60 minutes. The chair is chandankumar. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:28 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:01:31 The meeting name has been set to 'rdo_meeting___2017_12_13' 15:01:41 #topic Roll Call 15:01:42 o/ 15:01:52 \o 15:01:55 o/ 15:02:01 #chair amoralej PagliaccisCloud jpena 15:02:01 Current chairs: PagliaccisCloud amoralej chandankumar jpena 15:02:02 Current chairs: PagliaccisCloud amoralej chandankumar jpena 15:02:10 o/ 15:02:16 o/ 15:02:16 #chair rbowen 15:02:16 Current chairs: PagliaccisCloud amoralej chandankumar jpena rbowen 15:02:18 Current chairs: PagliaccisCloud amoralej chandankumar jpena rbowen 15:02:23 #chair number80 15:02:23 Current chairs: PagliaccisCloud amoralej chandankumar jpena number80 rbowen 15:02:24 Current chairs: PagliaccisCloud amoralej chandankumar jpena number80 rbowen 15:04:09 EmilienM: apevec meeting time 15:04:23 hi there o/ 15:04:35 #chair dmsimard 15:04:35 Current chairs: PagliaccisCloud amoralej chandankumar dmsimard jpena number80 rbowen 15:04:36 Current chairs: PagliaccisCloud amoralej chandankumar dmsimard jpena number80 rbowen 15:05:04 so starting with today's first topic 15:05:09 hello 15:05:11 #topic Dry-run/test/firedrill of the queens release ? (in retrospect of the challenges from pike) 15:05:17 #chair EmilienM 15:05:17 Current chairs: EmilienM PagliaccisCloud amoralej chandankumar dmsimard jpena number80 rbowen 15:05:18 Current chairs: EmilienM PagliaccisCloud amoralej chandankumar dmsimard jpena number80 rbowen 15:05:22 dmsimard: please go ahead 15:05:58 I'd like to propose we change the way we ship new stable releases 15:06:36 Historically, we've populated the testing repositories with RCs and eventually the signed tarballs and then pushed everything to the stable release mirror when everything was ready 15:07:13 The problem with this, is that we only do the initial push to stable repos once every 6 months and the process is largely prone to code rot or other problems in general 15:07:41 We've seen this for pike, it took us several days to get the release out 15:08:31 What I think would be good would be to build everything in -testing and then as soon as everything is bootstrapped, we push to the stable repositories -- at this point we do not ship neither rdo-release or centos-release-openstack-*, we would only release those on the release day 15:08:32 So? 15:09:04 that means pushing RC releases for all packages to mirror.c.o 15:09:16 before GA, right? 15:09:19 This way, we are not chasing KB and other things in an emergency to ship ASAP 15:09:27 we just push hotfixes or new builds as required 15:09:29 amoralej: yes 15:10:02 i'm not sure if there is any policy in CentOS, about it, but i'm ok with that altough it may help less that we expect 15:10:09 amoralej: the fact that the packages are there does not make anyone start using them unless they manually change their repo files, install rdo release or centos-release 15:10:19 the only exception to this might be the common repo 15:10:26 but these are updated throughout the cycle already 15:10:58 amoralej: why would it help less ? We would have like 2 weeks to address potential issues with releasing instead of 2 days 15:11:05 dmsimard, depends 15:11:09 let me ask something first 15:11:11 we cannot remove the RC packages once pushed, right? 15:11:22 those pushes would have to pass promotion criteria? 15:11:30 jobs in rdoinfo gate 15:11:42 those pushes would have to pass whatever criteria there would be for a release 15:11:44 KB has also asked me to communicate that if we can give plenty of warning of an upcoming release, and several reminders a week, day, etc out, that will greatly improve the chances that he'll be available. 15:11:44 i'd say so 15:11:49 ok, then 15:12:09 the problem is that trailing services (tripleo, etc...) 15:12:19 push RC tags short time before GA 15:12:32 i think in pike it was 4 days or so 15:12:40 so yeah, we'd have that time 15:12:58 did we ever get around to wrapping up the definition of done ? I think I had an action on that.. 15:12:59 that could help, but maybe not as much as we'd like 15:13:19 because that reminds me we had the discussion or whether or not we actually shipped on release day or waited until cycle trailing was out 15:13:33 yes, in fact this is closely related 15:14:17 ini Pike, for example we almost couldn't push RCs to testing repo because of missing tags until very close of GA 15:14:41 RDO historically released (or at least tried to) on upstream release 15:15:09 yes, that's still the goal, i guess 15:15:29 but if we do that, we ship RC of cycle-trailing :p 15:15:46 which might be blocked due to bugs or whatever 15:15:51 (I mean promotion criteria) 15:16:03 so goal is to provide 1) working packages 2)as soon as posible 15:16:36 in last releases RC for deployment tools have been good enough to promote 15:16:42 and push on GA 15:16:46 number80: what are your thoughts on trying to ship to stable mirrors ASAP to give us more time to troubleshoot ahead of upstream release ? 15:16:48 amoralej: yeah 15:17:18 at some point they may not be but, i think the compromise is to keep it as-is and *try* to promote with RC for cycle-training 15:17:30 keeping in mind that at some point it may not be possible 15:17:45 but TBH, it may happen even with GA releases, right? 15:17:50 yeah 15:17:54 dmsimard: I think we need specific process to ship GA as it is as you said once-per-release thing 15:18:26 but relaxing CI jobs for GA push could be a different way 15:18:27 number80: the process (or the code involved in the process) is highly prone to bit rot 15:18:37 automation, scripts, documentation 15:18:40 we have worked to implement some improvements in tooling based on pike retrospective 15:18:41 a lot of things can change in 6 months 15:18:53 yeah 15:18:58 if we have a way to test this process from end to end without pushing to stable repos, I'm all ears 15:19:02 We have a document here - http://rdoproject.org/rdo/release-checklist/ - that talks about the peripheral things around a release, too, by the way. 15:19:03 and i think it should work better for queens but 15:19:07 because I mean, that's the purpose 15:19:12 Mostly that's more about communication than the actual release process. 15:19:39 But it's helpful to keep it in mind with regard to timing and whatnot. 15:20:05 if we think that we should move to a different model in queens to make it shorter and less prone to automation errors, we can do it 15:20:08 rbowen: yeah, it's about the technical aspect of things -- how we build RC releases, the automation involved, updating specs, updating rdoinfo, get them tagged to stable, getting someone to sign the packages, ship rdo-release and centos-release 15:20:51 amoralej, number80: despite having quorum, I would love to have apevec_'s opinion so maybe we can take this one to the ML 15:20:52 in fact, with current automation model based on rdoinfo, it's easy to push packages manually 15:21:03 ack 15:21:04 either with cbs or graffiti 15:21:11 it will not break anything 15:21:16 dmsimard: +1 on taking to mailing list for more eyes and ideas 15:21:22 for improvements 15:21:23 yeah 15:21:30 ok 15:21:47 dmsimard: can you take the action item for the same? 15:21:57 #action dmsimard to create a ML thread about testing the process of shipping a new stable release 15:22:07 dmsimard: thanks :-) 15:22:31 i think i can move to next topic then. 15:22:51 #topic Test day TOMORROW 15:23:11 RDO test day is tomorrow https://www.rdoproject.org//testday/queens/milestone2/ 15:23:27 Please join us, and please help get the word out to anybody you think might have an hour or two to try it out. 15:23:39 dmsimard is working on our testing cloud. 15:23:46 oh, not just me 15:23:48 amoralej and jpena too :D 15:25:40 Here is the instructions on how to get a test cloud 15:25:43 #link https://etherpad.openstack.org/p/rdo-queens-m2-cloud 15:26:15 Also if you have suggestions of what people might should test, please update the test matrix 15:26:29 That's a place where we have historically not done a great job. People that do show up aren't sure what to do. 15:27:23 rbowen: is it a good idea to create a meetup event in my local openstack user group if needed people can join from there? 15:27:53 That's actually a great idea. Thanks. 15:28:04 rbowen: i will create it right now 15:28:20 oh 15:28:20 I'll make a note of that for next time. It's a little late now but every bit helps. 15:28:26 Next time we can do it in advance. 15:28:28 I happen to be a speaker at the Montreal OpenStack meetup tonight 15:28:34 I'll totally advertise the test day there 15:28:41 dmsimard: awesome. Tell them all to show up and bang on it. 15:29:29 offer them a free poutine 15:29:33 they'll help 15:29:46 EmilienM: the poutine at openstack canada day in ottawa wasn't even that good :/ 15:29:56 I think this topic is exhausted. 15:29:56 disappointing. 15:30:12 moving to next topic, 15:30:26 #topic ICYMI: http://rdoproject.org/newsletter/2017/december/ (Thanks to mary_grace) 15:30:30 I wanted to mention here that I had a lot of help putting together December's newsletter 15:30:41 mary_grace did almost all the work there, and it's the best one we've ever had 15:30:51 So if you want to show off what we're doing, please do tell folks about it. 15:31:06 And help is always appreciated suggesting items for that newsletter. 15:31:18 It goes out to 2000+ people, and I have evidence that at least 3 of them actually read it. 15:31:19 Merged rdoinfo master: Add python-octavia-tests-tempest https://review.rdoproject.org/r/10523 15:31:35 15:32:18 thank you rbowen and mary_grace for the nice newslettter :-) 15:32:23 moving to next topic 15:32:47 #topic Review "rebase if necessary" strategy in gerrit (e.g rdoinfo) 15:32:52 o/ 15:32:58 #chiar EmilienM 15:33:17 #chair EmilienM 15:33:17 Current chairs: EmilienM PagliaccisCloud amoralej chandankumar dmsimard jpena number80 rbowen 15:33:19 Current chairs: EmilienM PagliaccisCloud amoralej chandankumar dmsimard jpena number80 rbowen 15:33:30 I want to know why rdoinfo project is configured in Gerrit in a way each patch has to be rebased to be merged. 15:33:43 it's super painful 15:33:51 it is too irirating. 15:33:56 That's also the case in the 'website' repo, and it is a fairly recent change. 15:33:56 and it refrains me to review from now 15:34:01 yeah, i understand it 15:34:08 It didn't used to be that way. I can't pinpoint when it changed. 15:34:15 so unless that thing changes, i'll probably stop reviewing this project 15:34:16 amoralej: ^ would it break things to put merge ? 15:34:17 but it's not so easy for rdoinfo 15:34:26 so 15:34:32 there are two problems 15:34:56 1. make sure it doesn't breake the diff-tags command ( i need to check it) 15:35:19 2. we are running deployment jobs (weirdo and tripleo) only in check pipeline, not in gate one 15:35:27 to minimize ci resources usage 15:36:00 and changing the policy to merge would mean that we would be merging untested rdoinfo content 15:36:04 so the problem I see is 15:36:07 Merged openstack/cinder-distgit rpm-master: Remove policy.json file https://review.rdoproject.org/r/10962 15:36:12 If we allow merge in rdoinfo, what can happen 15:36:28 we test change A and it works, we test change B and it works -- but testing them together fails 15:36:35 yeah 15:36:51 i see two options: 15:36:53 so zuul is kind of built to prevent exactly that 15:37:16 except I'm not sure how to prevent that scenario from happening 15:37:30 1) add jobs to gate pipeline + fixing rdopkg if needed + changing t merge 15:37:34 why would changing the merge stragegy allow untested code to be merged into rdoinfo? 15:37:48 you'd still need a +1 from jenkins / zuul right? 15:37:57 my understanding is that the solution is to add old jobs to gate pipeline 15:38:19 Merged rdoinfo master: Import ansible-role-redhat-subscription https://review.rdoproject.org/r/10934 15:38:29 pabelanger, i may be wrong here, but the problem is that we only run a subset of jobs in gate pipeline, not all in check one 15:38:48 so in check we are testing master + review, right? 15:39:10 but if there is an automatic rebase it's only tested in gate one 15:39:11 pabelanger: there's two changes, change A and change B -- each are individually tested irrespective of each other. If these two merge, the end result might not be the same 15:40:01 dmsimard: sure, you could fix that my making a dependant pipeline for rdoinfo, some sort of special check to support that 15:40:37 or, just keep gerrit merge stragegy the same for rdoinfo and change it globally every place else. IIRC, you can override it per project 15:41:15 I believe there's just two projects where we have this kind of issue -- the config repo and the rdoinfo repo 15:41:17 Merged rdoinfo master: Remove python-django-openstack-auth from queens https://review.rdoproject.org/r/10921 15:41:20 dmsimard: but the issue you are discribing is the same issue that happens upstream with tripleo-check today 15:41:48 EmilienM is suggesting we change the policy for rdoinfo to merge if required (which I guess is the default upstream) but the concern is about the scenario I told you about. 15:42:13 pabelanger, to be sure i understand correctly, if we have two reviews for rdoinfo, review A and B 15:42:15 I'd be happy to change the merge policy if zuul can help us test the changes together under the hood (which I guess by design it is supposed to do ?) 15:42:28 in check master + A and Master + B are tested 15:42:34 but with merge policies 15:42:36 I'm not very familiar with the concept of queue and dependant pipeline in zuul 15:43:04 in gate pipeline, jobs for review B wouldn't use master + A + Merge B ? 15:43:18 right 15:43:32 but that is not specific to gate, that is a dependant pipeline 15:43:42 which, you could make check or some other pipeline 15:44:09 https://docs.openstack.org/infra/zuul/feature/zuulv3/user/config.html#value-pipeline.manager.dependent 15:44:19 then, let's asume we make rdoinfo-check pipeline dependent 15:44:25 pabelanger: ah, okay so we could (theoretically) make check a dependant pipeline 15:44:46 however, at the risk of increasing complexity, why not just add jobs into gate and issue is solved by default. I guess you don't have resources? 15:44:47 it would convert in a kind of sequential ? 15:44:53 amoralej: right 15:45:03 but, at the risk of being more complex 15:45:26 because if patch A is broken, then all new reviews are broken 15:45:31 and will slow development 15:45:38 there is another option that would help to optimize it 15:45:48 re-distribute data files in rdoinfo 15:45:54 rdo-trunk created config master: Create project for octavia-tempest-plugin https://review.rdoproject.org/r/10977 15:46:05 that is one reason why having check / gate is powerful 15:46:18 pabelanger: the jobs that run on rdoinfo are pretty expensive 15:46:43 dmsimard, the problem with that if that if a job fail in that pipeline 15:46:46 for a review 15:46:57 all chained reviews need to be re-executed 15:47:10 dmsimard: sure, cloud resources are cheap :) 15:47:44 dmsimard, jpena, back to option 2), change rdoinfo files 15:47:51 i explained it in the meeting etherpad 15:48:03 if we would have per-repo files 15:48:08 amoralej: yah, you'd need a high rate of success for a dependant pipeline, otherwise you'll get into a gate reset loop 15:48:22 cloudsig-newton-testing-tags.yaml 15:48:27 cloudsig-ocata-testing-tags.yaml 15:48:28 and so on 15:48:38 then we'd only have to rebase when it's actually needed 15:48:47 to ensure tests consistency 15:48:52 but that needs change in rdoinfo 15:48:58 and in automation scripts 15:49:10 ok, so, just to summarize 15:49:30 the merge strategy in rdoinfo is the way it is for good reasons for the time being 15:49:41 we can change that, but it requires changes in configuration and tooling which might take a while 15:49:49 did I get that right ? 15:49:58 yes 15:50:00 EmilienM: ^ 15:51:24 the "easy" solution is fix rdoinfo (probably not much work) + more CI resources (add jobs to gate pipeline) 15:51:44 or change check pipeline to dependant for rdoinfo 15:52:13 more ci resources 15:52:14 hah 15:52:24 the "long" solution is to change rdoinfo data files, but i'd like that to be agreed with the team 15:52:46 as it may fit with the effort to transform rdoinfo in a python module 15:52:50 + data files 15:52:54 in a separated repo 15:53:08 ok 15:53:09 that has been discussed in the past 15:53:23 rdoinfo hasn't exactly scaled very well 15:53:33 it grew organically as we added stuff we needed 15:53:33 rdo-trunk created config master: Create project for ansible-role-redhat-subscription https://review.rdoproject.org/r/10978 15:53:42 I enjoyed doing review on this project! 15:53:51 it might be worth thinking about how we would like things to be 15:54:39 yeah, i agree 15:54:55 that's what i'd like before start changing things 15:55:39 I guess we're done with this topic 15:55:42 open floor ? 15:56:04 #topic chair for next meeting 15:56:26 Anyone up for volunteering for next meeting 15:56:28 ? 15:56:55 i can take it 15:56:56 Do we want to keep meeting on 27th Dec, 2017? As it is shutdown 15:57:04 A lot of people are going to be out the next 2 - 3 weeks. 15:57:09 #action amoralej to chair for next meeting 15:57:17 no, i think we'll cancel on 27th 15:57:22 Cancel on the 27th 15:57:53 #info 27th Dec, 2017 RDO community meeting is cancelled due to shutdown :-) 15:58:02 #topic openfloor 15:58:19 we still have 3 mins left any topic to kickstart 15:59:33 if not then let's end on a count of 3 15:59:35 ... 15:59:37 .. 15:59:40 . 15:59:45 #endmeeting