15:00:53 #startmeeting RDO meeting - 2016-10-05 15:00:53 Meeting started Wed Oct 5 15:00:53 2016 UTC. The chair is jpena. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:53 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:00:53 The meeting name has been set to 'rdo_meeting_-_2016-10-05' 15:00:59 Meeting started Wed Oct 5 15:00:52 2016 UTC and is due to finish in 60 minutes. The chair is jpena. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:01 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:01:03 The meeting name has been set to 'rdo_meeting___2016_10_05' 15:01:17 dmsimard: can you remove voice from the openstack bot? 15:01:27 #topic roll call 15:01:30 \o 15:01:33 \0 15:01:41 o/ 15:01:42 o/ 15:01:43 Here, but only partially. Finishing up in another meeting. 15:01:43 o/ 15:01:47 o/ 15:01:51 #chair dmsimard weshay rbowen imcsk8 jruzicka coolsvap apevec tosky 15:01:51 Current chairs: apevec coolsvap dmsimard imcsk8 jpena jruzicka rbowen tosky weshay 15:01:57 * coolsvap copies rbowen 15:02:02 o/ 15:02:17 \0/ 15:02:53 EmilienM: cool! thanks 15:02:55 o/ 15:03:05 #chair jschlueter 15:03:05 Current chairs: apevec coolsvap dmsimard imcsk8 jpena jruzicka jschlueter rbowen tosky weshay 15:03:22 o/ 15:03:27 #chair amoralej 15:03:27 Current chairs: amoralej apevec coolsvap dmsimard imcsk8 jpena jruzicka jschlueter rbowen tosky weshay 15:04:15 let's start with the agenda topics 15:04:27 #topic DLRN Ocata branch 15:04:45 o/ 15:04:52 #chair trown 15:04:52 Current chairs: amoralej apevec coolsvap dmsimard imcsk8 jpena jruzicka jschlueter rbowen tosky trown weshay 15:04:52 rdo-trunk created openstack/openstackclient-distgit: python-openstackclient: failed to build ea7f28f http://review.rdoproject.org/r/3063 15:05:21 ihrachys, amoralej: http://logs.openstack.org/51/382351/2/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/94511dd/logs/neutron/server.txt.gz#_2016-10-05_14_27_58_887 15:05:28 I probably missed something else in config 15:05:46 so we now have a first ocata repo (which should become centos-master) -> https://trunk.rdoproject.org/centos7-ocata/consistent/ 15:05:51 EmilienM, you have to install the plugin in the server 15:06:04 jpena woohoo 15:06:06 mhh ok, let me see 15:06:30 rdo-trunk created openstack/ec2-api-distgit: openstack-ec2-api: failed to build a5e408a http://review.rdoproject.org/r/3064 15:06:34 EmilienM is testing it for the Puppet OpenStack Integration, and tripleo CI should be ok with it once we switch 15:06:46 jpena: did you try tripleo? 15:06:54 EmilienM: is package with the plugin installed? 15:06:54 EmilienM, not yet 15:07:01 EmilienM: python-neutron-lbaas I believe 15:07:01 ihrachys: I'm checking 15:07:05 o/ 15:07:10 #chair flepied 15:07:10 Current chairs: amoralej apevec coolsvap dmsimard flepied imcsk8 jpena jruzicka jschlueter rbowen tosky trown weshay 15:07:21 ihrachys: yes 15:07:23 http://logs.openstack.org/51/382351/2/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/94511dd/logs/rpm-qa.txt.gz 15:07:38 ihrachys: let's take it somewhere else, there is a meeting here now 15:07:43 the next step would be to have the ocata repo become trunk.rdo/centos7-master, and keep Newton only in trunk.rdo/centos-newton 15:07:43 EmilienM, jpena - can we not trigger tripleoci against centos7-ocata easily? 15:07:52 apevec: I guess so, yeah 15:08:13 EmilienM: that's ocata? 15:08:31 EmilienM: sorry, moving to another venue 15:09:04 would be nice to switch today, so we can focus on newton GA tomorrow 15:09:26 or Friday, right after GA 15:09:45 we release on a Friday? 15:09:54 jpena, on Friday we'll probably have 0day updates after tripleo rc3 ? 15:10:16 EmilienM, it's first release so nothing to break 15:10:23 updates are not released on Friday 15:10:33 apevec: ack. 15:10:33 but (next topic) we have outage on Monday 15:10:37 So, I have drafted that release announcement blog post, but I will be out Thursday, Friday, Monday. If/When the release goes out, can someone change the published flag to 'true' and push that out? 15:10:43 apevec: I'm happy with whatever day, just want to make sure I don't break anyting Newton-related by moving centos-master to Ocata 15:10:44 so need to release before 15:11:23 jpena, ack - sure, CI pass is a blocker 15:11:44 I'm specifically worried about hardcoded centos-master urls 15:11:54 jpena, good luck with that... just need to quickly fix it :-p 15:11:55 where we have those? 15:11:57 jpena: not just that 15:12:06 they'll be 404 after the switch 15:12:19 jpena: we probably need to go through jobs that are testing centos7-master but actually promoting in /home/centos-newton 15:12:24 stuff like that 15:12:26 trown: ^ 15:12:39 I believe tripleoci might do something like that as well 15:12:40 ok, let's add those as a checklist in the card 15:12:46 with their remote promotion 15:12:54 I am really confused by what the point of master dlrn worker is 15:13:16 I thought the whole point of naming it master and not newton was that it would always track master branch 15:13:33 we had to change stuff around when we introduced pinning to u-c 15:13:44 before that master == master 15:13:49 u-c? 15:13:53 upper constraints 15:14:00 version pins 15:14:06 trown: centos7-master is "current work-in-progress release, pinning certain libraries/clients to what's in upper-constraints.txt" 15:14:15 centos7-master-head is "real master" 15:14:42 jpena: granted, but what's confusing is the mapping on the dlrn instance (/home/) 15:14:43 and -head is not used for promotion, just as pre-warning system 15:15:05 dmsimard, well yes... but that's internal to us 15:15:06 :) 15:15:16 like, hey, I'm testing centos7-master but I'm actually promoting /home/centos-newton 15:15:25 ok... so for an upstream project using our packages to validate changes on master... what dlrn should they use 15:15:27 I get confused at times too :( 15:15:36 using newton does not make sense 15:15:39 dmsimard, we could just test centos7-ocata, they both point to the same /home 15:16:02 should we just abandone centos7-master symlink? 15:16:12 jpena: is it called /home/centos7-ocata ? or centos-ocata? 15:16:13 we can't really make it continuous 15:16:34 dmsimard it's /home/centos-ocata, but the url is centos7-ocata 15:16:43 ok, and that's ocata pinned to u-c 15:16:46 right ? 15:16:49 right 15:16:57 apevec: that is the thing... why cant we make it continuous? 15:16:58 ok so this is what we want to test 15:17:33 but I agree if it is not actually continuous, then "master" just adds to the confusion 15:17:37 I was under the assumption master remained master through releases 15:18:04 the issue with continuous master is infinitely growing database and repo storage. We can run some cleanups from time to time to remove old commits, but it's veeery slow 15:18:41 yes, we use it as cleanup point also we need to switch deps repo at one point to add new deps 15:18:55 which requires few steps see https://trello.com/c/sl4Xs6wy/353-recreate-centos-master-worker-on-newton-rc 15:19:22 ok, at some point we changed this assumption though... and that was not communicated anywhere 15:19:23 trown, apevec: do we have anyone looking a "testing tripleo with ocata repo"? 15:19:25 ideally we'd do it on RC and keep illusion of continuous master 15:19:46 so we have tripleo-ci pointed at master for testing master... and they are not testing master 15:19:47 EmilienM, ugh.. that should just be master 15:20:15 master is newton iiuc 15:20:19 (currently) 15:20:20 right 15:20:28 jpena, if it's just a clean up issue.. can't we focus on that .. and keep delorean master in step w/ git master 15:20:49 which is bonkers to me... we should definitely get rid of "master" name if it is not tracking master 15:21:15 I'm afraid we're not explaining it properly, just a sec 15:21:18 rdo-trunk created openstack/gnocchi-distgit: openstack-gnocchi: failed to build dbf7751 http://review.rdoproject.org/r/3065 15:22:00 master should track master, *but* we're pinning certain packages to their versions in upper-constraints.txt because that's what is being tested in the gate 15:22:47 so upper-constraints is now authoritative requirements source? 15:22:52 the fact that today master is still newton is a temporary issue. We know that if we just switch, we'll break CIs 15:23:08 jpena: temporary or not, it is not right 15:23:36 jpena: for instance https://trunk.rdoproject.org/centos7-master/77/2b/772ba65266708fb0945842f68a39bc2ebcc9ef67_783e10e8/instack-undercloud-5.0.0-0.20161003213232.9d2a655.el7.centos.noarch.rpm is a package that tripleo-ci will use to test changes to master 15:23:42 that is from a commit on stable/newton 15:24:14 hmm, it should be 5.2.0 if from stable/newton 15:24:19 so problem 1) confusing /home/centos7-master symlink - either make that follow master or remove 15:24:33 problem 2) borken CI on symlink update 15:24:44 if we are changing the commitment that master always tracks master... we need to communicate that loudly, as it is contrary to the assumption people make when we call something master 15:25:07 I'm pretty sure that was well communicated during last Summer 15:25:09 but anyway 15:25:28 apevec: I am more pointing out that is the latest commit available in centos-master repo, and it is HEAD of stable/newton (932a655) 15:25:37 https://github.com/openstack/instack-undercloud/commits/stable/newton 15:26:10 ie, all commits that have not been backported to stable/newton are not available in tripleo-ci 15:26:16 coin o/ 15:26:32 Duck: in english it's quack :p 15:26:42 ガーガー 15:26:45 trown, nm, I mixed up projects 15:26:59 #chair Duck 15:26:59 Current chairs: Duck amoralej apevec coolsvap dmsimard flepied imcsk8 jpena jruzicka jschlueter rbowen tosky trown weshay 15:27:20 sorry to be late, problem with my IRC seat server 15:28:17 apevec: so for instance, if tripleo merges an instack-undercloud patch that depends-on a tripleo-heat-templates patch it will pass CI, because both will be built in CI... but as soon as those are merged, CI will be broken 15:28:24 because the merged patch is not available 15:28:37 alright, so problem is we are late with setting up new dlrn instance 15:28:45 we need to have it ready by RC 15:28:57 we can improve that for Ocata 15:29:16 but original question remains when can we switch for Newton ? 15:29:51 my vote would be asap, since it is only a matter of time before ^ happens 15:29:54 EmilienM: ^ 15:30:37 do we want to switch w/o verification then fixup if anything breaks? 15:30:55 or we could run job against centos7-ocata ? 15:30:59 that would be my vote, but not sure how much is in flight for tripleo rc3 15:31:26 rdo-trunk created openstack/heat-distgit: openstack-heat: failed to build 68cb414 http://review.rdoproject.org/r/3066 15:32:01 trown: everything in flight needs to land tomorrow 15:32:30 ok, so we delay -master switch to -ocata to Friday? 15:32:50 or we switch tripleo/master to use -newton? 15:32:55 EmilienM: apevec, not sure... we could end up with broken CI quite easily if we **dont** switch 15:33:18 I can easily launch weirdo jobs against centos7-ocata and see what happens 15:34:00 EmilienM: we have to be really careful with depends-on patches if we decide to wait to switch 15:34:18 why don't we switch tripleo/master to use -ocata (like we're testing with puppet ci)? 15:34:20 I'm afraid of not switching and being broken 15:34:31 ya that is my concern as well 15:34:43 the current master jobs are downloading newton packages from stable/newton 15:34:50 it makes our backporting thing not easy 15:35:01 jpena: yes let's do it. 15:35:10 I am pretty sure even a tht depends-on puppet-tripleo will break us... which happens quite often 15:35:18 if it can stop the fact it download stable/newton dependencies 15:35:22 then yes 15:35:26 trown: you see what I mean? 15:35:47 currently, if you send a patch to THT, it will download puppet-tripleo from stable/newton builds 15:35:50 not from master 15:35:54 EmilienM: exactly 15:35:59 if we can fix that, let's go ahead ASAP 15:36:13 is the switch going to fix that? 15:36:32 EmilienM: yep... it might break something else though... 15:36:47 but, I think it is lower risk than doing nothing 15:36:49 trown: right. So before, let's run a CI job in ocata, using tripleo-ci 15:36:59 trown: let's do it now. If it pass, let's do the switch today. 15:37:10 also upgrades will not work until we merge all of https://review.openstack.org/#/q/topic:open-ocata 15:37:15 how is taking that action? 15:37:24 apevec: who? 15:37:26 since versions on master will be lower than versions on stable/newton 15:37:28 me if nobody 15:37:30 eer who 15:37:34 trown: right 15:37:39 I +2 it, can't do more 15:37:52 EmilienM: ya I will bug folks in #tripleo after the meeting 15:37:53 trown: please ask other reviewers to get it asap 15:38:11 trown: do you feel confortable to patch tripleo-ci with the ocata repo and see how it works? 15:38:25 ya I can do that 15:38:29 ok 15:38:36 jpena, ^ action recording time 15:38:43 #action trown submit test patch to tripleo-ci to test ocata repo 15:38:46 if it pass OVB/multinode jobs, ok let's do the switch 15:39:03 #action jpena to move centos-master to ocata worker as soon as tripleo-ci and puppet ci jobs are ok 15:39:21 David Moreau Simard created rdo-infra/ci-config: centos7-master now maps to centos-ocata, not centos-newton http://review.rdoproject.org/r/3067 15:39:26 rdo-trunk created openstack/ironicclient-distgit: python-ironicclient: failed to build 5b67658 http://review.rdoproject.org/r/3068 15:39:32 trown ^ 15:39:42 ok, next topic? 15:39:47 trown: also need to patch tripleoci to promote the right worker 15:39:52 oh fuck 15:39:57 oops sorry bad word 15:40:02 we also need to notify KB for CDN updates 15:40:16 jpena: /me working on puppet CI testing for ocata 15:40:18 I'll take that 15:40:24 dmsimard: yes, that's on the list. 15:40:31 jpena: where is that list ? 15:40:31 rdo-trunk created openstack/ironic-inspector-distgit: openstack-ironic-inspector: failed to build b8f49a0 http://review.rdoproject.org/r/3069 15:40:33 we shouldn't do that until we've switched 15:40:34 jpena: if lbaas reminds blocker we'll disable its testing and re-enable it afterwards 15:40:39 but ihrachys is highly helping here 15:40:43 dmsimard: https://trello.com/c/sl4Xs6wy/353-recreate-centos-master-worker-on-newton-rc 15:40:57 jpena: added myself ty 15:41:41 #topic Preparation for CentOS outage October 10th 15:42:04 this will be 48h outage 15:42:10 so we need GA released before that 15:42:17 latest Monday morning 15:42:37 is that monday morning UTC? 15:42:40 long story short: there's way more devil in the details for the DLRN+nodepool idea than we originally thought, so we need another way of building packages during the 48h outage if needed 15:42:42 UTC 15:42:58 so I'll be asleep, welp.. good luck :p 15:43:02 oh yes, DLRN is other part 15:43:21 repos will be available 15:43:46 jpena: we can fallback to trystack or rcip-dev for another dlrn instance 15:43:51 EmilienM: I think it's harsh for you folks to try to fix lbaas tests just because upstream just landed a patch with a plugin 15:43:53 but both have proven very unreliable. 15:44:04 EmilienM: I would think you would better have a guarding regex for tests you execute 15:44:09 ihrachys: not a problem. Feedback loop is short 15:44:15 EmilienM: and that you expand your gate scope on your time 15:44:21 dmsimard, jpena - yeah, I prefer not to build new packages during outage 15:44:27 ihrachys: we run what smoke gives us 15:44:34 apevec: ack 15:44:35 apevec, jpena: agreed -- it's a huge pain to roll things backwards after 15:44:36 ihrachys: we are tempest users, smoke is the right use case for us 15:44:42 EmilienM: well it's cool if everything works, but trying to fix it in fire drill instead of doing it more calmly is not ideal 15:44:46 when we migrated from rcip-dev to ci.c.o, we had a bunch of issues 15:44:47 just ensure we have latest consistent on Monday before outage 15:44:52 ihrachys: right 15:44:53 mismatching repo hashes, stuff like that 15:44:59 EmilienM: I would think it would be smoke with no plugins 15:45:03 apevec: what time does the outage start 15:45:05 ? 15:45:08 EmilienM: you never know what a random plugin can tag with smoke 15:45:19 Monday afternoon iirc 15:45:25 EmilienM: f.e. in neutron we had that tag applied to most test cases till Newton 15:45:30 ok then, I'll be on it 15:45:37 EmilienM: later we realized it's not right and cleaned it up 15:45:39 Migration is scheduled for """"Monday October 10th, 1:00 pm UTC time"""". 15:45:41 You can convert to local time with $(date -d '2016-10-10 13:00 UTC') 15:45:45 ihrachys: right, that's why i'm working on adding more CI jobs in core projects 15:45:49 ihrachys: eg: tripleo 15:46:00 ihrachys: devstack jobs can't be 100% trusted 15:46:07 anyhoo, it's up to you. I just think you would have a better experience with more guards against crazy (aka active) upstream community 15:46:15 #info no new packages built by DLRN during the CentOS outage 15:46:18 oh, side note 15:46:23 regarding the CentOS maintenance 15:46:32 EmilienM: I appreciate that. I don't know whether experimental jobs are that effective though. 15:46:32 we will lose the cloud slaves, even when the infrastructure is back online 15:46:39 EmilienM: but I guess that's all you have available 15:46:51 ihrachys: experimental is a first step. I expect them in check -nv soon. 15:46:55 I'll bring them back up ASAP but it might be a while, the openstack cloud will be re-installed, maybe on newton 15:47:01 ihrachys: it's hard to convaince people. So I'm doing baby steps. 15:47:43 dmsimard: so we'll only have the trystack slaves for review.rdo 15:48:02 jpena: no, by cloud slaves I meant for ci.c.o 15:48:12 jpena: we have one "static" slave and three "cloud" slaves right now 15:48:14 EmilienM: check -nv is cool 15:48:15 dmsimard: ah ok 15:48:31 ihrachys: I want to demonstrate that the job is useful before and i'll propose it 15:48:31 jpena: review.r.o, rcip-dev and trystack aren't affected by the maintenance 15:48:37 EmilienM: one question - does it make the overal job run longer? 15:48:55 next topic? 15:49:05 we're running out of time 15:49:14 #topic Preparation for RCIP-DEV move October 17th 15:49:14 ihrachys: shouldn't. The job takes ~ 50 min but use 2-nodes setup 15:49:29 dmsimard, is that yours? 15:49:44 EmilienM: so like another multinode 15:49:56 I didn't put that up 15:50:08 We can discuss it next week if anything 15:50:09 jpena: that's mine 15:50:21 yes we can discuss later 15:50:25 ok 15:50:26 * dmsimard has to go pick up kid from school for lunch 15:50:34 #topic GA rebuilds tomorrow 15:50:48 that's mine reminder 15:50:59 anyone knows of any blockers, please speak up! 15:51:25 apevec: puppet-heat is already known? 15:51:26 next is to review cloudsig-newton pipeline: 15:51:43 that is what is blocking tripleo jobs in cloudsig-newton pipeline 15:51:44 so that's the only failure? 15:51:57 9.4.1 was released upstream 15:52:09 well it blocks further testing... I have been trying (unsuccessfully) to hack around it locally 15:52:11 is the fix you pointed included? 15:52:51 yep 15:53:03 https://git.openstack.org/cgit/openstack/puppet-heat/commit/?h=9.4.1&id=7c2489b23cc6fbaaff8c486a7fd139afccd9d1b4 is in there 15:53:22 ok 15:53:32 we'll get that into -testing asap 15:54:12 puppet-cinder if that lands 15:54:15 #action apevec to make sure puppet 9.4.1 are in newton-testing 15:54:28 jschlueter, do you have o/releases proposed? 15:54:36 but it didn't land in time for 9.4.1 15:54:43 it's per project 15:54:44 apevec: so likely won't land for GA 15:54:57 you can still propose it 15:55:22 apevec: change was not yet in stable/newton last time I checked... 15:55:40 yeah need that first 15:55:42 Mike Burns created openstack/tripleo-heat-templates-compat-distgit: match tripelo-heat-templates packaging under /usr/share http://review.rdoproject.org/r/3070 15:56:01 mburned, ^ another blocker? 15:56:38 apevec: i think so, yes 15:56:51 please let's use topic:newton-branching for reviews that must get into GA builds 15:57:15 ok, that's all, we'll continue reviewing blockers on irc 15:57:19 next topic 15:57:52 #topic Newton GA test day Oct 13/14 - https://www.rdoproject.org/testday/newton/final/ 15:57:53 [sensu] NEW: master.monitoring.rdoproject.org - check-delorean-newton-head-consistent @ http://tinyurl.com/hlq9rv8 |#| Could not successfully retrieve repository info 15:57:55 Please look at test scenarios doc - https://etherpad.openstack.org/p/rdo-newton-ga-testday-testplan - and add tests (and instructions). 15:58:15 that seems very close to the outage 15:58:22 Is that a problem? 15:58:31 so we will have less than 24 hours to verify we have things working 15:58:47 On the other end, we'd be getting very close to summit. 15:58:58 What do you recommend? We can move it, but it's always a tight time. 15:59:00 ya outage is supremely bad timing :( 15:59:16 timing is never good 15:59:27 lets just keep it and hope for the best 15:59:34 yep 15:59:45 repos are not affected 15:59:45 ok. 15:59:49 just build/ci 15:59:56 #topic Announcements 16:00:18 Usual stuff here. I am still looking for folks to assist/demo at the booth in Barcelona. 16:00:18 just paste it all 16:00:19 rdo-trunk created openstack/ironic-inspector-client-distgit: python-ironic-inspector-client: failed to build 5ad4331 http://review.rdoproject.org/r/3071 16:00:25 for minutes 16:00:30 RDO/Ceph meetup at OpenStack Summit, Barcelona - https://www.rdoproject.org/blog/2016/10/ceph-rdo-meetup-in-barcelona-at-openstack-summit/ 16:00:30 Haven't registered for OpenStack Summit yet? Use RED_OPENSTACKSUMMIT for 20% off 16:00:30 If you have any "free" time at OpenStack Summit, consider helping out at the RDO booth - https://etherpad.openstack.org/p/rdo-barcelona-summit-booth 16:00:30 If you are considering going to OpenStack Days Natal, 19th November 2016 in the Praiamar Hotel & Convention Center, Natal, Brazil, please talk to RBowen 16:00:47 16:00:51 and finally 16:00:53 gotta go, see you tomorrow 16:00:55 #topic chair for next meeting 16:01:06 rbowen: count me in 16:01:13 Excellent. Thank you. 16:01:30 any volunteers? :) 16:01:42 o/ 16:01:54 #action jruzicka to chair next meeting 16:01:58 thanks a lot jruzicka ;) 16:02:06 ok, we're out of time, let's close the meeting 16:02:07 rdo-trunk created openstack/muranoclient-distgit: python-muranoclient: failed to build df35a2e http://review.rdoproject.org/r/3072 16:02:13 #endmeeting