15:01:10 #startmeeting RDO meeting - 2017-10-25 15:01:10 Meeting started Wed Oct 25 15:01:10 2017 UTC. The chair is amoralej. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:10 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:01:10 The meeting name has been set to 'rdo_meeting_-_2017-10-25' 15:01:14 Meeting started Wed Oct 25 15:01:10 2017 UTC and is due to finish in 60 minutes. The chair is amoralej. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:15 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:01:18 The meeting name has been set to 'rdo_meeting___2017_10_25' 15:01:28 #topic roll call 15:01:31 * Duck o/ 15:01:34 \o/ 15:01:35 o/ 15:01:49 o/ 15:02:08 o/ 15:02:13 #chair chandankumar ykarel jjoyce Duck jpena number80 15:02:13 Current chairs: Duck amoralej chandankumar jjoyce jpena number80 ykarel 15:02:14 Current chairs: Duck amoralej chandankumar jjoyce jpena number80 ykarel 15:02:26 o/ 15:03:23 \o/ 15:03:48 #chair PagliaccisCloud 15:03:48 Current chairs: Duck PagliaccisCloud amoralej chandankumar jjoyce jpena number80 ykarel 15:03:49 Current chairs: Duck PagliaccisCloud amoralej chandankumar jjoyce jpena number80 ykarel 15:03:57 o/ 15:04:06 #chair jrist 15:04:06 Current chairs: Duck PagliaccisCloud amoralej chandankumar jjoyce jpena jrist number80 ykarel 15:04:07 Current chairs: Duck PagliaccisCloud amoralej chandankumar jjoyce jpena jrist number80 ykarel 15:04:08 Merged rdo-infra/ci-config master: Remove "static" option from include_role https://review.rdoproject.org/r/10280 15:04:19 let's start with chandankumar topic 15:04:33 #topic Make RDO Office Hour biweekly and duration to one hour 15:04:39 chandankumar, ^ 15:04:57 So currently we are running RDO office hour each tuesday for 2hours. 15:05:28 We found that most of the people have conflicting meeting and z hours appears to be too much more time. 15:05:46 So can we bring down to one hour and make it biweekly? 15:05:50 yeah, biweekly makese sense 15:06:17 +2 15:06:19 +1 15:06:31 +1 15:06:35 +1 15:06:40 +1 15:06:47 +1 15:06:52 o/+1 15:06:55 locked down, merged! +W 15:06:58 ok, it seems there is consensous 15:07:21 next is do we need to change the timing? 15:07:24 chandankumar, will you send a mail to comunicate it? 15:07:43 amoralej: i will send an email about the changes. 15:07:45 chandankumar: let's keep it on the same time for now 15:08:10 number80: sure. 15:08:38 #agreed to do RDO Office Hour beweekly and duration to one hour 15:08:41 so starting from next week, we will be keeping it biweekly. 15:08:58 #action chandankumar will send a mail to communicate new schedule 15:09:10 that's it from my side. 15:09:20 thanks chandankumar 15:09:36 #topic infra: any problem to report after the ML migration? 15:09:59 Duck, is this yours? 15:10:09 quack 15:10:15 yes 15:10:24 that's not me to talk :-) 15:10:41 from my side, we changed mail to notify failures in ci to infra@ 15:10:45 and it's working fine 15:11:19 I'll be away from 31-08 for RubyWorld Conference and some PTO, so it's the right time to fix things if any problem 15:11:26 nothing special 15:11:44 ok, then that's all for me :-) 15:12:17 Duck, thanks for the lists work! 15:12:20 \o sorry didn't realize meeting had started 15:12:30 now we are a proper project :) 15:12:36 Duck: yeah thanks, it looks like there was no issues in the migration at all 15:12:37 apevec: that's just step 1 :-) 15:12:38 #chair apevec dmsimard jjoyce 15:12:38 Current chairs: Duck PagliaccisCloud amoralej apevec chandankumar dmsimard jjoyce jpena jrist number80 ykarel 15:12:38 with its own lists.* 15:12:39 Current chairs: Duck PagliaccisCloud amoralej apevec chandankumar dmsimard jjoyce jpena jrist number80 ykarel 15:12:54 everything looks fine 15:13:09 Duck, where is the machine running btw? 15:13:19 apevec: RDO Cloud 15:13:20 * apevec hopes rdocloud 15:13:24 excellent 15:13:25 the doc has been updated 15:13:45 the doc you started on service continuity, and some other things in the site 15:14:27 so the only thing to change is to finish the ansible base roles topic, and move the rules in your repo 15:14:45 backup is done, but monitoring is waiting for this fix 15:14:46 Duck, iirc, there was some plan to deploy some fancy webui, right? 15:15:09 yes, that's step 2 but blocked because of a bug 15:15:18 ok, cool 15:15:20 so this will be Mailman _3_ 15:15:26 and later Ponymail is possible 15:15:29 o/ 15:15:30 no hyperkitty ? 15:15:30 if 15:15:39 dmsimard: yes, in step 2 15:15:48 is there Unicornmail too? :P 15:16:02 OMG, please choose one guys 15:16:11 :D 15:16:21 hyperkitty comes with MM3 and is already handled by our roles and scripts 15:16:27 but the rest needs work 15:16:51 Hyperkitty 15:17:26 alright let's move on? 15:17:29 i have no preference, so the simpler is ok 15:17:31 yeah 15:17:43 next 15:17:48 number80: rbowen tested Ponymail est found it better 15:17:52 and 15:18:08 rbowen is biased, Ponymail is an apache project :P 15:18:28 anyway, let's defer that discussion to some other time :) 15:18:34 #info no issues have been reported after mailing lists migration 15:18:39 let's move on to the next topic 15:18:42 #topic given master CI status, delay Queens milestone 1 to the next week? https://www.rdoproject.org/testday/queens/milestone1/ 15:19:07 so this should be a nobrainer given status: 15:19:18 https://dashboards.rdoproject.org/rdo-dev 15:19:28 master didn't pass for > 40days 15:19:39 there is not other option than to delay to next week 15:19:46 yeah 15:19:51 yep, let's delay 15:19:56 yep 15:20:02 migration to cinder v3 seems fixed but there are some open issues yet 15:20:05 ok, I'll send update to the website page 15:20:07 has it really not passed for 40 days ? 15:20:11 yeah 15:20:11 yes 15:20:14 sadly 15:20:28 usually when it doesn't pass for >7 days the world is on fire 15:20:38 why are we not making a bigger deal out of this ? 15:20:41 so it was world on fire for 40days 15:20:42 we are 15:20:46 dmsimard, many people is still working in pike 15:20:50 stabilization 15:20:50 you are just not on right calls :) 15:21:01 amoralej: yeah I've seen a lot of work on pike.. 15:21:03 we live in our happy world dmsimard 15:21:08 :) 15:21:21 I might be a bit more disconnected from CI and promotion recently :/ 15:21:28 btw, we have now ansible 2.4 in centos extras repos 15:21:50 heh 15:21:52 in case you start seeing weird breakages 15:21:54 yeah it appeared sometime recently, 2.3.2 is still available 15:22:03 but there should be workarounds available in ooo I think? 15:22:20 it actually broke something in software factory because of a "mutual" bug between ara and ansible in 2.4.0 15:22:44 2.4.1 has been delayed for like 2 weeks now.. it should hopefully be tagged today so I can tag a new release for ara that works for 2.4.1 15:22:53 so actions 15:23:09 for ansible 2.4 there is a fix: https://review.openstack.org/#/c/513701/ 15:23:23 #action apevec to update testdays page and move queens1 testday to Nov2/3 15:23:27 containers build should work also after https://review.rdoproject.org/r/#/c/10280/ 15:24:13 we should probably consider pinning the package versions we get from extras like we did on pip .. 15:24:20 ansible upgrades are dangerous :( 15:24:41 there was some discussion about that in the past 15:25:13 but i think we need to keep compatible to whatever is in centos extras 15:25:29 yeah we can't really pin extras 15:25:42 instead we need to plug into ansible ci pre-release 15:26:08 there is work on that Cc flepied :) 15:26:52 ok, I think we're done with the topic, we can discuss general CI later 15:26:54 ok, let's hope we can get a promotion before Nov the 1nd 15:27:04 2nd 15:27:09 ack 15:27:15 if we don't I'll be sad 15:27:25 let's move on 15:27:32 #topic longer EOL goodbye for Newton - keep trunk running for some projects longer (apevec) 15:27:44 #info http://lists.openstack.org/pipermail/openstack-dev/2017-October/123624.html 15:27:54 * eggmaster perks up 15:27:59 o/ 15:28:04 so deployment projects want to stay around longer 15:28:08 #chair eggmaster 15:28:08 Current chairs: Duck PagliaccisCloud amoralej apevec chandankumar dmsimard eggmaster jjoyce jpena jrist number80 ykarel 15:28:09 Current chairs: Duck PagliaccisCloud amoralej apevec chandankumar dmsimard eggmaster jjoyce jpena jrist number80 ykarel 15:28:10 question is how can we support them 15:28:21 post newton EOL is pushed upstream 15:28:28 which is happening ~now 15:28:38 There's one thing for sure -- it's that we can't build openstack projects in general past EOL since they delete the branch and tag eol. 15:28:39 * jrist waves newton a farewell 15:28:42 we could pin EOled projects 15:28:56 So if the deployment projects happens to need a patch that isn't in newton-eol for, say, nova, there's nothing we can do about it 15:28:59 and build only deployment projects 15:29:05 we could change the branch in DLRN to newton-eol 15:29:08 dmsimard, and we won't 15:29:11 I mean, in rdoinfo 15:29:11 that would be a limitation 15:29:28 jpena, or last tag which should be == 15:29:30 as we disable fallback to master, they will fail 15:29:39 It would be nice to see a promotion with all of the eol tags if possible. 15:29:41 dmsimard: I get EmilienM proposal to allow people pushing fixes in installers, not in projects themselves 15:29:48 apevec: right, but this also means we need to keep the trunk, testing and stable repositories around -- which are no longer supported (especially from a security perspective) 15:29:59 dmsimard, only trunk 15:30:24 jpena: 15:30:25 +1 15:30:31 apevec: do you consider -deps (-testing) as trunk ? 15:30:43 apevec: who's on the hook for maintaining that ? 15:31:09 same suspects 15:31:14 the main problem i see is to ensure things keep working with RHEL updates 15:31:43 the main problem I see is a resource one (both human and physical) 15:31:44 that's a good point and we could say this is only for the current know working rhel minor release 15:31:54 i.e. 7.4 15:31:55 if we want to keep testing that over time, it may happen that patches in EOLed repos 15:32:13 amoralej, we are not going to that 15:32:17 apevec, the problem is how to enforce using 7.4 in CI 15:32:24 proposal is to keep non-EOLed projects building in trunkj 15:32:41 after 7.5 is published 15:32:52 amoralej, then we EOL this 15:33:06 7.5 is in like what, 6 months maybe ? 15:33:16 so it's 6 months proposal then 15:33:23 dmsimard, there isn't fixed schedule 15:33:31 so, it's like a best effort 15:33:32 it follows rhel and that's not public sched 15:33:33 apevec: yeah it's a hand-wavy approximation 15:33:36 dmsimard: no assumptions, can be earlier or later 15:33:43 re. resources, how much we need for Newton? 15:33:50 we could trim trunk 15:33:58 so this means we have manage an additional openstack release per cycle 15:34:35 because if we do this for pike, we'll be doing it for ocata and so on 15:34:41 errrrr s/pike/newton/ 15:34:54 what does that actually mean? 15:35:03 e.g. https://review.rdoproject.org/r/#/q/topic:rdo-FTBFS-newton is not that frequent 15:35:12 (as it should be in a stable release) 15:35:31 in terms of storage it should not mean much, with purging it will get down to a minimal footprint pretty soon 15:35:36 it means 1 more dlrn process and disk storage 15:35:39 maintaining DLRN builder and fix FTBFS shouldn't be really an issue, i think 15:35:48 apevec: it means instead of the ~1 month overlap during which we support 3 releases (N, N-1, N-2), we extend that a few months 15:36:05 not too different from what we had to do during the ocata short cycle 15:36:08 dmsimard, ok, let's quantify the work 15:36:29 main problem is the time to debug CI and keep promotions passing when we start having issues 15:36:32 I'd say let's try this post EOL trunk with Newton 15:36:33 EmilienM: are you there ? 15:36:38 and note that for newton there is no upstream promotion 15:36:42 then we see what comes out of it 15:36:44 only RDO CI promotion pipeline 15:36:47 I'm very busy now 15:36:56 dealing with tripleo gate & stuff 15:36:57 whats up 15:37:12 oh newton EOL? 15:37:15 EmilienM: we're talking about extending EOL for deployment projects 15:37:16 EmilienM, we're figuring out how to give you more work :) 15:37:25 bah, do what you like 15:37:27 b/c you asked for it! 15:37:30 I said it sucks if we EOL 15:37:40 I guess I already said my opinions and the why etc 15:37:45 do I need to repeat? 15:38:00 we can experiment with post EOL rdo trunk 15:38:14 EmilienM: We're interested in what in means in terms of resources to keep it going -- what jobs we need to keep, what packages still need to be maintained, etc. 15:38:59 i guess we just want to run tripleo jobs 15:39:07 because we still have backports in tripleo / newton 15:39:15 we want to make sure they actually work (HEH) 15:39:55 weshay|PTO, ^ what ooo newton jobs can we run in review.rdo? 15:39:57 ah PTO 15:40:01 so, we'd maintain only RDO Trunk repos pinned for EOLed projects and following stable/newton for the non EOLed 15:40:08 adarazs, trown ^ 15:40:21 amoralej, yes 15:40:35 EmilienM: so we need a promotion pipeline ? periodic jobs ? etc 15:40:37 questions are about: deps repo and storage 15:40:58 newton uses... hammer? which has been EOL for several months already 15:41:05 sorry, on a meeting at the moment. 15:41:06 we have however not retired the repositories (yet) 15:41:21 in ci.centos we could keep a reduced pipeline with only tripleo jobs 15:41:24 dmsimard, I think periodic jobs were no't there for newton? 15:41:33 amoralej, +1 15:41:38 dmsimard: It would be helpful to keep the pipeline up until we are through all of the eol tags and see promotions with them. 15:41:50 apevec: there was a promotion pipeline for newton, wether it was in tripleo-ci, ci.centos or review.rdo 15:42:09 jjoyce: yeah, that's a given, regardless if we keep newton around for longer or not 15:42:12 dmsimard: no promotion 15:42:15 for newton we just have a single tripleo periodic job 15:42:16 just gate in tripleo-ci 15:42:25 on patches sent to tripleo / newton 15:42:27 there wasn't https://trunk.rdoproject.org/centos7-newton/current-tripleo/ 15:42:32 it's 404 15:42:47 in newton is old model 15:42:52 only RDO-CI promotion 15:42:57 not upstream promotion 15:43:03 not tripleo-ci promotion 15:43:05 dmsimard: So we would see dlrn hashes that have been promoted for newton correct? 15:43:19 i'd keep some periodic jobs in ci.centos 15:43:55 or it'll be a mess when jobs fail in upstream gates 15:43:56 amoralej, which jobs are periodic in ci.c.o ? 15:44:11 or you mean promotion running periodically? 15:44:15 https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo-delorean-promote-newton/ 15:44:21 yes 15:44:22 ok, so the latter 15:44:27 yes 15:44:46 we could get rid of weirdo if it's a problem 15:45:09 and remove all jobs for cloudsig repos 15:45:11 alright, so can we agree to try this for Newton then review what we do for Ocata? 15:45:22 it would be, because p-o-i will be EOL'd and I suppose we won't make packstack follow an extended cycle 15:45:42 yeah, no patches for packstack 15:45:48 it's rock stable ;) 15:45:55 well, hang on, EmilienM -- would your thing also run for puppet ? 15:45:56 bug-free software 15:45:59 EmilienM: as in, extended EOL for puppet modules 15:46:07 if mnaser is ok yes 15:46:13 if too late no 15:46:44 yeah it's a bit last minute considering the draft was not broadly discussed upstream 15:46:52 you have to understand we have customers (HEH!) running OSP10 and the more backports we do downstream-only, less testing we have when shipping these backports. I see value in keeping Newton a little bit more 15:47:48 EmilienM: we can bring the production chain back 2 years and do backports manually even for supported releases if you want :P 15:48:59 we can keep p-o-i jobs while stable/newton branch exist 15:49:05 so for actions 15:49:28 looks like I dropped 15:49:39 i think newton-EOL tags are still not in place, right? 15:49:55 jpena, did you already deploy dlrn with the new option to prevent fallback to master? 15:49:59 they are coming 15:50:00 amoralej: They are starting to show up now. 15:50:04 guys do what you like 15:50:07 apevec: not yet 15:50:11 I just gave some feedback on why I think we should keep newton 15:50:12 EmilienM, we want to make you happy! 15:50:17 no you don't ! 15:50:22 that'd be important to avoid getting builds from master in newton 15:50:24 btw, I'll never be happy. 15:50:26 I mean amoralej 15:50:28 yep 15:51:10 ok, so we can refined this in newton eol card, which I need to create from a template https://trello.com/c/fV69VODx/165-rdo-release-eol 15:51:27 #action apevec to create Newton EOL card in rdo trello 15:51:27 ok 15:51:41 then we discuss details there 15:51:49 ok 15:51:50 let's move on 15:52:13 there are no more topics in the etherpad 15:52:24 #topic who will chair next week? 15:52:28 any volunteer? 15:54:06 ok, i will do it again 15:54:43 Is Nov 1 a non-working day everywhere? 15:54:53 it is in Spain, so amoralej and I will be off 15:55:05 i was checking, thanks jpena 15:55:32 oh right, I'm off too 15:55:58 dmsimard, number80 chandankumar ykarel do you want to keep the meeting? 15:56:19 amoralej, no item from my side 15:56:31 or is it non-working day there also on Nov-1st? 15:56:31 for those outside the core-catholic states :) 15:56:37 :) 15:56:42 and it's working here that day 15:57:13 ykarel, can you run the meeting, we might still need a sync to confirm testday? 15:57:13 ykarel, that sounded to me as presenting as volunteer to chair? :) 15:57:21 amoralej, sure 15:57:42 #action ykarel will chair the meeting on next week 15:57:50 #topic open floor 15:57:56 we still have a couple of minutes 15:58:14 so quick on general CI, 15:58:15 some topic you'd like to bring? 15:58:27 looks like people are not aware to look where are the current issues 15:58:42 and which series of issues we were hitting 15:59:22 amoralej, and now I'm not sure where we have it publicly presented... 15:59:36 https://trello.com/b/WXJTwsuU/tripleo-and-rdo-ci-status is not clear since it's just a scratchpad 15:59:49 and it even links to the internal docs ;( 16:00:04 we need to fix that so folks like jpena and dmsimard know what is going on ... 16:00:05 yeah, i'm trying with a search in LP 16:00:08 gimme a while 16:00:15 there is https://bugs.launchpad.net/tripleo/+bugs?field.tag=alert 16:00:28 yeah, that's what i was looking for 16:00:55 although that's not complete, to be honest 16:01:18 yeah, ok, I'll discuss that internall and see what we can do to increase visibilty 16:01:24 that's it from me 16:01:43 ok, thanks 16:01:48 we are over time 16:01:48 amoralej: network got slower at the office 16:01:54 so I'm good 16:02:11 i'm closing the meeting unless there is something else 16:02:31 #endmeeting