00:03:00 #startmeeting Ansible Azure Working Group 00:03:00 Meeting started Thu Apr 12 00:03:00 2018 UTC. The chair is Kylie_. Information about MeetBot at http://wiki.debian.org/MeetBot. 00:03:00 Useful Commands: #action #agreed #halp #info #idea #link #topic. 00:03:00 The meeting name has been set to 'ansible_azure_working_group' 00:03:37 #topic Issue/PR raised by top customer 00:04:04 yeah, i think we shoudl start with Catherine's pr 00:04:13 https://github.com/ansible/ansible/pull/37909 00:04:48 I know Matt D was looking at that 00:04:57 Hi Matt and Jordan, this one fix is for top customer and validated by the customer. 00:05:09 It's a pretty big change and requires some close review 00:05:21 authentication is always a big thing and we want to get it right 00:05:39 In addition, let me ask one process question. To approve one bug fix, two "ship it" is needed. But if it is you or Matt D (core committer), one approval is enough. Right? 00:06:43 Depends on the scope, usually if it is small either one of us will just merge it in 00:07:23 the `ship it` side is affected by the files touched and how ansibot classifies the components 00:10:12 Understand the quality control for big change, especially authentication. It was confirmed by the submitter 8 days ago. It is better Matt could take time review it. Since Matt is on event this week. @zikalino, you guys will meet Matt next Monday. Right? 00:10:32 I believe that was the time frame 00:10:34 yes, so we can talk about this 00:10:46 There's not major rush right now, 2.6 is still some time away 00:10:50 s/not/no/ 00:11:15 Please, Zim. 00:11:24 You want that PR in 2.5, correct? 00:11:31 I don't think we can 00:11:32 Yes. 00:11:35 it's a new feature not bug fix 00:11:43 Yeah, that's what I was going to point out. 00:12:00 Have you discussed that PR with Matt Davis at all yet (with regards to inclusion in 2.5)? 00:12:52 I'm pretty sure it won't fit the criteria, new args and functionality shouldn't be added in dot releases 00:13:00 Not yet for this one. 00:13:25 Opps. Customer wants to take an official release with those fixes but 2.6 is too far. 00:13:58 mattclay: what is the timeframe for 2.6? 00:13:58 How about https://github.com/ansible/ansible/pull/36826? Can this be in 2.5.x? 00:14:16 that's fine 00:14:23 do you know how to raise a backport PR? 00:14:51 Here is a guide https://docs.ansible.com/ansible/latest/community/development_process.html#backport-pull-request-process 00:15:27 @zikalino, could you please follow the guidance to ask backport today? 00:15:28 oh, that's very useful doc, i will create a pr 00:15:38 I believe 2.6 is expected sometime in July, although I'd need to confirm that. 00:15:43 yes 00:15:49 The only thing that is missing is to create a changelog fragment https://github.com/ansible/ansible/tree/stable-2.5/changelogs/fragments 00:16:37 trying to find the bulk text as the docs still need to be created 00:17:05 #action @zikalino, create backport PR for #36826 00:17:13 Here it is https://www.irccloud.com/pastebin/kDA2DbRj/ 00:18:03 #action @nitzmahone, review https://github.com/ansible/ansible/pull/37909 00:18:11 Thank you Jordan. 00:18:20 you're welcome 00:18:56 we have two prs from external contributors to (2 facts modules) 00:18:57 These two are top ones. We still have a bunch of PRs need your review, some are already confirmed by community contributors. 00:19:12 #topic PRs need core committer's review 00:19:40 https://github.com/ansible/ansible/pull/38279 00:20:15 Send through the links and I'll add them to my review list 00:20:23 ah wait you sent an email recently 00:20:26 I'll go through that 00:20:41 https://github.com/ansible/ansible/pull/37333 00:21:03 For 38279 I don't see any integration tests. 00:22:52 yes indeed, i will ask the author to add 00:25:57 Thank you Matt. And thank you Zim, please add integration test into your checklist for other PR review:). These two are good examples contributed by community contributor. We should respond issues and PRs raised by customers and community contributors timely. And then they will continue to contribute to Ansible project. 00:26:22 And this is list in my email. 00:26:29 https://github.com/ansible/ansible/pull/38127 00:26:29 https://github.com/ansible/ansible/pull/38108 00:26:29 https://github.com/ansible/ansible/pull/38077 00:26:29 https://github.com/ansible/ansible/pull/38022 00:26:29 https://github.com/ansible/ansible/pull/36109 00:26:29 https://github.com/ansible/ansible/pull/35156 00:26:30 https://github.com/ansible/ansible/pull/33063 00:26:31 https://github.com/ansible/ansible/pull/32984 00:26:31 https://github.com/ansible/ansible/pull/19513 00:26:32 https://github.com/ansible/ansible/pull/37769 00:26:32 https://github.com/ansible/ansible/pull/37627 00:26:32 https://github.com/ansible/ansible/pull/35888 00:26:40 @yuwzho, any missing or new one? 00:27:11 let me check 00:27:51 Thank you Yuwei. 00:28:21 yes, we try to keep external developers engaged 00:28:21 Zim, I remember you mentioned some test cases were disabled and you would like to discuss how to open them. Still a valid topic? 00:29:45 All the prs listed 00:30:08 well, still working on it, but slowed down due to other things we had to do, but other than that i think that goes pretty well. mattclay is very responsive here, and test prs are merged when ready. 00:30:56 :+1: Nice. Thank you Zim and Matt Clay. 00:31:03 Ok. 00:31:24 i don't want to enable tests too early while i am not sure if they are stable enough. 00:31:51 https://www.irccloud.com/pastebin/HpPYGc61/ 00:32:14 I've added myself to those PRs for review, will hopefully knock a few off in the coming days 00:32:29 * Kylie_ #action @jborean93 and @nitzmahone review below PRs 00:32:47 We're tracking unstable Azure tests here: https://github.com/ansible/ansible/projects/22 00:33:01 For those still here, we're finalising the dates for 2.6. Should have an update by Friday morning PDT. 00:33:04 Some of those have been disabled to avoid CI failures as they were failing quite frequently. 00:33:26 Y, please. Thank you Jordan 00:33:40 oh, didn't know about this place 00:33:46 I've been working on a better way to handle unstable tests so we don't have to completely disable them. I'm hoping to have a solution this week. 00:34:20 The issues on that project board will automatically move to `Done` once they've been closed. 00:34:26 Thank you Dylan. As for branch strategy, will it happen on 2.6 or 2.7? 00:35:56 zikalino: I've started tracking all unstable integration tests using issues and projects. All issues with Azure integration tests will show up on that project board. If the tests are frequently failing they'll be disabled, with that noted on their respective issue. If the failure is infrequent the test may remain enabled while awaiting a fix. 00:37:03 Once we have better handling of unstable tests in place, we should be able to disable them for unrelated CI runs while still running them for PRs that affect the module or the test itself, so we can still prevent regressions. 00:37:45 Right now if a test is unstable our only option is to disable it, which can lead to regressions when changes are made and the contributor doesn't realize the tests were not run. 00:37:48 ok, that sounds good. we are also performing full tests on our side. unfortunately some tests are unstable quite randomly.... 00:37:53 but getting there. 00:37:53 Zim, I think we are refining test cases in our playbook role. Once stable, you will merge to upstream. Right? 00:38:40 yes, however sometimes tests behave differently in our framework... 00:38:40 The most common causes of unstable tests I've seen are: 1) missing/inadequate retries within module logic 2) tests assuming they have exclusive use of a resource group 00:39:10 Tests have exclusive use of a resource group while they're running, but another test before or after that test can use the same resource group. 00:40:01 yes, there was one dependency where one test was deleting resource created by previous test, sth like that. 00:40:37 so (2) doesn't happen in our test framework because every test runs in a new resource group... 00:41:10 In most cases (2) should be fairly easy to fix, by having tests only manipulate resources created during the test run. 00:41:37 also sometimes deleting resorce, and then creating again resource with the same name in the same resource group may cause issues. 00:44:51 BTW, as for test environment will you test running Ansible on different OS platforms? 00:44:52 Using the resource prefix (or a derivative of it) should solve that issue. 00:44:53 We run most of our non-cloud related integration tests on multiple platforms. 00:44:53 The expectation for cloud modules is that they're not OS-specific and would not benefit from testing on various platforms. But we do test them on both Python 2.7 and 3.6 to ensure compatibility there. 00:44:53 btw, i have a question related to aliases and groups: 00:44:53 posix/ci/cloud/group2/azure 00:44:54 what is the rule to allocate test to particular group? 00:44:54 Good to know. 00:45:10 The groups are used to split up tests so no single group when run will exceed our CI timeout (under 1 hour). 00:45:23 Additionally, we try to keep cloud tests for the same cloud platform in the same group(s). 00:45:38 Periodically I rebalance the groups to keep the CI run times fairly even across groups. 00:45:53 So the general rule is to use the same group as existing tests for the same cloud platform. 00:46:38 ah, so i should use group with lowest execution time for new tests. 00:47:06 zikalino: That would be nice, but unless you look at a nightly run that executes all tests, you won't know which one to choose. 00:47:36 When you submit a PR it will only run tests relevant for that PR, which for cloud modules will not be all tests. 00:48:01 So it's easiest to pick an existing group already used by Azure tests and leave the balancing of groups to me. 00:48:05 i noticed when i moify for instance azure_rm_common then it runs all tests, right? 00:48:27 If you modify module_utils code then it will run tests for all modules depending on that code. 00:48:38 If you modify a module or module's test, only the test for the module runs. 00:49:15 also woudn't it be possible to validate aliases somehow? once we had an issue that test was not running because alias was completely wrong, and we just haven't noticed that. 00:49:44 Yes, that is part of the project I'm working on to handle unstable tests. I hope to have it done this week still, assuming I don't have too many other things to deal. 00:49:45 (i guess alias was just old because pr was a few months old, and i think aliases have changed in a meanwghile) 00:49:55 s/deal/deal with/. 00:50:39 We will soon have sanity tests to make sure that aliases are correct. I'll also be adding notices to PRs when modules do not have integration tests, or the tests are disabled or unstable. 00:50:45 That part won't be done this week though. 00:51:35 ok, that sounds very good :-) 00:51:43 There will also be some new documentation about using aliases, once the sanity tests are in place. 00:53:21 zikalino: Are you continuing to work on improving the stability of the unstable tests I've reported? 00:54:06 yes, i was just disrupted by some other tasks, but i think we can enable a few tests even this week. 00:54:26 i will bother you tomorrow i think 00:54:59 Good discussion here:) 00:55:15 Since we are overrun, I will ask any other topic? 00:55:18 Oh, before I forget. I don't know if Matt Davis mentioned it, but I'll be coming with him to meet with your team on Monday. 00:55:50 Nice, nice though I am not there. You guys could have very efficient F2F. 00:56:00 Matt, do you also locate at Oregon? 00:56:10 Kylie_: Yes 00:57:30 Great. 00:59:31 I did not see meeting room info. I will ask Catherine update the room info and leave contact info in case you guys cannot find the right building. 00:59:44 It would be a wonderful meeting. 00:59:56 Ok. Then if no other topic, I will end meeting. 01:00:16 #endmeeting Ansible Azure Working Group