00:03:00 <Kylie_> #startmeeting Ansible Azure Working Group
00:03:00 <zodbot> Meeting started Thu Apr 12 00:03:00 2018 UTC.  The chair is Kylie_. Information about MeetBot at http://wiki.debian.org/MeetBot.
00:03:00 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
00:03:00 <zodbot> The meeting name has been set to 'ansible_azure_working_group'
00:03:37 <Kylie_> #topic Issue/PR raised by top customer
00:04:04 <zikalino> yeah, i think we shoudl start with Catherine's pr
00:04:13 <Kylie_> https://github.com/ansible/ansible/pull/37909
00:04:48 <jborean93> I know Matt D was looking at that
00:04:57 <Kylie_> Hi Matt and Jordan, this one fix is for top customer and validated by the customer.
00:05:09 <jborean93> It's a pretty big change and requires some close review
00:05:21 <jborean93> authentication is always a big thing and we want to get it right
00:05:39 <Kylie_> In addition, let me ask one process question. To approve one bug fix, two "ship it" is needed. But if it is you or Matt D (core committer), one approval is enough. Right?
00:06:43 <jborean93> Depends on the scope, usually if it is small either one of us will just merge it in
00:07:23 <jborean93> the `ship it` side is affected by the files touched and how ansibot classifies the components
00:10:12 <Kylie_> Understand the quality control for big change, especially authentication. It was confirmed by the submitter 8 days ago. It is better Matt could take time review it. Since Matt is on event this week. @zikalino, you guys will meet Matt next Monday. Right?
00:10:32 <jborean93> I believe that was the time frame
00:10:34 <zikalino> yes, so we can talk about this
00:10:46 <jborean93> There's not major rush right now, 2.6 is still some time away
00:10:50 <jborean93> s/not/no/
00:11:15 <Kylie_> Please, Zim.
00:11:24 <mattclay> You want that PR in 2.5, correct?
00:11:31 <jborean93> I don't think we can
00:11:32 <Kylie_> Yes.
00:11:35 <jborean93> it's a new feature not bug fix
00:11:43 <mattclay> Yeah, that's what I was going to point out.
00:12:00 <mattclay> Have you discussed that PR with Matt Davis at all yet (with regards to inclusion in 2.5)?
00:12:52 <jborean93> I'm pretty sure it won't fit the criteria, new args and functionality shouldn't be added in dot releases
00:13:00 <Kylie_> Not yet for this one.
00:13:25 <Kylie_> Opps. Customer wants to take an official release with those fixes but 2.6 is too far.
00:13:58 <jborean93> mattclay: what is the timeframe for 2.6?
00:13:58 <Kylie_> How about https://github.com/ansible/ansible/pull/36826? Can this be in 2.5.x?
00:14:16 <jborean93> that's fine
00:14:23 <jborean93> do you know how to raise a backport PR?
00:14:51 <jborean93> Here is a guide https://docs.ansible.com/ansible/latest/community/development_process.html#backport-pull-request-process
00:15:27 <Kylie_> @zikalino, could you please follow the guidance to ask backport today?
00:15:28 <zikalino> oh, that's very useful doc, i will create a pr
00:15:38 <mattclay> I believe 2.6 is expected sometime in July, although I'd need to confirm that.
00:15:43 <zikalino> yes
00:15:49 <jborean93> The only thing that is missing is to create a changelog fragment https://github.com/ansible/ansible/tree/stable-2.5/changelogs/fragments
00:16:37 <jborean93> trying to find the bulk text as the docs still need to be created
00:17:05 <Kylie_> #action @zikalino, create backport PR for #36826
00:17:13 <jborean93> Here it is https://www.irccloud.com/pastebin/kDA2DbRj/
00:18:03 <Kylie_> #action @nitzmahone, review https://github.com/ansible/ansible/pull/37909
00:18:11 <Kylie_> Thank you Jordan.
00:18:20 <jborean93> you're welcome
00:18:56 <zikalino> we have two prs from external contributors to (2 facts modules)
00:18:57 <Kylie_> These two are top ones. We still have a bunch of PRs need your review, some are already confirmed by community contributors.
00:19:12 <Kylie_> #topic PRs need core committer's review
00:19:40 <zikalino> https://github.com/ansible/ansible/pull/38279
00:20:15 <jborean93> Send through the links and I'll add them to my review list
00:20:23 <jborean93> ah wait you sent an email recently
00:20:26 <jborean93> I'll go through that
00:20:41 <zikalino> https://github.com/ansible/ansible/pull/37333
00:21:03 <mattclay> For 38279 I don't see any integration tests.
00:22:52 <zikalino> yes indeed, i will ask the author to add
00:25:57 <Kylie_> Thank you Matt. And thank you Zim, please add integration test into your checklist for other PR review:). These two are good examples contributed by community contributor. We should respond issues and PRs raised by customers and community contributors timely. And then they will continue to contribute to Ansible project.
00:26:22 <Kylie_> And this is list in my email.
00:26:29 <Kylie_> https://github.com/ansible/ansible/pull/38127
00:26:29 <Kylie_> https://github.com/ansible/ansible/pull/38108
00:26:29 <Kylie_> https://github.com/ansible/ansible/pull/38077
00:26:29 <Kylie_> https://github.com/ansible/ansible/pull/38022
00:26:29 <Kylie_> https://github.com/ansible/ansible/pull/36109
00:26:29 <Kylie_> https://github.com/ansible/ansible/pull/35156
00:26:30 <Kylie_> https://github.com/ansible/ansible/pull/33063
00:26:31 <Kylie_> https://github.com/ansible/ansible/pull/32984
00:26:31 <Kylie_> https://github.com/ansible/ansible/pull/19513
00:26:32 <Kylie_> https://github.com/ansible/ansible/pull/37769
00:26:32 <Kylie_> https://github.com/ansible/ansible/pull/37627
00:26:32 <Kylie_> https://github.com/ansible/ansible/pull/35888
00:26:40 <Kylie_> @yuwzho, any missing or new one?
00:27:11 <yuwzho> let me check
00:27:51 <Kylie_> Thank you Yuwei.
00:28:21 <zikalino> yes, we try to keep external developers engaged
00:28:21 <Kylie_> Zim, I remember you mentioned some test cases were disabled and you would like to discuss how to open them. Still a valid topic?
00:29:45 <yuwzho> All the prs listed
00:30:08 <zikalino> well, still working on it, but slowed down due to other things we had to do, but other than that i think that goes pretty well. mattclay is very responsive here, and test prs are merged when ready.
00:30:56 <Kylie_> :+1: Nice. Thank you Zim and Matt Clay.
00:31:03 <Kylie_> Ok.
00:31:24 <zikalino> i don't want to enable tests too early while i am not sure if they are stable enough.
00:31:51 <Kylie_> https://www.irccloud.com/pastebin/HpPYGc61/
00:32:14 <jborean93> I've added myself to those PRs for review, will hopefully knock a few off in the coming days
00:32:29 * Kylie_ #action @jborean93 and @nitzmahone review below PRs
00:32:47 <mattclay> We're tracking unstable Azure tests here: https://github.com/ansible/ansible/projects/22
00:33:01 <thaumos> For those still here, we're finalising the dates for 2.6. Should have an update by Friday morning PDT.
00:33:04 <mattclay> Some of those have been disabled to avoid CI failures as they were failing quite frequently.
00:33:26 <Kylie_> Y, please. Thank you Jordan
00:33:40 <zikalino> oh, didn't know about this place
00:33:46 <mattclay> I've been working on a better way to handle unstable tests so we don't have to completely disable them. I'm hoping to have a solution this week.
00:34:20 <mattclay> The issues on that project board will automatically move to `Done` once they've been closed.
00:34:26 <Kylie_> Thank you Dylan. As for branch strategy, will it happen on 2.6 or 2.7?
00:35:56 <mattclay> zikalino: I've started tracking all unstable integration tests using issues and projects. All issues with Azure integration tests will show up on that project board. If the tests are frequently failing they'll be disabled, with that noted on their respective issue. If the failure is infrequent the test may remain enabled while awaiting a fix.
00:37:03 <mattclay> Once we have better handling of unstable tests in place, we should be able to disable them for unrelated CI runs while still running them for PRs that affect the module or the test itself, so we can still prevent regressions.
00:37:45 <mattclay> Right now if a test is unstable our only option is to disable it, which can lead to regressions when changes are made and the contributor doesn't realize the tests were not run.
00:37:48 <zikalino> ok, that sounds good. we are also performing full tests on our side. unfortunately some tests are unstable quite randomly....
00:37:53 <zikalino> but getting there.
00:37:53 <Kylie_> Zim, I think we are refining test cases in our playbook role. Once stable, you will merge to upstream. Right?
00:38:40 <zikalino> yes, however sometimes tests behave differently in our framework...
00:38:40 <mattclay> The most common causes of unstable tests I've seen are: 1) missing/inadequate retries within module logic 2) tests assuming they have exclusive use of a resource group
00:39:10 <mattclay> Tests have exclusive use of a resource group while they're running, but another test before or after that test can use the same resource group.
00:40:01 <zikalino> yes, there was one dependency where one test was deleting resource created by previous test, sth like that.
00:40:37 <zikalino> so (2) doesn't happen in our test framework because every test runs in a new resource group...
00:41:10 <mattclay> In most cases (2) should be fairly easy to fix, by having tests only manipulate resources created during the test run.
00:41:37 <zikalino> also sometimes deleting resorce, and then creating again resource with the same name in the same resource group may cause issues.
00:44:51 <Kylie_> BTW, as for test environment will you test running Ansible on different OS platforms?
00:44:52 <mattclay> Using the resource prefix (or a derivative of it) should solve that issue.
00:44:53 <mattclay> We run most of our non-cloud related integration tests on multiple platforms.
00:44:53 <mattclay> The expectation for cloud modules is that they're not OS-specific and would not benefit from testing on various platforms. But we do test them on both Python 2.7 and 3.6 to ensure compatibility there.
00:44:53 <zikalino> btw, i have a question related to aliases and groups:
00:44:53 <zikalino> posix/ci/cloud/group2/azure
00:44:54 <zikalino> what is the rule to allocate test to particular group?
00:44:54 <Kylie_> Good to know.
00:45:10 <mattclay> The groups are used to split up tests so no single group when run will exceed our CI timeout (under 1 hour).
00:45:23 <mattclay> Additionally, we try to keep cloud tests for the same cloud platform in the same group(s).
00:45:38 <mattclay> Periodically I rebalance the groups to keep the CI run times fairly even across groups.
00:45:53 <mattclay> So the general rule is to use the same group as existing tests for the same cloud platform.
00:46:38 <zikalino> ah, so i should use group with lowest execution time for new tests.
00:47:06 <mattclay> zikalino: That would be nice, but unless you look at a nightly run that executes all tests, you won't know which one  to choose.
00:47:36 <mattclay> When you submit a PR it will only run tests relevant for that PR, which for cloud modules will not be all tests.
00:48:01 <mattclay> So it's easiest to pick an existing group already used by Azure tests and leave the balancing of groups to me.
00:48:05 <zikalino> i noticed when i moify for instance azure_rm_common then it runs all tests, right?
00:48:27 <mattclay> If you modify module_utils code then it will run tests for all modules depending on that code.
00:48:38 <mattclay> If you modify a module or module's test, only the test for the module runs.
00:49:15 <zikalino> also woudn't it be possible to validate aliases somehow? once we had an issue that test was not running because alias was completely wrong, and we just haven't noticed that.
00:49:44 <mattclay> Yes, that is part of the project I'm working on to handle unstable tests. I hope to have it done this week still, assuming I don't have too many other things to deal.
00:49:45 <zikalino> (i guess alias was just old because pr was a few months old, and i think aliases have changed in a meanwghile)
00:49:55 <mattclay> s/deal/deal with/.
00:50:39 <mattclay> We will soon have sanity tests to make sure that aliases are correct. I'll also be adding notices to PRs when modules do not have integration tests, or the tests are disabled or unstable.
00:50:45 <mattclay> That part won't be done this week though.
00:51:35 <zikalino> ok, that sounds very good :-)
00:51:43 <mattclay> There will also be some new documentation about using aliases, once the sanity tests are in place.
00:53:21 <mattclay> zikalino: Are you continuing to work on improving the stability of the unstable tests I've reported?
00:54:06 <zikalino> yes, i was just disrupted by some other tasks, but i think we can enable a few tests even this week.
00:54:26 <zikalino> i will bother you tomorrow i think
00:54:59 <Kylie_> Good discussion here:)
00:55:15 <Kylie_> Since we are overrun, I will ask any other topic?
00:55:18 <mattclay> Oh, before I forget. I don't know if Matt Davis mentioned it, but I'll be coming with him to meet with your team on Monday.
00:55:50 <Kylie_> Nice, nice though I am not there. You guys could have very efficient F2F.
00:56:00 <Kylie_> Matt, do you also locate at Oregon?
00:56:10 <mattclay> Kylie_: Yes
00:57:30 <Kylie_> Great.
00:59:31 <Kylie_> I did not see meeting room info. I will ask Catherine update the room info and leave contact info in case you guys cannot find the right building.
00:59:44 <Kylie_> It would be a wonderful meeting.
00:59:56 <Kylie_> Ok. Then if no other topic, I will end meeting.
01:00:16 <Kylie_> #endmeeting Ansible Azure Working Group