17:02:34 <mattclay> #startmeeting Testing Working Group 17:02:34 <zodbot> Meeting started Thu Jul 13 17:02:34 2017 UTC. The chair is mattclay. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:02:34 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 17:02:34 <zodbot> The meeting name has been set to 'testing_working_group' 17:02:52 <mattclay> #chair gundalow 17:02:52 <zodbot> Current chairs: gundalow mattclay 17:02:59 * mattclay waves 17:03:01 * Pilou waves 17:03:06 <mattclay> #chair Pilou 17:03:06 <zodbot> Current chairs: Pilou gundalow mattclay 17:03:46 * mattclay skims https://github.com/ansible/community/issues/114 for open items to discuss 17:04:23 * spredzy waves 17:04:35 <mattclay> #chair spredzy 17:04:35 <zodbot> Current chairs: Pilou gundalow mattclay spredzy 17:04:47 <mattclay> A few updates before we get started: 17:05:08 <mattclay> We have a new "import" sanity test in ansible-test / CI now. 17:05:13 <gundalow> #info :) 17:05:22 <mattclay> gundalow: Thanks 17:05:28 <gundalow> For all the people that look at the logs *cough* 17:06:02 <mattclay> #info We have a new "import" sanity test in ansible-test and CI now. It will detect imports of python modules not in the standard library in either modules or module_utils which are not protected by a try/except block. 17:06:29 <mattclay> #info We will be extending this test to cover plugins as well. 17:07:00 <mattclay> #info We will be dropping testing of Fedora 24 and 25 with python 2 on Shippable and adding Fedora 25 and 26 with python 3. 17:07:34 <mattclay> gundalow: Do you have anything? 17:08:49 <gundalow> nopt 17:08:54 * gundalow -> afk 17:09:06 <mattclay> #topic Replacing pep8 with pycodestyle (https://github.com/ansible/ansible/pull/25947) 17:09:24 <mattclay> This has been on the agenda for a while, but hasn't been discussed yet. 17:09:52 <mattclay> Pilou: This one's yours. :) 17:10:03 <Pilou> yep :) 17:11:07 <mattclay> The changes look good. I'm OK with keeping the test named pep8 even though the tool running the test is pycodestyle, especially since it's ultimately enforcing PEP 8 rules. 17:11:22 <mattclay> So +1 from me 17:11:40 <mattclay> Does anyone else have comments? 17:11:51 <Pilou> we might want to fix E741 17:12:00 <Pilou> which is "ambiguous variable name" 17:13:06 <Pilou> there isn't many occurrence of this error in the current code 17:13:23 <mattclay> That's the biggest change with moving to pycodestyle. Using pep8 has been stable since it's no longer updated. Using pycodestyle is going to give us a moving target as more rules are added. 17:13:36 <mattclay> Which will periodically break CI until we either fix or ignore the issues. 17:14:01 <mattclay> If it gets to be a problem we can switch to a whitelist of rules to enforce instead of a blacklist. 17:14:21 <mattclay> Pilou: Once it's merged you can submit a separate PR to fix all the E741 issues at once. 17:14:34 <Pilou> ok 17:14:42 <mattclay> How many are there anyways? 17:14:43 <Pilou> hum, the version of pycodestyle package should be enforced 17:15:04 <spredzy> For the migration to be seemless you can also make test fail only is they are more failures on current commit then previous 17:15:24 <spredzy> This way one doesn't get penalized by a -1 unless is code is doing something wrong 17:15:26 <mattclay> spredzy: We've got that covered already with our ignore list. 17:15:45 <spredzy> ack 17:16:07 <mattclay> Pilou: We typically only require specific version when they're either 1) broken or 2) older versions lack features we require 17:16:22 <mattclay> Shippable will always run with the latest version. 17:16:31 <mattclay> Users are free to use whatever they want locally. 17:16:38 <Pilou> but new version of pycodestyle could break the CI 17:16:42 <mattclay> Obviously that can lead to differences between local testing and CI. 17:17:09 <mattclay> Yes, which makes it obvious we have something to address. We can either decide to ignore or fix. 17:17:39 <mattclay> If it becomes a frequent occurrence we'll change our approach. 17:17:58 <Pilou> ok 17:18:22 <mattclay> If we pin the version then new issues that could be detected will go unnoticed. 17:18:45 <mattclay> Until someone runs locally with a newer version and doesn't use our requirements file. 17:19:18 <mattclay> Are there any objections on merging this PR then? 17:20:05 <spredzy> nop 17:21:07 <mattclay> I'm re-running CI on the PR now. 17:21:15 <mattclay> #action mattclay to merge PR once CI passes 17:22:11 <mattclay> #topic Community Wiki (https://github.com/ansible/community/issues/114#issuecomment-310875236) 17:22:22 <mattclay> I don't believe we've discussed the community wiki here yet. 17:23:07 <mattclay> Does anyone have anything to add to the testing group page here? https://github.com/ansible/community/tree/master/group-testing 17:23:42 <mattclay> Or the wiki page here? https://github.com/ansible/community/wiki 17:24:11 <mattclay> Sorry, testing specific link for the wiki: https://github.com/ansible/community/wiki/Testing 17:24:17 <gundalow> back 17:24:41 * mattclay away for a minute 17:25:51 <gundalow> #info Testing Wiki exists at https://github.com/ansible/community/wiki/Testing 17:26:12 <gundalow> Thanks to those that have added stuff so far 17:26:37 * mattclay is back 17:26:39 <gundalow> Pilou: Do you have have edit permissions on the wiki? 17:27:08 <gundalow> if not, would you like them? 17:27:27 <Pilou> gundalow: yep, i have 17:27:33 <gundalow> Great 17:28:10 <mattclay> Speaking of things to put on the wiki. Does anyone want to volunteer to fix existing modules/module_utils which fail the new import sanity test? 17:30:19 <gundalow> mattclay: Where is the test defined, e.g. how can people see the todo list? 17:30:41 <mattclay> #info Existing files which don't pass the import sanity test: https://github.com/ansible/ansible/blob/devel/test/sanity/import/skip.txt 17:31:09 <mattclay> They're pretty easy to fix, but there's a lot of them. 17:31:21 * gundalow wonders if I should add that list to the wiki, feels like anyone that's comfortable with Python could chip away at the list 17:31:29 * mattclay nods 17:33:03 <gundalow> and it's: ansible-test sanity --test import 17:34:36 <mattclay> #topic Open Floor 17:35:06 <mattclay> One thing I forget in my earlier update. We now have our largest module with 100% code coverage: ping :) 17:35:07 <mattclay> https://codecov.io/gh/ansible/ansible/src/devel/lib/ansible/modules/system/ping.py 17:35:41 <gundalow> mattclay: for import/skip are we specifically lookin for module_utils, or any/all to be fixed? 17:35:47 <mattclay> This is a side-effect of the new import sanity test, as it covers the other half of the `if __name__ == '__main__'` branch not hit during normal module execution. 17:36:03 <mattclay> gundalow: module_utils and modules both 17:36:10 <gundalow> ack 17:36:30 <mattclay> Plugins too, once the test is updated to include them. 17:36:59 <gundalow> What's the point of `data: crash` in the ping module? 17:37:37 <mattclay> I believe it's there for testing when you want a module to raise an unhandled exception. 17:38:34 <gundalow> ah, OK 17:38:59 <mattclay> Now that's it possible for modules to reach 100% code coverage, I'm hoping we'll start seeing some slightly more complex modules hitting that. 17:39:16 <gundalow> https://github.com/ansible/community/wiki/Testing%3A-progress-tracker/_edit#fixing-broken-imports added 17:39:24 <gundalow> :) 17:39:38 <Pilou> about pycodestyle and E741, there are only 9 occurrences of this error 17:40:19 <Pilou> #25798 : a test_pull_requests & stale_ci labele PR, reviews are welcome 17:43:24 <mattclay> Pilou: Looks good. I've restarted CI on that one. 17:44:47 <mattclay> Does anyone have anything else to discuss? 17:44:49 <Pilou> about new 'import' test, i locally deleted 'lib/ansible/modules/cloud/amazon/cloudformation.py' from 'test/sanity/import/skip.txt' and 'ansible-test sanity --python 2.7 --tox --test import' fails with 17:44:51 <Pilou> 'ERROR: lib/ansible/modules/cloud/amazon/cloudformation.py:259:0: ImportError: No module named utils.hashing' 17:45:08 <Pilou> is that a false positive ? 17:45:19 <mattclay> Pilou: No, it's importing something from outside module_utils. 17:45:45 <mattclay> Pilou: That code needs to be copied to module_utils or the module itself. 17:46:11 <Pilou> ok, thanks for the explanation :) 17:46:40 <mattclay> It probably works for most people because they run the module on their local host where ansible is installed. 17:46:49 <mattclay> If you ran the module on a remote host without ansible it would fail. 17:47:31 <Pilou> when the code is copied to module_utils, is there a licence issue ? 17:48:08 <mattclay> You'll need to look at the licenses in both places. 17:48:28 <mattclay> It might be better to copy the code to the module, since there's only one usage. 17:48:56 <Pilou> ok 17:49:38 <Pilou> when the code should be moved to module_utils, what is the workflow to follow ? asking all authors of the file is they are ok ? 17:50:08 <mattclay> abadger1999: ^ I believe you've had to deal with this before? 17:50:38 * abadger1999 looks 17:50:48 <abadger1999> Pilou: yeah :-( that's the way to do it. 17:51:01 <abadger1999> We can talk about alternatives if there's authors you can't find. 17:51:21 <abadger1999> but that will involve real legal-knowledgable people 17:52:09 <abadger1999> Pilou: email (with complete headers) is one way to get consent. And we have started to use acceptance on a github ticket for that purpose as well. 17:52:45 <gundalow> Pilou: start with the easy ones :) 17:53:01 <Pilou> :) 17:53:26 <gundalow> It's about reducing the list, doesn't matter what order that's done in 17:53:36 <gundalow> And thanks in advance for your help with it :) 17:53:50 <mattclay> Pilou: If you copy to the module it should be OK, they're both GPL 3. 17:54:54 <jtanner> can i interject a testing question? 17:55:02 <mattclay> jtanner: Sure 17:55:18 <jtanner> is there a way to tell ansible-test not to tear down the containers at the end of the test? 17:55:27 <jtanner> specifically for the vcsim container 17:55:46 <mattclay> jtanner: Not currently. 17:55:50 <jtanner> crud 17:55:58 <jtanner> i guess i'll just do it manually 17:56:02 <jtanner> first world problems 17:56:02 <gundalow> We have --remote-terminate for AWS, don't seem to have the same for Docker 17:57:31 <mattclay> OK, anything else? We're almost at the end of our scheduled meeting. 17:57:40 <gundalow> Nothing else from me 17:58:53 <mattclay> #endmeeting