17:00:02 #startmeeting Testing Working Group 17:00:02 Meeting started Thu Mar 9 17:00:02 2017 UTC. The chair is gundalow. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:00:02 Useful Commands: #action #agreed #halp #info #idea #link #topic. 17:00:02 The meeting name has been set to 'testing_working_group' 17:00:07 #chair mattclay 17:00:07 Current chairs: gundalow mattclay 17:00:19 Matt will join in a few minutes 17:00:31 * gundalow waits for people to appear 17:00:47 howdy 17:01:59 hey sivel 17:02:19 #topic Module DOCUMENTATION validation 17:03:26 #info Mass tidy up of module documentation has been completed https://github.com/ansible/ansible/pull/22297 17:03:55 #info gundalow still working on the tool to enforce this https://github.com/ansible/ansible/pull/22353 17:04:29 assert alikins == present 17:04:31 #info Added support for "suboptions" e.g. dict's as options https://gist.githubusercontent.com/gundalow/4bdc3669d696268328ccc18528cc6718/raw/5c7cfabb6e33d6ff127fddb86bccd5d4e5ffedc0/nested-provider.png 17:04:36 hey alikins :) 17:04:53 gundalow: where did the PR for the `module` name validation go? 17:05:08 sivel: I killed it and I'll move it into 22353 17:05:22 Ok, I was going to recommend that 17:05:36 * mattclay waves 17:05:38 Your suggestion made sense, just not got to it yet 17:05:40 since you were modifying doc_schema anyway, converting into a factory could go there 17:05:40 hey mattclay 17:05:48 :) 17:05:59 Bingo :) 17:06:40 So 22353 will contain the updated schema validation & updates to eveloping_modules_documenting.html 17:06:44 developing_modules_documenting.html* 17:07:03 nice 17:07:21 In my travels around hacking/template I found that we have an undocumented feature of "type: bool" which expands to choices Yes/No. So I'll document that 17:07:36 mattclay: Thanks for your suggestions https://gist.githubusercontent.com/gundalow/4bdc3669d696268328ccc18528cc6718/raw/5c7cfabb6e33d6ff127fddb86bccd5d4e5ffedc0/nested-provider.png 17:07:46 My aim is to get this in before we branch 17:07:53 cool 17:07:58 gundalow: It's looking good now. Nice work. 17:08:06 As we will be having branch documenttion soon* 17:08:14 I have a big E1 pep8 clean up going in after we branch 17:08:28 should free up a lot of time when doing reviewing 17:08:52 yeah, the better we can get our validation, the less time needed on manual validation 17:09:06 Still some work left, like making sure version is a string of X.Y 17:09:32 and to add in nested schemas (though I found some notes) 17:10:27 I looked at your voluptuous reference, and seems like that is in their docs for recursive schemas, so +1 on that 17:10:39 ah, cool 17:10:54 Thanks 17:11:01 Any other suggestions on that? 17:12:35 #topic General updates 17:12:36 Sounds like you have plans on better validation on version_added, if you have questions let me know. but I'm guessing you are OK there. Voluptuous is pretty straight forward 17:12:55 sivel: Yup, I just need to read more of their docs 17:13:00 so far it's been really nice 17:13:06 once again, thanks for the tool 17:13:10 * gundalow hands over to mattclay 17:13:15 I use it everyday, so if you have questions, let me know 17:13:22 Thanks to jtanner's bot work, the bot is now commenting on failed PRs with the details of compile and sanity failures: https://github.com/ansible/ansible/pull/22419#issuecomment-285391904 17:13:55 yay! I saw he got validate-modules being included yesterday 17:14:01 Eventually we'll be able to expand that to include unit and integration test results. 17:14:13 sivel: If you have any other big schemas examples you could point me at, that would be cool 17:14:28 That's great, will save everyone loads of time 17:14:30 mattclay: My only "concern" with those is how we display them, since they are likely to be really big 17:15:01 I'm working on updating ansible-test to distinguish between failures that are PR related and those that are not, so we can hopefully avoid false positives due to failing tests merged to devel. 17:15:28 sivel: Yeah, we probably will link to the Shippable tests tab after we get those cleaned up. That's why they aren't in there right now. 17:16:12 We've been focusing on the majority of failures, which is compile and sanity tests. 17:16:47 I feel like I am a bad person, but I get a decent amount of enjoyment everytime I see CI fail 17:17:13 Means we are testing useful stuff 17:17:20 Once we're able to check compile/sanity failures against PR changes, the bot will be able to label issues with `ci_verified`, which will reduce the amount of manual review required for failed PRs. 17:18:32 I'm also working on updating how we write our code-smell tests so that they can behave more like a normal sanity test. That will include testing only changed files, as well as having lint friendly output. 17:19:17 The end goal being that most sanity test failures will be "lint like" in the form: file:line:column: message 17:20:20 I have on my to do list, to update how errors/warnings/traces are added to the reports in validate-modules. I don't think a tuple is the best long term soltuion, especially when line/column is moved out of the message 17:21:48 I kinda like how ansible-test handles the messages using SanityMessage (some implementation like that) 17:22:12 After we branch for stable-2.3 I'm going to be back porting improvements to test infrastructure (mainly ansible-test) to that branch when possible. Most new tests won't be able to be enabled, unless they happen to already pass in stable-2.3, as I don't want to be backporting fixes just to keep tests happy. 17:22:56 +1 17:23:30 Once I merge the E1 fixes into devel after the 2.3 branching, we have line length errors left in legacy. Once we empty out legacy, then current should become legacy right? 17:23:40 sivel: Yes 17:23:47 cool, just double confirming :) 17:23:54 a feature I would (sometimes) like that everyone else would hate: for CI failures that can point to file:line, comment on the line with the info in the pull request 'files changed view' 17:24:21 I'm not sure people would hate it 17:25:21 I've thought about that. I think it's something worth considering later, once things have stabilized more. 17:26:30 I am probably forgetting, but do we have a planned date for the 2.3 branching? 17:26:36 However, to avoid spamming lots of comments to users, it would need to be a review rather than individual comments. 17:27:50 I believe around March 15th, when we expect RC1 to be ready. 17:29:01 That's it for me on updates. 17:29:20 Nothing else from me 17:30:10 #topic Open Floor 17:31:20 how would I go about getting the 'cryptography' module installed/available for the unit tests envs? 17:32:09 alikins: Add it here: https://github.com/ansible/ansible/blob/devel/test/runner/requirements/units.txt 17:32:10 it's a compiled ext, so just adding it to requirements doesnt work with docker (since the docker images dont have a compiler) 17:32:33 You're using docker for unit tests? Or you want this for integration tests? 17:33:20 I do not use docker for unit tests, but adding the requirement seemed to break the docker intg tests iirc 17:33:49 let me confirm 17:33:57 needs to go into the dockerfiles and built+pushed to the docker registry? 17:34:18 suspect so 17:34:22 If you only need it for unit tests, you can add it to `units.txt`. However, that will prevent us from running unit tests inside docker, so I'll need to update the docker containers. 17:34:39 jtanner: Yeah, we update the Dockerfiles in the repo then I rebuild them on Docker Hub. 17:35:16 (oh, and 'cryptography' is just to speed up the vault tests) 17:35:26 I was just going to ask why you wanted it. 17:36:08 Are you planning on committing the change, or you just wanted to do this for local testing? 17:37:58 alikins: I assume you're using the --tox option to run unit tests? If so, and you have cryptography installed globally, you could use the --tox-sitepackages option to make it available without any further changes. 17:42:02 ah, adding 'cryptography' to requirements/integration.txt and 'ansible-test integration --docker centos7 --requirements' fails (presumably needs cryptography installed/built at docker build time) 17:42:38 Yeah, that's why I was asking if you wanted it for just unit tests. 17:43:03 If we want it in docker, we should use distro packages instead of pip. 17:43:05 haven't benchmarked it for intg tests, likely insignificant 17:43:18 * gundalow has to head off now 17:44:25 but I run just the vault unit tests often, and it's 2-3x different for that case but also solvable by local installs of cryptography 17:44:39 alikins: Does the --tox-sitepackages option work for your needs? 17:44:53 I'd rather not add cryptography to units.txt to avoid making it a requirement for tests. 17:46:20 I currently dont use the --tox options. But I also have it working for my local testing. It's just slow if you dont know to get that setup. 17:47:10 If you're not using --tox then adding it to units.txt won't do anything for you anyways. 17:50:59 If there's nothing else, I'll end the meeting in 1 minute. 17:53:03 #endmeeting