17:00:02 #startmeeting Testing Working Group 17:00:02 Meeting started Thu Jul 6 17:00:02 2017 UTC. The chair is gundalow. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:00:02 Useful Commands: #action #agreed #halp #info #idea #link #topic. 17:00:02 The meeting name has been set to 'testing_working_group' 17:00:08 #chair mattclay 17:00:09 Current chairs: gundalow mattclay 17:00:14 mattclay: over to you :) 17:00:15 o/ 17:00:26 #chair spredzy 17:00:26 Current chairs: gundalow mattclay spredzy 17:00:28 Hey :) 17:00:31 I rebranded from davidn to newswangerd because IRC kept yelling at me over duplicate names 17:00:31 * mattclay waves 17:00:41 #chair newswangerd 17:00:41 Current chairs: gundalow mattclay newswangerd spredzy 17:01:37 Hi! 17:01:51 There isn't a lot to announce this week. However, one change I thought worth mentioning is that ansible-test now checks to make sure ~/.ssh/known_hosts isn't changed during a test run. This was added to avoid issues like the one I fixed with the git test where it deleted the file. 17:03:35 (back) 17:04:34 #info There isn't a lot to announce this week. However, one change I thought worth mentioning is that ansible-test now checks to make sure ~/.ssh/known_hosts isn't changed during a test run. This was added to avoid issues like the one I fixed with the git test where it deleted the file. 17:04:48 #topic Use of test_pull_requests label 17:06:07 #info Currently the `test_pull_requests` label gets applied to a PR if *any* file under ./test gets changed. So we often end up with that label added even if someone has just tied up pep8. This makes searching for PRs that add/alter tests a pain 17:06:26 #info Should we ignore ./test/sanity/? 17:06:58 +1 17:07:04 what do other people think? 17:07:30 I think that would be too much. I like the idea of ignoring just the pep8 legacy-exclusions.txt file. 17:07:46 sure, I'm happy with that 17:07:47 That should take care of the main issue of people cleaning up PEP 8 on files but otherwise not changing tests. 17:08:09 ^ +1 17:08:20 #chair shaps 17:08:20 Current chairs: gundalow mattclay newswangerd shaps spredzy 17:08:25 Evening :) 17:08:42 hi :) 17:09:15 #action just ignore legacy-files.txt 17:12:08 mattclay, gundalow suggests chatting about the requirements.txt here as there isn't much in the agenda 17:12:29 #topic Installation of Integration Test Requirements 17:12:47 * resmo waves 17:13:17 Currently we handle integration test requirements in several ways. Some tests install requirements at the start of the test, others rely on the test environment already having the necessary requirements. 17:13:57 For the tests that expect the environment to already be set up, we use test/runner/requirements/integration.txt for pip requirements. 17:13:58 huch, ignoring the pep8? -1 17:14:13 I mean ignoring legacy-files.txt -1 17:14:44 if I do a bug fix of a legacy file, I don't want to fix pep8 17:15:12 otherwise the patches to backport get cluttered 17:15:22 resmo: The proposed change for legacy-files.txt is only for labeling PRs, nothing else. 17:15:43 ah, ok, +1 then ;() 17:16:23 For macOS and FreeBSD integration tests we also use test/runner/setup/remote.sh to configure the environment, mostly to install OS packages. 17:16:30 mattclay, so from a quick look there shouldn't be much in requirements.txt 17:17:03 other than 'cryptography and junit-xml (?)' 17:17:21 We're moving away from installing requirements before tests start to doing so during the tests. As we do so, we should be able to remove test-specific requirements from test/runner/requirements/integration.txt as well as test/runner/setup/remote.sh 17:17:38 The things that should remain are either required to run ansible or the test infrastructure. 17:18:37 Basically, the criteria should be: Is this requirement needed only for specific test(s)? If so, the test itself should handle requirements installation. 17:19:38 well, at a guess ansible deps are already installed when integration tests run, right? 17:20:04 shaps: No, that's not the case. 17:20:11 also, are we only considering integration.txt or integration*.txt? 17:20:30 Just integration.txt -- the specialized ones for cloud tests are a different matter. 17:20:34 resmo: context: We want it to be easier for the various working groups to do label:aws label:testing_pull_request to find PRs that add/extend tests 17:20:38 ok 17:21:03 gundalow: I see 17:22:05 mattclay, I know what the ansible requirements are, not sure about the test infra ones though 17:22:07 I believe these requirements can be moved into tests: jmespath passlib pexpect 17:22:42 jmespath is used by the json_query filter 17:22:44 yeah, that's what I was looking for 17:23:03 pexpect is using the expect module 17:23:08 s/using/used by/ 17:23:13 yep, I know roughly where those deps are used 17:24:08 worst case 'grep -r ' in the docs will find the ones I'm not aware of :) 17:24:34 CI should be able to help out here a lot. If we remove a requirement, the test(s) that fail depend on it. If everything fails, that's not one we should be changing. 17:25:02 that's a bit of a drastic way, but yeah, it works 17:25:26 I wasn't suggesting we use that as the means to identify them -- only that it will catch any mistakes we make. 17:26:38 Once we've taken care of requirements.txt, we can look at the packages installed by test/runner/setup/remote.sh and see which of those need to go into tests. For those, it's most likely just a case of adding support for macOS and FreeBSD, since the tests already handle their requirements for Linux. 17:27:04 yeah, had a look at the file, doesn't seem to scary 17:27:12 s/to/too/ 17:27:29 Well, actually that's not true. The docker images have a lot of pre-installed packages. 17:27:52 So we'll have tests which don't install any of their own requirements, even for Linux. 17:28:28 I'm only interested in focusing on the ones that are in remote.sh though. The fewer we have there, the less overhead we have to run tests there. 17:28:57 My suggestion would be to have a common 'setup_env' role, which will install the packages required, makes it easier for people to add requirements later 17:29:28 I believe we can probably have a closer look to the ones not installing their requirements later 17:29:31 shaps: How would that work? The tests are already roles, and we need requirements installation to be per-test, not global for all tests. 17:31:55 we could use include_role 17:32:30 How does splitting it into a role help? Most of the requirements will only be included once, since most tests don't share requirements. 17:33:10 yeah, that's true. Doesn't really make sense 17:34:05 For example, looking at remote.sh there's the installation of mercurial. That's only needed for the hg test. 17:34:15 Ones where there are groups, such as mysql_db/ mysql_user/ mysql_variables/ there often is a common role, such as setup_mysql_db which can be used 17:34:30 So instead of remote.sh installing it for every test run, the hg test should install it before running the tests. 17:34:43 * mattclay nods 17:35:00 yep, makes sense 17:36:02 The goal here is to simplify global test requirements and speed up environment set-up. This benefits us both for CI (where we wait for fewer requirements to install when running a subset of tests) as well as manual testing locally, as people often focus on a specific test. There's no need to wait for requirements to be installed that you won't be needing. 17:36:39 agree 17:36:56 shaps: Does this give you enough to work with? 17:36:59 I believe I have enough info 17:37:09 yep sure. 17:37:11 Does anyone else want to work with shaps on this? 17:37:56 * gundalow will be around for general questions 17:38:02 * shaps looks at tumbleweed :D 17:38:08 shaps: FYI - I'm going to be updating remote.sh soon, as I'm adding support for testing RHEL. I'll start with requirements installed there just like we do for the other platforms. We can move them out of there one test at a time. 17:38:27 As always #ansible-devel for Testing questions (outside of Testing Working Group) 17:38:57 shaps: Are you going to start with requirements.txt first? 17:39:00 mattclay, yeah. I'll focus on the integration.txt first, I'll have a look at remote.sh later 17:39:18 Well, test/runner/requirements/integration.txt I should say. 17:39:30 Sounds good. 17:39:33 I was going to keep them as 2 separate PRs anyway. So I think it works well with what you need to do 17:39:37 Thanks for volunteering to work on this. 17:39:56 np 17:40:18 #topic Open Floor 17:40:49 sorry if it took me a bit to get back to you, it's been a hell-week. 17:41:15 No problem. 17:46:47 Thanks for coming everyone! 17:46:49 #endmeeting