17:03:43 <mattclay> #startmeeting Ansible Testing Working Group
17:03:43 <zodbot> Meeting started Thu Jul 18 17:03:43 2019 UTC.
17:03:43 <zodbot> This meeting is logged and archived in a public location.
17:03:43 <zodbot> The chair is mattclay. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:03:43 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
17:03:43 <zodbot> The meeting name has been set to 'ansible_testing_working_group'
17:03:49 <jtanner> o/
17:04:30 <mattclay> #info Agenda: https://github.com/ansible/community/issues/248
17:04:36 <mattclay> #chair jtanner pabelanger
17:04:36 <zodbot> Current chairs: jtanner mattclay pabelanger
17:04:44 <mattclay> pabelanger: Do you have any Zuul updates for us this week?
17:07:00 <mattclay> jtanner: Did you have anything you wanted to discuss?
17:07:07 <jtanner> nope
17:07:31 <pabelanger> mattclay: nope, things are working really well
17:07:43 <pabelanger> we are just working on 2.9 features now
17:08:44 <mattclay> Early support for testing collections with ansible-test will be available soon. The current PR is here: https://github.com/ansible/ansible/pull/59197
17:09:21 <mattclay> There will be a lot of work to do once that is merged, but it provides enough of a foundation to run some tests on collections.
17:11:54 <mattclay> One important thing to note about the change is that ansible-test will require your current working directory to be within either the Ansible source tree (when testing Ansible) or an Ansible collection under a collection root (when testing a collection).
17:12:25 <mattclay> Currently you can run ansible-test from any cwd and it will test the current Ansible source, even if you're outside that tree.
17:15:32 <mattclay> That's all I have for today, unless someone has questions.
17:17:06 <mattclay> sdoran: Did you want to give any updates on where things are at with the Python 3.8 + coverage related freezes in tests?
17:18:36 <jtanner> mattclay: do you want me to shop https://github.com/ansible/ansible/pull/59197 around and get people playing with it?
17:18:43 <jtanner> before you demo it
17:19:23 <pabelanger> where are we pulling python 3.8 from?
17:19:53 <mattclay> pabelanger: We're using Python 3.8 Beta 2 from our default test container: https://github.com/ansible/default-test-container
17:20:32 <pabelanger> mattclay: ack, we could start doing python3.8 for network integration too, if interested. But haven't really pushed on doing it yet
17:22:10 <pabelanger> will also give ansible-test changes a run, to ensure nothing breaks on our side
17:22:28 <mattclay> jtanner: It will probably be more useful for people to play with once it's merged, since that will make it easier to expose issues that need to be worked on.
17:22:36 <jtanner> alrighty
17:23:46 <mattclay> pabelanger: It wouldn't hurt to do a pass with Python 3.8. You might run into the same freezes we're seeing in CI, at least if you enable code coverage.
17:24:04 <pabelanger> ++
17:24:50 <mattclay> jtanner: I'm putting together a list of things that will need to be worked on to support collections testing once the initial PR is merged. There's definitely work that can be split up if people are available to work on issues.
17:25:11 <jtanner> create the issues and we'll farm them out
17:25:47 <mattclay> Will do.
17:27:52 <mattclay> jtanner: The main thing I'm concerned with at this point for my PR is that it doesn't break existing user's workflow testing the Ansible source. Between CI and pabelanger using it for networking tests I think we'll be OK.
17:28:17 <jtanner> that's fine
17:28:23 <jtanner> if we break something, we can fix it
17:29:18 <jtanner> a lot fewer people will be testing ansible/ansible anyway if this goes well
17:29:33 <jtanner> since most are just trying to get their module into core
17:29:56 <pabelanger> when do we want to merge it?
17:30:07 <pabelanger> I can get us results today on network side, so it doesn't break us
17:30:56 <mattclay> pabelanger: I'll wait until Monday. That way I don't need to worry about breaking anything over the weekend.
17:34:23 <pabelanger> k, we'll have some results in 90mins I'd say
17:35:58 <sdoran> Sorry I'm a bit late.
17:36:30 <sdoran> I don't have much to report on the Python 3.8 hanging other than it is still present in 3.8.0b2 and it is related to the `coverage` plugin.
17:37:05 <sdoran> I am able to reproduce it very easily with an isolated test. I just need to dig into the `coverage` code to see where the problem is.
17:37:25 <sdoran> I do not think it is related to our coverage pytest plugin.
17:37:42 <sdoran> It seems it is getting stuck when making a call to a method in `coverage`.
17:37:47 <jtanner> can repro locally?
17:38:38 <sdoran> Yup. I have it nailed down pretty good.
17:39:06 <jtanner> i can try to debug if you want. i work with hangs a lot =)
17:39:13 <sdoran> I probably should try to see if I can reproduce it with some sample code that calls `coverage` rather than being invoked via `pytest` in `ansible-test`.
17:39:50 <mattclay> sdoran: How about the win_ping hang? That's still ansible-test, but no pytest.
17:40:39 <mattclay> With a bit of work you could set it up to run coverage directly w/o ansible-test.
17:41:38 <sdoran> Hmm, well the `win_ping` test that's hanging is running `ansible-test` with the `--coverage` flag. So I'm guessing that still makes a call to `coverage` somewhere. I would have to look more.
17:41:46 <sdoran> But it seems like all roads lead to `coverage`
17:41:56 <sdoran> Well, `coverage` + Python 3.8
17:42:15 <mattclay> sdoran: Yes, all of them go through the `coverage` cli ultimately.
17:43:01 <dag> acozine: you cannot default to string for docs, we have had that discussion three times already
17:43:34 <dag> bcoca: We added proper types to most (if not all) Windows modules, but we don't do doc/argspec checking in validation
17:43:51 <mattclay> dag: Wrong channel?
17:44:33 <dag> mattclay: no, responding to previous messages where I was mentioned
17:44:45 <mattclay> dag: Ah, previous meeting then.
17:45:26 <dag> hmm, is there a new meeting ongoing ? sorry for that
17:46:09 <mattclay> dag: Yeah, Testing Working Group right now. I'm guessing you're responding to comments from the core meeting before this.
17:46:59 <mattclay> Does anyone else have something to discuss?
17:53:26 <bcoca> actually, that question is testing related
17:53:46 <bcoca> do we have 'validate-modules' for windows? does it handle the doc/spec sync?
17:54:33 <bcoca> ^ no need to answer now
17:59:55 <mattclay> bcoca: Module validation does support windows modules. I don't know all of the differences in what is/isn't checked -- but it does do at least some checking of module arguments against docs.
18:00:15 <bcoca> gtk
18:00:53 <mattclay> OK, I think that's it for today.
18:00:57 <mattclay> Thanks everyone.
18:01:01 <mattclay> #endmeeting