17:00:23 <mattclay> #startmeeting Ansible Testing Working Group
17:00:23 <zodbot> Meeting started Thu May 16 17:00:23 2019 UTC.
17:00:23 <zodbot> This meeting is logged and archived in a public location.
17:00:23 <zodbot> The chair is mattclay. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:23 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
17:00:23 <zodbot> The meeting name has been set to 'ansible_testing_working_group'
17:00:27 <mattclay> #info Agenda: https://github.com/ansible/community/issues/248
17:00:31 <mattclay> #chair pabelanger
17:00:31 <zodbot> Current chairs: mattclay pabelanger
17:00:38 <mattclay> pabelanger: Do you have any Zuul updates for us today?
17:01:02 <pabelanger> sure, I have to drop shortly, but can update
17:01:27 <pabelanger> we are making good process on importing network appliances into zuul.a.c, today we have eos / junos / vyos
17:01:58 <pabelanger> we are also running periodic jobs (every 24 hours) of ansible-test network-integration using them
17:02:24 <pabelanger> so far, ansible/ansible works as expects (same with ansible-test) and just working thought failures due to local inventory issues / new images
17:02:41 <mattclay> pabelanger: Have you looked at running those jobs in sync with the Shippable nightly jobs so we can generate a unified code coverage report? We were doing this with DCI, at least at one point.
17:02:43 <pabelanger> next up will be cisco, maybe start on iosv today
17:03:12 <pabelanger> mattclay: not yet, but we have the ability to run then at any time. For now, it is just 00:00 UTC
17:03:23 <pabelanger> if you want to point me to location, I can start looking into that
17:03:40 <mattclay> Basically use a cron job to check for the SHA used by the Shippable nightly run and run tests on that same SHA with coverage reporting enabled and sending to codecov.io.
17:03:47 <jtanner> o/
17:04:04 <mattclay> That way we have a single report with coverage data for both the Zuul and Shippable CI runs.
17:04:12 <pabelanger> mattclay: sure, we can do that. Currently we just use the the tip of branch for periodic
17:04:17 <pabelanger> ack
17:04:21 <mattclay> Using the same SHA is what allows them to be merged.
17:04:50 <pabelanger> Oh, so the sha you are using is not merged?
17:04:57 <pabelanger> or am I not understanding
17:05:20 <mattclay> Both Shippable and Zuul need to test the same ansible/ansible SHA when reporting the coverage results.
17:05:40 <mattclay> That will allow codecov.io to merge the results from both CI systems into a single report.
17:05:52 <pabelanger> oh, I see
17:05:55 <pabelanger> merged reporting
17:05:58 <pabelanger> not merged code
17:06:00 <pabelanger> got it
17:06:28 <pabelanger> I'll start looking into that, and likely have some questions out of band
17:06:29 <mattclay> The way gundalow had set it up was to have a cron job run shortly after the Shippable nightly runs. That job used the Shippable API to get the SHA from the nightly run and then test the same SHA on DCI.
17:07:10 <pabelanger> yah, we should be able to do the same in zuul. Will required us to enqueue a specific sha into the pipeline
17:08:07 <mattclay> pabelanger: Anything else to report on?
17:08:26 <pabelanger> mattclay: aside from that, the only other item I wanted to discuss again was the PR integration for ansible-zuul github app on ansible/ansible. Do you think we can enable that, then work on ansiblebot itegration for branch protection?
17:08:57 <pabelanger> we'd like to demo what premerge testing looks like to ansible network team, and need to events from github first
17:10:18 <pabelanger> the only other zuul related issue, is I am hoping to land ansible-2.8 support in zuul shortly (once ansible 2.8.0 is released). Then jobs will be able to request ansible-2.8 for running jobs, today we support 2.5 / 2.6 / 2.7.
17:10:31 <mattclay> pabelanger: Are you planning on using the GitHub status API (the same one Shippable is using) to report the results on PRs?
17:11:29 <pabelanger> mattclay: yes, it should be the same. We need to bikeshed I think on what the name will be, I for now I think it would show up as ansible/third-party-ci.  But we can name it anything we want, third-party-ci represents the pipeline in zuul configuration
17:12:34 <mattclay> pabelanger: OK, so Zuul will report a pending/running status if it decides it needs to run tests on the PR, and then report results on completion -- otherwise it will report nothing? Or will it always report something, even if no tests need to run?
17:13:37 <pabelanger> mattclay: correct, for now, we won't report anything, but jobs will run.  I figure, once we figure out the naming for check api stuff, and you are happy with it, we can start reporting back to ansible/ansible. We have a lot of control of what we want to send back and when
17:14:07 <mattclay> pabelanger: Oh, so you want to start by running tests but not reporting any results back at all?
17:14:31 <pabelanger> mattclay: yah, I think that is the safest step first, unless you want to just right to reporting.
17:14:39 <pabelanger> I have no preference honestly
17:15:00 <pabelanger> but, we will only report back on network related stuff, for vyos / eos / junos ATM
17:15:01 <mattclay> pabelanger: I'm fine with starting out with no reporting. I just wanted to understand what you were planning on doing.
17:15:38 <pabelanger> okay, cool. I'd say, enable github app, no reporting is step 0.  That way we can ensure jobs are working on a selected PR
17:15:57 <pabelanger> then, once we understand ansiblebot check api intergation, maybe have you review and enable
17:16:21 <pabelanger> I still need to understand some of the branch protection stuff ansiblebot does (if any)
17:16:38 <mattclay> We'll probably need jtanner's help to understand how the bot will handle the additional status reports before we turn them on.
17:16:40 <pabelanger> I know you shared that last week, but haven't looked yet
17:16:49 <pabelanger> mattclay: sure, wfm
17:17:25 <mattclay> pabelanger: Is there anything else you need to set up before we can turn on the integration without the reporting?
17:17:52 <jtanner> it depends on what is wanted
17:17:58 <jtanner> i can simply ignore the ci endpoint
17:17:59 <pabelanger> mattclay: no, we are ready right now on zuul side. Once github app is installed, we start getting the events
17:18:08 <jtanner> or i can add it to the needs_revision calculation
17:18:26 <mattclay> jtanner: Initially we could start with just having the bot ignore the endpoint, and then work on adding functionality from there.
17:18:39 <jtanner> yeah, that was my first plan for sure
17:18:48 <jtanner> hence telling paul to tell me before it goes live
17:18:54 <mattclay> I know we had problems in the past with multiple endpoints -- I wasn't sure if the bot ignored new ones by default now or not.
17:19:13 <jtanner> it excludes anything !shippable right now
17:19:17 <jtanner> just checking real quick
17:19:31 <pabelanger> +1 I don't want to start reporting anything until everybody is onboard, and we've at least tested it.  Breaking ansible/ansible would be bad for zuul
17:19:46 <mattclay> pabelanger: OK, why don't we plan on turning it on on Monday, to avoid making changes on release day or a Friday. :)
17:19:56 <jtanner> https://github.com/ansible/ansibullbot/blob/master/ansibullbot/triagers/plugins/needs_revision.py#L135-L136
17:20:01 <pabelanger> mattclay: wfm, thanks!
17:20:11 <jtanner> yeah, if the context isn't shippable, it'll be ignored ... currently
17:20:29 <mattclay> jtanner: Perfect. That's where we'd like to start.
17:21:04 <pabelanger> k, I'll read up on ansibullbot for next week
17:21:25 <mattclay> pabelanger: OK, we can sync up on Monday then and enable the app for ansible/ansible.
17:21:37 <pabelanger> mattclay: thanks
17:22:19 <mattclay> Does anyone have anything else to discuss?
17:22:42 <pabelanger> nothing from me, also relocation again
17:22:45 <pabelanger> thanks everybody
17:22:49 <jtanner> nothing
17:24:41 <mattclay> Thanks everyone.
17:24:43 <mattclay> #endmeeting