20:00:43 <jborean93> #startmeeting Ansible Windows Working Group
20:00:43 <zodbot> Meeting started Tue Feb 26 20:00:43 2019 UTC.
20:00:43 <zodbot> This meeting is logged and archived in a public location.
20:00:43 <zodbot> The chair is jborean93. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00:43 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:00:43 <zodbot> The meeting name has been set to 'ansible_windows_working_group'
20:00:46 <dag> o/
20:00:49 <nitzmahone> o/
20:00:50 <jborean93> hey
20:00:57 <jborean93> #chair dag nitzmahone
20:00:57 <zodbot> Current chairs: dag jborean93 nitzmahone
20:01:06 <vutkin> Hi
20:01:21 * nitzmahone feels like he should \o or something just to be different ;)
20:01:30 <jborean93> #chair vutkin
20:01:30 <zodbot> Current chairs: dag jborean93 nitzmahone vutkin
20:01:31 <jborean93> hey
20:01:39 <dag> I think it is time to warn people with illegible PRs that the freeze is coming up
20:01:40 <jborean93> \o/
20:01:41 <tgadiev> hey
20:01:52 <jborean93> #chair tgadiev
20:01:52 <zodbot> Current chairs: dag jborean93 nitzmahone tgadiev vutkin
20:02:15 <jhawkesworth> hey
20:02:19 <jhawkesworth> sorry I'm late
20:02:28 <jborean93> #chair jhawkesworth
20:02:28 <zodbot> Current chairs: dag jborean93 jhawkesworth nitzmahone tgadiev vutkin
20:02:40 <jborean93> no worries, it's always a guessing game whether your nick would have the _ on the end :)
20:03:05 * jborean93 brings up the agenda
20:03:14 <jhawkesworth> something irccloud does for me, hopefully fixed now though
20:04:17 <jborean93> looks like I forgot to update last week's agenda items, will do that after today
20:04:25 <jborean93> #topic https://github.com/ansible/community/issues/420#issuecomment-467431381 newline newline_sequence
20:04:30 <jborean93> dag this one is yours
20:05:11 <dag> so, that comment summarized it
20:05:54 <dag> do we want to bring consistency, or leave it as-is ? I was confused about the difference (e.g. expected newline to have '\n')
20:05:54 <nitzmahone> so long as we never need to support wackadoo old-Mac style newlines or something, I'd probably lean toward the `unix`/`windows` choice vs explicit sequence, but 🤷
20:05:57 <jhawkesworth> I guess the 3 options available for `newline_sequence` are more flexible than the 2 options supported by `newline` in current implementations
20:06:21 <jborean93> sounds like a question for the main Ansible meetings as well. Not something I feel comfortable with deciding here
20:06:37 <dag> ok, I don't mind moving it up the chain
20:06:42 <jborean93> I don't feel strongly either way
20:06:53 <marwatk> I'd like to point out, though, that afaik nothing uses \r anymore.
20:06:53 <dag> but the newline was added by us, both in template and win_template
20:07:04 <dag> and wasn't discussed up the ladder either
20:07:47 <jhawkesworth> true, but in the past collectively we didn't take enough care with consistency between modules.
20:07:49 <dag> marwatk: I am sure we did not add that in vain
20:08:02 <jborean93> I'm happy with the windows/unix choice but I'm a +0 either way
20:08:05 <dag> it must have come from somewhere
20:09:15 <it-praktyk[m]> hi
20:09:22 <jborean93> hey
20:09:33 <jhawkesworth> fwiw personally I use jinja directive in templates for line endings and hardly use (win_)lineinfile at all
20:09:51 <jborean93> that's a good point, what's the syntax for jinja2?
20:10:02 <jhawkesworth> in fact have been replacing some win_lineinfiles with win_xml
20:10:07 <jhawkesworth> oh its ugly, let me check
20:13:02 <jhawkesworth> yeah its
20:13:04 <jhawkesworth> #jinja2: newline_sequence:'\r\n'
20:13:12 <jhawkesworth> (to get windows line ending)
20:13:18 <jborean93> ok
20:13:41 <jborean93> well I think this should be a question for the greater Ansible space as this could be used in other Python modules
20:13:46 <jhawkesworth> yeah.
20:13:53 <jborean93> I'm happy to go with whatever choice they want
20:14:09 <jhawkesworth> I wonder how much use block/lineinfile gets in comparison with template/win_template
20:14:22 <jborean93> well lineinfile shouldn't span across newlines
20:14:26 <jhawkesworth> since template has been around for the longest time
20:14:46 <jborean93> I assume blockinfile uses `\n`
20:15:06 <jhawkesworth> yeah but optionally writing a consistent line ending is not a bad feature I think
20:15:36 <jborean93> yep, dag are you wanting to drive that in the main meeting?
20:16:15 <jhawkesworth> but yeah ask in the core meeting.  I'd be sad if I had to change all my templace win_template invocations, but I shouldn't have to as long as the jinja2 directive is still honoured
20:16:50 <jborean93> I doubt that would change, it's a core interface and pretty integral for a change in behaviour
20:17:14 <jborean93> by doubt, I mean that any changes we make shouldn't break existing behaviour
20:17:17 <dag> ok
20:17:25 <marwatk> There's another option to have them be aliases of each other and allow for \r\n, \r, \n or windows/unix
20:17:27 <jborean93> cool
20:17:45 <jborean93> #action dag to bring newline discussion to the main Ansible meetings
20:18:09 <jborean93> #topic https://github.com/ansible/community/issues/420#issuecomment-467451424 - win_get_url checksums
20:18:13 <dag> jhawkesworth: that is where the parameter newline_sequence came from in template ! That makes sense
20:19:30 <jhawkesworth> yeah I suppose they are slightly different things with template/win_template being very much jinja2-powered
20:19:44 <jborean93> tgadiev: was there anything in particular you wanted to talk about. I've yet to go back over your replies in my comments and see what you have changed
20:19:46 <dag> marwatk: So the option for '\r' comes directly from Jinja:http://jinja.pocoo.org/docs/2.10/api/#high-level-api
20:20:43 <tgadiev> jborean93: I'd like to note about how to perform http tests
20:21:08 <tgadiev> I think we need single solution for all Windows modules
20:21:43 <tgadiev> Currently Linux modules use python based http server for that purpose
20:21:50 <tgadiev> And so did we
20:22:06 <jborean93> Linux has the benefit of having Python already available, this isn't something I feel comfortable trying to add to Windows
20:22:11 <dag> yeah, it's been an item on our agenda for over 2 years now
20:22:14 <jborean93> CI already takes time and now we add an external dependency
20:22:39 <nitzmahone> We kicked around the possibility of a native-Windows HTTP server (eg using a lightweight .NET HttpServer-based impl), but nobody's had the time to make it cover all the bases
20:22:40 <tgadiev> agree, thus we need to decide what tool would be better to use
20:22:55 <nitzmahone> IIS is too heavy
20:28:05 <nitzmahone> (and doesn't really gain us much anyway)
20:28:06 <jborean93> we can give access to the httptester container which is the same as what's used in uri and win_uri
20:28:06 <jborean93> but that doesn't give you the ability to serve your own files
20:28:07 <jhawkesworth> .net core 2 kestrel would be lighter.. but probably won't run on all windows
20:28:07 <tgadiev> right
20:28:08 <tgadiev> I propose two options there
20:28:09 <tgadiev> nginx, which is quite easy to install
20:28:10 <tgadiev> and Ran - very light single binary built especially for such cases
20:28:10 * jborean93 looks at Ran
20:28:10 <jborean93> do you have a link, I just get results for the Royal Australian Navy
20:28:11 <jhawkesworth> this perhaps? https://github.com/m3ng9i/ran
20:28:11 <tgadiev> Yes, right
20:28:11 <nitzmahone> Ideally whatever we choose would also be able to mimic the stuff that badssl does
20:28:12 <tgadiev> But Navy sounds good as well :)
20:28:12 <nitzmahone> (eg, so we can be sure cert validation etc is working properly)
20:28:12 <jborean93> nitzmahone: you can use httptester for that right now
20:28:12 <nitzmahone> right, I meant if we do a Windows-native thibgy
20:28:13 <jborean93> this is more for testing the checksum support that's being added to the PR
20:28:13 <jborean93> httptester works on Windows
20:28:13 <nitzmahone> sounds like someone's proposing a replacement though
20:28:13 <nitzmahone> "I think we need single solution for all Windows modules"
20:28:13 <jborean93> I'm mostly proposing a way to easily test this checksum behaviour
20:28:19 <jborean93> get_url tests use some Python http library and setup the files to service
20:28:38 <jborean93> I don't think we will find a one solution fits all unless we write it ourselves
20:28:40 <nitzmahone> yep (ala .NET HttpServer)
20:29:09 <dag> there was an IRC split
20:29:11 <dag> (21:22:58) dag: originally I wrote a basic Powershell HTTP server, but in order to have the full functionality from the python http server was just too much work
20:29:11 <dag> (21:23:31) dag: the solution back then was to open up the python http-service to the Windows VMs
20:29:11 <dag> (21:23:41) dag: but I don't think it ever happened
20:29:11 <dag> (21:24:03) dag: mattclay ?
20:29:11 <dag> (21:25:01) dag: https://meetbot.fedoraproject.org/ansible-meeting/2017-06-29/testing_working_group.2017-06-29-17.00.log.html
20:29:11 <dag> (21:25:04) dag: See 17:06
20:29:18 <tgadiev> How about nginx?
20:29:36 <jborean93> The Ran is good to me because it's a binary that doesn't need to be installed
20:29:45 <dag> tgadiev: the old meeting logs from the testing WG explains the solution
20:29:48 <nitzmahone> nginx is still a pretty heavyweight solution
20:29:49 <jborean93> We cna easily host that in our S3 bucket to reduce the possibilities of a network failure
20:30:03 <jborean93> dag: we do support httptester in Windows but that won't help with this problem
20:30:19 <tgadiev> to me Ran is also good
20:30:31 * dag missed part of the discussion :-/
20:30:39 <jborean93> the only worry I have is it looks like we need to use async which I have concerns with in CI
20:30:44 <tgadiev> It should fit test needs
20:30:46 <nitzmahone> httptester *could* if we wanted to put a few static files in it that we know won't go away, but yeah
20:30:58 <jhawkesworth> dag: looking at RAN webserver https://github.com/m3ng9i/ran
20:31:19 <jborean93> yea, that would require some more work on the controller side. I probably would like something like that eventually but I don't want to hold this PR up too much
20:32:52 <jborean93> tgadiev: what exactly do you need for testing checksums?
20:33:05 <tgadiev> so, for that PR what shall we try? or leave it with python?
20:33:14 <nitzmahone> def no on the Python
20:33:23 <jborean93> yea I'm -1 on Python
20:33:53 <tgadiev> I agree that Python it not a good choice for Windows
20:34:28 <tgadiev> But I'd like to avoid different approaches in modules
20:34:31 <jborean93> I'm just thinking you could achieve all this right now without a special http host
20:34:43 <tgadiev> With each one using its own thing
20:34:50 <nitzmahone> `ran` from the S3 bucket + async is far preferably to Python, but ideally it'd be async-free...
20:35:13 <nitzmahone> a temp surrogate service *might* be better?
20:35:35 <tgadiev> I guess we need async if we'd like to test authorization options etc.
20:35:37 <jborean93> you can use https://httpbin.org/ (like our httptester) to get a file and compare the checksums against a known value
20:35:57 <jborean93> the only thing you can't do is checksum_url but you could use an ftp or file source for that
20:37:00 <dag> So what is the problem with the solution from mattclay in 2017 ? Setting up a tunnel from the Windows infra to the Unix infra and use the http test service we use for Unix ?
20:37:10 <jborean93> dag: we have that, that's what I'm proposing
20:37:43 * dag is confused about Ran and half of the discussion, sorry
20:37:43 <tgadiev> Problem is with static files required to be downloaded for tests
20:37:52 <jborean93> what static files are those?
20:38:08 <tgadiev> vutkin: can you shed some light?
20:38:11 <jhawkesworth> can you just win_copy the static files into place first?
20:38:33 <dag> aren't we using the same tests for uri and get_url, if not, why not ? Any static files for those tests should be fine for Windows ?
20:38:52 <jborean93> dag: get_url uses some Python http library to do these tests
20:38:57 <jborean93> I just don't think it's necessary
20:39:04 <tgadiev> currently get_url don't use httptester for file download tests either
20:39:12 <jborean93> what files are those
20:40:18 <vutkin> we use same test as get_url uses: create files with known checksums and compare them in assets
20:40:37 <jborean93> but all you need is some data, why can't you use https://httpbin.org/image which is a static image and then compare the checksum
20:41:16 <vutkin> we generate them in our tests every run
20:41:22 <jborean93> why?
20:41:52 <vutkin> because get_url has same tests
20:42:18 <jborean93> let's ignore get_url for now, we can't do the exact same thing there because it relies on Python
20:42:39 <tgadiev> yes, we tried to make module as close as possible to the things done with get_url
20:42:48 <tgadiev> to avoid different approach
20:42:59 <jborean93> so my proposal is to use the existing httptester framework we have, you have static files you can use for checksum tests
20:43:06 <nitzmahone> dynamically-generated data is not necessary to exercise this module
20:43:22 <jborean93> the only thing that won't work is `checksum_url` but you can specify a file URI for a local file to test the checksum logic
20:43:49 <tgadiev> is it possible to put specific static files with some known checksum to the storage httptester uses?
20:43:56 <jborean93> no but you don't ened to
20:44:23 <jborean93> there's a bunch of images you cna use https://httpbin.org/#/Images for checksum tests
20:44:33 <jborean93> they stay static and have a few options
20:44:36 <nitzmahone> just fetch the file, generate the checksum locally, then re-fetch with checksum
20:44:45 <jborean93> ^ or that
20:44:58 <nitzmahone> (verifies idempotence inline)
20:45:37 <tgadiev> is it ok to bind to this external resource for ansible tests?
20:45:42 <vutkin> httpbin is written on python too
20:45:57 <jborean93> ansible-test has some logic to do this all for you
20:46:06 <jborean93> checkout the win_uri tests for an example https://github.com/ansible/ansible/tree/devel/test/integration/targets/win_uri
20:46:38 <jborean93> it uses the ansible-test httpbin instance if possible, otherwise it will use the external one. The external one shouldn't be used in CI which is what is important
20:47:00 <jhawkesworth> nice
20:47:07 <jborean93> so for example, you need to set `needs/httptester` and `skip/windows/2008` in the aliases file
20:47:19 <jborean93> create a meta/main.yml file like https://github.com/ansible/ansible/blob/devel/test/integration/targets/win_uri/meta/main.yml
20:47:47 <jborean93> then refer to the URL like `https://{{ httpbin_host }}/`
20:48:17 <jborean93> `httpbin_host` is templated to either be the local instance of httpbin or the external one if httptester wasn't being used (which won't be the case in CI)
20:50:07 <tgadiev> so, your proposal it to use checksum_downloads tests with local source only right?
20:50:31 <jborean93> you could use the ftp service that is used in the test but yes
20:51:00 <jborean93> using a local file URI is probably a lot easier to test though
20:51:44 <tgadiev> yes, but it doesn't represent a real life case for http download
20:51:55 <vutkin> unfortunately it has different logic for http and local file check
20:52:01 <vutkin> >there's a bunch of images you cna use https://httpbin.org/#/Images for checksum tests
20:52:03 <tgadiev> but if it's acceptable, ok
20:52:04 <jborean93> only for checksum_url
20:52:41 <tgadiev> ok, agreed
20:53:16 <vutkin> files on https://httpbin.org/#/Images could be changed in future and we should keep it in mind
20:53:23 <jborean93> you could also have a failure for checksum_url pointing to that another static page and verify it couldn't get the checksum
20:53:39 <jborean93> vutkin: that's why nitzmahone said to download it once and get the checksum first
20:54:04 <nitzmahone> (calculate the checksum externally with a `pipe` lookup or something)
20:54:29 <vutkin> so, if we download a broken file first tim, we get failed checksum:)
20:54:33 <jborean93> so that way you verify your checksum_url actually get's a valid response for web URIs but the actual checksum parsing is still testing with the far easier file sources
20:55:17 <vutkin> failed=wrong
20:55:47 <jborean93> just downlaod the file without the checksum first and then use something like win_stat to get the real checksum
20:56:13 <jborean93> then you can do the same thing with the correct checksum already known
20:56:30 <tgadiev> I think vutkin means that file can be corrupted or broken during first download
20:56:31 <nitzmahone> ah yeah, forgot not downloading to controller ;)
20:56:48 <nitzmahone> If that happens, the test will fail either way
20:57:03 <tgadiev> than checksum will differ even if it's right
20:57:03 <jborean93> yea you would still have the same issue with the other way
20:57:06 <jborean93> not much we could do there
20:57:22 <nitzmahone> if we can't rely on the http downloads working, the whole test is going to fail regardless
20:58:13 <vutkin> exactly for that we generate static files with known checksums
20:58:36 <nitzmahone> but if the infra is failing, you're still going to fail the checksum anyway
20:58:41 <jborean93> what he is saying if there's a corrupt download the whole purpose of win_get_url is failing
20:58:58 <nitzmahone> (so we want it to fail in that case anyway)
20:59:44 <vutkin> ok, let use ftp for now
21:00:42 <jborean93> anyway we are running out of time
21:00:48 <jborean93> was there anything else you wanted to discuss there
21:01:01 <tgadiev> are there any other points preventing PR to go forward?
21:01:25 <jborean93> testing and making the changes a bit more concise, right now there are a few things in there that are duplicated and could be simplified
21:01:49 <jborean93> I think there were some points around the `msg` return value as well
21:02:39 <jborean93> I'll need to go back over the PR and see what you've replied with based on my review comments
21:03:08 <tgadiev> ok, is there a chance that PR can go forward until freeze?
21:03:25 <jborean93> if we address all the points I don't see why not
21:04:05 <tgadiev> ok, so please take a look and let us know. We tried to address all the items you mentioned
21:04:27 <jborean93> if you need help with the testing side, feel free to ping me
21:04:42 <jborean93> Depending on time zones it may be hard for us to overlap, my day is just starting
21:04:49 <tgadiev> ok, will do. thanks a lot for your help!
21:04:50 <jborean93> but I'll try my best
21:05:02 <tgadiev> yeah, we know :)
21:05:20 <jborean93> thanks all for coming
21:05:23 <jborean93> have a good one all
21:05:28 <dag> o/
21:05:29 <jhawkesworth> cheers
21:05:32 <jborean93> #endmeeting