20:02:07 <nitzmahone> #startmeeting Ansible Windows Working Group
20:02:07 <zodbot> Meeting started Tue Jun 19 20:02:07 2018 UTC.
20:02:07 <zodbot> This meeting is logged and archived in a public location.
20:02:07 <zodbot> The chair is nitzmahone. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:02:07 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:02:07 <zodbot> The meeting name has been set to 'ansible_windows_working_group'
20:02:15 <jborean93> wow zodbot is back!
20:02:20 <jhawkesworth_> wecome back zodbot
20:02:32 <nitzmahone> hooray
20:02:45 <nitzmahone> #chair jborean93 jhawkesworth_
20:02:45 <zodbot> Current chairs: jborean93 jhawkesworth_ nitzmahone
20:04:05 * jborean93 checks agenda
20:04:22 * daBONDi sitting in the corner and making the coffe for the table :-)
20:05:16 <jborean93> #topic https://github.com/ansible/community/issues/294#issuecomment-397296168 default become method
20:05:28 <nitzmahone> That's covered by the ansible_platform stuff in 2.7
20:05:43 <nitzmahone> Basically "sane defaults for a given platform"
20:05:59 <jhawkesworth_> great.  well that was dealt with pretty quick
20:05:59 <jborean93> sounds good to me
20:06:04 <jborean93> haha yea
20:06:17 <nitzmahone> But you'd have to actually use it- some of my old prototypes tried to work backwards for things that didn't use it
20:06:40 <nitzmahone> But the way we're likely to do it won't do that, so unless you've specified `ansible_platform=windows` somewhere, it'll still fail like that
20:06:55 <jhawkesworth_> I recall some chat long ago about needing to separate concerns better between the connection plugins and the code surrounding them
20:06:58 <nitzmahone> If it works out right, there will be a lot of muscle memory and old content to ditch
20:07:10 <nitzmahone> Yeah, that's sort of related but also a different issue
20:07:20 <jborean93> I'm used to specifying all those extra vars for Windows
20:07:23 <nitzmahone> With the advent of "real" ssh on Windows and (maybe) PSRP
20:07:28 <nitzmahone> we'll have to revisit
20:07:50 <nitzmahone> Platform solves most of those problems on the user side
20:07:57 <jhawkesworth_> tbh I set as much of that stuff as possible in group_vars and then basically modify it ... almost never
20:09:14 <jborean93> yea I personally err on the side of explicit vars but that's because I know/understand most of how they interact in Ansible
20:09:21 <jborean93> for end users that's usually never the case
20:09:21 <jhawkesworth_> sounds great.  if you can set ansible_platform=windows on your group_vars that would make it a one-time setup sort of thing
20:09:32 <bcoca> taht is the plan
20:09:33 <nitzmahone> exactly
20:09:49 <bcoca> even better, let aws_ec2 dynamic inventory do it ...
20:09:50 <nitzmahone> Platform won't prevent people from doing that stuff either
20:10:02 <bcoca> it just makes it less neccessary
20:10:07 <jhawkesworth_> ok well I'll update the agenda
20:10:57 <jborean93> #topic open floor
20:11:23 <jhawkesworth_> something that came up earlier on IRC was `setup` doesn't have a default fact path for windows
20:11:35 <bcoca> for ansible_local
20:11:45 <bcoca> i thought there was PR that tried to add it
20:11:56 <nitzmahone> Yeah, I was going to respond to that
20:12:09 <jhawkesworth_> oh I have missed that PR
20:12:36 <jborean93> on Windows though?
20:12:43 <jborean93> or is this something different?
20:12:48 <jhawkesworth_> having a default fact path would be nice and a linux parity thing, but would it be possible to agree on a default location
20:13:03 <jborean93> is this to point to a script that gathers more facts?
20:13:17 <nitzmahone> Yeah, %PROGRAMDATA%\AnsibleFacts or something would probably be the closest to /etc/something
20:13:20 <jhawkesworth_> yes this is that
20:13:24 <bcoca> jborean93: its a dir with either scripts or json files  that then get pushed into ansible_local. dict
20:13:28 <nitzmahone> Yeah, custom local facts
20:13:51 <nitzmahone> Windows support for local facts was added ~2.2 IIRC, but it always required explicit call
20:13:58 <nitzmahone> with fact_path specified
20:14:04 <jborean93> I don't see why we can't just default and check the path
20:14:07 <bcoca> ah, that might be what i remember
20:14:29 <nitzmahone> Yeah- I'm not opposed to adding a default path to check
20:14:57 <jhawkesworth_> %PROGRAMDATA%\AnsibleFacts sounds good to me
20:15:00 <bcoca> actual equivalent to /etc/ in windows is the registry ...
20:15:07 <jhawkesworth_> quite
20:15:30 <jhawkesworth_> not sure we'd want a default registry path for local facts
20:15:32 <bcoca> programdata is per user?
20:15:39 <jborean93> C:\ProgramData
20:15:41 <daBONDi> Programdata is not writable per default with user rights, so maybe there will be a problem in the long run, only Creators rights so if you switch with become?
20:15:46 <nitzmahone> Kinda, but orthogonal to this issue- ProgramData is the closest thing to "a filesystem place for stuff to store config"
20:16:06 <nitzmahone> We don't want it to be writable
20:16:07 <jhawkesworth_> yeah /etc files are usually fairly locked down for writes
20:16:09 <bcoca> +1 for programdata
20:16:12 <nitzmahone> We want it as locked down as possible
20:16:19 <nitzmahone> (which ProgramData is)
20:16:37 <nitzmahone> But it's a supported place that admins could drop custom facts modules securely
20:16:54 <jhawkesworth_> sounds like a plan
20:17:06 <nitzmahone> IIRC it is world *readable* though (which we want)
20:17:07 * nitzmahone looks
20:17:12 <jborean93> I'm cool with that
20:17:27 <jborean93> arguably most WinRM connections are with an admin user anyway
20:17:28 <nitzmahone> Yep, Users has read/execute
20:17:38 <jhawkesworth_> I'll make a PR for 2.7 then
20:17:42 <nitzmahone> Yeah- we just want to make sure we don't bomb facts on non-admiin
20:17:49 <nitzmahone> We've fixed some bugs around that, might be some more
20:17:58 <jborean93> wouldn't it be a try and look, ignore if you can't?
20:18:06 <nitzmahone> Yep
20:18:20 * nitzmahone can't remember if we added setup tests for nonadmin
20:18:21 * jhawkesworth_ makes notes
20:19:34 <jborean93> cool
20:19:34 <nitzmahone> howdy dag!
20:19:40 <jborean93> hey
20:19:40 <daBONDi> a non admin user can put a File on %ProgramData%\<SubFolder> with the Creation rights, so this is maybe a security hole, when the Script get created with User Rights, and executed remote over WinRM with Admin rights, this is my only concern
20:19:48 * nitzmahone typed that very carefully today
20:21:35 <jborean93> hmm
20:21:38 <nitzmahone> @daBONDi - I'm not seeing how that's possible without someone explicitly allowing write access to nonadmins on the AnsibleFacts subdir
20:21:40 <jhawkesworth_> that does sound like a security hole
20:22:23 * jborean93 tests it out
20:23:44 <daBONDi> ,@nitzmahone look on the NTFS Policy on a system, it says  Creater User: Full Access Subfolder and Files, so when Ansible create the folder "\ansible_facts" and Default Execute a Script on "\ansible_facts\get_facts.ps1" but Script is not there, a User can create it and put his stuff in it, or i'm thinking wrong about this ?
20:24:55 <jborean93> yea he's right, as a standard user I can create the folder and a ps1 script
20:25:44 <jborean93> even if the folder exists I still have rights as a standard user in ProgramData
20:26:26 <nitzmahone> balls
20:26:30 <nitzmahone> Well, good catch there
20:26:39 <jhawkesworth_> yeah, thanks
20:27:18 <jhawkesworth_> I'll play around with it a bit
20:27:18 <daBONDi> think better approche would be the Profile Path of the WinRM User maybe, so there only the User and Admin can write it, or the System Profile
20:27:35 <nitzmahone> So I guess unless we want to default to something that's actually default-secured under %WINDIR% or %ProgramFiles%...
20:27:41 <jborean93> trouble with user based paths is that you need to do it per user
20:27:50 <nitzmahone> Yeah, it needs to be a global path
20:27:54 <nitzmahone> (or both)
20:28:32 <daBONDi> then prbly programFiles\Ansible there only a Admin User can write
20:28:43 <jhawkesworth_> I'd be happy with only a global path
20:29:02 <nitzmahone> Yeah, that's how posix facts does it now IIRC
20:29:04 * nitzmahone looks
20:29:07 <daBONDi> lol don't know that, there is also the "CREATER_OWNER2 rule
20:29:34 <jborean93> standard user can't create folders or files in Program Files
20:29:40 <daBONDi> on Program Files there is also the Same creater/owner rule like in programdata
20:29:41 <jborean93> "by default"
20:30:18 <daBONDi> you can when the User is the Owner of the folder
20:30:27 * jhawkesworth_ thinking user-specific facts should be a different thing from facts about a host, but that's another debate
20:30:35 <dag> o/
20:30:39 <jborean93> what user would be an owner of the Program Files folder?
20:31:00 <daBONDi> lets hope know one :D
20:31:31 <daBONDi> i would suggest a default location for ansible_facts but only as non executable maybe ?
20:31:53 <jborean93> I feel like you would have a bigger program is a non admin can create/modify in Program Files
20:32:13 <jborean93> I don't think we currently support non executable
20:32:17 <jborean93> just ps scripts right now
20:32:24 <jhawkesworth_> yes, just ps scripts now
20:32:48 <jhawkesworth_> if I was going to have read only facts, I'd pick a registry location
20:33:23 <nitzmahone> Regardless, no longer a slam dunk, so ... *punt*
20:33:31 <jhawkesworth_> but that's going away from just getting to feature parity
20:33:34 <jhawkesworth_> agreed
20:33:50 <jhawkesworth_> anybody got anything else, didn't mean for this to take over!
20:34:13 <jborean93> only wanted to mention that I was working on a psrp connection plugin based on https://github.com/jborean93/pypsrp
20:34:29 <jborean93> no PR ready just yet as I'm trying to iron out the problems
20:35:06 * jhawkesworth_ looks
20:35:23 * daBONDi looks, and add star + fav :D
20:36:38 <daBONDi> Whats the different to pywinrm?
20:37:18 <jborean93> it uses WinRM but through the PowerShell Runspaces, similar to Enter-PSSession or Invoke-Command
20:37:46 <daBONDi> i was thinking pywinrm do this also, then i'm probly missinformed
20:37:49 <jborean93> it's meant to be faster than WinRM as you don't need to start a ps interpreter on each command but it does have more messages to exchange
20:38:01 <nitzmahone> Some potential performance improvements, but only once we implement connection persistence
20:38:19 <jborean93> yep, it is probably dependent on persistance
20:38:21 <nitzmahone> (which we can't turn on by default until we solve the HTTPS wedging issue, lest we brand ourselves as masochists)
20:38:39 <jborean93> There are other benefits it brings but not really relevant to Ansible
20:38:51 <jborean93> nitzmahone: I still don't think we should limit it because of that
20:39:04 <jborean93> I've yet to come across it and rarely see something who is potentially affected by it
20:39:05 <jhawkesworth_> well looking forward to having it as an option
20:39:05 <nitzmahone> Nor do I, just not on by default
20:39:12 <jborean93> even still
20:39:25 <jborean93> why limit ourselves by default when it only affects a tiny subset of users
20:39:25 <nitzmahone> It's more common than you think, but only people that are doing >5M transfers hit it
20:39:50 <jborean93> if it was more common we would have more reproducers
20:39:55 <nitzmahone> Even if it hits 1% (which I think is low, actually), we'd be shooting ourselves in the foot
20:40:14 <nitzmahone> Not necessarily- only people using win_copy or large task loops are likely to hit today
20:40:15 <jhawkesworth_> if file transfer is speedier and everything else balances out speedwise it would be worth all your hard work to have
20:40:31 <jborean93> file transfer is still the same with psrp unfortunately
20:40:36 <jborean93> fetch should be faster though
20:40:41 <nitzmahone> If we turn it on by default, we'll find out pretty quickly I'm sure ;)
20:40:49 <jborean93> yea that's true
20:41:20 <nitzmahone> So long as there's an opt-out, I suppose we could try on-by-default, but makes me really nervous
20:41:40 <jborean93> we shall even see if persistance even gives us better performance anyway
20:42:17 <nitzmahone> It does
20:42:29 <nitzmahone> I've prototyped it before
20:42:29 * jhawkesworth_ slightly wishes I hadn't made win_copy a thing.
20:42:49 <nitzmahone> Depending on various factors, it can give a 5-20% boost
20:42:58 <jborean93> jhawkesworth_: we've pretty much gotten to a point where win_copy is the best we can do with winrm
20:43:00 <jhawkesworth_> all speedups will be appreciated by me
20:43:16 <jborean93> probably some small tweaks here and there but nitzmahone it really pushing the limit of winrm there :)
20:43:41 <jhawkesworth_> yep its as good as it gets now
20:44:05 * nitzmahone is also curious to see how SSH performs over WInRM in side-by-side tests
20:44:19 <jborean93> from tests, standard execution was roughly the same
20:44:27 <jborean93> file copy was a way faster
20:44:36 <jhawkesworth_> I still wonder about doing a stealthy reverse win copy which starts tcp server on ansible and then basically uses win_get_url to fetch the file from the controller
20:44:55 <nitzmahone> DANGER WILL ROBINSON, DANGER!
20:45:05 <nitzmahone> ;)
20:45:07 <jhawkesworth_> yeah I know
20:45:11 <jborean93> I was toying with the idea of having a psexec_copy that uses SMB as a different module
20:45:43 <bcoca> net copy ...
20:45:59 <daBONDi> to be honnest, i never was in the need of win_copy, every normal decent windows shop, got some configuration asset server on https :D
20:46:06 <nitzmahone> --^
20:46:11 <bcoca> or cifs
20:46:15 <jborean93> decent being the key word
20:46:18 <nitzmahone> That's the reason I think the HTTPS wedging thing is bigger than we think
20:46:45 <jborean93> bcoca: cifs/smb same thing
20:47:11 <daBONDi> @nitzmahone what is that https wedging thing ? never heard about
20:48:15 <jborean93> potentially this https://github.com/ansible/ansible/issues/41145
20:48:35 <jborean93> there's some older issues floating around somewhere as well that has more info
20:48:41 <nitzmahone> Yep, almost certainly
20:48:49 <nitzmahone> I get pingged about it on average at least once a month
20:49:03 <jborean93> but we think it's some incompatibility with OpenSSH, SChannel or something with certain ciphers
20:49:13 <nitzmahone> Sometimes it'll come up 2-3 times in a day from different folks, then I won't hear about it for months
20:49:36 <nitzmahone> But it only occurs once you've sent ~4M of data over the pipe
20:50:00 <nitzmahone> So a typical Ansible task doesn't trigger it, and since each task gets a new connection, win_copy is effectively the only trigger
20:50:14 <nitzmahone> (plus since we reuse connections on looped tasks, a big loop can hit it)
20:50:25 <daBONDi> ok i fall also over this, but thats a restriction on WinRM side and the http.sys in the backend i was thinking, so i never post it, and use win_url to fetch from http asset server from client side :-)
20:51:13 <jborean93> if you get it we should talk and exchange info
20:51:30 <jborean93> we are trying to find a way to replicate it ourselves so we can identify where the problem actually lies and find a fix
20:53:27 <daBONDi> @nitzmahone ahh now i'm having i clue why a decent Block not work, and after i split it up with include tasks it starts working!
20:54:12 <nitzmahone> :)
20:54:51 <nitzmahone> Once it happens, the control-side will almost always fail with a read timeout
20:54:59 <nitzmahone> (and it doesn't matter how long you set the timeout for)
20:55:22 <daBONDi> yes that what i was to try first, but the ps process on the server looked like it hangs
20:55:56 <jhawkesworth_> having never had it, does winrm recover on next connection or do you have to reboot or restart to get it to clear?
20:56:12 <daBONDi> i killed the hanging ps process and try again
20:56:51 <nitzmahone> The PS process may hang, but WinRM is still good
20:57:10 <daBONDi> but maybe winrm has timeout because, powershell process not return anything (Windows Threading thinks it hangs) and tell WinRM hey wi hanging around, and winrm returns timeout
20:57:13 <nitzmahone> But if you do it a bunch in a row, you can end up DoSing WinRM
20:57:31 <nitzmahone> (default maximum session limits, etc)
20:57:43 <nitzmahone> We used to struggle with this a lot when there were shutdown bugs in Ansible
20:57:59 <nitzmahone> (and when win_copy used to spawn a new WinRM session for each chunk, OMG)
20:59:10 <daBONDi> so prbly there will be no workaround, beside trying to make a pywinrm file uploade multithreaded/multisession stuff :D
20:59:14 <jhawkesworth_> that sounds like the version I made!
20:59:38 <daBONDi> cut the files in chunks, pull it over and reassamble
21:00:04 <daBONDi> let the thread always return something
21:00:04 <jhawkesworth_> sounds like rsync!
21:00:08 <daBONDi> yes
21:01:19 <daBONDi> a Powershell WinRM Rsync clone, lol quite a heavy project :D
21:01:19 <jhawkesworth_> I keep meaning to try this https://syncthing.net/
21:01:57 <jhawkesworth_> there are some .net rsync libraries kicking around but... still sounds like something I couldn't do after work!
21:02:55 <daBONDi> the problem is that you must put the rsync stream into the winrm transport, will be quite a heavy task, if not impossible, and will it be worth it with a nearly timeframe on ssh on windows?
21:03:31 <jhawkesworth_> agreed.  ssh on windows seems like a better way to spend your time
21:03:38 <nitzmahone> Regardless, the wedging thing appears to be at the http.sys level, not WinRM- the exact same traffic works fine over HTTP
21:04:25 * jhawkesworth_ wonders if connecting via ssh would also wind up using http.sys
21:04:35 <daBONDi> don't think so
21:05:14 <jhawkesworth_> you'd think it would be more likely to be based on openssl but I've not looked into it
21:05:43 <jborean93> it's a fork of openssl-portable
21:06:10 <nitzmahone> nope
21:06:15 <daBONDi> @nitzmahone maybe it has something todo with a installed IIS on the client side, and it alters the http.sys params
21:06:19 <nitzmahone> It's a usermode process
21:06:48 <nitzmahone> I don't think so- the one repro I had a few years back IIRC had no IIS installed
21:07:31 <nitzmahone> The workaround for most people has been to use (now message-encrypted) HTTP
21:08:36 <nitzmahone> I've gotten it to work/change in some cases by recompiling Python + OpenSSL + pyopenssl  on the controller as well, but on the one repro I did have, it failed *very* consistently down to the byte
21:09:00 <daBONDi> i don't have a clue how depending http.sys is on aspnetcore http.sys server but there is a minSendBytesPerSecond Value, and on a File copy, the Client would not send anything back to ansible
21:09:07 <daBONDi> https://docs.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.server.httpsys.timeoutmanager?view=aspnetcore-2.1
21:11:04 <nitzmahone> I'd have to decompile it to be sure, but IIRC that class is basically just a wrapper over the http.sys registry stuff. From you experience, it sounds like that setting is global vs. per-listener
21:11:15 <nitzmahone> (as are some other things that wreak havoc like SNI support)
21:11:18 <jhawkesworth_> catch you next time everyone
21:11:31 * jhawkesworth_ afk
21:11:34 <nitzmahone> *most* http.sys config things are pretty well isolated per-listener
21:15:33 <nitzmahone> Anyway, we're over time and I'm hungry- til next week!
21:15:36 <nitzmahone> #endmeeting