15:07:59 #startmeeting ansible core irc meeting 15:07:59 Meeting started Thu Dec 20 15:07:59 2018 UTC. 15:07:59 This meeting is logged and archived in a public location. 15:07:59 The chair is bcoca. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:07:59 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:07:59 The meeting name has been set to 'ansible_core_irc_meeting' 15:08:18 #topic https://github.com/ansible/ansible/pull/46561 15:08:22 ericzolf? 15:10:05 4th meeting w/o anyone to advocate for this so i'm going to remove from agenda, feel free to readd 15:10:14 #topic https://github.com/ansible/ansible/issues/49992 15:10:21 @akasurde ? 15:10:37 lucastheisen ? 15:10:43 here 15:11:18 so, one reason set_fact and include-vars are hardcoded here is to avoid the situation with ansible_Facts (which all modules can provide) in dealing with injection and security issues 15:11:33 i dont know that we want to open that door 15:12:57 so, i need a way to provide merged dict's from vars files that actually get interpolated. my team follows the _best practice_ approach where you `somesecret: {{vault_somesecret}}`, so without the interpolation, were stuck 15:13:42 wouldnt setting hash_behaviour= merge do this? 15:13:53 or am i missing something? 15:14:34 also , i dont know what _best_practice_ is here .. i personally think the term is totally incorrect and leads to people trying to use square peg in round hole 15:15:07 kindof... but not really. first that is global, and given all the warnings about that behavior it seems like a bad idea to switch the global behavior... second, the issue i found as to that behavior seemed to indicate it gets ignored for some cases (like `include_vars`)... 15:15:48 This is the best practice i speak of: https://docs.ansible.com/ansible/latest/user_guide/playbooks_best_practices.html?highlight=best%20practices#always-name-tasks 15:16:10 sorry, wrong section, this one: https://docs.ansible.com/ansible/latest/user_guide/playbooks_best_practices.html?highlight=best%20practices#variables-and-vaults 15:16:10 include_vars follows it at task level, but you might be refering to the 'directory' option which does things differently internally 15:17:24 ^ as i said 'personally' i think that should not be part of our docs in that manner, i would rephrase to 'one way to do this' .... but it is actually not a good way, much less 'best' for many contexts 15:18:08 hmm... well thats unfortunate cause i point to the _official_ documentation for all of our teams conventions (and everyone else i talk to) 15:18:21 specially since group_vars/name/vault force you into several things, a) you always need to pass in vault secrets even if you dont use them 15:18:43 ... lets leave at that, you are not here to see my lists 15:18:59 sure..., but i _AM_ interested in your reasons. 15:19:10 (you being someone with WAY more experience on this) 15:19:30 but that is 'my' view, there are many others to consider 15:19:43 and i cannot really tell you what to do w/o a lot more information about your context 15:20:31 that is why im more in favor to show 'you can do X or you can do Y' since in the end you have a set of policies, internal worklow and actual deployed setup that you need to conform to and they are not always the same as the person writing the scenario 15:21:06 having to write a custom plugin to import vars already tells me ... there is an easier way to set this up, that should NOT be a need 15:21:41 i would also argue that you shoudl favor 'inline vaults' vs vaulted files 15:22:12 https://docs.ansible.com/ansible/latest/user_guide/playbooks_vault.html#single-encrypted-variable 15:22:21 but again, this is just my opinion 15:22:38 others here will surely have different ones 15:22:44 guess the downside to inline vaults is the difficulty in creating them (ansible-vault encrypt + copy/paste) 15:23:08 yeah, i had very diff design, but that is what was done in the end 15:23:18 > there is an easier way to set this up, that should NOT be a need 15:23:38 is your suggestion to just use global `hash_merge` behavior? 15:23:44 idk, i really have very little info of what are you trying to do? what are your limitations? 15:24:08 w/o knowing much i would say ... a diff way of organizing your varaibles might be a better solution 15:24:40 these are not easy questions to answer (why i hate the 'best practices') w/o intimate knowledge of the problems you face 15:25:28 well, should we discuss here? or should i write up more in the github issue to try to more fully outline my problem? 15:25:42 (i dont wanna take up ALL the time for just my issue) 15:26:02 there is nothing else on agenda ... and we seem to be only ones at meeting .... 15:26:29 which is also why i dont feel comfortable giving you one answer ... we normally have a set of commiters present for any decision 15:27:46 ok, so our situation is that we are managing a kubernetes cluster. to that end we decided that the _hosts_ concept, best mapped to kubernetes namespaces... 15:28:01 then we decided we wanted to keep our kubernetes secrets dry... 15:28:17 and we have 4 environments dev/qa/beta/prod 15:28:40 what kind of secrets? 15:28:49 k8s secrets. 15:28:58 (or configmaps when not sensitive) 15:29:01 why not use lookup or vars_plugin? 15:29:14 i assume those secrets are also in etcd? 15:29:35 because k8s is not our final authority on those values... our ansible project in git is... 15:29:37 its push only 15:29:43 no editing in place 15:29:44 ah 15:30:25 do you always need the secrets or just for certain plays? 15:30:46 so our varibles can be common accross project, or env, or specific to the proj and environment... for example projects foo and bar would result in foo, dev, foo-dev, bar.... 15:31:11 > do you always need the secrets or just for certain plays? 15:31:18 we dont make a distinction... 15:31:29 i dont understand 15:31:32 (we dont isolate plays to need/not need secrets) 15:32:11 i linked to this: https://github.com/lucastheisen/ansible-merge-vars/ 15:32:17 its a bit of an _example_ 15:32:17 would it make sense to do so? 15:32:56 probably not. 15:33:08 actually, the more you describe this .. the more i think you could use inventory or vars_plugin to resolve your issue 15:33:09 because our plays can work at many levels... 15:33:30 you have 'constructed/geenrated' inventory plugins that allow you to compose vars, which coudl depend on existing groups/environment/inventory source used 15:34:06 you could also restructure your vars, but you seem to want to keep them as close as possible to the acual k8 map 15:34:15 yes 15:34:58 i suggest you try using the compose plugin (or if already using another dynamic plugin that supports composing) to create the vars instead of assembling them indirectly via templates in group vars 15:35:03 essentially the inventory is about kubernetes infra... and the _config_ is about the stuff injected as secrets/configmaps 15:35:56 .. this is where terminology bites us ... 15:36:16 many people consider inventory 'config' or group-vars 'config' ... me i only see 'ansible.cfg' as 'config' 15:36:33 just think those are really arbitrary distinctions, in the end its all just 'data' 15:36:43 dont let the labeling restrict you 15:37:20 some consider host/group vars also as part of 'inventory' ... its closely related, but not really same thing, yet it all comes back to labeling vs actually finding 'what works in your case' 15:38:44 just note, all this complexity is in 'our heads' vs what ansible does in the end, it looks at a 'host' and a'task' .. everything else is already reduced to a property of one of those, the trick is setting up our abstractions to meet the 'simplified end' 15:39:00 my biggest problem is just that the behavior for hash's is replace, not merge... if it were merge, i would be fine... but i know i read github issues mentioning why merge is not the default and why it is a bad idea... 15:39:02 ^ bit grick 15:39:37 so i am trying to isolate the merging to a controlled subset of vars... 15:39:40 hence the plugin 15:39:45 biggest issue with that setting, your plays/roles will not be shareable, but if you ensure it is consistent in your environment, it should not be a big issue 15:40:24 and i agree, its a big hammer, i've been toying with a !merge directive for vars .. but its not easy to implement 15:41:10 scoped merge would solve a lot of problems (ive seen many others on StackOverflow and ansible gitub issues talking about it) 15:41:27 bcoca, pardon me for interrupting - does vars plugin 'merge' or 'replace' vars ? 15:42:33 akasurde: the plugin itself can do whatever it wants, the results from it follow hash_behaviour 15:44:31 bcoca, oK 15:44:42 it also seems like hash_behavior is not respected by `include_vars` https://github.com/ansible/ansible/issues/10446 15:45:12 https://github.com/ansible/ansible/pull/14450 <- fixed long ago, that was bug 15:46:25 guess i missed that (reading from bottom up, my error) 15:46:48 ah, had not read bottom .. but we dont get notified for closed issues ... not sure if those are same issue or not 15:47:25 the fix was using 'combine_vars' which is THE thing that implements HASH_BEHAVIOUR 15:47:43 yeah, i know, i use it in my workaround :) 15:48:05 there is one thing in i.nclude_vars that does NOT follow this and that is the 'directory' feature, buyt that is internal to the plubing 15:48:19 * bcoca hits cat to get off keyboard 15:49:24 so... we use the dir feature... (because `value: {{vault_value}}`...) if we got rid of that then include_vars would work... 15:50:04 so i would have to change my global setting for hash_behavior to merge, then switch vars to use inline vault values... 15:50:11 is that where we are? 15:51:01 i think that is simpler solution 15:51:14 but again, you are the one with the full context, i only have a glimpse 15:51:19 and, you dont want to expose the `fact_setter` option i proposed for security reasons? 15:51:46 yes, that woudl allow 'random module injection' which is alreayd a problem we have with ansible_facts (though we restrict these a lot in templating) 15:52:06 if we allow 'any fact' to be set, then we would be in an issue with not being able to trust ANY variable 15:52:25 as a remote compromised system can hijack module results and then inject vars into your play 15:52:57 the whoel reason i created the INJECT_FACTS_AS_VARS setting 15:53:00 technically, i did just write a module that _does_ this... (by setting self._task.action = include_vars: https://github.com/lucastheisen/ansible-merge-vars/blob/master/plugins/action/merge_vars.py#L22) 15:53:09 ^ which currently defaults to the unsafe 'on' 15:53:26 but i hear you. 15:53:49 well, actions are a bit diff (they are local), but what you suggest would affect al modules 15:54:12 would there be another way to scope it down? 15:54:34 and don't count on that working, some want to introduce 'namespaces' and then reserve the 'ansible' one so you cannot do things like that 15:54:52 not that i can think of, but might be a limitation of my imagination 15:55:18 yeah, i know... the _task part clearly tells me im breaking the rules... but didn't have another way... 15:55:30 sometimes you need to break the rules 15:55:49 is there an issue proposing a _scoped_ merge? 15:55:51 in the end, it needs to work for YOU 15:56:08 lucastheisen: see the INJECT_FACTS_AS_VARS commit, i do exactly that 15:56:15 (sure and it does _now_ but any update to ansible itself my cause my stuff to just stop) 15:56:26 https://github.com/ansible/proposals/issues/127 <= also 15:56:46 lucastheisen: and that is the risk of 'breaking the rules' 15:58:34 so thats at the proposal stage, so years out (if ever) correct? 15:58:48 also, breaks backwards compat so think 3.0 15:59:01 yeah, figured. sounds thorough. 15:59:08 but now you have chance to add input to it :-) 15:59:47 the only part i _currently_ care about is: set: varname= value= scope= update=overwrite|merge|aggregate 15:59:56 variable scoping is something that wasn't really design in ansible it more 'happened organically' 16:00:11 its been my BIGGEST frustration 16:00:19 (along with many others) 16:00:25 (along with many other people) 16:00:42 i know, i've been trying to tame that for a while, but i'm at the limit of what i think can be done with current structures and backwards compatiblity 16:00:59 i can fix it for you, while breaking it for a large part of the user base 16:01:33 understood... hence my action plugin that brought up this conversation. 16:02:22 also understood, and we are always open on ways to make this better/easier, but we have to filter those against other considerations, in this case, it opens a huge hole on tempating vars 16:02:58 ok, so final word... ansible core is not gonna change to allow my to use my action to set facts with interpolation. so my options are, continue using my module with the understanding that im using a private whose behavior may change at any time, or switch to global merge behavior and use inline vault values. that sum it up? 16:03:03 lucastheisen: im going to end the meeting now, but we can continue discussing this off the record, this channel or ansible-devel seem appropriate 16:04:00 lucastheisen: not final word, but i would vote -1 , my vote alone is not sufficient to do/undo it so i would bring this back up in a meeting with actual quorum (tempted to cancel meetings till next year since i dont think atttendance will improve till then) 16:04:18 #endmeeting