19:01:37 #startmeeting ansible core IRC meeting 19:01:37 Meeting started Tue Aug 20 19:01:37 2019 UTC. 19:01:37 This meeting is logged and archived in a public location. 19:01:37 The chair is jillr. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:37 Useful Commands: #action #agreed #halp #info #idea #link #topic. 19:01:37 The meeting name has been set to 'ansible_core_irc_meeting' 19:01:54 hello 19:01:56 #topic https://github.com/ansible/ansible/pull/60593 19:02:00 Hi there 19:02:06 hi Timoses 19:02:49 just started reading your PR, want to kick us off? 19:02:58 Oh, I suppose that's my PR number : ). 19:03:02 Yes sure. 19:03:11 I've written a little summary which I'll just paste: 19:03:24 I use Ansible on a daily basis. Perhaps for the purpose of this discussion it would be best to describe how we use Ansible. 19:03:25 Our goal is to use Ansible as an IAC (Infrastructure as Code) tool for our complete infrastructure (possibly including network devices). This covers several logically encapsulated environments present in various physical locations. 19:03:38 Our approach with Ansible is the following 19:03:56 We have * One git repository for the IAC code regarding our infrastructure containing basically the following directories: 19:04:08 *** * plays - containing themed sub-directories which themselves contain the playbooks 19:04:20 *** * The playbooks use roles from galaxy or a `roles` folder living besides the playbook 19:04:29 *** * The playbooks’ directories contain a `group_vars` directory which is only allowed to use the `all` group (for default values) 19:04:36 *** * inventory - containing inventory files and `group_vars`. Here the variables from the playbooks’ default `all` group_vars can be overwritten as required for each group 19:04:46 The goal with this approach is to have all infrastructure specific setup within the inventory directory while the plays directory contains a somewhat general default of the infrastructure. 19:05:03 (( Half way through, feel free to interrupt if it's too much )) 19:05:18 ** 19:05:20 Our hosts are grouped into the following groups: 19:05:28 * environment 19:05:30 * namespace (e.g. contains all kubernetes hosts from all environments) 19:05:32 * namespace group (useful for clustering, for example `kubernetes-1` group contains all hosts from the first kubernetes cluster; note that this can span multiple environments, meaning this group contains the `kubernetes-1` hosts from all environments even though `kubernetes-1` clusters are distinct between environments) 19:05:34 * function (e.g. `kubernetes-master`) 19:05:39 It now becomes vital within the inventory’s group_vars directory to have fine-grained control over which groups receive which variables. We’ve achieved a somewhat good prioritization via `ansible_group_priority`. However, currently there is no way to specifically target a group of hosts which exists in another group. `recurisve_group_vars` fills this gap. 19:05:42 ****** 19:05:44 ****** 19:06:21 ^ That is basically the reason why I came up with the idea of r_group_vars 19:07:39 We previously started splitting the environments up into inventory subfolders with each their own group_vars directory. 19:08:34 However: 1. Now I can not use variables cross-environment (would have to define them for each environment, even if they are equal) and 2. The problem of "having to split up the inventory due to group_vars limitation" does not stop at the environment level. 19:10:22 it looks like from sivel's comment in the PR we also need to do work to enable the plugins in the first place 19:10:53 whitelist* 19:11:14 ^ Yes, that's what I understood as well. 19:11:28 IMO, it would be great if it were possible to define in which order the plugins are loaded. 19:11:32 ok, the comment said both things so I'll defer to what jtanner says 19:11:59 yeah, explicitly whitelisted. I think all would be used by default right now, rather than opt-in 19:12:10 Since, group_vars could be used in conjunction with r_group_vars: E.g. group_vars defines defaults, and then r_group_vars overwrites variables if the need arises. 19:12:14 the whitelist would prevent vars plugins from auto-loading, which i think sivel is implying is currently the case 19:12:55 https://github.com/bcoca/ansible/commits/user_toggle_vars_loading 19:12:55 I"m not sure I understand the benefits specifically provided by the plugin, but I do think it'd be useful to specify group_vars as a combo of other groups, perhaps by having a group_vars file called `group1:&group2` though I'm not sure how much this overlaps w/ the proposed plugin 19:13:02 Yes, I believe they are automatically loaded, currently (it's the behaviour I am experiencing when putting the plugin into Ansible's plugin/vars library 19:13:28 whitelist would also set order, like it does for callbacks and inventory 19:14:21 @cyberpear: It would be an interesting idea to be able to name group_vars files/folders in that manner. Though, I do see a benefit in r_group_vars as it gives you a nice directory tree which is easily navigable. 19:15:01 Whereas names like "group1:&group2.yml" can easily lead to an unmanageable group_vars directory (imagine lots of those files/folders named in that manner) 19:15:14 Timoses: another way to handle your case is adding group_vars at the playbook dir, for those that are common across inventories 19:16:17 @bcoca, what if I'd need to set variables that are specific for a combination of groups which also need to be available to several playbooks? We do not keep all plays in the same directory (it would be overkill and unmanageable) 19:16:29 cyberpear, Timoses i would argue to use 'constructed' inventory for the case of matching multiple groups to create a var 19:16:32 (be aware that the playbook group_vars trump the inventory group_vars) 19:17:07 which works for the case in which you want same for all cases of a group, but not all possible cases 19:17:09 (@cyperpear that is why we only use the `all` group var in playbook (to set defaults)) 19:17:56 Can you explain what you mean by "constructed". You mean construct "imaginary" groups in the inventory like "atlanta-webserver-1-webserver-app"? 19:18:04 Timoses: ^ to clarify, im not against adding the plugin, but as sivel mentiones i prefer to wait till we have whitelisting 19:18:25 https://docs.ansible.com/ansible/latest/plugins/inventory/constructed.html 19:18:46 ^ Surely, whitelisting would be good. Although, since it uses another directory it might not be harmful or have any negative effects for current users. 19:19:30 it does as it will always run and add lag to current executions 19:19:58 which is a problem even with existing plugin, why i was also looking at allowing user to optimize 'when' they run, 'after inventory', 'at task' or 'both' 19:20:33 before 2.4 it ran at 'both', but to optimize large inventories, i moved to 'at task', but for small inventories this can be noticiable cost so some peopel have asked for 'at inventory' 19:20:43 Ah, yes. I've come across the constructed plugin before. IMO, it's the same as with "group1:&group2" or "atlanta-webserver-1-webserver-app". It becomes very hard to manage the inventory groups and the group_vars folder. 19:20:59 the current complaint will just multiply if we ship more 'always executing' vars plugins by default 19:21:26 Oh yes, I completely understand that argument. 19:21:27 I like the plugin, but would be against it without the whitelist 19:21:58 the bad part, i don't have time to implement the whitelist right now, the link above was to branch i started but then got pulled away due to other priorities 19:22:10 ^ I'd volunteer for that 19:22:34 ^ didnt you also just get pulled into another project that will monopolize your time? 19:22:37 (although I'm not sure if nitzmahone was already looking at it?) 19:22:48 I think that starts for 2.10... 19:22:52 correct 19:22:52 also sivel, but all mentioned have plenty on their plate 19:23:53 Are there any more questions regarding the r_group_vars plugin? 19:24:12 i think someone wanted dir name to be configurable, but other than that, no, its mostly of 'us not being ready' 19:24:40 I'm glad the idea found some resonance then : ). 19:24:53 i also want the mozilla sops one ... but stalled for same reason 19:25:10 correct, shertel's new project starts when 2.10 starts 19:25:41 that feature also fixes 'vars plugins not working in collections' 19:26:00 since it needs a target list to load, unlike current vars plugins which explores 'existing paths', but collections are not part of that 19:26:22 I've already been playing with bcoca's user_toggle_vars_loading branch, I can try to have a PR up later this week. 19:26:31 +10k 19:26:39 shertel: we don't have long before freeze begins, so you have to be careful you don't commit to getting this done without enough time 19:27:21 also need to make sure the behavior for collections is considered/pondered/hashed-out 19:28:08 Yeah, I think we have like 1 week until freeze 19:29:01 Yeah, good point. cutting it close... 19:31:25 Alright, I suppose r_group_vars topic is over? 19:31:27 shertel: is that too close, or "see what we can come up with given the time"? 19:31:44 Pretty sure there's time to get it done but I'm not sure about proper "consideration" for collections since everyone has a lot on their plate and I'd need help with that. 19:32:06 shertel: that code would look a lot like my fix for collection callback loading 19:32:08 nitz has some things he could definitely use help with 19:32:18 review/testing wise 19:32:44 Oh, okay, good. Then this is what I'm working on for the next week. 19:33:06 rad, thanks 19:33:15 * bcoca shoves branch into tardis and sends to 'when i have time' period 19:33:23 hehe 19:33:33 #topic https://github.com/ansible/ansible/issues/60833 19:34:01 resmo: you added this one? 19:34:47 the implicit gather always had an implicit 'always' tag 19:34:51 wonder if he's still around this time of day 19:34:53 closed earlier today though.. 19:35:04 cyberpear: not always, that was added after a long time of being skiipped 19:35:18 the 'always' tag didnt even 'always' exist 19:35:19 from my experience. if you do --skip-tags=always, the gather facts would not run 19:35:28 (I don't think I tried it on 2.8, though) 19:35:52 yes, specifying 'always' will skip the gather .. if not otherwise tagged 19:36:39 but first, the 'always' tag is not that old itself, then applying it to gather_facts happend a while after it was introduced 19:36:59 so 'it always worked this way' is not true to start 19:37:00 it's certainly an (unannounced?) change in behavior 19:37:15 it was a bugfix, didnt seem worthy of announcing otherwise 19:37:36 just like adding the 'always' tag to gather facts was also a bugfix and waas not announced separately, just as a bugfix 19:37:39 when did it regress? 19:38:00 depends on which 'regresssoin' we are talking about 19:38:22 I see, so the intended change there is to have the gather_facts play respect tags applied on the play level in the playbook 19:38:29 yes 19:38:47 previoustlly it was skipping gather facts when any tag (even not applied to play) was specified 19:38:52 so if the play would be skipped because of tags, its gather would also be skipped 19:38:52 tha tis when 'always' got added 19:39:04 yes, that is 'current' behaviour 19:39:15 and was 'previous' behavior at one point, before 'always' was introduced 19:39:21 but likely tasks w/in the play if tagged 'always' would also run despite the play-level tags not being included? (haven't tested it) 19:39:34 cyberpear: not included != skipped 19:39:42 there are serveral corner cases 19:39:53 so we need a way to specify tags for the implicit setup task? 19:40:04 (for the playbook author) 19:40:08 no, i think we need 2 adjustments, 'append always' to existing tags on play 19:40:19 add toggle for 'gather_facts' to ignore tags, default off 19:42:42 https://gist.github.com/bcoca/0278c8fb83a27daf6a89467ded5d3fda <= first change 19:44:22 we might be able to re-order some code in the PlayIterator to wait until we filter the tasks based on tags to know whether any other tasks will execute within the play 19:44:24 just a thought 19:45:02 i thoguht that too, but it seems people want to gather facts even if no tasks in play run 19:45:09 and then if there are tasks that will run, we prepend setup to the front 19:45:34 I'm really not in favor of adding a config toggle, fwiw 19:45:55 then we cannot support all cases 19:46:13 we mostly already recommend that config changes that affect playbook behavior are left alone 19:46:15 my fix should support the main case made in ticket closed, but still keep original ticket working 19:47:18 then the case of 'gathering should run no matter what' ... would be the one that the toggle serves 19:47:19 yeah, I don't know. I might need to think on it more 19:48:06 But I don't think we have enough time to debate it really for the 2.9 release 19:48:08 sivel: also one thing about checking if all tasks can run, tags: `{{ variable}}` is valid ... 19:48:29 we can debate, we probalby cannot implement 19:48:41 well sure, enough time to properly debate and implement 19:48:50 I don't have more time to think on it today 19:49:08 but fwiw, we are already filtering all tasks at the start in PlayIterator 19:49:21 for block in self._play.compile(): 19:49:24 sivel: how do we do that on tags if they are dynamic? 19:49:40 bcoca: honestly that is a good question that I do not have an answer for. 19:49:42 ^ i really wish they weren't 19:50:07 will showed me with his hack `tags: '{{ansible_run_tags}}'` but that is also a 'global/magic var' 19:50:16 so probably 'it works sometimes depending on vars scope' 19:50:21 weeee! 19:50:25 I see the current code, and am not sure how that works, when people do things like tags: "{{ item }}" 19:50:29 * bcoca finds corner to cry in 19:50:45 you can't cry and weeee! at the same time 19:50:49 so are we 'kick the can to a near-future meeting', or 'please reintroduce this topic in 2.10'? 19:50:55 <= multitasker 19:51:31 I'm not even sure. 19:51:41 is this a feature request now? probably needs a ticket explaining the desire, if so 19:51:48 I'd be just as happy not changing behavior again 19:52:10 jtanner: we "fixed" what we believed was a "bug" in functionality, but it changed behavior, and people want old behavior...sometimes... 19:52:16 ^ gist above would fix most cases and still keep original bug 'fixed' 19:52:20 typical 19:52:34 also that behaviour has changed several times 19:52:57 it needs to be one way or the other or have a toggle and a deprecation cycle, imo 19:53:18 i think gist fixes it more 'sanely' as you keep the always, but can also explicitly skip 19:53:38 thoguh it breaks teh case of 'all tasks tagged X' and using --tags Y 19:53:41 so ... idk 19:53:54 i'll take BDFL rulling on this one 19:53:54 before anything changes again, i'd be curious what the behavior in all known versions was (ansible-test-versions ...) 19:54:09 jtanner: he, all over 19:54:24 no one way was dominant? 19:54:32 has been changing almost every other version since 1.9 19:54:40 well that sucks for everyone 19:54:42 depends on the use case 19:56:39 most of the changes didn't conflict with what users filed bugs for until now 19:56:49 mostly that they didnt go 'far enough' 19:56:58 we're running near time, I feel like we're leaning in the direction of "it's complicated, and we don't have time to dig into it, so maybe try again in 2.10"? 19:57:15 porting guide entry just to cover bases for now? 19:57:39 who's writing it? 19:58:07 sivel: you got a few minutes this week? 19:58:12 for 2.8 porting? 19:58:26 jtanner: I'm sure I do 19:58:32 bcoca: or whenever it last changed 19:58:36 2.8 19:58:43 I've gotta bounce for now 19:58:44 2.6 before that 19:59:06 k, that works for me 19:59:27 im not exagerating on 'fact gathering' changing behaviour every other version 19:59:36 no doubt 19:59:46 but it was always in response to bug reports 19:59:48 you know my history with things that "change a lot" 20:00:01 * bcoca finds trash can 20:00:03 #action sivel to write porting guide for 2.8 20:00:12 #endmeeting