15:01:26 #startmeeting Ansible Core Public IRC Meeting 15:01:26 Meeting started Thu Jan 21 15:01:26 2021 UTC. 15:01:26 This meeting is logged and archived in a public location. 15:01:26 The chair is Shrews. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:26 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:01:26 The meeting name has been set to 'ansible_core_public_irc_meeting' 15:01:40 #chair mkrizek 15:01:40 Current chairs: Shrews mkrizek 15:01:58 o/ 15:02:04 \o 15:02:09 #info agenda at https://github.com/ansible/community/issues/584 15:02:21 * agaffney waves 15:02:36 looks like a light agenda 15:02:48 #topic https://github.com/ansible/ansible/pull/72419 15:03:01 bobo175: bcoca: you around? 15:03:10 I am 15:03:27 im bsquare 15:03:38 looks like bcoca manifested after the meeting and sayeth: "bobo175: plans, yes, timeline ... no clue" 15:04:24 \o 15:04:27 oh, missed that.  any idea if timeline will be in 2.9/2.10?  We're stuck at 2.9.9 til then 15:04:46 if you merge into devel, I'm happy to test (my bug and the others) 15:04:58 generally only bugfixes will be migrated to older stable versions 15:05:00 * Duck o/ 15:05:02 not in a state that can be merged 15:05:24 bobo175: why are you stuck at 2.9.9, and how does that PR address that? 15:05:54 My bug is: https://github.com/ansible/ansible/issues/72776 15:06:09 and it first appears in 2.9.10 and  persists into 2.10 15:06:27 not sure bcoca's PR fixes it, but he suggests it may 15:07:00 well, connection supports templates, but they are now relative to delegated host not the delegator, which was givinging you incorrect info unless both hosts used same data 15:07:16 which does occur in homogeneous environments, but not in heterogeneous ones 15:07:47 right, I would argue that's a bug. 15:07:51 one unintended consequence of the changes is the fact that hostvars[x][y] is not auto templated, you normally got values cause the 'y' var in 'current inventory host' was tempalted 15:08:07 yes and fixing that bug, created this new one 15:09:41 are you saying the behavior in 2.9.9 and before was incorrect? 15:09:59 There was behavior in 2.9.9 that was incorrect, but it's not fully correct now either 15:10:58 ok, now I'm confused.  What is incorrect about 2.9.9?  If I can understand that, maybe I can workaround this by modifying my playbook 15:11:02 yes, consider hosta and hostb, hosta uses an expression based on host property to select the user, its all fine while hosta and hostb have the same value to the property, but if hostb is different and you delegate to hostb .. you are using hosta info, which is incorrect 15:11:24 when hosta delegates to hostb 15:11:42 oh, but if I use hostvars[hostb][blah], that should work 15:12:10 yes, but the problem is that blah gets templated ONLY for inventory_hostname and NOT delegated host 15:12:16 ^ that is what my pr is trying to fix 15:12:41 ah, got it, makes sense 15:12:44 sivel: per your comments, vars should not be tempalted at task level, but on consumption as they can refer to diff hosts per loop item 15:12:47 ty for explaining 15:12:53 np, its complicated issue 15:13:15 bcoca: post_validate does template per host in TE 15:13:37 as you are in the context of a single host 15:13:41 per inventory_hostname, not per remote host 15:13:56 exactly, why we defer templating for connection/become/shell to the lop 15:13:59 the loop 15:14:01 Yes, but all templating of a task definition should be done as inventory_hostname 15:14:04 where we have the 'actual remote host' 15:14:25 sivel: for many things, yes, not for connection related info and since vars are reused in those ... 15:14:47 I don't think we should make that edge case a thing, I think it should be the *full* task definition 15:15:08 we're only going to continue creating a steady stream of bugs by trying to special case things 15:15:11 https://github.com/ansible/ansible/issues/66533 <= example 15:15:48 no, its not special case, its keeping connection part tied to the connection, otherwise we need to remove the featuers that change anything in task per loop item 15:16:38 and restrict any/all parameters to inventory_hostname at task def and any connection related items become immutable for that task 15:17:10 ^ and i would be fine if that is how we initially designed it, but we didnt, we just didnt do a good job of partitioning what is 'per task' and 'per loop iteration' 15:17:21 trying to fix that now 15:17:59 http://paste.debian.net/1182073/ < why im working on this 15:18:34 and once working ... hopefully document clearly so people know WHAT changes WHEN and how things depend on each other in the task cycle 15:21:34 bobo175: as a workaround you can use set_fact to create a static copy per host fo the expression you want to use for connection, but what yo have 'should work' if i get to fix it correctly 15:21:34 Thanks bcoca - I will try that. 15:21:35 Thanks all for the discussion.  I understand about 5% of it, but it helps 15:21:40 dont worry, im knee deep in it and get 6% of it 15:21:45 lol 15:22:32 I could just blame the Tower folks... they're the cause of all this! 15:22:59 no. always blame bcoca. that is the standard 15:23:03 nope, not even close, most of how delegation, loop and connections work was not really designed as a whoel but added onto and 'grew' .. now im trying to apply a logic and have clear demarcations on who it should work 15:23:21 Shrews: standard changed once you got hired, now `git blame` always returns Shrews 15:23:26 dang 15:23:40 s/who/how/ 15:23:55 i guess we can open the floor now 15:23:59 #topic Open floor 15:24:09 quack 15:24:23 I wanted to discuss hash_merge since that's a feature I use 15:24:29 oh boy 15:24:32 :) 15:24:35 I heard all was not decided yet 15:24:42 well, without trolls 15:25:17 I read somewhere I don't recall peopel saying some made magic around roles etc and that caused problems 15:25:26 but my only usage is in the inventory 15:25:44 🙉 🙈 15:25:47 the problem is that it's a global feature that can wreak havoc on third party code that isn't designed for it 15:25:59 first I like to use structured variables 15:26:25 and then you can use groups and priorities to "add" or override to dicts 15:26:34 and there's no way you can do all that with combine 15:26:59 agaffney: how so? 15:27:24 I don't have a specific example handy 15:27:31 I know I've seen one in the past 15:27:37 as said I only use it in the inventory, then I pass variables to include_role and the roles, which are the only possibel external stuff are not impacted 15:27:49 for inventory, one could simply create an inventory plugin which works like the default group_host_vars one, but uses hash merge 15:28:11 rolea and roleb use rolec (which updates firewall with a set of rules passed in a var) 15:28:11 so if there's a way to limit the scope and keep it that would help much 15:28:16 you can't only use it in the inventory. you may only *intentionally* be using it in the inventory, but it works everywhere, which can lead to unexpected results 15:29:00 The main issue is there is no clear way to know what the final merged variables are without running the play. While it can be useful, more often it is surprising and confusing. 15:29:07 rolea was written with merge in mind, roleb was not, you cannot use both roles toghether since they work against each other and give diff results depending on the value of hash_behaviour 15:29:31 in one case you get less rules than expected in the other you get many dupes 15:30:02 well, I've not seen these problem. if you could move the logic to the inventory loading that would be better than leaving peopel using without any solution than rewrite all their work 15:30:05 * bcoca needs to write macro with that example 15:30:40 bcoca:sorry, I would need an example, I don't understand how this happen 15:30:49 ^ i just gave it 15:30:56 also in prev meetings 15:30:58 Duck: when you say inventory, you're just referring to group_vars, right? 15:31:06 agaffney:yes 15:31:09 if so, that could be handled in a custom vars plugin 15:31:30 and host_vars too of course 15:31:32 https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/vars/host_group_vars.py 15:31:35 again, things discussed in prev meetings, we should try not to keep reshashing same, why i suggested we come up with set of options and vote on em 15:31:40 but nowhere else I expect a magic merge 15:31:41 host_vars and group_vars loading it handled by the above vars plugin 15:31:49 *is handled 15:31:56 bcoca: well, sorry I was not here 15:32:01 logs online 15:32:04 why we post em 15:32:27 I was told my use case was not discussed 15:32:49 Duck: not trying to blame you, we shoudl have organized the vote on clear options going forward already instead of opening up the same discussion again 15:32:53 anyway, a custom vars plugin would be enough for me 15:32:56 Duck: it was 15:33:42 ok, then I'll go check what was decided 15:33:57 Duck: possibly but it does not cover all cases, also PR for include_vars to have 'merge' option, yet you'll find things like var declarations in plays or other sources (facts, etc) will now overwrite and even though you thought you ONLY used it in inventory you'll see you relied on it in more locations 15:34:25 Duck: nothing after the discussions 15:34:57 I think there's a very different perscpective in this example: you CAN use combine 15:35:14 with the group and priorities I just cannot create a logic to emulate it 15:35:21 yes, you can, the issue is that most people did not 15:35:46 well, you can , but becomes a bit complex and requires heavy use of default() and or varsname lookup 15:36:06 then I guess you're stuck with the feature or you're going to make people very unhappy and deal with it :-/ 15:36:56 bcoca:and how do you load vars from a specific group? 15:37:16 kindof where we are at, why there were several options to mitigate each scenario i was trying to get a vote on 15:37:37 Duck: you namespace them and then have a var that aggregates them via combine 15:38:04 tomcat_app1 tomcat_app2 tomcat: lookup(varnames, 'tomcat_*' ..... 15:38:25 that means rewriting the whole inventory 15:38:29 and no support for structures 15:39:12 i didnt say it was trivial, just possible 15:39:21 I'd like to avoid rewriting all projects I'm involved in 15:39:23 yes, we've stated a few times that the alternative to the deprecated hash_behaviour=merge will require refactoring 15:39:35 There is not a no-refactor option in the current path 15:40:05 of course I can tell my boss it's your fault, as a fellow Red Hatter, but he's not gonna like the joke :-) 15:41:12 there is other options for people, stay on the last version supporting it, move away from Ansible, or use the fork by the guy you know 15:42:03 is there any reason to remove the feature other than it being a bad idea in general to use it and users just not realizing that? 15:42:28 agaffney: nothing that was completely convincing, I think 15:42:46 Duck: why do you need to refactor if you only use merge for inventory, but nowhere else? 15:43:10 (an inventory plugin would solve the inventory merge problem) 15:43:17 felixfontein: in the solution proposed by bcoca in case the feature goes away totally 15:43:41 now if there's a new vars plugin, or an option, then that's fine for my uses cases 15:43:56 1) it's a bad idea, 2) it makes your content dependent on a specific ansible.cfg config, 3) It produces unexpected results for many users, 4) it doesn't work everywhere, and continually breaks as new code is added or modified 15:44:07 vars plugin only goes so far since most of the merging happens AFTER the results of several invocations of the plugin 15:44:13 Those are 4 I can come spit out without much thought 15:44:32 sivel: in my case the inventory vars and ansible.cfg are all together, like most people I think 15:44:45 sivel: when has it broken ? i really have not seen any issues with it in a long time 15:45:02 i agree with 1-3, but 4 is not something i knew about 15:45:11 sivel: most of those are just UX issues when the feature is enabled 15:45:27 Right now I'm looking at various places in the code where we are just doing .update and in others dealing with the same vars are using combine_vars 15:45:36 so while people may not necessarily know it's broken, it is 15:46:52 afaik we dont update in generation, only in things like returning resutls and such, whcih is not really part of the feature 15:47:03 varsmanager (where the merging happens) uses combine_vars 15:47:16 .update and combine_vars are used inconsisetently throughout the codebase 15:47:37 so some use may adhere to merge, others wont, leading to inconsistent behavior 15:47:44 on the user variables? can you point me to where? 15:48:06 one example, we use `.update` in `get_vars` on some play objects, and `combine_vars` in others 15:48:47 meaning inheritance sometimes merges sometimes doesn't 15:49:05 based on different play objects 15:49:29 no, we use update in vars manager very sparingly, iirc just on vars plugin loading (internally to the vars plugin function) and when we use namespaced facts 15:49:57 I don't mean VarsManager.get_vars, I mean like Play.get_vars, Task.get_vars, Role.get_vars 15:50:15 some use .update, some use combine_vars 15:50:53 so even in regular inheritance of vars, via `get_vars` which often use `self._parent.get_vars` we merge some, not others 15:51:25 Duck: nobody stops you of creating a vars plugin. so the only thing that might go away completely is support in ansible-core. but at least for inventory host_vars/group_vars files, you can still have support that way. 15:51:31 base and role use combine, rest shoudl use update to keep scope 15:51:33 besides roles, should those even be using combine_vars? 15:51:52 not really inconsistent (i woudl argue roles are messed up .. but that is deeper discussion) 15:52:11 In any case, as I'm saying, it's inconsistent, and we cannot easily prevent further inconsistency from happening 15:52:32 as people introduce .update or combine_vars elsewhere in the code 15:53:02 anyway, we should eventually vote on this, because having these discussions weekly isn't useful 15:53:26 definitely. 15:54:07 I introduced the change, and stand behind my decision, so I am inclined to leave it as is, so I won't put it up for vote. Someone else can do so, or we maintain the current decision 15:55:48 felixfontein: thanks for adding to my todolist. I would have prefered this to be handled by the project sicne that's not a whim but something many people use 15:56:54 I'm willing to chair anyone who would like to start the vote, but I'm not entirely certain what the voting options should be. 15:57:37 But I agree these weekly discussions seem very "unfulfilling" 15:58:36 well I guess it's a sign that's bothering people more than other things Ansible might lack 15:58:54 but that was not my goal to bother, I just did not have time to follow 15:59:16 and it's a tad late for me, so I do not usually come to the meeting 15:59:17 from previous meeting: a) rescind teh deprecation for now, review when new options become available, b) keep deprecation, but create docs on migration path and possibly helper tools 15:59:40 c) create varsmanager plugins 15:59:59 #chair bcoca 15:59:59 Current chairs: Shrews bcoca mkrizek 16:00:26 I will vote if put to a vote fwiw 16:00:43 d) recind but add big warning and examples of bad behaviour 16:00:51 we do need 5 core committers for a quorum 16:02:34 i see sdoran, shertel, bcoca, sivel, myself 16:02:41 \o 16:02:43 Ahoy. 16:02:51 nitzmahone might be around with opinions, too 16:03:12 I don't know what the potential new options would be for a) 16:03:32 * relrod is here too but doesn't really have enough info to vote meaningfully 16:03:42 * mkrizek is here too 16:04:37 well, we seem to have 4 decent enough options 16:04:50 #startvote How should we deal with hash_behavior going forward? (A, B, C, D) 16:05:03 I think (a) is basically at some arbitrary point in the future we may find another option 16:05:09 No idea if that's the correct meetbot syntax 16:05:10 question about a, because it seems like the only full replacement is c 16:05:30 (a) is a yet to be dicsovered option 16:06:07 a == punt for later 16:06:22 is there a specific #vote thing for the bot? Or how do you want this done? 16:06:30 bcoca said there were other uses cases, I guess some interesting, so A would mean to think more about them 16:06:35 something like: #vote A 16:06:45 It's usually just a manual tally I thought. 16:06:47 #vote B 16:06:53 That'd be fancy if the bot had a vote counter. 16:07:06 get your votes in :) 16:07:12 Duck, vars manager plugins handle a variety of cases - I'm not sure which is would not 16:07:17 otherwise my vote still wins ;) 16:07:20 s/is/it 16:07:33 please vote within the next few minutes as we are already over time 16:07:37 #vote C 16:08:13 also fwiw, we'd have to still prioritize any large feature 16:08:27 shertel: it would have been nice to have a listing on wome wiki page instead of multiple IRC meeting logs mixed with other subjects, but eh 16:08:27 which means a replacement comes at some arbitrary point in the future 16:08:53 sorry, just got back was reading logs for other options .. just realized c) was a joke i made (bad one) ... please NO varsmanager plugin ! 16:09:06 lol, ok, wasn't sure that was still in jest 16:09:10 #vote B 16:09:14 #vote A 16:09:17 #vote B 16:09:20 shertel: you just want to see what that would look like! 16:09:27 I do 16:09:37 bcoca:??? 16:09:53 b and d seem like they should be combined 16:10:20 you mean a and d 16:10:21 oh, no, nm 16:10:34 #vote a + d 16:11:04 yeah, those two 16:11:09 yes, a+doc makes sense 16:11:40 sorry, was parsing meeting logs to find discussed 'option's so started pasting here w/o reading them too deeply 16:11:49 a) 2, b) 3, c) 1, d) 1 16:11:59 still don't get why C was a joke but A is fine too I guess, just feels like that's going to hit us in the future anyway 16:12:06 Yeah I think A+D is my vote as well, but I don't have much of a horse in the race. 16:12:24 Duck: because VarsManager is already a mostly unmaintainable mess and adding more complexity to it is a terrible idea 16:12:25 Duck: .. you would need to know varsmanager and then you would be in a nice white padded room with a swell vest 16:12:41 also, adding it to VarsManager is different than a vars plugin 16:12:48 #vote A + D 16:12:53 everyone loved it when I deprecated it, but caves under pressure ;) 16:13:05 i was not here when that happened :-P 16:13:38 i think maybe it *should* be deprecated still, but under different circumstances 16:13:41 anyway, whatever is decided, someone needs to action it now 16:13:41 I don't remember the deprecation at all, just the original author's reddit post that probably came up around the same time 16:13:44 sivel: im want to deprecate it, but i want to remove many things that im not able to (debug, roles, dependencies, varsmanager, etc) 16:14:32 Shrews: yeah, I also think it should probably be removed in the future 16:14:34 i will end the vote in a few minutes. any cores who have not voted yet? 16:14:34 having a user base means you don't get to remove 'include' w/o a 5 year migration to new functions 16:14:52 * bcoca needs to fix his pr to deprecate include: 16:18:00 meh, we're also not a democracy, so while the community can influence us, we make final decisions 16:18:46 the community doesn't have to maintain this shit :) 16:20:13 ok, i will end the vote now 16:20:25 #endvote 16:20:49 unsure if that does anything with our meetbot (/me shakes fist angrily at AI) 16:21:00 well, depends on the 'section' of community .. a lot of the community does maintain this 16:21:11 agaffney: you being one of em 16:21:51 I will consider myself a maintainer vs. community only when it's convenient for me :) 16:22:22 agaffney: I like it! :) 16:22:41 i think: a) 4 b) 3 c) 1 d) 3 16:22:43 my only dog in this race is around supporting users in #ansible when they run into weird issues when using 'hash_behavior = merge' 16:22:44 is that right? 16:23:22 mine should have been a + d, sorry 16:23:31 and I can mostly just tell them it's a bad idea to use it and that they have to deal with the consequences if they do 16:23:43 a) 4 b) 3 c) 1 d) 4 16:24:06 the C was reattributed 16:24:57 that was shertel I think 16:25:11 looks like A + D anyway. we will take this under consideration I suppose and now have something to point to 16:25:21 i will end the meeting now since we are WAAAAY over time 16:25:25 thanks all 16:25:34 #endmeeting