15:01:26 <Shrews> #startmeeting Ansible Core Public IRC Meeting
15:01:26 <zodbot> Meeting started Thu Jan 21 15:01:26 2021 UTC.
15:01:26 <zodbot> This meeting is logged and archived in a public location.
15:01:26 <zodbot> The chair is Shrews. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:26 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:01:26 <zodbot> The meeting name has been set to 'ansible_core_public_irc_meeting'
15:01:40 <Shrews> #chair mkrizek
15:01:40 <zodbot> Current chairs: Shrews mkrizek
15:01:58 <felixfontein> o/
15:02:04 <mkrizek> \o
15:02:09 <Shrews> #info agenda at https://github.com/ansible/community/issues/584
15:02:21 * agaffney waves
15:02:36 <Shrews> looks like a light agenda
15:02:48 <Shrews> #topic https://github.com/ansible/ansible/pull/72419
15:03:01 <Shrews> bobo175: bcoca: you around?
15:03:10 <bobo175> I am
15:03:27 <bcoca> im bsquare
15:03:38 <Shrews> looks like bcoca manifested after the meeting and sayeth: "bobo175: plans, yes, timeline ... no clue"
15:04:24 <sdoran> \o
15:04:27 <bobo175> oh, missed that.  any idea if timeline will be in 2.9/2.10?  We're stuck at 2.9.9 til then
15:04:46 <bobo175> if you merge into devel, I'm happy to test (my bug and the others)
15:04:58 <agaffney> generally only bugfixes will be migrated to older stable versions
15:05:00 * Duck o/
15:05:02 <bcoca> not in a state that can be merged
15:05:24 <agaffney> bobo175: why are you stuck at 2.9.9, and how does that PR address that?
15:05:54 <bobo175> My bug is: https://github.com/ansible/ansible/issues/72776
15:06:09 <bobo175> and it first appears in 2.9.10 and  persists into 2.10
15:06:27 <bobo175> not sure bcoca's PR fixes it, but he suggests it may
15:07:00 <bcoca> well, connection supports templates, but they are now relative to delegated host not the delegator, which was givinging you incorrect info unless both hosts used same data
15:07:16 <bcoca> which does occur in homogeneous environments, but not in heterogeneous ones
15:07:47 <bobo175> right, I would argue that's a bug.
15:07:51 <bcoca> one unintended consequence of the changes is the fact that hostvars[x][y] is not auto templated, you normally got values cause the 'y' var in 'current inventory host' was tempalted
15:08:07 <bcoca> yes and fixing that bug, created this new one
15:09:41 <bobo175> are you saying the behavior in 2.9.9 and before was incorrect?
15:09:59 <sivel> There was behavior in 2.9.9 that was incorrect, but it's not fully correct now either
15:10:58 <bobo175> ok, now I'm confused.  What is incorrect about 2.9.9?  If I can understand that, maybe I can workaround this by modifying my playbook
15:11:02 <bcoca> yes, consider hosta and hostb, hosta uses an expression based on host property to select the user, its all fine while hosta and hostb have the same value to the property, but if hostb is different and you delegate to hostb .. you are using hosta info, which is incorrect
15:11:24 <bcoca> when hosta delegates to hostb
15:11:42 <bobo175> oh, but if I use hostvars[hostb][blah], that should work
15:12:10 <bcoca> yes, but the problem is that blah gets templated ONLY for inventory_hostname and NOT delegated host
15:12:16 <bcoca> ^ that is what my pr is trying to fix
15:12:41 <bobo175> ah, got it, makes sense
15:12:44 <bcoca> sivel:  per your comments, vars should not be tempalted at task level, but on consumption as they can refer to diff hosts per loop item
15:12:47 <bobo175> ty for explaining
15:12:53 <bcoca> np, its complicated issue
15:13:15 <sivel> bcoca: post_validate does template per host in TE
15:13:37 <sivel> as you are in the context of a single host
15:13:41 <bcoca> per inventory_hostname, not per remote host
15:13:56 <bcoca> exactly, why we defer templating for connection/become/shell to the lop
15:13:59 <bcoca> the loop
15:14:01 <sivel> Yes, but all templating of a task definition should be done as inventory_hostname
15:14:04 <bcoca> where we have the 'actual remote host'
15:14:25 <bcoca> sivel: for many things, yes, not for connection related info and since vars are reused in those ...
15:14:47 <sivel> I don't think we should make that edge case a thing, I think it should be the *full* task definition
15:15:08 <sivel> we're only going to continue creating a steady stream of bugs by trying to special case things
15:15:11 <bcoca> https://github.com/ansible/ansible/issues/66533 <= example
15:15:48 <bcoca> no, its not special case, its keeping connection part tied to the connection, otherwise we need to remove the featuers that change anything in task per loop item
15:16:38 <bcoca> and restrict any/all parameters to inventory_hostname at task def and any connection related items become immutable for that task
15:17:10 <bcoca> ^ and i would be fine if that is how we initially designed it, but we didnt, we just didnt do a good job of partitioning what is 'per task' and 'per loop iteration'
15:17:21 <bcoca> trying to fix that now
15:17:59 <bcoca> http://paste.debian.net/1182073/ < why im working on this
15:18:34 <bcoca> and once working ... hopefully document clearly so people know WHAT changes WHEN and how things depend on each other in the task cycle
15:21:34 <bcoca> bobo175: as a workaround you can use set_fact to create a static copy per host fo the expression you want to use for connection, but what yo have 'should work' if i get to fix it correctly
15:21:34 <bobo175> Thanks bcoca - I will try that.
15:21:35 <bobo175> Thanks all for the discussion.  I understand about 5% of it, but it helps
15:21:40 <bcoca> dont worry, im knee deep in it and get 6% of it
15:21:45 <bobo175> lol
15:22:32 <bobo175> I could just blame the Tower folks... they're the cause of all this!
15:22:59 <Shrews> no. always blame bcoca. that is the standard
15:23:03 <bcoca> nope, not even close, most of how delegation, loop and connections work was not really designed as a whoel but added onto and 'grew' .. now im trying to apply a logic and have clear demarcations on who it should work
15:23:21 <bcoca> Shrews: standard changed once you got hired, now `git blame` always returns Shrews
15:23:26 <Shrews> dang
15:23:40 <bcoca> s/who/how/
15:23:55 <Shrews> i guess we can open the floor now
15:23:59 <Shrews> #topic Open floor
15:24:09 <Duck> quack
15:24:23 <Duck> I wanted to discuss hash_merge since that's a feature I use
15:24:29 <Shrews> oh boy
15:24:32 <Shrews> :)
15:24:35 <Duck> I heard all was not decided yet
15:24:42 <Duck> well, without trolls
15:25:17 <Duck> I read somewhere I don't recall peopel saying some made magic around roles etc and that caused problems
15:25:26 <Duck> but my only usage is in the inventory
15:25:44 <sdoran> 🙉 🙈
15:25:47 <agaffney> the problem is that it's a global feature that can wreak havoc on third party code that isn't designed for it
15:25:59 <Duck> first I like to use structured variables
15:26:25 <Duck> and then you can use groups and priorities to "add" or override to dicts
15:26:34 <Duck> and there's no way you can do all that with combine
15:26:59 <Duck> agaffney: how so?
15:27:24 <agaffney> I don't have a specific example handy
15:27:31 <agaffney> I know I've seen one in the past
15:27:37 <Duck> as said I only use it in the inventory, then I pass variables to include_role and the roles, which are the only possibel external stuff are not impacted
15:27:49 <felixfontein> for inventory, one could simply create an inventory plugin which works like the default group_host_vars one, but uses hash merge
15:28:11 <bcoca> rolea and roleb use rolec (which updates firewall with a set of rules passed in a var)
15:28:11 <Duck> so if there's a way to limit the scope and keep it that would help much
15:28:16 <agaffney> you can't only use it in the inventory. you may only *intentionally* be using it in the inventory, but it works everywhere, which can lead to unexpected results
15:29:00 <sdoran> The main issue is there is no clear way to know what the final merged variables are without running the play. While it can be useful, more often it is surprising and confusing.
15:29:07 <bcoca> rolea was written with merge in mind, roleb was not, you cannot use both roles toghether since they work against each other and give diff results depending on the value of hash_behaviour
15:29:31 <bcoca> in one case you get less rules than expected in the other you get many dupes
15:30:02 <Duck> well, I've not seen these problem. if you could move the logic to the inventory loading that would be better than leaving peopel using without any solution than rewrite all their work
15:30:05 * bcoca needs to write macro with that example
15:30:40 <Duck> bcoca:sorry, I would need an example, I don't understand how this happen
15:30:49 <bcoca> ^ i just gave it
15:30:56 <bcoca> also in prev meetings
15:30:58 <agaffney> Duck: when you say inventory, you're just referring to group_vars, right?
15:31:06 <Duck> agaffney:yes
15:31:09 <agaffney> if so, that could be handled in a custom vars plugin
15:31:30 <Duck> and host_vars too of course
15:31:32 <agaffney> https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/vars/host_group_vars.py
15:31:35 <bcoca> again, things discussed in prev meetings, we should try not to keep reshashing same, why i suggested we come up with set of options and vote on em
15:31:40 <Duck> but nowhere else I expect a magic merge
15:31:41 <agaffney> host_vars and group_vars loading it handled by the above vars plugin
15:31:49 <agaffney> *is handled
15:31:56 <Duck> bcoca: well, sorry I was not here
15:32:01 <bcoca> logs online
15:32:04 <bcoca> why we post em
15:32:27 <Duck> I was told my use case was not discussed
15:32:49 <bcoca> Duck: not trying to blame you, we shoudl have organized the vote on clear options going forward already instead of opening up the same discussion again
15:32:53 <Duck> anyway, a custom vars plugin would be enough for me
15:32:56 <bcoca> Duck: it was
15:33:42 <Duck> ok, then I'll go check what was decided
15:33:57 <bcoca> Duck: possibly but it does not cover all cases, also PR for include_vars to have 'merge' option, yet you'll find things like var declarations in plays or other sources (facts, etc) will now overwrite and even though you thought you ONLY used it in inventory you'll see you relied on it in more locations
15:34:25 <bcoca> Duck: nothing after the discussions
15:34:57 <Duck> I think there's a very different perscpective in this example: you CAN use combine
15:35:14 <Duck> with the group and priorities I just cannot create a logic to emulate it
15:35:21 <bcoca> yes, you can, the issue is that most people did not
15:35:46 <bcoca> well, you can , but becomes a bit complex and requires heavy use of default() and or varsname lookup
15:36:06 <Duck> then I guess you're stuck with the feature or you're going to make people very unhappy and deal with it :-/
15:36:56 <Duck> bcoca:and how do you load vars from a specific group?
15:37:16 <bcoca> kindof where we are at, why there were several options to mitigate each scenario i was trying to get a vote on
15:37:37 <bcoca> Duck: you namespace them and then have a var that aggregates them via combine
15:38:04 <bcoca> tomcat_app1 tomcat_app2  tomcat: lookup(varnames, 'tomcat_*' .....
15:38:25 <Duck> that means rewriting the whole inventory
15:38:29 <Duck> and no support for structures
15:39:12 <bcoca> i didnt say it was trivial, just possible
15:39:21 <Duck> I'd like to avoid rewriting all projects I'm involved in
15:39:23 <sivel> yes, we've stated a few times that the alternative to the deprecated hash_behaviour=merge will require refactoring
15:39:35 <sivel> There is not a no-refactor option in the current path
15:40:05 <Duck> of course I can tell my boss it's your fault, as a fellow Red Hatter, but he's not gonna like the joke :-)
15:41:12 <Duck> there is other options for people, stay on the last version supporting it, move away from Ansible, or use the fork by the guy you know
15:42:03 <agaffney> is there any reason to remove the feature other than it being a bad idea in general to use it and users just not realizing that?
15:42:28 <felixfontein> agaffney: nothing that was completely convincing, I think
15:42:46 <felixfontein> Duck: why do you need to refactor if you only use merge for inventory, but nowhere else?
15:43:10 <felixfontein> (an inventory plugin would solve the inventory merge problem)
15:43:17 <Duck> felixfontein: in the solution proposed by bcoca in case the feature goes away totally
15:43:41 <Duck> now if there's a new vars plugin, or an option, then that's fine for my uses cases
15:43:56 <sivel> 1) it's a bad idea, 2) it makes your content dependent on a specific ansible.cfg config, 3) It produces unexpected results for many users, 4) it doesn't work everywhere, and continually breaks as new code is added or modified
15:44:07 <bcoca> vars plugin only goes so far since most of the merging happens AFTER the results of several invocations of the plugin
15:44:13 <sivel> Those are 4 I can come spit out without much thought
15:44:32 <Duck> sivel: in my case the inventory vars and ansible.cfg are all together, like most people I think
15:44:45 <bcoca> sivel: when has it broken ? i really have not seen any issues with it in a long time
15:45:02 <bcoca> i agree with 1-3, but 4 is not something i knew about
15:45:11 <agaffney> sivel: most of those are just UX issues when the feature is enabled
15:45:27 <sivel> Right now I'm looking at various places in the code where we are just doing .update and in others dealing with the same vars are using combine_vars
15:45:36 <sivel> so while people may not necessarily know it's broken, it is
15:46:52 <bcoca> afaik we dont update in generation, only in things like returning resutls and such, whcih is not really part of the feature
15:47:03 <bcoca> varsmanager (where the merging happens) uses combine_vars
15:47:16 <sivel> .update and combine_vars are used inconsisetently throughout the codebase
15:47:37 <sivel> so some use may adhere to merge, others wont, leading to inconsistent behavior
15:47:44 <bcoca> on the user variables? can you point me to where?
15:48:06 <sivel> one example, we use `.update` in `get_vars` on some play objects, and `combine_vars` in others
15:48:47 <sivel> meaning inheritance sometimes merges sometimes doesn't
15:49:05 <sivel> based on different play objects
15:49:29 <bcoca> no, we use update in vars manager very sparingly, iirc just on vars plugin loading (internally to the vars plugin function) and when we use namespaced facts
15:49:57 <sivel> I don't mean VarsManager.get_vars, I mean like Play.get_vars, Task.get_vars, Role.get_vars
15:50:15 <sivel> some use .update, some use combine_vars
15:50:53 <sivel> so even in regular inheritance of vars, via `get_vars` which often use `self._parent.get_vars` we merge some, not others
15:51:25 <felixfontein> Duck: nobody stops you of creating a vars plugin. so the only thing that might go away completely is support in ansible-core. but at least for inventory host_vars/group_vars files, you can still have support that way.
15:51:31 <bcoca> base and role use combine, rest shoudl use update to keep scope
15:51:33 <shertel> besides roles, should those even be using combine_vars?
15:51:52 <bcoca> not really inconsistent (i woudl argue roles are messed up .. but that is deeper discussion)
15:52:11 <sivel> In any case, as I'm saying, it's inconsistent, and we cannot easily prevent further inconsistency from happening
15:52:32 <sivel> as people introduce .update or combine_vars elsewhere in the code
15:53:02 <sivel> anyway, we should eventually vote on this, because having these discussions weekly isn't useful
15:53:26 <felixfontein> definitely.
15:54:07 <sivel> I introduced the change, and stand behind my decision, so I am inclined to leave it as is, so I won't put it up for vote.  Someone else can do so, or we maintain the current decision
15:55:48 <Duck> felixfontein: thanks for adding to my todolist. I would have prefered this to be handled by the project sicne that's not a whim but something many people use
15:56:54 <Shrews> I'm willing to chair anyone who would like to start the vote, but I'm not entirely certain what the voting options should be.
15:57:37 <Shrews> But I agree these weekly discussions seem very "unfulfilling"
15:58:36 <Duck> well I guess it's a sign that's bothering people more than other things Ansible might lack
15:58:54 <Duck> but that was not my goal to bother, I just did not have time to follow
15:59:16 <Duck> and it's a tad late for me, so I do not usually come to the meeting
15:59:17 <bcoca> from previous meeting: a) rescind teh deprecation for now, review when new options become available, b) keep deprecation, but create docs on migration path and possibly helper tools
15:59:40 <bcoca> c) create varsmanager plugins
15:59:59 <Shrews> #chair bcoca
15:59:59 <zodbot> Current chairs: Shrews bcoca mkrizek
16:00:26 <sivel> I will vote if put to a vote fwiw
16:00:43 <bcoca> d) recind but add big warning and examples of bad behaviour
16:00:51 <sivel> we do need 5 core committers for a quorum
16:02:34 <Shrews> i see sdoran, shertel, bcoca, sivel, myself
16:02:41 <shertel> \o
16:02:43 <sdoran> Ahoy.
16:02:51 <Shrews> nitzmahone might be around with opinions, too
16:03:12 <shertel> I don't know what the potential new options would be for a)
16:03:32 * relrod is here too but doesn't really have enough info to vote meaningfully
16:03:42 * mkrizek is here too
16:04:37 <sivel> well, we seem to have 4 decent enough options
16:04:50 <Shrews> #startvote How should we deal with hash_behavior going forward? (A, B, C, D)
16:05:03 <sivel> I think (a) is basically at some arbitrary point in the future we may find another option
16:05:09 <Shrews> No idea if that's the correct meetbot syntax
16:05:10 <shertel> question about a, because it seems like the only full replacement is c
16:05:30 <sivel> (a) is a yet to be dicsovered option
16:06:07 <shertel> a == punt for later
16:06:22 <sivel> is there a specific #vote thing for the bot? Or how do you want this done?
16:06:30 <Duck> bcoca said there were other uses cases, I guess some interesting, so A would mean to think more about them
16:06:35 <Shrews> something like: #vote A
16:06:45 <sdoran> It's usually just a manual tally I thought.
16:06:47 <sivel> #vote B
16:06:53 <sdoran> That'd be fancy if the bot had a vote counter.
16:07:06 <sivel> get your votes in :)
16:07:12 <shertel> Duck, vars manager plugins handle a variety of cases - I'm not sure which is would not
16:07:17 <sivel> otherwise my vote still wins ;)
16:07:20 <shertel> s/is/it
16:07:33 <Shrews> please vote within the next few minutes as we are already over time
16:07:37 <shertel> #vote C
16:08:13 <sivel> also fwiw, we'd have to still prioritize any large feature
16:08:27 <Duck> shertel: it would have been nice to have a listing on wome wiki page instead of multiple IRC meeting logs mixed with other subjects, but eh
16:08:27 <sivel> which means a replacement comes at some arbitrary point in the future
16:08:53 <bcoca> sorry, just got back was reading logs for other options .. just realized c) was a joke i made (bad one) ... please NO varsmanager plugin !
16:09:06 <shertel> lol, ok, wasn't sure that was still in jest
16:09:10 <sdoran> #vote B
16:09:14 <shertel> #vote A
16:09:17 <mkrizek> #vote B
16:09:20 <bcoca> shertel: you just want to see what that would look like!
16:09:27 <shertel> I do
16:09:37 <Duck> bcoca:???
16:09:53 <Shrews> b and d seem like they should be combined
16:10:20 <bcoca> you mean a and d
16:10:21 <Shrews> oh, no, nm
16:10:34 <bcoca> #vote a + d
16:11:04 <Shrews> yeah, those two
16:11:09 <Duck> yes, a+doc makes sense
16:11:40 <bcoca> sorry, was parsing meeting logs to find discussed 'option's so started pasting here w/o reading them too deeply
16:11:49 <sivel> a) 2, b) 3, c) 1, d) 1
16:11:59 <Duck> still don't get why C was a joke but A is fine too I guess, just feels like that's going to hit us in the future anyway
16:12:06 <relrod> Yeah I think A+D is my vote as well, but I don't have much of a horse in the race.
16:12:24 <agaffney> Duck: because VarsManager is already a mostly unmaintainable mess and adding more complexity to it is a terrible idea
16:12:25 <bcoca> Duck:  .. you would need to know varsmanager and then you would be in a nice white padded room with a swell vest
16:12:41 <agaffney> also, adding it to VarsManager is different than a vars plugin
16:12:48 <Shrews> #vote A + D
16:12:53 <sivel> everyone loved it when I deprecated it, but caves under pressure ;)
16:13:05 <Shrews> i was not here when that happened  :-P
16:13:38 <Shrews> i think maybe it *should* be deprecated still, but under different circumstances
16:13:41 <sivel> anyway, whatever is decided, someone needs to action it now
16:13:41 <shertel> I don't remember the deprecation at all, just the original author's reddit post that probably came up around the same time
16:13:44 <bcoca> sivel: im want to deprecate it, but i want to remove many things that im not able to (debug, roles, dependencies, varsmanager, etc)
16:14:32 <shertel> Shrews: yeah, I also think it should probably be removed in the future
16:14:34 <Shrews> i will end the vote in a few minutes. any cores who have not voted yet?
16:14:34 <bcoca> having a user base means you don't get to remove 'include' w/o a 5 year migration to new functions
16:14:52 * bcoca needs to fix his pr to deprecate include:
16:18:00 <sivel> meh, we're also not a democracy, so while the community can influence us, we make final decisions
16:18:46 <agaffney> the community doesn't have to maintain this shit :)
16:20:13 <Shrews> ok, i will end the vote now
16:20:25 <Shrews> #endvote
16:20:49 <Shrews> unsure if that does anything with our meetbot (/me shakes fist angrily at AI)
16:21:00 <bcoca> well, depends on the 'section' of community .. a lot of the community does maintain this
16:21:11 <bcoca> agaffney: you being one of em
16:21:51 <agaffney> I will consider myself a maintainer vs. community only when it's convenient for me :)
16:22:22 <sivel> agaffney: I like it! :)
16:22:41 <Shrews> i think:  a) 4   b)  3   c)  1  d) 3
16:22:43 <agaffney> my only dog in this race is around supporting users in #ansible when they run into weird issues when using 'hash_behavior = merge'
16:22:44 <Shrews> is that right?
16:23:22 <shertel> mine should have been a + d, sorry
16:23:31 <agaffney> and I can mostly just tell them it's a bad idea to use it and that they have to deal with the consequences if they do
16:23:43 <Shrews> a) 4   b)  3   c)  1  d) 4
16:24:06 <Duck> the C was reattributed
16:24:57 <Duck> that was shertel I think
16:25:11 <Shrews> looks like A + D anyway.  we will take this under consideration I suppose and now have something to point to
16:25:21 <Shrews> i will end the meeting now since we are WAAAAY over time
16:25:25 <Shrews> thanks all
16:25:34 <Shrews> #endmeeting