19:00:50 <Shrews> #startmeeting Ansible Core Public IRC Meeting
19:00:50 <zodbot> Meeting started Tue Nov 10 19:00:50 2020 UTC.
19:00:50 <zodbot> This meeting is logged and archived in a public location.
19:00:50 <zodbot> The chair is Shrews. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:50 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:50 <zodbot> The meeting name has been set to 'ansible_core_public_irc_meeting'
19:00:56 <Shrews> #chair mattclay
19:00:56 <zodbot> Current chairs: Shrews mattclay
19:00:59 * mattclay waves
19:01:05 <Shrews> #info Agenda: https://github.com/ansible/community/issues/570
19:01:14 <sdoran> o/
19:01:15 <Shrews> who wants to discuss ansible type things today?
19:01:29 <nitzmahone> o/
19:01:49 <bcoca> no, nones, niente, niet
19:01:50 <bcoca> done
19:02:04 <Stas55> Hi
19:02:11 * bcoca waves
19:02:15 <Shrews> o/
19:02:24 <Shrews> #topic Add filter dict
19:02:30 <Shrews> #link https://github.com/ansible/ansible/pull/72413
19:02:41 <Shrews> is this yours Stas55 ?
19:02:49 <bcoca> not really against, since i see a use for map/select/reject combos, but overlaps with dict() function already in jinja
19:02:51 <Stas55> Mine. Thanks
19:03:21 <bcoca> but also question why not push to collection
19:04:33 <Shrews> Yes, the name would overlap. Is there an existing collection where this might fit, as an alternative to core?
19:04:44 <Shrews> maybe c.g?
19:05:23 <mattclay> If it goes into a collection, `community.general` does seem appropriate.
19:05:52 <Stas55> As to me this feels more like a code feature. Regarding the name, the reason I made it dict is because I tried to use dict multiple times (intuitively) and every time was surprised that dict is a function and cannot be used as a filter.
19:06:02 <Stas55> s/code/core/
19:06:20 <bcoca> well, its a jinja2 thing, not really core
19:06:28 <bcoca> dict() function is built in
19:07:03 <sdoran> I have long thought Jinja should have a `dict` filter.
19:07:17 <bcoca> i dont disagree, but there is a conflict in the name
19:07:31 <sivel> I recommended on the issue to call it dict_kv
19:07:44 <sivel> since it's got a different signature as well
19:08:09 <Stas55> initially I used `dict` function for this filter exactly
19:08:19 <sivel> actually seems move like dict_vk
19:08:23 <Stas55> then I realized that there is no point as it cannot take more arguments
19:08:37 <bcoca> but then we also have things like zip that overlap with this functionality
19:08:39 <nitzmahone> Plus while we can usually get away with a filter/func name conflict, it's probably not a good idea to create one purposely
19:08:46 <sivel> dict_vk(value, key) => {key: value}
19:09:11 <bcoca> sivel: more like  value|dict_vk(key)
19:09:28 <sivel> bcoca: yeah, I get that, was more expressing order of params
19:09:31 <bcoca> as stand alone its not really usefule, only in map context
19:09:42 <bcoca> as dict(key,value) already works
19:10:06 <felixfontein> darn, it's already past 19:00 UTC
19:10:26 <Shrews> yeah, the standalone version is rather unuseful
19:10:43 <bcoca> now that i think of it existing dict() + zip  already cover this case
19:11:04 <sivel> the `map` case is complicated though
19:11:05 <bcoca> dict(keys|zip(values))
19:11:08 <Stas55> quite not intuitive though
19:11:17 <bcoca> ^ no need for map, zip takes care of iteration
19:11:24 <sivel> bcoca: true
19:11:40 <bcoca> for 'fun' you can stil lhave map/select/reject feed zipo
19:11:55 <bcoca> dict( stuff|map(...)|zip(values))
19:11:59 <sivel> I think overall I'm -1 on this, and bcoca just pushed me over from -0.5
19:12:04 <nitzmahone> ditto
19:12:34 <bcoca> ^ just came to me .. need to check if we have example in data manip doc
19:13:17 <bcoca> also slice .. which we have example of
19:13:20 <Shrews> Stas55: any thoughts on the suggested alternative from bcoca?
19:13:24 <Stas55> > dict(keys|zip(values)) - also, how would we use it in the middle of a filter pipeline?
19:13:42 <bcoca> https://docs.ansible.com/ansible/latest/user_guide/complex_data_manipulation.html#create-dictionary-from-list
19:13:49 <bcoca> Stas55: above i show putting map in middle
19:14:08 <bcoca> though you can use it in any part
19:14:09 <felixfontein> regarding the name, I wouldn't use dict for this, but something more expressive. mainly to avoid burning the name dict for a filter
19:14:10 <Shrews> though there is a bug in that documenation
19:14:29 <bcoca> Shrews: list is redundant, but not really a bnug
19:14:35 <bcoca> also PR exists
19:14:51 <Shrews> bcoca: it should be `dict(single_list | slice(2))
19:14:53 <Shrews> `
19:15:19 <Shrews> it was the first issue from triage today (a doc PR)
19:16:13 <Stas55> How would you do something like "myservers | map('dict', 'server')" ?
19:17:45 <bcoca> zip_longest and use 'server' as filler
19:17:52 <Stas55> Here is an actual place where we are using it:
19:17:54 <Stas55> postgresql_pg_hba_default: "{{ postgresql_pg_hba_static + (pg_cluster_masters + pg_cluster_slaves) | map('extract', hostvars, 'ansible_host') | product(['/32']) | map('join') | map('sbi_dict', 'address') | map('combine', hba_replication_template) | list }}"
19:18:01 <Stas55> see "sbi_dict"
19:18:30 <sivel> there's also nothing from preventing this from being shipped in a collection, or kept as a custom private filter
19:18:50 <bcoca> well, but that is not for discussion here
19:18:57 <bcoca> here we only discuss if accepting into core
19:19:01 <Shrews> yeah, i'm leaning toward suggesting a collection now
19:19:16 <bcoca> and i think we are pretty much all leaning to same answer
19:19:42 <nitzmahone> There are innumerable special-case utility things for Jinja that come up, and we've historically been inconsistent about which random ones we accept into core and which ones we don't, but with collections, it's probably better to let things like this soak somewhere
19:19:45 <Stas55> pushing to collections + renaming?
19:20:21 <Shrews> ok, i think we have a majority on this one. Stas55, yes, i would suggest proposing to `community.general` or a private collection
19:20:23 <nitzmahone> You've got the primary `community.general` maintainer in the room right now, maybe see what he thinks? ;)
19:21:01 <bcoca> i would move on to next topic, since that detail can be discussed elsewhere and/or durign community meetings
19:21:08 <nitzmahone> +1
19:21:11 <Shrews> #agreed rejecting 72413 for inclusion in core, propose pushing to a collection
19:21:22 <Shrews> #topic hosts: reference parent groups from children (makes copy/pastes easy)
19:21:30 <bcoca> #info zip filters can mostly do same
19:21:49 <Shrews> #chair bcoca
19:21:49 <zodbot> Current chairs: Shrews bcoca mattclay
19:21:50 <nitzmahone> I wasn't clear on what the ask was here
19:22:01 <bcoca> yeah, what does 'reference' mean in this context
19:22:12 <bcoca> and what are you copy/pasting?
19:22:30 <Shrews> Stas55: care to elaborate a bit?
19:22:39 <Stas55> right now we can specify children in definitions of host groups
19:23:10 <Stas55> the problem is that every time I create a new group I have to go back to the parent one and add another children
19:23:26 <Stas55> it would be useful to be able to say "here is your parent"
19:23:31 <Stas55> or parents
19:23:46 <bcoca> that depends on which inventory plugin you are using, easy to accomplish with custom one
19:24:04 <bcoca> each one uses slightly diff way of referencing the relationship
19:24:29 <bcoca> ini, yaml and toml do 'mostly' work as you say but with diff sections depending on where you define
19:25:06 <Stas55> > ini, yaml and toml do 'mostly' work as you say but with diff sections depending on where you define
19:25:14 <Stas55> With the default inventory plugin?
19:25:19 <Stas55> I'm not sure what you mean
19:25:22 <bcoca> all of the above + auto
19:25:32 <bcoca> there is no ONE plugin
19:25:37 <bcoca> + script .. almost forgot
19:25:53 <Stas55> I see. I guess I need to explore this more.
19:26:07 <bcoca> default: ['host_list', 'script', 'auto', 'yaml', 'ini', 'toml']
19:26:14 <bcoca> ^ also for got host_list
19:27:21 <Shrews> ok, so Stas55 can explore a custom inventory plugin a bit to see if it will work
19:27:53 <Shrews> #agreed suggest looking into a custom inventory plugin and research existing plugins
19:28:01 <Shrews> #topic Empty value of --limit should raise an error (or at least be no-op)
19:28:11 <bcoca> hmm, its already a noop
19:28:18 <bcoca> ansible -m debug -i localhost, --limit '' all
19:28:23 <bcoca> 'works'tm
19:29:03 <Stas55> Were were hit badly by this one when running ansible-playbook someplay.yml --limit "$host"
19:29:17 <Stas55> if host is empty it runs for all hosts
19:29:21 <Stas55> at least in ansible 2.9
19:29:22 <bcoca> yes
19:29:29 <bcoca> that is what 'noop' means, limit is ignored
19:29:32 <bcoca> if empty
19:29:57 <Shrews> yeah, i don't see us changing that
19:29:59 <nitzmahone> Sounds like the ask is to distinguish "not specified" from "empty string"
19:30:13 <nitzmahone> but yeah, that'd almost certainly require deprecation period
19:30:15 <Stas55> right
19:30:27 <bcoca> empty string is 'valid limit' for many cases , i can point to many others that depend on that behaviour
19:30:29 <nitzmahone> (and not sure if it's even what we want, I'm sure *someone* is using that as a feature)
19:30:38 <Shrews> ^ this
19:30:41 <bcoca> use -e to avoid 'empty vars' in your scripts if that is a requirement
19:30:49 <bcoca> nitzmahone: many other projects that depend on us
19:31:38 <nitzmahone> and thennnn?
19:31:41 <bcoca> Stas55: if you cannot control at script level, you could always add an 'assert task' at top of play to ensure limit was not empty
19:32:16 <Stas55> no as I know I do this exactly, it's easy do make this mistake though
19:32:20 <bcoca> assert: that: ansible_limit is defined and true
19:32:29 <Stas55> I understand why you would want to keep it this way now
19:32:41 <Stas55> as somebody may depend on it
19:32:46 <bcoca> s/may/does/
19:32:51 <Stas55> I see
19:32:58 <Shrews> #agreed empty `limit` is valid and likely relied on by others. Suggest external validation for empty vars supplied to it.
19:33:12 <Shrews> #topic Playbooks: Inventory variables should override playbook lever variables by default
19:33:20 <Shrews> #link https://github.com/ansible/ansible/issues/63668
19:33:21 <bcoca> #info ansible_limit var is avaiable to play
19:33:59 <bcoca> no, this goes against the established precedence every ansible user already relies on
19:34:27 <Stas55> > every ansible user already relies on
19:34:32 <bcoca> this is why there is a  connection varaibles > play keywords precedence, but need to allow for play still to override, why play vars > inventory vars
19:35:11 <bcoca> example, overriding a task to force using root to login for setting up common user vars: asnbile_user: root, while ansible_user is specified in group_vars/all.yml as the 'general user'
19:35:45 <Stas55> do people actually rely on this? When searching found quite a few messages of being surprised by the default behaviour
19:35:50 <bcoca> ansible_user > remote_user as 'hosts might have specific needs', but having play being able to override ansible_user for specific cases is also needed
19:36:06 <Stas55> every newcomer to ansible does the same mistake also (from my experience)
19:36:13 <sivel> This precendence has been in place for years, and we cannot just change it because a few people have had some misunderstandings
19:36:15 <nitzmahone> guaranteed there are people relying on it
19:36:25 <bcoca> Stas55:  its because not everyone has homogeneous contexts, otherwise it would not be needed
19:37:18 <bcoca> but since people have heterogeneous servers and services .. we need a variety of precedences to approach hosts, if we did as you ask, there would be no way to 'bootsrap the common user' .. if it is set in inventory .. cause you would need all machines to already have it
19:38:22 <Stas55> understood
19:39:03 <Shrews> anything else on this topic then?
19:39:43 <sivel> Don't think so
19:39:48 <Stas55> fine for now
19:40:07 <nitzmahone> Next one is "custom callback plugin"
19:40:09 <Shrews> #info one reason for this long established var precedence is to bootstrap the common user
19:40:17 <Shrews> #topic Allow custom success/failure message (should work with no_log: true)
19:40:26 <bcoca> fail/assert modules allow this also
19:40:40 <bcoca> depending on what you mean by custom success/failure msg
19:40:50 <Stas55> How do this in a loop?
19:40:52 <bcoca> most actions return their own message
19:41:28 <bcoca> we just pass them through and add 'status' and 'host' most of time
19:41:48 <bcoca> Stas55: an example of what you want to do might be needed
19:41:51 <nitzmahone> customizing the output generally is done through a custom default callback plugin- you can just subclass the builtin one and hack up whatever you want
19:42:05 <Stas55> for example when I run a shell command in a loop it would result in the whole command being printed on error (which may include creds sometimes)
19:42:28 <Stas55> I'd want to hide the output but print some part of the item I'm iterating on
19:42:35 <Stas55> like lable=...
19:42:49 <Stas55> but label doesn't work with no_log: true
19:43:05 <bcoca> ignore_errors and use no_log, then use register var in assert/fail task behind it that iterates over results and gives out specific info
19:43:06 <nitzmahone> (another case for looped blocks w/ task+no_log and a following debug task, but I digress ;) )
19:43:28 <nitzmahone> you could do it today with a looped include, but it's a little messy
19:43:29 <bcoca> no need for loop blocks
19:43:41 <bcoca> just 2 tasks
19:43:58 <nitzmahone> yeah, I guess if you want to loop over the output again that could work too
19:44:16 <bcoca> simpler than looped include
19:44:26 <Stas55> it is getting messy when having lots of loops
19:44:56 <Stas55> though this similar to what we do now
19:44:58 <bcoca> can be, but not really designed around uncontrolled output from tasks, mostly rely on actions/modules to take care of the filtering
19:45:03 <nitzmahone> this might be one where register projections could help a bit too
19:45:09 <bcoca> using 'shell' in that way is not somethign we expect
19:45:30 <bcoca> nitzmahone: not really, would still need 2 tasks
19:45:37 <bcoca> and no_log/ignore errors on first
19:45:40 <Stas55> another example is looping over huge dicts
19:46:02 <Stas55> I would want to hide the whole dict but print only a small label
19:46:12 <bcoca> loop_control: label
19:46:14 <bcoca> already exists
19:46:40 <Stas55> the dict appears with a single -v iirc
19:46:40 <sivel> It doesn't sound like we're going to add any generic success_msg/failure_msg for tasks though. So I'm not sure we really need to continue the discussion here for what existing options can be used
19:46:54 <bcoca> -v means you WANT verbose ... we dont hide it in that case
19:46:56 <Stas55> if I add no_log - label doesn't work anymore
19:47:11 <bcoca> we dont have '-v_except_this_case'
19:47:31 <bcoca> no, since no_log is for security, not for minimizing output
19:47:39 <Stas55> is there a good reason to hide labels on no_log?
19:47:41 <bcoca> it will clobber ALL of it since there might be sensitive info
19:47:50 <bcoca> yes, label can expose sensitive data
19:47:58 <nitzmahone> (we've had CVEs on it before)
19:48:06 <bcoca> since its a jinja2 expresion
19:48:10 <nitzmahone> no_log is the bane of our existence
19:48:24 <Shrews> i agree with sivel. we should move on to the next topic. sounds like we can continue discussing in #ansible-devel
19:48:28 <bcoca> well, security is .. no_log is just one of the linchpins of security
19:49:03 <Shrews> #topic Roles: A variable should not be passed to a role unless specified in a “spec” file
19:49:10 <nitzmahone> Last one seems like a misunderstanding of how/why vars work in Ansible
19:49:28 <bcoca> roles inherit vars, they are not 'passed'
19:49:47 <sivel> when roles are parsed, everything from a role gets injected into the play
19:49:48 <bcoca> every object in herits from teh object it is contained in  play > roles > blocks > tasks
19:49:52 <sivel> roles are not self contained
19:49:59 <sivel> so a role basically disappears, and it's contents become direct parts of the play
19:50:08 <sivel> so it's all play scoped really
19:50:39 <nitzmahone> If we made that change, all connections would stop working in vars unless they were also passed explicitly
19:50:48 <nitzmahone> s/vars/roles/
19:50:56 <bcoca> ^ i have pr that did that .. .broke everything
19:50:59 <nitzmahone> (and numerous other things)
19:50:59 <Stas55> the thinking here was to "force" role creators somehow listing all possible variables in one place (e.g. defaults/)
19:51:09 <nitzmahone> That's what role argspec will do
19:51:11 <bcoca> Stas55: we have a role_spec project
19:51:19 <sivel> Stas55: we are working on a new feature called role arg spec, which will be included in 2.11
19:51:20 <bcoca> that does not 'force' but 'encourage'
19:51:25 <nitzmahone> It's not a matter of "force", but document/verify what's spec'd
19:51:28 <nitzmahone> yep
19:52:03 <Shrews> Stas55: you can see some smallish examples of role arg spec here: https://github.com/ansible/ansible/pull/72120
19:52:26 <Stas55> nice!
19:52:43 <Stas55> this would work
19:52:52 <Shrews> great
19:52:56 <nitzmahone> cool, sounds like we can sign you up to betatest 2.11 ;)
19:53:00 <Shrews> feedback welcome on that too
19:53:16 <Shrews> let's move to the last topic then...
19:53:22 <felixfontein> are role args considered for being backported to 2.9/2.10?
19:53:32 <nitzmahone> almost certainly not
19:53:36 <Stas55> We'll definitely have a look at this
19:53:43 <Shrews> #topic update on internal `plugin_utils` discussion
19:53:48 <Shrews> @nitzmahone go!
19:53:58 <nitzmahone> So we discussed this internally last week- to recap the original ask:
19:54:51 <nitzmahone> felixfontein was asking for a place to put shared non-module code in a collection (eg shared between plugins) that wouldn't be subject to the module_utils sanity test requirements (eg, support controller only Python version)
19:55:09 <nitzmahone> He'd proposed `plugins/plugin_utils`, and we discussed internally last week
19:55:22 <nitzmahone> The answer is "not now", since it'd be mainly changes to ansible-test to support it
19:55:41 <nitzmahone> So for now, folks are free to place code wherever they want in the collection
19:55:57 <nitzmahone> If sanity testing is desired, module_utils is the place for now
19:56:14 <nitzmahone> If the need to drift from module Python reqs arises, ignores can be used for now.
19:56:46 <bcoca> specialy since ignores will be compatible with already published versions
19:56:53 <nitzmahone> We may revisit next year, but with the move to AZP in the works, the folks that might make those changes and work through the implications are busy through the end of the year
19:57:25 <nitzmahone> That's it from me
19:57:33 <Shrews> awsome. thx nitzmahone
19:57:46 <Shrews> with the last couple of minutes, let's open up the floor
19:57:48 <Shrews> #topic Open floor
19:58:02 <Shrews> anybody have anything not on the agenda?
19:58:13 * nitzmahone ducks over for WWG
19:58:18 <sivel> just news, ansible-base is being renamed to ansible-core for 2.11
19:58:41 <Shrews> names will be randomly assigned from 2.12 onward
19:58:47 <sivel> a warning has been added to 2.11 if you are running a python older than 3.8, and 2.12 will drop support for 3.7 and older
19:59:48 <felixfontein> Shrews: I was about to suggest renaming it to ansible-basic for 2.12
20:00:00 <felixfontein> and ansible-seed for 2.13
20:00:05 <Shrews> lol
20:00:11 <bcoca> on odd releases we will rename the cli tools randomly
20:00:21 <Shrews> ok, we're at time. thanks for attending everyone!
20:00:26 <Shrews> #endmeeting