19:05:46 <bcoca> #startmeeting ansible core irc public meeting
19:05:46 <zodbot> Meeting started Tue May 19 19:05:46 2020 UTC.
19:05:46 <zodbot> This meeting is logged and archived in a public location.
19:05:46 <zodbot> The chair is bcoca. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:05:46 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:05:46 <zodbot> The meeting name has been set to 'ansible_core_irc_public_meeting'
19:06:02 <sdoran> \o
19:06:07 <sdoran> So many meetings.
19:06:18 <bcoca> #topic https://github.com/ansible/ansible/pull/69291
19:06:31 <bcoca> ^ actualy, this should go to testing meeting
19:07:08 <felixfontein> no
19:07:18 <felixfontein> the original question maybe yes, but the new one no
19:07:25 <bcoca> ?
19:07:53 <felixfontein> the main reference would be https://github.com/ansible/community/wiki/Version-numbers-for-documentation-deprecation-removal-in-the-collection-world and regarding the PR https://github.com/ansible/ansible/pull/69291#issuecomment-631014434
19:07:58 <bcoca> please keep sep topics, i really canot follow
19:08:07 <felixfontein> let me explain
19:08:29 <felixfontein> I was too busy with things and RL to fix the topic
19:08:56 <bcoca> also ... going to put a time limit, 14mins before meeting adding topic does not give time people to review
19:09:53 <felixfontein> we have the problem that there are cross-collection (I treat ansible-base as ansible.builtin) includes for doc fragments and module_util code, wihch both have influcence on options for modules and plugins
19:10:00 <felixfontein> bcoca: I've added the `(including a discussion on how version numbers should be handled that are potentially cross-collection; see also https://github.com/ansible/community/wiki/Version-numbers-for-documentation-deprecation-removal-in-the-collection-world)` part yesterday
19:11:05 <bcoca> so puting on first to push to testing meeting
19:11:19 <bcoca> #topic a discussion on how version numbers should be handled that are potentially cross-collection;
19:11:38 <bcoca> @sivel  @nitzmahone i know you guys have been discussing versions for a while
19:11:46 <felixfontein> should I sum up what is written in the wiki article, or do you want to read for yourself?
19:11:53 <bcoca> @abadger1999 i know community team also has been looking into this
19:12:09 <felixfontein> this came up during a discussion with mattclay and abadger1999 on saturday
19:12:58 <bcoca> fyi on fragments, they are overridable and stackable, so you dont really need to add to main fragment, you can add additional ones or just mask those things your module doesnt implement
19:13:04 <bcoca> ^ mostly my 2c on that
19:13:29 <bcoca> and adding to fragment CANNOT override module
19:13:39 <nitzmahone> The routing deprecation/removal stuff assumes dates, not version numbers
19:14:06 <bcoca> nitzmahone: which is a mixed bag since most collections seem to be going for version numbers
19:14:28 <sivel> people trying to fix square peg in a roung hole
19:14:32 <felixfontein> you need to know a collection's release date for deprecation dates to be useful
19:14:41 <bcoca> idc either way, just against having split standards
19:14:54 <nitzmahone> You don't really, it's just "some release after this date"
19:15:00 <felixfontein> also, there needs to be a way to indicate in which versions exactly a feature is and in which not
19:15:25 <felixfontein> if you say "removed in a release after 2020-06-01", does it mean that 1.0.7 released on 2020-06-15 no longer has this feature? no, because it is a patch release.
19:15:39 <sivel> if it impacts ansible/ansible if you draw a line around the repo without looking at anything external, that is my concern
19:15:42 <nitzmahone> I could make it either/both, but dates make more sense for Ansible's own things, at least for deprecation.
19:15:49 <felixfontein> dates for versioning don't work well with semver, and for some reason collections use semver
19:15:53 <jborean93> Yes based on the certified lifecycle I’m not sure how viable depredations per date really make sense
19:16:14 <jborean93> So far I don’t think it’s going to be possible to use dates and not versions for the Windows collections
19:16:19 <abadger1999> community is also going to encourage people to use version numbers rather than dates.  dates are too user-unfriendly.
19:16:25 <bcoca> well, galaxy/AH uses semver, collections can use whatever they want .. but if they want to be published to those, it requries semver
19:16:36 <abadger1999> (But I don't think that's what felixfontein is discussing ;-)
19:16:57 <felixfontein> no :) but I've already asked earlier today what happened to deprecation by date - because I never heard about it again since quite some time now
19:17:24 <nitzmahone> As far as the code goes, it doesn't really matter, but if we want to support both, they should be separate (and probably mutually exclusive) so external tools can consume that
19:17:47 <nitzmahone> (and understand what they're consuming)
19:18:40 * nitzmahone stuffing lunch in face, at-mention if need something from me
19:18:53 <felixfontein> right now deprecation by date is only available in PRs - for modules in nitz's routing PR, for options, features, aliases in my deprecate-by-date PR
19:19:04 <felixfontein> while versions already work, and they work with 2.9 too
19:19:48 <felixfontein> anyway. back to topic.
19:20:44 <felixfontein> right now version_added is a bit of a mess, since for example `files` doc fragment says `version_added: 2.3` for option `attributes`, which is right now shown as-is when you for example look at the module docs of community.crypto.openssl_privatekey
19:21:37 <bcoca> why migration removed most of those , makes no sense for collections to use ansible version
19:21:49 <bcoca> but on display, if they exists, there i no current distinction
19:22:01 <bcoca> aside from the plugin name having FQCN format
19:22:12 <felixfontein> the good thing about docs fragments is that Ansible knows where they come from. I tried to solve this problem as follows: put the collection source in front of the version, i.e. `version_added: 2.3` from ansible-base becomes `version_added: ansible.builtin:2.3`, and `version_added: 1.0.0` for a option in community.general becomes `version_added: community.general:1.0.0`
19:22:26 <abadger1999> felixfontein: I like your write-up of the version problems.  One thing I noticed was missing is that people also need to have code to enable the new options.  Files is kinda on the fence in that regard: If the only place you need to use the file args is set_file_attributes_if_different() then it works.  But if you are using them to set mode and ownership some other way then it doesn't.
19:22:33 <abadger1999> (I've seen a few of each)
19:23:12 <felixfontein> (and ansible-doc does remove the tag for the current collection, so `1.0.0` is shown in community.general, while the ansible-base version is shown as `ansible.builtin:2.3`)
19:23:19 <abadger1999> aws was really good since I never saw module using their argspec + doc_fragment that did not use their connection function.
19:23:21 <felixfontein> abadger1999: true.
19:23:42 <felixfontein> still `files` does this a lot better than `url` :)
19:23:47 <abadger1999> Yes :-)
19:24:24 <abadger1999> Just need to mention that hte code piece is both construct argspec *and* do something useful with the new arguments without requiring any changes to the module.
19:24:31 <bcoca> well, most of those using url subsets never bothred implementing the whole set, while using file .. kind of forces you to implement most
19:24:48 <felixfontein> exactly.
19:24:57 <felixfontein> yes.
19:25:05 <jborean93> That’s a good point. If there’s a doc fragment in a collection that is relied upon another collection then you need a way to determine whether the version added for for the dep or current collection
19:25:47 <bcoca> jborean93: fragments are overridable, so plugin can always supercede version_added (done by design, specifically for this case)
19:26:03 <felixfontein> but next to people having to be more careful in the NWO, there's also the issue of removed_in_version and deprecated_aliases.version (and removed_at_date and deprecated_aliases.date later on?). these are set by code in module_utils, and we don't know which module_util from which collection set them
19:26:24 <jborean93> bcoca: but you shouldn’t have to know what the fragment implements to override it
19:26:28 <felixfontein> bcoca: plugins can only override options they know of
19:26:36 <abadger1999> It is not for this case.... because then you would have to be able to read the future.
19:26:45 <bcoca> yes, but you should not use the fragment unless you know what it provides
19:27:14 <felixfontein> essentially it would be best to *not* use fragments or module_utils from other collections
19:27:15 <jborean93> it defeats the point in using a shared fragment to implement common functions
19:27:15 <abadger1999> In two years time, urls will add a foobar parameter.  So I have to modify my module in my collection to override and get rid of that parameter now.
19:27:19 <felixfontein> but it's too late for that :)
19:27:26 <jborean93> abadger: exactly
19:27:40 <bcoca> jborean93:  no, the only problem is when you do updates to fragments that do not have corresponding function
19:27:49 <cyberpear> I'd be concerned about treating them copylibs, though; you'll get divergence and fixes will only be in some places
19:27:52 <bcoca> at which point you should really use separate fragments
19:28:39 <jborean93> bcoca: so what happens when the underlying module util adds a new feature, now your module has a divergance with what the util actually provides
19:28:42 <abadger1999> felixfontein: it would probably be good to say that the "solution" to example 2 is to create a second version of the doc_fragment and suppotring code and leave the original to never change.
19:29:09 <felixfontein> "rot in peace"
19:29:15 <bcoca> jborean93:  depends on how that is added and if it is forced option to all consumers of the module_utils (then i would add to fragment)
19:29:32 <abadger1999> So when url wants to add the foobar feature, it has to ship both doc_fragments/urls.py and doc_fragments/urls2.py
19:29:35 <bcoca> mostly you are making the case for plugins in collections to minimize external deps since they cannot rely on them
19:29:58 <jborean93> what if that's not what you want
19:30:03 <felixfontein> bcoca: from experience, minimizing external deps is *ALWAYS* good :)
19:30:07 <bcoca> agian, depends on how foobar is implemented and who/what is consuming urls fragment
19:30:14 <jborean93> like say I want to allow community.windows to use the PowerShell URLs equivalent
19:30:27 <cyberpear> felixfontein: but not at the cost of carrying a fork of those external deps
19:30:27 <jborean93> instead of implementing it all over again which was the original problem when it didn't have the urls util
19:30:49 <bcoca> jborean93: not sure what you are looking for in this case, doc_fragments dont have introspection into the plugin they are used in
19:30:57 <felixfontein> cyberpear: yep, that's usually not what you want to do. but sometimes the alternatives are worse.
19:31:14 <bcoca> mostly they canno, since you can include them appart from the code (file in template.py which has only docs, not the code)
19:31:45 <felixfontein> jborean93: so if a user hits the wrong combination of ansible-base (which has the feature in url) and community.windows (which doesn't know about this feature yet), the user get surprises (and sanity tests fail)
19:31:58 <bcoca> jborean93: also note that collections have 'versioned' dependencies, so even if you change in next release, they might still require the prior one (ensuring compatiblity)
19:32:30 <bcoca> this is teh eternal problem when the 'lib you depend on changes'
19:32:34 <jborean93> this is an argument against your don't use doc fragments
19:32:37 <felixfontein> bcoca: you'd force collections to only promise support for the current release of ansible, since they have no idea what the next version will add to the fragments they use
19:32:45 <bcoca> jborean93: they are optional, you are free not to use em
19:32:53 <felixfontein> bcoca: and then users don't want to upgrade to the next ansible since their favorite collection doesn't support it (yet)
19:33:02 <jborean93> the alternative is a duplicate of code with something that needs to be maintained in parallel
19:33:11 <bcoca> felixfontein:  there is already proposal to add 'requires_ansilbe' versions for collections, but that is irrespective of this
19:33:39 <felixfontein> bcoca: yes, but it doesn't solve the underlying problem either
19:33:45 <bcoca> jborean93:  that is one, not the one i favor, but yes, authors can do that
19:34:36 <felixfontein> ok. I think we all agree that cross-collection dependencies can cause various problems.
19:34:39 <bcoca> felixfontein:  about versionoing? no, doc_fragments  just reflect it, the cause is both in core code and docs structure itself with existing version_added
19:34:51 * cyberpear would rather instead of "you're free not to use them", we find a way to ease the issues?
19:34:53 <bcoca> of course they can, but that is not limited to module_utils, nor fragments
19:35:28 <felixfontein> I'm right now mainly interested in making it easier for the user to know what's going on
19:35:47 <felixfontein> if I look at the docs for a module, and I find `version_added: 2.3` even though the collection version is `1.0.0`, I'll be confused
19:36:03 <bcoca> agreed, why i removed version_Added at migrtion
19:36:09 <felixfontein> or if a module says to me `This feature is deprecated and will be removed in version 2.10`, while I'm using version `3.0.0` of the collection, I'll beconfused
19:36:11 <bcoca> missed the retuns though ...
19:36:27 <felixfontein> bcoca: you need to remove it from ansible-base as well, because doc framents in ansible-base still have it
19:36:31 <bcoca> felixfontein: but that is outside of fragments, that is the basic version handling
19:36:46 <bcoca> but it is congruent in base, for the plugins it hosts
19:37:49 <felixfontein> what I'm proposing is that versions are either auto-tagged with the collection name (for version_added), or have to be manually tagged (for removed_in_version and deprecated_aliases.version and module.deprecate(version=)), so that users can see from the error messages WHICH version is meant
19:38:26 <bcoca> manually tagging would 'just work'tm but its a pain, fine with adding collection namespace to version_added in ansible-doc
19:38:32 <felixfontein> (and docs can display it accordingly: option attributes was added by ansible-base 2.3; option foo was added by version 1.0.0 of community.general)
19:38:37 <bcoca> that wont fix mismatching docs from code though
19:38:54 <felixfontein> version_added is the most frequently used version number, and that can luckily be auto-tagged
19:39:08 <felixfontein> the others are deprecation-related and fortunately used way less frequently
19:39:17 <felixfontein> I think requiring them to be tagged would be acceptable
19:39:31 <felixfontein> (ansible-test can validate that as good as possible)
19:39:32 <bcoca> you lost me
19:39:47 <bcoca> you want to populate all existing options with version_added?
19:39:54 <felixfontein> no
19:40:13 <felixfontein> version_added stays as it is - use it for new options/modules/plugins, don't change it for existing ones (that means having none for migrated content)
19:40:48 <felixfontein> but if you use removed_in_version=xxx in argspec, or module.deprecate('msg', version=xxx), or deprecated_aliases.version=xxx, then xxx *MUST* contain a collection tag
19:41:26 <felixfontein> so that when xxx is shown to the user, it is clear which collection will remove this feature
19:42:59 <felixfontein> right now most of these versions are still ansible versions all over the collection world, at least in migrated content, because nobody updated them yet (ok, maybe thats mainly true only in community.general and community.network)
19:43:22 <bcoca> yeah, since 'deprecation' was not a feature of collections for a long time, not something we looked into
19:43:42 <bcoca> soon the routing PR will add the feature, but as per @nitzmahone that is just using date
19:43:47 <felixfontein> so if we ask for this now, it will be a) useful for the users since they get more useful deprecation warnings, and b) less work for collection maintainers since they have to replace all these versions anyway
19:44:50 <felixfontein> well, there are some collections which will probably want to stay with version numbers
19:44:54 <felixfontein> (for various reasons)
19:45:44 <bcoca> well, 3 cases 'certified partners', 'collections that want to live in acd' and 'others'
19:45:44 <felixfontein> I would find it really strange to allow versions for everything *except* for deprecating modules/plugins
19:46:10 <felixfontein> also, the docs team really likes version_added, and apparently the ansible content team too
19:46:23 <bcoca> right now we allow versions, i just dont think its well defined what they shoudl be
19:46:41 <bcoca> those all 'still work'
19:46:43 <felixfontein> yes. and right now I'm trying to make it more well defined
19:50:17 <felixfontein> is anyone against tracking collection names in version_added?
19:50:54 <felixfontein> (I hope not, since that can be done totally transparent)
19:51:12 <felixfontein> also, are there reasons to *not* tag deprecation versions?
19:52:34 <abadger1999> I am +1 to doing so.
19:55:56 <felixfontein> (also, for deprecation dates, knowing *which* collection released after a date will stop making something work would be helpful.)
19:56:55 <bcoca> i dont think they are related that way, it has to do with support flaging a minor version released as the 'supported' one
19:57:09 <bcoca> ^ mostly for certified partners / automation hub
19:57:21 <bcoca> so it wont be for 'current release'
19:57:31 <bcoca> unless that happens to be 'the' suported release
20:00:09 <bcoca> well, we are out of time, felixfontein i would write up specific proposal, unsure what steps you wanted to come from this .. seems mostly related to collection authors updating their version_added fields?
20:00:51 <bcoca> not sure core team is needed in that, i do see a need to coordinate the 'deprecated by date' vs 'deprecated by version' pushes
20:01:17 <felixfontein> for one, increasing awareness of the problem, and then to see whether people can live with, or even like tags for deprecation versions (which can't be tagged automatically since added by code)
20:01:49 <bcoca> but in teh code from the plugin author
20:02:16 <felixfontein> also in ansible-base
20:02:23 <bcoca> only reason i can see for core involvement woudl be to 'auto tag' in the code when collection context is detected, which i think is a sensible idea
20:02:38 <bcoca> but weren't you saying asnible base can go untagged by default?
20:02:48 <felixfontein> the urls doc fragment uses deprecated_aliases.version='2.13'
20:02:58 <felixfontein> version_added can go untagged
20:03:05 <felixfontein> since this comes from docs / doc fragments
20:03:19 <felixfontein> deprecation versions (except that one on module level) comes from code
20:03:42 <felixfontein> and there it's not known which code (from which collection) put it into argspec, or ran module.deprecate()
20:03:48 <bcoca> felixfontein: not the doc fragment, they dont support that
20:04:02 <felixfontein> yes
20:04:10 <bcoca> this is module utils
20:04:14 <felixfontein> someone has to create a argspec dictionary in code, or call module.deprecate() in code
20:04:27 <bcoca> and its simple enough for modules to call module.deprecate .. i seem to be lost here to what the issue is
20:04:29 <felixfontein> module_utils in ansile-base, module_utils in collection foo.bar, or module code in foo.baz
20:04:55 <felixfontein> if a module says "this feature has been deprecatd and will be removed in version 1.2.3", does this refer to a) ansible-base version, b) foo.bar version, c) foo.baz versoin?
20:05:18 <bcoca> well, module author should specifiy, just like you were saying for version_added
20:05:44 <felixfontein> but module author can only do that in module code
20:05:46 <felixfontein> not in module_utils
20:06:00 <felixfontein> maybe the deprecate call is made somewhere in ansible-base's module_utils
20:06:00 <bcoca> well,d epends, he canot do it in module_utils he does not own
20:06:09 <felixfontein> or in foo.bar's module_utils that the module is using
20:06:10 <bcoca> and?
20:06:19 <bcoca> i still fail to see the problem
20:06:40 <bcoca> things deprecated in core, will be deprecated at core version schedule
20:06:42 <felixfontein> if some module_utils in ansible-case calls module.deprecate(), they will use ansible-base version for version
20:06:57 <bcoca> if they dont want to override, yes
20:07:23 <bcoca> and would still be correct, since same module used with 'newer ansible that did the deprecation' would reflect the lack of the feature
20:07:33 <bcoca> if the module relies on the base to provide teh feature
20:07:34 <felixfontein> if your module is in foo.baz and you included module_utils from foo.bar, call some function there, that calls a function in ansible's module_utils, and that one calls module.deprecate
20:08:00 <felixfontein> taht will result in a deprecation warning for a module in foo.baz with a version number relative to ansible-base
20:08:19 <bcoca> how is this differnt that using urllib which in version 3.1 deprecates (and warns about) function X going away?
20:08:37 <felixfontein> the warning usually says that it is urllib 3.5 that removes this feature
20:08:39 <bcoca> its not like this is a new consideration when using 'shared libs
20:08:49 <felixfontein> so the user sees "ok, urllib 3.5 removes this feature which the program I use right now apparently needs"
20:08:53 <bcoca> and thsi will use 'base' version
20:09:08 <bcoca> if it were collection version it will be namesapced
20:09:12 <felixfontein> if the warning says "feature will be removed in version 3.5" without further information, the user has no clue what's going on
20:09:20 <bcoca> so what youa re asking for is 'always autolabel asnible-base'?
20:09:52 <felixfontein> no, I'm asking that all collections and ansible-base should label (or tag or however you want to call it) their deprecation version numbers
20:10:04 <bcoca> ^ cause above you were advocating for NOT labeling in the case of 'base' version
20:10:14 <felixfontein> so that it says 'the feature will be removed in version ansible.builtin:2.14'
20:10:16 <bcoca> so no, but yes?
20:10:21 <felixfontein> I'm advocating to NOT label for version_added
20:10:28 <felixfontein> but to ALWAYS label for deprecation versions
20:10:29 <bcoca> confused
20:10:43 <felixfontein> because version_added needs no label/tag, since Ansible knows exactly where doc fragments come from
20:11:04 <bcoca> k, in any case, wer are way over time, going to end now, please write up what you mean, cause i keep getting lost (or i just cannot think with construction outside window)
20:11:09 <felixfontein> but deprecation version numbers always come from code, and there Ansible doesn't know (without tracing execution of module in Python) where the version numbre came from
20:11:23 <felixfontein> ok
20:11:26 <bcoca> #endmeeting