14:31:25 <acozine> #startmeeting Docs Working Group aka DaWGs
14:31:25 <zodbot> Meeting started Tue Jun  2 14:31:25 2020 UTC.
14:31:25 <zodbot> This meeting is logged and archived in a public location.
14:31:25 <zodbot> The chair is acozine. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:31:25 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:31:25 <zodbot> The meeting name has been set to 'docs_working_group_aka_dawgs'
14:31:38 <acozine> #topic agenda review
14:31:38 * samccann waves
14:31:46 <acozine> #chair abadger1999 samccann
14:31:46 <zodbot> Current chairs: abadger1999 acozine samccann
14:31:49 <acozine> who else is around?
14:31:52 <samccann> heh love the first topic!
14:32:05 <acozine> I think we need to prioritize a little
14:32:34 <acozine> bcoca dmellado gundalow relrod you folks talking docs today?
14:32:34 <samccann> for sure
14:32:48 <felixfontein> hey!
14:33:10 <samccann> hey dmellado changed roles... may not be involved in Ansible anymore... just fyi
14:33:35 <acozine> andersson007_: awcrosby__ briantist cbudz cyberpear madonius Pilou shaps tadeboro Tas-sos thedoubl3j Xaroth zoredache you joining the meeting?
14:33:45 <dmellado> hiya! I'm around but yeah, not that much involved anymore ;)
14:33:55 <acozine> welcome felixfontein!
14:34:00 <acozine> #chair felixfontein
14:34:00 <zodbot> Current chairs: abadger1999 acozine felixfontein samccann
14:34:05 <acozine> dmellado: ah, I didn't know that
14:34:27 <acozine> dmellado: do you want to participate anyway?
14:34:47 <briantist> hi I'm around but will be somewhat distracted, sorry if I miss something or don't answer right away
14:35:18 <acozine> welcome briantist - I'll make you furniture, participate as/when you can
14:35:21 <acozine> #chair briantist
14:35:21 <zodbot> Current chairs: abadger1999 acozine briantist felixfontein samccann
14:35:59 <acozine> our first two agenda items were PRs 69727 and 69268, both from felixfontein, and both have been merged
14:36:28 <acozine> other agenda items carried over from previous weeks include:
14:36:37 <acozine> porting guide strategy
14:36:40 <acozine> changelog strategy
14:36:45 <acozine> docs pipeline
14:37:09 <samccann> on the merge items  - have we covered the docs changes needed?
14:37:25 <samccann> I've got a little PR for the remove_by_date
14:37:58 <acozine> oh, my mistake - 69268 is from andersson007_
14:38:33 <acozine> that one was a change to the changelog instructions themselves, so it's self-documenting, though we may need more docs changes once the new changelog process is in place
14:38:38 <samccann> https://github.com/ansible/ansible/pull/69833 is the PR to cover docs for 69727
14:39:16 <cbudz> I'm around, just wrapping up a fix.
14:39:25 <felixfontein> samccann: the docs should mention that removed_in and removed_by_date are mutually exclusive, and that one must be specified
14:39:36 <felixfontein> and also that meta/runtime.yml must contain the same information
14:39:51 <samccann> yeah we have no docs afaik for runtime.yml
14:40:03 <felixfontein> yes, they are added in shertel's action groups PR
14:40:10 <felixfontein> i.e. will exist hopefully soon
14:40:11 <samccann> so was thinking that's a separate pr. But I can clarify that only one should be used in the docstring
14:40:21 <samccann> kewl good to know!
14:40:33 <samccann> do you have the pr number for thta?
14:40:34 <samccann> that?
14:40:50 <samccann> (oh sorry... this is supposed to be just a review of the agenda... shhh me)
14:41:19 <acozine> heh
14:42:15 <felixfontein> https://github.com/ansible/ansible/pull/67291
14:42:22 <acozine> the other three agenda items are about PRs 69795, 69796, and 69797
14:42:25 <samccann> my nickel - if we have docs needs, we add them to the tracking board (even if it's just as a note until an issue is opened).  If the docs need is covered in the PRs itself, we remove them from the main agenda
14:42:50 <acozine> two of those are directly related to the docs pipeline changes (because they change the plugin formatter)
14:43:36 <acozine> then we have follow-up on recently merged PRs to make sure the changes they introduced are documented . . . added to our mental agenda here
14:44:10 <acozine> I vote we start with the pipeline update, since that seems both broadest and most immediate . . .
14:44:33 <samccann> +1
14:44:40 <acozine> samccann: yes, adding stuff to the tracking board is a great idea
14:44:49 <felixfontein> the PRs (except 69797) are both for the docs meeting and the core meeting, the question for the docs meeting is whether you like this change from how the result looks
14:45:18 <acozine> mmm, and how it fits in with the other pipeline changes
14:45:32 <acozine> #topic collections docs pipeline update
14:45:40 <felixfontein> 69797 is a bit strange, I'm not sure whether it's really needed; but then I assumed that the plugin_formatter would have already been deleted by now
14:45:43 <felixfontein> (ping abadger1999)
14:45:45 <acozine> abadger1999: what's the latest from your end?
14:46:22 <felixfontein> 69795 and 69796 are about ansible-doc console output, they are unrelated to the pipeline changes
14:46:52 <abadger1999> I merged the initial antisbull docs pipeline PR last week and opened up tickets for known bugs (two) and needed features (the total is somewhere around 17 with about 5 that are high priority)
14:47:08 <acozine> hooray!
14:47:30 <abadger1999> yesterday I updated the ansible/ansible PR to allow building the website with the new code.  And discovered tons of new bugs.
14:47:43 <samccann> #info antsibull docs pipeline PR merged.. followon bugs and features to follow
14:48:15 <abadger1999> It generates a docsite but you're going to get lots of sphinx warnings about improper rst and you'll notice problems on the rendered output (for instance, the indexes don't have descriptions of hte plugins)
14:48:28 <abadger1999> I'm working through those today.
14:48:44 <abadger1999> Oh, another important thing.
14:48:45 <acozine> "it generates a docsite" is a big step forward
14:49:36 <acozine> felixfontein: ah, so the plugin formatter gets used by `ansible-doc`
14:49:40 <abadger1999> Over the weekend, I took a step back and tried to put together the high level usages of the docs pipeline:  https://github.com/ansible-community/antsibull/wiki/ansibulled-docs-design-plan#use-cases
14:49:52 <felixfontein> acozine: no, it's used by the old website build
14:50:30 <abadger1999> My intent is that all use cases will be taken care of with the new docs pipeline but probably not with the exact same UI.
14:50:41 <acozine> abadger1999: that's a great summary
14:51:10 <abadger1999> ie: make webdocs MODULES='yum,ping,zypper' might no longer be the command to run to test that those three modules' documentation builds
14:51:46 <samccann> #info some notes on high leve use of docs pipeline (draft) - https://github.com/ansible-community/antsibull/wiki/ansibulled-docs-design-plan#use-cases
14:51:48 <abadger1999> If you have other use cases, refinements on the ones I listed, or want a use case to work in a specifc way, please let me know.
14:52:26 <acozine> will do - I think we can change the UI as we transition to collections
14:52:31 <abadger1999> Cool :-)
14:52:39 <samccann> #info looking for feedback on those use cases or any others folks have in mind
14:53:48 <acozine> so where do PRs 69796 and 69797 fit in?
14:54:27 <abadger1999> I'll upload a copy of the built docs site for people to look at and at some point this week we can start having jenkins build the docs ite to the test server if desired.
14:54:28 <acozine> both update the plugin_formatter
14:54:38 <felixfontein> 69795 and 69796 are about the text output of ansible-doc, and unrelated to the docs pipeline
14:54:59 <felixfontein> 69797 fixes the old docs pipeline so that it understands tagged versions before the new docs pipeline replaces the old one
14:55:46 <acozine> abadger1999: thanks, that sounds great
14:56:13 <abadger1999> https://github.com/ansible/ansible/pull/69796/ <== This one touches plugin_formatter code but I believe the new code will already the handle the proposed changes so I think it will just be an enhancement for ansible-doc.
14:56:31 <acozine> excellent
14:57:18 <felixfontein> abadger1999: it does, only to avoid that it breaks when it gets already decoded YAML
14:57:55 <abadger1999> felixfontein: Oh :-(  I thought the schema would accept the already decoded YAML.
14:58:41 <felixfontein> abadger1999: I meant the old plugin_formatter
14:58:47 <felixfontein> no idea about the new one, haven't checked it :)
14:59:22 <acozine> is formatting plugins going to be handled in antsibull when the new pipeline is ready?
14:59:27 <abadger1999> felixfontein: Ah :-)
14:59:34 <abadger1999> acozine: yes.
14:59:54 <acozine> thanks
15:00:06 <abadger1999> make webdocs with the new docs pipeline PR should be a way to test that (but be warned, it is much slower than the old code right now)
15:00:54 <abadger1999> https://github.com/ansible/ansible/pull/69797 <== this one will probably need to be pushed into the new docss pipeline code (inside of antsibull)
15:01:28 <abadger1999> too_old is handled as a jinja filter there, instead of stripping the data.
15:01:49 <acozine> #info can test the WIP collections docs pipeline using the branch from https://github.com/ansible/ansible/pull/59761
15:01:55 <abadger1999> I've currently set the filter to only hide "historical" but let any version numbers through.
15:01:58 <felixfontein> https://github.com/ansible-community/antsibull/pull/62 already does that (partially)
15:02:46 <abadger1999> That might need to be updated since the fully tagged version is no longer a smeantic version/pypi version/etc.
15:03:17 <acozine> should we close 69797 with a pointer to antsibull/62?
15:03:39 <felixfontein> it depends on whether the old code should stay up to date before being removed or not :)
15:05:32 <abadger1999> felixfontein: Does that "Just work(TM)" ?  If so, I'll merge it right now.... I don't think we want to do too much to the old plugin_formatter, though.
15:05:50 <abadger1999> But if you have the code working, might as well ;-)
15:05:55 <acozine> +1 to merging if it's complete; the work is done already
15:06:18 <felixfontein> I think it works
15:06:24 <abadger1999> Cool /me merges
15:06:35 <abadger1999> Worst case... someone says it broke something as we revert.
15:06:51 <felixfontein> most of the code doesn't do anything because the plugin_formatter no longer gets tagged versions, because collection_name is correctly passed to plugin_docs.get_docstring
15:07:05 <felixfontein> but if someone tries to adapt this to collections (independently of the work in antsibull), it should work
15:07:49 <felixfontein> thanks!
15:08:32 <abadger1999> #action abadger199 merged https://github.com/ansible/ansible/pull/69797
15:08:51 <samccann> woot!
15:08:57 <felixfontein> \o/
15:08:59 <abadger1999> #info equivalent functionality of 69797 should be added to the new docs pipeline in https://github.com/ansible-community/antsibull/pull/62
15:09:49 <acozine> next steps for docs WG on the new docs pipeline include 1. review docs design plan, identify gaps; 2. test new pipeline by pulling the branch from PR 59761; anything else we can do?
15:10:11 <acozine> (other than add links to the above?)
15:10:46 <felixfontein> abadger1999: how much work is it to get the devel docs build working? that one would a lot more interesting, since the last published collections are sometimes pretty old
15:11:24 <felixfontein> I'll later try again to get the individual plugin .rst generation working (I looked at it a bit yesterday but didn't have enough time), I think that would be useful to test things out - like deprecation by date etc.
15:11:36 <felixfontein> waiting until the first collection using that is published will take too much time :)
15:11:41 <abadger1999> https://github.com/ansible-community/antsibull/issues/54#issue-627447828
15:11:54 <abadger1999> There's new code needed to handle the git repo.
15:12:00 <acozine> the new pipeline pulls from Galaxy, not from GitHub (or other code repository platforms), right?
15:12:14 <abadger1999> But I did just add support for passing a directory in via --ansible-base-cache last night.
15:12:19 <felixfontein> abadger1999: ah, so devel doesn't mean dev versions of collections too?
15:12:32 <abadger1999> So actually, three-quarters of the code needed for handling of a git repo is now there.
15:12:35 <samccann> #info next steps for docs WG on the new docs pipeline include 1. review docs design plan, identify gaps; 2. test new pipeline by pulling the branch from PR 59761
15:13:33 <abadger1999> felixfontein: correct.... It should mean latest version of collections.
15:13:45 <acozine> if we support docs from git repos, how does the devel pipeline decide which collections get built from Galaxy and which from their git repos?
15:13:51 <abadger1999> latest published version.
15:14:23 <abadger1999> But getting the git repo for those collections could be tricky because we don't know if those will actually make it into the next major release of ansible or not.
15:14:27 <abadger1999> felixfontein: there's also this: https://github.com/ansible-community/antsibull/issues/65 which I think touches on what you want too.
15:14:48 <felixfontein> maybe I'll try to hack something together which does the whole build for the currently installed ansible and collections, that should be enough for testing everything I want to test ;)
15:14:50 <acozine> would we pull from any publicly accessible git repo, with a fallback to Galaxy if the git repo isn't listed or isn't publicly accessible?
15:15:19 <abadger1999> It would mean that we use the git repo for ansible-base, latest comit to stable-X.Y , and the latest compatible published version to galaxy for the collections.
15:16:28 <acozine> hmm, I think I'll have to see that in action to understand it thoroughly
15:16:48 <acozine> and it sounds like you're way ahead of me :-)
15:16:50 <abadger1999> acozine:  yes, the new pipeline pulls from galaxy.
15:17:28 <abadger1999> I'm not sure how successful we'd be at pulling from git repos... git repos don't need to have any standard layout, for instance.
15:18:11 <felixfontein> abadger1999: I guess we could collect a list of git repo URLs which allow to install a collection via `ansible-galaxy collection install` (it supports git repos now).
15:18:13 <abadger1999> ansible-base we just handle as one specific one-off... but it will get harder if we start handling all the ways that people are laying out their repos, tagging them, etc.
15:18:18 <acozine> the main goal for the 2.10 freeze is still being ready to build a `latest` website for 2.10 that includes module docs from collections
15:18:27 <abadger1999> felixfontein: ah.... yeah, that's a possiblity.
15:18:50 <acozine> okay, we've only got twelve minutes left
15:18:52 <felixfontein> abadger1999: that way we could off-load the work to `ansible-galaxy collection install` :)
15:19:02 <felixfontein> yeah, let's focus on a proper build first
15:19:09 <felixfontein> no devel stuff :)
15:19:35 <acozine> let's see if we can address the other topics
15:19:56 <acozine> and add "how to build `devel` docs" to a future agenda
15:20:19 <acozine> it's important, and i don't want to lose it, but . . . .
15:20:20 <abadger1999> felixfontein: I don't know... there's a lot of questions questions swirling around in my head about how collection git repos would work in the pipeline... both from a "how does the code have to change to deal with it" and from policy "Do we want to be documenting something that won't be in the next version of ansible?"
15:20:42 <samccann> #action add 'how to build devel docs' to agenda
15:21:08 <abadger1999> felixfontein: Maybe something to do but not do immediately?  Maybe something to implement as one of two alternate ways to construct the devel docs and then see which we like better?
15:21:20 <abadger1999> <nod>
15:21:32 <abadger1999> Ready to move on if ya'll are.
15:21:36 <felixfontein> abadger1999: having both options sounds good.
15:21:46 <felixfontein> but let's do this later, not now :) and move on
15:21:50 <acozine> #topic changelogs and porting guides for 2.10
15:22:15 <acozine> we've been discussing changelogs for a long time, and we've made a lot of progress
15:22:47 <acozine> andersson007_'s PR specifying the changelog entry format got merged (finally, sorry about the delay there)
15:23:10 <acozine> felixfontein's work on collections changelogs got merged (hooray!)
15:23:22 <felixfontein> not yet, I think
15:23:26 <acozine> no?
15:23:35 <felixfontein> or which part do you mean?
15:23:48 <felixfontein> using antsibull-changelog for ansible/ansible hasn't been merged yet
15:24:06 <acozine> hm, I guess it doesn't really matter - what matters is figuring out which parts are still not done/merged/ready
15:25:04 <acozine> ah, you're right, https://github.com/ansible/ansible/pull/69313 is still open
15:25:53 * acozine looks at PR conversation
15:25:54 <samccann> #info https://github.com/ansible/ansible/pull/69313 still waiting on merge to get ansibull-changelog for ansible/ansible
15:26:03 <felixfontein> I've added a lot of tests (and fixed some small bugs) in the changelog generator over the weekend
15:26:17 <felixfontein> and released a new version yesterday (0.2.0, which is pinned in that PR)
15:26:33 <acozine> awesome, thank you!
15:27:44 <acozine> can you explain `always_refresh`?
15:28:00 <felixfontein> it's mostly (and probably only) useful for ansible/ansible
15:28:12 <felixfontein> core wants the changelog to adjust to changed fragments and updated plugin descriptions
15:28:13 <acozine> what exactly will get refreshed?
15:28:40 <felixfontein> since changelog.yaml contains copies of the fragment texts and copies of the plugin descriptions, it won't auto-update when you change the original fragments and/or plugins
15:28:56 <felixfontein> always_refresh makes sure that these are always updated (and not only if you explicitly request it)
15:30:09 <felixfontein> if you don't keep fragments, or don't start a new changelog for every new major release, refreshing can be dangerous as it could remove "new plugins" from old changelog entries
15:30:59 <acozine> can you give an example?
15:31:29 <felixfontein> version 2.1.0 adds a new plugin `foo`; version 2.5.0 deprecates it for removal in 3.0.0; version 3.0.0 removes the plugin
15:32:05 <acozine> the versions here are collection versions, right?
15:32:10 <felixfontein> if you then refresh the changelog.yaml with the current state, it will see that plugin `foo` does not exist and will remove all mentions of it from changelog.yaml (i.e. the new plugin notice for 2.1.0)
15:32:15 <felixfontein> yes
15:32:51 <acozine> gotcha, and all these collection releases are happening during the development of a single release of ansible-base
15:32:52 <felixfontein> the current changelog generator (in ansible/ansible) always refreshes, since it doesn't store all this information in changelog.yaml (resp. .changes.yaml)
15:33:02 <felixfontein> no, it's unrelated to that
15:33:17 <felixfontein> that's an example why refresh is dangerous in collections
15:33:27 <felixfontein> but it's what core wants in ansible/ansible
15:33:41 <felixfontein> there it's not dangerous since they start a new changelog every major version
15:33:56 <acozine> so you've turned it on across the board? or you've turned it on only for ansible/ansible?
15:34:09 <felixfontein> no, it's only turned on in ansible/ansible's config.yaml
15:34:17 <acozine> okay, I'm with you (finally)
15:34:23 <acozine> thanks for the explanation
15:34:25 <felixfontein> collection can turn it on, but only manually (and they shouldn't)
15:34:37 <acozine> "don't do this at home, kids"
15:34:53 <felixfontein> there's always the kid who ignores that warning :)
15:34:56 <acozine> "this is a professional driver on a closed course"
15:34:57 <acozine> heh
15:34:59 <acozine> yep
15:35:52 <felixfontein> yep :)
15:35:56 <acozine> so getting 69313 merged is a question of following up with the core devs
15:36:30 <felixfontein> yes, I think so
15:36:50 <felixfontein> mattclay is probably busy with other things right now
15:36:56 <acozine> no doubt
15:37:11 <acozine> there's a lot going on
15:37:35 <felixfontein> as always :)
15:37:43 <acozine> what else do we need to do/test/think about for changelogs and porting guides?
15:38:01 <felixfontein> changelogs: does ansible/ansible also wants to use the new categories (breaking_changes / security_fixes)?
15:38:33 <felixfontein> porting guides: are changelog entries enough (multi-paragraph enrties now work, example: https://github.com/felixfontein/ansible-versioning_test_collection/blob/master/changelogs/CHANGELOG-v2.rst#minor-changes), or do we need something else? like sections with titles?
15:38:58 <acozine> I'd say yes, we want to use all the categories to publish a unified changelog somewhere
15:39:20 <felixfontein> the PR doesn't add the new categories yet
15:39:21 <acozine> will that happen when 69313 gets merged, or is that separate work?
15:39:24 <acozine> okay
15:39:30 <felixfontein> it's probably better to do that in a follow-up PR which also documents them
15:39:42 <felixfontein> (and adjusts existing fragments, especially the security fixes)
15:39:44 <acozine> sounds good
15:40:42 <acozine> #agreed we will need a follow-up PR to 69313 to make ansible/ansible's changelogs include the new categories of changes, with documentation
15:40:43 <samccann> fwiw we are past the 1hr mark
15:40:55 <acozine> samccann: ah, thanks
15:41:09 <felixfontein> true
15:41:18 <felixfontein> thanks everyone!
15:42:20 <acozine> thanks abadger1999 briantist cbudz dmellado samccann
15:42:35 <felixfontein> thanks also to acozine
15:42:52 <acozine> :-)
15:42:52 <abadger1999> Thanks for having such a well-organized meeting :-)
15:44:06 <acozine> we have enough on the agenda for another "overflow meeting" but I think this week we should focus on testing the changes we have in flight
15:44:11 <acozine> and meet again this same time next week
15:44:27 <acozine> we have made amazing progress
15:44:29 <samccann> same DaWGs time... same DaWGs channel...
15:44:36 <acozine> heh, exactly!
15:44:45 * samccann just here for the comic relief
15:44:52 <acozine> #topic open floor
15:44:57 <acozine> anyone need to express themselves?
15:45:34 <acozine> during this incredibly busy time, the docs team can miss easy-to-merge PRs because we just get busy
15:45:47 <acozine> if you've got an open PR, please feel free to bring it up!
15:46:00 <acozine> now . . . or in any DaWGs meeting . . . or in the channel any time
15:46:17 <acozine> all right, that was a short open floor
15:46:25 <acozine> but it happened
15:46:28 <acozine> thanks again folks!
15:46:31 <acozine> #endmeeting