14:31:22 #startmeeting Docs Working Group aka DaWGs 14:31:22 Meeting started Tue Apr 28 14:31:22 2020 UTC. 14:31:22 This meeting is logged and archived in a public location. 14:31:22 The chair is acozine. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:31:22 Useful Commands: #action #agreed #halp #info #idea #link #topic. 14:31:22 The meeting name has been set to 'docs_working_group_aka_dawgs' 14:31:26 who's around? 14:31:35 * samccann waves 14:31:41 #chair samccann 14:31:41 Current chairs: acozine samccann 14:32:34 hi! 14:33:19 welcome felixfontein 14:33:23 #chair felixfontein 14:33:23 Current chairs: acozine felixfontein samccann 14:34:28 who else is around? 14:34:35 or is everyone watching Summit? 14:34:52 what Summit? 14:35:18 Red Hat Summit, it's the big RH conference - taking place online this year, for obvious reasons 14:35:19 red hat virtual summit 14:35:23 ah 14:35:26 if you mean the head of the construction workers that are making my new sidewalk, yes 14:35:33 it's streaming for free today and tomorrow. There are some Ansible presentations happening 14:35:37 welcome bcoca 14:35:41 * bcoca waves 14:35:42 #chair bcoca 14:35:42 Current chairs: acozine bcoca felixfontein samccann 14:36:17 https://www.redhat.com/en/summit 14:36:38 let's see, today's agenda is at √ 14:36:42 ugh 14:36:56 still not used to the keyboard changes I made 14:36:58 https://github.com/ansible/community/issues/521#issuecomment-617256701 14:37:00 that's better 14:38:27 #topic changelogs for ACD 14:38:46 felixfontein: thanks for the "live" mockups 14:39:17 how is the work going? 14:41:09 I haven't done anything since the last mock from yesterday evening, but I think it looks good. working with the .yaml changelog format is fine, working with the changelog from ansible-base is a bit harder. (but then, this only has to be run by RM and not by users so that should be ok.) 14:41:32 I should mention that I may be called away during this meeting - my father is in the hospital (not COVID-19) and we are waiting to hear if they will release him today 14:41:56 felixfontein: that's awesome 14:42:23 so it's ready for testing? 14:42:24 #info ACD live mockups based on felixfontein's scripts - https://github.com/felixfontein/ansible-changelog/blob/master/acd/acd.rst 14:43:02 fantastic commit message: https://github.com/felixfontein/ansible-changelog/commit/0a50d07a0991e46056866070f804391a632d75e5 14:43:05 woopsie - I should also mention then that I've got a video call up and waiting for the doctor to show up... we're a bit sporatic today! 14:43:36 #info ACD mockup with ACD version as first header - https://github.com/felixfontein/ansible-changelog/blob/master/acd/acd2.rst 14:43:39 the main thing I noticed is that it needs more than a simple collection - version mapping, but a ACD version -> collection mapping for every collection included 14:43:55 otherwise it's impossible to compile a proper changelog which contains multiple ACD releases (minor and/or patch) 14:44:30 something I haven't tested (and implemented) yet is tracking changelogs over multiple branches 14:44:49 which might be needed depending on how versioning/releasing works for some collections 14:44:51 is that reflected in the config file - https://github.com/felixfontein/ansible-changelog/blob/master/acd/acd.yml ?? 14:45:07 in the current version it is 14:45:20 there would also be entries such as 2.10.1, 2.10.2, etc. 14:45:27 and eventually 2.11, 2.11.1, 2.11.2, ... 14:45:35 #info this is controlled by a config file - https://github.com/felixfontein/ansible-changelog/blob/master/acd/acd.yml 14:46:12 So is our next step to create a dummy repo collection with multiple branches etc? something to really exercise it? 14:46:14 I'm currently working with the git repos the collection comes from, and not the compiled and uploaded collection 14:46:23 yep, that I want to do 14:46:25 (unless someone knows of one already that could help) 14:46:30 I guess I'll manage somewhen the next days 14:46:47 samccann: they probably won't have the changelog.yml file ;) 14:46:57 heh true 14:47:26 speaking of which, I'll try and write up a WIP PR for the docs on how to do all this so we have a process people can review/comment on 14:47:46 can you re-use toshio's code for docs generation to switch from using the git repos to using the compiled collections? 14:47:51 TBH I'm still waiting for progress on how collections (in particular the huge community ones) should be versioned (in particular from git branching point of view) 14:48:09 mmm, yes, that would be good information to have 14:48:17 ah there is an issue somewheres with some detail on that (at least from the perspective of certified collections) 14:48:39 but that's just about versioning, not about how this will be handled in the repos 14:48:54 like what / where are release branches, what is backported and by whom, etc 14:49:03 https://github.com/ansible-collections/overview/issues/37 14:49:07 this is also a lot simpler for small and/or focussed collections 14:49:18 are your questions/what still needs deciding on somewhere in that tracking issue already? 14:49:25 but a lot harder for huge diverse collections such as community.network and community.general 14:49:47 mostly yes, I think 14:50:06 I'll send some emails out to get hopefully more discussion on this (right now there seems to be essentially none) 14:50:07 #info still need details on how collections will be versioned, especially from a git branching point of view. That's a dependency on finishing the changelog scripts/processes. 14:50:17 yes, that may drive a trend toward more granular collections 14:50:54 #action felixfontein to try and kickstart that branching etc conversation soon. 14:51:09 if you need help with the kicking ^^ let us know! 14:51:23 * bcoca buys padded clothing 14:51:27 heh 14:51:45 bcoca: I thought you'd be buying hobnailed boots 14:52:08 * samccann munches popcorn and waits to see who is the kicker and who are the kickees... 14:53:28 i have the boots, but its also good to cover defense 14:53:39 #action samccann to draft the changelog process as we know it so far into a docs PR for community review/feedback 14:53:46 * bcoca expects circular kicking squad 14:54:22 oh another question comes to mind - right now, ansible-base is going to keep using the current changelog solution. Is there any reason/benefit/reasons not to ask them to use the same solution as collections will use? 14:54:28 just to confirm , the criteria for inclusion in docs.ansible.com are that the collections are in ACD? 14:54:37 bcoca: yep afaik 14:54:39 all joking aside, I'm not sure who gets to decide on the release cadence, backport policies, and branching protocols for the big community collections 14:54:52 bcoca: yes, that is correct 14:55:19 samccann: I've wondered that too; at least including more information on plugins would already be helpful, because extracting plugin data is not so simple without actually installing it in some way 14:55:22 samccann: depends on what you mean, the changelog solution is deeply tied to workflow i doubt all collections will follow same one, while 'product consumable by ACD/docs' will be posible from several workflows/solutions 14:55:23 we may change that policy as time goes on, depending on how the collections ecosystem evolves 14:56:05 bcoca: I copied ansible/ansible's changelog generator and extended it to also handle collections (next to still work with ansible/ansible): https://github.com/felixfontein/ansible-changelog/ 14:56:08 gettign collection docs, currently do require installation, but that is on a lot of people's wishlist to see in galaxy/AH 14:56:22 felixfontein: do you mean including more information on plugins that remain in ansible/ansible? or the ones in collections? or both? 14:57:03 also consider that a lot of the plugins migrated went to 'partners' which won't be as 'free' as the 'community' at large to decide on their workflows 14:57:17 acozine: the ones remaining in ansible/ansible. the current .changes.yml file format (used in ansible/ansible itself, see https://github.com/ansible/ansible/blob/stable-2.9/changelogs/.changes.yaml) only mentions the names of the new modules and plugins, but no more information (like namespace and short description) 14:57:27 in the extended format I included that in the changelog.yml 14:58:29 compare https://github.com/ansible/ansible/blob/stable-2.9/changelogs/.changes.yaml#L1144-L1152 to https://github.com/felixfontein/community.general/blob/changelog-1.0/changelogs/changelog.yaml#L616-L623 for example 14:59:18 ah, thanks 14:59:36 the missing details can be extracted by ansible-doc, but it's easier to work with .changes.yaml if that information would already be there (then one simply deals with files on disk, and not with having to call ansible-doc and first installing that repo in a venv or something) 14:59:38 the extended format provides good information 15:00:05 it's essentially the same as the old format, except that all data is in there and not referenced somewhere else 15:00:36 so for the core team, this would mean using the new generator and adding more detail to the changelog files themselves? 15:01:19 if so, that seems like a good return on the investment of time 15:01:20 well, it would be nice if we could stick to one code-base of the changelog generator ;) and also if the fileformat for core could also contain this extra info 15:01:30 agreed 15:01:33 I don't mind if the fragment contents aren't included 15:02:06 @relrod ^ 15:02:15 heh, I'd like it if the fragment contents were included . . . 15:02:41 acozine: me too, but from a handling point of view, not having them included is also ok - the main pain is extracting the plugin data 15:02:43 the fragments end up in final, so you really dont need both 15:03:20 felixfontein: consider that there shoujld be almost no plugin changes in core from now on, 99.99999923453999 will occur in collections 15:03:52 bcoca: true. still, if it happens, the ACD changelog generator has to deal with it :) 15:04:01 bcoca: that may be true, but if there are changes to plugins in core, they will probably affect a lot of users 15:04:09 true, but it looks generic enough already to do so 15:04:27 just need a ansible.builtin fallback namespace? or we fine w/o it? 15:05:03 acozine: core normally didnt add 'module changes' to changelog, but we should from now on, mostly we added 'new/deprecated/removed' 15:05:24 they are part of ansible-base, so I think it's ok without them. but separating them into a `Ansible.Builtin` changelog section would also be possible. but then the question is how to separate changelog fragments between 'affects module/plugin' and 'affects core' 15:05:25 but not internal changes about modules/plugins themselves 15:05:47 * relrod reads up 15:05:59 more furniture! 15:06:02 welcome relrod 15:06:06 #chair relrod 15:06:06 Current chairs: acozine bcoca felixfontein relrod samccann 15:06:23 relrod: tldr having core change changelog process to align more with collections new process (felix forked core changelog generator) 15:07:11 relrod: the fork is here: https://github.com/felixfontein/ansible-changelog/; the acd/ subdirectory is mostly hacking to get the ACD changelog mockups working 15:07:25 relrod: are you/will you be working on ansible-base changelogs? 15:07:31 felixfontein: I can be :) 15:08:37 felixfontein: can you tl;dr the changes that are required? Still reading all the scrollback... 15:08:42 ^ RE .. we are in middle of redefining our release process, so i tyhought he should be in loop for what diff groups are discussing 15:09:42 relrod: thought that even though you might not be deciding many things, you should at least know all the diff interests since i suspect you'll end up implementing them 15:10:00 bcoca: Yep, I appreciate it 15:10:07 relrod: right now, the collection changelog .yam format (looks like this: https://github.com/felixfontein/community.general/blob/changelog-1.0/changelogs/changelog.yaml as opposed to ansible 2.9 one: https://github.com/ansible/ansible/blob/stable-2.9/changelogs/.changes.yaml) contains the contents of the changelog fragments, and contains more info on the new plugins/modules (2.9 yaml only 15:10:13 contains their names, collection one contains also short desc and namespace) 15:10:38 relrod: it would be good if the formats could get more aligned (I don't mind the changelog fragments being separate files, but the missing additional info on the plugins is painful :) ) 15:11:13 (also it would be good if there aren't two different changelog generators, but one who can handle both ansible/ansible and collections and which is used by both ansible/ansible and all interested collections) 15:11:14 felixfontein: one suggestion, move security issues to own header vs having each 'entry' have to try to get attention for it 15:11:35 bcoca: fine for me, right now it does exactly the same as the old changelog generator 15:11:37 ^ we do have specific page for them in ansible 15:11:56 bcoca: the security issues are put into fragments as 'minor changes' or 'bugfixes' I think and thus end up where they do right now 15:12:00 felixfontein: i know, but we manually update the security issues lists, would be nice ... to automate, otherwisee its a pain 15:12:22 definitely! having their own section is easy, the sections are configurable 15:12:25 yep, jsut saying, right now is a good opportunity to change htat 15:12:52 indeed 15:13:02 we were also talking about a new section for `breaking changes` 15:13:10 like what currently goes into the porting guide 15:13:19 i would call those 'major changes' 15:13:45 consider that bugfixes and security issues can also create such a change 15:13:46 I guess some people would want to use major changes for other things than that as well 15:14:22 I guess we'll have to create a whole new discussion just for this :) 15:14:23 breaking/porting is not single home for a change entry, but additional one 15:14:41 heh 15:14:42 so im all for the section, but it should be 'in addition to' 15:15:26 the entries in both sections might be formulated differently, so just being able to specify multiple sections per changelog fragment (as possible right now) might already be enough 15:15:31 not sure why fragemnt list is needed, i would just include the contents in the appropriate sections 15:15:40 felixfontein: yep 15:15:56 that was how the changelog generator was implemented initially and nobody ever changed that :) 15:16:03 well, until the collection one now ;) 15:16:12 that was my thought security: .... \n breaking: ..... 15:16:33 so how many categories would we have? 15:16:46 felixfontein: it made more sense when we didnt include all changes, but now that we seem to want to change that ... 15:16:57 just 2 more, security and porting/breaking 15:17:10 Security Fixes, Breaking Changes/Porting Guide, Major Changes, Minor Changes? 15:17:21 deprecations and removals 15:17:25 and Bugfixes 15:17:27 and bugfixes 15:17:36 7 15:17:48 Security Fixes, Breaking Changes/Porting Guide, Major Changes, Minor Changes, Deprecations, Removals, Bugfixes 15:17:55 quite a list 15:18:02 if we have that many, we will definitely need good guidance on what goes in each one 15:18:09 yep 15:18:12 agreed 15:18:27 major/minor might be most 'nebulous' 15:18:36 and it would be nice to have a second changelog called "porting guide" which just contains sections Security Fixes, Breaking Changes/Porting Guide, Deprecations, Removals 15:18:46 (maybe even without security fixes) 15:18:47 is duplication allowed? do we pull Security Fixes into the porting guides as well as the Breaking Changes? 15:18:48 deprecations, removals, bugfixes and security are easy 15:19:01 acozine: yes, but diff texts as stated above 15:19:02 yes 15:19:03 acozine: right now no, but in one fragment you can add content to different sections 15:19:13 clog fragment for security that does breaking, should have 2 headers 15:19:40 it took me a minute to translate `clog` into `changelog` 15:19:48 for example https://github.com/ansible-collections/community.general/blob/master/changelogs/fragments/23-hashi-vault-lookup-refresh.yaml 15:20:18 a question about formatting, doies it make sense to separate 'engine' from 'plugins' ? 15:20:25 thanks felixfontein 15:20:29 back from my telehealth call. I'm going to play catchup and maybe add some infos for the meeting minutes... ignore my noise in the middle of the current conversation 15:20:47 felixfontein: Is your fork (ansible-changelog) feasibly usable for -base for 2.10? i.e. can we just switch over to it and remove the current generator and avoid having two versions of it like you said? 15:20:53 samccann: i'll trade you a jackhammer, 2 cranes and a cement truck 15:21:02 heh 15:21:05 relrod: right now it produces exactly the same changelog .rst file for stable-2.9 15:21:11 (if ansible-doc wouldn't crash :D ) 15:21:16 you live too civilized bcoca... no sidewalks here! 15:21:25 felixfontein: what crashes? 15:21:34 that is backward compatibility in action! 15:21:34 felixfontein: cool 15:21:40 samccann: it is NJ, 'civilized' is very subjective term 15:21:50 bcoca: broken module docs that can't be exported as JSON; see my issue/PR you've already commented on ;) 15:22:08 ah, k, thought was new issue, serialization will be fixed 15:22:26 relrod: right now it creates the same .changes.yml file for ansible-base (well, depending on the config it can be called differently) which includes the same info as for stable-2.9, but that can be adjusted 15:22:29 even if it fails, im adding new capture to avoid the TB and give 'nicer error' 15:22:50 relrod: my aim is to have a generator which works for both ansible-base and collections, and which can be used to create a unified changelog for ACD 15:24:27 * relrod nods 15:24:32 #info the collection changelog .yam format (looks like this: https://github.com/felixfontein/community.general/blob/changelog-1.0/changelogs/changelog.yaml as opposed to ansible 2.9 one: https://github.com/ansible/ansible/blob/stable-2.9/changelogs/.changes.yaml) contains the contents of the changelog fragments, and contains more info on the new plugins/modules 15:24:54 #info 2.9 yaml only contains their names, collection one contains also short desc and namespace 15:25:25 woudl ansible/ansible be interested to use the same format for .changes.yml as the collections will be using (from 2.10 on), i.e. inlined fragment content and extended plugin/module info? that would be the best I think :) 15:25:47 #info would be good if the formats could get more aligned (I don't mind the changelog fragments being separate files, but the missing additional info on the plugins is painful) - would also be good if there aren't 2 different changelog generators, but one that can handle both ansible-base and collections. 15:27:18 felixfontein: yeah I think having the formats aligned would be ideal. We probably will need an intermediate thing that "inlines" the fragment contents though, since we want to keep the fragment files (having all changelogs in one file is painful with git merges) 15:27:20 relrod: the idea for the .yml format for collections is to also specify an ancestor for a changelog (like the ancestor for ansible's 2.10 changelog is the 2.9.0 changelog entry from branch stable-2.9) 15:27:31 #info possible categories in the changelog - Security Fixes, Breaking Changes/Porting Guide, Major Changes, Minor Changes, Deprecations, Removals, Bugfixes - will need clear guidance on what goes in each category. 15:27:58 relrod: that's needed if collections use similar release branching as ansible/ansible does right now, and when a ACD changelog covers collection versions over multiple branches 15:28:32 relrod: right now there is an option `keep_fragments` which allows to keep the fragment files; I've used that for the collection .yaml files I've created so that the diff doesn't include 100 deleted files ;) 15:29:03 relrod: felixfontein i think a 'final product' is easier to deal with than trying to mandate intermedate fragments, otherwise we are forcing collecitno workflows and i don't think we can do that for all of them (think partners) 15:29:36 that the core tool is avaialble to commuinity collecitons i think is great, but i dont expect all to use it and i specially dont expect partners to do so 15:30:08 so 'requiring' a changelog.yml seems the easiest way to get the data and not having all to conform to a specific workflow/tool 15:30:15 bcoca: they don't have to; either they provide their changelog in the same .yaml format, or they will just get links to their changelog in the ACD changelog 15:30:27 The process is - use the new tool, or give us the changelog.yaml to match, or give us a link to whatever you have for your collection changelog 15:30:34 bcoca: I'm just saying, for -base, we will still want to be able to use fragment files. If those ultimately get imported into one "intermediate" yaml file that felixfontein's tool can parse, then that's fine. 15:31:09 relrod: understood, but I explained all of the above, cause this is not something we can ensure all collecitons follow and need to keep that in mind 15:31:11 relrod: for community.general (and c.crypto and c.network) also fragment files are used; the generator puts them into the .yaml file when a release is made 15:31:48 ^ would be a lot nicer if they did, since it makes contributing to multiple collections easy .. its a PITA to learn 10x rulesets if you contribute to 10 collections 15:31:58 definitely! 15:32:06 though there are some collection who will probably use reno 15:32:21 at least during the dicussions some people vocally advocated for it 15:32:22 yes, and I suspect if we offer a reasonably easy option, many collections will use it 15:32:22 felixfontein: cool. So where does the plugin/module description/namespace come from? 15:32:24 but i know better ... put 5 devs in a room and you get 7 opinions on what repo workflow should be 15:33:12 and yes, ++ for having simple tool they CAN use, relying on dev lazyness is always a good strategy 15:33:24 yes, but when those same 5 devs are presented with the choice to either adopt an existing tool with decent documentation or write their own, often they use the existing tool 15:33:30 relrod: `ansible-doc --json --metadata-dump -t `, but that's kind of deprecated since `--metadata-dump` is only for internal use and doesn't work for collections 15:33:30 heh, exactly bcoca 15:33:48 acozine: not all . sadly, not all, there are those that ALWAYS need to reimplement the wheel 15:33:53 relrod: I've written some code to "emulate" the output of that with `ansible-doc --json` and `ansible-doc --json --list` 15:34:14 relrod: https://github.com/felixfontein/ansible-changelog/blob/master/ansible_changelog/plugins.py#L23-L82 15:35:26 felixfontein: ok. I'm just trying to figure out what all we need to change in -base to use the aligned/unified format. It sounds like "not that much" 15:35:48 relrod: the aim is to have to change essentially nothing 15:35:55 except running another tool than the current one 15:36:00 cool :) 15:36:02 +1 here then 15:36:36 felixfontein: thank you for your work on this 15:36:44 felixfontein++ 15:36:44 #info goal is that ansible-base would not have to change anything to start using the new changelog tool 15:36:46 relrod: yw 15:37:04 relrod: I guess it would make sense to eventually move the code to a more "official" repo though (and clean it up some more) 15:37:10 * relrod nods 15:37:30 we went from managing docs/clogs for 1 project to doing for 30+ .... fun! 15:37:43 I think it doesn't make sense to put it back into ansible/ansible, since having another executable installed by ansible-base that isn't helpful for all but a few developers isn't a good idea IMO 15:37:54 bcoca: indeed! 15:38:10 felixfontein: I don't think I have access to create repos, but I agree when you're ready/willing we should move it to ansible/ 15:39:01 I'll do some more cleanup then and move the ACD hacks somewhere else 15:39:55 thanks felixfontein relrod bcoca samccann 15:40:13 chat can continue but I'm going to officially end the meeting 15:40:26 #endmeeting