15:12:14 #startmeeting core public irc meeting 15:12:14 Meeting started Thu May 21 15:12:14 2020 UTC. 15:12:14 This meeting is logged and archived in a public location. 15:12:14 The chair is bcoca. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:12:14 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:12:14 The meeting name has been set to 'core_public_irc_meeting' 15:13:07 o/ 15:13:09 james ? 15:13:21 good morning 15:13:41 #topic https://github.com/ansible/proposals/issues?q=sort%3Aupdated-desc+collection 15:13:56 @thaumos ^ 15:14:02 * thaumos waves 15:14:27 https://github.com/ansible/community/issues/536#issuecomment-630999040 15:15:12 Collections - what are its goals and features? 15:15:33 I kind of know "through the grapevine" but I can't find any design documentation, nor a proposal for it 15:15:46 is the Proposal still the way big changes are made to ansible? 15:16:00 s/made/agreed upon and-or discussed/ 15:16:03 not necessarily 15:16:42 it is ONE way 15:17:08 it's a facility for external contributors to lay out plans for big changes they want to make, and get feedback before working on code 15:17:57 also for internal contributors when looking for feedback from the community 15:18:08 so for plans made by "internal contributors", is there meant to be any transparency or feedback from the external contributors? 15:18:51 or is it meant to be more of a "throw it over the wall" type of Open Source, with occasionally accepted community contributions? 15:19:23 We do not necessarily ask for community feedback on all features or funcitonality we plan 15:19:32 neither and both 15:19:40 so, I guess it's a specific, "what's the documented design purpose and goals of Collections", but brings about a more general "how is ansible developed in relation to its community contributorrs" 15:19:57 Also, this blog post is largely the genesis of collections from an announcement perspective: https://www.ansible.com/blog/the-future-of-ansible-content-delivery 15:20:06 thx for pasting that sivel 15:20:14 I was just about to as well. 15:20:25 took me a little longer than anticipated to find it :) 15:20:47 @cyberpear, collections aren't a new thing we're just implementing now. So the discussions for all of this date back to more than a year ago. 15:21:00 yes, I'm aware it's been in progress for some time 15:21:11 What pre-dated even the Collection itself was discussions around the "ansible installer" as it was called before that. 15:21:21 but for the 2.8 timeframe, there was no public mention of it at all, and I only became aware of it because it was blocking the 2.8 release 15:21:54 and back during the 2.8 cycle, I was reading all the chat in #ansible-devel channel 15:22:06 (I don't have enough bandwidth for that in the past few months, though) 15:22:14 cyberpear: its been something that has been brewing for 6 years, 2.8 was just hte moment we decided, we cannot go on like this w/o it 15:23:03 why all the secrecy? 15:23:15 I am sorry, but there has been no secrecy around this. 15:23:24 We have and continue to communicate our goals with this. 15:23:28 https://github.com/ansible/ansible/projects/30#card-13708579 15:23:40 That is from the project board linked in the 2.8 roadmap 15:24:03 Maybe, let's step back for a second. What is the *real* goal of this discussion? 15:24:17 What do we hope to accomplish here? 15:24:49 I'm not sure convincing you that collections are beneficial is a worth while goal 15:25:49 @cyberpear We've communicated this aggressively in several channels and at every Ansible Contributor Summit for the past year. 15:26:07 I guess I'd like to see better documentation of "the ideal" trying to be achieved with the collections concept 15:26:09 more than that gregdek, and thanks for chiming in. 15:26:17 are there internal design documents that could be published? 15:26:35 cyberpear: what would that help us achieve at this point? 15:26:35 I assume the driving force is "too big backlog" on ansible/ansible 15:26:55 Did you read the blog posts that thaumos pointed you to? 15:27:07 A lot is in flux at the moment with regards to documentation, the ansible repo, collection structures for the community, etc. So of course, the eventual goal is to document this all approrpriately. 15:27:10 cyberpear: no, that is a side effect 15:27:27 cyberpear: read blog, it states the goals 15:27:51 https://www.ansible.com/blog/thoughts-on-restructuring-the-ansible-project 15:27:55 @cyberpear, the main driving force for this originally was to enable our content creators a means to provide content at a much more rapid pace than what the core framework could provide on it's own. 15:27:56 I'll re-read it. I didn't read it today. 15:28:27 but I definitely read each when it was published 15:28:35 cyberpear: I'm still interested to know what we hope to accomplish with this discussion. What is the real goal/puprose? 15:28:52 What outcome would you like to see as a result? 15:30:07 sivel: 2 parts. 1. better goals documentation, both conceptual and on-the-ground technical of the Collections world, though at this point, it'd likely be "as-implemented" rather than "as-designed" docs 15:30:27 2. better transparency for changes going forward, with more opportunity for community input 15:30:41 So yes, like I said earier. We have a lot in flux. The goal is to have documentation around this. 15:30:46 cyberpear: i can show you initial docs, they resemble little of what we have, cause it is a continious process with community and user feedback 15:31:07 Secondly, I don't know how much more transparent we could have been about this subject. As gregdek and I and others have said, we have communicated a lot about this. 15:31:42 From my perspective, the post I linked is clear about the goals, and pretty much laid out the conceptual roadmap that we've followed since. 15:32:06 okay, I'll re-read those. 15:32:08 also, its not only core team that wrote this, many parts are being written by community members 15:32:25 others are just influenced by them 15:32:26 I'm also concerned that collections are by-and-large... too large 15:33:00 well, depends on each collection and there is much debate on that subject, ... and that will probably change in the future as feedback is incorporated 15:33:01 in the roles world, I could download use a single role, or a single module distributed as a role 15:33:18 you can still do that will collections 15:33:19 now in the collections world, I'm forced to download the entire collection instead of just the part I need 15:33:35 Roles will continue to be supported. 15:33:43 also, not really true what you stated 15:33:51 roles are getting migrated from repo-per-role to monorepo-per-collection-of-100-roles 15:33:52 you needed to download the plugin and 'everything it depends on' 15:34:02 sure, there's dependencies 15:34:20 and finally w/ 2.10, the meta/requirements.yml is a thing that's honored by the tools 15:34:32 that still works the same, the only big diff, is from role you could throw away structure and use library/ with collection you still need the structure (just not all the other content) 15:35:38 i think we (the core team) has as much control over the size and shape of all collections as much as the rpm package inventors have over all the rpm packages out in the world. we build a framework and it's on the framework users to define what they want to put in their widget 15:35:51 but it's a lot harder to convince a corporate security departement, (either for software I'm shipping, or just for internal use) that my dependencies have been audited -- it's easy to audit a single role... much harder to audit a collection of 100 roles 15:36:00 sure, but you do push users toward one way vs another way 15:36:11 the galaxy.ansible.com is pushing people away from publishing roles 15:36:15 cyberpear dont use the collection with 100 roles ...use one with 1 role 15:36:16 and toward collections 15:36:25 not that roles are all same size and tiny 15:36:31 @cyberpear, but to jctanner's point; it is up to the content creators/curators whomeever that is creating that collection to make it that way. 15:36:33 i've seen roles with 100s of task files and plugins 15:36:50 Not every collection will have 100s of roles or whatever else could be in it. 15:37:09 The same structure could exist as you are suggesting. There could be a collection with a single role and a plugin. 15:37:13 collections does not alter the scope/size of content, just like roles , its up to the author 15:37:16 The directory structure is just changed. 15:37:30 specific to the collections that have been split off of core, there's now copied libraries and/or module_utils, which makes maintenance harder for those components -- which copy is canonical? 15:37:40 collections just make it a LOT easier to distribute and consume that content (no need to ikmport/execute role tasks for access to a plugin) 15:38:02 cyberpear: good question and a problem the maintainers will have to solve 15:38:12 the directory structure is quite annoying, and I haven't checked recently if the stupid `collections/ansible_collections` structure is still required 15:38:19 but that is not part of what the colleciton design is meant for, just what content authors chose to do 15:38:28 cyberpear: it is 15:38:37 The way that the community.general collection has been structured is a result of mustering people together as resources to curate together. It is not meant to be the standard by which everyone operates. 15:39:18 (what I do is set collections_paths=./, and that puts collections in ./ansible_collections/namespace/collection). But I'm still planning on using roles for my playbooks outside of collections, so they can be in ./roles/[rolename] without namespacing) 15:40:00 (otherwise for simple local roles that I don't share on Galaxy, they'd be in ./ansible_collections/example/local/roles/[rolename] which is just not fun to navigate when working on playbooks ;) 15:40:31 the one big benefit I see for ansible content creators is distribution of playbooks in a standard way, but `ansible-playbook namespace.collection.playbook` apparently isn't planned for 2.10 15:41:01 There is a lot still to be implemented. 15:42:07 This isn't done. However, we do have to pick and choose what we can do with the resources that we have. Right now, it felt like all focus should be on getting content into collections. That effort required focus from all the team. 15:42:24 After this release, we will have more to do with collections. 15:42:25 cyberpear: i have a PR that might make it, its not not planned, we just are not going to hold the release for it 15:42:42 I think that in a collections, world, it'd be much more feasible to implement something like Fedora's Change Process for all changes, given that the core is much smaller. 15:43:01 https://docs.fedoraproject.org/en-US/program_management/changes_policy/ 15:43:03 core wont handle content outside ansible/ansible 15:43:13 right 15:43:16 but for core itself 15:43:42 and maybe for ACD 15:43:48 ^ is that still what we're calling it? 15:44:02 Bear in mind that it took Fedora a decade to get to that change management process. 15:44:06 also note that like roles, collections will mainly be 3rd parties, we will have no say on what/how they are operated or distributed .. nor do we want to have a say 15:44:27 And no, ACD is now just Ansible. 15:44:38 yes, I well understand that third parties do whatever they want 15:45:00 so ACD->"Ansible" and the ansible/ansible -> "Ansible Base"? 15:45:01 and ACD is not under core, it has it's own team 15:45:19 * cyberpear confused by changing terminology 15:45:24 right 15:45:34 u are not alone, i just found out about the change as you did 15:45:42 That is correct. 15:45:50 I brought up separately that ACD should have its own meetings, and I believe they're now scheduled for Wednesdays 15:46:28 "Ansible Base" or "Ansible Core", or "Ansible Base maintained by the Core Team"? 15:46:36 ansible-base is ansible/ansible 15:47:06 effectively ansible-base is still "ansible", but from a packaging perspectice, "ansible" is what we used to call ACD 15:47:46 project vs package 15:48:36 can we make a glossary? 15:49:05 ansible/ansible is the ansible-base package maintained by Core Team 15:49:24 ACD (does it have a repo to track what goes in?) is the ansible package maintained by the Community Team? 15:49:31 * bcoca tempted to drop ansible=base for ansible-core to minimize num of names 15:49:58 I think a lot of these conversations are actually better for the Wednesday meeting, where we can expect your help :) 15:50:00 cyberpear: ACD doesn't really have code. It has some build scripts, and the community team is responsible for packaging "ansible" 15:50:29 the 2.10+ ansible package, only contains bundled collections, and has a dependency on ansible-base 15:50:38 it has some code or machinery that determines which parts get pulled in, though 15:50:49 That is correct. The build code. 15:51:11 The goal of which is to provide continuity to users who want to continue to consume batteries-included Ansible. 15:51:33 https://github.com/ansible-community/ansibulled/ 15:52:39 is batteries-included version expected ever to go away? 15:53:00 (will there be stats on who installs the "ansible" package vs just "ansible-base"?) 15:53:47 too early to answer that first question 15:54:12 that will depend on stats, which I assume we'll have via some method 15:54:16 I realize some of these might be better questions for the newly established ACD meeting... I hope it gets some turnout... 15:54:33 I don't think there is a plan to get rid of the batteries included model. 15:54:35 "ever" is a long time. 15:54:53 There are no immediate plans to get rid of the batteries-included distro, certainly. 15:54:54 right, ansible might be replaced by opsmop, some day :P 15:55:02 lmao 15:55:30 * bcoca suspects cfengine will be only one left 300 yrs from now 15:56:12 cyberpear: consider this .. with collections you can create your own ansible package customized the way you want 15:56:14 ansible-aws 15:56:17 ansible-azure 15:56:23 asnible-nomysqlbuteverythingelse 15:57:58 * cyberpear typing 15:58:36 so I'd propose: 1. glossary of terms in the New World Order (and simplification/deconflict as appropriate) 2. do more to involve the community in /designing/ future changes (not just implementing) (ansible-devel mailing list?) 3. better docs for /design/ of collections, even after the fact 15:59:02 Sounds like an excellent agenda for our Wednesday meeting. 15:59:15 cyberpear: define 'do more' .. since the process has alwasy invovled the community 15:59:41 unsure what else you want there, communty feedback was incorporated, community devs are contributing 15:59:41 send proposed changes to the ansible-devel list before they're ultimately decided upon 16:00:00 that was done 16:00:07 also blogs 16:00:29 and literally 6-7 yrs of conversations, this startted before IBM and even RH aquisitions 16:00:43 i remember greg and i discussing this at the 2nd ansible co all hands 16:01:00 and the same in the irc channel and MLs 16:01:18 bcoca: can you point me to the ML archive about collections? 16:01:28 not the 'name' but the concept, yes 16:01:49 read scrollback, at one point this was called ansible-installer (i believe thaumos mentioned) 16:01:53 it has gone by MANY names 16:01:56 (it's possible it was before my time, but I've been on there for probably 2 years or more) 16:02:37 apb ansible playbook bundle .. orb .. many many bad names ... 16:02:37 When I was hired in Dec 2017, we were discussing it then, and had been discussed far longer 16:02:48 indeed, changing the names makes it very hard to track a concept... especiallly if the introduction doesn't say "previously known as XYZ" 16:02:54 ^ and sivel (like myself) was hired from the community 16:04:05 but yes, at around the time I was hired, bcoca was using the name "ansible-installer" 16:04:12 well, we are over time and at this point i thnk we can close the meeting, to be continued in ACD weekly 16:04:38 seeing "ansible-installer", I would have glossed over it thinking, "that's what RPM is for" 16:04:44 "or pip" 16:05:01 but now that I know, I can trawl my archives 16:05:02 'content installer' .. yes, one of many reasons that name was dropped, also it was very confusing (does it install ansible?) 16:05:35 cyberpear: but name is the least of it, i probably have 100s of hours of discussion about the concept w/o naming it in this and #ansible-devel channels 16:06:00 a few posts in the ML, but i dont think we ever had a pure discussion thread on it until around 2.7/2.8 16:06:10 also .. lack of name .. makes it hard to pinpoint 16:07:07 I'd like to see more discussion on the mailing list since you can have threads related to a topic more easily, and better searching 16:07:27 (also, let's kick the "please help me use ansible" folks over to ansible-project list) 16:07:54 why? that is one of the reasons that list exists 16:08:09 to help people use ansilbe, ansible-devel is for those developing ansible or content for asnible 16:08:11 I assumed there was ansible-devel list for folks developing ansible 16:08:20 and ansible-project for announcements and end-user support 16:08:32 also ansible-announcements 16:08:37 also that 16:08:42 plase help me use ansible seems 'user support to me' 16:08:48 maybe we need a new list for development of ansible itself, not ansible content? 16:09:10 k, add that to next agenda, 10mins over time and need to close this one 16:09:17 considering original point discussed 16:09:19 #endmeeting