14:10:40 <gundalow> #startmeeting AnsibleFest Austin: Contributors Summit - Galaxy Part 2
14:10:40 <zodbot> Meeting started Thu Oct  4 14:10:40 2018 UTC.
14:10:40 <zodbot> This meeting is logged and archived in a public location.
14:10:40 <zodbot> The chair is gundalow. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:10:40 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:10:40 <zodbot> The meeting name has been set to 'ansiblefest_austin:_contributors_summit_-_galaxy_part_2'
14:11:13 <gundalow> #chair ryansb chouseknecht ryansb newswangerd
14:11:13 <zodbot> Current chairs: chouseknecht gundalow newswangerd ryansb
14:11:32 <newswangerd> can't hear house too well
14:14:08 <bcoca> cant hear anything
14:15:16 <chouseknecht> We're talking about the role arg spec work that alikins did:   https://github.com/ansible/ansible/pull/44983
14:17:12 * ryansb can hear
14:17:36 <bcoca> are they speaking right now?
14:17:44 <ryansb> now are
14:17:47 <bcoca> ah, now i heard tim
14:18:05 <ryansb> cool
14:19:23 <tima> Sorry we were doing AV things.
14:31:34 <tima> https://github.com/ansible/ansible/pull/44983
14:32:13 <tima> That is the PR for the ansible-role POC @alikins is talking about ^^^
14:32:27 <tima> The objection: https://github.com/ansible/ansible/pull/44983#issuecomment-424400520
14:32:43 <tima> (Or at least the main one he is referring to)
14:34:21 <sdoran> I'm working on this for 2.8 (pulling arg_spec out of AnsibleModule), taking into account the work done by alikins and sivel as well as discussions with abadger1999 and nitzmahone.
14:35:27 <sdoran> So I'm hoping to unify all these ideas and get something working that meets these needs.
14:42:18 <alikins> https://github.com/ansible/ansible/compare/devel...alikins:argument_spec  is the 'arg spec in its own py module' branch. sivel has one as well at...
14:51:38 <bmbouter> is there a link to this file calssifier code?
14:51:39 * bmbouter is curious
14:51:53 <bmbouter> classifier*    :)
14:55:36 <bcoca> sounds like it would simplify galaxy code if you can rely on pulp already providing the tests/scores
14:56:16 <gundalow> For those attending remotely, are you managing to follow along OK? Please do keep your ideas coming.
15:00:13 <bcoca> we have a lot of overlap with docs meeting on the argspec topic
15:06:51 <ryansb> Counterpoint on "you have a score of [low number] and no commits in X years"
15:07:33 <ryansb> that role may actually work great and not have needed changes. For example, I have roles for setting up my desktop machines that have been stable for 1.5-2 years since I wrote them
15:07:59 <bcoca> same, my 'tidy' role really has not needed updates, some things are just simple and stable
15:08:14 <tima> this is an AND condition though.
15:08:16 <bcoca> my oracle-java one ... well, that needs changes every year
15:09:15 <bcoca> i have no tests, probably violate all 'best practices'
15:10:04 <willthames> bmbouter - https://github.com/willthames/ansible-review/blob/master/lib/ansiblereview/__init__.py#L177-L208
15:10:24 <bcoca> some of those violations could be spurious, i.e using shell: apt cause apt module does not suport a specifc switch (--auto-fix)
15:10:45 <willthames> come on bcoca, just use warn=no
15:10:56 <bmbouter> willthames: ty
15:11:33 <alikins> how does one run the scoring from cli?
15:11:51 <willthames> although it would punish you for that because generally it should be `command: apt` :)
15:11:53 <bcoca> willthames: so the scoring relies on warning? ... so if i warn=no all my shell usage?
15:12:10 <bcoca> willthames: apt -- | grep
15:12:14 <willthames> yeah, ansible-lint would assume you know what you're doing
15:12:27 <bcoca> i can see that as a way to game the system
15:13:00 <willthames> there are always ways to game the system
15:13:34 <tima> leave it to @geerlingguy ;)
15:14:22 <sdoran> Well he revealed all his secrets now, so the gaming will increase. :)
15:15:37 <alikins> don't remove existing roles ever... what gets 'curated' into a 'good for stable ansible' set that is used for search is different
15:15:47 <ryansb> +1
15:15:56 <alikins> *cough*role distro*cough*
15:16:30 <bcoca> repeat answer from IRC
15:20:07 <bcoca> shell: rm -rf /
15:20:17 <bcoca> ^ i would check even outside of 'python'
15:22:41 <dmsimard> Bandit for python security things: https://github.com/PyCQA/bandit
15:25:51 <robertdebock> That "shell: rm -rf /" - https://github.com/willthames/ansible-lint/blob/master/lib/ansiblelint/rules/UseCommandInsteadOfShellRule.py ;-)
15:26:18 <bcoca> shell: '{{cmd}} {{options}}'
15:26:25 <bcoca> vars: cmd: rm options: -rf
15:27:09 <bcoca> when you have such dynamic posiblities, linting is limited to what it can handle
15:29:25 <alikins> For say, https://galaxy-dev.ansible.com/geerlingguy/php-mysql the survey could learn a 'flag this as harmful' etc
15:29:52 <bcoca> <click here to report a problem with this content>
15:30:44 <alikins> yeah, at some point there will have to be someone handling a issue/flagged/takedown/etc queue
15:48:08 <abadger1999> alikins: https://gist.github.com/abadger/3edc108f5a5b88c1a9a46c4869d778fd#file-core-rst
15:48:19 <abadger1999> alikins: sivel's branch is linked from a comment in there
15:48:23 <abadger1999> sdoran: ^ as well
15:49:16 <sdoran> 👍
15:57:04 <alikins> https://github.com/ansible/ansible/compare/devel...sivel:categorization-of-basic  to be explicit
16:10:18 <geerlingguy> well hello there, I was in the wrong channel.
16:10:22 <bcoca> [12:08] <geerlingguy> qq - how will we manage multiple versions of a role for different projects?
16:10:23 <bcoca> [12:09] <geerlingguy> Can I still install namespaced packages in a project subdir?
16:10:26 <geerlingguy> ^^
16:10:29 <bcoca> he, was pasting for you
16:12:03 <geerlingguy> That's my main concern—because I often have different projects using different versions of the same role.
16:12:52 <geerlingguy> One other concern: Is the idea a collection could reference a separate role from elsewhere? E.g. I have a "webserver" collection, and it has some roles etc, but it also pulls in a separate "geerlingguy.apache" role or "geerlingguy.nginx" role so I don't have to duplicate those roles across different collections?
16:13:47 <geerlingguy> Without that features, collections can be fun little demo thingies, but there's no ability to 'compose' collections and that would be worthless in all but a couple tiny demo projects for me.
16:17:38 <bcoca> runtime should be establish in play/role
16:18:58 <alikins> imo, fine grained collections (ie, say, 1 role per collection) should be the preferred approach
16:19:06 <bmbouter> geerlingguy: I can see the need to compose. where/when would you want to compose what makes up a collection? locally/on-galaxy/both?
16:19:14 <geerlingguy> locally, mainly
16:19:21 <alikins> but collections holding multiple collections/plugins/modules etc a possibility
16:19:29 <bcoca> also to note, collections are not 'ansilbe runtime concept' its a 'installer concept'
16:19:30 <geerlingguy> it would be cool to be able to make collections of collections on galaxy, but that sounds like it could be a little harder
16:20:00 <bcoca> ansible still just uses modules/plugins/roles as it does today
16:20:09 <bcoca> its the installation that collections make easier
16:20:29 <geerlingguy> okay, I think that was a little of my confusion, it sounded like you would use like "one collection" for a playbook
16:20:58 <bcoca> you could structure your collections per playbook, but that is not a requirement
16:21:01 <geerlingguy> So for the most part, my roles would become "a collection with one role"
16:21:07 <bcoca> probably
16:21:16 <geerlingguy> and that's fine by me :)
16:21:16 <bcoca> or if you have closely related roles/modules/plugins
16:21:22 <sdoran> Do collections allow for multiple versions of the same role/plugin/module to co-exist on the same system?
16:21:26 <geerlingguy> yeah that makes sense
16:21:47 <bcoca> sdoran: that is not inherit of collections but of the installer and how ansible executes
16:22:04 <tima> @sdoran: multiple versions of the same role in a collection is not handled
16:22:06 <bcoca> so mazer/ansible need to decide how to handle versions, collections just specify 'which version'
16:22:12 <geerlingguy> and yeah, back to the multiple version thing ^^ just being able to have playbook A use geerlingguy.apache 1.2.3 and playbook B use 3.2.1 is important (easy enough if mazer install can install locally instead of under ~)
16:22:19 <alikins> geerlingguy: re collections of collections...  I tend to agree. Some variation of that idea may be required to make it so a single git repo can be a collection and a trad style role simultaneously
16:23:00 <geerlingguy> alikins: yeah, for my own roles I always support n-2 Ansible releases (unless impossible), so I don't like the idea of having to fork things for a year or something weird like that
16:23:51 <sdoran> Ok. That's what I was wondering.
16:24:13 <sdoran> If you have a playbook that just says "apache" and you have "apache" in five different collections, which one gets used?
16:24:21 <sdoran> Namespacing roles makes it clearer.
16:24:24 <bcoca> collections use namespaces
16:24:44 <bcoca> the problem is 'non namespaced apache' module, it will use 'first in path'
16:24:50 <sdoran> So it'd be "collection1.apache".
16:24:54 <alikins> sdoran: only a single version of a fully qualified role/module (ie, alikins.my_collection.my_module).  But each namespace / collection could have different versions (ie, alikins.my_collection_old.my_module)
16:24:55 <sdoran> Right, that can be problematic.
16:25:02 <bcoca> so you cannot use 'apache' with a collection, always 'name.apache'
16:27:50 <tima> @alikins i forget, did we say that you could access a module from a collection without the namespace?
16:28:09 <tima> oh @chouseknecht just touched on that as i was typing that. ^^^
16:28:10 <geerlingguy> that seems like it would be dangerous/not recommended (even if possible)
16:28:51 <sdoran> I wonder if it'd be useful to have the collection namespace be an inventory var. So you could "switch" which set of roles/plugins/modules you're using based on inventory.
16:29:15 <sdoran> Because otherwise you could end up with plabooks that are all the exact same, just with a different collection namespace.
16:29:28 * geerlingguy will be busy writing a script to go back and run `mazer build` and deploy for each git tag in his repos :P
16:30:09 <geerlingguy> Another quick question—any load testing done or planned for when we start serving archives from Galaxy instead of GitHub?
16:30:14 * sdoran wants to see geerlingguy's scripts for managing his mountain of roles :)
16:30:37 <geerlingguy> I imagine it will be a lot of bandwidth
16:30:40 <tima> @geerlingguy meaning the galaxy hub itself?
16:30:42 <geerlingguy> yeah
16:31:19 <tima> yes that is something we are considering. galaxy moved to red hat's openshift online cluster recently.
16:31:47 <geerlingguy> Good; I just don't want the launch of hosted archives to be tainted by super slowness or outages :)
16:31:51 <tima> we are also going to be working closely with the Red Hat Insights ops team.
16:32:09 <tima> geo issues are being discussed and are "on the board."
16:32:52 <alikins> tima: let me verify, it's been a while but I think it can use short unambiquous names for roles
16:33:02 <alikins> but thats hardly set in stone
16:33:03 <geerlingguy> yeah one nice thing about github-hosted-releases is they have their own CDN with good performance in india, asia, australia, etc.
16:33:53 <geerlingguy> It would be nice to have something like, in the manifest file, have "${GIT_TAG}" or something so when it builds from Travis, Circle, etc., it would just look at the current git tag/commit ref or something?
16:34:08 <bcoca> people still use svn .. but that is their hell to bear
16:34:16 <geerlingguy> otherwise I'll have to script that process as part of my CI build
16:35:17 <geerlingguy> svn+-
16:35:20 <geerlingguy> at least it's better than cvs
16:35:29 <bmbouter> geerlingguy: pulp does scale testing regularly and it would actually be the thing serving back the bits
16:35:32 <bcoca> geerlingguy: lowbar
16:35:35 <bmbouter> re: bandwidth
16:35:39 <bmbouter> re: scalability rather
16:37:02 <geerlingguy> pulp is an implementation detail and it sounds like it'll work out fine, I'm more worried about the transition than anything else
16:37:19 <geerlingguy> OH! Forgot one question—how is auth handled?
16:37:21 <tima> so am i @geerlingguy
16:37:40 <tima> will get to your auth question in a second.
16:37:44 <geerlingguy> like how do I push my artifact and make sure it's me pushing my artifact? GitHub SSH pubkey synced to galaxy?
16:37:45 <geerlingguy> Thanks
16:38:40 <alikins> tima: the bits around https://github.com/ansible/ansible/compare/mazer_role_loader#diff-57063b91d9c7fc66f5b6f2001cb4237eR59
16:39:19 <bcoca> i hope not, if we allow collections to 'cross depend' now we have to be very careful with deletions
16:40:10 <geerlingguy> I would imagine deletions would not be allowed, except for like "this is a national nuclear launch code stored in the collection" or something insane like that
16:40:34 <bcoca> i would only delete for same reasons we retire a role, severe violations
16:41:31 <bcoca> i would flag as 'unsecure' or 'deprecated' or 'reallydontdothis' so installer can warn
16:44:02 <maxamillion> the suggestion is to not allow deletion of things in galaxy but to "hide" them so they don't show up in the UI or otherwise "advertise" that release of the role/content but if you explicitly ask for that version because you know it's there and you depend on it then you can still install it because it will exist
16:44:10 <maxamillion> (transcribed for posterity)
16:44:20 <geerlingguy> having a deprecated checkbox++
16:44:25 <geerlingguy> like for my tomcat6 role :P
16:44:33 <geerlingguy> java--
16:44:35 <bcoca> like for tomcat in general
16:44:37 <newswangerd> it's in dev. The feature allows people to deprecate repo's not individual releases
16:45:07 <geerlingguy> I'd be okay with repos only; I assume old releases are 'deprecated' anyways, because they're old :)
16:45:55 <geerlingguy> (and relatedly, signing—don't expect that right away, but someday maybe)
16:46:59 <bcoca> checksums and signing
16:47:12 <bcoca> checksums are easy part
16:47:36 <bcoca> installer should verify
16:47:49 <tima> @alikins ^^^
16:48:00 <bcoca> also, they need to verify 'source' so https is normally enough, but if we want to also 'chain sign' the distributor
16:48:27 <bcoca> i.e galaxy key as we have deb/yum repo keys
16:49:09 <geerlingguy> just having hosted archives is like 1000x better
16:49:12 <bcoca> no blood tests to verify that statement
16:49:16 <maxamillion> geerlingguy: +1
16:49:23 <geerlingguy> signing adds a little more value, but not nearly as much as getting off github releases
16:49:37 <bmbouter> +1
16:49:44 <geerlingguy> being able to update without spending 30 minutes doing so for every playbook +1
16:49:45 <bcoca> geerlingguy: also, now archive is controlled centrally vs each user being able to remove his repo
16:49:51 <geerlingguy> yeah
16:52:15 <gundalow> Food is ready
16:56:07 <alikins> signing artifacts and verifying artifacts before they are installed (ie, 'rpmsign' / gpg) is on the TODO list. As is client storing the pre file checksums somewhere (in the theoretical mazer collection db) for use by mazer to verify installed files (ie, `rpm -Va` style feature)
16:56:16 * alikins tries to catch up scrollback
16:57:52 <gundalow> We are breaking for food now, back in an hour
16:58:15 <gundalow> #endmeeting