19:06:31 <jillr> #startmeeting ansible core irc public meeting
19:06:31 <zodbot> Meeting started Tue Feb  4 19:06:31 2020 UTC.
19:06:31 <zodbot> This meeting is logged and archived in a public location.
19:06:31 <zodbot> The chair is jillr. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:06:31 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:06:31 <zodbot> The meeting name has been set to 'ansible_core_irc_public_meeting'
19:06:42 <cyberpear> felixfontein: I've got an idea to fix your ansible_facts with interpreter discovery
19:06:56 <felixfontein> cyberpear: cool!
19:07:21 <felixfontein> hmm, there's food now at the meetup
19:07:25 <felixfontein> I'll try to look in again later ;)
19:07:38 <jillr> hehe, priorities  :)
19:07:53 <felixfontein> about https://github.com/ansible/ansible/pull/66389: I'm still looking for reviewers :)
19:08:18 <felixfontein> about https://github.com/ansible/ansible/pull/66961: probably needs reviewer(s) as well, I'd really like more sanity checks in ansible before modules are moved out
19:08:32 <cyberpear> I've been meaning to give that one a look
19:08:53 <felixfontein> about: https://github.com/ansible/ansible/pull/66920 I think the main question is how deprecation tests should work in the future, now that there isn't just ansible/ansible but also collections
19:09:03 <felixfontein> and now I'm afk a bit ;)
19:09:23 <shertel> I'll add 66389 to my queue
19:09:40 <jillr> skipping all of felixfontein's items on the agenda then..  :)
19:09:49 <cyberpear> wow, feb already!
19:09:51 <jillr> #topic https://github.com/ansible/ansible/issues/66582
19:10:20 <jillr> is pdumais around?
19:10:33 <sdoran> I was hoping to get input on this from other core folks like bcoca and sivel.
19:10:48 <sdoran> Or at least have a discussion around it.
19:10:55 * jillr reading
19:11:17 <bcoca> sdoran: for some definition  of task end
19:11:41 <bcoca> is it ended per batch? all hosts? when next tasks starts (strategy dependent)?
19:12:30 <bcoca> right now 'new task start' != 'old task' is what we use for 'task end' but for linear strategies, this does not apply for free/host_pinned
19:12:47 <bcoca> main reason event  doesn't exist, not easy to define
19:14:27 <sdoran> It seems they are after a way to show 'task X ran on Y hosts', which is a bit of an aggregation compared to what we have currently.
19:15:53 <jillr> I can totally understand what they're looking for and why folks would want it, I guess the question is how doable is it?
19:16:10 <sdoran> Would we accept a PR adding this since the author says they are interested in doing the work.
19:16:24 <sdoran> I just wasn't sure of all the implications and wanted broader discussion.
19:16:42 <cyberpear> sounds like a valuable addition, assuming not overly complicated
19:16:44 <jillr> for sure
19:17:12 <jillr> like all things, it will depend on the implementation I imagine
19:17:58 <cyberpear> maybe ask for a WIP PR to get early review
19:18:28 <sdoran> Sounds like it won't be easy to clearly define "task end".
19:18:36 <bcoca> sdoran: we have that data in play stats
19:18:43 <jillr> maybe give them a heads up that it may be complex based on what bcoca was saying but we'd be interested to see their approach?
19:19:08 <sdoran> So maybe this would be straightforward to implement.
19:19:12 <sdoran> Seems like we're ok with it?
19:19:33 <bcoca> dont think we need new event, but you can easily add that data to output on 'task_start' using same trick i did for printing banners
19:19:33 <sdoran> Enough to ask for a PR to review at least, as cyberpear said.
19:19:33 <jillr> I have no objections, I can see the value to users
19:20:37 * bcoca tempted to create 'template callback' to allow users DIY templates per event
19:20:38 <cyberpear> ^might be cool
19:20:43 * sdoran doesn't want the CVEs from that.
19:21:11 <cyberpear> `!unsafe`
19:21:12 <bcoca> sdoran: probably less than now
19:21:33 <bcoca> no lookups, you basically remove most issues with security bugs
19:21:39 <sdoran> Ok, I'll comment on the PR that we'd be willing to review a PR adding that new methed.
19:21:47 <sdoran> s/methed/method
19:21:51 <sdoran> We can move on.
19:21:53 <jillr> thanks sdoran
19:21:58 * cyberpear waits for open floor
19:22:02 <bcoca> i would say to advise using alternate like we do to print task headers
19:22:30 <jillr> I think we're skipping felix's items so,
19:22:30 <bcoca> instead of new method, since it can be done just in callback plugin w/o altering core
19:22:30 <jillr> #topic open floor
19:22:30 <cyberpear> https://github.com/ansible/ansible/pull/66071
19:22:31 <jillr> cyberpear  :)
19:22:41 <jillr> #topic https://github.com/ansible/ansible/pull/66071
19:22:51 <bcoca> any changes from last time?
19:23:10 <cyberpear> we discussed the service_mgr=systemd for offline cases previously, but there were very few core folks: https://meetbot.fedoraproject.org/ansible-meeting/2020-01-21/ansible_core_irc_public_meeting.2020-01-21-19.00.log.html#l-117
19:23:24 <cyberpear> gathering the `service_mgr` fact, specifically
19:23:29 <bcoca> then same issues as before, this breaks existing use cases, which is why we removed it in first place
19:23:51 <cyberpear> bcoca: as far as I can tell, it was never removed... I couldn't find that
19:23:57 <cyberpear> bcoca was worried about multiple init systems being installed and detecting the wrong one
19:24:50 <bcoca> ^ my workstation setup was done so i could reproduce those cases and why i made change, it was long ago so having touble finding reports (might have been in irc since we did a lot more 'out of github' fixes back then)
19:24:56 <cyberpear> would it be better if I detected "safe" distros to enable the detection, such as Fedora, RHEL, CentOS, SUSE, and Ubuntu?
19:25:25 * cyberpear hates special cases for the common case
19:25:48 <bcoca> probably, the issue is mostly people using containers as VMs and doing custom init that containers really are designed against and lie about
19:26:02 <cyberpear> Red Hat promotes it, though, so it's going to be more and more common
19:26:04 <bcoca> yes, but sadly ... its allspecial cases...
19:26:33 <bcoca> i wish, but i would not be that optimistic
19:26:56 <cyberpear> to run systemd in a container in a RHEL 8 box: `podman run -d ubi8-init`
19:27:37 <bcoca> before that 'nspawn' was 'the way' ... didnt really work out
19:27:52 * cyberpear still likes nspawn
19:28:00 <bcoca> not saying its bad tech, it just didnt take off/over
19:28:05 <cyberpear> for build time, it is most common for the systemd daemon to not actually be running, such as with ansible-bender
19:28:46 <cyberpear> then when the container is run as-above, the enabled systemd services are started w/in the container by systemd running in the container
19:29:18 <cyberpear> the `rhel6-init` container runs similarly, but instead runs dumb-init and activates SysVinit services that are enabled on disk
19:30:17 <cyberpear> does anyone else have any thoughts on the PR?
19:30:50 <cyberpear> If upstart or openrc are installed, those checks will be hit first and this code won't be reached
19:31:14 <bcoca> but sysvinit si the 'last', which was the problem case
19:31:16 <cyberpear> it's only reached for cases that would currently return `sysvinit` or the generic `service`
19:31:47 <cyberpear> bcoca: you said last time that the problem was upstart with systemd on the same system, didn't you?
19:32:18 <bcoca> no, mostly sysvinit was the issue, upstart/openrc could be issues but that depended on priority which is easy to fix
19:32:27 <bcoca> sysvinit is the 'fallback' so hard to put ahead w/o breaking other stuff
19:32:28 <cyberpear> https://meetbot.fedoraproject.org/ansible-meeting/2020-01-21/ansible_core_irc_public_meeting.2020-01-21-19.00.log.html#l-144
19:33:05 <bcoca> that was one example, but sysvinit is always the problem child
19:33:09 <cyberpear> bcoca: so it comes down to find a case where a sysvinit system has `/sbin/init` as a symlink to `/usr/lib/systemd/systemd`
19:33:31 <cyberpear> Is there any sysvinit system where /sbin/init isn't itself a binary rather than an symlink?
19:33:42 <bcoca> its not as much symlink as to systemd is not actually runnable
19:33:56 <bcoca> it CAN be the symlink but still not usable
19:34:13 <bcoca> if systemd  was working on the system, it 'should' be able to handle systemd also
19:34:29 <bcoca> er sysvinit .. unless its jessie which had issues .. but at thsi point jessie should be minor problem
19:34:47 <bcoca> ^ really cannot wait till debian jessie goes away, that systemd implementation was just bad
19:34:55 <cyberpear> jessie is nearing end of life, but I have tested it there and seen no issues
19:35:30 <bcoca> it has known issues with init scripts and were flagged as 'wont fix' so until it is gone,will stll be a pain
19:35:43 <bcoca> ^ they were 'fixed' in subsequent implementation in next version
19:36:06 <bcoca> so while your detection might work, the service will still be a problem
19:36:25 <bcoca> but as i said, less worried about jessie, install share seems to be down a lot at this point
19:36:51 <cyberpear> the PR under discussion is purely about the `service_mgr` fact
19:37:19 <bcoca> which influences 'services' action plugin and how services will be attempted to be used
19:37:28 <cyberpear> true
19:37:50 <bcoca> so its not about the fact itself being wrong (or not 100% right since there is more than one cat in the pen) .. but the results of attempting to use that info
19:38:13 <bcoca> also why 'use=' is part of service, cause sometimes we just cannot win
19:38:25 <bcoca> and user just has to tell us 'the right way'TM
19:38:32 <cyberpear> right
19:38:39 <bcoca> ^ your change is manly so you can avoid doing use=systemd on your systems
19:38:48 <cyberpear> bcoca: any other problematic distros where you'd expect to see an issue?
19:39:15 <cyberpear> with both sysvinit and systemd on the same system, having `/sbin/init` as a symlink to `systemd`?
19:39:16 <bcoca> its not as much distros as container setups, as we discussed before
19:39:50 <cyberpear> basically the same as most containers are just a distro with fewer packages
19:40:15 <bcoca> cyberpear: sysvinit is really 'init scripts' which can and do exist even if using upstart or systemd, the problem becomes handling them when systemd is not actually running even if it 'appears to be' cause containers are frequently setup to 'lie' to the appications
19:40:22 <cyberpear> sure
19:41:07 <bcoca> my system is setup that way .. cause i had to find way to reproduce teh issues, but it is not 'out of box' for most, so i cannot point you to that, i can point you to tutorials to 'make the lie' but it would still be custom setup
19:42:57 <cyberpear> I don't expect it'll cause any, but I'd be happy to address any bugs that folks raise as a result merging this PR.
19:43:25 <cyberpear> bcoca: what would you want to see for this to merge?
19:43:43 <cyberpear> shertel: jillr: sdoran: any thoughts on this one?
19:44:00 <jillr> I'm undecided. I'm strongly in favor of generally being explicit, but I can also understand why folks would want to abstract that away. but there's just so many sharp edges.
19:44:02 <bcoca> cyberpear: i really dont know that there is a good solution, why i setup use=
19:45:01 <bcoca> and i undesrtand your offer and belive you are sincere, but i cannot count on it,  if I accept this im putting core team on teh hook
19:45:33 <bcoca> and i'm very cautious about doing that cause the team is already on the hook for far more than what is possible or reasonable
19:45:55 <shertel> I prefer relying on `use` but I don't have a strong opinion. If it is merged, sooner is better than later so people can iron out wrinkles early.
19:46:10 <bcoca> ^ and you have been great help at fixing many issues that we are on the hook for
19:46:56 <bcoca> im also overtly cautions for 2 reasons, 'systemd keeps biting us ' and the next version might already have a slew of breaking changes that i really dont want to add on top of
19:47:17 <bcoca> but i also have a tendency to paranoia when it comes to system services ...
19:47:26 * cyberpear cf https://github.com/ansible/ansible/pull/67040 interpreter discovery :P
19:47:57 <bcoca> ^ changes/new features can create many issues when introduced
19:50:00 <cyberpear> last time I'd caused a ci_complete test failure w/ a merged connection plugin, you folks pinged me and we had it solve in about 2 hours
19:50:08 <cyberpear> *solved
19:50:37 <cyberpear> bcoca: is there anything at all that would help you feel more comfortable w/ this PR?
19:50:47 <bcoca> a tardis
19:50:51 <jillr> it seems like this is one we might just need more core folks to weigh in on? since we're all waffling?
19:50:54 <cyberpear> what if I only limited it to running on RHEL?
19:51:27 <sdoran> I understand the use case, and container hackery is only going to become more common.
19:51:31 <bcoca> again, its not as much distro as setup
19:51:40 <jillr> sdoran: very true
19:51:51 <sdoran> But I am leery of the blowback from things we aren't thinking of.
19:52:05 <sdoran> And we get dinged a lot for "Ansible updates broke my stuff".
19:52:09 <bcoca> 'container hackery'  ... sdoran i will use this term from now on
19:52:55 <sdoran> This seems like a sound change, but I don't know what I don't know.
19:53:40 <bcoca> i think the change is correct 'in theory' .. my problem is too many people don't practice 'the theory'
19:53:51 <bcoca> and container setups are very often liars
19:54:03 <sdoran> Yeah.
19:54:08 <jillr> is it unreasonable to say, if you're doing container hackery you need to specify what hackery you've gone with via use~= ?
19:54:18 * cyberpear more than happy to tell folks "you're doing it wrong, do it this other way instead"
19:54:33 <sdoran> It may work brilliantly in a number of situations, but cause problems when encountering someone else's container hackery.
19:54:40 <bcoca> jillr: we did  initially, but had enough complaints that we changed position .. now if rest of core is fine with it, im fine with changign position again
19:54:54 <jillr> ah ok
19:55:11 <bcoca> but this was also a few years ago, so if willing to roll the dice ..
19:55:37 <cyberpear> would it help if I pull up the archaeology on this particular set of code? (I did the research once, can do it again if needed.)
19:55:42 <bcoca> since im not probably going to be the one fixing it in the end ... i really don't think i should talk for core here, those that are doing maintenance should be the ones to decide
19:57:19 <jillr> I live pretty much entirely in cloud modules now, so I also won't be the one fixing it, as such I'm going to abstain from a vote.
19:57:42 <sdoran> I don't think history is helpful. We're more concerned about the current and future of all the crazy container setups that will inevitably exist.
19:58:47 <sdoran> I suppose we can roll the dice on this one again. It seems to make more sense given the way things are headed.
19:59:36 * sdoran feels sorry for future self
19:59:36 <shertel> I would be okay with merging even if I'd prefer the explicit `use`
19:59:57 <bcoca> use is not going away, we are just shifting the subsectoin of users forced to use it
20:00:17 <shertel> sdoran: you'll have company :-)
20:00:28 <sdoran> Good company at that. ;)
20:00:43 <bcoca> right now cyberpear is only one i heard complaining about this detection, had many in past in the other case .. but dont know population now
20:01:01 <bcoca> so im remiss to make the chagne for one user and affect many others (quantity unknown)
20:01:24 <shertel> we can revert if we unearth problems early enough
20:01:25 <cyberpear> https://github.com/ansible/ansible/commit/99d775a0c12ff16c53aa2a19860d9293a2ef5b3c is where it was originally introduced in Aug 2015, virtually identical to today.
20:01:34 <sdoran> cyberpear: Do you have any idea how many others are affected by this behavior and also want this change?
20:01:48 <bcoca> shertel: early enough is normally after release ... then its a bit late to change behaviour again w/o deprecation cycle
20:02:29 <shertel> bcoca: yes, I meant before release. More people will be using devel than testing the individual PR. After... we'd have to deal with it.
20:02:30 <cyberpear> anyone using ansible-bender to build a systemd container, which we do in place of VMs for testing ansible roles since it's cheaper and faster
20:02:59 <cyberpear> I'd imagine anyone hacking on ansible will be running devel, if they are sending PRs
20:03:05 <cyberpear> (since all PRs must go to devel first)
20:03:14 <bcoca> shertel: used to be the case, but seems a lot less do so now and if intepreter discovery has shown anything is how  little diversity in environments we now have testing devel
20:03:45 <cyberpear> bcoca: I'd guess that a lot of that was folks were rrunning python2 distros which at the time were dominant
20:03:49 <jillr> so it sounds like sdoran and shertel are both in favor? in which case can y'all add your reviews to the PR?
20:04:00 <cyberpear> that's why I never ran into any of the interpreter discovery issues until recently
20:04:04 <shertel> I see bcoca's point though... so many times something significant has broken in a common module like ec2 or s3 and is only discovered long after releases... sometimes after 2 or 3 releases because people don't upgrade often enough.
20:04:47 <sdoran> Yeah, we _think_ it'll be discovered by merging early to `devel`, but reality is very few people actually test/use `devel` and don't find things until after release.
20:04:54 <bcoca> cyberpear: in theory, intepreter discovery was introduced after py3 became dominant  .... in theory ...
20:05:17 <cyberpear> "theory" but RHEL was still on python2, which is what many enterprises run
20:05:20 <bcoca> sdoran: again, that used to be the case ... when most ansilbe users were 'in devel' ... but has long not been true
20:05:43 <bcoca> since ansible went 'enteperise' the number of users dealing with 'devel' every day has shrunk
20:06:03 <jillr> this is more of an "over hot beverages convo/commisseration" now though  :)
20:06:12 <bcoca> before it used to be the place to get 'latest feature' .. right now they'll just wait for release
20:06:23 <jillr> what action do we have on this PR for now?
20:06:26 <felixfontein> shertel: thanks!
20:07:11 <jillr> I believe we're in favor of cyberpear's change, correct?
20:07:16 <shertel> jillr: I'm not in favor per se, but I'm not opposed. And if it is merged and broken I would try to help.
20:07:17 <bcoca> cyberpear: in theory most entpreises shoudl be runnign latest rhel .. but we still hear people complain aobut RHEL5
20:07:39 <cyberpear> yep, still some of those floating around here... RH supports it until November
20:07:45 <jillr> shertel: would you prefer to see more votes? I just worry we haven't gotten those for some time now
20:07:58 <jillr> if so we should identify specific people and try to pin them down to do so
20:08:06 <bcoca> i've put the caution tape up, i wont block proceeding ahead as long as that has been seen and ack''d
20:08:35 <cyberpear> felixfontein: any thoughts on discovering service_mgr fact for systemd in the offline case?
20:08:41 <bcoca> cyberpear: that code was copy/pasted + some inline fix
20:08:51 <sdoran> Seems we are "reluctantly and cautiously relenting" 😞
20:09:25 <felixfontein> cyberpear: sorry, no. I neither install fake service managers on my machines, nor do I use them in containers
20:09:38 <sdoran> felixfontein++
20:09:43 <cyberpear> lol
20:09:54 <bcoca> felixfontein is 'in practice of theory'
20:10:02 <felixfontein> i.e. it won't break my usecases, and I have no RL experience with usecases which it could theoretically break :)
20:10:39 <bcoca> if the unkown quantity == 0 ... i'll be happy to wipe this setup and start a saner one ...
20:10:56 <bcoca> though it is still useful for trying out diff non default inits on distros
20:11:42 <jillr> we're super over time now though, so let's be decisive. over several meetings cyberpear has brought this proposal forward and we haven't gotten any hard objections so much as caution,
20:11:53 <jillr> is it fair to say anyone that felt strongly has had an opportunity to speak up?
20:12:29 <sdoran> Yeah, I'll merge it after layering it in caution tape.
20:12:36 <jillr> rad, thanks sdoran
20:13:02 <jillr> thanks everyone for a long meeting today!
20:13:10 <jillr> #endmeeting