19:00:26 <nitzmahone> #startmeeting Core team public meeting 19:00:26 <zodbot> Meeting started Tue Jul 12 19:00:26 2016 UTC. The chair is nitzmahone. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:26 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 19:00:26 <zodbot> The meeting name has been set to 'core_team_public_meeting' 19:00:54 <ryansb> hi folks 19:00:57 * gundalow waves 19:01:02 * ttom waves 19:01:08 <nitzmahone> #chair ryansb gundalow 19:01:08 <zodbot> Current chairs: gundalow nitzmahone ryansb 19:01:15 <gundalow> thanks 19:01:28 <Qalthos> hiya 19:03:17 * nitzmahone looks for unfinished business from last meeting 19:04:11 <ttom> in the mean time, we discussed https://github.com/ansible/ansible/pull/16491 last meeting, and agreed that i'd remove the strategy and refie the callback plugin. So I did. how can I move this PR forward? 19:04:28 <ttom> refie == refine 19:06:59 <gundalow> nitzmahone: I don't know anything about call backs. Who would be good to do an review of https://github.com/ansible/ansible/pull/16491 since ttom did the update that was requested last meeting 19:08:08 <gundalow> ttom: Who was it that asked for that jtanner and jimi|ansible ? 19:08:13 <nitzmahone> Whoever requested the update would probably be a good candidate 19:08:20 <jtanner> ? 19:08:21 * nitzmahone looks at log from last meeting 19:08:45 <ttom> I think jimi|ansible and mattclay, but not sure. 19:09:45 <jtanner> i made tangential comments about the type of strategy i want for integration testing 19:09:54 <jtanner> not really relevant to current PR 19:09:59 <nitzmahone> Looks like it was jimi, but he's plenty busy. 19:10:15 <nitzmahone> I can take the review 19:10:35 <nitzmahone> #action nitzmahone to review #16491 19:10:40 <gundalow> Thanks :) 19:11:30 <nitzmahone> It sounds like 2.1.1 RC3 is a foregone conclusion- anybody want to advocate for high priority bugs that should be included? 19:11:41 <ttom> thanks nitzmahone 19:11:43 <nitzmahone> s/bugs/fixes 19:14:46 <nitzmahone> dag: you here? 19:16:14 <nitzmahone> Looks like no. Might be a short meeting... 19:16:17 <nitzmahone> #topic open floor 19:16:38 <kustodian> so I can suggest a topic? 19:16:44 <nitzmahone> fire away 19:17:07 <kustodian> What do you guys think about modules storing state on managed hosts in special directories/files 19:17:22 <kustodian> when it's not possible to keep state in e.g. configuration file 19:17:29 <kustodian> or service memory or whatever 19:17:44 <nitzmahone> I think I speak for most of us on the core team that we'd be very hesitant to ship modules that did that. You of course are free to write whatever you want. :) 19:17:47 <kustodian> I had this discussion with bcoca and abadger1999 a few months ago 19:18:02 <kustodian> bcoca: was against it, abadger1999 was ok with it 19:18:03 <nitzmahone> I vaguely recall 19:18:12 <abadger1999> yeah. 19:18:12 <MichaelBaydoun> kustodian: do you have an example use case? 19:18:19 <kustodian> I wrote a module and submitted a pull request to extrast 19:18:21 <kustodian> yup 19:18:29 <kustodian> https://github.com/ansible/ansible-modules-extras/pull/2531 19:18:45 <kustodian> iptables module which keeps state in /etc/ansible-iptables 19:18:59 <kustodian> since iptables-save dump doesn't have enough information for the module to work correctly 19:19:15 <abadger1999> maybe that's something we should put on the agenda for either core meetup or contributor summit or both. 19:19:29 <kustodian> and even if it does it would make it hard to put everything there and later parse it 19:19:42 <jtanner> i would think the distros have a persistent store of rules somewhere. i know EL did prior to firewalld 19:19:52 <kustodian> but also make it easy for a user changing iptables directly to f*** up something 19:19:53 <ryansb> yeah, adding stored state seems like a really big increase of the testing surface 19:20:00 <nitzmahone> My biggest concern with that (and I think other have mentioned as well) is that if changes are made out of band, the cached state will be wrong. Where when the state is gathered on the fly, it's much less likely for that to happen. 19:20:09 <abadger1999> We do it already though... 19:20:15 <abadger1999> For instance, the cron module 19:20:28 <jtanner> cron.d ? 19:20:33 <ryansb> isn't that just adding the #ansible comment to the crontab line? 19:20:37 <abadger1999> Stores state about whether ansible added a cron entry or not. 19:20:38 <kustodian> yeah cron uses a comment 19:20:42 <abadger1999> as a comment in the crontab 19:20:54 <jtanner> that's still just one store to rule them all though 19:21:00 <ryansb> I feel like that's different from a separate store 19:21:07 <kustodian> well the module has a locking mechanism 19:21:11 <abadger1999> One store or multiple stores doesn't really feel any different to me. 19:21:25 <abadger1999> You have the same usecases you're trying to satisfy and the same drawbacks. 19:21:45 <jtanner> i think we're being hesitant to have modules attempting to merge stores and keep track of priorities 19:22:07 <abadger1999> (Look at recent bugs on the cron module from a user who wanted ansible to magically know that it should replace an existing, manual cron entry with its own) 19:22:13 <kustodian> I completely agree that if possible external store shouldn't be used, but sometimes it's not and because of that it's limiting the full potential of a module 19:22:26 <ttom> I think that if you have a wrapper for iptables that manages it better (saves state) then you end up with the same issues (someoneusing regular iptables and mangles the state). 19:22:39 <jtanner> is /etc/sysconfig/iptables gone in the firewalld world? 19:22:42 <kustodian> this module doesn't have that problem 19:22:49 <kustodian> jtanner: it is 19:22:59 <kustodian> if you use firewalld you can use the firewalld module 19:23:05 <kustodian> but not everyone has and uses firewalld 19:23:14 <kustodian> Centos 6 for example 19:23:21 <jtanner> do the other distros have /etc/sysconfig/iptables equivalents? 19:23:29 <kustodian> some do, some don't 19:23:48 <kustodian> Debian doesn't even have an official store 19:23:51 <jtanner> what do they do on boot if no file holds the rules? 19:23:55 <kustodian> you either use iptables-persistent 19:23:57 <MichaelBaydoun> We have old rhel5 servers, and anything that would better manage iptables would get a +1 from me 19:24:02 <kustodian> or write your own script which restores 19:24:24 <kustodian> this module works with C5, 6 and 7 (if you disable firewalld) 19:24:40 <jtanner> debian/ubuntu? 19:24:58 <kustodian> yup debian/ubuntu, no official way to make iptables persitent 19:25:03 <jtanner> lame. 19:25:10 <kustodian> I also thought that as well :D 19:25:30 <MichaelBaydoun> We wanted to use https://github.com/debops/ansible-ferm but it only works on rhel6 and higher 19:25:34 <kustodian> iptables-persistent is the only way via package, but it doesn't come with core 19:26:09 <kustodian> MichaelBaydoun: check out this module I wrote a blog about it how we use iptables, you'll like it :) http://blog.nordeus.com/dev-ops/managing-iptables-with-ansible-the-easy-way.htm 19:26:12 <jtanner> this page (https://wiki.debian.org/iptables) claims there is a /etc/iptables.up.rules 19:26:49 <kustodian> jtanner: if you install iptables-persistent 19:26:50 <jtanner> is that not desirable because you have to then edit /etc/network/if-pre-up.d/iptables ? 19:26:50 <MichaelBaydoun> So I'm supporting the pain, and like the solution, while at the same time not feeling great about the idea of storing state. And with that confusing feedback, I need to head to a meeting. 19:26:53 <kustodian> but not everyone uses it 19:27:22 <kustodian> I had a discussion with a debian user a few days ago about it :D 19:27:23 <abadger1999> MichaelBaydoun: :-) 19:28:08 <jtanner> how do chef/puppet/salt store iptables rules on debian? 19:28:12 <abadger1999> I'm owndering if what we're really debating is: https://en.wikipedia.org/wiki/Single_source_of_truth 19:28:22 <nitzmahone> ^-- ding 19:28:25 <abadger1999> Which is not straightforward to solve here. 19:28:32 <ryansb> well...yeah 19:29:00 <abadger1999> Because all of these things have "prexisting config on the target machine" + "One or more ansible playbooks" 19:29:07 <kustodian> I don't know how others do it, but as far as I remember in puppet you define rules and it somehow keeps state :D 19:29:36 <ryansb> and not all people run the same group of playbooks every run, so the risk of accidentally undoing rules seems kinda high 19:29:49 <kustodian> what this module does is it distinguishes what ansible made and what didn't 19:29:54 <abadger1999> yep. Unless you keep a state local to the target machine. 19:29:59 <bcoca> puppet keeps a 'central repo' with all the rules and tracks things there, not on each machine 19:30:16 <jtanner> puppet can run in standalone mode though 19:30:18 <kustodian> bcoca: that isn't possible with ansible for now 19:30:25 <bcoca> jtanner: using teh rules from central repo 19:30:34 <bcoca> kustodian: nor do we want it 19:30:43 <bcoca> my whole point against keeping state 19:30:44 <kustodian> so what puppet does doesn't help :D 19:30:50 <bcoca> the PLAYS ARE THE STATE 19:30:52 <jtanner> our "central state" is vars 19:31:09 <bcoca> jtanner: inventory + plays + vars 19:31:12 <jtanner> or what bcoca said 19:31:20 <kustodian> to have a proper iptables module, it's not possible to do it with "external" state 19:31:34 <kustodian> current iptables module is mostly useless 19:31:45 <bcoca> kustodian: i disagree, as the current one has plenty of users 19:31:48 <kustodian> not much different from running iptables with command 19:31:57 <bcoca> if you WANT to keep state, tempalte the rules, use dump/restore 19:32:19 <bcoca> i dont want to ship any module that keeps it's own state databases 19:32:34 <kustodian> bcoca: makes it a lot harder to manage iptables like that, than with this module, at least read it once when you have time, a lot easier than anything with a template 19:32:36 <bcoca> ^ bad precedent, if you want to use the module or share it galaxy, go ahead 19:32:45 <abadger1999> bcoca: Then rewrite cron. 19:33:02 <bcoca> abadger1999: making my point for me 19:33:19 <jtanner> ah, salt doesn't attempt to persist across rebots 19:33:35 <jtanner> it manages current in-mem state only 19:33:47 <kustodian> does it have rules priority? 19:33:54 <abadger1999> bcoca: <nod> yeah -- cron is horrible but do we get rid of it or do we decide that there are things that people need to do where we can't do it without state so we have to allow it. 19:34:08 <jimi|ansible> salt is akin to puppet, except written in python and using yaml 19:34:10 <bcoca> abadger1999: this is where i stres the usfuleness of templates 19:34:13 <jimi|ansible> functionally, they're very similar 19:34:18 <jtanner> kustodian: https://docs.saltstack.com/en/latest/ref/states/all/salt.states.iptables.html 19:34:26 <bcoca> cron/lineinfile are really hackky bad ways of maintaining 'partial state' 19:34:55 <ryansb> bcoca: so you mean the alternative being templating jobs into cron.d 19:35:08 <ryansb> err...ignore if that's not the rabbit hole you want to go down 19:35:09 <kustodian> so how do I change a rule in salt? 19:35:11 <abadger1999> ryansb: templating al jobs for cron I think. 19:35:14 <abadger1999> *all 19:35:23 <jtanner> lineinfile is a lifesaver though if you don't want to maintain templates for heterogenous environments 19:35:27 <bcoca> ryansb: that is how i maintained my machines, current cron module is a horrible hack that causes many problems, i want less of that, not more 19:35:34 <kustodian> it doesn't give an ID to rules, so not sure if you can change a rule 19:35:49 <kustodian> also no order of rules 19:35:59 <bcoca> jtanner: replace is better, but still a bad way of maintaining config files 19:36:31 <bcoca> kustodian: they use big hammer and avoid the fine tuned existing/new rules interweve problem 19:36:44 <jtanner> kustodian: didn't mean to imply it was a superior tool, just that it existed 19:37:05 <kustodian> big hammer? 19:37:09 <kustodian> delete everythign else? 19:37:24 <bcoca> kustodian: that is the 'easy'way 19:37:43 <kustodian> but that means the module needs to be executed at once? right? 19:37:44 <bcoca> that is what puppet does essentially, declares itself 'source of truth' and then everything must match 19:37:51 <bcoca> ^ most 'centralized' tools do this 19:38:02 <ryansb> yes. It's only the easy way if you assume you are the 100% boss 19:38:29 <kustodian> this module can also do this if you set one param to 'no' 19:38:39 <kustodian> but it doesn't do it by default 19:38:42 <bcoca> which is why i recommend the template for those cases, for the ones in which you are not 100%, current iptables modules is enough, specially if used in a role for complex interactions (fetch existing, merge, publish) 19:38:56 <kustodian> it's not 19:38:58 <jtanner> chef has a "role" like cookbook to manage iptables: https://github.com/chef-cookbooks/iptables 19:39:11 <kustodian> we have 350+ hosts and it's pain to do it 19:39:32 <bcoca> and chef uses templates to manage it 19:39:33 <kustodian> now our roles open everything they need and everyone I showed what we do they liked it 19:39:54 <kustodian> and Ansible is still a single point of truth 19:40:13 <nitzmahone> Maybe to bring it back in- sounds like a quorum of the core team is still pretty strongly opposed to adding modules that do external state storage. 19:40:29 <kustodian> as far as I can see mostly bcoca :P 19:40:32 <bcoca> kustodian: not really, data on the 'servers' IS the source of truth, ansible becomes a middle man, not something i wantin a module 19:40:37 <jtanner> i am opposed too 19:40:50 <nitzmahone> I know I'm still at -1. I understand the issue, but I'm not sold that we want to take on another one that does that. 19:40:55 <kustodian> the state is on the server, isn't it? 19:41:10 <bcoca> kustodian: im not saying youre module is 'bad' or not useful, it jsut does things in a way that we don't want to support or deall with the implications 19:41:17 <nitzmahone> ^-- that 19:41:32 <kustodian> so if the whole state was in the iptables-save dump 19:41:35 <kustodian> that would be fine? 19:41:59 <bcoca> you can ALREADY do that, or just dump the 'firewall state' from the kernel 19:42:12 <bcoca> that is what current iptables module does 19:42:17 <kustodian> but it doesn't have everything you need to easily manage iptables 19:42:22 <kustodian> doesn't have order 19:42:23 <bcoca> hell, that is what iptables commadn does 19:42:28 <kustodian> ids of rules 19:42:39 <jimi|ansible> a state dump is not idempotent, so it's not a great way to do things 19:42:43 <kustodian> doesn't have what rules are from ansible, what not, a lot of things 19:42:43 <bcoca> iirc we updated it so you can insert the rule at specific position 19:42:50 <bcoca> before it was just prepend/append 19:43:24 <bcoca> kustodian: for those cases, i use template and i don't think that merrits having a 'data store' on the remote 19:43:44 <bcoca> ^ but you can do as you wish, i'm not stopping you, i'm just not going to maintain such a thing 19:43:47 <kustodian> just a question, what exactly is the problem having a "data store" on the remote? 19:44:10 <bcoca> it makes it impossible for ansible to have a coherent picture at source 19:44:19 <kustodian> bcoca: aren't the extas module now maintained by the community mostly? or is that a plan? 19:44:20 <bcoca> now inventory + plays + vars ARE NOT your source of truth 19:44:24 <jimi|ansible> kustodian: you can never know for sure what's there, ideally the module should ensure the state matches exactly what you specify, thus removing anything you don't have there 19:44:36 <nitzmahone> (without external assurances that nobody's diddling the "actual truth" out from underneath us) 19:44:44 <jtanner> those diddlers 19:44:47 <bcoca> kustodian: its a plan, once we don't ship/maintain it ... go at it ... till then i opose the inclusion 19:45:08 <abadger1999> jimi|ansible: But -- many of our modules are not like that. Not just cron either. 19:45:13 <nitzmahone> *you* can possibly make such assurances in your env, so use such a thing, but it's a trip hazard for people who can't make such assurances or use the module without understanding that 19:45:27 <kustodian> pretty sure there are a few modules like that 19:45:42 * nitzmahone doesn't want to add to that list 19:45:43 <bcoca> abadger1999: and i'll celebrate when we split extras off and they are not our problem anymore 19:45:57 <abadger1999> I bet if you choose 5 random cloud modules you'll find at least one where it doesn't work like the ideal. 19:46:02 <bcoca> kustodian: mistakes im trying to remedy 19:46:06 <jimi|ansible> the ec2 ones are 19:46:14 <bcoca> abadger1999: 5 is a low number 19:46:19 <abadger1999> bcoca: exactly. 19:46:28 <kustodian> you're removing a lot of potential from modules because of that :( 19:46:36 <bcoca> abadger1999: ^ repeating myself: that we made a mistake before does nto mean we should double down 19:46:39 <abadger1999> bcoca: if it was choose 20 random cloud modules... maybe that's not a big deal... 19:46:54 <abadger1999> bcoca: it's only a mistake if there's another way to do it. 19:47:07 <bcoca> kustodian: yes and no, lookups and info modules are intended to gather info, you can create same logic using multiple tasks, really do not want to build that into single modules 19:47:19 <bcoca> abadger1999: ^ the other way 19:47:23 <abadger1999> bcoca: so perhaps what we need is to see an example of doing it hte other way. And then kustodian and others can critique the approach. 19:47:32 <bcoca> modules should stay small and composable, you are trying to do too much in one 19:47:39 <kustodian> bcoca: the reason why is it in the module is that I don't have to write 3 tasks each time I want to change iptables 19:47:58 <kustodian> the module only manages iptables and does it very easy for the user 19:48:02 <kustodian> but anyway 19:48:02 <abadger1999> and then we'll know if it's truly lacking things that we need in order to support the things that people want to do. 19:48:02 <bcoca> role: task1 => iptables-dump, task2, local template with that data and 'ordered rules' task3: push template task4: iptables-restore 19:48:26 <kustodian> and if I have 10 rules which I want to change that 40 tasks 19:48:33 <kustodian> just for iptables instead of 10 tasks 19:48:49 <bcoca> kustodian: if you want it to be, you can pass single task to role or list (i would use list) 19:49:22 <bcoca> you can call the role 1 time or 40, i dont care, its up to you 19:49:30 <kustodian> role is not good enough, I want to open a port in side a role 19:49:37 <kustodian> not the same 19:49:45 <bcoca> port in side a role? 19:49:54 <kustodian> yeah 19:49:55 <kustodian> why not 19:49:56 <nitzmahone> include 19:50:01 <bcoca> ^ not sure what that means 19:50:02 <nitzmahone> + vars 19:50:18 <kustodian> e.g. nginx role 19:50:25 <kustodian> opens port 80 and 445 19:50:27 <kustodian> 443 19:50:44 <bcoca> make it depend on firewall role, passing list of 2 rules to open those ports ... 19:50:46 <kustodian> of course it's configurable in the role 19:50:56 <kustodian> doesn't scale 19:51:07 <bcoca> just as much as module does 19:51:18 <kustodian> if you have a bunch of rules you have to open, I don't want to check which role needs what port, my roles open ports 19:51:27 <kustodian> bcoca: not true 19:51:40 <kustodian> when I include my roles in a host, iptables is setup 19:51:49 <kustodian> nothing more I need to do 19:52:02 <kustodian> which is the point of a role, to setup a service 19:52:09 <kustodian> iptables is part of setting up a service 19:52:15 <bcoca> no, the point of a role is to group related tasks and resources 19:52:28 <bcoca> they are NOT equivalent to 'service setup' that is just ONE way to use them 19:52:37 <kustodian> of course 19:52:45 <kustodian> but in most situations it's the service 19:52:51 <nitzmahone> regardless, there are other ways to accomplish that (include + with_items/vars) 19:52:54 <bcoca> you CAN have an iptables_install role and an 'iptables_rules' role 19:52:58 <nitzmahone> and you can do that inside a role 19:53:11 <bcoca> ^ as he says, i said roles, but includes+vars also would work 19:53:36 <bcoca> many ways to do this, w/o building soo much into a module 19:53:42 <bcoca> specially state tracking 19:53:48 <kustodian> if that is so easy 19:54:11 <kustodian> why there is no explanation how to do it, best article I found is using ferm 19:54:22 <bcoca> lininfile and cron (which i also opposed) do this by using an EXISTING state which was the file or cron file that they modify, your case adds an additional 'data store' to the target 19:54:54 <bcoca> kustodian: many things are easy to do and no one has written an article about them 19:55:14 <bcoca> i'll add to my list, but that is already long 19:55:43 <kustodian> I wrote an article, if you have time read it, it's not that long 5 minutes 19:55:56 <kustodian> give me suggestions how to do it your way 19:55:57 <kustodian> :) 19:56:20 <nitzmahone> OK, any other topics for open floor before we wrap up? 19:56:54 <gundalow> Nothing from me 19:57:08 <nitzmahone> ending meeting in 10 19:57:08 <gundalow> Will the meeting be happening next week or week after? 19:57:16 <ryansb> not I 19:57:27 <nitzmahone> Oh, good question. We *will* mostly all be in the same room... :) 19:57:44 <nitzmahone> Which could be good or bad 19:57:58 <gundalow> Can we go to the pub instead? 19:58:12 <bcoca> gundalow++ 19:58:13 <gundalow> 9 19:58:14 <gundalow> 0~8 19:58:15 <gundalow> 7 19:58:15 <gundalow> 6 19:58:15 <nitzmahone> It'll be early in the morning 19:58:16 <gundalow> 5 19:58:18 <gundalow> 5 19:58:21 <gundalow> 6 19:58:23 <gundalow> 2 19:58:26 <nitzmahone> 42 19:58:26 <gundalow> 1 19:58:28 <gundalow> #endmeeting