18:00:31 <nirik> #startmeeting Infrastructure (2016-06-16) 18:00:31 <zodbot> Meeting started Thu Jun 16 18:00:31 2016 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:31 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:31 <zodbot> The meeting name has been set to 'infrastructure_(2016-06-16)' 18:00:31 <nirik> #meetingname infrastructure 18:00:31 <nirik> #topic aloha 18:00:31 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore threebean pingou puiterwijk pbrobinson 18:00:31 <zodbot> The meeting name has been set to 'infrastructure' 18:00:31 <zodbot> Current chairs: abadger1999 dgilmore lmacken nirik pbrobinson pingou puiterwijk relrod smooge threebean 18:00:31 <nirik> #topic New folks introductions / Apprentice feedback 18:01:00 * threebean waves 18:01:05 <puiterwijk> Hello 18:01:18 * aikidouke here 18:01:24 * skrzepto is here 18:01:29 * pcreech here 18:01:33 * subho here 18:01:39 * lousab is here 18:01:43 <lousab> hello 18:02:10 <smooge> hello 18:02:11 <athos> hello :) 18:02:31 <nirik> hello all. 18:02:41 <nirik> any new folks like to give a short one line introduction? 18:03:47 <nirik> ok, if no, will go on to status/info 18:03:57 <nirik> #topic announcements and information 18:03:57 <nirik> #info Still in F24 final freeze - everyone 18:03:58 <nirik> #info spam attacks continue - patrick/smooge/kevin 18:03:58 <nirik> #info (re)setup proxy07 on a cloud node at bodhost, seems working so far - kevin 18:03:58 <nirik> #info New 10G switches and new bladecenters arriving in phx2 to be setup in coming weeks - kevin/smooge/patrick 18:04:09 <nirik> any other status/info folks would like to note? or discuss from above? 18:05:06 <smooge> spam 18:05:33 <nirik> yeah, the spam... continues. ;) 18:05:40 * pingou here (late) 18:07:33 * nirik is a bit distracted, still in the go/no-go meeting. 18:07:45 <smooge> alright will run 18:07:53 <smooge> ok we are in final freeze 18:07:54 <nirik> I had one discussion item 18:07:57 <smooge> which may or may not occur 18:08:18 <nirik> #topic Some upcoming releng changes that may affect infra: 18:08:18 <nirik> Review F25 change that may impact Infra: Automated Docker Releases 18:08:18 <nirik> Review F25 change that may impact Infra: Fedora Docker Registry 18:08:42 <nirik> .ticket 5356 18:08:43 <zodbot> nirik: #5356 (Review F25 change that may impact Infra: Automated Docker Releases) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/5356 18:08:44 <nirik> .ticket 5357 18:08:46 <zodbot> nirik: #5357 (Review F25 change that may impact Infra: Fedora Docker Registry) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/5357 18:09:08 <nirik> I looked at these and added feedback, but if everyone else could take a look and do so too that would be great. 18:09:47 <smooge> ok those will require new services and also impcact deliverables so pleas edo 18:10:01 <smooge> nad my fingers aren't typing well in colocation :) 18:10:31 <smooge> does anyone have anything to say right now on those two tickets? 18:10:46 <smooge> I expect we will talk more about them in the upcoming months 18:10:50 <nirik> yeah. 18:11:22 <smooge> ok will move onto next topic. 18:11:25 <smooge> #topic Apprentice office hours 18:11:49 * bwood09 back from hiatus and ready to start working 18:11:51 <smooge> I first want to say ++ to linuxmodder for helping various apprentices in #fedora-admin 18:11:58 <pingou> linuxmodder++ 18:11:58 <zodbot> pingou: Karma for linuxmodder changed to 18 (for the f23 release cycle): https://badges.fedoraproject.org/tags/cookie/any 18:12:05 <smooge> linuxmodder++ 18:12:11 <vivek_> linuxmodder++ 18:12:11 <zodbot> vivek_: Karma for linuxmodder changed to 19 (for the f23 release cycle): https://badges.fedoraproject.org/tags/cookie/any 18:12:14 <smooge> I think I gave my cookies alreayd 18:12:32 <athos> linuxmodder++ 18:12:32 <zodbot> athos: Karma for linuxmodder changed to 20 (for the f23 release cycle): https://badges.fedoraproject.org/tags/cookie/any 18:12:41 <smooge> I have seen him helping a lot of people while I have been stuck dealing with SPAM issues. 18:12:50 <smooge> Hi bwood09 18:12:54 <smooge> welcome back 18:12:57 <bwood09> Thanks ^_^ 18:13:11 <smooge> are there any questions at the moment? 18:13:34 <athos> yes, I didn't even know he was on fi-apprentice since he's been that active in the channel helping ppl out 18:13:46 <lousab> smooge i have a question on nagios ansible 18:15:05 <smooge> hi lousab 18:15:08 <lousab> i've seen on batcave under ansible/roles i think and i didn't understood if there's the yalm file or not or if i have to build for #5337 18:15:10 <smooge> what is the question 18:15:10 <aikidouke> and i have something unrelated to lousab 18:15:29 <smooge> .ticket 5337 18:15:30 <zodbot> smooge: #5337 (add monitoring for fedoramagazine) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/5337 18:15:56 <bwood09> and I was thinking about picking up ticket 4973 if nobody's done anything with it 18:16:00 <bwood09> .ticket 4973 18:16:01 <zodbot> bwood09: #4973 (add nagios check for mailman01's REST interface.) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/4973 18:16:12 <smooge> ha everyone wants to do nagios :). 18:16:20 <lousab> :) 18:16:24 <smooge> and I am about to change out how nagios is done 18:16:26 <nirik> nagios doesn't use anything close to as sane as a yaml file. ;) 18:17:07 <smooge> so nagios is mostly done through a file format that is one of the many meh formats. Why is it like that? meh 18:17:42 <lousab> nirik: i normally configure nagios with kickstart if i have to install it and thruk and monarch plugins....i never used ansible..i'm learning it :( 18:17:48 * jflory7 is here 18:17:52 <smooge> We haven't been consistent on how we implement them across the system so they end up being very fragiole 18:19:28 <nirik> yeah, the best way I think to see is pick a similar host to the one you are adding and 'git grep' it and look at where it's defined in nagios 18:19:34 <bwood09> Is there anything we can do to streamline that process? What changes are we looking at for nagios? 18:19:40 <nirik> There's also some old commits likely where machines were added... look in git log 18:20:16 <lousab> nirik: ok thanks 18:20:33 <nirik> the plan is for smooge to reimplement it so ansible just adds the needed stuff when we add a host. 18:20:35 <lousab> aikidouke: i'm new here :) i'm an apprentice 18:20:37 <nirik> but thats not done yet. ;) 18:20:44 <bwood09> hm, that sounds like a cool idea 18:20:49 <aikidouke> maybe this needs to start in releng or something, but how long do we think it might be before we need an app/tracker for snap packages or flatpacks 18:21:04 <bwood09> I need to get my Ansible infra rebuilt and refresh my memory on how to use it 18:21:14 <bwood09> I think the last time I was here we might have been migrating over to it 18:21:31 <nirik> aikidouke: I think thats a ways out. :) 18:21:47 <smooge> lousab, so what I think we are looking for right now would be a nagios file just like you would use on your other places htat would monitor a website 18:22:22 <aikidouke> :) ack - was typing that out waiting to hit enter, guess im not good at waiting 18:22:39 <aikidouke> and sorry lousab: did not mean to step on your toes 18:22:40 <smooge> aikidouke, releng would be better 18:22:47 <aikidouke> thanks 18:23:08 <nirik> aikidouke: neither snappy nor flatpack is what I would consider ready for widespread use, despite PR people saying so. ;) 18:23:27 <lousab> aikidouke: no problems :) 18:23:37 <smooge> bwood09, my plan is to make a set of templates for hosts like we have in the ./roles/nagios/client/templates/check_fedmsg_gateway_proc.cfg.j2 but for services and hosts 18:23:41 <aikidouke> :) 18:24:04 <lousab> smooge: ok right 18:24:12 <smooge> so lousab and bwood09 what I would like is a file that meets that 18:24:15 <bwood09> smooge, that makes sense 18:24:50 <smooge> so when looking at say fedoramagazine we want to be able to change that out to be fedoraproject or ask.fedoraproject or ... 18:25:05 <bwood09> I'll need to get back in and look at that file 18:25:19 <smooge> those files use a {{{ }}} variable substitution 18:25:59 <smooge> so write the configs with sya fedoramagazine in mind but then look at 'this could be a var' in comments 18:26:37 <smooge> that way the work is being done once and can be 'redone' when time is ready 18:27:02 <smooge> for general help on what oges in the files.. I have to work on that this weekend 18:27:11 <smooge> does that help? 18:27:23 <bwood09> makes sense to me 18:27:56 <lousab> smooge: ok i'll find you because i'm starting to use ansible now so solving this ticket means undestarnd for me also how ansible works 18:28:04 <smooge> cool 18:28:17 <bwood09> same here 18:28:25 <smooge> any other questions? 18:28:46 <bwood09> How much incident management stuff do we do? A lot of my recent professional experience is in that area 18:29:27 <smooge> we don't have incidents. we have unicorns and butterflies 18:29:44 <bwood09> lol I can see that ;) 18:30:07 <athos> lol 18:30:36 <pingou> and http://media-cache-ak0.pinimg.com/736x/bf/3f/4c/bf3f4c4e4cbc909f957f939bb6bc7cc6.jpg 18:30:41 <aikidouke> if a butterfly flaps its wings in the phx2 data center.... 18:30:51 <smooge> puiterwijk, is our security lead. 18:30:56 <pingou> aikidouke: we're fine, it's the other end of the world that's not :) 18:31:07 <aikidouke> lol 18:31:29 <smooge> puiterwijk is in another meeting I think so I would aim a question towards his email to find out 18:31:53 <pingou> truth is we have some but not that many 18:32:01 <pingou> (luckily) 18:32:26 <bwood09> I work at Navy Federal Credit Union now and omg so many actual incidents it's crazy 18:32:54 <pingou> somehow, I can imagine that :) 18:32:57 <aikidouke> would it help to review our, what are they called, security guidelines? who to report X to, etc 18:33:03 <puiterwijk> We have some documents of how to manage incidents that I'm updating and writing some more. But I try to make sure we have as few incidents as possible. If you have any specific questions, please let me know 18:33:18 <nirik> we have CSI for that process currently... 18:33:25 <bwood09> puiterwijk, if there's anything I can do to help out, I'd be happy to 18:33:25 <aikidouke> thanks nirik 18:33:35 <puiterwijk> nirik: right. That CSI is what I'm reading through and updating 18:33:36 <nirik> https://docs.fedoraproject.org/en-US/Community_Services_Infrastructure/1/html/Security_Policy/index.html 18:33:42 <nirik> puiterwijk: excellent. 18:33:59 <puiterwijk> I'm also writing a bunch of documents for myself and others of what to do for specific roles/services. 18:34:11 <aikidouke> there are probably a few hosts that need CSI variables updated/added still 18:34:19 <aikidouke> that is a good place to start 18:34:20 <nirik> yes, there's a number of them. ;) 18:34:24 <nirik> patches welcome. 18:34:35 <puiterwijk> Yep, absolutely. If anyone sees any open or outdated CSI variables, fixes are welcome 18:34:58 <athos> 145 groups w/o csi variables 18:35:10 <lousab> aikidouke: yes but i didn't really understod how to..i mean i've undestood iptables roles but how can we start? 18:35:18 <athos> I am trying to understand how to codument those and will probably start shipping patches this weekend 18:35:37 <aikidouke> good q lousab 18:35:53 <nirik> theres a section on them on the apprentice page... and also some previous patches on the list right now. 18:35:59 <athos> though I will need to get the feeling on how to set security categories 18:36:04 <pingou> athos: cool, thanks 18:36:05 <aikidouke> so - updating CSI vars takes a bit of research 18:36:21 <aikidouke> reading through the security doc nirik linked will help 18:36:29 <athos> oh, I did 18:36:39 <aikidouke> if you are unsure, make an educated guess, then ask 18:36:48 <athos> ack 18:37:26 <aikidouke> you can get a feel for the categories, by looking at existing configurations 18:37:39 <athos> I am trying to figure out if I can write some scripts to set the relationships 18:37:48 <aikidouke> for example - if mirrormanager went down - that would be bad.... 18:38:19 <pingou> for everyone :) 18:38:39 <aikidouke> walk through each of the tasks for one particular service - see what it depends on 18:38:45 <pingou> it's one of these apps where nagios is slower than our users :) 18:39:06 <aikidouke> grep through everything else to see what depends on the service/host you are looking at 18:39:14 <aikidouke> :) 18:39:20 <athos> thx :) 18:40:00 <aikidouke> dont be afraid to make a mistake - I've made them and noone shows up at your house to take away your birthday 18:40:24 <bwood09> I would hope the birthday police don't show up 18:40:24 <lousab> nirik: could you please tell me exactly what section? of course when you have time...:) 18:40:33 <bwood09> although not aging would be cool 18:40:34 <smooge> For the most part, the worst you are going to get is a hohoho email from me 18:40:44 <aikidouke> :) 18:40:54 <nirik> https://fedoraproject.org/wiki/Infrastructure_Apprentice#Longer_term_quests 18:40:55 <athos> please take away 8 or 9 of my bdays! 18:41:01 <pingou> bwood09: the problem is: you get older but w/o the cakes 18:41:04 <aikidouke> haha...same here 18:41:06 <bwood09> ooooooooh 18:41:11 <pingou> (even if the cake is a lie) 18:41:15 <aikidouke> lol 18:41:28 <bwood09> it wouldn't be a triumph anymore 18:41:31 <aikidouke> im getting us off track...sorry 18:41:32 <bwood09> no huge success 18:41:36 <bwood09> lol yeah me too sorry 18:43:16 <smooge> ok back on topic 18:43:20 <smooge> any other questions? 18:43:44 <smooge> I will move to open floor 18:43:48 <smooge> #topic Open Floor 18:44:15 <nirik> FYI, the go/no-go meeting just ended a few minutes ago... and we are go for f24 release next tuesday. ;) 18:44:22 <smooge> yay 18:44:23 <puiterwijk> \o/ 18:44:25 <pingou> \ó/ 18:44:30 <athos> awesome :) 18:44:44 <bwood09> great :D 18:44:51 <smooge> so get your f24 ++ cookies while you can 18:44:53 * pingou got to step away 18:45:03 <pingou> thanks for chairing smooge and nirik :) 18:45:15 <smooge> ok pingou night 18:45:27 <smooge> ok I think we are done for today anyway 18:45:31 <smooge> going once 18:45:33 <lousab> :) 18:45:35 <smooge> going twice 18:45:36 <puiterwijk> I would like to propose replacing the word "spam" with "fish" from now on. My blood triggers anytime I read the word "spam" now, so I want to change it slightly :) 18:45:59 <bwood09> I had a <triggering intensifies> gif somewhere... lol 18:45:59 <puiterwijk> </sorta-kidding> 18:46:00 <pingou> puiterwijk: and vice-versa? 18:46:05 <smooge> canned spiced ham? 18:46:06 <puiterwijk> pingou: sure 18:46:19 <pingou> puiterwijk: so are dolphin spam or mamals? 18:46:35 <smooge> #endmeeting