18:00:17 <nirik> #startmeeting Infrastructure (2016-03-03) 18:00:17 <zodbot> Meeting started Thu Mar 3 18:00:17 2016 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:17 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:17 <zodbot> The meeting name has been set to 'infrastructure_(2016-03-03)' 18:00:17 <nirik> #meetingname infrastructure 18:00:17 <nirik> #topic aloha 18:00:17 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore threebean pingou puiterwijk pbrobinson 18:00:17 <zodbot> The meeting name has been set to 'infrastructure' 18:00:17 <zodbot> Current chairs: abadger1999 dgilmore lmacken nirik pbrobinson pingou puiterwijk relrod smooge threebean 18:00:17 <nirik> #topic New folks introductions / Apprentice feedback 18:00:23 <nirik> morning everyone. 18:00:34 <smdeep> morning nirik 18:00:38 <nirik> any new folks like to introduce themselves? or apprentices with questions or comments. 18:01:40 * doteast present 18:03:10 * nirik will wait a few more minutes for folks to wander in. 18:03:17 * sayan is here 18:05:22 <nirik> ok, I guess lets go on to status and info dump... hang on to your irc clients. 18:05:35 <nirik> #topic announcements and information 18:05:35 <nirik> #info [release] mdapi: 2.4 - Pierre-Yves - http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/UXJT3F2PZFGOTOC5CHG3YXDOZ4NZDPGX/ 18:05:35 <nirik> #info [release] pagure: 1.2 - Pierre-Yves - http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/6E42HJLJPMSPEJQVZK6VJWMEOCBN6IZL/ 18:05:35 <nirik> #info [release] fedora-packages: 3.0.4 - Ralph - http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/ZVUJKOD45IIMD3VQK6JV6CQCL76B6JO7/ 18:05:36 <nirik> #info [release] the-new-hotness: 0.7.1 - Ralph - http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/QDIL4PU2N7ODOGUR3HJOI6MP2IOM4AAS/ 18:05:39 <nirik> #info ALPHA FREEZE starts 2016-03-08 (next tuesday!) - everyone 18:05:41 <nirik> #info Staging rebuild to take place later today - kevin 18:05:43 <nirik> #info stg websites is now being built from pagure.io - kevin 18:05:45 <nirik> #info virthosts 03/04/18/19/21 reinstalled - kevin 18:05:47 <nirik> #info virthosts04 installed - smooge 18:05:49 <nirik> #info autosign02 started on.. needs playbook - smooge 18:05:51 <nirik> #info looking for an opensource splunk - smooge 18:05:55 <nirik> #info budget.stg.fedoraproject.org live - websites/decause 18:05:57 <nirik> #info March apprentice status update email is out, answer it today! - kevin 18:06:26 <nirik> anything in there anyone wants to expand on, discuss or add to? 18:07:14 <aikidouke> splunk is ?? 18:07:26 * aikidouke has heard of it - whats our use case? 18:07:49 <lmacken> logstash + kibana + elasticsearch = open source splunk? 18:07:56 <nirik> logs processor 18:07:59 <lmacken> (aka ELK) 18:08:17 <nirik> smooge: ^ 18:08:46 <smooge> sorry 2 meetings 18:08:47 <aikidouke> any particular features you are wanting smooge? 18:09:19 <smooge> well at this point I would like to get a better idea of what our logs have in them 18:09:28 <smooge> and then figure out how to make reports that work well 18:09:47 <smooge> we have multiple terabytes of old log data and other stuff put through different places 18:09:48 <nirik> smooge: this is for httpd logs? or ? 18:10:00 * doteast has a splunk T-shirt 18:10:03 <smooge> part of it 18:10:11 <lmacken> many years ago I had a node in our cloud setup with ELK, the only issue was a lack of memory & disk. https://www.elastic.co/products/kibana 18:10:12 * threebean is here 18:10:19 * pcreech is here 18:10:48 <nirik> elasticsearch is in fedora now right? but not sure about the other parts. 18:10:49 <smooge> the httpd is where I am starting but I would like to get an idea of fedmsg and other tools 18:10:59 <aikidouke> ok so the requirement is something generic and useful that lets us define our requirements :) 18:11:12 <lmacken> nirik: yeah it is 18:11:54 <nirik> looks like the other parts are not. ;) 18:12:14 <nirik> overall for system logs, epylog stuff works ok, but that might just be because I am used to it. 18:12:18 <smooge> yeah there are a bunch of things which are listed as "no one packages this up we just wget | su -" 18:12:19 <lmacken> logstash is in copr 18:12:31 <lmacken> and kibana 18:13:01 <smooge> anyway this is more of a long term project that I am working on this year. 18:13:31 <nirik> might be worth a cloud node to play with the ones around and see if any are so awesome they are worth packaging up more. 18:15:40 <nirik> anyhow, I guess we can move along... 18:15:55 <nirik> #topic Docs-focused Tech Debt Fighting week is coming up - threebean 18:16:14 <nirik> so when did we schedule this for again? I think I didn't note it. ;( 18:17:59 <threebean> iirc it was right after freeze 18:18:05 <threebean> and freeze is next tuesday (the 8th?) 18:18:15 <nirik> yeah, looking back for exactly when 18:18:28 <aikidouke> do you have any areas you want to focus on threebean? 18:18:53 <threebean> aikidouke: there's plenty. the infra-docs repo SOPs could use some work. 18:19:13 <threebean> there are the csi ansible vars for each host group which are about half complete (I think) 18:19:38 <threebean> a number of our apps are sufficiently complex that they could use an architecture diagram to help people understand what's going on. badges, bodhi, pagure, etc.. 18:20:17 <threebean> and then just about all of our apps could benefit from someone going through and adding inline comments to the python code, elaborating on what this does or what that does. 18:20:36 <nirik> man, I am not finding when we said we were going to do it. weird. 18:21:24 <threebean> we assessed it two or maybe three weeks after the originally scheduled one in early January. 18:21:32 <nirik> ah ha 18:21:35 <nirik> Docs clean up week (infra docs, applications documentation, wiki) on 2016-03-08 to 2016-03-14 (nirik, 18:37:59) 18:22:05 <nirik> so yeah, as soon as freeze starts. 18:22:37 <nirik> I think I might try and focus on the wiki. it's rotted a fair bit on our pages I think. 18:22:38 <threebean> yeah, https://meetbot.fedoraproject.org/teams/infrastructure/infrastructure.2016-01-21-18.00.log.html 18:22:49 * pingou late 18:22:53 <threebean> +1 to wiki work. 18:23:22 <threebean> lmacken: someone (maybe kparal) was asking about the bodhi sop last week.. and I think all the info there is still bodhi1-specific. those could use updating. 18:23:50 <nirik> so did we want to try and have some kind of short daily meetup to discuss what peopel are working on, etc? 18:24:16 <aikidouke> +1 18:24:27 * threebean nods 18:24:40 <threebean> how about at 15:00 UTC each day? 18:25:17 <pingou> +1 for the daily meetup 18:25:21 <nirik> thats a bit early for me, but could try 18:25:31 <threebean> nirik: 16:00 UTC, then. :) 18:25:31 <pingou> nirik: what time would this be for you? 18:25:52 <nirik> hard to say, depends on how many fires. ;) But yeah, 16 would be better (thats 9am) 18:26:24 <nirik> lets try for 16... in #fedora-admin? 18:26:31 * pingou prefers 15utc but is ok with 16 18:26:49 <pingou> looks like we've got a time and a location :) 18:27:05 * threebean nods 18:27:14 <nirik> sounds good. Hopefully things will be quiet during freeze and we can get some stuff done. ;) 18:27:24 <pingou> \ó/ 18:27:33 <pingou> so the global theme is: doc? 18:27:35 <threebean> sounds good here :) 18:27:37 <threebean> pingou: yeah 18:27:38 <nirik> pingou: yep. 18:27:56 <nirik> infra-docs, sops, wiki, etc. 18:27:58 <pingou> cool 18:27:58 * threebean will make a wiki page to record the stuff we did like last time. 18:28:37 <threebean> look for it at 18:28:39 <threebean> https://fedoraproject.org/wiki/Infrastructure/Debt 18:28:47 <nirik> #info Join us each day starting 2016-03-08 until 2016-03-14 at 16:00UTC and we will work on updating documentation. 18:28:55 <nirik> sounds good. 18:29:08 <nirik> anything else on this? or shall we move on? 18:29:33 <nirik> #topic Spring cleaning - kevin 18:29:42 <nirik> so, I sent a status report to the list... 18:29:54 <nirik> and I am planning on repaving most of stg today. ;) 18:30:03 <nirik> but we will not get everything reinstalled before tuesday I am pretty sure. 18:30:49 <pingou> do we want to even try rebuilding some of the prod hosts before freeze? 18:31:08 <nirik> we could if folks have time/desire... and they don't cause outages. 18:31:29 <pingou> once stg is done I'm ok with doing fedocal, pkgdb, elections & co :) 18:31:36 <nirik> one way I thought of: find everything with 2 frontends/webs, and go do all the 01's... then go back and do the 02's 18:31:55 <nirik> sounds good to me if you have time to... 18:32:11 <pingou> I can make it :) 18:32:33 <pingou> just want a green flag from you first to be sure I don't do anything stupid :) 18:32:36 <nirik> Oh, one other note: I setup our kickstarts to make a ansible local fact on install. 18:33:00 <nirik> so, any machines reinstalled in the last weeks will have a /etc/ansible/facts.d/install_time.fact on them 18:33:07 <nirik> which lists the date they were installed on 18:33:13 <pingou> cool 18:33:32 <nirik> so we can easily see ones we need to still reinstall with the new cool stuff. ;) 18:33:50 <nirik> also, I tested a live migration (busgateway01) and it worked just dandy 18:34:01 <nirik> doteast was going to commit a manual playbook that will do migrations. 18:34:44 <nirik> if I have time tomorrow/this weekend I might move some stuff around and do some more virthosts. 18:35:10 <nirik> Is everyone good with me redoing staging this afternoon? I have not heard any yells, but just want to make sure... 18:35:51 <nirik> ok. I guess I will see if anyone complains. ;) 18:36:03 <nirik> #topic Easyfixes - kevin 18:36:12 <nirik> I thought I would do a quick run of easyfix tickets again... 18:36:40 <nirik> https://fedorahosted.org/fedora-infrastructure/report/14 18:36:59 <nirik> .ticket 3294 18:37:02 <zodbot> nirik: #3294 (Enable varnish caching for applications) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/3294 18:37:20 <nirik> I think puiterwijk still hasn't had a chance to get to this one... 18:37:36 <nirik> is there anyone else who might know our varnish setup and be able to review? 18:38:08 <nirik> .ticket 4485 18:38:10 <zodbot> nirik: #4485 (run POC-Change script for EPEL) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/4485 18:38:27 <pcreech> That one I took a couple meetings ago 18:38:27 <nirik> still open and seems ok as an easyfix to me. 18:38:43 <nirik> pcreech: ah, ok, any news? anything we can help with? 18:38:46 <pcreech> i just haven't been able to get around to it. (We are at the end of a release cycle for dayjob) 18:38:53 <nirik> no worries. ;) 18:39:01 <nirik> .ticket 4507 18:39:02 <zodbot> nirik: #4507 (kojipkgs redirects from https to http for directories with missing trailing slashes) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/4507 18:39:14 <nirik> this is a slipperly one, but I think two folks wanted to work on it anyhow? 18:39:16 <pcreech> I started looking into the architecture on that one 18:39:17 * nirik can't recall who. 18:39:29 <pcreech> it was me and linuxmodder who took it last time 18:39:34 <nirik> ah ha. ok. 18:39:48 <nirik> .ticket 4973 18:39:49 <zodbot> nirik: #4973 (add nagios check for mailman01's REST interface.) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/4973 18:40:18 <nirik> abompard: you happen to be around? any input on that one? we should figure it out. 18:40:29 <nirik> .ticket 5096 18:40:31 <zodbot> nirik: #5096 (Add "Connection Info" back into CommBlog) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/5096 18:40:52 <nirik> I think we wanted to remove easyfix from this. 18:41:23 * nirik does so 18:41:32 <nirik> .ticket 5128 18:41:33 <zodbot> nirik: #5128 (find and clean up any references to puppet in infra-docs) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/5128 18:41:39 <nirik> this goes right along with docs week next week. ;) 18:41:46 <nirik> .ticket 5131 18:41:48 <zodbot> nirik: #5131 (Setup /etc/profile as a template that sets prompt per env) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/5131 18:41:59 <nirik> this hasn't been claimed yet. 18:42:02 <nirik> ok, thats all of them. 18:42:20 <nirik> I don't have any 'learn about' today... so on to open floor 18:42:23 <nirik> #topic Open Floor 18:42:25 <threebean> +1 to #5131 18:42:27 <threebean> that would be awesome :) 18:42:33 <nirik> anyone have anything for open floor? 18:42:39 <nirik> threebean: yeah, might be handy sometimes. 18:43:31 <nirik> ok, if nothing else will close out in a minute... 18:43:44 * pingou had something but can't remember it :] 18:43:57 <threebean> thinking out loud here.. I have this auto-generated nagios config project I keep poking at on the side 18:44:16 <threebean> I should be able to work on it during freeze, because I won't be operating on noc01, but on a dummy host I'll set up for testing. 18:44:26 <nirik> threebean: yeah, I was going to look at making time this year to revamp monitoring... but still that could be good (unless we ditch nagios) 18:44:36 <threebean> but.. I guess I'll need to get all the prod hosts to allow the dummy host to connect before freeze sets in.. to make it possible to really test it. 18:45:02 <nirik> hum. well, could make a noc01.stg and have it monitor stg hosts only? 18:45:10 <nirik> or that won't work if it's generating for everything? 18:45:28 <threebean> I could put a staging conditional in there and then remove it later. yeah, that would work. 18:45:39 <threebean> nirik: although if you're really considering ditching nagios, maybe I should hold off for now? 18:46:02 <nirik> threebean: well, I haven't had time to look around much... last time I did, everything kinda sucked, but nagios sucked least. ;) 18:46:19 <threebean> I think that might still be the case :p 18:46:22 <nirik> I'd like to give http://assimilationsystems.com/ a good look... 18:46:32 <nirik> but I am not sure it's far enough along to handle all the stuff we want. 18:47:14 <nirik> basically it autodescovers everything... so there's almost no config 18:47:37 <pingou> that's appealing 18:47:53 <doteast> snmp based? 18:47:56 <nirik> but not sure the story on web interface or alerting or a bunch of things. 18:48:35 <nirik> doteast: nope. You run nanoprobes on machines and they figure out whats running on the host and how the switches are connected (via lldp, etc). Then they talk to each other 18:49:12 <doteast> cool 18:49:31 <doteast> would this be hindered if setup during freez? 18:49:47 <nirik> anyhow, dunno. I guess perhaps you could hold off for a bit more if you have other stuff to do... or go ahead and we can see if nagios fixed would be up to all our needs 18:50:11 <nirik> doteast: well, we couldn't install any of the nanoprobes on frozen hosts without a freeze break. 18:50:36 <doteast> same on staging? 18:50:49 <nirik> no, we could test on stg. 18:50:56 <nirik> just a matter of time 18:51:02 <doteast> I see 18:51:29 <nirik> ok, lets discuss further out of meeting. ;) 18:51:35 <nirik> Thanks for coming everyone! 18:51:37 <nirik> #endmeeting