19:00:03 <nirik> #startmeeting Infrastructure (2014-02-06)
19:00:03 <zodbot> Meeting started Thu Feb  6 19:00:03 2014 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:03 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:03 <nirik> #meetingname infrastructure
19:00:04 <nirik> #topic welcome everyone
19:00:04 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk
19:00:04 <zodbot> The meeting name has been set to 'infrastructure'
19:00:04 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean
19:00:13 * abadger1999 here
19:00:20 * johe is here
19:00:22 * relrod here, but sick. :(
19:00:25 <mariocav> hi
19:00:31 <janeznemanic> hi guys
19:00:39 * ausmarton here
19:00:39 <nirik> relrod: :(
19:00:40 * willo says gidday
19:00:55 * lmacken 
19:01:51 <nirik> #topic New folks introductions and Apprentice tasks
19:02:02 <nirik> any new folks like to introduce themselves?
19:02:08 <nirik> or apprentices with questions or comments?
19:02:43 <nirik> threebean and pingou are traveling today...
19:02:53 * maayke is here
19:03:57 <nirik> welcome maayke
19:04:33 * oddshocks here
19:05:10 <nirik> #topic Applications status / discussion
19:05:19 <nirik> any applications news or plans?
19:05:38 * nirik doesn't think too many of the applications folks are able to make it today
19:05:40 <lmacken> I'm wrapping up a few bug fixes now, but I plan on deploying a new bodhi release this week
19:06:06 <lmacken> fixing some of the odd editing race conditions :)
19:06:19 <nirik> awesome.
19:06:36 <nirik> lmacken: oh, I was going to make a bodhi-dev cloud instance wasn't I? I can do that today... for bodhi2
19:07:08 <lmacken> nirik: cool, sounds good. then we can test out the ansible playbook that janeznemanic wrote
19:07:45 <nirik> lmacken: yeah, I added some more comments there too... I guess we need to finish that up, then move on to releng stuff so we can move releng04/repel01 to ansible...
19:07:58 <nirik> there's lots of things on those that aren't really directly bodhi...
19:08:47 <lmacken> yeah, other than bodhi it'll be a bunch of cron jobs right?
19:08:56 <lmacken> some signing stuff maybe
19:09:13 <relrod> I'm hoping to have the Fedora Mobile side of FMN notifications done by the end of this weekend, provided I feel good enough to do anything :(
19:09:26 <nirik> yeah, cron jobs, signing stuff (client end), nfs
19:09:42 <nirik> relrod: cool.
19:09:54 <lmacken> relrod: very cool. I need to get my SDK setup again with my n5 so I can start playing with it again
19:10:18 <lmacken> I'm really excited to start getting fedmsgs to my phone :)
19:10:28 <relrod> :)
19:10:32 <nirik> so one thing I was wondering with FMN... but likely should ask threebean when he's around...
19:11:22 <nirik> could we use it for larger announcements someday? ie, push an email via it to notify everyone about some big issue? or to reset their passwords or something?
19:11:38 <nirik> just an idle thought
19:11:58 <lmacken> I know there is a plan to be able to have a certain group of people be able to send announcements
19:12:23 <nirik> cool.
19:12:54 <nirik> ok, any other applications news?
19:13:22 <nirik> #topic Sysadmin status / discussion
19:13:27 <lmacken> nirik: looks like fedmsg-announce already exists :) https://github.com/fedora-infra/fedmsg/blob/53e851f68f106bff919a591a3516f26d5b07c375/fedmsg/commands/announce.py
19:13:29 <nirik> so, on the sysadmin side...
19:13:39 <nirik> sweet
19:14:24 <nirik> I've been working to clean up all our ansible playbooks some more... we now have a daily cron job that runs --check --diff on all playbooks, so I have been working to move that output to 0.
19:14:44 <nirik> ie, make sure all playbooks are idempotent and hosts are synced up to them.
19:15:25 <nirik> if anyone wants to look, our ansible stuff is logging everything... on lockbox use '/srv/web/infra/ansible/scripts/logview -d today -s CHECK_DIFF:CHANGED -s CHECK_DIFF:FAILED | less'
19:15:57 <nirik> smooge will be in phx2 datacenter next week.
19:16:12 <smooge> yay me!
19:16:13 * ianweller joins from class
19:16:23 <willo> thks nirik, I just learnt a new word "idempotent".
19:16:23 <nirik> smooge: we should come up with a list/agenda for you...
19:16:37 <nirik> willo: yeah, it's a weird word, but important. ;)
19:16:46 <smooge> yes please.. otherwise I will just fight the alligators and forget the swamp.
19:17:13 <nirik> smooge: do you know if any new hw will get there in time for you being there? (I think it unlikely, but... )
19:17:54 <sart> (I missed the introductions due to a phone call, but I'm here, by the way)
19:19:05 <nirik> hey sart. Welcome. :) are you more interested in sysadmin or application devel type things? or both?
19:19:07 <smooge> nirik, no idea
19:19:20 <smooge> I am expecting it to all arrive after I leave
19:19:33 <sart> both, but I'm somewhat less of a noob in sysadmin
19:19:49 <nirik> smooge: yeah, me too, thats the way it goes. ;(
19:20:05 <nirik> sart: great. see me in #fedora-admin after the meeting and I can help get you started.
19:20:37 <sart> awesome, thanks for the welcome nirik
19:20:58 <nirik> no problem. :)
19:21:14 * SmootherFrOgZ here
19:21:21 <nirik> hey SmootherFrOgZ. ;)
19:21:50 <nirik> so, on other sysadmin stuff, we have some new hardware that will be wandering in over the next few weeks...
19:22:46 <nirik> Oh, and I tried tuning our pkgdb and elections stuff recently to stop us getting alerts on it all the time
19:22:54 <nirik> so far it seems to be working. Knock on wood.
19:23:01 <smooge> heck I am not sure the PPC hardware will get there when I am to be there
19:23:16 <abadger1999> cool.
19:23:17 <nirik> smooge: you might take some dvd's there to leave...
19:23:57 <nirik> oh, this is likely not an easyfix, but would be great if someone could come up with it: we need a glusterfs role for ansible. ;)
19:25:03 <nirik> Anyhow thats all that leaps to mind..
19:25:17 <nirik> #topic Upcoming Tasks/Items
19:25:17 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
19:25:26 <nirik> any upcoming items folks would like to note or schedule?
19:25:48 <nirik> There may be netapp work next tuesday. They have pushed it back twice tho, so it could get pushed again.
19:27:08 <nirik> #topic Open Floor
19:27:11 <nirik> anything for open floor?
19:27:18 <SmootherFrOgZ> yep
19:27:21 <johe> me too
19:27:28 <nirik> Oh, one thing if we had enough folks interested, we could talk about hotfixes in ansible sometime.
19:27:35 <nirik> SmootherFrOgZ: go ahead and then johe. ;)
19:27:37 <johe> +1
19:28:09 <willo> +1 hotfixes in ansible
19:28:21 <mariocav> +1
19:28:50 <SmootherFrOgZ> I started working on FAS 3.0. You can check the branch on github. and also in a process of writing down everything note (technical & functional) I had to people can have a look on what's coming
19:28:59 <SmootherFrOgZ> this is for end of this week
19:29:16 <webpigeon> link? :)
19:29:27 <nirik> SmootherFrOgZ: cool!
19:29:35 <lmacken> SmootherFrOgZ: excellent
19:29:58 <nirik> https://github.com/fedora-infra/fas/tree/FAS_3.0
19:29:59 <SmootherFrOgZ> I choosed pyramid as framework
19:30:08 <SmootherFrOgZ> choose*
19:30:09 <webpigeon> thanks nirik :)
19:30:49 <lmacken> SmootherFrOgZ: cool. It'll be nice to have a test suite for fas this time around ;)
19:30:56 <SmootherFrOgZ> So if you guys have any ideas of features you would like to talk/add/whatever, go to the issues' page
19:31:20 <SmootherFrOgZ> lmacken: definitely!
19:31:31 <nirik> will do. ;)
19:32:07 <nirik> SmootherFrOgZ: thanks. Anything else on this?
19:32:50 <SmootherFrOgZ> yeah, last flock we talked about permission for specific access from apps, fas 3.0 will be (i guess) the first one to have it :)
19:33:17 * SmootherFrOgZ done
19:33:32 * relrod was one thing - today is my two-year anniversary at Red Hat. I just want to give a huge thank you to the entire team, you've been awesome, and I could not ask for a better job.
19:33:34 <SmootherFrOgZ> for anuthing else come talk to me
19:33:35 <relrod> s/was/has/
19:33:36 <nirik> nice.
19:33:46 <SmootherFrOgZ> anything*
19:33:51 <nirik> relrod: congrats! very nice to have you around!
19:33:59 <nirik> johe: you had an item?
19:34:01 <SmootherFrOgZ> relrod: congrats!
19:34:03 <johe> well, i thought some weeks ago, someone told he would like to do a network map or layout, any news on that?  i would find it useful to see how machines interact, like what is redundant, how is it balanced, and so one
19:34:05 <willo> relrod: awesme
19:34:06 <relrod> thanks :)
19:34:10 <johe> relrod: gz
19:34:39 <willo> johe: that was me
19:34:48 <willo> still working on it.
19:35:04 <johe> willo: ah, okay, do you need some help?
19:35:09 <nirik> :) there is the nagios map, but yeah, it's not so good in some senses.
19:35:26 <nirik> I'd be happy to try and explain things after meeting (time permitting)
19:35:42 <johe> for example last week or so i came across hosted03 when it swapped, according to nagios there is a hosted04 but i dont know if the are redundant or standalone
19:36:04 <johe> nirik: okay
19:36:26 <mariocav> nirik: are the chats of #fedora-admin logged and publicly available?
19:36:27 <nirik> hosted04 is a 'warm spare' (it syncs data from 03 via rsync). So, if 03 died we could bring up 04 pretty quickly (hopefully)
19:36:48 <nirik> mariocav: there's no public logs that I know of. Some people might log in their clients...
19:37:00 <johe> nirik: but if i would restart apachte on 03 , 04 would take over?
19:37:10 <johe> -t
19:37:19 <nirik> johe: nope, the process is manual. It needs someone to switch dns to 04.
19:37:20 <willo> johe: sure, but I really wanted to something up first for others to then add to
19:37:39 <johe> nirik: willo okay
19:37:42 <nirik> They are in different datacenters. ;)
19:37:56 <willo> :)
19:37:57 <johe> i see
19:38:28 <willo> nirik: do you use nagios to trigger events on host down
19:38:43 <willo> such as do that dns switch for you automatically
19:39:04 <nirik> willo: not currently no. It could be pretty complex to make that happen...
19:39:10 <johe> such an automatication is not always useful
19:39:21 <nirik> and what happens if it drops and comes back up 30sec later?
19:39:27 <nirik> or was always up, but unreachable.
19:39:35 <nirik> (from just noc01)
19:39:41 <willo> yeah, true
19:39:55 <nirik> we do have nagios doing a few triggered actions...
19:40:01 <willo> interesting point that
19:40:05 <nirik> it can restart httpds in some cases.
19:40:06 <willo> have you heard of shinken
19:40:30 <nirik> not off hand
19:40:38 <willo> nagios rewrite in python
19:40:52 <nirik> huh, cool.
19:40:56 <willo> deals with unreachable from noc01 issue
19:41:09 <willo> check can be run distributed
19:41:40 <nirik> sure, there's always ways and tradeoffs.
19:41:55 <willo> so, you can check a host from another host that better aligns with the way your traffic flows
19:42:14 <nirik> but then you have to run a monitoring agent on it and allow access, etc...
19:42:58 <johe> could turn out in a very long discussion
19:43:06 <willo> true, and true
19:43:33 <nirik> anyhow, anything else general for the meeting? or shall we continue in #fedora-admin, #fedora-noc and #fedora-apps?
19:43:37 <willo> used nagios for years, now a big fan of shinken
19:44:00 * nirik is waiting for http://linux-ha.org/source-doc/assimilation/html/index.html to be usable. ;)
19:44:22 <willo> are drinks a regular occurence
19:44:28 <willo> I came prepared
19:44:47 <webpigeon> willo: I can't fine any licence infomation on the site?
19:44:51 <nirik> I'm having some lovely vanilla coffee myself. ;)
19:45:39 <nirik> webpigeon: AGPL it looks like
19:45:44 <willo> webpigeon: https://github.com/naparuba/shinken
19:45:51 * relrod should make himself some tea with honey and lemon and hope that it helps re: being sick :/
19:45:54 <nirik> http://www.shinken-monitoring.org/participate/ (look down at the bottom)
19:46:27 <webpigeon> nirik/willo: thanks :)
19:46:39 * nirik notes it's not packaged in fedora/epel that I see.
19:46:57 <willo> it is, or maybe was
19:47:08 <nirik> anyhow will close out the meeting in a minute if nothing else...
19:47:11 <webpigeon> couldn't see COPYING or LICENCE in source :P
19:47:33 <willo> Slow Screw up against a Red Wall -- http://www.webtender.com/db/drink/6130
19:48:08 <nirik> alright, thanks for coming everyone!
19:48:11 <nirik> #endmeeting