18:00:48 <nirik> #startmeeting Infrastructure (2014-04-24)
18:00:48 <zodbot> Meeting started Thu Apr 24 18:00:48 2014 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:48 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:48 <nirik> #meetingname infrastructure
18:00:48 <nirik> #topic aloha
18:00:48 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk
18:00:48 <zodbot> The meeting name has been set to 'infrastructure'
18:00:48 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean
18:01:11 <janeznemanic> hi
18:01:20 * adimania is here
18:01:24 * pingou 
18:01:32 * ghostalker is here
18:01:54 <dgilmore> hola
18:02:23 * relrod here
18:02:23 * threebean is here
18:02:36 * bwood09 is here
18:03:06 <nirik> cool. lots of new folks too. ;)
18:03:09 <nirik> #topic New folks introductions and Apprentice tasks.
18:03:13 * lmacken 
18:03:22 <nirik> anyone new like to give a short introduction of themselves?
18:03:28 <nirik> or apprentices with questions or comments?
18:03:31 <ghostalker> Sure
18:03:55 <ghostalker> I'm a 28 year old Sr Systems Engineer looking to take my operations experience and development skills to the next level, while giving back to the community. I'm also the one who didn't include the IRC nick in the email to the infra distro yesterday.
18:04:17 <nirik> welcome ghostalker!
18:04:25 <bwood09> Guess I should too
18:04:26 <pingou> welcome on board :)
18:04:29 <nirik> are you more interested in sysadmin type things? or application development? or both?
18:05:25 <ghostalker> A little of both. I have been just lurking since yesterday trying to get an idea of how things are structured. My development tends to be around helping do SysAdmin things.
18:05:49 <nirik> excellent. Always a need for sysadmin tooling too. :)
18:05:54 <nirik> bwood09: welcome and go ahead. ;)
18:05:55 <kjs> hi
18:05:58 <bwood09> I'm a (mostly hobbiest) sysadmin that is really into open source projects, but not a full-on programmer. Figure it'd be fun to try working with you guys
18:06:04 <ghostalker> As stated though my Python is weak so I would like to focus on SysAdmin related things will learning my way around.
18:06:13 <ghostalker> while*
18:06:32 <bwood09> I'm trying to work on my python skills too
18:06:41 <threebean> welcome :)
18:07:01 <Penasio> .fas
18:07:01 <zodbot> Penasio: (fas <query>) -- Search the Fedora Account System usernames, full names, and email addresses for a match.
18:07:07 <kjs> I'm interested in women, drugs and alcohol...
18:07:16 <henderbj> Hello all... first time on Infraestructure meeting
18:07:24 <Penasio> hi all...
18:07:28 <kjs> hi bj
18:07:32 <nirik> great. :) See me after the meeting over in #fedora-admin and I can get anyone interested added to our apprentice group: https://fedoraproject.org/wiki/Infrastructure_Apprentice
18:07:38 <Penasio> first time to me too
18:07:55 <pingou> it's cool to see so many new folks :)
18:08:17 <kjs> nirik: is kevin ?
18:08:36 <henderbj> Tnx, nirik... i want to get into apprentice group
18:08:40 <nirik> kjs: thats me, yes. well, one of them anyhow.... it's a somewhat common name.
18:08:56 <kjs> thought i recognised your nick
18:08:59 <blob> hello -- I'm interested in volunteering with releng and/or infrastructure. I've got quite a bit of sysadmin and  development experience and currently work on a release engineering team for a large software project. I'm a long time fedora user and free software advocate that would like to give back.
18:09:19 <nirik> wow. lots of new folks ;) welcome blob as well.
18:09:39 <pingou> blob: cool
18:10:58 * oddshocks here late
18:11:03 * oddshocks in lecture
18:11:23 <pingou> oddshocks: enjoy
18:11:27 <nirik> Penasio / henderbj: care to introduce yourselves quickly and let us know if you are intested more in sysadmin or app devel?
18:11:51 <Penasio> My name is Claudio Penasio Jr.
18:11:54 * mpduty is here
18:12:13 <henderbj> Hello... i has been sysadmin a long time now... i can help there if you like
18:12:16 <Penasio> i'm Fedora user since 2003
18:12:37 <Penasio> sysadmin since 2000
18:12:45 <kjs> fyi im interested in sysadmin, but dont mind crossing the line every now and again, especially if it's security related.
18:12:49 <henderbj> I am willing to work in development too
18:12:57 <Penasio> i'm from ambassadors group
18:13:07 <webpigeon> O/
18:13:29 <henderbj> I have experience programming... most on C, and Visual C, for Windows and a little for linux
18:13:39 <Penasio> 2009 to now i'm working on IT support manager
18:13:40 <nirik> excellent. :) Welcome everyone. I can add interested sysadmin folks to our apprentice group after the meeting over in #fedora-admin and we have #fedora-apps for our application development stuff... we can point you to easyfix tickets to look at getting started, etc.
18:13:43 <Penasio> :(
18:13:55 <henderbj> I programmed on Pascal, Matlab, PIC (assembler and C)
18:14:22 <Penasio> easyfix
18:14:25 <nirik> Most of our application development is python related and various python frameworks. ;)
18:14:36 <Penasio> look's like good
18:14:46 <henderbj> Ok... time to learn puthon then ;)
18:14:47 <webpigeon> http://fedoraproject.org/easyfix/
18:14:47 <nirik> http://fedoraproject.org/easyfix/
18:14:53 <nirik> :)
18:14:56 <webpigeon> :)
18:14:58 <Penasio> i'll need some menthoring
18:15:09 <henderbj> python, sorry!!
18:15:32 <threebean> lots of us can help ;)  if you get stuck anywhere, be sure to ask (more than once.. sometimes, everyone is sleeping or something)
18:15:32 <nirik> Penasio: feel free to ask questions in any of our channels as you go and anyone available can answer and help you out. ;)
18:15:34 <henderbj> with u is kind of a bad word on spanish ;)
18:15:45 <nirik> henderbj: :)
18:16:06 <Penasio> thank's a lot
18:16:11 <pingou> henderbj: I guess it's close to the French one ^^
18:16:16 <Penasio> i'm looking the easyfix
18:16:19 <nirik> ok, anyone else new who hasn't typed up yet?
18:16:58 <nirik> ok, great. ;)
18:17:02 <nirik> welcome again everyone!
18:17:10 <nirik> #topic Applications status / discussion
18:17:16 <henderbj> Tnx for warm welcome :)
18:17:21 <nirik> any application development news this week?
18:17:28 <pingou> #info new fedocal in prod this week
18:17:29 <nirik> I saw there was a new copr release...
18:17:37 <pingou> #info new pkgdb2 in stg this week
18:17:39 <nirik> #info new copr release went out this morning with some minor bugfixes
18:17:43 <pingou> (likely another one coming tomorrow)
18:17:48 <pingou> (pkgdb2 not copr :))
18:18:01 <nirik> ok, cool.
18:18:14 <pingou> I spent today fix bugs and RFE on pkgdb2 after the announce I sent to the devel list
18:18:19 <pingou> which is nice :)
18:18:36 <lmacken> lots of bodhi2 development since PyCon. threebean has been working on a slick web fontend, and it's looking great.
18:18:41 * nirik looks for abompard. I wonder where we are with hyperkitty. Perhaps we could do some targeted testing on it and see about migrating a list or two?
18:18:43 <pingou> the other part of the day was spent on https://github.com/jmflinuxtx/kerneltest-harness/tree/master/frontend
18:19:02 <threebean> I bet abompard is vacationing still after pycon.. I think he was going to be travelling.
18:19:11 <pingou> which is a small app gather kernel test results
18:19:14 <nirik> for new folks: https://apps.fedoraproject.org/ has a overview of all these apps... where you can find more info on the source, etc.
18:19:27 <pingou> gathering*
18:19:37 <nirik> pingou: great. Thats redone in flask? or ?
18:19:43 <pingou> nirik: yes flask
18:19:54 <nirik> and it would use postgres?
18:20:00 <nirik> threebean: ah, gotcha
18:20:04 <pingou> Justing is working on import its data in it so we can test it a bit more
18:20:10 <pingou> I still have one idea for it
18:20:23 <pingou> nirik: yup, flask and sqlalchemy so we can use whatever we want :)
18:20:40 <threebean> great :)
18:20:43 <nirik> pingou: can you reply to the list post about that? so we keep people just watching the list in the loop
18:20:49 <pingou> nirik: will do
18:21:28 <threebean> i've been working on some monitoring stuff for fedmsg based on this stuff - http://threebean.org/blog/threading-moksha/
18:21:37 <nirik> lmacken: is that up on the dev instance? or just local ahcking so far?
18:21:55 <threebean> i've got new moksha rpms in our staging environment that are working well now.  soon to follow are some new nagios checks and collectd plugins.
18:22:29 <pingou> threebean: nice!
18:22:40 <pingou> threebean: you should blog about which approach you took finally :)
18:22:48 <threebean> oh, the last one.  :)
18:23:09 <pingou> dedicated queue per thread
18:23:28 * threebean nods
18:23:36 <nirik> another thing I wanted to bring up: the bugzilla cookie/token change. :) Seems it's pushed back now, but we could still try and be ready.
18:23:37 <lmacken> nirik: just local right now, it's super easy to setup and import a db snapshot
18:23:39 <pingou> oh on the app front, I made some nice progress on gitsync https://github.com/pypingou/gitsync (also with the help of threebean :))
18:24:11 * pingou sees the readme needs an update
18:24:15 <nirik> lmacken: nice. Question: is there anything private in bodhi db dumps? or could we make them public?
18:24:27 <lmacken> nirik: they've been public from day 1 of bodhi
18:24:44 <nirik> oh? I had no idea... :) where at?
18:24:54 <pingou> I also converted the DB model of mirrormanager to sqlalchemy, as a first step for MM2 :]
18:25:34 <threebean> nice.  abadger1999 was talking about working on a mirrormanager rewrite.  you may want to catch up with him about it when he's back from vacation.
18:25:39 <lmacken> nirik: https://fedorahosted.org/releases/b/o/bodhi/ - `make init` in bodhi1 pulls it in automatically, and the bodhi2 readme has instructions for importing it as well
18:25:50 <nirik> excellent. :)
18:25:52 <pingou> threebean: I've heard we would team up ;-)
18:26:46 <nirik> ok, any other application news or questions or comments?
18:27:24 <smooge> the sky is blue
18:27:30 <pingou> smooge: grey here
18:27:41 <adimania> dark black here :(
18:27:45 <nirik> sort of a off white ceiling color here. ;)
18:28:07 <nirik> #topic Sysadmin status / discussion
18:28:34 <nirik> ok, on the sysadmin side of things...
18:28:49 <nirik> #info sent out last announcement about ssl certs all redone for heartbleed
18:28:58 <nirik> hopefully we are done with heartbleed stuff now
18:29:17 <nirik> #info new virthost in ibiblio... ibiblio04 to replace 01 thats going end of life.
18:29:49 <bwood09> I have a quick question
18:29:50 <nirik> new machines in phx2 are going to be installed by smooge over the next few weeks.
18:29:56 <nirik> bwood09: sure, fire away anytime.
18:30:34 <bwood09> I was looking at the easyfix list; why are we switching from puppet to ansible? Since I'm new, I'd like to hear the thought behind that.
18:30:50 <bwood09> (Ticket 4220)
18:30:56 <nirik> bwood09: I suppose I should make a wiki page for it or something, as people keep asking about it. ;)
18:31:08 <nirik> basically ansible meets our needs better.
18:31:42 * mirek-hm is late
18:32:00 <relrod> I started working on porting the puppet nagios server module to ansible this morning, but still have a long way to go. No ETA yet.
18:32:37 <nirik> It's simpler, it's in python (and we are mostly a python shop), it's agentless so we don't need to run a agent on all machines, it uses ssh instead of a seperate protocol, it allows us to manage things that are different oses more easily (fedora and rhel), etc
18:32:45 <bwood09> alright
18:32:50 <henderbj> sysadmin work has levels, like level 1, level 2, level 3?
18:32:50 <bwood09> Makes sense, actually
18:33:11 <nirik> relrod: cool. threebean had some ideas around that a while back I think... how to autogenerate much of it?
18:33:54 <nirik> henderbj: well, there's various groups... the apprentice group allows people to login to a bunch of machines and see our git repos, etc... but it's read only, you need someone else to review and make the actual change...
18:33:55 <henderbj> where i worked, level 1 are those facing clients, level 2 is medium problems, and level 3 for developers problems
18:33:59 <relrod> hm ok. I just started doing a straight port, I figured we could automate it later if we decide to...but if we want to do it as we port, we can try
18:34:48 <nirik> henderbj: next are sysadmin-something groups... those are areas that can make changes... like 'sysadmin-noc' is people who can login to machines, has sudo on some and can commit changes, etc.
18:34:58 <threebean> relrod: yeah, I didn't have much :)
18:35:14 <henderbj> got it... tnx
18:35:18 <nirik> then there is sysadmin-main, which is the folks who have access to everything... thats a small group that people are added to only after a long record of contribution. ;)
18:35:35 <threebean> relrod: we could probably automatically determine things like dependencies between hosts and guests.  but services get trickier.
18:35:37 <nirik> so, speaking of nagios...
18:35:38 <henderbj> i read about it a little before
18:35:47 <pingou> nirik: and few craig of beers
18:36:03 <nirik> I wanted to do a new thing in the meeting every week in this section...
18:36:22 <nirik> I want to go over the last weeks nagios alerts, and see if we can come up with ways to cut them down. ;)
18:36:23 <pingou> oh, shiny new things every week! :)
18:36:33 <adimania> +1
18:36:48 <adimania> I am doing something every similar these days at work.
18:37:08 <adimania> We generate a report of all the alert sent out every day and review it.
18:37:27 <nirik> adimania: cool. nagios? do you have a script for it that we could reuse?
18:37:34 <nirik> I just manually looked before the meeting. ;)
18:37:49 <nirik> there were 77 alerts in the last 7 days.
18:38:00 <nirik> 11 of those were from unbound-telia01
18:38:14 <adimania> We just send a metric to graphite every time we create any noise.
18:38:31 <nirik> 29 of them were from network problems at coloamerica... various instances there
18:38:31 <adimania> and do an aggregate at the end of the day.
18:38:50 <nirik> http://paste.fedoraproject.org/96714/64701139 is the full list
18:40:05 <nirik> so, I think I will adjust unbound-telia01 to do more retries before alerting... or longer timeframe.
18:40:32 <nirik> any other patterns folks can see or things we could adjust?
18:41:13 <threebean> at very least, it can give admin apprentices something to look for in the coming week
18:41:17 <nirik> if you come up with any, please do let me know or propose changes. ;)
18:41:21 <pingou> ftr, there are a *bunch* of email on the queue on bastion
18:41:35 <pingou> -- 77340 Kbytes in 7426 Requests.
18:41:37 <nirik> pingou: thats always been the case... it's the mail gateway... so it gets deferred stuff
18:41:42 <henderbj> disconnection alerts are good
18:41:43 <pingou> ok
18:42:03 <henderbj> they inform about poor connections, bottlenecks, or something
18:42:22 <nirik> another source of alerts for a while (or perhaps it was just emails) was fas02... it was getting overloaded from time to time.
18:42:34 <webpigeon> couple of disk space alerts in there
18:42:34 <adimania> swap alerts?
18:42:35 <nirik> I tweaked it's config some and it's been much better
18:42:39 <henderbj> maybe the other side has high cpu usage, etc
18:43:41 <nirik> yeah, a few swap alerts... usually thats when something gets hammered with requests or there's some problem processing.
18:43:55 <nirik> you can all also look at nagios directly to see the exact times, etc
18:44:02 <nirik> https://admin.fedoraproject.org/nagios/
18:44:40 <nirik> also, there's a few easyfix tickets I filed about adding some machines to nagios. ;)
18:45:07 <nirik> #info ideas to reduce nagios alerts welcome.
18:45:20 <nirik> Finally on the sysadmin side, there may be a outage next week...
18:45:24 <adimania> I can pick that up, if no one else is looking for easyfixes.
18:45:41 <nirik> the storage folks are finally hoping to move our backend storage. They say it won't cause any problems, but we should schedule an outage just in case.
18:45:54 <pingou> +1
18:46:03 <nirik> adimania: feel free. I need to file another one too.... lots for everyone. ;)
18:46:22 <nirik> any other sysadmin news/questions/comments?
18:46:37 <nirik> #topic Upcoming Tasks/Items
18:46:38 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
18:46:46 <nirik> anything upcoming folks would like to note or schedule?
18:46:47 * webpigeon still needs to refine his dns script, but hasn't had time with work
18:46:50 <nirik> threebean: hows the fad looking?
18:47:01 <adimania> FUDCon APAC is in less than a month. I am looking for a really really easyfix puppet module which I can migrate to Ansible in a probable workshop.
18:47:06 <nirik> webpigeon: no worries. we will get there when we do. ;)
18:48:08 <threebean> nirik: good, I think.  we've got hotels and flights set at this point.
18:48:16 <threebean> https://fedoraproject.org/wiki/FAD_Bodhi2_Taskotron_2014
18:48:17 <pingou> threebean: nice :)
18:48:51 <nirik> cool.
18:48:57 <mirek-hm> I am working on ansible manifest to install recent open stack - I am sowhere in half a way. If somebody want to help, just ping me
18:49:02 <nirik> I still need to send out info asking for interest in an ansible one.
18:49:07 <threebean> unless I'm forgetting something, we just need to get there and make the magic happen.
18:49:30 <nirik> threebean: excellent. ;)
18:49:48 <nirik> mirek-hm: thanks for working on that... are you basing on icehouse?
18:49:58 <mirek-hm> nirik: yes
18:50:02 <nirik> great
18:50:56 <tflink> threebean: are there any plans to work on the 1st? I'm trying to figure out when to head up to Denver
18:50:57 <nirik> oh, one other thing upcoming... not next week, but the week after, I will be out mon-wed... really the 1st-5th.
18:51:06 <tflink> mirek-hm: what kind of help are you looking for?
18:51:47 <mirek-hm> tflink: at least some review would be welcome
18:51:52 <threebean> tflink: I was hoping to work then, but we haven't sorted that out.  bocecha is actually getting in on the 31st.
18:51:57 <pingou> nirik: the same week, I'll be out, Mond to Friday, 5th to 9th
18:52:27 <mirek-hm> tflink: or take some chapter from installation guide and implement it (that would be super cool)
18:52:39 <bwood09> Oh, wow, Nagios let me log in with my creds
18:53:02 <nirik> bwood09: yeah, should be open to any fas accounts for read only. It likely won't let you ack alerts or anything tho
18:53:10 <bwood09> Figured as much
18:53:44 <nirik> but you can see history and such.
18:53:54 <nirik> #topic Open Floor
18:53:59 <nirik> anyone have items for open floor?
18:54:18 <nirik> questions, comments, favorite chocolate?
18:54:22 <bwood09> nirik, also, I mentioned in my welcome email that I'm unemployed, I am planning on devoting my time to this pretty much like a full time job. That's not a bad thing, is it? lol
18:54:24 <smooge> blue
18:54:27 <pingou> nirik: do we have any web-monitor solution in place?
18:54:38 <adimania> how about the slimming advice :P
18:54:43 <pingou> I remember we had awstats but it's not working so much it is?
18:54:53 <nirik> bwood09: :) you're welcome to spend as much time as you like... just don't get burned out. :)
18:55:06 <tflink> threebean: I was waiting on that before making a hotel reservation but I suppose that I could just commute for a day or two. it's only an hour or so
18:55:09 <nirik> pingou: for web logs? or monitoring web pages for alerts?
18:55:17 <bwood09> I'll try not to. I really, really like doing sysadmin work
18:55:17 * pingou wonders if there is an equivalent of piwik in python
18:55:20 <smooge> fedoramagazine is updated
18:55:28 <pingou> nirik: web-logs
18:55:29 <nirik> smooge: cool! :)
18:55:36 <tflink> mirek-hm: where are the playbooks that you're working on?
18:55:36 <smooge> pingou, look for pywik?
18:55:38 <webpigeon> nirik: Galaxy, as it happens. (but cool that nagios lets people login with their FAS creds)
18:55:42 <nirik> pingou: we don't, but it would be good if we did.
18:55:45 <pingou> smooge: does it? :D
18:56:19 <smooge> pingou, maybe pywyk? or pypiwik? [I am just pulling your leg now.]
18:56:21 <pingou> https://github.com/stef/pywik
18:56:34 <mirek-hm> tflink: on my machine, only. But I will push them to github and send link to infra mailing list so you can review if you want, ok?
18:56:53 <nirik> mirek-hm: that would be great.
18:56:59 <threebean> tflink: well, we're all only in town for so long and I know we've got tons of stuff to do.. so I'd vote for "yes, let's plan to hack on June 1st"
18:57:28 <tflink> mirek-hm: yeah, that'd be great
18:57:48 <nirik> threebean: +1
18:57:54 <tflink> threebean: ok, I'll double check to make sure that mike can do the 1st as well
18:58:01 <lmacken> yeah, I'm not sure when people are showing up, but I'm cool with hacking any time :)
18:58:53 * nirik too
18:59:11 <lmacken> let's talk fad logistics in #fedora-apps. we need to figure out what to do other than slay code :)
18:59:15 <nirik> also, we should definitely be active in our usual channels and #fedora-fad during the fad. :)
18:59:49 <nirik> alright, we are out of time. Lets continue over in #fedora-admin, #fedora-apps and #fedora-noc. :)
18:59:53 <nirik> thanks for coming everyone!
18:59:57 <nirik> #endmeeting