18:00:56 <nirik> #startmeeting Infrastructure (2012-05-10)
18:00:56 <zodbot> Meeting started Thu May 10 18:00:56 2012 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:56 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:56 <nirik> #meetingname infrastructure
18:00:56 <zodbot> The meeting name has been set to 'infrastructure'
18:00:56 <nirik> #topic Robot Roll Call
18:00:56 <nirik> #chair smooge skvidal CodeBlock ricky nirik abadger1999 lmacken dgilmore mdomsch threebean
18:00:56 <zodbot> Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge threebean
18:01:01 * relrod here
18:01:02 * skvidal is here
18:01:04 * abadger1999 here
18:01:08 * wolfkit is here
18:01:08 <smooge> crrowbot here
18:01:12 * lmacken 
18:01:15 <dgilmore> hola
18:01:20 * jds2001 
18:01:21 <Neldogz> neldogz is here
18:01:42 <herlo> here
18:02:07 * nirik waves
18:02:13 * SilentBob is here
18:03:00 <nirik> #topic New folks introductions and Apprentice tasks.
18:03:00 <nirik> If any new folks want to give a quick one line bio or any apprentices
18:03:00 <nirik> would like to ask general questions, they can do so now. Anyone?
18:03:36 * nirik waits a min for any apprentices
18:03:49 <Neldogz> Sure, I can start.. I am new here , have about 10 years of general IT experience, strong within the server and networking side
18:04:19 <nirik> welcome Neldogz. You interested in sysadmin type stuff? or application development?
18:04:50 <Neldogz> thank you nirik, I am mostly interested in sysadmin work. monitoring, patching, fixing issues
18:05:09 <skvidal> Neldogz: which monitoring frameworks have you used?
18:05:13 <nirik> great. see me in #fedora-admin after the meeting and we can see about setting you up.
18:05:29 <skvidal> (yah - what nirik said) :)
18:05:42 <Neldogz> will do! skvidal, i have used PRTG and HP SIM for monitoring
18:05:53 <Neldogz> I am also currently learning Nagios
18:06:01 <nirik> any other new folks? or apprentices wanting to talk about tasks?
18:06:10 <whiterhino> I am new as well and about the same stats are Neldogz and would also like to get more involved in the system admin side
18:06:38 <nirik> welcome whiterhino
18:07:09 <whiterhino> I have used PRTG and nagios in the past, I currently use icinga and cacti
18:07:12 <whiterhino> thanks
18:07:40 <nirik> we use nagios here... and it needs some rework that various people have looked into. ;)
18:08:06 <nirik> anyhow, moving along.
18:08:13 <nirik> #topic two factor auth status
18:08:21 <nirik> I think we have some movement in this this week?
18:08:27 <wolfkit> this is coming along quite nicely :)
18:08:44 <wolfkit> I have the pam_url module escaping usernames and passwords, and has been committed to the pam_url source tree
18:08:52 <nirik> excellent.
18:08:58 <wolfkit> there is an odd segfault on 64-bit machines with it.. but I'm still looking in to that
18:09:24 <nirik> wolfkit: would you be willing also to package it up for Fedora/RHEL?
18:09:49 <wolfkit> hehe, kind of one step ahead of you, was playing around with a Fedora package for it last night :)
18:10:00 <wolfkit> haven't tested it on RHEL, can do though
18:10:06 <nirik> cool. I would be happy to review if you like.
18:10:26 <nirik> I think the cgi side also had a bunch of work this week..
18:10:33 <wolfkit> great!  will file a ticket when it is ready
18:11:07 <skvidal> and some more items
18:11:07 <nirik> thanks
18:11:13 <skvidal> mricon did new patches to the totp cgi
18:11:17 <nirik> #info pam_url is nearly ready to package up and use.
18:11:18 <skvidal> that does the checking of the keys
18:11:32 <skvidal> specifically he added an 'encrypt the seed with the users passphrase/pin'
18:11:53 <nirik> https://github.com/mricon/totp-cgi <- url for cgi
18:11:54 <skvidal> so the user can have the opened seed on their end - but we only have the one encrypted with their pin/passphrase
18:12:11 <skvidal> this means we never have any unencrypted goo sitting on our system somewhere
18:12:25 <skvidal> now a user could get compromised, of course
18:12:32 <skvidal> but that just gives them one key
18:12:34 <skvidal> not all of them
18:12:38 <skvidal> s/key/seed/
18:12:43 <nirik> great.
18:12:56 <skvidal> it is terribly neat, actuall
18:12:56 <skvidal> y
18:13:19 <nirik> provided the pins aren't easily brute forceable... ie, not like 4 char number only.
18:13:55 <nirik> ok, anything else on this? or shall we move on?
18:14:14 <skvidal> that's it
18:14:23 <skvidal> nirik: I  think the pins can be whatever...
18:14:40 <nirik> yeah, so we can make it something reasonable (although I don't think they need as much as passwords)
18:14:46 <skvidal> agreed
18:14:47 <nirik> anyhow, moving along...
18:14:49 <nirik> #topic Staging re-work status
18:14:50 <skvidal> nod
18:14:53 <nirik> this is done!
18:14:57 <nirik> #info done!
18:15:05 <skvidal> yay!
18:15:06 <abadger1999> Cool!
18:15:09 <skvidal> and there was much rejoicin
18:15:11 <herlo> +1
18:15:11 <wolfkit> awesome :D
18:15:12 <nirik> let me know if you run into any problems with staging in the new world order.
18:15:43 <nirik> #topic Applications status / discussion
18:15:58 <nirik> any applications news this week? abadger1999 / lmacken / threebean / pingou / CodeBlock
18:16:22 <lmacken> not really, just a lot of bodhi2.0 hacking.
18:16:33 <lmacken> I still need to find a couple of hours to wrap up the packages/tagger deployment
18:16:52 <abadger1999> none from me
18:16:53 <nirik> lmacken: cool. We should still setup a meeting sometime for 2.0 discussions... perhaps after f17 goes gold...
18:17:25 <lmacken> nirik: sounds good
18:18:06 <threebean> nothing major new with messaging.  lots of package reviews (almost done!)
18:18:16 <nirik> oh, I'll note that now in staging we no longer have a rhel5 host... but the only rhel5 thing we have left is app07 to run old community.
18:18:29 <nirik> so, hopefully that can putter along until we retire it.
18:18:29 <skvidal> yay for no rhel5!
18:18:36 <lmacken> no more py2.4 :)
18:19:22 <lmacken> sadly, 2.6 is already feeling old :(
18:19:34 <nirik> I think bapp01 and xen04 (soon to be stopped) and app07 are all we have left now. so, thats good.
18:20:16 <abadger1999> (community and smolt's cron job)
18:20:29 <nirik> oh, and app01.dev I guess...
18:20:33 <nirik> we need to redo that sometime.
18:20:46 * relrod can do that, just let me know when is good
18:21:03 <nirik> I think there's some active development going on on it... but I could be wrong... abadger1999 ?
18:21:04 <smooge> lmacken, its ok.. we can move to 2.8 when it comes out
18:21:15 <abadger1999> yeah
18:21:38 <abadger1999> relrod: I'll have to coordinate that -- backup stuff and put on the new hosts
18:22:04 <abadger1999> relrod: I'd also like to split it into two hosts -- a fas-development server (for developing fas, not a fas for the dev env)
18:22:10 <nirik> #info will coordinate app01.dev re-install.
18:22:10 <abadger1999> relrod: and a pkgdb development server.
18:22:24 <threebean> lmacken: we might think about building bodhi2 on py3.  pyramid is already compatible (and mako and sqlalchemy).
18:22:32 <abadger1999> relrod: But if you wanted to create the new hosts.. I could just start moving the code over?
18:22:54 <relrod> abadger1999: Sounds good. Will start on that tonight after finals or tomorrow.
18:22:54 <lmacken> threebean: good call, I'll create a py3 virtualenv for my local development.
18:23:00 <abadger1999> relrod: Cool.
18:23:19 <abadger1999> uhhh...
18:23:24 <nirik> is there a python3 in rhel/epel6?
18:23:29 <lmacken> yup
18:23:29 <abadger1999> Not sure if I'd support that unless
18:23:34 <threebean> :D
18:23:44 <abadger1999> we do something about the py3 version in RHEL5/6 vs Fedora
18:23:47 <abadger1999> err
18:23:54 <abadger1999> just epel6
18:24:16 <abadger1999> unles dmalcolm already updated?
18:24:19 * nirik doesn't see it.
18:24:27 * abadger1999 checks whats available
18:24:37 * abadger1999 doesn't either
18:24:53 <abadger1999> so... we'd need a whole new stack of deps...
18:25:01 <lmacken> I think he's working on it
18:25:04 <abadger1999> lots of packaging work if you want to go that route.
18:25:07 <lmacken> and yeah, it'll require a lot of packaging tweaks
18:25:12 <lmacken> but, inevitable tweaks.
18:25:14 <abadger1999> most spec files will have conditionals
18:25:31 <abadger1999> should be reasonable to get those changed.
18:26:08 <abadger1999> and since it's not in yet, you won't have to worry about it being a really old version of py3
18:26:10 <nirik> how do things like mod_wsgi handle multiple python versions?
18:26:29 <abadger1999> otoh... it does mean that we'll have to support both py2 and py3.
18:26:34 <abadger1999> nirik: They don't
18:26:42 <abadger1999> so we'd need two sets of app servers too....
18:26:51 <nirik> yeah, so we would need more servers...
18:26:51 <nirik> yeah
18:26:56 <abadger1999> logisitcs of this seems more and more suspect.
18:27:03 <lmacken> yeah, true :)
18:27:08 <jds2001> why two sets?
18:27:16 <jds2001> instead of two instances?
18:27:23 <abadger1999> jds2001: TG1 and TG2 will never be able to shift over to python3
18:27:31 <abadger1999> because some of there deps are never going to port.
18:27:35 <nirik> jds2001: you mean multiple httpds?
18:27:39 <wolfkit> perhaps not necessarily building bodhi2 specifically for py3, but testing on py3 to ensure it's working / compatible with it for a later deployment?
18:27:40 <jds2001> nirik: yeah
18:27:44 <threebean> lmacken: build it with python-six so bodhi2 can run on py3 someday... someday.
18:27:47 <threebean> wolfkit: yeah
18:27:56 <lmacken> threebean: yeah, python-six is probably the way to go
18:28:06 <abadger1999> so until all of those are ported to different toolkits, we'd need to support modules for python2 and modules for python3.
18:28:15 <nirik> I suppose that could work, but it's always a bit odd to do that... ie, never know which server restarts when you do a restart, etc.
18:28:15 <abadger1999> <nod>
18:28:51 <abadger1999> threebean: six +1... That could be a good way forward -- also the next py3 release will make u"string" valid again
18:28:54 <abadger1999> so that will help
18:29:15 <threebean> :)
18:29:18 <jds2001> nirik: many moons ago at my previous job, I'd ran 20+ httpd instances on one box.
18:29:25 <nirik> crazy. ;)
18:29:30 <jds2001> and could restart each independently :)
18:29:38 <nirik> anyhow, worth investigating... we don't have to decide now.
18:29:46 <nirik> any other application news?
18:29:50 <lmacken> speakin of mod_wsgi, I've been pinging upstream about my hash seed patch. dead air :(
18:30:09 <lmacken> in the mean time, we could potentially do the apache init script hack to enable hash seed randomization
18:30:27 <lmacken> or, we could ship a custom mod_wsgi with my patch
18:30:55 * lmacken looks to see if our python in production already supports it
18:31:03 <nirik> probibly the init script hack would be easier as a hotfix... custom mod_wsgi would require us to keep on our toes updating.
18:31:10 <lmacken> yep
18:31:22 <nirik> but also note that we are in freeze... so we need to be careful what we change
18:31:25 <lmacken> I already put the diff for that init script patch in the trac ticket
18:31:35 <lmacken> yeah, probably will have to wait in that case
18:32:01 <nirik> ok.
18:32:03 <threebean> oh -- ryansb contributed an ircbot for fedmsg.  fun!
18:32:28 <nirik> cool.... irc message busing. ;)
18:32:55 <lmacken> yeah, our python already supports -R... should be simple to enable it for our apps.
18:33:13 <nirik> excellent.
18:33:15 <lmacken> (even though there is a bug in the python hash randomization http://bugs.python.org/issue14621)
18:33:35 <lmacken> I doubt any of our apps rely on dict ordering, but we'll still want to test everything in staging first
18:34:18 <nirik> yeah, perhaps we could do that soon and deploy after freeze... or if it looks fine, just do a freeze break and push it out.
18:34:22 <lmacken> my init script patch is here: https://fedorahosted.org/fedora-infrastructure/ticket/3169
18:35:08 <nirik> yeah
18:35:22 <lmacken> oh wait, I was looking at the wrong python... looks like we don't have the patch in production
18:35:35 <nirik> ah, ;(
18:35:50 <lmacken> hopefully a yum update should pull it in though... not positive
18:36:03 <nirik> I thought I saw that update go by
18:36:23 <nirik> but I guess not
18:36:51 <nirik> lets discuss further outside of meeting I guess. I can look into where that update is.
18:37:01 <nirik> moving along...
18:37:21 <nirik> #topic Upcoming Tasks/Items
18:37:36 <nirik> #info 2012-05-08 to 2012-05-22 FINAL FREEZE
18:37:36 <nirik> #info 2012-05-10 - drop inactive fi-apprentices
18:37:36 <nirik> #info 2012-05-11 - Skvidal out.
18:37:36 <nirik> #info 2012-05-22 - F17 release
18:37:36 <nirik> #info 2012-06-01 - nag fi-apprentices.
18:37:36 <nirik> #info 2011-06-03 - gitweb-cache removal day.
18:37:38 <nirik> #info 2012-06-08 OOW: osuosl01.fedoraproject.org
18:37:41 <nirik> #info 2012-06-17 OOW: sign-vault02.phx2.fedoraproject.org
18:37:42 <nirik> #info 2012-06-21 to 2012-07-04 Kevin is off on trains and boats.
18:37:44 <nirik> thats what I have for upcoming.
18:37:59 <nirik> anyone want to schedule something or note some upcoming task/work?
18:38:06 <skvidal> I'll note something
18:38:28 <skvidal> I am getting buildvm-05 - 08 setup today - and I will be starting a ftbfs run on them
18:38:40 <skvidal> I'll be basing the run from lockbox01 into some disk space there
18:38:51 <nirik> skvidal: cool. Is this against rawhide I assume?
18:38:56 <skvidal> yeah it will be
18:39:02 <Neldogz> nirik: I belive we can close easyfix ticket 3231. After the restart of the unbound service, nagios appears to be performing the check successfully. https://admin.fedoraproject.org/nagios/cgi-bin//extinfo.cgi?type=2&host=unbound-telia01&service=Unbound+443%2Ftcp
18:39:39 <nirik> Neldogz: yeah, odd. It was doing it for a while, then stopped. ;) Oh well.
18:40:02 <nirik> skvidal: anything we need to know if we need to stop it or the like?
18:40:12 <skvidal> nirik: no
18:40:15 <Neldogz> I wish we could pinpoint what exactly was causing the problem.
18:40:22 <skvidal> nirik: just kill the processes running as me on lockbox01
18:40:31 <skvidal> nirik: and you can either kill or reboot buildvmhost-02
18:40:42 <skvidal> it might eat up some disk space on lockbox01
18:40:48 <skvidal> I'm looking to make sure that won't happen now
18:40:48 <nirik> Neldogz: me too. oh well, we will see if it happens again I guess.
18:40:53 <nirik> skvidal: ok.
18:41:12 <skvidal> nirik: hmm - not a lot of disk there, is there...
18:41:12 <nirik> #info FTBFS run starting on buildvm05-08 and 02.
18:41:18 <skvidal> not 02
18:41:25 <smooge> question.. when does skvidal get back
18:41:31 <skvidal> buildvm-05 -> 08 - which are all on buildvmhost-02
18:41:36 <skvidal> smooge: I'll be back monday
18:41:39 <nirik> ah, right, sorry. Could be more clear there.
18:41:42 <skvidal> smooge: my little brother is graduating from college
18:41:50 <skvidal> so I'm away for that this weekend
18:41:55 <jds2001> skvidal: mine just did :)
18:42:00 <skvidal> I should be available via phone some of the time
18:42:01 <skvidal> jds2001: :)
18:42:07 <skvidal> but I wouldn't bet on it
18:42:12 <smooge> you kids
18:42:36 <Neldogz> nirik: can someone that has access re-enable the notifications for the tls/ssh dns check to unbound-telia01
18:42:47 <nirik> Neldogz: yeah, I can see after the meeting.
18:42:57 <Neldogz> cool
18:43:11 <nirik> #topic Open Floor
18:43:16 <nirik> anyone have items for open floor?
18:43:35 <skvidal> nirik: I have one item, maybe?
18:43:41 <nirik> sure, shoot...
18:43:45 <skvidal> ansible made it into epel-6 and fedora now as pkgs
18:43:51 <skvidal> so it is easier for folks to play/test with
18:43:56 <skvidal> it is still evolving
18:44:03 <skvidal> but I wanted to encourage folks to mess with it
18:44:16 <nirik> cool.
18:44:16 <skvidal> and complain baout things if you find issues
18:44:16 * nirik makes a note to try it out here.
18:44:23 <smooge> will be messing with it next week as I redeploy my home system
18:44:25 <skvidal> I'm using ansible's api to drive the ftbfs stuff
18:44:38 <skvidal> and using the playbooks to setup the buildvm boxes I'm doing today
18:44:48 <skvidal> but more eyes/complaints are good
18:44:51 <skvidal> thx
18:45:32 <nirik> #info ansible is in fedora/epel, please play with it.
18:46:21 <nirik> ok, anything else or shall we call it a meeting?
18:47:49 <nirik> ok, thanks for coming everyone!
18:47:52 <nirik> #endmeeting