17:59:59 <nirik> #startmeeting Infrastructure (2012-06-07)
17:59:59 <zodbot> Meeting started Thu Jun  7 17:59:59 2012 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:59:59 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:00 <nirik> #meetingname infrastructure
18:00:00 <zodbot> The meeting name has been set to 'infrastructure'
18:00:00 <nirik> #topic Is anyone there?
18:00:00 <nirik> #chair smooge skvidal CodeBlock ricky nirik abadger1999 lmacken dgilmore mdomsch threebean
18:00:00 <zodbot> Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge threebean
18:00:06 <skvidal> yah yah yah
18:00:09 * relrod here
18:00:14 * pingou here
18:00:42 * marcdeop here
18:00:49 * ianweller_ here
18:01:28 <nirik> ok, lets go ahead and start in...
18:01:28 * fcami__ here
18:01:29 <smooge> here
18:01:37 * threebean is here
18:01:57 <nirik> #topic New folks introductions and Apprentice tasks.
18:01:57 <nirik> If any new folks want to give a quick one line bio or any apprentices
18:01:57 <nirik> would like to ask general questions, they can do so here.
18:02:00 * rossdylan is here
18:02:16 <nirik> any new folks like to introduce themselves? or any general apprentice questions or concerns?
18:02:39 <rossdylan> I am a new red hat intern working on integrating open badges into fedora infrastructure
18:02:45 <rossdylan> Hi everyone
18:02:48 <nirik> welcome rossdylan
18:02:53 <threebean> Hi :)
18:02:55 <nirik> cool on the badges. ;)
18:02:56 <marcdeop> welcome rossdylan :)
18:03:12 <rossdylan> thanks :D
18:03:51 <nirik> ok, I guess lets move along then...
18:03:53 <nirik> #topic two factor auth status
18:04:03 <nirik> any news on this this week?
18:04:06 <skvidal> nothing really. I tried to get ahold of wolfkit
18:04:14 <skvidal> not much
18:04:20 <nirik> yeah, I've not seen him on irc of late much
18:04:34 <nirik> I get the idea we are close to ready to setup a staging/test
18:04:39 <skvidal> we are
18:04:55 <nirik> which will be very nice. ;)
18:04:57 <skvidal> it's round-tuits and a little bit of  code
18:05:01 <skvidal> also - I wanted to ask
18:05:14 <skvidal> has anyone in here used http://code.google.com/p/mod-auth-external/
18:05:16 <skvidal> and
18:05:20 <skvidal> http://code.google.com/p/pwauth/
18:05:26 <relrod> Yeah I haven't talked to wolfkit much recently either, I can try poking him via other means (I have contact with him outside of Fedora) and get him to check in
18:05:28 <skvidal> it's somewhat related
18:05:37 * nirik hasn't
18:05:37 <skvidal> relrod: that'd be great
18:05:48 <skvidal> so the gist of it is
18:05:55 <skvidal> we use mod-auth-external
18:06:01 <skvidal> coupled with pwauth
18:06:20 <skvidal> to do auth to fas2 for things like hosted trac, nagios, etc etc
18:06:32 <skvidal> w/o needing a direct db connection to fas2
18:06:41 <nirik> yeah, and that gets them to the same level as other places where the fas servers do not need to be up
18:06:53 <skvidal> yah
18:07:14 <nirik> (or the database as the case may be)
18:07:19 <skvidal> I've read the security warnings about both
18:07:27 <skvidal> and they don't seem more dramatic than anything else
18:07:41 <nirik> are they packaged up yet?
18:07:43 <skvidal> yes
18:07:44 <skvidal> in epel
18:07:47 <skvidal> all ready to roll
18:08:04 <nirik> cool. Sounds like a good thing to test out and see if there are any gotchas.
18:08:14 <skvidal> nod
18:08:19 <skvidal> it would, of course, HAVE to be ssl'd
18:08:23 <skvidal> but I think we assume that now anyway
18:08:25 <nirik> I was looking at mod_auth_openid... but it doesn't solve that problem (still need the fas openid provider to login)
18:08:45 <skvidal> so the only thing I am hazy on is this
18:08:51 <skvidal> pwauth needs to be setuid
18:09:00 <skvidal> so it can make the pam check at all
18:09:06 <skvidal> so the gist of it is
18:09:33 <skvidal> apache -> mod-auth-external -> pwauth -> pam -> talks to local pam auth
18:09:42 <nirik> we might run this idea by the RH security folks too?
18:09:43 <skvidal> then you can make a fall through to check for a group membership
18:10:04 <skvidal> maybe...
18:10:29 <skvidal> okay - that's all.
18:10:37 <nirik> ok, sounds good.
18:10:44 <skvidal> if anyone wants to setup those and test them - feel free to let me know
18:10:48 <skvidal> otherwise roundtuits
18:10:55 <nirik> #info will look at testing mod_auth_external / pwauth for nagios/trac
18:11:33 <nirik> #topic Applications status / discussion
18:11:46 <nirik> Any application news this week?
18:11:52 * pingou has not much
18:12:01 <threebean> oo, oo.  me!
18:12:05 <threebean> https://admin.stg.fedoraproject.org/updates (bodhi) and http://community01.dev.fedoraproject.org/tagger (tagger) are emitting zmq messages now.
18:12:06 <nirik> I hear that we had actual messages going accross fedmsg bus in staging this week? :)
18:12:19 <threebean> :)
18:12:23 <threebean> You can see them by logging onto a staging machine and running:  "$ fedmsg-tail --really-pretty" while poking at the apps.
18:12:39 <ianweller_> and that means it's time for me to start work on datanommer ;)
18:13:00 <threebean> I'm splitting up the config into /etc/fedmsg.d/ now (because /etc/fedmsg-config.py is ballooning).
18:13:03 <nirik> threebean: does that need to be any specific machine? or just anything?
18:13:04 <threebean> After that, I'll try standing up mediawiki+fedmsg in staging.
18:13:20 <threebean> nirik: should be anything in puppetEnvironment == 'staging'
18:13:24 <nirik> cool.
18:13:38 <smooge> tres cool
18:14:12 <nirik> speaking of mediawiki... smooge / ianweller_: whats next for mediawiki 119?
18:14:34 * relrod has a few small things - I worked with abadger1999 last night on making a new python-fedora release which adds a method to create FAS groups, which I needed for the fedorahosted app. Also did some testing with him making sure fas01.dev is working fine.
18:14:52 <ianweller_> i think since we're still working on mw118, we do that
18:14:53 <smooge> I need to build it in EPEL. I need to get the plugins reviewed
18:14:57 <ianweller_> wait
18:14:59 <ianweller_> we are doing 119
18:15:00 <ianweller_> that's right
18:15:05 * ianweller_ shuts up
18:15:19 <nirik> I can help with plugins.
18:15:27 <nirik> would be nice to get it updated. ;)
18:15:29 <smooge> right now the plugins are the ones we use in our private idaho
18:16:11 <smooge> my steps would be then would be update in a .dev. environment to see if it will work cleanly. Then do so in staging and then production
18:16:21 <nirik> yeah.
18:16:23 <smooge> ianweller_, did the last "update" so I will need info from him
18:16:30 <relrod> As far as the fedorahosted app itself, I'm working on getting the various fedorahosted SCMs processable via the CLI. Git requests should almost kindasorta work. Mailing lists should get created via ansible when we move lists to hosted-lists01 (I need to test the ansible bits locally, but it *should* work as-is).
18:16:38 <nirik> we can probibly just go to stg with it... do it on one of the apps there.
18:16:47 <ianweller_> that's fun, because i don't remember doing the last updat e;)
18:17:03 <threebean> relrod: that's awesome
18:17:14 <nirik> excellent.
18:17:29 <threebean> smooge, nirik: if it winds up in app*.stg, let's just communicate so aren't both fighting over mediawiki at the same time.  :)
18:17:29 <nirik> ianweller_: I'm sure we can sort it out. ;)
18:17:32 <ianweller_> smooge: are you wanting to work on this stuff today?
18:17:35 <ianweller_> smooge: or when
18:17:45 <nirik> threebean: yeah, agreed. Please coordinate.
18:18:34 <smooge> GNOME server went tango uniform again.. will be focusing on that for a few minutes
18:18:51 <skvidal> smooge: I hear if you just reseat everything it'll be fine
18:18:52 <skvidal> ;)
18:19:04 <nirik> In other application news, we moved fas db to it's own db server. This should prevent it from getting hit by load issues on other apps.
18:19:20 <skvidal> w00t
18:19:23 <nirik> smooge: ;(
18:19:28 <nirik> ok, any other application news?
18:19:30 <skvidal> we also gave db02.stg more ram
18:19:38 <skvidal> so hopefully it will stop crapping out occasionally
18:19:39 <nirik> oh yeah.
18:19:51 <threebean> rossdylan is making rapid progress on badges for week #1.
18:20:00 <nirik> skvidal: are we overcommited on any virthosts now?
18:20:05 <nirik> (on mem)
18:20:18 <skvidal> umm lemme look
18:20:35 <nirik> threebean / rossdylan: cool. Is there any place with a plan/overview of the badges setup? or place were folks could step up to help?
18:20:37 <rossdylan> indeed I am, you can now award badges to people (based on email) by listening on the fedmsg bus
18:20:59 <skvidal> nirik: running
18:21:00 <rossdylan> current dev is hosted on github.com/rossdylan/fedmsg
18:21:31 <threebean> nirik: nothing written down :/
18:21:52 <threebean> rossdylan: let's definitely do that so infra peeps can review it
18:21:56 <nirik> also, dunno if it's worth looking at to see if you could pull in, but our askbot instance has badges...
18:22:00 <skvidal> nirik: virthost10.phx2.fedoraproject.org:-479
18:22:07 <skvidal> still a little short there - but I bet we never hit it
18:22:24 <nirik> http://ask.fedoraproject.org/badges/
18:22:29 <skvidal> nirik: oh and virthost06.phx2.fedoraproject.org:-814
18:22:29 <nirik> skvidal: ok, good to know.
18:22:45 <skvidal> hmm
18:22:58 <skvidal> releng01, ask01, bapp02, packages01
18:23:08 <nirik> releng01 is bvirthost06. ;)
18:23:19 <rossdylan> nirik, we were thinking about creating scripts to dump existing badges (or existing activities that need awarding) into the newer open badges system
18:23:44 <nirik> rossdylan: sounds reasonable.
18:23:53 <nirik> ok, shall we move on then?
18:23:55 <skvidal> nirik: right - sorry - bad grep
18:24:13 <nirik> #topic Upcoming Tasks/Items
18:24:27 <nirik> here's what I have on my list for upcoming:
18:24:29 <nirik> #info 2012-06-08 OOW: osuosl01.fedoraproject.org
18:24:29 <nirik> #info 2012-06-11 remove people with pkgdb bugzilla issues.
18:24:29 <nirik> #info 2012-06-14 23UTC class A and B reboots
18:24:29 <nirik> #info 2012-06-17 OOW: sign-vault02.phx2.fedoraproject.org
18:24:30 <nirik> #info 2012-06-21 to 2012-07-04 Kevin is off on trains and boats.
18:24:32 <nirik> #info 2012-06-26 Fedora 15 end of life.
18:24:34 <nirik> #info 2012-06-28 Seth at jury duty.
18:24:46 <nirik> I'm going to ask fesco about the people who have bugzilla email issues...
18:24:53 <nirik> get permission to just remove them all.
18:25:02 <nirik> next wed we will be doing mass reboots.
18:25:14 <nirik> anything else folks would like to schedule or note?
18:26:14 <nirik> we need to reschedule the lists.fedorahosted move sometime.
18:26:35 <skvidal> does anyone here have any dyndns or round-robin dns mgmt experience?
18:26:58 <skvidal> I was looking for some input on if there is a vastly simpler way to do something we're currently doing in a kludgy, horrible way
18:26:59 <skvidal> :)
18:27:20 * nirik has avoided those setups in the past. ;)
18:27:35 <skvidal> well we're stuck with roundrobin
18:27:38 <abadger1999> nirik: I don't have the date handy but -- date that RH is turning of old bugzilla compatibility should probably be noted
18:27:40 <skvidal> I do not know a way around it for our proxies
18:27:47 <nirik> skvidal: oh, we might ask letoto, he's mr dns.
18:27:59 <nirik> abadger1999: good idea. I think I saw it somewhere, let me look.
18:27:59 <skvidal> nirik: I thought he only cared about dnssec
18:28:01 <smooge> skvidal, I do
18:28:52 <skvidal> smooge: so right now the process we use for updating dns for hte proxies is....
18:28:56 <skvidal> ridiculously complicated
18:28:59 <skvidal> 1. edit in 12 locations
18:29:01 <skvidal> 2. edit serial
18:29:03 <nirik> #topic Dns
18:29:05 <skvidal> 3. commit to git
18:29:08 <skvidal> 4. push
18:29:15 <skvidal> 5. force-puppet on ns\*
18:29:17 <skvidal> 6. wait
18:29:22 <skvidal> 7. do what you want to do
18:29:30 <skvidal> 8. do those again to put it all back
18:29:33 <marcdeop> wow, that's a lot of steps
18:29:46 <skvidal> sorry 12 is an exaggeration
18:29:52 <skvidal> it's only 6 locations
18:29:56 <smooge> yeah.. pretty much standard fair for doing the "we want people in X to use this proxy over that proxy"
18:30:06 <skvidal> so here's what I was thinking
18:30:16 <smooge> listening
18:30:24 <skvidal> we use the proxies just as fedoraproject.org and wildcard.fedoraproject.org
18:31:03 <skvidal> in most cases the sites are CNAMES to wildcard
18:31:21 <skvidal> (admin.fp.o, for example or www)
18:31:36 <smooge> yes
18:31:36 <skvidal> what I was thinking is that we setup a subdomain
18:31:43 <skvidal> dyn.fedoraproject.org
18:32:06 <skvidal> so we can edit that one domain - for changes to the proxies
18:32:15 <skvidal> and bump only the serial in that domain
18:32:31 <skvidal> the website names, of course, don't change
18:32:35 <skvidal> that's the point of the cnames
18:32:42 <skvidal> so if I go to www.fedoraproject.org
18:32:49 <skvidal> it currently goes to wildcard.fedoraproject.org
18:33:05 <skvidal> which RR dns to one of our proxy ips - dependent on which GEOip you're coming from
18:33:22 <skvidal> all we'd be changing is instead of going to wildcard.fedoraproject.org
18:33:28 <skvidal> it would be wildcard.dyn.fedoraproject.org
18:33:50 <skvidal> and instead of hand-editing the zones in puppet/git and pushing them out
18:34:01 <skvidal> we could script the creation/updating of that zone
18:34:26 <skvidal> just create a dictionary of ips and which regions they should exist in
18:34:32 <nirik> I think this is worth persuing.
18:34:36 <skvidal> and populate the various region-specific files that way
18:34:47 <smooge> skvidal, that sounds good.
18:34:47 <skvidal> so I guess what I'm wondering is..
18:34:52 <skvidal> has someone else already done this?
18:35:02 <nirik> not only for less editing/stuff when we change things, but allowing for automation (example nagios pulls a site thats down from dns for us)
18:35:36 <skvidal> it also seems like
18:35:39 <smooge> well we were using cobbler to do zones for building stuff and editing bind files that way. At another spot we had a shell script which did pretty much what you said for a limited domain
18:35:42 <skvidal> it would make the process less error prine
18:35:51 <skvidal> s/prine/prone/
18:35:56 <nirik> yes.
18:36:28 <nirik> also, ideally I'd prefer if there were a nag when it's not in the 'normal' config state. Currently, we take something out, and when it's back up, we forget to re-add it
18:37:19 <skvidal> okay..
18:37:22 <nirik> anyhow, I don't know of anything done like this before, but we could look around on the net?
18:37:23 <skvidal> here's what I might do
18:37:33 <skvidal> there's no reason to NOT setup the dns zone
18:37:37 <skvidal> and just not USE it for anything
18:37:45 <skvidal> ie: just test populating/updating the zones
18:37:54 <skvidal> but nothing is ever referred to inside it
18:37:55 <skvidal> right?
18:37:59 <skvidal> that seems harm-free to me
18:38:11 <nirik> yep. Should be fine.
18:38:20 <skvidal> so I may look into that as a start
18:38:26 <skvidal> unless someone else wants to take this on
18:39:05 * marcdeop is not sure has the knowledge need but wants to help skvidal out
18:39:08 * nirik listens to the silence. ;)
18:39:19 * marcdeop s/need/needed
18:39:28 <skvidal> marcdeop: do you have any dns experience?
18:39:35 <nirik> #info skvidal to work on more automated/dynamic zone updates for our primary zone
18:39:35 <marcdeop> a little bit, yes
18:40:03 <skvidal> marcdeop: well - all we're talking about is a script that lets us generate a pretty simple dns zone
18:40:11 <skvidal> it's really just a template
18:40:22 <skvidal> I bet I could use python jinja to most of it
18:40:33 <skvidal> the only tricky part is state maintenance
18:40:42 <skvidal> and making the commands make sense to someone updating it
18:40:53 <skvidal> marcdeop: I'll sketch up some pseudocode
18:40:58 <skvidal> and post to infra for complaints
18:41:04 <skvidal> if you want to work on fleshing it out - feel free
18:41:10 <nirik> sounds good.
18:41:39 <nirik> #topic Open Floor
18:41:48 <nirik> Any other items anyone has? questions? comments?
18:41:58 <skvidal> how about the clean up issues we encountered this week?
18:42:08 <smooge> I will be out next Thursday
18:42:13 <skvidal> smooge: have fun!
18:42:22 <nirik> skvidal: sure, would be good to note those...
18:42:23 <smooge> I am going to a security conference in Santa Fe for a couple of hours
18:42:33 <smooge> should be learning some new things
18:42:52 <smooge> hopefully not all Windows related :)
18:42:58 <nirik> smooge: cool.
18:43:00 <skvidal> nirik: well I was thinking of these items:
18:43:10 <marcdeop> couple of hours = next week? wooow
18:43:12 <marcdeop> :P
18:43:32 <skvidal> 1. all of sign-* hosts redo/work/update
18:43:35 <skvidal> 2. old puppet crap
18:43:55 <nirik> 3. firewalls?
18:43:57 <skvidal> 3. httpd::site vs httpd::website and porting things around
18:44:02 <skvidal> (firewalls, yes)
18:44:06 <skvidal> 4. fakefas reinstall
18:44:36 <skvidal> 5. it's been a while since the last time we changed global root pws
18:44:58 <relrod> 5 I was thinking about the other day and agree with.
18:45:10 <nirik> right. So, on 1. I am wondering if we should consider a late summer FAD for infrastructure security stuff. We could gather somewhere and work on redoing our sign stuff, finish implementing the 2factor auth stuff, talk about signing commits, talk about any other secuity deliverables we might want to have.
18:45:32 <nirik> we could also do 5 at that time with many of us in person?
18:45:47 <skvidal> nirik: root pw change is triviall, actually
18:45:53 <nirik> well, yeah, true.
18:45:54 <skvidal> i have a func script which can do it globally
18:46:08 <skvidal> speaking of that I made a change to our global ssh configuration this week
18:46:16 * marcdeop is sorry but has to leave. Hopes to read any stuff left in the log :)
18:46:22 <skvidal> such that it is now possible for us to allow root to login - but only using ssh keys
18:46:28 <nirik> marcdeop: no problem.
18:46:32 <skvidal> however, I have not enabled any authorized_keys
18:46:55 <nirik> I'm ok with a rootpw change anytime.
18:47:32 <skvidal> nirik: okay I'll jot it down
18:47:53 * nirik was also wanting to change the default IMM passwords/other places thats used.
18:47:58 <nirik> but thats more work
18:48:07 <relrod> nirik: when/where are you thinking about the FAD? I was thinking 'VFAD' but you said in-person
18:48:14 <skvidal> nirik: can we do that via the ssh connection?
18:48:20 <skvidal> nirik: b/c that would be WAY easier
18:48:21 <nirik> skvidal: not sure.
18:48:24 <skvidal> relrod: VFADs suck
18:48:29 <skvidal> relrod: b/c people are still distracted
18:48:45 <skvidal> let me rephrase
18:48:50 <skvidal> vfads do not suck
18:48:53 <skvidal> they suck for things like this
18:48:54 <nirik> relrod: yeah, if we can get funding... it occured to me that no one does FAD's anymore... but perhaps infra could put together a compelling one.
18:49:11 * pingou likes the idea
18:49:14 <relrod> Yeah. I think it'd be cool
18:49:17 <skvidal> in person, relatively isolated could work for hammering out something.
18:49:19 <nirik> not that I like traveling, but I bet we could get a lot done in a weekend face to face with less distractions.
18:49:31 <skvidal> or better yet! not a weekend! :)
18:49:34 <nirik> or at least a lot of plans to bring back
18:49:38 <nirik> sure, that too.
18:49:43 <pingou> a long week-end ? :)
18:50:02 <nirik> anyhow, it's just an idea.
18:50:12 <nirik> I'll run it by the list and see what interest there is.
18:50:16 <pingou> nirik: in your idea, where would it be held ?
18:50:16 * relrod likes it - if I can help plan it somehow let me know
18:50:29 <nirik> pingou: good question.
18:50:37 <nirik> we are pretty spread out...
18:50:39 <pingou> nirik: US is the basic answer, but east/west cost ?
18:50:54 <skvidal> I bet we could get a conference room in the RDU office for free
18:50:57 <nirik> I suspect it would come down to costs...
18:51:19 <nirik> where can we get space/net/etc... where is cheap to fly people into...where is cheap to have people stay
18:52:03 * relrod likes the idea a lot
18:52:05 <smooge> let us do one in Phoenix :). In the summer
18:52:09 <nirik> another possibility would be to try something next to fudcon... have folks come in a few days sooner or leave a few days later.
18:52:17 <nirik> but thats next year
18:53:13 <nirik> smooge: we could all sit next to the servers in the datacenter. ;)
18:53:25 <pingou> \ó/
18:53:32 <pingou> nice and cold
18:53:41 <nirik> but unfortunately, loud.
18:53:46 <skvidal> incredibly loud
18:53:56 <pingou> hear plugs ? :)
18:53:59 <nirik> anyhow, I'll send out a thing to the list and see what folks think...
18:53:59 <skvidal> I've never been in a quiet datacenter
18:54:05 <skvidal> nirik: cool
18:54:06 <pingou> and we can chat on irc :)
18:54:24 <nirik> I'll note it's important for FAD's to have a list of specific deliverables...
18:54:36 <nirik> not just "work on stuff"
18:54:39 <skvidal> right
18:54:42 <skvidal> I think that list is not hard
18:54:51 <skvidal> sign* fixup/redo/document/process
18:54:56 <skvidal> 2fa
18:54:59 <nirik> yeah, I have a long list of things I would like to have done/do/plan
18:55:11 <skvidal> well we could focus it down to only auth/security if it would help
18:55:21 <pingou> abadger1999: webapp FAD ?
18:55:22 <nirik> yeah, thats what I was thinking too.
18:55:34 <abadger1999> Maybe Colorado is the closest to all of infra-sysadmin :-)
18:55:51 <jaysonr> sorry i almost missed the meeting - I was thinking it was #PM EDT :)
18:55:53 <nirik> abadger1999: we could make a graph. ;)
18:55:58 <jaysonr> 3PM*
18:55:58 <abadger1999> pingou: We could.  I bet Boston area is most central for those.
18:56:07 <skvidal> anyway - to the list?
18:56:12 <pingou> abadger1999: likely
18:56:27 <relrod> webapp/fedora-apps FAD would be cool too, yes :P
18:56:31 <nirik> yeah. (I would be happy with colorado, because then I wouldn't have to fly)
18:56:40 <nirik> anyhow, anything else? or shall we call it a meeting?
18:56:44 <skvidal> nirik: you could drive to raleigh if you really wanted to
18:56:58 <nirik> yeah.
18:57:16 <nirik> possibly could do train too, but I bet I would have to go via chicago or something silly.
18:57:48 <nirik> ok, thanks for coming everyone.
18:57:58 <nirik> #endmeeting