20:00:09 <mmcgrath> #startmeeting Infrastructure
20:00:09 <zodbot> Meeting started Thu Jun 10 20:00:09 2010 UTC.  The chair is mmcgrath. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00:09 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:00:11 <mmcgrath> #topic Who's here?
20:00:15 * skvidal is here
20:00:17 <smooge> here
20:00:17 * jokajak is here
20:01:05 * a-k is
20:01:08 * rigeld2 is here
20:01:15 <dgilmore> hola
20:01:30 * sijis is here
20:01:38 <mmcgrath> lets get started
20:01:40 * nirik is lurking
20:01:43 <mmcgrath> #topic 5.5 upgrade
20:01:48 <mmcgrath> so this went pretty well
20:01:54 <mmcgrath> smooge: any outstanding issues?
20:02:36 <smooge> no outstanding issues
20:02:39 <smooge> beyond moksha
20:02:39 <dgilmore> did we migrate to postgesql 8.4?
20:03:00 <smooge> I did not myself. I think Mike may have done so on stg
20:03:16 <mmcgrath> dgilmore: we did in staging but not in production
20:03:23 <mmcgrath> lmacken: ping, any news on the updated moksha rpms?
20:03:37 <mmcgrath> it's blocking our upgrading bapp1
20:03:48 <dgilmore> mmcgrath: ok
20:04:06 <mmcgrath> dgilmore: so far no problems.
20:04:09 * mdomsch 
20:04:13 <mmcgrath> there was one issue but I haven't been able to reproduce it
20:04:25 <dgilmore> ok
20:04:34 <mmcgrath> the problem happened on publictest3 but not in staging.
20:04:42 <smooge> oh?
20:05:16 <mmcgrath> smooge: yeah, the fedora user lost writes to two sequences.
20:05:19 <mmcgrath> a grant statement fixed it
20:05:26 <mmcgrath> so even if that happens in production, it's an easy fix.
20:05:37 <mmcgrath> Ok, anyone have any questions on that?
20:05:57 <a-k> I might have one later on pub test 3 for you
20:06:01 <a-k> But now right now
20:06:03 <smooge> no the only issues I had was I bumped dgilmore off in a middle of an upgrade.
20:06:23 <a-k> But not right now (oops)
20:06:30 <smooge> and app1 had a problem after a downgrade to Django
20:06:53 <mmcgrath> k
20:06:57 <mmcgrath> #topic Outage this weekend
20:07:04 <mmcgrath> so I'm pretty sure there is going to be a PHX outage this weekend.
20:07:06 <smooge> outage?
20:07:06 <mmcgrath> I'm not sure the times yet
20:07:11 <mmcgrath> just that there's PHX2 work going on.
20:07:17 <smooge> oh I wasn't aware of that
20:07:25 <mmcgrath> smooge: I wasn't either until Oxf13 told me :)
20:07:34 <mmcgrath> so I poked around a bit.
20:07:35 <smooge> great
20:07:50 <mmcgrath> I think it's supposed to be short but I'm still waiting for the official outage notification on the internal list that work is going on.
20:07:55 <mmcgrath> so everyone be aware it's coming
20:07:57 <mmcgrath> whatever it is ;)
20:08:06 <mmcgrath> #topic db2
20:08:22 <mmcgrath> If I figure out what time it is, I'm going to set an outage that starts a bit early because db2 is almost out of disk space.
20:08:25 <mmcgrath> I'm going to give it some more
20:09:08 <smooge> ah cool. how much more do we need
20:09:10 <rigeld2> mmcgrath: growing disks can be done online right? or is it policy not to do so?
20:09:30 <mmcgrath> rigeld2: not easily, the only way I know of to grow online is by adding another disk to the server, then on the server adding it.
20:09:48 <mmcgrath> If you grow a disk image, the underlying guest doesn't realize the disk is bigger so it can't be grown.
20:09:59 * lmacken arrives a bit late, due to a long concall
20:10:09 <mmcgrath> rigeld2: if you know how to do it please let me know because I've been trying to get it to work for a long time without success.
20:10:11 <rigeld2> mmcgrath: on the guest, fdisk should see the new space, but I've never dealt with kvm/xen
20:10:17 <mmcgrath> rigeld2: it doesn't :(
20:10:27 <rigeld2> ah, ignore me then.  Sorry.
20:10:31 <mmcgrath> I thought maybe re-scanning the scsi bus and stuff.
20:10:32 <smooge> yeah it won't until it has been 'rebooted.'
20:10:33 <dgilmore> mmcgrath: ive not found a way to trigger a rescan of the disk on a guest so it call all be done online
20:10:44 <smooge> I have tried various ways and ended up with odd system
20:10:45 <mmcgrath> you'd think that would work by now :(
20:10:56 <mmcgrath> ahwell
20:11:00 <mdomsch> grow it, then reboot the guest
20:11:08 <rigeld2> (sorry for the sidetrack)
20:11:19 <mmcgrath> rigeld2: no problem at all it is good to check in on stuff like that from time to time.
20:11:24 <mmcgrath> lmacken: any word on moksha?
20:11:44 <lmacken> do we have a SOP on growing guest disks?  I've been meaning to do that for one of my own
20:11:53 <lmacken> mmcgrath: yeah, so I'll have a new moksha at least pushed to staging today
20:11:59 <lmacken> I'm still trying to track down a problem with a handfull of unit tests that fail with our RPMs but work fine in a virtualenv.
20:12:07 <lmacken> Ended up writing a tool to compar
20:12:07 <lmacken> e versions of eggs in two different python site-packages try and track things do
20:12:12 <lmacken> wn
20:12:14 <mmcgrath> lmacken: k, sounds good.  Give us a shout when bapp1 is ready
20:12:17 <lmacken> will do
20:12:24 <mmcgrath> ok, next topic
20:12:28 <mmcgrath> #topic CDN
20:12:43 <mmcgrath> so, without much pomp, Fedora's CDN is mostly deployed now.
20:12:53 <mmcgrath> you get to a proxy server based on GEO dns
20:13:18 <smooge> it seems to be going well
20:13:40 <mmcgrath> Also, I've got some pretty good etags, caching, gzipping and expiration headers on some content like on docs.fedoraproject.org, start.fp.o and just regular fedoraproject.org
20:13:47 <mmcgrath> if you have a moment, go to http://start.fedoraproject.org/
20:13:50 <mmcgrath> and see how quick it loads :)
20:14:11 <mmcgrath> same with http://fedoraproject.org/
20:14:19 <mmcgrath> still work to be done but we're getting there.
20:14:25 <mmcgrath> yslow certainly likes our new setup
20:14:27 * jokajak claps
20:14:46 <lmacken> very nice
20:14:55 <sijis> awesome
20:15:09 <mmcgrath> sijis: though I did just notice on start.fp.o we're advertising F12 instead of F13 :)
20:15:28 <mmcgrath> anywho, I've never gone though this cdn process before so anyone that has any expertise in it please do step forward and help out.
20:15:43 <sijis> mmcgrath: that was changed. weird.
20:15:47 <mmcgrath> it's been an interesting project to work on learning about etags, when they're good when they're bad, how to deal with expiration headers properly, etc.
20:15:48 <rigeld2> mmcgrath: You might have the most expertise :p
20:16:07 <a-k> My start.fp.o has f13
20:16:09 <rigeld2> mmcgrath: I'm getting f13 on start.fp.o
20:16:18 <mmcgrath> rigeld2: yeah it dawns on me most people outsource that sort of thing to akamai
20:16:19 <sijis> mine is showing f12.
20:16:26 * ianweller messed with yslow and handling a lot of that stuff before, but not nearly as extensively as this
20:16:39 <ianweller> and with a weird web host that does weird things. :/
20:16:45 <rigeld2> ProxyServer: proxy3.fedoraproject.org
20:16:48 <rigeld2> has f13
20:16:53 <mmcgrath> ianweller: yah it's pretty nice.  Some of our pages have a score of 99/100 once yslow and the site were properly configured :)
20:17:01 <mmcgrath> anywho, we can investigate that a bit more later.
20:17:04 <ianweller> mmcgrath: rockin'!
20:17:07 <mmcgrath> anyone have any questions or concerns on it?
20:17:32 <mmcgrath> the tricky thing here is it involves changes and checks everywhere in the stack.
20:17:39 <mmcgrath> it's not just like an apache thing or just a webpage thing
20:17:56 <mmcgrath> anywho, I'm going to get an SOP printed out and hold a training session like I did with puppet a year or two ago.
20:17:57 <mdomsch> does it change how we manage DNS entries?
20:18:07 <mmcgrath> It's not complicated, it's just large so it needs to be thought of that way.
20:18:14 <mmcgrath> mdomsch: it only changes how we manage geodns entries.
20:18:24 <mmcgrath> so, for example, torrent doesn't change but fedoraproject.org does.
20:18:25 <smooge> no it looks well
20:18:47 <mmcgrath> mdomsch: I'll do an SOP for that too but check out modules/bind/files/master/
20:18:56 <mmcgrath> you'll see some new directories in there that are the geo based ns files.
20:19:08 <mmcgrath> it's not a perfect setup, but it is a good step forward
20:19:13 <mdomsch> it sure feels faster
20:19:45 <mmcgrath> mdomsch: yeah my first hope was that start.fedoraproject.org loaded instantly and it pretty much does.
20:20:06 <mmcgrath> we'll have to go through application by application and sometimes page by page to make sure it's all working
20:20:18 <mmcgrath> but I'm getting pretty good at finding and fixing issues
20:20:30 <mmcgrath> it'll get harder with our turbogears stack since that's actually doing work :)
20:20:39 <smooge> we just need a couple of partners in Asia and we will have a good set
20:20:55 <mmcgrath> smooge: yeah I'm thinking one in asia, one in SA and one in africa would do us a lot of good
20:21:19 * mmcgrath just got an email
20:21:24 <mmcgrath> looks like the outage will be on June 12th.
20:21:27 <smooge> ok
20:21:56 <smooge> I will be available most parts of that day I believe
20:21:56 <mmcgrath> anywho, anyone have any other questions or comments about the CDN stuff?
20:22:14 <mmcgrath> I'm also planning on giving a talk about this at the next FUDCon for those that can make it :)
20:22:31 <mmcgrath> alrighty
20:22:46 <mmcgrath> That's really all I had for the day
20:22:59 <mmcgrath> #topic Search Engine
20:23:01 <mmcgrath> a-k: any updates?
20:23:16 <a-k> There are a few minor candidates to check out yet
20:23:25 <a-k> But I think the major ones are the four in pub test right now
20:23:38 <a-k> I can send a summary to the list when I get the others done
20:23:45 <mmcgrath> a-k: yeah I look forward to reading them
20:23:56 <mmcgrath> some of us have checked them all out but it's hard to remember which ones did what:)
20:24:17 <a-k> Xapian and nutch are the ones that had trouble with unicode
20:24:22 <mmcgrath> yeah
20:24:33 <a-k> mnoGoSearch is the one that didn't want to compile on 64-bit
20:24:45 <a-k> And DataparkSearch is just way slow
20:24:52 <ianweller> heh.
20:24:57 <a-k> I'll save the details for an email
20:25:16 <mmcgrath> sounds good
20:25:25 <mmcgrath> anyone have anything else?
20:25:28 <mmcgrath> if not
20:25:30 <mmcgrath> #topic open floor
20:25:37 <mmcgrath> anyone have anything to discuss at all?
20:26:01 <smooge> I am working on zikula at the moment
20:26:08 * mdomsch will be mostly offline from June 23 - July 18
20:26:25 <smooge> I have really come to dislike php
20:26:31 <mmcgrath> smooge: :)
20:26:31 <rigeld2> I've come to like python :)
20:26:38 <mmcgrath> mdomsch: hopefully for vacation?
20:26:43 <mmcgrath> mdomsch: oh!  that reminds me
20:26:47 <mmcgrath> I'll be in training next week.
20:26:50 <mmcgrath> and summit the week after
20:26:54 <rigeld2> and I need to apologize for being worthless for the past week and a half - work is destroying my life atm
20:27:02 <mdomsch> mmcgrath: certainly some of it is.  I'll be at Red Hat Summit from Wed night through Friday
20:27:02 <mmcgrath> I mentioned this at a meeting a couple weeks back but then didn't mention it again :)
20:27:12 <mmcgrath> mdomsch: well I'll see you there :)
20:27:12 * smooge pulls out his Alexander Haig suit and declares he will be in charge
20:27:36 <mmcgrath> rigeld2: that's ok, $work happens :)
20:27:52 <jokajak> i would like to be able to contribute more towards mod_security, but i'm not sure where to go from where i'm at
20:28:11 <mmcgrath> mdomsch: BTW you were right about which pxe image to select.
20:28:33 <mmcgrath> jokajak: we got you in sysadmin-web right?
20:28:37 <smooge> jokajak, where is it currently?
20:28:39 <jokajak> mmcgrath: correct
20:28:45 <jokajak> smooge: on my machine only
20:28:54 <mmcgrath> jokajak: k, are you going to be around tomorrow sometime?
20:29:01 <mmcgrath> I'll get you in how to deploy it in staging for a few days.
20:29:08 <mmcgrath> then we can push what you have to production
20:29:09 <jokajak> mmcgrath: yes
20:29:28 <rigeld2> oh yeah - python-mechanize rocks face.  In case you werent aware.  PSA.
20:29:39 <smooge> jokajak, well lets work on getting it onto a pt, then into staging
20:30:01 <jokajak> smooge: i'll deploy it wherever you guys instruct me
20:30:12 <jokajak> i'm not currently in sysadmin-test
20:30:15 <mmcgrath> rigeld2: :) thanks for that
20:30:35 <mmcgrath> jokajak: yeah lets meet up tomorrow and we'll get a solid plan together.
20:30:54 <mmcgrath> smooge: I ended up having him skip publictest for security reasons.
20:30:55 <jokajak> awesome
20:31:06 <mmcgrath> though we can do a restricted publictest host that'd be like staging if we want.
20:31:22 <mmcgrath> my concern was him doing tests through it and passing usernames and passwords through the pt host which is something we try to avoid
20:31:28 <mmcgrath> anywho, for Friday
20:31:32 <mmcgrath> anyone have anything else to discuss?
20:32:31 <mmcgrath> alrighty, we'll close in 30
20:32:41 <smooge> mmcgrath, oh ok
20:33:04 <smooge> mmcgrath, I was going to do a publictest to make sure I have the zikula stuff working correctly
20:33:13 <smooge> ooooh
20:33:34 <mmcgrath> sure
20:33:40 <mmcgrath> I think one existed at one point, I'm sure it could use a rebuild
20:33:46 <mmcgrath> smooge: any timeline on that yet?
20:34:22 <smooge> I am working on the module and hope to have it done tonight. The big issue is the dump from mysql and loading it up to the db.stg systems.
20:34:42 <smooge> I will need help with that
20:34:58 <mmcgrath> sure thing
20:35:03 <mmcgrath> just let me know what I can do
20:35:32 <mmcgrath> Ok, if there's nothing else we'll close in 30
20:36:39 <mmcgrath> #endmeeting