20:00:27 <mmcgrath> #startmeeting Infrastructure
20:00:27 <zodbot> Meeting started Thu Jun  3 20:00:27 2010 UTC.  The chair is mmcgrath. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00:27 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:00:31 <mmcgrath> #topic Who's here?
20:00:37 * jsmith lurks
20:00:38 <jforbes> doh
20:00:57 * nirik is lurking around in the back.
20:01:41 * sgallagh listens with one ear
20:01:48 * sijis is here
20:01:55 <mmcgrath> Doesn't look like there are any meeting tickets to discuss so we'll just get started.
20:02:03 <mmcgrath> #topic Upgrade to RHEL5.5
20:02:08 <mmcgrath> smooge: how's that going?
20:02:22 <smooge> woops sorry
20:02:26 <smooge> ranting on eng-list
20:02:43 <smooge> ok RHEL-5.5 was updated on stging systems and publictest systems.
20:03:03 <mmcgrath> smooge: how'd our applications hold up?
20:03:16 <smooge> app07 and 2 live systems also got updated
20:03:41 <smooge> there are some problems we found today with mod_python and mod_wsgi I guess
20:03:47 <smooge> need to clean and fix those up
20:03:56 * mdomsch 
20:04:06 <mmcgrath> smooge: yeah that's a minor change but something to watch out for.
20:04:06 <smooge> the app systems seemed to work but I didn't go through a formal checklist to confirm
20:04:19 <mmcgrath> jokajak actually changed that package so mod_wsgi is commented out by default.
20:04:34 <mmcgrath> so we just need to make sure to deploy a /etc/httpd/conf.d/wsgi.conf that loads the module
20:04:40 <abadger1999> smooge: If you don't know where to start on that, I might be able to help ( I fixed the mod_wsgi/mod_python problem for RHEL-5.4 for infra)
20:04:55 <mmcgrath> smooge: are you planning on scheduling downtime sometime soon for rebooting and whatnot?
20:05:09 <smooge> abadger1999, I will need to talk with you in a bit.
20:05:14 <abadger1999> <nod>
20:05:18 * mmcgrath has some downtime to do soon.
20:05:25 <smooge> mmcgrath, next tuesday would be our scheduled patch day
20:05:34 <smooge> for production systems.
20:05:52 <mmcgrath> smooge: sounds good, send a note to the list to have everyone double check their apps in staging by then.
20:05:59 <smooge> ok will do so
20:06:03 <smooge> which list
20:06:03 <mmcgrath> I think I'm going to schedule my downtime for tomorrow (will explain in a bit)
20:06:08 <mmcgrath> fedora-infrastructure
20:06:15 <smooge> ok webgroup?
20:06:48 <mmcgrath> sure
20:07:28 <mmcgrath> smooge: anything else on that?
20:07:40 <smooge> not at the moment
20:07:43 <mmcgrath> k
20:07:50 <mmcgrath> #topic bastion1 is back baby!
20:07:56 <smooge> the big isuse was dealing with the various --exclude for various projects
20:08:01 <smooge> oh sorry
20:08:04 <smooge> bastion1
20:08:08 <mmcgrath> So I've had a request from the network team to alter bastion's external IP address.
20:08:16 <mmcgrath> so I'm working on getting that in order.
20:08:23 <mmcgrath> smooge: what --excludes have you been needing to use?
20:08:52 <smooge> mediawiki mostly
20:09:10 <mmcgrath> ah, k.
20:09:16 <mmcgrath> So yeah, bastion still runs our vpn and mail services.
20:09:21 <mmcgrath> bastion1's been dead for some time.
20:09:30 <sijis> shouldn't ibe bastion01 :)
20:09:32 <mmcgrath> I'm going to re-create it (already have) then get production traffic moved to it.
20:09:39 <mmcgrath> sijis: ugh
20:10:00 <mmcgrath> the whole 0 vs non 0 thing has been a disaster, it's been half a year and we're still not all on 0whatever.
20:10:07 <mmcgrath> anywho .. :)
20:10:34 <mmcgrath> I'm going to schedule downtime for tomorrow, I'm hoping for a quick blip but since the vpn is involved who knows.  I've done small failovers with success for testing.
20:10:53 <mmcgrath> I *really* want a full featured solution but it's remarkably complicated to design in our specific infrastructure.
20:10:58 <mmcgrath> it's a split brain problem.
20:12:08 <mmcgrath> that's all I have on that, any questions or comments?
20:12:55 <mmcgrath> alrighty
20:12:58 <mmcgrath> #topic CDN
20:13:00 <mmcgrath> nb: you around?
20:13:59 <mmcgrath> k, well I'll take this.  At some point soon we'll be sending people in Europe to Europe servers, people in the states will go to servers in the states.
20:14:10 <mmcgrath> this should provide a better browsing experience for people.
20:14:29 <mmcgrath> I'd also like to spend more time analyzing our caching setup.
20:14:33 <mmcgrath> we've had some odd things happening
20:14:40 <sijis> are we using some sort of geo-dns?
20:14:48 <mmcgrath> sijis: yeah
20:15:37 <mmcgrath> sijis: bind + some weird configs.
20:15:47 <mmcgrath> I'm a little worried about performance but it's a low risk thing because we can always revert.
20:16:00 <mmcgrath> anyone have any questions or comments on that?
20:16:08 <mmcgrath> its pretty quiet today.
20:16:13 <smooge> uhm
20:16:15 * Schmidt says hi then...
20:16:44 <smooge> is this just for mirrors or all content?
20:16:52 <mmcgrath> all fedoraproject.org websites.
20:16:57 <mmcgrath> the mirrors already have that.
20:17:01 <mdomsch> nothing for mirrors changes
20:17:04 <mmcgrath> the mirrorlist server doesn't though :)
20:17:07 <mmcgrath> but will
20:17:07 <smooge> and how much savings do people get when stuff still has to make a long haul back to PHX2 for the db's?
20:17:34 <mmcgrath> for that stuff not much, for all css, image, js and static / cached content (like the wiki) it could be significant.
20:17:55 <mmcgrath> also, with most of our applications the data coming in and out of the database is small, but the formatting of it is much larger.
20:17:58 <mmcgrath> which has network savings.
20:18:11 <mmcgrath> but we'll need to re-do some of our haproxy setup.  I'm not totally sure how on that yet without getting messy.
20:18:23 <mmcgrath> I'm thinking about having a /etc/hosts entry for localapp1.fedoraproject.org
20:18:32 <sijis> smooge: is your concenr the wiki data?
20:18:35 <mmcgrath> so if the proxy server is in the same location as an app server, it gets there.
20:19:18 <smooge> sijis, wiki data, upcoming zikula apps, etc
20:19:33 <mmcgrath> oh actually zikula would also have significant savings.
20:19:36 <sijis> is smolts related too?
20:19:38 <mmcgrath> it'd also be cached at the proxy layer.
20:19:46 <mmcgrath> some smolts pages would see savings
20:19:55 <smooge> ah ok
20:19:58 <mmcgrath> not submission, though there's new code coming down for that.
20:20:43 <mmcgrath> Anyone have anything else?
20:21:00 * mmcgrath will get with nb to see hwo far he is wrt getting the dnssec signed geo zones in place.
20:21:12 <mmcgrath> AFAIK they're there, just need a quick alteration and a named.conf change.
20:21:31 <mmcgrath> alrighty.
20:21:39 <mmcgrath> well I think that's all I had for this meeting so
20:21:43 <mmcgrath> #topic Open Floor
20:21:48 <mmcgrath> did anyone have anything else they'd like to discuss?
20:22:00 <mmcgrath> I had one minor note, starting not next week but the week after I'm not going to be around much.
20:22:09 <mmcgrath> Training the first week, summit the next week.
20:22:49 <sijis> mmcgrath: what's the weird caching thing you mentioned?
20:22:50 <smooge> not at the moment
20:24:06 <mmcgrath> sijis: well, when trying to build out the new internetx site, we ran into some oddities.
20:24:14 <mmcgrath> basically trying to download some files we got some errors
20:24:20 <mmcgrath> but bypassing our normal proxy layer caused it to work fine.
20:24:40 <mmcgrath> it was just some generally odd things.
20:24:50 <sijis> gotcha.
20:25:19 <sijis> also, how important is it renaming stuff to include the 0? i could help with that.
20:25:39 <sijis> i understand is updating the apps, servers too, not just the configs
20:25:44 <sijis> puppet ones
20:26:10 <mmcgrath> sijis: eh, that's the thing.  Renaming was always kind of asthetic and it's a pain in the ass to actually do.
20:26:38 <sijis> yeah, we run into that here... naming scheme changes.
20:26:45 <mmcgrath> yeah
20:26:47 <sijis> it is always easiert o to do it on a new build
20:26:55 <mmcgrath> Anyone have anything else there?
20:27:05 <mmcgrath> if not we'll close the meeting in 30.
20:27:55 <mmcgrath> #endmeeting