20:00:34 <mmcgrath> #startmeeting Infrastructure
20:00:34 <zodbot> Meeting started Thu Apr  1 20:00:34 2010 UTC.  The chair is mmcgrath. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00:35 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:00:37 <mmcgrath> #topic Who's here?
20:00:50 * nirik is lurking around.
20:01:24 * sijis is sorta here
20:01:56 <mmcgrath> Ok
20:02:01 <mmcgrath> #topic Fedora 13 Beta
20:02:03 <smooge> here
20:02:37 <smooge> is waiting for Lenova to get a support tech
20:02:46 <mmcgrath> #link https://fedorahosted.org/fedora-infrastructure/report/9
20:02:54 <mmcgrath> So as you saw from the list, we slipped by a week.
20:02:57 * ricky 
20:03:00 <mmcgrath> so the change freeze also slips by one week.
20:03:02 <mmcgrath> .ticket 2058
20:03:03 <zodbot> mmcgrath: #2058 (Verify Mirror Space) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2058
20:03:09 <mmcgrath> I closed that just yesterday, we're in good shape on the mirrors
20:03:11 <mmcgrath> .ticket 2059
20:03:13 <zodbot> mmcgrath: #2059 (Release Day ticket) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2059
20:03:15 <mmcgrath> Just a tracking ticket
20:03:18 <mmcgrath> .tiny 2060
20:03:19 <zodbot> mmcgrath: Error: '2060' is not a valid url.
20:03:23 <mmcgrath> .ticket 2060
20:03:25 <zodbot> mmcgrath: #2060 (Verify releng permissions) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2060
20:03:30 <mmcgrath> >:-|
20:03:35 <mmcgrath> smooge has that one
20:03:38 <mmcgrath> .ticket 2061
20:03:43 <zodbot> mmcgrath: #2061 (MM redirects) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2061
20:03:50 <mmcgrath> No mdomsch again, I wonder if the time change has made this slot bad for him.
20:03:55 <mmcgrath> smooge: do you want to take that one for now?
20:04:01 <mmcgrath> I think you may have done that for the Alpha
20:04:24 <ricky> I thought the redirects were automatic now or something
20:04:45 <mmcgrath> ricky: I think they are but mdomsch seemed to want to verify that so I'm not sure if it's done and we don't have to worry about it, or if it's an ongoing thing
20:04:53 <mmcgrath> we should check to see if that can be removed from our list altogether
20:05:11 <ricky> OK
20:05:24 <ricky> Here's the comment from mdomsch about it: https://fedorahosted.org/fedora-infrastructure/ticket/1993
20:06:05 <mmcgrath> so in theory if the bug is fixed, we don't have to worry about it?
20:06:13 * mmcgrath will make note to talk to sr. domsch next time he's on.
20:06:17 <mmcgrath> ok
20:06:19 <smooge> ok
20:06:22 <mmcgrath> .ticket 2062
20:06:24 <zodbot> mmcgrath: #2062 (Infrastructure Change Freeze) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2062
20:06:28 <mmcgrath> This one's in effect already
20:06:30 <smooge> sorry I lost net connection
20:06:31 <mmcgrath> .ticket 2063
20:06:32 <zodbot> mmcgrath: #2063 (Lessons Learned) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2063
20:06:33 <mmcgrath> smooge: now orries
20:06:39 <mmcgrath> And lessons learned isn't until after the release.
20:06:43 <mmcgrath> So that's the release stuff
20:06:51 * mmcgrath notes it's only 6 minutes since th emeeting started
20:06:58 <mmcgrath> that could be a new record for release tickets.
20:07:04 <mmcgrath> anyone have any questions or concerns about the release?
20:07:52 <mmcgrath> ricky: do you think websites is in good shape?
20:07:54 <mmcgrath> sijis: ^^
20:08:08 <sijis> yeah. just repo stuff.
20:08:15 <ricky> Yup, just need to get the banners ready this weekend
20:08:19 <sijis> tryign to finish up get-* pages..
20:08:30 <sijis> which can be seen in https://stg.fp.o
20:08:53 <mmcgrath> sijis: how'd that go for you?
20:08:56 <ricky> sijis: Mind sending a link to websites-list?  Looking good
20:08:57 <mmcgrath> the staging environment I mean.
20:09:14 <sijis> ricky: we need to chat about banner that too. i think tyhey are trying something different.
20:09:24 <sijis> mmcgrath: yeah, it was easy to setup. thanks
20:09:32 <sijis> ricky: will send to list
20:09:47 <smooge> mmcgrath, you need me to take 2061 correct?
20:10:03 <mmcgrath> smooge: yeah, we're pretty sure it's done already, just needs a verification
20:10:17 <mmcgrath> Ok, well with that we'll move on.
20:10:21 <mmcgrath> #topic Search Engines
20:10:29 <mmcgrath> Doah, no a-k
20:10:29 <smooge> okie dokie
20:10:35 <mmcgrath> I thought I saw him earlier.  We'll skip this.
20:11:08 <mmcgrath> #topic Monitoring
20:11:25 <mmcgrath> I've been poking around and changing some basic behaviors in nagios.
20:11:38 <mmcgrath> Stuff like flagging a host as dead if nrpe is down, and not going off of pings.
20:11:52 <mmcgrath> I'm presently (as in just before the meeting started) trying to get some decent servergroup views in order.
20:11:59 <mmcgrath> just lots of general config management and things
20:12:26 <mmcgrath> Anyone have any questions or comments on that?
20:12:35 <mmcgrath> I'm disappointed that we don't have a combined trend and monitoring solution
20:12:41 <mmcgrath> but I have to say, collectd has been awesome.
20:12:51 <mmcgrath> we've learned several things about the environment just by using it
20:12:52 <sijis> so collectd = trend?
20:12:53 <smooge> I like collectd too
20:13:00 <mmcgrath> sijis: yeah, have you seen it?
20:13:14 <mmcgrath> it's been interesting.
20:13:14 <smooge> I can see why so many comapnies wrap nagios + something like it into an overall product
20:13:17 <sijis> yeah, right after you noticed some db issues on smolts, i think
20:13:33 <mmcgrath> https://admin.fedoraproject.org/collectd/bin/index.cgi?hostname=memcached01&plugin=memcached&timespan=86400&action=show_selection&ok_button=OK
20:13:37 <mmcgrath> even stuff like that has been helpful
20:13:48 <mmcgrath> we've had a pretty solid 98% hit ratio in memcached.
20:14:39 <mmcgrath> and, surprisingly to me, all of our memcached stats fit in les then 90M of space.
20:14:47 <mmcgrath> I figured it would be much higher then that
20:14:52 <mmcgrath> but anyway, yeah.  Monitoring.
20:14:59 <mmcgrath> any questions or comments or suggestions about our monitoring systems?
20:15:05 <mmcgrath> I really want us to get smarter alerts.
20:15:17 <mmcgrath> right now when the fit hits the shan, everything kind of goes nuts and that's just not helpful.
20:15:31 <mmcgrath> I also want to start taking more metrics like MTTF and MTTR for our apps.
20:15:46 <mmcgrath> which is kind of tricky to do at the moment because of all of our entry points.
20:16:19 <mmcgrath> Ok, I'll assume nothing there?
20:16:31 <mmcgrath> #topic Blade Center
20:16:37 <mmcgrath> So I'm working to get a new blade center.
20:16:57 <smooge> whee
20:16:59 <mmcgrath> Due to some internal accounting bits, it's taking longer than expected.  I'm still hopeful I'll be flying out to Phoenix in mid april (just a few weeks from now) to install it.
20:17:03 <mmcgrath> jwb: you around?
20:17:10 <smooge> do you need me out there?
20:17:20 <mmcgrath> smooge: naw, and I don't think we have cash to send you anyway :)
20:17:33 <smooge> I could hitch-hike?
20:17:35 <mmcgrath> I *hope* it will be pretty straight forward, I have that and a few other servers to install.
20:17:46 <mmcgrath> smooge: maybe you could just will yourself there :)
20:18:30 <mmcgrath> Ok, with that I'll open the floor
20:18:34 <mmcgrath> #topic Open Floor
20:18:35 <smooge> no my teleportation skills have been pretty bad lately
20:18:43 <mmcgrath> anyone have anything they'd like to discuss?
20:19:01 <smooge> I missed anything on a week delay
20:19:04 <skvidal> mmcgrath: if the new blade center comes in - will we have some extra machines?
20:19:45 <mmcgrath> skvidal: until December we might.  We'll be migrating off the old BC to the new one.
20:19:50 <mmcgrath> then we'll be removing the old one.
20:19:56 <skvidal> darn
20:19:59 <mmcgrath> but yeah, capacity will be up, whats up?
20:20:15 <skvidal> just thinking about some of the koprs stuff
20:20:22 <skvidal> and where we're going to find the capacity for that
20:20:26 <skvidal> esp disk space and builders
20:20:37 <jwb> mmcgrath, hm?
20:20:49 <mmcgrath> skvidal: ah, well.  I suspect it'all take a bit to get it all together but I can certainly put it in next years budget for Q1.
20:20:58 <mmcgrath> and we can do some trails now.  We might be able to find space for all that.
20:21:07 <mmcgrath> the new builders are significantly beefier then the old.
20:21:20 <mmcgrath> jwb: hey, so.  PPC stuff.  Just wanted to make you aware of my intentions and see what we can do.
20:21:23 <smooge> mmcgrath, how much would it be to keep the old bladecenter under warranty even if slow/old
20:21:30 <mmcgrath> So the new blade center has two PPC blades in it for EPEL building.
20:21:38 <smooge> yeah!
20:21:50 <mmcgrath> that means all our old PPC blades will be EOLed at the end of the year.  Do you happen to have any use for them outside of the colo?
20:22:08 <jwb> mmcgrath, why EOLed?
20:22:10 <mmcgrath> smooge: it'd be a few thousand if they allow it.  I actually extended it to be supported for this year, it's on its 4th year.
20:22:14 <mmcgrath> jwb: no longer supported.
20:22:26 <jwb> mmcgrath, the HW or ppc as a release?
20:22:32 <mmcgrath> the hardware
20:22:48 <jwb> mmcgrath, so what are you planning on doing for F12 ppc builds?
20:22:51 <mmcgrath> it'd probably run just fine, but since its not under warranty (and because the power drops are pretty pricy) we won't be keeping two blade centers in there at the moment.
20:22:58 <jwb> because F12 will still be active at the end of the year
20:23:01 <mmcgrath> jwb: we'll still have some non-blade builders.
20:23:05 <jwb> oh, ok
20:23:13 <mmcgrath> ppc4 still has I think 2 years left, certainly 1 year.
20:23:48 <jwb> mmcgrath, i can't really host the blades anywhere myself.  i can ask around but needing a BC sort of makes it hard to make use of them
20:24:12 <mmcgrath> jwb: yeah that's what my fear was as well.  Doesn't hurt to ask though.  If you happen to find someone let us know.
20:24:34 <jwb> will do
20:25:20 <mmcgrath> Ok, and with that does anyone have anything else to discuss?
20:25:40 <smooge> not me
20:25:46 <mmcgrath> allllrighty!
20:25:51 <mmcgrath> #endmeeting