19:59:40 <smooge> #startmeeting Infrastructure
19:59:40 <zodbot> Meeting started Thu Jan 20 19:59:40 2011 UTC.  The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:59:40 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:00:05 <smooge> #meetingname infrastructure
20:00:05 <zodbot> The meeting name has been set to 'infrastructure'
20:00:16 <smooge> #chairs skvidal CodeBlock mmcgrath
20:00:21 <CodeBlock> ok
20:00:23 <smooge> #topic here
20:00:29 * nirik is lurking around
20:00:42 * CodeBlock was a tad off, thought the meeting was at 4 EST. whoops. ;)
20:01:01 <goozbach> CodeBlock: 3 eastern
20:01:03 <goozbach> :)
20:01:39 <CodeBlock> ok, so ... who's here? :)
20:01:49 <smooge> #topic Roll Call
20:01:59 <smooge> I am here.
20:02:08 <goozbach> present
20:02:33 <CodeBlock> Alright then, looks like this is going to be a short meeting ;D
20:02:56 <CodeBlock> #topic Jan 16 iscsi outage
20:03:03 <CodeBlock> or not.
20:03:18 <goozbach> smooge needs to make you a chair methings
20:03:22 <goozbach> if that didn't happen
20:03:26 <CodeBlock> he did
20:03:43 <CodeBlock> well then
20:03:44 <CodeBlock> anyway
20:03:53 <CodeBlock> there was an outage on jan 16th at midnight EST...lasted two hours
20:03:57 <nirik> the command is '#chair'
20:04:23 <CodeBlock> iscsi/netapp crashed, RHIT was called
20:04:33 <smooge> #chair skvidal CodeBlock
20:04:33 <zodbot> Current chairs: CodeBlock skvidal smooge
20:04:41 <smooge> fricking plurrals
20:04:47 <CodeBlock> problem was fixed by 2AM EST ... affected were db02 (thus FAS and all services which depend on it), ns04, and a few other boxes
20:05:10 <smooge> #topic what CodeBlock said
20:05:11 <CodeBlock> #topic Jan 16th iscsi outage
20:05:24 <CodeBlock> hehe
20:05:53 <CodeBlock> so that was a fun morning...
20:06:00 <CodeBlock> any comments on any of that?
20:06:03 <goozbach> anything other than iscsi?
20:06:11 <goozbach> or was it the entire netapp
20:06:12 <nirik> one additional thing to note is that that takes out the package signing server too... has to be restarted after a db outage it seems.
20:06:17 <goozbach> or does that netapp do just iscsi
20:06:29 <smooge> the entire netapp was down for a bit.
20:06:38 <smooge> system is single headed so there was no failover
20:06:49 <smooge> we are moving to dual headed in the next 10-14 dyas
20:07:02 <smooge> nirik, ah I didn't realize that.
20:07:06 <CodeBlock> smooge: 10-14? Over fudcon?
20:07:11 <smooge> I think we will want to move
20:07:17 <smooge> CodeBlock, either before or after
20:07:22 <smooge> but not during
20:07:34 <smooge> no sleep for the wicked it would seem
20:07:40 <CodeBlock> haha
20:07:41 <smooge> so I have better be a lot more wicked
20:08:10 <goozbach> where do we put the information on system dependancies?
20:08:26 <goozbach> such as the one nirik mentioned?
20:09:07 <CodeBlock> We should probably put that on whatever SOP talks about the package signing server
20:09:52 <CodeBlock> If someone wants to do that, feel free
20:10:00 <CodeBlock> Otherwise, I can do that a bit later
20:10:08 <CodeBlock> Any other comments on this?
20:10:13 <goozbach> there's also something to be said to put that on the SOP for the db server
20:10:15 * nirik could, but hopefully netapp won't die.
20:10:32 <goozbach> both directions of the dependandcy should be mentioned
20:10:42 <CodeBlock> nirik: it's just a good thing to note, probably. Things happen
20:11:03 <nirik> sorry, I misspoke... it's after the iscsi outage, not db.
20:11:27 <CodeBlock> yeah
20:11:40 <smooge> currently I think we need to look at several changes.
20:11:46 <smooge> 1) Do we need to be on iscsi?
20:12:01 <smooge> 2) What services should be/remain on iscsi?
20:12:46 <smooge> 3) Longer term: how do we break up single points of failure better.
20:13:04 <smooge> Those are things I hope to tackle with others during the FUDcon talks
20:13:20 <smooge> that way we can draw out a picture, agree we are all seeing the same thing and move forward.
20:14:43 <smooge> 4) Asset management needs to be dealt with so we can find these things easily.
20:15:12 <smooge> something that writes out static pages because if the box we are looking relies on the db wiki wont work
20:15:28 <smooge> anything else from people?
20:18:33 <smooge> CodeBlock, sorry I took over didn't I :).
20:19:49 * smooge wonders if he has been netsplit and not realized it.
20:20:07 <smooge> #topic FUDCon
20:20:49 <CodeBlock> smooge: Good thing you did..sorry my boss had called me into his office to ask me to do something
20:20:54 <CodeBlock> >.> I'm back now ;D
20:21:03 <smooge> ok FUDcon NA is going to happen in 9 days. I will be in Az from Thursday to Monday playing santa claus and giving out rpms to good developers (and debs to bad ones)
20:21:21 <smooge> what plans do people want to cover over Sunday/Monday hackfests?
20:22:13 <nirik> time permitting, it would be nice to try and drive the ticket count down again.
20:22:19 <nirik> it keeps going up. ;(
20:22:38 <CodeBlock> hm
20:23:44 <smooge> ok that sounds good. I would like to have a "Is this still be worked on?" for old ones, and kill those that dont have an answer in 30 days. If there is already one in the ticket... close it.
20:23:55 <smooge> they can be reopened later.
20:24:12 <nirik> yeah, there are some old ones with no response for a while.
20:24:35 <smooge> Also I would like people to generate logins for all fedora machines from the last 60 days
20:24:52 <smooge> it will be used to winnow down our sysadmin count
20:25:17 <CodeBlock> hmm
20:27:38 <smooge> I have to head to a budget talk for Fedora FY 2011->2012 so I am going to cut this short.
20:27:44 <smooge> #topic Open Floor
20:28:19 <CodeBlock> alright...doesn't look like anyone is here to go over tickets (most of the meeting tickets are assigned to ianweller, ricky, skvidal)
20:28:39 <CodeBlock> I've not touched our nagios 2 -> nagios 3 stuff yet, so ... no update there
20:28:46 <goozbach> I'll do another ticket triage this week
20:28:53 <goozbach> like I've done in the past
20:29:09 <goozbach> and I'll make sure the meeting reminder/agenda is posted weds of next week
20:30:49 <CodeBlock> Alright, anyone have anything else?
20:31:48 <CodeBlock> Alright then.. smooge are we having a meeting next week?
20:32:57 <smooge> yes.
20:33:15 <CodeBlock> alright
20:33:18 <smooge> thanks
20:33:23 <goozbach> agenda item for next week: the ecualiptys(sp) stuff
20:33:25 <goozbach> no?
20:33:30 <goozbach> or is that going to be a seperate meeting?
20:33:31 <smooge> yes that why we have meeting
20:33:39 <goozbach> top of the list then
20:33:41 <smooge> rbergeron will run it for us :)
20:33:41 <goozbach> got it
20:33:44 <CodeBlock> Ah, ok
20:33:51 <CodeBlock> Just wasn't sure because it's so close to fudcon
20:34:07 <CodeBlock> Anyway - I'm out of here to do some work-related work. ;)
20:34:08 <CodeBlock> #endmeeting