20:00:35 <smooge> #startmeeting fedora infrastructure 2010-11-18
20:00:35 <zodbot> Meeting started Thu Nov 18 20:00:35 2010 UTC.  The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00:35 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:00:43 <smooge> #meetingname infrastructure
20:00:43 <zodbot> The meeting name has been set to 'infrastructure'
20:00:49 <smooge> #chair mmcgrath
20:00:49 <zodbot> Current chairs: mmcgrath smooge
20:01:16 <skvidal> yo
20:01:19 <skvidal> hi
20:01:20 * skvidal is here
20:01:22 * mdomsch is in 2 other meetings; name me if you need me
20:01:23 * nirik is lurking around.
20:01:31 * ninjazjb here
20:01:32 <smooge> ok mdomsch
20:01:45 <goozbach> but there wasn't a roll call :)
20:01:51 <goozbach> red 5 checking in
20:01:58 <skvidal> red 6 checking in
20:02:01 * daMaestro lurks
20:02:03 * waltJ is till awake- yawn
20:02:08 <skvidal> You okay back there porkins?
20:02:14 * mmcgrath here
20:02:21 <skvidal> mmcgrath: you're not porkins!
20:02:25 <skvidal> mmcgrath: you're slim
20:02:29 <skvidal> or maybe 'chops
20:02:31 <smooge> #rollcall
20:02:32 <skvidal> but not porkins
20:02:36 * mmcgrath feels like a porkins.  had a big lunch
20:02:46 <skvidal> mmcgrath: did it include sausage and/or brats?
20:02:49 <smooge> #topic rollcall
20:02:58 <smooge> been done to death
20:03:00 <goozbach> I had porkchops for lunch so maybe I'm porkins
20:03:02 * lmacken 
20:03:09 * skvidal is here
20:03:10 * abadger1999 here
20:03:12 * daMaestro lurks
20:03:14 * goozbach here
20:03:19 * ninjazjb here (again :) )
20:03:19 <skvidal> smooge: sausage has been done to death?
20:03:29 <skvidal> smooge: I think, in fact, that is the point
20:03:39 * Elwell is here (but a fedora-meeting virgin, so be gentle...)
20:03:49 <smooge> ok first time for everything
20:03:56 <smooge> #topic agenda
20:04:18 <smooge> I didn't get a chance to change add an agenda item to trac so we are once again not having one
20:04:23 <smooge> my apologies..
20:04:36 <smooge> Please list items you would like covered
20:05:17 <abadger1999> outages -- probably good to update people on where we're at there
20:05:28 <smooge> new servers
20:05:38 <smooge> removing 11 and upcoming EOL of 12
20:05:40 <smooge> elections
20:05:56 <smooge> and uhm..
20:06:13 <smooge> #topic outages
20:06:27 <smooge> skvidal, you had the best language use on this.
20:06:33 <skvidal> umm
20:06:39 <skvidal> I don't believe I'll be saying that in public
20:06:45 <skvidal> but lemme see if I can make it family-friendly
20:07:13 <skvidal> 'WHAT THE <BEEEEEEEP> <bEEEEP> is <BEEP> going on <BEEEP> here?'
20:07:22 <skvidal> something like that
20:07:33 <skvidal> it's like inverted madlibs
20:07:40 <gholms> Heh
20:07:46 <abadger1999> :-)
20:07:50 <skvidal> so - we've been having a whole host of problems
20:08:07 <skvidal> and not all of them  are related, it seems
20:08:33 <skvidal> we were losing fas boxes due to a problem with the kernel running under xen that we moved fas onto
20:08:41 <skvidal> so we were commonly losing fas boxes
20:08:43 <skvidal> quite commonly
20:08:53 <skvidal> we've reinstalled one of the xen boxes to rhel6 + kvm
20:08:54 <skvidal> and
20:08:58 <skvidal> <knock on wood>
20:09:02 <skvidal> it appears to be doing better
20:09:10 <skvidal> (is that an okay assessment? everyone?)
20:09:15 <abadger1999> yep
20:09:21 <waltJ> aye
20:09:27 <skvidal> so smooge has setup a couple of new boxes
20:09:29 <skvidal> running rhel6 + kvm
20:09:47 <skvidal> and I think we'll be having a good old time doing a bubble sort to get all the xen boxes converted to kvm
20:09:51 <skvidal> and all the guests reinstalled
20:09:57 * CodeBlock here kind of
20:10:08 <skvidal> smooge: am I still in the realm of reality
20:10:31 <skvidal> so - the other problem is
20:10:38 <smooge> skvidal, yes
20:10:41 <skvidal> that we've been seeing some odd timeouts hitting one of the netapps
20:10:46 <skvidal> and rhit is working on helping us
20:10:53 <skvidal> and we're trying to figure out why it is slowing down
20:10:59 <smooge> I was going for a shell sort, but buble sort will work
20:11:09 <skvidal> as it APPEARS to be impacting puppet (and other things) runtime
20:11:46 <skvidal> finally, I've at least been making a list of all the things I've run into that don't quite work how I expected them to work wrt to reinstalling and config-mgmt-ing our systems
20:12:05 <skvidal> b/c  of my relative lack of experience with nuances of the process and b/c of the transition to kvm from xen
20:12:21 <skvidal> a lot of our documented processes don't QUITE match what has happened - especially on rhel6
20:12:31 <smooge> there are a couple of other issues that have happened due to slow networks between remote sites. at various times the ping drops went up to 60% and then went away
20:12:39 <skvidal> that's not the fault of anyone - it's just normal 'hey look, that file isn't in /sbin anymore - where the hell is it?' sort of thing
20:12:56 <smooge> and the "oh wait I thought we had that in the config files"
20:13:04 <smooge> "oh yeah box A"
20:13:40 <skvidal> smooge: right
20:13:47 <skvidal> so I've got this list of oddities I'm going to work on
20:14:12 <skvidal> one of those is the hotfixes thing
20:14:17 <skvidal> which I'll be stabbing in the face shortly, I hope
20:14:50 <smooge> just remember to clean up afterwords. I had a devil of a time with the blood stains last time
20:15:07 <skvidal> I always throw down a tarp and put on my tyvek coveralls
20:15:33 <smooge> its the wall ceiling splatters .. people forget about those.
20:15:45 <skvidal> nod
20:15:48 <smooge> anyway anything else on this?
20:15:52 <smooge> questions?
20:15:53 <skvidal> not at the moment
20:16:23 <phuzion> Did I miss everything?
20:16:42 <smooge> only about outages.
20:16:47 <phuzion> ok
20:16:49 <smooge> #topic new hardware
20:17:11 <smooge> ok we are in the process of removing old hardware with new supported stuff
20:17:28 <smooge> this happens constantly as old items go out of support and new things come in.
20:17:31 <mmcgrath> smooge: is any of that in yet?  I know skvidal has been looking for hosts
20:17:47 <skvidal> mmcgrath: yep
20:17:50 <smooge> Mike had set up a good process of this with RHIT.. I am just following through
20:18:07 <skvidal> virthost02 and build_virthost01 is making headway if I'm reading puppet logs right
20:18:17 <smooge> so this week I have been clawing out various eyeballs to bet ^^^ installed
20:18:33 <smooge> virthost02 should be pretty much ready after the next reboot.
20:18:52 <smooge> virthost_build01 is almost done.
20:19:05 <smooge> the issues encountered were several:
20:19:15 <smooge> 1) tftp decided not to work for a bit. and then did
20:19:23 <smooge> like "maaaaagic"
20:20:01 <smooge> 2) tftpboot had old files on it and some configs
20:20:36 <smooge> 3) dhcp mac addresses and dsylexia are fun
20:21:00 <smooge> but we persevered and the boxes are set up
20:21:28 <smooge> I just have to finish the last of the puppet configs and I believe they will be ready for someone to countercheck
20:21:54 <smooge> once that is done.. it will be time to move another app server and fas server over to the new virthost box
20:22:22 <smooge> other hardware that is being worked on is the new download servers.
20:22:45 <smooge> the ones in PHX2 are racked and have an initial build on them
20:23:04 <smooge> the ones in RDU arrived and are in crates in some closet until room opens up
20:23:41 <smooge> I hope to have PHX2 boxes ready to be 'tested' next week.. if we get some network issues dealt with
20:24:00 <smooge> EOF
20:24:07 <smooge> questions?
20:25:50 <skvidal> yah
20:25:52 <skvidal> next week
20:26:01 <skvidal> I'm going to ask for a small favor
20:26:12 <skvidal> can we put in a 'major holiday in the us don't mess up my PTO' freeze?
20:26:22 <skvidal> I'm not actually kidding
20:26:34 <skvidal> most of the folks who will get paged/emailed will be trying to have time with their families
20:27:54 <mmcgrath> skvidal: I'm not against that.
20:27:57 <skvidal> smooge: ?
20:28:02 <skvidal> abadger1999: ?
20:28:05 <skvidal> lmacken: ?
20:28:13 <skvidal> I assume y'all all have families and what not
20:28:19 <smooge> ok
20:28:31 <goozbach> second'ed
20:28:43 <abadger1999> +1
20:28:49 <waltJ> i'm with you too
20:28:50 <skvidal> so I think what I'm thinking is we stabilize things this week
20:28:51 <smooge> we are on a slush freeze starting Monday 2000 UTC
20:29:03 <gholms> Heh, slush freeze...
20:29:08 <skvidal> try to get the fas box nightmare at least stable-ish
20:29:10 * nirik notes he will be around FWIW.
20:29:18 <skvidal> and then just try to keep things running
20:29:19 <smooge> anything after that needs to go through 2 +1's on list or IRC
20:29:27 <skvidal> smooge: sounds round to me
20:29:36 <goozbach> that's pretty reasonable
20:29:54 <smooge> #topic removing 11
20:30:23 <smooge> ok I am going to start removing 11 from the main netapps. It has been in archives for a bit and the download flags pointing to archives
20:30:36 <smooge> it just needs the final nail
20:30:58 <skvidal> yay!
20:30:59 <skvidal> thank you
20:31:14 <smooge> 12 will be EOL at the end of December I believe
20:31:44 <smooge> I will repeat this before fudcon NA so we are much cleaner on disks
20:31:56 <nirik> @f12eol
20:32:17 <smooge> well I guess I can start earlier then
20:33:12 <smooge> I don't expect any problems but there may be some emails about "where the hell did my ppc go?" or some such
20:33:20 <smooge> when the main mirrors whack it
20:33:48 <smooge> #topic elections
20:33:58 <smooge> ok elections are happening ...
20:34:00 <smooge> I think
20:34:02 <smooge> maybe
20:34:21 <smooge> abadger1999, has been helping on getting various db items dealt with
20:34:25 <abadger1999> yep.
20:34:31 <abadger1999> Board and feso are setup.
20:34:45 <abadger1999> famsco had a problem that I found after jsmith got on an airplane.
20:34:46 <phuzion> Elections start in 2 days, FYI
20:34:58 <abadger1999> I'm working out how to put the correct information into there now.
20:35:05 <smooge> ok cool
20:35:16 <abadger1999> If all elase fails, I can set it up directly in the database.
20:37:03 <abadger1999> Any questions?  That's all I've got.
20:37:23 <smooge> nothing from me at the moment
20:37:30 <skvidal> sounds round
20:37:45 <smooge> ready for next to last topic
20:37:54 <smooge> #topic New People
20:38:18 <smooge> we have some new people helping out.. some may have already introduced but I figured I should say so
20:38:35 <smooge> phuzion, waltJ, ninjazjb and maybe goozbach ?
20:39:02 <smooge> just say hi and wave to the cameras :)?
20:39:44 <goozbach> hello *waves
20:39:45 <goozbach> *
20:39:46 * waltJ waving
20:39:53 <goozbach> I'm new-ish
20:39:59 <waltJ> So i'm around since a few weeks. Working on Nagios issues most time.
20:40:02 <ninjazjb> hello everyone
20:40:26 <goozbach> this is week three attending this meeting, I've yet to find time in my schedule to actually *work* on fedora stuffs
20:40:42 <abadger1999> greetings!
20:41:47 <gholms> [time passes]
20:42:13 <smooge> that is the general case goozbach .. when I had my previous job, my time was pretty short and mostly limited to whatever problem was happening at the time I could look at IRC
20:42:43 <Elwell> in that case, I'll wave from the back and say I'm hoping to get involved a bit too.
20:42:44 <nirik> welcome everyone.
20:43:17 <waltJ> thinking, my general case is my time zone ;)
20:43:23 <smooge> #topic Open Floor
20:43:58 <smooge> ok I am running 17 minutes ahead for once so someone has to have an open floor itme to take up 20 minutes
20:44:27 <ninjazjb> smooge: anything about the bacula project?
20:44:34 <skvidal> ninjazjb: not this week, I suspect :)
20:44:41 * phuzion waves
20:44:49 <goozbach> smooge: what are the sources you would use to generate an agenda?
20:44:58 <ninjazjb> ok
20:45:12 <smooge> in the future it will be a line anything with the keyword meeting in trac
20:45:31 <skvidal> open floor still?
20:45:34 <phuzion> sorry about being in and out so much, I'm currently at work, and we have visitors in from Spain, so I'm helping out a lot.
20:45:35 <abadger1999> yep
20:45:37 <goozbach> maybe a contribution I could do is generate a first-draft agenda to send your way, or ping the mailing list to make sure people make agenda items
20:45:47 <smooge> I just need to add a report in the trac to do it and go from there.
20:45:52 <goozbach> I see
20:45:55 <goozbach> that's easier
20:45:56 <daMaestro> I'd like to hear a little bit more about the setup with Xen that we had issues with.
20:46:01 <smooge> goozbach, that would be great for some parts.
20:46:07 <smooge> skvidal, yes it is still open floor
20:46:15 <daMaestro> Was it EL5 Xen Dom0 with EL5 Xen DomUs that were having issues?
20:46:33 <goozbach> smooge: I'll start pinging the mailing list and IRC every weds to make sure folks add their agenda items then
20:46:43 <skvidal> daMaestro: no
20:46:57 <skvidal> el5 xen dom0 on with rhel6 domu
20:47:10 <skvidal> and afaik - also f13+ domU
20:47:11 <daMaestro> Ah, k.
20:47:15 <ninjazjb> skvidal: is there an eta on when you guys want to start working on it?
20:47:21 <daMaestro> Did any of the Fedora domUs have issues?
20:47:33 * nirik notes there are a lot of tickets out there, we might want to go do a triage/purge/update sometime.
20:47:35 <smooge> skvidal, actually I think it would be f10--> for this little bug.. but not sure
20:47:36 <skvidal> smooge: ^^
20:47:50 <skvidal> daMaestro: so - yes - fedora domUs - of certain kernels had issues
20:48:13 <smooge> ninjazjb, my eta on bacula would be late December/early January due to other deliverables.
20:48:25 <phuzion> I figure now is a decent time to introduce myself.  If you didn't see my email on the list, my name is Chris Egeland, and I'm interested in joining the team.  If you guys wanted to know anything about me, feel free to ask.  My primary goals for joining the team are to learn new things, and improve my existing skills.  I'm on FAS, CLA signed, ready to go.  I just need to be assigned a task, and get the appropriate access, and get cracking.
20:48:37 <smooge> ninjazjb, however that doesn't mean we can't start getting questions that ened to be answered etc together
20:48:51 <skvidal> phuzion: saying you're going to crack once you get access frightens me :)
20:49:03 <skvidal> phuzion: :-D
20:49:07 <phuzion> skvidal: :)
20:49:07 * skvidal is just being silly
20:49:10 <smooge> nirik, yes. the idea will be to take 3 tickets a week and deal with them
20:49:12 <phuzion> Obviously.
20:49:18 <goozbach> skvidal: he has bad joints, give him a break :)
20:49:32 <phuzion> goozbach: you know about my knee problems???
20:49:45 <phuzion> :)
20:49:56 <goozbach> I could hear you walk in :)
20:49:57 * smooge doesn't want to talk too much about joint issues.. pain killers are almost done
20:49:58 <nirik> smooge: fair enough.
20:50:04 <abadger1999> phuzion: Are you interested in sysadmin or development stuff?
20:50:09 <phuzion> sysadmin.
20:50:42 <abadger1999> k
20:50:53 <abadger1999> smooge is your man then :-)
20:51:03 <ninjazjb> smooge: still need help with nagios?
20:51:15 * abadger1999 helps onboard people who are looking to do development.
20:51:34 <phuzion> I have to go, I'll try to jump on IRC in the car if I can (as a passenger of course)
20:52:12 <phuzion> If you guys have any questions, please feel free to either ping me in -admin, -noc, or private message me.
20:52:15 <casep> hey there, Hi abadger1999, I'm kinda rusted in development but I think I can help
20:52:27 <smooge> ninjazjb, yes. work with waltJ phuzion and CodeBlock as they are the guys who will need it :)
20:52:34 <Elwell> abadger1999: 'general I want to become a fedora contributor' development, or infra specific devel?
20:52:36 <ninjazjb> ok
20:52:53 <skvidal> Elwell: the latter, I think
20:53:10 <smooge> ninjazjb, once my new advil kicks in I will take it up in #fedora-noc (about an hour?)
20:53:23 <phuzion> smooge: What will I need to do to get access to publictest7 to work on nagios3?
20:53:38 <abadger1999> Elwell: mostly infra specific -- but I can point you in the right direction with general contribution.
20:53:47 <smooge> I have to sponsor you in sysadmin and in sysadmin-test
20:53:50 <abadger1999> casep: Excellent.
20:53:55 <abadger1999> casep: Do you know python?
20:53:56 <smooge> once that is done you can get on pt7
20:54:13 <phuzion> Alright, sounds good.
20:54:24 <smooge> that goes for ninjazjb too
20:54:37 <smooge> I am going to have to close now.. I can barely type
20:54:39 <ninjazjb> smooge: I'll be gone by then.  I'll get ahold of you tomorrow on IRC.  Also, wasn't sure if you got to my email on the ciphers
20:54:55 <smooge> ninjazjb, did and have a draft reply thanks
20:55:05 <casep> abadger1999: some, not completely "fluent" ;)
20:55:09 <phuzion> smooge: Let me know if you need any more information from me. :)
20:55:11 <smooge> ninjazjb, tomororw looks like a better day
20:55:13 <abadger1999> :-)
20:55:21 <ninjazjb> smooge: will do
20:55:29 <smooge> ok closing in 2 minutes.
20:55:37 <skvidal> no meeting next week, right?
20:55:44 <smooge> no meeting next week
20:55:52 <abadger1999> casep: We can talk more in #fedora-admin if you have time.
20:55:58 <smooge> goozbach, no meeting next week
20:56:00 <casep> ok, mobing
20:56:18 <smooge> #endmeeting