20:00:39 <smooge> #startmeeting infrastructure 20:00:39 <zodbot> Meeting started Thu Jan 6 20:00:39 2011 UTC. The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot. 20:00:39 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 20:00:46 <smooge> #meetingname infrastructure 20:00:46 <zodbot> The meeting name has been set to 'infrastructure' 20:00:47 * abadger1999 reminded that he stuck water in the microwave for tea... four hours ago 20:01:06 <smooge> I hope you didn't run it for 4 hours 20:02:26 <skvidal> smooge: he likes a few extra rem 20:02:32 <goozbach> abadger1999: we have a guy here who will do that for 24-48 hours 20:02:49 <smooge> #chair skvidal 20:02:49 <zodbot> Current chairs: skvidal smooge 20:02:50 <goozbach> "SAM! HERE'S YOUR now-cold WATER!" 20:02:54 * dgilmore is here 20:02:59 * nirik is lurking around if needed. 20:03:01 * ricky 20:03:02 <smooge> #topic Roll Call 20:03:07 <smooge> hi guys 20:03:13 <abadger1999> hey 20:03:13 <CodeBlock> hi 20:03:13 * goozbach here, not really paying attention though 20:03:44 * skvidal is here 20:04:15 <smooge> #topic End of Year Break 20:04:51 <smooge> Ok our slushy freeze for Devemver is over 20:05:22 <smooge> we had only one outage it looks like. 20:05:40 <smooge> I think it recovered over itself so that was nice 20:06:14 <smooge> I have just rememvered to remove the "Slushy Freeze" notice. 20:06:14 <CodeBlock> yeah it was a quickie 20:06:23 <smooge> Anything else come up CodeBlock ? 20:06:35 <CodeBlock> smooge: nothing worth noting really 20:06:51 <CodeBlock> Everything seems to have stuck together.... and hopefully everyone had a nice vacation 20:06:54 <smooge> thanks for you, ricky and dgilmore for covering things 20:07:18 <CodeBlock> :) 20:07:30 <smooge> #topic Upcoming Outages 20:07:38 <skvidal> all things considered - I think I would have rather been here :) 20:07:42 <skvidal> CodeBlock: but thanks 20:08:03 <smooge> oh sorry to hear that skvidal I had hoped you would ahve a nice relaxing break. 20:08:13 <smooge> Ok we have a couple of outages coming up 20:08:32 <smooge> 1) We have a rolling outage for reboots of servers to get them all running on updated kernels and glibs 20:08:58 <smooge> 2) We have a major mondo oh crap outage in PHX2 when we move to the new Netapps 20:09:21 <smooge> #1 I think we can do tonight/tomorrow after a notice is emailed out. 20:09:30 <smooge> #2 I do not have a firm date on. 20:10:16 <smooge> But basically we are getting a new netapp and all our netapp storage (iscsi, sata, etc) will have to be frozen while copied over to it I think 20:11:03 <smooge> This could mean a 24-48 hour outage by my back of my envelope estimates but I am hopefully overestimating. 20:11:32 <goozbach> is this stuff netapp snapclonable? 20:11:47 <ricky> Hm. 20:11:48 <goozbach> ie separate volumes, compatable netapps? 20:11:56 <smooge> some might be.. but some won't be as it is moving from SATA->FC 20:12:00 <ricky> mmcgrath mentioned something about moving db02 storage off of the netapp during that last outage 20:12:03 <smooge> if I understand the new info. 20:12:25 <ricky> Ooh, shiny FC. 20:13:48 <smooge> I am not sure exactly when/where this will be. I will need to let rbergeron know as it will definately effect/affect? her schedules 20:14:37 <ricky> So what will go down for this? 20:15:34 <smooge> well some of this will hopefully be snapcloned (the ISCSI shares). 20:15:55 <ricky> So that means that it'll still be read/write during the copy, and the switchover can be quick? 20:16:25 <smooge> The big problem will be the moving of nfs if they really move us from SATA->FC. As that will be a long rsync and then some sort of freeze 20:16:55 <smooge> I am asking for some estimates because I may have misunderstood some steps. 20:16:58 <ricky> OK, so for that, only releng/mirrormanager stuff will be halted? 20:17:12 <skvidal> smooge: no 20:17:13 <skvidal> you can just do 20:17:22 <skvidal> pre-rsync days in advance - if the new unit it up in advance 20:17:30 <skvidal> and then we schedule the outage 20:17:34 <skvidal> freeze everything 20:17:39 <skvidal> do a final re-rsync 20:17:39 <smooge> They may have meant that FC stuff stays FC and SATA gets moved over.. but I am not sure. 20:17:42 <skvidal> and move over 20:18:07 <smooge> skvidal, I think I meant what you said :) 20:18:10 <ricky> Probably a good idea to get a list of iscsi machines and find out how those will be affected. 20:18:14 <skvidal> nod 20:18:23 <smooge> working on it 20:18:29 <ricky> Hopefully we can move those over one by one while both old and new netapps are up 20:18:32 <smooge> its part of a ticket that skvidal asked about 20:18:44 <skvidal> nod 20:19:04 <smooge> since I don't ahve much more I think we can move along to other items. 20:19:18 <smooge> #topic Tickets 20:20:06 <smooge> ok I need to add a ticket for rebooting systems for this month but we discussed that already. 20:20:19 <smooge> .ticket 2519 20:20:20 <zodbot> smooge: #2519 (kill cvs with fire) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2519 20:20:34 <nirik> sadly, we are up to 292 tickets again. ;) 20:20:48 <skvidal> nirik: I added a bunch on monday 20:20:50 <smooge> nirik, I hope to close a couple this week/next week. 20:20:57 <nirik> yeah, such is life I fear. 20:21:25 <smooge> I think I will also look at purging a bunch next week. 20:21:35 <smooge> the equivalent of a bugzilla EOL. 20:21:54 <smooge> try to give the new guy/gal a clean slate to reopen things with 20:21:57 <smooge> ok so back to cvs 20:22:15 <smooge> we have a couple of projects still using it, and they need to move off of it. 20:22:46 <skvidal> I wanted to ask about this 20:22:50 <skvidal> how would we feel 20:22:58 <skvidal> about installing 'cvs' on people01 20:23:06 <skvidal> and letting the folks rest their project there? 20:23:16 <smooge> I was wondering about hosted instead 20:23:27 <dgilmore> skvidal: rest there project there in what sense? 20:23:27 <skvidal> I really don't want to run pserver anywhere 20:23:32 <smooge> I trust its broken hardware a lot more than people's broken hardware 20:23:36 <hicham> isn't cvs dead ? 20:23:53 * ricky also prefers just killing it if possible 20:24:02 <ricky> It'd be doing those projects a favor :-) 20:24:10 <abadger1999> fedorahosted would mean "more supported' I guess :-( 20:24:29 <skvidal> ricky: so there is just one or two projects that won't be migrated 20:24:36 <skvidal> I looked into the others and updated that ticketr 20:24:45 * nirik would be fine with: we are shutting this machine on XXXX-XX-XX. If you don't want to move to fedorahosted, we will be happy to give you a tar.gz of your cvs project. Good luck. 20:25:08 <skvidal> nirik: so - if they want to use hosted they have to change scms 20:25:09 <skvidal> OR 20:25:12 <skvidal> we have to support cvs 20:25:19 <skvidal> which I, at least, don't want to do 20:25:24 <nirik> right, and I don't think we should support cvs. 20:25:34 <skvidal> does anyone disagree with that? 20:25:37 <nirik> if they can't use svn, then they should find their own cvs server elsewhere. 20:25:40 <skvidal> does anyone here want to support cvs on hosted? 20:26:15 <gholms> [You notice a deafening silence] 20:26:16 <ricky> OK, looks like we all agree then :-) 20:26:36 <skvidal> gholms: [a grue eats you] :) 20:26:38 <smooge> or at least put in an RFR for us to build a pukecvs box 20:26:42 <nirik> really svn should be close enough so as not to matter... except it sucks less. 20:26:43 <abadger1999> skvidal: Ah without pserver fedorapeople seems okay . 20:26:57 <gholms> skvidal: D: 20:26:59 <skvidal> abadger1999: right - that's what I was saying - just let them use cvs + ssh 20:27:03 <abadger1999> <nod> 20:27:19 <skvidal> if we're cool w/it then let's float an EOL on cvs01 to the list(s) 20:27:21 <smooge> skvidal, with that in mind I think people01/02 would be ok 20:27:24 <skvidal> and stick a fork in it 20:27:40 <tibbs> I wonder if folks who can't handle moving from CVS can handle changing to cvs+ssh. 20:27:51 <ricky> Eh, not having anonymous access is kind of bad :-( 20:27:56 <smooge> I declare 2011-03-03 to be pulled pork day for CVS 20:28:00 <skvidal> ricky: not for these trees 20:28:09 <ricky> Those projects would probably be better off having a "real" CVS provider somewhere else 20:28:11 <skvidal> smooge: we have to wait that long? :) 20:28:35 <smooge> Ok... 2011-02-03 20:28:36 <skvidal> I'll see your 2011-03-03 and raise you 2011-02-14 20:28:44 <skvidal> we'll all be coming back from fudcon 20:28:50 <nirik> happy v-day cvs! :) 20:28:55 <goozbach> +1 for valentines 20:29:01 <CodeBlock> hah 20:29:03 <goozbach> tis memorable 20:29:16 <smooge> and we will see what else we can clean up on massacre day too. 20:29:21 <smooge> ok 2011-02-14 it is 20:29:37 <skvidal> .info CVS01 to be EOL'd on 2011-02-14 20:29:38 <zodbot> skvidal: (info <url|feed>) -- Returns information from the given RSS feed, namely the title, URL, description, and last update date, if available. 20:29:53 * skvidal doesn't know how to use this damn thing 20:29:56 <smooge> #agreed 2011-02-14 will be end of cvs system. Projects will move to people or other services 20:29:59 <ricky> #info CVS01 to be EOL'd on 2011-02-14 20:30:09 <smooge> #info CVS01 to be EOL'd on 2011-02-14 20:30:13 <ricky> (it's inconsistent for meetbot, yeah) 20:30:14 <smooge> ok next 20:30:23 * skvidal grumbles about newfangled irc technology 20:30:26 <skvidal> YOU KIDS GET OFF MY LAWN 20:30:30 <CodeBlock> XD 20:31:12 <smooge> .ticket 2275 20:31:13 <zodbot> smooge: #2275 (Upgrade Nagios) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2275 20:31:39 * CodeBlock just throws this out there: https://gist.github.com/754833 20:31:40 <smooge> ok moving noc01 to el6 and using newer nagios 20:32:18 <smooge> I guess that was your winter vacation CodeBlock 20:32:24 <ricky> Yay :-) 20:32:27 <CodeBlock> that link deals with (hopefully) moving nagios to a proper puppet module, which I'd like to try to do at the same time as upgrading nagios :) 20:32:30 <ricky> Also, configfile is kind of deprecated at this point :-/ 20:32:44 <CodeBlock> mmh 20:32:45 <CodeBlock> bah 20:32:49 <CodeBlock> ricky: that diff is reversed, oops 20:33:07 <ricky> Ah, OK - I was wondering why you were removing quotes :-) 20:33:09 <smooge> CodeBlock, ok so first we need to do that with noc01.stg 20:33:25 <smooge> so I would say start checking in and breaking staging 20:33:53 <CodeBlock> smooge: will do that this weekend 20:34:17 <smooge> okie dokie. sned out a notification if it will cause a pager storm somehow 20:34:55 <CodeBlock> smooge: last I head stg can't send mails out 20:35:02 <CodeBlock> so.. in theory that shouldn't be possible 20:35:08 <CodeBlock> least I heard* 20:35:27 <CodeBlock> ricky: bah, now I want to reverse that diff, but can't because the files are at home. :( 20:36:23 <ricky> There's apways patch -R to apply reversed diffs :-) 20:36:25 <smooge> s/^+/X/; s/^-/+/; s/^X/-/; [or something like that] 20:36:41 <smooge> thanks for the progress on that. 20:36:48 <CodeBlock> no problem :) 20:37:07 <CodeBlock> smooge: and somehow that only took one night to do, btw :P 20:38:07 <smooge> abadger1999, ricky ping 20:38:13 <abadger1999> smooge: here 20:38:16 <smooge> .ticket 2481 20:38:17 <zodbot> smooge: #2481 (Fedora switching from the CLA to FPCA) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2481 20:38:42 * ricky has not looked at this at all, sorry :-( 20:38:57 <ricky> The changes in FAS should be pretty simple though. 20:39:01 * abadger1999 hasn't looked since he outlined the steps to make it happen. 20:40:15 <smooge> ok I think we will look at this again after FudCON unless a sprint is needed there. 20:40:25 <smooge> .ticket 2542 20:40:26 <zodbot> smooge: #2542 (reinstall fas01 on rhel6 kvm host) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2542 20:40:54 <skvidal> fas01 is on rhel6 20:40:59 <skvidal> runnong on a xen server 20:41:10 <smooge> Ok the steps here require us really to make one more kvm host. This means moving some stuff off of a current Xen box and rebuilding it to el6 20:42:00 <ricky> Does the kickstart SOP apply exactly the same for KVM? 20:42:08 <skvidal> ricky: almost, yes 20:42:11 <skvidal> ricky: I made a couple of edits 20:42:13 <ricky> Cool 20:42:15 <skvidal> to the kickstart sop 20:42:19 <skvidal> for el6 20:42:26 <skvidal> but the virt-install are the same ultimately 20:42:29 <smooge> I think Xen15 is our 'best' candidate 20:42:31 <skvidal> there is one thing I've not fixed yet 20:42:36 <skvidal> smooge: quick thought 20:42:41 <skvidal> smooge: lemme see if this seems wrong 20:43:00 <skvidal> xb-01 20:43:25 <skvidal> is on bxen01 20:43:33 <skvidal> and dgilmore said xb-01 is doing nothing at all 20:43:33 <dgilmore> skvidal: we would need to get eth0 on bxen01 moved to the public network 20:43:39 <skvidal> damn 20:43:41 <skvidal> okay 20:43:43 <skvidal> dgilmore: thank you 20:43:48 <skvidal> smooge: carry on 20:43:53 <skvidal> sorry for the useless input 20:43:55 <dgilmore> skvidal: they can probably do that in the switch 20:44:06 <skvidal> ah - it's not a physical move? 20:44:16 <dgilmore> move the vlan the port is on 20:44:20 <skvidal> smooge: i guess I was thinking - use bxen01 as our bubble sort location 20:44:40 <skvidal> smooge: move hosts from xen15 over there - reinstall xen15 to rhel6 + kvm 20:44:51 <skvidal> and then migrate hosts back by reinstalling them 20:45:26 <smooge> ah ok 20:45:27 <skvidal> ricky: the kvm installs do a couple of wonky-ish things - you'll want to have vnc running to connect to the installing guest. 20:45:40 <smooge> we will need to get RHIT to move bxen01 over to the 126 network 20:45:40 <dgilmore> skvidal: wonky in what way? 20:45:41 <skvidal> ricky: and then I need to figure out how to tell kvm + rhel6 to open the console port properly 20:45:52 <skvidal> dgilmore: just those 2 20:45:54 <dgilmore> smooge: thats just a ticket 20:46:02 <ricky> Ah yeah, I remember running into the virsh console issue 20:46:14 <skvidal> ricky: I've found a way to fix the virsh console thing 20:46:17 <dgilmore> skvidal: virsh start guest --console 20:46:30 <skvidal> dgilmore: doesn't work for a running one 20:46:31 <ricky> Ah, nice. 20:46:41 <smooge> dgilmore, I have to bribe mgalgoci every time.. I am down 1 kidney, half a liver and a pancreas. I need to get an intern this summer instead 20:46:45 <skvidal> dgilmore: if init is not listening on it 20:46:53 <skvidal> dgilmore: s/init/getty/ 20:46:57 <smooge> dgilmore, but yeah its just a ticket 20:46:58 <dgilmore> skvidal: if we pass console=ttyS0 to the install it should just work 20:47:09 <skvidal> dgilmore: when we install, we can't do that, I believe 20:47:10 <dgilmore> skvidal: which i think is different to rhel5 and xen 20:47:20 <dgilmore> skvidal: we can 20:47:22 <skvidal> dgilmore: b/c txt consoles are no longer there for the instaqller 20:47:25 <dgilmore> we can try it anyway 20:47:30 <skvidal> dgilmore: it NEEDs to use vnc 20:47:46 <dgilmore> skvidal: text install is there its just not interactive at all 20:47:47 <smooge> db02 might be pretty easy.. its just a xenGuest on iscis. It can be moved to a LOT of boxes pretty quickly 20:47:47 <skvidal> dgilmore: definitely try it - but I tried a bunch of things when I was doing fas## and friends 20:48:07 <skvidal> dgilmore: we occasionally need the interactivity 20:48:11 <skvidal> for disk partitioning 20:48:18 <dgilmore> skvidal: yeah that has to be done via vnc 20:48:20 <skvidal> therefore - we need vnc 20:48:27 <skvidal> rather than have 2 different instructions 20:48:33 <skvidal> It makes sense to do it all via vnc 20:48:37 <smooge> especially because kickstart+RAID in EL6 is funky 20:48:46 <dgilmore> smooge: no its not 20:48:50 <skvidal> I was just trying to make the vnc ALSO open the console for virsh console 20:49:07 <smooge> dgilmore, we have had problems with every box when making more than one RAID array. 20:49:26 <smooge> it makes the first one, and then sometimes makes the second one, and sometimes makes up a new one. 20:49:40 <smooge> skvidal can go over that one :) 20:49:41 <skvidal> dgilmore: and when you migrate from an older raid array - it has had some issues - especially when spares are involved 20:49:56 <skvidal> dgilmore: I filed a bug on it - and it's being worked on 20:50:07 <skvidal> dgilmore: apparently we're the only people using rhel with spares :) 20:50:11 * skvidal is kidding 20:51:03 <skvidal> dgilmore: it's not horribly broken - but I had to nuke the drives on the hw to get anaconda to let me install the box 20:51:20 <skvidal> but we're WAY off in the weeds 20:51:37 <dgilmore> smooge: ive had no issues with raid arrays in el6. all the builders have arrays, as does quite a few other boxes ive build using kickstart 20:51:38 <skvidal> ricky: if you want to do rhel6 installs on kvm - yell at me if you run into anything 'odd' and I'll make sure I update any docs 20:51:44 <ricky> Sure thing, thanks 20:51:45 <dgilmore> skvidal: :( ok 20:52:40 <skvidal> so - fas01 migration 20:53:05 <skvidal> smooge: just out of curiosity do we have a xen host holding 2 less critical pieces of infrastructure? 20:53:29 <skvidal> alternatively 20:53:32 <smooge> dgilmore, we can test on bvirthost01 its been rebuilt 5 times with different results each time. If I have a problem with the kickstart I want to fix it. 20:53:35 <skvidal> dgilmore: do you need bxen01 at all? 20:53:37 <smooge> but back to the problem at hand 20:53:59 <dgilmore> skvidal: nope 20:54:02 <skvidal> okay 20:54:05 <skvidal> is it under warranty? 20:54:10 <dgilmore> skvidal: it was there just for xb-01 20:54:19 <dgilmore> skvidal: probably not 20:54:19 <smooge> skvidal, db02 can move and bastion01 is a backup box 20:54:35 <skvidal> dgilmore: ah :( sad face 20:54:36 <skvidal> okay 20:54:47 <dgilmore> skvidal: it could be still 20:54:53 <dgilmore> mmcgrath: would know 20:54:56 <smooge> is bxen01 a dell? 20:55:12 <skvidal> smooge: yes 20:55:50 <smooge> it is not under warranty anymore 20:55:55 <skvidal> womp womp 20:55:57 <skvidal> then never mind 20:56:03 <smooge> and was to be replaced with bvirthost01 20:56:08 <skvidal> okay - 20:56:12 <skvidal> then just ignore me 20:56:22 <skvidal> bastion01 goes off - and we move db02 20:56:23 <skvidal> reinstall xen15 20:56:34 <smooge> and use it for bublble sort 20:56:55 <skvidal> and put fas01 on xen15 20:57:09 <skvidal> then bastion01 and db02? 20:57:36 <smooge> I would go with bastion01 going back 20:57:44 <smooge> actually that would break the buble sort 20:57:53 <smooge> bastion01 can go on virthost01 20:58:21 <smooge> if I can get iscsi exported to virthost02 we can put db02 there 20:59:12 <smooge> then we work on say xen11 or xen12 20:59:35 * rbergeron pokes in just ahead of the cloud sig mtg and waves hi to her favorite infrastructure folks 20:59:49 <skvidal> we've got a lot of tickets left 20:59:58 <skvidal> can we continue - or do we need to move to -2? 21:00:09 <dgilmore> lets move 21:00:25 <ricky> Or #fedora-admin where people already are :-) 21:00:40 <skvidal> ricky: too much random noise 21:00:41 <skvidal> imo 21:00:42 * rbergeron feels bad about disrupting every week 21:00:51 <skvidal> rbergeron: doesn't seem to STOP you 21:01:04 <rbergeron> no, no, it surely doesn't. :) 21:01:07 <smooge> ok #fedora-meeting-2? 21:01:22 <smooge> rbergeron, did you see the poke earlier about possible effects on your schedule? 21:01:27 <abadger1999> smooge: Sure. Don't forget to stop meeting here. 21:01:30 <smooge> #endmeeting