18:59:51 <nirik> #startmeeting Infrastructure (2011-08-18)
18:59:51 <zodbot> Meeting started Thu Aug 18 18:59:51 2011 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:59:51 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:59:52 <nirik> #meetingname infrastructure
18:59:52 <zodbot> The meeting name has been set to 'infrastructure'
18:59:52 <nirik> #topic Robot Roll Call
18:59:52 <nirik> #chair smooge skvidal codeblock ricky nirik abadger1999
18:59:52 <zodbot> Current chairs: abadger1999 codeblock nirik ricky skvidal smooge
18:59:58 * skvidal is here
19:00:00 * Flare183 is here
19:00:10 * athmane is here
19:00:16 <CodeBlock> hi, here, etc
19:01:32 <nirik> ok, I guess lets dive on in.
19:01:43 <nirik> #topic New folks introductions and apprentice tasks/feedback
19:01:58 <nirik> any new folks? or apprentice tasks we should discuss?
19:02:39 * nirik listens to crickets.
19:02:49 <skvidal> the ones here at night
19:02:51 <nirik> as always, feel free to chime in in #fedora-admin/noc.
19:02:52 <skvidal> they've been really loud
19:02:54 <smooge> How many items have been added to the talk repo
19:02:57 <rfelsburg> Not new persay, just back after being gone.
19:03:01 <smooge> the infra-repo
19:03:13 <nirik> skvidal: same here of late.
19:03:29 <nirik> smooge: infra-docs? or infra-hosts?
19:03:33 <smooge> geez I can't type.. infra-docs
19:03:36 <nirik> rfelsburg: welcome back. ;)
19:03:40 <smooge> welcome back
19:03:46 <nirik> smooge: some... lots more need to be done tho
19:03:49 <rfelsburg> thanks nirik/smooge
19:04:26 <nirik> I'd be happy to assist anyone wanting to work on that...
19:04:44 <smooge> me too. email works great if I am not on IRC
19:04:51 <skvidal> smooge: some - not all of them - I've been doing them as I read them
19:05:15 * nirik means to do some too, but hasn't.
19:05:34 <nirik> ok, moving along...
19:05:37 <nirik> #topic F16 Alpha Freeze reminder and tickets.
19:05:58 <skvidal> smooge: mainly b/c some of them I have to fix as I migrate
19:06:00 <nirik> I think sijis is working on getting the alpha website squared away.
19:06:06 <skvidal> smooge: b/c some of our SoPs are badly out of date
19:06:07 <skvidal> (sorry)
19:06:12 <smooge> np
19:06:17 * pingou hi
19:06:21 <nirik> yeah, many need updating. ;)
19:06:43 <nirik> we should have alpha syncing out soon, so we can check perms and space and all that fun stuff.
19:07:10 <nirik> anyone have any alpha related issues or concerns?
19:07:25 <smooge> no this one is pretty quiet on our side
19:07:31 <smooge> knock on silicon wood
19:08:04 <nirik> yeah, hopefully it will all go smoothly.
19:08:09 <nirik> ok, moving on then...
19:08:21 <nirik> oh, and hi pingou. ;)
19:08:24 <nirik> #topic Fedora hosted plans
19:08:26 <Flare183> I'll try to deal with the simple ones as needed
19:09:27 <nirik> we talked about this on the list some. I think we have at least a short term plan hashed out... make new collab/hosted pairs much like they are now, move lists to new collab, and migrate stuff to hosted with a eye for expansion down the road if need be.
19:09:32 <skvidal> Flare183: thank you
19:09:35 <abadger1999> smooge: I looked at allura's install doc... I think it would make hosted more heavy-weight than it currently is.  Not sure if that's necessarily bad.
19:09:51 <nirik> Flare183: please do ask if anything looks unclear or out of date. ;)
19:10:08 <skvidal> Flare183: or migrate it and just mark it as OUT OF DATE at the top
19:10:21 * nirik has a dislike of sourceforge... possibly based on the old code, but it used to be really difficult to use.
19:10:21 <smooge> abadger1999, yeah.. this looks a lot more enterprise class than trac
19:10:29 <abadger1999> smooge: It might be hard to package... the download is a 25MB pybundle... whatever that is.
19:10:35 <skvidal> abadger1999: I thought the same thing when I looked at it - that allura looked like it had a lot of infrastructure we may or may not ever use ubt willl require a lot to keep it running
19:10:48 <skvidal> so
19:10:48 <abadger1999> <nod>
19:10:55 <skvidal> I actually have a ridiculous dream
19:11:01 <skvidal> if we could do redmine in openshift
19:11:06 <smooge> a git repo for every .pot?
19:11:12 <skvidal> and point it back at git repos on hosted
19:11:31 <skvidal> this is partly b/c I'm lazy and partly b/c I don't think it is a terrible thing to help openshift ;)
19:11:38 <abadger1999> <nod>
19:11:50 <nirik> yeah, worth exploring...
19:11:50 <skvidal> and b/c they are not bound by the 'need an rpm' rules
19:11:54 <skvidal> b/c of how they are laid out
19:12:06 <skvidal> anyway - I like the plan we've been discussing for hosted
19:12:12 <smooge> ok just as long we say this is an experiment and if openshift goes away for some reason we can roll with it
19:12:18 <nirik> does everyone agree on the short term plan? and we continue to look at longer term?
19:12:28 <skvidal> nirik: oh and even if we decide that merging hosted lists and collab lists is not a good plan - load-wise
19:12:39 <skvidal> nirik: I still think a single instance that just does lists for hosted saves us a bunch of pain
19:12:46 <skvidal> nirik: would you agree?
19:12:47 <nirik> yeah.
19:12:52 <nirik> agreed.
19:13:10 * nirik notes the archiver is still working on the 16th on hosted. Perhaps it will catch up someday.
19:13:21 <skvidal> nirik: right - do you know if it is cpu or io-bound??
19:13:35 <abadger1999> nirik: What's the short term plan precisely?  spinning new hosted boxes for backend and using dns to make it appear like one from the front?
19:13:52 <nirik> skvidal: cpu it looks like.
19:13:58 <abadger1999> is the splitting just for repos or for repos and trac?
19:14:05 <skvidal> nirik: can we throttle back other processes adn nice that one
19:14:28 <skvidal> abadger1999: i think it's for repos and trac?
19:14:40 <skvidal> note _think_
19:15:22 <nirik> abadger1999: make new collab03/04, and hosted03/04 that are rhel6 and similar to what we have now. Migrate projects over, but add in cname or the like for trac/scm. It's still 1 machine, but we can easily move things around to 2 or more machines if need be down the road...
19:15:53 <nirik> and all lists migrate to the new collab03
19:16:23 <smooge> sounds good
19:16:26 <smooge> what else is on collab?
19:16:29 <abadger1999> okay.  So actually.  the initial change is very small.
19:16:30 <nirik> some of our load issues are mailman, some of them are our old trac/gitplugin.
19:16:34 <smooge> the older collab01/02
19:16:39 <nirik> so, new version might be much better.
19:16:39 <rfelsburg> I didn't think the archiver normally chewed that much on CPU nirik, but perhaps I'm just out of the loops, is this something new?
19:16:43 <abadger1999> the only thing splitting off is mailing lists.
19:16:49 <nirik> abadger1999: yep.
19:16:54 <nirik> and moving to rhel6.
19:16:57 <nirik> new trac, etc.
19:16:58 <abadger1999> The other portion is just a one-by-one migration to rhel6/new trac
19:17:11 <nirik> rfelsburg: well, we have this list... autoqa-results.
19:17:12 <abadger1999> Sounds good to me.
19:17:15 <nirik> it gets a LOT of emails.
19:17:17 <skvidal> nirik: question
19:17:28 <skvidal> do we want to go ahead and generate the cnames now?
19:17:31 <rfelsburg> Say no more nirik
19:17:37 <skvidal> point them all at hosted01
19:17:47 <skvidal> and then move them as we migrate hosts off?
19:17:51 <skvidal> s/hosts/projects/
19:17:52 <skvidal> or
19:17:56 <skvidal> just add them as we migrate?
19:18:06 <nirik> skvidal: we could, but it might be better to add as migrated, so we have a good way to tell whats moved?
19:18:21 <nirik> rfelsburg: the 16th (which it's working on) had so far 11,000 emails.
19:18:25 <abadger1999> from a support pov, seems better to have the cnames now.
19:18:40 <abadger1999> so that supporting end users, we don't have to know what's moved.
19:18:42 <nirik> I guess it could be easy to tell from other ways.
19:18:44 <skvidal> abadger1999: b/c then we could tell everyone the same thing
19:18:50 <nirik> actually, we have the cnames now.
19:18:50 <abadger1999> exactly.
19:18:50 * skvidal is willing to do either :)
19:18:55 <skvidal> nirik: we do?
19:19:00 <rfelsburg> nirik: Geez, exceptionally bad day or has the email count really gone up that much.
19:19:07 <skvidal> nirik: I thought we just had a wildcard
19:19:15 <skvidal> rfelsburg: the latter
19:19:21 <rfelsburg> damn.
19:19:34 <nirik> skvidal: right, I was joking about the wildcard. They all resolve already. ;)
19:19:47 <nirik> due to wildcard
19:20:05 <skvidal> nirik: right - so we don't have the cnames now
19:20:07 <nirik> I guess I don't care strongly about making them now or later.
19:21:11 <nirik> I expect us to have a small pool of interested projects migrate soon...
19:21:22 <nirik> then move over time, then we set a flag day to move the laggers.
19:22:04 <rfelsburg> gotta love flag day
19:22:17 <nirik> Yep. ;)
19:23:15 <nirik> anyhow, anything more on the short term hosted plans? or long term?
19:23:39 <nirik> I'll note that I am also waiting to hear from serverbeach about a hardware refresh... would be nice to setup these new ones on new machines.
19:23:53 <skvidal> nirik: question
19:24:03 <skvidal> nirik: do we want to keep all of hosted in a single site colo-wise?
19:24:03 <nirik> or we could put them somewhere else.
19:24:15 <nirik> well, we could stick the secondary one somewhere else...
19:24:21 <nirik> it's a fair bit of BW probibly.
19:24:27 <nirik> but otherwise I don't see an issue...
19:24:28 <skvidal> ie: should we have some of them in ibiblio?
19:24:32 <skvidal> and some in osuosl, maybe?
19:24:50 <nirik> so, split them out now?
19:25:22 <skvidal> I was thinking maybe so?
19:25:34 <skvidal> do we gain anything by having them be on one site?
19:25:50 <nirik> well, ability to have the warm spare I suppose.
19:26:04 <nirik> or we could still do that on other sites, just more machines.
19:26:25 <nirik> I think osu is pretty low on disk
19:27:09 <skvidal> true
19:27:11 <nirik> I don't know how important the warm spare is.
19:27:19 <skvidal> one advantage of having lots of slices
19:27:23 <skvidal> for git repos/trac
19:27:29 <nirik> it means less downtime and less data lost in case of doom.
19:27:37 <skvidal> and less resource required to run a slice
19:27:52 <skvidal> once we have the base image
19:27:56 <skvidal> that we build from
19:28:12 <skvidal> I should be able to whip up something in func to select the slice with the least instances but most disk space
19:28:14 <skvidal> to allocate to
19:28:15 <abadger1999> we've also used the warm spare as a "staging" for hosted.
19:28:28 <abadger1999> which does kinda defeat using it as a warm spare, though :-(
19:28:28 <skvidal> abadger1999: to be fair we've done that kinda badly
19:28:47 <skvidal> abadger1999: the boxes are frequently not the same as one another
19:28:51 <abadger1999> <nod>
19:28:54 <skvidal> and if we want to test something out in the slice model
19:28:56 <nirik> well, we could still do warm spares/pairs... in more places. Just means more machines and such.
19:29:00 <skvidal> we can just have a slice
19:29:04 <rfelsburg> so turn it around, what are the benefits of moving it to osuosl or ibiblio
19:29:12 <skvidal> rfelsburg: I'm not suggesting moving it there
19:29:19 <skvidal> I'm suggesting allowing a slice to be there
19:29:23 <nirik> rfelsburg: diversity... if one site is down others are still up.
19:29:28 <nirik> less eggs in one basket. ;)
19:29:30 <skvidal> in other words do we need all of it in one colo
19:29:31 <abadger1999> we're juggling too many tasks with too few boxes... slices sound nice in that model.
19:29:37 <skvidal> abadger1999: +1
19:29:37 <rfelsburg> okay, that makes more sense. are we having any problems with reliability at this point?
19:29:48 <nirik> rfelsburg: nope, just paranoia. ;)
19:29:54 <skvidal> rfelsburg: all of our colos have had outages at one point or another
19:30:09 <skvidal> having it so all of any single resource is bound up behind a spof is, imo, a bad thing
19:30:13 <nirik> we do backup hosted to phx2, but thats daily...
19:30:25 <skvidal> we are trying to minimize the spofs
19:30:55 <rfelsburg> gotcha, sorry still trying to catch up to where we're at
19:31:27 <skvidal> rfelsburg: no sweat
19:31:29 <nirik> I guess the question is: do this now short term, or wait until we perhaps have a more cloudy setup later to better handle this...
19:31:44 <skvidal> nirik: with or without cloud instances
19:31:51 <skvidal> we need to setup the base slice
19:31:55 <nirik> right.
19:31:58 <skvidal> and figure out if this shit is gonna work at all :)
19:32:07 <nirik> so, how about this is 1.5?
19:32:18 <nirik> 1.0 -> rhel6 / base instance setup working.
19:32:33 <nirik> 1.5 -> duplicate on another site, and get working overall with the other one.
19:32:38 <nirik> 2.0 -> rainbows
19:33:02 <skvidal> I'm okay w/that
19:33:10 <skvidal> provided you add unicorns to 2.0
19:33:34 * skvidal is a holdout for unicorns
19:33:38 <nirik> once we have a hosted03 ready, it should be not too bad to duplicate it to a hosted05 at another site and migrate some things to it.
19:33:46 <nirik> aren't they messy tho?
19:33:48 <nirik> :)
19:33:51 <skvidal> nirik: they shit rainbows
19:33:53 <skvidal> so... no
19:34:04 <nirik> ha.
19:34:12 * skvidal is full of class today
19:34:13 <skvidal> okay
19:34:13 <nirik> ok, anything else on hosted?
19:34:23 <skvidal> do you care where hosted03 happens?
19:34:33 <nirik> I guess not...
19:34:33 <skvidal> or do we have a space for it at serverbeach?
19:34:44 <nirik> well, I wanted to wait and see if they were giving us new hardware.
19:34:55 <nirik> then we could setup the new stuff on the new hardware.
19:35:08 <skvidal> fair enough
19:35:13 * skvidal has no problem with waiting :0
19:35:14 <skvidal> :)
19:35:35 <nirik> I don't want to wait too long tho. RHEL6/trac0.12 has been waited for a long time.
19:35:57 <skvidal> nirik: agreed
19:36:29 <nirik> ok, moving along then...
19:36:31 <nirik> #topic Upcoming Tasks/Items (nirik)
19:36:41 <nirik> Freeze ends after the release next week.
19:36:52 <nirik> I have a few minor commits queued up.
19:37:12 <nirik> I'd like to get a rhel6 app server setup
19:37:24 <nirik> switch some backups to new backup box
19:37:48 <nirik> possibly look at re-installing tummy01 or one of the smaller remote hosts.
19:38:12 <nirik> I think we could look at migrating ibiblio01 stuff to new rhel6 versions on ibiblio02
19:38:34 <nirik> any folks have other items waiting for after the freeze?
19:38:48 <nirik> beta freeze starts 09-13
19:40:17 <nirik> we have some updates piled up... none of them seem reboot requiring.
19:40:45 <skvidal> always nice to hear
19:41:04 <nirik> although I am sure there will be a os minor release sometime to mess us up. ;)
19:41:48 <nirik> Of course I'd like to get the new boxes installed and usable hopefully soon too. ;)
19:42:00 <nirik> anyhow, if no one has any exciting plans, will move on in a min
19:42:41 <abadger1999> I've got the ambassadors raffle app about ready -- just suddenly realized that I was using CSS code from fedoracommunity and it's agpl so I have to fix that somehow before putting it someplace public for inode0 to evaluate.
19:42:52 <nirik> cool.
19:42:55 <skvidal> abadger1999: :(
19:42:58 * abadger1999 sent email to mizmo to see if she can relicense
19:43:09 <abadger1999> otherwise, I guess I throw out the css and reimplement clean.
19:43:26 <nirik> :(
19:43:36 <abadger1999> nirik: how's askbot coming?
19:43:45 <nirik> oh yeah, I was going to discuss that some.
19:43:52 <nirik> #topic RFR's
19:44:03 <nirik> so, askbot is looking to move to staging now...
19:44:15 <nirik> I posted to the list a idea/plan for setting it up.
19:44:31 <nirik> basically: add to proxy01.stg and make a new ask01.stg instance for backend.
19:44:49 <nirik> I am not sure if it makes sense to have it's db on the main db host or seperate on it's own instance.
19:45:07 <nirik> ask does need rhel6.
19:45:34 <nirik> so, feedback on that welcome.
19:46:10 <nirik> I think fpaste is still working on their dev instance to get everything aligned and ready...
19:47:17 <nirik> Hopefully people will like askbot and use it. :)
19:47:30 <nirik> #topic Meeting tagged tickets:
19:47:30 <nirik> https://fedorahosted.org/fedora-infrastructure/query?status=new&status=assigned&status=reopened&group=milestone&keywords=~Meeting&order=priority
19:47:37 <nirik> any other tickets folks would like to discuss?
19:48:35 <smooge> not me.. I am still giggling about unicorn farts
19:48:42 <skvidal> smooge: unicorn poop
19:48:44 <skvidal> get it right
19:48:52 <nirik> just as a side note, I'd like to get us under 200 tickets by the end of the year. ;) We will see...
19:48:55 <skvidal> unicorn farts are just methane like anyone else :)
19:48:57 <smooge> no I was thinking that if they poop rainbows what do they fart?
19:49:09 <nirik> skvidal: thats boring. ;
19:49:10 <skvidal> smooge: it's colorful methane, I admit
19:49:25 <skvidal> nirik: rainbow colored methane is boring?
19:49:27 <skvidal> hrmph
19:49:29 <nirik> #topic Open Floor
19:49:38 <nirik> Anyone have anything for open floor?
19:49:41 <smooge> I figure they farted lasers
19:49:48 <smooge> methane powered lasers
19:49:52 <nirik> perhaps ponys?
19:50:02 <skvidal> nirik: they fart ponies?
19:50:07 <nirik> yeah. ;)
19:50:08 <skvidal> dear god that would hurt
19:50:13 <nirik> I'd say so.
19:50:21 <skvidal> open floor
19:50:28 <skvidal> I emailed the epylog people
19:50:35 <skvidal> and they said "what? oh that? yah, okay"
19:50:52 <skvidal> so I'll probably rip it away from svn with all of its weirdnesses
19:50:58 <skvidal> and put it into git with all of its weirdnesses
19:51:03 <nirik> excellent.
19:51:04 <skvidal> but weirdnesses I have to deal w/more often
19:51:10 <nirik> would be good to upstream things.
19:51:11 <skvidal> and merge all the modules and patches I have
19:51:39 <nirik> any further thoughts on the auth issue?
19:51:47 <skvidal> that I hates it?
19:51:47 * nirik didn't come up with any brilliant solution.
19:52:44 <skvidal> if nothing else
19:52:48 <skvidal> I will make an htpasswd file
19:52:51 <skvidal> in the dir
19:52:58 <skvidal> and chown it to sysadmin-logs
19:53:04 <skvidal> and you can add your damned self
19:53:12 <nirik> sure. simple. I like it.
19:53:25 <skvidal> and to be fair
19:53:28 <skvidal> if you can't add yourself
19:53:33 <skvidal> then you don't need to look at the logs ;
19:53:34 <skvidal> )
19:54:09 <nirik> sounds fine. might add it to the log sop too then.
19:54:16 <skvidal> indeed
19:54:18 <rfelsburg> sounded a little jaded there skvidal lol
19:54:24 <skvidal> rfelsburg: a little?
19:54:27 <skvidal> hmm
19:54:31 <rfelsburg> tin y bit :)
19:54:32 <skvidal> I must be softening in my old age
19:54:36 <skvidal> I meant to sound a lot jaded
19:54:36 <skvidal> :)
19:54:44 <rfelsburg> lol
19:54:53 <skvidal> just in case it isn't clear
19:54:56 <skvidal> </sarcasm>
19:54:58 <skvidal> I'm just messing about
19:55:01 * nirik will close out the meeting in a minute or two if nothing else comes up.
19:56:13 <rfelsburg> If I didn't understand sarcasm I'd have no place in IT
19:56:47 <nirik> ok, thanks for coming everyone!
19:56:52 <nirik> #endmeeting