20:01:04 <CodeBlock> #startmeeting infrastructure
20:01:04 <zodbot> Meeting started Thu Mar  3 20:01:04 2011 UTC.  The chair is CodeBlock. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:01:04 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:01:11 <CodeBlock> #chair smooge skvidal
20:01:11 <zodbot> Current chairs: CodeBlock skvidal smooge
20:01:15 <CodeBlock> shall we?
20:01:21 <goozbach> yay meeting start
20:01:22 <CodeBlock> #topic roll call
20:01:27 * averi is around
20:01:29 * skvidal is here
20:01:32 * CodeBlock blinks a few times :)
20:01:57 <CodeBlock> alrighty
20:02:04 <CodeBlock> #topic Transifex migration to tx.n
20:02:13 <CodeBlock> so...someone talk. :)
20:02:14 <smooge> here
20:02:49 <goozbach> nirik: your topic :)
20:02:50 <skvidal> goozbach: what's on the agenda (or even the secret agenda)
20:02:50 <smooge> ok we are further along. a notice has gone out to the lists from jsmith-busy and we will be working out a plan for the last itmes
20:02:52 * sijis is here
20:03:14 <goozbach> skvidal: the first topic is the topic now
20:03:18 <glezos> Translation already started normally, we have 50 teams already on Tx.
20:03:20 <skvidal> oh I see
20:03:21 <goozbach> then infra meeting tickets
20:03:56 <nirik> I was going to suggest that I go and close out the tx related tickets we have...
20:04:01 <nirik> and point people to tx.net.
20:04:01 <goozbach> http://lists.fedoraproject.org/pipermail/infrastructure/2011-March/009974.html
20:04:05 <nirik> is that ok for everyone?
20:04:11 <CodeBlock> nirik: +1
20:04:19 <goozbach> +1
20:04:59 <CodeBlock> Alright then
20:05:05 * marchant is here
20:05:10 * nirik will go do that later today.
20:05:10 <CodeBlock> Any other comments/questions/topics about Transifex?
20:05:34 <smooge> how many were left?
20:06:12 * abadger1999 here
20:06:35 <nirik> tickets? there's a number... 10-20?
20:06:46 <nirik> mostly with old issues from our old tx.net instance.
20:07:23 <CodeBlock> Think he meant projects...but I have no clue how many
20:07:55 <CodeBlock> eh, he's busy in -noc :)
20:08:07 <nirik> yeah, not sure on that...
20:08:08 <CodeBlock> #topic Ticket time
20:08:34 <CodeBlock> hm
20:08:55 <CodeBlock> .ticket 2591
20:08:56 <zodbot> CodeBlock: #2591 (Move blogs.fp.o to wordpress.com) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2591
20:09:13 <CodeBlock> So, since we're talking about migrations and such, where is this at?
20:09:14 <sijis> yeah, i didn't get around to send you guys my finding on blogs and users
20:09:19 <sijis> you'll get that in the next day or two
20:09:39 <sijis> sorry about that. i know i said i was gonna send it last week
20:09:42 <skvidal> sijis: thx
20:09:56 <CodeBlock> sijis: alright, no worries
20:10:15 <CodeBlock> .ticket 2574
20:10:17 <zodbot> CodeBlock: #2574 (Perform regular inactive account prunings and possibly a password reset policy.) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2574
20:10:22 <CodeBlock> smooge: poke
20:10:32 <smooge> sorry
20:11:02 <CodeBlock> What's status on our pruning?
20:11:06 <smooge> I am updating the list after this meeting and will mail out this afternoon. Its just me  going through and putting them into groups
20:11:27 <CodeBlock> Alright
20:11:31 <smooge> so that we can say "oh wait notting needs that access even if he doesn't use it"
20:11:47 <notting> hm?
20:12:06 <goozbach> :)
20:12:19 <CodeBlock> smooge: alright
20:12:34 <smooge> notting, sorry name pulled ut of blue
20:12:39 <notting> heh
20:12:43 <CodeBlock> any other comments on that?
20:13:02 <CodeBlock> whee, this is going to be a short meeting :)
20:13:09 <CodeBlock> .ticket 2563
20:13:10 <zodbot> CodeBlock: #2563 (upgrade MediaWiki to 1.16) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2563
20:13:14 <CodeBlock> ianweller: you around?
20:13:19 <skvidal> CodeBlock: I think everyone is being pulled in other directions
20:13:26 <CodeBlock> skvidal: yeah
20:13:28 <CodeBlock> which is fine
20:14:06 <CodeBlock> well then
20:14:09 <CodeBlock> let's do this:
20:14:11 <CodeBlock> #topic open floor
20:14:21 <CodeBlock> any comments for the good of the meeting and life as we know it?
20:14:33 <LyosNorezel> heh... this "meeting" is like trying to have a meeting of "ADHD Anonymous"
20:14:44 <smooge> Ok so the mediawiki is basically waiting on doing it on staging
20:14:45 <sijis> how'd the ibliblio stuff go?
20:14:57 * skvidal can help on that one
20:14:57 <smooge> I think all modules are in epel now
20:15:04 <smooge> but need to double check
20:15:11 <skvidal> so the machine move to the new colo for ibiblio went smoothly
20:15:15 <smooge> sijis, it went very well. The only issue was ipv6
20:15:20 <skvidal> bringing up the hosts and migrating the ipv4 networks went well
20:15:26 <skvidal> ipv6 was a pain
20:15:37 <nirik> typical. ;)
20:16:00 <smooge> however I learned a lot and feel like I know why its such a security nightmare
20:16:02 <skvidal> the only other hiccup we ran into was that Iforgot to do a git-grep to look for puppet being dumb
20:16:10 <skvidal> and re-networking some boxes after I changed them
20:16:19 <skvidal> to be fair - I wasn't the only one who forgot :)
20:16:30 <skvidal> but it got straightened out and the box is running fine afaict
20:16:35 <smooge> no I forgot
20:16:50 <smooge> it was on my checklist and I missed it
20:16:55 <skvidal> smooge: yes - everyone forgot. I personally blame one of the ricky's but I'm not sure which (/me is completely kidding)
20:16:56 <smooge> I learned about git grep
20:17:12 <smooge> I was doing find . -type f | xargs grep 2002:
20:17:17 <CodeBlock> noc02 had issues with puppet wasn't running on it for some reason (even --enable wouldn't kill the lock), and so nagios-external didn't know about the new IPs. Then there were some other config issues with nagios-external that showed up after I fixed that
20:17:22 <CodeBlock> So those are taken care of
20:18:01 <skvidal> sijis: does that answer your question?
20:18:05 <skvidal> oh and in case anyone is interested
20:18:14 <skvidal> ibiblio has some VERY nice new digs
20:18:19 <skvidal> in the ITS manning building at unc-ch
20:18:24 <sijis> yup. was curious. i did see a bunch of stuff for ipv6 giving smooge problems ;)
20:18:29 <CodeBlock> skvidal: blame ricky one. :) He's not here, so we don't have to worry about him imposing death threats upon us, until he reads the log. :)
20:18:43 <skvidal> CodeBlock: it was definitely ONE of the ricky's
20:18:46 <skvidal> anyway
20:19:03 <skvidal> that's how ibiblio move went
20:19:32 <skvidal> what else is out there that people are concerned about?
20:19:43 <CodeBlock> #info ibiblio move went well, and one of the ricky's is automatically blamed for something, but we aren't sure which one. :)
20:19:44 <skvidal> (or just curious)
20:20:01 <skvidal> CodeBlock: good choice that should get ricky's attention ;)
20:20:11 <CodeBlock> :P
20:20:26 <kwright> dgilmore, ping. looking for f15 installation repos
20:20:31 <nirik> are there any other upcoming moves or the like?
20:20:40 <nirik> or should be quiet for a bit?
20:20:58 <dgilmore> kwright: there on every mirror
20:21:08 <skvidal> smooge: the new server?
20:21:15 <skvidal> smooge: the one you want to xfer backup02 to?
20:21:35 <smooge> well its now on its way to PHX2
20:21:41 <goozbach> #agreed nirik will go and clean out tx tickets
20:21:43 <skvidal> does it have an ETA?
20:21:55 <smooge> there is no power for it at RDU so we couldn't put it there
20:21:57 <goozbach> (don't mind me, trying to document earlier meeting for minutes)
20:22:04 <smooge> ok server installation:
20:22:20 <kwright> dgilmore, currently pointing at http://serverbeach1.fedoraproject.org/pub/alt/stage/15-Alpha.RC2/Fedora/i386/os/
20:22:30 <dgilmore> kwright: why
20:22:57 <smooge> we have several EOL and soon to be EOL xen boxes. Mike had put in for a couple last fiscal year
20:23:01 <kwright> dgilmore, this is the url that shows up in the installation gui.
20:23:10 <kwright> dgilmore, what should I be using?
20:23:15 <smooge> kwright, could you do this elsewhere please
20:23:24 <dgilmore> kwright: well alpha is not available yet
20:23:25 <kwright> smooge, sure
20:23:31 <kwright> dgilmore, okay. thanks
20:23:33 <goozbach> #info sijis to send items from findings on blogs and users
20:24:03 <nirik> also, FYI, app06 bodhi alerts have annoyed me, so I am going to dig more into why it's doing that and fix it, even if I have to strace things. ;)
20:24:09 <LyosNorezel> how many servers do ya'll maintain?
20:24:17 <smooge> I ordered the last one but sent it to the wrong place. It looked like a perfect fit as a replacement for backup02 (which is in ibiblio) but it turns out it could not stay in RDU
20:24:23 <ianweller> CodeBlock: yes?
20:24:27 * ianweller just got to his lab
20:24:31 <smooge> so it went to PHX2and will be there next week.
20:24:31 <ianweller> but you can bother me
20:24:42 <goozbach> #info smooge to send update RE accounts to close
20:24:42 <CodeBlock> ianweller: Was poking you about the mediawiki update as we were going through meeting tickets
20:24:51 <ianweller> CodeBlock: standstill on my end
20:24:54 <ianweller> -ENOTIME, etc
20:25:33 <smooge> PHX2 will have a second big KVM box then and we can continue the EL6 dance with fas01 moving onto better hardware
20:25:37 <CodeBlock> LyosNorezel: 136 being monitored by nagios, give or take some.
20:25:48 <LyosNorezel> holly holy
20:26:16 <LyosNorezel> CodeBlock: physical? or virtual/
20:26:18 <LyosNorezel> ?
20:26:26 <CodeBlock> more virt than physical
20:26:32 <LyosNorezel> ah
20:26:35 <smooge> skvidal, does that answer?
20:26:54 * nirik notes many virtuals... many less physical machines.
20:27:13 <LyosNorezel> how many physical servers/
20:27:13 <LyosNorezel> ?
20:27:33 * CodeBlock shrugs
20:27:42 <LyosNorezel> heh
20:27:57 <skvidal> smooge: yah
20:28:00 <skvidal> smooge: thanks
20:28:18 <smooge> #topic Freeze/Release
20:28:31 <smooge> We are going to be in Freeze til next week when Alpha should release
20:28:31 <LyosNorezel> ya'll gonna tell me the infra team doesn't know how many physical servers they manage?
20:28:55 <smooge> not in the middle of a meeting when I am dealing with other stuff
20:29:04 <skvidal> LyosNorezel: I can give you a specific number in a bit
20:29:25 <LyosNorezel> skvidal: ok
20:29:47 <smooge> we are on track for release other than the differences between our i2 downloaders and PHX2 ones
20:30:03 <smooge> I am sending in a change request for that but not sure it will happen
20:30:40 <smooge> after the release we will be rebuilding a set of boxes and taking of all main download services
20:31:52 <smooge> I don't think we have had any issues with this release circle. Hopefully the beta will be better organized :)
20:32:04 <smooge> #topic back to open floor
20:32:53 <nirik> any news on nagios3/noc01 to rhel6? More testing in stg needed?
20:33:12 <smooge> marchant sent out a test plan.
20:33:27 <smooge> it looks good and should be followed. once its checked off we can move to it after the freeze is done.
20:33:37 <CodeBlock> nirik: I believe nagios is gold. I'm more worried about the other services on noc01
20:33:57 <nirik> zodbot?
20:33:59 <smooge> we can then start rewriting the "modules" into being more parent-child so that when ibiblio is down we get one page and not 400
20:34:07 <nirik> cool. Where did the test plan go?
20:34:19 <CodeBlock> infra@ I think
20:34:26 * nirik looks, didn't see it.
20:35:34 <smooge> google is pausing very long for me
20:36:04 <CodeBlock> I'd eventually like to talk about fedorahosted a bit, but..think I'll hold off, maybe until after-freeze
20:36:19 <smooge> #topic FedoraHosted
20:36:24 <smooge> your ball
20:36:29 <nirik> the net is full of fail today.
20:36:30 <CodeBlock> smooge: eh
20:36:30 <nirik> still don't see it, be might just be me. ;)
20:36:30 <smooge> or conch shell
20:36:43 <skvidal> okay
20:36:44 <smooge> oh wait I didnt read the last of your sentance.
20:36:50 <skvidal> so question about fedorahosted
20:37:07 <nirik> new trac would be nice... which should be quite possible now.
20:37:09 <skvidal> do we get rid of projects which are requested to be removed?
20:37:10 <CodeBlock> smooge: I can do it now, it's just more I think most people have more important stuff on their plates atm
20:37:11 <skvidal> https://fedorahosted.org/fedora-infrastructure/ticket/2528
20:37:34 <CodeBlock> skvidal: basically what I was told was that it's fine to remove them, but generally keep a copy of them around in your homedir for a while, just to be safe
20:37:58 <skvidal> nirik: it'd be nice to have a more.... active failover config for the hosted* boxes
20:38:02 <CodeBlock> that's what... I believe mmcgrath told me.
20:38:12 <skvidal> nirik: or better yet an actual cluster for them
20:38:15 <skvidal> CodeBlock: sounds good
20:38:19 <skvidal> averi: ^^^^^^^
20:38:20 <CodeBlock> There's quite a few things that would make hosted "better" I think
20:38:47 <averi> skvidal, great, so keeping a copy of their git repo should be fine for a while
20:38:56 <skvidal> averi: nod
20:39:02 <CodeBlock> For one -- like skvidal just said, I really *really* want to see about doing a cluster-esque thing for it, or something to distribute it out. I'm tired of every time I log into hosted it has a load of 5+
20:39:04 <nirik> perhaps we should have a policy to keep old projects for X days to make sure no one has a "oops, I need X from there"
20:39:08 <averi> and what about the trac istance for that project?
20:39:21 <skvidal> nirik: 14 days is a fine one imo
20:39:29 <skvidal> averi: you can just tar the whole thing up
20:39:32 <skvidal> and stowe it
20:39:35 <skvidal> but I would like to suggest something
20:39:40 <skvidal> instead of doing it in a homedir
20:39:46 <smooge> does anyone know if the sourceforge keep it forever was for "legal" reasons or something else?
20:39:55 <nirik> clustering would be great, but not sure how well trac handles that. Perhaps we could investigate ways to do it with trac.
20:40:08 <skvidal> smooge: don't know and I don't want to become sourceforge
20:40:18 <skvidal> we have a dir /srv/tmp on hosted1
20:40:26 <skvidal> how about we stuff our backup/holdover copies in there
20:40:36 * nirik nods. Sounds reasonable to me.
20:40:39 <skvidal> what I've done in the past is mv the files to there with a new name
20:40:40 <CodeBlock> nirik: yeah. Or even if we can't, let's move some of the other things off of that box. Maybe make a new host for hosted-lists01 and such.
20:40:55 <nirik> CodeBlock: whats causing the load?
20:41:00 <skvidal> mv someproject /srv/tmp/someproject-hold-until-YYYY-MM-DD
20:41:10 <skvidal> nirik: git
20:41:13 <skvidal> nirik: and bz
20:41:15 <skvidal> and git
20:41:18 <skvidal> mostly git
20:41:19 <skvidal> from apache
20:41:35 <nirik> ok, wonder if newer git will help that. ;)
20:41:40 <skvidal> it might
20:41:45 <CodeBlock> what skvidal said + it spikes way high when we generate the list of projects for the front page I think
20:41:46 * nirik has to take the cat and dog into the vet... back in a bit.
20:41:53 <smooge> also the git-web that kernel.org would
20:41:59 <CodeBlock> Which...would it be worth looking into generating that list dynamically?
20:42:06 <averi> skvidal, fully agreed, a backup to be pushed on /srv/tmp should be fine
20:42:07 <skvidal> umm
20:42:08 <CodeBlock> well
20:42:13 <CodeBlock> not dynamically
20:42:14 <skvidal> I thin kgenerating it dynamically would be worse
20:42:16 <skvidal> okay
20:42:17 <skvidal> good
20:42:25 <CodeBlock> but.. hm
20:42:39 <nirik> there's a new trac/rhel6 on one of the publictest boxes, perhaps we could test and gather more info?
20:42:41 <CodeBlock> some way that doesn't require it to spike like it does >.>
20:42:50 * nirik really goes for a bit mow.
20:43:03 <abadger1999> smooge: To address your "are we like sourceforge" question -- it was not for legal reasons that I know.
20:43:13 <CodeBlock> skvidal: maybe a database of projects... maybe pull the list from FAS groups or something, I don't know
20:43:26 <abadger1999> smooge: The original plan was to allow getting rid of inactive hosted projects (ie: without an owner)
20:43:44 <skvidal> CodeBlock: <shrug> fine by me - easy to do in sqlite - sounds like a fun project for you :)
20:43:45 <abadger1999> smooge: But the first time we attempted to reap, a large number of people complained.
20:44:54 <smooge> abadger1999, it is my guess it is the case with every hosting place.
20:45:04 <CodeBlock> skvidal: Would you mind expanding on your cluster idea a bit? (I've lately been thinking of things/ideas for hosted.. I really would like to help improve it a bit)
20:45:12 <smooge> have to find out why the dog is barking brb
20:45:21 <skvidal> CodeBlock: so - there are two ways I can think of doing it
20:45:29 <skvidal> both require a lot of testing
20:45:34 <CodeBlock> sure
20:45:46 <skvidal> so trac clustered, I'm sure is hairy b/c of the sqlitedbs
20:46:05 <skvidal> but what if we distributed the features out to multiple hosts writing to a common disk space
20:46:19 <skvidal> so that if we lost a host we just lost a feature, we didn't lose the whole thing
20:46:46 <skvidal> option 2 is to figure out how to cluster trac
20:47:07 <skvidal> and then have a shared disk pool that the cluster nodes can access and either LB or rrdns them
20:47:21 <skvidal> it's simplistic, I admit - but so is our current setup
20:47:44 <skvidal> as far as mailing lists go - if we change how we do archives
20:47:46 <CodeBlock> fwiw, I just logged into it (hosted01), and it has 8.92 9.27 9.97 .. that is what I'm talking about
20:47:57 <skvidal> we could easily replicate the archives but not the mailing lists themselves
20:48:41 <smooge> CodeBlock, part of that can be fixed with a better gitweb and running git gc on projects every now and then
20:49:04 <skvidal> I suspect that some of our issues have known solutions  - it is a matter of investigating and testing them
20:49:08 <skvidal> but that all takes time
20:49:11 <tmz> I thought you guys ran gitweb caching?
20:50:58 <smooge> yep.. time is what we are lacking on
20:50:59 <nirik> we could also split projects by name or something... a-k and l-z or whatever. or some more even distribution.
20:51:28 <skvidal> nirik: sharding them out is fine by me too
20:51:34 <tmz> smooge: As for git gc, when hosted is updated to git-1.7, that should happen more automatically, IIRC.
20:51:45 <nirik> right now hosted2 is pretty much idle right? hot spare?
20:51:48 <skvidal> nirik: I still think a common disk backend that is replicated across N machines (or maybe to the netapp) might be worth the time
20:51:54 <skvidal> nirik: luke-warm spare
20:51:56 <nirik> yeah, could be.
20:51:59 <skvidal> nirik: if hosted1 DIES DIES
20:52:01 <skvidal> we'll have all the data
20:52:09 <skvidal> but I'm not positive it will pretty moving it over
20:52:25 <skvidal> I suspect we could move the hosted1 puppet config over
20:52:27 <smooge> and ugh
20:52:28 <skvidal> and MOST of it would come back
20:52:33 <smooge> its at serverbeach
20:52:34 <skvidal> but the mailq would be a problem
20:52:39 <CodeBlock> smooge: both are
20:52:43 <CodeBlock> 1 and 2
20:52:45 <smooge> yep..
20:53:42 <CodeBlock> skvidal: I do like the common disk space idea
20:53:55 <skvidal> CodeBlock: so - if we were going to start
20:54:08 <skvidal> I 'd like to see us extend our cloudfs testing
20:54:30 <skvidal> jdarcy: have you had any change in the patches to make auth happen 'server side' for that?
20:54:57 <jdarcy> skvidal: Not so far.  End User Summit kind of interrupted everything.
20:55:05 <skvidal> jdarcy: understood.
20:56:59 <CodeBlock> alright
20:58:00 <CodeBlock> skvidal: it's just something that's been on my mind. Hosted is used -- quite a lot (seen by hosting request tickets and by the constant load averages)... and I'd like to make it into something great
20:58:09 <skvidal> nod
20:59:42 <smooge> ok my head is not getting any better and I think we are getting to the end here.
21:00:04 <CodeBlock> 15
21:00:16 <abadger1999> One questin just to throw out there
21:00:32 <abadger1999> Is there anyone presently in infra who likes to do software release management?
21:00:33 <smooge> ok
21:00:44 <smooge> define pleasE?
21:01:25 <abadger1999> ie: get a herd of cat^Wdevelopers to finish up features, test, pester devs to fix found bugs, then tar things up, build rpms, and such?
21:01:46 <smooge> oh that.. I have done it multiple times
21:01:57 <smooge> I wonder if its something I should be doing and havent
21:02:04 <abadger1999> Rephrase.. does anyone like to and have spare time :-)
21:02:26 <CodeBlock> abadger1999: what project needs it :P
21:02:45 <abadger1999> Reason: some of our services (fas, pkgdb, etc) are getting patches slowly... but we aren't making releases as frequently as the patches come in.
21:02:46 * CodeBlock wouldn't be good for it probably, but is curious
21:03:13 <abadger1999> Mostly because someone needs to sanity check that the tree is in a releasable sttate and then get people to fix any obvious issues.
21:04:37 <abadger1999> Anyhow -- if anyone wants to do that sort of thing, I'll help get you started on either fas or pkgdb.
21:04:46 <averi> do you guys have any clue about where trac stores project files?
21:05:12 <smooge> Well I would like for us to have something like pkgdb for our services
21:05:19 <skvidal> averi: one sec
21:05:22 <skvidal> I was in multiple windows
21:05:24 <sijis> abadger1999: but it sounds like this person would need python experience, right?
21:05:44 <skvidal> averi: /srv/web/trac
21:05:44 <abadger1999> Thought I'd mention it as it's been one of those not-quite-easy-but-not-too-hard things that I've been thinking I could get someone else to manage much better than I have been.
21:05:49 <averi> skvidal, you rock, thanks
21:05:54 <skvidal> averi: /srv/web/trac/projects to be precise
21:05:54 <smooge> sijis, a little.. I have been RL manager and had no coding experience
21:06:03 <smooge> it was more about following a schedule and beating people
21:06:13 <sijis> haha. gotcha.
21:06:15 <abadger1999> sijis: Preferably.  But mostly, packaging, testing, and good people skills.
21:06:18 <smooge> we could clone rbergeron again
21:06:36 <abadger1999> Oooh... /me was unaware that we had an rbergeron cloning facility.
21:06:42 <abadger1999> I'll take five please.
21:06:51 <abadger1999> :-)
21:06:53 <sijis> although the way i understand abadger1999 talking about it, sounds like a 'programmer with managing skills'
21:07:27 <smooge> yes or a manager with programming skills
21:07:29 <smooge> they are rare
21:07:32 <sijis> anyhow, i just wanted to understand the idea a little better
21:07:42 <abadger1999> sijis: Well, I'm a programmer, so I understand how a programmer can work on it from that angle.. but the other way around would likely work well too.
21:08:11 <abadger1999> anyhow.  Just putting it out there as a way for someone to lend a hand where it's needed.
21:10:41 <CodeBlock> :)
21:10:54 <CodeBlock> 15
21:11:06 <CodeBlock> 5
21:11:15 <CodeBlock> #endmeeting