20:00:01 <nirik> #startmeeting Infrastructure (2012-04-12)
20:00:01 <zodbot> Meeting started Thu Apr 12 20:00:01 2012 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:00:01 <nirik> #meetingname infrastructure
20:00:01 <zodbot> The meeting name has been set to 'infrastructure'
20:00:01 <nirik> #topic Robot Roll Call
20:00:01 <nirik> #chair smooge skvidal Codeblock ricky nirik abadger1999 lmacken dgilmore mdomsch threebean
20:00:01 <zodbot> Current chairs: Codeblock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge threebean
20:00:07 * skvidal is here
20:00:25 * adrianhannah here
20:00:28 * lmacken 
20:00:30 * abadger1999 here
20:00:31 * athmane is around
20:00:33 * mdomsch 
20:00:36 * kwame is here o/
20:00:52 * CodeBlock here
20:01:09 * wolfkit is here
20:01:17 * nirik waits another minute for folks to wander in
20:02:12 * herlo is here
20:02:48 <nirik> ok, lets go ahead and dive in...
20:02:52 <nirik> #topic New folks introductions and Apprentice tasks.
20:02:53 <nirik> 
20:02:53 <nirik> If any new folks want to give a quick one line bio or any apprentices
20:02:53 <nirik> would like to ask general questions, they can do so now. anyone?
20:03:33 <smooge> is here.
20:03:59 <nirik> I'll note there's at least one apprentice ticket waiting for me to comment. I'm going to try and get to it in the next few days...
20:04:25 <nirik> ok, moving along then...
20:04:27 <nirik> #topic two factor auth status
20:04:31 <nirik> any news here?
20:05:09 <skvidal> not much change
20:05:18 <skvidal> mricon is working on radius support for somethign for them for cisco vpn
20:05:24 <wolfkit> I've been quite busy and the latest rawhide grub2-efi / kernel updates have been playing up on my laptop so haven't been able to hack on it
20:05:33 <nirik> ok.
20:05:43 <nirik> anything we can do to help move it along?
20:06:09 <skvidal> I just need some time to sit down and focus on it, I suspect
20:06:21 <herlo> where are we stuck?
20:06:24 <nirik> ok. I'd be happy to help too.. .would love to get it going.
20:06:26 <skvidal> we're not stuck
20:06:32 * herlo hasn't been able to help in a while, wants to help again
20:06:37 <skvidal> there aren't any roadblocks afaik
20:06:42 <skvidal> it's more just integrate + test
20:06:42 <wolfkit> as do I, time seems to be my most shrinking resource, heh
20:06:42 <herlo> k
20:06:52 <nirik> wolfkit: always the way I fear. ;)
20:07:13 <nirik> ok, lets try and move it forward in the next week, revisit next meeting.
20:08:21 <nirik> #topic Staging re-work status
20:08:28 <nirik> still waiting on freeze on this one...
20:08:41 <nirik> but the happy news is that we have a beta and freeze is over next week. :)
20:08:53 <nirik> #topic Applications status / discussion
20:09:05 <nirik> any application news?
20:09:21 <CodeBlock> dpsearch is evil. That is all ;) -- I have emailed daMaestro asking for help, waiting for response.
20:09:31 <nirik> CodeBlock: ok, thanks.
20:09:53 <nirik> Oh, I have a bit of news on paste... I'm looking at sticky-notes as a possibly fpaste.org replacement server.
20:10:09 <nirik> it might work out and let us take over fpaste.org...
20:10:20 <ianweller> nirik: sticky-notes?
20:10:30 <nirik> It's php, but it has an upstream and they sound like they might be responsive.
20:10:47 <nirik> http://gitorious.org/sticky-notes/sticky-notes/
20:11:06 <nirik> kde uses it.
20:11:07 <nirik> http://paste.kde.org/
20:11:31 <CodeBlock> nirik: is there a CLI app for it? I could look at hacking one up if needed
20:11:32 <herlo> it looks pretty good
20:11:41 <nirik> pastebinit can talk to it apparently.
20:11:44 <nirik> (already in fedora)
20:11:56 <herlo> CodeBlock: the plan there was to maybe modify fpaste to take a configuration file
20:11:56 <smooge> cool
20:12:00 <herlo> nirik: good news!
20:12:13 * herlo thinks that's better anyway
20:12:19 <nirik> so, possibly we could work something with fpaste calling that or have pastebinit ship a fpaste wrapper or something.
20:12:26 <herlo> yeah
20:12:29 <herlo> easy to do
20:12:46 <nirik> pastebinit already defaults to fpaste.org in fedora too.
20:13:03 <herlo> we'll have to change something
20:13:14 <nirik> yeah.
20:13:19 <herlo> but we could address that later
20:13:22 <nirik> can burn that bridge when we come to it...
20:13:40 <nirik> abadger1999 / threebean / pingou / lmacken: any other app news from this week/upcoming?
20:13:48 <abadger1999> nothing upcoming
20:14:04 <threebean> still working on packaging for the message bus
20:14:23 * mdomsch has been hacking MM late at night
20:14:40 <nirik> mdomsch: oh, did you see we had another issue with mirrorlist_server last night...
20:14:42 <mdomsch> v1.4 in progress again, mostly works, needs more testing
20:14:46 <mdomsch> nirik: no...
20:14:47 * athmane reported two vuls to sticky-notes (see merge request), will check the admin console this weekend
20:15:07 <mdomsch> nirik: what now?
20:15:17 <nirik> mdomsch: started taking up memory and causing problems, but then we stopped it all and I tried to get some debugging and started it and it all started working again.
20:15:33 <mdomsch> ah - heisenbug
20:15:41 <nirik> I have no idea if it was a DOS or what was happening. ;( Have any debugging tips or info we can gather if it happens again?
20:15:58 <mdomsch> get a copy of /var/lib/mirrormanager/
20:16:00 <nirik> athmane: yeah, dcr226 also mailed the author a pointer to your patches. ;)
20:16:12 <nirik> mdomsch: I have one from the last issue on april 2nd...
20:16:13 <mdomsch> so we can see if there's corruption of the netblock lists or the pickle
20:16:16 <lmacken> nirik: nothing upcoming, aside from finishing up the apps.fp.o/{packages,tagger} deployment, which I haven't had any cycles for lately
20:16:32 <nirik> mdomsch: bastion:~kevin/mirorrmanager.tar.xz
20:16:39 <athmane> nirik: feel free to ping me, if you need help with the deployment
20:16:52 <nirik> rorry, mirrormanager-2012-04-02.tar.xz is the file
20:17:02 <nirik> athmane: excellent. will do.
20:17:02 <mdomsch> I did spend considerable time teaching MM how to do database schema updates for various oddities not covered by SQLObject itself
20:17:02 <nb> nirik, so are we not planning on sticking with th current fpaste-server?
20:17:15 <mdomsch> particularly adding indexes
20:17:30 <herlo> nb: it's still a possibility, but there's a ton of work that needs to be put into fpaste-server before it's ready
20:17:45 <nirik> nb: we tried, but herlo and daMaestro both thought the hacks were too gross for a clean deployment, and they also were not really wanting to be an upstream for it...
20:18:02 <nb> oh ok
20:18:03 <herlo> I wasn't aware of the amount of work it would take and it's going to take some serious amount of time to hack upon it
20:18:11 <herlo> to get it into shape
20:18:38 <nirik> mdomsch: what does mirrorlist_server do? which part? anyhow, if you could look at that dump and/or suggest things for us to collect I can do that if it happens again.
20:19:06 <mdomsch> nirik: that's the URL endpoint that yum hits
20:19:20 <nirik> ok, so it could have been some kind of DOS... ?
20:19:22 <mdomsch> and that download.fp.o URLs hit to generate the redirect
20:19:26 <mdomsch> yes, could be a DOS
20:19:33 <mdomsch> though that'd be surprising
20:19:37 * nirik can try checking logs for unusual hits.
20:19:44 <nirik> yeah, it was odd.
20:19:55 <nirik> it happened on app01 first... caused it to crash.
20:20:03 <nirik> then a few minutes later the other apps started doing it too.
20:20:18 <mdomsch> normally when it does this, it means the communication path between apache wsgi app and mirrorlist_server.py app got interrupted
20:20:32 <mdomsch> and both sides go a little nuts waiting on the other
20:20:50 <mdomsch> but yes, pls check the proxy logs for spikes
20:20:57 <nirik> will do.
20:21:06 <mdomsch> I suspect it's a bug in mirrorlist_server.py somehow
20:21:24 <nirik> yeah, I tried to change 'log_file' and 'debug' but it didn't seem to do anything. ;(
20:21:24 <abadger1999> nirik: oh I took a look at the ipnetblocks file -- it's not corrupt although it is very big.
20:21:27 <mdomsch> that doesn't log output much if at all - and it's run by supervisor
20:21:34 <nirik> yeah
20:21:42 <mdomsch> yeah, these netblock lists look sane at a glance
20:21:53 <abadger1999> Loads fine using the same code as is in mirrormanager
20:22:06 <nirik> so there goes that theory. ;(
20:22:29 <mdomsch> the pickle has a much newer timestamp
20:22:37 <mdomsch> than the netblock lists
20:23:12 <mdomsch> anyhow - not obvious...
20:23:30 <mdomsch> page me if it happens again
20:23:38 <mdomsch> ideally while it's happening
20:23:54 <mdomsch> and look at ps auxgw | grep mirror
20:24:20 <mdomsch> to see what's there.  normally there are a few mirrorlist_server.py processes there, 1 master plus one for each request in process
20:24:34 <nirik> mdomsch: ok. we do have OOM dumps in logs from the kernel if it helps any?
20:24:50 <mdomsch> if there are more than say 20, and if any are in S state, that's a problem
20:25:17 <mdomsch> it might, yes
20:26:19 <nirik> ok. can get you that.
20:26:33 <nirik> ok, any more app news? or shall we move on?
20:27:37 <nirik> #topic Upcoming Tasks/Items
20:28:34 <nirik> 2012-03-20 to 2012-04-17 - F17 Beta Freeze
20:28:34 <nirik> 2012-04-17 - F17Beta release day
20:28:34 <nirik> 2012-05-08 to 2012-05-22 - F17 Final Freeze.
20:28:34 <nirik> 2012-05-01 - nag fi-apprentices.
20:28:34 <nirik> 2011-05-03 - gitweb-cache removal day.
20:28:35 <nirik> 2012-05-09 - Check if puppet works on f17 yet.
20:28:37 <nirik> 2012-05-10 - drop inactive fi-apprentices
20:28:39 <nirik> 2012-05-22 - F17 release
20:28:44 <nirik> so beta comes out tuesday..
20:29:03 <nirik> we have our usual release tickets we will need to act on once the trees are staged.
20:29:25 <nirik> .ticket 3218
20:29:27 <zodbot> nirik: #3218 (Fedora17 Beta - Verify mirror space) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/3218
20:29:29 <nirik> .ticket 3219
20:29:30 <CodeBlock> nirik: safe tavels back in time to 2011 to remove gitweb-cache ;)
20:29:31 <zodbot> nirik: #3219 (Fedora17 Beta - new website) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/3219
20:29:33 <nirik> .ticket 3220
20:29:35 <zodbot> nirik: #3220 (Fedora17 Beta - release day ticket) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/3220
20:29:41 <nirik> .ticket 3221
20:29:43 <zodbot> nirik: #3221 (Fedora17 Beta - verify permissions on content) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/3221
20:29:49 <nirik> CodeBlock: ha. Paying attention I see...
20:29:51 * nirik fixes
20:30:31 <nirik> anything else upcoming anyone would like to note or schedule?
20:31:04 <nirik> I've added a wiki page with our private cloud plans: https://fedoraproject.org/wiki/Infrastructure_private_cloud
20:31:24 <nirik> we don't have concrete dates on it yet, but I can add them in when we have finalized hardware and such
20:32:23 <nirik> use ideas, suggestions or comments welcome on the list
20:32:44 <nirik> #topic Open Floor
20:32:50 <nirik> anyone have items for open floor?
20:33:27 <nb> not i
20:33:40 <smooge> yay release
20:33:48 <nirik> nb: oh, any news on the keyserver thing? or haven't had a chance to try yet?
20:33:49 <smooge> not i
20:33:51 <smooge> oh wait
20:34:04 <CodeBlock> my online presence will be iffy tomorrow and the weekend of April 28th. Not schedule worthy, but just noting.
20:34:12 <nb> nirik, nope, not had time yet, just got pw yesterday
20:34:25 <nirik> nb: no worries. Will be interesting to see if it works
20:34:32 <smooge> I am currently working on removing ALL mailing list aliases from redhat.com -> lists.fedoraproject.org
20:34:40 <nirik> smooge: hurray!
20:34:52 <smooge> I will then work on moving epel and some other lists over to lists.fedoraproject.org
20:35:00 <smooge> which is always fun
20:35:30 <nirik> oh, I had another thing to note: We were going to migrate hosted monday, but postponed it. We want to shake out a timeout/failover issue with the new glusterfs setup before making it live.
20:35:38 <nirik> Hopefully we can migrate that next week or so.
20:37:01 <nirik> ok, if nothing else we can call it a meeting and get back to work. ;)
20:37:17 <smooge> call it
20:37:44 <nirik> thanks for coming everyone!
20:37:47 <nirik> #endmeeting