18:20:28 <bulde> #startmeeting
18:20:28 <zodbot> Meeting started Tue Jul 31 18:20:28 2012 UTC.  The chair is bulde. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:20:28 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:20:38 <bulde> (11:45:27 PM) a2: bulde, pong
18:20:38 <bulde> (11:49:47 PM) bulde: a2: was looking at enhancing logging.[ch]
18:20:38 <bulde> (11:50:26 PM) bulde: first thing i wanted was log variables to get into ctx->log.{varname}; or something similar
18:20:38 <bulde> (11:50:50 PM) bulde: but it will soon get into circular dependency... :-/ resolving that right now
18:20:38 <bulde> (11:52:08 PM) bulde: mean time, about the flush(); patch, that was not 'well-thought' patch... planning to send one, where all log will be happening to a 1M (or lesser/more) buffer, which will be synced to disk on EOF or on a GF_LOG_ERROR or higher level logs
18:20:39 <bulde> (11:52:16 PM) bulde: does that make sense?
18:20:39 <bulde> (
18:22:04 <bulde> also, asked sgowda to work on dht sending an extra flag (mostly as a key in xdata) with readdirp() to send only files from posix layer... (this nature will be triggered from a xlator-option)
18:22:26 <bulde> this helps for the case of readdirp() from dht to 'other than first' subvolumes
18:25:30 <bulde> imagine a readdirp(size=4k) on 2nd subvol (ie, done with first already), and the directory has ~100k dirs inside that, and volume has ~20 dht subvols.... there will be 'lots and lots' of frames created, n/w calls made, and also it can result in client hanging on readdirp () because dht is not able to hit EOF or proper dentry set to fill 4k buf
18:26:20 <bulde> with this one feature/option, we can cut on n/w calls to N->1, and numbers of frames used also to N*M -> M
18:27:28 <bulde> this may not be 'strictly posix' behavior, but on few mounts where we don't have to care much for it, and the task is just crawling the filesystem, this enhancement would still help....
18:28:06 <bulde> i will back up this theory with more solid numbers once the basic patch is ready (which is mostly by thursday IST)
18:31:53 <bulde> also, wanted patch http://review.gluster.com/3532 to be taken in... want to get lot of logs into in-memory, they fill up the disk fast in few cases
18:32:01 <a2> sorry, was not looking at the window
18:32:07 <a2> was trying to fix dev.gluster.com (seems to be hung)
18:32:11 <a2> reading your messages now
18:32:19 <bulde> a2: np, was recording the log :p
18:32:39 <bulde> a2: oh! thats good, review.gluster.com was pathetically slow whole day
18:33:03 <a2> bulde, what if glusterfsd dies before flushing out logs? valuable info might be lost
18:33:41 <a2> oh GF_LOG_ERROR would force flush
18:33:42 <a2> got it
18:33:50 <a2> but where will you log it? in memory?
18:34:40 <bulde> still thinking
18:35:02 <a2> readdirp option to skip dirs is fine
18:35:03 <bulde> as of now, yes, a buffer in ctx (or where ever it fits better)
18:35:07 <a2> we cannot do it in readdir though
18:35:19 <bulde> hmm...
18:35:55 <bulde> considering in most of the 'general' use cases we use readdirp() (as md-cache will mostly be present), that should be fine
18:36:14 <a2> yeah
18:36:57 <bulde> you don't want to rely on d_type for filtering at posix level? (that would save a stat() call)
18:37:20 <a2> i don't think there is a guarantee that d_type will be set
18:38:21 <a2> the logging 'big buffer' can have adverse effect on performance if it is actively used
18:38:55 <a2> it will keep dirtying hardware memory caches (l1/l2/l3) extensively
18:39:30 <bulde> hmm
18:39:43 <bulde> logging to memory is lesser of an issue..
18:40:24 <bulde> and about \n flushing to disk... i am not sure, i was thinking that it flushes only in case FILE * is  'std---'
18:40:33 <bulde> are you sure every
18:40:40 <bulde> \n is syncing to disk?
18:42:50 <a2> a \n does not fsync()
18:42:58 <a2> it only performs write() syscall when \n is reached
18:46:00 <a2> a \n == fflush()
18:46:36 <bulde> hmm
18:48:26 <bulde> best thing is to use event-history i guess... (for all the repeating logs, at the entry/exit points)
18:49:25 <a2> ok.. will review it..
19:07:03 <bulde> #endmeeting