18:20:28 #startmeeting 18:20:28 Meeting started Tue Jul 31 18:20:28 2012 UTC. The chair is bulde. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:20:28 Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:20:38 (11:45:27 PM) a2: bulde, pong 18:20:38 (11:49:47 PM) bulde: a2: was looking at enhancing logging.[ch] 18:20:38 (11:50:26 PM) bulde: first thing i wanted was log variables to get into ctx->log.{varname}; or something similar 18:20:38 (11:50:50 PM) bulde: but it will soon get into circular dependency... :-/ resolving that right now 18:20:38 (11:52:08 PM) bulde: mean time, about the flush(); patch, that was not 'well-thought' patch... planning to send one, where all log will be happening to a 1M (or lesser/more) buffer, which will be synced to disk on EOF or on a GF_LOG_ERROR or higher level logs 18:20:39 (11:52:16 PM) bulde: does that make sense? 18:20:39 ( 18:22:04 also, asked sgowda to work on dht sending an extra flag (mostly as a key in xdata) with readdirp() to send only files from posix layer... (this nature will be triggered from a xlator-option) 18:22:26 this helps for the case of readdirp() from dht to 'other than first' subvolumes 18:25:30 imagine a readdirp(size=4k) on 2nd subvol (ie, done with first already), and the directory has ~100k dirs inside that, and volume has ~20 dht subvols.... there will be 'lots and lots' of frames created, n/w calls made, and also it can result in client hanging on readdirp () because dht is not able to hit EOF or proper dentry set to fill 4k buf 18:26:20 with this one feature/option, we can cut on n/w calls to N->1, and numbers of frames used also to N*M -> M 18:27:28 this may not be 'strictly posix' behavior, but on few mounts where we don't have to care much for it, and the task is just crawling the filesystem, this enhancement would still help.... 18:28:06 i will back up this theory with more solid numbers once the basic patch is ready (which is mostly by thursday IST) 18:31:53 also, wanted patch http://review.gluster.com/3532 to be taken in... want to get lot of logs into in-memory, they fill up the disk fast in few cases 18:32:01 sorry, was not looking at the window 18:32:07 was trying to fix dev.gluster.com (seems to be hung) 18:32:11 reading your messages now 18:32:19 a2: np, was recording the log :p 18:32:39 a2: oh! thats good, review.gluster.com was pathetically slow whole day 18:33:03 bulde, what if glusterfsd dies before flushing out logs? valuable info might be lost 18:33:41 oh GF_LOG_ERROR would force flush 18:33:42 got it 18:33:50 but where will you log it? in memory? 18:34:40 still thinking 18:35:02 readdirp option to skip dirs is fine 18:35:03 as of now, yes, a buffer in ctx (or where ever it fits better) 18:35:07 we cannot do it in readdir though 18:35:19 hmm... 18:35:55 considering in most of the 'general' use cases we use readdirp() (as md-cache will mostly be present), that should be fine 18:36:14 yeah 18:36:57 you don't want to rely on d_type for filtering at posix level? (that would save a stat() call) 18:37:20 i don't think there is a guarantee that d_type will be set 18:38:21 the logging 'big buffer' can have adverse effect on performance if it is actively used 18:38:55 it will keep dirtying hardware memory caches (l1/l2/l3) extensively 18:39:30 hmm 18:39:43 logging to memory is lesser of an issue.. 18:40:24 and about \n flushing to disk... i am not sure, i was thinking that it flushes only in case FILE * is 'std---' 18:40:33 are you sure every 18:40:40 \n is syncing to disk? 18:42:50 a \n does not fsync() 18:42:58 it only performs write() syscall when \n is reached 18:46:00 a \n == fflush() 18:46:36 hmm 18:48:26 best thing is to use event-history i guess... (for all the repeating logs, at the entry/exit points) 18:49:25 ok.. will review it.. 19:07:03 #endmeeting