path: root/libglusterfs/src/syncop-utils.h
diff options
authorPranith Kumar K <>2016-03-17 09:32:02 +0530
committerJeff Darcy <>2016-04-04 17:36:33 -0700
commitc76a1690bbd909b1c2dd2c495e2a8352d599b14b (patch)
tree30906a06541623d7249e23134bd97fd17e562f9e /libglusterfs/src/syncop-utils.h
parent3394c555171775c6f74c2c556265b044a70a2fa9 (diff)
syncop: Add parallel dir scan functionality
Most of this functionality's ideas are contributed by Richard Wareing, in his patch: VERY BIG thanks to him :-). After starting porting/testing the patch above, I found a few things we can improve in this patch based on the results we got in testing. 1) We are reading all the indices before we launch self-heals. In some customer cases I worked on there were almost 5million files/directories that needed heal. With such a big number self-heal daemon will be OOM killed if we go this route. So I modified this to launch heals based on a queue length limit. 2) We found that for directory hierarchies, multi-threaded self-heal patch was not giving better results compared to single-threaded self-heal because of the order problems. We improved index xlator to give gfid type to make sure that all directories in the indices are healed before the files that follow in that iteration of readdir output( In our testing this lead to zero errors of self-heals as we were only doing self-heals in parallel for files and not directories. I think we can further improve self-heal speed for directories by doing name heals in parallel based on similar techniques Richard's patch showed. I think the best thing there would be to introduce synccond_t infra (pthread_cond_t kind of infra for syncops) which I am planning to implement for future releases. 3) Based on 1), 2) and the fact that afr already does retries of the indices in a loop I removed retries again in the threads. 4) After the refactor, the changes required to bring in multi-threaded self-heal for ec would just be ~10 lines, most of it will be about options initialization. Our tests found that we are able to easily saturate network :-). High level description of the final feature: Traditionally self-heal daemon reads the indices (gfids) that need to be healed from the brick and initiates heal one gfid at a time. Goal of this feature is to add parallelization to the way we do self-heals in a way we do not regress in any case but increase parallelization wherever we can. As part of this following knobs are introduced to improve parallelization: 1) We can launch 'max-jobs' number of heals in parallel. 2) We can keep reading indices as long as the wait-q for heals doesn't go over 'max-qlen' passed as arguments to multi-threaded dir_scan. As a first cut, we always do healing of directories in serial order one at a time but for files we launch heals in parallel. In future we can do name-heals of dir in parallel, but this is not implemented as of now. Reason for this is mentioned already in '2)' above. AFR/EC can introduce options like max-shd-threads/wait-qlength which can be set by users to increase the rate of heals when they want. Please note that the options will take effect only for the next crawl. BUG: 1221737 Change-Id: I8fc0afc334def87797f6d41e309cefc722a317d2 Signed-off-by: Pranith Kumar K <> Reviewed-on: NetBSD-regression: NetBSD Build System <> CentOS-regression: Gluster Build System <> Reviewed-by: Jeff Darcy <> Smoke: Gluster Build System <>
Diffstat (limited to 'libglusterfs/src/syncop-utils.h')
1 files changed, 7 insertions, 0 deletions
diff --git a/libglusterfs/src/syncop-utils.h b/libglusterfs/src/syncop-utils.h
index 7a9ccacb285..52bcfd99429 100644
--- a/libglusterfs/src/syncop-utils.h
+++ b/libglusterfs/src/syncop-utils.h
@@ -11,12 +11,19 @@
+typedef int (*syncop_dir_scan_fn_t) (xlator_t *subvol, gf_dirent_t *entry,
+ loc_t *parent, void *data);
syncop_ftw (xlator_t *subvol, loc_t *loc, int pid, void *data,
int (*fn) (xlator_t *subvol, gf_dirent_t *entry, loc_t *parent,
void *data));
+syncop_mt_dir_scan (xlator_t *subvol, loc_t *loc, int pid, void *data,
+ syncop_dir_scan_fn_t fn, dict_t *xdata, uint32_t max_jobs,
+ uint32_t max_qlen);
syncop_dir_scan (xlator_t *subvol, loc_t *loc, int pid, void *data,
int (*fn) (xlator_t *subvol, gf_dirent_t *entry, loc_t *parent,
void *data));