summaryrefslogtreecommitdiffstats
path: root/geo-replication/syncdaemon/master.py
diff options
context:
space:
mode:
authorAravinda VK <avishwan@redhat.com>2015-04-12 17:46:45 +0530
committerVijay Bellur <vbellur@redhat.com>2015-05-06 10:02:27 -0700
commit7ca09c3d23f224efd139b372a88788c7cbe90522 (patch)
tree91356f5d78c0409805539329b01c35708c21cbf4 /geo-replication/syncdaemon/master.py
parent6ebaa045ae8b2523e91c087cffae35cc4ec682c2 (diff)
geo-rep: Minimize rm -rf race in Geo-rep
While doing RMDIR worker gets ENOTEMPTY because same directory will have files from other bricks which are not deleted since that worker is slow processing. So geo-rep does recursive_delete. Recursive delete was done using shutil.rmtree. once started, it will not check disk_gfid in between. So it ends up deleting the new files created by other workers. Also if other worker creates files after one worker gets list of files to be deleted, then first worker will again get ENOTEMPTY again. To fix these races, retry is added when it gets ENOTEMPTY/ESTALE/ENODATA. And disk_gfid check added for original path for which recursive_delete is called. This disk gfid check executed before every Unlink/Rmdir. If disk gfid is not matching with GFID from Changelog, that means other worker deleted the directory. Even if the subdir/file present, it belongs to different parent. Exit without performing further deletes. Retry on ENOENT during create is ignored, since if CREATE/MKNOD/MKDIR failed with ENOENT will not succeed unless parent directory is created again. Rsync errors handling was handling unlinked_gfids_list only for one Changelog, but when processed in batch it fails to detect unlinked_gfids and retries again. Finally skips the entire Changelogs in that batch. Fixed this issue by moving self.unlinked_gfids reset logic before batch start and after batch end. Most of the Geo-rep races with rm -rf is eliminated with this patch, but in some cases stale directories left in some bricks and in mount point we get ENOTEMPTY.(DHT issue, Error will be logged in Slave log) BUG: 1218922 Change-Id: I8716b88e4c741545f526095bf789f7c1e28008cb Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/10204 Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com> Reviewed-on: http://review.gluster.org/10599 Tested-by: Gluster Build System <jenkins@build.gluster.com>
Diffstat (limited to 'geo-replication/syncdaemon/master.py')
-rw-r--r--geo-replication/syncdaemon/master.py3
1 files changed, 1 insertions, 2 deletions
diff --git a/geo-replication/syncdaemon/master.py b/geo-replication/syncdaemon/master.py
index cd20a490397..a98a9b8925f 100644
--- a/geo-replication/syncdaemon/master.py
+++ b/geo-replication/syncdaemon/master.py
@@ -236,7 +236,6 @@ class TarSSHEngine(object):
"""
def a_syncdata(self, files):
- self.unlinked_gfids = []
logging.debug('files: %s' % (files))
self.current_files_skipped_count = 0
del self.skipped_gfid_list[:]
@@ -275,7 +274,6 @@ class RsyncEngine(object):
"""Sync engine that uses rsync(1) for data transfers"""
def a_syncdata(self, files):
- self.unlinked_gfids = []
logging.debug('files: %s' % (files))
self.current_files_skipped_count = 0
del self.skipped_gfid_list[:]
@@ -922,6 +920,7 @@ class GMasterChangelogMixin(GMasterCommon):
def process(self, changes, done=1):
tries = 0
retry = False
+ self.unlinked_gfids = []
while True:
self.skipped_gfid_list = []