glusterfs.git/geo-replication/syncdaemon, branch v3.7.2

geo-rep: Ignore ESTALE during unlink/rmdir

2015-06-01T06:38:15+00:00

during unlink/rmdir of Parent_GFID/Basename, if parent
directory does not exists. Parent GFID will not get resolved
and DHT raises ESTALE instead of ENOENT.

Now ESTALE errors ignored during unlink/rmdir

BUG: 1223286
Change-Id: If275c89fb9fc7d16004550805a4cd65be818540d
Signed-off-by: Aravinda VK 
Reviewed On: http://review.gluster.org/#/c/10837/
Reviewed-on: http://review.gluster.org/10913
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Kotresh HR 
Reviewed-by: Venky Shankar

geo-rep: Fix Data counter issue in status

2015-05-31T14:44:20+00:00

ENTRY and META operations executed sequentially, DATA operations
are handled async, increment happens when a changelog parsed.
Decrement happens after the sync of all files.

files_in_batch was reset multiple times in batch instead of once.

BUG: 1224100
Change-Id: I87617f2fd5f4d3221a1c9f9d5a8efb0686c42bbe
Signed-off-by: Aravinda VK 
Reviewed On: http://review.gluster.org/#/c/10911/
Reviewed-on: http://review.gluster.org/10912
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System 
Reviewed-by: Venky Shankar

geo-rep: Disable xattrs and acls support with tar_ssh

2015-05-29T09:02:32+00:00

Geo-rep can't sync xattrs and acls with tar over ssh
for following reasons.

Issue 1: xattrs doesn't sync with tar over ssh.

Reason: untar doesn't respect '--overwrite' option when used along
        with '--xattrs'. So it sends unlink if the file exists on
        destination and re-creates afresh. But all entry operations
        are banned in aux-gfid-mount as it may lead to gfid-mismatch.
        Hence fails with EPERM. This happens only when some xattr is
        set on a file in master volume.

Issue2: acls on directories does not sync with tar over ssh.

Reason: tar tries to opendir ".gfid/" and is not supported
        by gfid-access-translator as readirp can't be handled on
        virtual inodes and hence fails with ENOTSUP where as it syncs
        for files.

Since the issue is with tar commmand it self and nothing could be
done from gluster side, disabling xattr and acls support with tar
over ssh option.

Geo-rep can sync xattrs and acls with 'rsync' as the sync engine.

BUG: 1223644
Change-Id: Ieba9e749ddbc41683b58ed8b1cc2ca50517fe58b
Reviewed On: http://review.gluster.org/#/c/10873
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/10899
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System
Reviewed-by: Aravinda VK

geo-rep: Fix corrupt gsyncd output

2015-05-09T09:18:04+00:00

When gsyncd fails with Python traceback, glusterd fails
parsing gsyncd output and shows error.

BUG: 1219938
Change-Id: Ic32fd897c49a5325294a6588351b539c6e124338
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/10694
Reviewed-on: http://review.gluster.org/10695
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System
Reviewed-by: Kotresh HR 
Reviewed-by: Vijay Bellur

geo-rep: Fix Default values in Xml output

2015-05-08T09:25:17+00:00

Default Values for last_synced, checkpoint_time and
checkpoint_completion_time was zero instead of 'N/A'

BUG: 1218586
Change-Id: I8a96688588dd6e22edf8bc06ee0f3afe12a9e0f5
Reviewed-on: http://review.gluster.org/10580
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/10636
Reviewed-by: Saravanakumar Arumugam 
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

geo-rep: Fix Rsync hang issue

2015-05-08T09:19:59+00:00

When rsync is executed using Python subprocess, by default
stdout of subprocess will be None. With the log rsync performance
patch stdout is assigned to PIPE. Rsync writes to that PIPE
whenever it syncs files. If log_rsync_performance is disabled
then nobody will consume stdout and that gets full. Rsync hangs
if PIPE is full.

log_rsync_performance option is introduced with patch 10070

With this patch stdout=PIPE only if log_rsync_performance is
enabled. Also removed -v option from Rsync.

Thanks Venky and Kotresh for RCA.

BUG: 1219444
Change-Id: I4b06ca2ebdcb93ac8319f60bc637182fb3d38091
Original-Author: Aravinda VK 
Reviewed-on: http://review.gluster.org/10556
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/10634
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

geo-rep: Limit number of changelogs to process in batch

2015-05-08T04:47:48+00:00

Changelog processing is done in batch, for example if 10 changelogs
available for processing then process all at once. Collect Entry, Meta
and Data operations separately, All the entry operations like CREATE,
MKDIR, MKNOD, LINK, UNLINK will be executed first then rsync will be
triggered for whole batch. Stime will get updated once the complete
batch is complete.

In case of large number of Changelogs in a batch, If geo-rep fails after
Entry operations, but before rsync then on restart, it again starts from the
beginning since stime is not updated. It has to process all the changelogs
again. While processing same changelogs again, all CREATE will get EEXIST
since all the files created in previous run. Big hit for performance.

With this patch, Geo-rep limits number of changelogs per batch based on
Changelog file size. So that when geo-rep fails it has to retry only last batch
changelogs since stime gets updated after each batch.

BUG: 1217930
Change-Id: I844448c4cdcce38a3a2e2cca7c9a50db8f5a9062
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/10202
Reviewed-by: Kotresh HR 
Reviewed-by: Vijay Bellur 
Reviewed-on: http://review.gluster.org/10499
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System
Reviewed-by: Venky Shankar

geo-rep: Minimize rm -rf race in Geo-rep

2015-05-06T17:02:27+00:00

While doing RMDIR worker gets ENOTEMPTY because same directory will
have files from other bricks which are not deleted since that worker
is slow processing. So geo-rep does recursive_delete.

Recursive delete was done using shutil.rmtree. once started, it will
not check disk_gfid in between. So it ends up deleting the new files
created by other workers. Also if other worker creates files after one
worker gets list of files to be deleted, then first worker will again
get ENOTEMPTY again.

To fix these races, retry is added when it gets ENOTEMPTY/ESTALE/ENODATA.
And disk_gfid check added for original path for which recursive_delete is
called. This disk gfid check executed before every Unlink/Rmdir. If disk
gfid is not matching with GFID from Changelog, that means other worker
deleted the directory. Even if the subdir/file present, it belongs to
different parent. Exit without performing further deletes.

Retry on ENOENT during create is ignored, since if CREATE/MKNOD/MKDIR
failed with ENOENT will not succeed unless parent directory is created
again.

Rsync errors handling was handling unlinked_gfids_list only for one
Changelog, but when processed in batch it fails to detect unlinked_gfids
and retries again. Finally skips the entire Changelogs in that batch.
Fixed this issue by moving self.unlinked_gfids reset logic before batch
start and after batch end.

Most of the Geo-rep races with rm -rf is eliminated with this patch,
but in some cases stale directories left in some bricks and in mount
point we get ENOTEMPTY.(DHT issue, Error will be logged in Slave log)

BUG: 1218922
Change-Id: I8716b88e4c741545f526095bf789f7c1e28008cb
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/10204
Reviewed-by: Kotresh HR 
Reviewed-by: Vijay Bellur 
Reviewed-on: http://review.gluster.org/10599
Tested-by: Gluster Build System

geo-rep: Status Enhancements

2015-05-06T16:12:35+00:00

Discussion in gluster-devel
http://www.gluster.org/pipermail/gluster-devel/2015-April/044301.html

MASTER NODE - Master Volume Node
MASTER VOL - Master Volume name
MASTER BRICK - Master Volume Brick
SLAVE USER - Slave User to which Geo-rep session is established
SLAVE - :: used in Geo-rep Create command
SLAVE NODE - Slave Node to which Master worker is connected
STATUS - Worker Status(Created, Initializing, Active, Passive, Faulty,
         Paused, Stopped)
CRAWL STATUS - Crawl type(Hybrid Crawl, History Crawl, Changelog Crawl)
LAST_SYNCED - Last Synced Time(Local Time in CLI output and UTC in XML output)
ENTRY - Number of entry Operations pending.(Resets on worker restart)
DATA - Number of Data operations pending(Resets on worker restart)
META - Number of Meta operations pending(Resets on worker restart)
FAILURES - Number of Failures
CHECKPOINT TIME - Checkpoint set Time(Local Time in CLI output and UTC
                  in XML output)
CHECKPOINT COMPLETED - Yes/No or N/A
CHECKPOINT COMPLETION TIME - Checkpoint Completed Time(Local Time in CLI
                             output and UTC in XML output)

XML output:

cliOutput>
  geoRep>
      volume>
        name>
        sessions>
          session>
             session_slave>
             pair>
                master_node>
                master_brick>
                slave_user>
                slave/>
                slave_node>
                status>
                crawl_status>
                entry>
                data>
                meta>
                failures>
                checkpoint_completed>
                master_node_uuid>
                last_synced>
                checkpoint_time>
                checkpoint_completion_time>

BUG: 1218586
Change-Id: I944a6c3c67f1e6d6baf9670b474233bec8f61ea3
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/10121
Reviewed-by: Kotresh HR 
Reviewed-by: Vijay Bellur 
Reviewed-on: http://review.gluster.org/10574
Tested-by: Gluster Build System

geo-rep: log ENTRY failures from slave on master

2015-05-06T16:07:16+00:00

ENTRY operations failures on slave left no trace for debugging purposes.
This patch captures such failures on slave cluster and forwards them to
the master and logs them. Failures of specific interest are the ones
which return code EEXIST on the failing operations.

Change-Id: Iecab876f16593c746d53f4b7ec2e0783367856bb
Signed-off-by: Milind Changire 
BUG: 1217429
Reviewed-on: http://review.gluster.org/10048
Reviewed-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/10517
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System