glusterfs.git/geo-replication/syncdaemon/syncdutils.py, branch v3.8.10

geo-rep: Use configured log_level for libgfchangelog logs

2016-09-14T07:07:47+00:00

libgfchangelog was not respecting the log_level configured
in Geo-replication. With this patch Libgfchangelog log level
can be configured using `config changelog_log_level TRACE`.
Default Changelog log level is INFO

> Reviewed-on: http://review.gluster.org/15078
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Kotresh HR 

BUG: 1374610
Change-Id: Ida714931129f6a1331b9d0815da77efcb2b898e3
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15450
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Atin Mukherjee

features/libgfchangelog: Log failure in gf_histroy_changelog

2016-08-19T06:05:25+00:00

Add error logs if gf_history_changelog fails. If requested
changelog range is not available, log the error and exit
instead of continuing the loop and exiting in readdir
without logging. Also fixed the duplicate MSGID number in
'changelog-lib-messages.h'

> Change-Id: Icd71b89ae23b48a71380657ba5649029c32fabfd
> BUG: 1362151
> Signed-off-by: Kotresh HR 
> Reviewed-on: http://review.gluster.org/15064
> Smoke: Gluster Build System 
> CentOS-regression: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> Reviewed-by: Aravinda VK 

Change-Id: Icd71b89ae23b48a71380657ba5649029c32fabfd
BUG: 1365879
Signed-off-by: Kotresh HR 
(cherry picked from commit 949472d7561d3bfd67d8204e433a25dbc8a596cc)
Reviewed-on: http://review.gluster.org/15138
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Aravinda VK

geo-rep: Error message cleanup

2016-06-21T06:05:11+00:00

If ssh returns 127 that means the remote gsyncd path is wrong
or push-pem failed during create. Existing error message was
pointing old documentation.

Change-Id: Ifbbb4a604fc0ae0fd5cb2746df6363bf28cde1e9
BUG: 1347207
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/14673
 (cherry picked from commit 3b69e25c0f3cf0b958fbd2b8459712ad21239caa)
Reviewed-on: http://review.gluster.org/14741
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR

geo-rep: Do not crash worker on ESTALE

2015-08-03T09:34:48+00:00

Handle ESTALE returned by lstat gracefully
by retrying it. Do not crash the worker.

Change-Id: I2527cd8bd1f7d2428cb4fa3f20782bebaf2df12a
BUG: 1247529
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/11772
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System 
Reviewed-by: Aravinda VK 
Reviewed-by: Venky Shankar

geo-rep: Minimize rm -rf race in Geo-rep

2015-05-05T13:45:01+00:00

While doing RMDIR worker gets ENOTEMPTY because same directory will
have files from other bricks which are not deleted since that worker
is slow processing. So geo-rep does recursive_delete.

Recursive delete was done using shutil.rmtree. once started, it will
not check disk_gfid in between. So it ends up deleting the new files
created by other workers. Also if other worker creates files after one
worker gets list of files to be deleted, then first worker will again
get ENOTEMPTY again.

To fix these races, retry is added when it gets ENOTEMPTY/ESTALE/ENODATA.
And disk_gfid check added for original path for which recursive_delete is
called. This disk gfid check executed before every Unlink/Rmdir. If disk
gfid is not matching with GFID from Changelog, that means other worker
deleted the directory. Even if the subdir/file present, it belongs to
different parent. Exit without performing further deletes.

Retry on ENOENT during create is ignored, since if CREATE/MKNOD/MKDIR
failed with ENOENT will not succeed unless parent directory is created
again.

Rsync errors handling was handling unlinked_gfids_list only for one
Changelog, but when processed in batch it fails to detect unlinked_gfids
and retries again. Finally skips the entire Changelogs in that batch.
Fixed this issue by moving self.unlinked_gfids reset logic before batch
start and after batch end.

Most of the Geo-rep races with rm -rf is eliminated with this patch,
but in some cases stale directories left in some bricks and in mount
point we get ENOTEMPTY.(DHT issue, Error will be logged in Slave log)

BUG: 1211037
Change-Id: I8716b88e4c741545f526095bf789f7c1e28008cb
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/10204
Reviewed-by: Kotresh HR 
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System
Reviewed-by: Vijay Bellur

geo-rep: log ENTRY failures from slave on master

2015-04-28T05:05:18+00:00

ENTRY operations failures on slave left no trace for debugging purposes.
This patch captures such failures on slave cluster and forwards them to
the master and logs them. Failures of specific interest are the ones
which return code EEXIST on the failing operations.

Change-Id: Iecab876f16593c746d53f4b7ec2e0783367856bb
BUG: 1207115
Signed-off-by: Milind Changire 
Reviewed-on: http://review.gluster.org/10048
Reviewed-by: Aravinda VK 
Tested-by: NetBSD Build System
Tested-by: Gluster Build System

geo-rep: Adhering to the common storage for geo-rep

2015-04-28T04:02:21+00:00

Making geo-rep use the common storage shared by nfs,
snapshot and geo-rep. The meta volume should be named
as gluster_shared_storage, and it should be mounted
at "/var/run/gluster/shared_storage/".

geo-rep will have create a directory called 'geo-rep'
in the meta-volume and all the lock files are created
inside it.

Change-Id: I82d0bff9be191f75f643606a9a21d53559047ac4
BUG: 1210344
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/10196
Reviewed-by: Aravinda VK 
Tested-by: NetBSD Build System
Tested-by: Gluster Build System

feature/geo-rep: Active Passive Switching logic flock

2015-03-16T04:53:59+00:00

CURRENT DESIGN AND ITS LIMITATIONS:
-----------------------------------
Geo-replication syncs changes across geography using changelogs captured
by changelog translator. Changelog translator sits on server side just
above posix translator. Hence, in distributed replicated setup, both
replica pairs collect changelogs w.r.t their bricks. Geo-replication
syncs the changes using only one brick among the replica pair at a time,
calling it as "ACTIVE" and other non syncing brick as "PASSIVE".

Let's consider below example of distributed replicated setup where
NODE-1 as b1 and its replicated brick b1r is in NODE-2

        NODE-1                         NODE-2
          b1                            b1r

At the beginning, geo-replication chooses to sync changes from NODE-1:b1
and NODE-2:b1r will be "PASSIVE". The logic depends on virtual getxattr
'trusted.glusterfs.node-uuid' which always returns first up subvolume
i.e., NODE-1. When NODE-1 goes down, the above xattr returns NODE-2 and
that is made 'ACTIVE'. But when NODE-1 comes back again, the above xattr
returns NODE-1 and it is made 'ACTIVE' again. So for a brief interval of
time, if NODE-2 had not finished processing the changelog, both NODE-2
and NODE-1 will be ACTIVE causing rename race as mentioned in the bug.

SOLUTION:
---------
1. Have a shared replicated storage, a glusterfs management volume specific
   to geo-replication.

2. Geo-rep creates a file per replica set on management volume.

3. fcntl lock on the above said file is used for synchronization
   between geo-rep workers belonging to same replica set.

4. If management volume is not configured, geo-replication will back
   to previous logic of using first up sub volume.

Each worker tries to lock the file on shared storage, who ever wins will
be ACTIVE. With this, we are able to solve the problem but there is an
issue when the shared replicated storage goes down (when all replicas
goes down). In that case, the lock state is lost. So AFR needs to rebuild the
lock state after brick comes up.

NOTE:
-----
This patch brings in the, pre-requisite step of setting up management volume
for geo-replication during creation.

1. Create mgmt-vol for geo-replicatoin and start it. Management volume should
   be part of master cluster and recommended to be three way replicated
   volume having each brick in different nodes for availability.
2. Create geo-rep session.
3. Configure mgmt-vol created with geo-replication session as follows.
   gluster vol geo-rep  slavenode:: config meta_volume \
   
4. Start geo-rep session.

Backward Compatiability:
-----------------------
If management volume is not configured, it falls back to previous logic of
using node-uuid virtual xattr. But it is not recommended.

Change-Id: I7319d2289516f534b69edd00c9d0db5a3725661a
BUG: 1196632
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/9759
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

geo-rep: Handle ENOENT during cleanup

2015-03-06T02:26:57+00:00

shutil.rmtree was failing to remove file if file was not
exists. Added error handling function to ignore ENOENT if
a file/dir not present.

BUG: 1198101
Change-Id: I1796db2642f81d9e2b5e52c6be34b4ad6f1c9786
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/9792
Tested-by: Gluster Build System 
Reviewed-by: Prashanth Pai 
Reviewed-by: Venky Shankar 
Reviewed-by: Kotresh HR 
Reviewed-by: Saravanakumar Arumugam

geo-rep: Fixing the typo errors

2015-02-03T18:03:41+00:00

Change-Id: Iacc67e4ba9ac45e0858f3befe84ffb8fccf7e1c3
BUG: 1075417
Signed-off-by: arao 
Reviewed-on: http://review.gluster.org/9502
Tested-by: Gluster Build System 
Reviewed-by: Humble Devassy Chirammal