glusterfs.git/geo-replication, branch v3.12.0

geo-rep: Fix syncing of hardlink of symlink

2017-08-29T12:42:31+00:00

Problem:
If there is a hardlink to a symlink on master
and if the symlink file is deleted on master,
geo-rep fails to sync the hardlink.

Typical Usecase:
It's easily hit with rsnapshot use case where
it uses hardlinks.

Example Reproducer:
Setup geo-replication between master and slave
volume and in master mount point, do the following.

 1. mkdir /tmp/symlinkbug
 2. ln -f -s /does/not/exist /tmp/symlinkbug/a_symlink
 3. rsync -a /tmp/symlinkbug ./
 4. cp -al symlinkbug symlinkbug.0
 5. ln -f -s /does/not/exist2 /tmp/symlinkbug/a_symlink
 6. rsync -a /tmp/symlinkbug ./
 7. cp -al symlinkbug symlinkbug.1

Cause:
If the source was not present while syncing hardlink,
it was always packing the blob as regular file.

Fix:
If the source was not present while syncing hardlink,
pack the blob based on the mode.

> Change-Id: Iaa12d6f99de47b18e0650e7c4eb455f23f8390f2
> BUG: 1432046
> Signed-off-by: Kotresh HR 
> Reported-by: Christian Lohmaier 
> Reviewed-on: https://review.gluster.org/18011
> Smoke: Gluster Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Aravinda VK 
(cherry picked from commit e893962deaabab8e934813f8a0443a8f94e009f2)

Change-Id: Iaa12d6f99de47b18e0650e7c4eb455f23f8390f2
BUG: 1486120
Signed-off-by: Kotresh HR 
Reviewed-on: https://review.gluster.org/18127
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

geo-rep: Convert gfid mismatch logs to structured logging

2017-08-03T14:11:51+00:00

Convert the logs related to entry failures fix
due to gfid mismatch logs into structured logging
format

Change-Id: I9bce950c5339b48d3ec8b84bddee38b0473b7634
> Updates: #246
> Signed-off-by: Kotresh HR 
> Reviewed-on: https://review.gluster.org/17896
> Smoke: Gluster Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Aravinda VK 
(cherry picked from commit de69fb7ac819c7e92d132a5ead12ea317ef1ec20)

Change-Id: I9bce950c5339b48d3ec8b84bddee38b0473b7634
Updates: #246
Signed-off-by: Kotresh HR 
Reviewed-on: https://review.gluster.org/17920
Reviewed-by: Aravinda VK 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System

geo-rep: Fix syncing of self healed hardlinks

2017-07-31T16:29:32+00:00

Problem:
In a distribute replicate volume, if the hardlinks
are created when a subvolume is down, it gets
healed from other subvolume when it comes up.
If this subvolume becomes ACTIVE in geo-rep
there are chances that those hardlinks won't
be synced to slave.

Cause:
AFR can't detect hardlinks during self heal.
It just create those files using mknod and
the same is recorded in changelog. Geo-rep
processes these mknod and ignores it as
it finds gfid already on slave.

Solution:
Geo-rep should process the mknod as link
if the gfid already exists on slave.

> Change-Id: I2f721b462b38a74c60e1df261662db4b99b32057
> BUG: 1475308
> Signed-off-by: Kotresh HR 
> Reviewed-on: https://review.gluster.org/17880
> Smoke: Gluster Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Aravinda VK 
(cherry picked from commit d685e4238fafba8f58bf01174c79cb5ca35203e5)

Change-Id: I2f721b462b38a74c60e1df261662db4b99b32057
Signed-off-by: Kotresh HR 
BUG: 1476208
Reviewed-on: https://review.gluster.org/17905
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Aravinda VK

geo-rep: Handle possible entry failures gracefully

2017-07-21T10:38:12+00:00

Updates: #246

Change-Id: If0ce83fe8dd3068bfb671f398b2e82ac831288d0
Signed-off-by: Kotresh HR 
Reviewed-on: https://review.gluster.org/17577
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Aravinda VK

geo-rep: Fix changelog encoding to encode only space and newline

2017-07-21T04:44:30+00:00

libgfchangelog was encoding path using spec rfc3986, but encoding only
required for SPACE and NEWLINE chars since the NEWLINE char is used as
record separator and SPACE as field separator in the parsed changelogs
output.

Changed the encoding function to encode only SPACE and NEWLINE.

BUG: 1451724
Change-Id: I1936efad31788a9e636f912c832ed7d7efea4fe2
Signed-off-by: Aravinda VK 
Reviewed-on: https://review.gluster.org/17787
Reviewed-by: Prashanth Pai 
Reviewed-by: Kotresh HR 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System

geo-rep: Fix entry failure because parent dir doesn't exist

2017-07-05T15:44:23+00:00

In a distributed volume on master, it can so happen that
the RMDIR followed by MKDIR is recorded in changelog on
a particular subvolume with same gfid and pargfid/bname
but not on all subvolumes as below.

E 61c67a2e-07f2-45a9-95cf-d8f16a5e9c36 RMDIR \
9cc51be8-91c3-4ef4-8ae3-17596fcfed40%2Ffedora2
E 61c67a2e-07f2-45a9-95cf-d8f16a5e9c36 MKDIR 16877 0 0 \
9cc51be8-91c3-4ef4-8ae3-17596fcfed40%2Ffedora2

While processing this changelog, geo-rep thinks RMDIR is
successful and does recursive rmdir on slave. But in the
master the directory still exists. This could lead to
data discrepancy between master and slave.

Cause:
RMDIR-MKDIR pair gets recorded so in changelog when the
directory removal is successful on cached subvolume and
failed in one of hashed subvol for some reason
(may be down). In this case, the directory is re-created
on cached subvol which gets recorded as MKDIR again in
changelog.

Solution:
So while processing RMDIR geo-replication should stat on
master with gfid and should not delete it if it's present.

Change-Id: If5da1d6462eb4d9ebe2e88b3a70cc454411a133e
BUG: 1467718
Signed-off-by: Kotresh HR 
Reviewed-on: https://review.gluster.org/17695
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Aravinda VK

geo-rep: Structured log support

2017-06-20T06:00:47+00:00

Changed all log messages to structured log format

Change-Id: Idae25f8b4ad0bbae38f4362cbda7bbf51ce7607b
Updates: #240
Signed-off-by: Aravinda VK 
Reviewed-on: https://review.gluster.org/17551
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR

geo-rep: Added metrics related to Sync Time

2017-06-15T07:20:56+00:00

In Geo-rep, Sync jobs can be configured using, `config sync-jobs 3`. This
patch adds following information related to the sync job(Rsync/Tarssh)

Example output:

    [2017-06-13 09:09:32.532181] I [master(/bricks/b1):1713:syncjob] Syncer: \
        Sync Time Taken (Job:2 Files:5484 ReturnCode:0): 4.8774 secs

Change-Id: Ifceb96d4b8d14e00fd1290c0aeff60d64b4d7f37
BUG: 1455179
Signed-off-by: Aravinda VK 
Reviewed-on: https://review.gluster.org/17531
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Amar Tumballi 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR

geo-rep: Fix string format issue caused due to #17489

2017-06-13T06:09:05+00:00

With Patch #17489, values from Geo-rep config always represented
as Unicode string, which is not compatible with rest of the code.

Changed the format with this patch to fix the issue.

BUG: 1459620
Change-Id: I935fca0d24f02e90757f688f92ef73fad9f9b8e1
Signed-off-by: Aravinda VK 
Reviewed-on: https://review.gluster.org/17503
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Kotresh HR

geo-rep: Fix ConfigInterface Template issue

2017-06-08T11:16:20+00:00

ConfigParser uses string Template to substitute the dynamic values
for config. For some of the configurations, Geo-rep worker will
not restart. Due to this conf object may have non string values.

If val is not string in Template(val), then it fails with
"TypeError: expected string or buffer"

BUG: 1459620
Change-Id: I25b8bbc1df42f6f29e9563a55b3e27a228321c44
Signed-off-by: Aravinda VK 
Reviewed-on: https://review.gluster.org/17489
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR