glusterfs.git/geo-replication/syncdaemon, branch v6.6

geo-rep: Fix Permission denied traceback on non root setup

2019-10-24T15:15:10+00:00

Problem:
While syncing rename of directory in hybrid crawl, geo-rep
crashes as below.

Traceback (most recent call last):
  File "/usr/local/libexec/glusterfs/python/syncdaemon/repce.py", line 118, in worker
    res = getattr(self.obj, rmeth)(*in_data[2:])
  File "/usr/local/libexec/glusterfs/python/syncdaemon/resource.py", line 588, in entry_ops
    src_entry = get_slv_dir_path(slv_host, slv_volume, gfid)
  File "/usr/local/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 687, in get_slv_dir_path
    [ENOENT], [ESTALE])
  File "/usr/local/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 546, in errno_wrap
    return call(*arg)
PermissionError: [Errno 13] Permission denied: '/bricks/brick1/b1/.glusterfs/8e/c0/8ec0fcd4-d50f-4a6e-b473-a7943ab66640'

Cause:
Conversion of gfid to path for a directory uses readlink on backend
.glusterfs gfid path. But this fails for non root user with
permission denied.

Fix:
Use gfid2path interface to get the path from gfid

Backport of:
 > Patch: https://review.gluster.org/23570
 > Change-Id: I9d40c713a1b32cea95144cbc0f384ada82972222
 > BUG: 1763439
 > Signed-off-by: Kotresh HR 

Change-Id: I9d40c713a1b32cea95144cbc0f384ada82972222
fixes: bz#1764183
Signed-off-by: Kotresh HR

geo-rep: Fix config upgrade on non-participating node

2019-10-22T12:50:28+00:00

After upgrade, if the config files are of old format, it
gets migrated to new format. Monitor process migrates it.
Since monitor doesn't run on nodes where bricks are not
hosted, it doesn't get migrated there. So this patch fixes
the config upgrade on nodes which doesn't host bricks.
This happens during config either on get/set/reset.

Backport of:
 > Patch: https://review.gluster.org/23555
 > Change-Id: Ibade2f2310b0f3affea21a3baa1ae0eb71162cba
 > Signed-off-by: Kotresh HR 
 > BUG: 1762220

Change-Id: Ibade2f2310b0f3affea21a3baa1ae0eb71162cba
fixes: bz#1763028
Signed-off-by: Kotresh HR

geo-rep: Fix the name of changelog archive file

2019-10-22T12:50:27+00:00

Background:
The processed changelogs are archived each month in a single tar file.
The default format is "archive_YYYYMM.tar" which is specified as "%%Y%%m"
in configuration file.

Problem:
The created changelog archive file didn't have corresponding year
and month. It created as "archive_%Y%m.tar" on python2 only systems.

Cause and Fix:
Geo-rep expects "%Y%m" after the ConfigParser reads it from config file.
Since it was "%%Y%%m" in config file, geo-rep used to get correct value
"%Y%m" in python3 and "%%Y%%m" in python2 which is incorrect.
The fix can be to use "%Y%m" in config file but that fails in python3.
So the fix is to use "RawConfigParser" in geo-rep and use "%Y%m". This
works both in python2 and python3.

Backport of:
 > Patch: https://review.gluster.org/23248
 > Change-Id: Ie5b7d2bc04d0d53cd1769e064c2d67aaf95d557c
 > BUG: 1741890
 > Signed-off-by: Kotresh HR 

Change-Id: Ie5b7d2bc04d0d53cd1769e064c2d67aaf95d557c
fixes: bz#1764176
Signed-off-by: Kotresh HR

geo-rep: Fix Config Get Race

2019-10-22T12:50:27+00:00

When two threads(sync jobs) in Geo-rep worker calls `gconf.get` and
`gconf.getr`(realtime) at the sametime, `getr` resets the conf object
and other one gets None. Thread Lock is introduced to fix the issue.

```
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py",
  line 368, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1987,
  in syncjob
    po = self.sync_engine(pb, self.log_err)
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py",
  line 1444, in rsync
    rconf.ssh_ctl_args + \
AttributeError: 'NoneType' object has no attribute 'split'
```

Backport of:
 > Patch: https://review.gluster.org/23158
 > Change-Id: I9c245e5c36338265354e158f5baa32b119eb2da5
 > BUG: 1737484
 > Signed-off-by: Aravinda VK 

Change-Id: I9c245e5c36338265354e158f5baa32b119eb2da5
fixes: bz#1764174
Signed-off-by: Kotresh HR

geo-rep: Test case for upgrading config file

2019-10-22T12:50:27+00:00

Added test case for the patch
https://review.gluster.org/#/c/glusterfs/+/22894/4

Also updated if else structure in gsyncdconfig.py to avoid
repeated occurance of values in new configfile.

Backport of:
 > Patch: https://review.gluster.org/22982
 > BUG: 1707731
 > Change-Id: If97e1d37ac52dbd17d47be6cb659fc5a3ccab6d7
 > Signed-off-by: Shwetha K Acharya 

fixes: bz#1764171
Change-Id: If97e1d37ac52dbd17d47be6cb659fc5a3ccab6d7
Signed-off-by: Kotresh HR

geo-rep : fix gluster command path for non-root session

2019-10-22T12:50:27+00:00

Problem:
gluster command not found.

Cause:
In Volinfo class we issue command 'gluster vol info' to get information
about volume like getting brick_root to perform various operation.
When geo-rep session is configured for non-root user Volinfo class
fails to issue gluster command due to unavailability of gluster
binary path for non-root user.

Solution:
Use config value 'slave-gluster-command-dir'/'gluster-command-dir' to get path
for gluster command based on caller.

Backport of:
 > Patch: https://review.gluster.org/22920
 > BUG: 1722740
 > Change-Id: I4ec46373da01f5d00ecd160c4e8c6239da8b3859
 > Signed-off-by: Sunny Kumar 

fixes: bz#1764172
Change-Id: I4ec46373da01f5d00ecd160c4e8c6239da8b3859
Signed-off-by: Kotresh HR

geo-rep: Upgrading config file to new version

2019-10-22T12:50:27+00:00

- configuration handling is enhanced with patch
https://review.gluster.org/#/c/glusterfs/+/18257/
- hence, the old configurations are not applied when
Geo-rep session is created in the old version and upgraded.

This patch solves the issue. It,
- checks if the config file is old.
- parses required values from old config file and stores in new
  config file, which ensures that configerations are applied on
  upgrade.
- stores old config file as backup.
- handles changes in options introduced in
  https://review.gluster.org/#/c/glusterfs/+/18257/

Backport of:
 > Patch: https://review.gluster.org/22894
 > BUG: bz#1707731
 > Change-Id: Iad8da6c1e1ae8ecf7c84dfdf8ea3ac6966d8a2a0
 > Signed-off-by: Shwetha K Acharya 

updates: bz#1764171
Change-Id: Iad8da6c1e1ae8ecf7c84dfdf8ea3ac6966d8a2a0
Signed-off-by: Kotresh HR

geo-rep: Convert gfid conflict resolutiong logs into debug

2019-05-21T05:14:53+00:00

The gfid conflict resolution code path is not supposed
to hit in generic code path. But few of the heavy rename
workload (BUG: 1694820) makes it a generic case. So
logging the entries to be fixed as INFO floods the log
in these particular workloads. Hence convert them to DEBUG.

Backport of:
 > Patch: https://review.gluster.org/22720
 > BUG: 1709653
 > Change-Id: I4d5e102b87be5fe5b54f78f329e588882d72b9d9
 > Signed-off-by: Kotresh HR 

fixes: bz#1712223
Change-Id: I4d5e102b87be5fe5b54f78f329e588882d72b9d9
Signed-off-by: Kotresh HR

geo-rep: Fix sync hang with tarssh

2019-05-21T05:14:53+00:00

Problem:
Geo-rep sync hangs when tarssh is used as sync
engine at heavy workload.

Analysis and Root cause:
It's found out that the tar process was hung.
When debugged further, it's found out that stderr
buffer of tar process on master was full i.e., 64k.
When the buffer was copied to a file from /proc/pid/fd/2,
the hang is resolved.

This can happen when files picked by tar process
to sync doesn't exist on master anymore. If this count
increases around 1k, the stderr buffer is filled up.

Fix:
The tar process is executed using Popen with stderr as PIPE.
The final execution is something like below.

tar | ssh  root@slave tar --overwrite -xf - -C 

It was waiting on ssh process first using communicate() and then tar.
Note that communicate() reads stdout and stderr. So when stderr of tar
process is filled up, there is no one to read until untar via ssh is
completed. This can't happen and leads to deadlock.
Hence we should be waiting on both process parallely, so that stderr is
read on both processes.

Backport of:
 > Patch: https://review.gluster.org/22684/
 > Change-Id: I609c7cc5c07e210c504771115b4d551a2e891adf
 > BUG: 1707728
 > Signed-off-by: Kotresh HR 

Change-Id: I609c7cc5c07e210c504771115b4d551a2e891adf
fixes: bz#1709738
Signed-off-by: Kotresh HR

geo-rep: Fix sync-method config

2019-05-17T07:47:53+00:00

Problem:
When 'use_tarssh' is set to true, it exits with successful
message but the default 'rsync' was used as sync-engine.
The new config 'sync-method' is not allowed to set from cli.

Analysis and Fix:
The 'use_tarssh' config is deprecated with new
config framework and 'sync-method' is the new
config to choose sync-method i.e. tarssh or rsync.
This patch fixes the 'sync-method' config. The allowed
values are tarssh and rsync.

Backport of:
 > Patch: https://review.gluster.org/22683
 > Change-Id: I0edb0319cad0455b29e49f2f08a64ce324735e84
 > BUG: 1707686
 > Signed-off-by: Kotresh HR 

Change-Id: I0edb0319cad0455b29e49f2f08a64ce324735e84
fixes: bz#1709737
Signed-off-by: Kotresh HR