glusterfs.git/geo-replication, branch v7dev

geo-rep: Fix configparser import issue

2019-02-05T04:19:59+00:00

'configparser' is backported to python2 and can
be installed using pip (pip install configparser).
So trying to import 'configparser' first and later
'ConfigParser' can cause issues w.r.t unicode strings.

Always try importing 'ConfigParser' first and then
'configparser'. This solves python2/python3 compat
issues.

Change-Id: I2a87c3fc46476296b8cb547338f35723518751cc
fixes: bz#1671637
Signed-off-by: Kotresh HR

geo-rep : fix rename sync on hybrid crawl

2019-01-17T10:13:52+00:00

Problem: When geo-rep is configured as hybrid crawl
         directory renames are not synced to the slave.

Solution: Rename sync of directory was failing due to incorrect
          destination path calculation.
          During check for existence on slave we miscalculated
          realpath. .

Change-Id: I23f1ea60e86a917598fe869d5d24f8da654d8a0a
fixes: bz#1665826
Signed-off-by: Sunny Kumar

geo-rep: Make slave volume read-only (by default)

2018-12-07T09:37:54+00:00

Added a command to set "features.read-only" option
to a default value "on" for slave volume.
Changes are made in:
$SRC//extras/hook-scripts/S56glusterd-geo-rep-create-post.sh
for root geo-rep and
$SRC/geo-replication/src/set_geo_rep_pem_keys.sh
for non-root geo-rep.

Fixes: bz#1654187

Change-Id: I15beeae3506f3f6b1dcba0a5c50b6344fd468c7c
Signed-off-by: Harpreet Kaur

libglusterfs: Move devel headers under glusterfs directory

2018-12-05T21:47:04+00:00

libglusterfs devel package headers are referenced in code using
include semantics for a program, this while it works can be better
especially when dealing with out of tree xlator builds or in
general out of tree devel package usage.

Towards this, the following changes are done,
- moved all devel headers under a glusterfs directory
- Included these headers using system header notation <> in all
code outside of libglusterfs
- Included these headers using own program notation "" within
libglusterfs

This change although big, is just moving around the headers and
making it correct when including these headers from other sources.

This helps us correctly include libglusterfs includes without
namespace conflicts.

Change-Id: Id2a98854e671a7ee5d73be44da5ba1a74252423b
Updates: bz#1193929
Signed-off-by: ShyamsundarR

geo-rep: Fix syncing of files with non-ascii filenames

2018-12-04T09:15:44+00:00

Problem:
Creation of files/directories with non-ascii names fails
to sync to the slave. It crashes with below traceback on
slave.
...
File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/repce.py", line 118, in worker
res = getattr(self.obj, rmeth)(*in_data[2:])
File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py", line 709, in entry_ops
[ESTALE, EINVAL, EBUSY])
File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/syncdutils.py", line 546, in errno_wrap
return call(*arg)
File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/libcxattr.py", line 83, in lsetxattr
cls.raise_oserr()
File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/libcxattr.py", line 38, in raise_oserr
raise OSError(errn, os.strerror(errn))
OSError: [Errno 12] Cannot allocate memory

Cause:
The length calculation arguments passed to blob creation was done before encoding. Hence
was failing in gfid-access layer.

Fix:
It appears that the calculating lenght properly fixes this issue. But it will cause
issues in other places in 'python2' and not in 'python3'. So encoding and decoding
each required string to make geo-rep compatible with both 'python2' and 'python3'
is a nightmare and is not fool proof. Hence kept 'python2' code as is with out
encode/decode and applied encode/decode only to 'python3'

Added non-ascii filename tests to regression

fixes: bz#1650893
Change-Id: I35cfaf848e07b1a0b5cb93c01b98b472f08271a6
Signed-off-by: Kotresh HR

geo-rep: Fix permissions with non-root setup

2018-11-26T04:20:57+00:00

Problem:
In non-root fail-over/fail-back(FO/FB), when slave is
promoted as master, the session goes to 'Faulty'

Cause:
The command 'gluster-mountbroker  '
is run as a pre-requisite on slave in non-root setup.
It modifies the permission and group of following required
directories and files recursively

  [1] /var/lib/glusterd/geo-replication
  [2] /var/log/glusterfs/geo-replication-slaves

In a normal setup, this is executed on slave node and hence
doing it recursively is not an issue on [1]. But when original
master becomes slave in non-root during FO/FB, it contains
ssh public keys and modifying permissions on them causes
geo-rep to fail with incorrect permissions.

Fix:
Don't do permission change recursively. Fix permissions for
required files.

fixes: bz#1651498
Change-Id: I68a744644842e3b00abc26c95c06f123aa78361d
Signed-off-by: Kotresh HR

geo-rep: Fix traceback with symlink metadata sync

2018-11-06T02:22:35+00:00

While syncing metadata, 'os.chmod', 'os.chown',
'os.utime' should be used without de-reference.
But python supports only 'os.chown' without
de-reference. That's mostly because Linux
doesn't support 'chmod' on symlink file itself
but it does support 'chown'.

So while syncing metadata ops, if it's symlink
we should only sync 'chown' and not do 'chmod'
and 'utime'. It will lead to tracebacks with
errors like EROFS, EPERM, ACCESS, ENOENT.
All the three errors (EPERM, ACCESS, ENOENT)
were handled except EROFS. But the way it was
handled was not fool proof. The operation is
tried and failure was handled based on the errors.
All the errors with symlink file for 'chown',
'utime' had to be passed to safe errors list of
'errno_wrap'. This patch handles it better by
avoiding 'chmod' and 'utime' if it's symlink
file.

fixes: bz#1646104
Change-Id: Ic354206455cdc7ab2a87d741d81f4efe1f19d77d
Signed-off-by: Kotresh HR

geo-rep: Add more intelligence to automatic error handling

2018-10-30T13:13:41+00:00

Geo-rep's automatic error handling does gfid conflict
resolution. But if there are ENOENT errors because the
parent is not synced to slave, it doesn' handle them.
This patch adds the intelligence to create missing
parent directories on slave. It can create the missing
directories upto the depth of 10.

fixes: bz#1643402
Change-Id: Ic97ed1fa5899c087e404d559e04f7963ed7bb54c
Signed-off-by: Kotresh HR

geo-rep/scripts: Fix traceback in gluster-mountbroker

2018-10-30T13:13:08+00:00

When 'gluster-mountbroker status' was issued, it
crashes in a corner case with 'str object has not
attribute get'. Fixed the same.

fixes: bz#1643929
Signed-off-by: Kotresh HR 
Change-Id: Iaf1a937ed0136b3b2058230c75fa89a215d8a5eb

georep: python2 to python3 compat - scheduler

2018-10-30T13:12:46+00:00

1. scheduler - Popen
2. syncdutils - corner case on failure

fixes: bz#1643932
Change-Id: I65af97a244a8790e976acedc2728db6ebbf2ae10
Signed-off-by: Kotresh HR