glusterfs.git/geo-replication, branch v5.8

eventsapi: Fix Python3 compatibility issues

2019-02-26T05:11:49+00:00

- Fixed Relative import and non-package import related issues.
- socketserver import issues fix
- Renamed installed directory name to `gfevents` from `events`(To
  avoid any issues with other global libs)

Fixes: bz#1649054
Change-Id: I3dc38bc92b23387a6dfbcc0ab8283178235bf756
Signed-off-by: Aravinda VK 
(cherry picked from commit cd68f7b88b9a2c9a4e4ff9fca61517384e54130a)

geo-rep: Fix syncing of files with non-ascii filenames

2018-12-26T16:50:01+00:00

Problem:
  Creation of files/directories with non-ascii names fails
  to sync to the slave. It crashes with below traceback on
  slave.
  ...
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/repce.py", line 118, in worker
    res = getattr(self.obj, rmeth)(*in_data[2:])
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py", line 709, in entry_ops
    [ESTALE, EINVAL, EBUSY])
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/syncdutils.py", line 546, in errno_wrap
    return call(*arg)
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/libcxattr.py", line 83, in lsetxattr
    cls.raise_oserr()
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/libcxattr.py", line 38, in raise_oserr
    raise OSError(errn, os.strerror(errn))
  OSError: [Errno 12] Cannot allocate memory

Cause:
  The length calculation arguments passed to blob creation was done before encoding. Hence
  was failing in gfid-access layer.

Fix:
  It appears that the calculating lenght properly fixes this issue. But it will cause
  issues in other places in 'python2' and not in 'python3'. So encoding and decoding
  each required string to make geo-rep compatible with both 'python2' and 'python3'
  is a nightmare and is not fool proof. Hence kept 'python2' code as is with out
  encode/decode and applied encode/decode only to 'python3'

Added non-ascii filename tests to regression

Backport of:
 > Patch: https://review.gluster.org/21668
 > BUG: 1650893
 > Change-Id: I35cfaf848e07b1a0b5cb93c01b98b472f08271a6
 > Signed-off-by: Kotresh HR 

fixes: bz#1648642
Change-Id: I35cfaf848e07b1a0b5cb93c01b98b472f08271a6
Signed-off-by: Kotresh HR

geo-rep: Fix traceback with symlink metadata sync

2018-12-12T12:54:37+00:00

While syncing metadata, 'os.chmod', 'os.chown',
'os.utime' should be used without de-reference.
But python supports only 'os.chown' without
de-reference. That's mostly because Linux
doesn't support 'chmod' on symlink file itself
but it does support 'chown'.

So while syncing metadata ops, if it's symlink
we should only sync 'chown' and not do 'chmod'
and 'utime'. It will lead to tracebacks with
errors like EROFS, EPERM, ACCESS, ENOENT.
All the three errors (EPERM, ACCESS, ENOENT)
were handled except EROFS. But the way it was
handled was not fool proof. The operation is
tried and failure was handled based on the errors.
All the errors with symlink file for 'chown',
'utime' had to be passed to safe errors list of
'errno_wrap'. This patch handles it better by
avoiding 'chmod' and 'utime' if it's symlink
file.
Backport of :
< Patch: https://review.gluster.org/21546/
> BUG: bz#1646104
> Change-Id: Ic354206455cdc7ab2a87d741d81f4efe1f19d77d
> Signed-off-by: Kotresh HR 
(cherry picked from commit 3c6cf9a4a1b46cab2dc53c1ee0afca0fe993102e)


fixes: bz#1654115
Change-Id: Ic354206455cdc7ab2a87d741d81f4efe1f19d77d
Signed-off-by: Kotresh HR

geo-rep: Fix permissions with non-root setup

2018-11-29T15:35:41+00:00

Problem:
In non-root fail-over/fail-back(FO/FB), when slave is
promoted as master, the session goes to 'Faulty'

Cause:
The command 'gluster-mountbroker  '
is run as a pre-requisite on slave in non-root setup.
It modifies the permission and group of following required
directories and files recursively

  [1] /var/lib/glusterd/geo-replication
  [2] /var/log/glusterfs/geo-replication-slaves

In a normal setup, this is executed on slave node and hence
doing it recursively is not an issue on [1]. But when original
master becomes slave in non-root during FO/FB, it contains
ssh public keys and modifying permissions on them causes
geo-rep to fail with incorrect permissions.

Fix:
Don't do permission change recursively. Fix permissions for
required files.

Backport of:
 > Patch: https://review.gluster.org/#/c/glusterfs/+/21689/
 > BUG: bz#1651498
 > Change-Id: I68a744644842e3b00abc26c95c06f123aa78361d
 > Signed-off-by: Kotresh HR 
(cherry picked from commit b2776b1ec1ad845ba568c4439bca3b57cc4d2592)


fixes: bz#1654117
Change-Id: I68a744644842e3b00abc26c95c06f123aa78361d
Signed-off-by: Kotresh HR

geo-rep: Add more intelligence to automatic error handling

2018-11-09T18:46:54+00:00

Geo-rep's automatic error handling does gfid conflict
resolution. But if there are ENOENT errors because the
parent is not synced to slave, it doesn' handle them.
This patch adds the intelligence to create missing
parent directories on slave. It can create the missing
directories upto the depth of 10.

Backport of:

 > Patch: https://review.gluster.org/21498/
 > fixes: bz#1643402
 > Change-Id: Ic97ed1fa5899c087e404d559e04f7963ed7bb54c
 > Signed-off-by: Kotresh HR 
(cherry picked from commit 19775e0445411cca9ddd9d294fd54d0b6fbe6a03)

fixes: bz#1646896
Change-Id: Ic97ed1fa5899c087e404d559e04f7963ed7bb54c
Signed-off-by: Kotresh HR

geo-rep/scripts: Fix traceback in gluster-mountbroker

2018-11-08T14:36:32+00:00

When 'gluster-mountbroker status' was issued, it
crashes in a corner case with 'str object has not
attribute get'. Fixed the same.

Backport of:
> BUG: 1643929
> Signed-off-by: Kotresh HR 
> Change-Id: Iaf1a937ed0136b3b2058230c75fa89a215d8a5eb
(cherry picked from commit 5987b3388126a3c5e77481913cbaa4142117d19a)

fixes: bz#1644515
Signed-off-by: Kotresh HR 
Change-Id: Iaf1a937ed0136b3b2058230c75fa89a215d8a5eb

georep: python2 to python3 compat - scheduler

2018-11-08T14:35:30+00:00

1. scheduler - Popen
2. syncdutils - corner case on failure

Backport of:
> Patch: https://review.gluster.org/21505
> BUG: 1643932
> Change-Id: I65af97a244a8790e976acedc2728db6ebbf2ae10
> Signed-off-by: Kotresh HR 
(cherry picked from commit 33e96100e17e9a293db6d63d9d5449d6c2d69376)

fixes: bz#1644514
Change-Id: I65af97a244a8790e976acedc2728db6ebbf2ae10
Signed-off-by: Kotresh HR

geo-rep: Fix issue in gfid-conflict-resolution

2018-11-08T14:33:17+00:00

Problem:
During gfid-conflict-resolution, geo-rep crashes
with 'ValueError: list.remove(x): x not in list'

Cause and Analysis:
During gfid-conflict-resolution, the entry blob is
passed back to master along with additional
information to verify it's integrity. If everything
looks fine, the entry creation is ignored and is
deleted from the original list.  But it is crashing
during removal of entry from the list saying entry
not in list. The reason is that the stat information
in the entry blob was modified and sent back to
master if present.

Fix:
Send back the correct stat information for
gfid-conflict-resolution.


Backport of:
> BUG: bz#1642865
> Change-Id: I47a6aa60b2a495465aa9314eebcb4085f0b1c4fd
> Signed-off-by: Kotresh HR 
(cherry picked from commit ff18121945bff394f3234e9f1a9d61ac97d4d493)

fixes: bz#1644158
Change-Id: I47a6aa60b2a495465aa9314eebcb4085f0b1c4fd
Signed-off-by: Kotresh HR

georep: python2 to python3 compat - syscalls

2018-10-10T14:09:10+00:00

1. ctypes/syscalls
    A) arguments is expected to be encoded
    B) Raw conversion of return value from bytearray into string
2. struct pack/unpack - Raw converstion of string to bytearray
3. basestring -> str

Updates: #411
Change-Id: I80f939adcdec0ed0022c87c0b76d057ad5559e5a
Signed-off-by: Kotresh HR 
(cherry picked from commit fb6e8d0d0ca21b16d331fa69da9b9dadf6c5c35d)

georep: Fix python3 compatibility (configparser)

2018-10-05T14:23:00+00:00

'%' needs special handling in config and also removed
duplicate misspelled 'changelog-archive-format' config.

Updates: #411
Change-Id: I33621a62bdf5f781ee62e6cedec0c2df3f5d70cf
Signed-off-by: Kotresh HR 
(cherry picked from commit 62164ac0d242a271f19b0e0bb352af5f7df082ce)