glusterfs.git/geo-replication/syncdaemon, branch v3.9rc1

geo-rep: Fix Geo-rep status if monitor.pid file not exists

2016-09-15T06:48:01+00:00

If monitor.pid file not exists, gsyncd fails with following traceback

Traceback (most recent call last):
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py",
  line 201, in main
    main_i()
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py",
  line 681, in main_i
    brick_status.print_status(checkpoint_time=checkpoint_time)
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncdstatus.py",
  line 343, in print_status
    for key, value in self.get_status(checkpoint_time).items():
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncdstatus.py",
  line 262, in get_status
    with open(self.monitor_pid_file, "r+") as f:
IOError: [Errno 2] No such file or directory: '/var/lib/glusterd/
  geo-replication/master_node_slave/monitor.pid'

If Georep status command this worker's status will not be displayed
since not returning expected status output.

> Reviewed-on: http://review.gluster.org/15416
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Kotresh HR 

BUG: 1374630
Change-Id: I600a2f5d9617f993d635b9bc6e393108500db5f9
Signed-off-by: Aravinda VK 
(cherry picked from commit c7118a92f52a2fa33ab69f3e3ef1bdabfee847cf)
Reviewed-on: http://review.gluster.org/15446
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Saravanakumar Arumugam 
Smoke: Gluster Build System

geo-rep: Defunct tar process after sync

2016-09-15T06:30:42+00:00

After every sync iteration with tarssh mode leaves defunct tar
process.

Added wait for tar process to prevent this issue.

> Reviewed-on: http://review.gluster.org/15426
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> Reviewed-by: Saravanakumar Arumugam 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Kotresh HR 

BUG: 1375543
Change-Id: I9953239ef601cc1970c814b00074b45eb00f481e
Signed-off-by: Aravinda VK 
(cherry picked from commit 6b30e9bf5a612e105eb7ded0a89ef25fd8530ba5)
Reviewed-on: http://review.gluster.org/15488
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
Reviewed-by: Saravanakumar Arumugam 
CentOS-regression: Gluster Build System

geo-rep: Fix ESTALE/EINVAL issue during set_{xtime,stime}

2016-09-15T06:30:02+00:00

Setfattr may get ESTALE/EINVAL if a file is being unlinked.
To prevent worker crashing, added retry for these error messages.

On second retry it will get ENOENT and that error is handled by
ignoring.

> Reviewed-on: http://review.gluster.org/15404
> Reviewed-by: Saravanakumar Arumugam 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Kotresh HR 
> Smoke: Gluster Build System 

BUG: 1374626
Change-Id: Ic660fa13208366d57c8d3d492bbef611475e45b7
Signed-off-by: Aravinda VK 
(cherry picked from commit 9db2826cd8dbcc0fb58f5434c21d00ca5c15c491)
Reviewed-on: http://review.gluster.org/15445
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Saravanakumar Arumugam

geo-rep: Use configured log_level for libgfchangelog logs

2016-09-15T06:29:47+00:00

libgfchangelog was not respecting the log_level configured
in Geo-replication. With this patch Libgfchangelog log level
can be configured using `config changelog_log_level TRACE`.
Default Changelog log level is INFO

> Reviewed-on: http://review.gluster.org/15078
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Kotresh HR 

BUG: 1374608
Change-Id: Ida714931129f6a1331b9d0815da77efcb2b898e3
Signed-off-by: Aravinda VK 
(cherry picked from commit f41ec4fcfaa9ca976fddbe9e91aabf840c20111f)
Reviewed-on: http://review.gluster.org/15444
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Saravanakumar Arumugam

geo-rep: Fix logging sync failures

2016-09-15T06:28:37+00:00

If Rsync/Tar subprocess dies, while logging error Geo-rep fails
with EBADF while accessing error file. Also worker dies while
accessing elines before it is set.

> Reviewed-on: http://review.gluster.org/15379
> NetBSD-regression: NetBSD Build System 
> Smoke: Gluster Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Kotresh HR 

BUG: 1374597
Change-Id: I9cfce116e8aafa4a98654f5190d40a455af8ec95
Signed-off-by: Aravinda VK 
(cherry picked from commit c0f877c0374d97e0bee17aac4850d7655a35e61b)
Reviewed-on: http://review.gluster.org/15441
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Saravanakumar Arumugam

geo-rep: Handle EISDIR error during Unlink

2016-09-15T06:28:05+00:00

During Rename, If Source and Target has same inode then
Geo-rep unlinks source. But if source is a directory then
this will fail with below traceback

Traceback (most recent call last):
  File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 113, in worker
    res = getattr(self.obj, rmeth)(*in_data[2:])
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 772,
    in entry_ops
    os.unlink(entry)
OSError: [Errno 21] Is a directory: '.gfid/12711ebf-7fdc-4f4b-9850-2d75581eb
   452/New folder'

With this patch, if EISDIR, rmdir is tried. Logs error in Slave log in case
of ENOTEMPTY.

> Reviewed-on: http://review.gluster.org/15132
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Smoke: Gluster Build System 
> Reviewed-by: Kotresh HR 

BUG: 1374581
Change-Id: I099af4192adac5125c0a23988ceb6506f91e987f
Signed-off-by: Aravinda VK 
 (cherry picked from commit 8a6236e4cfc7bf86e881e5c770a19040a6060ad0)
Change-Id: Ic3fcf3fa3c049c0d820db86b93e04f6e239deb45
Reviewed-on: http://review.gluster.org/15435
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Saravanakumar Arumugam

geo-rep: Fix History post process

2016-09-08T16:14:08+00:00

This patch removes changelogsdb part of post processing since
not got much performance advantage as expected.

Entry stime and other logging improvements retained.

> Reviewed-on: http://review.gluster.org/15371
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Kotresh HR 
> Smoke: Gluster Build System 

BUG: 1374153
Change-Id: Ib99d23f09d96c14bc28225b47d9134260f5551bf
Signed-off-by: Aravinda VK 

(cherry picked from commit 5de500cd0116796ff797099c60d33258bd48ce3c)

Change-Id: If4ac94de557dd79a1e7c43e01ea8a1d6566569f4
Reviewed-on: http://review.gluster.org/15424
Tested-by: Aravinda VK 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR

geo-rep: add geo-rep events for server side changes

2016-08-31T14:14:08+00:00

Event Type defined in #15351 to avoid merge conflicts

Add geo-rep events applicable to changes in
geo-rep session in the server side.

Change-Id: Ia66574d2abccad7fce6a96667efbc7c6c8903fc6
BUG: 1370445
Signed-off-by: Saravanakumar Arumugam 
Reviewed-on: http://review.gluster.org/15328
Tested-by: Aravinda VK 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Aravinda VK

geo-rep: Post process Data and Meta Changelogs

2016-08-26T17:45:58+00:00

With this patch, Data and Meta GFIDs are post processed. If Changelog has
UNLINK entry then remove from Data and Meta GFIDs list(If stat on GFID is
ENOENT in Master).

While processing Changelogs,

- Collect all the data and meta operations in a temporary database
- Delete all Data and Meta GFIDs which are already unlinked as per Changelogs
  (unlink only if stat on GFID is ENOENT)
- Process all Entry operations as usual
- Process data and meta operations in batch(Fetch from Db in batch)
- Data sync is again batched based on number of changelogs(Default 1day
  changelogs). Once the sync is complete, Update last Changelog's time as last_synced
  time as usual.

Additionally maintain entry_stime on Brick root, ignore Entry ops if changelog
suffix time is less than entry_stime. If data stime is more than entry_stime,
this can happen only when passive worker updates stime by itself by getting
mount point stime. Use entry_stime = data_stime in this case.

New configurations:

max-rsync-retries - Default Value is 10
max-data-changelogs-in-batch - Max number of changelogs to be considered in a
batch for syncing. Default value is 5760(4 changelogs per min * 60 min *
24 hours)
max-history-changelogs-in-batch - Max number of history changelogs to be
processed at once. Default value 86400(4 changelogs per min * 60 min * 24
hours * 15 days)

BUG: 1364420
Change-Id: I7b665895bf4806035c2a8573d361257cbadbea17
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15110
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Kotresh HR 
CentOS-regression: Gluster Build System

features/libgfchangelog: Log failure in gf_histroy_changelog

2016-08-09T07:27:12+00:00

Add error logs if gf_history_changelog fails. If requested
changelog range is not available, log the error and exit
instead of continuing the loop and exiting in readdir
without logging. Also fixed the duplicate MSGID number in
'changelog-lib-messages.h'

Change-Id: Icd71b89ae23b48a71380657ba5649029c32fabfd
BUG: 1362151
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/15064
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Aravinda VK