glusterfs.git/geo-replication/syncdaemon/syncdutils.py, branch v3.10.0rc0

geo-rep: Use Host UUID to find local Gluster node

2016-12-14T06:38:14+00:00

To spawn workers for each local brick, Geo-rep was collecting all
the machine IPs based on hostname and finds based on the connectivity.

With this patch, Geo-rep finds local brick if host UUID matches with
UUID of the brick from Volume info.

BUG: 1401801
Change-Id: Ic83c65df89e43cb86346e3ede227aa84d17ffd79
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/16035
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR

geo-rep/eventsapi: Additional Events

2016-10-18T10:56:03+00:00

Added following events
EVENT_GEOREP_ACTIVE
    {
        "nodeid": NODEID,
        "ts": TIMESTAMP,
        "event": "GEOREP_ACTIVE",
        "message": {
            "master_volume": MASTER_VOLUME_NAME,
            "slave_host": SLAVE_HOST,
            "slave_volume": SLAVE_VOLUME,
            "brick_path": BRICK_PATH
        }
    }

EVENT_GEOREP_PASSIVE
    {
        "nodeid": NODEID,
        "ts": TIMESTAMP,
        "event": "GEOREP_PASSIVE",
        "message": {
            "master_volume": MASTER_VOLUME_NAME,
            "slave_host": SLAVE_HOST,
            "slave_volume": SLAVE_VOLUME,
            "brick_path": BRICK_PATH
        }
    }
EVENT_GEOREP_CHECKPOINT_COMPLETED
    {
        "nodeid": NODEID,
        "ts": TIMESTAMP,
        "event": "GEOREP_ACTIVE",
        "message": {
            "master_volume": MASTER_VOLUME_NAME,
            "slave_host": SLAVE_HOST,
            "slave_volume": SLAVE_VOLUME,
            "brick_path": BRICK_PATH,
            "checkpoint_time": CHECKPOINT_TIME,
            "checkpoint_completion_time": CHECKPOINT_COMPLETION_TIME
        }
    }

BUG: 1379330
Change-Id: I90716175868c59dd65c8d202e73e0ede90347b6a
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15630
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR 
Tested-by: Kotresh HR

eventsapi/geo-rep: Geo-rep will not work without eventsapi rpms

2016-09-29T06:22:28+00:00

If glusterfs-events rpm is not installed, Geo-replication will
fail since it imports eventtypes.

Any call to gsyncd will fail with Import error. Glusterd start
fails since it runs `gsyncd.py --version`

Traceback (most recent call last):
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py",
    line 29, in 
      from syncdutils import FreeObject, norm, grabpidfile, finalize
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py",
    line 28, in 
      from events import eventtypes
ImportError: No module named events

BUG: 1378057
Change-Id: I1a9bc086c3d52449ec7296cb2f9ceb16cd41a8a4
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15539
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Saravanakumar Arumugam 
Reviewed-by: Kotresh HR

geo-rep: Use configured log_level for libgfchangelog logs

2016-09-09T07:23:30+00:00

libgfchangelog was not respecting the log_level configured
in Geo-replication. With this patch Libgfchangelog log level
can be configured using `config changelog_log_level TRACE`.
Default Changelog log level is INFO

BUG: 1363965
Change-Id: Ida714931129f6a1331b9d0815da77efcb2b898e3
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15078
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR

geo-rep: add geo-rep events for server side changes

2016-08-31T14:14:08+00:00

Event Type defined in #15351 to avoid merge conflicts

Add geo-rep events applicable to changes in
geo-rep session in the server side.

Change-Id: Ia66574d2abccad7fce6a96667efbc7c6c8903fc6
BUG: 1370445
Signed-off-by: Saravanakumar Arumugam 
Reviewed-on: http://review.gluster.org/15328
Tested-by: Aravinda VK 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Aravinda VK

geo-rep: Post process Data and Meta Changelogs

2016-08-26T17:45:58+00:00

With this patch, Data and Meta GFIDs are post processed. If Changelog has
UNLINK entry then remove from Data and Meta GFIDs list(If stat on GFID is
ENOENT in Master).

While processing Changelogs,

- Collect all the data and meta operations in a temporary database
- Delete all Data and Meta GFIDs which are already unlinked as per Changelogs
  (unlink only if stat on GFID is ENOENT)
- Process all Entry operations as usual
- Process data and meta operations in batch(Fetch from Db in batch)
- Data sync is again batched based on number of changelogs(Default 1day
  changelogs). Once the sync is complete, Update last Changelog's time as last_synced
  time as usual.

Additionally maintain entry_stime on Brick root, ignore Entry ops if changelog
suffix time is less than entry_stime. If data stime is more than entry_stime,
this can happen only when passive worker updates stime by itself by getting
mount point stime. Use entry_stime = data_stime in this case.

New configurations:

max-rsync-retries - Default Value is 10
max-data-changelogs-in-batch - Max number of changelogs to be considered in a
batch for syncing. Default value is 5760(4 changelogs per min * 60 min *
24 hours)
max-history-changelogs-in-batch - Max number of history changelogs to be
processed at once. Default value 86400(4 changelogs per min * 60 min * 24
hours * 15 days)

BUG: 1364420
Change-Id: I7b665895bf4806035c2a8573d361257cbadbea17
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15110
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Kotresh HR 
CentOS-regression: Gluster Build System

features/libgfchangelog: Log failure in gf_histroy_changelog

2016-08-09T07:27:12+00:00

Add error logs if gf_history_changelog fails. If requested
changelog range is not available, log the error and exit
instead of continuing the loop and exiting in readdir
without logging. Also fixed the duplicate MSGID number in
'changelog-lib-messages.h'

Change-Id: Icd71b89ae23b48a71380657ba5649029c32fabfd
BUG: 1362151
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/15064
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Aravinda VK

geo-rep: Error message cleanup

2016-06-15T17:28:54+00:00

If ssh returns 127 that means the remote gsyncd path is wrong
or push-pem failed during create. Existing error message was
pointing old documentation.

Change-Id: Ifbbb4a604fc0ae0fd5cb2746df6363bf28cde1e9
BUG: 1343943
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/14673
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Milind Changire 
Reviewed-by: Kotresh HR

geo-rep: Do not crash worker on ESTALE

2015-08-03T09:34:48+00:00

Handle ESTALE returned by lstat gracefully
by retrying it. Do not crash the worker.

Change-Id: I2527cd8bd1f7d2428cb4fa3f20782bebaf2df12a
BUG: 1247529
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/11772
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System 
Reviewed-by: Aravinda VK 
Reviewed-by: Venky Shankar

geo-rep: Minimize rm -rf race in Geo-rep

2015-05-05T13:45:01+00:00

While doing RMDIR worker gets ENOTEMPTY because same directory will
have files from other bricks which are not deleted since that worker
is slow processing. So geo-rep does recursive_delete.

Recursive delete was done using shutil.rmtree. once started, it will
not check disk_gfid in between. So it ends up deleting the new files
created by other workers. Also if other worker creates files after one
worker gets list of files to be deleted, then first worker will again
get ENOTEMPTY again.

To fix these races, retry is added when it gets ENOTEMPTY/ESTALE/ENODATA.
And disk_gfid check added for original path for which recursive_delete is
called. This disk gfid check executed before every Unlink/Rmdir. If disk
gfid is not matching with GFID from Changelog, that means other worker
deleted the directory. Even if the subdir/file present, it belongs to
different parent. Exit without performing further deletes.

Retry on ENOENT during create is ignored, since if CREATE/MKNOD/MKDIR
failed with ENOENT will not succeed unless parent directory is created
again.

Rsync errors handling was handling unlinked_gfids_list only for one
Changelog, but when processed in batch it fails to detect unlinked_gfids
and retries again. Finally skips the entire Changelogs in that batch.
Fixed this issue by moving self.unlinked_gfids reset logic before batch
start and after batch end.

Most of the Geo-rep races with rm -rf is eliminated with this patch,
but in some cases stale directories left in some bricks and in mount
point we get ENOTEMPTY.(DHT issue, Error will be logged in Slave log)

BUG: 1211037
Change-Id: I8716b88e4c741545f526095bf789f7c1e28008cb
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/10204
Reviewed-by: Kotresh HR 
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System
Reviewed-by: Vijay Bellur