glusterfs.git/geo-replication/syncdaemon/gsyncd.py, branch v3.11.0beta1

geo-rep: Improve worker log messages

2017-04-07T06:09:34+00:00

Monitor process expects worker to establish SSH Tunnel to slave node
and mount master volume locally with in 60 secs and acknowledge monitor
process by closing feedback fd. If something goes wrong and worker
does not close feedback fd with in 60 secs, monitor kills the worker.
But there was no clue in log message about the actual issue. This patch
adds log and indicates whether the worker is hung during SSH
or master mount.

Change-Id: Id08a12fa6f3bba1d4fe8036728dbc290e6c14c8c
BUG: 1261689
Signed-off-by: Kotresh HR 
Reviewed-on: https://review.gluster.org/16997
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Aravinda VK

geo-rep: Optionally allow access to mounts

2017-03-20T03:33:36+00:00

In order to improve debuggability, it is important
to have access to geo-rep master and slave mounts.
With the default behaviour, geo-rep lazy unmounts
the mounts after changing the current working
directory into the mount point. It also cleans
up the mount points. So only geo-rep worker has
the access and it becomes impossible to take the
client profile info and do any other client statck
analysis. Hence the following new config is being
introduced to allow access to mounts.

gluster vol geo-rep  :: \
config access_mount true

The default value of 'access_mount' is false.

Change-Id: I53dce4ea86a6ffc979c82f9330e8954327180ca3
BUG: 1433506
Signed-off-by: Kotresh HR 
Reviewed-on: https://review.gluster.org/16912
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Vijay Bellur

geo-rep: Separate slave mount logs for each connection

2017-01-18T08:47:01+00:00

Geo-rep worker mounts the slave volume on the slave
node. If multiple worker connects to same slave node,
all workers share the same mount log file. This
is very difficult to debug as logs are cluttered from
different mounts. Hence creating separate mount log
file for each connection from worker. Each connection
from worker is identified uniquely using 'mastervol uuid',
'master host', 'master brickpath', 'salve vol'. The log
file name will be combination of the above.

Change-Id: I67871dc8e8ea5864e2ad55e2a82063be0138bf0c
BUG: 1412689
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/16384
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Aravinda VK

geo-rep/eventsapi: Add Master node information in Geo-rep Events

2016-12-02T10:16:32+00:00

Added Master node information to GEOREP_ACTIVE, GEOREP_PASSIVE, GEOREP_FAULTY
and GEOREP_CHECKPOINT_COMPLETED events.

EVENT_GEOREP_ACTIVE(master_node and master_node_id are new fields)
{
    "nodeid": NODEID,
    "ts": TIMESTAMP,
    "event": "GEOREP_ACTIVE",
    "message": {
        "master_volume": MASTER_VOLUME_NAME,
        "master_node": MASTER_NODE,
        "master_node_id": MASTER_NODE_ID,
        "slave_host": SLAVE_HOST,
        "slave_volume": SLAVE_VOLUME,
        "brick_path": BRICK_PATH
    }
}

EVENT_GEOREP_PASSIVE(master_node and master_node_id are new fields)
{
    "nodeid": NODEID,
    "ts": TIMESTAMP,
    "event": "GEOREP_PASSIVE",
    "message": {
        "master_volume": MASTER_VOLUME_NAME,
        "master_node": MASTER_NODE,
        "master_node_id": MASTER_NODE_ID,
        "slave_host": SLAVE_HOST,
        "slave_volume": SLAVE_VOLUME,
        "brick_path": BRICK_PATH
    }
}

EVENT_GEOREP_FAULTY(master_node and master_node_id are new fields)
{
    "nodeid": NODEID,
    "ts": TIMESTAMP,
    "event": "GEOREP_FAULTY",
    "message": {
        "master_volume": MASTER_VOLUME_NAME,
        "master_node": MASTER_NODE,
        "master_node_id": MASTER_NODE_ID,
        "current_slave_host": CURRENT_SLAVE_HOST,
        "slave_host": SLAVE_HOST,
        "slave_volume": SLAVE_VOLUME,
        "brick_path": BRICK_PATH
    }
}

EVENT_GEOREP_CHECKPOINT_COMPLETED(master_node and master_node_id are new fields)
{
    "nodeid": NODEID,
    "ts": TIMESTAMP,
    "event": "GEOREP_CHECKPOINT_COMPLETED",
    "message": {
        "master_volume": MASTER_VOLUME_NAME,
        "master_node": MASTER_NODE,
        "master_node_id": MASTER_NODE_ID,
        "slave_host": SLAVE_HOST,
        "slave_volume": SLAVE_VOLUME,
        "brick_path": BRICK_PATH,
        "checkpoint_time": CHECKPOINT_TIME,
        "checkpoint_completion_time": CHECKPOINT_COMPLETION_TIME
    }
}

BUG: 1395660
Change-Id: Ic91af52fa248c8e982e93a06be861dfd69689f34
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15858
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR

geo-rep: Upgrade conf file only if it is session config

2016-10-24T15:26:24+00:00

Ignore config upgrade if it is template config file present in
/var/lib/glusterd/geo-replication/gsyncd_template.conf

BUG: 1386123
Change-Id: I2cbba3103b6801c16ff57f778a90b9a0bb2467cf
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15669
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR 
NetBSD-regression: NetBSD Build System

geo-rep: Logging improvements

2016-10-24T07:01:48+00:00

- Redundant log messages removed.
- Worker and connected slave node details added in "starting worker" log
- Added log for Monitor state change
- Added log for Worker status change(Initializing/Active/Passive/Faulty)
- Added log for Crawl status Change
- Added log for config set and reset
- Added log for checkpoint set, reset and completion

BUG: 1359612
Change-Id: Icc7173ff3c93de4b862bdb1a61760db7eaf14271
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15684
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR

geo-rep/eventsapi: Additional Events

2016-10-18T10:56:03+00:00

Added following events
EVENT_GEOREP_ACTIVE
    {
        "nodeid": NODEID,
        "ts": TIMESTAMP,
        "event": "GEOREP_ACTIVE",
        "message": {
            "master_volume": MASTER_VOLUME_NAME,
            "slave_host": SLAVE_HOST,
            "slave_volume": SLAVE_VOLUME,
            "brick_path": BRICK_PATH
        }
    }

EVENT_GEOREP_PASSIVE
    {
        "nodeid": NODEID,
        "ts": TIMESTAMP,
        "event": "GEOREP_PASSIVE",
        "message": {
            "master_volume": MASTER_VOLUME_NAME,
            "slave_host": SLAVE_HOST,
            "slave_volume": SLAVE_VOLUME,
            "brick_path": BRICK_PATH
        }
    }
EVENT_GEOREP_CHECKPOINT_COMPLETED
    {
        "nodeid": NODEID,
        "ts": TIMESTAMP,
        "event": "GEOREP_ACTIVE",
        "message": {
            "master_volume": MASTER_VOLUME_NAME,
            "slave_host": SLAVE_HOST,
            "slave_volume": SLAVE_VOLUME,
            "brick_path": BRICK_PATH,
            "checkpoint_time": CHECKPOINT_TIME,
            "checkpoint_completion_time": CHECKPOINT_COMPLETION_TIME
        }
    }

BUG: 1379330
Change-Id: I90716175868c59dd65c8d202e73e0ede90347b6a
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15630
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR 
Tested-by: Kotresh HR

geo-rep: Use configured log_level for libgfchangelog logs

2016-09-09T07:23:30+00:00

libgfchangelog was not respecting the log_level configured
in Geo-replication. With this patch Libgfchangelog log level
can be configured using `config changelog_log_level TRACE`.
Default Changelog log level is INFO

BUG: 1363965
Change-Id: Ida714931129f6a1331b9d0815da77efcb2b898e3
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15078
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR

geo-rep: Fix History post process

2016-09-08T06:11:18+00:00

This patch removes changelogsdb part of post processing since
not got much performance advantage as expected.

Entry stime and other logging improvements retained.

BUG: 1364420
Change-Id: Ib99d23f09d96c14bc28225b47d9134260f5551bf
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15371
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR 
Smoke: Gluster Build System

geo-rep: Post process Data and Meta Changelogs

2016-08-26T17:45:58+00:00

With this patch, Data and Meta GFIDs are post processed. If Changelog has
UNLINK entry then remove from Data and Meta GFIDs list(If stat on GFID is
ENOENT in Master).

While processing Changelogs,

- Collect all the data and meta operations in a temporary database
- Delete all Data and Meta GFIDs which are already unlinked as per Changelogs
  (unlink only if stat on GFID is ENOENT)
- Process all Entry operations as usual
- Process data and meta operations in batch(Fetch from Db in batch)
- Data sync is again batched based on number of changelogs(Default 1day
  changelogs). Once the sync is complete, Update last Changelog's time as last_synced
  time as usual.

Additionally maintain entry_stime on Brick root, ignore Entry ops if changelog
suffix time is less than entry_stime. If data stime is more than entry_stime,
this can happen only when passive worker updates stime by itself by getting
mount point stime. Use entry_stime = data_stime in this case.

New configurations:

max-rsync-retries - Default Value is 10
max-data-changelogs-in-batch - Max number of changelogs to be considered in a
batch for syncing. Default value is 5760(4 changelogs per min * 60 min *
24 hours)
max-history-changelogs-in-batch - Max number of history changelogs to be
processed at once. Default value 86400(4 changelogs per min * 60 min * 24
hours * 15 days)

BUG: 1364420
Change-Id: I7b665895bf4806035c2a8573d361257cbadbea17
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15110
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Kotresh HR 
CentOS-regression: Gluster Build System