glusterfs.git/geo-replication, branch v3.7.0beta1

geo-rep: Log Rsync performance

2015-04-02T13:47:16+00:00

Introducing configurable option to log the rsync performance.

gluster volume geo-replication  :: \
        config log-rsync-performance true

Default value is False.

Example log:
[2015-03-31 16:48:34.572022] I [resource(/bricks/b1):857:rsync] SSH: rsync
performance: Number of files: 2 (reg: 1, dir: 1), Number of regular files
transferred: 1, Total file size: 178 bytes, Total transferred file
size: 178 bytes, Literal data: 178 bytes, Matched data: 0 bytes,
Total bytes sent: 294, Total bytes received: 32, sent 294 bytes
received 32 bytes  652.00 bytes/sec

Change-Id: If11467e29e6ac502fa114bd5742a8434b7084f98
Signed-off-by: Aravinda VK 
BUG: 764827
Reviewed-on: http://review.gluster.org/10070
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

geo-rep: Add support for acls

2015-03-30T13:57:16+00:00

This patch adds support for ACLS. When it sees SETXATTR
in Changelog, it adds the file to data queue. rsync/tar+ssh
will take care of syncing ACLS. User set ACLS will be
synced to Slave.

This requires "system.posix_acl_access" to go through when
client-pid is equal GF_CLIENT_PID_GSYNCD in fuse layer.

New config interface is introduced, sync-acls
Which can be set using geo-rep config(Default is True)

gluster volume geo-replication  :: \
               config sync-acls false

Change-Id: I7eb3523fa72b8fed830efc98138891244e830d65
BUG: 1187021
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/10001
Reviewed-by: Aravinda VK 
Tested-by: Gluster Build System 
Reviewed-by: Venky Shankar 
Tested-by: Venky Shankar

geo-rep: Use gfchangelog context init API

2015-03-27T09:04:09+00:00

With the RPC based changes to {libgf}changelog, changelog_init
is required before changelog_register.

Change-Id: Id125b2bd2e51aaaffa22ecab463dfb739c50d83c
Signed-off-by: Venky Shankar 
BUG: 1170075
Reviewed-on: http://review.gluster.org/9993
Reviewed-by: Saravanakumar Arumugam 
Tested-by: Saravanakumar Arumugam 
Tested-by: Gluster Build System

geo-rep/gverify: Ignore .trashcan while checking slave is empty.

2015-03-20T17:27:05+00:00

With trash translator, '.trashcan' is created in gluster volume
during init. So geo-rep create used to fail, with slave not
being empty. The gverify script should ignore '.trashcan' while
checking whether slave is empty.

Change-Id: Ib7ee265c6aa99b63c7ccf103747d47a6f756ad35
BUG: 1203086
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/9921
Reviewed-by: Aravinda VK 
Tested-by: Gluster Build System 
Reviewed-by: Venky Shankar 
Tested-by: Venky Shankar

geo-rep: Ignore .trashcan during xsync crawl.

2015-03-19T04:22:55+00:00

With trash feature, .trashcan directory gets created
at each export directory. Xsync picks .trashcan to sync
and fails with EPERM. Xsync should ignore .trashcan
directory.

Change-Id: I45bd226c96011ace2c40dd2de878d886c7d34ce5
BUG: 1203293
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/9934
Reviewed-by: Vijay Bellur 
Reviewed-by: Aravinda VK 
Reviewed-by: Venky Shankar 
Tested-by: Gluster Build System

geo-rep / gsyncd: use RTLD_GLOBAL while loading libgfchangelog

2015-03-19T01:22:50+00:00

With the RPC based changes to {libgf}changelog, loading shared
objects dynamically would need symbols to be available from
other shared libraries. As an example, creating an RPC listner
loads the RPC transport shared object which requires symbols
to be available from already loaded shared objects.

Using RTLD_GLOBAL makes the symbols available for symbol
resolution of subsequently loaded libraries.

Change-Id: I3d3ef790eded82911f05836c707509157680645c
BUG: 1170075
Signed-off-by: Venky Shankar 
Reviewed-on: http://review.gluster.org/9814
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

geo-rep: Remove Generated file From git

2015-03-18T09:55:29+00:00

geo-replication/src/peer_mountbroker

BUG: 1136312
Change-Id: Ib9b287b4e1183cb44acbf01184a240be7f09be7c
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/9923
Reviewed-by: Vijay Bellur 
Tested-by: Vijay Bellur

feature/glusterfind: A tool to find incremental changes

2015-03-18T09:54:25+00:00

Documentation is available in patch:
http://review.gluster.org/#/c/9800/

A tool which helps to get list of modified files or list of all files in
GlusterFS Volume using Changelog or find command.

Usage
=====
glusterfind --help

Create:
-------
glusterfind create --help

The tool creates status file $GLUSTERD_WORKDIR/SESSION/VOLUME/status
and records current timestamp to initiate the session. This timestamp
will be used as start time for next runs.

As part of create also generates ssh key and distributes to all peers.
and enables build.pgfid and changelog using volume set command.

Pre:
----
glusterfind pre --help

This command is used to generate the list of files modified after session
creation time or after last run. To get list of all files/dirs in Volume,
run pre command with `--full` argument.

The tool gets all nodes details using gluster volume info and runs node
agent for each brick in respective nodes via ssh command. Once these node
agents generate the output file, tool copies to local using scp. Merges all
the output files to generate the final output file.

Post:
-----
glusterfind post --help

After consuming the list, this sub command is called to update the session
time based on pre command status file.

List:
-----
glusterfind list --help

To view all the sessions

Delete:
-------
glusterfind delete --help

Delete session.

Known Issues
------------
1. Deleted files will not get listed, since we can't convert GFID to
   Path if file/dir is deleted.
2. Only new name will get listed if Renamed.
3. All hardlinks will get listed.

Change-Id: I82991feb0aea85cb6ec035fddbf80a2b276e86b0
BUG: 1193893
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/9682
Tested-by: Gluster Build System 
Reviewed-by: Venky Shankar 
Reviewed-by: Prashanth Pai 
Reviewed-by: Kotresh HR 
Reviewed-by: Vijay Bellur

geo-rep: mountbroker user management

2015-03-17T11:56:24+00:00

Non root geo-replication setup is now simplified. This
patch provides cli for mountbroker user and options management

To set Options,
    gluster system:: execute mountbroker opt  
    # for example,
    gluster system:: execute mountbroker opt mountbroker-root /var/mountbroker-root
    gluster system:: execute mountbroker opt geo-replication-log-group geogroup
    gluster system:: execute mountbroker opt rpc-auth-allow-insecure on

To remove option,
    gluster system:: execute mountbroker optdel 
    # for example,
    gluster system:: execute mountbroker optdel geo-replication-log-group

To add/edit user,
    gluster system:: execute mountbroker user  
    # for example
    gluster system:: execute mountbroker user geoaccount slavevol1,slavevol2

To remove user,
    gluster system:: execute mountbroker userdel 
    # for example
    gluster system:: execute mountbroker userdel geoaccount

For info,
    gluster system:: execute mountbroker info
    gluster system:: execute mountbroker -j info

For JSON output add -j after mountbroker, for example,
    gluster system:: execute mountbroker -j user geoaccount slavevol1,slavevol2

PS: Each peer prints its own JSON output, aggregator required from consumer side

BUG: 1136312
Change-Id: Ie52210c0bcc91ac2ffd3ba58988222ffca62b47f
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/9398
Tested-by: Gluster Build System 
Reviewed-by: darshan n 
Reviewed-by: Kotresh HR 
Reviewed-by: Vijay Bellur

geo-rep: Do not use xsync_upper_limit for change detection

2015-03-16T05:55:31+00:00

Use register time(xsync_upper_limit) only for stime update, do not
use for change detection.

Problem 1:
If a file created before geo-rep, xtime xattr does not exist.
Geo-rep updates xtime of the file to current time if not exists.
xtime > upper_limit so geo-rep will not pick those files. Changelog
either will have SETXATTR, and fails to sync the file.

Problem 2:
If a file is created before geo-rep create and updated after
geo-rep start. xtime of the file is greater than upper limit(geo-rep
start time/changelog register time). Geo-rep(XSync) will not pick this
file for syncing. Changelog will have only DATA recorded for that file.
Geo-rep tries DATA without any ENTRY ops and fails with rsync error.

BUG: 1200733
Change-Id: Ie4e8f284db689d2c755ef8e7ecbb658db1c0785f
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/9855
Reviewed-by: Kotresh HR 
Reviewed-by: Saravanakumar Arumugam 
Tested-by: Saravanakumar Arumugam 
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur