summaryrefslogtreecommitdiffstats
path: root/geo-replication/syncdaemon/master.py
Commit message (Collapse)AuthorAgeFilesLines
* geo-rep: Use configured log_level for libgfchangelog logsAravinda VK2016-09-131-1/+0
| | | | | | | | | | | | | | | | | | | | | | libgfchangelog was not respecting the log_level configured in Geo-replication. With this patch Libgfchangelog log level can be configured using `config changelog_log_level TRACE`. Default Changelog log level is INFO > Reviewed-on: http://review.gluster.org/15078 > Smoke: Gluster Build System <jenkins@build.gluster.org> > NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> > CentOS-regression: Gluster Build System <jenkins@build.gluster.org> > Reviewed-by: Kotresh HR <khiremat@redhat.com> BUG: 1374606 Change-Id: Ida714931129f6a1331b9d0815da77efcb2b898e3 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/15449 Smoke: Gluster Build System <jenkins@build.gluster.org> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Saravanakumar Arumugam <sarumuga@redhat.com> CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
* georep: add reset-sync-time option for session deleteMilind Changire2016-07-211-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Set the stime xattr at all the brick roots to (0,0) if the argument reset-sync-time has been provided on the command-line. To avoid testing against directory specific stime, the remote stime is assumed to be minus_infinity, if the root directory stime is set to (0,0), before the directory scan begins. This triggers a full volume resync to slave in the case of a geo-rep session recreation with the same master-slave volume pair. Command synopsis: gluster volume geo-replication <MASTERVOL> <SLAVE>::<SLAVEVOL> delete \ [reset-sync-time] Update gluster cli man page to include new sub-command reset-sync-time. Change-Id: Ie4ce03b9425ed9bb81eda8681058c0fc6f990948 BUG: 1357772 Signed-off-by: Milind Changire <mchangir@redhat.com> Reviewed-on: http://review.gluster.org/14051 Reviewed-by: Kotresh HR <khiremat@redhat.com> Smoke: Gluster Build System <jenkins@build.gluster.org> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Aravinda VK <avishwan@redhat.com> (cherry picked from commit 70fd68d94f768c098b3178c151fa92c5079a8cfd) Reviewed-on: http://review.gluster.org/14952
* geo-rep: Safely handle if unliked GFID not present in data listAravinda VK2016-06-271-1/+2
| | | | | | | | | | | | | | | | | If unlinked GFID is not present in data list to be synced then Geo-rep worker was crashing with KeyError. Handled KeyError with this patch. BUG: 1348085 Change-Id: I5a1c9ca4473e32606df2e5c7e26c95faf55d44c0 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/14706 (cherry picked from commit 4797ca3778d82a671716d4913c14f285591ae959) Reviewed-on: http://review.gluster.org/14766 Smoke: Gluster Build System <jenkins@build.gluster.org> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Kotresh HR <khiremat@redhat.com> CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
* geo-rep: Handling Rsync/Tar errors efficientlyAravinda VK2016-03-081-64/+44
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Geo-rep processes Changelogs in Batch, if one file in batch fails with rsync error that Changelog file is reprocessed multiple times. After MAX_RETRY, it logs all the GFIDs from that batch as Skipped. This patch addresses following issues, 1. When Rsync/Tar fails do not parse Changelog again for retry 2. When Rsync/Tar fails do not replay Entry operations, only retry rsync/tar for those GFIDs 3. Log Error in Rsync/Tar only in the last Retry 4. Do not log Skipped GFIDs since Rsync/Tar errors are logged for only failed files. 5. Changed Entry failures as Error instead of Warning BUG: 1313309 Change-Id: Ie134ce2572693056ab9b9008cd8aa5b5d87f7975 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/12856 Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Saravanakumar Arumugam <sarumuga@redhat.com> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> Smoke: Gluster Build System <jenkins@build.gluster.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Venky Shankar <vshankar@redhat.com> (cherry picked from commit d136a789258e8f600e536717da156a242d8ed9a5) Reviewed-on: http://review.gluster.org/13558
* geo-rep: Handle hardlink in Tiering based volumeSaravanakumar Arumugam2016-03-071-10/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Problem: Hardlinks are synced as Sticky bit files to Slave in a Tiering based volume. In a Tiering based volume, cold tier is hashed subvolume and geo-rep captures all namespace operations in cold tier. While syncing a file and its corresponding hardlink, it is recorded as MKNOD in cold tier(for both) and We end up creating two different files in Slave. Solution: If MKNOD with Sticky bit set is present, record it as LINK. This way it will create a HARDLINK if source file exists (on slave), else it will create a new file. This way, Slave can create Hardlink file itself (instead of creating a new file) in case of hardlink. Change-Id: Ic50dc6e64df9ed01799c30539a33daace0abe6d4 BUG: 1302979 Signed-off-by: Saravanakumar Arumugam <sarumuga@redhat.com> Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/13281 Reviewed-on: http://review.gluster.org/13315 Smoke: Gluster Build System <jenkins@build.gluster.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
* geo-rep: Symlink Rename issueAravinda VK2016-01-031-1/+7
| | | | | | | | | | | | | | | | | | | | | | | | If ENTRY creation failed for symlink in Slave and symlink renamed in Master. If Source not exists to Rename in Slave Geo-rep interprets as Create of Target file. Geo-rep sends blob of regular file to create symlink instead of sending blob of symlink. With this patch, Geo-rep identifies symlink and sends respective blob. BUG: 1292697 Change-Id: If9351974d1945141a1d3abb838b7d0de7591e48e Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/12917 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Milind Changire <mchangir@redhat.com> Tested-by: Milind Changire <mchangir@redhat.com> Tested-by: NetBSD Build System <jenkins@build.gluster.org> (cherry picked from commit 3151194fad1aed3ab340b53aeeb4e8d131538d12) Reviewed-on: http://review.gluster.org/12998 Reviewed-by: Venky Shankar <vshankar@redhat.com>
* geo-rep: use cold tier bricks for namespace operationsSaravanakumar Arumugam2015-12-071-4/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Problem: symlinks are not getting synced to slave in a Tiering based volume. Solution: Now, symlinks are created directly in cold tier bricks( in the backend). Earlier, cold tier was avoided for namespace operations and only hot tier was used while processing changelogs. Now, cold tier is HASH subvolume in a Tiering volume. So, carry out namespace operation only in cold tier subvolume and avoid hot tier subvolume to avoid any races. Earlier, XSYNC was used(and changeloghistory avoided) during initial sync in order to avoid race while processing historychangelog in Hot tier. This is no longer required as there is no race from Hot tier. Also, avoid both live and history changelog ENTRY operations from Hot tier to avoid any race with cold tier. Change-Id: Ia8fbb7ae037f5b6cb683f36c0df5c3fc2894636e BUG: 1288027 Signed-off-by: Saravanakumar Arumugam <sarumuga@redhat.com> Reviewed-on: http://review.gluster.org/12844 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> (cherry picked from commit 93f31189ce8f6e2980a39b02568ed17088e0a667) Reviewed-on: http://review.gluster.org/12891
* geo-rep: fd close and fcntl issueAravinda VK2015-12-071-15/+19
| | | | | | | | | | | | | | | | | | | | | When any of the open fd of a file is closed on which fcntl lock is taken even though another fd of the same file is open on which lock is taken, all fcntl locks will be released. This causes both replica workers to be ACTIVE sometimes. This patche fixes that issue. BUG: 1287456 Change-Id: I8f113230d16075f18020044898a7eda5e2a5a0ac Original-Author: Aravinda VK <avishwan@redhat.com> Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/12752 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Milind Changire <mchangir@redhat.com> Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Aravinda VK <avishwan@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Reviewed-on: http://review.gluster.org/12842
* geo-rep: Avoid cold tier bricks during ENTRY operationSaravanakumar Arumugam2015-12-071-0/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a series of patch which aims to fix geo-replication in a Tiering Volume. Problem: Consider, a file is placed in volume initially and then hot tier is attached. During any operation on the file, due to lookup a linkto file is created in hot tier. Now, any namespace operation carried out on the file is recorded in both cold and hot tier. There is a room for races when both changelogs are replayed. Solution: So, We are going to replay (namespace related)operations only in the hot tier. Why? a. If the file is directly placed in Hot tier , all fops will be recorded in HOT tier. b. If the file is already present in Cold tier, and if any fop is carried out, it creates linkto file in Hot tier. Now, operations like UNLINK, RENAME are captured in Hot tier(by means of linkto file). This way, we can get both tier's operation in HOT tier itself. Now, once the file is demoted to COLD tier, any namespace operation carried out on the cold tier can be avoided as we directly RECORD the same in HOT tier. How? 1. Check whether the brick is cold tier and skip ENTRY operation. 2. Also, if it is cold tier brick, use Xsync(which is used during initial run). This will help in getting all cold tier bricks changes using File System crawl and helps in avoiding races with hot tier brick(which can happen if historychangelog used in cold tier brick). Dependent patches: 1. http://review.gluster.org/12239 2. http://review.gluster.org/12326 Change-Id: I7692b1dbb8813a7e253451bca02f8f09a5782dde BUG: 1275173 Signed-off-by: Saravanakumar Arumugam <sarumuga@redhat.com> Reviewed-on: http://review.gluster.org/12355 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Aravinda VK <avishwan@redhat.com> (cherry picked from commit 6188b5fcebc56b3d8af1956beeec9988f3e8f268) Reviewed-on: http://review.gluster.org/12429 Reviewed-by: Venky Shankar <vshankar@redhat.com>
* geo-rep: Fix FD leak from Active Geo-rep workerAravinda VK2015-12-011-0/+15
| | | | | | | | | | | | | | | | | | | | | Active worker tries to acquire lock in each iteration. On every successfull lock acqusition it was not closing previously opened lock fd. To see the leak, get the PID of worker, ps -ax | grep feedback-fd watch ls /proc/$pid/fd BUG: 1225567 Change-Id: Ic476c24c306e7ab372c5560fbb80ef39f4fb31af Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/12332 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Milind Changire <mchangir@redhat.com> Reviewed-by: Saravanakumar Arumugam <sarumuga@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> (cherry picked from commit 42def948ac8e5d24278cb000cc8f8906b83a8592) Reviewed-on: http://review.gluster.org/12650 Tested-by: Gluster Build System <jenkins@build.gluster.com>
* geo-rep: Handle FXATTROP and XATTROPKotresh HR2015-11-271-1/+2
| | | | | | | | | | | | | | | | | | | GEO-REP INTEROP WITH SHARD FEATURE If it is FXATTROP or XATTROP in changelog, add the gfid to rsync queue. BUG: 1284453 Change-Id: If598da1543bbff7fe9228ba57c7e813ed2a3a7f2 Reviewed-on: http://review.gluster.org/12226 Tested-by: Gluster Build System <jenkins@build.gluster.com> Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Aravinda VK <avishwan@redhat.com> Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/12721 Reviewed-by: Saravanakumar Arumugam <sarumuga@redhat.com> Reviewed-by: Milind Changire <mchangir@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com>
* geo-rep: Fix syncing chown in xsync crawlKotresh HR2015-11-271-5/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | GEO-REP INTEROP WITH SHARD FEATURE Problem: The sequence of entry creation and chown in master is recorded as creation of entry with resulted user:group in xsync changelog. During sync, entry creation is always split into two ops, MKNOD and SETATTR. Hence the issue is not being hit otherwise it would have failed with EPERM if parent is owned by different user. But with shard translator being enabled on slave, doing entry creation with MKNOD and SETATTR is not allowed, SETATTR fails as it accesses inode structure which is not linked. Solution: The sequence of entry creation and chown in master should be recorded as MKNOD and SETATTR separately always and do entry creation with single op in gfid-access xlator. The gfid-access patch will be sent separately. BUG: 1284453 Change-Id: Ia577aa5270cb96b86830885d6c4c01fb0133eeed Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/12205 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/12729 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Milind Changire <mchangir@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com>
* geo-rep: Update last_synced_time in XSyncAravinda VK2015-11-091-14/+9
| | | | | | | | | | | | | | | | | During XSync crawl, last_synced time in status file was not updated. This patch fixes the issue by updating status file when stime xattr is updated after Xsync or Changelog Crawl. Change-Id: I4dc3a2d4c3d8378a939da0868caf1aef4f789599 Signed-off-by: Aravinda VK <avishwan@redhat.com> BUG: 1279306 Reviewed-on: http://review.gluster.org/11771 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Jeff Darcy <jdarcy@redhat.com> (cherry picked from commit ee0b1a3bf11a4d05696212d91c932ddb7c7091ee) Reviewed-on: http://review.gluster.org/12545 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Venky Shankar <vshankar@redhat.com>
* geo-rep: Add data operation if mknod with tier attributeSaravanakumar Arumugam2015-10-291-0/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a series of patches which aims to fix geo-replication in a Tiering Volume. Problem: Consider, a file is placed in volume initially and then hot tier is attached. During any operation on the file, due to lookup a linkto file is created in hot tier. Now, any namespace operation carried out on the file is recorded in both cold and hot tier. There is a room for races when both changelogs are replayed. Solution: So, We are going to replay (namespace related)operations only in the hot tier. Why? a. If the file is directly placed in Hot tier, all fops will be recorded in HOT tier. b. If the file is already present in Cold tier, and if any fop is carried out, it creates linkto file in Hot tier. Now, operations like UNLINK, RENAME are captured in Hot tier(by means of linkto file). This way, we can get both tier's operation in HOT tier itself. But, We may miss initial Data sync immediately after creating the file as it is only recording MKNOD. So, if MKNOD encountered with sticky bit set, queue DATA operation for the corresponding gfid. (This is addressed here in this patch) So, If tier-gfid linkto is set, we need to record the corresponding MKNOD. Earlier this was avoided as it was set as INTERNAL fop. (This changelog related changes are addressed in the patch: - http://review.gluster.org/12417) Change-Id: I2fa84cfa2b0f86506c3d15d484138ab9651e4f83 BUG: 1275173 Signed-off-by: Saravanakumar Arumugam <sarumuga@redhat.com> Reviewed-on: http://review.gluster.org/12326 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Aravinda VK <avishwan@redhat.com> (cherry picked from commit ffc39c9d8807464b5c78959bc43dc12b22f5a37b) Reviewed-on: http://review.gluster.org/12427 Reviewed-by: Venky Shankar <vshankar@redhat.com>
* geo-rep: Fix history failureKotresh HR2015-08-051-5/+13
| | | | | | | | | | | | | | | | | | | | | | | | | Both ACTIVE and PASSIVE workers register to changelog at almost same time. When PASSIVE worker becomes ACTIVE, the start and end time would be current stime and register_time repectively for history API. Hence register_time would be less then stime for which history obviously fails. But it will be successful for the next restart as new register_time > stime. Fix is to pass current time as the end time to history call instead of the register_time. Also improvised the logging for ACTIVE/PASSIVE switching. BUG: 1247882 Change-Id: I40c582cc32fe29a6c30340ec81a3b5d30e461e71 Reviewed-on: http://review.gluster.org/11524 Tested-by: Gluster Build System <jenkins@build.gluster.com> Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Aravinda VK <avishwan@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/11784 Reviewed-by: Milind Changire <mchangir@redhat.com>
* geo-rep: Fix fd referenced before assignmentKotresh HR2015-07-121-2/+6
| | | | | | | | | | | | | | | | Fix fd reference before assignment in mgmt_lock function. BUG: 1240607 Change-Id: Ib8a5207a516eecb55a5f906ff5d49abc98cab2a6 Reviewed-on: http://review.gluster.org/11318 Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/11563 Tested-by: Gluster Build System <jenkins@build.gluster.com> Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Saravanakumar Arumugam <sarumuga@redhat.com> Reviewed-by: Milind Changire <mchangir@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com>
* geo-rep: ignore ESTALE as ENOENTAravinda VK2015-06-281-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | When DHT can't resolve a File it raises ESTALE, ignore ESTALE errors same as ENOENT after retry. Affected places: Xattr.lgetxattr os.listdir os.link Xattr.lsetxattr os.chmod os.chown os.utime os.readlink BUG: 1236093 Change-Id: I53f8dfa47911da93e0dcc20213afcbb47a14ccd8 Reviewed-On: http://review.gluster.org/11296 Original-Author: Aravinda VK <avishwan@redhat.com> Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/11430 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Milind Changire <mchangir@redhat.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* geo-rep: Fix geo-rep fanout setup with meta volumeKotresh HR2015-06-251-2/+2
| | | | | | | | | | | | | | | | | | | Lock filename was formed with 'master volume id' and 'subvol number'. Hence multiple slaves try acquiring lock on same file and become PASSIVE ending up not syncing data. Using 'slave volume id' in lock filename will fix the issue making lock file unique across different slaves. Change-Id: I64c84670a5d9e1b0dfbdeb4479ee6b8e0c6b829e BUG: 1234898 Reviewed-On: http://review.gluster.org/11367 Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/11366 Tested-by: Gluster Build System <jenkins@build.gluster.com> Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Saravanakumar Arumugam <sarumuga@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com>
* geo-rep: Fix toggling of use_meta_volume configKotresh HR2015-06-251-1/+1
| | | | | | | | | | | | | | | | | If meta-volume is deleted and use_meta_volume is set to false, geo-rep still fails complaining meta volume is not mounted. The patch fixes that issue. BUG: 1234695 Change-Id: I5ec7ad85399b90e85bca1d36750dd3fb29222db9 Reviewe-On: http://review.gluster.org/11358 Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/11359 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Saravanakumar Arumugam <sarumuga@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com>
* geo-rep: Fix Data counter issue in statusAravinda VK2015-05-311-2/+12
| | | | | | | | | | | | | | | | | ENTRY and META operations executed sequentially, DATA operations are handled async, increment happens when a changelog parsed. Decrement happens after the sync of all files. files_in_batch was reset multiple times in batch instead of once. BUG: 1224100 Change-Id: I87617f2fd5f4d3221a1c9f9d5a8efb0686c42bbe Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed On: http://review.gluster.org/#/c/10911/ Reviewed-on: http://review.gluster.org/10912 Tested-by: Gluster Build System <jenkins@build.gluster.com> Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Venky Shankar <vshankar@redhat.com>
* geo-rep: Disable xattrs and acls support with tar_sshKotresh HR2015-05-291-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Geo-rep can't sync xattrs and acls with tar over ssh for following reasons. Issue 1: xattrs doesn't sync with tar over ssh. Reason: untar doesn't respect '--overwrite' option when used along with '--xattrs'. So it sends unlink if the file exists on destination and re-creates afresh. But all entry operations are banned in aux-gfid-mount as it may lead to gfid-mismatch. Hence fails with EPERM. This happens only when some xattr is set on a file in master volume. Issue2: acls on directories does not sync with tar over ssh. Reason: tar tries to opendir ".gfid/<gfid1>" and is not supported by gfid-access-translator as readirp can't be handled on virtual inodes and hence fails with ENOTSUP where as it syncs for files. Since the issue is with tar commmand it self and nothing could be done from gluster side, disabling xattr and acls support with tar over ssh option. Geo-rep can sync xattrs and acls with 'rsync' as the sync engine. BUG: 1223644 Change-Id: Ieba9e749ddbc41683b58ed8b1cc2ca50517fe58b Reviewed On: http://review.gluster.org/#/c/10873 Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/10899 Tested-by: Gluster Build System <jenkins@build.gluster.com> Tested-by: NetBSD Build System Reviewed-by: Aravinda VK <avishwan@redhat.com>
* geo-rep: Limit number of changelogs to process in batchAravinda VK2015-05-071-6/+32
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Changelog processing is done in batch, for example if 10 changelogs available for processing then process all at once. Collect Entry, Meta and Data operations separately, All the entry operations like CREATE, MKDIR, MKNOD, LINK, UNLINK will be executed first then rsync will be triggered for whole batch. Stime will get updated once the complete batch is complete. In case of large number of Changelogs in a batch, If geo-rep fails after Entry operations, but before rsync then on restart, it again starts from the beginning since stime is not updated. It has to process all the changelogs again. While processing same changelogs again, all CREATE will get EEXIST since all the files created in previous run. Big hit for performance. With this patch, Geo-rep limits number of changelogs per batch based on Changelog file size. So that when geo-rep fails it has to retry only last batch changelogs since stime gets updated after each batch. BUG: 1217930 Change-Id: I844448c4cdcce38a3a2e2cca7c9a50db8f5a9062 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/10202 Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com> Reviewed-on: http://review.gluster.org/10499 Tested-by: Gluster Build System <jenkins@build.gluster.com> Tested-by: NetBSD Build System Reviewed-by: Venky Shankar <vshankar@redhat.com>
* geo-rep: Minimize rm -rf race in Geo-repAravinda VK2015-05-061-2/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | While doing RMDIR worker gets ENOTEMPTY because same directory will have files from other bricks which are not deleted since that worker is slow processing. So geo-rep does recursive_delete. Recursive delete was done using shutil.rmtree. once started, it will not check disk_gfid in between. So it ends up deleting the new files created by other workers. Also if other worker creates files after one worker gets list of files to be deleted, then first worker will again get ENOTEMPTY again. To fix these races, retry is added when it gets ENOTEMPTY/ESTALE/ENODATA. And disk_gfid check added for original path for which recursive_delete is called. This disk gfid check executed before every Unlink/Rmdir. If disk gfid is not matching with GFID from Changelog, that means other worker deleted the directory. Even if the subdir/file present, it belongs to different parent. Exit without performing further deletes. Retry on ENOENT during create is ignored, since if CREATE/MKNOD/MKDIR failed with ENOENT will not succeed unless parent directory is created again. Rsync errors handling was handling unlinked_gfids_list only for one Changelog, but when processed in batch it fails to detect unlinked_gfids and retries again. Finally skips the entire Changelogs in that batch. Fixed this issue by moving self.unlinked_gfids reset logic before batch start and after batch end. Most of the Geo-rep races with rm -rf is eliminated with this patch, but in some cases stale directories left in some bricks and in mount point we get ENOTEMPTY.(DHT issue, Error will be logged in Slave log) BUG: 1218922 Change-Id: I8716b88e4c741545f526095bf789f7c1e28008cb Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/10204 Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com> Reviewed-on: http://review.gluster.org/10599 Tested-by: Gluster Build System <jenkins@build.gluster.com>
* geo-rep: Status EnhancementsAravinda VK2015-05-061-321/+53
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Discussion in gluster-devel http://www.gluster.org/pipermail/gluster-devel/2015-April/044301.html MASTER NODE - Master Volume Node MASTER VOL - Master Volume name MASTER BRICK - Master Volume Brick SLAVE USER - Slave User to which Geo-rep session is established SLAVE - <SLAVE_NODE>::<SLAVE_VOL> used in Geo-rep Create command SLAVE NODE - Slave Node to which Master worker is connected STATUS - Worker Status(Created, Initializing, Active, Passive, Faulty, Paused, Stopped) CRAWL STATUS - Crawl type(Hybrid Crawl, History Crawl, Changelog Crawl) LAST_SYNCED - Last Synced Time(Local Time in CLI output and UTC in XML output) ENTRY - Number of entry Operations pending.(Resets on worker restart) DATA - Number of Data operations pending(Resets on worker restart) META - Number of Meta operations pending(Resets on worker restart) FAILURES - Number of Failures CHECKPOINT TIME - Checkpoint set Time(Local Time in CLI output and UTC in XML output) CHECKPOINT COMPLETED - Yes/No or N/A CHECKPOINT COMPLETION TIME - Checkpoint Completed Time(Local Time in CLI output and UTC in XML output) XML output: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> cliOutput> geoRep> volume> name> sessions> session> session_slave> pair> master_node> master_brick> slave_user> slave/> slave_node> status> crawl_status> entry> data> meta> failures> checkpoint_completed> master_node_uuid> last_synced> checkpoint_time> checkpoint_completion_time> BUG: 1218586 Change-Id: I944a6c3c67f1e6d6baf9670b474233bec8f61ea3 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/10121 Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com> Reviewed-on: http://review.gluster.org/10574 Tested-by: Gluster Build System <jenkins@build.gluster.com>
* geo-rep: log ENTRY failures from slave on masterMilind Changire2015-05-061-2/+10
| | | | | | | | | | | | | | | | ENTRY operations failures on slave left no trace for debugging purposes. This patch captures such failures on slave cluster and forwards them to the master and logs them. Failures of specific interest are the ones which return code EEXIST on the failing operations. Change-Id: Iecab876f16593c746d53f4b7ec2e0783367856bb Signed-off-by: Milind Changire <mchangir@redhat.com> BUG: 1217429 Reviewed-on: http://review.gluster.org/10048 Reviewed-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/10517 Tested-by: Gluster Build System <jenkins@build.gluster.com> Tested-by: NetBSD Build System
* geo-rep: Changes required as part of changelog version 1.2Kotresh HR2015-05-041-0/+6
| | | | | | | | | | | | | | | | | | | | The index value for UNLINK and RMDIR in changelog is no more the last index. It varies based on whether the 'changelog.capture-del-path' is enabled or not. Hence, fixed index is used. The option to capture deleted path in changelog comes with the patch: http://review.gluster.org/#/c/10288/ And the parser changes with http://review.gluster.org/#/c/10166/ BUG: 1217935 Change-Id: I6d8a941129416381e24226faed9803f7e5fae8d0 Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/10344 Reviewed-on: http://review.gluster.org/10500 Tested-by: NetBSD Build System Reviewed-by: Aravinda VK <avishwan@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* geo-rep: Fix minor bugs in meta-volume setupKotresh HR2015-05-041-1/+2
| | | | | | | | | | | | | | | | | | | | | | | 1. Access unreferenced access of fd: In meta volume configuration for geo-rep, if geo-rep directory is not created yet, open fails with no fd, but it is accessed in close(fd). So after creating 'geo-rep' directory in meta-volume, open the lock file to get fd. 2. Fix volume_id in forming lock file name. For the very first time, gconf.volume_id would be null, as config is not reloaded yet. Hence, use 'uuid' function to get the volume id. BUG: 1217939 Change-Id: I06bf659ec506943c7ebf5303cc56a2e7748fbad0 Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/10458 Reviewed-on: http://review.gluster.org/10506 Tested-by: Gluster Build System <jenkins@build.gluster.com> Tested-by: NetBSD Build System Reviewed-by: Aravinda VK <avishwan@redhat.com>
* geo-rep: Adhering to the common storage for geo-repKotresh HR2015-05-041-10/+20
| | | | | | | | | | | | | | | | | | | | Making geo-rep use the common storage shared by nfs, snapshot and geo-rep. The meta volume should be named as gluster_shared_storage, and it should be mounted at "/var/run/gluster/shared_storage/". Geo-rep will create a directory called 'geo-rep' in the meta-volume and all the lock files are created inside it. BUG: 1217939 Change-Id: I1d88798376d68340e2b2eff018c7e4f0121a608a Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/10196 Reviewed-on: http://review.gluster.org/10503 Tested-by: NetBSD Build System Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Aravinda VK <avishwan@redhat.com>
* geo-rep: Re Enable ignore_deletes OptionAravinda VK2015-05-031-2/+3
| | | | | | | | | | | | | | | | | | | | | | | If this option is set, Deletes will not be propogated to Slave. This option is applicable for UNLINK and RMDIR. gluster volume geo-replication <MASTER> <SLAVEHOST>::<SLAVEVOL> \ config ignore_deletes true Default value is false. PS: Use this option with caution, If you create the file in master with same path then it fails to sync to slave. Old file in Slave will have different GFID compared to New. BUG: 1217929 Change-Id: I1f7816d1ea36460a654873739d3fb1b6c13e0f8d Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/9583 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/10498 Tested-by: NetBSD Build System
* geo-rep: Add support for aclsKotresh HR2015-03-301-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | This patch adds support for ACLS. When it sees SETXATTR in Changelog, it adds the file to data queue. rsync/tar+ssh will take care of syncing ACLS. User set ACLS will be synced to Slave. This requires "system.posix_acl_access" to go through when client-pid is equal GF_CLIENT_PID_GSYNCD in fuse layer. New config interface is introduced, sync-acls Which can be set using geo-rep config(Default is True) gluster volume geo-replication <VOLUME> <SLAVEHOST>::<SLAVEVOL> \ config sync-acls false Change-Id: I7eb3523fa72b8fed830efc98138891244e830d65 BUG: 1187021 Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/10001 Reviewed-by: Aravinda VK <avishwan@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* geo-rep: Do not use xsync_upper_limit for change detectionAravinda VK2015-03-151-30/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Use register time(xsync_upper_limit) only for stime update, do not use for change detection. Problem 1: If a file created before geo-rep, xtime xattr does not exist. Geo-rep updates xtime of the file to current time if not exists. xtime > upper_limit so geo-rep will not pick those files. Changelog either will have SETXATTR, and fails to sync the file. Problem 2: If a file is created before geo-rep create and updated after geo-rep start. xtime of the file is greater than upper limit(geo-rep start time/changelog register time). Geo-rep(XSync) will not pick this file for syncing. Changelog will have only DATA recorded for that file. Geo-rep tries DATA without any ENTRY ops and fails with rsync error. BUG: 1200733 Change-Id: Ie4e8f284db689d2c755ef8e7ecbb658db1c0785f Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/9855 Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Saravanakumar Arumugam <sarumuga@redhat.com> Tested-by: Saravanakumar Arumugam <sarumuga@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* feature/geo-rep: Active Passive Switching logic flockKotresh HR2015-03-151-3/+36
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | CURRENT DESIGN AND ITS LIMITATIONS: ----------------------------------- Geo-replication syncs changes across geography using changelogs captured by changelog translator. Changelog translator sits on server side just above posix translator. Hence, in distributed replicated setup, both replica pairs collect changelogs w.r.t their bricks. Geo-replication syncs the changes using only one brick among the replica pair at a time, calling it as "ACTIVE" and other non syncing brick as "PASSIVE". Let's consider below example of distributed replicated setup where NODE-1 as b1 and its replicated brick b1r is in NODE-2 NODE-1 NODE-2 b1 b1r At the beginning, geo-replication chooses to sync changes from NODE-1:b1 and NODE-2:b1r will be "PASSIVE". The logic depends on virtual getxattr 'trusted.glusterfs.node-uuid' which always returns first up subvolume i.e., NODE-1. When NODE-1 goes down, the above xattr returns NODE-2 and that is made 'ACTIVE'. But when NODE-1 comes back again, the above xattr returns NODE-1 and it is made 'ACTIVE' again. So for a brief interval of time, if NODE-2 had not finished processing the changelog, both NODE-2 and NODE-1 will be ACTIVE causing rename race as mentioned in the bug. SOLUTION: --------- 1. Have a shared replicated storage, a glusterfs management volume specific to geo-replication. 2. Geo-rep creates a file per replica set on management volume. 3. fcntl lock on the above said file is used for synchronization between geo-rep workers belonging to same replica set. 4. If management volume is not configured, geo-replication will back to previous logic of using first up sub volume. Each worker tries to lock the file on shared storage, who ever wins will be ACTIVE. With this, we are able to solve the problem but there is an issue when the shared replicated storage goes down (when all replicas goes down). In that case, the lock state is lost. So AFR needs to rebuild the lock state after brick comes up. NOTE: ----- This patch brings in the, pre-requisite step of setting up management volume for geo-replication during creation. 1. Create mgmt-vol for geo-replicatoin and start it. Management volume should be part of master cluster and recommended to be three way replicated volume having each brick in different nodes for availability. 2. Create geo-rep session. 3. Configure mgmt-vol created with geo-replication session as follows. gluster vol geo-rep <mastervol> slavenode::<slavevol> config meta_volume \ <meta-vol-name> 4. Start geo-rep session. Backward Compatiability: ----------------------- If management volume is not configured, it falls back to previous logic of using node-uuid virtual xattr. But it is not recommended. Change-Id: I7319d2289516f534b69edd00c9d0db5a3725661a BUG: 1196632 Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/9759 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* geo-rep: Add support for xattrsAravinda VK2015-03-021-0/+5
| | | | | | | | | | | | | | | | | | | | | | This patch adds support for xattrs. When it sees SETXATTR in Changelog, it adds the file to data queue. rsync/tar+ssh will take care of syncing xattrs. User set xattrs will be synced to Slave. New config interface is introduced, sync-xattrs Which can be set using geo-rep config(Default is True) gluster volume geo-replication <VOLUME> <SLAVEHOST>::<SLAVEVOL> \ config sync-xattrs false Change-Id: I70626d854a0d616469dd54d61e5ef155ed8b67d8 BUG: 1196690 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/9499 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Saravanakumar Arumugam <sarumuga@redhat.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* geo-rep: Archive Changelogs and avoid generating empty XSync changelogsAravinda VK2015-02-191-5/+77
| | | | | | | | | | | | | | | With this patch, - Hybrid Crawl will not generate empty Changelogs - Archives Changelogs when processed(Hybrid(XSync), History, and Changelog Crawl - Passive worker cleans up its processing directory BUG: 1169331 Change-Id: I1383ffaed261cdf50da91b14260b4d43177657d1 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/9453 Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* geo-rep: Fixing the typo errorsarao2015-02-031-6/+6
| | | | | | | | | Change-Id: Iacc67e4ba9ac45e0858f3befe84ffb8fccf7e1c3 BUG: 1075417 Signed-off-by: arao <arao@redhat.com> Reviewed-on: http://review.gluster.org/9502 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Humble Devassy Chirammal <humble.devassy@gmail.com>
* geo-rep: Error handling in tar+ssh modeAravinda VK2014-12-291-12/+12
| | | | | | | | | | | | | | | | | | | | | | | Georep raises exception if tar+ssh fails and worker dies due to the exception. This patch adds resilience to tar+ssh error and geo-rep worker retries when error, and skips those changelogs after maximum retries.(same as rsync mode) Removed warning messages for each rsync/tar+ssh failure per GFID, since skipped list will be populated after Max retry. Retry changelog files log also available, hence warning message for each GFID is redundent. BUG: 1177527 Change-Id: I3019c5c1ada7fc0822e4b14831512d283755b1ea Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/9356 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* geo-rep: Sync atime, mtime to slave when SETATTRAravinda VK2014-12-251-3/+11
| | | | | | | | | | | | Existing geo-rep only syncs chown and chmod, with this patch geo-rep also syncs atime and mtime. Change-Id: Iea52d86682873bb4a47eeb0d325f5b9ddf2de2cf Signed-off-by: Aravinda VK <avishwan@redhat.com> BUG: 1176934 Reviewed-on: http://review.gluster.org/9331 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Venky Shankar <vshankar@redhat.com>
* feature/geo-rep: Fix skipped files logging.Kotresh H R2014-10-161-0/+10
| | | | | | | | | | | | | | | | When changelog is failed to process even after max retries, files are skipped but were not logging. This patch addresses this issue to log the skipped files. Change-Id: Ic617e4151231fe18a171efa57a119e21164e1a6a BUG: 1103577 Signed-off-by: Kotresh H R <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/7945 Reviewed-by: Aravinda VK <avishwan@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* geo-rep: fix same file different gfid in master and slaveAravinda VK2014-09-261-15/+35
| | | | | | | | | | | | | | | | | | | | | While processing RENAME in changelog, if the file is unlinked in master, then geo-rep was sending UNLINK to slave instead of RENAME. If rsync job fails if one of the file failed to sync in the job. This patch adds logic to remove GFID from data list if the same changelog has UNLINK entry for it after the DATA. Or it removes those GFIDs during retry of changelogs processing. BUG: 1143853 Change-Id: I982dc976397cd0ab676bb912583f66a28f821926 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/8761 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Jeff Darcy <jdarcy@redhat.com> Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* geo-rep: Fixing issue with xsync upper limitAravinda VK2014-08-191-39/+34
| | | | | | | | | | | | | | | | | | | | While identifying the file/dir to sync, xtime of the file was compared with xsync_upper_limit as `xtime < xsync_upper_limit` After the sync, xtime of parent directory is updated as stime. With the upper limit condition, stime is updated as MIN(xtime_parent, xsync_upper_limit) With this files will get missed if `xtime_of_file == xsync_upper_limit` With this patch xtime_of_file is compared as xtime_of_file <= xsync_upper_limit BUG: 1128093 Change-Id: I5022ce67ba503ed4621531a649a16fc06b2229d9 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/8439 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* geo-rep: minimize xsync crawl usage and set upper limit to xsync crawlAravinda VK2014-07-231-19/+82
| | | | | | | | | | | | | | | | | | | | | | | | | | | | For effective handling of deletes and renames use history crawl as much as possible. History crawl will run in loop till it syncs all data before live changelog time. When it uses xsync crawl(fallback when changelog not available, or very first crawl) it sets upper limit to crawl. After completing History crawl, it checks actual end time returned by history api to compare with register time, if actual end is less than register time then run history crawl one more time. If first turn history processing time is less than the CHANGELOG ROLLOVER TIME then sleep for the difference, After sleep if it is guaranteed that rollover will happen and switches to live changelog consumption without switching to xsync. This sleep is only when history processing completed < CHANGELOG_ROLLOVER_TIME and sleep only after the first turn, So will not affect the performance. BUG: 1112238 Change-Id: I644ef24b07e42e81cec96a025ebd21244a555ec0 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/8151 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* geo-rep: Avoid duplicate stat in xsync changelog processingAravinda VK2014-07-091-6/+24
| | | | | | | | | | | | | | | | | | | | When A file/dir is identified for metadata sync, it was doing duplicate stat to get the metadata to sync. With this patch it avoids doing one additional stat call. Xsync performance will improve. rsync will copy files metadata, so no need to include for processing. BUG: 1111490 Change-Id: I79dad6375fa4742d9aaca7d9856993c184a744dc Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/8124 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* geo-rep: History Change detector method select issueAravinda VK2014-07-011-1/+2
| | | | | | | | | | | | | | | | | Geo-rep does history crawl even if change-detector is set to xsync. This patch fixes by taking priority to user configured change detector. BUG: 1113525 Change-Id: Ic6c34e187c9cb6608c9ef8a010ea07015ba60a80 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/8183 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* geo-rep: Making replica failover check interval configurableAravinda VK2014-06-111-1/+1
| | | | | | | | | | | | | | | | | | Replica failover check interval is hardcoded to 60 sec by default. Now this option is made configurable and defaulted to 1 sec. To change the default value gluster volume geo-replication <MASTERVOL> \ <SLAVEHOST>::<SLAVEVOL> config replica_failover_interval 15 Change-Id: Iada1b80d510452dcfedebd8a21bebd62394b0597 BUG: 1066410 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/8003 Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* gsyncd / geo-rep: FSH recommended log locationsVenky Shankar2014-06-101-5/+6
| | | | | | | | | | | | | | | | Upgrading "working_dir" on the fly is a bit unclean yet (though it works) as currently config upgrade does not support "old" values to be expanded by using configuration variables. Change-Id: I44ed65c281f2e0ce3b6b467addc5c1c88ac674e7 BUG: 1077516 Signed-off-by: Venky Shankar <vshankar@redhat.com> Signed-off-by: Kotresh H R <khiremat@redhat.com> Signed-off-by: Aravinda VK <avishwan@redhat.com> Signed-off-by: Ajeet Jha <ajha@redhat.com> Reviewed-on: http://review.gluster.org/7375 Tested-by: Gluster Build System <jenkins@build.gluster.com>
* gsyncd / geo-rep: Xsync crawl metadata synchronizationVenky Shankar2014-06-101-0/+2
| | | | | | | | | | | | | Added "metadata" record for directory and file creations during the intial crawl. Change-Id: I811ae26e0144cadf7249cb64541ec354ab83fe66 BUG: 1106604 Signed-off-by: Venky Shankar <vshankar@redhat.com> Reviewed-on: http://review.gluster.org/8018 Reviewed-by: Aravinda VK <avishwan@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* geo-rep: Changelog History API changesAravinda VK2014-05-131-6/+7
| | | | | | | | | | | | | | | Additional argument added to API gf_history_changelog, actual_end - The end time till where changelogs are available. Added sort to history_get_changes API output. BUG: 1091961 Change-Id: Id043409882a83cd0a7b9adc3d34d5147d17e532e Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/7747 Reviewed-by: ajeet jha <ajha@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* geo-rep: Pause and Resume feature for geo-replicationAravinda VK2014-05-091-16/+18
| | | | | | | | | | | | | | | | Changelog consumption/processing now happens in seperate process group than monitor. When monitor process group gets SIGSTOP all worker process, ssh, rsync will be paused except the changelog processing. When it gets SIGCONT it resumes its operation. Changelog agent runs as RepceServer, geo-rep worker communicates with changelog agent using RepceClient. Change-Id: I35c333e4d8b13d03a7808aed601960eef23cfa04 BUG: 1093602 Signed-off-by: Venky Shankar <vshankar@redhat.com> Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/7322
* geo-rep: Changelog History consumption more fixesAravinda VK2014-05-091-8/+2
| | | | | | | | | | | Number of parallel threads to process changelog history is made configurable via sync_jobs Change-Id: Idcd8e655d9df540cfa48648b9e98af941f95e9d0 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/7660 Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* geo-rep: Consume Changelog History APIAravinda VK2014-04-301-18/+75
| | | | | | | | | | | | | | | | | | | | | | | | Every time when geo-rep restarts it first does FS crawl using XCrawl and then switches to Changelog Mode. This is because changelog only had live API, that is we can get changes only after registering. Now this(http://review.gluster.org/#/c/6930/) patch introduces History API for changelogs. If history is available then geo-rep will use it instead of FS Crawl. History API returns TS till what time history is available for given start and end time. If TS < endtime then switch to FS Crawl. (History => FS Crawl => Live Changelog) If TS >= endtime, then switch directly to Changelog mode (History => Live Changelog) Change-Id: I4922f62b9f899c40643bd35720e0c81c36b2f255 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/6938 Reviewed-by: Venky Shankar <vshankar@redhat.com> Reviewed-by: Humble Devassy Chirammal <humble.devassy@gmail.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
generated by cgit (git 2.34.1) at 2024-04-24 08:16:30 +0000