summaryrefslogtreecommitdiffstats
path: root/geo-replication
Commit message (Collapse)AuthorAgeFilesLines
* geo-rep: Fix py2/py3 compatibility in repceKotresh HR2019-12-241-3/+2
| | | | | | | | | | | | | | | | | Geo-rep fails to start on python2 only machine like centos6. It fails with "ImportError no module named _io". This patch fixes the same. Backport of: > Patch: https://review.gluster.org//23702/ > BUG: 1771577 > Change-Id: I8228458a853a230546f9faf29a0e9e0f23b3efec > Signed-off-by: Kotresh HR <khiremat@redhat.com> (cherry picked from commit 9595ecca3de49fdf37d30b151f5c3e071e0a80d0) fixes: bz#1771842 Change-Id: I8228458a853a230546f9faf29a0e9e0f23b3efec Signed-off-by: Kotresh HR <khiremat@redhat.com>
* libgfchangelog : use find_library to locate shared librarySunny Kumar2019-11-081-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Issue: libgfchangelog.so: cannot open shared object file Due to hardcoded shared library name runtime loader looks for particular version of a shared library. Solution: Using find_library to locate shared library at runtime solves this issue. Traceback (most recent call last): File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 323, in main func(args) File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 82, in subcmd_worker local.service_loop(remote) File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1261, in service_loop changelog_agent.init() File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 233, in __call__ return self.ins(self.meth, *a) File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 215, in __call__ raise res OSError: libgfchangelog.so: cannot open shared object file: No such file or directory Backport of: > Patch: https://review.gluster.org/22557 > Change-Id: I3dd013d701ed1cd99ba7ef20d1898f343e1db8f5 > BUG: 1699394 > Signed-off-by: Sunny Kumar <sunkumar@redhat.com> (cherry picked from commit f316c8b797283818bd800569771870a4b9bf1310) Change-Id: I3dd013d701ed1cd99ba7ef20d1898f343e1db8f5 fixes: bz#1770100 Signed-off-by: Sunny Kumar <sunkumar@redhat.com>
* geo-rep: Fix Permission denied traceback on non root setupKotresh HR2019-10-242-10/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Problem: While syncing rename of directory in hybrid crawl, geo-rep crashes as below. Traceback (most recent call last): File "/usr/local/libexec/glusterfs/python/syncdaemon/repce.py", line 118, in worker res = getattr(self.obj, rmeth)(*in_data[2:]) File "/usr/local/libexec/glusterfs/python/syncdaemon/resource.py", line 588, in entry_ops src_entry = get_slv_dir_path(slv_host, slv_volume, gfid) File "/usr/local/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 687, in get_slv_dir_path [ENOENT], [ESTALE]) File "/usr/local/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 546, in errno_wrap return call(*arg) PermissionError: [Errno 13] Permission denied: '/bricks/brick1/b1/.glusterfs/8e/c0/8ec0fcd4-d50f-4a6e-b473-a7943ab66640' Cause: Conversion of gfid to path for a directory uses readlink on backend .glusterfs gfid path. But this fails for non root user with permission denied. Fix: Use gfid2path interface to get the path from gfid Backport of: > Patch: https://review.gluster.org/23570 > Change-Id: I9d40c713a1b32cea95144cbc0f384ada82972222 > BUG: 1763439 > Signed-off-by: Kotresh HR <khiremat@redhat.com> Change-Id: I9d40c713a1b32cea95144cbc0f384ada82972222 fixes: bz#1764183 Signed-off-by: Kotresh HR <khiremat@redhat.com>
* geo-rep: Fix config upgrade on non-participating nodeKotresh HR2019-10-221-1/+2
| | | | | | | | | | | | | | | | | | | After upgrade, if the config files are of old format, it gets migrated to new format. Monitor process migrates it. Since monitor doesn't run on nodes where bricks are not hosted, it doesn't get migrated there. So this patch fixes the config upgrade on nodes which doesn't host bricks. This happens during config either on get/set/reset. Backport of: > Patch: https://review.gluster.org/23555 > Change-Id: Ibade2f2310b0f3affea21a3baa1ae0eb71162cba > Signed-off-by: Kotresh HR <khiremat@redhat.com> > BUG: 1762220 Change-Id: Ibade2f2310b0f3affea21a3baa1ae0eb71162cba fixes: bz#1763028 Signed-off-by: Kotresh HR <khiremat@redhat.com>
* geo-rep: Fix the name of changelog archive fileKotresh HR2019-10-222-8/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Background: The processed changelogs are archived each month in a single tar file. The default format is "archive_YYYYMM.tar" which is specified as "%%Y%%m" in configuration file. Problem: The created changelog archive file didn't have corresponding year and month. It created as "archive_%Y%m.tar" on python2 only systems. Cause and Fix: Geo-rep expects "%Y%m" after the ConfigParser reads it from config file. Since it was "%%Y%%m" in config file, geo-rep used to get correct value "%Y%m" in python3 and "%%Y%%m" in python2 which is incorrect. The fix can be to use "%Y%m" in config file but that fails in python3. So the fix is to use "RawConfigParser" in geo-rep and use "%Y%m". This works both in python2 and python3. Backport of: > Patch: https://review.gluster.org/23248 > Change-Id: Ie5b7d2bc04d0d53cd1769e064c2d67aaf95d557c > BUG: 1741890 > Signed-off-by: Kotresh HR <khiremat@redhat.com> Change-Id: Ie5b7d2bc04d0d53cd1769e064c2d67aaf95d557c fixes: bz#1764176 Signed-off-by: Kotresh HR <khiremat@redhat.com>
* geo-rep: Fix Config Get RaceAravinda VK2019-10-221-6/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | When two threads(sync jobs) in Geo-rep worker calls `gconf.get` and `gconf.getr`(realtime) at the sametime, `getr` resets the conf object and other one gets None. Thread Lock is introduced to fix the issue. ``` File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 368, in twrap tf(*aargs) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1987, in syncjob po = self.sync_engine(pb, self.log_err) File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1444, in rsync rconf.ssh_ctl_args + \ AttributeError: 'NoneType' object has no attribute 'split' ``` Backport of: > Patch: https://review.gluster.org/23158 > Change-Id: I9c245e5c36338265354e158f5baa32b119eb2da5 > BUG: 1737484 > Signed-off-by: Aravinda VK <avishwan@redhat.com> Change-Id: I9c245e5c36338265354e158f5baa32b119eb2da5 fixes: bz#1764174 Signed-off-by: Kotresh HR <khiremat@redhat.com>
* geo-rep: Test case for upgrading config fileShwetha K Acharya2019-10-221-6/+5
| | | | | | | | | | | | | | | | | | Added test case for the patch https://review.gluster.org/#/c/glusterfs/+/22894/4 Also updated if else structure in gsyncdconfig.py to avoid repeated occurance of values in new configfile. Backport of: > Patch: https://review.gluster.org/22982 > BUG: 1707731 > Change-Id: If97e1d37ac52dbd17d47be6cb659fc5a3ccab6d7 > Signed-off-by: Shwetha K Acharya <sacharya@redhat.com> fixes: bz#1764171 Change-Id: If97e1d37ac52dbd17d47be6cb659fc5a3ccab6d7 Signed-off-by: Kotresh HR <khiremat@redhat.com>
* geo-rep : fix gluster command path for non-root sessionSunny Kumar2019-10-222-5/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | Problem: gluster command not found. Cause: In Volinfo class we issue command 'gluster vol info' to get information about volume like getting brick_root to perform various operation. When geo-rep session is configured for non-root user Volinfo class fails to issue gluster command due to unavailability of gluster binary path for non-root user. Solution: Use config value 'slave-gluster-command-dir'/'gluster-command-dir' to get path for gluster command based on caller. Backport of: > Patch: https://review.gluster.org/22920 > BUG: 1722740 > Change-Id: I4ec46373da01f5d00ecd160c4e8c6239da8b3859 > Signed-off-by: Sunny Kumar <sunkumar@redhat.com> fixes: bz#1764172 Change-Id: I4ec46373da01f5d00ecd160c4e8c6239da8b3859 Signed-off-by: Kotresh HR <khiremat@redhat.com>
* geo-rep: Upgrading config file to new versionShwetha K Acharya2019-10-222-0/+46
| | | | | | | | | | | | | | | | | | | | | | | | | | - configuration handling is enhanced with patch https://review.gluster.org/#/c/glusterfs/+/18257/ - hence, the old configurations are not applied when Geo-rep session is created in the old version and upgraded. This patch solves the issue. It, - checks if the config file is old. - parses required values from old config file and stores in new config file, which ensures that configerations are applied on upgrade. - stores old config file as backup. - handles changes in options introduced in https://review.gluster.org/#/c/glusterfs/+/18257/ Backport of: > Patch: https://review.gluster.org/22894 > BUG: bz#1707731 > Change-Id: Iad8da6c1e1ae8ecf7c84dfdf8ea3ac6966d8a2a0 > Signed-off-by: Shwetha K Acharya <sacharya@redhat.com> updates: bz#1764171 Change-Id: Iad8da6c1e1ae8ecf7c84dfdf8ea3ac6966d8a2a0 Signed-off-by: Kotresh HR <khiremat@redhat.com>
* geo-rep : fix mountbroker setupSunny Kumar2019-09-271-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Problem: Unable to setup mountbroker root directory while creating geo-replication session for non-root user. Casue: With patch[1] which defines the max-port for glusterd one extra sapce got added in field of 'option max-port'. [1]. https://review.gluster.org/#/c/glusterfs/+/21872/ In geo-rep spliting of key-value pair form vol file was done on the basis of space so this additional space caused "ValueError: too many values to unpack". Solution: Use split so that it can treat consecutive whitespace as a single separator. Backport of https://review.gluster.org/#/c/glusterfs/+/22716/ >Fixes: bz#1709248 >Change-Id: Ia22070a43f95d66d84cb35487f23f9ee58b68c73 >Signed-off-by: Sunny Kumar <sunkumar@redhat.com> >(cherry picked from commit 3dd03146bb7037ae2ebea0579d0b81be27fdd927) Fixes: bz#1750228 Signed-off-by: Sunny Kumar <sunkumar@redhat.com> Change-Id: Ia22070a43f95d66d84cb35487f23f9ee58b68c73
* geo-rep: Fix mount broker setup issueKotresh HR2019-08-071-3/+6
| | | | | | | | | | | | | | | | | | | | | | Even the use builtin 'type' command as in patch [1] causes issues if argument in question is not part of PATH environment variable for that user. This patch fixes the same by doing source /etc/profile. This was already being used in another part of script. [1] https://review.gluster.org/23089 Backport of: > Patch: https://review.gluster.org/23136/ > Change-Id: Iceb78835967ec6a4350983eec9af28398410c002 > BUG: 1734738 > Signed-off-by: Kotresh HR <khiremat@redhat.com> (cherry picked from commit 84f7794547522463841068063b22fd3a8d8fca2b) Change-Id: Iceb78835967ec6a4350983eec9af28398410c002 fixes: bz#1737712 Signed-off-by: Kotresh HR <khiremat@redhat.com>
* geo-rep: Fix mount broker setup issueKotresh HR2019-08-051-3/+3
| | | | | | | | | | | | | | | | | | | | | The patch [1] added validation in gverify.sh to check if the gluster binary exists on slave by executing gluster directly on slave. But for non-root users, even though gluster binary is present, path is not found when executed via ssh. Hence validate the gluster binary using bash builtin 'type' command. [1] https://review.gluster.org/19224 Backport of: > Patch: https://review.gluster.org/23089/ > Change-Id: I93ca62c0c5b1e16263e586ddbbca8407d60ca126 > BUG: 1731920 > Signed-off-by: Kotresh HR <khiremat@redhat.com> (cherry picked from commit 2aa731a259ea457c07494e3c3edf6d5f7c02fe77) Change-Id: I93ca62c0c5b1e16263e586ddbbca8407d60ca126 fixes: bz#1733880 Signed-off-by: Kotresh HR <khiremat@redhat.com>
* geo-rep: Convert gfid conflict resolutiong logs into debugKotresh HR2019-05-211-9/+12
| | | | | | | | | | | | | | | | | | The gfid conflict resolution code path is not supposed to hit in generic code path. But few of the heavy rename workload (BUG: 1694820) makes it a generic case. So logging the entries to be fixed as INFO floods the log in these particular workloads. Hence convert them to DEBUG. Backport of: > Patch: https://review.gluster.org/22720 > BUG: 1709653 > Change-Id: I4d5e102b87be5fe5b54f78f329e588882d72b9d9 > Signed-off-by: Kotresh HR <khiremat@redhat.com> fixes: bz#1712223 Change-Id: I4d5e102b87be5fe5b54f78f329e588882d72b9d9 Signed-off-by: Kotresh HR <khiremat@redhat.com>
* geo-rep: Fix sync hang with tarsshKotresh HR2019-05-211-4/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Problem: Geo-rep sync hangs when tarssh is used as sync engine at heavy workload. Analysis and Root cause: It's found out that the tar process was hung. When debugged further, it's found out that stderr buffer of tar process on master was full i.e., 64k. When the buffer was copied to a file from /proc/pid/fd/2, the hang is resolved. This can happen when files picked by tar process to sync doesn't exist on master anymore. If this count increases around 1k, the stderr buffer is filled up. Fix: The tar process is executed using Popen with stderr as PIPE. The final execution is something like below. tar | ssh <args> root@slave tar --overwrite -xf - -C <path> It was waiting on ssh process first using communicate() and then tar. Note that communicate() reads stdout and stderr. So when stderr of tar process is filled up, there is no one to read until untar via ssh is completed. This can't happen and leads to deadlock. Hence we should be waiting on both process parallely, so that stderr is read on both processes. Backport of: > Patch: https://review.gluster.org/22684/ > Change-Id: I609c7cc5c07e210c504771115b4d551a2e891adf > BUG: 1707728 > Signed-off-by: Kotresh HR <khiremat@redhat.com> Change-Id: I609c7cc5c07e210c504771115b4d551a2e891adf fixes: bz#1709738 Signed-off-by: Kotresh HR <khiremat@redhat.com>
* geo-rep: Fix sync-method configKotresh HR2019-05-172-7/+9
| | | | | | | | | | | | | | | | | | | | | | | | Problem: When 'use_tarssh' is set to true, it exits with successful message but the default 'rsync' was used as sync-engine. The new config 'sync-method' is not allowed to set from cli. Analysis and Fix: The 'use_tarssh' config is deprecated with new config framework and 'sync-method' is the new config to choose sync-method i.e. tarssh or rsync. This patch fixes the 'sync-method' config. The allowed values are tarssh and rsync. Backport of: > Patch: https://review.gluster.org/22683 > Change-Id: I0edb0319cad0455b29e49f2f08a64ce324735e84 > BUG: 1707686 > Signed-off-by: Kotresh HR <khiremat@redhat.com> Change-Id: I0edb0319cad0455b29e49f2f08a64ce324735e84 fixes: bz#1709737 Signed-off-by: Kotresh HR <khiremat@redhat.com>
* geo-rep: Fix rename with existing destination with same gfidSunny Kumar2019-05-172-5/+46
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Problem: Geo-rep fails to sync the rename properly if destination exists. It results in source to be remained on slave causing more number of files on slave. Also heavy rename workload like logrotate caused lot of ESTALE errors Cause: Geo-rep fails to sync rename if destination exists if creation of source file also falls into single batch of changelogs being processed. This is because, after fixing problematic gfids verifying from master, while re-processing original entries, CREATE also was re-processed causing more files on slave and rename to be failed. Solution: Entries need to be removed from retrial list after fixing problematic gfids on slave so that it's not re-created again on slave. Also treat ESTALE as EEXIST so that the error is properly handled verifying the op on master volume. Backport of: > Patch: https://review.gluster.org/22519/ > Change-Id: I50cf289e06b997adddff0552bf2466d9201dd1f9 > BUG: 1694820 > Signed-off-by: Kotresh HR <khiremat@redhat.com> > Signed-off-by: Sunny Kumar <sunkumar@redhat.com> Change-Id: I50cf289e06b997adddff0552bf2466d9201dd1f9 fixes: bz#1709734 Signed-off-by: Kotresh HR <khiremat@redhat.com>
* geo-rep: Fix entries and metadata counters in geo-rep statusKotresh HR2019-05-171-3/+2
| | | | | | | | | | | | | | | | | Entries counter was incremented twice and decremented only once. And entries count was being used in place of metadata entries. This patch fixes both of them. Backport of: > Patch: https://review.gluster.org/22603 > BUG: 1512093 > Change-Id: I5601a5fe8d25c9d65b72eb529171e7117ebbb67f > Signed-off-by: Kotresh HR <khiremat@redhat.com> (cherry picked from commit e0a6941af6ed352911698012ada895d1296b549e) fixes: bz#1709685 Change-Id: I5601a5fe8d25c9d65b72eb529171e7117ebbb67f Signed-off-by: Kotresh HR <khiremat@redhat.com>
* geo-rep: fix integer config validationAravinda VK2019-04-171-5/+18
| | | | | | | | | | | | ssh-port validation is mentioned as `validation=int` in template `gsyncd.conf`, but not handled this during geo-rep config set. Backport of https://review.gluster.org/22418 Fixes: bz#1695445 Change-Id: I3f19d9b471b0a3327e4d094dfbefcc58ed2c34f6 Signed-off-by: Aravinda VK <avishwan@redhat.com> (cherry picked from commit c574984e19d59e351372eacce0ce11fb36e96dd4)
* geo-rep: IPv6 supportAravinda VK2019-04-174-7/+38
| | | | | | | | | | | | | | | | `address_family=inet6` needs to be added while mounting master and slave volumes in gverify script. New option introduced to gluster cli(`--inet6`) which will be used internally by geo-rep while calling `gluster volume info --remote-host=<ipv6>`. Backport of https://review.gluster.org/22363 Fixes: bz#1695436 Change-Id: I1e0d42cae07158df043e64a2f991882d8c897837 Signed-off-by: Aravinda VK <avishwan@redhat.com> (cherry picked from commit 240e1d6821fbb779c3dd73f6f0225d755a5b7cc6)
* geo-rep: Fix syncing multiple rename of symlinkKotresh HR2019-04-031-9/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Problem: Geo-rep fails to sync rename of symlink if it's renamed multiple times if creation and rename happened successively Worker crash at slave: Traceback (most recent call last): File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", in worker res = getattr(self.obj, rmeth)(*in_data[2:]) File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", in entry_ops [ESTALE, EINVAL, EBUSY]) File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", in errno_wrap return call(*arg) File "/usr/libexec/glusterfs/python/syncdaemon/libcxattr.py", in lsetxattr cls.raise_oserr() File "/usr/libexec/glusterfs/python/syncdaemon/libcxattr.py", in raise_oserr raise OSError(errn, os.strerror(errn)) OSError: [Errno 12] Cannot allocate memory Geo-rep Behaviour: 1. SYMLINK doesn't record target path in changelog. So while syncing SYMLINK, readlink is done on master to get target path. 2. Geo-rep will create destination if source is not present while syncing RENAME. Hence while syncing RENAME of SYMLINK, target path is collected from destination. Cause: If symlink is created and renamed multiple times, creation of symlink is ignored, as it's no longer present on master at that path. While symlink is renamed multiple times at master, when syncing first RENAME of SYMLINK, both source and destination is not present, hence target path is not known. In this case, while creating destination directly at slave, regular file attributes were encoded into blob instead of symlink, causing failure in gfid-access translator while decoding blob. Solution: While syncing of RENAME of SYMLINK, when target is not known and when src and destination is not present on the master, don't create destination. Ignore the rename. It's ok to ignore. If it's unliked, it's fine. If it's renamed to something else, it will be synced then. Backport of: > Change-Id: Ibdfa495513b7c05b5370ab0b89c69a6802338d87 > BUG: bz#1693648 > Signed-off-by: Kotresh HR <khiremat@redhat.com> (cherry picked from commit 877af725b3e35b548d6d7aeec5adb21721d8bf8b) Change-Id: Ibdfa495513b7c05b5370ab0b89c69a6802338d87 fixes: bz#1694002 Signed-off-by: Kotresh HR <khiremat@redhat.com> (cherry picked from commit 877af725b3e35b548d6d7aeec5adb21721d8bf8b)
* eventsapi: Fix Python3 compatibility issuesAravinda VK2019-02-261-5/+5
| | | | | | | | | | | | - Fixed Relative import and non-package import related issues. - socketserver import issues fix - Renamed installed directory name to `gfevents` from `events`(To avoid any issues with other global libs) Fixes: bz#1683008 Change-Id: I3dc38bc92b23387a6dfbcc0ab8283178235bf756 Signed-off-by: Aravinda VK <avishwan@redhat.com> (cherry picked from commit cd68f7b88b9a2c9a4e4ff9fca61517384e54130a)
* geo-rep: Fix configparser import issueKotresh HR2019-02-051-2/+2
| | | | | | | | | | | | | | | 'configparser' is backported to python2 and can be installed using pip (pip install configparser). So trying to import 'configparser' first and later 'ConfigParser' can cause issues w.r.t unicode strings. Always try importing 'ConfigParser' first and then 'configparser'. This solves python2/python3 compat issues. Change-Id: I2a87c3fc46476296b8cb547338f35723518751cc fixes: bz#1671637 Signed-off-by: Kotresh HR <khiremat@redhat.com>
* geo-rep : fix rename sync on hybrid crawlSunny Kumar2019-01-172-13/+11
| | | | | | | | | | | | | | Problem: When geo-rep is configured as hybrid crawl directory renames are not synced to the slave. Solution: Rename sync of directory was failing due to incorrect destination path calculation. During check for existence on slave we miscalculated realpath. <host:brickpath/dir>. Change-Id: I23f1ea60e86a917598fe869d5d24f8da654d8a0a fixes: bz#1665826 Signed-off-by: Sunny Kumar <sunkumar@redhat.com>
* geo-rep: Make slave volume read-only (by default)Harpreet Kaur2018-12-071-0/+1
| | | | | | | | | | | | | | | Added a command to set "features.read-only" option to a default value "on" for slave volume. Changes are made in: $SRC//extras/hook-scripts/S56glusterd-geo-rep-create-post.sh for root geo-rep and $SRC/geo-replication/src/set_geo_rep_pem_keys.sh for non-root geo-rep. Fixes: bz#1654187 Change-Id: I15beeae3506f3f6b1dcba0a5c50b6344fd468c7c Signed-off-by: Harpreet Kaur <hlalwani@redhat.com>
* libglusterfs: Move devel headers under glusterfs directoryShyamsundarR2018-12-052-9/+9
| | | | | | | | | | | | | | | | | | | | | | | | libglusterfs devel package headers are referenced in code using include semantics for a program, this while it works can be better especially when dealing with out of tree xlator builds or in general out of tree devel package usage. Towards this, the following changes are done, - moved all devel headers under a glusterfs directory - Included these headers using system header notation <> in all code outside of libglusterfs - Included these headers using own program notation "" within libglusterfs This change although big, is just moving around the headers and making it correct when including these headers from other sources. This helps us correctly include libglusterfs includes without namespace conflicts. Change-Id: Id2a98854e671a7ee5d73be44da5ba1a74252423b Updates: bz#1193929 Signed-off-by: ShyamsundarR <srangana@redhat.com>
* geo-rep: Fix syncing of files with non-ascii filenamesKotresh HR2018-12-044-68/+174
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Problem: Creation of files/directories with non-ascii names fails to sync to the slave. It crashes with below traceback on slave. ... File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/repce.py", line 118, in worker res = getattr(self.obj, rmeth)(*in_data[2:]) File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py", line 709, in entry_ops [ESTALE, EINVAL, EBUSY]) File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/syncdutils.py", line 546, in errno_wrap return call(*arg) File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/libcxattr.py", line 83, in lsetxattr cls.raise_oserr() File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/libcxattr.py", line 38, in raise_oserr raise OSError(errn, os.strerror(errn)) OSError: [Errno 12] Cannot allocate memory Cause: The length calculation arguments passed to blob creation was done before encoding. Hence was failing in gfid-access layer. Fix: It appears that the calculating lenght properly fixes this issue. But it will cause issues in other places in 'python2' and not in 'python3'. So encoding and decoding each required string to make geo-rep compatible with both 'python2' and 'python3' is a nightmare and is not fool proof. Hence kept 'python2' code as is with out encode/decode and applied encode/decode only to 'python3' Added non-ascii filename tests to regression fixes: bz#1650893 Change-Id: I35cfaf848e07b1a0b5cb93c01b98b472f08271a6 Signed-off-by: Kotresh HR <khiremat@redhat.com>
* geo-rep: Fix permissions with non-root setupKotresh HR2018-11-261-3/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Problem: In non-root fail-over/fail-back(FO/FB), when slave is promoted as master, the session goes to 'Faulty' Cause: The command 'gluster-mountbroker <mountbroker-root> <group>' is run as a pre-requisite on slave in non-root setup. It modifies the permission and group of following required directories and files recursively [1] /var/lib/glusterd/geo-replication [2] /var/log/glusterfs/geo-replication-slaves In a normal setup, this is executed on slave node and hence doing it recursively is not an issue on [1]. But when original master becomes slave in non-root during FO/FB, it contains ssh public keys and modifying permissions on them causes geo-rep to fail with incorrect permissions. Fix: Don't do permission change recursively. Fix permissions for required files. fixes: bz#1651498 Change-Id: I68a744644842e3b00abc26c95c06f123aa78361d Signed-off-by: Kotresh HR <khiremat@redhat.com>
* geo-rep: Fix traceback with symlink metadata syncKotresh HR2018-11-061-15/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | While syncing metadata, 'os.chmod', 'os.chown', 'os.utime' should be used without de-reference. But python supports only 'os.chown' without de-reference. That's mostly because Linux doesn't support 'chmod' on symlink file itself but it does support 'chown'. So while syncing metadata ops, if it's symlink we should only sync 'chown' and not do 'chmod' and 'utime'. It will lead to tracebacks with errors like EROFS, EPERM, ACCESS, ENOENT. All the three errors (EPERM, ACCESS, ENOENT) were handled except EROFS. But the way it was handled was not fool proof. The operation is tried and failure was handled based on the errors. All the errors with symlink file for 'chown', 'utime' had to be passed to safe errors list of 'errno_wrap'. This patch handles it better by avoiding 'chmod' and 'utime' if it's symlink file. fixes: bz#1646104 Change-Id: Ic354206455cdc7ab2a87d741d81f4efe1f19d77d Signed-off-by: Kotresh HR <khiremat@redhat.com>
* geo-rep: Add more intelligence to automatic error handlingKotresh HR2018-10-301-24/+46
| | | | | | | | | | | | | Geo-rep's automatic error handling does gfid conflict resolution. But if there are ENOENT errors because the parent is not synced to slave, it doesn' handle them. This patch adds the intelligence to create missing parent directories on slave. It can create the missing directories upto the depth of 10. fixes: bz#1643402 Change-Id: Ic97ed1fa5899c087e404d559e04f7963ed7bb54c Signed-off-by: Kotresh HR <khiremat@redhat.com>
* geo-rep/scripts: Fix traceback in gluster-mountbrokerKotresh HR2018-10-301-1/+1
| | | | | | | | | | When 'gluster-mountbroker status' was issued, it crashes in a corner case with 'str object has not attribute get'. Fixed the same. fixes: bz#1643929 Signed-off-by: Kotresh HR <khiremat@redhat.com> Change-Id: Iaf1a937ed0136b3b2058230c75fa89a215d8a5eb
* georep: python2 to python3 compat - schedulerKotresh HR2018-10-301-1/+1
| | | | | | | | | 1. scheduler - Popen 2. syncdutils - corner case on failure fixes: bz#1643932 Change-Id: I65af97a244a8790e976acedc2728db6ebbf2ae10 Signed-off-by: Kotresh HR <khiremat@redhat.com>
* geo-rep: Fix issue in gfid-conflict-resolutionKotresh HR2018-10-261-17/+24
| | | | | | | | | | | | | | | | | | | | | | | | | Problem: During gfid-conflict-resolution, geo-rep crashes with 'ValueError: list.remove(x): x not in list' Cause and Analysis: During gfid-conflict-resolution, the entry blob is passed back to master along with additional information to verify it's integrity. If everything looks fine, the entry creation is ignored and is deleted from the original list. But it is crashing during removal of entry from the list saying entry not in list. The reason is that the stat information in the entry blob was modified and sent back to master if present. Fix: Send back the correct stat information for gfid-conflict-resolution. fixes: bz#1642865 Change-Id: I47a6aa60b2a495465aa9314eebcb4085f0b1c4fd Signed-off-by: Kotresh HR <khiremat@redhat.com>
* geo-rep : fix high sev coverity isuueSunny Kumar2018-10-251-1/+0
| | | | | | | | This patch fixes CID: 1382374: USE_AFTER_FREE. Change-Id: If408f52ee291312fb83095126ebd6bb79ae95e26 updates: bz#789278 Signed-off-by: Sunny Kumar <sunkumar@redhat.com>
* georep: python2 to python3 compat - syscallsKotresh HR2018-10-087-35/+88
| | | | | | | | | | | | 1. ctypes/syscalls A) arguments is expected to be encoded B) Raw conversion of return value from bytearray into string 2. struct pack/unpack - Raw converstion of string to bytearray 3. basestring -> str Updates: #411 Change-Id: I80f939adcdec0ed0022c87c0b76d057ad5559e5a Signed-off-by: Kotresh HR <khiremat@redhat.com>
* python: remove shebangs of non-main scripts and make others executableNiels de Vos2018-10-025-4/+0
| | | | | | | | | | | | | | | Some of the scripts that have a #!/usr/bin/python3 shebang do not have a main() like function. These scripts will not get executed but only imported. They do not need the shebang. A few others are not installed with 'make install', but do have a main() like function. These scripts are expected to be used by developers for different tasks (mostly code generation). Marking these scripts executable to make it easier to identify them. Change-Id: I73541471deb7e0830766b804786244e73dfe4221 Updates: #411 Signed-off-by: Niels de Vos <ndevos@redhat.com>
* georep: python2 to python3 compat - pickleKotresh HR2018-10-021-4/+12
| | | | | | | | | | | | | | | | | | | | | | Handle py2 and py3 compatibility for pickling and unpickling. Geo-rep pickles and unpickles reading sys.stdin and sys.stdout streams. py2 and py3 compatibility expects the streams to be opened in binary mode but the sys.stdout objects are different in python2 and python3 python2: >>> type(sys.stdout) <type 'file'> python3: >>> type(sys.stdout) <class '_io.TextIOWrapper'> So in order to access binary stream, using sys.stdin.buffer in python3 Updates: #411 Change-Id: I1a633ccdddff5baf0cf05a8b493add39ddf75bd7 Signed-off-by: Kotresh HR <khiremat@redhat.com>
* georep: python2 and python3 compat - bytes-strKotresh HR2018-10-021-3/+5
| | | | | | | | | 1. Fix fdopen used for pid file 2. Fix sha256 checksum calculation Updates: #411 Change-Id: Ic173d104a73822c29aca260ba6de872cd8d23f86 Signed-off-by: Kotresh HR <khiremat@redhat.com>
* georep: python2 to python3 compatibility-mount writeKotresh HR2018-10-021-3/+9
| | | | | | | | | | python3 expects byte string for os.write. This works for both py2 and py3. Fixed the same for geo-rep mount testing code path. Updates: #411 Change-Id: I2dfedcb0869457707bcca4d2847ef0d52bff1987 Signed-off-by: Kotresh HR <khiremat@redhat.com>
* georep: python3 compatibility (Popen)Kotresh HR2018-09-253-12/+21
| | | | | | | | | | | | | | The file objects for python3 by default is opened in binary mode where as in python2 it's opened as text by default. The geo-rep code parses the output of Popen assuming it as text, hence used the 'universal_newlines' flag which provides backward compatibility for the same. Change-Id: I371a03b6348af9666164cb2e8b93d47475431ad9 Updates: #411 Signed-off-by: Kotresh HR <khiremat@redhat.com>
* georep: Fix python3 compatibility (configparser)Kotresh HR2018-09-251-5/+2
| | | | | | | | | '%' needs special handling in config and also removed duplicate misspelled 'changelog-archive-format' config. Updates: #411 Change-Id: I33621a62bdf5f781ee62e6cedec0c2df3f5d70cf Signed-off-by: Kotresh HR <khiremat@redhat.com>
* georep: Fix python3 compatibility (os.pipe)Kotresh HR2018-09-253-7/+19
| | | | | | | | | | | | | | 'os.pipe' returns pair of file descriptors which are non-inheritable by child processes. But geo-rep uses te inheritable nature of pipe fds to communicate between parent and child processes. Hence wrote a compatiable pipe routine which works well both with python2 and python3 with inheritable nature. Updates: #411 Change-Id: I869d7a52eeecdecf3851d44ed400e69b32a612d9 Signed-off-by: Kotresh HR <khiremat@redhat.com>
* geo-rep : fix log message in master.pySunny Kumar2018-09-241-2/+2
| | | | | | | | This patch fixes spell mistake in log message. Change-Id: I84779c64aef6698cbc1a60ae1a82533e8a6a6e3d updates: bz#1193929 Signed-off-by: Sunny Kumar <sunkumar@redhat.com>
* geo-rep: Fix issues related config setKotresh HR2018-09-183-7/+11
| | | | | | | | | | | | | | | | | | | | 1. '--ignore-mising-args' option for rsync is not being used even though the rsync version is greater than 3.1.0. Fixed the same. 2. '--existing' option for rsync is also not being used. Fixed the same. 3. geo-rep config fails to set rsync-options as the value contains '--'. Interestingly, python argsparse treats the value with '--' (e.g., --ignore-missing-args) as option. But when passed with something like --value=--ignore-missing-args, it succeeds. Fixed the same. Change-Id: Iaeb838acaff1c2920fee9c7f920c99edce13a0a1 Signed-off-by: Kotresh HR <khiremat@redhat.com> fixes: bz#1629561
* Land part 2 of clang-format changesGluster Ant2018-09-122-385/+396
| | | | | Change-Id: Ia84cc24c8924e6d22d02ac15f611c10e26db99b4 Signed-off-by: Nigel Babu <nigelb@redhat.com>
* Land clang-format changesGluster Ant2018-09-121-4/+5
| | | | Change-Id: I6f5d8140a06f3c1b2d196849299f8d483028d33b
* multiple: Fix python2 to python3 compatKotresh HR2018-09-072-3/+3
| | | | | | Change-Id: Iac241166d7a35dc7cc6cf07850f9f1bce38fe207 Updates: #411 Signed-off-by: Kotresh HR <khiremat@redhat.com>
* core: switch to python3Sanju Rakonde2018-09-051-1/+1
| | | | | | | | | | Commit af0d5a9b5375a5cd87ac10b429e2b9934718ce5b changes python2 -> python3. This patch fixes a small issue introduced by it. Change-Id: Ib23c73683f570e8891f41476b661f37c89635fb5 updates: #411 Signed-off-by: Sanju Rakonde <srakonde@redhat.com>
* core: python3Kaleb S. KEITHLEY2018-09-0311-11/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | see https://review.gluster.org/#/c/19788/, https://review.gluster.org/#/c/19871/, https://review.gluster.org/#/c/19952/, https://review.gluster.org/#/c/20104/, https://review.gluster.org/#/c/20162/, https://review.gluster.org/#/c/20185/, https://review.gluster.org/#/c/20207/, https://review.gluster.org/#/c/20227/, https://review.gluster.org/#/c/20307/, https://review.gluster.org/#/c/20320/, https://review.gluster.org/#/c/20332/, https://review.gluster.org/#/c/20364/, https://review.gluster.org/#/c/20441/, and https://review.gluster.org/#/c/20484 shebangs changed from /usr/bin/python2 to /usr/bin/python3. (Reminder, various distribution packaging guidelines require use of explicit python version and don't allow '#!/usr/bin/env python', regardless of how handy that idiom may be.) glusterfs.spec(.in) package python{2,3}-gluster and python2 or python3 dependencies as appropriate. configure(.ac): + test for and use python2 or python3 as appropriate. If build machine has python2 and python3, use python3. Override by setting PYTHON=/usr/bin/python2 when running configure. + PYTHONDEV_CPPFLAGS from python[23]-config --includes is a better match to the original python sysconfig.get_python_inc(). All those other extraneous flags breaks the build. + Only change the shebangs once. Changing them over and over again, e.g., during a `make glusterrpms` in extras/LinuxRPM just sends make (is it really make that's looping?) into an infinite loop. If you figure out why, let me know. + Oldest python2 is python2.6 on CentOS 6 and Debian 8 (Jessie). Everything else has 2.7 or 3.x + logic from https://review.gluster.org/c/glusterfs/+/21050, which needs to be removed/merged after that patch is merged. Builds on CentOS 6, CentOS 7, Fedora 28, Fedora rawhide, and the mysterious RHEL > 7. Change-Id: Idae21d3b6f58b32372e1daa0d234e491e563198f updates: #411 Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
* geo-rep: Make automatic gfid conflict resolution optionalKotresh HR2018-08-272-10/+18
| | | | | | | | | | | | | | | | | | | | | | | | Autmatic gfid conflict resolution needs to be disabled during failover/failback as it might lead to data loss in the following scenario. 1. Master went down without syncing directory "dir1" to slave. 2. When slave is failed over to master, if a new file is written inside "dir1", creating dir1 again if not present, "dir1" ends up with different gfid on original slave. 3. When original master is up and failed back, due to automatic gfid conflict resolution, "dir1" present in original master is deleted losing all files and only new file created on original slave is restored. Hence during failover/failback, automatic gfid conflict resolution should be disabled. So in these cases, appropriate decision is taken. fixes: bz#1622076 Signed-off-by: Kotresh HR <khiremat@redhat.com> Change-Id: I433616f5d3e13d4b6eb675475bd554ca34928573
* geo-rep : fix few coverity issues in procdiggy.cSunny Kumar2018-08-201-4/+6
| | | | | | | | | | | | | This patch fixes BAD_FREE at line number 75,105 and UNUSED_VALUE at line number 108. CID : 1274068 scan detail at [1]. [1] https://scan6.coverity.com/reports.htm#v42401/p10714/fileInstanceId=84384726&defectInstanceId=25600457&mergedDefectId=727233 Change-Id: I8dff42546204bf78c178c9dcaba534888c5354e9 updates: bz#789278 Signed-off-by: Sunny Kumar <sunkumar@redhat.com>
generated by cgit (git 2.34.1) at 2024-04-19 11:37:09 +0000