glusterfs.git/tools, branch v3.12dev

build/packaging: Debian and Ubuntu don't have /usr/libexec

2017-03-13T14:18:07+00:00

GLUSTERFS_LIBEXECDIR is effectively hard-coded to /usr/libexec/glusterfs
in configure(.ac)

Debian-based distributions don't have a /usr/libexec/ directory

This issues is partially mitigated by the use of $libexecdir in
some of the Makefile.am files, but even so the incorrectly defined
GLUSTERFS_LIBEXECDIR results in various things such as gsyncd,
glusterfind, eventsd, etc., trying to invoke other scripts and
programs from a location that doesn't exist.

And once we correctly define GLUSTERFS_LIBEXECDIR, then we might as
well use it appropriatedly.

Change-Id: If5219cadc51ae316f7ba2e2831d739235c77902d
BUG: 1430841
Signed-off-by: Kaleb S. KEITHLEY 
Reviewed-on: https://review.gluster.org/16880
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Milind Changire 
Reviewed-by: Joe Julian 
Reviewed-by: Shyamsundar Ranganathan

tools/glusterfind: delete temporary folder

2017-01-30T05:27:25+00:00

Problem:
Run specific temporary folder created under
/usr/var/lib/misc/glusterfsd/glusterfind///
remains undeleted.

Solution:
Delete the temporary folder from all nodes.

Change-Id: I0edaf868aebb01b15c489434bbb26fe853351384
BUG: 1413526
Signed-off-by: Milind Changire 
Reviewed-on: https://review.gluster.org/16416
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Aravinda VK

tools/glusterfind: avoid deleting keys directory

2016-12-08T06:05:48+00:00

Problem:
gluster volume delete mistakenly deletes the .keys directory under
/var/lib/glusterd/glusterfind.

Solution:
Check for ".keys" directory and avoid deleting it.

Change-Id: Ia595c8bf3f423c1ad5d6faa183a29598c07a11f9
BUG: 1402369
Signed-off-by: Milind Changire 
Reviewed-on: http://review.gluster.org/16052
Reviewed-by: Aravinda VK 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

tools/glusterfind: xml parsing fix for tiered volumes

2016-11-17T11:30:54+00:00

gluster volume info  --xml for non-tiered volumes have
'bricks/brick' elements under the 'volInfo/volumes/volume' element.
However, tiered volumes have a 'bricks/hotBricks/brick' and
'bricks/coldBricks/brick' elements under the 'volInfo/volumes/volume'
element.

Fix main.py::get_nodes()

BUG: 1389481
Change-Id: I2f4465bfa8a55e7fa87917d3ec3e69b05d5241b9
Signed-off-by: Milind Changire 
Reviewed-on: http://review.gluster.org/15746
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Aravinda VK

tools/glusterfind: kill remote processes and separate run-time directories

2016-10-25T12:00:54+00:00

Problem #1:
Hitting CTRL+C leaves stale processes on remote nodes if glusterfind pre
has been initiated.

Solution #1:
Adding "-t -t" to ssh command-line forces pseudo-terminal to be assigned
to remote process. When local process receives Keyboard Interrupt,
SIGHUP is immediately conveyed to the remote terminal causing remote
changelog.py process to terminate immediately.

Problem #2:
Concurrent glusterfind pre runs are not possible on the same glusterfind
session in case of a runaway process.

Solution #2:
glusterfind pre runs now add random directory name to the working
directory to store and manage temporary database and changelog
processing.
If KeyboardInterrupt is received, the function call
run_cmd_nodes("cleanup", args, tmpfilename=gtmpfilename)
cleans up the remote run specific directory.

Patch:
7571380 cli/xml: Fix wrong XML format in volume get command
broke "gluster volume get  changelog.rollover-time --xml"
Now fixed function utils.py::get_changelog_rollover_time()

Fixed spurious trailing space getting written if second path is empty in
main.py::write_output()
Fixed repetitive changelog processing in changelog.py::get_changes()

Change-Id: Ia8d96e2cd47bf2a64416bece312e67631a1dbf29
BUG: 1382236
Signed-off-by: Milind Changire 
Reviewed-on: http://review.gluster.org/15609
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Aravinda VK

core: use readdir(3) with glibc, and associated cleanup

2016-07-18T11:59:42+00:00

Starting with glibc-2.23 (i.e. what's in Fedora 25), readdir_r(3)
is marked as deprecated. Specifically the function decl in 
has the deprecated attribute, and now warnings are thrown during the
compile on Fedora 25 builds.

The readdir(_r)(3) man page (on Fedora 25 at least) and World+Dog say
that glibc's readdir(3) is, and always has been, MT-SAFE as long as
only one thread is accessing the directory object returned by opendir().
World+Dog also says there is a potential buffer overflow in readdir_r().
World+Dog suggests that it is preferable to simply use readdir(). There's
an implication that eventually readdir_r(3) will be removed from glibc.
POSIX has, apparently deprecated it in the standard, or even removed it
entirely.

Over and above that, our source near the various uses of readdir(_r)(3)
has a few unsafe uses of strcpy()+strcat().

(AFAIK nobody has looked at the readdir(3) implemenation in *BSD to see
if the same is true on those platforms, and we can't be sure of MacOS
even though we know it's based on *BSD.)

Change-Id: I5481f18ba1eebe7ee177895eecc9a80a71b60568
BUG: 1356998
Signed-off-by: Kaleb S. KEITHLEY 
Reviewed-on: http://review.gluster.org/14838
Smoke: Gluster Build System 
Reviewed-by: Niels de Vos 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Kotresh HR 
Reviewed-by: Jeff Darcy

packaging: Remove ".py" extension from symlink target

2016-07-15T13:41:16+00:00

S57glusterfind-delete-post.py* is packaged in
$LIBEXEC/glusterfs/glusterfind directory, symlink created to
/var/lib/glusterd/hooks directory

ln -s $(libexecdir)/glusterfs/glusterfind/S57glusterfind-delete-post.py \
$(DESTDIR)$(GLUSTERD_WORKDIR)/hooks/1/delete/post/S57glusterfind-delete-post.py

".py" should be removed from symlink target so that build system will
not try to compile the symlink files and generate .pyc/.pyo files.

BUG: 1356868
Change-Id: I9155378aa72eb8559ba8af76d91a4985f69f38f9
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/14928
Smoke: Gluster Build System 
Reviewed-by: Kotresh HR 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kaleb KEITHLEY

scripts: bash-isms in scripts

2016-05-18T09:22:35+00:00

Change-Id: I05f8718fb73c16dbb37c1f7460bb7b55cc0ffe97
BUG: 1336853
Reported-by: Patrick Matthäi 
Signed-off-by: Kaleb S KEITHLEY 
Reviewed-on: http://review.gluster.org/14401
NetBSD-regression: NetBSD Build System 
Reviewed-by: Niels de Vos 
Smoke: Gluster Build System 
Reviewed-by: Kotresh HR 
CentOS-regression: Gluster Build System

tools/glusterfind: add --full option to query command

2016-03-31T04:51:32+00:00

The --full option will use brickfind.py to list all files in the volume.
The output file will contain url-encoded file names prefixed with the
tag string to indicate that all files should be considered as new.
The default tag string for the --full option is "NEW". This can be
changed with the --tag-for-full-find command-line option.

Change-Id: Ic85ba5db062e19df13ae9dc2de8a08eacb5c9792
BUG: 1286279
Signed-off-by: Milind Changire 
Reviewed-on: http://review.gluster.org/12779
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Aravinda VK

tools/glusterfind: Handling Unicode file names

2016-03-30T13:37:18+00:00

Unicode filenames handled cleanly with this patch. Changelog
files and output files are opened with utf-8 encoding using codecs.open.

urllib.quote_plus and unquote_plus will not handle Unicode so, encode
Unicode to 8-bit string version before calling unquote. urllib.quote_plus
requires 8-bit string itself so do not decode to Unicode if we need to use
quote_plus(when --no-encode=false). Decode to unicode in --no-encode is set.

BUG: 1319717
Change-Id: If5561c749ab5529445650d322c831eb4da22b65a
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/13798
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Milind Changire 
Reviewed-by: Kotresh HR