glusterfs.git/tools, branch exp

tools/glusterfind: avoid deleting keys directory

2016-12-08T06:05:48+00:00

Problem:
gluster volume delete mistakenly deletes the .keys directory under
/var/lib/glusterd/glusterfind.

Solution:
Check for ".keys" directory and avoid deleting it.

Change-Id: Ia595c8bf3f423c1ad5d6faa183a29598c07a11f9
BUG: 1402369
Signed-off-by: Milind Changire 
Reviewed-on: http://review.gluster.org/16052
Reviewed-by: Aravinda VK 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

tools/glusterfind: xml parsing fix for tiered volumes

2016-11-17T11:30:54+00:00

gluster volume info  --xml for non-tiered volumes have
'bricks/brick' elements under the 'volInfo/volumes/volume' element.
However, tiered volumes have a 'bricks/hotBricks/brick' and
'bricks/coldBricks/brick' elements under the 'volInfo/volumes/volume'
element.

Fix main.py::get_nodes()

BUG: 1389481
Change-Id: I2f4465bfa8a55e7fa87917d3ec3e69b05d5241b9
Signed-off-by: Milind Changire 
Reviewed-on: http://review.gluster.org/15746
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Aravinda VK

tools/glusterfind: kill remote processes and separate run-time directories

2016-10-25T12:00:54+00:00

Problem #1:
Hitting CTRL+C leaves stale processes on remote nodes if glusterfind pre
has been initiated.

Solution #1:
Adding "-t -t" to ssh command-line forces pseudo-terminal to be assigned
to remote process. When local process receives Keyboard Interrupt,
SIGHUP is immediately conveyed to the remote terminal causing remote
changelog.py process to terminate immediately.

Problem #2:
Concurrent glusterfind pre runs are not possible on the same glusterfind
session in case of a runaway process.

Solution #2:
glusterfind pre runs now add random directory name to the working
directory to store and manage temporary database and changelog
processing.
If KeyboardInterrupt is received, the function call
run_cmd_nodes("cleanup", args, tmpfilename=gtmpfilename)
cleans up the remote run specific directory.

Patch:
7571380 cli/xml: Fix wrong XML format in volume get command
broke "gluster volume get  changelog.rollover-time --xml"
Now fixed function utils.py::get_changelog_rollover_time()

Fixed spurious trailing space getting written if second path is empty in
main.py::write_output()
Fixed repetitive changelog processing in changelog.py::get_changes()

Change-Id: Ia8d96e2cd47bf2a64416bece312e67631a1dbf29
BUG: 1382236
Signed-off-by: Milind Changire 
Reviewed-on: http://review.gluster.org/15609
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Aravinda VK

core: use readdir(3) with glibc, and associated cleanup

2016-07-18T11:59:42+00:00

Starting with glibc-2.23 (i.e. what's in Fedora 25), readdir_r(3)
is marked as deprecated. Specifically the function decl in 
has the deprecated attribute, and now warnings are thrown during the
compile on Fedora 25 builds.

The readdir(_r)(3) man page (on Fedora 25 at least) and World+Dog say
that glibc's readdir(3) is, and always has been, MT-SAFE as long as
only one thread is accessing the directory object returned by opendir().
World+Dog also says there is a potential buffer overflow in readdir_r().
World+Dog suggests that it is preferable to simply use readdir(). There's
an implication that eventually readdir_r(3) will be removed from glibc.
POSIX has, apparently deprecated it in the standard, or even removed it
entirely.

Over and above that, our source near the various uses of readdir(_r)(3)
has a few unsafe uses of strcpy()+strcat().

(AFAIK nobody has looked at the readdir(3) implemenation in *BSD to see
if the same is true on those platforms, and we can't be sure of MacOS
even though we know it's based on *BSD.)

Change-Id: I5481f18ba1eebe7ee177895eecc9a80a71b60568
BUG: 1356998
Signed-off-by: Kaleb S. KEITHLEY 
Reviewed-on: http://review.gluster.org/14838
Smoke: Gluster Build System 
Reviewed-by: Niels de Vos 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Kotresh HR 
Reviewed-by: Jeff Darcy

packaging: Remove ".py" extension from symlink target

2016-07-15T13:41:16+00:00

S57glusterfind-delete-post.py* is packaged in
$LIBEXEC/glusterfs/glusterfind directory, symlink created to
/var/lib/glusterd/hooks directory

ln -s $(libexecdir)/glusterfs/glusterfind/S57glusterfind-delete-post.py \
$(DESTDIR)$(GLUSTERD_WORKDIR)/hooks/1/delete/post/S57glusterfind-delete-post.py

".py" should be removed from symlink target so that build system will
not try to compile the symlink files and generate .pyc/.pyo files.

BUG: 1356868
Change-Id: I9155378aa72eb8559ba8af76d91a4985f69f38f9
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/14928
Smoke: Gluster Build System 
Reviewed-by: Kotresh HR 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kaleb KEITHLEY

scripts: bash-isms in scripts

2016-05-18T09:22:35+00:00

Change-Id: I05f8718fb73c16dbb37c1f7460bb7b55cc0ffe97
BUG: 1336853
Reported-by: Patrick Matthäi 
Signed-off-by: Kaleb S KEITHLEY 
Reviewed-on: http://review.gluster.org/14401
NetBSD-regression: NetBSD Build System 
Reviewed-by: Niels de Vos 
Smoke: Gluster Build System 
Reviewed-by: Kotresh HR 
CentOS-regression: Gluster Build System

tools/glusterfind: add --full option to query command

2016-03-31T04:51:32+00:00

The --full option will use brickfind.py to list all files in the volume.
The output file will contain url-encoded file names prefixed with the
tag string to indicate that all files should be considered as new.
The default tag string for the --full option is "NEW". This can be
changed with the --tag-for-full-find command-line option.

Change-Id: Ic85ba5db062e19df13ae9dc2de8a08eacb5c9792
BUG: 1286279
Signed-off-by: Milind Changire 
Reviewed-on: http://review.gluster.org/12779
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Aravinda VK

tools/glusterfind: Handling Unicode file names

2016-03-30T13:37:18+00:00

Unicode filenames handled cleanly with this patch. Changelog
files and output files are opened with utf-8 encoding using codecs.open.

urllib.quote_plus and unquote_plus will not handle Unicode so, encode
Unicode to 8-bit string version before calling unquote. urllib.quote_plus
requires 8-bit string itself so do not decode to Unicode if we need to use
quote_plus(when --no-encode=false). Decode to unicode in --no-encode is set.

BUG: 1319717
Change-Id: If5561c749ab5529445650d322c831eb4da22b65a
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/13798
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Milind Changire 
Reviewed-by: Kotresh HR

tools/glusterfind: Do not fail cleanup script on volume delete

2016-03-08T09:58:10+00:00

If glusterfind dir is not present inside GLUSTERD_WORKDIR, then
cleanup script Post Volume delete was failing. This patch safely
handles the absence of glusterfind directory in GLUSTERD_WORKDIR

BUG: 1289935
Change-Id: I9f7e6035e3dc6079927803df942b0f921218c79a
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/12923
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR

tools/glusterfind: New option --no-encode

2016-02-26T10:11:08+00:00

New option added to skip encoding path in output file. Also handled
Unicode strings.

File paths can have newline characters, to differentiate between each
path patch is encoded according to
RFC3986(https://www.ietf.org/rfc/rfc3986.txt). Due to this consumer
applications have to decode the path before consuming it. With this
option Paths are not encoded, can be directly consumed by
applications.

Unicode encoding is handled automatically

BUG: 1310080
Change-Id: I83d59831997dbd1264b48e9b1aa732c7dfc700b5
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/13477
Smoke: Gluster Build System 
Reviewed-by: Milind Changire 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Kotresh HR 
CentOS-regression: Gluster Build System 
Reviewed-by: Venky Shankar