glusterfs.git/tools/glusterfind/src/changelog.py, branch v3.11.3

tools/glusterfind: kill remote processes and separate run-time directories

2016-10-25T12:00:54+00:00

Problem #1:
Hitting CTRL+C leaves stale processes on remote nodes if glusterfind pre
has been initiated.

Solution #1:
Adding "-t -t" to ssh command-line forces pseudo-terminal to be assigned
to remote process. When local process receives Keyboard Interrupt,
SIGHUP is immediately conveyed to the remote terminal causing remote
changelog.py process to terminate immediately.

Problem #2:
Concurrent glusterfind pre runs are not possible on the same glusterfind
session in case of a runaway process.

Solution #2:
glusterfind pre runs now add random directory name to the working
directory to store and manage temporary database and changelog
processing.
If KeyboardInterrupt is received, the function call
run_cmd_nodes("cleanup", args, tmpfilename=gtmpfilename)
cleans up the remote run specific directory.

Patch:
7571380 cli/xml: Fix wrong XML format in volume get command
broke "gluster volume get  changelog.rollover-time --xml"
Now fixed function utils.py::get_changelog_rollover_time()

Fixed spurious trailing space getting written if second path is empty in
main.py::write_output()
Fixed repetitive changelog processing in changelog.py::get_changes()

Change-Id: Ia8d96e2cd47bf2a64416bece312e67631a1dbf29
BUG: 1382236
Signed-off-by: Milind Changire 
Reviewed-on: http://review.gluster.org/15609
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Aravinda VK

tools/glusterfind: Handling Unicode file names

2016-03-30T13:37:18+00:00

Unicode filenames handled cleanly with this patch. Changelog
files and output files are opened with utf-8 encoding using codecs.open.

urllib.quote_plus and unquote_plus will not handle Unicode so, encode
Unicode to 8-bit string version before calling unquote. urllib.quote_plus
requires 8-bit string itself so do not decode to Unicode if we need to use
quote_plus(when --no-encode=false). Decode to unicode in --no-encode is set.

BUG: 1319717
Change-Id: If5561c749ab5529445650d322c831eb4da22b65a
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/13798
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Milind Changire 
Reviewed-by: Kotresh HR

tools/glusterfind: New option --no-encode

2016-02-26T10:11:08+00:00

New option added to skip encoding path in output file. Also handled
Unicode strings.

File paths can have newline characters, to differentiate between each
path patch is encoded according to
RFC3986(https://www.ietf.org/rfc/rfc3986.txt). Due to this consumer
applications have to decode the path before consuming it. With this
option Paths are not encoded, can be directly consumed by
applications.

Unicode encoding is handled automatically

BUG: 1310080
Change-Id: I83d59831997dbd1264b48e9b1aa732c7dfc700b5
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/13477
Smoke: Gluster Build System 
Reviewed-by: Milind Changire 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Kotresh HR 
CentOS-regression: Gluster Build System 
Reviewed-by: Venky Shankar

tools/glusterfind: add query command to list files

2015-11-25T06:29:07+00:00

When session information is maintained outside Gluster, there needs to
be some mechanism to list files starting from a time-stamp. This patch
implements the feature via the "query" command-line option.

The only caveat is that the first time the query command is run for the
volume, it will likely report that "historical changelogs are not
available". This is due to the fact that changelogs had not been turned
on for the volume so far. So the volume options need to be turned on
outside glusterfind or the since_time need to be greater than the
current time when the query command is run for the very first time for
the volume. The query command turns on the required volume options for
collecting changelogs.

Change-Id: I6cb7a57a5ecd166210e2eb4deede06d40ccfa996
BUG: 1272006
Signed-off-by: Milind Changire 
Reviewed-on: http://review.gluster.org/12362
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Aravinda VK

tools/glusterfind: Prepend prefix in case of delete

2015-08-27T06:21:23+00:00

In case of delete operation alone, output prefix adding
was not handled earlier. 
Output prefix is added now.

Change-Id: Ia91444dddbff501b26a864f6185ca4c0aaf4c802
BUG: 1244144
Signed-off-by: Saravanakumar Arumugam 
Reviewed-on: http://review.gluster.org/11712
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Aravinda VK 
Reviewed-by: Milind Changire

tools/glusterfind: ignoring deleted files

2015-06-12T14:13:18+00:00

OSError and IOError exceptions were being thrown if files
were deleted after session was created and a subsequent
glusterfind pre was attmepted. glusterfind now detects
this scenario and safely ignores these changes to the
file-system. We also avoid recording deleted file paths
into database in the case where gfid to path resolution
cannot be performed for deleted files.

Also, we now turn on volume option to capture delete paths.

Change-Id: Iaf3883463f5e64643497bd77229ad80a9b2d5da5
BUG: 1228111
Signed-off-by: Milind Changire 
Reviewed-on: http://review.gluster.org/11194
Reviewed-by: Aravinda VK 
Tested-by: NetBSD Build System

tools/glusterfind: Fix GFID to Path conversion for dir

2015-06-11T19:28:12+00:00

For Directories, GFID to Path conversion was fine, but wrong
filter was used while updating to db.

Change-Id: Id588adbb25ff5fd357f8cc2d5485bd2eb4a52b65
Signed-off-by: Aravinda VK 
BUG: 1230127
Reviewed-on: http://review.gluster.org/11158
Tested-by: Gluster Build System 
Reviewed-by: Kotresh HR 
Reviewed-by: Vijay Bellur

tools/glusterfind: GFID to Path conversion using Changelog

2015-05-09T04:59:10+00:00

Records fop information collected from Changelogs in sqlite database.
This is only working database, not required after processing.

After post processing, output file is generated by reading these
database files.

This is applicable only in incremental run, When a changelog is
parsed, all the details are saved in Db. GFID to Path is converted
to those files for which information is available in Changelogs.

For all the failed cases, it tries to convert to Path using Pgfid,
if not found GFID to Path is done using find.

BUG: 1201284
Change-Id: I53f168860dae15a0149004835e67f97aebd822be
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/10463
Reviewed-by: Kotresh HR 
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

tools/glusterfind: Partial Find

2015-05-09T04:19:06+00:00

Enabled by default, if one node fails Glusterfind will not fail
to return list of files from other nodes. This behavior can be changed
using --disable-partial

Now session is maintained in each nodes as well as in initiator node.
Every pre command will pick the status file from respective node and
start collecting list of changes happened after the status time.

--reset-session-time, new option to force reset the session time.
Next incremental run will start from this time.

Change-detector argument is removed since Changelog mode is required
to detect deletes and Renames.

Change-Id: I0b83bc7c0e1b30b13de772b2d21fe968db4ff964
Signed-off-by: Aravinda VK 
BUG: 1201289
Reviewed-on: http://review.gluster.org/10320
Tested-by: Gluster Build System 
Reviewed-by: Kotresh HR 
Reviewed-by: Vijay Bellur

tools/glusterfind: Changelog_init before changelog_register

2015-03-30T12:50:25+00:00

With the new Changelog RPC based approach, Changelog_init
needs to be called before changelog_register

BUG: 1206127
Change-Id: I817b84016d3f9ffdf2ea7a4e177953fe14ccc323
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/10009
Tested-by: Gluster Build System 
Reviewed-by: Kotresh HR 
Reviewed-by: Prashanth Pai 
Reviewed-by: Kaleb KEITHLEY