glusterfs.git/tools/glusterfind/src/changelog.py, branch v3.7.7

tools/glusterfind: add query command to list files

2015-12-11T08:29:12+00:00

When session information is maintained outside Gluster, there needs to
be some mechanism to list files starting from a time-stamp. This patch
implements the feature via the "query" command-line option.

The only caveat is that the first time the query command is run for the
volume, it will likely report that "historical changelogs are not
available". This is due to the fact that changelogs had not been turned
on for the volume so far. So the volume options need to be turned on
outside glusterfind or the since_time need to be greater than the
current time when the query command is run for the very first time for
the volume. The query command turns on the required volume options for
collecting changelogs.

Change-Id: I6cb7a57a5ecd166210e2eb4deede06d40ccfa996
BUG: 1272007
Signed-off-by: Milind Changire 
Reviewed-on: http://review.gluster.org/12362
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Aravinda VK 
 (cherry picked from commit a56e32e19703c0fbe2cedebcaf5edc8a6307d5a1)
Reviewed-on: http://review.gluster.org/12742

tools/glusterfind: Prepend prefix in case of delete

2015-11-25T07:20:34+00:00

In case of delete operation alone, output prefix adding
was not handled earlier. 
Output prefix is added now.

Change-Id: Ia91444dddbff501b26a864f6185ca4c0aaf4c802
Reviewed-on: http://review.gluster.org/11712/
BUG: 1257141
Signed-off-by: Saravanakumar Arumugam 
Reviewed-on: http://review.gluster.org/12016
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Milind Changire 
Reviewed-by: Aravinda VK

tools/glusterfind: ignoring deleted files

2015-06-13T06:46:11+00:00

OSError and IOError exceptions were being thrown if files
were deleted after session was created and a subsequent
glusterfind pre was attmepted. glusterfind now detects
this scenario and safely ignores these changes to the
file-system. We also avoid recording deleted file paths
into database in the case where gfid to path resolution
cannot be performed for deleted files.

Also, we now turn on volume option to capture delete paths.

Change-Id: Iaf3883463f5e64643497bd77229ad80a9b2d5da5
BUG: 1230783
Reviewed-on: http://review.gluster.org/#/c/11194/
Signed-off-by: Milind Changire 
Reviewed-on: http://review.gluster.org/11199
Tested-by: Gluster Build System 
Reviewed-by: Aravinda VK

tools/glusterfind: Fix GFID to Path conversion for dir

2015-06-12T10:51:02+00:00

For Directories, GFID to Path conversion was fine, but wrong
filter was used while updating to db.

Change-Id: Id588adbb25ff5fd357f8cc2d5485bd2eb4a52b65
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/#/c/11158/
BUG: 1230712
Reviewed-on: http://review.gluster.org/11184
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

tools/glusterfind: GFID to Path conversion using Changelog

2015-05-10T03:44:00+00:00

Records fop information collected from Changelogs in sqlite database.
This is only working database, not required after processing.

After post processing, output file is generated by reading these
database files.

This is applicable only in incremental run, when a changelog is
parsed, all the details are saved in DB. GFID to Path is converted
to those files for which information is available in Changelogs.

For all the failed cases, it tries to convert to Path using Pgfid,
if not found GFID to Path is done using find.

BUG: 1219475
Change-Id: Iaccc9d063b1187d973137e592f024979b724e85c
Reviewed-On: http://review.gluster.org/#/c/10463/
Original-Author: Aravinda VK 
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/10640
Tested-by: Gluster Build System 
Reviewed-by: Aravinda VK

tools/glusterfind: Partial Find

2015-05-09T10:25:07+00:00

This is optional and enabled by default, if one node fails Glusterfind
will not fail to return list of files from other nodes. This behavior
can be changed using --disable-partial

Now session is maintained in each nodes as well as in initiator node.
Every pre command will pick the status file from respective node and
start collecting list of changes happened after the status time.

--reset-session-time, new option to force reset the session time.
Next incremental run will start from this time.

Change-detector argument is removed since Changelog mode is required
to detect deletes and Renames.

BUG: 1219467
Change-Id: I1d0a0629facc3d26780200ccdf39b221ab4037c4
Original-Author: Aravinda VK 
Reviewed-On: http://review.gluster.org/#/c/10320
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/10639
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

tools/glusterfind: Changelog_init before changelog_register

2015-03-30T12:50:25+00:00

With the new Changelog RPC based approach, Changelog_init
needs to be called before changelog_register

BUG: 1206127
Change-Id: I817b84016d3f9ffdf2ea7a4e177953fe14ccc323
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/10009
Tested-by: Gluster Build System 
Reviewed-by: Kotresh HR 
Reviewed-by: Prashanth Pai 
Reviewed-by: Kaleb KEITHLEY

tools/glusterfind: Ignore .trashcan dir during Brick Crawl

2015-03-30T12:35:32+00:00

BUG: 1203650
Change-Id: Icf81726ad34bd5cfec6d8e06763b02a159e84c2e
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/10010
Reviewed-by: Kotresh HR 
Reviewed-by: Prashanth Pai 
Tested-by: Gluster Build System 
Reviewed-by: Kaleb KEITHLEY

feature/glusterfind: A tool to find incremental changes

2015-03-18T09:54:25+00:00

Documentation is available in patch:
http://review.gluster.org/#/c/9800/

A tool which helps to get list of modified files or list of all files in
GlusterFS Volume using Changelog or find command.

Usage
=====
glusterfind --help

Create:
-------
glusterfind create --help

The tool creates status file $GLUSTERD_WORKDIR/SESSION/VOLUME/status
and records current timestamp to initiate the session. This timestamp
will be used as start time for next runs.

As part of create also generates ssh key and distributes to all peers.
and enables build.pgfid and changelog using volume set command.

Pre:
----
glusterfind pre --help

This command is used to generate the list of files modified after session
creation time or after last run. To get list of all files/dirs in Volume,
run pre command with `--full` argument.

The tool gets all nodes details using gluster volume info and runs node
agent for each brick in respective nodes via ssh command. Once these node
agents generate the output file, tool copies to local using scp. Merges all
the output files to generate the final output file.

Post:
-----
glusterfind post --help

After consuming the list, this sub command is called to update the session
time based on pre command status file.

List:
-----
glusterfind list --help

To view all the sessions

Delete:
-------
glusterfind delete --help

Delete session.

Known Issues
------------
1. Deleted files will not get listed, since we can't convert GFID to
   Path if file/dir is deleted.
2. Only new name will get listed if Renamed.
3. All hardlinks will get listed.

Change-Id: I82991feb0aea85cb6ec035fddbf80a2b276e86b0
BUG: 1193893
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/9682
Tested-by: Gluster Build System 
Reviewed-by: Venky Shankar 
Reviewed-by: Prashanth Pai 
Reviewed-by: Kotresh HR 
Reviewed-by: Vijay Bellur