glusterfs-snapshot.git/xlators/features/marker/utils, branch development

move 'xlators/marker/utils/' to 'geo-replication/' directory

2013-07-22T08:53:03+00:00

Change-Id: Ibd0faefecc15b6713eda28bc96794ae58aff45aa
BUG: 847839
Original Author: Amar Tumballi 
Signed-off-by: Avra Sengupta 
Reviewed-on: http://review.gluster.org/5133
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

features/changelog: changelog translator

2013-07-22T08:52:24+00:00

This is the initial version of the Changelog Translator.

What is it
-----------
Goal is to capture changes performed on a GlusterFS volume.
The translator needs to be loaded on the server (bricks) and
captures changes in a plain text file inside a configured
directory path (controlled by "changelog-dir", should be
somewhere in /.glusterfs/changelog by default).

Changes are classified into 3 types:
    - Data:     : TYPE-I
    - Metadata  : TYPE-II
    - Entry     : TYPE-III

Changelog file is rolled over after a certain time interval
(defauls to 60 seconds) after which a changelog is started.
The thing to be noted here is that for a time interval
(time slice) multiple changes for an inode are recorded only
once (ie. say for 100+ writes on an inode that happens within
the time slice has only a single corresponding entry in the
changelog file). That way we do not bloat up the changelog
and also save lots of writes.

Changelog Format
-----------------
TYPE-I and TYPE-II changes have the gfid on the entity on
which the operation happened. TYPE-III being a entry op
requires the parent gfid and the basename. Changelog format
has been kept to a minimal and it's upto the consumers to
do the heavy loading of figuring out deletes, renames etc..
A single changelog file records all three types of changes,
with each change starting with an identifier ("D": DATA,
"M": METADATA and "E": ENTRY). Option is provided for the
encoding type (See TUNABLES).

Consumers
----------
The only consumer as of today would be geo-replication, although
backup utilities, self-heal, bit-rot detection could be possible
consumers in the future.

CLI
----
By default, change-logging is disabled (the translator is present
in the server graph but does nothing). When enabled (via cli) each
brick starts to log the changes. There are a set of tunable that
can be used to change the translators behaviour:

- enable/disable changelog (disabled by default)
  gluster volume set  changelog {on|off}

- set the logging directory (/.glusterfs/changelogs is the
  default)
  gluster volume set  changelog-dir /path/to/dir

- select encoding type (binary (default) or ascii)
  gluster volume set  encoding {binary|ascii}

- change the rollover time for the logs (60 secs by default)
  gluster volume set  rollover-time 

- when secs > 0, changelog file is not open()'d with O_SYNC flag
- and fsync is trigerred periodically every  seconds.
  gluster volume set  fsync-interval 

features/changelog: changelog consumer library (libgfchangelog)

A shared library is provided for the consumer of the changelogs
for easy acess via APIs. Application can link against this library
and request for changelog updates. Conversion of binary logs to
human-readable ascii format is also taken care by the library which
keeps a copy of the changelog in application provided working
directory.

Change-Id: I75575fb7f1c53d2bec3dba1a329ea7bb3c628497
BUG: 847839
Original Author: Venky Shankar 
Signed-off-by: Avra Sengupta 
Reviewed-on: http://review.gluster.org/5127
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

gsync: Display additional information in status command

2013-04-04T08:04:19+00:00

Added code to display extra information when status command
is executed.
 Information shown now are
1 Number of files synced
2 crawl time
3 total sync time
4 bytes synced

bytes synced is taken from rsync output .
--stats option of rsync gives extra infor
mation about the sync.In stats output there
is a field called Total transferred file
size which states the ammount of bytes synced .
This information is parsed from stdout output
using regular expressions.Bytes synced information
can be used to calculate throughput.

Change-Id: Id9bba9fff45ee7049bb8257c6fd918e5237e05b1
BUG: 947774
Signed-off-by: sarvotham s pai 
Reviewed-on: http://review.gluster.org/4749
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

geo-rep: retire old style ssh setup

2013-03-14T07:23:23+00:00

Users are still using geo-rep with the old, deprecated, insecure, unsupported
ssh setup. Not their fault -- the implementation of the new method had the
following charasteristics:
- old method is possible, but with default settings it's not working
- it can be made operational by fiddling with "remote-gsyncd" tunable
- with default setting, an unhelpful, actually misleading error message is
  produced
- the UI gave no hint to the changes in the ssh setup

http://review.gluster.org/4392 tried to fix these; what it accomplished was
unrestricted support to the bad practice (by making the default old setup
operational).

From this on:
- we disable the old method by reserving the "remote-gsyncd" tunable
- if the old method is attempted, give a hint what to do

Change-Id: Icade94725d8d8d2d4c89cab992d4226351637b86
BUG: 895656
Signed-off-by: Csaba Henk 
Reviewed-on: http://review.gluster.org/4602
Reviewed-by: Venky Shankar 
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

gsyncd: allow the override of the compiled-in python path

2013-02-08T01:17:18+00:00

.. using the environment variable $PYTHON

Change-Id: Ieaad8be98b826c803268216826e250d9944c8190
BUG: 882127
Signed-off-by: Joe Julian 
Signed-off-by: Anand Avati 
Reviewed-on: http://review.gluster.org/4252
Tested-by: Gluster Build System

geo-rep / gsyncd: Separate log file directory for Mountbroker sessions

2013-02-05T01:34:56+00:00

... so that a mountbroker session which is initiated b/w master
and slave does not use the same log file if it's started after
a normal geo-rep session b/w master and slave. This results in
EPERM as the log file is owned by root and the geo-rep slave
process (now running as a non privileged user) does not have
access to it.

Also, having separate log file directory for mountbroker sessions
looks clean.

NOTE: geo-rep's client mount log file location remains unchanged.

Change-Id: Ic7a732e250aee5393b9c3f6ebf6dfe2c310b7fe4
BUG: 893960
Signed-off-by: Venky Shankar 
Reviewed-on: http://review.gluster.org/4407
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

geo-rep: do not access BaseException.message in syncdutils

2012-12-18T22:17:35+00:00

http://www.python.org/dev/peps/pep-0352/ explains that the .message
property of BaseException is being removed. Most of the other exception
handlers access .args[] which should be suitable for this
case too.

Change-Id: I1810450b78d2b3d7f8bd07f2beb02cbe9e2adecb
BUG: 888346
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/4328
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

geo-rep / gsyncd: play nicely with peer multiplexing when setting a checkpoint

2012-12-04T20:13:02+00:00

The gsyncd invocation that instruments the "geo-rep config" command is
multiplexed over peers to ensure the uniformity of configuration.
In general, that works well, but checkpoint setting is a special case,
because (unlike other instances of config-set) it is logged (as recording
of checkpoint events is part of the feature).

Problem is that the path components leading to the log file are
created only on the original node, where gsyncd was started.
Therefore the logging attempt will fail on the other nodes.

Fix: ignore if opening the logfile on behalf of checkpoint setting
fails with ENOENT.

Change-Id: I677f3f081bf4b9e3ba4d25d58979d86931e6beb4
BUG: 881997
Signed-off-by: Csaba Henk 
Reviewed-on: http://review.gluster.org/4248
Reviewed-by: Niels de Vos 
Tested-by: Christos Triantafyllidis 
Reviewed-by: Christos Triantafyllidis 
Reviewed-by: Anand Avati

geo-rep / gsyncd,glusterd: do not hardcode socket path

2012-11-29T00:54:07+00:00

... in gsyncd python code. Indeed, use the configuration
mechanism to set it suitably from glusterd.

Change-Id: I9fe2088b14d28588d1e64fe892740cc5755b8365
BUG: 868877
Signed-off-by: Csaba Henk 
Reviewed-on: http://review.gluster.org/4143
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

geo-replication: catch select.error on select()

2012-11-29T00:23:49+00:00

tailer() in resource.py does not correctly catch exceptions from
select(). select() can raise an instance of the select.error class and
the current expression only catches ValueError (and the instance will
have reference called selecterror).

The geo-rep log contains a call trace like this:
> E [syncdutils:190:log_raise_exception] : FAIL:
> Traceback (most recent call last):
> File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 216, in twrap
> tf(*aa)
> File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 123, in tailer
> poe, _ ,_ = select([po.stderr for po in errstore], [], [], 1)
> File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 276, in select
> return eintr_wrap(oselect.select, oselect.error, *a)
> File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 269, in eintr_wrap
> return func(*a)
> error: (9, 'Bad file descriptor')

BUG: 880308
Change-Id: I2babe42918950d0e9ddb3d08fa21aa3548ccf7c5
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/4233
Reviewed-by: Peter Portante 
Reviewed-by: Csaba Henk 
Tested-by: Gluster Build System