glusterfs-quota.git/libglusterfs, branch test

core: increase the auxillary group limit to 65536

2013-07-24T18:36:46+00:00

Make the allocation of groups dynamic and increase the limit
to 65536.

Change-Id: I702364ff460e3a982e44ccbcb3e337cac9c2df51
BUG: 953694
Signed-off-by: Anand Avati 
Reviewed-on: http://review.gluster.org/5111
Reviewed-by: Xavier Hernandez 
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

features/changelog: changelog translator

2013-07-22T08:52:24+00:00

This is the initial version of the Changelog Translator.

What is it
-----------
Goal is to capture changes performed on a GlusterFS volume.
The translator needs to be loaded on the server (bricks) and
captures changes in a plain text file inside a configured
directory path (controlled by "changelog-dir", should be
somewhere in /.glusterfs/changelog by default).

Changes are classified into 3 types:
    - Data:     : TYPE-I
    - Metadata  : TYPE-II
    - Entry     : TYPE-III

Changelog file is rolled over after a certain time interval
(defauls to 60 seconds) after which a changelog is started.
The thing to be noted here is that for a time interval
(time slice) multiple changes for an inode are recorded only
once (ie. say for 100+ writes on an inode that happens within
the time slice has only a single corresponding entry in the
changelog file). That way we do not bloat up the changelog
and also save lots of writes.

Changelog Format
-----------------
TYPE-I and TYPE-II changes have the gfid on the entity on
which the operation happened. TYPE-III being a entry op
requires the parent gfid and the basename. Changelog format
has been kept to a minimal and it's upto the consumers to
do the heavy loading of figuring out deletes, renames etc..
A single changelog file records all three types of changes,
with each change starting with an identifier ("D": DATA,
"M": METADATA and "E": ENTRY). Option is provided for the
encoding type (See TUNABLES).

Consumers
----------
The only consumer as of today would be geo-replication, although
backup utilities, self-heal, bit-rot detection could be possible
consumers in the future.

CLI
----
By default, change-logging is disabled (the translator is present
in the server graph but does nothing). When enabled (via cli) each
brick starts to log the changes. There are a set of tunable that
can be used to change the translators behaviour:

- enable/disable changelog (disabled by default)
  gluster volume set  changelog {on|off}

- set the logging directory (/.glusterfs/changelogs is the
  default)
  gluster volume set  changelog-dir /path/to/dir

- select encoding type (binary (default) or ascii)
  gluster volume set  encoding {binary|ascii}

- change the rollover time for the logs (60 secs by default)
  gluster volume set  rollover-time 

- when secs > 0, changelog file is not open()'d with O_SYNC flag
- and fsync is trigerred periodically every  seconds.
  gluster volume set  fsync-interval 

features/changelog: changelog consumer library (libgfchangelog)

A shared library is provided for the consumer of the changelogs
for easy acess via APIs. Application can link against this library
and request for changelog updates. Conversion of binary logs to
human-readable ascii format is also taken care by the library which
keeps a copy of the changelog in application provided working
directory.

Change-Id: I75575fb7f1c53d2bec3dba1a329ea7bb3c628497
BUG: 847839
Original Author: Venky Shankar 
Signed-off-by: Avra Sengupta 
Reviewed-on: http://review.gluster.org/5127
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

log: send current gf_log to syslog conditionally

2013-07-19T10:56:47+00:00

When compile time option GF_USE_SYSLOG is enabled (which is default),
generated logs are sent to syslog with error code ERR_DEV.

User can opt to use traditional log at run time by creating
/var/log/glusterd/logger.conf file and restarting respective gluster
services.

Change-Id: I9837d0f99da1afc2189d7ecd214c4293ec53715a
BUG: 928648
Signed-off-by: Bala.FA 
Reviewed-on: http://review.gluster.org/5002
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

log: enhance syslog logging using CEE format

2013-07-19T10:55:35+00:00

This patch enables to use syslog as log target in addition to the
default.  The logs are sent in CEE format (http://cee.mitre.org/).
This logging can be disabled using compile time option by

./configure --disable-syslog
(or)
rpmbuild glusterfs.tar.gz --without syslog

The framework provides two api

void gf_openlog (const char *ident, int option, int facility);
void gf_syslog (int error_code, int facility_priority, char *format, ...);

consumers need to call gf_openlog() prior to gf_syslog() like the way
traditional syslog function calls.  error_code is mandatory when using
gf_syslog().  For example,

gf_openlog (NULL, -1, -1);
gf_syslog (GF_ERR_DEV, LOG_ERR, "error reading configuration file");

Using syslog, admin is free to configure logger to
* reduce repeated log messages
* forward logs to remote logger
* execute a command on certain log pattern
* alert people for certain log pattern by email, snmp etc
* and many more

Change-Id: Ibacbcbbc547192893fc4a46b387496b622e4811f
BUG: 928648
Signed-off-by: Bala.FA 
Reviewed-on: http://review.gluster.org/4915
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

log: error code generation support

2013-07-19T10:55:15+00:00

error code and message are generated at compile time by reading a json
file which contains information of elements for each error code.  This
framework provides error handling and ability to do more cleaner log
messages to users.

error-codes.json file contains error description is below format

{
    "ERR_NAME": {"code": ERR_NUM,
                 "message": {"LOCALE": "ERR_MESSAGE"}}
}

At compile time autogen.sh calls gen-headers.py which produces C
header file libglusterfs/src/gf-error-codes.h.  This header has a
function

const char *_gf_get_message (int code);

which returns respective ERR_MESSAGE for given ERR_NUM.

Change-Id: Ieefbf4c470e19a0175c28942e56cec98a3c94ff0
BUG: 928648
Signed-off-by: Bala.FA 
Reviewed-on: http://review.gluster.org/4977
Reviewed-by: Niels de Vos 
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

fuse: auxiliary gfid mount support

2013-07-19T08:14:08+00:00

* files can be accessed directly through their gfid and not just
  through their paths. For eg., if the gfid of a file is
  f3142503-c75e-45b1-b92a-463cf4c01f99, that file can be accessed
  using /.gfid/f3142503-c75e-45b1-b92a-463cf4c01f99

  .gfid is a virtual directory used to seperate out the namespace
  for accessing files through gfid. This way, we do not conflict with
  filenames which can be qualified as uuids.

* A new file/directory/symlink can be created with a pre-specified
  gfid. A setxattr done on parent directory with fuse_auxgfid_newfile_args_t
  initialized with appropriate fields as value to key "glusterfs.gfid.newfile"
  results in the entry /bname whose gfid is set to args.gfid. The
  contents of the structure should be in network byte order.

  struct auxfuse_symlink_in {
        char     linkpath[]; /* linkpath is a null terminated string */
  } __attribute__ ((__packed__));

  struct auxfuse_mknod_in {
        unsigned int   mode;
        unsigned int   rdev;
        unsigned int   umask;
  } __attribute__ ((__packed__));

  struct auxfuse_mkdir_in {
        unsigned int   mode;
        unsigned int   umask;
  } __attribute__ ((__packed__));

  typedef struct {
        unsigned int  uid;
        unsigned int  gid;
        char          gfid[UUID_CANONICAL_FORM_LEN + 1]; /* a null terminated gfid string
                                                      * in canonical form.
                                                      */
        unsigned int  st_mode;
        char          bname[];     /* bname is a null terminated string */

        union {
                struct auxfuse_mkdir_in   mkdir;
                struct auxfuse_mknod_in   mknod;
                struct auxfuse_symlink_in symlink;
        } __attribute__ ((__packed__)) args;
  } __attribute__ ((__packed__)) fuse_auxgfid_newfile_args_t;

  An initial consumer of this feature would be geo-replication to
  create files on slave mount with same gfids as that on master.
  It will also help gsyncd to access files directly through their
  gfids. gsyncd in its newer version will be consuming a changelog
  (of master) containing operations on gfids and sync corresponding
  files to slave.

* Also, bring in support to heal gfids with a specific value.
  fuse-bridge sends across a gfid during a lookup, which storage
  translators assign to an inode (file/directory etc) if there is
  no gfid associated it. This patch brings in support
  to specify that gfid value from an application, instead of relying
  on random gfid generated by fuse-bridge.

  gfids can be healed through setxattr interface. setxattr should be
  done on parent directory. The key used is "glusterfs.gfid.heal"
  and the value should be the following structure whose contents
  should be in network byte order.

  typedef struct {
        char      gfid[UUID_CANONICAL_FORM_LEN + 1]; /* a null terminated gfid
                                                      * string in canonical form
                                                      */
        char      bname[]; /* a null terminated basename */
  } __attribute__((__packed__)) fuse_auxgfid_heal_args_t;

  This feature can be used for upgrading older geo-rep setups where gfids
  of files are different on master and slave to newer setups where they
  should be same. One can delete gfids on slave using setxattr -x and
  .glusterfs and issue stat on all the files with gfids from master.

Thanks to "Amar Tumballi"  and "Csaba Henk"
 for their inputs.

Signed-off-by: Raghavendra G 
Change-Id: Ie8ddc0fb3732732315c7ec49eab850c16d905e4e
BUG: 952029
Reviewed-on: http://review.gluster.com/#/c/4702
Reviewed-by: Amar Tumballi 
Tested-by: Amar Tumballi 
Reviewed-on: http://review.gluster.org/4702
Reviewed-by: Xavier Hernandez 
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

core: avoid infinite mutual recursion between THIS and gf_log

2013-07-19T06:12:11+00:00

This caused a stack overflow when (for some unknown reason) pthread_setspecific
was failing.  Therefore __glusterfs_this_location called gf_log which used THIS
which wraps __glusterfs_this_location which . . . you get the idea.  We have to
break the loop somewhere, and we can't reasonably make _gf_log stop using THIS,
so we make __glusterfs_this_location stop using _gf_log.

Change-Id: I79c3ea40dd7980bb8ac76a52cdbf5c057b2e1c3c
Signed-off-by: Jeff Darcy 
Reviewed-on: http://review.gluster.org/5341
Reviewed-by: Krishnan Parthasarathi 
Reviewed-by: Raghavendra G 
Reviewed-by: Raghavendra Bhat 
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

include for PATH_MAX

2013-07-18T14:58:51+00:00

I need to include  in order to use PATH_MAX, Otherwise it will
not build at mine. I believe it is standard compliant to do so:
http://pubs.opengroup.org/onlinepubs/009695399/basedefs/limits.h.html

BUG: 764655
Change-Id: I3f124466f7f7742e94a9d1256bc9239ec16aab04
Signed-off-by: Emmanuel Dreyfus 
Reviewed-on: http://review.gluster.org/5340
Reviewed-by: Kaleb KEITHLEY 
Reviewed-by: Jeff Darcy 
Tested-by: Gluster Build System

locks: Added an xdata-based 'cmd' for inodelk count in a given domain

2013-07-18T08:38:16+00:00

Following is the semantics of the 'cmd':
1) If @domain is NULL - returns no. of locks blocked/granted in all domains
2) If @domain is non-NULL- returns no. of locks blocked/granted in that
domain
3) If @domain is non-existent - returns '0'; This is important since
locks xlator creates a domain in a lazy manner.

where @domain - a string representing the domain.

Change-Id: I5e609772343acc157ca650300618c1161efbe72d
BUG: 951195
Original-author: Krishnan Parthasarathi 
Signed-off-by: Krishnan Parthasarathi 
Signed-off-by: shishir gowda 
Reviewed-on: http://review.gluster.org/4889
Tested-by: Gluster Build System 
Reviewed-by: Amar Tumballi

cluster/*: get logic to calculate min() of the 'stime' xattr

2013-07-15T04:06:42+00:00

* in both distribute and replicate (ignoring stripe for now),
  add logic to calculate the min() of stime values.

* What is a 'stime' ? Why is this required:
  -  stime means 'slave xtime', mainly used to keep track of slave
  node's sync status when distributed geo-replication is used.
  Logic of calculating 'min()' for this stime is very important as
  in case of crashes/reboots/shutdown, we will have to 'restart'
  with crawling from stime time value from the mount point, which
  gives the 'min()' of all the bricks, which means, we don't miss
  syncing any files in the above cases.

Change-Id: I2be8d434326572be9d4986db665570a6181db1ee
BUG: 847839
Original Author: Amar Tumballi 
Signed-off-by: Avra Sengupta 
Reviewed-on: http://review.gluster.org/4893
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur