glusterfs.git, branch v3.5.1beta

logging: remove unused message-id scripts

2014-05-24T15:26:08+00:00

The current unused implementation for message-ids in the logs depends on
automatically generated files. The generated files are not included in
the distributed tarball. This causes issues when distributions build
packages, they need to re-run ./autogen.sh to create the needed files.

I thought of including the generated files in the distribution tarball.
However, the contents of these files are not actively used, so it seems
to make more sense to drop it all together. These functions were the
only users of libintl and gettext too, so dropped the requirement
checking from configure.ac.

A replacement for the message-id logging framework is in progress. Any
changes that this patch makes, can be reverted in the submission of
patches for the new framework.

Cherry picked from commit cec37c9b66b8711b213f114875d215f56b8120aa:
> Reference: http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/6212
> Change-Id: Iea82dd3910944a5c6be3ee393806eccabd575e11
> BUG: 1038391
> Signed-off-by: Niels de Vos 
> Reviewed-on: http://review.gluster.org/7714
> Tested-by: Gluster Build System 
> Reviewed-by: Vijay Bellur 

Change-Id: Iea82dd3910944a5c6be3ee393806eccabd575e11
BUG: 1089054
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/7851
Tested-by: Gluster Build System

doc: distributed geo-replication

2014-05-23T23:55:06+00:00

Change-Id: Id89089aa52066634e8a6428c2bf00ea4f30f1943
BUG: 1086758
Signed-off-by: Venky Shankar 
Reviewed-on: http://review.gluster.org/7864
Reviewed-by: Humble Devassy Chirammal 
Reviewed-by: Niels de Vos 
Tested-by: Niels de Vos

tests/rpm: always run ./autogen.sh to create missing files

2014-05-23T18:20:56+00:00

In some occasions 'install-sh' seems to be missing in the 'make dist'
tarball when ./autogen.sh has not been run (skipped when 'configure'
exists). With this changes, 'autogen.sh' is always run so that missing
files should get added to the tarball.

Also write the logs from mock to a known location, and copy them to the
'/var/log/' directory that Jenkins archives after a regression test
failure. This makes it easier to find build issues that mock detects.

Cherry picked from commit 79de5b63775f1ab8e2e498fd51b55509a30cd896:
> Change-Id: I8d0cf1afef61ebab0137aa0d20521e75a35ddbdd
> BUG: 1038391
> Signed-off-by: Niels de Vos 
> Reviewed-on: http://review.gluster.org/7786
> Tested-by: Gluster Build System 
> Reviewed-by: Kaleb KEITHLEY 
> Reviewed-by: Anand Avati 

Change-Id: I8d0cf1afef61ebab0137aa0d20521e75a35ddbdd
BUG: 1089054
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/7850
Reviewed-by: Kaleb KEITHLEY 
Tested-by: Gluster Build System

doc: changelog consumer library

2014-05-23T14:10:53+00:00

Change-Id: I1e641e1a61b78c84ed610a56eabdad1075865cd3
BUG: 1086758
Signed-off-by: Venky Shankar 
Reviewed-on: http://review.gluster.org/7856
Reviewed-by: Humble Devassy Chirammal 
Reviewed-by: Niels de Vos 
Tested-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/7863

rpc: implement server.manage-gids for group resolving on the bricks

2014-05-23T08:34:02+00:00

The new volume option 'server.manage-gids' can be enabled in
environments where a user belongs to more than the current absolute
maximum of 93 groups. This option triggers the following behavior:

1. The AUTH_GLUSTERFS structure sent by GlusterFS clients (fuse, nfs or
   libgfapi) will contain only one (1) auxiliary group, instead of
   a full list. This reduces network usage and prevents problems in
   encoding the AUTH_GLUSTERFS structure which should fit in 400 bytes.
2. The single group in the RPC Calls received by the server is replaced
   by resolving the groups server-side. Permission checks and similar in
   lower xlators are applied against the full list of groups where the
   user belongs to, and not the single auxiliary group that the client
   sent.

Cherry picked from commit 2fd499d148fc8865c77de8b2c73fe0b7e1737882:
> BUG: 1053579
> Signed-off-by: Niels de Vos 
> Reviewed-on: http://review.gluster.org/7501
> Tested-by: Gluster Build System 
> Reviewed-by: Santosh Pradhan 
> Reviewed-by: Harshavardhana 
> Reviewed-by: Anand Avati 

Change-Id: I9e540de13e3022f8b63ff893ecba511129a47b91
BUG: 1096425
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/7830
Tested-by: Gluster Build System 
Reviewed-by: Santosh Pradhan

rpc: warn and truncate grouplist if RPC/AUTH can not hold everything

2014-05-22T13:02:21+00:00

The GlusterFS protocol currently uses AUTH_GLUSTERFS_V2 in the RPC/AUTH
header. This header contains the uid, gid and auxiliary groups of the
user/process that accesses the Gluster Volume.

The AUTH_GLUSTERFS_V2 structure allows up to 65535 auxiliary groups to
be passed on. Unfortunately, the RPC/AUTH header is limited to 400 bytes
by the RPC specification: http://tools.ietf.org/html/rfc5531#section-8.2

In order to not cause complete failures on the client-side when trying
to encode a AUTH_GLUSTERFS_V2 that would result in more than 400 bytes,
we can calculate the expected size of the other elements:

    1 | pid
    1 | uid
    1 | gid
    1 | groups_len
   XX | groups_val (GF_MAX_AUX_GROUPS=65535)
    1 | lk_owner_len
   YY | lk_owner_val (GF_MAX_LOCK_OWNER_LEN=1024)
  ----+-------------------------------------------
    5 | total xdr-units

  one XDR-unit is defined as BYTES_PER_XDR_UNIT = 4 bytes
  MAX_AUTH_BYTES = 400 is the maximum, this is 100 xdr-units.
  XX + YY can be 95 to fill the 100 xdr-units.

  Note that the on-wire protocol has tighter requirements than the
  internal structures. It is possible for xlators to use more groups and
  a bigger lk_owner than that can be sent by a GlusterFS-client.

This change prevents overflows when allocating the RPC/AUTH header. Two
new macros are introduced to calculate the number of groups that fit in
the RPC/AUTH header, when taking the size of the lk_owner in account. In
case the list of groups exceeds the maximum possible, only the first
groups are passed over the RPC/GlusterFS protocol to the bricks.
A warning is added to the logs, so that most system administrators will
get informed.

The reducing of the number of groups is not a new inventions. The
RPC/AUTH header (AUTH_SYS or AUTH_UNIX) that NFS uses has a limit of 16
groups. Most, if not all, NFS-clients will reduce any bigger number of
groups to 16. (nfs.server-aux-gids can be used to workaround the limit
of 16 groups, but the Gluster NFS-server will be limited to a maximum of
93 groups, or fewer in case the lk_owner structure contains more items.)

Cherry picked from commit 8235de189845986a535d676b1fd2c894b9c02e52:
> BUG: 1053579
> Signed-off-by: Niels de Vos 
> Reviewed-on: http://review.gluster.org/7202
> Tested-by: Gluster Build System 
> Reviewed-by: Harshavardhana 
> Reviewed-by: Santosh Pradhan 
> Reviewed-by: Vijay Bellur 

Change-Id: I8410e59d0fd246d601b54b961d3ae9cb5a858c10
BUG: 1096425
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/7829
Reviewed-by: Lalatendu Mohanty 
Tested-by: Gluster Build System

libgfapi: Added Handle-based ops to get/set/remove extended attributes in the libgfapi.

2014-05-22T09:33:49+00:00

Cherry picked from commit Change-Id: I1a8e666018d7b93e0bba2d9882935681da909980
and Change-Id: I62f63da37edf722d6d79c75f72ee7403e93e4936

> BUG: 1089414
> Signed-off-by: Soumya Koduri 
> Reviewed-on: http://review.gluster.org/7308
> Reviewed-by: Vijay Bellur 
> Tested-by: Vijay Bellur 

This patch differs a bit from its earlier master patch -
* 'DECODE_SYNCOP_ERR' macro is not defined in this code branch.
So lines where this macro is used are skipped.
* 'syncop_removexattr(..)' definition differs from the one in the master branch.

BUG: 1099878
Change-Id: I1e9cce4efeec038b9736065d39887c35752caead
Signed-off-by: Soumya Koduri 
Reviewed-on: http://review.gluster.org/7825
Reviewed-by: Raghavendra G 
Tested-by: Gluster Build System 
Reviewed-by: Niels de Vos

doc: Add block device xlator details

2014-05-21T16:11:06+00:00

> Change-Id: I4c24a7f0a4339364a80e5d1be0fc8de6b9c189c1
> BUG: 1086762
> Signed-off-by: Humble Chirammal 
> Reviewed-on: http://review.gluster.org/7809
> Reviewed-by: Niels de Vos 
> Reviewed-by: Humble Devassy Chirammal 
> Reviewed-by: Vijay Bellur 
> Tested-by: Vijay Bellur 
(cherry picked from commit 99afc640f27841ec819c0f32c01f567fd0b345f8)
Signed-off-by: Humble Chirammal 

Change-Id: I4ccbc234d0c88674e855d91529e286b24f1bed84
BUG: 1086762
Reviewed-on: http://review.gluster.org/7832
Reviewed-by: Niels de Vos 
Tested-by: Niels de Vos

doc: zerofill api documentation

2014-05-21T12:35:08+00:00

Change-Id: I437735d68fdcc4a55f12fd51b1d40cd7a32ad783
BUG: 1086756
Signed-off-by: Humble Chirammal 
Reviewed-on: http://review.gluster.org/7800
Reviewed-by: Niels de Vos 
Reviewed-by: Anand Avati 
Tested-by: Anand Avati 
Reviewed-on: http://review.gluster.org/7822
Tested-by: Niels de Vos

doc: afr-statistics command documentation

2014-05-20T16:15:50+00:00

BUG: 1086748
Change-Id: I67ca9958e4e369b5902fc32d8d9863073da76c77
Signed-off-by: Venkatesh Somyajulu 
Reviewed-on: http://review.gluster.org/7792
Reviewed-by: Humble Devassy Chirammal 
Tested-by: Niels de Vos