glusterfs.git/xlators, branch v3.5.1beta

rpc: implement server.manage-gids for group resolving on the bricks

2014-05-23T08:34:02+00:00

The new volume option 'server.manage-gids' can be enabled in
environments where a user belongs to more than the current absolute
maximum of 93 groups. This option triggers the following behavior:

1. The AUTH_GLUSTERFS structure sent by GlusterFS clients (fuse, nfs or
   libgfapi) will contain only one (1) auxiliary group, instead of
   a full list. This reduces network usage and prevents problems in
   encoding the AUTH_GLUSTERFS structure which should fit in 400 bytes.
2. The single group in the RPC Calls received by the server is replaced
   by resolving the groups server-side. Permission checks and similar in
   lower xlators are applied against the full list of groups where the
   user belongs to, and not the single auxiliary group that the client
   sent.

Cherry picked from commit 2fd499d148fc8865c77de8b2c73fe0b7e1737882:
> BUG: 1053579
> Signed-off-by: Niels de Vos 
> Reviewed-on: http://review.gluster.org/7501
> Tested-by: Gluster Build System 
> Reviewed-by: Santosh Pradhan 
> Reviewed-by: Harshavardhana 
> Reviewed-by: Anand Avati 

Change-Id: I9e540de13e3022f8b63ff893ecba511129a47b91
BUG: 1096425
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/7830
Tested-by: Gluster Build System 
Reviewed-by: Santosh Pradhan

rpc: warn and truncate grouplist if RPC/AUTH can not hold everything

2014-05-22T13:02:21+00:00

The GlusterFS protocol currently uses AUTH_GLUSTERFS_V2 in the RPC/AUTH
header. This header contains the uid, gid and auxiliary groups of the
user/process that accesses the Gluster Volume.

The AUTH_GLUSTERFS_V2 structure allows up to 65535 auxiliary groups to
be passed on. Unfortunately, the RPC/AUTH header is limited to 400 bytes
by the RPC specification: http://tools.ietf.org/html/rfc5531#section-8.2

In order to not cause complete failures on the client-side when trying
to encode a AUTH_GLUSTERFS_V2 that would result in more than 400 bytes,
we can calculate the expected size of the other elements:

    1 | pid
    1 | uid
    1 | gid
    1 | groups_len
   XX | groups_val (GF_MAX_AUX_GROUPS=65535)
    1 | lk_owner_len
   YY | lk_owner_val (GF_MAX_LOCK_OWNER_LEN=1024)
  ----+-------------------------------------------
    5 | total xdr-units

  one XDR-unit is defined as BYTES_PER_XDR_UNIT = 4 bytes
  MAX_AUTH_BYTES = 400 is the maximum, this is 100 xdr-units.
  XX + YY can be 95 to fill the 100 xdr-units.

  Note that the on-wire protocol has tighter requirements than the
  internal structures. It is possible for xlators to use more groups and
  a bigger lk_owner than that can be sent by a GlusterFS-client.

This change prevents overflows when allocating the RPC/AUTH header. Two
new macros are introduced to calculate the number of groups that fit in
the RPC/AUTH header, when taking the size of the lk_owner in account. In
case the list of groups exceeds the maximum possible, only the first
groups are passed over the RPC/GlusterFS protocol to the bricks.
A warning is added to the logs, so that most system administrators will
get informed.

The reducing of the number of groups is not a new inventions. The
RPC/AUTH header (AUTH_SYS or AUTH_UNIX) that NFS uses has a limit of 16
groups. Most, if not all, NFS-clients will reduce any bigger number of
groups to 16. (nfs.server-aux-gids can be used to workaround the limit
of 16 groups, but the Gluster NFS-server will be limited to a maximum of
93 groups, or fewer in case the lk_owner structure contains more items.)

Cherry picked from commit 8235de189845986a535d676b1fd2c894b9c02e52:
> BUG: 1053579
> Signed-off-by: Niels de Vos 
> Reviewed-on: http://review.gluster.org/7202
> Tested-by: Gluster Build System 
> Reviewed-by: Harshavardhana 
> Reviewed-by: Santosh Pradhan 
> Reviewed-by: Vijay Bellur 

Change-Id: I8410e59d0fd246d601b54b961d3ae9cb5a858c10
BUG: 1096425
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/7829
Reviewed-by: Lalatendu Mohanty 
Tested-by: Gluster Build System

cluster/afr: remove unused variable

2014-05-15T13:31:03+00:00

Fix a compiler warning about unused variable. Missed to catch this in
http://review.gluster.org/7723

Change-Id: Iac97051edb97bdd7dbc5f0fd0f2528d4d404d16d
BUG: 1096040
Signed-off-by: Ravishankar N 
Reviewed-on: http://review.gluster.org/7765
Tested-by: Gluster Build System 
Reviewed-by: Niels de Vos

cluster/afr: send opendirs to all children for entry self-heal

2014-05-14T08:34:42+00:00

Problem: In entry self-heal, opendir was sent only to one source because of
which afr_sh_erase_pending() failed to clear the changelogs of other sources. So
heals were happening multiple times.

Fix :Send opendir to all sources.

Change-Id: Ief4f131848b24a0da782f29b9f1d40e136d3fcff
BUG: 1096040
Signed-off-by: Ravishankar N 
Reviewed-on: http://review.gluster.org/7723
Reviewed-by: Pranith Kumar Karampuri 
Tested-by: Gluster Build System 
Reviewed-by: Niels de Vos

cluster/afr: Remove eager-lock stub on finodelk failure

2014-05-14T08:25:44+00:00

Problem:
For write fops afr's transaction eager-lock init adds transactions
that can share eager-lock to fdctx list. But if eager-lock finodelk
fop fails the stub remains in the list. This could later lead to
corruption of the list and lead to infinite loop on the list
leading to a mount hang.

Fix:
Remove the stub when finodelk fails.

Change-Id: Ic9d1368907c32edb4ea2e6db623e869e4f50180d
BUG: 1063190
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/7748
Tested-by: Gluster Build System 
Reviewed-by: Niels de Vos 
Tested-by: Niels de Vos

libgfapi: Added support to fetch volume info from glusterd and store in glfs object.

2014-05-12T15:14:52+00:00

Defined new APIs in the libgfapi module, given a glfs object,
 * to send handshake RPC call to glusterd process to fetch UUID of the volume
 * store it in the glusterfs_context linked to the glfs object.
 * to parse UUID from its cannonical string format into 16-byte array
   before sending it to the libgfapi users.

Defined a RPC call in glusterd which can be used to query volume related
info by other processes using 'clnt_handshake_procs'.

Note - Currently this RPC call to glusterd process is used only to fetch UUID.
But it can be extended to get other volume related structures as well.

In addition to the above, defined a new variable to keep track of such handshake
RPCs still in progress to make sure all the corresponding RPC callbacks have been
processed before libgfapi returns the glfs object initialized.

Also bumping up the GFAPI current version number since there is a new API
"glfs_get_volume_id" defined and exposed by libgfapi as part of these changes.

Cherry picked from commit 5adb10b9ac1c634334f29732e062b12d747ae8c5:
> Change-Id: I303f76d7177d32d25bdb301b1dbcf5cd73f42807
> BUG: 1095775
> Signed-off-by: Soumya Koduri 
> Reviewed-on: http://review.gluster.org/7218
> Reviewed-by: Anand Avati 
> Reviewed-by: Harshavardhana 
> Tested-by: Gluster Build System 
> Reviewed-by: Vijay Bellur 

This change differs a little from the patch in the master branch:
- libgfapi in 3.5 does not have glfs_get_volfile(), so there were some
  merge conflicts resolved,
- libgfapi in 3.5 is not versioned, the configure.ac changes related to
  the versioning have been skipped,
- in the master branch only the XDR .x files are available, release-3.5
  requires a manual re-generation of the relates .h and .c files.

Change-Id: I52c32d0e69a52a7f4285f74164bca6fd83c4f3b3
BUG: 1095775
Signed-off-by: Soumya Koduri 
Reviewed-on: http://review.gluster.org/7741
Reviewed-by: Niels de Vos 
Tested-by: Niels de Vos

Removed the old and unused non-Glupy Python bindings code

2014-05-11T10:39:41+00:00

Cherry picked from commit 0a8abcc5d0b727746a2ac6c0d5729700bfea742b:
> BUG: 1064096
> Change-Id: Ifd34f46f0f0b45bbc27d35953e6ecbca36753886
> Signed-off-by: Justin Clift 
> Reviewed-on: http://review.gluster.org/6980
> Reviewed-by: Jeff Darcy 

Change-Id: I330dca1d53cb85ffc9ee823afa6bca056f34c824
BUG: 1064096
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/7731
Tested-by: Gluster Build System

features/marker-quota: more stringent error handling in rename.

2014-05-11T02:36:10+00:00

If an error occurs and op_errno is not set to non-zero value, we can
end up in loosing a frame resulting in a hung syscall. This patch
adds code setting op_errno appropriately in storage/posix and
makes marker to set err to a default non-zero value in case of
op_errno being zero.

Change-Id: Idc2c3e843b932709a69b32ba67deb284547168f2
BUG: 833586
Signed-off-by: Raghavendra G 
Reviewed-on: http://review.gluster.org/6913
Tested-by: Gluster Build System 
Reviewed-by: Niels de Vos

cluster/afr: Fix bugs in quorum implementation

2014-05-10T20:30:48+00:00

- Have common place to perform quorum fop wind check
- Check if fop succeeded in a way that matches quorum
  to avoid marking changelog in split-brain.

Change-Id: I663072ece0e1de6e1ee9fccb03e1b6c968793bc5
BUG: 1066996
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/7513
Reviewed-by: Ravishankar N 
Tested-by: Gluster Build System 
Reviewed-by: Niels de Vos

posix: if brick-uid or brick-gid is not specified, do not set

2014-05-09T14:36:02+00:00

Current code would set owner uid/gid explicitly to 0/0 on start
even if none was specified. Fix it.

Cherry picked from commit 8bdc329:
> Change-Id: I72dec9e79c51bd1eb3af5334c42b7c23b01d0258
> BUG: 1040275
> Signed-off-by: Anand Avati 
> Reviewed-on: http://review.gluster.org/6476
> Tested-by: Gluster Build System 
> Tested-by: Lukáš Bezdička 
> Reviewed-by: Krishnan Parthasarathi 
> Reviewed-by: Vijay Bellur 

Change-Id: Ie0396c1a4e6e0979ea9c855d33db963544a75c42
BUG: 1095971
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/7720
Tested-by: Gluster Build System