glusterfs.git/xlators/features, branch v3.11.0rc0

Halo Replication feature for AFR translator

2017-05-08T05:37:07+00:00

	Backport of https://review.gluster.org/16177
		    https://review.gluster.org/17174

Merged both these patches to make sure IPV6 changes don't make it to 3.11 at all.

Summary:
Halo Geo-replication is a feature which allows Gluster or NFS clients to write
locally to their region (as defined by a latency "halo" or threshold if you
like), and have their writes asynchronously propagate from their origin to the
rest of the cluster.  Clients can also write synchronously to the cluster
simply by specifying a halo-latency which is very large (e.g. 10seconds) which
will include all bricks.

In other words, it allows clients to decide at mount time if they desire
synchronous or asynchronous IO into a cluster and the cluster can support both
of these modes to any number of clients simultaneously.

There are a few new volume options due to this feature:
  halo-shd-latency:  The threshold below which self-heal daemons will
  consider children (bricks) connected.

  halo-nfsd-latency: The threshold below which NFS daemons will consider
  children (bricks) connected.

  halo-latency: The threshold below which all other clients will
  consider children (bricks) connected.

  halo-min-replicas: The minimum number of replicas which are to
  be enforced regardless of latency specified in the above 3 options.
  If the number of children falls below this threshold the next
  best (chosen by latency) shall be swapped in.

New FUSE mount options:
  halo-latency & halo-min-replicas: As descripted above.

This feature combined with multi-threaded SHD support (D1271745) results in
some pretty cool geo-replication possibilities.

Operational Notes:
- Global consistency is gaurenteed for synchronous clients, this is provided by
  the existing entry-locking mechanism.
- Asynchronous clients on the other hand and merely consistent to their region.
  Writes & deletes will be protected via entry-locks as usual preventing
  concurrent writes into files which are undergoing replication.  Read operations
  on the other hand should never block.
- Writes are allowed from _any_ region and propagated from the origin to all
  other regions.  The take away from this is care should be taken to ensure
  multiple writers do not write the same files resulting in a gfid split-brain
  which will require resolution via split-brain policies (majority, mtime &
  size).  Recommended method for preventing this is using the nfs-auth feature to
  define which region for each share has RW permissions, tiers not in the origin
  region should have RO perms.

TODO:
- Synchronous clients (including the SHD) should choose clients from their own
  region as preferred sources for reads.  Most of the plumbing is in place for
  this via the child_latency array.
- Better GFID split brain handling & better dent type split brain handling
  (i.e. create a trash can and move the offending files into it).
- Tagging in addition to latency as a means of defining which children you wish
  to synchronously write to

Test Plan:
- The usual suspects, clang, gcc w/ address sanitizer & valgrind
- Prove tests

Reviewers: jackl, dph, cjh, meyering

Reviewed By: meyering

Subscribers: ethanr

Differential Revision: https://phabricator.fb.com/D1272053

Tasks: 4117827

 >Change-Id: I694a9ab429722da538da171ec528406e77b5e6d1
 >BUG: 1428061
 >Signed-off-by: Kevin Vigor 
 >Reviewed-on: http://review.gluster.org/16099
 >Reviewed-on: https://review.gluster.org/16177
 >Tested-by: Pranith Kumar Karampuri 
 >Smoke: Gluster Build System 
 >NetBSD-regression: NetBSD Build System 
 >CentOS-regression: Gluster Build System 
 >Reviewed-by: Pranith Kumar Karampuri 

BUG: 1448416
Change-Id: I694a9ab429722da538da171ec528406e77b5e6d1
Signed-off-by: Pranith Kumar K 
Reviewed-on: https://review.gluster.org/17192
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kaushal M

SELinux : implementation of SELinux translator

2017-05-04T12:20:26+00:00

The patch implement a part of SELinux translator to support setting
SELinux contexts on files in a glusterfs volume.

URL: https://github.com/gluster/glusterfs-specs/blob/master/accepted/SELinux-client-support.md

Upstream reference :
>Change-Id: Id8916bd8e064ccf74ba86225ead95f86dc5a1a25
>BUG: 1318100
>Fixes : #55
>Signed-off-by: Manikandan Selvaganesh 
>Signed-off-by: Jiffin Tony Thottan 
>Signed-off-by: Niels de Vos 
>Reviewed-on: https://review.gluster.org/13762
>Smoke: Gluster Build System 
>NetBSD-regression: NetBSD Build System 
>CentOS-regression: Gluster Build System 
>Reviewed-by: Manikandan Selvaganesh 
>Reviewed-by: Atin Mukherjee 

Change-Id: Id8916bd8e064ccf74ba86225ead95f86dc5a1a25
BUG: 1447597
Signed-off-by: Jiffin Tony Thottan 
Reviewed-on: https://review.gluster.org/17159
Reviewed-by: Niels de Vos 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

libglusterfs/stack.h: reduce duplication of code

2017-04-27T14:54:40+00:00

* Use STACK_UNWIND_STRICT everywhere.
* Provide STACK_WIND_COMMON as both STACK_WIND_COOKIE
  and STACK_WIND differ by just 1 line and 1 option.

Updates gluster/glusterfs#137

Change-Id: Ifbb6b9c4702b02f4a02834824f509fd10c78f0ce
Signed-off-by: Amar Tumballi 
Reviewed-on: https://review.gluster.org/16915
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Jeff Darcy

features/bit-rot-stub: bring in optional versioning

2017-04-19T03:29:46+00:00

* As of now bit-rot-stub does versioning always. This leads
  lots of getxattr calls being made in lookups. So make
  object versioning optional.

Change-Id: I83713e45ae59fb28004bb3cfa008f2d69edebbfa
BUG: 1359599
Signed-off-by: Raghavendra Bhat 
Signed-off-by: Kotresh HR 
Reviewed-on: https://review.gluster.org/14442
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Vijay Bellur

worm: add check for internal processes in ftruncate()

2017-04-18T18:35:12+00:00

The patch fixes the recently seen issues with worm_sh.t test.

RCA:

-
$ git log --oneline xlators/features/read-only/src/worm.c
1b01bdc worm: allow Self-heal-Daemon to perform some operations
c5a4a77 features/worm: Adding implementation for ftruncate
-

These two patches were merged in reverse order of their submission,
and hence the check added for internal processes got missed in
new fop 'ftruncate()'. The worm_sh.t passed the tests as while
that patch got submitted there was no ftruncate() in worm xlator.

Change-Id: I81a8a45fa2679917a2c859c4f5224a2c3edbc784
BUG: 1423413
Signed-off-by: Amar Tumballi 
Reviewed-on: https://review.gluster.org/17048
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Zhou Zhengping 
Reviewed-by: David Spisla 
Reviewed-by: Vijay Bellur

worm: allow Self-heal-Daemon to perform some operations

2017-04-12T11:27:09+00:00

The Self-Heal-Daemon should be allowed to trigger unlink, link,
trauncate, rename and write operation. The value of frame->root->pid
can be used to detect internal (by SHD) operations.

Change-Id: I7526148100bef1e2837d69df5c119dc97d91fffd
BUG: 1423413
Signed-off-by: David Spisla 
Reviewed-on: https://review.gluster.org/16661
Tested-by: jiffin tony Thottan 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos 
Smoke: Gluster Build System 
Reviewed-by: jiffin tony Thottan 
Reviewed-by: Amar Tumballi

features/shard: Initialize local->fop in readv

2017-04-11T13:32:19+00:00

Change-Id: I9008ca9960df4821636501ae84f93a68f370c67f
BUG: 1440051
Signed-off-by: Krutika Dhananjay 
Reviewed-on: https://review.gluster.org/17014
NetBSD-regression: NetBSD Build System 
Reviewed-by: Pranith Kumar Karampuri 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Vijay Bellur

upcall: correct the cleanup path in init() when GF_CALLOC() fails

2017-04-10T15:19:24+00:00

Coverity found that "priv" can be NULL when the 1st GF_CALLOC() fails,
and still is being used with "priv->xattrs" before the functon returns.

While cleaning this up, also removing the log message for out of memory,
as this is already logged through GF_CALLOC().

BUG: 789278
Change-Id: I887eaa9136dc25a39c107cd2152b1ba11f9fa925
Signed-off-by: Niels de Vos 
Reviewed-on: https://review.gluster.org/17023
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: soumya k 
CentOS-regression: Gluster Build System 
Reviewed-by: Amar Tumballi

features/shard: Fix vm corruption upon fix-layout

2017-04-07T18:52:00+00:00

shard's writev implementation, as part of identifying
presence of participant shards that aren't in memory,
first sends an MKNOD on these shards, and upon EEXIST error,
looks up the shards before proceeding with the writes.

The VM corruption was caused when the following happened:
1. DHT had n subvolumes initially.
2. Upon add-brick + fix-layout, the layout of .shard changed
   although the existing shards under it were yet to be migrated
   to their new hashed subvolumes.
3. During this time, there were writes on the VM falling in regions
   of the file whose corresponding shards were already existing under
   .shard.
4. Sharding xl sent MKNOD on these shards, now creating them in their
   new hashed subvolumes although there already exist shard blocks for
   this region with valid data.
5. All subsequent writes were wound on these newly created copies.

The net outcome is that both copies of the shard didn't have the correct
data. This caused the affected VMs to be unbootable.

FIX:
For want of better alternatives in DHT, the fix changes shard fops to do
a LOOKUP before the MKNOD and upon EEXIST error, perform another lookup.

Change-Id: I8a2e97d91ba3275fbc7174a008c7234fa5295d36
BUG: 1440051
RCA'd-by: Raghavendra Gowdappa 
Reported-by: Mahdi Adnan 
Signed-off-by: Krutika Dhananjay 
Reviewed-on: https://review.gluster.org/17010
Smoke: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

shared: simplify and fix crash when we run out of memory

2017-04-07T03:48:06+00:00

Since mem_get0 can return NULL, local->op_ret is gonna
crash. Found by coverity. And since we only have ENOMEM
as potential error, we can also simplify the code by avoiding
using 'local' for that.

Change-Id: I778747b57f520b1a52347c0fc9f27efd7a7c5ca0
BUG: 789278
Signed-off-by: Michael Scherer 
Reviewed-on: https://review.gluster.org/16739
Smoke: Gluster Build System 
Tested-by: Michael Scherer 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Vijay Bellur 
Reviewed-by: Krutika Dhananjay 
Reviewed-by: Pranith Kumar Karampuri