glusterfs.git, branch v3.11.0rc0

doc: Added initial version of 3.11.0 release-notes

2017-05-08T18:15:00+00:00

This is just a skeleton, each feature further needs to provide
additional content as noted.

Further list of bugs fixed needs to be added to the list.

Change-Id: Id909cd4559c8d828eb9a77a0b4f27b7b59dfca4a
Signed-off-by: Shyam 
Reviewed-on: https://review.gluster.org/17207
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System

Halo Replication feature for AFR translator

2017-05-08T05:37:07+00:00

	Backport of https://review.gluster.org/16177
		    https://review.gluster.org/17174

Merged both these patches to make sure IPV6 changes don't make it to 3.11 at all.

Summary:
Halo Geo-replication is a feature which allows Gluster or NFS clients to write
locally to their region (as defined by a latency "halo" or threshold if you
like), and have their writes asynchronously propagate from their origin to the
rest of the cluster.  Clients can also write synchronously to the cluster
simply by specifying a halo-latency which is very large (e.g. 10seconds) which
will include all bricks.

In other words, it allows clients to decide at mount time if they desire
synchronous or asynchronous IO into a cluster and the cluster can support both
of these modes to any number of clients simultaneously.

There are a few new volume options due to this feature:
  halo-shd-latency:  The threshold below which self-heal daemons will
  consider children (bricks) connected.

  halo-nfsd-latency: The threshold below which NFS daemons will consider
  children (bricks) connected.

  halo-latency: The threshold below which all other clients will
  consider children (bricks) connected.

  halo-min-replicas: The minimum number of replicas which are to
  be enforced regardless of latency specified in the above 3 options.
  If the number of children falls below this threshold the next
  best (chosen by latency) shall be swapped in.

New FUSE mount options:
  halo-latency & halo-min-replicas: As descripted above.

This feature combined with multi-threaded SHD support (D1271745) results in
some pretty cool geo-replication possibilities.

Operational Notes:
- Global consistency is gaurenteed for synchronous clients, this is provided by
  the existing entry-locking mechanism.
- Asynchronous clients on the other hand and merely consistent to their region.
  Writes & deletes will be protected via entry-locks as usual preventing
  concurrent writes into files which are undergoing replication.  Read operations
  on the other hand should never block.
- Writes are allowed from _any_ region and propagated from the origin to all
  other regions.  The take away from this is care should be taken to ensure
  multiple writers do not write the same files resulting in a gfid split-brain
  which will require resolution via split-brain policies (majority, mtime &
  size).  Recommended method for preventing this is using the nfs-auth feature to
  define which region for each share has RW permissions, tiers not in the origin
  region should have RO perms.

TODO:
- Synchronous clients (including the SHD) should choose clients from their own
  region as preferred sources for reads.  Most of the plumbing is in place for
  this via the child_latency array.
- Better GFID split brain handling & better dent type split brain handling
  (i.e. create a trash can and move the offending files into it).
- Tagging in addition to latency as a means of defining which children you wish
  to synchronously write to

Test Plan:
- The usual suspects, clang, gcc w/ address sanitizer & valgrind
- Prove tests

Reviewers: jackl, dph, cjh, meyering

Reviewed By: meyering

Subscribers: ethanr

Differential Revision: https://phabricator.fb.com/D1272053

Tasks: 4117827

 >Change-Id: I694a9ab429722da538da171ec528406e77b5e6d1
 >BUG: 1428061
 >Signed-off-by: Kevin Vigor 
 >Reviewed-on: http://review.gluster.org/16099
 >Reviewed-on: https://review.gluster.org/16177
 >Tested-by: Pranith Kumar Karampuri 
 >Smoke: Gluster Build System 
 >NetBSD-regression: NetBSD Build System 
 >CentOS-regression: Gluster Build System 
 >Reviewed-by: Pranith Kumar Karampuri 

BUG: 1448416
Change-Id: I694a9ab429722da538da171ec528406e77b5e6d1
Signed-off-by: Pranith Kumar K 
Reviewed-on: https://review.gluster.org/17192
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kaushal M

gfapi: Fix few severe coverity issues

2017-05-07T15:34:01+00:00

From https://download.gluster.org/pub/gluster/glusterfs/static-analysis/master/glusterfs-coverity/2017-05-03-6484558c/html/
fixes coverity#189, 248, 250, 488, 546

This is backport of below mainline fix -
- https://review.gluster.org/17161

>Change-Id: I7c4aa5bf6c367344602748dfd736de27f341cca2
>BUG: 789278
>Signed-off-by: Soumya Koduri 
>Reviewed-on: https://review.gluster.org/17161
>Smoke: Gluster Build System 
>Reviewed-by: Niels de Vos 
>NetBSD-regression: NetBSD Build System 
>Reviewed-by: Amar Tumballi 
>CentOS-regression: Gluster Build System 
>(cherry picked from commit fa10c8a3528b95dc575bfedd30bcd68cc2dff25b)

Change-Id: I7c4aa5bf6c367344602748dfd736de27f341cca2
BUG: 1447571
Signed-off-by: Soumya Koduri 
Reviewed-on: https://review.gluster.org/17194
Reviewed-by: Niels de Vos 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System

glusterd: skip nfs svc reconfigure if nfs xlator is not installed

2017-05-07T15:33:44+00:00

With 83abcba, nfs svc is not (re)started or stopped if NFS so file is
not installed. However the same check was missing in nfs svc reconfigure
which was causing all volume set command to fail.

>Reviewed-on: https://review.gluster.org/17149
>Smoke: Gluster Build System 
>NetBSD-regression: NetBSD Build System 
>Reviewed-by: Niels de Vos 
>CentOS-regression: Gluster Build System 
>Reviewed-by: Jeff Darcy 
>(cherry picked from commit 316e3300cfaa646b7fa45fcc7f57b81c7bb15a0e)

Change-Id: Ie87b5dba44ac59e890cbd60f85944f8e685ad52b
BUG: 1447604
Signed-off-by: Atin Mukherjee 
Reviewed-on: https://review.gluster.org/17156
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Shyamsundar Ranganathan

gfapi/handleops: Introducing glfs_xreaddirplus_r() fop for handleops

2017-05-05T11:12:48+00:00

Its known that readdirplus operation fetches stat as well for each of the
dirents. But often applications may need extra information, like for eg.,
NFS-Ganesha which operates on handles needs handles for each of those
dirents returned. So this would require extra calls to the backend, in this
case LOOKUP (which is very expensive operation) resulting in very low
readdir performance.

To address that introducing this new API using which applications can
make request for any extra information to be returned as part of
readdirplus response.

Currently this new api returns stat and handles as demanded by application.
The synopsis of the API is noted in glfs.h.

@todo:
* Enhance test script using this new API

Below were the perf results on single brick volume with and without
these changes -

Dataset used -
10*100 directories and each directory containing 100 empty files.

I used NFS-Ganesha application to test these changes -
>for i in {1..5}; do systemctl restart nfs-ganesha; sleep 10; mount -t nfs -o vers=4 localhost:/brick_vol /mnt; cd /mnt; echo "ITERATION$i"; date; find . > tmp-nfs.log; date; cd /; umount /mnt; sleep 2; done;

Without these changes -
ITERATION1
Mon Mar 20 17:22:26 IST 2017
Mon Mar 20 17:23:18 IST 2017
ITERATION2
Mon Mar 20 17:23:39 IST 2017
Mon Mar 20 17:24:28 IST 2017
ITERATION3
Mon Mar 20 17:24:49 IST 2017
Mon Mar 20 17:25:36 IST 2017
ITERATION4
Mon Mar 20 17:30:57 IST 2017
Mon Mar 20 17:31:37 IST 2017
ITERATION5
Mon Mar 20 17:31:57 IST 2017
Mon Mar 20 17:32:40 IST 2017
[root@dhcp35-197 /]#

On an average ~46.2 sec

With these changes applied -
ITERATION1
Mon Mar 20 17:35:03 IST 2017
Mon Mar 20 17:35:15 IST 2017
ITERATION2
Mon Mar 20 17:35:36 IST 2017
Mon Mar 20 17:35:46 IST 2017
ITERATION3
Mon Mar 20 17:36:06 IST 2017
Mon Mar 20 17:36:17 IST 2017
ITERATION4
Mon Mar 20 17:41:38 IST 2017
Mon Mar 20 17:41:49 IST 2017
ITERATION5
Mon Mar 20 17:42:10 IST 2017
Mon Mar 20 17:42:20 IST 2017

On an average ~10.8 sec

This is backport of below upstream patch -
        https://review.gluster.org/15663

>Updates #174
>BUG: 1442950
>Change-Id: I0f74f74dc62085ca4c4a23c38e3edc84bd850876
>Signed-off-by: Soumya Koduri 
>Reviewed-on: https://review.gluster.org/15663
>Smoke: Gluster Build System 
>NetBSD-regression: NetBSD Build System 
>Reviewed-by: Niels de Vos 
>CentOS-regression: Gluster Build System 

BUG: 1447571
Change-Id: I0f74f74dc62085ca4c4a23c38e3edc84bd850876
Signed-off-by: Soumya Koduri 
Reviewed-on: https://review.gluster.org/17164
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

extras/hook-scripts: SELinux brick file context management scripts

2017-05-04T12:25:14+00:00

The SELinux policy for gluster defines the glusterd_brick_t type to
support server side SELinux (e.g., server side labels). Add
convenience hook scripts that users/packagers can install to ensure
that new bricks are labeled correctly.

The volume create hook script adds a new SELinux file context for
each brick path and runs a restorecon to label the brick. The
volume delete hook removes the per-brick SELinux file context.

Cherry picked from commit 859669759f7fa0f2114add13660ce3bf16c77f30:
> Change-Id: I5f102db5382d813c4d822ff74e873a7a669b41db
> BUG: 1047975
> Signed-off-by: Brian Foster 
> Signed-off-by: Niels de Vos 
> Signed-off-by: Jiffin Tony Thottan 
> Reviewed-on: https://review.gluster.org/6630
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Kaleb KEITHLEY 

Change-Id: I5f102db5382d813c4d822ff74e873a7a669b41db
BUG: 1447597
Signed-off-by: Niels de Vos 
Reviewed-on: https://review.gluster.org/17157
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: jiffin tony Thottan

SELinux : implementation of SELinux translator

2017-05-04T12:20:26+00:00

The patch implement a part of SELinux translator to support setting
SELinux contexts on files in a glusterfs volume.

URL: https://github.com/gluster/glusterfs-specs/blob/master/accepted/SELinux-client-support.md

Upstream reference :
>Change-Id: Id8916bd8e064ccf74ba86225ead95f86dc5a1a25
>BUG: 1318100
>Fixes : #55
>Signed-off-by: Manikandan Selvaganesh 
>Signed-off-by: Jiffin Tony Thottan 
>Signed-off-by: Niels de Vos 
>Reviewed-on: https://review.gluster.org/13762
>Smoke: Gluster Build System 
>NetBSD-regression: NetBSD Build System 
>CentOS-regression: Gluster Build System 
>Reviewed-by: Manikandan Selvaganesh 
>Reviewed-by: Atin Mukherjee 

Change-Id: Id8916bd8e064ccf74ba86225ead95f86dc5a1a25
BUG: 1447597
Signed-off-by: Jiffin Tony Thottan 
Reviewed-on: https://review.gluster.org/17159
Reviewed-by: Niels de Vos 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

libglusterfs: Correct op-version for 3.11

2017-05-03T13:02:43+00:00

Change-Id: Ida58626b3c25ed97f3b0b94b3fa5be58d607fede
BUG: 1447543
Signed-off-by: Kaushal M 
Reviewed-on: https://review.gluster.org/17155
Smoke: Gluster Build System 
Reviewed-by: Niels de Vos 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

core: remove experimental xlators and associated tests

2017-05-03T12:33:13+00:00

experimental xlators not included in 3.11

Cherry picked from 4231c40973c60999f5ef759db450d25e129ef6ba:
> Change-Id: I547480ee5e7912664784643e436feb198b6d16d0
> BUG: 1415866
> Signed-off-by: Kaleb S. KEITHLEY 
> Reviewed-on: https://review.gluster.org/16447
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Atin Mukherjee 
> Reviewed-by: Shyamsundar Ranganathan 

Change-Id: I547480ee5e7912664784643e436feb198b6d16d0
BUG: 1447543
Signed-off-by: Kaushal M 
Reviewed-on: https://review.gluster.org/17154
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Amar Tumballi 
CentOS-regression: Gluster Build System

libgfapi: remove/revert glfs_ipc() changes targeted for 4.0

2017-05-03T12:32:50+00:00

revert glfs_ipc() to 3.7.0 signature and associated symbol versioning

Cherry picked from commit e549df2c0d8b5dbfa6e9c129109185aaf4cea11d:
> Change-Id: Ieffe1d966234652091a4a9ae0b2c4b23f1297147
> BUG: 1415279
> Signed-off-by: Kaleb S. KEITHLEY 
> Reviewed-on: https://review.gluster.org/16450
> Smoke: Gluster Build System 
> Reviewed-by: Shyamsundar Ranganathan 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 

Change-Id: Ieffe1d966234652091a4a9ae0b2c4b23f1297147
BUG: 1447543
Signed-off-by: Kaushal M 
Reviewed-on: https://review.gluster.org/17153
Smoke: Gluster Build System 
Reviewed-by: Niels de Vos 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System