glusterfs.git/libglusterfs/src/common-utils.h, branch v3.7.12rc1

cluster/afr: Don't let NFS cache stat after writes.

2016-06-03T13:23:47+00:00

Problem:
Afr does post-ops after write but the stat buffer it unwinds is at the
time of write, so if nfs client caches this, it will see different
ctime when it does stat on it after post-op is done. From NFS client's
perspective it thinks the file is changed. Tar which depends on this
to be correct keeps giving 'file changed as we read it' warning.
If Afr instead has to choose to unwind after post-op, eager-lock,
delayed-post-op will have to be disabled which will lead to bad
performance for all write usecases.

Fix:
Don't let client cache stat after write.

 >Change-Id: Ic6062acc6e5cdd97a9c83c56bd529ec83cee8a23
 >BUG: 1302948
 >Signed-off-by: Pranith Kumar K 
 >Signed-off-by: Anuradha Talur 
 >Reviewed-on: http://review.gluster.org/13785
 >Smoke: Gluster Build System 
 >NetBSD-regression: NetBSD Build System 
 >CentOS-regression: Gluster Build System 
 >Reviewed-by: Niels de Vos 

BUG: 1312721
Change-Id: I42a5d524bcf2a2034fe48ee8454812ca26a98c37
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/14454
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System

rpc: define client port range

2016-05-05T03:24:42+00:00

Problem:
when bind-insecure is 'off', all the clients bind to secure ports,
if incase all the secure ports exhaust the client will no more bind
to secure ports and tries gets a random port which is obviously insecure.

we have seen the client obtaining a port number in the range 49152-65535
which are actually reserved as part of glusterd's pmap_registry for bricks,
hence this will lead to port clashes between client and brick processes.

Solution:
If we can define different port ranges for clients incase where secure ports
exhaust, we can avoid the maximum port clashes with in gluster processes.

Still we are prone to have clashes with other non-gluster processes, but
the chances being very low, but that's a different story on its own, which
will be handled in upcoming patches.

> Change-Id: Ib5ce05991aa1290ccb17f6f04ffd65caf411feaf
> BUG: 1322805
> Signed-off-by: Prasanna Kumar Kalever 
> Reviewed-on: http://review.gluster.org/13998
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Atin Mukherjee 
> Reviewed-by: Raghavendra G 
> Signed-off-by: Prasanna Kumar Kalever 

Change-Id: I712676d3e79145d78a17f2c361525e6ef82a4732
BUG: 1323564
Signed-off-by: Prasanna Kumar Kalever 
Reviewed-on: http://review.gluster.org/14205
Tested-by: Prasanna Kumar Kalever 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Raghavendra G

features/trash: wind mkdir with special pid

2016-05-03T11:22:48+00:00

Recent changes done w.r.t handling of mkdir calls in posix translator
resulted in crashing the brick process from trash translator. This was
due to the changes made in posix translator to return EPERM for every
mkdir calls without 'gfid-req' set in dictionary. In order to avoid
gfid mismatches during directory creation from brick side trash
translator does not set 'gfid-req'. This patch is to have an exemption
for trash based on a special pid set for those mkdir calls originating
from trash translator and to reset it in callback.

This patch also includes a small optimization to the existing test case for trash feature.

> Reviewed-on: http://review.gluster.org/13776
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Jeff Darcy 

(cherry picked from commit b5cfe948cb3569f034da80ac97b5d2f028b3b0e5)

Change-Id: I59f084ac875e54342ecf2bffa6e43ebd84814153
BUG: 1332372
Signed-off-by: Anoop C S 
Reviewed-on: http://review.gluster.org/14173
Reviewed-by: Pranith Kumar Karampuri 
Tested-by: Pranith Kumar Karampuri 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System

Rename enum _gf_client_pid to _gf_special_pid

2016-05-03T05:47:44+00:00

Till now _gf_client_pid enum has been used to define special PIDs used
by glusterfs clients like shd, quotad etc. In order to have this enum
capable of holding all other special PIDs including the one used by
trash translator, _gf_client_pid is being renamed to _gf_special_pid.

> Change-Id: Id123127771f18aa55d39f335801a54810848d7bc
> BUG: 1330616
> Reviewed-on: http://review.gluster.org/14083
> Reviewed-by: Joseph Fernandes
> Reviewed-by: Jeff Darcy 
> Reviewed-by: jiffin tony Thottan 
> Smoke: Gluster Build System 
> CentOS-regression: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
(cherry picked from commit 2a6c6de35130328cfb6f95a6d7794dcb01b4004d)

Change-Id: Id123127771f18aa55d39f335801a54810848d7bc
Signed-off-by: Anoop C S 
Reviewed-on: http://review.gluster.org/14097
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri

cli/quota: Sort the list output alphabetically by path

2016-04-27T16:46:50+00:00

This is a backport of http://review.gluster.org/14000

> Change-Id: I0b124e119d167817be2ae3eb52ac6c80fc7db5d1
> BUG: 1320716
> Signed-off-by: vmallika 
> Reviewed-on: http://review.gluster.org/14000
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Kaushal M 

Change-Id: I87e12d58c8e267b2af67e287998e7313efc70af4
BUG: 1330018
Signed-off-by: vmallika 
Reviewed-on: http://review.gluster.org/14061
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System

Revert "glusterd: Bug fixes for IPv6 support"

2016-04-16T02:18:23+00:00

This reverts commit b33f3c95ec9c8112e6677e09cea05c4c462040d0.

This commit exposes some issues with management encryption that prevents
GlusterFS from operating properly. This will be added again once
problems with management encryption are fixed.

glusterd: Bug fixes for IPv6 support

2016-03-21T17:52:50+00:00

Backport of http://review.gluster.org/#/c/11988/

Problem:
Glusterd not working using ipv6 transport. The idea is with proper glusterd.vol configuration,
1. glusterd needs to listen on default port (240007) as IPv6 TCP listner.
2. Volume creation/deletion/mounting/add-bricks/delete-bricks/peer-probe
needs to work using ipv6 addresses.
3. Bricks needs to listen on ipv6 addresses.
All the above functionality is needed to say that glusterd supports ipv6 transport and this is broken.

Fix:
When "option transport.address-family inet6" option is present in glusterd.vol
file, it is made sure that glusterd creates listeners using ipv6 sockets only and also the same information is saved
inside brick volume files used by glusterfsd brick process when they are starting.

Tests Run:
Regression tests using ./run-tests.sh
IPv4: Regression tests using ./run-tests.sh for release-3.7 branch verified by comparing with clean repo.
IPv6: (Need to add the above mentioned config and also add an entry for "hostname ::1" in /etc/hosts)
    Started failing at ./tests/basic/glusterd/arbiter-volume-probe.t and ran successfully till here

Change-Id: Idd7513aa2347ce0de2b1f68daeecce1b7a39a7af
BUG: 1310445
Signed-off-by: Nithin D 
Reviewed-on: http://review.gluster.org/13787
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Atin Mukherjee

cluster/ec: Mark self-heal fops as internal

2016-03-05T01:26:21+00:00

 >Change-Id: I8ae7af266d3e00460f0cfdc9389a926e5f2fee36
 >BUG: 1282761
 >Signed-off-by: Pranith Kumar K 
 >Reviewed-on: http://review.gluster.org/12598
 >Tested-by: Gluster Build System 
 >Tested-by: NetBSD Build System 
 >Reviewed-by: Xavier Hernandez 

BUG: 1283757
Change-Id: Ic20d4ee031265305db1a6ed2cf591ce94b7d0749
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/12668
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

performance/write-behind: retry "failed syncs to backend"

2016-02-16T09:09:34+00:00

1. When sync fails, the cached-write is still preserved unless there
   is a flush/fsync waiting on it.

2. When a sync fails and there is a flush/fsync waiting on the
   cached-write, the cache is thrown away and no further retries will
   be made. In other words flush/fsync act as barriers for all the
   previous writes. The behaviour of fsync acting as a barrier is
   controlled by an option (see below for details). All previous
   writes are either successfully synced to backend or forgotten in
   case of an error. Without such barrier fop (especially flush which
   is issued prior to a close), we end up retrying for ever even after
   fd is closed.

3. If a fop is waiting on cached-write and syncing to backend fails,
   the waiting fop is failed.

4. sync failures when no fop is waiting are ignored and are not
   propagated to application. For eg.,
   a. first attempt of sync of a cached-write w1 fails
   b. second attempt of sync of w1 succeeds

   If there are no fops dependent on w1 are issued b/w a and b,
   application won't know about failure encountered in a.

5. The effect of repeated sync failures is that, there will be no
   cache for future writes and they cannot be written behind.

fsync as a barrier and resync of cached writes post fsync failure:
==================================================================
Whether to keep retrying failed syncs post fsync is controlled by an
option "resync-failed-syncs-after-fsync". By default, this option is
set to "off".

If sync of "cached-writes issued before fsync" (to backend) fails,
this option configures whether to retry syncing them after fsync or
forget them. If set to on, cached-writes are retried till a "flush"
fop (or a successful sync) on sync failures. fsync itself is failed
irrespective of the value of this option, when there is a sync failure
of any cached-writes issued before fsync.

Change-Id: I6097c0257bfb9ee5b1f616fbe6a0576ae9af369a
Signed-off-by: Raghavendra G 
BUG: 1293534
Signed-off-by: Raghavendra Talur 
Reviewed-on: http://review.gluster.org/13057
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

storage/posix: Implement .unlink directory

2016-01-25T11:37:10+00:00

Problem: For EC volume, If a file descriptor is open and
file has been unlinked, any further write on that fd will
fail. When a write request comes, EC internally reads some
blocks using anonymous fd. This read will fail as the file
has already been unlinked.

Solution: To solve this issue, we are using .unlink directory
to keep track of unlinked file. If a file is to be unlinked
while its fd is open, move this to .unlink directory and unlink
it from .glusterfs and real path. Once all the fd will be closed,
remove this entry form .unlink directory.

master -
http://review.gluster.org/#/c/12816/

Change-Id: I8344edb0d340bdb883dc46458c16edbc336916b9
BUG: 1291557
Signed-off-by: Ashish Pandey 
Reviewed-on: http://review.gluster.org/12968
Reviewed-by: Pranith Kumar Karampuri 
Tested-by: Pranith Kumar Karampuri 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System