glusterfs.git/xlators/cluster/dht/src/dht-helper.c, branch v3.7.15

cluster/dht: Fix unsafe iteration on inode->fd_list

2016-06-24T10:20:05+00:00

When DHT traverses the inode->fd_list, it does that in an unsafe
way that can generate races with fd_unref() called from other threads.

This patch fixes this problem taking the inode->lock and adding a
reference to the fd while it's being used outside of the mutex
protected region.

A minor change in storage/posix has been done to also access the
inode->fd_list in a safe way.

Backport of:
> Change-Id: I10d469ca6a8f76e950a8c9779ae9c8b70f88ef93
> BUG: 1344340
> Signed-off-by: Xavier Hernandez 
> Reviewed-on: http://review.gluster.org/14682
> CentOS-regression: Gluster Build System 
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> Reviewed-by: Raghavendra G 

Change-Id: I10d469ca6a8f76e950a8c9779ae9c8b70f88ef93
BUG: 1346751
Signed-off-by: Xavier Hernandez 
Reviewed-on: http://review.gluster.org/14734
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G

dht:remember locked subvol and send unlock to the same

2016-05-06T12:26:26+00:00

During locking we send lock request to cached subvol,
and normally we unlock to the cached subvol
But with parallel fresh lookup on a directory, there
is a race window where the cached subvol can change
and the unlock can go into a different subvol from
which we took lock.

This will result in a stale lock held on one of the
subvol.

So we will store the details of subvol which we took the lock
and will unlock from the same subvol

back port of>
>Change-Id: I47df99491671b10624eb37d1d17e40bacf0b15eb
>BUG: 1311002
>Signed-off-by: Mohammed Rafi KC 
>Reviewed-on: http://review.gluster.org/13492
>Reviewed-by: N Balachandran 
>Smoke: Gluster Build System 
>NetBSD-regression: NetBSD Build System 
>Reviewed-by: Raghavendra G 
>CentOS-regression: Gluster Build System 

Change-Id: Ia847e7115d2296ae9811b14a956f3b6bf39bd86d
BUG: 1333645
Signed-off-by: Mohammed Rafi KC 
Reviewed-on: http://review.gluster.org/14236
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G

cluster/distribute: detect stale layouts in entry fops

2016-04-25T09:30:45+00:00

dht_mkdir ()
{
      first-hashed-subvol = hashed-subvol for "bname" in in-memory
                            layout of "parent";
      inodelk (SETLKW, parent, "LAYOUT_HEAL_DOMAIN", "can be any
               subvol, but we choose first-hashed-subvol randomly");
      {
begin:
            hashed-subvol = hashed-subvol for "bname" in in-memory
                            layout of "parent";
            hash-range = extract hashe-range from layout of "parent";

            ret = mkdir (parent/bname, hashed-subvol, hash-range);
            if (ret == "hash-value doesn't fall into layout stored on
                       the brick (this error is returned by posix-mkdir)")
            {
                refresh_parent_layout ();
                goto begin;
            }

      }
      inodelk (UNLCK, parent, "LAYOUT_HEAL_DOMAIN",
               "first-hashed-subvol");

      proceed with other parts of dht_mkdir;
}

posix_mkdir (parent/bname, client-hash-range)
{

       disk-hash-range = getxattr (parent, "dht-layout-key");
       if (disk-hash-range != client-hash-range) {
              fail-with-error ("hash-value doesn't fall into layout
                                stored on the brick");
              return 0;
       }

       continue-with-posix-mkdir;
}

Similar changes need to be done for dentry operations like create,
symlink, link, unlink, rmdir, rename. These will be addressed in
subsequent patches. This patch addresses only mkdir codepath.

This change breaks stripe tests, as on some striped subvols dht layout
xattrs are not set for some reason. This results in failure of
mkdir. Since striped volumes are always created with dht, some tests
associated with stripe also fail. So, I am making following tests
changes (since stripe is out of maintainance):
* modify ./tests/basic/rpc-coverage.t to not to use striped volumes
* mark all (2) tests in tests/bugs/stripe/ as bad tests

Change-Id: Idd1ae879f24a48303dc743c1bb4d91f89a629e25
BUG: 1329062
Signed-off-by: Raghavendra G 
Reviewed-on: http://review.gluster.org/14040
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

dht: lock on subvols to prevent lookup vs rmdir race

2016-04-06T10:08:06+00:00

There is a possibility that while an rmdir is completed on
some non-hashed subvol and proceeding to others, a lookup
selfheal can recreate the same directory on those subvols
for which the rmdir had succeeded. Now the deletion of the
parent directory will fail with an ENOTEMPTY.

To fix this take blocking inodelk on the subvols before
starting rmdir. Selfheal must also take blocking inodelk
before creating the entry.

Backport of http://review.gluster.org/13528

> Change-Id: I168a195c35ac1230ba7124d3b0ca157755b3df96
> BUG: 1245065
> Signed-off-by: Sakshi 
> Reviewed-on: http://review.gluster.org/13528
> CentOS-regression: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> Smoke: Gluster Build System 
> Reviewed-by: Raghavendra G 
> Tested-by: Raghavendra G 

Change-Id: I168a195c35ac1230ba7124d3b0ca157755b3df96
BUG: 1257894
Signed-off-by: Sakshi 
Reviewed-on: http://review.gluster.org/13915
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Raghavendra G 
CentOS-regression: Gluster Build System

dht: report constant directory size

2016-03-28T10:56:48+00:00

Directory size is meaningless.  Every filesystem has its own
unpredictable way of increasing or decreasing it, based on internal data
structures and even transient conditions.  Some filesystems (e.g. ext4)
never decrease it at all.  Others (e.g. btrfs) don't even report it.
Very few programs look at it, and those that do are broken.
Unfortunately, one such program is GNU tar, which will complain when it
sees different values because at different times we got the value from
different DHT subvolumes.  To avoid such problems, just report a
constant value.

Backport of http://review.gluster.org/#/c/13770/

> Change-Id: Id64ce917c75b5f7ff50cb55b6e997f3b3556e7e3
> BUG: 1302948
> Original-author: Shyam 
> Signed-off-by: Jeff Darcy 
> Signed-off-by: N Balachandran 
> Reviewed-on: http://review.gluster.org/13770
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Shyamsundar Ranganathan 
> Reviewed-by: Vijay Bellur 

Change-Id: Id64ce917c75b5f7ff50cb55b6e997f3b3556e7e3
BUG: 1312721
Original-author: Shyam 
Signed-off-by: Jeff Darcy 
Signed-off-by: N Balachandran 
Reviewed-on: http://review.gluster.org/13831
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G

cluster/dht : Ftruncate on migrating file fails with EINVAL

2016-01-29T08:42:30+00:00

What:
If dht_open is called on a migrating file after the inode_ctx is set,
subsequent FOPs on that fd do not open the fd on the dst subvol.
This is seen when the open-ftruncate-close sequence is repeatedly
called on a migrating file.
A second call to the sequence described above causes dht_truncate_cbk
to call dht_truncate2 as the dht_inode_ctx was already set by the first
call. As dht_rebalance_in_progress_check is not called, the fd is not
opened on the dst subvol.
On a distributed-replicate volume, this causes AFR to
open the fd using afr_fix_open, but with the wrong flags, causing
posix_ftruncate to fail with EINVAL.
The fix: We require fd specific information to make a decision while
handling migrating files.
Set the fd_ctx to indicate the fd has been opened on the dst subvol
and check if it has been set while processing Phase1/Phase2 checks
in the FOP callback functions.

> Change-Id: I43cdcd8017b4a11e18afdd210469de7cd9a5ef14
> Signed-off-by: N Balachandran 
> Reviewed-on: http://review.gluster.org/12985
> Reviewed-by: Raghavendra G 
> Tested-by: Gluster Build System 
> Reviewed-by: Dan Lambright 
> Tested-by: Dan Lambright 

Change-Id: I99f8aceec105f16631def06a263f0561954d14b3
BUG: 1293827
Signed-off-by: N Balachandran 
Reviewed-on: http://review.gluster.org/13071
Reviewed-by: Pranith Kumar Karampuri 
Tested-by: Pranith Kumar Karampuri 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

all: reduce "inline" usage

2016-01-18T09:02:34+00:00

There are three kinds of inline functions: plain inline, extern inline,
and static inline.  All three have been removed from .c files, except
those in "contrib" which aren't our problem.  Inlines in .h files, which
are overwhelmingly "static inline" already, have generally been left
alone.  Over time we should be able to "lower" these into .c files, but
that has to be done in a case-by-case fashion requiring more manual
effort.  This part was easy to do automatically without (as far as I can
tell) any ill effect.

In the process, several pieces of dead code were flagged by the
compiler, and were removed.

backport of Change-Id: I56a5e614735c9e0a6ee420dab949eac22e25c155,
http://review.gluster.org/11769, BUG: 1245331

Change-Id: Iba1efb0bc578ea4a5e9bf76b7bd93dc1be9eba44
BUG: 1283302
Signed-off-by: Kaleb S KEITHLEY 
Reviewed-on: http://review.gluster.org/12646
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

cluster/tier: fix loading tier.so into glusterd

2015-12-04T13:44:32+00:00

The glusterd process loads the shared libraries of client translators.
This failed for tiering due to a reference to dht_methods which is
defined as a global variable which is not necessary.
The global variable has been removed and this is now a member of
dht_conf and is now initialised in the *_init calls.

> Change-Id: Ifa0a21e3962b5cd8d9b927ef1d087d3b25312953
> Signed-off-by: N Balachandran 
> Reviewed-on: http://review.gluster.org/12863
> Tested-by: NetBSD Build System 
> Tested-by: Gluster Build System 
> Reviewed-by: Dan Lambright 
>Tested-by: Dan Lambright 
(cherry picked from commit 96fc7f64da2ef09e82845a7ab97574f511a9aae5)

Change-Id: If3cc908ebfcd1f165504f15db2e3079d97f3132e
BUG: 1288352
Signed-off-by: N Balachandran 
Reviewed-on: http://review.gluster.org/12877
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

dht: heal directory path if the directory is not present

2015-11-09T10:52:34+00:00

        back port of http://review.gluster.org/#/c/12376/

After a successful nameless lookup if the directory is not
present on any of the subvol, then we will get the path of
the directory and will recursively send a named lookp on
each parent directory.

This will help particularly for the scenarios like add brick
and attach-tier.

Change-Id: I97f3178adb08b88910f709f0acf1d86c147be017
BUG: 1279095
Signed-off-by: Mohammed Rafi KC 
Reviewed-on: http://review.gluster.org/12539
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: N Balachandran 
Reviewed-by: Raghavendra G

dht/rebalance: fix mem-leak in rebalance

2015-10-06T11:10:19+00:00

Change-Id: I37faf983fc02996541f3d96a17cb2a2c2cdb6781
BUG: 1261234
Reviewed-on: http://review.gluster.org/12235
Reviewed-by: Raghavendra G 
Tested-by: Raghavendra G 
Signed-off-by: Susant Palai 
Reviewed-on: http://review.gluster.org/12296