glusterfs.git/xlators/cluster/dht/src, branch v3.8.11

cluster/dht: Modify local->loc.gfid in thread safe manner

2017-04-07T12:08:00+00:00

	Backport of https://review.gluster.org/16986

Problem:
local->loc.gfid in dht_lookup_directory() will be null-gfid for a fresh lookup.
dht_lookup_dir_cbk() updates local->loc.gfid while in other thread dht_lookup_directory()
is still winding lookup calls to subvolumes so there is a chance of partial gfid being
seen by EC.

We saw in 12x(4+2) volume, ec is receiving an loc where the gfid has last 10 bytes matching
with the gfid of the directory and the first 4 bytes are all-zeros. This is leading to EC
erroring out the lookup with EINVAL which leads to NFS failing lookup with EIO.

snip from gdb:
$37 = (dht_local_t *) 0x7fde5de5b3cc
(gdb) p /x $37->loc.gfid
$39 = {0x3b, 0x82, 0x10, 0x5e, 0x40, 0x65, 0x43, 0x14, 0xa0, 0xc6, 0x8, 0xf5,
0x6c, 0x2c, 0xb8, 0x56}
(gdb) fr 7
state=) at ec-generic.c:837
837	                ec_lookup_rebuild(fop->xl->private, fop, cbk);
(gdb) p /x fop->loc[0].gfid
$40 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x43, 0x14, 0xa0, 0xc6, 0x8, 0xf5, 0x6c,
0x2c, 0xb8, 0x56}

snip from log:
[2017-01-29 03:22:30.132328] W [MSGID: 122019]
[ec-helpers.c:354:ec_loc_gfid_check] 0-butcher-disperse-4: Mismatching GFID's
in loc [2017-01-29 03:22:30.132709] W [MSGID: 112199]
[nfs3-helpers.c:3515:nfs3_log_newfh_res] 0-nfs-nfsv3:
/linux-4.9.5/Documentation => (XID: b27b9474, MKDIR: NFS: 5(I/O error), POSIX:
5(Input/output error)), FH: exportid 00000000-0000-0000-0000-000000000000, gfid
00000000-0000-0000-0000-000000000000, mountid
00000000-0000-0000-0000-000000000000 [Invalid argument]

Fix:
update local->loc.gfid in last-call to make sure there are no races.

 >BUG: 1438411
 >Change-Id: Ifcb7e911568c1f1f83123da6ff0cf742b91800a0
 >Signed-off-by: Pranith Kumar K 

BUG: 1438424
Change-Id: If039956205cfac5e798c2c90e92a9a47b404e804
Signed-off-by: Pranith Kumar K 
Reviewed-on: https://review.gluster.org/16988
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G

cluster/dht: Fix crash in "nuke-dir" feature

2017-03-10T23:00:58+00:00

        Backport of: https://review.gluster.org/16829

My patch at https://review.gluster.org/16419 is resulting
in core dumps everytime I run tests/features/nuke.t.

Turns out dht, upon successfully "nuking" a directory,
which was initiated through a setxattr, unwinds the operation
with rmdir fop signature, resulting in readdir-ahead casting
a struct iatt (preparent) to dict_t, leading to a crash.

Change-Id: Ib970b3198185a6c641092b00e115a672cb3f9111
BUG: 1428743
Signed-off-by: Krutika Dhananjay 
Reviewed-on: https://review.gluster.org/16840
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Raghavendra G 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

cluster/dht Fix error assignment in dht_*xattr2 functions

2017-03-10T22:42:42+00:00

Corrected the op_errno assignments and NULL checks in
the dht_sexattr2 and dht_removexattr2 functions. Earlier,
they unwound with the default EINVAL op_errno if the
file had been deleted.

> Change-Id: Iaf837a473d769cea40132487a966c7f452990071
> BUG: 1421653
> Signed-off-by: N Balachandran 
> Reviewed-on: https://review.gluster.org/16610
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: MOHIT AGRAWAL 
> Reviewed-by: Shyamsundar Ranganathan 
(cherry picked from commit 028626a86ea409f908783b9007c02877f20be43e)
Signed-off-by: N Balachandran 

Change-Id: Id2e91df47bcd734dda18700fb075608c1627a608
BUG: 1424915
Reviewed-on: https://review.gluster.org/16678
Tested-by: N Balachandran 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

cluster/dht: Don't update layout in rebalance_task_completion

2017-02-13T08:32:11+00:00

Updating the layout in the dht inode_ctx in
rebalance_task_completion after the file is migrated
is erroneous in case of files with hardlinks.
This step can be skipped as the layout will be set
in the syncop_lookup call post the migration in
dht_migrate_file.

> Change-Id: I24ac798a919585d91a117d6a207e6a31b88486c6
> BUG: 1415761
> Signed-off-by: N Balachandran 
> Reviewed-on: https://review.gluster.org/16457
> Reviewed-by: Raghavendra G 
> Reviewed-by: Susant Palai 
Signed-off-by: N Balachandran 

Change-Id: Ifccffd67b8bc12208efb23101366a1ac7a8c60f5
BUG: 1420184
Reviewed-on: https://review.gluster.org/16561
Smoke: Gluster Build System 
Tested-by: N Balachandran 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Susant Palai 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

cluster/dht: Incorrect migration checks in fsync

2017-01-23T04:15:17+00:00

Fixed the order of the migration phase checks
in dht_fsync_cbk. Phase1 should never be hit if op_ret
is non zero.

> Change-Id: I9222692e04868bffa93498059440f0aa553c83ec
> BUG: 1410777
> Reviewed-on: http://review.gluster.org/16350
> Smoke: Gluster Build System 
> Reviewed-by: Shyamsundar Ranganathan 
> Tested-by: Raghavendra G 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Raghavendra G 
(cherry picked from commit 83117c71482c9f87a0b6812094cbb22497eb3faa)

Change-Id: I0455871aef230371db5ecb1b2e6b1468e8a6d079
BUG: 1412119
Signed-off-by: N Balachandran 
Reviewed-on: https://review.gluster.org/16374
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G

cluster/dht: Do rename cleanup as root

2017-01-18T04:03:30+00:00

Problem:
Rename linkfile cleanup is done as non-root which may not have priviliges to do
the rename so it fails with EACCESS. MKDIR on that name in future will start to
hole on this subvolume. It is not easy to hit on fuse mounts because vfs takes
care of the permission checks even before rename fop is wound. But with
nfs-ganesha mounts it happens.

Fix:
Do rename cleanup as root

 >BUG: 1409727
 >Change-Id: I414c1eb6dce76b4516a6c940557b249e6c3f22f4
 >Signed-off-by: Pranith Kumar K 
 >Reviewed-on: http://review.gluster.org/16317
 >Smoke: Gluster Build System 
 >CentOS-regression: Gluster Build System 
 >Reviewed-by: Raghavendra G 
 >Reviewed-by: N Balachandran 
 >NetBSD-regression: NetBSD Build System 

BUG: 1412913
Change-Id: I7f891034150d7a0e3210202fb0788040c91e1c09
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/16390
Smoke: Gluster Build System 
Reviewed-by: N Balachandran 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System

dht/rebalance: remove errno check for failure detection

2017-01-08T11:03:50+00:00

> BUG: 1410355
> Change-Id: I867419ca36a81ef7209e6911a46c1c2c898b8eab
> Signed-off-by: Susant Palai 
> Reviewed-on: http://review.gluster.org/16328
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Smoke: Gluster Build System 
>Reviewed-by: Raghavendra G 
(cherry picked from commit 451ca272d12f2d49522c845d53585520f71525f8)

Change-Id: If05094346fc1fc48f4982db6a195740171ae2fad
BUG: 1410764
Signed-off-by: Susant Palai 
Reviewed-on: http://review.gluster.org/16348
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

cluster/dht: Fix dict_leak in migration check tasks

2017-01-08T10:53:49+00:00

Fixed a memleak where dict was not being unrefed
in the dht_migration_complete_check_task and
dht_rebalance_inprogress_task functions.

BUG: 1410369
> Change-Id: I3d42e9a2e5c8596c985bf6431a68fd3905227383
> BUG: 1409186
> Signed-off-by: N Balachandran 
> Reviewed-on: http://review.gluster.org/16308
> Smoke: Gluster Build System 
> Reviewed-by: MOHIT AGRAWAL 
> CentOS-regression: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> Reviewed-by: Shyamsundar Ranganathan 
> (cherry picked from commit 11b6a2c9fc5232b58774cab29873406c0fbfef19)

Change-Id: I4e92e4c87b52f6f5ffb859005aa3d525225b4bda
Signed-off-by: N Balachandran 
Reviewed-on: http://review.gluster.org/16331
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

cluster/dht: Fix memory corruption while accessing regex stored in

2017-01-04T03:41:24+00:00

private

If reconfigure is executed parallely (or concurrently with dht_init),
there are races that can corrupt memory. One such race is modification
of regexes stored in conf (conf->rsync_regex_valid and
conf->extra_regex_valid) through dht_init_regex. With change [1],
reconfigure codepath can get executed parallely (with itself or with
dht_init) and this fix is needed.

Also, a reconfigure can race with any thread doing dht_layout_search,
resulting in dht_layout_search accessing regex freed up by reconfigure
(like in bz 1399134).

[1] http://review.gluster.org/15046

>Change-Id: I039422a65374cf0ccbe0073441f0e8c442ebf830
>BUG: 1399134
>Signed-off-by: Raghavendra G 
>Reviewed-on: http://review.gluster.org/15945
>Smoke: Gluster Build System 
>NetBSD-regression: NetBSD Build System 
>Reviewed-by: N Balachandran 
>CentOS-regression: Gluster Build System 
>Reviewed-by: Shyamsundar Ranganathan 

Change-Id: I039422a65374cf0ccbe0073441f0e8c442ebf830
BUG: 1399423
Signed-off-by: Raghavendra G 
(cherry picked from commit 64451d0f25e7cc7aafc1b6589122648281e4310a)
Reviewed-on: http://review.gluster.org/15793
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System

dht/rename : Incase of failure remove linkto file properly

2017-01-02T13:07:25+00:00

Generally linkto file is created using root user. Consider following
case, a user is trying to rename a file which he is not permitted.
So the rename fails with EACESS and when rename tries to cleanup the
linkto file, it fails.

The above issue happens when rename/00.t test executed on nfs-ganesha
clients :
Steps executed in script
* create a file "abc" using root
* rename the file "abc" to "xyz" using a non root user, it fails with EACESS
* delete "abc"
* create directory "abc" using root
* again try ot rename "abc" to "xyz" using non root user, test hungs here
which slowly leds to OOM kill of ganesha process

RCA put forwarded by Du for OOM kill of ganesha
Note that when we hit this bug, we've a scenario of a dentry being
present as:
    * a linkto file on one subvol
    * a directory on rest of subvols

When a lookup happens on the dentry in such a scenario, the control flow
goes into an infinite loop of:

    dht_lookup_everywhere
    dht_lookup_everywhere_cbk
    dht_lookup_unlink_cbk
    dht_lookup_everywhere_done
    dht_lookup_directory (as local->dir_count > 0)
    dht_lookup_dir_cbk (sets to local->need_selfheal = 1 as the entry is a linkto file on one of the subvol)
    dht_lookup_everywhere (as need_selfheal = 1).

This infinite loop can cause increased consumption of memory due to:
1) dht_lookup_directory assigns a new layout to local->layout unconditionally
2)  Most of the functions in this loop do a stack_wind of various fops.

This results in growing of call stack (note that call-stack is destroyed only after lookup response is
received by fuse - which never happens in this case)

Thanks Du for root causing the oom kill and Sushant for suggesting the fix

Upstream reference :
>Change-Id: I1e16bc14aa685542afbd21188426ecb61fd2689d
>BUG: 1397052
>Signed-off-by: Jiffin Tony Thottan 
>Reviewed-on: http://review.gluster.org/15894
>NetBSD-regression: NetBSD Build System 
>CentOS-regression: Gluster Build System 
>Smoke: Gluster Build System 
>Reviewed-by: Raghavendra G 
>(cherry picked from commit 57d59f4be205ae0c7888758366dc0049bdcfe449)

Change-Id: I1e16bc14aa685542afbd21188426ecb61fd2689d
BUG: 1401029
Signed-off-by: Jiffin Tony Thottan 
Reviewed-on: http://review.gluster.org/16015
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos