glusterfs.git/xlators/features/shard/src, branch v3.8.12

features/shard: Fix vm corruption upon fix-layout

2017-04-10T15:45:51+00:00

        Backport of: https://review.gluster.org/17010

shard's writev implementation, as part of identifying
presence of participant shards that aren't in memory,
first sends an MKNOD on these shards, and upon EEXIST error,
looks up the shards before proceeding with the writes.

The VM corruption was caused when the following happened:
1. DHT had n subvolumes initially.
2. Upon add-brick + fix-layout, the layout of .shard changed
   although the existing shards under it were yet to be migrated
   to their new hashed subvolumes.
3. During this time, there were writes on the VM falling in regions
   of the file whose corresponding shards were already existing under
   .shard.
4. Sharding xl sent MKNOD on these shards, now creating them in their
   new hashed subvolumes although there already exist shard blocks for
   this region with valid data.
5. All subsequent writes were wound on these newly created copies.

The net outcome is that both copies of the shard didn't have the correct
data. This caused the affected VMs to be unbootable.

FIX:
For want of better alternatives in DHT, the fix changes shard fops to do
a LOOKUP before the MKNOD and upon EEXIST error, perform another lookup.

Change-Id: I1a5d3515b42e2e5583c407d1b4aff44d7ce472eb
BUG: 1440635
RCA'd-by: Raghavendra Gowdappa 
Reported-by: Mahdi Adnan 
Signed-off-by: Krutika Dhananjay 
Reviewed-on: https://review.gluster.org/17019
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
Reviewed-by: jiffin tony Thottan

features/shard: Initialize local->fop in readv

2017-04-10T11:27:44+00:00

        Backport of: https://review.gluster.org/17014

Change-Id: I4d2f0a3f533009038d48579db5a8a2a048b77ca1
BUG: 1440635
Signed-off-by: Krutika Dhananjay 
Reviewed-on: https://review.gluster.org/17020
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Pranith Kumar Karampuri 
CentOS-regression: Gluster Build System

features/shard: Pass the correct iatt for cache invalidation

2017-04-04T08:54:24+00:00

        Backport of: https://review.gluster.org/16961

This fixes a performance issue with shard which was causing
the translator to trigger unusually high number of lookups
for cache invalidation even when there was no modification to
the file.

In shard_common_stat_cbk(), it is local->prebuf that contains the
aggregated size and block count as opposed to buf which only holds the
attributes for the physical copy of base shard. Passing buf for
inode_ctx invalidation would always set refresh to true since the file
size in inode ctx contains the aggregated size and would never be same
as @buf->ia_size. This was leading to every write/read being preceded
by a lookup on the base shard even when the file underwent no
modification.

Change-Id: I85940b4b33e77b98e97e277d880ab35b1496c89a
BUG: 1437330
Signed-off-by: Krutika Dhananjay 
Reviewed-on: https://review.gluster.org/16968
Smoke: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: jiffin tony Thottan

features/shard: Fix EIO error on add-brick

2017-03-10T22:47:05+00:00

        Backport of: https://review.gluster.org/14419

DHT seems to link inode during lookup even before initializing
inode ctx with layout information, which comes after
directory healing.

Consider two parallel writes. As part of the first write,
shard sends lookup on .shard which in its return path would
cause DHT to link .shard inode. Now at this point, when a
second write is wound, inode_find() of .shard succeeds and
as a result of this, shard goes to create the participant
shards by issuing MKNODs under .shard. Since the layout is
yet to be initialized, mknod fails in dht call path with EIO,
leading to VM pauses.

The fix involves shard maintaining a flag to denote whether
a fresh lookup on .shard completed one network trip. If it
didn't, all inode_find()s in fop path will be followed by a
lookup before proceeding with the next stage of the fop.

Big thanks to Raghavendra G and Pranith Kumar K for the RCA
and subsequent inputs and feedback on the patch.

Change-Id: I66a7adf177e338a7691f441f199dde7c2b90c292
BUG: 1387878
Signed-off-by: Krutika Dhananjay 
Reviewed-on: https://review.gluster.org/16750
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Pranith Kumar Karampuri 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

features/shard: Put onus of choosing the inode to resolve on individual fops

2017-03-10T22:44:47+00:00

        Backport of: https://review.gluster.org/16709

... as opposed to adding checks in "common" functions to choose the inode
to resolve based local->fop, which is rather ugly and prone to errors.

Change-Id: Ib26d3dd5a7ae43cd27839752bdae2cce56d73e8a
BUG: 1387878
Signed-off-by: Krutika Dhananjay 
Reviewed-on: https://review.gluster.org/16749
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Pranith Kumar Karampuri 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

features/shard: Fill loc.pargfid too for named lookups on individual shards

2016-11-09T07:28:19+00:00

        Backport of: http://review.gluster.org/#/c/15788/

On a sharded volume when a brick is replaced while IO is going on, named
lookup on individual shards as part of read/write was failing with
ENOENT on the replaced brick, and as a result AFR initiated name heal in
lookup callback. But since pargfid was empty (which is what this patch
attempts to fix), the resolution of the shards by protocol/server used
to fail and the following pattern of logs was seen:

Brick-logs:

[2016-11-08 07:41:49.387127] W [MSGID: 115009]
[server-resolve.c:566:server_resolve] 0-rep-server: no resolution type
for (null) (LOOKUP)
[2016-11-08 07:41:49.387157] E [MSGID: 115050]
[server-rpc-fops.c:156:server_lookup_cbk] 0-rep-server: 91833: LOOKUP(null)
(00000000-0000-0000-0000-000000000000/16d47463-ece5-4b33-9c93-470be918c0f6.82)
==> (Invalid argument) [Invalid argument]

Client-logs:
[2016-11-08 07:41:27.497687] W [MSGID: 114031]
[client-rpc-fops.c:2930:client3_3_lookup_cbk] 2-rep-client-0: remote
operation failed. Path: (null) (00000000-0000-0000-0000-000000000000)
[Invalid argument]
[2016-11-08 07:41:27.497755] W [MSGID: 114031]
[client-rpc-fops.c:2930:client3_3_lookup_cbk] 2-rep-client-1: remote
operation failed. Path: (null) (00000000-0000-0000-0000-000000000000)
[Invalid argument]
[2016-11-08 07:41:27.498500] W [MSGID: 114031]
[client-rpc-fops.c:2930:client3_3_lookup_cbk] 2-rep-client-2: remote
operation failed. Path: (null) (00000000-0000-0000-0000-000000000000)
[Invalid argument]
[2016-11-08 07:41:27.499680] E [MSGID: 133010]

Also, this patch makes AFR by itself choose a non-NULL pargfid even if
its ancestors fail to initialize all pargfid placeholders.

Change-Id: Ica9e1b5b196ac37aafe6128e7aa0694a07245fdb
BUG: 1392846
Signed-off-by: Krutika Dhananjay 
Reviewed-on: http://review.gluster.org/15796
Reviewed-by: Pranith Kumar Karampuri 
Reviewed-by: Ravishankar N 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

build: correctly format some (s)size_t messages

2016-09-07T14:00:48+00:00

On 32-bit builds the are are warnings like these:

    posix.c:6438: warning: format '%ld' expects type 'long int', but argument 11 has type 'ssize_t'

Instead of using "%l" for (signed) size_t variables, "%z" should be
used.

Cherry picked from commit 3af889f02722f4636d2ea30570de6477e8b5a3a9:
> BUG: 1198849
> Change-Id: I6f57b5e8ea174dd9e3056aff5da685e497894ccf
> Signed-off-by: Niels de Vos 
> Reviewed-on: http://review.gluster.org/14933
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Kaleb KEITHLEY 
> Smoke: Gluster Build System 

This patch is not really recommended for backporting, but we do have a
new smoke test that fails when these warnings pop-up. It is cleaner to
correct the code in the release-3.7 branch then to modify the smoke test
to skip this branch.

Change-Id: I6f57b5e8ea174dd9e3056aff5da685e497894ccf
BUG: 1373530
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/15410
Reviewed-by: Kaleb KEITHLEY 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

features/shard: Don't modify readv size

2016-06-27T15:18:40+00:00

        Backport of: http://review.gluster.org/14623

For o-direct reads application sends aligned size which needs to be
sent as is, otherwise o-direct writes where the file-size is not
aligned fails.

 >Change-Id: I66afcba41f3484da11e9a12fe2671d2051fafc8a
 >BUG: 1342903
 >Signed-off-by: Pranith Kumar K 
 >Reviewed-on: http://review.gluster.org/14658
 >Tested-by: Krutika Dhananjay 
 >Smoke: Gluster Build System 
 >CentOS-regression: Gluster Build System 
 >Reviewed-by: Krutika Dhananjay 
 >NetBSD-regression: NetBSD Build System 

BUG: 1347553
Change-Id: I1adcdaccbd5686986a057314efab9a646afa3d47
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/14752
Smoke: Gluster Build System 
Reviewed-by: Krutika Dhananjay 
Tested-by: Krutika Dhananjay 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

core, shard: Make shards inherit main file's O_DIRECT flag if present

2016-06-27T15:17:55+00:00

        Backport of: http://review.gluster.org/14191

If the application opens a file with O_DIRECT, the shards'
anon fds would also need to inherit the flag. Towards this,
shard xl would be passing the odirect flag in the @flags parameter
to the WRITEV fop. This will be used in anon fd resolution
and subsequent opening by posix xl.

 >Change-Id: I3a0593fa46cc25e390a5762a0354b469c2a1532d
 >BUG: 1342903
 >Signed-off-by: Krutika Dhananjay 
 >Reviewed-on: http://review.gluster.org/14663
 >Smoke: Gluster Build System 
 >CentOS-regression: Gluster Build System 
 >NetBSD-regression: NetBSD Build System 
 >Reviewed-by: Jeff Darcy 

Change-Id: Ibfc164aa7f9eecd6993255f1c03557f2ec35ac8c
BUG: 1347553
Signed-off-by: Krutika Dhananjay 
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/14754
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

posix, shard: Use page-aligned buffer for o-direct reads

2016-06-27T15:17:12+00:00

        Backport of: http://review.gluster.org/#/c/14639/

and also make shard_readv_do() pass the correct flags when
the original fd is opened with O_DIRECT.

 >Change-Id: I253f74459656b047de652b6aace487c0e2ac71f6
 >BUG: 1342903
 >Signed-off-by: Krutika Dhananjay 
 >Reviewed-on: http://review.gluster.org/14650
 >Smoke: Gluster Build System 
 >CentOS-regression: Gluster Build System 
 >NetBSD-regression: NetBSD Build System 
 >Reviewed-by: Pranith Kumar Karampuri 

Change-Id: Ic774d2c21efc05566a1b842f7dce24509cbeaac4
BUG: 1347553
Signed-off-by: Krutika Dhananjay 
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/14753
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System