glusterfs.git/xlators/features/shard/src, branch v4.1dev

core: fix some of the dict_{get,set} with proper APIs

2018-01-17T00:00:11+00:00

updates #220

Change-Id: I6e25dbb69b2c7021e00073e8f025d212db7de0be
Signed-off-by: Amar Tumballi

all: Simplify component message id's definition

2017-12-14T02:33:51+00:00

This patch creates a new way of defining message id's that is easier
and less error prone because it doesn't require so many manual changes
each time a new component is defined or a new message created.

Change-Id: I71ba8af9ac068f5add7e74f316a2478bc991c67b
Signed-off-by: Xavier Hernandez

features/shard: Coverity fixes

2017-11-08T18:04:27+00:00

This patch fixes coverity issues 242 and 453.

Change-Id: If18f40539dccc7c2fcdcf8ef9b6fa3efbb3e462f
BUG: 789278
Signed-off-by: Krutika Dhananjay

features/shard: Change default shard-block-size to 64MB

2017-09-14T03:23:15+00:00

Change-Id: I55fa87e07136cff10b0d725ee24dd3151016e64e
BUG: 1489823
Signed-off-by: Krutika Dhananjay 
Reviewed-on: https://review.gluster.org/18243
Reviewed-by: Pranith Kumar Karampuri 
Smoke: Gluster Build System 
Tested-by: Sunil Kumar Acharya 
CentOS-regression: Gluster Build System

features/shard: Increment counts in locks

2017-09-06T02:31:29+00:00

Problem:
Because create_count/eexist_count are incremented without locks, all the shards may not
be created because call_count will be lesser than what it needs to be. This can lead
to crash in shard_common_inode_write_do() because inode on which we want to do
fd_anonymous() is NULL

Fix:
Increment the counts in frame->lock

Change-Id: Ibc87dcb1021e9f4ac2929f662da07aa7662ab0d6
BUG: 1488354
Signed-off-by: Pranith Kumar K 
Reviewed-on: https://review.gluster.org/18203
Smoke: Gluster Build System 
Reviewed-by: Krutika Dhananjay 
CentOS-regression: Gluster Build System

features/shard: Return aggregated size in stbuf of LINK fop

2017-09-06T02:31:09+00:00

Change-Id: I42df7679d63fec9b4c03b8dbc66c5625f097fac0
BUG: 1488546
Signed-off-by: Krutika Dhananjay 
Reviewed-on: https://review.gluster.org/18209
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri

features/shard: Remove ctx from LRU in shard_forget

2017-06-30T05:05:15+00:00

Problem:
There is a race when the following two commands are executed on the mount in
parallel from two different terminals on a sharded volume,
which leads to use-after-free.

Terminal-1:
while true; do dd if=/dev/zero of=file1 bs=1M count=4; done

Terminal-2:
while true; do cat file1 > /dev/null; done

In the normal case this is the life-cycle of a shard-inode
1) Shard is added to LRU when it is first looked-up
2) For every operation on the shard it is moved up in LRU
3) When "unlink of the shard"/"LRU limit is hit" happens it is removed from LRU

But we are seeing a race where the inode stays in Shard LRU even after it is
forgotten which leads to Use-after-free and then some memory-corruptions.

These are the steps:
1) Shard is added to LRU when it is first looked-up
2) For every operation on the shard it is moved up in LRU

Reader-handler                                    Truncate-handler
1) Reader handler needs shard-x to be read.     1) Truncate has just deleted shard-x
2) In shard_common_resolve_shards(), it does
   inode_resolve() and that leads to
   a hit in LRU, so it is going to call
__shard_update_shards_inode_list() to move the
   inode to top of LRU
						2) shard-x gets unlinked from the itable
						   and inode_forget(inode, 0) is called
						   to make sure the inode can be purged
						   upon last unref
3) when __shard_update_shards_inode_list() is
   called it finds that the inode is not in LRU
   so it adds it back to the LRU-list

Both these operations complete and call inode_unref(shard-x) which leads to the inode
getting freed and forgotten, even when it is in Shard LRU list. When more inodes are
added to LRU, use-after-free will happen and it leads to undefined behaviors.

Fix:
I see that the inode can be removed from LRU even by the protocol layers like gfapi/gNFS
when LRU limit is reached. So it is better to add a check in shard_forget() to remove itself
from LRU list if it exists.

BUG: 1466037
Change-Id: Ia79c0c5c9d5febc56c41ddb12b5daf03e5281638
Signed-off-by: Pranith Kumar K 
Reviewed-on: https://review.gluster.org/17644
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Krutika Dhananjay

features/shard: Handle offset in appending writes

2017-05-27T16:00:50+00:00

When a file is opened with append, all writes are appended at the end of file
irrespective of the offset given in the write syscall. This needs to be
considered in shard size update function and also for choosing which shard to
write to.

At the moment shard piggybacks on queuing from write-behind
xlator for ordering of the operations. So if write-behind is disabled and
two parallel appending-writes come both of which can increase the file size
beyond shard-size the file will be corrupted.

BUG: 1455301
Change-Id: I9007e6a39098ab0b5d5386367bd07eb5f89cb09e
Signed-off-by: Pranith Kumar K 
Reviewed-on: https://review.gluster.org/17387
Smoke: Gluster Build System 
Reviewed-by: Krutika Dhananjay 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

features/shard: Set size in inode ctx before size update for truncate too

2017-05-10T12:33:53+00:00

Change-Id: I7e984bb0f50c7d42764c0648e697d94d6c768dc7
BUG: 1448299
Signed-off-by: Krutika Dhananjay 
Reviewed-on: https://review.gluster.org/17184
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Pranith Kumar Karampuri 
Smoke: Gluster Build System

features/shard: Initialize local->fop in readv

2017-04-11T13:32:19+00:00

Change-Id: I9008ca9960df4821636501ae84f93a68f370c67f
BUG: 1440051
Signed-off-by: Krutika Dhananjay 
Reviewed-on: https://review.gluster.org/17014
NetBSD-regression: NetBSD Build System 
Reviewed-by: Pranith Kumar Karampuri 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Vijay Bellur