glusterfs.git/xlators/features/shard/src, branch v6dev

Land part 2 of clang-format changes

2018-09-12T12:22:45+00:00

Change-Id: Ia84cc24c8924e6d22d02ac15f611c10e26db99b4
Signed-off-by: Nigel Babu

Land clang-format changes

2018-09-12T11:52:48+00:00

Change-Id: I6f5d8140a06f3c1b2d196849299f8d483028d33b

Multiple files: calloc -> malloc

2018-09-04T04:58:14+00:00

xlators/storage/posix/src/posix-inode-fd-ops.c:
xlators/storage/posix/src/posix-helpers.c:
xlators/storage/bd/src/bd.c:
xlators/protocol/client/src/client-lk.c:
xlators/performance/quick-read/src/quick-read.c:
xlators/performance/io-cache/src/page.c
xlators/nfs/server/src/nfs3-helpers.c
xlators/nfs/server/src/nfs-fops.c
xlators/nfs/server/src/mount3udp_svc.c
xlators/nfs/server/src/mount3.c
xlators/mount/fuse/src/fuse-helpers.c
xlators/mount/fuse/src/fuse-bridge.c
xlators/mgmt/glusterd/src/glusterd-utils.c
xlators/mgmt/glusterd/src/glusterd-syncop.h
xlators/mgmt/glusterd/src/glusterd-snapshot.c
xlators/mgmt/glusterd/src/glusterd-rpc-ops.c
xlators/mgmt/glusterd/src/glusterd-replace-brick.c
xlators/mgmt/glusterd/src/glusterd-op-sm.c
xlators/mgmt/glusterd/src/glusterd-mgmt.c
xlators/meta/src/subvolumes-dir.c
xlators/meta/src/graph-dir.c
xlators/features/trash/src/trash.c
xlators/features/shard/src/shard.h
xlators/features/shard/src/shard.c
xlators/features/marker/src/marker-quota.c
xlators/features/locks/src/common.c
xlators/features/leases/src/leases-internal.c
xlators/features/gfid-access/src/gfid-access.c
xlators/features/cloudsync/src/cloudsync-plugins/src/cloudsyncs3/src/libcloudsyncs3.c
xlators/features/bit-rot/src/bitd/bit-rot.c
xlators/features/bit-rot/src/bitd/bit-rot-scrub.c
bxlators/encryption/crypt/src/metadata.c
xlators/encryption/crypt/src/crypt.c
xlators/performance/md-cache/src/md-cache.c:

Move to GF_MALLOC() instead of GF_CALLOC() when possible

It doesn't make sense to calloc (allocate and clear) memory
when the code right away fills that memory with data.
It may be optimized by the compiler, or have a microscopic
performance improvement.

In some cases, also changed allocation size to be sizeof some
struct or type instead of a pointer - easier to read.
In some cases, removed redundant strlen() calls by saving the result
into a variable.

1. Only done for the straightforward cases. There's room for improvement.
2. Please review carefully, especially for string allocation, with the
terminating NULL string.

Only compile-tested!

.. and allocate memory as much as needed.

xlators/nfs/server/src/mount3.c :

Don't blindly allocate PATH_MAX, but strlen() the string and allocate
appropriately.
Also, align error messges.

updates: bz#1193929
Original-Author: Yaniv Kaul 
Signed-off-by: Yaniv Kaul 
Signed-off-by: Yaniv Kaul 
Change-Id: Ibda6f33dd180b7f7694f20a12af1e9576fe197f5

xlators: move from strlen() to sizeof()

2018-08-31T02:14:06+00:00

xlators/features/index/src/index.c
xlators/features/shard/src/shard.c
xlators/features/upcall/src/upcall-internal.c
xlators/mgmt/glusterd/src/glusterd-bitrot.c
xlators/mgmt/glusterd/src/glusterd-locks.c
xlators/mgmt/glusterd/src/glusterd-mountbroker.c
xlators/mgmt/glusterd/src/glusterd-op-sm.c

For const strings, just do compile time size calc instead of runtime.

Compile-tested only!

Change-Id: I995b2b89f14454b3855a4cd0ca90b3f01d5e080f
updates: bz#1193929
Signed-off-by: Yaniv Kaul

feature/shard: Fix Coverity issue

2018-08-27T03:29:00+00:00

Fix following coverity issues-

CID:
1394660
1394668
1394667
1389008
1389434

https://scan6.coverity.com/reports.htm#v42607/p10714/fileInstanceId=84880983&defectInstanceId=25821108&mergedDefectId=1389008
https://scan6.coverity.com/reports.htm#v42607/p10714/fileInstanceId=84880983&defectInstanceId=25821101&mergedDefectId=1389434
https://scan6.coverity.com/reports.htm#v42607/p10714/fileInstanceId=84880983&defectInstanceId=25821001&mergedDefectId=1394660
https://scan6.coverity.com/reports.htm#v42607/p10714/fileInstanceId=84880983&defectInstanceId=25821010&mergedDefectId=1394667
https://scan6.coverity.com/reports.htm#v42607/p10714/fileInstanceId=84880983&defectInstanceId=25821017&mergedDefectId=1394668

Change-Id: I08f09649dbe758ba0d367ae5330b48b18784dec3
updates: bz#789278
Signed-off-by: Ashish Pandey

features/shard: Fix crash and test case in RENAME fop

2018-08-14T13:49:42+00:00

Setting the refresh flag in inode ctx in shard_rename_src_cbk()
is applicable only when the dst file exists and is sharded and
has a hard link > 1 at the time of rename.

But this piece of code is exercised even when dst doesn't exist.
In this case, the mount crashes because local->int_inodelk.loc.inode
is NULL.

Change-Id: Iaf85a5ee3dff8b01a76e11972f10f2bb9dcbd407
Updates: bz#1611692
Signed-off-by: Krutika Dhananjay

features/shard: Make lru limit of inode list configurable

2018-07-23T07:32:32+00:00

Currently this lru limit is hard-coded to 16384. This patch makes it
configurable to make it easier to hit the lru limit and enable testing
of different cases that arise when the limit is reached.

The option is features.shard-lru-limit. It is by design allowed to
be configured only in init() but not in reconfigure(). This is to avoid
all the complexity associated with eviction of least recently used shards
when the list is shrunk.

Change-Id: Ifdcc2099f634314fafe8444e2d676e192e89e295
updates: bz#1605056
Signed-off-by: Krutika Dhananjay

features/shard: Perform shards deletion in the background

2018-06-20T09:44:56+00:00

A synctask is created that would scan the indices from
.shard/.remove_me, to delete the shards associated with the
gfid corresponding to the index bname and the rate of deletion
is controlled by the option features.shard-deletion-rate whose
default value is 100.
The task is launched on two accounts:
1. when shard receives its first-ever lookup on the volume
2. when a rename or unlink deleted an inode

Change-Id: Ia83117230c9dd7d0d9cae05235644f8475e97bc3
updates: bz#1568521
Signed-off-by: Krutika Dhananjay

features/shard: Introducing ".shard/.remove_me" for atomic shard deletion (part 1)

2018-06-13T09:57:14+00:00

PROBLEM:
Shards are deleted synchronously when a sharded file is unlinked or
when a sharded file participating as the dst in a rename() is going to
be replaced. The problem with this approach is it makes the operation
really slow, sometimes causing the application to time out, especially
with large files.

SOLUTION:
To make this operation atomic, we introduce a ".remove_me" directory.
Now renames and unlinks will simply involve two steps:
1. creating an empty file under .remove_me named after the gfid of the file
participating in unlink/rename
2. carrying out the actual rename/unlink
A synctask is created (more on that in part 2) to scan this directory
after every unlink/rename operation (or upon a volume mount) and clean
up all shards associated with it. All of this happens in the background.
The task takes care to delete the shards associated with the gfid in
.remove_me only if this gfid doesn't exist in backend, ensuring that the
file was successfully renamed/unlinked and its shards can be discarded now
safely.

Change-Id: Ia1d238b721a3e99f951a73abbe199e4245f51a3a
updates: bz#1568521
Signed-off-by: Krutika Dhananjay

features/shard: Fix missing unlock in shard_fsync_shards_cbk()

2018-06-01T09:33:01+00:00

updates: bz#789278

Change-Id: I745a98e957cf3c6ba69247fcf6b58dd05cf59c3c
Signed-off-by: Vijay Bellur