glusterfs.git/xlators/features/shard, branch v5.9

features/shard: Ref shard inode while adding to fsync list

2019-02-04T15:07:46+00:00

PROBLEM:

Lot of the earlier changes in the management of shards in lru, fsync
lists assumed that if a given shard exists in fsync list, it must be
part of lru list as well. This was found to be not true.

Consider this - a file is FALLOCATE'd to a size which would make the
number of participant shards to be greater than the lru list size.
In this case, some of the resolved shards that are to participate in
this fop will be evicted from lru list to give way to the rest of the
shards. And once FALLOCATE completes, these shards are added to fsync
list but without a ref. After the fop completes, these shard inodes
are unref'd and destroyed while their inode ctxs are still part of
fsync list. Now when an FSYNC is called on the base file and the
fsync-list traversed, the client crashes due to illegal memory access.

FIX:

Hold a ref on the shard inode when adding to fsync list as well.
And unref under following conditions:
1. when the shard is evicted from lru list
2. when the base file is fsync'd
3. when the shards are deleted.

Change-Id: Iab460667d091b8388322f59b6cb27ce69299b1b2
fixes: bz#1669382
Signed-off-by: Krutika Dhananjay 
(cherry picked from commit 72922c1fd69191b220f79905a23395c3a87f86ce)

features/shard: Fix launch of multiple synctasks for background deletion

2019-01-15T04:57:11+00:00

PROBLEM:

When multiple sharded files are deleted in quick succession, multiple
issues were observed:
1. misleading logs corresponding to a sharded file where while one log
   message said the shards corresponding to the file were deleted
   successfully, this was followed by multiple logs suggesting the very
   same operation failed. This was because of multiple synctasks
   attempting to clean up shards of the same file and only one of them
   succeeding (the one that gets ENTRYLK successfully), and the rest of
   them logging failure.

2. multiple synctasks to do background deletion would be launched, one
   for each deleted file but all of them could readdir entries from
   .remove_me at the same time could potentially contend for ENTRYLK on
   .shard for each of the entry names. This is undesirable and wasteful.

FIX:
Background deletion will now follow a state machine. In the event that
there are multiple attempts to launch synctask for background deletion,
one for each file deleted, only the first task is launched. And if while
this task is doing the cleanup, more attempts are made to delete other
files, the state of the synctask is adjusted so that it restarts the
crawl even after reaching end-of-directory to pick up any files it may
have missed in the previous iteration.

This patch also fixes uninitialized lk-owner during syncop_entrylk()
which was leading to multiple background deletion synctasks entering
the critical section at the same time and leading to illegal memory access
of base inode in the second syntcask after it was destroyed post shard deletion
by the first synctask.

Change-Id: Ib33773d27fb4be463c7a8a5a6a4b63689705324e
updates: bz#1665803
Signed-off-by: Krutika Dhananjay 
(cherry picked from commit c0c2022e7d7097e96270a74f37813eda0c4e6339)

features/shard: Assign fop id during background deletion to prevent excessive logging

2019-01-14T06:41:20+00:00

... of the kind

"[2018-12-26 05:22:44.195019] E [MSGID: 133010]
[shard.c:2253:shard_common_lookup_shards_cbk] 0-volume1-shard: Lookup
on shard 785 failed. Base file gfid = cd938e64-bf06-476f-a5d4-d580a0d37416
[No such file or directory]"

shard_common_lookup_shards_cbk() has a specific check to ignore ENOENT error without
logging them during specific fops. But because background deletion is done in a new
frame (with local->fop being GF_FOP_NULL), the ENOENT check is skipped and the
absence of shards gets logged everytime.

To fix this, local->fop is initialized to GF_FOP_UNLINK during background deletion.

Change-Id: I0ca8d3b3bfbcd354b4a555eee520eb0479bcda35
updates: bz#1665803
Signed-off-by: Krutika Dhananjay 
(cherry picked from commit aa28fe32364e39981981d18c784e7f396d56153f)

shard: prevent segfault in shard_unlink_block_inode()

2018-12-20T14:19:55+00:00

gluster-blockd sometimes segfaults with the following backtrace: 

    Core was generated by `/usr/sbin/gluster-blockd --glfs-lru-count 5 --log-level INFO'.
    Program terminated with signal 11, Segmentation fault.
    #0  0x00007fbb9cd639b9 in shard_unlink_block_inode (local=local@entry=0x7fbb80000a78, shard_block_num=) at shard.c:2929
    2929	            base_ictx->fsync_count--;
    (gdb) bt
    #0  0x00007fbb9cd639b9 in shard_unlink_block_inode (local=local@entry=0x7fbb80000a78, shard_block_num=) at shard.c:2929
    #1  0x00007fbb9cd64311 in shard_unlink_shards_do_cbk (frame=frame@entry=0x7fbb9010a768, cookie=, this=, op_ret=, op_errno=, preparent=preparent@entry=0x7fbb7470dcf8, 
        postparent=postparent@entry=0x7fbb7470dd90, xdata=xdata@entry=0x0) at shard.c:2987

A fix for this has already been provided through a Converity report.

Backport of:

> Change-Id: Ic5d302a5e32d375acf8adc412763ab94e6dabc3d
> Signed-off-by: Sunny Kumar 
> (cherry picked from commit 145e180517054626d07892219fdee689b703c218)

Change-Id: I699a039e9c5115eb3376190dd8014427d12a293b
Updates: bz#1659563
Signed-off-by: Niels de Vos

all: fix the format string exceptions

2018-11-09T14:03:02+00:00

Currently, there are possibilities in few places, where a user-controlled
(like filename, program parameter etc) string can be passed as 'fmt' for
printf(), which can lead to segfault, if the user's string contains '%s',
'%d' in it.

While fixing it, makes sense to make the explicit check for such issues
across the codebase, by making the format call properly.

Fixes: CVE-2018-14661

Fixes: bz#1647666
Change-Id: Ib547293f2d9eb618594cbff0df3b9c800e88bde4
Signed-off-by: Amar Tumballi

features/shard: Hold a ref on base inode when adding a shard to lru list

2018-10-25T13:11:49+00:00

Backport of:
> Change-Id: Ic15ca41444dd04684a9458bd4a526b1d3e160499
> Signed-off-by: Krutika Dhananjay 
> (cherry picked from commit e627977)
> BUG: 1605056

In __shard_update_shards_inode_list(), previously shard translator
was not holding a ref on the base inode whenever a shard was added to
the lru list. But if the base shard is forgotten and destroyed either
by fuse due to memory pressure or due to the file being deleted at some
point by a different client with this client still containing stale
shards in its lru list, the client would crash at the time of locking
lru_base_inode->lock owing to illegal memory access.

So now the base shard is ref'd into the inode ctx of every shard that
is added to lru list until it gets lru'd out.

The patch also handles the case where none of the shards associated
with a file that is about to be deleted are part of the LRU list and
where an unlink at the beginning of the operation destroys the base
inode (because there are no refkeepers) and hence all of the shards
that are about to be deleted will be resolved without the existence
of a base shard in-memory. This, if not handled properly, could lead
to a crash.

Change-Id: Ic15ca41444dd04684a9458bd4a526b1d3e160499
updates: bz#1641440
Signed-off-by: Krutika Dhananjay

glusterd: Update op-version from 4.2 to 5.0

2018-09-17T14:26:25+00:00

Post changing the max op-version to 4.2, after release
4.1 branching, the decision was to go with increasing
release numbers. Thus this needs to change to 5.0.

This commit addresses the above change.

Fixes: bz#1628668
Change-Id: Ifcc0c6da90fdd51e4eceea40749511110a432cce
Signed-off-by: ShyamsundarR

Land part 2 of clang-format changes

2018-09-12T12:22:45+00:00

Change-Id: Ia84cc24c8924e6d22d02ac15f611c10e26db99b4
Signed-off-by: Nigel Babu

Land clang-format changes

2018-09-12T11:52:48+00:00

Change-Id: I6f5d8140a06f3c1b2d196849299f8d483028d33b

Multiple files: calloc -> malloc

2018-09-04T04:58:14+00:00

xlators/storage/posix/src/posix-inode-fd-ops.c:
xlators/storage/posix/src/posix-helpers.c:
xlators/storage/bd/src/bd.c:
xlators/protocol/client/src/client-lk.c:
xlators/performance/quick-read/src/quick-read.c:
xlators/performance/io-cache/src/page.c
xlators/nfs/server/src/nfs3-helpers.c
xlators/nfs/server/src/nfs-fops.c
xlators/nfs/server/src/mount3udp_svc.c
xlators/nfs/server/src/mount3.c
xlators/mount/fuse/src/fuse-helpers.c
xlators/mount/fuse/src/fuse-bridge.c
xlators/mgmt/glusterd/src/glusterd-utils.c
xlators/mgmt/glusterd/src/glusterd-syncop.h
xlators/mgmt/glusterd/src/glusterd-snapshot.c
xlators/mgmt/glusterd/src/glusterd-rpc-ops.c
xlators/mgmt/glusterd/src/glusterd-replace-brick.c
xlators/mgmt/glusterd/src/glusterd-op-sm.c
xlators/mgmt/glusterd/src/glusterd-mgmt.c
xlators/meta/src/subvolumes-dir.c
xlators/meta/src/graph-dir.c
xlators/features/trash/src/trash.c
xlators/features/shard/src/shard.h
xlators/features/shard/src/shard.c
xlators/features/marker/src/marker-quota.c
xlators/features/locks/src/common.c
xlators/features/leases/src/leases-internal.c
xlators/features/gfid-access/src/gfid-access.c
xlators/features/cloudsync/src/cloudsync-plugins/src/cloudsyncs3/src/libcloudsyncs3.c
xlators/features/bit-rot/src/bitd/bit-rot.c
xlators/features/bit-rot/src/bitd/bit-rot-scrub.c
bxlators/encryption/crypt/src/metadata.c
xlators/encryption/crypt/src/crypt.c
xlators/performance/md-cache/src/md-cache.c:

Move to GF_MALLOC() instead of GF_CALLOC() when possible

It doesn't make sense to calloc (allocate and clear) memory
when the code right away fills that memory with data.
It may be optimized by the compiler, or have a microscopic
performance improvement.

In some cases, also changed allocation size to be sizeof some
struct or type instead of a pointer - easier to read.
In some cases, removed redundant strlen() calls by saving the result
into a variable.

1. Only done for the straightforward cases. There's room for improvement.
2. Please review carefully, especially for string allocation, with the
terminating NULL string.

Only compile-tested!

.. and allocate memory as much as needed.

xlators/nfs/server/src/mount3.c :

Don't blindly allocate PATH_MAX, but strlen() the string and allocate
appropriately.
Also, align error messges.

updates: bz#1193929
Original-Author: Yaniv Kaul 
Signed-off-by: Yaniv Kaul 
Signed-off-by: Yaniv Kaul 
Change-Id: Ibda6f33dd180b7f7694f20a12af1e9576fe197f5