glusterfs.git/tests, branch v5.3

fuse: add --lru-limit option

2019-01-16T15:59:03+00:00

The inode LRU mechanism is moot in fuse xlator (ie. there is no
limit for the LRU list), as fuse inodes are referenced from
kernel context, and thus they can only be dropped on request of
the kernel. This might results in a high number of passive
inodes which are useless for the glusterfs client, causing a
significant memory overhead.

This change tries to remedy this by extending the LRU semantics
and allowing to set a finite limit on the fuse inode LRU.

A brief history of problem:

When gluster's inode table was designed, fuse didn't have any
'invalidate' method, which means, userspace application could
never ask kernel to send a 'forget()' fop, instead had to wait
for kernel to send it based on kernel's parameters. Inode table
remembers the number of times kernel has cached the inode based
on the 'nlookup' parameter. And 'nlookup' field is not used by
no other entry points (like server-protocol, gfapi etc).

Hence the inode_table of fuse module always has to have lru-limit
as '0', which means no limit. GlusterFS always had to keep all
inodes in memory as kernel would have had a reference to it.
Again, the reason for this is, kernel's glusterfs inode reference
was pointer of 'inode_t' structure in glusterfs. As it is a
pointer, we could never free it (to prevent segfault, or memory
corruption).

Solution:

In the inode table, handle the prune case of inodes with 'nlookup'
differently, and call a 'invalidator' method, which in this case is
fuse_invalidate(), and it sends the request to kernel for getting
the forget request.

When the kernel sends the forget, it means, it has dropped all
the reference to the inode, and it will send the forget with the
'nlookup' parameter too. We just need to make sure to reduce the
'nlookup' value we have when we get forget. That automatically
cause the relevant prune to happen.

Credits: Csaba Henk, Xavier Hernandez, Raghavendra Gowdappa, Nithya B

fixes: bz#1623107
Change-Id: Ifee0737b23b12b1426c224ec5b8f591f487d83a2
Signed-off-by: Amar Tumballi

geo-rep: Fix syncing of files with non-ascii filenames

2018-12-26T16:50:01+00:00

Problem:
  Creation of files/directories with non-ascii names fails
  to sync to the slave. It crashes with below traceback on
  slave.
  ...
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/repce.py", line 118, in worker
    res = getattr(self.obj, rmeth)(*in_data[2:])
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py", line 709, in entry_ops
    [ESTALE, EINVAL, EBUSY])
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/syncdutils.py", line 546, in errno_wrap
    return call(*arg)
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/libcxattr.py", line 83, in lsetxattr
    cls.raise_oserr()
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/libcxattr.py", line 38, in raise_oserr
    raise OSError(errn, os.strerror(errn))
  OSError: [Errno 12] Cannot allocate memory

Cause:
  The length calculation arguments passed to blob creation was done before encoding. Hence
  was failing in gfid-access layer.

Fix:
  It appears that the calculating lenght properly fixes this issue. But it will cause
  issues in other places in 'python2' and not in 'python3'. So encoding and decoding
  each required string to make geo-rep compatible with both 'python2' and 'python3'
  is a nightmare and is not fool proof. Hence kept 'python2' code as is with out
  encode/decode and applied encode/decode only to 'python3'

Added non-ascii filename tests to regression

Backport of:
 > Patch: https://review.gluster.org/21668
 > BUG: 1650893
 > Change-Id: I35cfaf848e07b1a0b5cb93c01b98b472f08271a6
 > Signed-off-by: Kotresh HR 

fixes: bz#1648642
Change-Id: I35cfaf848e07b1a0b5cb93c01b98b472f08271a6
Signed-off-by: Kotresh HR

cluster/dht: sync brick root perms on add brick

2018-12-26T16:40:30+00:00

If a single brick is added to the volume and the
newly added brick is the first to respond to a
dht_revalidate call, its stbuf will not be merged
into local->stbuf as the brick does not yet have
a layout. The is_permission_different check therefore
fails to detect that an attr heal is required as it
only considers the stbuf values from existing bricks.
To fix this, merge all stbuf values into local->stbuf
and use local->prebuf to store the correct directory
attributes.

Change-Id: Ic9e8b04a1ab9ed1248b6b056e3450bbafe32e1bc
fixes: bz#1660736
Signed-off-by: N Balachandran

tests: Fix zero-flag.t script

2018-12-19T16:11:34+00:00

The default value of shard-block-size was changed from 4MB
to 64MB sometime back. The script "fallocate"s a 6MB file
and expects it to have 1 shard under .shard. This worked when
the shard-block-size was 4MB. With the default value now at 64MB,
file "file1" won't have any shards under .shard and the stat on the
1st shard's path fails with ENOENT.

Changed the script to explicitly set shard-block-size to 4MB.

Change-Id: I7f1785922287d16d74c95fa57cbbe12e6e66e4f7
fixes: bz#1660932
Signed-off-by: Krutika Dhananjay 
(cherry picked from commit e69fc87593334b24432978dbf592fa73fe5fc38b)

afr: thin-arbiter 2 domain locking and in-memory state

2018-12-12T14:26:42+00:00

2 domain locking + xattrop for write-txn failures:
--------------------------------------------------
- A post-op wound on TA takes AFR_TA_DOM_NOTIFY range lock and
AFR_TA_DOM_MODIFY full lock, does xattrop on TA and releases
AFR_TA_DOM_MODIFY lock and stores in-memory which brick is bad.

- All further write txn failures are handled based on this in-memory
value without querying the TA.

- When shd heals the files, it does so by requesting full lock on
AFR_TA_DOM_NOTIFY domain. Client uses this as a cue (via upcall),
releases AFR_TA_DOM_NOTIFY range lock and invalidates its in-memory
notion of which brick is bad. The next write txn failure is wound on TA
to again update the in-memory state.

- Any incomplete write txns before the AFR_TA_DOM_NOTIFY upcall release
request is got is completed before the lock is released.

- Any write txns got after the release request are maintained in a ta_waitq.

- After the release is complete, the ta_waitq elements are spliced to a
separate queue which is then processed one by one.

- For fops that come in parallel when the in-memory bad brick is still
unknown, only one is wound to TA on wire. The other ones are maintained
in a ta_onwireq which is then processed after we get the response from
TA.

Change-Id: I32c7b61a61776663601ab0040e2f0767eca1fd64
updates: bz#1648205
Signed-off-by: Ravishankar N 
Signed-off-by: Ashish Pandey

afr: assign gfid during name heal when no 'source' is present.

2018-12-12T12:57:59+00:00

Problem:
If parent dir is in split-brain or has dirty xattrs set, and the file
has gfid missing on one of the bricks, then name heal won't assign the
gfid.

Fix:
Use the brick we select the gfid from as the 'source'.

Note: Problem was found while trying to debug a split-brain issue on
Cynthia Zhou's setup.

fixes: bz#1655545
Change-Id: Id088d4f0fb017aa35122de426654194e581ed742
Reported-by: Cynthia Zhou 
Signed-off-by: Ravishankar N 
(cherry picked from commit 4d58730c0cd6ab5db39aec8a15276f7bd3371b04)

gfapi: Offload callback notifications to synctask

2018-12-12T12:53:23+00:00

Upcall notifications are received from server via epoll
and same thread is used to forward these notifications
to the application. This may lead to deadlock and hang
in the following scenario.

Consider if as part of handling these callbacks,
application has to do some operations which involve
sending I/Os to gfapi stack which inturn have to wait for
epoll threads to receive repsonse. Thus this may lead to
deadlock if all the epoll threads are waiting to complete
these callback notifications.

To address it, instead of using epoll thread itself,
make use of synctask to send those notificaitons to the
application.

Change-Id: If614e0d09246e4279b9d1f40d883a32a39c8fd90
updates: bz#1651323
Signed-off-by: Soumya Koduri 
(cherry picked from commit ad35193718a99494ab1b852ca4cbdf054f73de88)

lease: Treat unlk request as noop if lease not found

2018-12-12T12:53:10+00:00

When the glusterfs server recalls the lease, it expects
client to flush data and unlock the lease. If not it sets
a timer (starting from the time it sends RECALL request) and post
timeout, it revokes it.

Here we could have a race where in client did send UNLK
lease request but because of network delay it may have reached
after server revokes it. To handle such situations, treat
such requests as noop and return sucesss.

Change-Id: I166402d10273f4f115ff04030ecbc14676a01663
updates: bz#1651323
Signed-off-by: Soumya Koduri 
(cherry picked from commit c2e758b54d8a3f778e3e63db0000bb8b63de9b25)

cluster/afr: Use 2 domain locking in SHD for thin-arbiter

2018-11-29T15:33:37+00:00

With this change when SHD starts the index crawl it requests
all the clients to release the AFR_TA_DOM_NOTIFY lock so that
clients will know the in memory state is no more valid and
any new operations needs to query the thin-arbiter if required.

When SHD completes healing all the files without any failure, it
will again take the AFR_TA_DOM_NOTIFY lock and gets the xattrs on
TA to see whether there are any new failures happened by that time.
If there are new failures marked on TA, SHD will start the crawl
immediately to heal those failures as well. If there are no new
failures, then SHD will take the AFR_TA_DOM_MODIFY lock and unsets
the xattrs on TA, so that both the data bricks will be considered
as good there after.

>Change-Id: I037b89a0823648f314580ba0716d877bd5ddb1f1
>fixes: bz#1579788
>Signed-off-by: karthik-us 
(cherry picked from commit 5784a00f997212d34bd52b2303e20c097240d91c)

Change-Id: I037b89a0823648f314580ba0716d877bd5ddb1f1
fixes: bz#1648205

glusterd: ensure volinfo->caps is set to correct value.

2018-11-09T18:46:34+00:00

With the commit febf5ed4848, during the volume create op,
we are setting volinfo->caps to 0, only if any of the bricks
belong to the same node and brickinfo->vg[0] is null.
Previously, we used to set volinfo->caps to 0, when
either brick doesn't belong to the same node or brickinfo->vg[0]
is null.

With this patch, we set volinfo->caps to 0, when either brick
doesn't belong to the same node or brickinfo->vg[0] is null.
(as we do earlier without commit febf5ed4848).

> BUG: bz#1635820
> Change-Id: I00a97415786b775fb088ac45566ad52b402f1a49
> Signed-off-by: Sanju Rakonde 
(cherry picked from commit aae1c402b74fd02ed2f6473b896f108d82aef8e3)

fixes: bz#1647968
Change-Id: I00a97415786b775fb088ac45566ad52b402f1a49
Signed-off-by: Sanju Rakonde