glusterfs.git/xlators/performance/readdir-ahead/src/readdir-ahead.c, branch v3.11.3

readdir-ahead: Fix duplicate listing and cache size calculation

2017-06-21T12:48:09+00:00

Issue:
If a opendir is followed by a closedir without readdir, though
the prefetched entries were freed, the freed size was not accounted
in priv->rda_cache_size. Thus the cache limit will exceed if there
are multiple opendir followed by closedir.

Fix:
Fix the pric->rda_cache_size calculation. Also have removed the
inode_ctx_size. Each perf xlator has its own cache limit that
it works with. Also the inode_ctx size can change, if a forget/
invalidate or any other factor triggers the inode_ctx size.


> Reviewed-on: https://review.gluster.org/17504
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Raghavendra G 
> (cherry picked from commit e97c32ee9913969a726f8a8286cf714f907729d6)

Change-Id: I9707ec558076ce046e58a55989ec9513c70ea029
BUG: 1460898
Signed-off-by: Poornima G 
Reviewed-on: https://review.gluster.org/17529
Reviewed-by: Raghavendra G 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

rda, glusterd: Change the max of rda-cache-limit to INFINITY

2017-05-22T15:05:43+00:00

Issue:
The max value of rda-cache-limit is 1GB before this patch.
When parallel-readdir is enabled, there will be many instances of
readdir-ahead, hence the rda-cache-limit depends on the number of
instances. Eg: On a volume with distribute count 4, rda-cache-limit
when parallel-readdir is enabled, will be 4GB instead of 1GB.
Consider a followinf sequence of operations:
- Enable parallel readdir
- Set rda-cache-limit to lets say 3GB
- Disable parallel-readdir, this results in one instance of readdir-ahead
  and the rda-cache-limit will be back to 1GB, but the current value is 3GB
  and hence the mount will stop working as 3GB > max 1GB.

Solution:
To fix this, we can limit the cache to 1GB even when parallel-readdir
is enabled. But there is no necessity to limit the cache to 1GB, it
can be increased if the system has enough resources. Hence getting rid
of the rda-cache-limit max value is more apt. If we just change the
rda-cache-limit max to INFINITY, we will render older(<3.11) clients
broken, when the rda-cache-limit is set to > 1GB (as the older clients
still expect a value < 1GB). To safely change the max value of
rda-cache-limit to INFINITY, add a check in glusted to verify all
the clients are > 3.11 if the value exceeds 1GB.

>Reviewed-on: https://review.gluster.org/17338
>Smoke: Gluster Build System 
>Reviewed-by: Atin Mukherjee 
>NetBSD-regression: NetBSD Build System 
>CentOS-regression: Gluster Build System 
>(cherry picked from commit e43b40296956d132c70ffa3aa07b0078733b39d4)

Change-Id: Id0cdda3b053287b659c7bf511b13db2e45b92032
BUG: 1453152
Signed-off-by: Poornima G 
Reviewed-on: https://review.gluster.org/17354
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

dht: The xattrs sent in readdirp should be sent in opendir aswell

2017-04-06T04:04:03+00:00

As readdir-ahead can be loaded as a child of dht, dht has to specify
the xattrs it is intrested in, as part of opendir call itself.

Change-Id: I012ef96cc143b0cef942df78aa7150d85ec38606
BUG: 1431908
Signed-off-by: Poornima G 
Reviewed-on: https://review.gluster.org/16902
Smoke: Gluster Build System 
Reviewed-by: Raghavendra G 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

reddir-ahead: Fix EOD propagation problem

2017-04-05T06:24:32+00:00

In readdirp fop, op_errno is overloaded to indicate the EOD detection.
If op_errno contains ENOENT, then it indicates that there are no
further entries pending read in the directory. Currently NFS uses the
ENOENT to identify the EOD.

Issue:
NFS clients issues a 4K buffer for readdirp, readdir-ahead converts it
to 128K buffer as its reading ahead. If there are 100 entries in the
bricks, 128K can get all 100 and store in readdir-ahead, but only 23
entries that can be fit in 4K will be sent to NFS. Since the whole
100 entries were read from brick, the op_errno is set to ENOENT, and
the op_errno is propagated as is when sent to NFS. Hence NFS client
in reading 23 entries thinks it reached EOD.

Solution:
Do not propogate ENOENT errno, unless all the entries are read
from the readdir ahead buffer.

Change-Id: I4f173a77b21ab9e98ae35e291a45b8fc0cde65bd
BUG: 1436086
Signed-off-by: Poornima G 
Reviewed-on: https://review.gluster.org/16953
Smoke: Gluster Build System 
Reviewed-by: Raghavendra G 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

glusterd, readdir-ahead: Fix backward incompatibility

2017-02-18T07:03:11+00:00

Issue:
Any opion is spcified in two places: In the options[] of xlator
itself and glusterd-volume-set.c. The default value of this option
can be specified in both the places. If its specified only in xlator
then the volfile generated will not have the option and default value,
it will be assigned during graph initialization.
With patch [1] the option rda-request-size was changed from INT to SIZET
type, and default was changed from 131072 to 128KB, but was specified
only in the readdir-ahead.c. Thus with this patch alone the volfile
entry for readdir-ahead looks like:
volume patchy-readdir-ahead
    type performance/readdir-ahead
    subvolumes patchy-read-ahead
end-volume

With patch [2], the default of option rda-request-size was specified
in glusterd-volume-set.c as well(as it was necessary fr parallel readdir).
With this patch the readdir entry in the volfile will look like:
volume patchy-readdir-ahead
    type performance/readdir-ahead
    option rda-cache-limit 10MB
    option rda-request-size 128KB
    option parallel-readdir off
    subvolumes patchy-read-ahead
end-volume


Now consider the server has both these patches and client doesn't.
Server will generate a volfile with entry:

The old clients which thought the option rda-request-size is of type
INT will now recieve the value 128KB which it willn't understand,
and hence fail the mount.

The issue is seen only with the combination of [1] and [2].

Solution:
Instead of specifying 128KB as default in glusterd we specify 131072
so that the old clients will interpret as INT and new ones as 128KB

Credits: Raghavendra G

Change-Id: I0c269a5890957fd8a38e9a05bdec088645a6688a
BUG: 1423410
Signed-off-by: Poornima G 
Reviewed-on: https://review.gluster.org/16657
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan 
Reviewed-by: Raghavendra G 
Reviewed-by: Atin Mukherjee

readdir-ahead: Remove unnecessary logging

2017-02-17T12:27:38+00:00

dict_get_int can return < 0 when key is not found is a valid case.
Hence no need to log.

Change-Id: If0795b0f178adbb94b10efc563506993f7411962
BUG: 1423369
Signed-off-by: Poornima G 
Reviewed-on: https://review.gluster.org/16654
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Jeff Darcy

Readdir-ahead : Honor readdir-optimise option of dht

2017-01-24T05:07:42+00:00

Change-Id: I9c5e65b32e316e6a2fc7e1f5c79fce79386b78e2
BUG: 1401812
Signed-off-by: Poornima G 
Reviewed-on: https://review.gluster.org/16071
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Raghavendra G

readdir-ahead : Perform STACK_UNWIND outside of mutex locks

2017-01-10T04:52:28+00:00

Currently STACK_UNWIND is performnd within ctx->lock.
If readdir-ahead is loaded as a child of dht, then there
can be scenarios where the function calling STACK_UNWIND
becomes re-entrant. Its a good practice to not call
STACK_WIND/UNWIND within local mutex's

Change-Id: If4e869849d99ce233014a8aad7c4d5eef8dc2e98
BUG: 1401812
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/16068
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Jeff Darcy 
Reviewed-by: Raghavendra G

readdir-ahead: Enhance EOD detection logic

2017-01-06T04:27:50+00:00

Issue:
Currently end of directory is identified on obtaining a
readdirp_cbk with op_ret = 0 (i.e. 0 entries fetched in
readdirp). Thus an extra readdirp is required for every
directory just to identify EOD. Consider a case of listing
large number of small directories. The readdirp fops required
are doubled in that case.

Solution:
On reaching the EOD, posix sets the op_errno to ENOENT,
hence along with looking for 'op_ret == 0' we also
look for 'operrno == ENOENT' ehile checking for EOD condition

Change-Id: I7a5b52e7b98f5dc236c387635fcc651dac0171b3
BUG: 1401812
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/16042
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Raghavendra G

performance/readdir-ahead: limit cache size

2016-12-22T11:43:14+00:00

This patch introduces a new option called "rda-cache-limit", which is
the maximum value the entire readdir-ahead cache can grow into. Since,
readdir-ahead holds a reference to inode through dentries, this patch
also accounts memory stored by various xlators in inode contexts.

Change-Id: I84cc0ca812f35e0f9041f8cc71effae53a9e7f99
BUG: 1356960
Signed-off-by: Raghavendra G 
Reviewed-on: http://review.gluster.org/16137
NetBSD-regression: NetBSD Build System 
Reviewed-by: Poornima G 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System