glusterfs.git/xlators/performance/readdir-ahead, branch v3.10.3

dht: The xattrs sent in readdirp should be sent in opendir aswell

2017-04-27T10:49:47+00:00

As readdir-ahead can be loaded as a child of dht, dht has to specify
the xattrs it is intrested in, as part of opendir call itself.

>Reviewed-on: https://review.gluster.org/16902
>Smoke: Gluster Build System 
>Reviewed-by: Raghavendra G 
>NetBSD-regression: NetBSD Build System 
>CentOS-regression: Gluster Build System 
>(cherry picked from commit 0f71338e1d7c0b70f4fe3b19c68612fe730d9de2)

Change-Id: I012ef96cc143b0cef942df78aa7150d85ec38606
BUG: 1435942
Signed-off-by: Poornima G 
Reviewed-on: https://review.gluster.org/16947
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Prashanth Pai 
Reviewed-by: Raghavendra G

reddir-ahead: Fix EOD propagation problem

2017-04-13T15:38:18+00:00

In readdirp fop, op_errno is overloaded to indicate the EOD detection.
If op_errno contains ENOENT, then it indicates that there are no
further entries pending read in the directory. Currently NFS uses the
ENOENT to identify the EOD.

Issue:
NFS clients issues a 4K buffer for readdirp, readdir-ahead converts it
to 128K buffer as its reading ahead. If there are 100 entries in the
bricks, 128K can get all 100 and store in readdir-ahead, but only 23
entries that can be fit in 4K will be sent to NFS. Since the whole
100 entries were read from brick, the op_errno is set to ENOENT, and
the op_errno is propagated as is when sent to NFS. Hence NFS client
in reading 23 entries thinks it reached EOD.

Solution:
Do not propogate ENOENT errno, unless all the entries are read
from the readdir ahead buffer.

> Reviewed-on: https://review.gluster.org/16953
> Smoke: Gluster Build System 
> Reviewed-by: Raghavendra G 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> (cherry picked from commit 61f76f318faed395660f5bbcfe39616b39c158f0)

Change-Id: I4f173a77b21ab9e98ae35e291a45b8fc0cde65bd
BUG: 1439148
Signed-off-by: Poornima G 
Reviewed-on: https://review.gluster.org/17001
Smoke: Gluster Build System 
Reviewed-by: Raghavendra G 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

glusterd, readdir-ahead: Fix backward incompatibility

2017-02-19T14:18:08+00:00

Backport of https://review.gluster.org/#/c/16657/

Issue:
Any opion is spcified in two places: In the options[] of xlator
itself and glusterd-volume-set.c. The default value of this option
can be specified in both the places. If its specified only in xlator
then the volfile generated will not have the option and default value,
it will be assigned during graph initialization.
With patch [1] the option rda-request-size was changed from INT to SIZET
type, and default was changed from 131072 to 128KB, but was specified
only in the readdir-ahead.c. Thus with this patch alone the volfile
entry for readdir-ahead looks like:
volume patchy-readdir-ahead
    type performance/readdir-ahead
    subvolumes patchy-read-ahead
end-volume

With patch [2], the default of option rda-request-size was specified
in glusterd-volume-set.c as well(as it was necessary fr parallel readdir).
With this patch the readdir entry in the volfile will look like:
volume patchy-readdir-ahead
    type performance/readdir-ahead
    option rda-cache-limit 10MB
    option rda-request-size 128KB
    option parallel-readdir off
    subvolumes patchy-read-ahead
end-volume


Now consider the server has both these patches and client doesn't.
Server will generate a volfile with entry:

The old clients which thought the option rda-request-size is of type
INT will now recieve the value 128KB which it willn't understand,
and hence fail the mount.

The issue is seen only with the combination of [1] and [2].

Solution:
Instead of specifying 128KB as default in glusterd we specify 131072
so that the old clients will interpret as INT and new ones as 128KB

Credits: Raghavendra G

> Reviewed-on: https://review.gluster.org/16657
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Shyamsundar Ranganathan 
> Reviewed-by: Raghavendra G 
> Reviewed-by: Atin Mukherjee 

Change-Id: I0c269a5890957fd8a38e9a05bdec088645a6688a
BUG: 1423412
Signed-off-by: Poornima G 
Reviewed-on: https://review.gluster.org/16658
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Atin Mukherjee

readdir-ahead: Remove unnecessary logging

2017-02-17T16:56:00+00:00

dict_get_int can return < 0 when key is not found is a valid case.
Hence no need to log.

> Reviewed-on: https://review.gluster.org/16654
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Smoke: Gluster Build System 
> Reviewed-by: Jeff Darcy  

Change-Id: If0795b0f178adbb94b10efc563506993f7411962
BUG: 1423429
Signed-off-by: Poornima G 
Reviewed-on: https://review.gluster.org/16659
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

Readdir-ahead : Honour readdir-optimise option of dht

2017-01-27T14:01:15+00:00

>Change-Id: I9c5e65b32e316e6a2fc7e1f5c79fce79386b78e2
>BUG: 1401812
>Signed-off-by: Poornima G 
>Reviewed-on: https://review.gluster.org/16071
>Smoke: Gluster Build System 
>CentOS-regression: Gluster Build System 
>NetBSD-regression: NetBSD Build System 
>Reviewed-by: Raghavendra G 

Change-Id: I9c5e65b32e316e6a2fc7e1f5c79fce79386b78e2
BUG: 1417027
Signed-off-by: Poornima G 
(cherry picked from commit 7c6538f6c8f9a015663b4fc57c640a7c451c87f7)
Reviewed-on: https://review.gluster.org/16461
Tested-by: Raghavendra G 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G 
Reviewed-by: Shyamsundar Ranganathan

readdir-ahead : Perform STACK_UNWIND outside of mutex locks

2017-01-10T04:52:28+00:00

Currently STACK_UNWIND is performnd within ctx->lock.
If readdir-ahead is loaded as a child of dht, then there
can be scenarios where the function calling STACK_UNWIND
becomes re-entrant. Its a good practice to not call
STACK_WIND/UNWIND within local mutex's

Change-Id: If4e869849d99ce233014a8aad7c4d5eef8dc2e98
BUG: 1401812
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/16068
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Jeff Darcy 
Reviewed-by: Raghavendra G

readdir-ahead: Enhance EOD detection logic

2017-01-06T04:27:50+00:00

Issue:
Currently end of directory is identified on obtaining a
readdirp_cbk with op_ret = 0 (i.e. 0 entries fetched in
readdirp). Thus an extra readdirp is required for every
directory just to identify EOD. Consider a case of listing
large number of small directories. The readdirp fops required
are doubled in that case.

Solution:
On reaching the EOD, posix sets the op_errno to ENOENT,
hence along with looking for 'op_ret == 0' we also
look for 'operrno == ENOENT' ehile checking for EOD condition

Change-Id: I7a5b52e7b98f5dc236c387635fcc651dac0171b3
BUG: 1401812
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/16042
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Raghavendra G

performance/readdir-ahead: limit cache size

2016-12-22T11:43:14+00:00

This patch introduces a new option called "rda-cache-limit", which is
the maximum value the entire readdir-ahead cache can grow into. Since,
readdir-ahead holds a reference to inode through dentries, this patch
also accounts memory stored by various xlators in inode contexts.

Change-Id: I84cc0ca812f35e0f9041f8cc71effae53a9e7f99
BUG: 1356960
Signed-off-by: Raghavendra G 
Reviewed-on: http://review.gluster.org/16137
NetBSD-regression: NetBSD Build System 
Reviewed-by: Poornima G 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System

build: out-of-tree builds generates files in the wrong directory

2016-09-18T16:34:37+00:00

And minor cleanup of a few of the Makefile.am files while we're
at it.

Rewrite the make rules to do what xdrgen does. Now we can get rid
of xdrgen.

Note 1. netbsd6's sed doesn't do -i. Why are we still running
smoke tests on netbsd6 and not netbsd7? We barely support netbsd7
as it is.

Note 2. Why is/was libgfxdr.so (.../rpc/xdr/src/...) linked with
libglusterfs? A cut-and-paste mistake? It has no references to
symbols in libglusterfs.

Note3. "/#ifndef\|#define\|#endif/" (note the '\'s) is a _basic_
regex that matches the same lines as the _extended_ regex
"/#(ifndef|define|endif)/". To match the extended regex sed needs to
be run with -r on Linux; with -E on *BSD. However NetBSD's and
FreeBSD's sed helpfully also provide -r for compatibility. Using a
basic regex avoids having to use a kludge in order to run sed with
the correct option on OS X.

Note 4. Not copying the bit of xdrgen that inserts copyright/license
boilerplate. AFAIK it's silly to pretend that machine generated
files like these can be copyrighted or need license boilerplate.
The XDR source files have their own copyright and license; and
their copyrights are bound to be more up to date than old
boilerplate inserted by a script. From what I've seen of other
Open Source projects -- e.g. gcc and its C parser files generated
by yacc and lex -- IIRC they don't bother to add copyright/license
boilerplate to their generated files.

It appears that it's a long-standing feature of make (SysV, BSD,
gnu) for out-of-tree builds to helpfully pretend that the source
files it can find in the VPATH "exist" as if they are in the $cwd.
rpcgen doesn't work well in this situation and generates files
with "bad" #include directives.

E.g. if you `rpcgen ../../../../$srcdir/rpc/xdr/src/glusterfs3-xdr.x`,
you get an #include directive in the generated .c file like this:

  ...
  #include "../../../../$srcdir/rpc/xdr/src/glusterfs3-xdr.h"
  ...

which (obviously) results in compile errors on out-of-tree build
because the (generated) header file doesn't exist at that location.
Compared to `rpcgen ./glusterfs3-xdr.x` where you get:

  ...
  #include "glusterfs3-xdr.h"
  ...

Which is what we need. We have to resort to some Stupid Make Tricks
like the addition of various .PHONY targets to work around the VPATH
"help".

Warning: When doing an in-tree build, -I$(top_builddir)/rpc/xdr/...
looks exactly like -I$(top_srcdir)/rpc/xdr/...  Don't be fooled though.
And don't delete the -I$(top_builddir)/rpc/xdr/... bits

Change-Id: Iba6ab96b2d0a17c5a7e9f92233993b318858b62e
BUG: 1330604
Signed-off-by: Kaleb S KEITHLEY 
Reviewed-on: http://review.gluster.org/14085
Tested-by: Niels de Vos 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

readdir-ahead: Prefetch xattrs needed by md-cache

2016-05-10T09:05:38+00:00

Problem:
Negative cache feature implementation in md-cache requires xattrs
returned by posix to be intercepted for every call that can possibly
return xattrs. This includes readdirp(). This is crucial to treat
missing keys in cache as a case of negative entry (returns ENODATA)

md-cache puts names of xattrs that it wants to cache in xdata and
passes it down to posix which returns the specified xattrs in the
callback. This is done in lookup() and readdirp(). Hence, a xattr
that is cached can be invalidated during readdirp_cbk too.

This is based on the assumption that readdirp() will always return
all xattrs that md-cache is interested in. However, this is not the
case when readdirp() call is served from readdir-ahead's cache.
readdir-ahead xlator will pre-fetch dentries during opendir_cbk
and readdirp. These internal readdirp() calls made by readdir-ahead
xlator does not set xdata in it's requests. Hence, no xattrs are
fetched and stored in it's internal cache.

This causes metadata loss in gluster-swift. md-cache returns ENODATA
during getxattr() call even though the xattr for that object exists on
the brick. On receiving ENODATA, gluster-swift will create new metadata
and do setxattr(). This results in loss of information stored in
existing xattr.

Fix:
During opendir, md-cache will communicate to readdir-ahead asking it
to store the names of xattrs it's interested in so that readdir-ahead
can fetch those in all subsequent internal readdirp() calls issued by
it. This stored names of xattrs is invalidated/updated on the next
real readdirp() call issued by application. This readdirp() call will
have xdata set correctly by md-cache xlator.

BUG: 1333023
Change-Id: I32d46f93a99d4ec34c741f3c52b0646d141614f9
Reviewed-on: http://review.gluster.org/14214
Tested-by: Prashanth Pai 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Tested-by: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Raghavendra G 
Tested-by: Raghavendra G