glusterfs.git/xlators/features/shard, branch v3.11dev

features/shard: Fill loc.pargfid too for named lookups on individual shards

2016-11-08T11:05:27+00:00

On a sharded volume when a brick is replaced while IO is going on, named
lookup on individual shards as part of read/write was failing with
ENOENT on the replaced brick, and as a result AFR initiated name heal in
lookup callback. But since pargfid was empty (which is what this patch
attempts to fix), the resolution of the shards by protocol/server used
to fail and the following pattern of logs was seen:

Brick-logs:

[2016-11-08 07:41:49.387127] W [MSGID: 115009]
[server-resolve.c:566:server_resolve] 0-rep-server: no resolution type
for (null) (LOOKUP)
[2016-11-08 07:41:49.387157] E [MSGID: 115050]
[server-rpc-fops.c:156:server_lookup_cbk] 0-rep-server: 91833: LOOKUP(null)
(00000000-0000-0000-0000-000000000000/16d47463-ece5-4b33-9c93-470be918c0f6.82)
==> (Invalid argument) [Invalid argument]

Client-logs:
[2016-11-08 07:41:27.497687] W [MSGID: 114031]
[client-rpc-fops.c:2930:client3_3_lookup_cbk] 2-rep-client-0: remote
operation failed. Path: (null) (00000000-0000-0000-0000-000000000000)
[Invalid argument]
[2016-11-08 07:41:27.497755] W [MSGID: 114031]
[client-rpc-fops.c:2930:client3_3_lookup_cbk] 2-rep-client-1: remote
operation failed. Path: (null) (00000000-0000-0000-0000-000000000000)
[Invalid argument]
[2016-11-08 07:41:27.498500] W [MSGID: 114031]
[client-rpc-fops.c:2930:client3_3_lookup_cbk] 2-rep-client-2: remote
operation failed. Path: (null) (00000000-0000-0000-0000-000000000000)
[Invalid argument]
[2016-11-08 07:41:27.499680] E [MSGID: 133010]

Also, this patch makes AFR by itself choose a non-NULL pargfid even if
its ancestors fail to initialize all pargfid placeholders.

Change-Id: I5f85b303ede135baaf92e87ec8e09941f5ded6c1
BUG: 1392445
Signed-off-by: Krutika Dhananjay 
Reviewed-on: http://review.gluster.org/15788
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Ravishankar N 
Reviewed-by: Pranith Kumar Karampuri 
Smoke: Gluster Build System

build: out-of-tree builds generates files in the wrong directory

2016-09-18T16:34:37+00:00

And minor cleanup of a few of the Makefile.am files while we're
at it.

Rewrite the make rules to do what xdrgen does. Now we can get rid
of xdrgen.

Note 1. netbsd6's sed doesn't do -i. Why are we still running
smoke tests on netbsd6 and not netbsd7? We barely support netbsd7
as it is.

Note 2. Why is/was libgfxdr.so (.../rpc/xdr/src/...) linked with
libglusterfs? A cut-and-paste mistake? It has no references to
symbols in libglusterfs.

Note3. "/#ifndef\|#define\|#endif/" (note the '\'s) is a _basic_
regex that matches the same lines as the _extended_ regex
"/#(ifndef|define|endif)/". To match the extended regex sed needs to
be run with -r on Linux; with -E on *BSD. However NetBSD's and
FreeBSD's sed helpfully also provide -r for compatibility. Using a
basic regex avoids having to use a kludge in order to run sed with
the correct option on OS X.

Note 4. Not copying the bit of xdrgen that inserts copyright/license
boilerplate. AFAIK it's silly to pretend that machine generated
files like these can be copyrighted or need license boilerplate.
The XDR source files have their own copyright and license; and
their copyrights are bound to be more up to date than old
boilerplate inserted by a script. From what I've seen of other
Open Source projects -- e.g. gcc and its C parser files generated
by yacc and lex -- IIRC they don't bother to add copyright/license
boilerplate to their generated files.

It appears that it's a long-standing feature of make (SysV, BSD,
gnu) for out-of-tree builds to helpfully pretend that the source
files it can find in the VPATH "exist" as if they are in the $cwd.
rpcgen doesn't work well in this situation and generates files
with "bad" #include directives.

E.g. if you `rpcgen ../../../../$srcdir/rpc/xdr/src/glusterfs3-xdr.x`,
you get an #include directive in the generated .c file like this:

  ...
  #include "../../../../$srcdir/rpc/xdr/src/glusterfs3-xdr.h"
  ...

which (obviously) results in compile errors on out-of-tree build
because the (generated) header file doesn't exist at that location.
Compared to `rpcgen ./glusterfs3-xdr.x` where you get:

  ...
  #include "glusterfs3-xdr.h"
  ...

Which is what we need. We have to resort to some Stupid Make Tricks
like the addition of various .PHONY targets to work around the VPATH
"help".

Warning: When doing an in-tree build, -I$(top_builddir)/rpc/xdr/...
looks exactly like -I$(top_srcdir)/rpc/xdr/...  Don't be fooled though.
And don't delete the -I$(top_builddir)/rpc/xdr/... bits

Change-Id: Iba6ab96b2d0a17c5a7e9f92233993b318858b62e
BUG: 1330604
Signed-off-by: Kaleb S KEITHLEY 
Reviewed-on: http://review.gluster.org/14085
Tested-by: Niels de Vos 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

shard: fix unused variable warnings/errors

2016-08-29T15:14:46+00:00

http://review.gluster.org/14085 fixes a/the "leak" - via the
generated rpc/xdr headers - of pragmas that mask these warnings.

However 14085 won't pass the smoke test until all the warnings are
fixed.

Change-Id: I3e53f12371b002d7f2fe2e281b63794c7769f823
BUG: 1369124
Signed-off-by: Kaleb S. KEITHLEY 
Reviewed-on: http://review.gluster.org/15258
Reviewed-by: Krutika Dhananjay 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos 
NetBSD-regression: NetBSD Build System

build: correctly format some (s)size_t messages

2016-07-17T13:36:58+00:00

On 32-bit builds the are are warnings like these:

    posix.c:6438: warning: format '%ld' expects type 'long int', but argument 11 has type 'ssize_t'

Instead of using "%l" for (signed) size_t variables, "%z" should be
used.

BUG: 1198849
Change-Id: I6f57b5e8ea174dd9e3056aff5da685e497894ccf
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/14933
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kaleb KEITHLEY 
Smoke: Gluster Build System

core, shard: Make shards inherit main file's O_DIRECT flag if present

2016-06-07T07:19:26+00:00

If the application opens a file with O_DIRECT, the shards'
anon fds would also need to inherit the flag. Towards this,
shard xl would be passing the odirect flag in the @flags parameter
to the WRITEV fop. This will be used in anon fd resolution
and subsequent opening by posix xl.

Change-Id: Iddb75c9ed14ce5a8c5d2128ad09b749f46e3b0c2
BUG: 1342171
Signed-off-by: Krutika Dhananjay 
Reviewed-on: http://review.gluster.org/14191
Tested-by: Pranith Kumar Karampuri 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Pranith Kumar Karampuri

features/shard: Don't modify readv size

2016-06-06T15:12:55+00:00

For o-direct reads application sends aligned size which needs to be
sent as is, otherwise o-direct writes where the file-size is not
aligned fails.

Change-Id: I097418ad92eda6c835d7352a3d2e53ea9d8e2424
BUG: 1342298
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/14623
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Krutika Dhananjay 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Jeff Darcy

posix, shard: Use page-aligned buffer for o-direct reads

2016-06-04T01:58:10+00:00

and also make shard_readv_do() pass the correct flags when
the original fd is opened with O_DIRECT.

Change-Id: Ic2f8ad900743ed3f7cab56948bcf1358d247a311
BUG: 1342171
Signed-off-by: Krutika Dhananjay 
Reviewed-on: http://review.gluster.org/14639
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri

features/shard: Get hard-link-count in {unlink,rename}_cbk before deleting shards

2016-05-20T14:40:00+00:00

Change-Id: I0606b74f11f5412c4d9af44a6505635ed9022c15
BUG: 1335858
Signed-off-by: Krutika Dhananjay 
Reviewed-on: http://review.gluster.org/14334
Reviewed-by: Pranith Kumar Karampuri 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

Revert "features/shard: Make o-direct writes work with sharding"

2016-05-17T11:01:39+00:00

This reverts commit c272c71391cea9db817f4e7e38cfc25a7cff8bd5.

This is for two reasons:
1) It introduces high fop latencies
2) Even with the patch, there is no true odirect behavior since the
   workaround in the patch doesn't reduce the caching done in kernel's
   page cache as far as writes on anon fds associated with individual
   shards is concerned.

Change-Id: Iaf411198b56237ca0601deaf17f69ef178d7e769
BUG: 1335818
Signed-off-by: Krutika Dhananjay 
Reviewed-on: http://review.gluster.org/14328
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Pranith Kumar Karampuri 
CentOS-regression: Gluster Build System

features/shard: Make o-direct writes work with sharding

2016-04-11T19:48:53+00:00

With files opened with o-direct, the expectation is that
the IO performed on the fds is byte aligned wrt the sector size
of the underlying device. With files getting sharded, a single
write from the application could be broken into more than one write
falling on different shards which _might_ cause the original byte alignment
property to be lost. To get around this, shard translator will send fsync
on odirect writes to emulate o-direct-like behavior in the backend.

Change-Id: Ie8a6c004df215df78deff5cf4bcc698b4e17a7ae
BUG: 1322214
Signed-off-by: Krutika Dhananjay 
Reviewed-on: http://review.gluster.org/13846
Smoke: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System