glusterfs.git/xlators/features/upcall/src, branch v3.10.2

marker: Fix inode value in loc, in setxattr fop

2017-02-20T12:24:01+00:00

On recieving a rename fop, marker_rename() stores the,
oldloc and newloc in its 'local' struct, once the rename
is done, the xtime marker(last updated time) is set on
the file, but sending a setxattr fop. When upcall
receives the setxattr fop, the loc->inode is NULL and
it crashes. The loc->inode can be NULL only in one valid
case, i.e. in rename case where the inode of new loc
can be NULL. Hence, marker should have filled the inode
of the new_loc before issuing a setxattr.

marker_rename_cbk was already fixed in a previous commit.
Fixing marker_rename_done to send valid inode in this commit.

Also in upcall check for NULL inode so that there is no crash.

>Reviewed-on: https://review.gluster.org/16633
>Reviewed-by: Kotresh HR 
>Smoke: Gluster Build System 
>CentOS-regression: Gluster Build System 
>Reviewed-by: soumya k 
>NetBSD-regression: NetBSD Build System 
>Reviewed-by: Raghavendra G 
>(cherry picked from commit 73defab8be16b73241225bb1c2588a61e3e425d5)

Change-Id: I3ed2a05118fed3367dfe3251ce4477310cb480d0
BUG: 1424937
Signed-off-by: Poornima G 
Reviewed-on: https://review.gluster.org/16684
Tested-by: Atin Mukherjee 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

upcall: Resolve dict leak from up_(f)removexattr in upcall code path

2017-01-16T09:32:13+00:00

Problem: In up_(f)removexattr() dict_for_key_value() is used to create a
         new dict. This dict is not correctly unref'd and gets leaked.

Solution: To avoid the leak up_(f)removexattr() now also does a
          dict_unref() on the newly created dict.

While reviewing the code in up_(f)setxattr() for a similar problem, it
was noticed that there is an extra dict created. There is no need for
this copy, upcall_local_init() can just take the dict that was passed as
argument to the FOP.

BUG: 1412917
Change-Id: I5bb9a7d99f5087af11c19ae722de62bdb5ad1498
Signed-off-by: Mohit Agrawal 
Reviewed-on: http://review.gluster.org/16392
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos 
Smoke: Gluster Build System

Upcall: Fix possible memleak when inode_ctx_set fails

2017-01-12T15:47:23+00:00

In __upcall_inode_ctx_set(), if inode_ctx_set fails we should
free allocated memory for ctx. This patch takes care of the same.

Change-Id: Iafb42787151a579caf6f396c9b414ea48d16e6b4
BUG: 1412489
Reported-by: Nithya Balachandran 
Signed-off-by: Soumya Koduri 
Reviewed-on: http://review.gluster.org/16381
Reviewed-by: N Balachandran 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Jeff Darcy

upcall: Fix 'use after free' in a log message

2016-12-13T14:48:41+00:00

There is chance of accessing freed pointer in a log message at TRACE
level while cleaning up expired client entries.

Change-Id: I06b4dad755df63978ab04ca52442bfd4600d139a
BUG: 1404168
Reported-by: Ravishankar N 
Signed-off-by: Soumya Koduri 
Reviewed-on: http://review.gluster.org/16117
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos 
Smoke: Gluster Build System

upcall: Fix a log level

2016-11-08T20:44:47+00:00

In upcall_cache_invalidation(), the gfid can be NULL in certain
valid test cases(eg: entry for ".." in readdirp), hence change
the log level from WARNING to DEBUG.

Change-Id: Ic90167a0e2076694e9131913114460df7b939b30
BUG: 1392167
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/15777
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Rajesh Joseph 
Reviewed-by: Jeff Darcy

md-cache, afr: Reduce the window of stale read

2016-10-20T07:07:55+00:00

Problem:
Consider a replica setup, where one mount writes data to a
file and the other mount reads the file. In afr, read operations
are not transaction based, a brick(read subvolume) is chosen as
a part of lookup or other operations, read is always wound only
to the read subvolume, even if there was write from a different client
that failed on this brick. This stale read continues until there is
a lookup or any write operation from the mount point. Currently, this
is not a major issue, as a lookup is issued before every read and it will
switch the read subvolume to a correct one. But with the plan of
increasing md-cache timeout to 600s, the stale read problem will be
more pronounced, i.e. stale read can continue for 600s(or more if cascaded
with readdirp), as there will be no lookups.

Solution:
Afr doesn't have any built-in solution for stale read(without affecting
the performance). The solution that came up, was to use upcall. When a file
on any brick is marked bad for the first time, upcall sends a notification
to all the clients that had recently accessed the file. The solution has
2 parts:
- Identifying when a file is marked bad, on any of the bricks,
  for the first time
- Client side actions on recieving the notifications

Identifying when a file is marked bad on any of the bricks for the first time:
-----------------------------------------------------------------------------
The idea is to track xattrop in upcall. xattrop currently comes with 2 afr
xattrs - afr dirty bit and afr pending xattrs.
   Dirty xattr is set to 1 before every write, and is unset if write succeeds.
In certain scenarios, dirty xattr can be 0 and still the file could be bad
copy. Hence do not track dirty xattr.
   Pending xattr is set on the good copy, indicating the other bricks that have
bad copy. It is still not as simple as, notifying when any of the pending xattrs
change. It could lead to flood of notifcations, in case the other brick is
completely down or consistantly failing. Hence it is important to notify only
once, the first time a good copy is marked bad.

Client side actions on recieving pending xattr change, notification:
--------------------------------------------------------------------
md-cache will invalidate the cache of that file, so that further lookup is
passed down to afr and hence update the read subvolume. Invalidating only in
md-cache is not enough, consider the folling oder of opertaions:
- pending xattr invalidation - invalidate md-cache
- readdirp on the bad read subvolume - fill md-cache
- lookup (served from md-cache)
- read - wound to the old read subvol.
Hence, along with invalidating md-cache, it is very important to reset the
read subvolume for that file, in afr.

Design Credit: Anuradha Talur, Ravishankar N

1. xattrop doesn't carry info saying post op/pre op.
2. Pre xattrop will have 0 value for all pending xattrs,
   the cbk of pre xattrop carries the on-disk xattr value.
   Non zero indicated healing is required.
3. Post xattrop will have non zero value for any of the
   pending xattrs, if the fop failed on any of the bricks.

Change-Id: I469cbc111714c433984fe1c922be2ef113c25804
BUG: 1211863
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/15398
Reviewed-by: Pranith Kumar Karampuri 
Tested-by: Pranith Kumar Karampuri 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

build: out-of-tree builds generates files in the wrong directory

2016-09-18T16:34:37+00:00

And minor cleanup of a few of the Makefile.am files while we're
at it.

Rewrite the make rules to do what xdrgen does. Now we can get rid
of xdrgen.

Note 1. netbsd6's sed doesn't do -i. Why are we still running
smoke tests on netbsd6 and not netbsd7? We barely support netbsd7
as it is.

Note 2. Why is/was libgfxdr.so (.../rpc/xdr/src/...) linked with
libglusterfs? A cut-and-paste mistake? It has no references to
symbols in libglusterfs.

Note3. "/#ifndef\|#define\|#endif/" (note the '\'s) is a _basic_
regex that matches the same lines as the _extended_ regex
"/#(ifndef|define|endif)/". To match the extended regex sed needs to
be run with -r on Linux; with -E on *BSD. However NetBSD's and
FreeBSD's sed helpfully also provide -r for compatibility. Using a
basic regex avoids having to use a kludge in order to run sed with
the correct option on OS X.

Note 4. Not copying the bit of xdrgen that inserts copyright/license
boilerplate. AFAIK it's silly to pretend that machine generated
files like these can be copyrighted or need license boilerplate.
The XDR source files have their own copyright and license; and
their copyrights are bound to be more up to date than old
boilerplate inserted by a script. From what I've seen of other
Open Source projects -- e.g. gcc and its C parser files generated
by yacc and lex -- IIRC they don't bother to add copyright/license
boilerplate to their generated files.

It appears that it's a long-standing feature of make (SysV, BSD,
gnu) for out-of-tree builds to helpfully pretend that the source
files it can find in the VPATH "exist" as if they are in the $cwd.
rpcgen doesn't work well in this situation and generates files
with "bad" #include directives.

E.g. if you `rpcgen ../../../../$srcdir/rpc/xdr/src/glusterfs3-xdr.x`,
you get an #include directive in the generated .c file like this:

  ...
  #include "../../../../$srcdir/rpc/xdr/src/glusterfs3-xdr.h"
  ...

which (obviously) results in compile errors on out-of-tree build
because the (generated) header file doesn't exist at that location.
Compared to `rpcgen ./glusterfs3-xdr.x` where you get:

  ...
  #include "glusterfs3-xdr.h"
  ...

Which is what we need. We have to resort to some Stupid Make Tricks
like the addition of various .PHONY targets to work around the VPATH
"help".

Warning: When doing an in-tree build, -I$(top_builddir)/rpc/xdr/...
looks exactly like -I$(top_srcdir)/rpc/xdr/...  Don't be fooled though.
And don't delete the -I$(top_builddir)/rpc/xdr/... bits

Change-Id: Iba6ab96b2d0a17c5a7e9f92233993b318858b62e
BUG: 1330604
Signed-off-by: Kaleb S KEITHLEY 
Reviewed-on: http://review.gluster.org/14085
Tested-by: Niels de Vos 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

xlators/upcall: fix unused variable warnings/errors

2016-09-13T16:50:21+00:00

http://review.gluster.org/14085 fixes a "pragma leak" where the
generated rpc/xdr headers have a pair of pragmas that disable these
warnings. With the warnings disabled, many unused variables have
crept into the code base.

And 14085 won't pass its own smoke test until all these warnings are
fixed.

BUG: 1369124
Change-Id: I48c0d9bbf13ccf79975dd7b35d524f6d7428ac52
Signed-off-by: Kaleb S. KEITHLEY 
Reviewed-on: http://review.gluster.org/15484
Reviewed-by: Niels de Vos 
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System

upcall: Mark the clients as accessed on readdirp entries

2016-08-31T13:44:01+00:00

Currently when a client performs a readdirp it is not stored
in upcall, as one of the clients that have accessed the files.
Hence, when any other client modifies the file, the client that
had performed readdirp will not get any notifications.

Fix this by adding the clients to upcall database when they
perform readdirp.

Change-Id: I7767f1e26bf1bd1f67702a6d01f8aa64526ccc46
BUG: 1369430
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/15313
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: soumya k 
Reviewed-by: Niels de Vos

md-cache: Register the list of xattrs with cache-invalidation

2016-08-31T06:07:01+00:00

Issue:
md-cache caches a specified list of xattrs, and when cache invalidation
is enabled, it makes sense to recieve invalidation only when those xattrs
are modified by other clients. But the current implementation of upcall
is that, it will send invalidation when any of the on-disk xattrs is modified.

Solution:
md-cache sends a list of xattrs that it is interested in, to upcall by
issuing an ipc(). The challenge here is to make sure everytime a brick
goes offline and comes back up, the ipc() needs to be issued to the
bricks. Hence ipc() is sent from md-cache every time there is a
CHILD_UP/CHILD_MODIFIED event.

TODO:
There will be patches following, in cluster xlators, to implement ipc fop.

Change-Id: I6efcf3df474f5ce6eabd3d6694c00c7bd89bc25d
BUG: 1211863
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/15002
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Rajesh Joseph 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Prashanth Pai 
Reviewed-by: Raghavendra G