glusterfs.git/tests/basic, branch v3.9.0rc2

compound fops: Fix file corruption issue

2016-10-25T04:14:54+00:00

        Backport of: http://review.gluster.org/#/c/15654/

1. Address of a local variable @args is copied into state->req
in server3_3_compound (). But even after the function has gone out of
scope, in server_compound_resume () this pointer is accessed and
dereferenced. This patch fixes that.

2. Compound fops, by virtue of NOT having a vector sizer (like the one
writev has), ends up having both the header and the data (in case one of
its member fops is WRITEV) in the same hdr_iobuf. This buffer was not
being preserved through the lifetime of the compound fop, causing it to
be overwritten by a parallel write fop, even when the writev associated
with the currently executing compound fop is yet to hit the desk, thereby
corrupting the file's data. This is fixed by associating the hdr_iobuf with
the iobref so its memory remains valid through the lifetime of the fop.

3. Also fixed a use-after-free bug in protocol/client in compound fops cbk,
missed by Linux but caught by NetBSD.

Finally, big thanks to Pranith Kumar K and Raghavendra Gowdappa for their
help in debugging this file corruption issue.

Change-Id: I58da39ae544ad81192849926399a971c4c01c986
BUG: 1387984
Signed-off-by: Krutika Dhananjay 
Reviewed-on: http://review.gluster.org/15709
NetBSD-regression: NetBSD Build System 
Reviewed-by: Raghavendra G 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System

cluster/ec: set/unset dirty flag for data/metadata update

2016-10-13T20:04:11+00:00

Currently, for all the update operations, metadata or data,
we set the dirty flag at the end of the operation only if
a brick is down. This leads to delay in healing and in some
cases not at all.
In this patch we set (+1) the dirty flag
at the start of the metadata or data update operations and
after successfull completion of the fop, we unset (-1) it again.

>Change-Id: Ide5668bdec7b937a61c5c840cdc79a967598e1e9
>BUG: 1316873
>Signed-off-by: Ashish Pandey 
>Reviewed-on: http://review.gluster.org/13733
>Tested-by: Pranith Kumar Karampuri 
>Smoke: Gluster Build System 
>NetBSD-regression: NetBSD Build System 
>CentOS-regression: Gluster Build System 
>Reviewed-by: Xavier Hernandez 

Change-Id: Ide5668bdec7b937a61c5c840cdc79a967598e1e9
BUG: 1377570
Signed-off-by: Ashish Pandey 
Reviewed-on: http://review.gluster.org/15534
Reviewed-by: Pranith Kumar Karampuri 
Tested-by: Pranith Kumar Karampuri 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

cluster/ec: Implement heal info with lock

2016-10-13T14:35:46+00:00

Problem: Currently heal info command prints all
the files/directories if the index for the
file/directory is present in .glusterfs/indices folder.
After implementing patch http://review.gluster.org/#/c/13733/
indices of the file which is going through update fop
will also be present in .glusterfs/indices even
if the fop is successful on all the brick. At this time
if heal info command is being used, it will also display this
file which is actually healthy and does not require any heal.

Solution: Take lock on a file corresponding to the indices
and inspect xattrs to decide if the file needs heal or not.

>Reviewed-on: http://review.gluster.org/15543
>NetBSD-regression: NetBSD Build System 
>Reviewed-by: Pranith Kumar Karampuri 
>Reviewed-by: Xavier Hernandez 
>CentOS-regression: Gluster Build System 
>Smoke: Gluster Build System 
>Signed-off-by: Ashish Pandey 

Change-Id: I6361e2813ece369be12d02e74816df4eddb81cfa
BUG: 1383913
Signed-off-by: Ashish Pandey 
Reviewed-on: http://review.gluster.org/15627
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Xavier Hernandez

gfapi: redesign the public interface for upcall consumers

2016-10-11T12:18:56+00:00

The glfs_callback_arg and glfs_callback_inode_arg were allocated by
gfapi, and expected to be free()'d by the application. However it is not
reasonable to expect that applications use the same memory allocator to
as the compiled libgfapi.so. For instance, it is possible that gfapi
uses glibc malloc/free, and an application like NFS-Ganesha the versions
from jemalloc. Mismatching of the malloc() and free() functions causes
segmentation faults at best.

In order to prevent problems like this in the future, the API for
applications that consume upcalls has been remodeled. Any of the
structures that gfapi allocates, should be free'd with glfs_free(). The
members of the structures can not be accessed directly anymore, each
has its own function to access now.

Correcting the naming of the functions, structures and constants is a
continuation of commit 2775dc64101ed37c8d9809bf9852dbf0746ee2b6. These
new improvements not only have correct prefixes for the functions and
structures, the naming also reflects more to the upcall framework and
does not use "callback" anymore.

Cherry picked from commit 4721188a154acd9a0a4c096d8d73e97f3bf1b2a9:
> Change-Id: I2b8bd5a0a82036d2abea1a217f5e5975a1d4fe93
> BUG: 1344714
> Signed-off-by: Niels de Vos 
> Reviewed-on: http://review.gluster.org/14701
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Kaleb KEITHLEY 
> Reviewed-by: soumya k 
> Reviewed-by: jiffin tony Thottan 

Change-Id: I2b8bd5a0a82036d2abea1a217f5e5975a1d4fe93
BUG: 1378948
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/15597
NetBSD-regression: NetBSD Build System 
Reviewed-by: jiffin tony Thottan 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Kaleb KEITHLEY

cluster/ec: Add support for hardware acceleration

2016-09-10T02:19:13+00:00

This patch implements functionalities for fast encoding/decoding
using hardware support. Currently optimized x86_64, SSE and AVX is
added.

Additionally this patch implements a caching mecanism for inverse
matrices to reduce computation time, as well as a new method for
computing the inverse that takes quadratic time instead of cubic.

Finally some unnecessary memory copies have been eliminated to
further increase performance.

 >Change-Id: I26c75f26fb4201bd22b51335448ea4357235065a
 >BUG: 1289922
 >Signed-off-by: Xavier Hernandez 
 >Reviewed-on: http://review.gluster.org/12837
 >Tested-by: Pranith Kumar Karampuri 
 >Smoke: Gluster Build System 
 >NetBSD-regression: NetBSD Build System 
 >CentOS-regression: Gluster Build System 
 >Reviewed-by: Pranith Kumar Karampuri 

BUG: 1374841
Change-Id: I83731663922ed11ca84536deab5737463416e1e0
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/15455
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

tests: Fix spurious failures because of wrong shd up function

2016-08-31T18:01:39+00:00

Fixed the way shd up check is done to prevent self-heal daemon
not running error when heal full command is executed.

Change-Id: I93c4a0da12316373d62cd4ea74432cd9bf2b090c
BUG: 1370053
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/15341
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Anuradha Talur

gfapi: SSL connection for mgmt connection is not working

2016-08-29T06:19:33+00:00

Problem: libgfapi does not enable SSL on mgmt connection.

Fix: Enable SSL when it is enabled on mgmt connection is enabled,
i.e. presence of /var/lib/glusterd/secure-access file

Change-Id: I1ce4935b04e6140aeab819e42076defd580b0727
BUG: 1362602
Signed-off-by: Rajesh Joseph 
Reviewed-on: http://review.gluster.org/15073
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos 
Reviewed-by: Kaushal M

tests: change EXPECT_WITHIN timeouts

2016-08-29T06:09:42+00:00

Use defined HEAL and PROCESS_UP timeouts rather than
hard code them in self-heald.t.

Change-Id: I21586811904c8417b7208bb643f14dff20dc4832
BUG: 1370074
Signed-off-by: Anuradha Talur 
Reviewed-on: http://review.gluster.org/15316
Reviewed-by: Ravishankar N 
Tested-by: Ravishankar N 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Krutika Dhananjay 
CentOS-regression: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri

gfapi: do not cache upcalls if the application is not interested

2016-08-25T20:35:43+00:00

When the volume option 'features.cache-invalidation' is enabled, upcall
events are sent from the brick process to the client. Even if the client
is not interested in upcall events itself, md-cache or other xlators may
benefit from them.

By adding a new 'cache_upcalls' boolean in the 'struct glfs', we can
enable the caching of upcalls when the application called
glfs_h_poll_upcall(). NFS-Ganesha sets up a thread for handling upcalls
in the initialization phase, and calls glfs_h_poll_upcall() before any
NFS-client accesses the NFS-export.

In the future there will be a more flexible registration API for
enabling certain kind of upcall events. Until that is available, this
should work just fine.

Verificatio of this change is not trivial within our current regression
test framework. The bug report contains a description on how to reliably
reproduce the problem with the glusterfs-coreutils.

Change-Id: I818595c92db50e6e48f7bfe287ee05103a4a30a2
BUG: 1368842
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/15191
Smoke: Gluster Build System 
Reviewed-by: Poornima G 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: soumya k 
Reviewed-by: Kaleb KEITHLEY

snapshot/cli: Fix snapshot status xml output

2016-08-23T07:11:54+00:00

snap status --xml errors out if a brick is down and
doesn't have pid. It is handled in the cli of the snap
status where "N/A" is displayed in such a scenario.
Handled the same in xml

snap status  --xml fails as the writer is
not initialised for the same. Using GF_SNAP_STATUS_TYPE_ITER
instead of GF_SNAP_STATUS_TYPE_SNAP for all snap's
status to differentiate between the two scenarios.

Added testcase volume-snapshot-xml.t to check
all snapshot commands xml outputs

Change-Id: I99563e8f3e84f1aaeabd865326bb825c44f5c745
BUG: 1325831
Signed-off-by: Avra Sengupta 
Reviewed-on: http://review.gluster.org/14018
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Rajesh Joseph