glusterfs.git/api/src, branch experimental

gfapi: Access fs->oldvolfile under mutex lock

2018-12-25T17:36:01+00:00

In some cases (for eg., when there are multiple
RPC_CLNT_CONNECT notifications), multiple threads may fetch
volfile and try to update it in 'fs' object simultaneously.
Hence protect those variables' access under fs->mutex lock.

Change-Id: Idaee9548560db32d83f4c04ebb1f375fee7864a9
fixes: bz#1660577
Signed-off-by: Soumya Koduri

clang: Fix various missing checks for empty list

2018-12-14T04:33:15+00:00

When using list_for_each_entry(_safe) functions, care needs
to be taken that the list passed in are not empty, as these
functions are not empty list safe.

clag scan reported various points where this this pattern
could be caught, and this patch fixes the same.

Additionally the following changes are present in this patch,
- Added an explicit op_ret setting in error case in the
macro MAKE_INODE_HANDLE to address another clang issue reported
- Minor refactoring of some functions in quota code, to address
possible allocation failures in certain functions (which in turn
cause possible empty lists to be passed around)

Change-Id: I1e761a8d218708f714effb56fa643df2a3ea2cc7
Updates: bz#1622665
Signed-off-by: ShyamsundarR

xlator: make 'xlator_api' mandatory

2018-12-13T09:11:50+00:00

* Remove the options to load old symbol.
* keep only 'xlator_api' symbol from being exported using xlator.sym
* add xlator_api to all the xlators where its missing

NOTE: This covers all the xlators which has at least a test case
to validate its loading. If there is a translator, which doesn't
have any test, then we should probably remove that from codebase.

fixes: #164
Change-Id: Ibcdc8c9844cda6b4463d907a15813745d14c1ebb
Signed-off-by: Amar Tumballi

copy_file_range support in GlusterFS

2018-12-12T15:56:55+00:00

    * libglusterfs changes to add new fop

    * Fuse changes:
      - Changes in fuse bridge xlator to receive and send responses

    * posix changes to perform the op on the backend filesystem

    * protocol and rpc changes for sending and receiving the fop

    * gfapi changes for performing the fop

    * tools: glfs-copy-file-range tool for testing copy_file_range fop

      - Although, copy_file_range support has been added to the upstream
	    fuse kernel module, no release has been made yet of a kernel
        which contains the support. It is expected to come in the
        upcoming release of linux-4.20

        So, as of now, executing copy_file_range fop on a fused based
        filesystem results in fuse kernel module sending read on the
	    source fd and write on the destination fd.

	    Therefore a small gfapi based tool has been written to be able
        test the copy_file_range fop. This tool is similar (in functionality)
	    to the example program given in copy_file_range man page.

	    So, running regular copy_file_range on a fuse mount point and
	    running gfapi based glfs-copy-file-range tool gives some idea about
	    how fast, the copy_file_range (or reflink) can be.

	    On the local machine this was the result obtained.

	    mount -t glusterfs workstation:new /mnt/glusterfs
	    [root@workstation ~]# cd /mnt/glusterfs/
	    [root@workstation glusterfs]# ls
	    file
	    [root@workstation glusterfs]# cd
	    [root@workstation ~]# time /tmp/a.out /mnt/glusterfs/file /mnt/glusterfs/new
	    real  0m6.495s
	    user  0m0.000s
	    sys   0m1.439s
	    [root@workstation ~]# time glfs-copy-file-range $(hostname) new /tmp/glfs.log /file /rrr
	    OPEN_SRC: opening /file is success
	    OPEN_DST: opening /rrr is success
	    FSTAT_SRC: fstat on /rrr is success
	    copy_file_range successful

        real  0m0.309s
        user  0m0.039s
        sys   0m0.017s

        This tool needs following arguments
         1) hostname
         2) volume name
         3) log file path
         4) source file path (relative to the gluster volume root)
         5) destination file path (relative to the gluster volume root)

        "glfs-copy-file-range     "

      - Added a testcase as well to run glfs-copy-file-range tool

    * io-stats changes to capture the fop for profiling

    * NOTE:

      - Added conditional check to see whether the copy_file_range syscall
        is available or not. If not, then return ENOSYS.

      - Added conditional check for kernel minor version in fuse_kernel.h
        and fuse-bridge while referring to copy_file_range. And the kernel
        minor version is kept as it is. i.e. 24. Increment it in future
        when there is a kernel release which contains the support for
        copy_file_range fop in fuse kernel module.

    * The document which contains a writeup on this enhancement can be found at
      https://docs.google.com/document/d/1BSILbXr_knynNwxSyyu503JoTz5QFM_4suNIh2WwrSc/edit

Change-Id: I280069c814dd21ce6ec3be00a884fc24ab692367
updates: #536
Signed-off-by: Raghavendra Bhat

libglusterfs: Move devel headers under glusterfs directory

2018-12-05T21:47:04+00:00

libglusterfs devel package headers are referenced in code using
include semantics for a program, this while it works can be better
especially when dealing with out of tree xlator builds or in
general out of tree devel package usage.

Towards this, the following changes are done,
- moved all devel headers under a glusterfs directory
- Included these headers using system header notation <> in all
code outside of libglusterfs
- Included these headers using own program notation "" within
libglusterfs

This change although big, is just moving around the headers and
making it correct when including these headers from other sources.

This helps us correctly include libglusterfs includes without
namespace conflicts.

Change-Id: Id2a98854e671a7ee5d73be44da5ba1a74252423b
Updates: bz#1193929
Signed-off-by: ShyamsundarR

gfapi: Offload callback notifications to synctask

2018-11-26T04:20:05+00:00

Upcall notifications are received from server via epoll
and same thread is used to forward these notifications
to the application. This may lead to deadlock and hang
in the following scenario.

Consider if as part of handling these callbacks,
application has to do some operations which involve
sending I/Os to gfapi stack which inturn have to wait for
epoll threads to receive repsonse. Thus this may lead to
deadlock if all the epoll threads are waiting to complete
these callback notifications.

To address it, instead of using epoll thread itself,
make use of synctask to send those notificaitons to the
application.

Change-Id: If614e0d09246e4279b9d1f40d883a32a39c8fd90
updates: bz#1648768
Signed-off-by: Soumya Koduri

gfapi: Send fop_attr dict as part of syncop_open

2018-11-17T09:16:13+00:00

Leaseid (stored in thread locals) is sent to server via xdata.
This dict variable is set but not passed as argument in glfs_h_open().
Fixed the same.

Change-Id: Idd2f8a0ec184b4b6b1ad1e6e5d75df551c36a96d
updates: bz#1648768
Signed-off-by: Soumya Koduri

gfapi: fix bad dict setting of lease-id

2018-11-08T15:12:22+00:00

lease_id is a 16 bits opaque data, copying it by gf_strdup is wrong.

Invalid read of size 2
   at 0x483FA2F: memmove (vg_replace_strmem.c:1270)
   by 0xE2EF6FB: ??? (in /usr/lib64/libtirpc.so.3.0.0)
   by 0xE2EE047: xdr_opaque (in /usr/lib64/libtirpc.so.3.0.0)
   by 0x107A97DC: xdr_gfx_value (glusterfs4-xdr.c:207)
   by 0x107A98C0: xdr_gfx_dict_pair (glusterfs4-xdr.c:321)
   by 0xE2EF35E: xdr_array (in /usr/lib64/libtirpc.so.3.0.0)
   by 0x107A9A89: xdr_gfx_dict (glusterfs4-xdr.c:335)
   by 0x107AA97B: xdr_gfx_write_req (glusterfs4-xdr.c:897)
   by 0x107A181E: xdr_serialize_generic (xdr-generic.c:25)
   by 0x231044A2: client_submit_request (client.c:205)
   by 0x2314D3C1: client4_0_writev (client-rpc-fops_v2.c:3863)
   by 0x230FD5FA: client_writev (client.c:956)
 Address 0xad659e18 is 72 bytes inside a block of size 73 alloc'd
   at 0x483880B: malloc (vg_replace_malloc.c:299)
   by 0x106BA7EC: __gf_malloc (mem-pool.c:136)
   by 0x1064521E: gf_strndup (mem-pool.h:166)
   by 0x1064521E: gf_strdup (mem-pool.h:183)
   by 0x1064521E: get_fop_attr_thrd_key (glfs.c:627)
   by 0x1064D8E9: glfs_pwritev@@GFAPI_3.4.0 (glfs-fops.c:1154)
   by 0x10610C0C: glusterfs_write2 (handle.c:2092)
   by 0x54D30C: mdcache_write2 (mdcache_file.c:647)
   by 0x48A3FC: nfs4_write (nfs4_op_write.c:459)
   by 0x48A44D: nfs4_op_write (nfs4_op_write.c:487)
   by 0x4634F5: nfs4_Compound (nfs4_Compound.c:947)
   by 0x460155: nfs_rpc_process_request (nfs_worker_thread.c:1329)
   by 0x4608A3: nfs_rpc_valid_NFS (nfs_worker_thread.c:1539)
   by 0x488F12F: svc_vc_decode (svc_vc.c:825)

Updates bz#1647651
Change-Id: Ib9fff55c897bc43c15036a869888e763df133757
Signed-off-by: Kinglong Mee

api: fill out attribute information if not valid

2018-10-17T13:44:30+00:00

translators like readdir-ahead selectively retain entry information of
iatt (gfid and type) when rest of the iatt is invalidated (for write
invalidating ia_size, (m)(c)times etc). Fuse-bridge uses this
information and sends only entry information in readdirplus
response. However such option doesn't exist in gfapi. This patch
modifies gfapi to populate the stat by forcing an extra lookup.

Thanks to Shyamsundar Ranganathan  and Prashanth
Pai  for tests.

Change-Id: Ieb5f8fc76359c327627b7d8420aaf20810e53000
Fixes: bz#1630804
Signed-off-by: Raghavendra Gowdappa 
Signed-off-by: Soumya Koduri

libgfapi: Send path with "." for the root when loc.parent is set

2018-10-16T06:42:47+00:00

fixes: bz#1582516
Change-Id: Ia9649d7c3c0930dfec656d364cf6e9a98c774dd2
Signed-off-by: Kinglong Mee