glusterfs.git/contrib, branch v3.11.1

fuse: implement "-oauto_unmount"

2017-06-20T13:44:39+00:00

libfuse has an auto_unmount option which,
if enabled, ensures that the file system
is unmounted at FUSE server termination
by running a separate monitor process
that performs the unmount when that
occurs. (This feature would probably
better be called "robust auto-unmount",
as FUSE servers usually do try to unmount
their file systems upon termination,
it's just this mechanism is not crash
resilient.)

This change implements that option and
behavior for glusterfs.

Note that "auto unmount" (robust or not) is
a leaky abstraction, as the kernel cannot
guarantee that at the path where the FUSE
fs is mounted is actually the toplevel mount
at the time of the umount(2) call, for
multiple reasons, among others, see:

  fuse-devel: "fuse: feasible to distinguish between umount and abort?"
  http://fuse.996288.n3.nabble.com/fuse-feasible-to-distinguish-between-umount-and-abort-tt14358.html
  https://github.com/libfuse/libfuse/issues/122

Updates #153

> Change-Id: Ia4432580c9fd2c156d9c73c3a44f4bfd42437599
> Signed-off-by: Csaba Henk 
> Reviewed-on: https://review.gluster.org/17230
> Tested-by: Amar Tumballi 
> CentOS-regression: Gluster Build System 
> Smoke: Gluster Build System 
> Reviewed-by: Amar Tumballi 
> NetBSD-regression: NetBSD Build System 
(cherry picked from commit 461888bb63b2409f8245c7766aa799ca22f734e6)

Change-Id: Ia4432580c9fd2c156d9c73c3a44f4bfd42437599
Signed-off-by: Csaba Henk 
Reviewed-on: https://review.gluster.org/17573
Smoke: Gluster Build System 
Reviewed-by: Amar Tumballi 
Tested-by: Amar Tumballi 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

libglusterfs: extract some functionality to functions

2017-06-20T13:44:34+00:00

- code in run.c to close all file descriptors,
  except for specified ones is extracted to

    int close_fds_except (int *fdv, size_t count);

- tokenizing and editing a string that consists
  of comma-separated tokens (as done eg. in
  mount_param_to_flag() of contrib/fuse/mount.c
  is abstacted into the following API:

    char *token_iter_init (char *str, char sep, token_iter_t *tit);
    gf_boolean_t next_token (char **tokenp, token_iter_t *tit);
    void drop_token (char *token, token_iter_t *tit);

Updates #153

> Change-Id: I7cb5bda38f680f08882e2a7ef84f9142ffaa54eb
> Signed-off-by: Csaba Henk 
> Reviewed-on: https://review.gluster.org/17229
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Amar Tumballi 
(cherry picked from commit 98db583e9b2e7aa8e095a75a6bb5f42b0d65ae79)

Change-Id: I7cb5bda38f680f08882e2a7ef84f9142ffaa54eb
Signed-off-by: Csaba Henk 
Reviewed-on: https://review.gluster.org/17569
Tested-by: Amar Tumballi 
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Amar Tumballi

contrib/timerwheel: bad 32-bit, use builtin fls(), fix copyright

2017-05-16T00:36:21+00:00

It's bad form to remove other people's copyright and license when you
copy their source for your own use.

Defining BITS_PER_LONG as 64 is incorrect on 32-bit platforms.

The mismatch between the unsigned long of the timer and the int
param to fls() means on 64-bit platforms that any bits set in the
high 32-bits of the the timer are lost/ignored.

gf_tw_find_last_bit() is meant to find the last bit in an array of
longs. It's overkill for gluster's timerwheel  where we only ever pass
a single long; replacing it with a direct call to fls() which is
renamed to gf_tw_fls()

The timer routines are slightly modified from the kernel timer
functions that first appeared circa 2.6.x in .../kernel/timer.c
AFAICT.

find_last_bit() comes from the (linux) kernel (.../lib/find_bit.c
in 4.x kernels, .../lib/find_last_bit.c in 3.x kernels) but as noted
above, it is removed with this patch.

__fls() comes from the linux kernel (.../include/asm-generic/
bitops/{__fls.h,builtin-__fls.h}

Restoring/updating the copyright and license to the version from
the 4.x kernel find_bit.c. (timer.c does not have a license, __fls.h
and builtin-__fls.h do not have a copyright or license, but the whole
kernel is licensed under GPLv2 anyway.)

  mainline BUG: none
  Master: https://review.gluster.org/17146

Change-Id: I2d2defccf1ccc74f55d99e94212747a36a1dff35
BUG: 1451033
Signed-off-by: Kaleb S. KEITHLEY 
Reviewed-on: https://review.gluster.org/17301
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

fuse: clean up mount flag processing

2017-04-27T17:38:48+00:00

In general, when one invokes a mount helper program -- basically
anything that mounts something based on its command line, so thinking of
mount(8), mount. or fusermount, but also of FUSE servers in
general, including glusterfs -- the command line arguments that are to
affect mount(2) are mapped to a bitmask called the mount flags, which is
passed to mount(2), so that the kernel can interpret the flag bits and
adjusts properties of the mount accordingly.

There is a traditional syntax for this mechanism as implemented in
mount(8): one passes "-ocomma,separated,mount,options" and the
individual option name strings are mapped to flag bits in mount(8).

FUSE further explores this idea and typically the FUSE server command
lines allow further option names to be used in the "-ooption,name,list"
which are then separated from the kernel sanctioned option names (to
which we'll refer as "system mount options") and are passed to a
platform specific lower level fuse mount helper interface.

The separation of system mount option names and FUSE specific option
names is also platform specific, so the general mount interface
function, which in case of glusterfs is gf_fuse_mount(), should abstract
this away.

Therefore we change the signature of this function from

        int gf_fuse_mount (const char *mountpoint, char *fsname,
                           unsigned long mountflags, char *mnt_param,
                           pid_t *mtab_pid, int status_fd);

to

        int gf_fuse_mount (const char *mountpoint, char *fsname,
                           char *mnt_param, pid_t *mtab_pid,
                           int status_fd);

and deal with flag extraction in platform specific mount code. Note that
the sole purpose of the mountflags argument was to indicate read-only
mounting. The other system mount option names were expected to reside in
the comma-separated mnt_param string, but they were not properly
processed (see the referred BUG). With the new gf_fuse_mount signature
read-only mounting is to be indicated as a "ro" component in mnt_param.

- For Darwin, which has a dedicated, separate gf_fuse_mount
  implementation, gf_fuse_mount was ignoring mountflags, so only the
  signature had to to be adjusted. However, as bonus, we gain read-only
  support for Darwin, which was missing so far, given that it was
  indicated via the ignored mountflags. Darwin's low level mount helper
  relies on the "ro" component of the option string, which agrees with
  the new calling convention of gf_fuse_mount.

- On Linux, system mount option name handling (apart from the
  distinguished read-only option) used to have the inadvertent side
  effect of adding "nosuid,nodev" as indicated in BUG; since
  Ia89d975d1e27fcfa5ab2036ba546aa8fa0d2d1b0 this side effect is removed,
  but system mount option name handling was left broken (passing system
  mount options other than "ro" fails to mount).

- On other platforms, system mount option name handling is broken
  (expect for the distinguished read-only option).

As of this change, in the general (non-Darwin) implementation of
gf_fuse_mount we take care of proper separation of system mount names
and their conversion to mount flags. For Linux, we adopt the conversion
table from FUSE upstream. For other systems we just provide a best
effort to support those system mount options which are understood across
all Unices (nosuid,nodev,noatime,noexec,ro). (This can be improved later
to provide proper plaform support.)

BUG: 1297182
Change-Id: I5d10b5df46feba7a02bf5bf1018db69e6b52260a
Signed-off-by: Csaba Henk 
Reviewed-on: https://review.gluster.org/16313
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Amar Tumballi 
Tested-by: Amar Tumballi

cli: keep 'gluster volume status detail' consistent

2017-01-19T23:52:26+00:00

The output of the command 'gluster volume status  detail' is
not consistent between operating systems. On linux hosts it shows the
file system type, the device name, mount options and inode size of each
brick. However the same command executed on a FreeBSD host doesn't show
all this information, even for bricks stored on a linux.

Additionally, for hosts other than linux, this information is shown as
'N/A' many times. This has been fixed to show as much information as it
can be retrieved from the operating system.

The file contrib/mount/mntent.c has been mostly rewriten because it
contained many errors that caused mount information to not be retrieved
on some operating systems.

Change-Id: Icb6e19e8af6ec82255e7792ad71914ef679fc316
BUG: 1411334
Signed-off-by: Xavier Hernandez 
Reviewed-on: http://review.gluster.org/16371
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Atin Mukherjee 
Reviewed-by: Kaleb KEITHLEY

fuse: limit fusermount fallback to EPERM cases

2017-01-18T07:19:58+00:00

There are two mount mechanims for fuse:
1) Call mount(2) syscall directly -- implemented by fuse_mount_sys
2) Call out to fusermount(1) helper utilty to do the mount --
   implemented by fuse_mount_fusermount
   [Note: both libfuse and glusterfs ships a variant of this helper
   utility; named, respectively, fusermount and fusermount-glusterfs.
   The two has diverged, and are not compatible at the moment.]

The intended use of 1) is privileged mounting, ie. when root
is invoking the glusterfs client. (It cannot work for non-privileged
users as the kernel enforces privilege for mount(2), more precisely,
caller context needs CAP_SYS_ADMIN, see capabilities(7).)

The intended use of 2) is unprivileged mountig, ie. when
the glusterfs client is invoked by an unprivileged user.
The helper utility is a setuid binary, so it can perform
mount(2) on behalf of the user.

The main mount routine, gf_fuse_mount, calls fuse_mount_sys first,
and if that fails, tries also with fuse_mount_fusermount. This
is what we call "fusermount fallback". However, in the light of
the above remarks about intended use, this logic should apply if
the fuse_mount_fusermount fails because of a privilege shortage,
ie. with error "Operation not permitted" (errno EPERM).

So far the fallback was unconditional (masking bugs of
fuser_mount_sys, as it happens in referred BUG). Now we
add the "errno == EPERM" condition.

BUG: 1297182
Change-Id: Ia89d975d1e27fcfa5ab2036ba546aa8fa0d2d1b0
Signed-off-by: Csaba Henk 
Reviewed-on: http://review.gluster.org/15766
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

build: out-of-tree builds generates files in the wrong directory

2016-09-18T16:34:37+00:00

And minor cleanup of a few of the Makefile.am files while we're
at it.

Rewrite the make rules to do what xdrgen does. Now we can get rid
of xdrgen.

Note 1. netbsd6's sed doesn't do -i. Why are we still running
smoke tests on netbsd6 and not netbsd7? We barely support netbsd7
as it is.

Note 2. Why is/was libgfxdr.so (.../rpc/xdr/src/...) linked with
libglusterfs? A cut-and-paste mistake? It has no references to
symbols in libglusterfs.

Note3. "/#ifndef\|#define\|#endif/" (note the '\'s) is a _basic_
regex that matches the same lines as the _extended_ regex
"/#(ifndef|define|endif)/". To match the extended regex sed needs to
be run with -r on Linux; with -E on *BSD. However NetBSD's and
FreeBSD's sed helpfully also provide -r for compatibility. Using a
basic regex avoids having to use a kludge in order to run sed with
the correct option on OS X.

Note 4. Not copying the bit of xdrgen that inserts copyright/license
boilerplate. AFAIK it's silly to pretend that machine generated
files like these can be copyrighted or need license boilerplate.
The XDR source files have their own copyright and license; and
their copyrights are bound to be more up to date than old
boilerplate inserted by a script. From what I've seen of other
Open Source projects -- e.g. gcc and its C parser files generated
by yacc and lex -- IIRC they don't bother to add copyright/license
boilerplate to their generated files.

It appears that it's a long-standing feature of make (SysV, BSD,
gnu) for out-of-tree builds to helpfully pretend that the source
files it can find in the VPATH "exist" as if they are in the $cwd.
rpcgen doesn't work well in this situation and generates files
with "bad" #include directives.

E.g. if you `rpcgen ../../../../$srcdir/rpc/xdr/src/glusterfs3-xdr.x`,
you get an #include directive in the generated .c file like this:

  ...
  #include "../../../../$srcdir/rpc/xdr/src/glusterfs3-xdr.h"
  ...

which (obviously) results in compile errors on out-of-tree build
because the (generated) header file doesn't exist at that location.
Compared to `rpcgen ./glusterfs3-xdr.x` where you get:

  ...
  #include "glusterfs3-xdr.h"
  ...

Which is what we need. We have to resort to some Stupid Make Tricks
like the addition of various .PHONY targets to work around the VPATH
"help".

Warning: When doing an in-tree build, -I$(top_builddir)/rpc/xdr/...
looks exactly like -I$(top_srcdir)/rpc/xdr/...  Don't be fooled though.
And don't delete the -I$(top_builddir)/rpc/xdr/... bits

Change-Id: Iba6ab96b2d0a17c5a7e9f92233993b318858b62e
BUG: 1330604
Signed-off-by: Kaleb S KEITHLEY 
Reviewed-on: http://review.gluster.org/14085
Tested-by: Niels de Vos 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

qemu-block: deprecated/defunct, remove from tree

2016-03-07T11:34:59+00:00

qemu-block xlator is not used by anyone, or so I'm told.

It's also substantially out of date. There's little reason to keep
it in our sources. (And FedoraProject doesn't like bundled software
either.)

Change-Id: I4aeb2fdfd962ec6d93de6bae126874121272220a
Signed-off-by: Kaleb S KEITHLEY 
Reviewed-on: http://review.gluster.org/13473
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Shyamsundar Ranganathan 
Reviewed-by: Jeff Darcy

fuse: add support for SEEK_HOLE and SEEK_DATA through lseek()

2016-02-11T06:16:50+00:00

The Linux FUSE kernel module has gained support for passing SEEK_HOLE
and SEEK_DATA on through lseek(). This can greatly improve performance
when working with sparse files.

Linux FUSE introduced support for lseek() with version 4.5. The commit
in mainline Linux is 0b5da8db145bfd44266ac964a2636a0cf8d7c286.

URL: http://thread.gmane.org/gmane.comp.file-systems.fuse.devel/14752
Change-Id: I12496d788e59461a3023ddd30e0ea3179007f77e
BUG: 1220173
Signed-off-by: Ravishankar N 
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/11474
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Raghavendra G

fuse: update fuse_kernel.h to version 23

2016-02-07T05:56:46+00:00

The following changes were made upstream:
 - add FUSE_WRITEBACK_CACHE
 - add time_gran to fuse_init_out
 - add reserved space to fuse_init_out
 - add FATTR_CTIME
 - add ctime and ctimensec to fuse_setattr_in
 - add FUSE_RENAME2 request
 - add FUSE_NO_OPEN_SUPPORT flag

Including these changes will make it easier to backport support for
lseek().

Because the fuse_init_out structure changed its size, older versions of
FUSE would fail initializing. When an older version of FUSE is detected,
the fuse_init_out structure is reduced to the previous size. This is
harmless, as the attributes that are not passed, are not used for
earlier versions anyway.

BUG: 1220173
Change-Id: I58c74e161638b2d4ce12fc91a206fdc1b96de14d
Signed-off-by: Ravishankar N 
[ndevos: splitted from http://review.gluster.org/11474
         old version fuse_init_out size correction]
Reviewed-on: http://review.gluster.org/11537
Smoke: Gluster Build System 
Reviewed-by: Kaleb KEITHLEY 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System