glusterfs.git, branch v3.12dev

fuse: enhance fusedump to include timestamp and a signature

2017-04-30T08:36:25+00:00

(Also referred to as "fusedump v2".)

Change-Id: I837944024efd1b9055c2f5f91bd5723ef350e688
Signed-off-by: Csaba Henk 
Reviewed-on: https://review.gluster.org/16422
Smoke: Gluster Build System 
Reviewed-by: Amar Tumballi 
Tested-by: Raghavendra G 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G

mount/fuse: Replace GF_LOG_OCCASIONALLY with gf_log() to report fop failure at all times

2017-04-30T08:35:47+00:00

Change-Id: Ibd8e1c6172812951092ff6097ba4bed943051b7c
BUG: 1440051
Signed-off-by: Krutika Dhananjay 
Reviewed-on: https://review.gluster.org/17086
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra Bhat

cluster/dht: Make rebalance throttle option tuned by number

2017-04-29T14:29:34+00:00

Current rebalance throttle options: lazy/normal/aggressive may not always be
sufficient for the purpose of throttling.  In our recent test, we observed for
certain setups, normal and aggressive modes behaved similarly consuming full
disk bandwidth. So in cases like this admin should be able to  tune it
down(or vice versa) depending on the need.

Along with old throttle configurations, thread counts are tuned based on number.
e.g. gluster v set vol-name cluster-rebal.throttle  5.

Admin can tune up/down between 0 and the number of cores available.

Note: For heterogenous servers, validation will fail on the old server if "number"
is given for throttle configuration.
The message looks something like this:
"volume set: failed: Staging failed on vm2. Error: cluster.rebal-throttle should be {lazy|normal|aggressive}"

Test: Manual test by logging active thread number after reconfiguring throttle option.
testcase: tests/basic/distribute/throttle-rebal.t

Change-Id: I46e3cde546900307831028b344ecf601fd9b02c3
BUG: 1438370
Signed-off-by: Susant Palai 
Reviewed-on: https://review.gluster.org/16980
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Atin Mukherjee 
Reviewed-by: Raghavendra G

dht: send lookup on old name inside rename with bname and pargfid

2017-04-29T14:28:38+00:00

Inside rename, a lookup is done on the source name to make sure that
the file is there. But we used to do a gfid based lookup and hence,
even if the source name was renamed to a new name from some other client,
lookup will be successful as server3_3_lookup will fetch the new path
based on the gfid.

So even if the source file does not exist any more rename will carry on,
and as server3_3_link(destination is hashed to a different brick other
than source cached scenario) also does gfid based resolve, it wont
detect that the source name does not exist and hardlink creation will be
successful (since gfid based resolve will get the new dentry).

To solve this problem, do a name based lookup inside rename. So that
rename will fail right away if the source does not exist.

Change-Id: Ieba8bdd6675088dbf18de90ed4622df043d163bd
BUG: 1412135
Signed-off-by: Susant Palai 
Reviewed-on: https://review.gluster.org/16375
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: N Balachandran 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G

cluster/dht: rebalance perf enhancement

2017-04-29T14:28:19+00:00

Problem: Throttle settings "normal" and "aggressive" for rebalance
did not have performance difference.

normal mode spawns $(no. of cores - 4)/2 threads and aggressive
spawns $(no. of cores - 4) threads. Though aggressive mode has twice
the number of threads compared to that of normal mode, there was no
performance gain when switched to aggressive mode from normal mode.

RCA:
During the course of debugging the above problem, we tried assigning
migration job to migration threads spawned by rebalance, rather than
synctasks(as there is more overhead associated to manage the task
queue and threads). This gave us a significant improvement over rebalance
under synctasks. This patch does not really gurantee that there will be a
clear performance difference between normal and aggressive mode, but this
patch certainly maximized the disk utilization for 1GBfiles run.

Results:

Test enviroment:
Gluster Config:
Number of Bricks: 2 (one brick per disk(RAID-6 12 disk))
Bricks:
Brick1: server1:/brick/test1/1
Brick2: server2:/brick/test1/1
Options Reconfigured:
performance.readdir-ahead: on
server.event-threads: 4
client.event-threads: 4

1000 files with 1GB each were created/renamed such that all files will have
server1 as cached and server2 as hashed, so that all files will be migrated.

Test machines had 24 cores each.

Results  with/without synctask based migration:
-----------------------------------------------

mode                    normal(10threads)          aggressive(20threads)

timetaken               0:55:30 (h:m:s)            0:56:3 (h:m:s)
withsynctask

timetaken
with migrator           0:38:3 (h:m:s)             0:23:41 (h:m:s)
threads

From above table it can be seen that, there is a clear 2x perf gain between
rebalance with synctask vs rebalance with migrator threads.

Additionally this patch modifies the code so that caller will have the exact error
number returned by dht_migrate_file(earlier the errno meaning was overloaded). This
will help avoiding scenarios where migration failure due to ENOENT, can result in
rebalance abort/failure.

Change-Id: I8904e2fb147419d4a51c1267be11a08ffd52168e
BUG: 1420166
Signed-off-by: Susant Palai 
Reviewed-on: https://review.gluster.org/16427
Smoke: Gluster Build System 
Reviewed-by: N Balachandran 
Reviewed-by: Raghavendra G 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

api: fix two typos of GFAPI_PUBLIC function define

2017-04-29T12:55:43+00:00

Change-Id: I2f9915b3572c4cbce6748e99a037daba439a187e
BUG: 1446273
Signed-off-by: Kinglong Mee 
Reviewed-on: https://review.gluster.org/17112
Reviewed-by: Niels de Vos 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System

glusterd: Fix removing pmap entry on rpc disconnect

2017-04-28T17:15:30+00:00

Problem:
The following line of code intended to remove pmap entry for the
connection during disconnects:

    pmap_registry_remove (this, 0, NULL, GF_PMAP_PORT_NONE, xprt);

However, no pmap entry will have it's type set to GF_PMAP_PORT_NONE
at any point in time. So a call to pmap_registry_search_by_xprt() in
pmap_registry_remove() will always fail to find a match.

Fix:
Optionally ignore pmap entry's type in pmap_registry_search_by_xprt().

BUG: 1193929
Change-Id: I705f101739ab1647ff52a92820d478354407264a
Signed-off-by: Prashanth Pai 
Reviewed-on: https://review.gluster.org/17129
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Jeff Darcy

build: conditionally build legacy gNFS server and associated sub-packaging

2017-04-28T14:08:41+00:00

Plus some additional logic in glusterd to ensure gnfs (glusterfs)
daemons are never started if server/nfs xlator is not installed.

As a service, nfs is still initialized. The glusterfs-gnfs RPM
may be installed or uninstalled independent of anything else,
including on a system where gluster is actively running, so the
existence of the xlator is always tested before trying to start
gnfs.

Change-Id: I56743ad1cb36a84917226d7d26cb9d015d441e66
BUG: 1326219
Signed-off-by: Kaleb S. KEITHLEY 
Reviewed-on: https://review.gluster.org/16958
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System

cluster/dht: Pass the correct xdata in fremovexattr fop

2017-04-28T04:27:14+00:00

Change-Id: Id84bc87e48f435573eba3b24d3fb3c411fd2445d
BUG: 1440051
Signed-off-by: Krutika Dhananjay 
Reviewed-on: https://review.gluster.org/17126
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Raghavendra G

fuse: clean up mount flag processing

2017-04-27T17:38:48+00:00

In general, when one invokes a mount helper program -- basically
anything that mounts something based on its command line, so thinking of
mount(8), mount. or fusermount, but also of FUSE servers in
general, including glusterfs -- the command line arguments that are to
affect mount(2) are mapped to a bitmask called the mount flags, which is
passed to mount(2), so that the kernel can interpret the flag bits and
adjusts properties of the mount accordingly.

There is a traditional syntax for this mechanism as implemented in
mount(8): one passes "-ocomma,separated,mount,options" and the
individual option name strings are mapped to flag bits in mount(8).

FUSE further explores this idea and typically the FUSE server command
lines allow further option names to be used in the "-ooption,name,list"
which are then separated from the kernel sanctioned option names (to
which we'll refer as "system mount options") and are passed to a
platform specific lower level fuse mount helper interface.

The separation of system mount option names and FUSE specific option
names is also platform specific, so the general mount interface
function, which in case of glusterfs is gf_fuse_mount(), should abstract
this away.

Therefore we change the signature of this function from

        int gf_fuse_mount (const char *mountpoint, char *fsname,
                           unsigned long mountflags, char *mnt_param,
                           pid_t *mtab_pid, int status_fd);

to

        int gf_fuse_mount (const char *mountpoint, char *fsname,
                           char *mnt_param, pid_t *mtab_pid,
                           int status_fd);

and deal with flag extraction in platform specific mount code. Note that
the sole purpose of the mountflags argument was to indicate read-only
mounting. The other system mount option names were expected to reside in
the comma-separated mnt_param string, but they were not properly
processed (see the referred BUG). With the new gf_fuse_mount signature
read-only mounting is to be indicated as a "ro" component in mnt_param.

- For Darwin, which has a dedicated, separate gf_fuse_mount
  implementation, gf_fuse_mount was ignoring mountflags, so only the
  signature had to to be adjusted. However, as bonus, we gain read-only
  support for Darwin, which was missing so far, given that it was
  indicated via the ignored mountflags. Darwin's low level mount helper
  relies on the "ro" component of the option string, which agrees with
  the new calling convention of gf_fuse_mount.

- On Linux, system mount option name handling (apart from the
  distinguished read-only option) used to have the inadvertent side
  effect of adding "nosuid,nodev" as indicated in BUG; since
  Ia89d975d1e27fcfa5ab2036ba546aa8fa0d2d1b0 this side effect is removed,
  but system mount option name handling was left broken (passing system
  mount options other than "ro" fails to mount).

- On other platforms, system mount option name handling is broken
  (expect for the distinguished read-only option).

As of this change, in the general (non-Darwin) implementation of
gf_fuse_mount we take care of proper separation of system mount names
and their conversion to mount flags. For Linux, we adopt the conversion
table from FUSE upstream. For other systems we just provide a best
effort to support those system mount options which are understood across
all Unices (nosuid,nodev,noatime,noexec,ro). (This can be improved later
to provide proper plaform support.)

BUG: 1297182
Change-Id: I5d10b5df46feba7a02bf5bf1018db69e6b52260a
Signed-off-by: Csaba Henk 
Reviewed-on: https://review.gluster.org/16313
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Amar Tumballi 
Tested-by: Amar Tumballi