glusterfs.git/tests/basic, branch v3.10.0rc1

gfapi: add API to trigger events for debugging and troubleshooting

2017-02-14T01:53:23+00:00

Introduce glfs_sysrq() as a generic API for triggering debug and
troubleshoot events. This interface will be used by the feature to get
statedumps for applications using libgfapi.

The current events that can be requested through this API are:
 - 'h'elp: log a mesage with all supported events
 - 's'tatedump: trigger a statedump for the passed glfs_t

In future, this API can be used by a CLI to trigger statedumps from
storage servers. At the moment it is limited to take statedumps, but it
is extensible to set the log-level, clear caches, force reconnects and
much more.

> BUG: 1169302
> Change-Id: I18858359a3957870cea5139c79efe1365a15a992
> Original-author: Poornima G 
> Signed-off-by: Niels de Vos 
> Reviewed-on-master: http://review.gluster.org/16414
> Reviewed-by: Prashanth Pai 
> Reviewed-by: Kaleb KEITHLEY 

BUG: 1418981
Change-Id: I18858359a3957870cea5139c79efe1365a15a992
Signed-off-by: Shyam 
Reviewed-on: https://review.gluster.org/16600
Smoke: Gluster Build System 
Reviewed-by: Niels de Vos 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

trash: fix problem with trash feature under multiplexing

2017-02-10T11:44:01+00:00

With multiplexing, the trash translator gets a reconfigure call before
a notify(CHILD_UP).  In this case, priv->trash_itable was not yet
initialized, so the reconfigure would get a SEGV.  Moving the itable
allocation to init seems to fix it, so trash can be reenabled.

Backport of:
> Change-Id: I21ac2d7fc66bac1bc4ec70fbc8bae306d73ac565
> BUG: 1420434
> Reviewed-on: https://review.gluster.org/16567

Change-Id: I43a6de6ac5070848619c5f905f075e4a4099c1bd
BUG: 1420808
Signed-off-by: Jeff Darcy 
Reviewed-on: https://review.gluster.org/16582
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Anoop C S 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

tests: fix online_brick_count for multiplexing

2017-02-08T20:03:37+00:00

The number of brick processes no longer matches the number of bricks,
therefore counting processes doesn't work.  Counting *pidfiles* does.
Ironically, the fix broke multiplex.t which used this function, so it
now uses a different function with the old process-counting behavior.
Also had to fix online_brick_count and kill_node in cluster.rc to be
consistent with the new reality.

Backport of:
> Change-Id: I4e81a6633b93227e10604f53e18a0b802c75cbcc
> BUG: 1385758
> Reviewed-on: https://review.gluster.org/16527

Change-Id: I70b5cd169eafe3ad5b523bc0a30d21d864b3036a
BUG: 1418091
Signed-off-by: Jeff Darcy 
Reviewed-on: https://review.gluster.org/16565
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

CLI/TIER: removing old tier commands under rebalance

2017-02-07T14:28:39+00:00

        back-port of : https://review.gluster.org/#/c/16463/

PROBLEM: gluster v rebalance  tier start works even after
the switch of tier to service framework.
This lets the user have two tierd for the same volume.

FIX: checking for each process will make the new code hard
to maintain. So we are removing the support for old commands.

>Change-Id: I5b0974b2dbb74f0bee8344b61c7f924300ad73f2
>BUG: 1415590
>Signed-off-by: hari gowtham 
>Reviewed-on: https://review.gluster.org/16463
>Smoke: Gluster Build System 
>Tested-by: hari gowtham 
>NetBSD-regression: NetBSD Build System 
>CentOS-regression: Gluster Build System 
>Reviewed-by: N Balachandran 
>Reviewed-by: Atin Mukherjee 

Change-Id: Ib996d89b1bd250176a3f5eeb369b71b0a4f95968
BUG: 1419868
Signed-off-by: hari gowtham 
Reviewed-on: https://review.gluster.org/16555
Smoke: Gluster Build System 
Tested-by: hari gowtham 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Atin Mukherjee 
CentOS-regression: Gluster Build System

tests: Marked 2 tests in master that bad, the same in release-3.10

2017-02-07T13:30:05+00:00

The 2 tests that were marked bad in master are,
tests/basic/ec/ec-background-heals.t
tests/bitrot/bug-1373520.t

The same are marked bad in release-3.10 to prevent any spurious
failures during backporting other commits.

When the failures are analyzed and fixed in master the same would
need to be backported, as applicable, to release-3.10 branch.

Change-Id: I7d686bc90e1ee443475d3a3512fa7dfb7cbeede7
BUG: 1419696
Signed-off-by: Shyam 
Reviewed-on: https://review.gluster.org/16549
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Atin Mukherjee

extras: Provide group set for md-cache and invalidation options

2017-02-06T15:38:07+00:00

To enable the integration of md-cache and invalidation features
we need to perform 3 volume set options in a specific order.
In order to ease this for user provide a group volume set option.

Usage: gluster vol set  group metadata-cache

>Reviewed-on: https://review.gluster.org/16503
>Smoke: Gluster Build System 
>NetBSD-regression: NetBSD Build System 
>CentOS-regression: Gluster Build System 
>Reviewed-by: Rajesh Joseph 
>Reviewed-by: Atin Mukherjee 

Change-Id: I9bf0fd4217aa2a1c7ffbdc93e879b10f87addeac
BUG: 1419306
Signed-off-by: Poornima G 
Reviewed-on: https://review.gluster.org/16546
Tested-by: Atin Mukherjee 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

libglusterfs: make memory pools more thread-friendly

2017-02-03T00:44:09+00:00

Early multiplexing tests revealed *massive* contention on certain
pools' global locks - especially for dictionaries and secondarily for
call stubs.  For the thread counts that multiplexing can create, a
more lock-free solution is clearly needed.  Also, the current mem-pool
implementation does a poor job releasing memory back to the system,
artificially inflating memory usage to match whatever the worst case
was since the process started.  This is bad in general, but especially
so for multiplexing where there are more pools and a major point of
the whole exercise is to reduce memory consumption.

The basic ideas for the new design are these

  There is one pool, globally, for each power-of-two size range.
  Every attempt to create a new pool within this range will instead
  add a reference to the existing pool.

  Instead of adding pools for each translator within each multiplexed
  brick (potentially infinite and quite possibly thousands), we
  allocate one set of size-based pools per *thread* (hundreds at
  worst).

  Each per-thread pool is divided into hot and cold lists.  Every
  allocation first attempts to use the hot list, then the cold list.
  When objects are freed, they always go on the hot list.

  There is one global "pool sweeper" thread, which periodically
  reclaims everything in each pool's cold list and then "demotes" the
  current hot list to be the new cold list.

  For normal allocation activity, only a per-thread lock need be
  taken, and even that only to guard against very rare contention from
  the pool sweeper.  When threads start and stop, a global lock must
  be taken to add them to the pool sweeper's list.  Lock contention is
  therefore extremely low, and the hot/cold lists also provide good
  locality.

A more complete explanation (of a similar earlier design) can be found
here:

 http://www.gluster.org/pipermail/gluster-devel/2016-October/051160.html

Backport of:
> Change-Id: I5bc8a1ba57cfb553998f979a498886e0d006e665
> BUG: 1385758
> Reviewed-on: https://review.gluster.org/15645

BUG: 1418091
Change-Id: Id09bbea41f65fcd245822607bc204f3a34904dc2
Signed-off-by: Jeff Darcy 
Reviewed-on: https://review.gluster.org/16531
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

core: run many bricks within one glusterfsd process

2017-02-02T00:54:58+00:00

This patch adds support for multiple brick translator stacks running in
a single brick server process.  This reduces our per-brick memory usage
by approximately 3x, and our appetite for TCP ports even more.  It also
creates potential to avoid process/thread thrashing, and to improve QoS
by scheduling more carefully across the bricks, but realizing that
potential will require further work.

Multiplexing is controlled by the "cluster.brick-multiplex" global
option.  By default it's off, and bricks are started in separate
processes as before.  If multiplexing is enabled, then *compatible*
bricks (mostly those with the same transport options) will be started in
the same process.

Backport of:
> Change-Id: I45059454e51d6f4cbb29a4953359c09a408695cb
> BUG: 1385758
> Reviewed-on: https://review.gluster.org/14763

Change-Id: I4bce9080f6c93d50171823298fdf920258317ee8
BUG: 1418091
Signed-off-by: Jeff Darcy 
Reviewed-on: https://review.gluster.org/16496
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

md-cache: Cache security.ima xattrs

2017-01-30T14:25:03+00:00

Backport of http://review.gluster.org/16296

From kernel version 3.X or greater, creating of a file
results in removexattr call on security.ima xattr. But
this xattr is not set on the file unless IMA feature
is active. With this patch, removxattr call returns
ENODATA if it is not found in the cache.

> Change-Id: I8136096598a983aebc09901945eba1db1b2f93c9
> Signed-off-by: Poornima G 
> Reviewed-on: http://review.gluster.org/16296
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Raghavendra G 
> (cherry picked from commit ac629e574935a8aed6526936bc83b1c6d295ae67)

Change-Id: I27abc23024c8fcf07389608df61ef6e64736d414
BUG: 1415918
Signed-off-by: Poornima G 
Reviewed-on: https://review.gluster.org/16460
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G

core: remove experimental xlators and associated tests

2017-01-24T13:25:58+00:00

experimental xlators not included in 3.10

Change-Id: I547480ee5e7912664784643e436feb198b6d16d0
BUG: 1415866
Signed-off-by: Kaleb S. KEITHLEY 
Reviewed-on: https://review.gluster.org/16447
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Atin Mukherjee 
Reviewed-by: Shyamsundar Ranganathan