glusterfs.git/tests/basic, branch v3.10.1

glusterd: disallow increasing replica count for arbiter volumes

2017-03-07T13:54:18+00:00

Problem: add-brick command to increase replica count in an arbiter
volume succeeds, causing undesirable effects like the 4th brick being
loaded with the arbiter xlator, the 3rd one losing the arbiter xlator
(when the brick process is restarted), arbitration logic in afr going
for a toss etc.

Fix: Arbiter configuration should always be a replica 3 volume (of
which 3rd brick is arbiter). Hence disallow increasing replica count for
arbiter volume configurations.

> Reviewed-on: https://review.gluster.org/16845
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Atin Mukherjee 
(cherry picked from commit b7ba77ab3ffb641d06223f7af5145d3d670b032a)

Change-Id: I9fe4edac880d0f711e6d44324ad5562974e53e51
BUG: 1429773
Signed-off-by: Ravishankar N 
Reviewed-on: https://review.gluster.org/16863
Smoke: Gluster Build System 
Reviewed-by: Atin Mukherjee 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

cluster/ec: Don't trigger data/metadata heal on Lookups

2017-02-27T15:34:08+00:00

Problem-1
If Lookup which doesn't take any locks observes version mismatch it can't be
trusted. If we launch a heal based on this information it will lead to
self-heals which will affect I/O performance in the cases where Lookup is
wrong. Considering self-heal-daemon and operations on the inode from client
which take locks can still trigger heal we can choose to not attempt a heal on
Lookup.

Problem-2:
Fixed spurious failure of
tests/bitrot/bug-1373520.t
For the issues above, what was happening was that ec_heal_inspect()
is preventing 'name' heal to happen

Problem-3:
tests/basic/ec/ec-background-heals.t
To be honest I don't know what the problem was, while fixing
the 2 problems above, I made some changes to ec_heal_inspect() and
ec_need_heal() after which when I tried to recreate the spurious
failure it just didn't happen even after a long time.

 >BUG: 1414287
 >Signed-off-by: Pranith Kumar K 
 >Change-Id: Ife2535e1d0b267712973673f6d474e288f3c6834
 >Reviewed-on: https://review.gluster.org/16468
 >Smoke: Gluster Build System 
 >NetBSD-regression: NetBSD Build System 
 >Reviewed-by: Xavier Hernandez 
 >CentOS-regression: Gluster Build System 
 >Reviewed-by: Ashish Pandey 

BUG: 1419824
Change-Id: I340b48cd416b07890bf3a5427562f5e3f88a481f
Signed-off-by: Pranith Kumar K 
Reviewed-on: https://review.gluster.org/16765
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Xavier Hernandez 
Smoke: Gluster Build System

gfapi: add API to trigger events for debugging and troubleshooting

2017-02-14T01:53:23+00:00

Introduce glfs_sysrq() as a generic API for triggering debug and
troubleshoot events. This interface will be used by the feature to get
statedumps for applications using libgfapi.

The current events that can be requested through this API are:
 - 'h'elp: log a mesage with all supported events
 - 's'tatedump: trigger a statedump for the passed glfs_t

In future, this API can be used by a CLI to trigger statedumps from
storage servers. At the moment it is limited to take statedumps, but it
is extensible to set the log-level, clear caches, force reconnects and
much more.

> BUG: 1169302
> Change-Id: I18858359a3957870cea5139c79efe1365a15a992
> Original-author: Poornima G 
> Signed-off-by: Niels de Vos 
> Reviewed-on-master: http://review.gluster.org/16414
> Reviewed-by: Prashanth Pai 
> Reviewed-by: Kaleb KEITHLEY 

BUG: 1418981
Change-Id: I18858359a3957870cea5139c79efe1365a15a992
Signed-off-by: Shyam 
Reviewed-on: https://review.gluster.org/16600
Smoke: Gluster Build System 
Reviewed-by: Niels de Vos 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

trash: fix problem with trash feature under multiplexing

2017-02-10T11:44:01+00:00

With multiplexing, the trash translator gets a reconfigure call before
a notify(CHILD_UP).  In this case, priv->trash_itable was not yet
initialized, so the reconfigure would get a SEGV.  Moving the itable
allocation to init seems to fix it, so trash can be reenabled.

Backport of:
> Change-Id: I21ac2d7fc66bac1bc4ec70fbc8bae306d73ac565
> BUG: 1420434
> Reviewed-on: https://review.gluster.org/16567

Change-Id: I43a6de6ac5070848619c5f905f075e4a4099c1bd
BUG: 1420808
Signed-off-by: Jeff Darcy 
Reviewed-on: https://review.gluster.org/16582
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Anoop C S 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

tests: fix online_brick_count for multiplexing

2017-02-08T20:03:37+00:00

The number of brick processes no longer matches the number of bricks,
therefore counting processes doesn't work.  Counting *pidfiles* does.
Ironically, the fix broke multiplex.t which used this function, so it
now uses a different function with the old process-counting behavior.
Also had to fix online_brick_count and kill_node in cluster.rc to be
consistent with the new reality.

Backport of:
> Change-Id: I4e81a6633b93227e10604f53e18a0b802c75cbcc
> BUG: 1385758
> Reviewed-on: https://review.gluster.org/16527

Change-Id: I70b5cd169eafe3ad5b523bc0a30d21d864b3036a
BUG: 1418091
Signed-off-by: Jeff Darcy 
Reviewed-on: https://review.gluster.org/16565
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

CLI/TIER: removing old tier commands under rebalance

2017-02-07T14:28:39+00:00

        back-port of : https://review.gluster.org/#/c/16463/

PROBLEM: gluster v rebalance  tier start works even after
the switch of tier to service framework.
This lets the user have two tierd for the same volume.

FIX: checking for each process will make the new code hard
to maintain. So we are removing the support for old commands.

>Change-Id: I5b0974b2dbb74f0bee8344b61c7f924300ad73f2
>BUG: 1415590
>Signed-off-by: hari gowtham 
>Reviewed-on: https://review.gluster.org/16463
>Smoke: Gluster Build System 
>Tested-by: hari gowtham 
>NetBSD-regression: NetBSD Build System 
>CentOS-regression: Gluster Build System 
>Reviewed-by: N Balachandran 
>Reviewed-by: Atin Mukherjee 

Change-Id: Ib996d89b1bd250176a3f5eeb369b71b0a4f95968
BUG: 1419868
Signed-off-by: hari gowtham 
Reviewed-on: https://review.gluster.org/16555
Smoke: Gluster Build System 
Tested-by: hari gowtham 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Atin Mukherjee 
CentOS-regression: Gluster Build System

tests: Marked 2 tests in master that bad, the same in release-3.10

2017-02-07T13:30:05+00:00

The 2 tests that were marked bad in master are,
tests/basic/ec/ec-background-heals.t
tests/bitrot/bug-1373520.t

The same are marked bad in release-3.10 to prevent any spurious
failures during backporting other commits.

When the failures are analyzed and fixed in master the same would
need to be backported, as applicable, to release-3.10 branch.

Change-Id: I7d686bc90e1ee443475d3a3512fa7dfb7cbeede7
BUG: 1419696
Signed-off-by: Shyam 
Reviewed-on: https://review.gluster.org/16549
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Atin Mukherjee

extras: Provide group set for md-cache and invalidation options

2017-02-06T15:38:07+00:00

To enable the integration of md-cache and invalidation features
we need to perform 3 volume set options in a specific order.
In order to ease this for user provide a group volume set option.

Usage: gluster vol set  group metadata-cache

>Reviewed-on: https://review.gluster.org/16503
>Smoke: Gluster Build System 
>NetBSD-regression: NetBSD Build System 
>CentOS-regression: Gluster Build System 
>Reviewed-by: Rajesh Joseph 
>Reviewed-by: Atin Mukherjee 

Change-Id: I9bf0fd4217aa2a1c7ffbdc93e879b10f87addeac
BUG: 1419306
Signed-off-by: Poornima G 
Reviewed-on: https://review.gluster.org/16546
Tested-by: Atin Mukherjee 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

libglusterfs: make memory pools more thread-friendly

2017-02-03T00:44:09+00:00

Early multiplexing tests revealed *massive* contention on certain
pools' global locks - especially for dictionaries and secondarily for
call stubs.  For the thread counts that multiplexing can create, a
more lock-free solution is clearly needed.  Also, the current mem-pool
implementation does a poor job releasing memory back to the system,
artificially inflating memory usage to match whatever the worst case
was since the process started.  This is bad in general, but especially
so for multiplexing where there are more pools and a major point of
the whole exercise is to reduce memory consumption.

The basic ideas for the new design are these

  There is one pool, globally, for each power-of-two size range.
  Every attempt to create a new pool within this range will instead
  add a reference to the existing pool.

  Instead of adding pools for each translator within each multiplexed
  brick (potentially infinite and quite possibly thousands), we
  allocate one set of size-based pools per *thread* (hundreds at
  worst).

  Each per-thread pool is divided into hot and cold lists.  Every
  allocation first attempts to use the hot list, then the cold list.
  When objects are freed, they always go on the hot list.

  There is one global "pool sweeper" thread, which periodically
  reclaims everything in each pool's cold list and then "demotes" the
  current hot list to be the new cold list.

  For normal allocation activity, only a per-thread lock need be
  taken, and even that only to guard against very rare contention from
  the pool sweeper.  When threads start and stop, a global lock must
  be taken to add them to the pool sweeper's list.  Lock contention is
  therefore extremely low, and the hot/cold lists also provide good
  locality.

A more complete explanation (of a similar earlier design) can be found
here:

 http://www.gluster.org/pipermail/gluster-devel/2016-October/051160.html

Backport of:
> Change-Id: I5bc8a1ba57cfb553998f979a498886e0d006e665
> BUG: 1385758
> Reviewed-on: https://review.gluster.org/15645

BUG: 1418091
Change-Id: Id09bbea41f65fcd245822607bc204f3a34904dc2
Signed-off-by: Jeff Darcy 
Reviewed-on: https://review.gluster.org/16531
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

core: run many bricks within one glusterfsd process

2017-02-02T00:54:58+00:00

This patch adds support for multiple brick translator stacks running in
a single brick server process.  This reduces our per-brick memory usage
by approximately 3x, and our appetite for TCP ports even more.  It also
creates potential to avoid process/thread thrashing, and to improve QoS
by scheduling more carefully across the bricks, but realizing that
potential will require further work.

Multiplexing is controlled by the "cluster.brick-multiplex" global
option.  By default it's off, and bricks are started in separate
processes as before.  If multiplexing is enabled, then *compatible*
bricks (mostly those with the same transport options) will be started in
the same process.

Backport of:
> Change-Id: I45059454e51d6f4cbb29a4953359c09a408695cb
> BUG: 1385758
> Reviewed-on: https://review.gluster.org/14763

Change-Id: I4bce9080f6c93d50171823298fdf920258317ee8
BUG: 1418091
Signed-off-by: Jeff Darcy 
Reviewed-on: https://review.gluster.org/16496
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan