glusterfs.git/xlators/cluster/dht/src/dht-shared.c, branch v3.7.16

tier/dht : Default value for demote-freq, max files and mb

2016-02-14T14:04:21+00:00

Default value for tier-demote-frequency is 3600 sec to avoid
frequent demotions.

Default value for tier-max-mb is 4000 mb

Default value for tier-max-files is 10000 files

Backport of http://review.gluster.org/13270

> Change-Id: Ie60951c478a7462c425059699ab82511aa13fa0a
> BUG: 1300412
> Signed-off-by: Joseph Fernandes 
> Reviewed-on: http://review.gluster.org/13270
> Smoke: Gluster Build System 
> Tested-by: Dan Lambright 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Dan Lambright 
Signed-off-by: Joseph Fernandes 

Change-Id: Ic4458b1b78dc3c6059713f3871e9e2dc8f38b39a
BUG: 1306302
Reviewed-on: http://review.gluster.org/13424
Smoke: Gluster Build System 
Reviewed-by: Joseph Fernandes
Tested-by: Joseph Fernandes
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Dan Lambright

cluster/tier: fix loading tier.so into glusterd

2015-12-04T13:44:32+00:00

The glusterd process loads the shared libraries of client translators.
This failed for tiering due to a reference to dht_methods which is
defined as a global variable which is not necessary.
The global variable has been removed and this is now a member of
dht_conf and is now initialised in the *_init calls.

> Change-Id: Ifa0a21e3962b5cd8d9b927ef1d087d3b25312953
> Signed-off-by: N Balachandran 
> Reviewed-on: http://review.gluster.org/12863
> Tested-by: NetBSD Build System 
> Tested-by: Gluster Build System 
> Reviewed-by: Dan Lambright 
>Tested-by: Dan Lambright 
(cherry picked from commit 96fc7f64da2ef09e82845a7ab97574f511a9aae5)

Change-Id: If3cc908ebfcd1f165504f15db2e3079d97f3132e
BUG: 1288352
Signed-off-by: N Balachandran 
Reviewed-on: http://review.gluster.org/12877
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

cluster/tier: add pause tier for snapshots

2015-10-22T02:46:18+00:00

This is a backport of 12304

Snaps of tiered volumes cannot handle files undergoing migration.
We implement a helper mechanism to "pause" migration. Any files
undergoing migration are aborted. Clean up is done to remove
sticky bits and data at the destination. Migration is restarted
after snap completes.

For testing an internal switch is added. It is not exposed externally.

gluster volume set vol1 tier-pause [true|false]

> Change-Id: Ia85bbf89ac142e9b7e73fcbef98bb9da86097799
> BUG: 1267950
> Signed-off-by: Dan Lambright 
> Reviewed-on: http://review.gluster.org/12304
> Reviewed-by: N Balachandran 
> Tested-by: NetBSD Build System 
> Tested-by: Gluster Build System 
Signed-off-by: Dan Lambright 

Conflicts:
	xlators/mgmt/glusterd/src/glusterd-messages.h

Change-Id: I5f039d8d38a4c915bd873969f336b96755a0b8f1
BUG: 1274101
Reviewed-on: http://review.gluster.org/12411
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

cluster/tier: add watermarks and policy driver

2015-10-10T21:01:44+00:00

Backport fix 12039

This fix introduces infrastructure to support different
policies for promotion and demotion.

Currently the tier feature automatically promotes and demotes
files periodically based on access. This is good for testing
but too stringent for most real workloads. It makes it
difficult to fully utilize a hot tier- data will be demoted
before it is touched- its unlikely a 100GB hot SSD will have
all its data touched in a window of time.

A new parameter "mode" allows the user to pick promotion/demotion
polcies.

The "test mode" will be used for *.t and other general testing.
This is the current mechanism.

The "cache mode" introduces watermarks. The watermarks
represent levels of data residing on the hot tier.

"cache mode" policy:

The % the hot tier is full is called P.

Do not promote or demote more than D MB or F files.

A random number [0-100] is called R.

Rules for migration:

if (P < watermark_low) don't demote, always promote.

if (P >= watermark_low) && (P < watermark_hi) demote if R < P; promote if R > P.

if (P > watermark_hi) always demote, don't promote.

gluster volume set {vol} cluster.watermark-hi %
gluster volume set {vol} cluster.watermark-low %
gluster volume set {vol} cluster.tier-max-mb {D}
gluster volume set {vol} cluster.tier-max-files {F}
gluster volume set {vol} cluster.tier-mode {test|cache}

> Change-Id: I157f19667ec95aa1d53406041c1e3b073be127c2
> BUG: 1257911
> Signed-off-by: Dan Lambright 
> Reviewed-on: http://review.gluster.org/12039
> Tested-by: Gluster Build System 
> Reviewed-by: Atin Mukherjee 
Signed-off-by: Dan Lambright 

Signed-off-by: Dan Lambright 

Conflicts:
	xlators/cluster/dht/src/dht-rebalance.c
	xlators/cluster/dht/src/tier.c

Change-Id: Ibfe6b89563ceab98708325cf5d5ab0997c64816c
BUG: 1270527
Reviewed-on: http://review.gluster.org/12330
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

glusterd, dht: volume set for use-readdirp in dht

2015-10-05T06:49:38+00:00

 >Change-Id: Icab246b1d02808864d878d949fa56f9f889b538a
 >BUG: 1265677
 >Signed-off-by: Pranith Kumar K 
 >Reviewed-on: http://review.gluster.org/12221
 >Tested-by: NetBSD Build System 
 >Tested-by: Gluster Build System 
 >Reviewed-by: Raghavendra G 
 >Reviewed-by: Raghavendra Bhat 
 >Reviewed-by: Kaushal M 
 >(cherry picked from commit 059db0254f5670a34f1a928155c0c7d1cd03b53a)

Change-Id: Ifc46ed08fc10b32f5e814aa09c155e11e8c93138
BUG: 1267822
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/12269
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Raghavendra G

dht: Adding log messages to the new logging framework

2015-07-27T07:21:48+00:00

        Backported from: http://review.gluster.org/10021

> Change-Id: Ib3bb61c5223f409c23c68100f3fe884918d2dc3f
> BUG: 1194640
> Reviewed-on: http://review.gluster.org/10021
> Reviewed-by: N Balachandran 
> Reviewed-by: Joseph Fernandes 
> Tested-by: Joseph Fernandes 
> Reviewed-by: Dan Lambright 
> Reviewed-by: Raghavendra G 
> Tested-by: Raghavendra G 
> Signed-off-by: arao 

BUG: 1217722
Change-Id: Ide79c6c1e6a466fb52f955c90a2b22711bec794a
Signed-off-by: arao 
Signed-off-by: Anusha Rao 
Reviewed-on: http://review.gluster.org/11350
Tested-by: NetBSD Build System 
Reviewed-by: Raghavendra G 
Tested-by: Gluster Build System

dht: Add lookup-optimize configuration option for DHT

2015-06-05T09:16:13+00:00

Currently with commit 4eaaf5 a mixed version cluster would
have issues if lookup-uhashed is set to auto, as older clients
would fail to validate the layouts if newer clients (i.e 3.7 or
upwards) create directories. Also, in a mixed version cluster
rebalance daemon would set commit hash for some subvolumes and
not for the others.

This commit fixes this problem by moving the enabling of the
functionality introduced in the above mentioned commit to a
new dht option. This option also has a op_version of 3_7_1
thereby preventing it from being set in a mixed version
cluster. It brings in the following changes,
- Option can be set only if min version of the cluster is
3.7.1 or more
- Rebalance and mkdir update the layout with the commit hashes
only if this option is set, hence ensuring rebalance works in a
mixed version cluster, and also directories created by newer
clients do not cause layout errors when read by older clients
- This option also supersedes lookup-unhased, to enable the
optimization for lookups more deterministic and not conflict
with lookup-unhashed settings.

Option added is cluster.lookup-optimize, which is a boolean.

Usage: # gluster volume set VOLNAME cluster.lookup-optimize on

Change-Id: Ifd1d4ce3f6438fcbcd60ffbfdbfb647355ea1ae0
BUG:  1225940
Signed-off-by: Shyam 
Reviewed-on: http://review.gluster.org/10976
Tested-by: Gluster Build System 
Reviewed-by: N Balachandran 
Reviewed-by: Raghavendra G 
Tested-by: Raghavendra G

dht: make lookup-unhashed=auto do something actually useful

2015-05-10T04:55:09+00:00

The key concept here is to determine whether a directory is "clean" by
comparing its last-known-good topology to the current one for the
volume.  These are stored as "commit hashes" on the directory and the
volume root respectively.  The volume's commit hash changes whenever a
brick is added or removed, and a fix-layout is done.  A directory's
commit hash changes only when a full rebalance (not just fix-layout)
is done on it.  If all bricks are present and have a directory
commit hash that matches the volume commit hash, then we can assume
that every file is in its "proper" place. Therefore, if we look for
a file in that proper place and don't find it, we can assume it's not
on any other subvolume and *safely* skip the global (broadcast to all)
lookup.

Change-Id: Id6ce4593ba1f7daffa74cfab591cb45960629ae3
BUG: 1220064
Reviewed-on-master: http://review.gluster.org/#/c/7702/
Signed-off-by: Jeff Darcy 
Signed-off-by: Shyam 
Reviewed-on: http://review.gluster.org/10729
Tested-by: Gluster Build System 
Reviewed-by: Krishnan Parthasarathi 
Reviewed-by: Vijay Bellur

Restore build on non Linux systems

2015-05-07T18:52:10+00:00

This change broke the build on NetBSD, FreeBSD, and MacOS X:
http://review.gluster.org/10526/

We restore the build with two fixes:
- Use POSIX-compliant sysconf(_SC_NPROCESSORS_ONLN) to get the
  number of processors, instead of Linux specific get_nprocs().
  That let us remove Linux-specific #include 
- Only define MAX() if it is not already defined. NetBSD defines
  it in  which is already included

Backport of: I62341c670598670e47ea2f69ab94864f96588b18
BUG: 1212676
Signed-off-by: Emmanuel Dreyfus 

Change-Id: I0f098153e76954bb85b5dca3f054a069e31dd94c
Reviewed-on: http://review.gluster.org/10653
Reviewed-by: Shyamsundar Ranganathan 
Reviewed-by: Kaleb KEITHLEY 
Tested-by: Vijay Bellur

dht/rebalance: Throttle rebalance

2015-05-07T17:26:54+00:00

Throttle value will be "normal" by default. For throttling down,
a thread will be put in to sleep. And for throttling up,
gf_defrag_process_dir will wake up the sleeping threads.

Change-Id: I4892ab14982a1ff305aeb2d8bbd33c79d6877b69
BUG: 1219579
Signed-off-by: Susant Palai 
Reviewed-on: http://review.gluster.org/10526
Reviewed-by: Raghavendra G 
Tested-by: Raghavendra G 
Reviewed-on: http://review.gluster.org/10629
Reviewed-by: Vijay Bellur 
Tested-by: Vijay Bellur