glusterfs.git/tests/basic/tier, branch v3.7.6

cluster/tier correct promotion cycle calculation

2015-11-07T21:44:38+00:00

This is a backport of 12480

The tier translator should only choose candidate files for promotion
from the most recent cycle, not a multiple of the most recent cycles.
Otherwise user observed behavior can be inconsistent. Remove related
test in tier.t that is subject to race condition.

> Change-Id: I9ad1523cac00f904097ce468efa6ddd515857024
> BUG: 1275524
> Signed-off-by: root 
> Signed-off-by: Dan Lambright 
> Reviewed-on: http://review.gluster.org/12480
> Reviewed-by: Joseph Fernandes
> Tested-by: Gluster Build System 
Signed-off-by: Dan Lambright 

Signed-off-by: Dan Lambright 

Conflicts:
	tests/basic/tier/tier.t
	xlators/cluster/dht/src/tier.c

Change-Id: Ic4587bf1b5d26ba377a12a4ce8e329362988a33b
BUG: 1275483
Reviewed-on: http://review.gluster.org/12536
Tested-by: NetBSD Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

cluster/tier : Files skipped during tier query parsing

2015-11-04T11:55:08+00:00

The tier query parsing code was using fscanf to read each record.
As space is a delimiter for fscanf, filenames containing spaces
caused the parsing to return unexpected values causing various
issues in the tier process, including crashes due to buffer
 overflows.

> Change-Id: Ife602cb7ecb158fccbc2c89e4d2959bd97098a87
> Signed-off-by: N Balachandran 
> Reviewed-on: http://review.gluster.org/12469
> Tested-by: NetBSD Build System 
> Tested-by: Gluster Build System 
> Reviewed-by: Dan Lambright 
> Tested-by: Dan Lambright 
(cherry picked from commit 499b43058049572e33b525ac669ef623d476fe41)

Change-Id: Id60f9c484dfbb02de6ebb44032160ad4cc94cb7f
BUG: 1277587
Signed-off-by: N Balachandran 
Reviewed-on: http://review.gluster.org/12502
Tested-by: NetBSD Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

tests/tier: Move common functions to tier.rc

2015-11-03T18:44:29+00:00

Move common functions in tier .t files to tier.rc

> Change-Id: Ibc312d987be9d93e7cc7fc47d0bf598bb1c944c2
> Signed-off-by: N Balachandran 
> Reviewed-on: http://review.gluster.org/12404
> Tested-by: Gluster Build System 
> Reviewed-by: Dan Lambright 
> Tested-by: Dan Lambright 
(cherry picked from commit 594a03b030577bf0ed6960199e920cc5fa7e7afc)

Change-Id: Ie13bb86f85575164e62120e29bbbedb7789b0af5
BUG: 1277590
Signed-off-by: N Balachandran 
Reviewed-on: http://review.gluster.org/12497
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

cluster/tier enable CTR on attach tier

2015-10-31T03:20:45+00:00

CTR is currently disabled by default, and must be manually enabled
for tiering to start. This is an overhead on the administrator and
easy to overlook. Enable it automatically when a tier is attached.

This is a backport of 12420

> Change-Id: I0c29de8762faec1bfe6d1376a57eeef3357ad15a
> BUG: 1274847
> Signed-off-by: Dan Lambright 
> Reviewed-on: http://review.gluster.org/12420
> Tested-by: Gluster Build System 
> Reviewed-by: mohammed rafi  kc 
Signed-off-by: Dan Lambright 

Change-Id: I43a32ac0c88b44d10aa0f40b18b0564ae1e17321
BUG: 1276671
Reviewed-on: http://review.gluster.org/12474
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

cluster/tier: add pause tier for snapshots

2015-10-22T02:46:18+00:00

This is a backport of 12304

Snaps of tiered volumes cannot handle files undergoing migration.
We implement a helper mechanism to "pause" migration. Any files
undergoing migration are aborted. Clean up is done to remove
sticky bits and data at the destination. Migration is restarted
after snap completes.

For testing an internal switch is added. It is not exposed externally.

gluster volume set vol1 tier-pause [true|false]

> Change-Id: Ia85bbf89ac142e9b7e73fcbef98bb9da86097799
> BUG: 1267950
> Signed-off-by: Dan Lambright 
> Reviewed-on: http://review.gluster.org/12304
> Reviewed-by: N Balachandran 
> Tested-by: NetBSD Build System 
> Tested-by: Gluster Build System 
Signed-off-by: Dan Lambright 

Conflicts:
	xlators/mgmt/glusterd/src/glusterd-messages.h

Change-Id: I5f039d8d38a4c915bd873969f336b96755a0b8f1
BUG: 1274101
Reviewed-on: http://review.gluster.org/12411
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

cluster/dht : Do not migrate files with POSIX locks held

2015-10-18T14:23:27+00:00

dht_migrate_file does not migrate file locks to the dst file.
Any locks held on the source file are lost once the migration
is complete. This issue is magnified in the case of a tier volume
as file migrations occur more frequently and repeatedly as compared
to a DHT rebalance.

The fix makes 2 changes:
1. Before starting the actual migration process, check if there are
 any locks held on the file. If yes, do not migrate the file.
2. The rebalance process tries to lock on the entire file just before
 moving into the Phase 2 of the file migration. If the lock acquisition
fails, the file migration does not proceed.
If the lock is granted, the file migration proceeds.

This still leaves a small window where conflicting locks can be granted to
different clients. If client1 requests a lock on the src file just after
it is converted to a linkto file and client2 requests a lock on the dst
data file, they will both be granted, but all FOPs will be redirected
to the dst data file. This issue will be taken up in a subsequent patch.

Change-Id: I8c895fc3cced50dd2894259d40a827c7b43d58ac
BUG: 1272331
Signed-off-by: N Balachandran 
> Reviewed-on: http://review.gluster.org/12347
> Tested-by: NetBSD Build System 
> Tested-by: Gluster Build System 
> Reviewed-by: Dan Lambright 
> Tested-by: Dan Lambright 
> Signed-off-by: N Balachandran 
Reviewed-on: http://review.gluster.org/12369
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

tier/shd: make shd commands compatible with tiering

2015-10-14T02:47:39+00:00

tiering volfiles may contain afr and disperse together
or multiple time based on configuration. And the
informations for those configurations are stored in
tier_info. So most of the volgen code generation
need to be changed to make compatible with it.

Back port of>
>Change-Id: I563d1ca6f281f59090ebd470b7fda1cc4b1b7e1d
>BUG: 1261276
>Signed-off-by: Mohammed Rafi KC 
>Reviewed-on: http://review.gluster.org/12135
>Tested-by: NetBSD Build System 
>Tested-by: Gluster Build System 
>Reviewed-by: Pranith Kumar Karampuri 

(cherry picked from commit 0ef62933649392051e73fe01c028e41baddec489)

BUG: 1261744
Change-Id: Iff1b27ae8ce61f1f38fbbd6c92894b3d3516e4d4
Signed-off-by: Mohammed Rafi KC 
Reviewed-on: http://review.gluster.org/12344
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Joseph Fernandes
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

tier/ctr: CTR DB named lookup heal of cold tier during attach tier

2015-10-11T01:10:18+00:00

Heal hardlink in the db for already existing data in the cold
tier during attach tier. i.e during fix layout do lookup to files
in the cold tier.

CTR xlator on the  brick/server side does db update/insert of the hardlink on a namelookup.
Currently the namedlookup is done synchronous to the fixlayout that is
triggered by attach tier. This is not performant, adding more time to
fixlayout. The performant approach is record the hardlinks on a compressed
datastore and then do the namelookup asynchronously later, giving the ctr db
eventual consistency

master patch : http://review.gluster.org/#/c/11828/

>>Change-Id: I4ffc337fffe7d447804786851a9183a51b5044a9
>>BUG: 1252586
>>Signed-off-by: Joseph Fernandes 
>>Reviewed-on: http://review.gluster.org/11828
>>Tested-by: Gluster Build System 
>>Reviewed-by: Dan Lambright 
>>Tested-by: Dan Lambright 
Signed-off-by: Joseph Fernandes 

Change-Id: I61b185a54ae4e8c1d82804b95a278bfbea870987
BUG: 1261146
Reviewed-on: http://review.gluster.org/12331
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

cluster/tier: add watermarks and policy driver

2015-10-10T21:01:44+00:00

Backport fix 12039

This fix introduces infrastructure to support different
policies for promotion and demotion.

Currently the tier feature automatically promotes and demotes
files periodically based on access. This is good for testing
but too stringent for most real workloads. It makes it
difficult to fully utilize a hot tier- data will be demoted
before it is touched- its unlikely a 100GB hot SSD will have
all its data touched in a window of time.

A new parameter "mode" allows the user to pick promotion/demotion
polcies.

The "test mode" will be used for *.t and other general testing.
This is the current mechanism.

The "cache mode" introduces watermarks. The watermarks
represent levels of data residing on the hot tier.

"cache mode" policy:

The % the hot tier is full is called P.

Do not promote or demote more than D MB or F files.

A random number [0-100] is called R.

Rules for migration:

if (P < watermark_low) don't demote, always promote.

if (P >= watermark_low) && (P < watermark_hi) demote if R < P; promote if R > P.

if (P > watermark_hi) always demote, don't promote.

gluster volume set {vol} cluster.watermark-hi %
gluster volume set {vol} cluster.watermark-low %
gluster volume set {vol} cluster.tier-max-mb {D}
gluster volume set {vol} cluster.tier-max-files {F}
gluster volume set {vol} cluster.tier-mode {test|cache}

> Change-Id: I157f19667ec95aa1d53406041c1e3b073be127c2
> BUG: 1257911
> Signed-off-by: Dan Lambright 
> Reviewed-on: http://review.gluster.org/12039
> Tested-by: Gluster Build System 
> Reviewed-by: Atin Mukherjee 
Signed-off-by: Dan Lambright 

Signed-off-by: Dan Lambright 

Conflicts:
	xlators/cluster/dht/src/dht-rebalance.c
	xlators/cluster/dht/src/tier.c

Change-Id: Ibfe6b89563ceab98708325cf5d5ab0997c64816c
BUG: 1270527
Reviewed-on: http://review.gluster.org/12330
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

cluster/tier re-enable tier.t in automatic tests

2015-09-30T02:48:22+00:00

This is a backport of 12208

> Re-enable tier.t in automatic tests. Disable check
> for BSD until recurring problem with SQLlite on it is understood.

> Change-Id: Ib13b269ab841a59a0a41d8478c8627b180b16c61
> BUG: 1231268
> Signed-off-by: Dan Lambright 

Change-Id: Id462e9f67cc90c79aa7a230f2baae87c0746ff6f
BUG: 1262335
Signed-off-by: Dan Lambright 
Reviewed-on: http://review.gluster.org/12253
Tested-by: Gluster Build System 
Reviewed-by: mohammed rafi  kc