glusterfs.git/xlators/cluster/dht/src/dht-rebalance.c, branch v3.8.6

dht: Proper log message if data migration is skipped

2016-11-03T05:33:02+00:00

There was a misleading message from logs about 
available disk space while rebalancing of bricks
while calculating free space.

Bug: 1390870

Backprt of http://review.gluster.org/#/c/15345/

Change-Id: Ie9df0b2cbf00faaf13a0a3f0dbd657377a082755
Signed-off-by: ankitraj 
Reviewed-on: http://review.gluster.org/15765
Tested-by: ankitraj
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Raghavendra G

dht/rebalance: allocate migrator thread pool dynamically

2016-08-10T10:21:54+00:00

Problems: The maximum number of migratior threads created was static set
to "40". And the number of these threads get created in rebalance depends
on the number of cores user has. If the number of cores exceeds 40, a
crash or memory corruption can be seen.

Fix: Make the migratior thread pool dynamic.

> Change-Id: Ifbdac8a1a396363dd75e2f6bcb454070cfdbf839
> BUG: 1362069
> Reviewed-on: http://review.gluster.org/15000
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Raghavendra G 
(cherry picked from commit b8e8bfc7e4d3eaf76bb637221bc6392ec10ca54b)

Change-Id: Ifbdac8a1a396363dd75e2f6bcb454070cfdbf839
BUG: 1362069
Signed-off-by: Susant Palai 
Reviewed-on: http://review.gluster.org/15061
Smoke: Gluster Build System 
Tested-by: N Balachandran 
Reviewed-by: N Balachandran 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System

dht : add metalock/unlock

2016-06-06T06:33:57+00:00

Change-Id: I842a7ea1b286f1b893b200fe647597e7fd0f2105
BUG: 1338501
Signed-off-by: Susant Palai 
Reviewed-on: http://review.gluster.org/14631
Tested-by: Kotresh HR 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Raghavendra G 
CentOS-regression: Gluster Build System

tier/detach: Clear tier-fix-layout-complete xattr after migration threads join

2016-05-18T20:04:33+00:00

Previously we had wrongly placed the clearing tier-fix-layout-complete
xattr before the joining of migration threads. This would lead to
situations where failure of clearing the xattr would cause the
premature death of migration threads.

Now we clear the xattr only after the data movement threads join,
ensuring that all migration is done.

This is a backport of 14285

> Change-Id: I829b671efa165ae13dbff7b00707434970b37a09
> BUG: 1334839
> Signed-off-by: Joseph Fernandes 
Signed-off-by: Dan Lambright 

Change-Id: I08cc92db1fb8d48026ca51743be6cafe385d1b79
BUG: 1336152
Reviewed-on: http://review.gluster.org/14342
Smoke: Gluster Build System 
Reviewed-by: Joseph Fernandes
Tested-by: Dan Lambright 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

tier/detach : During detach check if background fixlayout is done

2016-05-06T20:11:38+00:00

During detach check if background fixlayout is done, if not done ignore
the case and continue detach.

Backport of http://review.gluster.org/14147

> Change-Id: I5d5cfc0e73d0eb217fdeab54c432dc4af8bc598d
> BUG: 1332136
> Signed-off-by: Joseph Fernandes 
> Reviewed-on: http://review.gluster.org/14147
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: N Balachandran 
> Reviewed-by: Dan Lambright 
> Signed-off-by: Joseph Fernandes 

Change-Id: I16f0320a2c07ac1d137bd4f27697722049df803c
BUG: 1333803
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/14241
Smoke: Gluster Build System 
Tested-by: Joseph Fernandes
NetBSD-regression: NetBSD Build System 
Reviewed-by: Niels de Vos 
CentOS-regression: Gluster Build System

dht/rebalance: add lock migration fop to dht_migrate_file

2016-05-02T01:05:14+00:00

Change-Id: Id0e7400c8ae950c90d42a3ddf8b558a14959a1f8
BUG: 1326085
Signed-off-by: Susant Palai 
Reviewed-on: http://review.gluster.org/14074
Smoke: Gluster Build System 
Reviewed-by: Niels de Vos 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

dht/rebalance: Handle GF_DEFRAG_STOP

2016-04-25T14:31:27+00:00

Problem: On a rebal stop, the migrator threads don't intimate the
crawler thread to wake up in case it is waiting on signal from
migrator thread.

Change-Id: I3cc4be41a4db25f48fee059ebb79a97ee99dcd00
BUG: 1327507
Signed-off-by: Susant Palai 
Reviewed-on: http://review.gluster.org/14004
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: N Balachandran 
Reviewed-by: Jeff Darcy

tier/dht : Attach tier fix layout to run in background

2016-03-28T16:17:59+00:00

1. Spawn a thread for background fix-layout for tier process.

2. Once the fix-layout is completed a marker xttr is set on the root of
volume to mark the completion of the background fixlayout, so that
even if the tier process is spawned again, fixlayout will not be
issued, if it was completed last time.

3. Please note that promotion of legacy files will happen eventually as
the ctr lookup heal in the fixlayout slowly heals the ctr db for legacy
files OR the ctr lookup heal happend due to a name lookup.

4. When a detach tier is successful in evacuation data from hot tier, we remove
the marker xattr is removed. So that next attach tier runs the background
tier fixlayout.

what is remaining ?
1. Instead of clearing the marker xattr of tiering fix layout at the end of detach start
clear it during detach commit. But the issue is detach commit is a glusterd operation
and the volume is not mounted in glusterd.
The reason we want to do it in detach commit is that if the admin wants to attach the
same tier again, then a background fixlayout will be triggered, which would not be needed.
2. Clearing the CTR DB of the cold bricks when there is a detach commit, as it will be having
entries which will be stale when the volume is used, with ctr off (ctr is switched off only when
we have detach commit.)

Change-Id: Ibe343572e95865325cd0eef4d0b976b626a3c0c5
BUG: 1313228
Signed-off-by: Joseph Fernandes
Reviewed-on: http://review.gluster.org/13491
Smoke: Gluster Build System
Tested-by: Joseph Fernandes
NetBSD-regression: NetBSD Build System
CentOS-regression: Gluster Build System
Reviewed-by: Dan Lambright

cluster/dht : Rebalance process crashes due to double fd_unref

2016-01-11T05:19:30+00:00

The dst_fd was being unrefed twice in case the call
to __dht_rebalance_create_dst_file failed.

Change-Id: I56c5aff7fa3827887e67936b8aa1ecbd1a08a9e9
BUG: 1296611
Signed-off-by: N Balachandran 
Reviewed-on: http://review.gluster.org/13193
Reviewed-by: Susant Palai 
Reviewed-by: Raghavendra G 
Tested-by: Raghavendra G

cluster/tier: allow db queries to be interruptable

2016-01-07T22:01:31+00:00

A query to the database may take a long time if the database
has many entries. The tier daemon also sends IPC calls to the
bricks which can run slowly, espcially in RHEL6. While it is
possible to track down each such instance, the snapshot
feature should not be affected by database operations. It requires
no migration be underway. Therefore it is okay to pause tiering
at any time except when DHT is moving a file.  This fix implements
this strategy by monitoring when control passes to DHT to
migrate a file using the GF_XATTR_FILE_MIGRATE_KEY trigger. If it
is not, the pause operation is successful.

Change-Id: I21f168b1bd424077ad5f38cf82f794060a1fabf6
BUG: 1287842
Signed-off-by: Dan Lambright 
Reviewed-on: http://review.gluster.org/13104
Reviewed-by: Joseph Fernandes
Tested-by: Gluster Build System