glusterfs.git/xlators/cluster/dht/src/dht-rebalance.c, branch v3.7.15

dht/rebalance: allocate migrator thread pool dynamically

2016-08-05T10:55:39+00:00

Problems: The maximum number of migratior threads created was static set
to "40". And the number of these threads get created in rebalance depends
on the number of cores user has. If the number of cores exceeds 40, a
crash or memory corruption can be seen.

Fix: Make the migratior thread pool dynamic.

> Change-Id: Ifbdac8a1a396363dd75e2f6bcb454070cfdbf839
> BUG: 1362070
> Reviewed-on: http://review.gluster.org/15000
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Raghavendra G 
(cherry picked from commit b8e8bfc7e4d3eaf76bb637221bc6392ec10ca54b)

Change-Id: Ifbdac8a1a396363dd75e2f6bcb454070cfdbf839
BUG: 1362070
Signed-off-by: Susant Palai 
Reviewed-on: http://review.gluster.org/15062
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Raghavendra G

tier/detach: Clear tier-fix-layout-complete xattr after migration threads join

2016-05-16T12:21:56+00:00

Previously we had wrongly placed the clearing tier-fix-layout-complete
xattr before the joining of migration threads. This would lead to
situations where failure of clearing the xattr would cause the
premature death of migration threads.

Now we clear the xattr only after the data movement threads join,
ensuring that all migration is done.

This is a backport of 14285

> Change-Id: I829b671efa165ae13dbff7b00707434970b37a09
> BUG: 1334839
> Signed-off-by: Joseph Fernandes 

Signed-off-by: Dan Lambright 

Change-Id: I475242e6a05cacd2252dc5c29b160e7abc5d1791
BUG: 1336148
Reviewed-on: http://review.gluster.org/14341
Smoke: Gluster Build System 
Tested-by: N Balachandran 
NetBSD-regression: NetBSD Build System 
Reviewed-by: N Balachandran 
CentOS-regression: Gluster Build System 
Reviewed-by: Dan Lambright

tier/detach : During detach check if background fixlayout is done

2016-05-14T10:23:32+00:00

During detach check if background fixlayout is done, if not done ignore
the case and continue detach.

Backport of http://review.gluster.org/14147

> Change-Id: I5d5cfc0e73d0eb217fdeab54c432dc4af8bc598d
> BUG: 1332136
> Signed-off-by: Joseph Fernandes 
> Reviewed-on: http://review.gluster.org/14147
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: N Balachandran 
> Reviewed-by: Dan Lambright 
>Signed-off-by: Joseph Fernandes 

Change-Id: I2161673cf6861b02a8e323366a13a13587258bef
BUG: 1333934
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/14246
Smoke: Gluster Build System 
Tested-by: Joseph Fernandes
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: N Balachandran 
Reviewed-by: Dan Lambright

dht/rebalance: Handle GF_DEFRAG_STOP

2016-04-26T19:02:27+00:00

Backport of http://review.gluster.org/14004

Problem: On a rebal stop, the migrator threads don't intimate the
crawler thread to wake up in case it is waiting on signal from
migrator thread.

BUG: 1330529
Change-Id: I9019a715c7b4673b8bb5a75d7d33a18add85ce33
Reviewed-by: N Balachandran 
Signed-off-by: Susant Palai 
Reviewed-on: http://review.gluster.org/14076
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Jeff Darcy 
Smoke: Gluster Build System

cluster/dht/rebalance: rebalance failure handling

2016-04-06T06:33:48+00:00

At current state rebalance aborts basically on any failure
like fix-layout of a directory, readdirp, opendir etc. Unless it is
not a remove-brick process we can ignore these failures.

Major impact:  Any failure in the gf_defrag_process_dir means there
are files left unmigrated in the directory.

Fix-layout(setxattr) failure will impact it's child subtree i.e.
the child subtree will not be rebalanced.

Settle-hash (commit-hash)failure will trigger lookup_everywhere for
immediate children until the next commit-hash.

Note: Remove-brick opertaion is still sensitive to any kind of failure.

Change-Id: I2f67a490e4e7d06423bb5bc010a1373a74a6af1d
BUG: 1318196
Signed-off-by: Susant Palai 
Reviewed-on: http://review.gluster.org/12013
Tested-by: NetBSD Build System 
Reviewed-by: Raghavendra G 
Signed-off-by: Susant Palai 
Reviewed-on: http://review.gluster.org/13749
Tested-by: N Balachandran 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System

tier/dht : Attach tier fix layout to run in background

2016-04-01T10:55:32+00:00

1. Spawn a thread for background fix-layout for tier process.

2. Once the fix-layout is completed a marker xttr is set on the root of
   volume to mark the completion of the background fixlayout, so that
   even if the tier process is spawned again, fixlayout will not be
   issued, if it was completed last time.

3. Please note that promotion of legacy files will happen eventually as
   the ctr lookup heal in the fixlayout slowly heals the ctr db for legacy
   files OR the ctr lookup heal happend due to a name lookup.

4. When a detach tier is successful in evacuation data from hot tier, we remove
   the marker xattr is removed. So that next attach tier runs the background
   tier fixlayout.

what is remaining ?
1. Instead of clearing the marker xattr of tiering fix layout at the end of detach start
   clear it during detach commit. But the issue is detach commit is a glusterd operation
   and the volume is not mounted in glusterd.
   The reason we want to do it in detach commit is that if the admin wants to attach the
   same tier again, then a background fixlayout will be triggered, which would not be needed.
2. Clearing the CTR DB of the cold bricks when there is a detach commit, as it will be having
   entries which will be stale when the volume is used, with ctr off (ctr is switched off only when
   we have detach commit.)

Backport of http://review.gluster.org/13491

> Change-Id: Ibe343572e95865325cd0eef4d0b976b626a3c0c5
> BUG: 1313228
> Signed-off-by: Joseph Fernandes 
> Reviewed-on: http://review.gluster.org/13491
> Smoke: Gluster Build System 
> Tested-by: Joseph Fernandes
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Dan Lambright 
Signed-off-by: Joseph Fernandes 

Change-Id: Ic28affdf78d2ac0f394f3dd59f0126df7915d609
BUG: 1323016
Reviewed-on: http://review.gluster.org/13879
Smoke: Gluster Build System 
Reviewed-by: Joseph Fernandes
Tested-by: Joseph Fernandes
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Dan Lambright

cluster/tier: allow db queries to be interruptable

2016-02-18T13:54:58+00:00

A query to the database may take a long time if the database
has many entries. The tier daemon also sends IPC calls to the
bricks which can run slowly, espcially in RHEL6. While it is
possible to track down each such instance, the snapshot
feature should not be affected by database operations. It requires
no migration be underway. Therefore it is okay to pause tiering
at any time except when DHT is moving a file.  This fix implements
this strategy by monitoring when control passes to DHT to
migrate a file using the GF_XATTR_FILE_MIGRATE_KEY trigger. If it
is not, the pause operation is successful.

> Change-Id: I21f168b1bd424077ad5f38cf82f794060a1fabf6
> BUG: 1287842
> Signed-off-by: Dan Lambright 
> Reviewed-on: http://review.gluster.org/13104
> Reviewed-by: Joseph Fernandes
> Tested-by: Gluster Build System 
Signed-off-by: Dan Lambright 

Change-Id: I667e0af24eaa66afefa860c4d73b324e4f39b997
BUG: 1288352
Signed-off-by: Dan Lambright 
Reviewed-on: http://review.gluster.org/13199
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

cluster/dht : Rebalance process crashes due to double fd_unref

2016-01-25T02:28:22+00:00

The dst_fd was being unrefed twice in case the call
to __dht_rebalance_create_dst_file failed.

> BUG: 1296611
> Signed-off-by: N Balachandran 
> Reviewed-on: http://review.gluster.org/13193
> Reviewed-by: Susant Palai 
> Reviewed-by: Raghavendra G 
> Tested-by: Raghavendra G 
(cherry picked from commit 36fcaf275952202ce3f1e621d3b3a34d58054c99)

Change-Id: I56c5aff7fa3827887e67936b8aa1ecbd1a08a9e9
BUG: 1297309
Signed-off-by: N Balachandran 
Reviewed-on: http://review.gluster.org/13212
Reviewed-by: Raghavendra G 
Reviewed-by: Pranith Kumar Karampuri 
Tested-by: Pranith Kumar Karampuri 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Vijay Bellur 
NetBSD-regression: NetBSD Build System

all: reduce "inline" usage

2016-01-18T09:02:34+00:00

There are three kinds of inline functions: plain inline, extern inline,
and static inline.  All three have been removed from .c files, except
those in "contrib" which aren't our problem.  Inlines in .h files, which
are overwhelmingly "static inline" already, have generally been left
alone.  Over time we should be able to "lower" these into .c files, but
that has to be done in a case-by-case fashion requiring more manual
effort.  This part was easy to do automatically without (as far as I can
tell) any ill effect.

In the process, several pieces of dead code were flagged by the
compiler, and were removed.

backport of Change-Id: I56a5e614735c9e0a6ee420dab949eac22e25c155,
http://review.gluster.org/11769, BUG: 1245331

Change-Id: Iba1efb0bc578ea4a5e9bf76b7bd93dc1be9eba44
BUG: 1283302
Signed-off-by: Kaleb S KEITHLEY 
Reviewed-on: http://review.gluster.org/12646
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

tier: Demotion failed if the file was renamed when it was in cold

2015-12-30T16:29:03+00:00

During migration if the file is present we just open the file
in hashed subvol. Now if the linkfile present on hashed is just
linkfile to another subvol, we actually open in hashed subvol.

But subsequent operation will go to linkto subvol ie,
to non-hashed subvol. This operation will get failed
since we haven't opened d on non-hashed.

Backport of>
>Change-Id: I9753ad3a48f0384c25509612ba76e7e10645add3
>BUG: 1292067
>Signed-off-by: Mohammed Rafi KC 
>Reviewed-on: http://review.gluster.org/12980
>Tested-by: NetBSD Build System 
>Reviewed-by: Susant Palai 
>Tested-by: Gluster Build System 
>Reviewed-by: Dan Lambright 
>Tested-by: Dan Lambright 

(cherry picked from commit d2f48214d436be633efb1136ee951b0736935143)


Change-Id: I562ac4ba73e6b572bc1be91e55a029fa75262b33
BUG: 1293342
Signed-off-by: Mohammed Rafi KC 
Reviewed-on: http://review.gluster.org/13045
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright