glusterfs.git/xlators/cluster/dht/src, branch v3.7.6

tier/libgfdb: Replacing ASCII query file with binary

2015-11-08T13:22:02+00:00

Earlier, when the database was queried we used to save
all the queried records in an ASCII format in the query file.
This caused issues like filename having ASCII delimiter and used
to take a lot of space. The tier.c file also had a lot of parsing code.

Here we changed the format of the query file to binary.
All the logic of serialization and formating of query record is done
by libgfdb. Libgfdb provides API,
gfdb_write_query_record() and gfdb_read_query_record(),
which the user i.e tier migrator and CTR xlator can use to
write to and read from query file.
With this binary format we save on disk space i.e reduce to 50% atleast
as we are saving GFID's in binary format 16 bytes and not the string format
which takes 36 bytes + We are not saving path of the file + we are also saving on
ASCII delimiters.

The on disk format of query record is as follows,

+---------------------------------------------------------------------------+
| Length of serialized query record |       Serialized Query Record         |
+---------------------------------------------------------------------------+
             4 bytes                     Length of serialized query record
                                                      |
                                                      |
     -------------------------------------------------|
     |
     |
     V
   Serialized Query Record Format:
   +---------------------------------------------------------------------------+
   | GFID |  Link count   |    |.....                      | FOOTER |
   +---------------------------------------------------------------------------+
     16 B        4 B         Link Length                                  4 B
                                |                                          |
                                |                                          |
   -----------------------------|                                          |
   |                                                                       |
   |                                                                       |
   V                                                                       |
   Each  will be serialized as                                  |
   +-----------------------------------------------+                       |
   | PGID | BASE_NAME_LENGTH |      BASE_NAME      |                       |
   +-----------------------------------------------+                       |
     16 B       4 B             BASE_NAME_LENGTH                           |
                                                                           |
                                                                           |
   ------------------------------------------------------------------------|
   |
   |
   V
   FOOTER is a magic number 0xBAADF00D indicating the end of the record.
   This also serves as a serialized schema validator.

Backport of http://review.gluster.org/#/c/12354/

> Change-Id: I9db7416fd421e118dd44eafab8b535caafe50d5a
> BUG: 1272207
> Signed-off-by: Joseph Fernandes 
> Reviewed-on: http://review.gluster.org/12354
> Reviewed-by: N Balachandran 
> Tested-by: Gluster Build System 
> Reviewed-by: Dan Lambright 
> Tested-by: Dan Lambright 

Change-Id: I170c579027f2594a58706f826e3ddf89e34022f4
BUG: 1263619
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/12535
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

cluster/tier correct promotion cycle calculation

2015-11-07T21:44:38+00:00

This is a backport of 12480

The tier translator should only choose candidate files for promotion
from the most recent cycle, not a multiple of the most recent cycles.
Otherwise user observed behavior can be inconsistent. Remove related
test in tier.t that is subject to race condition.

> Change-Id: I9ad1523cac00f904097ce468efa6ddd515857024
> BUG: 1275524
> Signed-off-by: root 
> Signed-off-by: Dan Lambright 
> Reviewed-on: http://review.gluster.org/12480
> Reviewed-by: Joseph Fernandes
> Tested-by: Gluster Build System 
Signed-off-by: Dan Lambright 

Signed-off-by: Dan Lambright 

Conflicts:
	tests/basic/tier/tier.t
	xlators/cluster/dht/src/tier.c

Change-Id: Ic4587bf1b5d26ba377a12a4ce8e329362988a33b
BUG: 1275483
Reviewed-on: http://review.gluster.org/12536
Tested-by: NetBSD Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

cluster/tier: fix lookup-unhashed on tiered volumes

2015-11-06T16:24:47+00:00

During attach tier the commit hash must be copied to the hot tier.

This is a backport of 12498.

> Change-Id: I91b92fd8e98696993433856e1436409b657c439d
> BUG: 1277716
> Signed-off-by: Dan Lambright 
> Reviewed-on: http://review.gluster.org/12498
> Tested-by: NetBSD Build System 
> Tested-by: Gluster Build System 
> Reviewed-by: Jeff Darcy 
Signed-off-by: Dan Lambright 

Change-Id: I012c8ce9ef1dbb9550f109a1141afbecaa4fa54d
BUG: 1278603
Reviewed-on: http://review.gluster.org/12525
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

tier/dht: Ignoring replica for migration counting

2015-11-06T02:31:35+00:00

We used to count replica files for migration counting even though
they were ignore for migration as the replica brick didnt have
the ownership (as per the replication xlator either AFR/EC).
As a result the number of files migrated would show a wrong count,
i.e each replicated file would be counted 1 + number of replica.

This patch ignores such cases.

Backport of http://review.gluster.org/#/c/12453/

Change-Id: Ib005fedaee16f171e0499782b91f3eeedf8860ed
BUG: 1262860
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/12511
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

cluster/tier : Files skipped during tier query parsing

2015-11-04T11:55:08+00:00

The tier query parsing code was using fscanf to read each record.
As space is a delimiter for fscanf, filenames containing spaces
caused the parsing to return unexpected values causing various
issues in the tier process, including crashes due to buffer
 overflows.

> Change-Id: Ife602cb7ecb158fccbc2c89e4d2959bd97098a87
> Signed-off-by: N Balachandran 
> Reviewed-on: http://review.gluster.org/12469
> Tested-by: NetBSD Build System 
> Tested-by: Gluster Build System 
> Reviewed-by: Dan Lambright 
> Tested-by: Dan Lambright 
(cherry picked from commit 499b43058049572e33b525ac669ef623d476fe41)

Change-Id: Id60f9c484dfbb02de6ebb44032160ad4cc94cb7f
BUG: 1277587
Signed-off-by: N Balachandran 
Reviewed-on: http://review.gluster.org/12502
Tested-by: NetBSD Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

quota: add version to quota xattrs

2015-11-03T05:03:46+00:00

This is a backport of http://review.gluster.org/#/c/12386/

When a quota is disable and the clean-up process terminated
without completely cleaning-up the quota xattrs.
Now when quota is enabled again, this can mess-up the accounting

A version number is suffixed for all quota xattrs and this version
number is specific to marker xaltor, i.e when quota xattrs are
requested by quotad/client marker will remove the version suffix in the
key before sending the response

> Change-Id: I1ca2c11460645edba0f6b68db70d476d8d26e1eb
> BUG: 1272411
> Signed-off-by: vmallika 
> Reviewed-on: http://review.gluster.org/12386
> Tested-by: NetBSD Build System 
> Reviewed-by: Krishnan Parthasarathi 
> Tested-by: Gluster Build System 
> Reviewed-by: Manikandan Selvaganesh 
> Reviewed-by: Raghavendra G 

Change-Id: I67b1b930b28411d76b2d476a4e5250c52aa495a0
BUG: 1277080
Signed-off-by: vmallika 
Reviewed-on: http://review.gluster.org/12487
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Raghavendra G 
Tested-by: Raghavendra G

cluster/tier dont log error on lookup heal for files on hot tier

2015-10-30T12:53:32+00:00

This is a backport of 12430

On fix-layout heal files are scanned. Files found are exist on the hot or cold
subvolume. Those not found in the cold tier would exist on the hot. They
should not be flagged as an error.

Replace INFO with TRACE for common tier migration logs. Frequent migration
was growing the log files too quickly.

On migratation failures, do not acrue files towards cycle limit's budget.

> Change-Id: Ie832ee07c43bce5477ae81c939d1fe8416a11615
> BUG: 1275383
> Signed-off-by: Dan Lambright 
> Reviewed-on: http://review.gluster.org/12430
> Tested-by: Gluster Build System 
> Reviewed-by: Joseph Fernandes
Signed-off-by: Dan Lambright 

Change-Id: Ia1ce5c3ac9c8c43cf3f3f7e0bd6161aa13affe5f
BUG: 1272398
Signed-off-by: Dan Lambright 
Reviewed-on: http://review.gluster.org/12465
Tested-by: Gluster Build System

cluster/dht : op_ret not set correctly in dht_fsync_cbk

2015-10-25T13:41:31+00:00

local->op_ret was not set correctly in dht_fsync_cbk in case
the file was being migrated

> Change-Id: If73ae04368ea0c7f6868c8704dfc2deb2faee753
> Signed-off-by: N Balachandran 
> Reviewed-on: http://review.gluster.org/12401
> Tested-by: Gluster Build System 
> Reviewed-by: Raghavendra G 
> Reviewed-by: Jeff Darcy 
(cherry picked from commit 9710f58e5874bccb4b328abef80ea226ccf9c798)

Change-Id: I2addb86083c1d8305cf91e0b0385deeb227216c8
BUG: 1272036
Signed-off-by: N Balachandran 
Reviewed-on: http://review.gluster.org/12409
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

cluster/tier: add pause tier for snapshots

2015-10-22T02:46:18+00:00

This is a backport of 12304

Snaps of tiered volumes cannot handle files undergoing migration.
We implement a helper mechanism to "pause" migration. Any files
undergoing migration are aborted. Clean up is done to remove
sticky bits and data at the destination. Migration is restarted
after snap completes.

For testing an internal switch is added. It is not exposed externally.

gluster volume set vol1 tier-pause [true|false]

> Change-Id: Ia85bbf89ac142e9b7e73fcbef98bb9da86097799
> BUG: 1267950
> Signed-off-by: Dan Lambright 
> Reviewed-on: http://review.gluster.org/12304
> Reviewed-by: N Balachandran 
> Tested-by: NetBSD Build System 
> Tested-by: Gluster Build System 
Signed-off-by: Dan Lambright 

Conflicts:
	xlators/mgmt/glusterd/src/glusterd-messages.h

Change-Id: I5f039d8d38a4c915bd873969f336b96755a0b8f1
BUG: 1274101
Reviewed-on: http://review.gluster.org/12411
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

cluster/tier do not abort migration if a single brick is down

2015-10-21T13:11:21+00:00

backport fix 12397

When a bricks are down, promotion/demotion should still be possible.
For example, if an EC brick is down, the other bricks are able to
recover the data and migrate it.

> Change-Id: I8e650c640bce22a3ad23d75c363fbb9fd027d705
> BUG: 1273215
> Signed-off-by: Dan Lambright 
> Reviewed-on: http://review.gluster.org/12397
> Tested-by: NetBSD Build System 
> Tested-by: Gluster Build System 
> Reviewed-by: Joseph Fernandes
Signed-off-by: Dan Lambright 

Change-Id: I6688757eaf97426c8e1ea1038c598b34bf6b8ccc
BUG: 1272334
Signed-off-by: Dan Lambright 
Reviewed-on: http://review.gluster.org/12405
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System