glusterfs.git/xlators, branch v3.7.6

tier/libgfdb: Replacing ASCII query file with binary

2015-11-08T13:22:02+00:00

Earlier, when the database was queried we used to save
all the queried records in an ASCII format in the query file.
This caused issues like filename having ASCII delimiter and used
to take a lot of space. The tier.c file also had a lot of parsing code.

Here we changed the format of the query file to binary.
All the logic of serialization and formating of query record is done
by libgfdb. Libgfdb provides API,
gfdb_write_query_record() and gfdb_read_query_record(),
which the user i.e tier migrator and CTR xlator can use to
write to and read from query file.
With this binary format we save on disk space i.e reduce to 50% atleast
as we are saving GFID's in binary format 16 bytes and not the string format
which takes 36 bytes + We are not saving path of the file + we are also saving on
ASCII delimiters.

The on disk format of query record is as follows,

+---------------------------------------------------------------------------+
| Length of serialized query record |       Serialized Query Record         |
+---------------------------------------------------------------------------+
             4 bytes                     Length of serialized query record
                                                      |
                                                      |
     -------------------------------------------------|
     |
     |
     V
   Serialized Query Record Format:
   +---------------------------------------------------------------------------+
   | GFID |  Link count   |    |.....                      | FOOTER |
   +---------------------------------------------------------------------------+
     16 B        4 B         Link Length                                  4 B
                                |                                          |
                                |                                          |
   -----------------------------|                                          |
   |                                                                       |
   |                                                                       |
   V                                                                       |
   Each  will be serialized as                                  |
   +-----------------------------------------------+                       |
   | PGID | BASE_NAME_LENGTH |      BASE_NAME      |                       |
   +-----------------------------------------------+                       |
     16 B       4 B             BASE_NAME_LENGTH                           |
                                                                           |
                                                                           |
   ------------------------------------------------------------------------|
   |
   |
   V
   FOOTER is a magic number 0xBAADF00D indicating the end of the record.
   This also serves as a serialized schema validator.

Backport of http://review.gluster.org/#/c/12354/

> Change-Id: I9db7416fd421e118dd44eafab8b535caafe50d5a
> BUG: 1272207
> Signed-off-by: Joseph Fernandes 
> Reviewed-on: http://review.gluster.org/12354
> Reviewed-by: N Balachandran 
> Tested-by: Gluster Build System 
> Reviewed-by: Dan Lambright 
> Tested-by: Dan Lambright 

Change-Id: I170c579027f2594a58706f826e3ddf89e34022f4
BUG: 1263619
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/12535
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

tier/ctr: Ignore bitrot related fops

2015-11-08T04:04:18+00:00

Ignore bitrot related fops since they are internal fops.

Backport of http://review.gluster.org/#/c/12512/

Change-Id: I5b676fe450d266b95bbd25aeca0d54436e098917
BUG: 1278640
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/12533
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright

cluster/tier correct promotion cycle calculation

2015-11-07T21:44:38+00:00

This is a backport of 12480

The tier translator should only choose candidate files for promotion
from the most recent cycle, not a multiple of the most recent cycles.
Otherwise user observed behavior can be inconsistent. Remove related
test in tier.t that is subject to race condition.

> Change-Id: I9ad1523cac00f904097ce468efa6ddd515857024
> BUG: 1275524
> Signed-off-by: root 
> Signed-off-by: Dan Lambright 
> Reviewed-on: http://review.gluster.org/12480
> Reviewed-by: Joseph Fernandes
> Tested-by: Gluster Build System 
Signed-off-by: Dan Lambright 

Signed-off-by: Dan Lambright 

Conflicts:
	tests/basic/tier/tier.t
	xlators/cluster/dht/src/tier.c

Change-Id: Ic4587bf1b5d26ba377a12a4ce8e329362988a33b
BUG: 1275483
Reviewed-on: http://review.gluster.org/12536
Tested-by: NetBSD Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

tier/ctr: Correcting the internal fop calculation

2015-11-07T02:36:44+00:00

Correcting the internal fop calculation method, as it had wrong logic.

backport of http://review.gluster.org/#/c/12418/

Change-Id: I1deeae8b67dd967159853b494e89a3f46572c962
BUG: 1275483
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/12423
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

cluster/tier: fix lookup-unhashed on tiered volumes

2015-11-06T16:24:47+00:00

During attach tier the commit hash must be copied to the hot tier.

This is a backport of 12498.

> Change-Id: I91b92fd8e98696993433856e1436409b657c439d
> BUG: 1277716
> Signed-off-by: Dan Lambright 
> Reviewed-on: http://review.gluster.org/12498
> Tested-by: NetBSD Build System 
> Tested-by: Gluster Build System 
> Reviewed-by: Jeff Darcy 
Signed-off-by: Dan Lambright 

Change-Id: I012c8ce9ef1dbb9550f109a1141afbecaa4fa54d
BUG: 1278603
Reviewed-on: http://review.gluster.org/12525
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

tier/dht: Ignoring replica for migration counting

2015-11-06T02:31:35+00:00

We used to count replica files for migration counting even though
they were ignore for migration as the replica brick didnt have
the ownership (as per the replication xlator either AFR/EC).
As a result the number of files migrated would show a wrong count,
i.e each replicated file would be counted 1 + number of replica.

This patch ignores such cases.

Backport of http://review.gluster.org/#/c/12453/

Change-Id: Ib005fedaee16f171e0499782b91f3eeedf8860ed
BUG: 1262860
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/12511
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

snapshot: Don't display snapshot's hard-limit and soft-limit in vol info

2015-11-05T11:20:46+00:00

    Backport of http://review.gluster.org/#/c/12443/

The snap-max-hard-limit being displayed in the volume info
currently is propagated from system's snap-max-hard-limit as
that is a global option common for all volumes, and hence ends
up showing the system's snap-max-hard-limit.

We should not be displaying snap-max-hard-limit and
snap-max-soft-limit in the volume info at all, as these are
snap config options and should be set and displayed via snap
config command.

Modified bug-1113476.t to test the same behaviour.

Change-Id: I90891f0cf7fb39fd686787297c7f7cd8c1e7daa1
BUG: 1277394
Signed-off-by: Avra Sengupta 
Reviewed-on: http://review.gluster.org/12443
Tested-by: Gluster Build System 
Reviewed-by: mohammed rafi  kc 
Reviewed-by: Rajesh Joseph 
Tested-by: NetBSD Build System 
(cherry picked from commit 2e56bde3ea952beabd27cdf8a3a10da563a00bcc)
Reviewed-on: http://review.gluster.org/12493

v info for disperse count fails while upgrading

2015-11-04T18:55:06+00:00

        back-port of : http://review.gluster.org/#/c/12495/

The upgrade from 3.7.5-3 to 3.7.5-5 causes the type and number
of bricks for the cold tier to be printed wrong.

>Change-Id: Ia45b97c35fef88f9c66e15e5bdb93fd30cb342af
>BUG: 1277481
>Signed-off-by: Hari Gowtham 
>Reviewed-on: http://review.gluster.org/12495
>Tested-by: NetBSD Build System 
>Reviewed-by: Dan Lambright 
>Tested-by: Dan Lambright 

Change-Id: Ic61dd1378c8efe37d797328719ba16e64ff76f55
BUG: 1277984
Signed-off-by: Hari Gowtham 
Reviewed-on: http://review.gluster.org/12506
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

cluster/tier : Files skipped during tier query parsing

2015-11-04T11:55:08+00:00

The tier query parsing code was using fscanf to read each record.
As space is a delimiter for fscanf, filenames containing spaces
caused the parsing to return unexpected values causing various
issues in the tier process, including crashes due to buffer
 overflows.

> Change-Id: Ife602cb7ecb158fccbc2c89e4d2959bd97098a87
> Signed-off-by: N Balachandran 
> Reviewed-on: http://review.gluster.org/12469
> Tested-by: NetBSD Build System 
> Tested-by: Gluster Build System 
> Reviewed-by: Dan Lambright 
> Tested-by: Dan Lambright 
(cherry picked from commit 499b43058049572e33b525ac669ef623d476fe41)

Change-Id: Id60f9c484dfbb02de6ebb44032160ad4cc94cb7f
BUG: 1277587
Signed-off-by: N Balachandran 
Reviewed-on: http://review.gluster.org/12502
Tested-by: NetBSD Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

features/changelog: fix buffer overrun in changelog-helpers

2015-11-04T06:52:37+00:00

changelog-helpers.c:1911:17: warning: Size argument is greater than the free
space in the destination buffer strncat (result, pre_dir_name, PATH_MAX);
                                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

changelog-helpers.c:1919:17: warning: Size argument is greater than the free
space in the destination buffer strncat (result, bname, PATH_MAX);
                                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Backport of:
> Change-Id: I60ca7fe762f07cb72fe7b69f0253835becaff7b9
> BUG: 1222238
> Signed-off-by: Prasanna Kumar Kalever 
> Reviewed-on: http://review.gluster.org/10802
> Tested-by: NetBSD Build System 
> Reviewed-by: Venky Shankar 
> (cherry picked from commit 09530dfd822c8c3cc8da20a4600b5d2aec1ebf9d)

Change-Id: I46e1bf48b62f95e21f6615ac4afc22032f16f5a2
BUG:1252057
Signed-off-by: Prasanna Kumar Kalever 
Reviewed-on: http://review.gluster.org/12494
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Venky Shankar