glusterfs.git/xlators/features, branch v3.7.6

tier/libgfdb: Replacing ASCII query file with binary

2015-11-08T13:22:02+00:00

Earlier, when the database was queried we used to save
all the queried records in an ASCII format in the query file.
This caused issues like filename having ASCII delimiter and used
to take a lot of space. The tier.c file also had a lot of parsing code.

Here we changed the format of the query file to binary.
All the logic of serialization and formating of query record is done
by libgfdb. Libgfdb provides API,
gfdb_write_query_record() and gfdb_read_query_record(),
which the user i.e tier migrator and CTR xlator can use to
write to and read from query file.
With this binary format we save on disk space i.e reduce to 50% atleast
as we are saving GFID's in binary format 16 bytes and not the string format
which takes 36 bytes + We are not saving path of the file + we are also saving on
ASCII delimiters.

The on disk format of query record is as follows,

+---------------------------------------------------------------------------+
| Length of serialized query record |       Serialized Query Record         |
+---------------------------------------------------------------------------+
             4 bytes                     Length of serialized query record
                                                      |
                                                      |
     -------------------------------------------------|
     |
     |
     V
   Serialized Query Record Format:
   +---------------------------------------------------------------------------+
   | GFID |  Link count   |    |.....                      | FOOTER |
   +---------------------------------------------------------------------------+
     16 B        4 B         Link Length                                  4 B
                                |                                          |
                                |                                          |
   -----------------------------|                                          |
   |                                                                       |
   |                                                                       |
   V                                                                       |
   Each  will be serialized as                                  |
   +-----------------------------------------------+                       |
   | PGID | BASE_NAME_LENGTH |      BASE_NAME      |                       |
   +-----------------------------------------------+                       |
     16 B       4 B             BASE_NAME_LENGTH                           |
                                                                           |
                                                                           |
   ------------------------------------------------------------------------|
   |
   |
   V
   FOOTER is a magic number 0xBAADF00D indicating the end of the record.
   This also serves as a serialized schema validator.

Backport of http://review.gluster.org/#/c/12354/

> Change-Id: I9db7416fd421e118dd44eafab8b535caafe50d5a
> BUG: 1272207
> Signed-off-by: Joseph Fernandes 
> Reviewed-on: http://review.gluster.org/12354
> Reviewed-by: N Balachandran 
> Tested-by: Gluster Build System 
> Reviewed-by: Dan Lambright 
> Tested-by: Dan Lambright 

Change-Id: I170c579027f2594a58706f826e3ddf89e34022f4
BUG: 1263619
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/12535
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

tier/ctr: Ignore bitrot related fops

2015-11-08T04:04:18+00:00

Ignore bitrot related fops since they are internal fops.

Backport of http://review.gluster.org/#/c/12512/

Change-Id: I5b676fe450d266b95bbd25aeca0d54436e098917
BUG: 1278640
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/12533
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright

tier/ctr: Correcting the internal fop calculation

2015-11-07T02:36:44+00:00

Correcting the internal fop calculation method, as it had wrong logic.

backport of http://review.gluster.org/#/c/12418/

Change-Id: I1deeae8b67dd967159853b494e89a3f46572c962
BUG: 1275483
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/12423
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

features/changelog: fix buffer overrun in changelog-helpers

2015-11-04T06:52:37+00:00

changelog-helpers.c:1911:17: warning: Size argument is greater than the free
space in the destination buffer strncat (result, pre_dir_name, PATH_MAX);
                                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

changelog-helpers.c:1919:17: warning: Size argument is greater than the free
space in the destination buffer strncat (result, bname, PATH_MAX);
                                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Backport of:
> Change-Id: I60ca7fe762f07cb72fe7b69f0253835becaff7b9
> BUG: 1222238
> Signed-off-by: Prasanna Kumar Kalever 
> Reviewed-on: http://review.gluster.org/10802
> Tested-by: NetBSD Build System 
> Reviewed-by: Venky Shankar 
> (cherry picked from commit 09530dfd822c8c3cc8da20a4600b5d2aec1ebf9d)

Change-Id: I46e1bf48b62f95e21f6615ac4afc22032f16f5a2
BUG:1252057
Signed-off-by: Prasanna Kumar Kalever 
Reviewed-on: http://review.gluster.org/12494
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Venky Shankar

quota: add version to quota xattrs

2015-11-03T05:03:46+00:00

This is a backport of http://review.gluster.org/#/c/12386/

When a quota is disable and the clean-up process terminated
without completely cleaning-up the quota xattrs.
Now when quota is enabled again, this can mess-up the accounting

A version number is suffixed for all quota xattrs and this version
number is specific to marker xaltor, i.e when quota xattrs are
requested by quotad/client marker will remove the version suffix in the
key before sending the response

> Change-Id: I1ca2c11460645edba0f6b68db70d476d8d26e1eb
> BUG: 1272411
> Signed-off-by: vmallika 
> Reviewed-on: http://review.gluster.org/12386
> Tested-by: NetBSD Build System 
> Reviewed-by: Krishnan Parthasarathi 
> Tested-by: Gluster Build System 
> Reviewed-by: Manikandan Selvaganesh 
> Reviewed-by: Raghavendra G 

Change-Id: I67b1b930b28411d76b2d476a4e5250c52aa495a0
BUG: 1277080
Signed-off-by: vmallika 
Reviewed-on: http://review.gluster.org/12487
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Raghavendra G 
Tested-by: Raghavendra G

features/shard: Force cache-refresh when lookup/readdirp/stat detect that xattr value has changed

2015-10-30T06:36:57+00:00

        Backport of: http://review.gluster.org/12400

Change-Id: Ifa51979bc530e31d36781759ca62bcac1de7af24
BUG: 1274600
Signed-off-by: Krutika Dhananjay 
Reviewed-on: http://review.gluster.org/12457
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System 
Reviewed-by: Pranith Kumar Karampuri

features/shard: Support geo-rep for sharded volume

2015-10-29T08:40:02+00:00

Approach:
      Shard xlator on slave side is by passed for all the fops
to geo-rep mount. So each shard on master is considered as a
separate file for geo-rep and it syncs them separately on to
slave. The extended attribute in which shard maintains the
size is also synced from master and shard on slave doesn't
calculate by itself.

Pre-requisites:
      1. If master is sharded volume, slave also should be sharded.
      2. Slave's shard configurations should be same as master.
      3. Geo-rep config of xattr sync should not be disabled.

All other dependant patches:
      1. http://review.gluster.org/#/c/12205/
      2. http://review.gluster.org/#/c/12206/
      3. http://review.gluster.org/#/c/12225/
      4. http://review.gluster.org/#/c/12226/

BUG: 1275972
Change-Id: Ieba70e75ebaebd70851454e1b85c0fe86022ad8d
Reviewed-on: http://review.gluster.org/12228
Tested-by: NetBSD Build System 
Reviewed-by: Krutika Dhananjay 
Reviewed-by: Pranith Kumar Karampuri 
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/12438
Tested-by: Gluster Build System

features/changelog: record mknod if tier-dht linkto is set

2015-10-29T07:17:47+00:00

This is a series of patches which aims to fix geo-replication
in a Tiering Volume.

Problem:
Consider, a file is placed in volume initially and then hot tier is
attached. During any operation on the file, due to lookup a linkto
file is created in hot tier.

Now, any namespace operation carried out on the file is recorded in
both cold and hot tier.
There is a room for races when both changelogs are replayed.

Solution:
So, We are going to replay (namespace related)operations
only in the hot tier.

Why?
a. If the file is directly placed in Hot tier, all fops will be
recorded in HOT tier.

b. If  the file is already present in Cold tier, and if any fop is
carried out, it creates linkto file in Hot tier.
Now, operations like UNLINK, RENAME are captured in Hot tier(by means of linkto file).

This way, we can get both tier's operation in HOT tier itself.

But, We may miss initial Data sync immediately after creating the
file as it is only recording MKNOD. So, if MKNOD encountered
with sticky bit set, queue DATA operation for the corresponding gfid.
( This geo-rep related changes are addressed in this patch: http://review.gluster.org/12326/ )

So, If tier-dht linkto is set, we need to record the corresponding
MKNOD. Earlier this was avoided as it was set as INTERNAL fop.
(This is addressed here in this patch)

Change-Id: I25514fe3e25f68592a8d6361507f8c8a4fcb70b1
BUG: 1275173
Reviewed-on: http://review.gluster.org/12417
Signed-off-by: Saravanakumar Arumugam 
Reviewed-on: http://review.gluster.org/12428
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System 
Reviewed-by: Venky Shankar

features/ctr: Reduce the log-level for ctr-disabled message

2015-10-28T11:47:17+00:00

     Backport of http://review.gluster.org/#/c/11592/

Since by default CTR translator is disabled, the following
log message is being displayed in brick logs.

[ctr-helper.c:256:extract_ctr_options] 0-gfdbdatastore: CTR Xlator is disabled.

Therefore this change is to reduce the log-level to INFO.

>Change-Id: I3b82d6b0dc0445286f91490fb497167a36914a2b
>BUG: 1241379
>Signed-off-by: Anoop C S 
>Reviewed-on: http://review.gluster.org/11592
>Reviewed-by: Joseph Fernandes
>Tested-by: Joseph Fernandes
>Tested-by: NetBSD Build System 
BUG: 1275157
Signed-off-by: Anoop C S 
(cherry picked from commit b6a7a44079c724cbf6fb24c7ac83892f551dc5f6)

Change-Id: I1bcea5258da525b955606f2353f749b2e37449cc
Reviewed-on: http://review.gluster.org/12424
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

features/changelog: ignore recording tiering rebalance fops

2015-10-28T10:31:03+00:00

Recording of tiering rebalance process's fops like Creation
and Deletion of file must be avoided.

Ignore the fops using corresponding pid.

Change-Id: Ifdc7765598d04d033f93e6339e9b188f7566cb65
BUG: 1275173
Signed-off-by: Saravanakumar Arumugam 
Reviewed-on: http://review.gluster.org/12239/
Reviewed-on: http://review.gluster.org/12425
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Venky Shankar