glusterfs.git/libglusterfs, branch v3.7.6

tier/libgfdb: Replacing ASCII query file with binary

2015-11-08T13:22:02+00:00

Earlier, when the database was queried we used to save
all the queried records in an ASCII format in the query file.
This caused issues like filename having ASCII delimiter and used
to take a lot of space. The tier.c file also had a lot of parsing code.

Here we changed the format of the query file to binary.
All the logic of serialization and formating of query record is done
by libgfdb. Libgfdb provides API,
gfdb_write_query_record() and gfdb_read_query_record(),
which the user i.e tier migrator and CTR xlator can use to
write to and read from query file.
With this binary format we save on disk space i.e reduce to 50% atleast
as we are saving GFID's in binary format 16 bytes and not the string format
which takes 36 bytes + We are not saving path of the file + we are also saving on
ASCII delimiters.

The on disk format of query record is as follows,

+---------------------------------------------------------------------------+
| Length of serialized query record |       Serialized Query Record         |
+---------------------------------------------------------------------------+
             4 bytes                     Length of serialized query record
                                                      |
                                                      |
     -------------------------------------------------|
     |
     |
     V
   Serialized Query Record Format:
   +---------------------------------------------------------------------------+
   | GFID |  Link count   |    |.....                      | FOOTER |
   +---------------------------------------------------------------------------+
     16 B        4 B         Link Length                                  4 B
                                |                                          |
                                |                                          |
   -----------------------------|                                          |
   |                                                                       |
   |                                                                       |
   V                                                                       |
   Each  will be serialized as                                  |
   +-----------------------------------------------+                       |
   | PGID | BASE_NAME_LENGTH |      BASE_NAME      |                       |
   +-----------------------------------------------+                       |
     16 B       4 B             BASE_NAME_LENGTH                           |
                                                                           |
                                                                           |
   ------------------------------------------------------------------------|
   |
   |
   V
   FOOTER is a magic number 0xBAADF00D indicating the end of the record.
   This also serves as a serialized schema validator.

Backport of http://review.gluster.org/#/c/12354/

> Change-Id: I9db7416fd421e118dd44eafab8b535caafe50d5a
> BUG: 1272207
> Signed-off-by: Joseph Fernandes 
> Reviewed-on: http://review.gluster.org/12354
> Reviewed-by: N Balachandran 
> Tested-by: Gluster Build System 
> Reviewed-by: Dan Lambright 
> Tested-by: Dan Lambright 

Change-Id: I170c579027f2594a58706f826e3ddf89e34022f4
BUG: 1263619
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/12535
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

tier/ctr: Ignore CTR Lookup heal insert errors

2015-11-07T18:32:33+00:00

CTR doesnt read from the DB, so to make sure that file records are
created it does a heal during a lookup. It remembers the decision in
the inode context cache and retrys periodically. When the volume is
restarted it looses all the inode cache from the previous time and CTR
lookup heals tries the heal again, but this time it finds that the records
are already there from sql and logs this error, and remembers this until the
volume is restarted or inode is flushed out of inode cache of the brick.

Solution: the log levels should be reduced to trace for this case and
customers need not see this.

Backport of http://review.gluster.org/#/c/12491/

Change-Id: I5f11e83c306ae912a90f12220f62bb7563b8eb96
BUG: 1279059
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/12534
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

quota: add version to quota xattrs

2015-11-03T05:03:46+00:00

This is a backport of http://review.gluster.org/#/c/12386/

When a quota is disable and the clean-up process terminated
without completely cleaning-up the quota xattrs.
Now when quota is enabled again, this can mess-up the accounting

A version number is suffixed for all quota xattrs and this version
number is specific to marker xaltor, i.e when quota xattrs are
requested by quotad/client marker will remove the version suffix in the
key before sending the response

> Change-Id: I1ca2c11460645edba0f6b68db70d476d8d26e1eb
> BUG: 1272411
> Signed-off-by: vmallika 
> Reviewed-on: http://review.gluster.org/12386
> Tested-by: NetBSD Build System 
> Reviewed-by: Krishnan Parthasarathi 
> Tested-by: Gluster Build System 
> Reviewed-by: Manikandan Selvaganesh 
> Reviewed-by: Raghavendra G 

Change-Id: I67b1b930b28411d76b2d476a4e5250c52aa495a0
BUG: 1277080
Signed-off-by: vmallika 
Reviewed-on: http://review.gluster.org/12487
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Raghavendra G 
Tested-by: Raghavendra G

cli : 'gluster volume help' output sorted alphabetically

2015-11-02T10:03:28+00:00

'gluster volume help' output is not sorted alphabetically.
This makes little harder for the user to search or get to know of
few gluster volume commands usage just from gluster cli.

    Backport of http://review.gluster.org/11663

Cherry picked from 323e71617fee5020324540776d0d4469577f0afe
Change-Id: I855da2e4748a5c2ff3be319c50fa9548d676ee8a
BUG: 1276208
Signed-off-by: Mohamed Ashiq 
>Reviewed-on: http://review.gluster.org/11663
>Tested-by: Gluster Build System 
>Reviewed-by: Manikandan Selvaganesh 
>Reviewed-by: Anand Nekkunti 
Reviewed-on: http://review.gluster.org/12452
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Anand Nekkunti

logging : GF_LOG_NONE logs always

2015-10-28T10:08:20+00:00

Shouldn't GF_LOG_NONE mean "Never log this"? If so, it's not being
tested for and is, instead, treated as a higher priority than
CRITICAL thus is always logged.

	Backport of http://review.gluster.org/11797
Cherry picked from 7b390983768e833a8df15b994038155c53d71bdd

>Change-Id: Icad1e02a720a05ff21bd54ebf19c0032e6d5ce03
>BUG: 1246794
>Signed-off-by: Mohamed Ashiq 
>Reviewed-on: http://review.gluster.org/11797
>Reviewed-by: Raghavendra Talur 
>Tested-by: NetBSD Build System 

Change-Id: Icad1e02a720a05ff21bd54ebf19c0032e6d5ce03
BUG: 1259167
Signed-off-by: Mohamed Ashiq 
Reviewed-on: http://review.gluster.org/12087
Tested-by: NetBSD Build System 
Reviewed-by: Raghavendra Talur 
Tested-by: Gluster Build System 
Reviewed-by: Niels de Vos

tier/ctr: CTR DB named lookup heal of cold tier during attach tier

2015-10-11T01:10:18+00:00

Heal hardlink in the db for already existing data in the cold
tier during attach tier. i.e during fix layout do lookup to files
in the cold tier.

CTR xlator on the  brick/server side does db update/insert of the hardlink on a namelookup.
Currently the namedlookup is done synchronous to the fixlayout that is
triggered by attach tier. This is not performant, adding more time to
fixlayout. The performant approach is record the hardlinks on a compressed
datastore and then do the namelookup asynchronously later, giving the ctr db
eventual consistency

master patch : http://review.gluster.org/#/c/11828/

>>Change-Id: I4ffc337fffe7d447804786851a9183a51b5044a9
>>BUG: 1252586
>>Signed-off-by: Joseph Fernandes 
>>Reviewed-on: http://review.gluster.org/11828
>>Tested-by: Gluster Build System 
>>Reviewed-by: Dan Lambright 
>>Tested-by: Dan Lambright 
Signed-off-by: Joseph Fernandes 

Change-Id: I61b185a54ae4e8c1d82804b95a278bfbea870987
BUG: 1261146
Reviewed-on: http://review.gluster.org/12331
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

cluster/tier: add watermarks and policy driver

2015-10-10T21:01:44+00:00

Backport fix 12039

This fix introduces infrastructure to support different
policies for promotion and demotion.

Currently the tier feature automatically promotes and demotes
files periodically based on access. This is good for testing
but too stringent for most real workloads. It makes it
difficult to fully utilize a hot tier- data will be demoted
before it is touched- its unlikely a 100GB hot SSD will have
all its data touched in a window of time.

A new parameter "mode" allows the user to pick promotion/demotion
polcies.

The "test mode" will be used for *.t and other general testing.
This is the current mechanism.

The "cache mode" introduces watermarks. The watermarks
represent levels of data residing on the hot tier.

"cache mode" policy:

The % the hot tier is full is called P.

Do not promote or demote more than D MB or F files.

A random number [0-100] is called R.

Rules for migration:

if (P < watermark_low) don't demote, always promote.

if (P >= watermark_low) && (P < watermark_hi) demote if R < P; promote if R > P.

if (P > watermark_hi) always demote, don't promote.

gluster volume set {vol} cluster.watermark-hi %
gluster volume set {vol} cluster.watermark-low %
gluster volume set {vol} cluster.tier-max-mb {D}
gluster volume set {vol} cluster.tier-max-files {F}
gluster volume set {vol} cluster.tier-mode {test|cache}

> Change-Id: I157f19667ec95aa1d53406041c1e3b073be127c2
> BUG: 1257911
> Signed-off-by: Dan Lambright 
> Reviewed-on: http://review.gluster.org/12039
> Tested-by: Gluster Build System 
> Reviewed-by: Atin Mukherjee 
Signed-off-by: Dan Lambright 

Signed-off-by: Dan Lambright 

Conflicts:
	xlators/cluster/dht/src/dht-rebalance.c
	xlators/cluster/dht/src/tier.c

Change-Id: Ibfe6b89563ceab98708325cf5d5ab0997c64816c
BUG: 1270527
Reviewed-on: http://review.gluster.org/12330
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

tier/ctr: Solution for db locks for tier migrator and ctr using sqlite version less than 3.7 i.e rhel 6.7

2015-10-09T23:58:53+00:00

Problem: On RHEL 6.7, we have sqlite version 3.6.2 which doesnt support
WAL journaling mode, as this journaling mode is only available in sqlite 3.7 and above.
As a result we cannot have to progreses concurrently accessing sqlite, without
running into db locks! Well WAL is also need for performace on CTR side.

Solution: This solution is to use CTR db connection for doing queries when WAL mode is
absent. i,e tier migrator will send sync_op ipc calls to CTR, which in turn will
do the query and create/update the query file suggested by tier migrator.

Pending: Well this solution will stop the db locks but the performance is still an issue for CTR.
We are developing an in-Memory Transaction Log (iMeTaL) which will help boost the CTR
performance by doing in memory udpates on the IO path and later flush the updates to
the db in a batch/segment flush.

Master patch: http://review.gluster.org/#/c/12191
>> Change-Id: Ie3149643ded159234b5cc6aa6cf93b9022c2f124
>> BUG: 1240577
>> Signed-off-by: Joseph Fernandes 
>> Signed-off-by: Dan Lambright 
>> Signed-off-by: Joseph Fernandes 
>> Reviewed-on: http://review.gluster.org/12191
>> Tested-by: Gluster Build System 
>> Reviewed-by: Luis Pabon 
Signed-off-by: Joseph Fernandes 

Change-Id: Ie8c7a7e9566244c104531b579126bb57fbc6e32b
BUG: 1270123
Reviewed-on: http://review.gluster.org/12325
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

cluster/ec : Mark new entry changelog in entry self-heal

2015-10-06T16:38:42+00:00

Problem :
When a new entry is created dirty mark xattrs are not
created this will need full heal to be performed, even
when there are partial failures.

Solution :
Marks new entry changelog in self-heal.

PS: Also fixed erasing of dirty markers when no data heal
is required.

BUG: 1258313
Signed-off-by: Ashish Pandey 
Change-Id: I156e3d3201afa77efe118e1aaace1d91c90a9613
Reviewed-on: http://review.gluster.org/12306
Tested-by: NetBSD Build System 
Reviewed-by: Pranith Kumar Karampuri

fd: Do fd_bind on successful open

2015-10-06T06:11:02+00:00

- fd_unref should decrement fd->inode->fd_count only if it is present in the
inode's fd list.
- successful open/opendir should perform fd_bind.

 >Change-Id: I81dd04f330e2fee86369a6dc7147af44f3d49169
 >BUG: 1207735
 >Signed-off-by: Pranith Kumar K 
 >Reviewed-on: http://review.gluster.org/11044
 >Reviewed-by: Anoop C S 
 >Tested-by: NetBSD Build System 
 >Tested-by: Gluster Build System 
 >Reviewed-by: Raghavendra G 

BUG: 1259697
Change-Id: I73b79dd3519aa085fb84dde74b321511cbccce1a
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/12100
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System 
Reviewed-by: Raghavendra G