glusterfs.git/libglusterfs/src/gfdb/gfdb_data_store.c, branch v3.8rc0

ctr/sql: Providing for vol set for sqlcachesize and sqlWALsize and skip recording path

2015-12-22T15:14:10+00:00

1. Providing vol set option for cache size and wal autocheck point
   so that performance can be tuned.
2. Removed recording of file path in the db. Trimming database columns.
   Path need not be stored in the db, as PARGFID, GFID, Basename is suffice
   to derive the path during migration.

Change-Id: I2cb590451a6d244bc91fe66c6dbffe2c2059dfb8
BUG: 1293034
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/12972
Reviewed-by: N Balachandran 
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

tier/ctr: Ignore CTR Lookup heal insert errors

2015-11-07T01:14:03+00:00

CTR doesnt read from the DB, so to make sure that file records are
created it does a heal during a lookup. It remembers the decision in
the inode context cache and retrys periodically. When the volume is
restarted it looses all the inode cache from the previous time and CTR
lookup heals tries the heal again, but this time it finds that the records
are already there from sql and logs this error, and remembers this until the
volume is restarted or inode is flushed out of inode cache of the brick.

Solution: the log levels should be reduced to trace for this case and
customers need not see this.

Change-Id: I67b568fb6904f8597e2c6d32894a247c4f500b94
BUG: 1277352
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/12491
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

tier/libgfdb: Replacing ASCII query file with binary

2015-11-06T18:30:45+00:00

Earlier, when the database was queried we used to save
all the queried records in an ASCII format in the query file.
This caused issues like filename having ASCII delimiter and used
to take a lot of space. The tier.c file also had a lot of parsing code.

Here we changed the format of the query file to binary.
All the logic of serialization and formating of query record is done
by libgfdb. Libgfdb provides API,
gfdb_write_query_record() and gfdb_read_query_record(),
which the user i.e tier migrator and CTR xlator can use to
write to and read from query file.
With this binary format we save on disk space i.e reduce to 50% atleast
as we are saving GFID's in binary format 16 bytes and not the string format
which takes 36 bytes + We are not saving path of the file + we are also saving on
ASCII delimiters.

The on disk format of query record is as follows,

+---------------------------------------------------------------------------+
| Length of serialized query record |       Serialized Query Record         |
+---------------------------------------------------------------------------+
             4 bytes                     Length of serialized query record
                                                      |
                                                      |
     -------------------------------------------------|
     |
     |
     V
   Serialized Query Record Format:
   +---------------------------------------------------------------------------+
   | GFID |  Link count   |    |.....                      | FOOTER |
   +---------------------------------------------------------------------------+
     16 B        4 B         Link Length                                  4 B
                                |                                          |
                                |                                          |
   -----------------------------|                                          |
   |                                                                       |
   |                                                                       |
   V                                                                       |
   Each  will be serialized as                                  |
   +-----------------------------------------------+                       |
   | PGID | BASE_NAME_LENGTH |      BASE_NAME      |                       |
   +-----------------------------------------------+                       |
     16 B       4 B             BASE_NAME_LENGTH                           |
                                                                           |
                                                                           |
   ------------------------------------------------------------------------|
   |
   |
   V
   FOOTER is a magic number 0xBAADF00D indicating the end of the record.
   This also serves as a serialized schema validator.

Change-Id: I9db7416fd421e118dd44eafab8b535caafe50d5a
BUG: 1272207
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/12354
Reviewed-by: N Balachandran 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

tier/ctr: Solution for db locks for tier migrator and ctr using sqlite version less than 3.7 i.e rhel 6.7

2015-10-08T19:00:31+00:00

Problem: On RHEL 6.7, we have sqlite version 3.6.2 which doesnt support
WAL journaling mode, as this journaling mode is only available in sqlite 3.7 and above.
As a result we cannot have to progreses concurrently accessing sqlite, without
running into db locks! Well WAL is also need for performace on CTR side.

Solution: This solution is to use CTR db connection for doing queries when WAL mode is
absent. i,e tier migrator will send sync_op ipc calls to CTR, which in turn will
do the query and create/update the query file suggested by tier migrator.

Pending: Well this solution will stop the db locks but the performance is still an issue for CTR.
We are developing an in-Memory Transaction Log (iMeTaL) which will help boost the CTR
performance by doing in memory udpates on the IO path and later flush the updates to
the db in a batch/segment flush.

Change-Id: Ie3149643ded159234b5cc6aa6cf93b9022c2f124
BUG: 1240577
Signed-off-by: Joseph Fernandes 
Signed-off-by: Dan Lambright 
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/12191
Tested-by: Gluster Build System 
Reviewed-by: Luis Pabon

cluster/tier fix bug with sql includes introduced by 12031

2015-09-12T00:12:55+00:00

We accidentally introduced a bug where client translators have a
dependency on sql. This broke freebsd smoke tests. Fix is to
abstract from the client those dependencies.

Change-Id: I7152573a489bacc8f32e6eb139f9ff4408288f5b
BUG: 1260730
Signed-off-by: Dan Lambright 
Reviewed-on: http://review.gluster.org/12155
Tested-by: NetBSD Build System

tier/ctr: Solving DB Lock issue due to write contention from db connections

2015-09-08T12:13:00+00:00

Problem: The DB on the brick is been accessed by CTR, for write and
tier migrator, for read and write. The write from tier migrator is reseting
the heat counters after a cycle. Since we are using sqlite, two connections
trying to write would cause a db lock contention. As a result CTR used to fail
to update the db.

Solution: Using the same db connection of CTR for reseting the heat counters.
1) Introducted a new IPC FOP for CTR
2) After the query do a ipc syncop to the underlying client xlator associated
   to the brick.
3) CTR in brick will catch the IPC FOP and cleat the heat counters.

Change-Id: I53306bfc08dcdba479deb4ccc154896521336150
BUG: 1260730
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/12031
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

all: reduce "inline" usage

2015-09-01T11:55:15+00:00

There are three kinds of inline functions: plain inline, extern inline,
and static inline.  All three have been removed from .c files, except
those in "contrib" which aren't our problem.  Inlines in .h files, which
are overwhelmingly "static inline" already, have generally been left
alone.  Over time we should be able to "lower" these into .c files, but
that has to be done in a case-by-case fashion requiring more manual
effort.  This part was easy to do automatically without (as far as I can
tell) any ill effect.

In the process, several pieces of dead code were flagged by the
compiler, and were removed.

Change-Id: I56a5e614735c9e0a6ee420dab949eac22e25c155
BUG: 1245331
Signed-off-by: Jeff Darcy 
Reviewed-on: http://review.gluster.org/11769
Tested-by: NetBSD Build System 
Reviewed-by: Dan Lambright 
Reviewed-by: Kaleb KEITHLEY 
Reviewed-by: Niels de Vos 
Reviewed-by: Raghavendra Bhat 
Reviewed-by: Venky Shankar

gfdb/libglusterfs : Porting to a new logging framework

2015-06-10T08:54:14+00:00

Change-Id: I61874561fdf2c175d2b3eec0c85c25f12dc60552
BUG: 1194640
Signed-off-by: Mohamed Ashiq 
Reviewed-on: http://review.gluster.org/10819
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Joseph Fernandes

tier/tier.t: Fixing tier.t crash in regression runs

2015-05-27T10:58:41+00:00

1) If the database file exists
   a. Dont try re-creating the db schema
   b. Dont try re-configuring the db.

2) Dont assert in fini_db () when connection is NULL

Change-Id: I15dd103fe7542f70113c1d5e539a99f8cd062be4
BUG: 1163543
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/10870
Tested-by: NetBSD Build System
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

cluster/dht: fix tier.c problems found prior to feature freeze

2015-04-06T09:35:37+00:00

This patch resolves tiering translator issues taken from the list
in bug 1203776. These issues have been selected to be fixed
first. The rest will be fixed in a subsequent patch (or are not a
problem).

3. Replace hardcoded #defines of promote/demote file names

6. Use loc_wipe() in migrate_using_query_file()

9. Only promote/demote files on the same node on which they reside.

14. Replace calloc with GF_CALLOC in tier.c and ensure freeing done
properly.

15. Handle if parse_query_str fails

22. Only load gfdb library on server side, remove SQL references
    from client.

Change-Id: I6563b11e58ab2e4c6b1ce44db755781ad6d930fb
BUG: 1203776
Signed-off-by: Dan Lambright 
Reviewed-on: http://review.gluster.org/9987
Tested-by: Gluster Build System 
Reviewed-by: N Balachandran 
Reviewed-by: Niels de Vos