glusterfs.git/libglusterfs/src/gfdb/gfdb_data_store.c, branch v3.7.20

all: reduce "inline" usage

2016-01-18T09:02:34+00:00

There are three kinds of inline functions: plain inline, extern inline,
and static inline.  All three have been removed from .c files, except
those in "contrib" which aren't our problem.  Inlines in .h files, which
are overwhelmingly "static inline" already, have generally been left
alone.  Over time we should be able to "lower" these into .c files, but
that has to be done in a case-by-case fashion requiring more manual
effort.  This part was easy to do automatically without (as far as I can
tell) any ill effect.

In the process, several pieces of dead code were flagged by the
compiler, and were removed.

backport of Change-Id: I56a5e614735c9e0a6ee420dab949eac22e25c155,
http://review.gluster.org/11769, BUG: 1245331

Change-Id: Iba1efb0bc578ea4a5e9bf76b7bd93dc1be9eba44
BUG: 1283302
Signed-off-by: Kaleb S KEITHLEY 
Reviewed-on: http://review.gluster.org/12646
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

ctr/sql: Providing for vol set for sqlcachesize and sqlWALsize and skip recording path

2015-12-22T22:45:20+00:00

1. Providing vol set option for cache size and wal autocheck point
   so that performance can be tuned.
2. Removed recording of file path in the db. Trimming database columns.
   Path need not be stored in the db, as PARGFID, GFID, Basename is suffice
   to derive the path during migration.

Backport of http://review.gluster.org/12972

> Change-Id: I2cb590451a6d244bc91fe66c6dbffe2c2059dfb8
> BUG: 1293034
> Signed-off-by: Joseph Fernandes 
> Reviewed-on: http://review.gluster.org/12972
> Reviewed-by: N Balachandran 
> Tested-by: Gluster Build System 
> Tested-by: NetBSD Build System 
> Reviewed-by: Dan Lambright 
> Tested-by: Dan Lambright 
Signed-off-by: Joseph Fernandes 

Change-Id: Ia1c109983ec6ce75ed27b8c08f454f5b6283c31d
BUG: 1293659
Reviewed-on: http://review.gluster.org/13067
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

tier/libgfdb: Replacing ASCII query file with binary

2015-11-08T13:22:02+00:00

Earlier, when the database was queried we used to save
all the queried records in an ASCII format in the query file.
This caused issues like filename having ASCII delimiter and used
to take a lot of space. The tier.c file also had a lot of parsing code.

Here we changed the format of the query file to binary.
All the logic of serialization and formating of query record is done
by libgfdb. Libgfdb provides API,
gfdb_write_query_record() and gfdb_read_query_record(),
which the user i.e tier migrator and CTR xlator can use to
write to and read from query file.
With this binary format we save on disk space i.e reduce to 50% atleast
as we are saving GFID's in binary format 16 bytes and not the string format
which takes 36 bytes + We are not saving path of the file + we are also saving on
ASCII delimiters.

The on disk format of query record is as follows,

+---------------------------------------------------------------------------+
| Length of serialized query record |       Serialized Query Record         |
+---------------------------------------------------------------------------+
             4 bytes                     Length of serialized query record
                                                      |
                                                      |
     -------------------------------------------------|
     |
     |
     V
   Serialized Query Record Format:
   +---------------------------------------------------------------------------+
   | GFID |  Link count   |    |.....                      | FOOTER |
   +---------------------------------------------------------------------------+
     16 B        4 B         Link Length                                  4 B
                                |                                          |
                                |                                          |
   -----------------------------|                                          |
   |                                                                       |
   |                                                                       |
   V                                                                       |
   Each  will be serialized as                                  |
   +-----------------------------------------------+                       |
   | PGID | BASE_NAME_LENGTH |      BASE_NAME      |                       |
   +-----------------------------------------------+                       |
     16 B       4 B             BASE_NAME_LENGTH                           |
                                                                           |
                                                                           |
   ------------------------------------------------------------------------|
   |
   |
   V
   FOOTER is a magic number 0xBAADF00D indicating the end of the record.
   This also serves as a serialized schema validator.

Backport of http://review.gluster.org/#/c/12354/

> Change-Id: I9db7416fd421e118dd44eafab8b535caafe50d5a
> BUG: 1272207
> Signed-off-by: Joseph Fernandes 
> Reviewed-on: http://review.gluster.org/12354
> Reviewed-by: N Balachandran 
> Tested-by: Gluster Build System 
> Reviewed-by: Dan Lambright 
> Tested-by: Dan Lambright 

Change-Id: I170c579027f2594a58706f826e3ddf89e34022f4
BUG: 1263619
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/12535
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

tier/ctr: Ignore CTR Lookup heal insert errors

2015-11-07T18:32:33+00:00

CTR doesnt read from the DB, so to make sure that file records are
created it does a heal during a lookup. It remembers the decision in
the inode context cache and retrys periodically. When the volume is
restarted it looses all the inode cache from the previous time and CTR
lookup heals tries the heal again, but this time it finds that the records
are already there from sql and logs this error, and remembers this until the
volume is restarted or inode is flushed out of inode cache of the brick.

Solution: the log levels should be reduced to trace for this case and
customers need not see this.

Backport of http://review.gluster.org/#/c/12491/

Change-Id: I5f11e83c306ae912a90f12220f62bb7563b8eb96
BUG: 1279059
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/12534
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

tier/ctr: Solution for db locks for tier migrator and ctr using sqlite version less than 3.7 i.e rhel 6.7

2015-10-09T23:58:53+00:00

Problem: On RHEL 6.7, we have sqlite version 3.6.2 which doesnt support
WAL journaling mode, as this journaling mode is only available in sqlite 3.7 and above.
As a result we cannot have to progreses concurrently accessing sqlite, without
running into db locks! Well WAL is also need for performace on CTR side.

Solution: This solution is to use CTR db connection for doing queries when WAL mode is
absent. i,e tier migrator will send sync_op ipc calls to CTR, which in turn will
do the query and create/update the query file suggested by tier migrator.

Pending: Well this solution will stop the db locks but the performance is still an issue for CTR.
We are developing an in-Memory Transaction Log (iMeTaL) which will help boost the CTR
performance by doing in memory udpates on the IO path and later flush the updates to
the db in a batch/segment flush.

Master patch: http://review.gluster.org/#/c/12191
>> Change-Id: Ie3149643ded159234b5cc6aa6cf93b9022c2f124
>> BUG: 1240577
>> Signed-off-by: Joseph Fernandes 
>> Signed-off-by: Dan Lambright 
>> Signed-off-by: Joseph Fernandes 
>> Reviewed-on: http://review.gluster.org/12191
>> Tested-by: Gluster Build System 
>> Reviewed-by: Luis Pabon 
Signed-off-by: Joseph Fernandes 

Change-Id: Ie8c7a7e9566244c104531b579126bb57fbc6e32b
BUG: 1270123
Reviewed-on: http://review.gluster.org/12325
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

tier/ctr: Solving DB Lock issue due to write contention from db connections

2015-09-11T20:46:50+00:00

This is a backport of 12031.

> Problem: The DB on the brick is been accessed by CTR, for write and
> tier migrator, for read and write. The write from tier migrator is reseting
> the heat counters after a cycle. Since we are using sqlite, two connections
> trying to write would cause a db lock contention. As a result CTR used to fail
> to update the db.

> Solution: Using the same db connection of CTR for reseting the heat counters.
> 1) Introducted a new IPC FOP for CTR
> 2) After the query do a ipc syncop to the underlying client xlator associated
>    to the brick.
> 3) CTR in brick will catch the IPC FOP and cleat the heat counters.

> Change-Id: I53306bfc08dcdba479deb4ccc154896521336150
> BUG: 1260730
> Signed-off-by: Joseph Fernandes 
> Reviewed-on: http://review.gluster.org/12031
> Tested-by: NetBSD Build System 
> Tested-by: Gluster Build System 

Signed-off-by: Dan Lambright 

Conflicts:
	xlators/cluster/dht/src/tier.c

Change-Id: I88aa289cdf21e216b42c3d8ccfb4e7e828b43772
BUG: 1262341
Reviewed-on: http://review.gluster.org/12161
Reviewed-by: Joseph Fernandes
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Tested-by: Dan Lambright

gfdb/libglusterfs : Port log messages to a new framework

2015-06-26T12:58:08+00:00

        Backport of http://review.gluster.org/#/c/10819/

Cherry picked from 2847b03a38515a625e36d52c7e0a1d37107d72ba
>Change-Id: I61874561fdf2c175d2b3eec0c85c25f12dc60552
>BUG: 1194640
>Signed-off-by: Mohamed Ashiq 
>Reviewed-on: http://review.gluster.org/10819
>Tested-by: NetBSD Build System 
>Tested-by: Gluster Build System 
>Reviewed-by: Joseph Fernandes

Change-Id: I61874561fdf2c175d2b3eec0c85c25f12dc60552
BUG: 1217722
Signed-off-by: Mohamed Ashiq 
Reviewed-on: http://review.gluster.org/11284
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

tier/tier.t: Fixing tier.t crash in regression runs

2015-05-30T15:07:35+00:00

1) If the database file exists
   a. Dont try re-creating the db schema
   b. Dont try re-configuring the db.

2) Dont assert in fini_db () when connection is NULL

>> Signed-off-by: Joseph Fernandes 
>> Reviewed-on: http://review.gluster.org/10870
>> Tested-by: NetBSD Build System
>> Tested-by: Gluster Build System 
>> Reviewed-by: Vijay Bellur 
>> Signed-off-by: Joseph Fernandes 
>>Change-Id: Idd2833f55caaa6b3a77d935d877d6c4d2994da6a

Change-Id: Ia2e044fce4e41bccc8fdada1cb21f240fdbd55df
BUG: 1225077
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/10992
Reviewed-by: Vijay Bellur 
Tested-by: Vijay Bellur

cluster/dht: fix tier.c problems found prior to feature freeze

2015-04-06T09:35:37+00:00

This patch resolves tiering translator issues taken from the list
in bug 1203776. These issues have been selected to be fixed
first. The rest will be fixed in a subsequent patch (or are not a
problem).

3. Replace hardcoded #defines of promote/demote file names

6. Use loc_wipe() in migrate_using_query_file()

9. Only promote/demote files on the same node on which they reside.

14. Replace calloc with GF_CALLOC in tier.c and ensure freeing done
properly.

15. Handle if parse_query_str fails

22. Only load gfdb library on server side, remove SQL references
    from client.

Change-Id: I6563b11e58ab2e4c6b1ce44db755781ad6d930fb
BUG: 1203776
Signed-off-by: Dan Lambright 
Reviewed-on: http://review.gluster.org/9987
Tested-by: Gluster Build System 
Reviewed-by: N Balachandran 
Reviewed-by: Niels de Vos

Adding Libgfdb to GlusterFS

2015-03-18T17:36:42+00:00

*************************************************************************
			Libgfdb						|
*************************************************************************
Libgfdb provides abstract mechanism to record extra/rich metadata
required for data maintenance, such as data tiering/classification.
It provides consumer with API for recording and querying, keeping
the consumer abstracted from the data store used beneath for storing data.
It works in a plug-and-play model, where data stores can be plugged-in.
Presently we have plugin for Sqlite3. In the future will provide recording
and querying performance optimizer. In the current implementation the schema
of metadata is fixed.

Schema:
~~~~~~
      GF_FILE_TB Table:
      ~~~~~~~~~~~~~~~~~
      This table has one entry per file inode. It holds the metadata required to
      make decisions in data maintenance.
      GF_ID (Primary key)	: File GFID (Universal Unique IDentifier in the namespace)
      W_SEC, W_MSEC 		: Write wind time in sec & micro-sec
      UW_SEC, UW_MSEC		: Write un-wind time in sec & micro-sec
      W_READ_SEC, W_READ_MSEC 	: Read wind time in sec & micro-sec
      UW_READ_SEC, UW_READ_MSEC : Read un-wind time in sec & micro-sec
      WRITE_FREQ_CNTR INTEGER	: Write Frequency Counter
      READ_FREQ_CNTR INTEGER	: Read Frequency Counter

      GF_FLINK_TABLE:
      ~~~~~~~~~~~~~~
      This table has all the hardlinks to a file inode.
      GF_ID		: File GFID               (Composite Primary Key)``|
      GF_PID		: Parent Directory GFID  (Composite Primary Key)   |-> Primary Key
      FNAME 		: File Base Name          (Composite Primary Key)__|
      FPATH 		: File Full Path (Its redundant for now, this will go)
      W_DEL_FLAG 	: This Flag is used for crash consistancy, when a link is unlinked.
                  	  i.e Set to 1 during unlink wind and during unwind this record
                          is deleted
      LINK_UPDATE 	: This Flag is used when a link is changed i.e rename.
                          Set to 1 when rename wind and set to 0 in rename unwind

Libgfdb API:
~~~~~~~~~~~
Refer libglusterfs/src/gfdb/gfdb_data_store.h

Change-Id: I2e9fbab3878ce630a7f41221ef61017dc43db11f
BUG: 1194753
Signed-off-by: Joseph Fernandes 
Signed-off-by: Dan Lambright 
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/9683
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur