glusterfs.git/libglusterfs/src/gfdb/Makefile.am, branch v3.8.10

tier/libgfdb: Replacing ASCII query file with binary

2015-11-06T18:30:45+00:00

Earlier, when the database was queried we used to save
all the queried records in an ASCII format in the query file.
This caused issues like filename having ASCII delimiter and used
to take a lot of space. The tier.c file also had a lot of parsing code.

Here we changed the format of the query file to binary.
All the logic of serialization and formating of query record is done
by libgfdb. Libgfdb provides API,
gfdb_write_query_record() and gfdb_read_query_record(),
which the user i.e tier migrator and CTR xlator can use to
write to and read from query file.
With this binary format we save on disk space i.e reduce to 50% atleast
as we are saving GFID's in binary format 16 bytes and not the string format
which takes 36 bytes + We are not saving path of the file + we are also saving on
ASCII delimiters.

The on disk format of query record is as follows,

+---------------------------------------------------------------------------+
| Length of serialized query record |       Serialized Query Record         |
+---------------------------------------------------------------------------+
             4 bytes                     Length of serialized query record
                                                      |
                                                      |
     -------------------------------------------------|
     |
     |
     V
   Serialized Query Record Format:
   +---------------------------------------------------------------------------+
   | GFID |  Link count   |    |.....                      | FOOTER |
   +---------------------------------------------------------------------------+
     16 B        4 B         Link Length                                  4 B
                                |                                          |
                                |                                          |
   -----------------------------|                                          |
   |                                                                       |
   |                                                                       |
   V                                                                       |
   Each  will be serialized as                                  |
   +-----------------------------------------------+                       |
   | PGID | BASE_NAME_LENGTH |      BASE_NAME      |                       |
   +-----------------------------------------------+                       |
     16 B       4 B             BASE_NAME_LENGTH                           |
                                                                           |
                                                                           |
   ------------------------------------------------------------------------|
   |
   |
   V
   FOOTER is a magic number 0xBAADF00D indicating the end of the record.
   This also serves as a serialized schema validator.

Change-Id: I9db7416fd421e118dd44eafab8b535caafe50d5a
BUG: 1272207
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/12354
Reviewed-by: N Balachandran 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

build: make contrib/uuid dependency optional

2015-04-10T11:39:52+00:00

On Linux systems we should use the libuuid from the distribution and not
bundle and statically link the contrib/uuid/ bits.

libglusterfs/src/compat-uuid.h has been introduced and should become an
abstraction layer for different UUID APIs. Non-Linux operating systems
should implement their compatibility layer there.

Once all operating systems have an implementation in compat-uuid.h, we
can remove contrib/uuid/ from the repository completely.

Change-Id: I345e5357644be2521685e00358bb8c83c4ea0577
BUG: 1206587
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/10129
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

build: pass the correct CFLAGS and LDFLAGS when linking (to) libgfdb

2015-03-19T20:16:54+00:00

Change-Id: Id9a7d0f457d9759ab7d0a52a4000b5ae36d211f8
BUG: 1194753
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/9946
Tested-by: Gluster Build System 
Reviewed-by: Kaleb KEITHLEY

Adding Libgfdb to GlusterFS

2015-03-18T17:36:42+00:00

*************************************************************************
			Libgfdb						|
*************************************************************************
Libgfdb provides abstract mechanism to record extra/rich metadata
required for data maintenance, such as data tiering/classification.
It provides consumer with API for recording and querying, keeping
the consumer abstracted from the data store used beneath for storing data.
It works in a plug-and-play model, where data stores can be plugged-in.
Presently we have plugin for Sqlite3. In the future will provide recording
and querying performance optimizer. In the current implementation the schema
of metadata is fixed.

Schema:
~~~~~~
      GF_FILE_TB Table:
      ~~~~~~~~~~~~~~~~~
      This table has one entry per file inode. It holds the metadata required to
      make decisions in data maintenance.
      GF_ID (Primary key)	: File GFID (Universal Unique IDentifier in the namespace)
      W_SEC, W_MSEC 		: Write wind time in sec & micro-sec
      UW_SEC, UW_MSEC		: Write un-wind time in sec & micro-sec
      W_READ_SEC, W_READ_MSEC 	: Read wind time in sec & micro-sec
      UW_READ_SEC, UW_READ_MSEC : Read un-wind time in sec & micro-sec
      WRITE_FREQ_CNTR INTEGER	: Write Frequency Counter
      READ_FREQ_CNTR INTEGER	: Read Frequency Counter

      GF_FLINK_TABLE:
      ~~~~~~~~~~~~~~
      This table has all the hardlinks to a file inode.
      GF_ID		: File GFID               (Composite Primary Key)``|
      GF_PID		: Parent Directory GFID  (Composite Primary Key)   |-> Primary Key
      FNAME 		: File Base Name          (Composite Primary Key)__|
      FPATH 		: File Full Path (Its redundant for now, this will go)
      W_DEL_FLAG 	: This Flag is used for crash consistancy, when a link is unlinked.
                  	  i.e Set to 1 during unlink wind and during unwind this record
                          is deleted
      LINK_UPDATE 	: This Flag is used when a link is changed i.e rename.
                          Set to 1 when rename wind and set to 0 in rename unwind

Libgfdb API:
~~~~~~~~~~~
Refer libglusterfs/src/gfdb/gfdb_data_store.h

Change-Id: I2e9fbab3878ce630a7f41221ef61017dc43db11f
BUG: 1194753
Signed-off-by: Joseph Fernandes 
Signed-off-by: Dan Lambright 
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/9683
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur