glusterfs.git/libglusterfs/src/gfdb, branch v3.7.5

tier/ctr: Solving DB Lock issue due to write contention from db connections

2015-09-11T20:46:50+00:00

This is a backport of 12031.

> Problem: The DB on the brick is been accessed by CTR, for write and
> tier migrator, for read and write. The write from tier migrator is reseting
> the heat counters after a cycle. Since we are using sqlite, two connections
> trying to write would cause a db lock contention. As a result CTR used to fail
> to update the db.

> Solution: Using the same db connection of CTR for reseting the heat counters.
> 1) Introducted a new IPC FOP for CTR
> 2) After the query do a ipc syncop to the underlying client xlator associated
>    to the brick.
> 3) CTR in brick will catch the IPC FOP and cleat the heat counters.

> Change-Id: I53306bfc08dcdba479deb4ccc154896521336150
> BUG: 1260730
> Signed-off-by: Joseph Fernandes 
> Reviewed-on: http://review.gluster.org/12031
> Tested-by: NetBSD Build System 
> Tested-by: Gluster Build System 

Signed-off-by: Dan Lambright 

Conflicts:
	xlators/cluster/dht/src/tier.c

Change-Id: I88aa289cdf21e216b42c3d8ccfb4e7e828b43772
BUG: 1262341
Reviewed-on: http://review.gluster.org/12161
Reviewed-by: Joseph Fernandes
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Tested-by: Dan Lambright

cluster/tier: add gluster v tier

2015-09-10T17:59:11+00:00

This is a backport of 11984.

> Currently the tier feature piggy backs off the rebalance command
> syntax to obtain status and this is clumsy. Introduce a new
> tier command that can do tier specific operations, starting
> with volume status to display counters.

> Old commands:
> gluster volume attach-tier  [replica count] {bricklist..}
> gluster volume detach-tier  {start|stop|commit}

> New commands:
> gluster volume tier  attach [replica count] {bricklist} |
>                           detach {start|stop|commit} |
>                           status

> Change-Id: Ic07b3c6260588162de7d34380f8cbd3d8a7f35d3
> BUG: 1255693
> Signed-off-by: Dan Lambright 
> Reviewed-on: http://review.gluster.org/11984
> Reviewed-by: Gaurav Kumar Garg 
> Tested-by: Gluster Build System 
Signed-off-by: Dan Lambright 

Change-Id: Id45bd0fa6b8606dd47863de83a694908da393229
BUG: 1261664
Signed-off-by: Dan Lambright 
Reviewed-on: http://review.gluster.org/12143
Tested-by: Gluster Build System 
Reviewed-by: Gaurav Kumar Garg

cluster/tier: fix 64 bit issue with sql query using times

2015-08-13T16:08:43+00:00

This is a backport of 11885.

> We overflowed when converting seconds to usecs in preperation for
> sql queries. The fix uses uint64_t throughout including subexpressions.

> Change-Id: I59bdb742197400dede97f54735b52030920b0d19
> BUG: 1231268
> Signed-off-by: Dan Lambright 

Change-Id: Iaef7083dac7a162080f4bc23e29dedfe700fa342
BUG: 1252903
Signed-off-by: Dan Lambright 
Reviewed-on: http://review.gluster.org/11913
Tested-by: NetBSD Build System 
Reviewed-by: Joseph Fernandes

libgfdb/ctr: Fixing rename or update of link

2015-08-12T13:46:53+00:00

Backport of http://review.gluster.org/11584 to 3.7.2

Fixing rename or updating of link in
1) libgfdb : handling return from updatelink
2) ctr xlator: correcting freeing of oldbase name in
               hardlink list in the update link

>Change-Id: I1789c492b81876123cd456051a9ae495e876ad46
>BUG: 1241054
>Signed-off-by: Joseph Fernandes 
>Reviewed-on: http://review.gluster.org/11584
>Tested-by: Joseph Fernandes
>Reviewed-by: Dan Lambright 
>Tested-by: Dan Lambright 
>Tested-by: Gluster Build System 
Signed-off-by: Joseph Fernandes 

Change-Id: I6da86c06e390f62fe8a44ade7b1efda5771e9ca8
BUG: 1242536
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/11649
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright

libgfdb/sql: Fixing broken query of find_unchanged

2015-07-18T13:46:05+00:00

 The find_unchanged query should be

          "write_heat <= defined_heat" AND "read_heat <= defined_heat"
   and not
          "write_heat <= defined_heat" OR "read_heat <= defined_heat"

>>Backport of http://review.gluster.org/#/c/11577/
>>Change-Id: Ie82e02aafbb7ea14563007307de3350ea022049a
>>BUG: 1240970
>>Signed-off-by: Joseph Fernandes 

Change-Id: Ie5d06fa2e3a3552741d2e6bd76807a69808c3d65
BUG: 1241776
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/11608
Reviewed-by: Dan Lambright 
Reviewed-by: Pamela Ousley 
Tested-by: Gluster Build System 
Tested-by: Dan Lambright

tier/ctr: Ignore creation of T file and Ctr Lookup heal improvememnts

2015-06-28T01:11:00+00:00

This is a back port of 11334

1) Ignore creation of T file in ctr_mknod
2) Ignore lookup for T file in ctr_lookup
3) Ctr_lookup:
    a. If the gfid and pgfid in empty dont record
    b. Decreased log level for multiple heal attempts
    c. Inode/File heal happens after an expiry period, which is configurable.
    d. Hardlink heal happens after an expiry period, which is configurable.

> Change-Id: Id8eb5092e78beaec22d05f5283645081619e2452
> BUG: 1235269
> Signed-off-by: Joseph Fernandes 
> Reviewed-on: http://review.gluster.org/11334
> Tested-by: Gluster Build System 
> Reviewed-by: Dan Lambright 
> Tested-by: Dan Lambright 

Change-Id: Ia28a5cf975e41d318906f707deca447aaa35630f
BUG: 1236288
Signed-off-by: Dan Lambright 
Reviewed-on: http://review.gluster.org/11446
Reviewed-by: Joseph Fernandes
Tested-by: Joseph Fernandes
Tested-by: Gluster Build System

gfdb/libglusterfs : Port log messages to a new framework

2015-06-26T12:58:08+00:00

        Backport of http://review.gluster.org/#/c/10819/

Cherry picked from 2847b03a38515a625e36d52c7e0a1d37107d72ba
>Change-Id: I61874561fdf2c175d2b3eec0c85c25f12dc60552
>BUG: 1194640
>Signed-off-by: Mohamed Ashiq 
>Reviewed-on: http://review.gluster.org/10819
>Tested-by: NetBSD Build System 
>Tested-by: Gluster Build System 
>Reviewed-by: Joseph Fernandes

Change-Id: I61874561fdf2c175d2b3eec0c85c25f12dc60552
BUG: 1217722
Signed-off-by: Mohamed Ashiq 
Reviewed-on: http://review.gluster.org/11284
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

tier/tier.t: Fixing tier.t crash in regression runs

2015-05-30T15:07:35+00:00

1) If the database file exists
   a. Dont try re-creating the db schema
   b. Dont try re-configuring the db.

2) Dont assert in fini_db () when connection is NULL

>> Signed-off-by: Joseph Fernandes 
>> Reviewed-on: http://review.gluster.org/10870
>> Tested-by: NetBSD Build System
>> Tested-by: Gluster Build System 
>> Reviewed-by: Vijay Bellur 
>> Signed-off-by: Joseph Fernandes 
>>Change-Id: Idd2833f55caaa6b3a77d935d877d6c4d2994da6a

Change-Id: Ia2e044fce4e41bccc8fdada1cb21f240fdbd55df
BUG: 1225077
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/10992
Reviewed-by: Vijay Bellur 
Tested-by: Vijay Bellur

CTR/Libgfdb: Log typo fix

2015-05-10T09:30:18+00:00

Log typo fix for CTR Xlator and Libgfdb

Change-Id: Ia39069a5ce9c48bbee937f1b5c5d749a30c9ac56
BUG: 1220100
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/10742
Reviewed-by: N Balachandran 
Tested-by: Gluster Build System

ctr/xlator: Named lookup heal of pre-existing files, before ctr was ON.

2015-05-08T07:34:30+00:00

Problem: The CTR xlator records file meta (heat/hardlinks)
into the data. This works fine for files which are created
after ctr xlator is switched ON. But for files which were
created before CTR xlator is ON, CTR xlator is not able to
record either of the meta i.e heat or hardlinks. Thus making
those files immune to promotions/demotions.

Solution: The solution that is implemented in this patch is
do ctr-db heal of all those pre-existent files, using named lookup.
For this purpose we use the inode-xlator context variable option
in gluster.
The inode-xlator context variable for ctr xlator will have the
following,
    a. A Lock for the context variable
    b. A hardlink list: This list represents the successful looked
       up hardlinks.
These are the scenarios when the hardlink list is updated:
1) Named-Lookup: Whenever a named lookup happens on a file, in the
   wind path we copy all required hardlink and inode information to
   ctr_db_record structure, which resides in the frame->local variable.
   We dont update the database in wind. During the unwind, we read the
   information from the ctr_db_record and ,
   Check if the inode context variable is created, if not we create it.
   Check if the hard link is there in the hardlink list.
      If its not there we add it to the list and send a update to the
      database using libgfdb.
      Please note: The database transaction can fail(and we ignore) as there
      already might be a record in the db. This update to the db is to heal
      if its not there.
      If its there in the list we ignore it.
2) Inode Forget: Whenever an inode forget hits we clear the hardlink list in
   the inode context variable and delete the inode context variable.
   Please note: An inode forget may happen for two reason,
   a. when the inode is delete.
   b. the in-memory inode is evicted from the inode table due to cache limits.
3) create: whenever a create happens we create the inode context variable and
   add the hardlink. The database updation is done as usual by ctr.
4) link: whenever a hardlink is created for the inode, we create the inode context
   variable, if not present, and add the hardlink to the list.
5) unlink: whenever a unlink happens we delete the hardlink from the list.
6) mknod: same as create.
7) rename: whenever a rename happens we update the hardlink in list. if the hardlink
   was not present for updation, we add the hardlink to the list.

What is pending:
1) This solution will only work for named lookups.
2) We dont track afr-self-heal/dht-rebalancer traffic for healing.

> http://review.gluster.org/#/c/10370/
> Cherry picked from commit cb11dd91a6cc296e4a3808364077f4eacb810e48
> Change-Id: Ia4bbaf84128ad6ce8c3ddd70bcfa82894c79585f
> BUG: 1212037
> Signed-off-by: Joseph Fernandes 
> Signed-off-by: Dan Lambright 
> Reviewed-on: http://review.gluster.org/10370
> Reviewed-by: Jeff Darcy 
> Tested-by: Gluster Build System 
> Tested-by: NetBSD Build System
> Reviewed-by: Vijay Bellur 

Change-Id: I367aa46c3f4b8f912248fb8be75866507f2538df
BUG: 1219075
Signed-off-by: Joseph Fernandes 
Signed-off-by: Dan Lambright 
Reviewed-on: http://review.gluster.org/10370
Reviewed-by: Jeff Darcy 
Reviewed-by: Vijay Bellur 
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/10615
Tested-by: Vijay Bellur