glusterfs.git/tests/basic/tier, branch v3.7.4

glusterd/tier: configure tier daemon during volume restart

2015-08-19T08:11:03+00:00

rebalance daemon will be running on every tier volume for
promoting/demoting the files. When volume/glusterd is restarted,
then we need to configure the daemon.

Back port of >
>Change-Id: Ib565240a70edea2ec8bc1601c52b40c0783491d3
>BUG: 1225330
>Signed-off-by: Mohammed Rafi KC 
>Signed-off-by: Joseph Fernandes 
>Reviewed-on: http://review.gluster.org/10933
>Reviewed-by: Dan Lambright 
>Tested-by: Dan Lambright 

(cherry picked from commit 7aaa414c67be460aca3c1c5eed047e3db17bd435)

Change-Id: Id15f5a18fd2ebaf8642b02561ae3666e619e3f83
BUG: 1235202
Signed-off-by: Mohammed Rafi KC 
Reviewed-on: http://review.gluster.org/11376
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

tiering/rebalance: tier daemon stopped with out updating status

2015-06-26T15:36:44+00:00

When a subvol goes down, tier daemon stopped immediately, and
the status shows as "Progressing".

With this change, with respect to tier xlator, when a subvol
goes offline it will update the status as failed.

Back port of:
>Change-Id: I9f722ed0d35cda8c7fc1a7e75af52222e2d0fdb7
>BUG: 1227803
>Signed-off-by: Mohammed Rafi KC 
>Reviewed-on: http://review.gluster.org/11068
>Tested-by: NetBSD Build System 
>Reviewed-by: Dan Lambright 
>Tested-by: Dan Lambright 

(cherry picked from commit d3714f252d91f4d1d5df05c4dcc8bc7c2ee75326)


Change-Id: I268b7e2631779d15db5d9aec495dfd00ded94c5c
BUG: 1235203
Signed-off-by: Mohammed Rafi KC 
Reviewed-on: http://review.gluster.org/11414
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

tier/volume set: Validate volume set option for tier

2015-06-17T05:13:18+00:00

Volume set option related to tier volume can only be set
for tier volume, also currently all volume set i for tier
option accepts a non-negative integer. This patch validate
both condition.

Back port of:
>Change-Id: I3611af048ff4ab193544058cace8db205ea92336
>BUG: 1216960
>Signed-off-by: Mohammed Rafi KC 
>Signed-off-by: Dan Lambright 
>Reviewed-on: http://review.gluster.org/10751
>Tested-by: Gluster Build System 
>Tested-by: NetBSD Build System 
>Reviewed-by: Joseph Fernandes


(cherry picked from commit f6a062044a3447bea5bf0fcf21a3f85c00fb6c7d)

Change-Id: Ic6081f0ce7ae7effac69ba192bd35c8d382a11d5
BUG: 1230560
Signed-off-by: Mohammed Rafi KC 
Reviewed-on: http://review.gluster.org/11173
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Joseph Fernandes
Tested-by: Joseph Fernandes

tiering/nfs: duplication of nodes in client graph

2015-06-01T04:40:47+00:00

        Back port of http://review.gluster.org/10820

When creating client volfiles, xlator tier-dht will
be loaded for each volume. So for services like nfs
have one or more volumes . So for each volume in the
graph a tier-dht xlator will be created. So the graph
parser will fail because of the redundant node in
graph.

By this change tier-dht will be renamed as volname-tier-dht

 >Change-Id: I3c9b9c23ddcb853773a8a02be7fd8a5d09a7f972
 >BUG: 1222840
 >Signed-off-by: Mohammed Rafi KC 
 >Reviewed-on: http://review.gluster.org/10820
 >Reviewed-by: Atin Mukherjee 
 >Tested-by: Gluster Build System 
 >Tested-by: NetBSD Build System
 >Reviewed-by: Kaushal M 

Change-Id: I5629d48d4d1dbec8790f75e2fee66729aa2f6eed
BUG: 1226029
Signed-off-by: Mohammed Rafi KC 
Reviewed-on: http://review.gluster.org/10981
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Joseph Fernandes
Reviewed-by: Kaushal M

tiering: Send both attach-tier and tier-start together

2015-05-10T04:51:36+00:00

        Back port of http://review.gluster.org/10363

After attaching tier, we have to start tier rebalance process.
This patch is to trigger tier start along with attch-tier.

>Change-Id: I39380f95123f0087a82213ef263f9f33adcc5adc
>BUG: 1214222
>Signed-off-by: Mohammed Rafi KC 
>Reviewed-on: http://review.gluster.org/10363
>Tested-by: Gluster Build System 
>Reviewed-by: Dan Lambright 

Change-Id: Ia27e97a9dc0772f3f56b991ce4019281ab7bf1b5
BUG: 1219848
Signed-off-by: Mohammed Rafi KC 
Reviewed-on: http://review.gluster.org/10711
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System
Reviewed-by: Vijay Bellur

tiering/cli: Check replica count and bricks are proper or not

2015-05-10T03:25:13+00:00

        Back port of http://review.gluster.org/10428

Right now, attach-tier calls parsing function for add-brick.
Add-brick does not have any check for brick count and replca
count compatibility.

>Change-Id: I44ec13eadffc003a9ebf8c4eb0193df559933a68
>BUG: 1215122
>Signed-off-by: Mohammed Rafi KC 
>Reviewed-on: http://review.gluster.org/10428
>Reviewed-by: Dan Lambright 
>Tested-by: Gluster Build System 
>Reviewed-by: Vijay Bellur 

Change-Id: Ie86d5b61b044386a062bfc9782178853e3af0f11
BUG: 1219850
Signed-off-by: Mohammed Rafi KC 
Reviewed-on: http://review.gluster.org/10710
Reviewed-by: Dan Lambright 
Tested-by: Gluster Build System

cli/tiering: Enhance cli output for tiering

2015-05-09T17:26:36+00:00

        back port of http://review.gluster.org/10284

Fix for handling cli output for attach-tier and
detach-tier

>Change-Id: I4d17f4b09612754fe1b8cec6c2e14927029b9678
>BUG: 1211562
>Signed-off-by: Mohammed Rafi KC 
>Reviewed-on: http://review.gluster.org/10284
>Reviewed-by: Dan Lambright 
>Tested-by: Gluster Build System 
>Tested-by: NetBSD Build System
>Reviewed-by: Vijay Bellur 

Change-Id: Ic30c8f0a104d89bf98f5d0069937a34674ee3f2d
BUG: 1220052
Signed-off-by: Mohammed Rafi KC 
Reviewed-on: http://review.gluster.org/10713
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

glusterd: support for tier volumes 'detach start' and 'detach commit'

2015-05-09T15:35:14+00:00

        Back port of http://review.gluster.org/10108

These commands work in a manner analagous to rebalancing when removing a
brick. The existing migration daemon detects "detach start" and switches
to moving data off the hot tier. While in this state all lookups are
directed to the cold tier.

gluster v detach-tier  start
gluster v detach-tier  commit

The status and stop cli commands shall be submitted separately.

>Change-Id: I24fda5cc3ba74f5fb8aa9a3234ad51f18b80a8a0
>BUG: 1205540
>Signed-off-by: Dan Lambright 
>Signed-off-by: root 
>Signed-off-by: Dan Lambright 
>Reviewed-on: http://review.gluster.org/10108
>Reviewed-by: Kaleb KEITHLEY 

Change-Id: I212d748d077fb5870ee84b316c653acbafbea3f7
BUG: 1220047
Signed-off-by: Mohammed Rafi KC 
Reviewed-on: http://review.gluster.org/10708
Reviewed-by: Dan Lambright 
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

ctr/xlator: Named lookup heal of pre-existing files, before ctr was ON.

2015-05-08T07:34:30+00:00

Problem: The CTR xlator records file meta (heat/hardlinks)
into the data. This works fine for files which are created
after ctr xlator is switched ON. But for files which were
created before CTR xlator is ON, CTR xlator is not able to
record either of the meta i.e heat or hardlinks. Thus making
those files immune to promotions/demotions.

Solution: The solution that is implemented in this patch is
do ctr-db heal of all those pre-existent files, using named lookup.
For this purpose we use the inode-xlator context variable option
in gluster.
The inode-xlator context variable for ctr xlator will have the
following,
    a. A Lock for the context variable
    b. A hardlink list: This list represents the successful looked
       up hardlinks.
These are the scenarios when the hardlink list is updated:
1) Named-Lookup: Whenever a named lookup happens on a file, in the
   wind path we copy all required hardlink and inode information to
   ctr_db_record structure, which resides in the frame->local variable.
   We dont update the database in wind. During the unwind, we read the
   information from the ctr_db_record and ,
   Check if the inode context variable is created, if not we create it.
   Check if the hard link is there in the hardlink list.
      If its not there we add it to the list and send a update to the
      database using libgfdb.
      Please note: The database transaction can fail(and we ignore) as there
      already might be a record in the db. This update to the db is to heal
      if its not there.
      If its there in the list we ignore it.
2) Inode Forget: Whenever an inode forget hits we clear the hardlink list in
   the inode context variable and delete the inode context variable.
   Please note: An inode forget may happen for two reason,
   a. when the inode is delete.
   b. the in-memory inode is evicted from the inode table due to cache limits.
3) create: whenever a create happens we create the inode context variable and
   add the hardlink. The database updation is done as usual by ctr.
4) link: whenever a hardlink is created for the inode, we create the inode context
   variable, if not present, and add the hardlink to the list.
5) unlink: whenever a unlink happens we delete the hardlink from the list.
6) mknod: same as create.
7) rename: whenever a rename happens we update the hardlink in list. if the hardlink
   was not present for updation, we add the hardlink to the list.

What is pending:
1) This solution will only work for named lookups.
2) We dont track afr-self-heal/dht-rebalancer traffic for healing.

> http://review.gluster.org/#/c/10370/
> Cherry picked from commit cb11dd91a6cc296e4a3808364077f4eacb810e48
> Change-Id: Ia4bbaf84128ad6ce8c3ddd70bcfa82894c79585f
> BUG: 1212037
> Signed-off-by: Joseph Fernandes 
> Signed-off-by: Dan Lambright 
> Reviewed-on: http://review.gluster.org/10370
> Reviewed-by: Jeff Darcy 
> Tested-by: Gluster Build System 
> Tested-by: NetBSD Build System
> Reviewed-by: Vijay Bellur 

Change-Id: I367aa46c3f4b8f912248fb8be75866507f2538df
BUG: 1219075
Signed-off-by: Joseph Fernandes 
Signed-off-by: Dan Lambright 
Reviewed-on: http://review.gluster.org/10370
Reviewed-by: Jeff Darcy 
Reviewed-by: Vijay Bellur 
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/10615
Tested-by: Vijay Bellur

Tests: use a portable way to flush kernel cache

2015-05-08T03:10:49+00:00

On Linux, kernel cache can be flushed using
echo 3 > /proc/sys/vm/drop_caches

This non-portable approach can be replaced by an on-purpose
failed attempt to unmount: if the mount point is the current
directory and umount is called, the kernel will flush inodes
until it realize it cannot complete the operation because
root of filesystem is busy:
( cd $M0 ; umount $M0 )

Unfortunately this does not flush everything. Entries may
still be present in the kenrel FUSE cache. Using $GFS to
mount the filesystem ensure --entry-timeout=0 and clears
this problem.

Some stall information may also remain in glusterfs caches,
and that may have to be adressed by appropriate volume option.
For instance tests/bugs/rpc/bug-954057.t needs to disable
performance.stat-prefetch. Qtherwise, root's new credentials
are not evaluated after root-quash is enabled. The test could
also be done with performance.stat-prefetch enabled using
various tricks: copying the file to read, creating a hard link
on it, or just waiting long enough for metadata cache to expire.

Backport of: I54929e899d55c04dcd9d947809133549f01fd0e1
BUG: 1212676
Signed-off-by: Emmanuel Dreyfus 

Change-Id: I2849a27acaa0334ef30aae3b852019b5a6eeb419
Reviewed-on: http://review.gluster.org/10648
Reviewed-by: Niels de Vos 
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System