glusterfs.git/xlators/cluster/dht/src/dht-layout.c, branch v3.6.3

cluster/dht: Do layout self healing of directory for nameless lookup

2014-06-17T12:39:22+00:00

Problem: Currently in the  nameless lookup code path, if at the
         end of the lookup, even if it detects that layout
         anamolies are there, layout healing will not be done as
         there is no code to heal it.
         So there can be race between mkdir and lookup.

         Assume mkdir is going on from some other mount point,
         Say, M1. Directories are created on some nodes but layout
         is not set yet.

         Now from M2, nameless lookup goes, lookup will be success
         full as the directory is present on some of the nodes, but
         it won't heal layout. Now if create goes after lookup fop,
         because layout is absent, file creation will fail.

Fix:     Included the code of layout self-heal in the nameless
         lookup path. At the end of lookup, layout will be computed
         as it would have been in the named lookup, but it will be
         set to those node only, where directory is present.
         So after that if create fop goes, the probabiliy to get the
         subvolume with proper hash-range is high now, so reduces
         the race window.

Other:  Whenever a directory is created, we have to choose a brick
        from which we start allocating layout in a circular fashion.
        To calculate this starting brick, I have changed the candidate
        from name of the directory to gfid of the directory

        But to compute where a given file belongs, we will still
        use the name of the file. Hash computed from the name of the
        file should belong to any one of the directory-hash-range

        Calculation of hash for a file is acting as a consumer and the
        setting of directory layout based on gfid is acting as a producer,
        which are independent from each other.

Change-Id: I3808c55082cd1b5c72d2c77cbbc063f55aa38bee
BUG: 1095888
Signed-off-by: Venkatesh Somyajulu 
Reviewed-on: http://review.gluster.org/7493
Tested-by: Gluster Build System 
Reviewed-by: Raghavendra Bhat 
Reviewed-by: Vijay Bellur

Cluster/DHT: New logging framework

2014-06-16T13:25:51+00:00

Moved all relevant DHT gf_log calls to the new logging
framework.

Change-Id: I3af3cfe0416e332774a6c4ff6a091d006c400af2
BUG: 1075611
Signed-off-by: Nithya Balachandran 
Reviewed-on: http://review.gluster.org/7929
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

build: Remove cmockery2 from repo

2014-03-17T18:12:08+00:00

While we wait for cmockery2 to be available from Fedora,
we can remove cmockery2 from the repo.

BUG: 1077011

Change-Id: I75d462c607cd376a5d838ea83f4d12eb59757e73
Signed-off-by: Luis Pabon 
Reviewed-on: http://review.gluster.org/7281
Reviewed-by: Justin Clift 
Tested-by: Gluster Build System 
Reviewed-by: Harshavardhana 
Reviewed-by: Niels de Vos 
Reviewed-by: Anand Avati

build: GlusterFS Unit Test Framework

2014-03-06T12:10:46+00:00

This patch will allow for developers to create unit tests for
their code.  Documentation has been added to the patch and
is available here:

doc/hacker-guide/en-US/markdown/unittest.md

Also, unit tests are run when RPM is created.

BUG: 1067059
Change-Id: I95cf8bb0354d4ca4ed4476a0f2385436a17d2369
Signed-off-by: Vijay Bellur 
Signed-off-by: Luis Pabon 
Reviewed-on: http://review.gluster.org/7145
Tested-by: Gluster Build System 
Reviewed-by: Rajesh Joseph 
Reviewed-by: Justin Clift 
Tested-by: Justin Clift

cluster/dht: Fix layout sorting

2014-02-04T01:24:13+00:00

The layout was not being sorted in the ascending order leading
to the wrong detection of holes/overlaps.

From looking at the previous git commits it appears that the initial version itself had the err comparison code.
Deductions from the current dht_layout_sort():
1. The zero'ed out layouts should be in the from of list, if needed
2. The layout should be sorted in the ascending order of layout error value.
3. The layout should be sorted in the ascending order of the layout 'start'.
But In some cases, with the err comparison code its not sorted in the ascending order. Example: If the input is as below for dht_layout_sort(), the sorting doesn't happen in ascending order.
Input:
0-1 err:0    2-3 err:0   6-7 err:0    0-0 err:20   4-5 err:0
          
With the current sort, Output:
4-5 err:0    0-0 err:0    0-1 err:0    2-3 err:0    6-7 err:0
Expected: 0-0 err:20 0-1 err:0 2-3 err:0 4-5 err:0 6-7 err:0
Looking at dht_layout_anomalies() it appears that, it doesn't require the layout to be sorted based on error value.
The other solution was to replace line 468 with:
 if ((layout->list[i].err || layout->list[j].err) && (layout->list[i].start > layout->list[j].start))
Since dht_layout_anomalies() didn't expect the layout to be sorted based on the error, removed the err comparison.

Change-Id: I1215f6cd53efc7dba01c0958ba6cc7609dab6ff5
BUG: 1056406
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/6757
Reviewed-by: Anand Avati 
Tested-by: Anand Avati

cluster/dht: Set quota limit key in dht_selfheal of dirs.

2014-01-23T05:39:57+00:00

Also fixed check in dht_is_subvol_in_layout to check if the
layouts are zero'ed out.

Change-Id: I4bf8ebf66d3ef1946309b6c9aac9e79bf8a6d495
BUG: 969461
Signed-off-by: shishir gowda 
Signed-off-by: Varun Shastry 
Reviewed-on: http://review.gluster.org/6392
Reviewed-by: Raghavendra G 
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

dht: Ignore directory with missing xattrs, which have err == 0, and start == stop

2014-01-16T01:20:09+00:00

From the history (Patch: http://review.gluster.org/4668/)

When subvols-per-directory is < available subvols, then there are layouts
which are not populated. This leads to incorrect identification of holes or
overlaps. We need to ignore layouts, which have err == 0, and start == stop.
In the current scenario (start == stop == 0).

Additionally, in layout-merge, treat missing xattrs as err = 0. In case of
missing layouts, anomalies will reset them.

For any other valid subvoles, err != 0 in case of layouts being zeroed out.
Also reverted back dht_selfheal_dir_xattr, which does layout calculation only
on subvols which have errors.

Change-Id: Idb72a869f1a6f103046bb7e6fe0019f6ac853fd4
BUG: 1047331
Signed-off-by: Vijaykumar M 
Reviewed-on: http://review.gluster.org/6618
Reviewed-by: Krishnan Parthasarathi 
Tested-by: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan 
Reviewed-by: Anand Avati

core: fix errno for non-existent GFID

2013-11-26T18:29:23+00:00

When clients refer to a GFID which does not exist, the errno to
be returned in ESTALE (and not ENOENT). Even though ENOENT might
look "proper" most of the time, as the application eventually expects
ENOENT even if a parent directory does not exist, not returning
ESTALE results in resolvers (FUSE and GFAPI) to not retry resolution
in uncached mode. This can result in spurious ENOENTs during
concurrent path modification operations.

Change-Id: I7a06ea6d6a191739f2e9c6e333a1969615e05936
BUG: 1032894
Signed-off-by: Anand Avati 
Reviewed-on: http://review.gluster.org/6318
Tested-by: Gluster Build System 
Reviewed-by: Amar Tumballi 
Reviewed-by: Brian Foster 
Reviewed-by: Vijay Bellur

Revert "cluster/dht: Return success in dht_discover if layout issues"

2013-09-25T06:48:04+00:00

This reverts commit a3e593f9f17cb1e68db97bb5a0d8074793a33964 which
was bought into fix dht_layout_anomalies error handling.

We still see applications error'ing out due to graph switches.

To fix the above issue -

We cannot heal in dht_discover, as it is a gfid based lookup, and not
path based. So, returning error here would lead to app's to see failure.

Also, update the layout in inode_ctx even if it has anomalies. Let
subsequent heals fix the issue.

Conflicts:
	xlators/cluster/dht/src/dht-common.c

Signed-off-by: shishir gowda 
Change-Id: I68c1056c3587e04a02344548546ddd06034489c5
BUG: 960348
Reviewed-on: http://review.gluster.org/5443
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

cluster/dht: Fix anomaly check

2013-09-23T18:40:33+00:00

We were wrongly detecting holes/overlaps for already accounted
errors. Additionally, sort should also handle zero'ed out layout

Change-Id: Ic3d13e1d735b914f9acc01fe919bc90656baea48
BUG: 1003851
Signed-off-by: shishir gowda 
Reviewed-on: http://review.gluster.org/5762
Tested-by: Gluster Build System 
Reviewed-by: Amar Tumballi 
Reviewed-by: Anand Avati