glusterfs.git/xlators/cluster/dht/src/dht-layout.c, branch v3.7.16

cluster/distribute: detect stale layouts in entry fops

2016-04-25T09:30:45+00:00

dht_mkdir ()
{
      first-hashed-subvol = hashed-subvol for "bname" in in-memory
                            layout of "parent";
      inodelk (SETLKW, parent, "LAYOUT_HEAL_DOMAIN", "can be any
               subvol, but we choose first-hashed-subvol randomly");
      {
begin:
            hashed-subvol = hashed-subvol for "bname" in in-memory
                            layout of "parent";
            hash-range = extract hashe-range from layout of "parent";

            ret = mkdir (parent/bname, hashed-subvol, hash-range);
            if (ret == "hash-value doesn't fall into layout stored on
                       the brick (this error is returned by posix-mkdir)")
            {
                refresh_parent_layout ();
                goto begin;
            }

      }
      inodelk (UNLCK, parent, "LAYOUT_HEAL_DOMAIN",
               "first-hashed-subvol");

      proceed with other parts of dht_mkdir;
}

posix_mkdir (parent/bname, client-hash-range)
{

       disk-hash-range = getxattr (parent, "dht-layout-key");
       if (disk-hash-range != client-hash-range) {
              fail-with-error ("hash-value doesn't fall into layout
                                stored on the brick");
              return 0;
       }

       continue-with-posix-mkdir;
}

Similar changes need to be done for dentry operations like create,
symlink, link, unlink, rmdir, rename. These will be addressed in
subsequent patches. This patch addresses only mkdir codepath.

This change breaks stripe tests, as on some striped subvols dht layout
xattrs are not set for some reason. This results in failure of
mkdir. Since striped volumes are always created with dht, some tests
associated with stripe also fail. So, I am making following tests
changes (since stripe is out of maintainance):
* modify ./tests/basic/rpc-coverage.t to not to use striped volumes
* mark all (2) tests in tests/bugs/stripe/ as bad tests

Change-Id: Idd1ae879f24a48303dc743c1bb4d91f89a629e25
BUG: 1329062
Signed-off-by: Raghavendra G 
Reviewed-on: http://review.gluster.org/14040
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

dht: Adding log messages to the new logging framework

2015-07-27T07:21:48+00:00

        Backported from: http://review.gluster.org/10021

> Change-Id: Ib3bb61c5223f409c23c68100f3fe884918d2dc3f
> BUG: 1194640
> Reviewed-on: http://review.gluster.org/10021
> Reviewed-by: N Balachandran 
> Reviewed-by: Joseph Fernandes 
> Tested-by: Joseph Fernandes 
> Reviewed-by: Dan Lambright 
> Reviewed-by: Raghavendra G 
> Tested-by: Raghavendra G 
> Signed-off-by: arao 

BUG: 1217722
Change-Id: Ide79c6c1e6a466fb52f955c90a2b22711bec794a
Signed-off-by: arao 
Signed-off-by: Anusha Rao 
Reviewed-on: http://review.gluster.org/11350
Tested-by: NetBSD Build System 
Reviewed-by: Raghavendra G 
Tested-by: Gluster Build System

dht: make lookup-unhashed=auto do something actually useful

2015-05-10T04:55:09+00:00

The key concept here is to determine whether a directory is "clean" by
comparing its last-known-good topology to the current one for the
volume.  These are stored as "commit hashes" on the directory and the
volume root respectively.  The volume's commit hash changes whenever a
brick is added or removed, and a fix-layout is done.  A directory's
commit hash changes only when a full rebalance (not just fix-layout)
is done on it.  If all bricks are present and have a directory
commit hash that matches the volume commit hash, then we can assume
that every file is in its "proper" place. Therefore, if we look for
a file in that proper place and don't find it, we can assume it's not
on any other subvolume and *safely* skip the global (broadcast to all)
lookup.

Change-Id: Id6ce4593ba1f7daffa74cfab591cb45960629ae3
BUG: 1220064
Reviewed-on-master: http://review.gluster.org/#/c/7702/
Signed-off-by: Jeff Darcy 
Signed-off-by: Shyam 
Reviewed-on: http://review.gluster.org/10729
Tested-by: Gluster Build System 
Reviewed-by: Krishnan Parthasarathi 
Reviewed-by: Vijay Bellur

dht : coverity fixes

2015-04-07T10:52:05+00:00

CID : 1124352,1124365 (unchecked return value),
      1124377 ( logically dead code),
      1124511 (null dereference)

Change-Id: I61e029a078559cfe15d36bf0aa53418f6214e5cb
BUG: 789278
Signed-off-by: Manikandan Selvaganesh 
Reviewed-on: http://review.gluster.org/9622
Tested-by: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan 
Reviewed-by: Kaleb KEITHLEY 
Reviewed-by: N Balachandran 
Reviewed-by: Vijay Bellur

Avoid conflict between contrib/uuid and system uuid

2015-04-04T17:48:35+00:00

glusterfs relies on Linux uuid implementation, which
API is incompatible with most other systems's uuid. As
a result, libglusterfs has to embed contrib/uuid,
which is the Linux implementation, on non Linux systems.
This implementation is incompatible with systtem's
built in, but the symbols have the same names.

Usually this is not a problem because when we link
with -lglusterfs, libc's symbols are trumped. However
there is a problem when a program not linked with
-lglusterfs will dlopen() glusterfs component. In
such a case, libc's uuid implementation is already
loaded in the calling program, and it will be used
instead of libglusterfs's implementation, causing
crashes.

A possible workaround is to use pre-load libglusterfs
in the calling program (using LD_PRELOAD on NetBSD for
instance), but such a mechanism is not portable, nor
is it flexible. A much better approach is to rename
libglusterfs's uuid_* functions to gf_uuid_* to avoid
any possible conflict. This is what this change attempts.

BUG: 1206587
Change-Id: I9ccd3e13afed1c7fc18508e92c7beb0f5d49f31a
Signed-off-by: Emmanuel Dreyfus 
Reviewed-on: http://review.gluster.org/10017
Tested-by: Gluster Build System 
Reviewed-by: Niels de Vos

Xlators : Fixed typos

2015-04-02T10:21:08+00:00

Change-Id: I948f85cb369206ee8ce8b8cd5e48cae9adb971c9
BUG: 1075417
Signed-off-by: Manikandan Selvaganesh 
Reviewed-on: http://review.gluster.org/9529
Reviewed-by: Niels de Vos 
Tested-by: Gluster Build System 
Reviewed-by: Kaleb KEITHLEY 
Reviewed-by: Humble Devassy Chirammal

cluster/dht: Change the subvolume encoding in d_off to be a "global"

2015-03-18T11:47:41+00:00

position in the graph rather than relative (local) to a particular
translator.

Encoding the volume in this way allows a single translator to manage
which brick is currently being scanned for directory entries. Using a
single translator minimizes allocated bits in the d_off. It also allows
multiple DHT translators in the same graph to have a common frame of
reference (the graph position) for which brick is being read. Multiple
DHT translators are needed for the Tiering feature.

The fix builds off a previous change (9332) which removed subvolume
encoding from AFR. The fix makes an equivalent change to the EC
translator.

More background can be found in fix 9332 and gluster-dev discussions [1].

DHT and AFR/EC are responsibile (as before) for choosing which brick to
enumerate directory entries in over the readdir lifecycle.

The client translator receiving the readdir fop encodes the dht_t. It
is referred to as the "leaf node" in the graph and corresponds to the
brick being scanned.

When DHT decodes the d_off, it translates the leaf node to a local
subvolume, which represents the next node in the graph leading to
the brick.

Tracking of leaf nodes is done in common utility functions. Leaf nodes
counts and positional information are updated on a graph switch.

[1] www.gluster.org/pipermail/gluster-devel/2015-January/043592.html

Change-Id: Iaf0ea86d7046b1ceadbad69d88707b243077ebc8
BUG: 1190734
Signed-off-by: Dan Lambright 
Reviewed-on: http://review.gluster.org/9688
Reviewed-by: Xavier Hernandez 
Reviewed-by: Krishnan Parthasarathi 
Reviewed-by: Vijay Bellur 
Tested-by: Vijay Bellur

testing: Switch to cmocka the successor of cmockery2

2015-03-05T21:51:37+00:00

This uses https://cmocka.org/ as the unit testing framework.

With this change, unit testing is made optional as well. We assume there
is no cmocka available while building. cmocka will be enabled by default
later on. For now, to build with cmocka run:

    $ ./configure --enable-cmocka

This change is based on the work of Andreas (replacing cmockery2 with
cmocka) and Kaleb (make cmockery2 an optional build dependency).

The only modifications I made, are additional #defines in unittest.h for
making sure the unit tests function as expected.

Change-Id: Iea4cbcdaf09996b49ffcf3680c76731459cb197e
BUG: 1067059
Merged-change: http://review.gluster.org/9762/
Signed-off-by: Andreas Schneider 
Signed-off-by: Kaleb S. KEITHLEY 
Signed-off-by: Niels de Vos 
Change-Id: Ia2e955481c102d5dce17695a9205395a6030e985
Reviewed-on: http://review.gluster.org/9738
Tested-by: Gluster Build System

cluster/dht: synchronize with other concurrent healers while healing layout.

2015-02-20T10:35:00+00:00

Current layout heal code assumes layout setting is idempotent. This
allowed multiple concurrent healers to set the layout without any
synchronization. However, this is not the case as different healers
can come up with different layout for same directory and making layout
setting non-idempotent. So, we bring in synchronization among healers
to
   1. Not to overwrite an ondisk well-formed layout.
   2. Refresh the in-memory layout with the ondisk layout if in-memory
   layout needs healing and ondisk layout is well formed.

This patch can synchronize
   1. among multiple healers.
   2. among multiple fix-layouts (which extends layout to consider
   added or removed brick)
   3. (but) not between healers and fix-layouts. So, the problem of
   in-memory stale layouts (not matching with layout ondisk), is not
   _completely_ fixed by this patch.

Signed-off-by: Raghavendra G 
Change-Id: Ia285f25e8d043bb3175c61468d0d11090acee539
BUG: 1176008
Reviewed-on: http://review.gluster.org/9302
Reviewed-by: N Balachandran

build: Support for unit tests using Cmockery2

2014-07-18T17:55:51+00:00

This patch will allow for developers to create unit tests for
their code.  Documentation has been added to the patch and
is available here:

doc/hacker-guide/en-US/markdown/unittest.md

Also, unit tests are run when RPM is created.

This patch is a replacement for http://review.gluster.org/#/c/7281
which removed unit test infrastucture from the repo due to multiple
conflicts.  Cmockery2 is now available in Fedora and EPEL, and soon
to be available in Debian and Ubuntu.  For all other operating
systems, please install from the source:

https://github.com/lpabon/cmockery2

BUG: 1067059

Change-Id: I1b36cb1f56fd10916f9bf535e8ad080a3358289f
Signed-off-by: Luis Pabón 
Reviewed-on: http://review.gluster.org/7538
Tested-by: Gluster Build System 
Reviewed-by: Niels de Vos 
Reviewed-by: Vijay Bellur