glusterfs-afrv1.git/rpc/xdr, branch development

glusterd: Volume locks and transaction specific opinfos

2014-02-11T07:25:40+00:00

With this patch we are replacing the existing cluster-wide
lock taken on glusterds across the cluster, with volume locks
which are also taken on glusterds across the cluster, but are
volume specific. So with the volume locks we are able to perform
more than one gluster operation at the same time, as long as the
operations are being performed on different volumes.

We maintain a global list of volume-locks (using a dict for a list)
where the key is the volume name, and which saves the uuid of the
originator glusterd. These locks are held and released per volume
transaction.

In order to acheive multiple gluster operations occuring at the
same time, we also separate opinfos in the op-state-machine, as a
part of this patch. To do so, we generate a unique transaction-id
(uuid) per gluster transaction. An opinfo is then associated with
this transaction id, which is used throughout the transaction. We
maintain a run-time global list(using a dict) of transaction-ids,
and their respective opinfos to achieve this.

Upstream Feature Page: http://www.gluster.org/community/documentation/index.php/Features/glusterd-volume-locks

Change-Id: Iaad505a854bac8de8f83beec0357eb6cde3f7ea8
BUG: 1011470
Signed-off-by: Avra Sengupta 
Reviewed-on: http://review.gluster.org/5994
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

cli: Add options to the CLI that let the user control the reset of

2014-01-24T10:05:35+00:00

stats

"volume profile info" automatically clears incremental stats. There
isn't a command to:
- fetch stats without clearing incremental stats and
- clear cumulative and incremental stats

This change introduces two arguments (i.e. peek and clear). 'clear'
will wipe both incremental and cumulative stats. 'peek' fetches stats
without wiping incremental stats.

'volume profile info peek' - fetches incremental and cumulative stats
                             without wiping incremental stats

'volume profile info incremental peek' - fetches incremental stats
                                     without wiping incremental stats

'volume profile info clear' - clears both incremental and cumultiave
                              stats

Change-Id: I91834515ad672eca5f882809941147d7d997c4c9
BUG: 1047416
Signed-off-by: Dawit Alemu 
Reviewed-on: http://review.gluster.org/6620
Tested-by: Gluster Build System 
Reviewed-by: Kaushal M 
Reviewed-by: Vijay Bellur

build: Start using library versioning for various libraries

2014-01-18T16:51:14+00:00

According to libtool three individual numbers stand for
CURRENT:REVISION:AGE, or C:R:A for short. The libtool
script typically tacks these three numbers onto the end
of the name of the .so file it creates. The formula for
calculating the file numbers on Linux and Solaris is

   /path/to/library/.(C - A).(A).(R)

As you release new versions of your library, you will
update the library's C:R:A. Although the rules for changing
these version numbers can quickly become confusing, a few
simple tips should help keep you on track. The libtool
documentation goes into greater depth.

In essence, every time you make a change to the library and
release it, the C:R:A should change. A new library should start
with 0:0:0. Each time you change the public interface
(i.e., your installed header files), you should increment the
CURRENT number. This is called your interface number. The main
use of this interface number is to tag successive revisions
of your API.

The AGE number is how many consecutive versions of the API the
current implementation supports. Thus if the CURRENT library
API is the sixth published version of the interface and it is
also binary compatible with the fourth and fifth versions
(i.e., the last two), the C:R:A might be 6:0:2. When you break
binary compatibility, you need to set AGE to 0 and of course
increment CURRENT.

The REVISION marks a change in the source code of the library
that doesn't affect the interface-for example, a minor bug fix.
Anytime you increment CURRENT, you should set REVISION back to 0.

Change-Id: Id72e74c1642c804fea6f93ec109135c7c16f1810
BUG: 862082
Signed-off-by: Harshavardhana 
Reviewed-on: http://review.gluster.org/5645
Tested-by: Gluster Build System 
Reviewed-by: Niels de Vos 
Reviewed-by: Vijay Bellur

cli: Add an option to fetch just incremental or cumulative I/0

2013-12-04T02:09:29+00:00

information

'volume profile info' fetches both cumulative and incremental
I/O statistics. There isn't a way to fetch just cumulative or
incremental statistics.

This change introduces two optional arguments, namely "incremental"
and "cumulative", that can be tacked on to 'volume profile info'.
In other words, the new command format is

volume profile  {start | info [incremental | cumulative]
| stop} [nfs]

'volume profile info incremental' - fetches incremental stats
'volume profile info cumulative' - fetches cumulative stats
'volume profile info' - fetches incremental and cumulative stats

Change-Id: I5ddb45d990542ea611d23d251efebfec46f472d0
BUG: 1030580
Signed-off-by: Dawit Alemu 
Reviewed-on: http://review.gluster.org/6264
Tested-by: Gluster Build System 
Reviewed-by: Krishnan Parthasarathi

cli: More checks in rebalance status output

2013-12-03T23:19:25+00:00

Change-Id: Ibd2edc5608ae6d3370607bff1c626c8347c4deda
BUG: 1031887
Signed-off-by: Kaushal M 
Reviewed-on: http://review.gluster.org/6337
Reviewed-by: Krishnan Parthasarathi 
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

cli/glusterd: Changes to quota command Quota feature

2013-11-26T18:25:27+00:00

 re-work.

Following are the cli commands that are new/re-worked:
======================================================

volume quota  {enable|disable|list [ ...]|remove | default-soft-limit } |
volume quota  {limit-usage   []} |
volume quota  {alert-time|soft-timeout|hard-timeout} {}
volume status [all |  [nfs|shd||quotad]] [detail|clients|mem|inode|fd|callpool]
volume statedump  [nfs|quotad] [all|mem|iobuf|callpool|priv|fd|inode|history]

glusterd changes:
=================
* Quota limits are now set as extended attributes by glusterd from
  the aux mount created by the cli.
* The gfids of the directories on which quota limits are set
  for a given volume are stored in
  /var/lib/glusterd/vols//quota.conf file in binary format,
  and whose cksum and version is stored in
  /var/lib/glusterd/vols//quota.cksum.

Original-author: Krutika Dhananjay 
Original-author: Krishnan Parthasarathi 

BUG: 969461
Change-Id: If32bba36c67f9c2a30417af9c6389045b2b7c13b
Signed-off-by: Krutika Dhananjay 
Signed-off-by: Raghavendra G 
Reviewed-on: http://review.gluster.org/6003
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

glusterfs: zerofill support

2013-11-11T05:25:49+00:00

Add support for a new ZEROFILL fop. Zerofill writes zeroes to a file in
the specified range. This fop will be useful when a whole file needs to
be initialized with zero (could be useful for zero filled VM disk image
provisioning or  during scrubbing of VM disk images).

Client/application can issue this FOP for zeroing out. Gluster server
will zero out required range of bytes ie server offloaded zeroing. In
the absence of this fop,  client/application has to repetitively issue
write (zero) fop to the server, which is very inefficient method because
of the overheads involved in RPC calls  and acknowledgements.

WRITESAME is a  SCSI T10 command that takes a block of data as input and
writes the same data to other blocks and this write is handled
completely within the storage and hence is known as offload . Linux ,now
has support for SCSI WRITESAME command which is exposed to the user in
the form of BLKZEROOUT ioctl.  BD Xlator can exploit BLKZEROOUT ioctl to
implement this fop. Thus zeroing out operations can be completely
offloaded to the storage device , making it highly efficient.

The fop takes two arguments offset and size. It zeroes out 'size' number
of bytes in an opened file starting from 'offset' position.

This patch adds zerofill support to the following areas:
	- libglusterfs
	- io-stats
	- performance/md-cache,open-behind
	- quota
	- cluster/afr,dht,stripe
	- rpc/xdr
	- protocol/client,server
	- io-threads
	- marker
	- storage/posix
	- libgfapi

Client applications can exloit this fop by using glfs_zerofill introduced in
libgfapi.FUSE support to this fop has not been added as there is no system call
for this fop.

Changes from previous version 3:
* Removed redundant memory failure log messages

Changes from previous version 2:
* Rebased and fixed build error

Changes from previous version 1:
* Rebased for latest master

TODO :
     * Add zerofill support to trace xlator
     * Expose zerofill capability as part of gluster volume info

Here is a performance comparison of server offloaded zeofill vs zeroing
out using repeated writes.

[root@llmvm02 remote]# time ./offloaded aakash-test log 20

real	3m34.155s
user	0m0.018s
sys	0m0.040s
[root@llmvm02 remote]# time ./manually aakash-test log 20

real	4m23.043s
user	0m2.197s
sys	0m14.457s
[root@llmvm02 remote]# time ./offloaded aakash-test log 25;

real	4m28.363s
user	0m0.021s
sys	0m0.025s
[root@llmvm02 remote]# time ./manually aakash-test log 25

real	5m34.278s
user	0m2.957s
sys	0m18.808s

The argument log is a file which we want to set for logging purpose and
the third argument is size in GB .

As we can see there is a performance improvement of around 20% with this
fop.

Change-Id: I081159f5f7edde0ddb78169fb4c21c776ec91a18
BUG: 1028673
Signed-off-by: Aakash Lal Das 
Signed-off-by: M. Mohan Kumar 
Reviewed-on: http://review.gluster.org/5327
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

cli,glusterd: Implement 'volume status tasks'

2013-10-09T06:13:16+00:00

oVirt's Gluster Integration needs an inexpensive command that can be
executed every 10 seconds to monitor async tasks and their parameters,
for all volumes.

The solution involves adding a 'tasks' sub-command to 'volume status'
to fetch only the async task IDs, type and other relevant parameters.
Only the originator glusterd participates in this command as all the
information needed is available on all the nodes. This is to make the
command suitable for being executed every 10 seconds.

Change-Id: I1edc607baf29b001a5585079dec681d7c641b3d1
BUG: 1012346
Signed-off-by: Krutika Dhananjay 
Reviewed-on: http://review.gluster.org/6006
Tested-by: Gluster Build System 
Reviewed-by: Kaushal M

cli/glusterd: improve rebalance fix-layout status reporting

2013-09-19T16:22:36+00:00

Problem:
Currenly the CLI rebalance status command output does not indicate the
'type' of rebalance, i.e. whether a full rebalance or only a fix-layout
was carried out.

Fix: After the rebalance status of all peers is received by the
originator glusterd, alter it to reflect the type of rebalance
before passing it on to the CLI process.

Change-Id: I1940ffda0d36e25e5b33c84a0ea210394cc9e1d3
BUG: 1004744
Signed-off-by: Ravishankar N 
Reviewed-on: http://review.gluster.org/5826
Reviewed-by: Krishnan Parthasarathi 
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

glusterd/cli: Geo-Replication "status detail" cmd

2013-09-05T02:50:49+00:00

Provides detailed status info in the following format

                    MASTER   SLAVE 

NODE   HEALTH   UPTIME  FILES SYNCD  FILES PENDING  BYTES PENDING  DELETES PENDING
-----------------------------------------------------------------------------------

This patch introdues "status detail" command to show crawl related
information in CLI. These values are "pulled" from gsyncd when
"status detail" is executed.

Change-Id: I1fdaf7180eacce054a864d34971dc160bd7301e1
BUG: 990420
Signed-off-by: Venky Shankar 
Reviewed-on: http://review.gluster.org/5590
Tested-by: Gluster Build System 
Reviewed-by: Avra Sengupta 
Tested-by: Avra Sengupta 
Reviewed-by: Anand Avati