glusterfs.git/rpc/xdr/src, branch v3.7.0alpha0

build: add more files to .gitignore

2015-04-06T13:00:39+00:00

Change-Id: Icef0d7f443f7caf3aa386d3a6978f98cf3a5a4af
BUG: 1198849
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/10132
Reviewed-by: Kaleb KEITHLEY 
Tested-by: Gluster Build System

glusterd: CLI commands to create and manage tiered volumes.

2015-03-19T13:32:28+00:00

A tiered volume is a normal volume with some number of new bricks
representing "hot" storage. The "hot" bricks can be attached or
detached dynamically to a normal volume. When this happens, a new graph
is constructed. The root of the new graph is an instance of the tier
translator. One subvolume of the tier translator leads to the old volume,
and another leads to the new hot bricks.

attach-tier  [ ]  ... [force]

volume detach-tier  [replica ] 
... 

gluster volume rebalance  tier start

gluster volume rebalance  tier stop

gluster volume rebalance  tier status

The "tier start" CLI command starts a server side daemon. The daemon
initiates file level migration based on caching policies. The daemon's
status can be monitored and stopped.

Note development on the "tier status" command is incomplete. It will be
added in a subsequent patch.

When the "hot" storage is detached, the tier translator is removed
from the graph and the tiered volume reverts to its original state as
described in the volume's info file.

For more background and design see the feature page [1].

[1]
http://www.gluster.org/community/documentation/index.php/Features/data-classification

Change-Id: Ic8042ce37327b850b9e199236e5be3dae95d2472
BUG: 1194753
Signed-off-by: Dan Lambright 
Reviewed-on: http://review.gluster.org/9753
Reviewed-by: Vijay Bellur 
Tested-by: Vijay Bellur

cli/glusterd: cli command implementation for bitrot features

2015-03-19T01:34:07+00:00

CLI command for bitrot features.

volume bitrot  enable|disable

Above command will enable/disable bitrot feature for particular volume.

BUG: 1170075
Change-Id: Ie84002ef7f479a285688fdae99c7afa3e91b8b99
Signed-off-by: Gaurav Kumar Garg     
Signed-off-by: Anand nekkunti        
Signed-off-by: Dominic P Geevarghese 
Reviewed-on: http://review.gluster.org/9866
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

features/quota : Introducing inode quota

2015-03-19T01:24:12+00:00

==========================================================================
                             Inode quota
==========================================================================
= Currently, the only way to retrieve the number of files/objects in a   =
= directory or volume is to do a crawl of the entire directory/volume.   =
= This is expensive and is not scalable.                                 =
=                                                                        =
= The proposed mechanism will provide an easier alternative to determine =
= the count of files/objects in a directory or volume.                   =
=                                                                        =
= The new mechanism proposes to store count of objects/files as part of  =
= an extended attribute of a directory. Each directory's extended        =
= attribute value will indicate the number of files/objects present      =
= in a tree with the directory being considered as the root of the tree. =
=                                                                        =
= The count value can be accessed by performing a getxattr().            =
= Cluster translators like afr, dht and stripe will perform aggregation  =
= of count values from various bricks when getxattr() happens on the key =
= associated with file/object count.                                     =

A new interface is introduced:
------------------------------
        limit-objects  : limit the number of inodes at directory level
        list-objects   : list the directories where the limit is set
        remove-objects : remove the limit from the directory

==========================================================================

CLI COMMAND:
gluster volume quota  limit-objects   []

*  is a hard-limit for number of objects limitation for path ""
  If hard-limit is exceeded, creation of file/directory is no longer
permitted.

*  is a soft-limit for number of objects creation for path ""
  If soft-limit is exceeded, a warning is issued for each creation.

CLI COMMAND:
gluster volume quota  remove-objects [path]

==========================================================================

CLI COMMAND:
gluster volume quota  list-objects [path] ...

Sample output:
------------------
  Path                   Hard-limit Soft-limit   Used  Available
Soft-limit exceeded?
Hard-limit exceeded?
  ------------------------------------------------------------------------
--------------------------------------
  /dir                      10       80%          10       0
Yes
        Yes

==========================================================================

[root@snapshot-28 dir]# ls
a  b  file11  file12  file13  file14  file15  file16  file17
[root@snapshot-28 dir]# touch a1
touch: cannot touch `a1': Disk quota exceeded
* Nine files are created in directory "dir" and directory is included in
* the
count too. Hence the limit "10" is reached and further file creation
fails

==========================================================================

Note: We have also done some re-factoring in cli for volume name
validation. New function cli_validate_volname is created

==========================================================================

Change-Id: I1823497de4f790a2a20ebb1770293472ea33ee2b
BUG: 1190108
Signed-off-by: Sachin Pandit 
Signed-off-by: vmallika 
Reviewed-on: http://review.gluster.org/9769
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

features/changelog: RPC'fy {libgf}changelog

2015-03-19T01:22:36+00:00

This patch introduces RPC based communication between the changelog
translator and libgfchangelog. It replaces the old pathetic stream
based interaction that existed earlier (due to time constraints :-/).

Changelog, upon initialization starts a RPC server (rpcsvc) allowing
clients to invoke a probe API as a bootup mechanism to request for
event notifications. During probe, clients can choose an event
filter specifying the type(s) of events they are interested in. As
of now there is no way to change the event notification set once
the probe RPC call is made, but that is easier to implement.

The actual event notifications is done on a separate RPC session.
The client (libgfchangelog) itself starts and RPC server which the
changelog translator "connects back" during probe. Notifications
are dispatched by a bunch of threads from the server (translator)
and the client optionally orders them if ordered notifications
are requried. FOPs fill in their respective event details in a
buffer (rot-buffs to be particular) and a bunch of threads
(consumers) swap the buffers out of roatation and dispatch them
via RPC. To avoid writer starvation, then number of dispatcher
threads is one less than the number of buffer list in rot-buffs.x

libgfchangelog becomes purely callback based -- upon event
notification from the server (and re-ordering them if required)
invoke a callback routine specified by consumer(s).

A major part of the patch is also aimed at providing backward
compatibility for geo-replication, which was one of the main
consumer of the stream based API. Also, this patch does not\
"turn on" event notifications for all fops, just a bunch which
is currently in requirement. Another pain point is that the
server does not filter events before dispatching it to the
clients. That load is taken up by the client itself (although
it's done at the library layer rather than making it hard on
the callback implementor). This needs improvement and care
needs to be taken to not load the server up with expensive
filtering mechanisms.

Change-Id: Ibf60a432b68f2dfa60c6f9add2bcfd37a9c41395
BUG: 1170075
Signed-off-by: Venky Shankar 
Reviewed-on: http://review.gluster.org/9708
Reviewed-by: Jeff Darcy 
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

Snapshot/clone: clone of a snapshot that will act as a regular volume

2015-03-18T07:07:55+00:00

snapshot clone will allow us to take a snpahot of a snapshot.
Newly created clone volume will be a regular volume with read/write
permissions.

CLI command
snapshot clone  

Change-Id: Icadb993fa42fff787a330f8f49452da54e9db7de
BUG: 1199894
Signed-off-by: Mohammed Rafi KC 
Reviewed-on: http://review.gluster.org/9750
Tested-by: Gluster Build System 
Reviewed-by: Rajesh Joseph 
Reviewed-by: Vijay Bellur

gfapi: APIs to store and process upcall notifications received

2015-03-17T21:01:21+00:00

In case of any upcall cbk events received by the protocol/client,
gfapi will be notified which queues them up in a list ().

Applicatons are responsible to provide APIs to process & notify them in case
of any such upcall events queued.

Added a new API which will be used by Ganesha to repeatedly poll for any
such upcall event notified ().

A new test-file has been added to test the cache_invalidation upcall events.

Below link has a writeup which explains the code changes done -
        URL: https://soumyakoduri.wordpress.com/2015/02/25/glusterfs-understanding-upcall-infrastructure-and-cache-invalidation-support/

Change-Id: Iafc6880000c865fd4da22d0cfc388ec135b5a1c5
BUG: 1200262
Signed-off-by: Soumya Koduri 
Reviewed-on: http://review.gluster.org/9536
Tested-by: Gluster Build System 
Reviewed-by: Kaleb KEITHLEY

every/where: add GF_FOP_IPC for inter-translator communication

2015-03-17T14:02:15+00:00

Several features - e.g. encryption, erasure codes, or NSR - involve
multiple cooperating translators which sometimes need a "private" means
of communication amongst themselves.  Historically we've used virtual or
synthetic xattrs, but that's not very elegant and clutters up the
getxattr/setxattr path which must also handle real xattr requests.  This
new fop should address that.

The only argument is an int32_t "op" which should be recognized by the
target translator.  It is recommended that translators using these
feature follow some convention regarding the ops that they define, to
avoid conflicts.  Using a hash of the target translator's type string as
a base for a series of ops would probably be a good start.  Any other
information can be passed in both directions using xdata.

The default behavior for this fop, as with any other, is to pass through
to FIRST_CHILD.  That makes use of this fop "transparent" to other
translators that were written before it existed, but it also means that
it only really works with pass-through translators.  If a routing
translator (such as DHT) or a fan-out translator (such as AFR) is
involved, the IPC might not reach its intended destination unless those
translators are modified to forward IPC fops along all paths.

If an IPC gets all the way to storage/posix it is considered an error,
much like an uncaught exception.  We don't actually *do* anything in
that case, but we do log it send back an EOPNOTSUPP error.  This makes
the "unrecognized opcode" condition distinguishable from the "no IPC
support" condition (which would yield an RPC error instead) so clients
can probe for the presence of a handler for their own favorite opcode
and either use that or use old-school xattrs depending on the result.

BUG: 1158628
Signed-off-by: Venky Shankar 
Signed-off-by: Jeff Darcy 
Change-Id: I84af1b17babe5b30ec03ecf027ae37d09b873968
Reviewed-on: http://review.gluster.org/8812
Reviewed-by: Vijay Bellur

Upcall: New xlator to store various states and send cbk events

2015-03-17T12:08:07+00:00

Framework on the server-side, to handle certain state of the files
accessed and send notifications to the clients connected.

A generic and extensible framework, used to maintain states in
the glusterfsd process for each of the files accessed
(including the clients info doing the fops) and send
notifications to the respective glusterfs clients incase of
any change in that state.

This patch handles "Inode Update/Invalidation" upcall event.

Feature page:
        URL: http://www.gluster.org/community/documentation/index.php/Features/Upcall-infrastructure

Below link has a writeup which explains the code changes done -
        URL: https://soumyakoduri.wordpress.com/2015/02/25/glusterfs-understanding-upcall-infrastructure-and-cache-invalidation-support/

Change-Id: Ie3d724be9a3419fcf18901a753e8ec2df2ac802f
BUG: 1200262
Signed-off-by: Soumya Koduri 
Reviewed-on: http://review.gluster.org/9535
Reviewed-by: Kaleb KEITHLEY 
Tested-by: Gluster Build System

rdma:rdma fuse mount hangs for tcp,rdma volumes if brick is down.

2014-11-18T07:30:55+00:00

When we try to mount a tcp,rdma volume as rdma
transport using FUSE protocol, then mount will
hang if the brick is down. When we kill a process,
signal will be received in glusterfsd process and
it will call pmap_signout with port listening on tcp only.
In case of the tcp,rdma there will be two ports,
and port which is listening for rdma will not
called for sign out. 
So the mount process will try to connect to a port
which is not open and it will keep trying to connect.
This patch will call pmap_signout for rdma port also,
So when mount tries to get the brick port,it will fail.

Change-Id: I23676f65f96eb90b69b76478f7a21412a6aba70f
BUG: 1143886
Signed-off-by: Mohammed Rafi KC 
Reviewed-on: http://review.gluster.org/8762
Tested-by: Gluster Build System 
Reviewed-by: Krishnan Parthasarathi 
Reviewed-by: Raghavendra G 
Tested-by: Raghavendra G