glusterfs.git/libglusterfs, branch v3.10dev

io-stats: Add stats for upcall notifications

2016-09-01T03:18:42+00:00

With this patch, there will be additional entries seen in
the profile info:

UPCALL : Total number of upcall events that were sent from
         the brick(in brick profile), and number of upcall
         notifications recieved by client(in client profile)

Cache invalidation events:
-------------------------
CI_IATT : Number of upcalls that were cache invalidation and
         had one of the IATT_UPDATE_FLAGS set. This indicates
         that one of the iatt value was changed.

CI_XATTR : Number of upcalls that were cache invalidation, and
         had one of the UP_XATTR or UP_XATTR_RM set. This indicates
         that an xattr was updated or deleted.

CI_RENAME : Number of upcalls that were cache invalidation,
         resulted by the renaming of a file or directory

CI_UNLINK : Number of upcalls that were cache invalidation,
         resulted by the unlink of a file.

CI_FORGET : Number of upcalls that were cache invalidation,
         resulted by the forget of inode on the server side.

Lease events:
------------
LEASE_RECALL : Number of lease recalls sent by the brick (in
         brick profile), and number of lease recalls recieved
         by client(in client profile)

Note that the sum of CI_IATT, CI_XATTR, CI_RENAME, CI_UNLINK,
CI_FORGET, LEASE_RECALL may not be equal to UPCALL. This is
because, each cache invalidation can carry multiple flags.
Eg:
- Every CI_XATTR will have CI_IATT
- Every CI_UNLINK will also increment CI_IATT as link count is an
iatt attribute.

Also UP_PARENT_DENTRY_FLAGS is currently not accounted for,
as CI_RENAME and CI_UNLINK will always have the flag
UP_PARENT_DENTRY_FLAGS

Change-Id: Ieb8cd21dde2c4c7618f12d025a5e5156f9cc0fe9
BUG: 1371543
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/15193
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Rajesh Joseph 
CentOS-regression: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri

compound fops: Some fixes to compound fops framework

2016-08-31T06:50:40+00:00

Change-Id: I808fd5f9f002a35bff94d310c5d61a781e49570b
BUG: 1360169
Signed-off-by: Anuradha Talur 
Reviewed-on: http://review.gluster.org/15010
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G

dht, md-cache, upcall: Add invalidation of IATT when the layout changes

2016-08-31T06:08:54+00:00

Issue:
dht_layout is built as a part of lookup only. The layout can be
modified by rebalance process. Since every IO fop is preceded
by a lookup, there are very less issues of stale layout. But
with enhancements of aggressive caching of stats in md-cache,
the lookup will reduce and expose the stale layout issue often.

Solution:
Since stale layout is already an issue on dht, there is already
a plan to fix this at the dht layer, but this fix is not currently
planned for any release. Until this fix comes out, we can have
a workaround where, the upcall will send a notification to md-cache
when a layout xattr is changed. As a part of layout change notification
the existing cache is invalidated and the next lookup will fetch the
latest layout.

This is not a foolproof solution as the window between the layout change
and the next lookup(after invalidation of stat), where there will be stale
layout. But until the final fix comes in, this reduces the stale layout
window.

Change-Id: Iacf871a38b35880c1fc0bc68fe7ce291265e71d4
BUG: 1369638
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/15300
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Raghavendra G

md-cache: Register the list of xattrs with cache-invalidation

2016-08-31T06:07:01+00:00

Issue:
md-cache caches a specified list of xattrs, and when cache invalidation
is enabled, it makes sense to recieve invalidation only when those xattrs
are modified by other clients. But the current implementation of upcall
is that, it will send invalidation when any of the on-disk xattrs is modified.

Solution:
md-cache sends a list of xattrs that it is interested in, to upcall by
issuing an ipc(). The challenge here is to make sure everytime a brick
goes offline and comes back up, the ipc() needs to be issued to the
bricks. Hence ipc() is sent from md-cache every time there is a
CHILD_UP/CHILD_MODIFIED event.

TODO:
There will be patches following, in cluster xlators, to implement ipc fop.

Change-Id: I6efcf3df474f5ce6eabd3d6694c00c7bd89bc25d
BUG: 1211863
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/15002
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Rajesh Joseph 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Prashanth Pai 
Reviewed-by: Raghavendra G

eventsapi: Add support for Client side Events

2016-08-31T01:34:59+00:00

Client side gf_event uses ctx->cmd_args.volfile_server to push
notifications to the eventsd.

Socket server changed from Unix domain socket to UDP to support
external events.

Following to be addressed in different patch
- Port used for eventsd is 24009. Make it configurable
  Already configurable in Server side. Configurable in gf_event API
  is required.
- Auth Token yet to be added as discussed in
  https://www.gluster.org/pipermail/gluster-devel/2016-August/050324.html

Change-Id: I159acf80b681d10b82d52cfb3ffdf85cb896542d
BUG: 1367774
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15189
Smoke: Gluster Build System 
Reviewed-by: Prashanth Pai 
Reviewed-by: Atin Mukherjee 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System

upcall: Add permission change flag to iatt flag

2016-08-29T12:15:15+00:00

Currently IATT_UPDATE_FLAGS is (UP_NLINK | UP_MODE | UP_OWN | UP_SIZE |
UP_TIMES | UP_ATIME). But it should also have UP_PERM and permission bits
are apart of IATT.

This change will have no effect on the existing users as
IATT_UPDATE_FLAGS is currently used only by md-cache. And the changes in
md-cache to consume cache invalidation is not part of any release yet.

Change-Id: I7efad44972add152f5f981d733fb8191dc7ef8ef
BUG: 1369432
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/15301
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Raghavendra G 
Reviewed-by: Prashanth Pai 
Reviewed-by: Jeff Darcy

dht: define GF_IPC_TARGET_UPCALL

2016-08-29T03:50:58+00:00

In patch http://review.gluster.org/#/c/15225/
GF_IPC_TARGET_UPCALL was used but not defined,
because of which build is broken

Hence fixing it here.

Change-Id: I81e1e63cba25a49fd72a8fd7a3b4a445cadae103
BUG: 1370862
Signed-off-by: Prasanna Kumar Kalever 
Reviewed-on: http://review.gluster.org/15331
Smoke: Gluster Build System 
Tested-by: Atin Mukherjee 
Reviewed-by: Atin Mukherjee 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System

md-cache: Add cache hit and miss counters

2016-08-27T11:14:36+00:00

These counters can be accessed either by .meta interface
or statedump.

From meta: cat on the private file in md-cache directory.
Eg: cat /mnt/glusterfs/0/.meta/graphs/active/patchy-md-cache/private
[performance/md-cache.patchy-md-cache]
stat_hit_count = 2
stat_miss_count = 8
xattr_hit_count = 4
xattr_miss_count = 3
nameless_lookup_count = 1
negative_lookup_count = 0
stat_invalidations_recieved = 1
xattr_invalidations_recieved = 1

Change-Id: Ib62a8822f263a9f75858b15832d0119fbe382629
BUG: 1211863
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/15185
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Raghavendra G

glusterd/cli: cli to get local state representation from glusterd

2016-08-26T15:23:37+00:00

Currently there is no existing CLI that can be used to get the
local state representation of the cluster as maintained in glusterd
in a readable as well as parseable format.

The CLI added has the following usage:

 # gluster get-state [daemon] [odir ] [file ]

This would dump data points that reflect the local state
representation of the cluster as maintained in glusterd (no other
daemons are supported as of now) to a file inside the specified
output directory. The default output directory and filename is
/var/run/gluster and glusterd_state_ respectively. The
option for specifying the daemon name leaves room to add support for
other daemons in the future. Following are the data points captured
as of now to represent the state from the local glusterd pov:

 * Peer:
    - Primary hostname
    - uuid
    - state
    - connection status
    - List of hostnames

 * Volumes:
    - name, id, transport type, status
    - counts: bricks, snap, subvol, stripe, arbiter, disperse,
 redundancy
    - snapd status
    - quorum status
    - tiering related information
    - rebalance status
    - replace bricks status
    - snapshots

 * Bricks:
    - Path, hostname (for all bricks these info will be shown)
    - port, rdma port, status, mount options, filesystem type and
signed in status for bricks running locally.

 * Services:
    - name, online status for initialised services

 * Others:
    - Base port, last allocated port
    - op-version
    - MYUUID

Change-Id: I4a45cc5407ab92d8afdbbd2098ece851f7e3d618
BUG: 1353156
Signed-off-by: Samikshan Bairagya 
Reviewed-on: http://review.gluster.org/14873
Reviewed-by: Avra Sengupta 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Atin Mukherjee

build: fix eventtypes.h generation

2016-08-26T12:38:00+00:00

make 'eventtypes.py' as dependent i.e. must build eventtypes.h

Else the consequence will be like:
error: 'EVENT_CLIENT_GRACE_TIMER_START' undeclared (first use in this function)

Change-Id: I5fe2491d8d1e0c430b307026695d25475250ae79
BUG: 1370406
Signed-off-by: Prasanna Kumar Kalever 
Reviewed-on: http://review.gluster.org/15327
Tested-by: Prasanna Kumar Kalever 
Smoke: Gluster Build System 
Reviewed-by: Aravinda VK 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System