glusterfs.git/rpc, branch v3.10.0rc0

socket: retry connect immediately if it fails

2017-02-03T00:43:24+00:00

Previously we relied on a complex dance of setting flags, shutting
down the socket, tearing stuff down, getting an event, tearing more
stuff down, and waiting for a higher-level retry.  What we really
need, in the case where we're just trying to connect prematurely e.g.
to a brick that hasn't fully come up yet, is a simple retry of the
connect(2) call.

This was discovered by observing failures in ec-new-entry.t with
multiplexing enabled, but probably fixes other random failures as
well.

Backport of:
> Change-Id: Ibedb8942060bccc96b02272a333c3002c9b77d4c
> BUG: 1385758
> Reviewed-on: https://review.gluster.org/16510

BUG: 1418091
Change-Id: I4bac26929a12cabcee4f9e557c8b4d520948378b
Signed-off-by: Jeff Darcy 
Reviewed-on: https://review.gluster.org/16533
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

rpc/socket.c : Bonnie++ hangs during rewrites in ganesha + SSL

2017-02-02T17:33:45+00:00

Problem: Bonnie++ rewrite operation hangs in ganesha + SSL environment

Solution: Bonnie++ hangs during execution of rewrite operation in
          ganesha + SSL environment.It was hanged due to blocking on poll
          call in ssl_do because no POLLOUT event was getting on socket.
          Socket is not getting POLLOUT event because all other threads
          are waiting to get lock and lock is not released ssl_do
          because it is not getting any event on poll.To correct it
          update the condition in ssl_do as same in getting error
          SSL_ERROR_WANT_READ.

Test:     To test the patch followed below procedure
          1) Setup 2X2 Ganesha + SSL environment.
          2) Run bonnie from 3 nfs client parallely
          3) After run "Rewwrite operation" by bonnie it is hanged.
          4) After apply the patch it is not hanged.

> BUG: 1418213
> Change-Id: I5985cbbc4cfdac5d287268d791e31c274abc3c8d
> Signed-off-by: Mohit Agrawal 
> Reviewed-on: https://review.gluster.org/16501
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> Reviewed-by: Jeff Darcy 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Vijay Bellur 
> (cherry picked from commit d7077bca4b372a056d23416294e729637e9af94e)

Change-Id: Id029c71382025477bb5ff31f28ec537e4fe58b03
BUG: 1418541
Reviewed-on: https://review.gluster.org/16513
Tested-by: MOHIT AGRAWAL 
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

libglusterfs+transport+io-threads: fix 256KB stack abuse

2017-02-02T17:32:49+00:00

Some functions were allocating 64K booleans, which are (crazily)
mapped to 4-byte ints, for a total of 256KB per call.  Changed to use
bitfields instead, so usage is now only 8KB per call.  This was the
impediment to changing the io-threads stack size, so that has been
adjusted too.

Backport of:
> Change-Id: I8781c4f2c8f2b830f4535e366995fac8dd0a8653
> BUG: 1418095
> Reviewed-on: https://review.gluster.org/15745

Change-Id: Ia5dada61703e6bea95f2511da71feb573fc9a429
BUG: 1418536
Signed-off-by: Jeff Darcy 
Reviewed-on: https://review.gluster.org/16511
Reviewed-by: N Balachandran 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Vijay Bellur 
CentOS-regression: Gluster Build System

core: run many bricks within one glusterfsd process

2017-02-02T00:54:58+00:00

This patch adds support for multiple brick translator stacks running in
a single brick server process.  This reduces our per-brick memory usage
by approximately 3x, and our appetite for TCP ports even more.  It also
creates potential to avoid process/thread thrashing, and to improve QoS
by scheduling more carefully across the bricks, but realizing that
potential will require further work.

Multiplexing is controlled by the "cluster.brick-multiplex" global
option.  By default it's off, and bricks are started in separate
processes as before.  If multiplexing is enabled, then *compatible*
bricks (mostly those with the same transport options) will be started in
the same process.

Backport of:
> Change-Id: I45059454e51d6f4cbb29a4953359c09a408695cb
> BUG: 1385758
> Reviewed-on: https://review.gluster.org/14763

Change-Id: I4bce9080f6c93d50171823298fdf920258317ee8
BUG: 1418091
Signed-off-by: Jeff Darcy 
Reviewed-on: https://review.gluster.org/16496
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

tier : Tier as a service

2017-01-17T04:49:47+00:00

tierd is implemented by separating from rebalance process.

The commands affected:

1) Attach tier will trigger this process instead of old one
2) tier start and tier start force will also trigger this process.
3) volume status [tier] will show tier daemon as a process instead
of task and normal tier status and tier detach status works.
4) tier stop implemented.
5) detach tier implemented separately along with new detach tier
status
6) volume tier volname status will work using the changes.
7) volume set works

This patch has separated the tier translator from the legacy
DHT rebalance code. It now sends the RPCs from the CLI
to glusterd separate to the DHT rebalance code.
The daemon is now a service, similar to the snapshot daemon,
and can be viewed using the volume status command.

The code for the validation and commit phase are the same
as the earlier tier validation code in DHT rebalance.

The “brickop” phase has been changed so that the status
command can use this framework.

The service management framework is now used.
DHT rebalance does not use this framework.

This service framework takes care of :

*) spawning the daemon, killing it and other such processes.
*) volume set options , which are written on the volfile.
*) restart and reconfigure functions. Restart is to restart
the daemon at two points
        1)after gluster goes down and comes up.
        2) to stop detach tier.
*) reconfigure is used to make immediate volfile changes.
By doing this, we don’t restart the daemon.
it has the code to rewrite the volfile for topological
changes too (which comes into place during add and remove brick).

With this patch the log, pid, and volfile are separated
and put into respective directories.

Change-Id: I3681d0d66894714b55aa02ca2a30ac000362a399
BUG: 1313838
Signed-off-by: hari gowtham 
Reviewed-on: http://review.gluster.org/13365
Smoke: Gluster Build System 
Tested-by: hari gowtham 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Dan Lambright 
Reviewed-by: Atin Mukherjee

glusterfs : Removing logically dead code

2017-01-07T14:56:47+00:00

CID = 1356502

BUG: 789278
Change-Id: I11a814addc6607902c12aca8f4efec5741cbd7d3
Signed-off-by: Muthu-vigneshwaran 
Reviewed-on: http://review.gluster.org/16028
Tested-by: Muthu Vigneshwaran 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Jeff Darcy 
Reviewed-by: Niels de Vos

socket: socket disconnect should wait for poller thread exit

2016-12-22T04:49:19+00:00

When SSL is enabled or if "transport.socket.own-thread" option is set
then socket_poller is run as different thread. Currently during
disconnect or PARENT_DOWN scenario we don't wait for this thread
to terminate. PARENT_DOWN will disconnect the socket layer and
cleanup resources used by socket_poller.

Therefore before disconnect we should wait for poller thread to exit.

Change-Id: I71f984b47d260ffd979102f180a99a0bed29f0d6
BUG: 1404181
Signed-off-by: Rajesh Joseph 
Reviewed-on: http://review.gluster.org/16141
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kaushal M 
Reviewed-by: Raghavendra Talur 
Reviewed-by: Raghavendra G

socket: Keepalives should happen on IPv6 as well as IPv4

2016-12-16T20:30:05+00:00

Summary:
- Check for AF_INET *and* AF_INET6.
- This is a cherry-pick of D3057373 to 3.8

Signed-off-by: Shreyas Siravara 
Change-Id: I53eb79284eddfee6e13821c6570809f575b96769
BUG: 1405478
Reviewed-on: http://review.gluster.org/16167
Reviewed-by: Jeff Darcy 
Tested-by: Jeff Darcy 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Vijay Bellur

rpc: fix for race between rpc and protocol/client

2016-12-05T14:09:48+00:00

It is possible that the notification thread which notifies
protocol/client layer about the disconnection is put to sleep
and meanwhile, a fuse thread or a timer thread initiates and
completes reconnection to the brick. The notification thread
is then woken up and protocol/client layer updates its flags
to indicate that network is disconnected. No reconnection is
initiated because reconnection is rpc-lib layer's responsibility
and its flags indicate that connection is connected.

Fix: Serialize connect and disconnect notify

Credit: Raghavendra Talur 
Change-Id: I8ff5d1a3283b47f5c26848a42016a40bc34ffc1d
BUG: 1386626
Signed-off-by: Rajesh Joseph 
Reviewed-on: http://review.gluster.org/15916
Reviewed-by: Raghavendra G 
Smoke: Gluster Build System 
Reviewed-by: Raghavendra Talur 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

dht/md-cache: Filter invalidate if the file is made a linkto file

2016-12-02T10:46:58+00:00

Upcall as a part of setattr, sends an invalidation and the
invalidation carries the resulting stat value. When a file
is converted to linkto files, even then an invalidation
is set and as a result the mountpoint shows the sticky
bit in the stat of the file.
eg: ---------T. 945 root root 0 Nov  8 10:14 hardlink.999

Fix:
When dht recieves a notification of sticky bit change, it updates
the flag, to indicate md-cache to send the subsequent lookup.

Change-Id: Ic2fd7a5b196db0754f9b97072e644e6bf69da606
BUG: 1392713
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/15789
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Niels de Vos 
CentOS-regression: Gluster Build System 
Reviewed-by: Susant Palai 
Reviewed-by: Rajesh Joseph