path: root/rpc
diff options
authorPranith Kumar K <>2016-10-19 15:50:50 +0530
committerRaghavendra G <>2016-10-24 23:42:20 -0700
commita6b63e11b7758cf1bfcb67985e25ec02845f0995 (patch)
treea091bd67a68903ebeb5fdc67062f7610b6c9eb4d /rpc
parentbca6d0ba54d12d389cfb5c2b37fb8cc12a7e044b (diff)
rpc: Fix the race between notification and reconnection
Problem: There was a hang because unlock on an entry failed with ENOTCONN. Client thinks the connection is down where as server thinks the connection is up. This is the race we are seeing: 1) Connection from client to the brick disconnects. 2) Saved frames unwind is called which unwinds all frames that were wound before disconnect. 3) connection from client to the brick happens and setvolume. 4) Disconnect notification for the connection in 1) comes now and calls client_rpc_notify() which marks the connection to be offline even when the connection is up. This is happening because I/O can retrigger connection before disconnect notification is sent to the higher layers in rpc. Fix: Notify the higher layers that a disconnect happened and then go ahead with reconnect logic. For the logs which point to the information above check: Thanks to Raghavendra G for suggesting the correct fix. BUG: 1386626 Change-Id: I3c84ba1f17010bd69049fa88ec5f0ae431f8cda9 Signed-off-by: Pranith Kumar K <> Reviewed-on: NetBSD-regression: NetBSD Build System <> Reviewed-by: Niels de Vos <> CentOS-regression: Gluster Build System <> Smoke: Gluster Build System <> Reviewed-by: Raghavendra G <>
Diffstat (limited to 'rpc')
1 files changed, 4 insertions, 3 deletions
diff --git a/rpc/rpc-lib/src/rpc-clnt.c b/rpc/rpc-lib/src/rpc-clnt.c
index e8a8ea2..3caab98 100644
--- a/rpc/rpc-lib/src/rpc-clnt.c
+++ b/rpc/rpc-lib/src/rpc-clnt.c
@@ -898,6 +898,10 @@ rpc_clnt_notify (rpc_transport_t *trans, void *mydata,
switch (event) {
+ if (clnt->notifyfn)
+ ret = clnt->notifyfn (clnt, clnt->mydata,
rpc_clnt_connection_cleanup (conn);
pthread_mutex_lock (&conn->lock);
@@ -921,9 +925,6 @@ rpc_clnt_notify (rpc_transport_t *trans, void *mydata,
pthread_mutex_unlock (&conn->lock);
- if (clnt->notifyfn)
- ret = clnt->notifyfn (clnt, clnt->mydata,
if (unref_clnt)
rpc_clnt_ref (clnt);