glusterd: fix brick restart parallelism

glusterd's brick restart logic is not always sequential as there is
atleast three different ways how the bricks are restarted.
1. through friend-sm and glusterd_spawn_daemons ()
2. through friend-sm and handling volume quorum action
3. through friend handshaking when there is a mimatch on quorum on
friend import.

In a brick multiplexing setup, glusterd ended up trying to spawn the
same brick process couple of times as almost in fraction of milliseconds
two threads hit glusterd_brick_start () because of which glusterd didn't
have any choice of rejecting any one of them as for both the case brick
start criteria met.

As a solution, it'd be better to control this madness by two different
flags, one is a boolean called start_triggered which indicates a brick
start has been triggered and it continues to be true till a brick dies
or killed, the second is a mutex lock to ensure for a particular brick
we don't end up getting into glusterd_brick_start () more than once at
same point of time.

Change-Id: I292f1e58d6971e111725e1baea1fe98b890b43e2
BUG: 1508283
Signed-off-by: Atin Mukherjee <amukherj@redhat.com>
(cherry picked from commit 82be66ef8e9e3127d41a4c843daf74c1d8aec4aa)

1 files changed, 12 insertions, 3 deletions

diff --git a/xlators/mgmt/glusterd/src/glusterd-server-quorum.c b/xlators/mgmt/glusterd/src/glusterd-server-quorum.c
index 659ff9ddfc9..3084c1eacb9 100644
--- a/xlators/mgmt/glusterd/src/glusterd-server-quorum.c
+++ b/xlators/mgmt/glusterd/src/glusterd-server-quorum.c
@@ -361,10 +361,19 @@ glusterd_do_volume_quorum_action (xlator_t *this, glusterd_volinfo_t *volinfo,
         list_for_each_entry (brickinfo, &volinfo->bricks, brick_list) {
                 if (!glusterd_is_local_brick (this, volinfo, brickinfo))
                         continue;
-                if (quorum_status == DOESNT_MEET_QUORUM)
+                if (quorum_status == DOESNT_MEET_QUORUM) {
                         glusterd_brick_stop (volinfo, brickinfo, _gf_false);
-                else
-                        glusterd_brick_start (volinfo, brickinfo, _gf_false);
+                } else {
+                        if (!brickinfo->start_triggered) {
+                                pthread_mutex_lock (&brickinfo->restart_mutex);
+                                {
+                                        glusterd_brick_start (volinfo,
+                                                              brickinfo,
+                                                              _gf_false);
+                                }
+                                pthread_mutex_unlock (&brickinfo->restart_mutex);
+                        }
+                }
         }
         volinfo->quorum_status = quorum_status;
         if (quorum_status == MEETS_QUORUM) {