glusterd: Don't start bricks if server quorum is not met

Upon glusterd restart if it is observered that the server quorum
isn't met anymore due to changes to the "server-quorum-ratio"
global option, the bricks should be stopped if they are running.
Also if glusterd has been restarted, and if server quorum is not
applicable for a volume, do not restart the bricks corresponding
to the volume to make sure that bricks that have been brought
down purposely, say for maintenance, are not brought up. This
commit moves this check that was previously inside
"glusterd_spawn_daemons" to "glusterd_restart_bricks" instead.

> Change-Id: I0a44a2e7cad0739ed7d56d2d67ab58058716de6b
> BUG: 1345727
> Signed-off-by: Samikshan Bairagya <samikshan@gmail.com>
> Reviewed-on: http://review.gluster.org/14758
> Smoke: Gluster Build System <jenkins@build.gluster.org>
> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
> CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
> Reviewed-by: Jeff Darcy <jdarcy@redhat.com>

(cherry picked from commit 807b9a135d697f175fc9933f1d23fb67b0cc6c7d)

Change-Id: I0a44a2e7cad0739ed7d56d2d67ab58058716de6b
BUG: 1353814
Signed-off-by: Samikshan Bairagya <samikshan@gmail.com>
Reviewed-on: http://review.gluster.org/14876
Smoke: Gluster Build System <jenkins@build.gluster.org>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Atin Mukherjee <amukherj@redhat.com>

1 files changed, 62 insertions, 0 deletions

diff --git a/tests/bugs/glusterd/bug-1345727-bricks-stop-on-no-quorum-validation.t b/tests/bugs/glusterd/bug-1345727-bricks-stop-on-no-quorum-validation.t
new file mode 100644
index 00000000000..e5951e0a039
--- /dev/null
+++ b/tests/bugs/glusterd/bug-1345727-bricks-stop-on-no-quorum-validation.t
@@ -0,0 +1,62 @@
+#!/bin/bash
+
+# Test case for quorum validation in glusterd for syncop framework
+
+. $(dirname $0)/../../include.rc
+. $(dirname $0)/../../volume.rc
+. $(dirname $0)/../../cluster.rc
+
+cleanup;
+
+TEST launch_cluster 3
+
+TEST $CLI_1 peer probe $H2;
+TEST $CLI_1 peer probe $H3;
+EXPECT_WITHIN $PROBE_TIMEOUT 2 peer_count
+
+# Lets create the volume and set quorum type as a server
+TEST $CLI_1 volume create $V0 $H1:$B1/${V0}1 $H2:$B2/${V0}2 $H3:$B3/${V0}3
+TEST $CLI_1 volume set $V0 cluster.server-quorum-type server
+
+# Start the volume
+TEST $CLI_1 volume start $V0
+
+EXPECT_WITHIN $PROCESS_UP_TIMEOUT "1" brick_up_status_1 $V0 $H1 $B1/${V0}1
+EXPECT_WITHIN $PROCESS_UP_TIMEOUT "1" brick_up_status_1 $V0 $H2 $B2/${V0}2
+EXPECT_WITHIN $PROCESS_UP_TIMEOUT "1" brick_up_status_1 $V0 $H3 $B3/${V0}3
+
+# Bring down 2nd and 3rd glusterd
+TEST kill_glusterd 2
+TEST kill_glusterd 3
+
+# Server quorum is not met. Brick on 1st node must be down
+EXPECT "0" brick_up_status_1 $V0 $H1 $B1/${V0}1
+
+# Set quorum ratio 95. means 95 % or more than 95% nodes of total available node
+# should be available for performing volume operation.
+# i.e. Server-side quorum is met if the number of nodes that are available is
+# greater than or equal to 'quorum-ratio' times the number of nodes in the
+# cluster
+
+TEST $CLI_1 volume set all cluster.server-quorum-ratio 95
+
+# Bring back 2nd glusterd
+TEST $glusterd_2
+
+EXPECT_WITHIN $PROBE_TIMEOUT 1 peer_count
+
+# Server quorum is still not met. Bricks should be down on 1st and 2nd nodes
+EXPECT "0" brick_up_status_1 $V0 $H1 $B1/${V0}1
+EXPECT "0" brick_up_status_1 $V0 $H2 $B2/${V0}2
+
+# Bring back 3rd glusterd
+TEST $glusterd_3
+EXPECT_WITHIN $PROBE_TIMEOUT 2 peer_count
+
+# Server quorum is met now. Bricks should be up on all nodes
+EXPECT_WITHIN $PROCESS_UP_TIMEOUT "1" brick_up_status_1 $V0 $H1 $B1/${V0}1
+EXPECT_WITHIN $PROCESS_UP_TIMEOUT "1" brick_up_status_1 $V0 $H2 $B2/${V0}2
+EXPECT_WITHIN $PROCESS_UP_TIMEOUT "1" brick_up_status_1 $V0 $H3 $B3/${V0}3
+
+cleanup;
+