glusterd: Brick process fails to come up with brickmux on

Issue:
1- In a cluster of 3 Nodes N1, N2, N3. Create 3 volumes vol1,
vol2, vol3 with 3 bricks (one from each node)
2- Set cluster.brick-multiplex on
3- Start all 3 volumes
4- Check if all bricks on a node are running on same port
5- Kill N1
6- Set performance.readdir-ahead for volumes vol1, vol2, vol3
7- Bring N1 up and check volume status
8- All bricks processes not running on N1.

Root Cause -
Since, There is a diff in volfile versions in N1 as compared
to N2 and N3 therefore glusterd_import_friend_volume() is called.
glusterd_import_friend_volume() copies the new_volinfo and deletes
old_volinfo and then calls glusterd_start_bricks().
glusterd_start_bricks() looks for the volfiles and sends an rpc
request to glusterfs_handle_attach(). Now, since the volinfo
has been deleted by glusterd_delete_stale_volume()
from priv->volumes list before glusterd_start_bricks() and
glusterd_create_volfiles_and_notify_services() and
glusterd_list_add_order is called after glusterd_start_bricks(),
therefore the attach RPC req gets an empty volfile path
and that causes the brick to crash.

Fix- Call glusterd_list_add_order() and
glusterd_create_volfiles_and_notify_services before
glusterd_start_bricks() cal is made in glusterd_import_friend_volume

> Change-Id: Idfe0e8710f7eb77ca3ddfa1cabeb45b2987f41aa
> Bug: bz#1773856
> Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com>

Change-Id: Idfe0e8710f7eb77ca3ddfa1cabeb45b2987f41aa
Fixes: bz#1808966
Signed-off-by: Sanju Rakonde <srakonde@redhat.com>

1 files changed, 15 insertions, 13 deletions

diff --git a/xlators/mgmt/glusterd/src/glusterd-utils.c b/xlators/mgmt/glusterd/src/glusterd-utils.c
index 73e2dc26e65..90fcd5af24c 100644
--- a/xlators/mgmt/glusterd/src/glusterd-utils.c
+++ b/xlators/mgmt/glusterd/src/glusterd-utils.c
@@ -4690,16 +4690,6 @@ glusterd_import_friend_volume(dict_t *peer_data, int count)
         glusterd_volinfo_unref(old_volinfo);
     }
 
-    if (glusterd_is_volume_started(new_volinfo)) {
-        (void)glusterd_start_bricks(new_volinfo);
-        if (glusterd_is_snapd_enabled(new_volinfo)) {
-            svc = &(new_volinfo->snapd.svc);
-            if (svc->manager(svc, new_volinfo, PROC_START_NO_WAIT)) {
-                gf_event(EVENT_SVC_MANAGER_FAILED, "svc_name=%s", svc->name);
-            }
-        }
-    }
-
     ret = glusterd_store_volinfo(new_volinfo, GLUSTERD_VOLINFO_VER_AC_NONE);
     if (ret) {
         gf_msg(this->name, GF_LOG_ERROR, 0, GD_MSG_VOLINFO_STORE_FAIL,
@@ -4709,19 +4699,31 @@ glusterd_import_friend_volume(dict_t *peer_data, int count)
         goto out;
     }
 
-    ret = glusterd_create_volfiles_and_notify_services(new_volinfo);
+    ret = glusterd_create_volfiles(new_volinfo);
     if (ret)
         goto out;
 
+    glusterd_list_add_order(&new_volinfo->vol_list, &priv->volumes,
+                            glusterd_compare_volume_name);
+
+    if (glusterd_is_volume_started(new_volinfo)) {
+        (void)glusterd_start_bricks(new_volinfo);
+        if (glusterd_is_snapd_enabled(new_volinfo)) {
+            svc = &(new_volinfo->snapd.svc);
+            if (svc->manager(svc, new_volinfo, PROC_START_NO_WAIT)) {
+                gf_event(EVENT_SVC_MANAGER_FAILED, "svc_name=%s", svc->name);
+            }
+        }
+    }
+
     ret = glusterd_import_quota_conf(peer_data, count, new_volinfo, "volume");
     if (ret) {
         gf_event(EVENT_IMPORT_QUOTA_CONF_FAILED, "volume=%s",
                  new_volinfo->volname);
         goto out;
     }
-    glusterd_list_add_order(&new_volinfo->vol_list, &priv->volumes,
-                            glusterd_compare_volume_name);
 
+    ret = glusterd_fetchspec_notify(this);
 out:
     gf_msg_debug("glusterd", 0, "Returning with ret: %d", ret);
     return ret;