Add halo-min-samples option, better swap logic, edge case fixes

Summary:
- Changes halo-decision to be based on the lowest halo value observed
- Adds halo-min-sample option to wait until N latency samples have been
  gathered prior to activating halos.
- Fixed 3 edge cases where halo's weren't being correctly
  config'd, or not configured as quickly as is possible.  Namely:
  1. Don't mark a child down if there's no better alternative (and you'd
  no longer satisfy min/max replicas); fixes unneccessary flapping.
  2. If a child goes down and this causes us to fall below max_replicas,
  swap in a warm child immediately if it is within our halo latency
  (don't wait around for the next "ping"); swaps in a new child
  immediately helping with resiliency.
  3. If the child latency is within the halo, and it's currently marked
  up, mark it down if it's the highest latency child and the number of
  children is > max_replicas; this will allow us to support the
  SHD use-case where we can "beam" a single copy to a geo and have it
  replicate within the geo after that.
- More commenting

Test Plan:
- Run halo prove tests
- Pointed compiled code at gfsglobal.prn2, tested out an NFS daemon and
  FUSE mounts to ensure they worked as expected on a large scale
  cluster.

Reviewers: dph, jackl, cjh, mmckeen

Reviewed By: mmckeen

FB-commit-id: 7e2e8ae6b8ec62a5e0b31c9fd6100c81795b3424

Change-Id: Iba2b2f1bc848b4546cb96117ff1895f83953a4f8
Signed-off-by: Kevin Vigor <kvigor@fb.com>
Reviewed-on: http://review.gluster.org/16304
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
Smoke: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Shreyas Siravara <sshreyas@fb.com>

1 files changed, 21 insertions, 10 deletions

diff --git a/rpc/rpc-lib/src/rpc-clnt-ping.c b/rpc/rpc-lib/src/rpc-clnt-ping.c
index a48a265b71d..7ce066dec5f 100644
--- a/rpc/rpc-lib/src/rpc-clnt-ping.c
+++ b/rpc/rpc-lib/src/rpc-clnt-ping.c
@@ -173,17 +173,28 @@ out:
 }
 
 void
-_update_client_latency (call_frame_t *frame, double elapsed_usec)
+_update_client_latency (const rpc_clnt_connection_t *conn,
+                        call_frame_t *frame,
+                        uint64_t elapsed_usec)
 {
         fop_latency_t *lat;
 
         lat = &frame->this->client_latency;
 
+        if (elapsed_usec < lat->min) {
+                lat->min = elapsed_usec;
+        }
+
+        if (elapsed_usec > lat->max) {
+                lat->max = elapsed_usec;
+        }
+
         lat->total += elapsed_usec;
         lat->count++;
         lat->mean = lat->mean + (elapsed_usec - lat->mean) / lat->count;
-        gf_log (THIS->name, GF_LOG_DEBUG, "Ping latency is %0.6lf ms, "
-                "avg: %0.6lf ms, count:%ld", elapsed_usec / 1000.0,
+        gf_log (THIS->name, GF_LOG_DEBUG, "%s - Ping latency is %0.6lf ms, "
+                "avg: %0.6lf ms, count:%ld",
+                conn->trans->peerinfo.identifier, elapsed_usec / 1000.0,
                 lat->mean / 1000.0, lat->count);
 }
 
@@ -217,13 +228,6 @@ rpc_clnt_ping_cbk (struct rpc_req *req, struct iovec *iov, int count,
 
         pthread_mutex_lock (&conn->lock);
         {
-                timespec_now (&now);
-                timespec_sub (&local->submit_time, &now, &delta);
-                latency_usec = delta.tv_sec * 1000000UL +
-                               delta.tv_nsec / 1000UL;
-
-                _update_client_latency (frame, (double)latency_usec);
-
                 if (req->rpc_status == -1) {
                         unref = rpc_clnt_remove_ping_timer_locked (local->rpc);
                         if (unref) {
@@ -240,6 +244,13 @@ rpc_clnt_ping_cbk (struct rpc_req *req, struct iovec *iov, int count,
                         goto unlock;
                 }
 
+                timespec_now (&now);
+                timespec_sub (&local->submit_time, &now, &delta);
+                latency_usec = delta.tv_sec * 1000000UL +
+                               delta.tv_nsec / 1000UL;
+
+                _update_client_latency (conn, frame, latency_usec);
+                call_notify = _gf_true;
                 unref = rpc_clnt_remove_ping_timer_locked (local->rpc);
                 if (__rpc_clnt_rearm_ping_timer (local->rpc,
                                                  rpc_clnt_start_ping) == -1) {