debug/io-stats: Adding stat for weighted & unweighted average latency

Summary:
- Our current approach to measuring "average fop latency" is badly
  flawed in that it doesn't weight the FOPs correctly according to how
  many occurred in the time interval.  This makes Statisticians very
  sad.  This patch adds an internally computed weighted average
  latency which will be far more efficient to display via ODS, as well
  as having the benefit of not being complete nonsense.
- This is a port of D3148415 & D3405772 to 3.8

Reviewers: kvigor, dph, sshreyas

Reviewed By: sshreyas

Change-Id: Ie3618f279b545610b7ed1a8482243fcc8dc53217
Reviewed-on: https://review.gluster.org/18192
Reviewed-by: Shreyas Siravara <sshreyas@fb.com>
Smoke: Gluster Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>

1 files changed, 43 insertions, 0 deletions

diff --git a/tests/basic/ios-dump.t b/tests/basic/ios-dump.t
new file mode 100644
index 00000000000..ddeb19d01ac
--- /dev/null
+++ b/tests/basic/ios-dump.t
@@ -0,0 +1,43 @@
+#!/bin/bash
+
+. $(dirname $0)/../include.rc
+. $(dirname $0)/../volume.rc
+
+cleanup;
+
+function check_brick_inter_stats() {
+  local counter="$1"
+  local inter_cnt=""
+
+  inter_cnt=$(grep -h "\".*inter.*$counter\"" \
+    /var/lib/glusterd/stats/glusterfsd*.dump 2>/dev/null | 
+    grep -v '\"0.0000\"' | wc -l)
+  if (( $inter_cnt == 3 )); then
+    echo "Y"
+  else
+    echo "N"
+  fi
+}
+
+TEST glusterd
+TEST pidof glusterd
+TEST $CLI volume create $V0 replica 3 $H0:$B0/${V0}{0,1,2}
+TEST $CLI volume set $V0 diagnostics.stats-dump-interval 5
+TEST $CLI volume set $V0 diagnostics.count-fop-hits on
+TEST $CLI volume set $V0 diagnostics.latency-measurement on
+TEST $CLI volume start $V0
+TEST glusterfs --volfile-id=/$V0 --volfile-server=$H0 $M0 --attribute-timeout=0 --entry-timeout=0
+
+# Generate some FOPs
+cd $M0
+for i in {1..10}; do
+  mkdir a
+  cd a
+  for g in {1..10}; do
+    dd if=/dev/zero of=test$g bs=128k count=1
+  done
+done
+
+EXPECT_WITHIN 30 "Y" check_brick_inter_stats fop.weighted_latency_ave_usec
+
+cleanup