debug/delay-gen: Implement delay-generation feature

Background:
I was working on a customer issue where the disks were responding some times
after seconds. It was becoming very difficult to recreate the issues in our
labs, so had to come up with this feature.

Requirements:
We need an xlator which can delay x% of ops for y micro seconds.
We should be able to enable delays for specific fops.

This feature is modeled after error-gen. Most of the logic
is borrowed from that xlator. This is a minimum implementation
of the feature which satisfied the requirements I had. May be
in future with more requirements and understanding of the problem
further we can improve upon this implementation.

Here are the commands and what they do:
Enable delay-gen: (This is similar to how err-gen is enabled on the brick side)
- gluster volume set <volname> delay-gen posix

Set the percentage of fops that need to be delayed
- gluster volume set <volname> delay-gen.delay-percentage 50
Default is 10%

Set the delay in micro seconds
- gluster volume set <volname> delay-gen.delay-duration 500000
Default is 100000

Set comma separated fops to be delayed
- gluster v set r2 delay-gen.enable read,write
Default is all fops.

Fixes #257
Change-Id: Ib547bd39cc024c9cdb63754d21e3aa62fc9d6473
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: https://review.gluster.org/17591
Smoke: Gluster Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Jeff Darcy <jeff@pl.atyp.us>

1 files changed, 39 insertions, 0 deletions

diff --git a/tests/features/delay-gen.t b/tests/features/delay-gen.t
new file mode 100755
index 00000000000..38b2a499dc0
--- /dev/null
+++ b/tests/features/delay-gen.t
@@ -0,0 +1,39 @@
+#!/bin/bash
+
+. $(dirname $0)/../include.rc
+. $(dirname $0)/../volume.rc
+
+cleanup;
+
+TEST glusterd
+TEST pidof glusterd
+
+TEST $CLI volume create $V0 $H0:$B0/${V0}1
+
+EXPECT "$V0" volinfo_field $V0 'Volume Name'
+EXPECT 'Created' volinfo_field $V0 'Status'
+
+TEST $CLI volume set $V0 delay-gen posix
+TEST $CLI volume set $V0 delay-gen.delay-duration 1000000
+TEST $CLI volume set $V0 delay-gen.delay-percentage 100
+TEST $CLI volume set $V0 delay-gen.enable read,write
+
+TEST $CLI volume start $V0
+EXPECT 'Started' volinfo_field $V0 'Status'
+
+TEST $CLI volume profile $V0 start
+## Mount FUSE with caching disabled (read-write)
+TEST $GFS -s $H0 --volfile-id $V0 $M0
+
+TEST dd if=/dev/zero of=$M0/1 count=1 bs=128k oflag=sync
+
+#Write should take at least a second
+write_max_latency=$($CLI volume profile $V0 info | grep WRITE | awk 'BEGIN {max = 0} {if ($6 > max) max=$6;} END {print max}' | cut -d. -f 1 | egrep "[0-9]{7,}")
+
+#Create should not take a second
+create_max_latency=$($CLI volume profile $V0 info | grep CREATE | awk 'BEGIN {max = 0} {if ($6 > max) max=$6;} END {print max}' | cut -d. -f 1 | egrep "[0-9]{7,}")
+
+TEST [ ! -z $write_max_latency ];
+TEST [ -z $create_max_latency ];
+
+cleanup;