LU-1897 test: replay-single test_70b dbench not found

dbench is exiting before test_70b completes is work.

dbench is being used as a background load in this test.
In one observed case test was taking a little under 90s but
dbench was exiting at 62s. Duration was set at 90.

Presently dbench starts for 60s then the system sleeps
for 12s, then we start a new 60s timer in which we expect
dbench to be running.  There is a large window in which
this test will fail.

The main issue is the use of Duration for use as both the
timer of the real test and the timer of the background load.

I raised the real Duration and started the timer for the real
test before starting dbench such that the real test should
finish before dbench does or about the same time.

Signed-off-by: Keith Mannthey <keith.mannthey@intel.com>
Change-Id: Iba243fe3132a5fa677e5a7bcd09e491727ba092a
Reviewed-on: http://review.whamcloud.com/4973
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Hudson
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

diff --git a/lustre/tests/replay-single.sh b/lustre/tests/replay-single.sh
index 20f7b09..7750d01 100755 (executable)
--- a/lustre/tests/replay-single.sh
+++ b/lustre/tests/replay-single.sh
@@ -1880,10 +1880,11 @@ test_70b () {
        zconf_mount_clients $clients $MOUNT
 
        local duration=300
-       [ "$SLOW" = "no" ] && duration=60
+       [ "$SLOW" = "no" ] && duration=120
        # set duration to 900 because it takes some time to boot node
        [ "$FAILURE_MODE" = HARD ] && duration=900
 
+       local start_ts=$(date +%s)
        local cmd="rundbench 1 -t $duration"
        local pid=""
        do_nodesv $clients "set -x; MISSING_DBENCH_OK=$MISSING_DBENCH_OK \
@@ -1895,12 +1896,11 @@ test_70b () {
 
        # give rundbench a chance to start, bug 24118
        sleep 12
-       local elapsed=0
+       local elapsed=$(($(date +%s) - start_ts))
        local num_failovers=0
-       local start_ts=$(date +%s)
        while [ $elapsed -lt $duration ]; do
                if ! check_for_process $clients dbench; then
-                       error_noexit "dbench not found on some of $clients!"
+                       error_noexit "dbench not running on some of $clients!"
                        killall_process $clients dbench
                        break
                fi