EX-585 tests: add hot-pools.sh for lamigo and lpurge tests

EX-585 tests: add hot-pools.sh for lamigo and lpurge tests

This patch adds hot-pools.sh test script to test lamigo and
lpurge utilities for Hot Pools feature.

Lustre-commit: a57e6a22dfec43fbcda5bc259c7be7b076f465c2
Lustre-change: https://review.whamcloud.com/39616

EX-1859 tests: sleep some time in hot-pools.sh test_6()

This patch fixes hot-pools.sh test_6() to sleep some time
before dumping lamigo stats and verifying chlg_user param.

Lustre-commit: 92fedefe242a1ec6d486929b6c355b6d2238e56b
Lustre-change: https://review.whamcloud.com/40532

EX-1860 tests: flush client cache in hot-pools.sh test_56()

This patch fixes hot-pools.sh test_56() to flush client cache
before resynchronizing the mirrored file and also shorten the
test running time by skipping checksum validation on files
larger than 2GB.

Lustre-commit: 6d4c30106bdef4aea13145f297dc92bc6f0d5bfa
Lustre-change: https://review.whamcloud.com/40627

EX-585 tests: improve hot-pools.sh to support multiple MDTs

This patch improves hot-pools.sh to support multiple MDSs and MDTs.

Lustre-commit: 5be8906ae03f6fb62185198a1a25ddb23b3bf29a
Lustre-change: https://review.whamcloud.com/40719

EX-2745 tests: add hot-pools.sh test 56 into except list

Temporarily add hot-pools.sh test 56 into except list because
the test caused EX-2745 and EX-2757.

Lustre-commit: c00b98d2e5e215923090fec7e8d7b24be07745e9
Lustre-change: https://review.whamcloud.com/41873

EX-2689 tests: initiate hot pools test env inside subtests

Before running hot pools tests, we need to register changelog user,
create OST pools and mount Lustre client on server node. Since
changelog user deregistration and OST pools destroy are called inside
changelog_register() and create_ost_pools() as trap commands on EXIT,
we need to initiate the test env inside subtests.

This patch also fixes hot-pools.sh subtests 4,7,8,10 to redirect
both stdout and stderr to a temporary debug file.

Lustre-commit: 5e9341303abe776a549971c70ee2705c482a1d71
Lustre-change: https://review.whamcloud.com/41820

Test-Parameters: trivial clientdistro=el8.3 testlist=hot-pools \
serverextra_install_params="--lipe-job lipe --lipe-build 0 -k zfs"

Test-Parameters: trivial testlist=hot-pools \
mdscount=2 mdtcount=4 \
serverextra_install_params="--lipe-job lipe --lipe-build 0 -k zfs"

Test-Parameters: trivial testlist=hot-pools,ost-pools,sanity-flr \
serverextra_install_params="--lipe-job lipe --lipe-build 0 -k zfs"

Change-Id: I03ca24d4e74a3152be6880b01683147965194f00
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/41842
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>