Whamcloud - gitweb
LU-14361 statahead: regularized fname statahead pattern
Some applications do stat() calls under a directory within which
all children files have regularized file name:
- mdtest benchmark tool: mdtest.$rank.$i
- ML/AI with ingested data that have typically a format rule of
the filename in the directory.
The most common format for regularized file name is that the
suffix part of the file name is number-indexing.
However, in the current statahead mechanism, the statahead is
populated by the order of the hash of the file name via readdir()
calls, not a kind of sorting order.
In this patch, we improve the statahead to prefetch attributes for
the files with regularized indexing file name via asynchronous
batching RPC.
This patch adds the support to do statahead for these kinds of
applications, which can be optimized, but without opendir()/
close() to start/stop statahead thread explicitly.
Instead, the statahead thread will stop and quit when found
that there was no acitivy for more than a certain time period
(i.e. 30 seconds).
Test-Parameters: mdtcount=4 mdscount=2 testlist=sanity env=ONLY=27p,ONLY_REPEAT=5
Test-Parameters: mdtcount=4 mdscount=2 testlist=sanity env=ONLY=27p,ONLY_REPEAT=5
Test-Parameters: mdtcount=4 mdscount=2 testlist=sanity env=ONLY=123f,ONLY_REPEAT=10
Test-Parameters: mdtcount=4 mdscount=2 testlist=sanity env=ONLY=123f,ONLY_REPEAT=10
Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: Ide11ec5a651ae74884ddbe1cecede4f5c961e38d
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/41308
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>