Whamcloud - gitweb
LU-14361 statahead: regularized fname statahead pattern 08/41308/40
authorQian Yingjin <qian@ddn.com>
Mon, 10 Oct 2022 08:17:31 +0000 (04:17 -0400)
committerOleg Drokin <green@whamcloud.com>
Wed, 3 Jan 2024 03:02:07 +0000 (03:02 +0000)
commite10bf68d7c3eb71a7e84986089c020aa4c9f98f7
treef62e7e9d762977145e3122805c5230799d6f42d8
parent0031ab371ddd0ffda157629fb64c101203a3b1e2
LU-14361 statahead: regularized fname statahead pattern

Some applications do stat() calls under a directory within which
all children files have regularized file name:
- mdtest benchmark tool: mdtest.$rank.$i
- ML/AI with ingested data that have typically a format rule of
  the filename in the directory.

The most common format for regularized file name is that the
suffix part of the file name is number-indexing.
However, in the current statahead mechanism, the statahead is
populated by the order of the hash of the file name via readdir()
calls, not a kind of sorting order.

In this patch, we improve the statahead to prefetch attributes for
the files with regularized indexing file name via asynchronous
batching RPC.

This patch adds the support to do statahead for these kinds of
applications, which can be optimized, but without opendir()/
close() to start/stop statahead thread explicitly.

Instead, the statahead thread will stop and quit when found
that there was no acitivy for more than a certain time period
(i.e. 30 seconds).

Test-Parameters: mdtcount=4 mdscount=2 testlist=sanity env=ONLY=27p,ONLY_REPEAT=5
Test-Parameters: mdtcount=4 mdscount=2 testlist=sanity env=ONLY=27p,ONLY_REPEAT=5
Test-Parameters: mdtcount=4 mdscount=2 testlist=sanity env=ONLY=123f,ONLY_REPEAT=10
Test-Parameters: mdtcount=4 mdscount=2 testlist=sanity env=ONLY=123f,ONLY_REPEAT=10
Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: Ide11ec5a651ae74884ddbe1cecede4f5c961e38d
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/41308
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
lustre/llite/file.c
lustre/llite/llite_internal.h
lustre/llite/llite_lib.c
lustre/llite/lproc_llite.c
lustre/llite/statahead.c
lustre/ptlrpc/layout.c
lustre/tests/sanity.sh