Whamcloud - gitweb
LU-18351 obdclass: jobstat scaling 07/56607/35
authorShaun Tancheff <shaun.tancheff@hpe.com>
Fri, 6 Dec 2024 00:11:50 +0000 (07:11 +0700)
committerOleg Drokin <green@whamcloud.com>
Mon, 16 Dec 2024 08:14:48 +0000 (08:14 +0000)
commitcad59b9b72fc72182574f4ce782e12d1e98f84fd
treefc720d461bdc69899e2f600f7b9b25d1d6b089c8
parentf824ccc8f1db3246dd1d64ba9ae4e52e765f0fa8
LU-18351 obdclass: jobstat scaling

When there is a large number of jobstats (>100k) list walking in
seq_write does not scale and can trigger cpus soft lock-ups.

Prefer red black trees for easier lookup, removal and simplified
tree walk during seq_write()

Add an rbtree with a generated sequential id as jobs are added
to simplify iteration with restarts. The larger the amount of
data being written via seq_write() the more likely and frequent
a restart is needed to seek to new position which can be done
quickly with an rbtree.

Add an lru list to so periodic trim of jobs does not need to
walk the entire tree of jobs.

Unpatched:
  jobs  50000    100000   200000  350000  500000
  time  2.096s   3.672s   7.504s  11.875s 16.721s

Patched:
  jobs  50000    100000   200000  350000  500000
  time  1.366s   3.296s   6.763s  8.895s  11.965s

HPE-bug-id: LUS-12621
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: Icca1365af5db761ed89ee9a8a97ce4ded65b8832
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/56607
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
libcfs/autoconf/lustre-libcfs.m4
libcfs/include/libcfs/linux/linux-misc.h
lustre/include/lprocfs_status.h
lustre/include/obd_support.h
lustre/mdt/mdt_lproc.c
lustre/obdclass/lprocfs_jobstats.c
lustre/ofd/ofd_internal.h
lustre/target/tgt_main.c
lustre/tests/sanity.sh