Whamcloud - gitweb
LU-18556 hsm: optimize llog record modification 28/57428/18
authorAlexander Boyko <alexander.boyko@hpe.com>
Fri, 13 Dec 2024 12:57:17 +0000 (13:57 +0100)
committerOleg Drokin <green@whamcloud.com>
Thu, 12 Jun 2025 06:31:51 +0000 (06:31 +0000)
commit07f6a59ad71324ed253c2eba519cf095f346bde9
tree688771b71d06b7bfe9cccd4e7091ebc3ad36bb8d
parent4a9fc7cebaad24cadf3213906fda193d8c681226
LU-18556 hsm: optimize llog record modification

This commit introduces a new llog modification mechanism for HSM
operations to address inefficiencies caused by prior reliance on
catalog processing. The new approach directly modifies llog record,
eliminating the need for catalog-based processing and reducing
latency.

Key changes include:
* Replacing the hsm_action_item (HAI) with a full in-memory llog
 record representation, increasing memory usage by ~80 bytes per
 record but removing the need for a dedicated llog cookie hash
 table.
* Unifying the coordinator's read/store logic for HAI data into a
 single in-memory item shared by mdt_hsm_agent_send() and
 mdt_hsm_add_hsr(). This reduces memory allocation steps: only one
 cdt_agent_req allocation is now required during llog read
 operations, eliminating subsequent allocations/copies.

Performance results on VMs 2 MDTs/2 OSTs/2 Clients no-op copytool:
Test 1 (1M archive requests): 572s -> 187s (~3 times faster)
Test 2 (1M archive + 1M queued): 558s -> 392s (~1.4 times faster)

HPE-bug-id: LUS-12583
Signed-off-by: Alexander Boyko <alexander.boyko@hpe.com>
Change-Id: I4b6e697bc3b1f0cf2c76f5433b49affbc933c653
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/57428
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Nikitas Angelinas <nikitas.angelinas@hpe.com>
Reviewed-by: Etienne AUJAMES <eaujames@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
lustre/include/lustre_log.h
lustre/include/uapi/linux/lustre/lustre_idl.h
lustre/mdt/mdt_coordinator.c
lustre/mdt/mdt_hsm_cdt_actions.c
lustre/mdt/mdt_hsm_cdt_agent.c
lustre/mdt/mdt_hsm_cdt_requests.c
lustre/mdt/mdt_internal.h
lustre/mgs/mgs_llog.c
lustre/obdclass/llog.c
lustre/obdclass/llog_cat.c
lustre/obdclass/llog_osd.c