LU-16235 hsm: check CDT state before adding actions llog

LU-16235 hsm: check CDT state before adding actions llog

Don't allow HSM requests to be added to the actions llog when
cdt_state is in CDT_STOPPED/CDT_STOPPING as the CDT is unavailable, or
in CDT_INIT as any HSM requests in the llog may not have been fully
processed and so cdt_last_cookie may not have been set appropriately,
otherwise a colliding cookie value can be reused in
mdt_agent_record_add() and the assertions in
cdt_agent_record_hash_add() can be triggered:

"ASSERTION( carl0->carl_cat_idx == carl1->carl_cat_idx ) failed"
"ASSERTION( carl0->carl_rec_idx == carl1->carl_rec_idx ) failed"

Requests needed to implement the Remove Archive on Last Unlink (RAoLU)
policy are allowed when the CDT is shutdown, as those are safe
operations. They are also allowed during CDT initialization, even
though this can lead to the assertions being triggered, as doing so
maintains administrator expectations regarding file archives always
being removed when the RAoLU policy is enabled. This could possibly be
improved by e.g. failing when mdt_handle_last_unlink() is not able to
add an HSM remove request, or saving the requests in an llog so they
can be sent if the CDT is available later.

For the same reason, the llog needs to be processed before setting
cdt_state to CDT_RUNNING in the coordinator thread.

Change-Id: I4b5f5ee22f74827b31d8ed5917a8fc16e35d1f16
Signed-off-by: Nikitas Angelinas <nikitas.angelinas@hpe.com>
HPE-bug-id: LUS-8231, LUS-11064
Fixes: e26d7cc3 ("LU-14399 hsm: process hsm_actions in coordinator")
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/48842
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Etienne AUJAMES <eaujames@ddn.com>