LU-5216 hsm: cancel hsm actions running on CT when killed

LU-5216 hsm: cancel hsm actions running on CT when killed

This patch handles:

1. Unexpected client (data mover node) eviction could
cause on going hsm requests to be stuck in "STARTED"
state as the copy tool running on the data mover node
is not available anymore and requests could not be
finished. This patch unregisters the copy tool and
cancels all the requests on the copytool's agent.

2. The way to stop a copytool is to KILL it. In this
case also, as explained above, on going hsm operations
get stuck in "STARTED" state and if another copytool
tries to do any hsm activity on that file, it will not be
processed as the hsm status of the file is still STARTED.

In such cases i.e
if the copytool's agent is killed then we are just going
to add the hsm cancel request to agent record
and mark request state as ARS_CANCELED. mdt_cordinator()
thread running in the back ground looks into the request state
and marks it as CANCELED. This allows hsm status to be
maintained at proper state and allows any further
operation to proceed as expected.

Adding test_62 to sanity-hsm.sh to verify the fix

Test-Parameters: trivial testlist=sanity-hsm
Seagate-bug-id: MRP-2464
Signed-off-by: vinayakswami hariharmath <vinayakswami.hariharmath@seagate.com>
Signed-off-by: Sergey Cheremencev <c17829@cray.com>
Change-Id: Iceb1a3450bcbcec287fe11c6a9fce45fc6097e3c
Reviewed-on: https://review.whamcloud.com/24238
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Quentin Bouget <quentin.bouget@cea.fr>
Reviewed-by: Ben Evans <bevans@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>