LU-17840 kfilnd: Race between peer del RKEY reuse

LU-17840 kfilnd: Race between peer del RKEY reuse

kfilnd_peer object deletion is a two step process. First a flag
(kfilnd_peer::kp_remove_peer = 1) is atomically set in the object to
mark it for removal via a call to kfilnd_peer_del(). Then, the next
caller of kfilnd_peer_put() will atomically modify this flag
(kfilnd_peer::kp_remove_peer = 2) again to denote that it is removing
the peer from the rhashtable before actually removing the object.

The window between marking a peer for deletion and removing it from
the peer cache allows a race where an RKEY may be re-used. For
example:

Thread 1: Posts tagged receive with RKEY based on
      peerA::kp_local_session_key X and tn_mr_key Y
Thread 1: Cancels tagged receive
Thread 1: kfilnd_peer_del() -> peerA::kp_remove_peer = 1
Thread 2: kfilnd_peer_put() -> peerA::kp_remove_peer = 2
Thread 1: kfilnd_peer_put() -> kfilnd_tn_finalize() -> releases
	tn_mr_key Y
Thread 3: allocates tn_mr_key Y
Thread 3: Fetches peerA with kp_local_session_key X
Thread 2: Removes peerA from rhashtable

At this point, thread 3 has the same RKEY used by thread 1.

The fix is to check on the peer lookup path whether a peer found in
the rhashtable has been marked for removal. If it has then we perform
the lookup again. We do this in a loop until either no peer is found,
or a peer is found that has not been marked for removal.

To reduce the size of this window, the process for kfilnd_peer
deletion is modified so that the first thread to call
kfilnd_peer_del() will also remove the peer from the rhashtable.

HPE-bug-id: LUS-12312
Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: Ibbbb38cd5ee2d90956791f8350dafbee5fe5d888
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/55071
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Ian Ziemba <ian.ziemba@hpe.com>
Reviewed-by: Ron Gredvig <ron.gredvig@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>