LU-17510 obdclass: fix wake up when queuing close request.

LU-17510 obdclass: fix wake up when queuing close request.

The waitqueue for requests that need to be sent but that haven't been
allocated a slot is kept ordered by request arrival for fairness.  So
new requests are added to the end.

For requests other than 'close' there is a limit to the number of
active requests (slots) and requests are assigned to slot on a
first-come-first-served basis, so they are simply removed from the
head of the list.

For 'close' requests it is important that these not block indefinitely
behind other other requests so there is one slot that can only be used
by a close request - and only if no other slots are used by a close
request.  These requests do not follow a strict FIFO order.

When a non-"close" request completes we wake the first request on the
list.  There is no point searching all the way down the list for a
close request that could also be woken.  We only do that when a
"close" request completes.  This optimises the common case.

However: when a request is first queued we add it to the end of the
queue and then wake up the first deserving request if there is one.
When there are free slots, this is expected to wake the request just
queued.  When there are no free slots, nothing is woken.

When a "close" request is queued and added to the end of the queue
after other non-close requests, we need to potentially search to the
end of the queue for a close request to wake, just as we do when a
close request completes.  Unfortunately we don't.  This can result in
a close request blocking indefinitely.

So: change the wakeup in obd_get_mod_rpc_slot() to match the wakeup in
obd_put_mod_rpc_slot().  This ensure consistent handling and in
particular will handle a close request immediately if there are no
other close requests in flight.

Clarify comment in claim_mod_rpc_function() to make and perform minor
code cleanup there.

Lustre-change: https://review.whamcloud.com/54259
Lustre-commit: 7a2296a397381a5f6f9473b297f0062e8ff15948

Fixes: b5fde4d6c023 ("LU-17197 obdclass: preserve fairness when waiting for rpc slot")
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I7b658efc0298a091166f0f18ce460fc3148047eb
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/54689
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>