LU-12222 lnet: Check if we're sending to ourselves

It's desirable to avoid taking a send credit when sending messages to
ourselves. Check if dst_nid is one of our own, and use the lolnd for
the send accordingly.

There are two exceptions:
1. Recovery messages must be sent to the lnet_ni that is being
   recovered.
2. If a source NID is specified then we need to send via the
   associated NI.

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I656c6b1ef18ccb9b18bca65839de7c487981ebdd
Reviewed-on: https://review.whamcloud.com/35778
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexandr Boyko <c17825@cray.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

diff --git a/lnet/lnet/lib-move.c b/lnet/lnet/lib-move.c
index 75a0650..262e432 100644 (file)
--- a/lnet/lnet/lib-move.c
+++ b/lnet/lnet/lib-move.c
@@ -2634,13 +2634,20 @@ lnet_select_pathway(lnet_nid_t src_nid, lnet_nid_t dst_nid,
 again:
 
        /*
-        * If we're being asked to send to the loopback interface, there
-        * is no need to go through any selection. We can just shortcut
-        * the entire process and send over lolnd
+        * If we're sending to ourselves then there is no need to go through
+        * any selection. We can shortcut the entire process and send over
+        * lolnd.
+        *
+        * However, we make two exceptions to this rule:
+        * 1. If the src_nid is specified then our API defines that we must send
+        *    via that interface.
+        * 2. Recovery messages must be sent to the lnet_ni that is being
+        *    recovered.
         */
        send_data.sd_msg = msg;
        send_data.sd_cpt = cpt;
-       if (LNET_NETTYP(LNET_NIDNET(dst_nid)) == LOLND) {
+       if (src_nid == LNET_NID_ANY && !msg->msg_recovery &&
+           lnet_nid2ni_locked(dst_nid, cpt)) {
                rc = lnet_handle_lo_send(&send_data);
                lnet_net_unlock(cpt);
                return rc;