LU-13606 lnet: Allow router to forward to healthier NID

LU-13606 lnet: Allow router to forward to healthier NID

When a final-hop router (aka edge router) is forwarding a message,
if both the originator and destination of the message are mutli-rail
capable, then allow the router to choose a new destination lpni if
the one selected by the message originator is unhealthy or down.

Test-Parameters: trivial
HPE-bug-id: LUS-8905
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I4676edc0395584c9a8c396930f2db3d6ffd99eba
Reviewed-on: https://review.whamcloud.com/38798
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>