LU-6261 gnilnd: Use trylock for conn mutex.

LU-6261 gnilnd: Use trylock for conn mutex.

When converting to thread safe implementation, I missed the need for
the conn mutex to trylock because we may be holding the kgn_peer_conn
lock.
Change conn mutex to use a trylock in kgnilnd_sendmsg_trylock().
Add the module parameter thread_safe.
Disable thread safe on gemini.
In kgnilnd_create_conn(), use NOFS flag when vmalloc the tx_ref_table
to avoid possible hangs because of OOM condition.
Clean up info message for down event.

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I1896c12b421ae35f65d1816bbe3eb5599b664498
Reviewed-on: http://review.whamcloud.com/15434
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: James Shimek <jshimek@cray.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>