LU-3008 lnet: Update support for Cray's interconnects

LU-3008 lnet: Update support for Cray's interconnects

This patch updates gnilnd to include all of Cray's
patches for the last year since the initial push.

Included changes

----------------------------------------------------------------------
Subject
Reverse rdma kgnilnd fixes
Description
A LNET_PUT when matched on the receiving side is parsed it
can call kgnilnd_recv with a mlen == 0, previously the reverse_rdma
code for kgnilnd did not handle this and asserted. This mod adds
handling of the case when mlen is set to 0 and also adds handling
when an LNET_GET's lnetmsg is == NULL, which is another case which
is handled in non reverse_rdma path but not in the reverse_rdma path.

----------------------------------------------------------------------
Subject
Gnilnd refcount changes
Description
This mod adjusts connection refcount handling to bring the
reference adding and removing in line with what was expected, this
was brought up during the whamcloud review but left undone on their
end.

----------------------------------------------------------------------
Subject
kgnilnd peer_timeout enhancement for peer_health
Description
Currently on router nodes kgnilnd peer_health is enabled, when
peer_health is enabled it sets a default timeout factor of
kgn_timeout+kgn_timeout/8. This value currently cannot be adjusted
except through adjust kgn_timeout. This mod allows for the user to
increase the value by setting the module parameter peer_timeout in
conjunction with peer_health.

When peer_timeout is set and peer_health is enabled the timeout
passed to lnet will be what the user has specified as long as it is
greater than the previous fudge calculation. If the user specifies a
value less than fudge kgnilnd will fail to load and throw an error
to the console.

Changes
1. Added module parameter peer_timeout, when peer_health is enabled
   this allows manipulation of the ni_peertimeout value passed to
   lnet.

----------------------------------------------------------------------
Subject
kgnilnd conn double free refcount fix
Description
Currently kgnilnd has a possible race condition on service nodes
between two scheduler threads. When a connection is scheduled another
scheduler can act upon the conn before the first has decremented its
reference.
Currently kgnilnd_conn_decref uses a seperate atomic_read after it
decrefs to decide what to do next. There is the possibility that two
threads calling kgnilnd_conn_decref could see the same value of zero
even though one thread would have brought the refcount to one and the
other to zero. The same issue can occur with kgnilnd_peer_decref.

This mod introduces changes to the scheduler to prevent two decrefs
at the same time in different scheduler threads. Also it updates
kgnilnd_conn_decref to utilize the value that is returned by
atomic_dec_return instead of doing a second atomic_read to verify
the reference count.

Changes
1. Changed kgnilnd_conn_decref to use the val returned by
   atomic_sub_return instead of doing atomic_reads to get the value.
2. Changed kgnilnd_peer_decref to use the val returned by
   atomic_sub_return instead of doing atomic_reads to get the value.
3. Updated kgnilnd_schedule_conn and kgnilnd_schedule_process conn
   so that when a connection is scheduled from within a scheduler
   thread it carries the reference forward instead of removing it.
   This in addition to the kgnilnd_conn_decref change should remove
   the double free problem.
4. Changed assertions in kgnilnd_peer_addref, kgnilnd_conn_addref so
   they catch when the value is incremented up from 0 to 1.
5. Use magic value to verify conn is not being free twice.

----------------------------------------------------------------------
Subject
Debug for mailbox corrruption.
Description
We have two peers (routers) writing to the same mailbox of a compute
node.

Add more debug to identify the cause of two peers getting the same
mailbox information.
- Store both the previous nid and the previous purgatory nid for this
  mailbox.
- Store the dgram type in the conn so we can tell if the conn
  resulted from a matched wildcard or a direct connection request.
- Keep track of the total allocations of a mailbox and the current
  number of allocations.
- Add a proc file peer_conns with information containing the peer's
  connection information.
  - writing a nid value (echo 1234 > /proc/kgnilnd/peer_conns) will
    allow the read (cat /proc/kgnilnd/peer_conns) to produce a list
    of conns associated with the specified nid.

----------------------------------------------------------------------
Subject
Ignore events generated from 'xtcli set/clr_reserve'
Description
'xtcli set_reserve' and 'xtcli clr_reserve' operations overload the
ec_node_unavailable event as described in bug 785850.  Since gnilnd
uses ec_node_unavailable events, we need to ignore them when they
originate from those commands.

----------------------------------------------------------------------
Subject
Close connection upon receipt of RCA unavailable event.
Description
When a blade is powered down, messages sent to the nodes will
cause ORB timeouts which causes a quiesce and ORB scrub. The quiesce
causes gnilnd to bump it's timeouts so we continue sending traffic
causing more ORB timeouts.

----------------------------------------------------------------------
Subject
kgnilnd_dgram_mover thread runtime deadline
Description
Currently there is no deadline associated with starting outbound
dgrams within the kgnilnd_dgram_mover thread. The thread will loop
while the list is not empty. When there is a large amount of network
problems the thread could run for a very long time. This mod adds a
deadline check to make sure the dgram thread stops attempting to post
dgrams after the deadline passes, the thread will schedule itself and
be woken up normally after time has passed to continue its work.

Changes
1. Added deadline to kgnilnd_dgram_mover so
   kgnilnd_start_outbound_dgrams is bounded in runtime by size of
   list and by a maximum runtime deadline.
2. Added error injection to verify dgram deadline.
3. Added module parameter to adjust deadline of dgram thread.

----------------------------------------------------------------------
Subject
fix peer_conn_lock deadlock
Description
kgnilnd_tx_done() called with lock held.
There is an error case whereby kgnilnd_tx_done will be called by
kgnilnd_queue_tx(). This can cause a deadlock if lnet calls back
needing the write lock.

Remove call to kgnilnd_tx_done since the tx will be processsed by
kgnilnd_process_fmaq() (like the EAGAIN case).

----------------------------------------------------------------------
Subject
Make kgnilnd_bump_timeouts aware of DONE connections
Description
Currently when kgnilnd comes out of quiesce all connections timeouts
are bumped so they dont close from the period they were paused.
kgnilnd_bump_timeouts schedules all the connections on a peer
including ones that are in purgatory in the GNILND_CONN_DONE state.
These connections are not supposed to be put through the scheduler
once they are in the DONE state.

A LBUG can occur if after the quiesce occurs the scheduler thread
does not push the newly scheduled conns through the state machine
fast enough. This can leave DONE conns on the scheduled list when
stack reset is triggered. Stack reset then puts any scheduled conns
through kgnilnd_complete_closed_conn which when the function sees a
conn in the GNILND_CONN_DONE state it asserts.

Changes
1. Add if statement so kgnilnd_bump_timeouts does not schedule DONE
   connections.

----------------------------------------------------------------------
Subject
Subscribe GNILND to UXACT errors
Description
Aries has a new type of error that GNILND needs to be subscribed to
for stack reset initiation. This mod adds that error type to our
callback subscription routine.

Changes
1. Add GNI_ERRMASK_UNKNOWN_TRANSACTION to mask passed into
   kgnilnd_subscribe_errors function.

----------------------------------------------------------------------
Subject
kgnilnd reverse bte rdma transactions
Description
Currently GNILND executes all of its kgni bte rdma transactions
using GNI_POST_RDMA_PUT, on cascade systems this can cause IOMMU
thrashing on router nodes from the many computes initiating rdma
to the single service node. This can cause linear performance
degradation as more and more computes attempt to write into a single
service nodes memory space. To alleviate this problem we will change
how rdmas are done we will use GNI_POST_RDMA_GET, so the service node
will initiate the transfer of data to it instead of thousands of
clients all trying at once. By adding a run time tunable that allows
us to switch to using GNI_POST_RDMA_GET we can govern the RDMA from
the receiving node.

Changes
1. Added new message types that exist side by side with current
   infrastructure so different nodes can have rdma setting tuned
   and all nodes will handle the messages.
2. Added tunables so that the REVERSE setting can be adjusted at
   run time.
3. Added support for non byte aligned data transfers so that gets
   will succeed when non byte aligned offsets and lengths are
   provided to kgnilnd.
4. Added the capability to send checksum information in the message
   being sent to the side that will be initiating the rdma.
   This works side by side with existing rdma checksum capabilities.
5. Corrected rdma nak problems when RDMA mapping fails for a specific
   type of tx.
6. Added counters to rdma when a copy needs to be made due to
   unaligned data, this will allow us to see if performance is
   hindered because of a large number of vmalloc calls have to be
   made.
7. Changed the entire call tree for rdma to support the handling of
   the new message types.
8. On Aries platforms service nodes will be defaulted to
   GNILND_REVERSE_GET, compute nodes defaulted to GNILND_REVERSE_PUT.

----------------------------------------------------------------------
Subject
Generate/check checksum over the number of bytes actually transferred
Description
It is possible for PUTs to have a different length than the
length stored in lntmsg->msg_len since LNET can adjust this
length based on it's buffer size and offset.
lnet_try_match_md() sets the mlength that we use to do the
RDMA transfer.

Therfore we need to compute checksum using tx->tx_rdma_desc.length
and verify the checksum using length returned in the
msg->gnm_u.completion.gncm_retval which contains the actual number
of bytes transmitted.

----------------------------------------------------------------------
Subject
GniLND needs to filter accelerator events.
Description
Change the kgnilnd_rca thread to filter out accelerator events.
----------------------------------------------------------------------
Subject
kgnilnd BTE Delivery MODE tunable
Description
Currently kgnilnd only exposes a few options to tune for kgni's rdma
bte delivery mode. This works well for Gemini systems, but on Cascade
we would like finer grained control. This mod allows us to change the
delivery mode at run time through the exposed tunable interface.
Giving us the capability to tune the delivery modes without having to
restart the system or make code changes.

Changes
1.  Added tunable bte_dlvr_mode which takes a mask/number for the
    delivery mode and uses that to set the bte delivery option for
    rdma.
2.  Removed extraneous tunables that were only single tunable
    specific.
3.  Added Gemini and Aries header options if in the future we need to
    change the defaults on Aries or Gemini.

----------------------------------------------------------------------
Subject
GniLND connection serialization, debug for compute bad message type.
Description
Introduce a semaphore for connection processing serialization within
the scheduler thread for bugs 789853 and 789855.
  - The main work of the scheduler thread is now protected by a read
    semaphore.
  - When kgnilnd_process_conns needs to do work on a connection, it
    takes a write semaphore.

----------------------------------------------------------------------
Subject
GniLND rca_thread exit fix.
Description
Change the kgnilnd_rca thread from exiting when receiving an error
from krca_wait_event.

----------------------------------------------------------------------
Subject
GniLND kgnilnd_recv message type unknown
Description
Add debug to print out more info in kgnilnd_recv() default case of
the gnm_type switch statement.

----------------------------------------------------------------------
Subject
fix fma_blk state when mdd is invalidated.
Description
Currently when an VIRT_MAPPED fma_blk is unmapped kgnilnd doesnt
change its state to IDLE. Since it doesnt the code that finds a free
mbox will use mboxes within the fma_blk even though its mdd has been
invalidated, causing dgram exchanges to contain bad mailboxes.

This change will mark the fma_blk as having its mdd invalidated.

----------------------------------------------------------------------
Subject
gnilnd/rca integration
Description
Subscribe for the rca events ec_node_unavailable, ec_node_available
and ec_node_failed to prevent reconnect attempts to downed nodes.
We do not use the event to kill a live connection.

----------------------------------------------------------------------
Subject
kgnilnd eager_recv double free fix
Description
Currently the function call kgnilnd_eager_recv does no verification
that the connection passed into it with an rx message is alive and
valid. Normally this is without issue except when connections are
being closed and opened on routers. A connection could be in the
process of being destroyed and have its refcount incremented.
The next call to kgnilnd_recv could cause a double free.

This mod alleviates this by doing a reverse lookup on the connection
based on the information we can validate within the rx message. By
using a read_lock on kgn_peer_conn_lock we can then lookup the
connection based on its nid and verify it conn_stamp matches the one
the message is expecting. If we find a valid connection that matches
we then increment that connections refcount while the lock is held,
preventing it from disappearing until after the receive. Without the
lock and reverse lookup we could end up looking at already freed
memory.

This race was showing itself through an fma_blk assertion on the
router nodes, when 2 destroy_conn calls occured in parallel sometimes
one would get past an if(fma_blk) check and then find that the
fma_blk had already been set to 0.

----------------------------------------------------------------------
Subject
Sequence kgnilnd tx use with close of connections.
Description
Currently kgnilnd makes an incorrect assumption
that when a conn is closed and the connection is removed from
the cqid lookup table that no tx's are in use by other threads.

What can happen is one of the other scheduler threads can be
in the process of using a tx and has called
kgnilnd_tx_del_state_locked. This can race against
kgnilnd_complete_closed_conn in a different scheduler thread as it
attempts to remove all existing tx's from the conn's tx_ref_table.
That kgnilnd_complete_closed_conn calls kgnilnd_tx_del_state_locked
on the connection's tx's, and since a tx could still be in use in the
first scheduler thread an exception can occur.

This mod marks the conn as having tx's in use when the first thread
has a read_lock on the kgnilnd_peer_conn_lock.

Changes
1. Added to kgn_conn_t an atomic gnc_tx_in_use that is incremented
   any time kgnilnd_validate_tx_ev_id is called.
2. Added a decref to the conn's gnc_tx_in_use after the function
   is finished using the tx.
3. Added a check in kgnilnd_process_conns that barriers entry for a
   given connection into kgnilnd_complete_closed_conn until
   gnc_tx_in_use is 0. Once the conn is removed by the close call from
   the cqid hash table only existing in use tx's from before the close
   will prevent the close from completing so no livelocks should be
   possible.

----------------------------------------------------------------------
Subject
Add kgnilnd scheduler thread runtime deadline
Description
This mod makes sure that the kgnilnd scheduler threads
are not sitting on the cpu longer than neccessary by adding a deadline
that forces a yield after the deadline is hit. The amount of time
that the scheduler will allow itself to run without scheduling
is configurable via module parameter in 1 second intervals.

It was also found that the nice value of the scheduler threads
is preventing the heartbeat system from working correctly on
compute nodes with only a single scheduler thread. So we are
changing default nice value of thread to 0 to allow other
threads to run.

Changes
1. Added sched_timeout module parameter to allow changing of
   default scheduler thread deadline.
2. Added deadline check to kgnilnd_process_conns so it does
   not spin in its while loop forever.
3. Added error injection to verify deadline is checked and
   calls to yield occur.
3. Added sched_nice module parameter to allow adjustment of
   scheduler thread priority seperate from other kgnilnd
   threads.

----------------------------------------------------------------------
Subject
Cleanup kgnilnd_schedule_conn races during conn close
Description
This patch reworks the previous debug patch and adds a
debug framework that addresses the shortcomings previous patch.

We are also removing an extraneous kgnilnd_schedule_conn
call from kgnilnd_finish_connect that was causing a large number of
the schedule after close occurences.

There is still a chance that a conn can be scheduled after close but
the current refcount framework is designed to counteract issues that
arise when that happens, making the removal of the assertion valid.

----------------------------------------------------------------------
Subject
Repost WC dgram when OOM event occurs
Description
Currently when kgnilnd runs out of GART space while attempting to
repost a wildcard datagram, the system asserts and tips over. Instead
we can put into place a mechanism that allows WC datagrams to be
reposted when the OOM conditon resolves.

This mod removes the assertion and puts into place a mechanism within
the dgram mover thread to post wildcards when neccessary. This allows
the system to stay up instead of crashing. When posting a dgram
fails a D_NETERROR message will be written out to the console.

----------------------------------------------------------------------
Subject
Workaround and additional debug for scheduler assertion
Description
This mod adds debug to get a better analysis of the gnc_scheduled
problem. It also has a workaround; the call to
kgnilnd_complete_closed_conn will short circuit and let
kgnilnd_process_conns handle the schedule normally when it sees that
gnc_scheduled != GNILND_CONN_PROCESS instead of asserting. I have also
added debug to all the calls to kgnilnd_schedule_conn so we can find
the call that is causing the assertion.

----------------------------------------------------------------------
Subject
Remove assertion and attempt recovery on mailbox corruption
Description
Previous mods have addressed the sequencing that could cause mailbox
corruption by fixing the state machine and adding timeouts. This mod
builds on those and makes the detection of issues relating to the
mailbox a correctable error. Instead of asserting we will now close
the connection when we detect corruption occuring and utilize the
purgatory system to attempt to get things back in order.  The previous
changes allow us to do this as they prevent the close sequence
corruption from spiraling out of control.

Changes
        1. Removed assertion in kgnilnd_check_fma_rx on seqno
           corruption and replace with a statement that closes the
           connection and returns -EIO. This should allow the system
           to continue without causing the node to come down.
        2. Added debug so when we do detect corruption it will be
           tagged in the console. This will allow us to see how often
           the problem occurs and if it contributes to system
           problems.

----------------------------------------------------------------------
Subject
Fix race condition and sequence kgnilnd connection closing
Description
There is a race between the scheduler thread and
kgnilnd_close_conn_locked. While we take the kgn_peer_conn_lock to
close the connection, the scheduler threads dont look at it when they
check the gnc_state. We could end up all the way through the close
state machine by the time the kgnilnd_close_conn_locked function
finishes tripping an assertion. To correct this race and improve
sequencing we need to make sure when changing the conn's gnc_state
we grab the write_lock on kgn_peer_conn_lock.

Changes
        1. In kgnilnd_send_conn_close when setting the conn's
           gnc_state to GNILND_CONN_CLOSED added a write_lock to make
           sure we are sequencing the close with other threads that
           might be changing the connections state.
----------------------------------------------------------------------

Signed-off-by: James R. Shimek <jshimek@cray.com>
Change-Id: I5b8de3b72cdc17b32134cb2532c9ad7dc4fa621c
Reviewed-on: http://review.whamcloud.com/5815
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>