LU-6261 gnilnd: Cray interconnect rollup

LU-6261 gnilnd: Cray interconnect rollup

I am leaving a few lines in structure definitions that are
longer than 80 columns. It's not the time to reformat the
whole structure.
-------------------------------------------------------------
Subject: Update debug messages for rca and quiesce events
Description:
Change informational message when receiving down event for
better tracking of RCA event issues to display under console
logging.
Clarify the message printed when we receive connection request
from a down node.
Simplify quiesce messages to just report the start and end of
quiesce.
-------------------------------------------------------------
Subject: Limit fma block allocations.
Description:
Under network pressure whereby thousands of nodes need to
reconnect all at the same time, routers can run out of memory
allocating fma blocks for mailboxes since the previous ones
cannot be cleaned up until a new connection is established.
Limit the amount of fma blocks that can be allocated to 3
quarters of total memory. This leaves memory free for other
allocations which tend to be much smaller than the mailboxs.
This should only be needed on service nodes.
Clean up some whitespace in kgn_data_t.
-------------------------------------------------------------
Subject: Double deregistration error.
Description:
lustre:18920 introduced a bug which causes us to deregister
the same memory twice when the transfer is unaligned.
Clean up the tx_buffer_copy after a deregistration so that
kgnilnd_rdma can properly register the memory on the retry.
-------------------------------------------------------------
Subject: Stack reset is causing pings to timeout instead of
failing immediately.
Description:
It is possible to register with the same MDD after a stack
reset causing pings to timeout instead of failing right away.
During a stack reset, we need to deregister with a hold
timeout set so we don't use the same mdd after the stack reset
is complete.
This was found by gnilnd regression test 110c.
-------------------------------------------------------------
Subject: Post rdma resource error
Description:
Handle kgni_post_rdma resource error by unmapping the tx and
put it back on the TX_MAPQ.
Also fixed:
fast_reconn variable check was using the pointer instead of
it's value.
bug that causes a stall when calling
kgnilnd_wakeup_rca_thread() when regression test causes
startup failures and the rca thread has not started yet.
Only call sock_release if socket was created.
Changed some stats prints to print unsigned values so they
don't show as negative.
-------------------------------------------------------------
Subject: limit kgnilnd conns in purgatory
Description:
Currently kgnilnd allows for an infinite number of connections
in purgatory, which in the face of a missed rca event can
cause nodes to slowly run out of memory from continued timed
out connection requests to those halted or dead nodes.
This mod makes the following changes to alleviate this issue:
1. Add a module parameter and live tunable allowing us to
limit
   number of connections per peer held in purgatory.
2. Remove the fast reconnect path on the server by making
   that tunable contain different settings for computes
   and service nodes. fast_reconnect is on for computes and
   off for service nodes. This setting can be changed on a
   live system.
3. In the kgnilnd reaper code utilize the tunable and remove
   the oldest purgatoried connections as new connections are
   put into purgatory. This will keep memory usage down and
   allow a system to stay up in the face of nodes being down
   and rca not informing us that they are down.
-------------------------------------------------------------
Subject: Update kgnilnd to be KNC aware.
Description:
Kgnilnd currently ignores rt_accel nodetype events coming from
RCA. This is incorrect as KNC's down and up events are
reported as rt_accel.
Since we currently ignore rt_accel events this causes us to
continually attempt to talk to down KNC nodes.
With this mod we now recognize rt_accel events allowing us to
prevent
communications with down KNC nodes.
-------------------------------------------------------------
Subject: Always notify LNET on GNILND_RCA_NODE_DOWN
Description:
When an LNET router fails it can take router_ping_timeout +
live_router_check_interval seconds for all peers to detect the
down router. For peers on a gni network this can be over two
minutes. During this time peers will continue to use the
failed router.
In some situations gnilnd will receive an event from RCA
notifying that the node is down within 30 seconds of the node
failure. This is much faster than relying on the router
pinger, so gnilnd should call lnet_notify() to notify LNET,
upon receipt of the RCA event, that a peer is down.
-------------------------------------------------------------
Subject: Add fast reconnect path and update lnet_notify last
alive timestamp.
Description:
A lustre client can time out a router during a blade failure
which causes multiple quiesce cycles.
When we time out a connection, reconnect even if there are no
tx's waiting to be sent. This causes an lnet_notify up
notification so we don't need to wait
for the router pinger to bring the connection back up.
At the end of a quiesce, call lnet_notify that the peer is
still up which updates the last alive timestamp.
Various debug message cleanup.
-------------------------------------------------------------
Subject: gnilnd proc_dir_entry port - part 2
Description:
PDE_DATA is defined by libcfs in Cray-master and therefore
only needed by b2_5
-------------------------------------------------------------
Subject: gnilnd proc_dir_entry port
Description:
In SLES12 create_proc_entry and create_proc_read_entry have
been removed, and struct proc_dir_entry is no longer public.
This mod ports all proc functions to use seq_file.
-------------------------------------------------------------
Subject: Remove system.h from gnilnd
Description:
There is no longer system.h for x86 and gnilnd doesn't seem to
need it.
Remove it from gnilnd include.
-------------------------------------------------------------

Signed-off-by: Chuck Fossen <chuckf@cray.com>
Change-Id: Iad14538751cc50fbd03fd3d4876ca41f4c0a223f
Reviewed-on: http://review.whamcloud.com/13812
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Patrick Farrell <paf@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>