LU-4431 lnet: 1/3/2014 update for Cray interconnects

LU-4431 lnet: 1/3/2014 update for Cray interconnects

This is a rollup of changes for gnilnd containing bug fixes and
enhancements since the LU-3008 submission.
The new header file gni_pub.h contains code that will allow gnilnd to
be built upstream. It will not pass the checkpatch.pl script since it
was developed previously for gni drivers and ugni.

To build a lustre client including gnilnd for Aries (XC30):
sh autogen.sh
./configure --with-linux=/path_to_centos6.4_kernel
--disable-ldiskfs-build --disable-doc --disable-liblustre
--disable-server --with-o2ib=no --without-sysio --disable-checksum
--enable-utils --enable-gni
GNICPPFLAGS='-DCONFIG_CRAY_ARIES -I$PWD/lnet/klnds/gnilnd'

Included changes:
----------------------------------------------------------------------
Subject: Unused VIRT fma block does not get cleaned up during stack
reset.
Description:
This is an edge case where we allocate a GNILND_FMABLK_VIRT fma block
but before using it, a GNILND_FMABLK_PHYS is freed up for use. The
GNILND_FMABLK_VIRT fma block didn't get associated with a conn thus,
during a stack reset, the fma block will not be cleaned up and we
assert.
Changed kgnilnd_unmap_phys_fmablk to unmap all fma blocks instead of
just PHYS blocks.
Renamed kgnilnd_unmap_phys_fmablk to kgnilnd_unmap_fma_blocks.
---------------------------------------------------------------------
Subject: New LND type "gip" gnilnd use of IP addresses.
Description:
Add a new LND type to the libcfs_netstrfns array that converts IP
address to/from addresses.
This change also allows us to use hostnames or IP addresses to specify
the direct attached file systems in the /etc/fstab file.
----------------------------------------------------------------------
Subject: Changes to gnilnd for non-Cray modified kernel.
Description:
Get MAC address from arp table for generating a nicaddr.
Change the rca_inject proc file name to peer_state.
Add the GNIIPLND type to lnd_type for Apollo builds.
Add TO_GNILND_timeout for building upstream without gni-headers.
Add gni_pub.h for use by upstream builds.
Fix slab-freed debug statement which references a freed structure.
Remove unused code that was needed for the gemini simulator.
----------------------------------------------------------------------
Subject: Gnilnd upstream sync LU-4069
Description:
LU-4069 build: cleanup from GOTO(label, -ERRNO)
Cleanup the code from GOTO(label, -ERRNO) and other bad GOTOs.
----------------------------------------------------------------------
Subject: Merge gnilnd changes from LU-2800.
Description:
Upstream changes from LU-2800 need merging to gnilnd.
----------------------------------------------------------------------
Subject: Adjust to Cray-master cfs changes.
Description:
ll_proc_doxxxx macros have been removed in the cfs layer. Use the
corresponding proc_doxxxx function instead.
----------------------------------------------------------------------
Subject: Fix offset problem in reverse rdma edge case.
Description:
The call to lnet_copy_flat2kiov() was used incorrectly passing in an
offset to the source buffer being copied from. The offset is used to
decide how many bytes will be copied from the first iov which causes
the routine to only copy the difference between the nob and the offset
to the first iov. Since only one iov is ever passed in, all the bytes
need to come from that first iov.
----------------------------------------------------------------------
Subject: Remove CFS kernel abstraction dependencies from GniLND
Description:
The CFS kernel abstactions are being removed upstream.
Incorporate LU-1346 changes to GniLND.
We should not call deamonize as we are using kthread_create/
kthread_run.
----------------------------------------------------------------------
Subject: Fix race in closing connection in response to EFAULT error
Description:
The previous mod causes a regression in closing a conn in two threads
at once. We should use kngilnd_close_conn() instead of
kgnilnd_close_conn_locked() since it checks the state of the conn
before actually closing.
----------------------------------------------------------------------
Subject: Close connection in response to EFAULT error
Description:
After the companion node's GPU fell off the bus, we get an mdd invalid
hardware error even though the mdd's that have been inspected look ok.
The hardware error is returned in the rdma cq event in
kgnilnd_check_rdma_cq() and we respond by nak'ing the message.
The plan is to close the connection since the connection is still
alive and subsequent rdma's will continue to fail.
If the connection cannot be reestablished then the communication to
this node will cease so at least jobs will not continue to be
scheduled on this node.
----------------------------------------------------------------------
Subject: Fix outstanding conns issue during kgnilnd_base_shutdown
Description:
Currently in kgnilnd_base_shutdown there is a small race with the
datagram thread that can cause a wildcard dgram to match while in the
process of shutting down and will generate a nak datagram to be
generated. This new datagram needs to be canceled, currently we go
straight into full shutdown without doing so causing us to assert.

This mod adds a cancel function that iterates over all outstanding
non-wc dgrams regardless of net and cancels them. It then schedules
the device to clean up the remaining conns.
----------------------------------------------------------------------
Subject: Canceled dgram deadlock
Description:
When adding conns of canceled dgrams to purgatory, a call was made to
kgnilnd_destroy_conn_ep().
This is inappropriate since we are inside the kgn_peer_conn_lock and
kgnilnd_destroy_conn_ep() takes a mutex lock.
Avoid this behavior by setting the conn state to CLOSED instead of
DONE and allowing the scheduler thread to finish the conn's
processing.
----------------------------------------------------------------------
Subject: LND obtains node up/down information when creating peer.
Description:
Before creating a peer, check the state of the node to see if it is up
or down.
This is done by calling krca_get_sysnodes() and walking through the
array for the nid of interest and checking it's state.
kgnilnd_finish_connect() also creates peers but we do not need to
check in this instance since the request is from the peer.
----------------------------------------------------------------------
Subject: debug and lbug capability for kgnilnd client EFAULT errors
Description:
Currently when kgnilnd encounters an EFAULT within a nak message it
kills the TX and prints a message to the screen. It does not crash or
print enough information for us to diagnose if the problem is hardware
or software.
This patch will allow us to bug a compute when it starts getting a
large number EFAULTs programatically. It also prints out the memhandle
of the mdd that we should be inspecting for validity.
----------------------------------------------------------------------
Subject: Fix kgnilnd q_time setting
Description:
When we recieve a GNILND_MSG_PUT_REQ we send a GNILND_MSG_PUT_ACK in
response when we send that response we were not setting the tx's
q_time.
This mod fixes that problem and allows us to see the correct tx age
when calling kgnilnd_tx_done.
Changed a cast from long to unsigned long.
Corrected a tab issue.
----------------------------------------------------------------------
Subject: Add gnilnd eager receive limit
Description:
Add module parameter eager_credits which limits the amount of messages
that can be eager received.
Currently, we continue to allocate memory with each message which can
cause out of memory issues if a IB interface goes down.
Add counter to track eager allocations. Return -ENOMEM to lnet if we
exceed the number of credits allocated. Lnet will drop messages when
eager receive returns with an error.
Set the default eager_credits to 256k - this limits us to using 512 MB
of memory. This path is used mainly when there is an imbalance in the
either "side" of the router. There should be no performance impact
provided the normal tuning is done.
----------------------------------------------------------------------
Subject: kgnilnd static analysis fixes
Description:
Static analysis has found different bugs to fix.
This mod is a package of minor static analysis fixes.

1. Remove unsigned compare against 0 in kgnilnd_setup_immediate_buffer
2. Fix unintentional integer overflow in kgnilnd_proc_run_cksum_test
3. Fix Nesting issue in kgnilnd_map_phys_fmablk
4. Fix kgnilnd_process_nak return code nto being used.
5. Remove unneeded code in kgnilnd_del_conn_or_peer
6. Fix uninitialized value in kgnilnd_queue_tx
----------------------------------------------------------------------
Subject: kgnilnd_probe_for_dgram() race during shutdown.
Description:
Canceling dgrams while shutting down can cause an assertion in
kgnilnd_probe_for_dgram().
If the shutdown thread calls kgnilnd_probe_for_dgram concurrently with
the dram mover thread,
both may get the same dgram from the postdata_probe_by_id kgni
function.
Move the lock release to after postdata_test_by_id which actually
removes the dgram from the list.
Added fail_loc to test fix.
----------------------------------------------------------------------
Subject: Mailbox corruption fix
Description:
Canceled dgrams could have been completed at the peer during the
cancelation.
The mailbox could then be used for another peer therefore allowing two
peers to use the same mailbox.
The conns for canceled dgrams need to be put in purgatory so they
don't get reused until a connection has been established for the peer.
During release of a canceled dgram, we hook up the conn to the peer
then put it in purgatory.
Added flag to kgnilnd_release_dgram to indicate we are shutting down
or going through a stack reset.
Added some tracing of gnd_ndgrams.
----------------------------------------------------------------------
Subject: LND support for knc
Description:
For knc nodes, use GNI_PTAG_LND_KNC. Use two scheduler threads for
better performance.
libcfs includes calls to cfs_crypto_crc32_pclmul_register() and
cfs_crypto_crc32_pclmul_unregister() but those files are not built for
k1om architecture.
----------------------------------------------------------------------

Signed-off-by: Chuck Fossen <chuckf@cray.com>
Change-Id: Ie8be6d7e8b6623a49d7a75ec878a23cf5385cc46
Reviewed-on: http://review.whamcloud.com/8744
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>