From: Doug Oucharek Date: Wed, 7 Jun 2017 00:09:36 +0000 (-0700) Subject: LUDOC-375 lnet: Add ko2iblnd tuning parameters X-Git-Tag: 2.10.0~4 X-Git-Url: https://git.whamcloud.com/?a=commitdiff_plain;h=0efc02f3e813b8d0576cfc97b8f2242f2cfcef91;p=doc%2Fmanual.git LUDOC-375 lnet: Add ko2iblnd tuning parameters The Jira ticket calls for the new parameter wrq_sge to be added to the manual. I'm using this as an opportunity to add a table with all the ko2iblnd tuning parameters. Signed-off-by: Doug Oucharek Change-Id: Ic01da4ccdfaafb7662dd9dc498560a512957aabc Reviewed-on: https://review.whamcloud.com/27474 Tested-by: Jenkins Reviewed-by: Joseph Gmitter --- diff --git a/LustreTuning.xml b/LustreTuning.xml index b3854f0..0b0a2c5 100644 --- a/LustreTuning.xml +++ b/LustreTuning.xml @@ -626,6 +626,429 @@ cpu_partition_table= default values are automatically set and are chosen to work well across a number of typical scenarios. +
+ ko2iblnd Tuning + The following table outlines the ko2iblnd module parameters to be used + for tuning: + + + + + + + + + + Module Parameter + + + + + Default Value + + + + + Description + + + + + + + + + service + + + + + 987 + + + + Service number (within RDMA_PS_TCP). + + + + + + cksum + + + + + 0 + + + + Set non-zero to enable message (not RDMA) checksums. + + + + + + timeout + + + + + 50 + + + + Timeout in seconds. + + + + + + nscheds + + + + + 0 + + + + Number of threads in each scheduler pool (per CPT). Value of + zero means we derive the number from the number of cores. + + + + + + conns_per_peer + + + + + 4 (OmniPath), 1 (Everything else) + + + + Introduced in 2.10. Number of connections to each peer. Messages + are sent round-robin over the connection pool. Provides signifiant + improvement with OmniPath. + + + + + + ntx + + + + + 512 + + + + Number of message descriptors allocated for each pool at + startup. Grows at runtime. Shared by all CPTs. + + + + + + credits + + + + + 256 + + + + Number of concurrent sends on network. + + + + + + peer_credits + + + + + 8 + + + + Number of concurrent sends to 1 peer. Related/limited by IB + queue size. + + + + + + peer_credits_hiw + + + + + 0 + + + + When eagerly to return credits. + + + + + + peer_buffer_credits + + + + + 0 + + + + Number per-peer router buffer credits. + + + + + + peer_timeout + + + + + 180 + + + + Seconds without aliveness news to declare peer dead (less than + or equal to 0 to disable). + + + + + + ipif_name + + + + + ib0 + + + + IPoIB interface name. + + + + + + retry_count + + + + + 5 + + + + Retransmissions when no ACK received. + + + + + + rnr_retry_count + + + + + 6 + + + + RNR retransmissions. + + + + + + keepalive + + + + + 100 + + + + Idle time in seconds before sending a keepalive. + + + + + + ib_mtu + + + + + 0 + + + + IB MTU 256/512/1024/2048/4096. + + + + + + concurrent_sends + + + + + 0 + + + + Send work-queue sizing. If zero, derived from + map_on_demand and peer_credits. + + + + + + + map_on_demand + + + + + 0 (pre-4.8 Linux) 1 (4.8 Linux onward) 32 (OmniPath) + + + + Number of fragments reserved for connection. If zero, use + global memory region (found to be security issue). If non-zero, use + FMR or FastReg for memory registration. Value needs to agree between + both peers of connection. + + + + + + fmr_pool_size + + + + + 512 + + + + Size of fmr pool on each CPT (>= ntx / 4). Grows at runtime. + + + + + + + fmr_flush_trigger + + + + + 384 + + + + Number dirty FMRs that triggers pool flush. + + + + + + fmr_cache + + + + + 1 + + + + Non-zero to enable FMR caching. + + + + + + dev_failover + + + + + 0 + + + + HCA failover for bonding (0 OFF, 1 ON, other values reserved). + + + + + + + require_privileged_port + + + + + 0 + + + + Require privileged port when accepting connection. + + + + + + use_privileged_port + + + + + 1 + + + + Use privileged port when initiating connection. + + + + + + wrq_sge + + + + + 2 + + + + Introduced in 2.10. Number scatter/gather element groups per + work request. Used to deal with fragmentations which can consume + double the number of work requests. + + + + + +