1 <?xml version='1.0' encoding='utf-8'?>
2 <chapter xmlns="http://docbook.org/ns/docbook"
3 xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"
5 <title xml:id="lustretuning.title">Tuning a Lustre File System</title>
6 <para>This chapter contains information about tuning a Lustre file system for
7 better performance.</para>
9 <para>Many options in the Lustre software are set by means of kernel module
10 parameters. These parameters are contained in the
11 <literal>/etc/modprobe.d/lustre.conf</literal> file.</para>
13 <section xml:id="dbdoclet.50438272_55226">
16 <primary>tuning</primary>
19 <primary>tuning</primary>
20 <secondary>service threads</secondary>
21 </indexterm>Optimizing the Number of Service Threads</title>
22 <para>An OSS can have a minimum of two service threads and a maximum of 512
23 service threads. The number of service threads is a function of how much
24 RAM and how many CPUs are on each OSS node (1 thread / 128MB * num_cpus).
25 If the load on the OSS node is high, new service threads will be started in
26 order to process more requests concurrently, up to 4x the initial number of
27 threads (subject to the maximum of 512). For a 2GB 2-CPU system, the
28 default thread count is 32 and the maximum thread count is 128.</para>
29 <para>Increasing the size of the thread pool may help when:</para>
32 <para>Several OSTs are exported from a single OSS</para>
35 <para>Back-end storage is running synchronously</para>
38 <para>I/O completions take excessive time due to slow storage</para>
41 <para>Decreasing the size of the thread pool may help if:</para>
44 <para>Clients are overwhelming the storage capacity</para>
47 <para>There are lots of "slow I/O" or similar messages</para>
50 <para>Increasing the number of I/O threads allows the kernel and storage to
51 aggregate many writes together for more efficient disk I/O. The OSS thread
52 pool is shared--each thread allocates approximately 1.5 MB (maximum RPC
53 size + 0.5 MB) for internal I/O buffers.</para>
54 <para>It is very important to consider memory consumption when increasing
55 the thread pool size. Drives are only able to sustain a certain amount of
56 parallel I/O activity before performance is degraded, due to the high
57 number of seeks and the OST threads just waiting for I/O. In this
58 situation, it may be advisable to decrease the load by decreasing the
59 number of OST threads.</para>
60 <para>Determining the optimum number of OSS threads is a process of trial
61 and error, and varies for each particular configuration. Variables include
62 the number of OSTs on each OSS, number and speed of disks, RAID
63 configuration, and available RAM. You may want to start with a number of
64 OST threads equal to the number of actual disk spindles on the node. If you
65 use RAID, subtract any dead spindles not used for actual data (e.g., 1 of N
66 of spindles for RAID5, 2 of N spindles for RAID6), and monitor the
67 performance of clients during usual workloads. If performance is degraded,
68 increase the thread count and see how that works until performance is
69 degraded again or you reach satisfactory performance.</para>
71 <para>If there are too many threads, the latency for individual I/O
72 requests can become very high and should be avoided. Set the desired
73 maximum thread count permanently using the method described above.</para>
78 <primary>tuning</primary>
79 <secondary>OSS threads</secondary>
80 </indexterm>Specifying the OSS Service Thread Count</title>
82 <literal>oss_num_threads</literal> parameter enables the number of OST
83 service threads to be specified at module load time on the OSS
86 options ost oss_num_threads={N}
88 <para>After startup, the minimum and maximum number of OSS thread counts
90 <literal>{service}.thread_{min,max,started}</literal> tunable. To change
91 the tunable at runtime, run:</para>
94 lctl {get,set}_param {service}.thread_{min,max,started}
98 This works in a similar fashion to
99 binding of threads on MDS. MDS thread tuning is covered in
100 <xref linkend="dbdoclet.mdsbinding" />.</para>
104 <literal>oss_cpts=[EXPRESSION]</literal> binds the default OSS service
106 <literal>[EXPRESSION]</literal>.</para>
110 <literal>oss_io_cpts=[EXPRESSION]</literal> binds the IO OSS service
112 <literal>[EXPRESSION]</literal>.</para>
115 <para>For further details, see
116 <xref linkend="dbdoclet.50438271_87260" />.</para>
118 <section xml:id="dbdoclet.mdstuning">
121 <primary>tuning</primary>
122 <secondary>MDS threads</secondary>
123 </indexterm>Specifying the MDS Service Thread Count</title>
125 <literal>mds_num_threads</literal> parameter enables the number of MDS
126 service threads to be specified at module load time on the MDS
128 <screen>options mds mds_num_threads={N}</screen>
129 <para>After startup, the minimum and maximum number of MDS thread counts
131 <literal>{service}.thread_{min,max,started}</literal> tunable. To change
132 the tunable at runtime, run:</para>
135 lctl {get,set}_param {service}.thread_{min,max,started}
138 <para>For details, see
139 <xref linkend="dbdoclet.50438271_87260" />.</para>
140 <para>The number of MDS service threads started depends on system size
141 and the load on the server, and has a default maximum of 64. The
142 maximum potential number of threads (<literal>MDS_MAX_THREADS</literal>)
145 <para>The OSS and MDS start two threads per service per CPT at mount
146 time, and dynamically increase the number of running service threads in
147 response to server load. Setting the <literal>*_num_threads</literal>
148 module parameter starts the specified number of threads for that
149 service immediately and disables automatic thread creation behavior.
152 <para condition='l23'>Lustre software release 2.3 introduced new
153 parameters to provide more control to administrators.</para>
157 <literal>mds_rdpg_num_threads</literal> controls the number of threads
158 in providing the read page service. The read page service handles
159 file close and readdir operations.</para>
163 <literal>mds_attr_num_threads</literal> controls the number of threads
164 in providing the setattr service to clients running Lustre software
170 <section xml:id="dbdoclet.mdsbinding" condition='l23'>
173 <primary>tuning</primary>
174 <secondary>MDS binding</secondary>
175 </indexterm>Binding MDS Service Thread to CPU Partitions</title>
176 <para>With the introduction of Node Affinity (
177 <xref linkend="nodeaffdef" />) in Lustre software release 2.3, MDS threads
178 can be bound to particular CPU partitions (CPTs) to improve CPU cache
179 usage and memory locality. Default values for CPT counts and CPU core
180 bindings are selected automatically to provide good overall performance for
181 a given CPU count. However, an administrator can deviate from these setting
182 if they choose. For details on specifying the mapping of CPU cores to
183 CPTs see <xref linkend="dbdoclet.libcfstuning"/>.
188 <literal>mds_num_cpts=[EXPRESSION]</literal> binds the default MDS
189 service threads to CPTs defined by
190 <literal>EXPRESSION</literal>. For example
191 <literal>mds_num_cpts=[0-3]</literal> will bind the MDS service threads
193 <literal>CPT[0,1,2,3]</literal>.</para>
197 <literal>mds_rdpg_num_cpts=[EXPRESSION]</literal> binds the read page
198 service threads to CPTs defined by
199 <literal>EXPRESSION</literal>. The read page service handles file close
200 and readdir requests. For example
201 <literal>mds_rdpg_num_cpts=[4]</literal> will bind the read page threads
203 <literal>CPT4</literal>.</para>
207 <literal>mds_attr_num_cpts=[EXPRESSION]</literal> binds the setattr
208 service threads to CPTs defined by
209 <literal>EXPRESSION</literal>.</para>
212 <para>Parameters must be set before module load in the file
213 <literal>/etc/modprobe.d/lustre.conf</literal>. For example:
214 <example><title>lustre.conf</title>
215 <screen>options lnet networks=tcp0(eth0)
216 options mdt mds_num_cpts=[0]</screen>
220 <section xml:id="dbdoclet.50438272_73839">
223 <primary>LNet</primary>
224 <secondary>tuning</secondary>
227 <primary>tuning</primary>
228 <secondary>LNet</secondary>
229 </indexterm>Tuning LNet Parameters</title>
230 <para>This section describes LNet tunables, the use of which may be
231 necessary on some systems to improve performance. To test the performance
232 of your Lustre network, see
233 <xref linkend='lnetselftest' />.</para>
235 <title>Transmit and Receive Buffer Size</title>
236 <para>The kernel allocates buffers for sending and receiving messages on
239 <literal>ksocklnd</literal> has separate parameters for the transmit and
240 receive buffers.</para>
242 options ksocklnd tx_buffer_size=0 rx_buffer_size=0
244 <para>If these parameters are left at the default value (0), the system
245 automatically tunes the transmit and receive buffer size. In almost every
246 case, this default produces the best performance. Do not attempt to tune
247 these parameters unless you are a network expert.</para>
250 <title>Hardware Interrupts (
251 <literal>enable_irq_affinity</literal>)</title>
252 <para>The hardware interrupts that are generated by network adapters may
253 be handled by any CPU in the system. In some cases, we would like network
254 traffic to remain local to a single CPU to help keep the processor cache
255 warm and minimize the impact of context switches. This is helpful when an
256 SMP system has more than one network interface and ideal when the number
257 of interfaces equals the number of CPUs. To enable the
258 <literal>enable_irq_affinity</literal> parameter, enter:</para>
260 options ksocklnd enable_irq_affinity=1
262 <para>In other cases, if you have an SMP platform with a single fast
263 interface such as 10 Gb Ethernet and more than two CPUs, you may see
264 performance improve by turning this parameter off.</para>
266 options ksocklnd enable_irq_affinity=0
268 <para>By default, this parameter is off. As always, you should test the
269 performance to compare the impact of changing this parameter.</para>
271 <section condition='l23'>
274 <primary>tuning</primary>
275 <secondary>Network interface binding</secondary>
276 </indexterm>Binding Network Interface Against CPU Partitions</title>
277 <para>Lustre software release 2.3 and beyond provide enhanced network
278 interface control. The enhancement means that an administrator can bind
279 an interface to one or more CPU partitions. Bindings are specified as
280 options to the LNet modules. For more information on specifying module
282 <xref linkend="dbdoclet.50438293_15350" /></para>
284 <literal>o2ib0(ib0)[0,1]</literal> will ensure that all messages for
285 <literal>o2ib0</literal> will be handled by LND threads executing on
286 <literal>CPT0</literal> and
287 <literal>CPT1</literal>. An additional example might be:
288 <literal>tcp1(eth0)[0]</literal>. Messages for
289 <literal>tcp1</literal> are handled by threads on
290 <literal>CPT0</literal>.</para>
295 <primary>tuning</primary>
296 <secondary>Network interface credits</secondary>
297 </indexterm>Network Interface Credits</title>
298 <para>Network interface (NI) credits are shared across all CPU partitions
299 (CPT). For example, if a machine has four CPTs and the number of NI
300 credits is 512, then each partition has 128 credits. If a large number of
301 CPTs exist on the system, LNet checks and validates the NI credits for
302 each CPT to ensure each CPT has a workable number of credits. For
303 example, if a machine has 16 CPTs and the number of NI credits is 256,
304 then each partition only has 16 credits. 16 NI credits is low and could
305 negatively impact performance. As a result, LNet automatically adjusts
307 <literal>peer_credits</literal>(
308 <literal>peer_credits</literal> is 8 by default), so each partition has 64
310 <para>Increasing the number of
311 <literal>credits</literal>/
312 <literal>peer_credits</literal> can improve the performance of high
313 latency networks (at the cost of consuming more memory) by enabling LNet
314 to send more inflight messages to a specific network/peer and keep the
315 pipeline saturated.</para>
316 <para>An administrator can modify the NI credit count using
317 <literal>ksoclnd</literal> or
318 <literal>ko2iblnd</literal>. In the example below, 256 credits are
319 applied to TCP connections.</para>
323 <para>Applying 256 credits to IB connections can be achieved with:</para>
327 <note condition="l23">
328 <para>In Lustre software release 2.3 and beyond, LNet may revalidate
329 the NI credits, so the administrator's request may not persist.</para>
335 <primary>tuning</primary>
336 <secondary>router buffers</secondary>
337 </indexterm>Router Buffers</title>
338 <para>When a node is set up as an LNet router, three pools of buffers are
339 allocated: tiny, small and large. These pools are allocated per CPU
340 partition and are used to buffer messages that arrive at the router to be
341 forwarded to the next hop. The three different buffer sizes accommodate
342 different size messages.</para>
343 <para>If a message arrives that can fit in a tiny buffer then a tiny
344 buffer is used, if a message doesn’t fit in a tiny buffer, but fits in a
345 small buffer, then a small buffer is used. Finally if a message does not
346 fit in either a tiny buffer or a small buffer, a large buffer is
348 <para>Router buffers are shared by all CPU partitions. For a machine with
349 a large number of CPTs, the router buffer number may need to be specified
350 manually for best performance. A low number of router buffers risks
351 starving the CPU partitions of resources.</para>
355 <literal>tiny_router_buffers</literal>: Zero payload buffers used for
356 signals and acknowledgements.</para>
360 <literal>small_router_buffers</literal>: 4 KB payload buffers for
361 small messages</para>
365 <literal>large_router_buffers</literal>: 1 MB maximum payload
366 buffers, corresponding to the recommended RPC size of 1 MB.</para>
369 <para>The default setting for router buffers typically results in
370 acceptable performance. LNet automatically sets a default value to reduce
371 the likelihood of resource starvation. The size of a router buffer can be
372 modified as shown in the example below. In this example, the size of the
373 large buffer is modified using the
374 <literal>large_router_buffers</literal> parameter.</para>
376 lnet large_router_buffers=8192
378 <note condition="l23">
379 <para>In Lustre software release 2.3 and beyond, LNet may revalidate
380 the router buffer setting, so the administrator's request may not
387 <primary>tuning</primary>
388 <secondary>portal round-robin</secondary>
389 </indexterm>Portal Round-Robin</title>
390 <para>Portal round-robin defines the policy LNet applies to deliver
391 events and messages to the upper layers. The upper layers are PLRPC
392 service or LNet selftest.</para>
393 <para>If portal round-robin is disabled, LNet will deliver messages to
394 CPTs based on a hash of the source NID. Hence, all messages from a
395 specific peer will be handled by the same CPT. This can reduce data
396 traffic between CPUs. However, for some workloads, this behavior may
397 result in poorly balancing loads across the CPU.</para>
398 <para>If portal round-robin is enabled, LNet will round-robin incoming
399 events across all CPTs. This may balance load better across the CPU but
400 can incur a cross CPU overhead.</para>
401 <para>The current policy can be changed by an administrator with
403 <replaceable>value</replaceable>>
404 /proc/sys/lnet/portal_rotor</literal>. There are four options for
406 <replaceable>value</replaceable>
411 <literal>OFF</literal>
413 <para>Disable portal round-robin on all incoming requests.</para>
417 <literal>ON</literal>
419 <para>Enable portal round-robin on all incoming requests.</para>
423 <literal>RR_RT</literal>
425 <para>Enable portal round-robin only for routed messages.</para>
429 <literal>HASH_RT</literal>
431 <para>Routed messages will be delivered to the upper layer by hash of
432 source NID (instead of NID of router.) This is the default
438 <title>LNet Peer Health</title>
439 <para>Two options are available to help determine peer health:
443 <literal>peer_timeout</literal>- The timeout (in seconds) before an
444 aliveness query is sent to a peer. For example, if
445 <literal>peer_timeout</literal> is set to
446 <literal>180sec</literal>, an aliveness query is sent to the peer
447 every 180 seconds. This feature only takes effect if the node is
448 configured as an LNet router.</para>
449 <para>In a routed environment, the
450 <literal>peer_timeout</literal> feature should always be on (set to a
451 value in seconds) on routers. If the router checker has been enabled,
452 the feature should be turned off by setting it to 0 on clients and
454 <para>For a non-routed scenario, enabling the
455 <literal>peer_timeout</literal> option provides health information
456 such as whether a peer is alive or not. For example, a client is able
457 to determine if an MGS or OST is up when it sends it a message. If a
458 response is received, the peer is alive; otherwise a timeout occurs
459 when the request is made.</para>
461 <literal>peer_timeout</literal> should be set to no less than the LND
462 timeout setting. For more information about LND timeouts, see
463 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
464 linkend="section_c24_nt5_dl" />.</para>
466 <literal>o2iblnd</literal>(IB) driver is used,
467 <literal>peer_timeout</literal> should be at least twice the value of
469 <literal>ko2iblnd</literal> keepalive option. for more information
470 about keepalive options, see
471 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
472 linkend="section_ngq_qhy_zl" />.</para>
476 <literal>avoid_asym_router_failure</literal>– When set to 1, the
477 router checker running on the client or a server periodically pings
478 all the routers corresponding to the NIDs identified in the routes
479 parameter setting on the node to determine the status of each router
480 interface. The default setting is 1. (For more information about the
481 LNet routes parameter, see
482 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
483 linkend="dbdoclet.50438216_71227" /></para>
484 <para>A router is considered down if any of its NIDs are down. For
485 example, router X has three NIDs:
486 <literal>Xnid1</literal>,
487 <literal>Xnid2</literal>, and
488 <literal>Xnid3</literal>. A client is connected to the router via
489 <literal>Xnid1</literal>. The client has router checker enabled. The
490 router checker periodically sends a ping to the router via
491 <literal>Xnid1</literal>. The router responds to the ping with the
492 status of each of its NIDs. In this case, it responds with
493 <literal>Xnid1=up</literal>,
494 <literal>Xnid2=up</literal>,
495 <literal>Xnid3=down</literal>. If
496 <literal>avoid_asym_router_failure==1</literal>, the router is
497 considered down if any of its NIDs are down, so router X is
498 considered down and will not be used for routing messages. If
499 <literal>avoid_asym_router_failure==0</literal>, router X will
500 continue to be used for routing messages.</para>
502 </itemizedlist></para>
503 <para>The following router checker parameters must be set to the maximum
504 value of the corresponding setting for this option on any client or
509 <literal>dead_router_check_interval</literal>
514 <literal>live_router_check_interval</literal>
519 <literal>router_ping_timeout</literal>
522 </itemizedlist></para>
523 <para>For example, the
524 <literal>dead_router_check_interval</literal> parameter on any router must
528 <section xml:id="dbdoclet.libcfstuning" condition='l23'>
531 <primary>tuning</primary>
532 <secondary>libcfs</secondary>
533 </indexterm>libcfs Tuning</title>
534 <para>Lustre software release 2.3 introduced binding service threads via
535 CPU Partition Tables (CPTs). This allows the system administrator to
536 fine-tune on which CPU cores the Lustre service threads are run, for both
537 OSS and MDS services, as well as on the client.
539 <para>CPTs are useful to reserve some cores on the OSS or MDS nodes for
540 system functions such as system monitoring, HA heartbeat, or similar
541 tasks. On the client it may be useful to restrict Lustre RPC service
542 threads to a small subset of cores so that they do not interfere with
543 computation, or because these cores are directly attached to the network
546 <para>By default, the Lustre software will automatically generate CPU
547 partitions (CPT) based on the number of CPUs in the system.
548 The CPT count can be explicitly set on the libcfs module using
549 <literal>cpu_npartitions=<replaceable>NUMBER</replaceable></literal>.
550 The value of <literal>cpu_npartitions</literal> must be an integer between
551 1 and the number of online CPUs.
553 <para condition='l29'>In Lustre 2.9 and later the default is to use
554 one CPT per NUMA node. In earlier versions of Lustre, by default there
555 was a single CPT if the online CPU core count was four or fewer, and
556 additional CPTs would be created depending on the number of CPU cores,
557 typically with 4-8 cores per CPT.
560 <para>Setting <literal>cpu_npartitions=1</literal> will disable most
561 of the SMP Node Affinity functionality.</para>
564 <title>CPU Partition String Patterns</title>
565 <para>CPU partitions can be described using string pattern notation.
566 If <literal>cpu_pattern=N</literal> is used, then there will be one
567 CPT for each NUMA node in the system, with each CPT mapping all of
568 the CPU cores for that NUMA node.
570 <para>It is also possible to explicitly specify the mapping between
571 CPU cores and CPTs, for example:</para>
575 <literal>cpu_pattern="0[2,4,6] 1[3,5,7]</literal>
577 <para>Create two CPTs, CPT0 contains cores 2, 4, and 6, while CPT1
578 contains cores 3, 5, 7. CPU cores 0 and 1 will not be used by Lustre
579 service threads, and could be used for node services such as
580 system monitoring, HA heartbeat threads, etc. The binding of
581 non-Lustre services to those CPU cores may be done in userspace
582 using <literal>numactl(8)</literal> or other application-specific
583 methods, but is beyond the scope of this document.</para>
587 <literal>cpu_pattern="N 0[0-3] 1[4-7]</literal>
589 <para>Create two CPTs, with CPT0 containing all CPUs in NUMA
590 node[0-3], while CPT1 contains all CPUs in NUMA node [4-7].</para>
593 <para>The current configuration of the CPU partition can be read via
594 <literal>lctl get_parm cpu_partition_table</literal>. For example,
595 a simple 4-core system has a single CPT with all four CPU cores:
596 <screen>$ lctl get_param cpu_partition_table
597 cpu_partition_table=0 : 0 1 2 3</screen>
598 while a larger NUMA system with four 12-core CPUs may have four CPTs:
599 <screen>$ lctl get_param cpu_partition_table
601 0 : 0 1 2 3 4 5 6 7 8 9 10 11
602 1 : 12 13 14 15 16 17 18 19 20 21 22 23
603 2 : 24 25 26 27 28 29 30 31 32 33 34 35
604 3 : 36 37 38 39 40 41 42 43 44 45 46 47
609 <section xml:id="dbdoclet.lndtuning">
612 <primary>tuning</primary>
613 <secondary>LND tuning</secondary>
614 </indexterm>LND Tuning</title>
615 <para>LND tuning allows the number of threads per CPU partition to be
616 specified. An administrator can set the threads for both
617 <literal>ko2iblnd</literal> and
618 <literal>ksocklnd</literal> using the
619 <literal>nscheds</literal> parameter. This adjusts the number of threads for
620 each partition, not the overall number of threads on the LND.</para>
622 <para>Lustre software release 2.3 has greatly decreased the default
623 number of threads for
624 <literal>ko2iblnd</literal> and
625 <literal>ksocklnd</literal> on high-core count machines. The current
626 default values are automatically set and are chosen to work well across a
627 number of typical scenarios.</para>
630 <title>ko2iblnd Tuning</title>
631 <para>The following table outlines the ko2iblnd module parameters to be used
633 <informaltable frame="all">
635 <colspec colname="c1" colwidth="50*" />
636 <colspec colname="c2" colwidth="50*" />
637 <colspec colname="c3" colwidth="50*" />
642 <emphasis role="bold">Module Parameter</emphasis>
647 <emphasis role="bold">Default Value</emphasis>
652 <emphasis role="bold">Description</emphasis>
661 <literal>service</literal>
666 <literal>987</literal>
670 <para>Service number (within RDMA_PS_TCP).</para>
676 <literal>cksum</literal>
685 <para>Set non-zero to enable message (not RDMA) checksums.</para>
691 <literal>timeout</literal>
696 <literal>50</literal>
700 <para>Timeout in seconds.</para>
706 <literal>nscheds</literal>
715 <para>Number of threads in each scheduler pool (per CPT). Value of
716 zero means we derive the number from the number of cores.</para>
722 <literal>conns_per_peer</literal>
727 <literal>4 (OmniPath), 1 (Everything else)</literal>
731 <para>Introduced in 2.10. Number of connections to each peer. Messages
732 are sent round-robin over the connection pool. Provides signifiant
733 improvement with OmniPath.</para>
739 <literal>ntx</literal>
744 <literal>512</literal>
748 <para>Number of message descriptors allocated for each pool at
749 startup. Grows at runtime. Shared by all CPTs.</para>
755 <literal>credits</literal>
760 <literal>256</literal>
764 <para>Number of concurrent sends on network.</para>
770 <literal>peer_credits</literal>
779 <para>Number of concurrent sends to 1 peer. Related/limited by IB
786 <literal>peer_credits_hiw</literal>
795 <para>When eagerly to return credits.</para>
801 <literal>peer_buffer_credits</literal>
810 <para>Number per-peer router buffer credits.</para>
816 <literal>peer_timeout</literal>
821 <literal>180</literal>
825 <para>Seconds without aliveness news to declare peer dead (less than
826 or equal to 0 to disable).</para>
832 <literal>ipif_name</literal>
837 <literal>ib0</literal>
841 <para>IPoIB interface name.</para>
847 <literal>retry_count</literal>
856 <para>Retransmissions when no ACK received.</para>
862 <literal>rnr_retry_count</literal>
871 <para>RNR retransmissions.</para>
877 <literal>keepalive</literal>
882 <literal>100</literal>
886 <para>Idle time in seconds before sending a keepalive.</para>
892 <literal>ib_mtu</literal>
901 <para>IB MTU 256/512/1024/2048/4096.</para>
907 <literal>concurrent_sends</literal>
916 <para>Send work-queue sizing. If zero, derived from
917 <literal>map_on_demand</literal> and <literal>peer_credits</literal>.
924 <literal>map_on_demand</literal>
929 <literal>0 (pre-4.8 Linux) 1 (4.8 Linux onward) 32 (OmniPath)</literal>
933 <para>Number of fragments reserved for connection. If zero, use
934 global memory region (found to be security issue). If non-zero, use
935 FMR or FastReg for memory registration. Value needs to agree between
936 both peers of connection.</para>
942 <literal>fmr_pool_size</literal>
947 <literal>512</literal>
951 <para>Size of fmr pool on each CPT (>= ntx / 4). Grows at runtime.
958 <literal>fmr_flush_trigger</literal>
963 <literal>384</literal>
967 <para>Number dirty FMRs that triggers pool flush.</para>
973 <literal>fmr_cache</literal>
982 <para>Non-zero to enable FMR caching.</para>
988 <literal>dev_failover</literal>
997 <para>HCA failover for bonding (0 OFF, 1 ON, other values reserved).
1004 <literal>require_privileged_port</literal>
1009 <literal>0</literal>
1013 <para>Require privileged port when accepting connection.</para>
1019 <literal>use_privileged_port</literal>
1024 <literal>1</literal>
1028 <para>Use privileged port when initiating connection.</para>
1034 <literal>wrq_sge</literal>
1039 <literal>2</literal>
1043 <para>Introduced in 2.10. Number scatter/gather element groups per
1044 work request. Used to deal with fragmentations which can consume
1045 double the number of work requests.</para>
1053 <section xml:id="dbdoclet.nrstuning" condition='l24'>
1056 <primary>tuning</primary>
1057 <secondary>Network Request Scheduler (NRS) Tuning</secondary>
1058 </indexterm>Network Request Scheduler (NRS) Tuning</title>
1059 <para>The Network Request Scheduler (NRS) allows the administrator to
1060 influence the order in which RPCs are handled at servers, on a per-PTLRPC
1061 service basis, by providing different policies that can be activated and
1062 tuned in order to influence the RPC ordering. The aim of this is to provide
1063 for better performance, and possibly discrete performance characteristics
1064 using future policies.</para>
1065 <para>The NRS policy state of a PTLRPC service can be read and set via the
1066 <literal>{service}.nrs_policies</literal> tunable. To read a PTLRPC
1067 service's NRS policy state, run:</para>
1069 lctl get_param {service}.nrs_policies
1071 <para>For example, to read the NRS policy state of the
1072 <literal>ost_io</literal> service, run:</para>
1074 $ lctl get_param ost.OSS.ost_io.nrs_policies
1075 ost.OSS.ost_io.nrs_policies=
1114 high_priority_requests:
1152 <para>NRS policy state is shown in either one or two sections, depending on
1153 the PTLRPC service being queried. The first section is named
1154 <literal>regular_requests</literal> and is available for all PTLRPC
1155 services, optionally followed by a second section which is named
1156 <literal>high_priority_requests</literal>. This is because some PTLRPC
1157 services are able to treat some types of RPCs as higher priority ones, such
1158 that they are handled by the server with higher priority compared to other,
1159 regular RPC traffic. For PTLRPC services that do not support high-priority
1160 RPCs, you will only see the
1161 <literal>regular_requests</literal> section.</para>
1162 <para>There is a separate instance of each NRS policy on each PTLRPC
1163 service for handling regular and high-priority RPCs (if the service
1164 supports high-priority RPCs). For each policy instance, the following
1165 fields are shown:</para>
1166 <informaltable frame="all">
1168 <colspec colname="c1" colwidth="50*" />
1169 <colspec colname="c2" colwidth="50*" />
1174 <emphasis role="bold">Field</emphasis>
1179 <emphasis role="bold">Description</emphasis>
1188 <literal>name</literal>
1192 <para>The name of the policy.</para>
1198 <literal>state</literal>
1202 <para>The state of the policy; this can be any of
1203 <literal>invalid, stopping, stopped, starting, started</literal>.
1204 A fully enabled policy is in the
1205 <literal>started</literal> state.</para>
1211 <literal>fallback</literal>
1215 <para>Whether the policy is acting as a fallback policy or not. A
1216 fallback policy is used to handle RPCs that other enabled
1217 policies fail to handle, or do not support the handling of. The
1219 <literal>no, yes</literal>. Currently, only the FIFO policy can
1220 act as a fallback policy.</para>
1226 <literal>queued</literal>
1230 <para>The number of RPCs that the policy has waiting to be
1237 <literal>active</literal>
1241 <para>The number of RPCs that the policy is currently
1248 <para>To enable an NRS policy on a PTLRPC service run:</para>
1250 lctl set_param {service}.nrs_policies=
1251 <replaceable>policy_name</replaceable>
1253 <para>This will enable the policy
1254 <replaceable>policy_name</replaceable>for both regular and high-priority
1255 RPCs (if the PLRPC service supports high-priority RPCs) on the given
1256 service. For example, to enable the CRR-N NRS policy for the ldlm_cbd
1257 service, run:</para>
1259 $ lctl set_param ldlm.services.ldlm_cbd.nrs_policies=crrn
1260 ldlm.services.ldlm_cbd.nrs_policies=crrn
1263 <para>For PTLRPC services that support high-priority RPCs, you can also
1265 <replaceable>reg|hp</replaceable>token, in order to enable an NRS policy
1266 for handling only regular or high-priority RPCs on a given PTLRPC service,
1269 lctl set_param {service}.nrs_policies="
1270 <replaceable>policy_name</replaceable>
1271 <replaceable>reg|hp</replaceable>"
1273 <para>For example, to enable the TRR policy for handling only regular, but
1274 not high-priority RPCs on the
1275 <literal>ost_io</literal> service, run:</para>
1277 $ lctl set_param ost.OSS.ost_io.nrs_policies="trr reg"
1278 ost.OSS.ost_io.nrs_policies="trr reg"
1282 <para>When enabling an NRS policy, the policy name must be given in
1283 lower-case characters, otherwise the operation will fail with an error
1289 <primary>tuning</primary>
1290 <secondary>Network Request Scheduler (NRS) Tuning</secondary>
1291 <tertiary>first in, first out (FIFO) policy</tertiary>
1292 </indexterm>First In, First Out (FIFO) policy</title>
1293 <para>The first in, first out (FIFO) policy handles RPCs in a service in
1294 the same order as they arrive from the LNet layer, so no special
1295 processing takes place to modify the RPC handling stream. FIFO is the
1296 default policy for all types of RPCs on all PTLRPC services, and is
1297 always enabled irrespective of the state of other policies, so that it
1298 can be used as a backup policy, in case a more elaborate policy that has
1299 been enabled fails to handle an RPC, or does not support handling a given
1301 <para>The FIFO policy has no tunables that adjust its behaviour.</para>
1306 <primary>tuning</primary>
1307 <secondary>Network Request Scheduler (NRS) Tuning</secondary>
1308 <tertiary>client round-robin over NIDs (CRR-N) policy</tertiary>
1309 </indexterm>Client Round-Robin over NIDs (CRR-N) policy</title>
1310 <para>The client round-robin over NIDs (CRR-N) policy performs batched
1311 round-robin scheduling of all types of RPCs, with each batch consisting
1312 of RPCs originating from the same client node, as identified by its NID.
1313 CRR-N aims to provide for better resource utilization across the cluster,
1314 and to help shorten completion times of jobs in some cases, by
1315 distributing available bandwidth more evenly across all clients.</para>
1316 <para>The CRR-N policy can be enabled on all types of PTLRPC services,
1317 and has the following tunable that can be used to adjust its
1322 <literal>{service}.nrs_crrn_quantum</literal>
1325 <literal>{service}.nrs_crrn_quantum</literal> tunable determines the
1326 maximum allowed size of each batch of RPCs; the unit of measure is in
1327 number of RPCs. To read the maximum allowed batch size of a CRR-N
1330 lctl get_param {service}.nrs_crrn_quantum
1332 <para>For example, to read the maximum allowed batch size of a CRR-N
1333 policy on the ost_io service, run:</para>
1335 $ lctl get_param ost.OSS.ost_io.nrs_crrn_quantum
1336 ost.OSS.ost_io.nrs_crrn_quantum=reg_quantum:16
1340 <para>You can see that there is a separate maximum allowed batch size
1342 <literal>reg_quantum</literal>) and high-priority (
1343 <literal>hp_quantum</literal>) RPCs (if the PTLRPC service supports
1344 high-priority RPCs).</para>
1345 <para>To set the maximum allowed batch size of a CRR-N policy on a
1346 given service, run:</para>
1348 lctl set_param {service}.nrs_crrn_quantum=
1349 <replaceable>1-65535</replaceable>
1351 <para>This will set the maximum allowed batch size on a given
1352 service, for both regular and high-priority RPCs (if the PLRPC
1353 service supports high-priority RPCs), to the indicated value.</para>
1354 <para>For example, to set the maximum allowed batch size on the
1355 ldlm_canceld service to 16 RPCs, run:</para>
1357 $ lctl set_param ldlm.services.ldlm_canceld.nrs_crrn_quantum=16
1358 ldlm.services.ldlm_canceld.nrs_crrn_quantum=16
1361 <para>For PTLRPC services that support high-priority RPCs, you can
1362 also specify a different maximum allowed batch size for regular and
1363 high-priority RPCs, by running:</para>
1365 $ lctl set_param {service}.nrs_crrn_quantum=
1366 <replaceable>reg_quantum|hp_quantum</replaceable>:
1367 <replaceable>1-65535</replaceable>"
1369 <para>For example, to set the maximum allowed batch size on the
1370 ldlm_canceld service, for high-priority RPCs to 32, run:</para>
1372 $ lctl set_param ldlm.services.ldlm_canceld.nrs_crrn_quantum="hp_quantum:32"
1373 ldlm.services.ldlm_canceld.nrs_crrn_quantum=hp_quantum:32
1376 <para>By using the last method, you can also set the maximum regular
1377 and high-priority RPC batch sizes to different values, in a single
1378 command invocation.</para>
1385 <primary>tuning</primary>
1386 <secondary>Network Request Scheduler (NRS) Tuning</secondary>
1387 <tertiary>object-based round-robin (ORR) policy</tertiary>
1388 </indexterm>Object-based Round-Robin (ORR) policy</title>
1389 <para>The object-based round-robin (ORR) policy performs batched
1390 round-robin scheduling of bulk read write (brw) RPCs, with each batch
1391 consisting of RPCs that pertain to the same backend-file system object,
1392 as identified by its OST FID.</para>
1393 <para>The ORR policy is only available for use on the ost_io service. The
1394 RPC batches it forms can potentially consist of mixed bulk read and bulk
1395 write RPCs. The RPCs in each batch are ordered in an ascending manner,
1396 based on either the file offsets, or the physical disk offsets of each
1397 RPC (only applicable to bulk read RPCs).</para>
1398 <para>The aim of the ORR policy is to provide for increased bulk read
1399 throughput in some cases, by ordering bulk read RPCs (and potentially
1400 bulk write RPCs), and thus minimizing costly disk seek operations.
1401 Performance may also benefit from any resulting improvement in resource
1402 utilization, or by taking advantage of better locality of reference
1403 between RPCs.</para>
1404 <para>The ORR policy has the following tunables that can be used to
1405 adjust its behaviour:</para>
1409 <literal>ost.OSS.ost_io.nrs_orr_quantum</literal>
1412 <literal>ost.OSS.ost_io.nrs_orr_quantum</literal> tunable determines
1413 the maximum allowed size of each batch of RPCs; the unit of measure
1414 is in number of RPCs. To read the maximum allowed batch size of the
1415 ORR policy, run:</para>
1417 $ lctl get_param ost.OSS.ost_io.nrs_orr_quantum
1418 ost.OSS.ost_io.nrs_orr_quantum=reg_quantum:256
1422 <para>You can see that there is a separate maximum allowed batch size
1424 <literal>reg_quantum</literal>) and high-priority (
1425 <literal>hp_quantum</literal>) RPCs (if the PTLRPC service supports
1426 high-priority RPCs).</para>
1427 <para>To set the maximum allowed batch size for the ORR policy,
1430 $ lctl set_param ost.OSS.ost_io.nrs_orr_quantum=
1431 <replaceable>1-65535</replaceable>
1433 <para>This will set the maximum allowed batch size for both regular
1434 and high-priority RPCs, to the indicated value.</para>
1435 <para>You can also specify a different maximum allowed batch size for
1436 regular and high-priority RPCs, by running:</para>
1438 $ lctl set_param ost.OSS.ost_io.nrs_orr_quantum=
1439 <replaceable>reg_quantum|hp_quantum</replaceable>:
1440 <replaceable>1-65535</replaceable>
1442 <para>For example, to set the maximum allowed batch size for regular
1443 RPCs to 128, run:</para>
1445 $ lctl set_param ost.OSS.ost_io.nrs_orr_quantum=reg_quantum:128
1446 ost.OSS.ost_io.nrs_orr_quantum=reg_quantum:128
1449 <para>By using the last method, you can also set the maximum regular
1450 and high-priority RPC batch sizes to different values, in a single
1451 command invocation.</para>
1455 <literal>ost.OSS.ost_io.nrs_orr_offset_type</literal>
1458 <literal>ost.OSS.ost_io.nrs_orr_offset_type</literal> tunable
1459 determines whether the ORR policy orders RPCs within each batch based
1460 on logical file offsets or physical disk offsets. To read the offset
1461 type value for the ORR policy, run:</para>
1463 $ lctl get_param ost.OSS.ost_io.nrs_orr_offset_type
1464 ost.OSS.ost_io.nrs_orr_offset_type=reg_offset_type:physical
1465 hp_offset_type:logical
1468 <para>You can see that there is a separate offset type value for
1470 <literal>reg_offset_type</literal>) and high-priority (
1471 <literal>hp_offset_type</literal>) RPCs.</para>
1472 <para>To set the ordering type for the ORR policy, run:</para>
1474 $ lctl set_param ost.OSS.ost_io.nrs_orr_offset_type=
1475 <replaceable>physical|logical</replaceable>
1477 <para>This will set the offset type for both regular and
1478 high-priority RPCs, to the indicated value.</para>
1479 <para>You can also specify a different offset type for regular and
1480 high-priority RPCs, by running:</para>
1482 $ lctl set_param ost.OSS.ost_io.nrs_orr_offset_type=
1483 <replaceable>reg_offset_type|hp_offset_type</replaceable>:
1484 <replaceable>physical|logical</replaceable>
1486 <para>For example, to set the offset type for high-priority RPCs to
1487 physical disk offsets, run:</para>
1489 $ lctl set_param ost.OSS.ost_io.nrs_orr_offset_type=hp_offset_type:physical
1490 ost.OSS.ost_io.nrs_orr_offset_type=hp_offset_type:physical
1492 <para>By using the last method, you can also set offset type for
1493 regular and high-priority RPCs to different values, in a single
1494 command invocation.</para>
1496 <para>Irrespective of the value of this tunable, only logical
1497 offsets can, and are used for ordering bulk write RPCs.</para>
1502 <literal>ost.OSS.ost_io.nrs_orr_supported</literal>
1505 <literal>ost.OSS.ost_io.nrs_orr_supported</literal> tunable determines
1506 the type of RPCs that the ORR policy will handle. To read the types
1507 of supported RPCs by the ORR policy, run:</para>
1509 $ lctl get_param ost.OSS.ost_io.nrs_orr_supported
1510 ost.OSS.ost_io.nrs_orr_supported=reg_supported:reads
1511 hp_supported=reads_and_writes
1514 <para>You can see that there is a separate supported 'RPC types'
1516 <literal>reg_supported</literal>) and high-priority (
1517 <literal>hp_supported</literal>) RPCs.</para>
1518 <para>To set the supported RPC types for the ORR policy, run:</para>
1520 $ lctl set_param ost.OSS.ost_io.nrs_orr_supported=
1521 <replaceable>reads|writes|reads_and_writes</replaceable>
1523 <para>This will set the supported RPC types for both regular and
1524 high-priority RPCs, to the indicated value.</para>
1525 <para>You can also specify a different supported 'RPC types' value
1526 for regular and high-priority RPCs, by running:</para>
1528 $ lctl set_param ost.OSS.ost_io.nrs_orr_supported=
1529 <replaceable>reg_supported|hp_supported</replaceable>:
1530 <replaceable>reads|writes|reads_and_writes</replaceable>
1532 <para>For example, to set the supported RPC types to bulk read and
1533 bulk write RPCs for regular requests, run:</para>
1536 ost.OSS.ost_io.nrs_orr_supported=reg_supported:reads_and_writes
1537 ost.OSS.ost_io.nrs_orr_supported=reg_supported:reads_and_writes
1540 <para>By using the last method, you can also set the supported RPC
1541 types for regular and high-priority RPC to different values, in a
1542 single command invocation.</para>
1549 <primary>tuning</primary>
1550 <secondary>Network Request Scheduler (NRS) Tuning</secondary>
1551 <tertiary>Target-based round-robin (TRR) policy</tertiary>
1552 </indexterm>Target-based Round-Robin (TRR) policy</title>
1553 <para>The target-based round-robin (TRR) policy performs batched
1554 round-robin scheduling of brw RPCs, with each batch consisting of RPCs
1555 that pertain to the same OST, as identified by its OST index.</para>
1556 <para>The TRR policy is identical to the object-based round-robin (ORR)
1557 policy, apart from using the brw RPC's target OST index instead of the
1558 backend-fs object's OST FID, for determining the RPC scheduling order.
1559 The goals of TRR are effectively the same as for ORR, and it uses the
1560 following tunables to adjust its behaviour:</para>
1564 <literal>ost.OSS.ost_io.nrs_trr_quantum</literal>
1566 <para>The purpose of this tunable is exactly the same as for the
1567 <literal>ost.OSS.ost_io.nrs_orr_quantum</literal> tunable for the ORR
1568 policy, and you can use it in exactly the same way.</para>
1572 <literal>ost.OSS.ost_io.nrs_trr_offset_type</literal>
1574 <para>The purpose of this tunable is exactly the same as for the
1575 <literal>ost.OSS.ost_io.nrs_orr_offset_type</literal> tunable for the
1576 ORR policy, and you can use it in exactly the same way.</para>
1580 <literal>ost.OSS.ost_io.nrs_trr_supported</literal>
1582 <para>The purpose of this tunable is exactly the same as for the
1583 <literal>ost.OSS.ost_io.nrs_orr_supported</literal> tunable for the
1584 ORR policy, and you can use it in exactly the sme way.</para>
1588 <section xml:id="dbdoclet.tbftuning" condition='l26'>
1591 <primary>tuning</primary>
1592 <secondary>Network Request Scheduler (NRS) Tuning</secondary>
1593 <tertiary>Token Bucket Filter (TBF) policy</tertiary>
1594 </indexterm>Token Bucket Filter (TBF) policy</title>
1595 <para>The TBF (Token Bucket Filter) is a Lustre NRS policy which enables
1596 Lustre services to enforce the RPC rate limit on clients/jobs for QoS
1597 (Quality of Service) purposes.</para>
1599 <title>The internal structure of TBF policy</title>
1602 <imagedata scalefit="1" width="100%"
1603 fileref="figures/TBF_policy.svg" />
1606 <phrase>The internal structure of TBF policy</phrase>
1610 <para>When a RPC request arrives, TBF policy puts it to a waiting queue
1611 according to its classification. The classification of RPC requests is
1612 based on either NID or JobID of the RPC according to the configure of
1613 TBF. TBF policy maintains multiple queues in the system, one queue for
1614 each category in the classification of RPC requests. The requests waits
1615 for tokens in the FIFO queue before they have been handled so as to keep
1616 the RPC rates under the limits.</para>
1617 <para>When Lustre services are too busy to handle all of the requests in
1618 time, all of the specified rates of the queues will not be satisfied.
1619 Nothing bad will happen except some of the RPC rates are slower than
1620 configured. In this case, the queue with higher rate will have an
1621 advantage over the queues with lower rates, but none of them will be
1623 <para>To manage the RPC rate of queues, we don't need to set the rate of
1624 each queue manually. Instead, we define rules which TBF policy matches to
1625 determine RPC rate limits. All of the defined rules are organized as an
1626 ordered list. Whenever a queue is newly created, it goes though the rule
1627 list and takes the first matched rule as its rule, so that the queue
1628 knows its RPC token rate. A rule can be added to or removed from the list
1629 at run time. Whenever the list of rules is changed, the queues will
1630 update their matched rules.</para>
1634 <literal>ost.OSS.ost_io.nrs_tbf_rule</literal>
1636 <para>The format of the rule start command of TBF policy is as
1639 $ lctl set_param x.x.x.nrs_tbf_rule=
1640 "[reg|hp] start <replaceable>rule_name</replaceable> <replaceable>arguments</replaceable>..."
1643 <replaceable>rule_name</replaceable>' argument is a string which
1644 identifies a rule. The format of the '
1645 <replaceable>arguments</replaceable>' is changing according to the
1646 type of the TBF policy. For the NID based TBF policy, its format is
1649 $ lctl set_param x.x.x.nrs_tbf_rule=
1650 "[reg|hp] start <replaceable>rule_name</replaceable> {<replaceable>nidlist</replaceable>} <replaceable>rate</replaceable>"
1652 <para>The format of '
1653 <replaceable>nidlist</replaceable>' argument is the same as the
1654 format when configuring LNet route. The '
1655 <replaceable>rate</replaceable>' argument is the RPC rate of the
1656 rule, means the upper limit number of requests per second.</para>
1657 <para>Following commands are valid. Please note that a newly started
1658 rule is prior to old rules, so the order of starting rules is
1659 critical too.</para>
1661 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=
1662 "start other_clients {192.168.*.*@tcp} 50"
1665 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=
1666 "start loginnode {192.168.1.1@tcp} 100"
1668 <para>General rule can be replaced by two rules (reg and hp) as
1671 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=
1672 "reg start loginnode {192.168.1.1@tcp} 100"
1675 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=
1676 "hp start loginnode {192.168.1.1@tcp} 100"
1679 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=
1680 "start computes {192.168.1.[2-128]@tcp} 500"
1682 <para>The above rules will put an upper limit for servers to process
1683 at most 5x as many RPCs from compute nodes as login nodes.</para>
1684 <para>For the JobID (please see
1685 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
1686 linkend="dbdoclet.jobstats" /> for more details) based TBF
1687 policy, its format is as follows:</para>
1689 $ lctl set_param x.x.x.nrs_tbf_rule=
1690 "[reg|hp] start <replaceable>name</replaceable> {<replaceable>jobid_list</replaceable>} <replaceable>rate</replaceable>"
1692 <para>Following commands are valid:</para>
1694 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=
1695 "start user1 {iozone.500 dd.500} 100"
1698 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=
1699 "start iozone_user1 {iozone.500} 100"
1701 <para>Same as nid, could use reg and hp rules separately:</para>
1703 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=
1704 "hp start iozone_user1 {iozone.500} 100"
1707 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=
1708 "reg start iozone_user1 {iozone.500} 100"
1710 <para>The format of the rule change command of TBF policy is as
1713 $ lctl set_param x.x.x.nrs_tbf_rule=
1714 "[reg|hp] change <replaceable>rule_name</replaceable> <replaceable>rate</replaceable>"
1716 <para>Following commands are valid:</para>
1718 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule="change loginnode 200"
1721 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule="reg change loginnode 200"
1724 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule="hp change loginnode 200"
1726 <para>The format of the rule stop command of TBF policy is as
1729 $ lctl set_param x.x.x.nrs_tbf_rule="[reg|hp] stop
1730 <replaceable>rule_name</replaceable>"
1732 <para>Following commands are valid:</para>
1734 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule="stop loginnode"
1737 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule="reg stop loginnode"
1740 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule="hp stop loginnode"
1745 <section xml:id="dbdoclet.delaytuning" condition='l2A'>
1748 <primary>tuning</primary>
1749 <secondary>Network Request Scheduler (NRS) Tuning</secondary>
1750 <tertiary>Delay policy</tertiary>
1751 </indexterm>Delay policy</title>
1752 <para>The NRS Delay policy seeks to perturb the timing of request
1753 processing at the PtlRPC layer, with the goal of simulating high server
1754 load, and finding and exposing timing related problems. When this policy
1755 is active, upon arrival of a request the policy will calculate an offset,
1756 within a defined, user-configurable range, from the request arrival
1757 time, to determine a time after which the request should be handled.
1758 The request is then stored using the cfs_binheap implementation,
1759 which sorts the request according to the assigned start time.
1760 Requests are removed from the binheap for handling once their start
1761 time has been passed.</para>
1762 <para>The Delay policy can be enabled on all types of PtlRPC services,
1763 and has the following tunables that can be used to adjust its behavior:
1768 <literal>{service}.nrs_delay_min</literal>
1771 <literal>{service}.nrs_delay_min</literal> tunable controls the
1772 minimum amount of time, in seconds, that a request will be delayed by
1773 this policy. The default is 5 seconds. To read this value run:</para>
1775 lctl get_param {service}.nrs_delay_min</screen>
1776 <para>For example, to read the minimum delay set on the ost_io
1777 service, run:</para>
1779 $ lctl get_param ost.OSS.ost_io.nrs_delay_min
1780 ost.OSS.ost_io.nrs_delay_min=reg_delay_min:5
1781 hp_delay_min:5</screen>
1782 <para>To set the minimum delay in RPC processing, run:</para>
1784 lctl set_param {service}.nrs_delay_min=<replaceable>0-65535</replaceable></screen>
1785 <para>This will set the minimum delay time on a given service, for both
1786 regular and high-priority RPCs (if the PtlRPC service supports
1787 high-priority RPCs), to the indicated value.</para>
1788 <para>For example, to set the minimum delay time on the ost_io service
1791 $ lctl set_param ost.OSS.ost_io.nrs_delay_min=10
1792 ost.OSS.ost_io.nrs_delay_min=10</screen>
1793 <para>For PtlRPC services that support high-priority RPCs, to set a
1794 different minimum delay time for regular and high-priority RPCs, run:
1797 lctl set_param {service}.nrs_delay_min=<replaceable>reg_delay_min|hp_delay_min</replaceable>:<replaceable>0-65535</replaceable>
1799 <para>For example, to set the minimum delay time on the ost_io service
1800 for high-priority RPCs to 3, run:</para>
1802 $ lctl set_param ost.OSS.ost_io.nrs_delay_min=hp_delay_min:3
1803 ost.OSS.ost_io.nrs_delay_min=hp_delay_min:3</screen>
1804 <para>Note, in all cases the minimum delay time cannot exceed the
1805 maximum delay time.</para>
1809 <literal>{service}.nrs_delay_max</literal>
1812 <literal>{service}.nrs_delay_max</literal> tunable controls the
1813 maximum amount of time, in seconds, that a request will be delayed by
1814 this policy. The default is 300 seconds. To read this value run:
1816 <screen>lctl get_param {service}.nrs_delay_max</screen>
1817 <para>For example, to read the maximum delay set on the ost_io
1818 service, run:</para>
1820 $ lctl get_param ost.OSS.ost_io.nrs_delay_max
1821 ost.OSS.ost_io.nrs_delay_max=reg_delay_max:300
1822 hp_delay_max:300</screen>
1823 <para>To set the maximum delay in RPC processing, run:</para>
1824 <screen>lctl set_param {service}.nrs_delay_max=<replaceable>0-65535</replaceable>
1826 <para>This will set the maximum delay time on a given service, for both
1827 regular and high-priority RPCs (if the PtlRPC service supports
1828 high-priority RPCs), to the indicated value.</para>
1829 <para>For example, to set the maximum delay time on the ost_io service
1832 $ lctl set_param ost.OSS.ost_io.nrs_delay_max=60
1833 ost.OSS.ost_io.nrs_delay_max=60</screen>
1834 <para>For PtlRPC services that support high-priority RPCs, to set a
1835 different maximum delay time for regular and high-priority RPCs, run:
1837 <screen>lctl set_param {service}.nrs_delay_max=<replaceable>reg_delay_max|hp_delay_max</replaceable>:<replaceable>0-65535</replaceable></screen>
1838 <para>For example, to set the maximum delay time on the ost_io service
1839 for high-priority RPCs to 30, run:</para>
1841 $ lctl set_param ost.OSS.ost_io.nrs_delay_max=hp_delay_max:30
1842 ost.OSS.ost_io.nrs_delay_max=hp_delay_max:30</screen>
1843 <para>Note, in all cases the maximum delay time cannot be less than the
1844 minimum delay time.</para>
1848 <literal>{service}.nrs_delay_pct</literal>
1851 <literal>{service}.nrs_delay_pct</literal> tunable controls the
1852 percentage of requests that will be delayed by this policy. The
1853 default is 100. Note, when a request is not selected for handling by
1854 the delay policy due to this variable then the request will be handled
1855 by whatever fallback policy is defined for that service. If no other
1856 fallback policy is defined then the request will be handled by the
1857 FIFO policy. To read this value run:</para>
1858 <screen>lctl get_param {service}.nrs_delay_pct</screen>
1859 <para>For example, to read the percentage of requests being delayed on
1860 the ost_io service, run:</para>
1862 $ lctl get_param ost.OSS.ost_io.nrs_delay_pct
1863 ost.OSS.ost_io.nrs_delay_pct=reg_delay_pct:100
1864 hp_delay_pct:100</screen>
1865 <para>To set the percentage of delayed requests, run:</para>
1867 lctl set_param {service}.nrs_delay_pct=<replaceable>0-100</replaceable></screen>
1868 <para>This will set the percentage of requests delayed on a given
1869 service, for both regular and high-priority RPCs (if the PtlRPC service
1870 supports high-priority RPCs), to the indicated value.</para>
1871 <para>For example, to set the percentage of delayed requests on the
1872 ost_io service to 50, run:</para>
1874 $ lctl set_param ost.OSS.ost_io.nrs_delay_pct=50
1875 ost.OSS.ost_io.nrs_delay_pct=50
1877 <para>For PtlRPC services that support high-priority RPCs, to set a
1878 different delay percentage for regular and high-priority RPCs, run:
1880 <screen>lctl set_param {service}.nrs_delay_pct=<replaceable>reg_delay_pct|hp_delay_pct</replaceable>:<replaceable>0-100</replaceable>
1882 <para>For example, to set the percentage of delayed requests on the
1883 ost_io service for high-priority RPCs to 5, run:</para>
1884 <screen>$ lctl set_param ost.OSS.ost_io.nrs_delay_pct=hp_delay_pct:5
1885 ost.OSS.ost_io.nrs_delay_pct=hp_delay_pct:5
1891 <section xml:id="dbdoclet.50438272_25884">
1894 <primary>tuning</primary>
1895 <secondary>lockless I/O</secondary>
1896 </indexterm>Lockless I/O Tunables</title>
1897 <para>The lockless I/O tunable feature allows servers to ask clients to do
1898 lockless I/O (the server does the locking on behalf of clients) for
1899 contended files to avoid lock ping-pong.</para>
1900 <para>The lockless I/O patch introduces these tunables:</para>
1904 <emphasis role="bold">OST-side:</emphasis>
1907 ldlm.namespaces.filter-<replaceable>fsname</replaceable>-*.
1910 <literal>contended_locks</literal>- If the number of lock conflicts in
1911 the scan of granted and waiting queues at contended_locks is exceeded,
1912 the resource is considered to be contended.</para>
1914 <literal>contention_seconds</literal>- The resource keeps itself in a
1915 contended state as set in the parameter.</para>
1917 <literal>max_nolock_bytes</literal>- Server-side locking set only for
1918 requests less than the blocks set in the
1919 <literal>max_nolock_bytes</literal> parameter. If this tunable is
1920 set to zero (0), it disables server-side locking for read/write
1925 <emphasis role="bold">Client-side:</emphasis>
1928 /proc/fs/lustre/llite/lustre-*
1931 <literal>contention_seconds</literal>-
1932 <literal>llite</literal> inode remembers its contended state for the
1933 time specified in this parameter.</para>
1937 <emphasis role="bold">Client-side statistics:</emphasis>
1940 <literal>/proc/fs/lustre/llite/lustre-*/stats</literal> file has new
1941 rows for lockless I/O statistics.</para>
1943 <literal>lockless_read_bytes</literal> and
1944 <literal>lockless_write_bytes</literal>- To count the total bytes read
1945 or written, the client makes its own decisions based on the request
1946 size. The client does not communicate with the server if the request
1947 size is smaller than the
1948 <literal>min_nolock_size</literal>, without acquiring locks by the
1953 <section condition="l29">
1956 <primary>tuning</primary>
1957 <secondary>with lfs ladvise</secondary>
1959 Server-Side Advice and Hinting
1961 <section><title>Overview</title>
1962 <para>Use the <literal>lfs ladvise</literal> command give file access
1963 advices or hints to servers.</para>
1964 <screen>lfs ladvise [--advice|-a ADVICE ] [--background|-b]
1965 [--start|-s START[kMGT]]
1966 {[--end|-e END[kMGT]] | [--length|-l LENGTH[kMGT]]}
1967 <emphasis>file</emphasis> ...
1970 <informaltable frame="all">
1972 <colspec colname="c1" colwidth="50*"/>
1973 <colspec colname="c2" colwidth="50*"/>
1977 <para><emphasis role="bold">Option</emphasis></para>
1980 <para><emphasis role="bold">Description</emphasis></para>
1987 <para><literal>-a</literal>, <literal>--advice=</literal>
1988 <literal>ADVICE</literal></para>
1991 <para>Give advice or hint of type <literal>ADVICE</literal>.
1992 Advice types are:</para>
1993 <para><literal>willread</literal> to prefetch data into server
1995 <para><literal>dontneed</literal> to cleanup data cache on
2001 <para><literal>-b</literal>, <literal>--background</literal>
2005 <para>Enable the advices to be sent and handled asynchronously.
2011 <para><literal>-s</literal>, <literal>--start=</literal>
2012 <literal>START_OFFSET</literal></para>
2015 <para>File range starts from <literal>START_OFFSET</literal>
2021 <para><literal>-e</literal>, <literal>--end=</literal>
2022 <literal>END_OFFSET</literal></para>
2025 <para>File range ends at (not including)
2026 <literal>END_OFFSET</literal>. This option may not be
2027 specified at the same time as the <literal>-l</literal>
2033 <para><literal>-l</literal>, <literal>--length=</literal>
2034 <literal>LENGTH</literal></para>
2037 <para>File range has length of <literal>LENGTH</literal>.
2038 This option may not be specified at the same time as the
2039 <literal>-e</literal> option.</para>
2046 <para>Typically, <literal>lfs ladvise</literal> forwards the advice to
2047 Lustre servers without guaranteeing when and what servers will react to
2048 the advice. Actions may or may not triggered when the advices are
2049 recieved, depending on the type of the advice, as well as the real-time
2050 decision of the affected server-side components.</para>
2051 <para>A typical usage of ladvise is to enable applications and users with
2052 external knowledge to intervene in server-side cache management. For
2053 example, if a bunch of different clients are doing small random reads of a
2054 file, prefetching pages into OSS cache with big linear reads before the
2055 random IO is a net benefit. Fetching that data into each client cache with
2056 fadvise() may not be, due to much more data being sent to the client.
2058 <para>The main difference between the Linux <literal>fadvise()</literal>
2059 system call and <literal>lfs ladvise</literal> is that
2060 <literal>fadvise()</literal> is only a client side mechanism that does
2061 not pass the advice to the filesystem, while <literal>ladvise</literal>
2062 can send advices or hints to the Lustre server side.</para>
2064 <section><title>Examples</title>
2065 <para>The following example gives the OST(s) holding the first 1GB of
2066 <literal>/mnt/lustre/file1</literal>a hint that the first 1GB of the
2067 file will be read soon.</para>
2068 <screen>client1$ lfs ladvise -a willread -s 0 -e 1048576000 /mnt/lustre/file1
2070 <para>The following example gives the OST(s) holding the first 1GB of
2071 <literal>/mnt/lustre/file1</literal> a hint that the first 1GB of file
2072 will not be read in the near future, thus the OST(s) could clear the
2073 cache of the file in the memory.</para>
2074 <screen>client1$ lfs ladvise -a dontneed -s 0 -e 1048576000 /mnt/lustre/file1
2078 <section condition="l29">
2081 <primary>tuning</primary>
2082 <secondary>Large Bulk IO</secondary>
2084 Large Bulk IO (16MB RPC)
2086 <section><title>Overview</title>
2087 <para>Beginning with Lustre 2.9, Lustre is extended to support RPCs up
2088 to 16MB in size. By enabling a larger RPC size, fewer RPCs will be
2089 required to transfer the same amount of data between clients and
2090 servers. With a larger RPC size, the OSS can submit more data to the
2091 underlying disks at once, therefore it can produce larger disk I/Os
2092 to fully utilize the increasing bandwidth of disks.</para>
2093 <para>At client connection time, clients will negotiate with
2094 servers what the maximum RPC size it is possible to use, but the
2095 client can always send RPCs smaller than this maximum.</para>
2096 <para>The parameter <literal>brw_size</literal> is used on the OST
2097 to tell the client the maximum (preferred) IO size. All clients that
2098 talk to this target should never send an RPC greater than this size.
2099 Clients can individually set a smaller RPC size limit via the
2100 <literal>osc.*.max_pages_per_rpc</literal> tunable.
2103 <para>The smallest <literal>brw_size</literal> that can be set for
2104 ZFS OSTs is the <literal>recordsize</literal> of that dataset. This
2105 ensures that the client can always write a full ZFS file block if it
2106 has enough dirty data, and does not otherwise force it to do read-
2107 modify-write operations for every RPC.
2111 <section><title>Usage</title>
2112 <para>In order to enable a larger RPC size,
2113 <literal>brw_size</literal> must be changed to an IO size value up to
2114 16MB. To temporarily change <literal>brw_size</literal>, the
2115 following command should be run on the OSS:</para>
2116 <screen>oss# lctl set_param obdfilter.<replaceable>fsname</replaceable>-OST*.brw_size=16</screen>
2117 <para>To persistently change <literal>brw_size</literal>, one of the following
2118 commands should be run on the OSS:</para>
2119 <screen>oss# lctl set_param -P obdfilter.<replaceable>fsname</replaceable>-OST*.brw_size=16</screen>
2120 <screen>oss# lctl conf_param <replaceable>fsname</replaceable>-OST*.obdfilter.brw_size=16</screen>
2121 <para>When a client connects to an OST target, it will fetch
2122 <literal>brw_size</literal> from the target and pick the maximum value
2123 of <literal>brw_size</literal> and its local setting for
2124 <literal>max_pages_per_rpc</literal> as the actual RPC size.
2125 Therefore, the <literal>max_pages_per_rpc</literal> on the client side
2126 would have to be set to 16M, or 4096 if the PAGESIZE is 4KB, to enable
2127 a 16MB RPC. To temporarily make the change, the following command
2128 should be run on the client to set
2129 <literal>max_pages_per_rpc</literal>:</para>
2130 <screen>client$ lctl set_param osc.<replaceable>fsname</replaceable>-OST*.max_pages_per_rpc=16M</screen>
2131 <para>To persistently make this change, the following command should
2133 <screen>client$ lctl conf_param <replaceable>fsname</replaceable>-OST*.osc.max_pages_per_rpc=16M</screen>
2134 <caution><para>The <literal>brw_size</literal> of an OST can be
2135 changed on the fly. However, clients have to be remounted to
2136 renegotiate the new maximum RPC size.</para></caution>
2139 <section xml:id="dbdoclet.50438272_80545">
2142 <primary>tuning</primary>
2143 <secondary>for small files</secondary>
2144 </indexterm>Improving Lustre I/O Performance for Small Files</title>
2145 <para>An environment where an application writes small file chunks from
2146 many clients to a single file can result in poor I/O performance. To
2147 improve the performance of the Lustre file system with small files:</para>
2150 <para>Have the application aggregate writes some amount before
2151 submitting them to the Lustre file system. By default, the Lustre
2152 software enforces POSIX coherency semantics, so it results in lock
2153 ping-pong between client nodes if they are all writing to the same
2154 file at one time.</para>
2155 <para>Using MPI-IO Collective Write functionality in
2156 the Lustre ADIO driver is one way to achieve this in a straight
2157 forward manner if the application is already using MPI-IO.</para>
2160 <para>Have the application do 4kB
2161 <literal>O_DIRECT</literal> sized I/O to the file and disable locking
2162 on the output file. This avoids partial-page IO submissions and, by
2163 disabling locking, you avoid contention between clients.</para>
2166 <para>Have the application write contiguous data.</para>
2169 <para>Add more disks or use SSD disks for the OSTs. This dramatically
2170 improves the IOPS rate. Consider creating larger OSTs rather than many
2171 smaller OSTs due to less overhead (journal, connections, etc).</para>
2174 <para>Use RAID-1+0 OSTs instead of RAID-5/6. There is RAID parity
2175 overhead for writing small chunks of data to disk.</para>
2179 <section xml:id="dbdoclet.50438272_45406">
2182 <primary>tuning</primary>
2183 <secondary>write performance</secondary>
2184 </indexterm>Understanding Why Write Performance is Better Than Read
2186 <para>Typically, the performance of write operations on a Lustre cluster is
2187 better than read operations. When doing writes, all clients are sending
2188 write RPCs asynchronously. The RPCs are allocated, and written to disk in
2189 the order they arrive. In many cases, this allows the back-end storage to
2190 aggregate writes efficiently.</para>
2191 <para>In the case of read operations, the reads from clients may come in a
2192 different order and need a lot of seeking to get read from the disk. This
2193 noticeably hampers the read throughput.</para>
2194 <para>Currently, there is no readahead on the OSTs themselves, though the
2195 clients do readahead. If there are lots of clients doing reads it would not
2196 be possible to do any readahead in any case because of memory consumption
2197 (consider that even a single RPC (1 MB) readahead for 1000 clients would
2198 consume 1 GB of RAM).</para>
2199 <para>For file systems that use socklnd (TCP, Ethernet) as interconnect,
2200 there is also additional CPU overhead because the client cannot receive
2201 data without copying it from the network buffers. In the write case, the
2202 client CAN send data without the additional data copy. This means that the
2203 client is more likely to become CPU-bound during reads than writes.</para>