1 <?xml version='1.0' encoding='utf-8'?>
2 <chapter xmlns="http://docbook.org/ns/docbook"
3 xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"
5 <title xml:id="lustretuning.title">Tuning a Lustre File System</title>
6 <para>This chapter contains information about tuning a Lustre file system for
7 better performance.</para>
9 <para>Many options in the Lustre software are set by means of kernel module
10 parameters. These parameters are contained in the
11 <literal>/etc/modprobe.d/lustre.conf</literal> file.</para>
13 <section xml:id="tuning_optimizing_service_threads">
16 <primary>tuning</primary>
19 <primary>tuning</primary>
20 <secondary>service threads</secondary>
21 </indexterm>Optimizing the Number of Service Threads</title>
22 <para>An OSS can have a minimum of two service threads and a maximum of 512
23 service threads. The number of service threads is a function of how much
24 RAM and how many CPUs are on each OSS node (1 thread / 128MB * num_cpus).
25 If the load on the OSS node is high, new service threads will be started in
26 order to process more requests concurrently, up to 4x the initial number of
27 threads (subject to the maximum of 512). For a 2GB 2-CPU system, the
28 default thread count is 32 and the maximum thread count is 128.</para>
29 <para>Increasing the size of the thread pool may help when:</para>
32 <para>Several OSTs are exported from a single OSS</para>
35 <para>Back-end storage is running synchronously</para>
38 <para>I/O completions take excessive time due to slow storage</para>
41 <para>Decreasing the size of the thread pool may help if:</para>
44 <para>Clients are overwhelming the storage capacity</para>
47 <para>There are lots of "slow I/O" or similar messages</para>
50 <para>Increasing the number of I/O threads allows the kernel and storage to
51 aggregate many writes together for more efficient disk I/O. The OSS thread
52 pool is shared--each thread allocates approximately 1.5 MB (maximum RPC
53 size + 0.5 MB) for internal I/O buffers.</para>
54 <para>It is very important to consider memory consumption when increasing
55 the thread pool size. Drives are only able to sustain a certain amount of
56 parallel I/O activity before performance is degraded, due to the high
57 number of seeks and the OST threads just waiting for I/O. In this
58 situation, it may be advisable to decrease the load by decreasing the
59 number of OST threads.</para>
60 <para>Determining the optimum number of OSS threads is a process of trial
61 and error, and varies for each particular configuration. Variables include
62 the number of OSTs on each OSS, number and speed of disks, RAID
63 configuration, and available RAM. You may want to start with a number of
64 OST threads equal to the number of actual disk spindles on the node. If you
65 use RAID, subtract any dead spindles not used for actual data (e.g., 1 of N
66 of spindles for RAID5, 2 of N spindles for RAID6), and monitor the
67 performance of clients during usual workloads. If performance is degraded,
68 increase the thread count and see how that works until performance is
69 degraded again or you reach satisfactory performance.</para>
71 <para>If there are too many threads, the latency for individual I/O
72 requests can become very high and should be avoided. Set the desired
73 maximum thread count permanently using the method described above.</para>
78 <primary>tuning</primary>
79 <secondary>OSS threads</secondary>
80 </indexterm>Specifying the OSS Service Thread Count</title>
82 <literal>oss_num_threads</literal> parameter enables the number of OST
83 service threads to be specified at module load time on the OSS
86 options ost oss_num_threads={N}
88 <para>After startup, the minimum and maximum number of OSS thread counts
90 <literal>{service}.thread_{min,max,started}</literal> tunable. To change
91 the tunable at runtime, run:</para>
94 lctl {get,set}_param {service}.thread_{min,max,started}
98 This works in a similar fashion to
99 binding of threads on MDS. MDS thread tuning is covered in
100 <xref linkend="mdsbinding" />.</para>
104 <literal>oss_cpts=[EXPRESSION]</literal> binds the default OSS service
106 <literal>[EXPRESSION]</literal>.</para>
110 <literal>oss_io_cpts=[EXPRESSION]</literal> binds the IO OSS service
112 <literal>[EXPRESSION]</literal>.</para>
115 <para>For further details, see
116 <xref linkend="tuning_setting_thread_count" />.</para>
118 <section xml:id="mdstuning">
121 <primary>tuning</primary>
122 <secondary>MDS threads</secondary>
123 </indexterm>Specifying the MDS Service Thread Count</title>
125 <literal>mds_num_threads</literal> parameter enables the number of MDS
126 service threads to be specified at module load time on the MDS
128 <screen>options mds mds_num_threads={N}</screen>
129 <para>After startup, the minimum and maximum number of MDS thread counts
131 <literal>{service}.thread_{min,max,started}</literal> tunable. To change
132 the tunable at runtime, run:</para>
135 lctl {get,set}_param {service}.thread_{min,max,started}
138 <para>For details, see
139 <xref linkend="tuning_setting_thread_count" />.</para>
140 <para>The number of MDS service threads started depends on system size
141 and the load on the server, and has a default maximum of 64. The
142 maximum potential number of threads (<literal>MDS_MAX_THREADS</literal>)
145 <para>The OSS and MDS start two threads per service per CPT at mount
146 time, and dynamically increase the number of running service threads in
147 response to server load. Setting the <literal>*_num_threads</literal>
148 module parameter starts the specified number of threads for that
149 service immediately and disables automatic thread creation behavior.
152 <para>Parameters are available to provide administrators control
153 over the number of service threads.</para>
157 <literal>mds_rdpg_num_threads</literal> controls the number of threads
158 in providing the read page service. The read page service handles
159 file close and readdir operations.</para>
164 <section xml:id="mdsbinding">
167 <primary>tuning</primary>
168 <secondary>MDS binding</secondary>
169 </indexterm>Binding MDS Service Thread to CPU Partitions</title>
170 <para>With the Node Affinity (<xref linkend="nodeaffinity" />) feature,
171 MDS threads can be bound to particular CPU partitions (CPTs) to improve CPU
172 cache usage and memory locality. Default values for CPT counts and CPU core
173 bindings are selected automatically to provide good overall performance for
174 a given CPU count. However, an administrator can deviate from these setting
175 if they choose. For details on specifying the mapping of CPU cores to
176 CPTs see <xref linkend="libcfstuning"/>.
181 <literal>mds_num_cpts=[EXPRESSION]</literal> binds the default MDS
182 service threads to CPTs defined by
183 <literal>EXPRESSION</literal>. For example
184 <literal>mds_num_cpts=[0-3]</literal> will bind the MDS service threads
186 <literal>CPT[0,1,2,3]</literal>.</para>
190 <literal>mds_rdpg_num_cpts=[EXPRESSION]</literal> binds the read page
191 service threads to CPTs defined by
192 <literal>EXPRESSION</literal>. The read page service handles file close
193 and readdir requests. For example
194 <literal>mds_rdpg_num_cpts=[4]</literal> will bind the read page threads
196 <literal>CPT4</literal>.</para>
199 <para>Parameters must be set before module load in the file
200 <literal>/etc/modprobe.d/lustre.conf</literal>. For example:
201 <example><title>lustre.conf</title>
202 <screen>options lnet networks=tcp0(eth0)
203 options mdt mds_num_cpts=[0]</screen>
207 <section xml:id="tuning_lnet_params">
210 <primary>LNet</primary>
211 <secondary>tuning</secondary>
214 <primary>tuning</primary>
215 <secondary>LNet</secondary>
216 </indexterm>Tuning LNet Parameters</title>
217 <para>This section describes LNet tunables, the use of which may be
218 necessary on some systems to improve performance. To test the performance
219 of your Lustre network, see
220 <xref linkend='lnetselftest' />.</para>
222 <title>Transmit and Receive Buffer Size</title>
223 <para>The kernel allocates buffers for sending and receiving messages on
226 <literal>ksocklnd</literal> has separate parameters for the transmit and
227 receive buffers.</para>
229 options ksocklnd tx_buffer_size=0 rx_buffer_size=0
231 <para>If these parameters are left at the default value (0), the system
232 automatically tunes the transmit and receive buffer size. In almost every
233 case, this default produces the best performance. Do not attempt to tune
234 these parameters unless you are a network expert.</para>
237 <title>Hardware Interrupts (
238 <literal>enable_irq_affinity</literal>)</title>
239 <para>The hardware interrupts that are generated by network adapters may
240 be handled by any CPU in the system. In some cases, we would like network
241 traffic to remain local to a single CPU to help keep the processor cache
242 warm and minimize the impact of context switches. This is helpful when an
243 SMP system has more than one network interface and ideal when the number
244 of interfaces equals the number of CPUs. To enable the
245 <literal>enable_irq_affinity</literal> parameter, enter:</para>
247 options ksocklnd enable_irq_affinity=1
249 <para>In other cases, if you have an SMP platform with a single fast
250 interface such as 10 Gb Ethernet and more than two CPUs, you may see
251 performance improve by turning this parameter off.</para>
253 options ksocklnd enable_irq_affinity=0
255 <para>By default, this parameter is off. As always, you should test the
256 performance to compare the impact of changing this parameter.</para>
261 <primary>tuning</primary>
262 <secondary>Network interface binding</secondary>
263 </indexterm>Binding Network Interface Against CPU Partitions</title>
264 <para>Lustre allows enhanced network interface control. This means that
265 an administrator can bind an interface to one or more CPU partitions.
266 Bindings are specified as options to the LNet modules. For more
267 information on specifying module options, see
268 <xref linkend="tuning_lnet_mod_params" /></para>
270 <literal>o2ib0(ib0)[0,1]</literal> will ensure that all messages for
271 <literal>o2ib0</literal> will be handled by LND threads executing on
272 <literal>CPT0</literal> and
273 <literal>CPT1</literal>. An additional example might be:
274 <literal>tcp1(eth0)[0]</literal>. Messages for
275 <literal>tcp1</literal> are handled by threads on
276 <literal>CPT0</literal>.</para>
281 <primary>tuning</primary>
282 <secondary>Network interface credits</secondary>
283 </indexterm>Network Interface Credits</title>
284 <para>Network interface (NI) credits are shared across all CPU partitions
285 (CPT). For example, if a machine has four CPTs and the number of NI
286 credits is 512, then each partition has 128 credits. If a large number of
287 CPTs exist on the system, LNet checks and validates the NI credits for
288 each CPT to ensure each CPT has a workable number of credits. For
289 example, if a machine has 16 CPTs and the number of NI credits is 256,
290 then each partition only has 16 credits. 16 NI credits is low and could
291 negatively impact performance. As a result, LNet automatically adjusts
293 <literal>peer_credits</literal>(
294 <literal>peer_credits</literal> is 8 by default), so each partition has 64
296 <para>Increasing the number of
297 <literal>credits</literal>/
298 <literal>peer_credits</literal> can improve the performance of high
299 latency networks (at the cost of consuming more memory) by enabling LNet
300 to send more inflight messages to a specific network/peer and keep the
301 pipeline saturated.</para>
302 <para>An administrator can modify the NI credit count using
303 <literal>ksoclnd</literal> or
304 <literal>ko2iblnd</literal>. In the example below, 256 credits are
305 applied to TCP connections.</para>
309 <para>Applying 256 credits to IB connections can be achieved with:</para>
314 <para>LNet may revalidate the NI credits, so the administrator's
315 request may not persist.</para>
321 <primary>tuning</primary>
322 <secondary>router buffers</secondary>
323 </indexterm>Router Buffers</title>
324 <para>When a node is set up as an LNet router, three pools of buffers are
325 allocated: tiny, small and large. These pools are allocated per CPU
326 partition and are used to buffer messages that arrive at the router to be
327 forwarded to the next hop. The three different buffer sizes accommodate
328 different size messages.</para>
329 <para>If a message arrives that can fit in a tiny buffer then a tiny
330 buffer is used, if a message doesn’t fit in a tiny buffer, but fits in a
331 small buffer, then a small buffer is used. Finally if a message does not
332 fit in either a tiny buffer or a small buffer, a large buffer is
334 <para>Router buffers are shared by all CPU partitions. For a machine with
335 a large number of CPTs, the router buffer number may need to be specified
336 manually for best performance. A low number of router buffers risks
337 starving the CPU partitions of resources.</para>
341 <literal>tiny_router_buffers</literal>: Zero payload buffers used for
342 signals and acknowledgements.</para>
346 <literal>small_router_buffers</literal>: 4 KB payload buffers for
347 small messages</para>
351 <literal>large_router_buffers</literal>: 1 MB maximum payload
352 buffers, corresponding to the recommended RPC size of 1 MB.</para>
355 <para>The default setting for router buffers typically results in
356 acceptable performance. LNet automatically sets a default value to reduce
357 the likelihood of resource starvation. The size of a router buffer can be
358 modified as shown in the example below. In this example, the size of the
359 large buffer is modified using the
360 <literal>large_router_buffers</literal> parameter.</para>
362 lnet large_router_buffers=8192
365 <para>LNet may revalidate the router buffer setting, so the
366 administrator's request may not persist.</para>
372 <primary>tuning</primary>
373 <secondary>portal round-robin</secondary>
374 </indexterm>Portal Round-Robin</title>
375 <para>Portal round-robin defines the policy LNet applies to deliver
376 events and messages to the upper layers. The upper layers are PLRPC
377 service or LNet selftest.</para>
378 <para>If portal round-robin is disabled, LNet will deliver messages to
379 CPTs based on a hash of the source NID. Hence, all messages from a
380 specific peer will be handled by the same CPT. This can reduce data
381 traffic between CPUs. However, for some workloads, this behavior may
382 result in poorly balancing loads across the CPU.</para>
383 <para>If portal round-robin is enabled, LNet will round-robin incoming
384 events across all CPTs. This may balance load better across the CPU but
385 can incur a cross CPU overhead.</para>
386 <para>The current policy can be changed by an administrator with
387 <literal>lctl set_param portal_rotor=value</literal>.
388 There are four options for
390 <replaceable>value</replaceable>
395 <literal>OFF</literal>
397 <para>Disable portal round-robin on all incoming requests.</para>
401 <literal>ON</literal>
403 <para>Enable portal round-robin on all incoming requests.</para>
407 <literal>RR_RT</literal>
409 <para>Enable portal round-robin only for routed messages.</para>
413 <literal>HASH_RT</literal>
415 <para>Routed messages will be delivered to the upper layer by hash of
416 source NID (instead of NID of router.) This is the default
422 <title>LNet Peer Health</title>
423 <para>Two options are available to help determine peer health:
427 <literal>peer_timeout</literal>- The timeout (in seconds) before an
428 aliveness query is sent to a peer. For example, if
429 <literal>peer_timeout</literal> is set to
430 <literal>180sec</literal>, an aliveness query is sent to the peer
431 every 180 seconds. This feature only takes effect if the node is
432 configured as an LNet router.</para>
433 <para>In a routed environment, the
434 <literal>peer_timeout</literal> feature should always be on (set to a
435 value in seconds) on routers. If the router checker has been enabled,
436 the feature should be turned off by setting it to 0 on clients and
438 <para>For a non-routed scenario, enabling the
439 <literal>peer_timeout</literal> option provides health information
440 such as whether a peer is alive or not. For example, a client is able
441 to determine if an MGS or OST is up when it sends it a message. If a
442 response is received, the peer is alive; otherwise a timeout occurs
443 when the request is made.</para>
445 <literal>peer_timeout</literal> should be set to no less than the LND
446 timeout setting. For more information about LND timeouts, see
447 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
448 linkend="section_c24_nt5_dl" />.</para>
450 <literal>o2iblnd</literal>(IB) driver is used,
451 <literal>peer_timeout</literal> should be at least twice the value of
453 <literal>ko2iblnd</literal> keepalive option. for more information
454 about keepalive options, see
455 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
456 linkend="section_ngq_qhy_zl" />.</para>
460 <literal>avoid_asym_router_failure</literal>– When set to 1,
461 this parameter adds the additional requirement that for a route to be
462 considered up the gateway of the route must have at least one NI up on
463 the remote network of the route.
464 This new requirement applies only to routes that are single-hop,
465 which means that either the route's hop value is explicitly set to 1,
466 or that it can be inferred that the route is single-hop. The default
468 <para>The inference of a single-hop routes works as follows:
469 If the router checker is running on a node, the node will periodically
470 ping all of its gateways, which are routers on the same lnet that are
471 listed in the node's routes. The gateways' responses will include the
472 status of all their network interfaces (NIs).
473 If a node <literal>A</literal> has a route <literal>R</literal>
474 through gateway <literal>B</literal> to network <literal>C</literal>,
475 and node <literal>A</literal> sees that router <literal>B</literal>
476 has at least one NI that connects directly to network
477 <literal>C</literal> (in a ping response from <literal>B</literal>),
478 node <literal>A</literal> will infer that <literal>R</literal> is a
480 This aspect of single-hop is independent of the hop value of the
481 route. For more information about the LNet routes parameter, see
482 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
483 linkend="lnet_module_routes" /></para>
484 <para>It is recommended to specify <literal>hop=1</literal> when
485 creating a single-hop
486 route when this feature is enabled. When a route truly has only 1 hop,
487 it is still recommended to explicitly set hop=1 because the single-hop
488 inference mechanism will fail in cases where NIs fail to ever come up
489 at all. This is because for the route inference to work, the NIs must
490 at least come up or the gateway won't even mention them in the ping
491 response, so the node won't see any mention of the remote net of the
492 route in the gateway's NIs, and will mistakenly consider the route to
493 be multi-hop, and <literal>avoid_asym_router_failure</literal>
494 will then have no effect unless hop=1 is set explicitly when the route
496 <para>In the following examples, nodes running lnet are circles,
497 networks are squares,
498 and NIs are lines labeled by their NIDs.
499 There is a client <literal>C</literal> and a router
500 <literal>X</literal>. <literal>C</literal>
501 has routes to networks <literal>o2ib0</literal> and
502 <literal>o2ib1</literal> with <literal>X</literal> as the gateway.
504 with it's name crossed out, <literal>C</literal> considers it
505 to be down, otherwise <literal>C</literal>
506 considers it to be up.</para>
507 <figure xml:id="avoid_asym_router_failure.fig.one_o2ib0_down">
508 <title>One of Two Connections to o2ib0 Down</title>
511 <imagedata scalefit="1" width="45%"
512 fileref="figures/Tuning_one_o2ib0_down.png" />
515 <phrase>One of Two Connections to o2ib0 Down</phrase>
519 <para>In the above figure, one of two NIs that connect to
520 <literal>o2ib0</literal> is up,
521 so the route to <literal>o2ib0</literal> is considered up.</para>
522 <figure xml:id="avoid_asym_router_failure.fig.both_o2ib0_down">
523 <title>Both Connections to o2ib0 Down</title>
526 <imagedata scalefit="1" width="45%"
527 fileref="figures/Tuning_both_o2ib0_down.png" />
530 <phrase>Both Connections to o2ib0 Down</phrase>
534 <para>In the above figure, zero NIs that connect to
535 <literal>o2ib0</literal> are up,
536 so the route to <literal>o2ib0</literal> is down.</para>
537 <figure xml:id="avoid_asym_router_failure.fig.o2ib1_down">
538 <title>Connection to o2ib1 Down</title>
541 <imagedata scalefit="1" width="45%"
542 fileref="figures/Tuning_o2ib1_down.png" />
545 <phrase>Connection to o2ib1 Down</phrase>
549 <para>In the above figure, zero NIs that connect to
550 <literal>o2ib1</literal> are up,
551 so the route to <literal>o2ib1</literal> is down.</para>
552 <figure xml:id="avoid_asym_router_failure.fig.o2ib1_missing">
553 <title>Connection to o2ib1 Never Came Up</title>
556 <imagedata scalefit="1" width="65%"
557 fileref="figures/Tuning_o2ib1_missing.png" />
560 <phrase>Connection to o2ib1 Never Came Up </phrase>
564 <para>Compare Figures 34.3 and 34.4. In 34.4,
565 <literal>X4@o2ib1</literal> never came up
566 (rather than coming up and then going down). Consequently,
567 <literal>X</literal> did not list <literal>X4@o2ib1</literal> in its
568 ping response, so <literal>C</literal> cannot infer that
569 <literal>X</literal> should be directly connected to
570 <literal>o2ib1</literal>. If <literal>C</literal> has a route to
571 <literal>o2ib1</literal> through <literal>X</literal>, and the hop
572 count is not set to 1 by the sysadmin, LNet assumes that
573 <literal>X</literal> has a route to <literal>o2ib1</literal> through
574 some remote router node, such as <literal>Y</literal>. The gray part of
575 Figure 34.4 shows the sort of configuration that LNet incorrectly
576 assumes in this situation. Therefore, <literal>C</literal> will try
577 to send messages for <literal>o2ib1</literal> through
578 <literal>X</literal>, where they will be dropped.
579 If the sysadmin explicitly sets <literal>hop=1</literal> for the route
580 to <literal>o2ib1</literal> (on <literal>C</literal>), LNet will know
581 that if <literal>X</literal> does not report an NI on
582 <literal>o2ib1</literal>, that the route should be marked as down.
585 </itemizedlist></para>
586 <para>The following router checker parameters must be set to the maximum
587 value of the corresponding setting for this option on any client or
592 <literal>dead_router_check_interval</literal>
597 <literal>live_router_check_interval</literal>
602 <literal>router_ping_timeout</literal>
605 </itemizedlist></para>
606 <para>For example, the
607 <literal>dead_router_check_interval</literal> parameter on any router must
611 <section xml:id="libcfstuning">
614 <primary>tuning</primary>
615 <secondary>libcfs</secondary>
616 </indexterm>libcfs Tuning</title>
617 <para>Lustre allows binding service threads via CPU Partition Tables
618 (CPTs). This allows the system administrator to fine-tune on which CPU
619 cores the Lustre service threads are run, for both OSS and MDS services,
620 as well as on the client.
622 <para>CPTs are useful to reserve some cores on the OSS or MDS nodes for
623 system functions such as system monitoring, HA heartbeat, or similar
624 tasks. On the client it may be useful to restrict Lustre RPC service
625 threads to a small subset of cores so that they do not interfere with
626 computation, or because these cores are directly attached to the network
629 <para>By default, the Lustre software will automatically generate CPU
630 partitions (CPT) based on the number of CPUs in the system.
631 The CPT count can be explicitly set on the libcfs module using
632 <literal>cpu_npartitions=<replaceable>NUMBER</replaceable></literal>.
633 The value of <literal>cpu_npartitions</literal> must be an integer between
634 1 and the number of online CPUs.
636 <para condition='l29'>In Lustre 2.9 and later the default is to use
637 one CPT per NUMA node. In earlier versions of Lustre, by default there
638 was a single CPT if the online CPU core count was four or fewer, and
639 additional CPTs would be created depending on the number of CPU cores,
640 typically with 4-8 cores per CPT.
643 <para>Setting <literal>cpu_npartitions=1</literal> will disable most
644 of the SMP Node Affinity functionality.</para>
647 <title>CPU Partition String Patterns</title>
648 <para>CPU partitions can be described using string pattern notation.
649 If <literal>cpu_pattern=N</literal> is used, then there will be one
650 CPT for each NUMA node in the system, with each CPT mapping all of
651 the CPU cores for that NUMA node.
653 <para>It is also possible to explicitly specify the mapping between
654 CPU cores and CPTs, for example:</para>
658 <literal>cpu_pattern="0[2,4,6] 1[3,5,7]</literal>
660 <para>Create two CPTs, CPT0 contains cores 2, 4, and 6, while CPT1
661 contains cores 3, 5, 7. CPU cores 0 and 1 will not be used by Lustre
662 service threads, and could be used for node services such as
663 system monitoring, HA heartbeat threads, etc. The binding of
664 non-Lustre services to those CPU cores may be done in userspace
665 using <literal>numactl(8)</literal> or other application-specific
666 methods, but is beyond the scope of this document.</para>
670 <literal>cpu_pattern="N 0[0-3] 1[4-7]</literal>
672 <para>Create two CPTs, with CPT0 containing all CPUs in NUMA
673 node[0-3], while CPT1 contains all CPUs in NUMA node [4-7].</para>
676 <para>The current configuration of the CPU partition can be read via
677 <literal>lctl get_parm cpu_partition_table</literal>. For example,
678 a simple 4-core system has a single CPT with all four CPU cores:
679 <screen>$ lctl get_param cpu_partition_table
680 cpu_partition_table=0 : 0 1 2 3</screen>
681 while a larger NUMA system with four 12-core CPUs may have four CPTs:
682 <screen>$ lctl get_param cpu_partition_table
684 0 : 0 1 2 3 4 5 6 7 8 9 10 11
685 1 : 12 13 14 15 16 17 18 19 20 21 22 23
686 2 : 24 25 26 27 28 29 30 31 32 33 34 35
687 3 : 36 37 38 39 40 41 42 43 44 45 46 47
692 <section xml:id="lndtuning">
695 <primary>tuning</primary>
696 <secondary>LND tuning</secondary>
697 </indexterm>LND Tuning</title>
698 <para>LND tuning allows the number of threads per CPU partition to be
699 specified. An administrator can set the threads for both
700 <literal>ko2iblnd</literal> and
701 <literal>ksocklnd</literal> using the
702 <literal>nscheds</literal> parameter. This adjusts the number of threads for
703 each partition, not the overall number of threads on the LND.</para>
705 <para>The default number of threads for
706 <literal>ko2iblnd</literal> and
707 <literal>ksocklnd</literal> are automatically set and are chosen to
708 work well across a number of typical scenarios, for systems with both
709 high and low core counts.</para>
712 <title>ko2iblnd Tuning</title>
713 <para>The following table outlines the ko2iblnd module parameters to be used
715 <informaltable frame="all">
717 <colspec colname="c1" colwidth="50*" />
718 <colspec colname="c2" colwidth="50*" />
719 <colspec colname="c3" colwidth="50*" />
724 <emphasis role="bold">Module Parameter</emphasis>
729 <emphasis role="bold">Default Value</emphasis>
734 <emphasis role="bold">Description</emphasis>
743 <literal>service</literal>
748 <literal>987</literal>
752 <para>Service number (within RDMA_PS_TCP).</para>
758 <literal>cksum</literal>
767 <para>Set non-zero to enable message (not RDMA) checksums.</para>
773 <literal>timeout</literal>
778 <literal>50</literal>
782 <para>Timeout in seconds.</para>
788 <literal>nscheds</literal>
797 <para>Number of threads in each scheduler pool (per CPT). Value of
798 zero means we derive the number from the number of cores.</para>
804 <literal>conns_per_peer</literal>
809 <literal>4 (OmniPath), 1 (Everything else)</literal>
813 <para>Introduced in 2.10. Number of connections to each peer. Messages
814 are sent round-robin over the connection pool. Provides significant
815 improvement with OmniPath.</para>
821 <literal>ntx</literal>
826 <literal>512</literal>
830 <para>Number of message descriptors allocated for each pool at
831 startup. Grows at runtime. Shared by all CPTs.</para>
837 <literal>credits</literal>
842 <literal>256</literal>
846 <para>Number of concurrent sends on network.</para>
852 <literal>peer_credits</literal>
861 <para>Number of concurrent sends to 1 peer. Related/limited by IB
868 <literal>peer_credits_hiw</literal>
877 <para>When eagerly to return credits.</para>
883 <literal>peer_buffer_credits</literal>
892 <para>Number per-peer router buffer credits.</para>
898 <literal>peer_timeout</literal>
903 <literal>180</literal>
907 <para>Seconds without aliveness news to declare peer dead (less than
908 or equal to 0 to disable).</para>
914 <literal>ipif_name</literal>
919 <literal>ib0</literal>
923 <para>IPoIB interface name.</para>
929 <literal>retry_count</literal>
938 <para>Retransmissions when no ACK received.</para>
944 <literal>rnr_retry_count</literal>
953 <para>RNR retransmissions.</para>
959 <literal>keepalive</literal>
964 <literal>100</literal>
968 <para>Idle time in seconds before sending a keepalive.</para>
974 <literal>ib_mtu</literal>
983 <para>IB MTU 256/512/1024/2048/4096.</para>
989 <literal>concurrent_sends</literal>
998 <para>Send work-queue sizing. If zero, derived from
999 <literal>map_on_demand</literal> and <literal>peer_credits</literal>.
1006 <literal>map_on_demand</literal>
1011 <literal>0 (pre-4.8 Linux) 1 (4.8 Linux onward) 32 (OmniPath)</literal>
1015 <para>Number of fragments reserved for connection. If zero, use
1016 global memory region (found to be security issue). If non-zero, use
1017 FMR or FastReg for memory registration. Value needs to agree between
1018 both peers of connection.</para>
1024 <literal>fmr_pool_size</literal>
1029 <literal>512</literal>
1033 <para>Size of fmr pool on each CPT (>= ntx / 4). Grows at runtime.
1040 <literal>fmr_flush_trigger</literal>
1045 <literal>384</literal>
1049 <para>Number dirty FMRs that triggers pool flush.</para>
1055 <literal>fmr_cache</literal>
1060 <literal>1</literal>
1064 <para>Non-zero to enable FMR caching.</para>
1070 <literal>dev_failover</literal>
1075 <literal>0</literal>
1079 <para>HCA failover for bonding (0 OFF, 1 ON, other values reserved).
1086 <literal>require_privileged_port</literal>
1091 <literal>0</literal>
1095 <para>Require privileged port when accepting connection.</para>
1101 <literal>use_privileged_port</literal>
1106 <literal>1</literal>
1110 <para>Use privileged port when initiating connection.</para>
1116 <literal>wrq_sge</literal>
1121 <literal>2</literal>
1125 <para>Introduced in 2.10. Number scatter/gather element groups per
1126 work request. Used to deal with fragmentations which can consume
1127 double the number of work requests.</para>
1135 <section xml:id="nrstuning">
1138 <primary>tuning</primary>
1139 <secondary>Network Request Scheduler (NRS) Tuning</secondary>
1140 </indexterm>Network Request Scheduler (NRS) Tuning</title>
1141 <para>The Network Request Scheduler (NRS) allows the administrator to
1142 influence the order in which RPCs are handled at servers, on a per-PTLRPC
1143 service basis, by providing different policies that can be activated and
1144 tuned in order to influence the RPC ordering. The aim of this is to provide
1145 for better performance, and possibly discrete performance characteristics
1146 using future policies.</para>
1147 <para>The NRS policy state of a PTLRPC service can be read and set via the
1148 <literal>{service}.nrs_policies</literal> tunable. To read a PTLRPC
1149 service's NRS policy state, run:</para>
1151 lctl get_param {service}.nrs_policies
1153 <para>For example, to read the NRS policy state of the
1154 <literal>ost_io</literal> service, run:</para>
1156 $ lctl get_param ost.OSS.ost_io.nrs_policies
1157 ost.OSS.ost_io.nrs_policies=
1196 high_priority_requests:
1234 <para>NRS policy state is shown in either one or two sections, depending on
1235 the PTLRPC service being queried. The first section is named
1236 <literal>regular_requests</literal> and is available for all PTLRPC
1237 services, optionally followed by a second section which is named
1238 <literal>high_priority_requests</literal>. This is because some PTLRPC
1239 services are able to treat some types of RPCs as higher priority ones, such
1240 that they are handled by the server with higher priority compared to other,
1241 regular RPC traffic. For PTLRPC services that do not support high-priority
1242 RPCs, you will only see the
1243 <literal>regular_requests</literal> section.</para>
1244 <para>There is a separate instance of each NRS policy on each PTLRPC
1245 service for handling regular and high-priority RPCs (if the service
1246 supports high-priority RPCs). For each policy instance, the following
1247 fields are shown:</para>
1248 <informaltable frame="all">
1250 <colspec colname="c1" colwidth="50*" />
1251 <colspec colname="c2" colwidth="50*" />
1256 <emphasis role="bold">Field</emphasis>
1261 <emphasis role="bold">Description</emphasis>
1270 <literal>name</literal>
1274 <para>The name of the policy.</para>
1280 <literal>state</literal>
1284 <para>The state of the policy; this can be any of
1285 <literal>invalid, stopping, stopped, starting, started</literal>.
1286 A fully enabled policy is in the
1287 <literal>started</literal> state.</para>
1293 <literal>fallback</literal>
1297 <para>Whether the policy is acting as a fallback policy or not. A
1298 fallback policy is used to handle RPCs that other enabled
1299 policies fail to handle, or do not support the handling of. The
1301 <literal>no, yes</literal>. Currently, only the FIFO policy can
1302 act as a fallback policy.</para>
1308 <literal>queued</literal>
1312 <para>The number of RPCs that the policy has waiting to be
1319 <literal>active</literal>
1323 <para>The number of RPCs that the policy is currently
1330 <para>To enable an NRS policy on a PTLRPC service run:</para>
1332 lctl set_param {service}.nrs_policies=
1333 <replaceable>policy_name</replaceable>
1335 <para>This will enable the policy
1336 <replaceable>policy_name</replaceable>for both regular and high-priority
1337 RPCs (if the PLRPC service supports high-priority RPCs) on the given
1338 service. For example, to enable the CRR-N NRS policy for the ldlm_cbd
1339 service, run:</para>
1341 $ lctl set_param ldlm.services.ldlm_cbd.nrs_policies=crrn
1342 ldlm.services.ldlm_cbd.nrs_policies=crrn
1345 <para>For PTLRPC services that support high-priority RPCs, you can also
1347 <replaceable>reg|hp</replaceable>token, in order to enable an NRS policy
1348 for handling only regular or high-priority RPCs on a given PTLRPC service,
1351 lctl set_param {service}.nrs_policies="
1352 <replaceable>policy_name</replaceable>
1353 <replaceable>reg|hp</replaceable>"
1355 <para>For example, to enable the TRR policy for handling only regular, but
1356 not high-priority RPCs on the
1357 <literal>ost_io</literal> service, run:</para>
1359 $ lctl set_param ost.OSS.ost_io.nrs_policies="trr reg"
1360 ost.OSS.ost_io.nrs_policies="trr reg"
1364 <para>When enabling an NRS policy, the policy name must be given in
1365 lower-case characters, otherwise the operation will fail with an error
1371 <primary>tuning</primary>
1372 <secondary>Network Request Scheduler (NRS) Tuning</secondary>
1373 <tertiary>first in, first out (FIFO) policy</tertiary>
1374 </indexterm>First In, First Out (FIFO) policy</title>
1375 <para>The first in, first out (FIFO) policy handles RPCs in a service in
1376 the same order as they arrive from the LNet layer, so no special
1377 processing takes place to modify the RPC handling stream. FIFO is the
1378 default policy for all types of RPCs on all PTLRPC services, and is
1379 always enabled irrespective of the state of other policies, so that it
1380 can be used as a backup policy, in case a more elaborate policy that has
1381 been enabled fails to handle an RPC, or does not support handling a given
1383 <para>The FIFO policy has no tunables that adjust its behaviour.</para>
1388 <primary>tuning</primary>
1389 <secondary>Network Request Scheduler (NRS) Tuning</secondary>
1390 <tertiary>client round-robin over NIDs (CRR-N) policy</tertiary>
1391 </indexterm>Client Round-Robin over NIDs (CRR-N) policy</title>
1392 <para>The client round-robin over NIDs (CRR-N) policy performs batched
1393 round-robin scheduling of all types of RPCs, with each batch consisting
1394 of RPCs originating from the same client node, as identified by its NID.
1395 CRR-N aims to provide for better resource utilization across the cluster,
1396 and to help shorten completion times of jobs in some cases, by
1397 distributing available bandwidth more evenly across all clients.</para>
1398 <para>The CRR-N policy can be enabled on all types of PTLRPC services,
1399 and has the following tunable that can be used to adjust its
1404 <literal>{service}.nrs_crrn_quantum</literal>
1407 <literal>{service}.nrs_crrn_quantum</literal> tunable determines the
1408 maximum allowed size of each batch of RPCs; the unit of measure is in
1409 number of RPCs. To read the maximum allowed batch size of a CRR-N
1412 lctl get_param {service}.nrs_crrn_quantum
1414 <para>For example, to read the maximum allowed batch size of a CRR-N
1415 policy on the ost_io service, run:</para>
1417 $ lctl get_param ost.OSS.ost_io.nrs_crrn_quantum
1418 ost.OSS.ost_io.nrs_crrn_quantum=reg_quantum:16
1422 <para>You can see that there is a separate maximum allowed batch size
1424 <literal>reg_quantum</literal>) and high-priority (
1425 <literal>hp_quantum</literal>) RPCs (if the PTLRPC service supports
1426 high-priority RPCs).</para>
1427 <para>To set the maximum allowed batch size of a CRR-N policy on a
1428 given service, run:</para>
1430 lctl set_param {service}.nrs_crrn_quantum=
1431 <replaceable>1-65535</replaceable>
1433 <para>This will set the maximum allowed batch size on a given
1434 service, for both regular and high-priority RPCs (if the PLRPC
1435 service supports high-priority RPCs), to the indicated value.</para>
1436 <para>For example, to set the maximum allowed batch size on the
1437 ldlm_canceld service to 16 RPCs, run:</para>
1439 $ lctl set_param ldlm.services.ldlm_canceld.nrs_crrn_quantum=16
1440 ldlm.services.ldlm_canceld.nrs_crrn_quantum=16
1443 <para>For PTLRPC services that support high-priority RPCs, you can
1444 also specify a different maximum allowed batch size for regular and
1445 high-priority RPCs, by running:</para>
1447 $ lctl set_param {service}.nrs_crrn_quantum=
1448 <replaceable>reg_quantum|hp_quantum</replaceable>:
1449 <replaceable>1-65535</replaceable>"
1451 <para>For example, to set the maximum allowed batch size on the
1452 ldlm_canceld service, for high-priority RPCs to 32, run:</para>
1454 $ lctl set_param ldlm.services.ldlm_canceld.nrs_crrn_quantum="hp_quantum:32"
1455 ldlm.services.ldlm_canceld.nrs_crrn_quantum=hp_quantum:32
1458 <para>By using the last method, you can also set the maximum regular
1459 and high-priority RPC batch sizes to different values, in a single
1460 command invocation.</para>
1467 <primary>tuning</primary>
1468 <secondary>Network Request Scheduler (NRS) Tuning</secondary>
1469 <tertiary>object-based round-robin (ORR) policy</tertiary>
1470 </indexterm>Object-based Round-Robin (ORR) policy</title>
1471 <para>The object-based round-robin (ORR) policy performs batched
1472 round-robin scheduling of bulk read write (brw) RPCs, with each batch
1473 consisting of RPCs that pertain to the same backend-file system object,
1474 as identified by its OST FID.</para>
1475 <para>The ORR policy is only available for use on the ost_io service. The
1476 RPC batches it forms can potentially consist of mixed bulk read and bulk
1477 write RPCs. The RPCs in each batch are ordered in an ascending manner,
1478 based on either the file offsets, or the physical disk offsets of each
1479 RPC (only applicable to bulk read RPCs).</para>
1480 <para>The aim of the ORR policy is to provide for increased bulk read
1481 throughput in some cases, by ordering bulk read RPCs (and potentially
1482 bulk write RPCs), and thus minimizing costly disk seek operations.
1483 Performance may also benefit from any resulting improvement in resource
1484 utilization, or by taking advantage of better locality of reference
1485 between RPCs.</para>
1486 <para>The ORR policy has the following tunables that can be used to
1487 adjust its behaviour:</para>
1491 <literal>ost.OSS.ost_io.nrs_orr_quantum</literal>
1494 <literal>ost.OSS.ost_io.nrs_orr_quantum</literal> tunable determines
1495 the maximum allowed size of each batch of RPCs; the unit of measure
1496 is in number of RPCs. To read the maximum allowed batch size of the
1497 ORR policy, run:</para>
1499 $ lctl get_param ost.OSS.ost_io.nrs_orr_quantum
1500 ost.OSS.ost_io.nrs_orr_quantum=reg_quantum:256
1504 <para>You can see that there is a separate maximum allowed batch size
1506 <literal>reg_quantum</literal>) and high-priority (
1507 <literal>hp_quantum</literal>) RPCs (if the PTLRPC service supports
1508 high-priority RPCs).</para>
1509 <para>To set the maximum allowed batch size for the ORR policy,
1512 $ lctl set_param ost.OSS.ost_io.nrs_orr_quantum=
1513 <replaceable>1-65535</replaceable>
1515 <para>This will set the maximum allowed batch size for both regular
1516 and high-priority RPCs, to the indicated value.</para>
1517 <para>You can also specify a different maximum allowed batch size for
1518 regular and high-priority RPCs, by running:</para>
1520 $ lctl set_param ost.OSS.ost_io.nrs_orr_quantum=
1521 <replaceable>reg_quantum|hp_quantum</replaceable>:
1522 <replaceable>1-65535</replaceable>
1524 <para>For example, to set the maximum allowed batch size for regular
1525 RPCs to 128, run:</para>
1527 $ lctl set_param ost.OSS.ost_io.nrs_orr_quantum=reg_quantum:128
1528 ost.OSS.ost_io.nrs_orr_quantum=reg_quantum:128
1531 <para>By using the last method, you can also set the maximum regular
1532 and high-priority RPC batch sizes to different values, in a single
1533 command invocation.</para>
1537 <literal>ost.OSS.ost_io.nrs_orr_offset_type</literal>
1540 <literal>ost.OSS.ost_io.nrs_orr_offset_type</literal> tunable
1541 determines whether the ORR policy orders RPCs within each batch based
1542 on logical file offsets or physical disk offsets. To read the offset
1543 type value for the ORR policy, run:</para>
1545 $ lctl get_param ost.OSS.ost_io.nrs_orr_offset_type
1546 ost.OSS.ost_io.nrs_orr_offset_type=reg_offset_type:physical
1547 hp_offset_type:logical
1550 <para>You can see that there is a separate offset type value for
1552 <literal>reg_offset_type</literal>) and high-priority (
1553 <literal>hp_offset_type</literal>) RPCs.</para>
1554 <para>To set the ordering type for the ORR policy, run:</para>
1556 $ lctl set_param ost.OSS.ost_io.nrs_orr_offset_type=
1557 <replaceable>physical|logical</replaceable>
1559 <para>This will set the offset type for both regular and
1560 high-priority RPCs, to the indicated value.</para>
1561 <para>You can also specify a different offset type for regular and
1562 high-priority RPCs, by running:</para>
1564 $ lctl set_param ost.OSS.ost_io.nrs_orr_offset_type=
1565 <replaceable>reg_offset_type|hp_offset_type</replaceable>:
1566 <replaceable>physical|logical</replaceable>
1568 <para>For example, to set the offset type for high-priority RPCs to
1569 physical disk offsets, run:</para>
1571 $ lctl set_param ost.OSS.ost_io.nrs_orr_offset_type=hp_offset_type:physical
1572 ost.OSS.ost_io.nrs_orr_offset_type=hp_offset_type:physical
1574 <para>By using the last method, you can also set offset type for
1575 regular and high-priority RPCs to different values, in a single
1576 command invocation.</para>
1578 <para>Irrespective of the value of this tunable, only logical
1579 offsets can, and are used for ordering bulk write RPCs.</para>
1584 <literal>ost.OSS.ost_io.nrs_orr_supported</literal>
1587 <literal>ost.OSS.ost_io.nrs_orr_supported</literal> tunable determines
1588 the type of RPCs that the ORR policy will handle. To read the types
1589 of supported RPCs by the ORR policy, run:</para>
1591 $ lctl get_param ost.OSS.ost_io.nrs_orr_supported
1592 ost.OSS.ost_io.nrs_orr_supported=reg_supported:reads
1593 hp_supported=reads_and_writes
1596 <para>You can see that there is a separate supported 'RPC types'
1598 <literal>reg_supported</literal>) and high-priority (
1599 <literal>hp_supported</literal>) RPCs.</para>
1600 <para>To set the supported RPC types for the ORR policy, run:</para>
1602 $ lctl set_param ost.OSS.ost_io.nrs_orr_supported=
1603 <replaceable>reads|writes|reads_and_writes</replaceable>
1605 <para>This will set the supported RPC types for both regular and
1606 high-priority RPCs, to the indicated value.</para>
1607 <para>You can also specify a different supported 'RPC types' value
1608 for regular and high-priority RPCs, by running:</para>
1610 $ lctl set_param ost.OSS.ost_io.nrs_orr_supported=
1611 <replaceable>reg_supported|hp_supported</replaceable>:
1612 <replaceable>reads|writes|reads_and_writes</replaceable>
1614 <para>For example, to set the supported RPC types to bulk read and
1615 bulk write RPCs for regular requests, run:</para>
1618 ost.OSS.ost_io.nrs_orr_supported=reg_supported:reads_and_writes
1619 ost.OSS.ost_io.nrs_orr_supported=reg_supported:reads_and_writes
1622 <para>By using the last method, you can also set the supported RPC
1623 types for regular and high-priority RPC to different values, in a
1624 single command invocation.</para>
1631 <primary>tuning</primary>
1632 <secondary>Network Request Scheduler (NRS) Tuning</secondary>
1633 <tertiary>Target-based round-robin (TRR) policy</tertiary>
1634 </indexterm>Target-based Round-Robin (TRR) policy</title>
1635 <para>The target-based round-robin (TRR) policy performs batched
1636 round-robin scheduling of brw RPCs, with each batch consisting of RPCs
1637 that pertain to the same OST, as identified by its OST index.</para>
1638 <para>The TRR policy is identical to the object-based round-robin (ORR)
1639 policy, apart from using the brw RPC's target OST index instead of the
1640 backend-fs object's OST FID, for determining the RPC scheduling order.
1641 The goals of TRR are effectively the same as for ORR, and it uses the
1642 following tunables to adjust its behaviour:</para>
1646 <literal>ost.OSS.ost_io.nrs_trr_quantum</literal>
1648 <para>The purpose of this tunable is exactly the same as for the
1649 <literal>ost.OSS.ost_io.nrs_orr_quantum</literal> tunable for the ORR
1650 policy, and you can use it in exactly the same way.</para>
1654 <literal>ost.OSS.ost_io.nrs_trr_offset_type</literal>
1656 <para>The purpose of this tunable is exactly the same as for the
1657 <literal>ost.OSS.ost_io.nrs_orr_offset_type</literal> tunable for the
1658 ORR policy, and you can use it in exactly the same way.</para>
1662 <literal>ost.OSS.ost_io.nrs_trr_supported</literal>
1664 <para>The purpose of this tunable is exactly the same as for the
1665 <literal>ost.OSS.ost_io.nrs_orr_supported</literal> tunable for the
1666 ORR policy, and you can use it in exactly the sme way.</para>
1670 <section xml:id="tbftuning" condition='l26'>
1673 <primary>tuning</primary>
1674 <secondary>Network Request Scheduler (NRS) Tuning</secondary>
1675 <tertiary>Token Bucket Filter (TBF) policy</tertiary>
1676 </indexterm>Token Bucket Filter (TBF) policy</title>
1677 <para>The TBF (Token Bucket Filter) is a Lustre NRS policy which enables
1678 Lustre services to enforce the RPC rate limit on clients/jobs for QoS
1679 (Quality of Service) purposes.</para>
1681 <title>The internal structure of TBF policy</title>
1684 <imagedata scalefit="1" width="50%"
1685 fileref="figures/TBF_policy.png" />
1688 <phrase>The internal structure of TBF policy</phrase>
1692 <para>When a RPC request arrives, TBF policy puts it to a waiting queue
1693 according to its classification. The classification of RPC requests is
1694 based on either NID or JobID of the RPC according to the configure of
1695 TBF. TBF policy maintains multiple queues in the system, one queue for
1696 each category in the classification of RPC requests. The requests waits
1697 for tokens in the FIFO queue before they have been handled so as to keep
1698 the RPC rates under the limits.</para>
1699 <para>When Lustre services are too busy to handle all of the requests in
1700 time, all of the specified rates of the queues will not be satisfied.
1701 Nothing bad will happen except some of the RPC rates are slower than
1702 configured. In this case, the queue with higher rate will have an
1703 advantage over the queues with lower rates, but none of them will be
1705 <para>To manage the RPC rate of queues, we don't need to set the rate of
1706 each queue manually. Instead, we define rules which TBF policy matches to
1707 determine RPC rate limits. All of the defined rules are organized as an
1708 ordered list. Whenever a queue is newly created, it goes though the rule
1709 list and takes the first matched rule as its rule, so that the queue
1710 knows its RPC token rate. A rule can be added to or removed from the list
1711 at run time. Whenever the list of rules is changed, the queues will
1712 update their matched rules.</para>
1713 <section remap="h4">
1714 <title>Enable TBF policy</title>
1715 <para>Command:</para>
1716 <screen>lctl set_param ost.OSS.ost_io.nrs_policies="tbf <<replaceable>policy</replaceable>>"
1718 <para>For now, the RPCs can be classified into the different types
1719 according to their NID, JOBID, OPCode and UID/GID. When enabling TBF
1720 policy, you can specify one of the types, or just use "tbf" to enable
1721 all of them to do a fine-grained RPC requests classification.</para>
1722 <para>Example:</para>
1723 <screen>$ lctl set_param ost.OSS.ost_io.nrs_policies="tbf"
1724 $ lctl set_param ost.OSS.ost_io.nrs_policies="tbf nid"
1725 $ lctl set_param ost.OSS.ost_io.nrs_policies="tbf jobid"
1726 $ lctl set_param ost.OSS.ost_io.nrs_policies="tbf opcode"
1727 $ lctl set_param ost.OSS.ost_io.nrs_policies="tbf uid"
1728 $ lctl set_param ost.OSS.ost_io.nrs_policies="tbf gid"</screen>
1730 <section remap="h4">
1731 <title>Start a TBF rule</title>
1732 <para>The TBF rule is defined in the parameter
1733 <literal>ost.OSS.ost_io.nrs_tbf_rule</literal>.</para>
1734 <para>Command:</para>
1735 <screen>lctl set_param x.x.x.nrs_tbf_rule=
1736 "[reg|hp] start <replaceable>rule_name</replaceable> <replaceable>arguments</replaceable>..."
1738 <para>'<replaceable>rule_name</replaceable>' is a string up to 15
1739 characters which identifies the TBF policy rule's name. Alphanumeric
1740 characters and underscores are accepted (e.g: "test_rule_A1").
1742 <para>'<replaceable>arguments</replaceable>' is a string to specify the
1743 detailed rule according to the different types.
1746 <para>Next, the different types of TBF policies will be described.</para>
1748 <para><emphasis role="bold">NID based TBF policy</emphasis></para>
1749 <para>Command:</para>
1750 <screen>lctl set_param x.x.x.nrs_tbf_rule=
1751 "[reg|hp] start <replaceable>rule_name</replaceable> nid={<replaceable>nidlist</replaceable>} rate=<replaceable>rate</replaceable>"
1753 <para>'<replaceable>nidlist</replaceable>' uses the same format
1754 as configuring LNET route. '<replaceable>rate</replaceable>' is
1755 the (upper limit) RPC rate of the rule.</para>
1756 <para>Example:</para>
1757 <screen>$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\
1758 "start other_clients nid={192.168.*.*@tcp} rate=50"
1759 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\
1760 "start computes nid={192.168.1.[2-128]@tcp} rate=500"
1761 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\
1762 "start loginnode nid={192.168.1.1@tcp} rate=100"</screen>
1763 <para>In this example, the rate of processing RPC requests from
1764 compute nodes is at most 5x as fast as those from login nodes.
1765 The output of <literal>ost.OSS.ost_io.nrs_tbf_rule</literal> is
1767 <screen>lctl get_param ost.OSS.ost_io.nrs_tbf_rule
1768 ost.OSS.ost_io.nrs_tbf_rule=
1771 loginnode {192.168.1.1@tcp} 100, ref 0
1772 computes {192.168.1.[2-128]@tcp} 500, ref 0
1773 other_clients {192.168.*.*@tcp} 50, ref 0
1774 default {*} 10000, ref 0
1775 high_priority_requests:
1777 loginnode {192.168.1.1@tcp} 100, ref 0
1778 computes {192.168.1.[2-128]@tcp} 500, ref 0
1779 other_clients {192.168.*.*@tcp} 50, ref 0
1780 default {*} 10000, ref 0</screen>
1781 <para>Also, the rule can be written in <literal>reg</literal> and
1782 <literal>hp</literal> formats:</para>
1783 <screen>$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\
1784 "reg start loginnode nid={192.168.1.1@tcp} rate=100"
1785 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\
1786 "hp start loginnode nid={192.168.1.1@tcp} rate=100"</screen>
1789 <para><emphasis role="bold">JobID based TBF policy</emphasis></para>
1790 <para>For the JobID, please see
1791 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
1792 linkend="jobstats" /> for more details.</para>
1793 <para>Command:</para>
1794 <screen>lctl set_param x.x.x.nrs_tbf_rule=
1795 "[reg|hp] start <replaceable>rule_name</replaceable> jobid={<replaceable>jobid_list</replaceable>} rate=<replaceable>rate</replaceable>"
1797 <para>Wildcard is supported in
1798 {<replaceable>jobid_list</replaceable>}.</para>
1799 <para>Example:</para>
1800 <screen>$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\
1801 "start iozone_user jobid={iozone.500} rate=100"
1802 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\
1803 "start dd_user jobid={dd.*} rate=50"
1804 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\
1805 "start user1 jobid={*.600} rate=10"
1806 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\
1807 "start user2 jobid={io*.10* *.500} rate=200"</screen>
1808 <para>Also, the rule can be written in <literal>reg</literal> and
1809 <literal>hp</literal> formats:</para>
1810 <screen>$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\
1811 "hp start iozone_user1 jobid={iozone.500} rate=100"
1812 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\
1813 "reg start iozone_user1 jobid={iozone.500} rate=100"</screen>
1816 <para><emphasis role="bold">Opcode based TBF policy</emphasis></para>
1817 <para>Command:</para>
1818 <screen>$ lctl set_param x.x.x.nrs_tbf_rule=
1819 "[reg|hp] start <replaceable>rule_name</replaceable> opcode={<replaceable>opcode_list</replaceable>} rate=<replaceable>rate</replaceable>"
1821 <para>Example:</para>
1822 <screen>$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\
1823 "start user1 opcode={ost_read} rate=100"
1824 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\
1825 "start iozone_user1 opcode={ost_read ost_write} rate=200"</screen>
1826 <para>Also, the rule can be written in <literal>reg</literal> and
1827 <literal>hp</literal> formats:</para>
1828 <screen>$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\
1829 "hp start iozone_user1 opcode={ost_read} rate=100"
1830 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\
1831 "reg start iozone_user1 opcode={ost_read} rate=100"</screen>
1834 <para><emphasis role="bold">UID/GID based TBF policy</emphasis></para>
1835 <para>Command:</para>
1836 <screen>$ lctl set_param ost.OSS.*.nrs_tbf_rule=\
1837 "[reg][hp] start <replaceable>rule_name</replaceable> uid={<replaceable>uid</replaceable>} rate=<replaceable>rate</replaceable>"
1838 $ lctl set_param ost.OSS.*.nrs_tbf_rule=\
1839 "[reg][hp] start <replaceable>rule_name</replaceable> gid={<replaceable>gid</replaceable>} rate=<replaceable>rate</replaceable>"</screen>
1840 <para>Exapmle:</para>
1841 <para>Limit the rate of RPC requests of the uid 500</para>
1842 <screen>$ lctl set_param ost.OSS.*.nrs_tbf_rule=\
1843 "start tbf_name uid={500} rate=100"</screen>
1844 <para>Limit the rate of RPC requests of the gid 500</para>
1845 <screen>$ lctl set_param ost.OSS.*.nrs_tbf_rule=\
1846 "start tbf_name gid={500} rate=100"</screen>
1847 <para>Also, you can use the following rule to control all reqs
1849 <para>Start the tbf uid QoS on MDS:</para>
1850 <screen>$ lctl set_param mds.MDS.*.nrs_policies="tbf uid"</screen>
1851 <para>Limit the rate of RPC requests of the uid 500</para>
1852 <screen>$ lctl set_param mds.MDS.*.nrs_tbf_rule=\
1853 "start tbf_name uid={500} rate=100"</screen>
1856 <para><emphasis role="bold">Policy combination</emphasis></para>
1857 <para>To support TBF rules with complex expressions of conditions,
1858 TBF classifier is extented to classify RPC in a more fine-grained
1859 way. This feature supports logical conditional conjunction and
1860 disjunction operations among different types.
1862 "&" represents the conditional conjunction and
1863 "," represents the conditional disjunction.</para>
1864 <para>Example:</para>
1865 <screen>$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\
1866 "start comp_rule opcode={ost_write}&jobid={dd.0},\
1867 nid={192.168.1.[1-128]@tcp 0@lo} rate=100"</screen>
1868 <para>In this example, those RPCs whose <literal>opcode</literal> is
1869 ost_write and <literal>jobid</literal> is dd.0, or
1870 <literal>nid</literal> satisfies the condition of
1871 {192.168.1.[1-128]@tcp 0@lo} will be processed at the rate of 100
1873 The output of <literal>ost.OSS.ost_io.nrs_tbf_rule</literal>is like:
1875 <screen>$ lctl get_param ost.OSS.ost_io.nrs_tbf_rule
1876 ost.OSS.ost_io.nrs_tbf_rule=
1879 comp_rule opcode={ost_write}&jobid={dd.0},nid={192.168.1.[1-128]@tcp 0@lo} 100, ref 0
1880 default * 10000, ref 0
1882 comp_rule opcode={ost_write}&jobid={dd.0},nid={192.168.1.[1-128]@tcp 0@lo} 100, ref 0
1883 default * 10000, ref 0
1884 high_priority_requests:
1886 comp_rule opcode={ost_write}&jobid={dd.0},nid={192.168.1.[1-128]@tcp 0@lo} 100, ref 0
1887 default * 10000, ref 0
1889 comp_rule opcode={ost_write}&jobid={dd.0},nid={192.168.1.[1-128]@tcp 0@lo} 100, ref 0
1890 default * 10000, ref 0</screen>
1891 <para>Example:</para>
1892 <screen>$ lctl set_param ost.OSS.*.nrs_tbf_rule=\
1893 "start tbf_name uid={500}&gid={500} rate=100"</screen>
1894 <para>In this example, those RPC requests whose uid is 500 and
1895 gid is 500 will be processed at the rate of 100 req/sec.</para>
1899 <section remap="h4">
1900 <title>Change a TBF rule</title>
1901 <para>Command:</para>
1902 <screen>lctl set_param x.x.x.nrs_tbf_rule=
1903 "[reg|hp] change <replaceable>rule_name</replaceable> rate=<replaceable>rate</replaceable>"
1905 <para>Example:</para>
1906 <screen>$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\
1907 "change loginnode rate=200"
1908 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\
1909 "reg change loginnode rate=200"
1910 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\
1911 "hp change loginnode rate=200"
1914 <section remap="h4">
1915 <title>Stop a TBF rule</title>
1916 <para>Command:</para>
1917 <screen>lctl set_param x.x.x.nrs_tbf_rule="[reg|hp] stop
1918 <replaceable>rule_name</replaceable>"</screen>
1919 <para>Example:</para>
1920 <screen>$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule="stop loginnode"
1921 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule="reg stop loginnode"
1922 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule="hp stop loginnode"</screen>
1924 <section remap="h4">
1925 <title>Rule options</title>
1926 <para>To support more flexible rule conditions, the following options
1930 <para><emphasis role="bold">Reordering of TBF rules</emphasis></para>
1931 <para>By default, a newly started rule is prior to the old ones,
1932 but by specifying the argument '<literal>rank=</literal>' when
1933 inserting a new rule with "<literal>start</literal>" command,
1934 the rank of the rule can be changed. Also, it can be changed by
1935 "<literal>change</literal>" command.
1937 <para>Command:</para>
1938 <screen>lctl set_param ost.OSS.ost_io.nrs_tbf_rule=
1939 "start <replaceable>rule_name</replaceable> <replaceable>arguments</replaceable>... rank=<replaceable>obj_rule_name</replaceable>"
1940 lctl set_param ost.OSS.ost_io.nrs_tbf_rule=
1941 "change <replaceable>rule_name</replaceable> rate=<replaceable>rate</replaceable> rank=<replaceable>obj_rule_name</replaceable>"
1943 <para>By specifying the existing rule
1944 '<replaceable>obj_rule_name</replaceable>', the new rule
1945 '<replaceable>rule_name</replaceable>' will be moved to the front of
1946 '<replaceable>obj_rule_name</replaceable>'.</para>
1947 <para>Example:</para>
1948 <screen>$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\
1949 "start computes nid={192.168.1.[2-128]@tcp} rate=500"
1950 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\
1951 "start user1 jobid={iozone.500 dd.500} rate=100"
1952 $ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=\
1953 "start iozone_user1 opcode={ost_read ost_write} rate=200 rank=computes"</screen>
1954 <para>In this example, rule "iozone_user1" is added to the front of
1955 rule "computes". We can see the order by the following command:
1957 <screen>$ lctl get_param ost.OSS.ost_io.nrs_tbf_rule
1958 ost.OSS.ost_io.nrs_tbf_rule=
1961 user1 jobid={iozone.500 dd.500} 100, ref 0
1962 iozone_user1 opcode={ost_read ost_write} 200, ref 0
1963 computes nid={192.168.1.[2-128]@tcp} 500, ref 0
1964 default * 10000, ref 0
1966 user1 jobid={iozone.500 dd.500} 100, ref 0
1967 iozone_user1 opcode={ost_read ost_write} 200, ref 0
1968 computes nid={192.168.1.[2-128]@tcp} 500, ref 0
1969 default * 10000, ref 0
1970 high_priority_requests:
1972 user1 jobid={iozone.500 dd.500} 100, ref 0
1973 iozone_user1 opcode={ost_read ost_write} 200, ref 0
1974 computes nid={192.168.1.[2-128]@tcp} 500, ref 0
1975 default * 10000, ref 0
1977 user1 jobid={iozone.500 dd.500} 100, ref 0
1978 iozone_user1 opcode={ost_read ost_write} 200, ref 0
1979 computes nid={192.168.1.[2-128]@tcp} 500, ref 0
1980 default * 10000, ref 0</screen>
1983 <para><emphasis role="bold">TBF realtime policies under congestion
1985 <para>During TBF evaluation, we find that when the sum of I/O
1986 bandwidth requirements for all classes exceeds the system capacity,
1987 the classes with the same rate limits get less bandwidth than if
1988 preconfigured evenly. The reason for this is the heavy load on a
1989 congested server will result in some missed deadlines for some
1990 classes. The number of the calculated tokens may be larger than 1
1991 during dequeuing. In the original implementation, all classes are
1992 equally handled to simply discard exceeding tokens.</para>
1993 <para>Thus, a Hard Token Compensation (HTC) strategy has been
1994 implemented. A class can be configured with the HTC feature by the
1995 rule it matches. This feature means that requests in this kind of
1996 class queues have high real-time requirements and that the bandwidth
1997 assignment must be satisfied as good as possible. When deadline
1998 misses happen, the class keeps the deadline unchanged and the time
1999 residue(the remainder of elapsed time divided by 1/r) is compensated
2000 to the next round. This ensures that the next idle I/O thread will
2001 always select this class to serve until all accumulated exceeding
2002 tokens are handled or there are no pending requests in the class
2004 <para>Command:</para>
2005 <para>A new command format is added to enable the realtime feature
2007 <screen>lctl set_param x.x.x.nrs_tbf_rule=\
2008 "start <replaceable>rule_name</replaceable> <replaceable>arguments</replaceable>... realtime=1</screen>
2009 <para>Example:</para>
2010 <screen>$ lctl set_param ost.OSS.ost_io.nrs_tbf_rule=
2011 "start realjob jobid={dd.0} rate=100 realtime=1</screen>
2012 <para>This example rule means the RPC requests whose JobID is dd.0
2013 will be processed at the rate of 100req/sec in realtime.</para>
2018 <section xml:id="delaytuning" condition='l2A'>
2021 <primary>tuning</primary>
2022 <secondary>Network Request Scheduler (NRS) Tuning</secondary>
2023 <tertiary>Delay policy</tertiary>
2024 </indexterm>Delay policy</title>
2025 <para>The NRS Delay policy seeks to perturb the timing of request
2026 processing at the PtlRPC layer, with the goal of simulating high server
2027 load, and finding and exposing timing related problems. When this policy
2028 is active, upon arrival of a request the policy will calculate an offset,
2029 within a defined, user-configurable range, from the request arrival
2030 time, to determine a time after which the request should be handled.
2031 The request is then stored using the cfs_binheap implementation,
2032 which sorts the request according to the assigned start time.
2033 Requests are removed from the binheap for handling once their start
2034 time has been passed.</para>
2035 <para>The Delay policy can be enabled on all types of PtlRPC services,
2036 and has the following tunables that can be used to adjust its behavior:
2041 <literal>{service}.nrs_delay_min</literal>
2044 <literal>{service}.nrs_delay_min</literal> tunable controls the
2045 minimum amount of time, in seconds, that a request will be delayed by
2046 this policy. The default is 5 seconds. To read this value run:</para>
2048 lctl get_param {service}.nrs_delay_min</screen>
2049 <para>For example, to read the minimum delay set on the ost_io
2050 service, run:</para>
2052 $ lctl get_param ost.OSS.ost_io.nrs_delay_min
2053 ost.OSS.ost_io.nrs_delay_min=reg_delay_min:5
2054 hp_delay_min:5</screen>
2055 <para>To set the minimum delay in RPC processing, run:</para>
2057 lctl set_param {service}.nrs_delay_min=<replaceable>0-65535</replaceable></screen>
2058 <para>This will set the minimum delay time on a given service, for both
2059 regular and high-priority RPCs (if the PtlRPC service supports
2060 high-priority RPCs), to the indicated value.</para>
2061 <para>For example, to set the minimum delay time on the ost_io service
2064 $ lctl set_param ost.OSS.ost_io.nrs_delay_min=10
2065 ost.OSS.ost_io.nrs_delay_min=10</screen>
2066 <para>For PtlRPC services that support high-priority RPCs, to set a
2067 different minimum delay time for regular and high-priority RPCs, run:
2070 lctl set_param {service}.nrs_delay_min=<replaceable>reg_delay_min|hp_delay_min</replaceable>:<replaceable>0-65535</replaceable>
2072 <para>For example, to set the minimum delay time on the ost_io service
2073 for high-priority RPCs to 3, run:</para>
2075 $ lctl set_param ost.OSS.ost_io.nrs_delay_min=hp_delay_min:3
2076 ost.OSS.ost_io.nrs_delay_min=hp_delay_min:3</screen>
2077 <para>Note, in all cases the minimum delay time cannot exceed the
2078 maximum delay time.</para>
2082 <literal>{service}.nrs_delay_max</literal>
2085 <literal>{service}.nrs_delay_max</literal> tunable controls the
2086 maximum amount of time, in seconds, that a request will be delayed by
2087 this policy. The default is 300 seconds. To read this value run:
2089 <screen>lctl get_param {service}.nrs_delay_max</screen>
2090 <para>For example, to read the maximum delay set on the ost_io
2091 service, run:</para>
2093 $ lctl get_param ost.OSS.ost_io.nrs_delay_max
2094 ost.OSS.ost_io.nrs_delay_max=reg_delay_max:300
2095 hp_delay_max:300</screen>
2096 <para>To set the maximum delay in RPC processing, run:</para>
2097 <screen>lctl set_param {service}.nrs_delay_max=<replaceable>0-65535</replaceable>
2099 <para>This will set the maximum delay time on a given service, for both
2100 regular and high-priority RPCs (if the PtlRPC service supports
2101 high-priority RPCs), to the indicated value.</para>
2102 <para>For example, to set the maximum delay time on the ost_io service
2105 $ lctl set_param ost.OSS.ost_io.nrs_delay_max=60
2106 ost.OSS.ost_io.nrs_delay_max=60</screen>
2107 <para>For PtlRPC services that support high-priority RPCs, to set a
2108 different maximum delay time for regular and high-priority RPCs, run:
2110 <screen>lctl set_param {service}.nrs_delay_max=<replaceable>reg_delay_max|hp_delay_max</replaceable>:<replaceable>0-65535</replaceable></screen>
2111 <para>For example, to set the maximum delay time on the ost_io service
2112 for high-priority RPCs to 30, run:</para>
2114 $ lctl set_param ost.OSS.ost_io.nrs_delay_max=hp_delay_max:30
2115 ost.OSS.ost_io.nrs_delay_max=hp_delay_max:30</screen>
2116 <para>Note, in all cases the maximum delay time cannot be less than the
2117 minimum delay time.</para>
2121 <literal>{service}.nrs_delay_pct</literal>
2124 <literal>{service}.nrs_delay_pct</literal> tunable controls the
2125 percentage of requests that will be delayed by this policy. The
2126 default is 100. Note, when a request is not selected for handling by
2127 the delay policy due to this variable then the request will be handled
2128 by whatever fallback policy is defined for that service. If no other
2129 fallback policy is defined then the request will be handled by the
2130 FIFO policy. To read this value run:</para>
2131 <screen>lctl get_param {service}.nrs_delay_pct</screen>
2132 <para>For example, to read the percentage of requests being delayed on
2133 the ost_io service, run:</para>
2135 $ lctl get_param ost.OSS.ost_io.nrs_delay_pct
2136 ost.OSS.ost_io.nrs_delay_pct=reg_delay_pct:100
2137 hp_delay_pct:100</screen>
2138 <para>To set the percentage of delayed requests, run:</para>
2140 lctl set_param {service}.nrs_delay_pct=<replaceable>0-100</replaceable></screen>
2141 <para>This will set the percentage of requests delayed on a given
2142 service, for both regular and high-priority RPCs (if the PtlRPC service
2143 supports high-priority RPCs), to the indicated value.</para>
2144 <para>For example, to set the percentage of delayed requests on the
2145 ost_io service to 50, run:</para>
2147 $ lctl set_param ost.OSS.ost_io.nrs_delay_pct=50
2148 ost.OSS.ost_io.nrs_delay_pct=50
2150 <para>For PtlRPC services that support high-priority RPCs, to set a
2151 different delay percentage for regular and high-priority RPCs, run:
2153 <screen>lctl set_param {service}.nrs_delay_pct=<replaceable>reg_delay_pct|hp_delay_pct</replaceable>:<replaceable>0-100</replaceable>
2155 <para>For example, to set the percentage of delayed requests on the
2156 ost_io service for high-priority RPCs to 5, run:</para>
2157 <screen>$ lctl set_param ost.OSS.ost_io.nrs_delay_pct=hp_delay_pct:5
2158 ost.OSS.ost_io.nrs_delay_pct=hp_delay_pct:5
2164 <section xml:id="tuning_lockless_IO">
2167 <primary>tuning</primary>
2168 <secondary>lockless I/O</secondary>
2169 </indexterm>Lockless I/O Tunables</title>
2170 <para>The lockless I/O tunable feature allows servers to ask clients to do
2171 lockless I/O (the server does the locking on behalf of clients) for
2172 contended files to avoid lock ping-pong.</para>
2173 <para>The lockless I/O patch introduces these tunables:</para>
2177 <emphasis role="bold">OST-side:</emphasis>
2180 ldlm.namespaces.filter-<replaceable>fsname</replaceable>-*.
2183 <literal>contended_locks</literal>- If the number of lock conflicts in
2184 the scan of granted and waiting queues at contended_locks is exceeded,
2185 the resource is considered to be contended.</para>
2187 <literal>contention_seconds</literal>- The resource keeps itself in a
2188 contended state as set in the parameter.</para>
2190 <literal>max_nolock_bytes</literal>- Server-side locking set only for
2191 requests less than the blocks set in the
2192 <literal>max_nolock_bytes</literal> parameter. If this tunable is
2193 set to zero (0), it disables server-side locking for read/write
2198 <emphasis role="bold">Client-side:</emphasis>
2200 <screen>llite.<replaceable>fsname</replaceable>-*</screen>
2202 <literal>contention_seconds</literal>-
2203 <literal>llite</literal> inode remembers its contended state for the
2204 time specified in this parameter.</para>
2208 <emphasis role="bold">Client-side statistics:</emphasis>
2211 <literal>llite.<replaceable>fsname</replaceable>-*.stats</literal>
2212 parameter has several entries for lockless I/O statistics.</para>
2214 <literal>lockless_read_bytes</literal> and
2215 <literal>lockless_write_bytes</literal>- To count the total bytes read
2216 or written, the client makes its own decisions based on the request
2217 size. The client does not communicate with the server if the request
2218 size is smaller than the
2219 <literal>min_nolock_size</literal>, without acquiring locks by the
2224 <section condition="l29">
2227 <primary>tuning</primary>
2228 <secondary>with lfs ladvise</secondary>
2230 Server-Side Advice and Hinting
2232 <section><title>Overview</title>
2233 <para>Use the <literal>lfs ladvise</literal> command to give file access
2234 advices or hints to servers.</para>
2235 <screen>lfs ladvise [--advice|-a ADVICE ] [--background|-b]
2236 [--start|-s START[kMGT]]
2237 {[--end|-e END[kMGT]] | [--length|-l LENGTH[kMGT]]}
2238 <emphasis>file</emphasis> ...
2241 <informaltable frame="all">
2243 <colspec colname="c1" colwidth="50*"/>
2244 <colspec colname="c2" colwidth="50*"/>
2248 <para><emphasis role="bold">Option</emphasis></para>
2251 <para><emphasis role="bold">Description</emphasis></para>
2258 <para><literal>-a</literal>, <literal>--advice=</literal>
2259 <literal>ADVICE</literal></para>
2262 <para>Give advice or hint of type <literal>ADVICE</literal>.
2263 Advice types are:</para>
2264 <para><literal>willread</literal> to prefetch data into server
2266 <para><literal>dontneed</literal> to cleanup data cache on
2268 <para><literal>lockahead</literal> Request an LDLM extent lock
2269 of the given mode on the given byte range </para>
2270 <para><literal>noexpand</literal> Disable extent lock expansion
2271 behavior for I/O to this file descriptor</para>
2276 <para><literal>-b</literal>, <literal>--background</literal>
2280 <para>Enable the advices to be sent and handled asynchronously.
2286 <para><literal>-s</literal>, <literal>--start=</literal>
2287 <literal>START_OFFSET</literal></para>
2290 <para>File range starts from <literal>START_OFFSET</literal>
2296 <para><literal>-e</literal>, <literal>--end=</literal>
2297 <literal>END_OFFSET</literal></para>
2300 <para>File range ends at (not including)
2301 <literal>END_OFFSET</literal>. This option may not be
2302 specified at the same time as the <literal>-l</literal>
2308 <para><literal>-l</literal>, <literal>--length=</literal>
2309 <literal>LENGTH</literal></para>
2312 <para>File range has length of <literal>LENGTH</literal>.
2313 This option may not be specified at the same time as the
2314 <literal>-e</literal> option.</para>
2319 <para><literal>-m</literal>, <literal>--mode=</literal>
2320 <literal>MODE</literal></para>
2323 <para>Lockahead request mode <literal>{READ,WRITE}</literal>.
2324 Request a lock with this mode.</para>
2331 <para>Typically, <literal>lfs ladvise</literal> forwards the advice to
2332 Lustre servers without guaranteeing when and what servers will react to
2333 the advice. Actions may or may not triggered when the advices are
2334 recieved, depending on the type of the advice, as well as the real-time
2335 decision of the affected server-side components.</para>
2336 <para>A typical usage of ladvise is to enable applications and users with
2337 external knowledge to intervene in server-side cache management. For
2338 example, if a bunch of different clients are doing small random reads of a
2339 file, prefetching pages into OSS cache with big linear reads before the
2340 random IO is a net benefit. Fetching that data into each client cache with
2341 fadvise() may not be, due to much more data being sent to the client.
2344 <literal>ladvise lockahead</literal> is different in that it attempts to
2345 control LDLM locking behavior by explicitly requesting LDLM locks in
2346 advance of use. This does not directly affect caching behavior, instead
2347 it is used in special cases to avoid pathological results (lock exchange)
2348 from the normal LDLM locking behavior.
2351 Note that the <literal>noexpand</literal> advice works on a specific
2352 file descriptor, so using it via lfs has no effect. It must be used
2353 on a particular file descriptor which is used for i/o to have any effect.
2355 <para>The main difference between the Linux <literal>fadvise()</literal>
2356 system call and <literal>lfs ladvise</literal> is that
2357 <literal>fadvise()</literal> is only a client side mechanism that does
2358 not pass the advice to the filesystem, while <literal>ladvise</literal>
2359 can send advices or hints to the Lustre server side.</para>
2361 <section><title>Examples</title>
2362 <para>The following example gives the OST(s) holding the first 1GB of
2363 <literal>/mnt/lustre/file1</literal>a hint that the first 1GB of the
2364 file will be read soon.</para>
2365 <screen>client1$ lfs ladvise -a willread -s 0 -e 1048576000 /mnt/lustre/file1
2367 <para>The following example gives the OST(s) holding the first 1GB of
2368 <literal>/mnt/lustre/file1</literal> a hint that the first 1GB of file
2369 will not be read in the near future, thus the OST(s) could clear the
2370 cache of the file in the memory.</para>
2371 <screen>client1$ lfs ladvise -a dontneed -s 0 -e 1048576000 /mnt/lustre/file1
2373 <para>The following example requests an LDLM read lock on the first
2374 1 MiB of <literal>/mnt/lustre/file1</literal>. This will attempt to
2375 request a lock from the OST holding that region of the file.</para>
2376 <screen>client1$ lfs ladvise -a lockahead -m READ -s 0 -e 1M /mnt/lustre/file1
2378 <para>The following example requests an LDLM write lock on
2379 [3 MiB, 10 MiB] of <literal>/mnt/lustre/file1</literal>. This will
2380 attempt to request a lock from the OST holding that region of the
2382 <screen>client1$ lfs ladvise -a lockahead -m WRITE -s 3M -e 10M /mnt/lustre/file1
2386 <section condition="l29">
2389 <primary>tuning</primary>
2390 <secondary>Large Bulk IO</secondary>
2392 Large Bulk IO (16MB RPC)
2394 <section><title>Overview</title>
2395 <para>Beginning with Lustre 2.9, Lustre is extended to support RPCs up
2396 to 16MB in size. By enabling a larger RPC size, fewer RPCs will be
2397 required to transfer the same amount of data between clients and
2398 servers. With a larger RPC size, the OSS can submit more data to the
2399 underlying disks at once, therefore it can produce larger disk I/Os
2400 to fully utilize the increasing bandwidth of disks.</para>
2401 <para>At client connection time, clients will negotiate with
2402 servers what the maximum RPC size it is possible to use, but the
2403 client can always send RPCs smaller than this maximum.</para>
2404 <para>The parameter <literal>brw_size</literal> is used on the OST
2405 to tell the client the maximum (preferred) IO size. All clients that
2406 talk to this target should never send an RPC greater than this size.
2407 Clients can individually set a smaller RPC size limit via the
2408 <literal>osc.*.max_pages_per_rpc</literal> tunable.
2411 <para>The smallest <literal>brw_size</literal> that can be set for
2412 ZFS OSTs is the <literal>recordsize</literal> of that dataset. This
2413 ensures that the client can always write a full ZFS file block if it
2414 has enough dirty data, and does not otherwise force it to do read-
2415 modify-write operations for every RPC.
2419 <section><title>Usage</title>
2420 <para>In order to enable a larger RPC size,
2421 <literal>brw_size</literal> must be changed to an IO size value up to
2422 16MB. To temporarily change <literal>brw_size</literal>, the
2423 following command should be run on the OSS:</para>
2424 <screen>oss# lctl set_param obdfilter.<replaceable>fsname</replaceable>-OST*.brw_size=16</screen>
2425 <para>To persistently change <literal>brw_size</literal>, the
2426 following command should be run:</para>
2427 <screen>oss# lctl set_param -P obdfilter.<replaceable>fsname</replaceable>-OST*.brw_size=16</screen>
2428 <para>When a client connects to an OST target, it will fetch
2429 <literal>brw_size</literal> from the target and pick the maximum value
2430 of <literal>brw_size</literal> and its local setting for
2431 <literal>max_pages_per_rpc</literal> as the actual RPC size.
2432 Therefore, the <literal>max_pages_per_rpc</literal> on the client side
2433 would have to be set to 16M, or 4096 if the PAGESIZE is 4KB, to enable
2434 a 16MB RPC. To temporarily make the change, the following command
2435 should be run on the client to set
2436 <literal>max_pages_per_rpc</literal>:</para>
2437 <screen>client$ lctl set_param osc.<replaceable>fsname</replaceable>-OST*.max_pages_per_rpc=16M</screen>
2438 <para>To persistently make this change, the following command should
2440 <screen>client$ lctl set_param -P obdfilter.<replaceable>fsname</replaceable>-OST*.osc.max_pages_per_rpc=16M</screen>
2441 <caution><para>The <literal>brw_size</literal> of an OST can be
2442 changed on the fly. However, clients have to be remounted to
2443 renegotiate the new maximum RPC size.</para></caution>
2446 <section xml:id="tuning_IO_small_files">
2449 <primary>tuning</primary>
2450 <secondary>for small files</secondary>
2451 </indexterm>Improving Lustre I/O Performance for Small Files</title>
2452 <para>An environment where an application writes small file chunks from
2453 many clients to a single file can result in poor I/O performance. To
2454 improve the performance of the Lustre file system with small files:</para>
2457 <para>Have the application aggregate writes some amount before
2458 submitting them to the Lustre file system. By default, the Lustre
2459 software enforces POSIX coherency semantics, so it results in lock
2460 ping-pong between client nodes if they are all writing to the same
2461 file at one time.</para>
2462 <para>Using MPI-IO Collective Write functionality in
2463 the Lustre ADIO driver is one way to achieve this in a straight
2464 forward manner if the application is already using MPI-IO.</para>
2467 <para>Have the application do 4kB
2468 <literal>O_DIRECT</literal> sized I/O to the file and disable locking
2469 on the output file. This avoids partial-page IO submissions and, by
2470 disabling locking, you avoid contention between clients.</para>
2473 <para>Have the application write contiguous data.</para>
2476 <para>Add more disks or use SSD disks for the OSTs. This dramatically
2477 improves the IOPS rate. Consider creating larger OSTs rather than many
2478 smaller OSTs due to less overhead (journal, connections, etc).</para>
2481 <para>Use RAID-1+0 OSTs instead of RAID-5/6. There is RAID parity
2482 overhead for writing small chunks of data to disk.</para>
2486 <section xml:id="write_vs_read_performance">
2489 <primary>tuning</primary>
2490 <secondary>write performance</secondary>
2491 </indexterm>Understanding Why Write Performance is Better Than Read
2493 <para>Typically, the performance of write operations on a Lustre cluster is
2494 better than read operations. When doing writes, all clients are sending
2495 write RPCs asynchronously. The RPCs are allocated, and written to disk in
2496 the order they arrive. In many cases, this allows the back-end storage to
2497 aggregate writes efficiently.</para>
2498 <para>In the case of read operations, the reads from clients may come in a
2499 different order and need a lot of seeking to get read from the disk. This
2500 noticeably hampers the read throughput.</para>
2501 <para>Currently, there is no readahead on the OSTs themselves, though the
2502 clients do readahead. If there are lots of clients doing reads it would not
2503 be possible to do any readahead in any case because of memory consumption
2504 (consider that even a single RPC (1 MB) readahead for 1000 clients would
2505 consume 1 GB of RAM).</para>
2506 <para>For file systems that use socklnd (TCP, Ethernet) as interconnect,
2507 there is also additional CPU overhead because the client cannot receive
2508 data without copying it from the network buffers. In the write case, the
2509 client CAN send data without the additional data copy. This means that the
2510 client is more likely to become CPU-bound during reads than writes.</para>
2514 vim:expandtab:shiftwidth=2:tabstop=8:
git://git.whamcloud.com / doc / manual.git / blob