<para><xref linkend="dbdoclet.lndtuning"/></para>
</listitem>
<listitem>
+ <para><xref linkend="dbdoclet.nrstuning"/></para>
+ </listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438272_25884"/></para>
</listitem>
<listitem>
<indexterm><primary>tuning</primary></indexterm>
<indexterm><primary>tuning</primary><secondary>service threads</secondary></indexterm>
Optimizing the Number of Service Threads</title>
- <para>An OSS can have a minimum of 2 service threads and a maximum of 512 service threads. The number of service threads is a function of how much RAM and how many CPUs are on each OSS node (1 thread / 128MB * num_cpus). If the load on the OSS node is high, new service threads will be started in order to process more requests concurrently, up to 4x the initial number of threads (subject to the maximum of 512). For a 2GB 2-CPU system, the default thread count is 32 and the maximum thread count is 128.</para>
+ <para>An OSS can have a minimum of two service threads and a maximum of 512 service threads. The
+ number of service threads is a function of how much RAM and how many CPUs are on each OSS node
+ (1 thread / 128MB * num_cpus). If the load on the OSS node is high, new service threads will
+ be started in order to process more requests concurrently, up to 4x the initial number of
+ threads (subject to the maximum of 512). For a 2GB 2-CPU system, the default thread count is
+ 32 and the maximum thread count is 128.</para>
<para>Increasing the size of the thread pool may help when:</para>
<itemizedlist>
<listitem>
</section>
<section xml:id="dbdoclet.50438272_73839">
<title>
- <indexterm><primary>LNET</primary><secondary>tuning</secondary>
- </indexterm><indexterm><primary>tuning</primary><secondary>LNET</secondary></indexterm>Tuning LNET Parameters</title>
- <para>This section describes LNET tunables. that may be necessary on some systems to improve performance. To test the performance of your Lustre network, see <link xl:href="LNETSelfTest.html#50438223_71556">Chapter 23</link>: <link xl:href="LNETSelfTest.html#50438223_21832">Testing Lustre Network Performance (LNET Self-Test)</link>.</para>
+ <indexterm>
+ <primary>LNet</primary>
+ <secondary>tuning</secondary>
+ </indexterm><indexterm>
+ <primary>tuning</primary>
+ <secondary>LNet</secondary>
+ </indexterm>Tuning LNET Parameters</title>
+ <para>This section describes LNET tunables, the use of which may be necessary on some systems to
+ improve performance. To test the performance of your Lustre network, see <xref linkend='lnetselftest'/>.</para>
<section remap="h3">
<title>Transmit and Receive Buffer Size</title>
<para>The kernel allocates buffers for sending and receiving messages on a network.</para>
<screen>options ksocklnd enable_irq_affinity=0</screen>
<para>By default, this parameter is off. As always, you should test the performance to compare the impact of changing this parameter.</para>
</section>
- <section><title><indexterm><primary>tuning</primary><secondary>Network interface binding</secondary></indexterm>Binding Network Interface Against CPU Partitions</title>
- <para>Luster 2.3 and beyond provide enhanced network interface control. The enhancement means that an administrator can bind an interface to one or more CPU Partitions. Bindings are specified as options to the lnet modules. For more information on specifying module options, see <xref linkend="dbdoclet.50438293_15350"/></para>
-<para>For example, <literal>o2ib0(ib0)[0,1]</literal> will ensure that all messages for <literal>o2ib0</literal> will be handled by LND threads executing on <literal>CPT0</literal> and <literal>CPT1</literal>. An additional example might be: <literal>tcp1(eth0)[0]</literal>. Messages for <literal>tcp1</literal> are handled by threads on <literal>CPT0</literal>.</para>
+ <section condition='l23'><title><indexterm><primary>tuning</primary><secondary>Network interface binding</secondary></indexterm>Binding Network Interface Against CPU Partitions</title>
+ <para>Lustre 2.3 and beyond provide enhanced network interface control. The enhancement means that
+ an administrator can bind an interface to one or more CPU partitions. Bindings are specified
+ as options to the LNET modules. For more information on specifying module options, see <xref
+ linkend="dbdoclet.50438293_15350"/></para>
+ <para>For example, <literal>o2ib0(ib0)[0,1]</literal> will ensure that all messages
+ for <literal>o2ib0</literal> will be handled by LND threads executing on
+ <literal>CPT0</literal> and <literal>CPT1</literal>. An additional example might be:
+ <literal>tcp1(eth0)[0]</literal>. Messages for <literal>tcp1</literal> are handled by
+ threads on <literal>CPT0</literal>.</para>
</section>
<section><title><indexterm><primary>tuning</primary><secondary>Network interface credits</secondary></indexterm>Network Interface Credits</title>
- <para>Network interface (NI) credits are shared across all CPU partitions (CPT). For example, a machine has 4 CPTs and NI credits is 512, then each partition will has 128 credits. If a large number of CPTs exist on the system, LNet will check and validate the NI credits value for each CPT to ensure each CPT has workable number of credits. For example, a machine has 16 CPTs and NI credits is set to 256, then each partition only has 16 credits. 16 NI credits is low and could negatively impact performance. As a result, LNet will automatically make an adjustment to 8*peer_credits (peer_credits is 8 by default), so credits for each partition is still 64.</para>
- <para>Modifying the NI Credit count can be performed by an administrator using <literal>ksoclnd</literal> or <literal>ko2iblnd</literal>. For example:</para>
+ <para>Network interface (NI) credits are shared across all CPU partitions (CPT). For example, a
+ machine has 4 CPTs and NI credits is 512, then each partition will has 128 credits. If a
+ large number of CPTs exist on the system, LNET will check and validate the NI credits value
+ for each CPT to ensure each CPT has workable number of credits. For example, a machine has
+ 16 CPTs and NI credits is set to 256, then each partition only has 16 credits. 16 NI credits
+ is low and could negatively impact performance. As a result, LNET will automatically make an
+ adjustment to 8*peer_credits (peer_credits is 8 by default), so credits for each partition
+ is still 64.</para>
+ <para>Modifying the NI credit count can be performed by an administrator using
+ <literal>ksoclnd</literal> or <literal>ko2iblnd</literal>. For example:</para>
<screen>ksocklnd credits=256</screen>
- <para>applies 256 credits to TCP connections. Applying 256 credits to IB connections can be achieved with:</para>
+ <para>In this example, 256 credits are applied to TCP connections. Applying 256 credits to IB
+ connections can be achieved with:</para>
<screen>ko2iblnd credits=256</screen>
- <note><para>From Lustre 2.3 and beyond, it is possible that LNet may revalidate the NI Credits and the administrator's request do not persist.</para></note>
+ <note><para condition="l23">From Lustre 2.3 and beyond, it is possible that LNET may
+ revalidate the NI Credits so that the administrator's request does not persist.</para></note>
</section>
<section><title><indexterm><primary>tuning</primary><secondary>router buffers</secondary></indexterm>Router Buffers</title>
- <para>Router buffers are shared by all CPU partitions. For a machine with a large number of CPTs, the router buffer number may need to be specified manually for best performance. A low number of router buffers risks starving the CPU Partitions of resources.</para>
- <para>The default setting for router buffers will typically perform well. LNet automatically sets a default value to reduce the likelihood of resource starvation</para>
- <para>An administrator may modify router buffers using the <literal>large_router_buffers</literal> parameter. For example:</para>
- <screen>lnet large_router_buffers=8192</screen>
- <note><para>From Lustre 2.3 and beyond, it is possible that LNet may revalidate the router buffer setting and the administrator's request do not persist.</para></note>
+ <para>Router buffers are shared by all CPU partitions and the client nodes accessing the router.
+ For a machine with a large number of CPTs, the router buffer number may need to be specified
+ manually for best performance. A low number of router buffers risks starving the CPU
+ partitions of resources.</para>
+ <para>The default setting for router buffers will typically perform well. LNET automatically sets a
+ default value to reduce the likelihood of resource starvation</para>
+ <para>An administrator may modify router buffers using the required parameter.</para>
+ <para>There are three different kinds of router buffer parameters:</para>
+ <itemizedlist>
+ <listitem>
+ <para><literal>tiny_router_buffers</literal>: Zero payload buffers which are used for signals and acknowledgements.</para>
+ </listitem>
+ <listitem>
+ <para><literal>small_router_buffers</literal>: 4KB payload buffers for small messages</para>
+ </listitem>
+ <listitem>
+ <para><literal>large_router_buffers</literal>: 1MB maximum payload buffers. This corresponds to the recommended size of 1MB for an RPC.</para>
+ </listitem>
+ </itemizedlist>
+ <para>These can be set using this syntax:</para>
+ <screen>options lnet large_router_buffers=8192</screen>
+ <note><para condition="l23">From Lustre 2.3 and beyond, it is possible that LNET may
+ revalidate the router buffer setting and the administrator's request do not
+ persist.</para></note>
+ <para>The purpose of the router buffers is to create a system of quotas where no single client can overwhelm the router. Router buffers are allotted to the clients.</para>
+ <para>So if increasing the router buffer parameters increases the total number of router buffers
+ available, the parameter <literal>peer_buffer_credit</literal> denotes the number of buffers
+ allotted to each client individually.</para>
+ <para>It can be set in a similar manner.</para>
+ <screen>options lnet peer_buffer_credits=256</screen>
</section>
<section><title><indexterm><primary>tuning</primary><secondary>portal round-robin</secondary></indexterm>Portal Round-Robin</title>
- <para>Portal round-robin defines the policy LNet applies to deliver events and messages to the upper layers. The upper layers are ptlrpc service or LNet selftest.</para>
- <para>If portal round-robin is disabled, LNet will deliver messages to CPTs based on a hash of the source NID. Hence, all messages from a specific peer will be handled by the same CPT. This can reduce data traffic between CPUs. However, for some workloads, this behavior may result in poorly balancing loads across the CPU.</para>
- <para>If portal round-robin is enabled, LNet will round-robin incoming events across all CPTs. This may balance load better across the CPU but can incur a cross CPU overhead.</para>
+ <para>Portal round-robin defines the policy LNET applies to deliver events and messages to the
+ upper layers. The upper layers are PLRPC service or LNET selftest.</para>
+ <para>If portal round-robin is disabled, LNET will deliver messages to CPTs based on a hash of the
+ source NID. Hence, all messages from a specific peer will be handled by the same CPT. This
+ can reduce data traffic between CPUs. However, for some workloads, this behavior may result
+ in poorly balancing loads across the CPU.</para>
+ <para>If portal round-robin is enabled, LNET will round-robin incoming events across all CPTs. This
+ may balance load better across the CPU but can incur a cross CPU overhead.</para>
<para>The current policy can be changed by an administrator with <literal>echo <replaceable>value</replaceable> > /proc/sys/lnet/portal_rotor</literal>. There are four options for <literal><replaceable>value</replaceable></literal>:</para>
<itemizedlist>
<listitem>
<para>LND tuning allows the number of threads per CPU partition to be specified. An administrator can set the threads for both <literal>ko2iblnd</literal> and <literal>ksocklnd</literal> using the <literal>nscheds</literal> parameter. This adjusts the number of threads for each partition, not the overall number of threads on the LND.</para>
<note><para>Lustre 2.3 has greatly decreased the default number of threads for <literal>ko2iblnd</literal> and <literal>ksocklnd</literal> on high-core count machines. The current default values are automatically set and are chosen to work well across a number of typical scenarios.</para></note>
</section>
+ <section xml:id="dbdoclet.nrstuning" condition='l24'>
+ <title><indexterm><primary>tuning</primary><secondary>Network Request Scheduler (NRS) Tuning</secondary></indexterm>Network Request Scheduler (NRS) Tuning</title>
+ <para>The Network Request Scheduler (NRS) allows the administrator to influence the order in which RPCs are handled at servers, on a per-PTLRPC service basis, by providing different policies that can be activated and tuned in order to influence the RPC ordering. The aim of this is to provide for better performance, and possibly discrete performance characteristics using future policies.</para>
+ <para>The NRS policy state of a PTLRPC service can be read and set via the <literal>{service}.nrs_policies</literal> tunable. To read a PTLRPC service's NRS policy state, run:</para>
+ <screen>lctl get_param {service}.nrs_policies</screen>
+ <para>For example, to read the NRS policy state of the ost_io service, run:</para>
+ <screen>$ lctl get_param ost.OSS.ost_io.nrs_policies
+ost.OSS.ost_io.nrs_policies=
+
+regular_requests:
+ - name: fifo
+ state: started
+ fallback: yes
+ queued: 0
+ active: 0
+
+ - name: crrn
+ state: stopped
+ fallback: no
+ queued: 0
+ active: 0
+
+ - name: orr
+ state: stopped
+ fallback: no
+ queued: 0
+ active: 0
+
+ - name: trr
+ state: started
+ fallback: no
+ queued: 2420
+ active: 268
+
+high_priority_requests:
+ - name: fifo
+ state: started
+ fallback: yes
+ queued: 0
+ active: 0
+
+ - name: crrn
+ state: stopped
+ fallback: no
+ queued: 0
+ active: 0
+
+ - name: orr
+ state: stopped
+ fallback: no
+ queued: 0
+ active: 0
+
+ - name: trr
+ state: stopped
+ fallback: no
+ queued: 0
+ active: 0
+ </screen>
+ <para>NRS policy state is shown in either one or two sections, depending on the PTLRPC service being queried. The first section is named <literal>regular_requests</literal> and is available for all PTLRPC services, optionally followed by a second section which is named <literal>high_priority_requests</literal>. This is because some PTLRPC services are able to treat some types of RPCs as higher priority ones, such that they are handled by the server with higher priority compared to other, regular RPC traffic. For PTLRPC services that do not support high-priority RPCs, you will only see the <literal>regular_requests</literal> section.</para>
+ <para>There is a separate instance of each NRS policy on each PTLRPC service for handling regular and high-priority RPCs (if the service supports high-priority RPCs). For each policy instance, the following fields are shown:</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Field</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <literal> name </literal></para>
+ </entry>
+ <entry>
+ <para>The name of the policy.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal> state </literal></para>
+ </entry>
+ <entry>
+ <para>The state of the policy; this can be any of <literal>invalid, stopping, stopped, starting, started</literal>. A fully enabled policy is in the <literal> started</literal> state.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal> fallback </literal></para>
+ </entry>
+ <entry>
+ <para>Whether the policy is acting as a fallback policy or not. A fallback policy is used to handle RPCs that other enabled policies fail to handle, or do not support the handling of. The possible values are <literal>no, yes</literal>. Currently, only the FIFO policy can act as a fallback policy.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal> queued </literal></para>
+ </entry>
+ <entry>
+ <para>The number of RPCs that the policy has waiting to be serviced.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal> active </literal></para>
+ </entry>
+ <entry>
+ <para>The number of RPCs that the policy is currently handling.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <para>To enable an NRS policy on a PTLRPC service run:</para>
+ <screen>lctl set_param {service}.nrs_policies=<replaceable>policy_name</replaceable></screen>
+ <para>This will enable the policy <replaceable>policy_name</replaceable> for both regular and high-priority RPCs (if the PLRPC service supports high-priority RPCs) on the given service. For example, to enable the CRR-N NRS policy for the ldlm_cbd service, run:</para>
+ <screen>$ lctl set_param ldlm.services.ldlm_cbd.nrs_policies=crrn
+ldlm.services.ldlm_cbd.nrs_policies=crrn
+ </screen>
+ <para>For PTLRPC services that support high-priority RPCs, you can also supply an optional <replaceable>reg|hp</replaceable> token, in order to enable an NRS policy for handling only regular or high-priority RPCs on a given PTLRPC service, by running:</para>
+ <screen>lctl set_param {service}.nrs_policies="<replaceable>policy_name</replaceable> <replaceable>reg|hp</replaceable>"</screen>
+ <para>For example, to enable the TRR policy for handling only regular, but not high-priority
+ RPCs on the <literal>ost_io</literal> service, run:</para>
+ <screen>$ lctl set_param ost.OSS.ost_io.nrs_policies="trr reg"
+ost.OSS.ost_io.nrs_policies="trr reg"
+ </screen>
+ <note>
+ <para>When enabling an NRS policy, the policy name must be given in lower-case characters, otherwise the operation will fail with an error message.</para>
+ </note>
+ <section>
+ <title><indexterm><primary>tuning</primary><secondary>Network Request Scheduler (NRS) Tuning</secondary><tertiary>First In, First Out (FIFO) policy</tertiary></indexterm>First In, First Out (FIFO) policy</title>
+ <para>The First In, First Out (FIFO) policy handles RPCs in a service in the same order as they arrive from the LNet layer, so no special processing takes place to modify the RPC handling stream. FIFO is the default policy for all types of RPCs on all PTLRPC services, and is always enabled irrespective of the state of other policies, so that it can be used as a backup policy, in case a more elaborate policy that has been enabled fails to handle an RPC, or does not support handling a given type of RPC.</para>
+ <para> The FIFO policy has no tunables that adjust its behaviour.</para>
+ </section>
+ <section>
+ <title><indexterm><primary>tuning</primary><secondary>Network Request Scheduler (NRS) Tuning</secondary><tertiary>Client Round-Robin over NIDs (CRR-N) policy</tertiary></indexterm>Client Round-Robin over NIDs (CRR-N) policy</title>
+ <para>The Client Round-Robin over NIDs (CRR-N) policy performs batched Round-Robin scheduling of all types of RPCs, with each batch consisting of RPCs originating from the same client node, as identified by its NID. CRR-N aims to provide for better resource utilization across the cluster, and to help shorten completion times of jobs in some cases, by distributing available bandwidth more evenly across all clients.</para>
+ <para>The CRR-N policy can be enabled on all types of PTLRPC services, and has the following tunable that can be used to adjust its behaviour:</para>
+ <itemizedlist>
+ <listitem>
+ <para><literal>{service}.nrs_crrn_quantum</literal></para>
+ <para>The <literal>{service}.nrs_crrn_quantum</literal> tunable determines the maximum allowed size of each batch of RPCs; the unit of measure is in number of RPCs. To read the maximum allowed batch size of a CRR-N policy, run:</para>
+ <screen>lctl get_param {service}.nrs_crrn_quantum</screen>
+ <para>For example, to read the maximum allowed batch size of a CRR-N policy on the ost_io service, run:</para>
+ <screen>$ lctl get_param ost.OSS.ost_io.nrs_crrn_quantum
+ost.OSS.ost_io.nrs_crrn_quantum=reg_quantum:16
+hp_quantum:8
+ </screen>
+ <para>You can see that there is a separate maximum allowed batch size value for regular (<literal>reg_quantum</literal>) and high-priority (<literal>hp_quantum</literal>) RPCs (if the PTLRPC service supports high-priority RPCs).</para>
+ <para>To set the maximum allowed batch size of a CRR-N policy on a given service, run:</para>
+ <screen>lctl set_param {service}.nrs_crrn_quantum=<replaceable>1-65535</replaceable></screen>
+ <para>This will set the maximum allowed batch size on a given service, for both regular and high-priority RPCs (if the PLRPC service supports high-priority RPCs), to the indicated value.</para>
+ <para>For example, to set the maximum allowed batch size on the ldlm_canceld service to 16 RPCs, run:</para>
+ <screen>$ lctl set_param ldlm.services.ldlm_canceld.nrs_crrn_quantum=16
+ldlm.services.ldlm_canceld.nrs_crrn_quantum=16
+ </screen>
+ <para>For PTLRPC services that support high-priority RPCs, you can also specify a different maximum allowed batch size for regular and high-priority RPCs, by running:</para>
+ <screen>$ lctl set_param {service}.nrs_crrn_quantum=<replaceable>reg_quantum|hp_quantum</replaceable>:<replaceable>1-65535</replaceable>"</screen>
+ <para>For example, to set the maximum allowed batch size on the ldlm_canceld service, for high-priority RPCs to 32, run:</para>
+ <screen>$ lctl set_param ldlm.services.ldlm_canceld.nrs_crrn_quantum="hp_quantum:32"
+ldlm.services.ldlm_canceld.nrs_crrn_quantum=hp_quantum:32
+ </screen>
+ <para>By using the last method, you can also set the maximum regular and high-priority RPC batch sizes to different values, in a single command invocation.</para>
+ </listitem>
+ </itemizedlist>
+ </section>
+ <section>
+ <title><indexterm><primary>tuning</primary><secondary>Network Request Scheduler (NRS) Tuning</secondary><tertiary>Object-based Round-Robin (ORR) policy</tertiary></indexterm>Object-based Round-Robin (ORR) policy</title>
+ <para>The Object-based Round-Robin (ORR) policy performs batched Round-Robin scheduling of bulk read write (brw) RPCs, with each batch consisting of RPCs that pertain to the same backend-filesystem object, as identified by its OST FID.</para>
+ <para>The ORR policy is only available for use on the ost_io service. The RPC batches it forms can potentially consist of mixed bulk read and bulk write RPCs. The RPCs in each batch are ordered in an ascending manner, based on either the file offsets, or the physical disk offsets of each RPC (only applicable to bulk read RPCs).</para>
+ <para>The aim of the ORR policy is to provide for increased bulk read throughput in some cases, by ordering bulk read RPCs (and potentially bulk write RPCs), and thus minimizing costly disk seek operations. Performance may also benefit from any resulting improvement in resource utilization, or by taking advantage of better locality of reference between RPCs.</para>
+ <para>The ORR policy has the following tunables that can be used to adjust its behaviour:</para>
+ <itemizedlist>
+ <listitem>
+ <para><literal>ost.OSS.ost_io.nrs_orr_quantum</literal></para>
+ <para>The <literal>ost.OSS.ost_io.nrs_orr_quantum</literal> tunable determines the maximum allowed size of each batch of RPCs; the unit of measure is in number of RPCs. To read the maximum allowed batch size of the ORR policy, run:</para>
+ <screen>$ lctl get_param ost.OSS.ost_io.nrs_orr_quantum
+ost.OSS.ost_io.nrs_orr_quantum=reg_quantum:256
+hp_quantum:16
+ </screen>
+ <para>You can see that there is a separate maximum allowed batch size value for regular (<literal>reg_quantum</literal>) and high-priority (<literal>hp_quantum</literal>) RPCs (if the PTLRPC service supports high-priority RPCs).</para>
+ <para>To set the maximum allowed batch size for the ORR policy, run:</para>
+ <screen>$ lctl set_param ost.OSS.ost_io.nrs_orr_quantum=<replaceable>1-65535</replaceable></screen>
+ <para>This will set the maximum allowed batch size for both regular and high-priority RPCs, to the indicated value.</para>
+ <para>You can also specify a different maximum allowed batch size for regular and high-priority RPCs, by running:</para>
+ <screen>$ lctl set_param ost.OSS.ost_io.nrs_orr_quantum=<replaceable>reg_quantum|hp_quantum</replaceable>:<replaceable>1-65535</replaceable></screen>
+ <para>For example, to set the maximum allowed batch size for regular RPCs to 128, run:</para>
+ <screen>$ lctl set_param ost.OSS.ost_io.nrs_orr_quantum=reg_quantum:128
+ost.OSS.ost_io.nrs_orr_quantum=reg_quantum:128
+ </screen>
+ <para>By using the last method, you can also set the maximum regular and high-priority RPC batch sizes to different values, in a single command invocation.</para>
+ </listitem>
+ <listitem>
+ <para><literal>ost.OSS.ost_io.nrs_orr_offset_type</literal></para>
+ <para>The <literal>ost.OSS.ost_io.nrs_orr_offset_type</literal> tunable determines whether the ORR policy orders RPCs within each batch based on logical file offsets or physical disk offsets. To read the offset type value for the ORR policy, run:</para>
+ <screen>$ lctl get_param ost.OSS.ost_io.nrs_orr_offset_type
+ost.OSS.ost_io.nrs_orr_offset_type=reg_offset_type:physical
+hp_offset_type:logical
+ </screen>
+ <para>You can see that there is a separate offset type value for regular (<literal>reg_offset_type</literal>) and high-priority (<literal>hp_offset_type</literal>) RPCs.</para>
+ <para>To set the ordering type for the ORR policy, run:</para>
+ <screen>$ lctl set_param ost.OSS.ost_io.nrs_orr_offset_type=<replaceable>physical|logical</replaceable></screen>
+ <para>This will set the offset type for both regular and high-priority RPCs, to the indicated value.</para>
+ <para>You can also specify a different offset type for regular and high-priority RPCs, by running:</para>
+ <screen>$ lctl set_param ost.OSS.ost_io.nrs_orr_offset_type=<replaceable>reg_offset_type|hp_offset_type</replaceable>:<replaceable>physical|logical</replaceable></screen>
+ <para>For example, to set the offset type for high-priority RPCs to physical disk offsets, run:</para>
+ <screen>$ lctl set_param ost.OSS.ost_io.nrs_orr_offset_type=hp_offset_type:physical
+ost.OSS.ost_io.nrs_orr_offset_type=hp_offset_type:physical
+ </screen>
+ <para>By using the last method, you can also set offset type for regular and high-priority RPCs to different values, in a single command invocation.</para>
+ <note><para>Irrespective of the value of this tunable, only logical offsets can, and are used for ordering bulk write RPCs.</para></note>
+ </listitem>
+ <listitem>
+ <para><literal>ost.OSS.ost_io.nrs_orr_supported</literal></para>
+ <para>The <literal>ost.OSS.ost_io.nrs_orr_supported</literal> tunable determines the type of RPCs that the ORR policy will handle. To read the types of supported RPCs by the ORR policy, run:</para>
+ <screen>$ lctl get_param ost.OSS.ost_io.nrs_orr_supported
+ost.OSS.ost_io.nrs_orr_supported=reg_supported:reads
+hp_supported=reads_and_writes
+ </screen>
+ <para>You can see that there is a separate supported 'RPC types' value for regular (<literal>reg_supported</literal>) and high-priority (<literal>hp_supported</literal>) RPCs.</para>
+ <para>To set the supported RPC types for the ORR policy, run:</para>
+ <screen>$ lctl set_param ost.OSS.ost_io.nrs_orr_supported=<replaceable>reads|writes|reads_and_writes</replaceable></screen>
+ <para>This will set the supported RPC types for both regular and high-priority RPCs, to the indicated value.</para>
+ <para>You can also specify a different supported 'RPC types' value for regular and high-priority RPCs, by running:</para>
+ <screen>$ lctl set_param ost.OSS.ost_io.nrs_orr_supported=<replaceable>reg_supported|hp_supported</replaceable>:<replaceable>reads|writes|reads_and_writes</replaceable></screen>
+ <para>For example, to set the supported RPC types to bulk read and bulk write RPCs for regular requests, run:</para>
+ <screen>$ lctl set_param ost.OSS.ost_io.nrs_orr_supported=reg_supported:reads_and_writes
+ost.OSS.ost_io.nrs_orr_supported=reg_supported:reads_and_writes
+ </screen>
+ <para>By using the last method, you can also set the supported RPC types for regular and high-priority RPC to different values, in a single command invocation.</para>
+ </listitem>
+ </itemizedlist>
+ </section>
+ <section>
+ <title><indexterm><primary>tuning</primary><secondary>Network Request Scheduler (NRS) Tuning</secondary><tertiary>Target-based Round-Robin (TRR) policy</tertiary></indexterm>Target-based Round-Robin (TRR) policy</title>
+ <para>The Target-based Round-Robin (TRR) policy performs batched Round-Robin scheduling of brw RPCs, with each batch consisting of RPCs that pertain to the same OST, as identified by its OST index.</para>
+ <para>The TRR policy is identical to the Object-based Round-Robin (ORR) policy, apart from using the brw RPC's target OST index instead of the backend-fs object's OST FID, for determining the RPC scheduling order. The goals of TRR are effectively the same as for ORR, and it uses the following tunables to adjust its behaviour:</para>
+ <itemizedlist>
+ <listitem>
+ <para><literal>ost.OSS.ost_io.nrs_trr_quantum</literal></para>
+ <para>The purpose of this tunable is exactly the same as for the <literal>ost.OSS.ost_io.nrs_orr_quantum</literal> tunable for the ORR policy, and you can use it in exactly the same way.</para>
+ </listitem>
+ <listitem>
+ <para><literal>ost.OSS.ost_io.nrs_trr_offset_type</literal></para>
+ <para>The purpose of this tunable is exactly the same as for the <literal>ost.OSS.ost_io.nrs_orr_offset_type</literal> tunable for the ORR policy, and you can use it in exactly the same way.</para>
+ </listitem>
+ <listitem>
+ <para><literal>ost.OSS.ost_io.nrs_trr_supported</literal></para>
+ <para>The purpose of this tunable is exactly the same as for the <literal>ost.OSS.ost_io.nrs_orr_supported</literal> tunable for the ORR policy, and you can use it in exactly the sme way.</para>
+ </listitem>
+ </itemizedlist>
+ </section>
+ </section>
<section xml:id="dbdoclet.50438272_25884">
<title><indexterm><primary>tuning</primary><secondary>lockless I/O</secondary></indexterm>Lockless I/O Tunables</title>
<para>The lockless I/O tunable feature allows servers to ask clients to do lockless I/O (liblustre-style where the server does the locking) on contended files.</para>
git://git.whamcloud.com / doc / manual.git / blobdiff