1 <?xml version="1.0" encoding="UTF-8"?>
2 <chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='lustreproc'>
4 <title xml:id='lustreproc.title'>LustreProc</title>
6 <para>The /proc file system acts as an interface to internal data structures in the kernel. The /proc variables can be used to control aspects of Lustre performance and provide information.</para>
7 <para>This chapter describes Lustre /proc entries and includes the following sections:</para>
8 <itemizedlist><listitem>
9 <para><xref linkend="dbdoclet.50438271_90999"/></para>
13 <para><xref linkend="dbdoclet.50438271_78950"/></para>
17 <para><xref linkend="dbdoclet.50438271_83523"/></para>
22 <section xml:id="dbdoclet.50438271_90999">
23 <title>31.1 Proc Entries for Lustre</title>
24 <para>This section describes /proc entries for Lustre.</para>
26 <title>31.1.1 Locating Lustre <anchor xml:id="dbdoclet.50438271_marker-1296151" xreflabel=""/>File Systems and Servers</title>
27 <para>Use the proc files on the MGS to locate the following:</para>
28 <itemizedlist><listitem>
29 <para> All known file systems</para>
33 <screen># cat /proc/fs/lustre/mgs/MGS/filesystems
37 <itemizedlist><listitem>
38 <para> The server names participating in a file system (for each file system that has at least one server running)</para>
42 <screen># cat /proc/fs/lustre/mgs/MGS/live/spfs
48 <para>All servers are named according to this convention: <fsname>-<MDT|OST><XXXX> This can be shown for live servers under /proc/fs/lustre/devices:</para>
49 <screen># cat /proc/fs/lustre/devices
51 1 UP mgc MGC192.168.10.34@tcp 1f45bb57-d9be-2ddb-c0b0-5431a49226705
52 2 UP mdt MDS MDS_uuid 3
53 3 UP lov lustre-mdtlov lustre-mdtlov_UUID 4
54 4 UP mds lustre-MDT0000 lustre-MDT0000_UUID 7
55 5 UP osc lustre-OST0000-osc lustre-mdtlov_UUID 5
56 6 UP osc lustre-OST0001-osc lustre-mdtlov_UUID 5
57 7 UP lov lustre-clilov-ce63ca00 08ac6584-6c4a-3536-2c6d-b36cf9cbdaa04
58 8 UP mdc lustre-MDT0000-mdc-ce63ca00 08ac6584-6c4a-3536-2c6d-b36cf9cbdaa05
59 9 UP osc lustre-OST0000-osc-ce63ca00 08ac6584-6c4a-3536-2c6d-b36cf9cbdaa05
60 10 UP osc lustre-OST0001-osc-ce63ca00 08ac6584-6c4a-3536-2c6d-b36cf9cbdaa05
62 <para>Or from the device label at any time:</para>
63 <screen># e2label /dev/sda
68 <title>31.1.2 Lustre <anchor xml:id="dbdoclet.50438271_marker-1296153" xreflabel=""/>Timeouts</title>
69 <para>Lustre uses two types of timeouts.</para>
70 <itemizedlist><listitem>
71 <para>LND timeouts that ensure point-to-point communications complete in finite time in the presence of failures. These timeouts are logged with the S_LND flag set. They may <emphasis>not</emphasis> be printed as console messages, so you should check the Lustre log for D_NETERROR messages, or enable printing of D_NETERROR messages to the console (echo + neterror > /proc/sys/lnet/printk).</para>
75 <para>Congested routers can be a source of spurious LND timeouts. To avoid this, increase the number of LNET router buffers to reduce back-pressure and/or increase LND timeouts on all nodes on all connected networks. You should also consider increasing the total number of LNET router nodes in the system so that the aggregate router bandwidth matches the aggregate server bandwidth.</para>
76 <itemizedlist><listitem>
77 <para>Lustre timeouts that ensure Lustre RPCs complete in finite time in the presence of failures. These timeouts should <emphasis>always</emphasis> be printed as console messages. If Lustre timeouts are not accompanied by LNET timeouts, then you need to increase the lustre timeout on both servers and clients.</para>
81 <para>Specific Lustre timeouts are described below.</para>
82 <para><emphasis role="bold">/proc/sys/lustre/timeout</emphasis></para>
83 <para>This is the time period that a client waits for a server to complete an RPC (default is 100s). Servers wait half of this time for a normal client RPC to complete and a quarter of this time for a single bulk request (read or write of up to 1 MB) to complete. The client pings recoverable targets (MDS and OSTs) at one quarter of the timeout, and the server waits one and a half times the timeout before evicting a client for being "stale."</para>
84 <note><para>Lustre sends periodic 'PING' messages to servers with which it had no communication for a specified period of time. Any network activity on the file system that triggers network traffic toward servers also works as a health check.</para></note>
85 <para><emphasis role="bold">/proc/sys/lustre/ldlm_timeout</emphasis></para>
86 <para>This is the time period for which a server will wait for a client to reply to an initial AST (lock cancellation request) where default is 20s for an OST and 6s for an MDS. If the client replies to the AST, the server will give it a normal timeout (half of the client timeout) to flush any dirty data and release the lock.</para>
87 <para><emphasis role="bold">/proc/sys/lustre/fail_loc</emphasis></para>
88 <para>This is the internal debugging failure hook.</para>
89 <para>See lustre/include/linux/obd_support.h for the definitions of individual failure locations. The default value is 0 (zero).</para>
90 <screen>sysctl -w lustre.fail_loc=0x80000122 # drop a single reply
92 <para><emphasis role="bold">/proc/sys/lustre/dump_on_timeout</emphasis></para>
93 <para>This triggers dumps of the Lustre debug log when timeouts occur. The default value is 0 (zero).</para>
94 <para><emphasis role="bold">/proc/sys/lustre/dump_on_eviction</emphasis></para>
95 <para>This triggers dumps of the Lustre debug log when an eviction occurs. The default value is 0 (zero). By default, debug logs are dumped to the /tmp folder; this location can be changed via /proc.</para>
98 <title>31.1.3 Adaptive <anchor xml:id="dbdoclet.50438271_marker-1293380" xreflabel=""/>Timeouts</title>
99 <para>Lustre offers an adaptive mechanism to set RPC timeouts. The adaptive timeouts feature (enabled, by default) causes servers to track actual RPC completion times, and to report estimated completion times for future RPCs back to clients. The clients use these estimates to set their future RPC timeout values. If server request processing slows down for any reason, the RPC completion estimates increase, and the clients allow more time for RPC completion.</para>
100 <para>If RPCs queued on the server approach their timeouts, then the server sends an early reply to the client, telling the client to allow more time. In this manner, clients avoid RPC timeouts and disconnect/reconnect cycles. Conversely, as a server speeds up, RPC timeout values decrease, allowing faster detection of non-responsive servers and faster attempts to reconnect to a server's failover partner.</para>
101 <para>In previous Lustre versions, the static obd_timeout (/proc/sys/lustre/timeout) value was used as the maximum completion time for all RPCs; this value also affected the client-server ping interval and initial recovery timer. Now, with adaptive timeouts, obd_timeout is only used for the ping interval and initial recovery estimate. When a client reconnects during recovery, the server uses the client's timeout value to reset the recovery wait period; i.e., the server learns how long the client had been willing to wait, and takes this into account when adjusting the recovery period.</para>
103 <title>31.1.3.1 Configuring <anchor xml:id="dbdoclet.50438271_marker-1293381" xreflabel=""/>Adaptive Timeouts</title>
104 <para>One of the goals of adaptive timeouts is to relieve users from having to tune the obd_timeout value. In general, obd_timeout should no longer need to be changed. However, there are several parameters related to adaptive timeouts that users can set. In most situations, the default values should be used.</para>
105 <para>The following parameters can be set persistently system-wide using lctl conf_param on the MGS. For example, lctl conf_param work1.sys.at_max=1500 sets the at_max value for all servers and clients using the work1 file system.</para>
106 <note><para>Nodes using multiple Lustre file systems must use the same at_* values for all file systems.)</para></note>
107 <informaltable frame="all">
109 <colspec colname="c1" colwidth="50*"/>
110 <colspec colname="c2" colwidth="50*"/>
113 <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
114 <entry><para><emphasis role="bold">Description</emphasis></para></entry>
119 <entry><para> <emphasis role="bold">at_min</emphasis></para></entry>
120 <entry><para> Sets the minimum adaptive timeout (in seconds). Default value is 0. The at_min parameter is the minimum processing time that a server will report. Clients base their timeouts on this value, but they do not use this value directly. If you experience cases in which, for unknown reasons, the adaptive timeout value is too short and clients time out their RPCs (usually due to temporary network outages), then you can increase the at_min value to compensate for this. Ideally, users should leave at_min set to its default.</para></entry>
123 <entry><para> <emphasis role="bold">at_max</emphasis></para></entry>
124 <entry><para> Sets the maximum adaptive timeout (in seconds). The at_max parameter is an upper-limit on the service time estimate, and is used as a 'failsafe' in case of rogue/bad/buggy code that would lead to never-ending estimate increases. If at_max is reached, an RPC request is considered 'broken' and should time out.</para><para>Setting at_max to 0 causes adaptive timeouts to be disabled and the old fixed-timeout method (obd_timeout) to be used. This is the default value in Lustre 1.6.5.</para><para> </para>
126 <note><para>It is possible that slow hardware might validly cause the service estimate to increase beyond the default value of at_max. In this case, you should increase at_max to the maximum time you are willing to wait for an RPC completion.</para></note></entry>
129 <entry><para> <emphasis role="bold">at_history</emphasis></para></entry>
130 <entry><para> Sets a time period (in seconds) within which adaptive timeouts remember the slowest event that occurred. Default value is 600.</para></entry>
133 <entry><para> <emphasis role="bold">at_early_margin</emphasis></para></entry>
134 <entry><para> Sets how far before the deadline Lustre sends an early reply. Default value is 5<footnote><para>This default was chosen as a reasonable time in which to send a reply from the point at which it was sent.</para></footnote>.</para></entry>
138 <entry><para> <emphasis role="bold">at_extra</emphasis></para></entry>
139 <entry><para> Sets the incremental amount of time that a server asks for, with each early reply. The server does not know how much time the RPC will take, so it asks for a fixed value. Default value is 30<footnote><para>This default was chosen as a balance between sending too many early replies for the same RPC and overestimating the actual completion time</para></footnote>. When a server finds a queued request about to time out (and needs to send an early reply out), the server adds the at_extra value. If the time expires, the Lustre client enters recovery status and reconnects to restore it to normal status.</para><para>If you see multiple early replies for the same RPC asking for multiple 30-second increases, change the at_extra value to a larger number to cut down on early replies sent and, therefore, network load.</para></entry>
142 <entry><para> <emphasis role="bold">ldlm_enqueue_min</emphasis></para></entry>
143 <entry><para> Sets the minimum lock enqueue time. Default value is 100. The ldlm_enqueue time is the maximum of the measured enqueue estimate (influenced by at_min and at_max parameters), multiplied by a weighting factor, and the ldlm_enqueue_min setting. LDLM lock enqueues were based on the obd_timeout value; now they have a dedicated minimum value. Lock enqueues increase as the measured enqueue times increase (similar to adaptive timeouts).</para></entry>
148 <para>Adaptive timeouts are enabled, by default. To disable adaptive timeouts, at run time, set at_max to 0. On the MGS, run:</para>
149 <screen>$ lctl conf_param <fsname>.sys.at_max=0
151 <note><para>Changing adaptive timeouts status at runtime may cause transient timeout, reconnect, recovery, etc.</para></note>
154 <title>31.1.3.2 Interpreting <anchor xml:id="dbdoclet.50438271_marker-1293383" xreflabel=""/>Adaptive Timeouts Information</title>
155 <para>Adaptive timeouts information can be read from /proc/fs/lustre/*/timeouts files (for each service and client) or with the lctl command.</para>
156 <para>This is an example from the /proc/fs/lustre/*/timeouts files:</para>
157 <screen>cfs21:~# cat /proc/fs/lustre/ost/OSS/ost_io/timeouts
159 <para>This is an example using the lctl command:</para>
160 <screen>$ lctl get_param -n ost.*.ost_io.timeouts
162 <para>This is the sample output:</para>
163 <screen>service : cur 33 worst 34 (at 1193427052, 0d0h26m40s ago) 1 1 33 2
165 <para>The ost_io service on this node is currently reporting an estimate of 33 seconds. The worst RPC service time was 34 seconds, and it happened 26 minutes ago.</para>
166 <para>The output also provides a history of service times. In the example, there are 4 "bins" of adaptive_timeout_history, with the maximum RPC time in each bin reported. In 0-150 seconds, the maximum RPC time was 1, with the same result in 150-300 seconds. From 300-450 seconds, the worst (maximum) RPC time was 33 seconds, and from 450-600s the worst time was 2 seconds. The current estimated service time is the maximum value of the 4 bins (33 seconds in this example).</para>
167 <para>Service times (as reported by the servers) are also tracked in the client OBDs:</para>
168 <screen>cfs21:# lctl get_param osc.*.timeouts
169 last reply : 1193428639, 0d0h00m00s ago
170 network : cur 1 worst 2 (at 1193427053, 0d0h26m26s ago) 1 1 1\
172 portal 6 : cur 33 worst 34 (at 1193427052, 0d0h26m27s ago) 33 33 33\
174 portal 28 : cur 1 worst 1 (at 1193426141, 0d0h41m38s ago) 1 1 1\
176 portal 7 : cur 1 worst 1 (at 1193426141, 0d0h41m38s ago) 1 0 1\
178 portal 17 : cur 1 worst 1 (at 1193426177, 0d0h41m02s ago) 1 0 0\
181 <para>In this case, RPCs to portal 6, the OST_IO_PORTAL (see lustre/include/lustre/lustre_idl.h), shows the history of what the ost_io portal has reported as the service estimate.</para>
182 <para>Server statistic files also show the range of estimates in the normal min/max/sum/sumsq manner.</para>
183 <screen>cfs21:~# lctl get_param mdt.*.mdt.stats
185 req_timeout 6 samples [sec] 1 10 15 105
191 <title>31.1.4 LNET <anchor xml:id="dbdoclet.50438271_marker-1296164" xreflabel=""/>Information</title>
192 <para>This section describes /proc entries for LNET information.</para>
193 <para><emphasis role="bold">/proc/sys/lnet/peers</emphasis></para>
194 <para>Shows all NIDs known to this node and also gives information on the queue state.</para>
195 <screen># cat /proc/sys/lnet/peers
200 192.168.10.35@tcp 1 ~rtr 8 8 \
202 192.168.10.36@tcp 1 ~rtr 8 8 \
204 192.168.10.37@tcp 1 ~rtr 8 8 \
207 <para>The fields are explained below:</para>
208 <informaltable frame="all">
210 <colspec colname="c1" colwidth="50*"/>
211 <colspec colname="c2" colwidth="50*"/>
214 <entry><para><emphasis role="bold">Field</emphasis></para></entry>
215 <entry><para><emphasis role="bold">Description</emphasis></para></entry>
220 <entry><para> <emphasis role="bold">refs</emphasis></para></entry>
221 <entry><para> A reference count (principally used for debugging)</para></entry>
224 <entry><para> <emphasis role="bold">state</emphasis></para></entry>
225 <entry><para> Only valid to refer to routers. Possible values:</para><itemizedlist><listitem>
226 <para> ~ rtr (indicates this node is not a router)</para>
229 <para> up/down (indicates this node is a router)</para>
232 <para> auto_fail must be enabled</para>
234 </itemizedlist></entry>
237 <entry><para> <emphasis role="bold">max</emphasis></para></entry>
238 <entry><para> Maximum number of concurrent sends from this peer</para></entry>
241 <entry><para> <emphasis role="bold">rtr</emphasis></para></entry>
242 <entry><para> Routing buffer credits.</para></entry>
245 <entry><para> <emphasis role="bold">min</emphasis></para></entry>
246 <entry><para> Minimum routing buffer credits seen.</para></entry>
249 <entry><para> <emphasis role="bold">tx</emphasis></para></entry>
250 <entry><para> Send credits.</para></entry>
253 <entry><para> <emphasis role="bold">min</emphasis></para></entry>
254 <entry><para> Minimum send credits seen.</para></entry>
257 <entry><para> <emphasis role="bold">queue</emphasis></para></entry>
258 <entry><para> Total bytes in active/queued sends.</para></entry>
263 <para>Credits work like a semaphore. At start they are initialized to allow a certain number of operations (8 in this example). LNET keeps a track of the minimum value so that you can see how congested a resource was.</para>
264 <para>If rtr/tx is less than max, there are operations in progress. The number of operations is equal to rtr or tx subtracted from max.</para>
265 <para>If rtr/tx is greater that max, there are operations blocking.</para>
266 <para>LNET also limits concurrent sends and router buffers allocated to a single peer so that no peer can occupy all these resources.</para>
267 <para><emphasis role="bold">/proc/sys/lnet/nis</emphasis></para>
268 <screen># cat /proc/sys/lnet/nis
273 192.168.10.34@tcp 4 8 256 \
276 <para>Shows the current queue health on this node. The fields are explained below:</para>
277 <informaltable frame="all">
279 <colspec colname="c1" colwidth="50*"/>
280 <colspec colname="c2" colwidth="50*"/>
283 <entry><para><emphasis role="bold">Field</emphasis></para></entry>
284 <entry><para><emphasis role="bold">Description</emphasis></para></entry>
289 <entry><para> <emphasis role="bold">nid</emphasis></para></entry>
290 <entry><para> Network interface</para></entry>
293 <entry><para> <emphasis role="bold">refs</emphasis></para></entry>
294 <entry><para> Internal reference counter</para></entry>
297 <entry><para> <emphasis role="bold">peer</emphasis></para></entry>
298 <entry><para> Number of peer-to-peer send credits on this NID. Credits are used to size buffer pools</para></entry>
301 <entry><para> <emphasis role="bold">max</emphasis></para></entry>
302 <entry><para> Total number of send credits on this NID.</para></entry>
305 <entry><para> <emphasis role="bold">tx</emphasis></para></entry>
306 <entry><para> Current number of send credits available on this NID.</para></entry>
309 <entry><para> <emphasis role="bold">min</emphasis></para></entry>
310 <entry><para> Lowest number of send credits available on this NID.</para></entry>
313 <entry><para> <emphasis role="bold">queue</emphasis></para></entry>
314 <entry><para> Total bytes in active/queued sends.</para></entry>
319 <para>Subtracting max - tx yields the number of sends currently active. A large or increasing number of active sends may indicate a problem.</para>
320 <screen># cat /proc/sys/lnet/nis
325 10.67.73.173@tcp 4 8 256 \
330 <title>31.1.5 Free Space <anchor xml:id="dbdoclet.50438271_marker-1296165" xreflabel=""/>Distribution</title>
331 <para>Free-space stripe weighting, as set, gives a priority of "0" to free space (versus trying to place the stripes "widely" -- nicely distributed across OSSs and OSTs to maximize network balancing). To adjust this priority (as a percentage), use the qos_prio_free proc tunable:</para>
332 <screen>$ cat /proc/fs/lustre/lov/<fsname>-mdtlov/qos_prio_free
334 <para>Currently, the default is 90%. You can permanently set this value by running this command on the MGS:</para>
335 <screen>$ lctl conf_param <fsname>-MDT0000.lov.qos_prio_free=90
337 <para>Setting the priority to 100% means that OSS distribution does not count in the weighting, but the stripe assignment is still done via weighting. If OST 2 has twice as much free space as OST 1, it is twice as likely to be used, but it is NOT guaranteed to be used.</para>
338 <para>Also note that free-space stripe weighting does not activate until two OSTs are imbalanced by more than 20%. Until then, a faster round-robin stripe allocator is used. (The new round-robin order also maximizes network balancing.)</para>
340 <title>31.1.5.1 Managing Stripe Allocation</title>
341 <para>The MDS uses two methods to manage stripe allocation and determine which OSTs to use for file object storage:</para>
342 <itemizedlist><listitem>
343 <para><emphasis role="bold">QOS</emphasis></para>
347 <para>Quality of Service (QOS) considers an OST's available blocks, speed, and the number of existing objects, etc. Using these criteria, the MDS selects OSTs with more free space more often than OSTs with less free space.</para>
348 <itemizedlist><listitem>
349 <para><emphasis role="bold">RR</emphasis></para>
353 <para>Round-Robin (RR) allocates objects evenly across all OSTs. The RR stripe allocator is faster than QOS, and used often because it distributes space usage/load best in most situations, maximizing network balancing and improving performance.</para>
354 <para>Whether QOS or RR is used depends on the setting of the qos_threshold_rr proc tunable. The qos_threshold_rr variable specifies a percentage threshold where the use of QOS or RR becomes more/less likely. The qos_threshold_rr tunable can be set as an integer, from 0 to 100, and results in this stripe allocation behavior:</para>
355 <itemizedlist><listitem>
356 <para> If qos_threshold_rr is set to 0, then QOS is always used</para>
360 <para> If qos_threshold_rr is set to 100, then RR is always used</para>
364 <para> The larger the qos_threshold_rr setting, the greater the possibility that RR is used instead of QOS</para>
371 <section xml:id="dbdoclet.50438271_78950">
372 <title>31.2 Lustre I/O <anchor xml:id="dbdoclet.50438271_marker-1290508" xreflabel=""/>Tunables</title>
373 <para>The section describes I/O tunables.</para>
374 <para><emphasis role="bold">/proc/fs/lustre/llite/<fsname>-<uid>/max_cache_mb</emphasis></para>
375 <screen># cat /proc/fs/lustre/llite/lustre-ce63ca00/max_cached_mb 128
377 <para>This tunable is the maximum amount of inactive data cached by the client (default is 3/4 of RAM).</para>
379 <title>31.2.1 Client I/O RPC<anchor xml:id="dbdoclet.50438271_marker-1290514" xreflabel=""/> Stream Tunables</title>
380 <para>The Lustre engine always attempts to pack an optimal amount of data into each I/O RPC and attempts to keep a consistent number of issued RPCs in progress at a time. Lustre exposes several tuning variables to adjust behavior according to network conditions and cluster size. Each OSC has its own tree of these tunables. For example:</para>
381 <screen>$ ls -d /proc/fs/lustre/osc/OSC_client_ost1_MNT_client_2 /localhost
382 /proc/fs/lustre/osc/OSC_uml0_ost1_MNT_localhost
383 /proc/fs/lustre/osc/OSC_uml0_ost2_MNT_localhost
384 /proc/fs/lustre/osc/OSC_uml0_ost3_MNT_localhost
385 $ ls /proc/fs/lustre/osc/OSC_uml0_ost1_MNT_localhost
386 blocksizefilesfree max_dirty_mb ost_server_uuid stats
388 <para>... and so on.</para>
389 <para>RPC stream tunables are described below.</para>
390 <para><emphasis role="bold">/proc/fs/lustre/osc/<object name>/max_dirty_mb</emphasis></para>
391 <para>This tunable controls how many MBs of dirty data can be written and queued up in the OSC. POSIX file writes that are cached contribute to this count. When the limit is reached, additional writes stall until previously-cached writes are written to the server. This may be changed by writing a single ASCII integer to the file. Only values between 0 and 512 are allowable. If 0 is given, no writes are cached. Performance suffers noticeably unless you use large writes (1 MB or more).</para>
392 <para><emphasis role="bold">/proc/fs/lustre/osc/<object name>/cur_dirty_bytes</emphasis></para>
393 <para>This tunable is a read-only value that returns the current amount of bytes written and cached on this OSC.</para>
394 <para><emphasis role="bold">/proc/fs/lustre/osc/<object name>/max_pages_per_rpc</emphasis></para>
395 <para>This tunable is the maximum number of pages that will undergo I/O in a single RPC to the OST. The minimum is a single page and the maximum for this setting is platform dependent (256 for i386/x86_64, possibly less for ia64/PPC with larger PAGE_SIZE), though generally amounts to a total of 1 MB in the RPC.</para>
396 <para><emphasis role="bold">/proc/fs/lustre/osc/<object name>/max_rpcs_in_flight</emphasis></para>
397 <para>This tunable is the maximum number of concurrent RPCs in flight from an OSC to its OST. If the OSC tries to initiate an RPC but finds that it already has the same number of RPCs outstanding, it will wait to issue further RPCs until some complete. The minimum setting is 1 and maximum setting is 32. If you are looking to improve small file I/O performance, increase the max_rpcs_in_flight value.</para>
398 <para>To maximize performance, the value for max_dirty_mb is recommended to be 4 * max_pages_per_rpc * max_rpcs_in_flight.</para>
399 <note><para>The <object name> varies depending on the specific Lustre configuration. For <object name> examples, refer to the sample command output.</para></note>
402 <title>31.2.2 Watching the <anchor xml:id="dbdoclet.50438271_marker-1290535" xreflabel=""/>Client RPC Stream</title>
403 <para>The same directory contains a rpc_stats file with a histogram showing the composition of previous RPCs. The histogram can be cleared by writing any value into the rpc_stats file.</para>
404 <screen># cat /proc/fs/lustre/osc/spfs-OST0000-osc-c45f9c00/rpc_stats
405 snapshot_time: 1174867307.156604 (secs.\
407 read RPCs in flight: 0
408 write RPCs in flight: 0
409 pending write pages: 0
410 pending read pages: 0
412 pages per rpc rpcs % cum % | rpcs \
418 rpcs in flight rpcs % cum % | rpcs \
424 offset rpcs % cum % | rpcs \
430 <informaltable frame="all">
432 <colspec colname="c1" colwidth="50*"/>
433 <colspec colname="c2" colwidth="50*"/>
436 <entry><para><emphasis role="bold">Field</emphasis></para></entry>
437 <entry><para><emphasis role="bold">Description</emphasis></para></entry>
442 <entry><para> <emphasis role="bold">{read,write} RPCs in flight</emphasis></para></entry>
443 <entry><para> Number of read/write RPCs issued by the OSC, but not complete at the time of the snapshot. This value should always be less than or equal to max_rpcs_in_flight.</para></entry>
446 <entry><para> <emphasis role="bold">pending {read,write} pages</emphasis></para></entry>
447 <entry><para> Number of pending read/write pages that have been queued for I/O in the OSC.</para></entry>
450 <entry><para> <emphasis role="bold">pages per RPC</emphasis></para></entry>
451 <entry><para> When an RPC is sent, the number of pages it consists of is recorded (in order). A single page RPC increments the 0: row.</para></entry>
454 <entry><para> <emphasis role="bold">RPCs in flight</emphasis></para></entry>
455 <entry><para> When an RPC is sent, the number of other RPCs that are pending is recorded. When the first RPC is sent, the 0: row is incremented. If the first RPC is sent while another is pending, the 1: row is incremented and so on. As each RPC *completes*, the number of pending RPCs is not tabulated.</para><para>This table is a good way to visualize the concurrency of the RPC stream. Ideally, you will see a large clump around the max_rpcs_in_flight value, which shows that the network is being kept busy.</para></entry>
458 <entry><para> <emphasis role="bold">offset</emphasis></para></entry>
459 <entry><para> </para></entry>
466 <title>31.2.3 Client Read-Write <anchor xml:id="dbdoclet.50438271_marker-1290564" xreflabel=""/>Offset Survey</title>
467 <para>The offset_stats parameter maintains statistics for occurrences where a series of read or write calls from a process did not access the next sequential location. The offset field is reset to 0 (zero) whenever a different file is read/written.</para>
468 <para>Read/write offset statistics are off, by default. The statistics can be activated by writing anything into the offset_stats file.</para>
469 <para>Example:</para>
470 <screen># cat /proc/fs/lustre/llite/lustre-f57dee00/rw_offset_stats
471 snapshot_time: 1155748884.591028 (secs.usecs)
472 R/W PID RANGE START RANGE END \
473 SMALLEST EXTENT LARGEST EXTENT OFF\
489 <informaltable frame="all">
491 <colspec colname="c1" colwidth="50*"/>
492 <colspec colname="c2" colwidth="50*"/>
495 <entry><para><emphasis role="bold">Field</emphasis></para></entry>
496 <entry><para><emphasis role="bold">Description</emphasis></para></entry>
501 <entry><para> <emphasis role="bold">R/W</emphasis></para></entry>
502 <entry><para> Whether the non-sequential call was a read or write</para></entry>
505 <entry><para> <emphasis role="bold">PID</emphasis></para></entry>
506 <entry><para> Process ID which made the read/write call.</para></entry>
509 <entry><para> <emphasis role="bold">Range Start/Range End</emphasis></para></entry>
510 <entry><para> Range in which the read/write calls were sequential.</para><para> </para></entry>
513 <entry><para> <emphasis role="bold">Smallest Extent</emphasis></para></entry>
514 <entry><para> Smallest extent (single read/write) in the corresponding range.</para></entry>
517 <entry><para> <emphasis role="bold">Largest Extent</emphasis></para></entry>
518 <entry><para> Largest extent (single read/write) in the corresponding range.</para></entry>
521 <entry><para> <emphasis role="bold">Offset</emphasis></para></entry>
522 <entry><para> Difference from the previous range end to the current range start.</para><para>For example, Smallest-Extent indicates that the writes in the range 100 to 1110 were sequential, with a minimum write of 10 and a maximum write of 500. This range was started with an offset of -150. That means this is the difference between the last entry's range-end and this entry's range-start for the same file.</para><para>The rw_offset_stats file can be cleared by writing to it:</para><screen>
523 echo > /proc/fs/lustre/llite/lustre-f57dee00/rw_offset_stats
531 <title>31.2.4 Client Read-Write <anchor xml:id="dbdoclet.50438271_marker-1290612" xreflabel=""/>Extents Survey</title>
532 <para><emphasis role="bold">Client-Based I/O Extent Size Survey</emphasis></para>
533 <para>The rw_extent_stats histogram in the llite directory shows you the statistics for the sizes of the read-write I/O extents. This file does not maintain the per-process statistics.</para>
534 <para>Example:</para>
535 <screen>$ cat /proc/fs/lustre/llite/lustre-ee5af200/extents_stats
536 snapshot_time: 1213828728.348516 (secs.usecs)
538 extents calls % cum% | calls % \
541 0K - 4K : 0 0 0 | 2 2 2
542 4K - 8K : 0 0 0 | 0 0 2
543 8K - 16K : 0 0 0 | 0 0 2
544 16K - 32K : 0 0 0 | 20 23 \
546 32K - 64K : 0 0 0 | 0 0 \
548 64K - 128K : 0 0 0 | 51 60 \
550 128K - 256K : 0 0 0 | 0 0 \
552 256K - 512K : 0 0 0 | 0 0 \
554 512K - 1024K : 0 0 0 | 0 0 \
556 1M - 2M : 0 0 0 | 11 13 100
559 <para>The file can be cleared by issuing the following command:</para>
560 <screen>$ echo > cat /proc/fs/lustre/llite/lustre-ee5af200/extents_stats
562 <para><emphasis role="bold">Per-Process Client I/O Statistics</emphasis></para>
563 <para>The extents_stats_per_process file maintains the I/O extent size statistics on a per-process basis. So you can track the per-process statistics for the last MAX_PER_PROCESS_HIST processes.</para>
564 <para>Example:</para>
565 <screen>$ cat /proc/fs/lustre/llite/lustre-ee5af200/extents_stats_per_process
566 snapshot_time: 1213828762.204440 (secs.usecs)
568 extents calls % cum% | calls % \
572 0K - 4K : 0 0 0 | 0 0 0
573 4K - 8K : 0 0 0 | 0 0 0
574 8K - 16K : 0 0 0 | 0 0 0
575 16K - 32K : 0 0 0 | 0 0 0
576 32K - 64K : 0 0 0 | 0 0 0
577 64K - 128K : 0 0 0 | 0 0 0
578 128K - 256K : 0 0 0 | 0 0 0
579 256K - 512K : 0 0 0 | 0 0 0
580 512K - 1024K : 0 0 0 | 0 0 0
581 1M - 2M : 0 0 0 | 10 100 100
584 0K - 4K : 0 0 0 | 0 0 0
585 4K - 8K : 0 0 0 | 0 0 0
586 8K - 16K : 0 0 0 | 0 0 0
587 16K - 32K : 0 0 0 | 20 100 100
590 0K - 4K : 0 0 0 | 0 0 0
591 4K - 8K : 0 0 0 | 0 0 0
592 8K - 16K : 0 0 0 | 0 0 0
593 16K - 32K : 0 0 0 | 0 0 0
594 32K - 64K : 0 0 0 | 0 0 0
595 64K - 128K : 0 0 0 | 16 100 100
598 0K - 4K : 0 0 0 | 1 100 100
601 0K - 4K : 0 0 0 | 1 100 100
606 <title>31.2.5 <anchor xml:id="dbdoclet.50438271_55057" xreflabel=""/> Watching the <anchor xml:id="dbdoclet.50438271_marker-1290631" xreflabel=""/>OST Block I/O Stream</title>
607 <para>Similarly, there is a brw_stats histogram in the obdfilter directory which shows you the statistics for number of I/O requests sent to the disk, their size and whether they are contiguous on the disk or not.</para>
608 <screen>cat /proc/fs/lustre/obdfilter/lustre-OST0000/brw_stats
609 snapshot_time: 1174875636.764630 (secs:usecs)
611 pages per brw brws % cum % | rpcs % \
615 discont pages rpcs % cum % | rpcs % \
619 discont blocks rpcs % cum % | rpcs % \
623 dio frags rpcs % cum % | rpcs % cum %
626 disk ios in flight rpcs % cum % | rpcs % cum %
629 io time (1/1000s) rpcs % cum % | rpcs % cum %
632 disk io size rpcs % cum % | rpcs % \
637 <para>The fields are explained below:</para>
638 <informaltable frame="all">
640 <colspec colname="c1" colwidth="50*"/>
641 <colspec colname="c2" colwidth="50*"/>
644 <entry><para><emphasis role="bold">Field</emphasis></para></entry>
645 <entry><para><emphasis role="bold">Description</emphasis></para></entry>
650 <entry><para> <emphasis role="bold">pages per brw</emphasis></para></entry>
651 <entry><para> Number of pages per RPC request, which should match aggregate client rpc_stats.</para></entry>
654 <entry><para> <emphasis role="bold">discont pages</emphasis></para></entry>
655 <entry><para> Number of discontinuities in the logical file offset of each page in a single RPC.</para></entry>
658 <entry><para> <emphasis role="bold">discont blocks</emphasis></para></entry>
659 <entry><para> Number of discontinuities in the physical block allocation in the file system for a single RPC.</para></entry>
664 <para>For each Lustre service, the following information is provided:</para>
665 <itemizedlist><listitem>
666 <para> Number of requests</para>
670 <para> Request wait time (avg, min, max and std dev)</para>
674 <para> Service idle time (% of elapsed time)</para>
678 <para>Additionally, data on each Lustre service is provided by service type:</para>
679 <itemizedlist><listitem>
680 <para> Number of requests of this type</para>
684 <para> Request service time (avg, min, max and std dev)</para>
690 <title>31.2.6 Using File <anchor xml:id="dbdoclet.50438271_marker-1294292" xreflabel=""/>Readahead and Directory Statahead</title>
691 <para>Lustre 1.6.5.1 introduced file readahead and directory statahead functionality that read data into memory in anticipation of a process actually requesting the data. File readahead functionality reads file content data into memory. Directory statahead functionality reads metadata into memory. When readahead and/or statahead work well, a data-consuming process finds that the information it needs is available when requested, and it is unnecessary to wait for network I/O.</para>
693 <title>31.2.6.1 Tuning <anchor xml:id="dbdoclet.50438271_marker-1295183" xreflabel=""/>File Readahead</title>
694 <para>File readahead is triggered when two or more sequential reads by an application fail to be satisfied by the Linux buffer cache. The size of the initial readahead is 1 MB. Additional readaheads grow linearly, and increment until the readahead cache on the client is full at 40 MB.</para>
695 <para><emphasis role="bold">/proc/fs/lustre/llite/<fsname>-<uid>/max_read_ahead_mb</emphasis></para>
696 <para>This tunable controls the maximum amount of data readahead on a file. Files are read ahead in RPC-sized chunks (1 MB or the size of read() call, if larger) after the second sequential read on a file descriptor. Random reads are done at the size of the read() call only (no readahead). Reads to non-contiguous regions of the file reset the readahead algorithm, and readahead is not triggered again until there are sequential reads again. To disable readahead, set this tunable to 0. The default value is 40 MB.</para>
697 <para><emphasis role="bold">/proc/fs/lustre/llite/<fsname>-<uid>/max_read_ahead_whole_mb</emphasis></para>
698 <para>This tunable controls the maximum size of a file that is read in its entirety, regardless of the size of the read().</para>
701 <title>31.2.6.2 Tuning Directory <anchor xml:id="dbdoclet.50438271_marker-1295184" xreflabel=""/>Statahead</title>
702 <para>When the ls -l process opens a directory, its process ID is recorded. When the first directory entry is ''stated'' with this recorded process ID, a statahead thread is triggered which stats ahead all of the directory entries, in order. The ls -l process can use the stated directory entries directly, improving performance.</para>
703 <para>/proc/fs/lustre/llite/*/statahead_max</para>
704 <para>This tunable controls whether directory statahead is enabled and the maximum statahead count. By default, statahead is active.</para>
705 <para>To disable statahead, set this tunable to:</para>
706 <para>echo 0 > /proc/fs/lustre/llite/*/statahead_max</para>
707 <para>To set the maximum statahead count (n), set this tunable to:</para>
708 <screen>echo n > /proc/fs/lustre/llite/*/statahead_max
710 <para>The maximum value of n is 8192.</para>
711 <para>/proc/fs/lustre/llite/*/statahead_status</para>
712 <para>This is a read-only interface that indicates the current statahead status.</para>
716 <title>31.2.7 OSS <anchor xml:id="dbdoclet.50438271_marker-1296183" xreflabel=""/>Read Cache</title>
717 <para>The OSS read cache feature provides read-only caching of data on an OSS. This functionality uses the regular Linux page cache to store the data. Just like caching from a regular filesytem in Linux, OSS read cache uses as much physical memory as is allocated.</para>
718 <para>OSS read cache improves Lustre performance in these situations:</para>
719 <itemizedlist><listitem>
720 <para> Many clients are accessing the same data set (as in HPC applications and when diskless clients boot from Lustre)</para>
724 <para> One client is storing data while another client is reading it (essentially exchanging data via the OST)</para>
728 <para> A client has very limited caching of its own</para>
732 <para>OSS read cache offers these benefits:</para>
733 <itemizedlist><listitem>
734 <para> Allows OSTs to cache read data more frequently</para>
738 <para> Improves repeated reads to match network speeds instead of disk speeds</para>
742 <para> Provides the building blocks for OST write cache (small-write aggregation)</para>
747 <title>31.2.7.1 Using OSS Read Cache</title>
748 <para>OSS read cache is implemented on the OSS, and does not require any special support on the client side. Since OSS read cache uses the memory available in the Linux page cache, you should use I/O patterns to determine the appropriate amount of memory for the cache; if the data is mostly reads, then more cache is required than for writes.</para>
749 <para>OSS read cache is enabled, by default, and managed by the following tunables:</para>
750 <itemizedlist><listitem>
751 <para>read_cache_enable controls whether data read from disk during a read request is kept in memory and available for later read requests for the same data, without having to re-read it from disk. By default, read cache is enabled (read_cache_enable = 1).</para>
755 <para>When the OSS receives a read request from a client, it reads data from disk into its memory and sends the data as a reply to the requests. If read cache is enabled, this data stays in memory after the client's request is finished, and the OSS skips reading data from disk when subsequent read requests for the same are received. The read cache is managed by the Linux kernel globally across all OSTs on that OSS, and the least recently used cache pages will be dropped from memory when the amount of free memory is running low.</para>
756 <para>If read cache is disabled (read_cache_enable = 0), then the OSS will discard the data after the client's read requests are serviced and, for subsequent read requests, the OSS must read the data from disk.</para>
757 <para>To disable read cache on all OSTs of an OSS, run:</para>
758 <screen>root@oss1# lctl set_param obdfilter.*.read_cache_enable=0
760 <para>To re-enable read cache on one OST, run:</para>
761 <screen>root@oss1# lctl set_param obdfilter.{OST_name}.read_cache_enable=1
763 <para>To check if read cache is enabled on all OSTs on an OSS, run:</para>
764 <screen>root@oss1# lctl get_param obdfilter.*.read_cache_enable
766 <itemizedlist><listitem>
767 <para>writethrough_cache_enable controls whether data sent to the OSS as a write request is kept in the read cache and available for later reads, or if it is discarded from cache when the write is completed. By default, writethrough cache is enabled (writethrough_cache_enable = 1).</para>
771 <para>When the OSS receives write requests from a client, it receives data from the client into its memory and writes the data to disk. If writethrough cache is enabled, this data stays in memory after the write request is completed, allowing the OSS to skip reading this data from disk if a later read request, or partial-page write request, for the same data is received.</para>
772 <para>If writethrough cache is disabled (writethrough_cache_enabled = 0), then the OSS discards the data after the client's write request is completed, and for subsequent read request, or partial-page write request, the OSS must re-read the data from disk.</para>
773 <para>Enabling writethrough cache is advisable if clients are doing small or unaligned writes that would cause partial-page updates, or if the files written by one node are immediately being accessed by other nodes. Some examples where this might be useful include producer-consumer I/O models or shared-file writes with a different node doing I/O not aligned on 4096-byte boundaries. Disabling writethrough cache is advisable in the case where files are mostly written to the file system but are not re-read within a short time period, or files are only written and re-read by the same node, regardless of whether the I/O is aligned or not.</para>
774 <para>To disable writethrough cache on all OSTs of an OSS, run:</para>
775 <screen>root@oss1# lctl set_param obdfilter.*.writethrough_cache_enable=0
777 <para>To re-enable writethrough cache on one OST, run:</para>
778 <screen>root@oss1# lctl set_param \obdfilter.{OST_name}.writethrough_cache_enable=1
780 <para>To check if writethrough cache is</para>
781 <screen>root@oss1# lctl set_param obdfilter.*.writethrough_cache_enable=1
783 <itemizedlist><listitem>
784 <para>readcache_max_filesize controls the maximum size of a file that both the read cache and writethrough cache will try to keep in memory. Files larger than readcache_max_filesize will not be kept in cache for either reads or writes.</para>
788 <para>This can be very useful for workloads where relatively small files are repeatedly accessed by many clients, such as job startup files, executables, log files, etc., but large files are read or written only once. By not putting the larger files into the cache, it is much more likely that more of the smaller files will remain in cache for a longer time.</para>
789 <para>When setting readcache_max_filesize, the input value can be specified in bytes, or can have a suffix to indicate other binary units such as <emphasis role="bold">K</emphasis>ilobytes, <emphasis role="bold">M</emphasis>egabytes, <emphasis role="bold">G</emphasis>igabytes, <emphasis role="bold">T</emphasis>erabytes, or <emphasis role="bold">P</emphasis>etabytes.</para>
790 <para>To limit the maximum cached file size to 32MB on all OSTs of an OSS, run:</para>
791 <screen>root@oss1# lctl set_param obdfilter.*.readcache_max_filesize=32M
793 <para>To disable the maximum cached file size on an OST, run:</para>
794 <screen>root@oss1# lctl set_param \obdfilter.{OST_name}.readcache_max_filesize=-1
796 <para>To check the current maximum cached file size on all OSTs of an OSS, run:</para>
797 <screen>root@oss1# lctl get_param obdfilter.*.readcache_max_filesize
802 <title>31.2.8 OSS Asynchronous Journal Commit</title>
803 <para>The OSS asynchronous journal commit feature synchronously writes data to disk without forcing a journal flush. This reduces the number of seeks and significantly improves performance on some hardware.</para>
804 <note><para>Asynchronous journal commit cannot work with O_DIRECT writes, a journal flush is still forced.</para></note>
805 <para>When asynchronous journal commit is enabled, client nodes keep data in the page cache (a page reference). Lustre clients monitor the last committed transaction number (transno) in messages sent from the OSS to the clients. When a client sees that the last committed transno reported by the OSS is >=bulk write transno, it releases the reference on the corresponding pages. To avoid page references being held for too long on clients after a bulk write, a 7 second ping request is scheduled (jbd commit time is 5 seconds) after the bulk write reply is received, so the OSS has an opportunity to report the last committed transno.</para>
806 <para>If the OSS crashes before the journal commit occurs, then the intermediate data is lost. However, new OSS recovery functionality (introduced in the asynchronous journal commit feature), causes clients to replay their write requests and compensate for the missing disk updates by restoring the state of the file system.</para>
807 <para>To enable asynchronous journal commit, set the sync_journal parameter to zero (sync_journal=0):</para>
808 <screen>$ lctl set_param obdfilter.*.sync_journal=0
809 obdfilter.lol-OST0001.sync_journal=0
811 <para>By default, sync_journal is disabled (sync_journal=1), which forces a journal flush after every bulk write.</para>
812 <para>When asynchronous journal commit is used, clients keep a page reference until the journal transaction commits. This can cause problems when a client receives a blocking callback, because pages need to be removed from the page cache, but they cannot be removed because of the extra page reference.</para>
813 <para>This problem is solved by forcing a journal flush on lock cancellation. When this happens, the client is granted the metadata blocks that have hit the disk, and it can safely release the page reference before processing the blocking callback. The parameter which controls this action is sync_on_lock_cancel, which can be set to the following values:</para>
814 <para>always: Always force a journal flush on lock cancellation</para>
815 <para>blocking: Force a journal flush only when the local cancellation is due to a blocking callback</para>
816 <para>never: Do not force any journal flush</para>
817 <para>Here is an example of sync_on_lock_cancel being set not to force a journal flush:</para>
818 <screen>$ lctl get_param obdfilter.*.sync_on_lock_cancel
819 obdfilter.lol-OST0001.sync_on_lock_cancel=never
821 <para>By default, sync_on_lock_cancel is set to never, because asynchronous journal commit is disabled by default.</para>
822 <para>When asynchronous journal commit is enabled (sync_journal=0), sync_on_lock_cancel is automatically set to always, if it was previously set to never.</para>
823 <para>Similarly, when asynchronous journal commit is disabled, (sync_journal=1), sync_on_lock_cancel is enforced to never.</para>
826 <title>31.2.9 mballoc <anchor xml:id="dbdoclet.50438271_marker-1297045" xreflabel=""/>History</title>
827 <para><emphasis role="bold">/proc/fs/ldiskfs/sda/mb_history</emphasis></para>
828 <para>Multi-Block-Allocate (mballoc), enables Lustre to ask ldiskfs to allocate multiple blocks with a single request to the block allocator. Typically, an ldiskfs file system allocates only one block per time. Each mballoc-enabled partition has this file. This is sample output:</para>
829 <screen>pid inode goal result found grps cr \ me\
831 2838 139267 17/12288/1 17/12288/1 1 0 0 \
833 2838 139267 17/12289/1 17/12289/1 1 0 0 \
835 2838 139267 17/12290/1 17/12290/1 1 0 0 \
837 2838 24577 3/12288/1 3/12288/1 1 0 0 \
839 2838 24578 3/12288/1 3/771/1 1 1 1 \
841 2838 32769 4/12288/1 4/12288/1 1 0 0 \
843 2838 32770 4/12288/1 4/12289/1 13 1 1 \
845 2838 32771 4/12288/1 5/771/1 26 2 1 \
847 2838 32772 4/12288/1 5/896/1 31 2 1 \
849 2838 32773 4/12288/1 5/897/1 31 2 1 \
851 2828 32774 4/12288/1 5/898/1 31 2 1 \
853 2838 32775 4/12288/1 5/899/1 31 2 1 \
855 2838 32776 4/12288/1 5/900/1 31 2 1 \
857 2838 32777 4/12288/1 5/901/1 31 2 1 \
859 2838 32778 4/12288/1 5/902/1 31 2 1 \
862 <para>The parameters are described below:</para>
863 <informaltable frame="all">
865 <colspec colname="c1" colwidth="50*"/>
866 <colspec colname="c2" colwidth="50*"/>
869 <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
870 <entry><para><emphasis role="bold">Description</emphasis></para></entry>
875 <entry><para> <emphasis role="bold">pid</emphasis></para></entry>
876 <entry><para> Process that made the allocation.</para></entry>
879 <entry><para> <emphasis role="bold">inode</emphasis></para></entry>
880 <entry><para> inode number allocated blocks</para></entry>
883 <entry><para> <emphasis role="bold">goal</emphasis></para></entry>
884 <entry><para> Initial request that came to mballoc (group/block-in-group/number-of-blocks)</para></entry>
887 <entry><para> <emphasis role="bold">result</emphasis></para></entry>
888 <entry><para> What mballoc actually found for this request.</para></entry>
891 <entry><para> <emphasis role="bold">found</emphasis></para></entry>
892 <entry><para> Number of free chunks mballoc found and measured before the final decision.</para></entry>
895 <entry><para> <emphasis role="bold">grps</emphasis></para></entry>
896 <entry><para> Number of groups mballoc scanned to satisfy the request.</para></entry>
899 <entry><para> <emphasis role="bold">cr</emphasis></para></entry>
900 <entry><para> Stage at which mballoc found the result:</para><para><emphasis role="bold">0</emphasis> - best in terms of resource allocation. The request was 1MB or larger and was satisfied directly via the kernel buddy allocator.</para><para><emphasis role="bold">1</emphasis> - regular stage (good at resource consumption)</para><para><emphasis role="bold">2</emphasis> - fs is quite fragmented (not that bad at resource consumption)</para><para><emphasis role="bold">3</emphasis> - fs is very fragmented (worst at resource consumption)</para></entry>
903 <entry><para> <emphasis role="bold">queue</emphasis></para></entry>
904 <entry><para> Total bytes in active/queued sends.</para></entry>
907 <entry><para> <emphasis role="bold">merge</emphasis></para></entry>
908 <entry><para> Whether the request hit the goal. This is good as extents code can now merge new blocks to existing extent, eliminating the need for extents tree growth.</para></entry>
911 <entry><para> <emphasis role="bold">tail</emphasis></para></entry>
912 <entry><para> Number of blocks left free after the allocation breaks large free chunks.</para></entry>
915 <entry><para> <emphasis role="bold">broken</emphasis></para></entry>
916 <entry><para> How large the broken chunk was.</para></entry>
921 <para>Most customers are probably interested in found/cr. If cr is 0 1 and found is less than 100, then mballoc is doing quite well.</para>
922 <para>Also, number-of-blocks-in-request (third number in the goal triple) can tell the number of blocks requested by the obdfilter. If the obdfilter is doing a lot of small requests (just few blocks), then either the client is processing input/output to a lot of small files, or something may be wrong with the client (because it is better if client sends large input/output requests). This can be investigated with the OSC rpc_stats or OST brw_stats mentioned above.</para>
923 <para>Number of groups scanned (grps column) should be small. If it reaches a few dozen often, then either your disk file system is pretty fragmented or mballoc is doing something wrong in the group selection part.</para>
926 <title>31.2.10 mballoc3<anchor xml:id="dbdoclet.50438271_marker-1290795" xreflabel=""/> Tunables</title>
927 <para>Lustre version 1.6.1 and later includes mballoc3, which was built on top of mballoc2. By default, mballoc3 is enabled, and adds these features:</para>
928 <itemizedlist><listitem>
929 <para> Pre-allocation for single files (helps to resist fragmentation)</para>
933 <para> Pre-allocation for a group of files (helps to pack small files into large, contiguous chunks)</para>
937 <para> Stream allocation (helps to decrease the seek rate)</para>
941 <para>The following mballoc3 tunables are available:</para>
942 <informaltable frame="all">
944 <colspec colname="c1" colwidth="50*"/>
945 <colspec colname="c2" colwidth="50*"/>
948 <entry><para><emphasis role="bold">Field</emphasis></para></entry>
949 <entry><para><emphasis role="bold">Description</emphasis></para></entry>
954 <entry><para> <emphasis role="bold">stats</emphasis></para></entry>
955 <entry><para> Enables/disables the collection of statistics. Collected statistics can be found</para><para>in /proc/fs/ldiskfs2/<dev>/mb_history.</para></entry>
958 <entry><para> <emphasis role="bold">max_to_scan</emphasis></para></entry>
959 <entry><para> Maximum number of free chunks that mballoc finds before a final decision to avoid livelock.</para></entry>
962 <entry><para> <emphasis role="bold">min_to_scan</emphasis></para></entry>
963 <entry><para> Minimum number of free chunks that mballoc finds before a final decision. This is useful for a very small request, to resist fragmentation of big free chunks.</para></entry>
966 <entry><para> <emphasis role="bold">order2_req</emphasis></para></entry>
967 <entry><para> For requests equal to 2^N (where N >= order2_req), a very fast search via buddy structures is used.</para></entry>
970 <entry><para> <emphasis role="bold">stream_req</emphasis></para></entry>
971 <entry><para> Requests smaller or equal to this value are packed together to form large write I/Os.</para></entry>
976 <para>The following tunables, providing more control over allocation policy, will be available in the next version:</para>
977 <informaltable frame="all">
979 <colspec colname="c1" colwidth="50*"/>
980 <colspec colname="c2" colwidth="50*"/>
983 <entry><para><emphasis role="bold">Field</emphasis></para></entry>
984 <entry><para><emphasis role="bold">Description</emphasis></para></entry>
989 <entry><para> <emphasis role="bold">stats</emphasis></para></entry>
990 <entry><para> Enables/disables the collection of statistics. Collected statistics can be found in /proc/fs/ldiskfs2/<dev>/mb_history.</para></entry>
993 <entry><para> <emphasis role="bold">max_to_scan</emphasis></para></entry>
994 <entry><para> Maximum number of free chunks that mballoc finds before a final decision to avoid livelock.</para></entry>
997 <entry><para> <emphasis role="bold">min_to_scan</emphasis></para></entry>
998 <entry><para> Minimum number of free chunks that mballoc finds before a final decision. This is useful for a very small request, to resist fragmentation of big free chunks.</para></entry>
1001 <entry><para> <emphasis role="bold">order2_req</emphasis></para></entry>
1002 <entry><para> For requests equal to 2^N (where N >= order2_req), a very fast search via buddy structures is used.</para></entry>
1005 <entry><para> <emphasis role="bold">small_req</emphasis></para></entry>
1006 <entry morerows="1"><para> All requests are divided into 3 categories:</para><para>< small_req (packed together to form large, aggregated requests)</para><para>< large_req (allocated mostly in linearly)</para><para>> large_req (very large requests so the arm seek does not matter)</para><para>The idea is that we try to pack small requests to form large requests, and then place all large requests (including compound from the small ones) close to one another, causing as few arm seeks as possible.</para></entry>
1009 <entry><para> <emphasis role="bold">large_req</emphasis></para></entry>
1012 <entry><para> <emphasis role="bold">prealloc_table</emphasis></para></entry>
1013 <entry><para> The amount of space to preallocate depends on the current file size. The idea is that for small files we do not need 1 MB preallocations and for large files, 1 MB preallocations are not large enough; it is better to preallocate 4 MB.</para></entry>
1016 <entry><para> <emphasis role="bold">group_prealloc</emphasis></para></entry>
1017 <entry><para> The amount of space preallocated for small requests to be grouped.</para></entry>
1023 <section remap="h3">
1024 <title>31.2.11 <anchor xml:id="dbdoclet.50438271_13474" xreflabel=""/>Lo<anchor xml:id="dbdoclet.50438271_marker-1290874" xreflabel=""/>cking</title>
1025 <para><emphasis role="bold">/proc/fs/lustre/ldlm/ldlm/namespaces/<OSC name|MDC name>/lru_size</emphasis></para>
1026 <para>The lru_size parameter is used to control the number of client-side locks in an LRU queue. LRU size is dynamic, based on load. This optimizes the number of locks available to nodes that have different workloads (e.g., login/build nodes vs. compute nodes vs. backup nodes).</para>
1027 <para>The total number of locks available is a function of the server's RAM. The default limit is 50 locks/1 MB of RAM. If there is too much memory pressure, then the LRU size is shrunk. The number of locks on the server is limited to {number of OST/MDT on node} * {number of clients} * {client lru_size}.</para>
1028 <itemizedlist><listitem>
1029 <para> To enable automatic LRU sizing, set the lru_size parameter to 0. In this case, the lru_size parameter shows the current number of locks being used on the export. (In Lustre 1.6.5.1 and later, LRU sizing is enabled, by default.)</para>
1033 <para> To specify a maximum number of locks, set the lru_size parameter to a value > 0 (former numbers are okay, 100 * CPU_NR). We recommend that you only increase the LRU size on a few login nodes where users access the file system interactively.</para>
1037 <para>To clear the LRU on a single client, and as a result flush client cache, without changing the lru_size value:</para>
1038 <screen>$ lctl set_param ldlm.namespaces.<osc_name|mdc_name>.lru_size=clear
1040 <para>If you shrink the LRU size below the number of existing unused locks, then the unused locks are canceled immediately. Use echo clear to cancel all locks without changing the value.</para>
1041 <note><para>Currently, the lru_size parameter can only be set temporarily with lctl set_param; it cannot be set permanently.</para></note>
1042 <para>To disable LRU sizing, run this command on the Lustre clients:</para>
1043 <screen>$ lctl set_param ldlm.namespaces.*osc*.lru_size=$((NR_CPU*100))
1045 <para>Replace NR_CPU value with the number of CPUs on the node.</para>
1046 <para>To determine the number of locks being granted:</para>
1047 <screen>$ lctl get_param ldlm.namespaces.*.pool.limit
1050 <section remap="h3">
1051 <title>31.2.12 <anchor xml:id="dbdoclet.50438271_87260" xreflabel=""/>Setting MDS and OSS Thread Counts</title>
1052 <para>MDS and OSS thread counts (minimum and maximum) can be set via the {min,max}_thread_count tunable. For each service, a new /proc/fs/lustre/{service}/*/thread_{min,max,started} entry is created. The tunable, {service}.thread_{min,max,started}, can be used to set the minimum and maximum thread counts or get the current number of running threads for the following services.</para>
1053 <informaltable frame="all">
1055 <colspec colname="c1" colwidth="50*"/>
1056 <colspec colname="c2" colwidth="50*"/>
1059 <entry><para> <emphasis role="bold">Service</emphasis></para></entry>
1060 <entry><para> <emphasis role="bold">Description</emphasis></para></entry>
1063 <entry><para> mdt.MDS.mds</para></entry>
1064 <entry><para> normal metadata ops</para></entry>
1067 <entry><para> mdt.MDS.mds_readpage</para></entry>
1068 <entry><para> metadata readdir</para></entry>
1071 <entry><para> mdt.MDS.mds_setattr</para></entry>
1072 <entry><para> metadata setattr</para></entry>
1075 <entry><para> ost.OSS.ost</para></entry>
1076 <entry><para> normal data</para></entry>
1079 <entry><para> ost.OSS.ost_io</para></entry>
1080 <entry><para> bulk data IO</para></entry>
1083 <entry><para> ost.OSS.ost_create</para></entry>
1084 <entry><para> OST object pre-creation service</para></entry>
1087 <entry><para> ldlm.services.ldlm_canceld</para></entry>
1088 <entry><para> DLM lock cancel</para></entry>
1091 <entry><para> ldlm.services.ldlm_cbd</para></entry>
1092 <entry><para> DLM lock grant</para></entry>
1097 <itemizedlist><listitem>
1098 <para> To temporarily set this tunable, run:</para>
1102 <screen># lctl {get,set}_param {service}.thread_{min,max,started}
1104 <itemizedlist><listitem>
1105 <para> To permanently set this tunable, run:</para>
1109 <screen># lctl conf_param {service}.thread_{min,max,started} </screen>
1110 <para>The following examples show how to set thread counts and get the number of running threads for the ost_io service.</para>
1111 <itemizedlist><listitem>
1112 <para> To get the number of running threads, run:</para>
1116 <screen># lctl get_param ost.OSS.ost_io.threads_started</screen>
1117 <para>The command output will be similar to this:</para>
1118 <screen>ost.OSS.ost_io.threads_started=128
1120 <itemizedlist><listitem>
1121 <para> To set the maximum number of threads (512), run:</para>
1125 <screen># lctl get_param ost.OSS.ost_io.threads_max
1127 <para>The command output will be:</para>
1128 <screen>ost.OSS.ost_io.threads_max=512
1130 <itemizedlist><listitem>
1131 <para> To set the maximum thread count to 256 instead of 512 (to avoid overloading the storage or for an array with requests), run:</para>
1135 <screen># lctl set_param ost.OSS.ost_io.threads_max=256
1137 <para>The command output will be:</para>
1138 <screen>ost.OSS.ost_io.threads_max=256
1140 <itemizedlist><listitem>
1141 <para> To check if the new threads_max setting is active, run:</para>
1145 <screen># lctl get_param ost.OSS.ost_io.threads_max
1147 <para>The command output will be similar to this:</para>
1148 <screen>ost.OSS.ost_io.threads_max=256
1150 <note><para>Currently, the maximum thread count setting is advisory because Lustre does not reduce the number of service threads in use, even if that number exceeds the threads_max value. Lustre does not stop service threads once they are started.</para></note>
1153 <section xml:id="dbdoclet.50438271_83523">
1154 <title>31.3 Debug</title>
1155 <para><emphasis role="bold">/proc/sys/lnet/debug</emphasis></para>
1156 <para>By default, Lustre generates a detailed log of all operations to aid in debugging. The level of debugging can affect the performance or speed you achieve with Lustre. Therefore, it is useful to reduce this overhead by turning down the debug level<footnote><para>This controls the level of Lustre debugging kept in the internal log buffer. It does not alter the level of debugging that goes to syslog.</para></footnote> to improve performance. Raise the debug level when you need to collect the logs for debugging problems. The debugging mask can be set with "symbolic names" instead of the numerical values that were used in prior releases. The new symbolic format is shown in the examples below.</para>
1157 <note><para>All of the commands below must be run as root; note the # nomenclature.</para></note>
1158 <para>To verify the debug level used by examining the sysctl that controls debugging, run:</para>
1159 <screen># sysctl lnet.debug
1160 lnet.debug = ioctl neterror warning error emerg ha config console
1162 <para>To turn off debugging (except for network error debugging), run this command on all concerned nodes:</para>
1163 <screen># sysctl -w lnet.debug="neterror"
1164 lnet.debug = neterror
1166 <para>To turn off debugging completely, run this command on all concerned nodes:</para>
1167 <screen># sysctl -w lnet.debug=0
1170 <para>To set an appropriate debug level for a production environment, run:</para>
1171 <screen># sysctl -w lnet.debug="warning dlmtrace error emerg ha rpctrace vfstrace"
1172 lnet.debug = warning dlmtrace error emerg ha rpctrace vfstrace
1174 <para>The flags above collect enough high-level information to aid debugging, but they do not cause any serious performance impact.</para>
1175 <para>To clear all flags and set new ones, run:</para>
1176 <screen># sysctl -w lnet.debug="warning"
1177 lnet.debug = warning
1179 <para>To add new flags to existing ones, prefix them with a "+":</para>
1180 <screen># sysctl -w lnet.debug="+neterror +ha"
1181 lnet.debug = +neterror +ha
1183 lnet.debug = neterror warning ha
1185 <para>To remove flags, prefix them with a "-":</para>
1186 <screen># sysctl -w lnet.debug="-ha"
1189 lnet.debug = neterror warning
1191 <para>You can verify and change the debug level using the /proc interface in Lustre. To use the flags with /proc, run:</para>
1192 <screen># cat /proc/sys/lnet/debug
1194 # echo "+ha" > /proc/sys/lnet/debug
1195 # cat /proc/sys/lnet/debug
1197 # echo "-warning" > /proc/sys/lnet/debug
1198 # cat /proc/sys/lnet/debug
1201 <para><emphasis role="bold">/proc/sys/lnet/subsystem_debug</emphasis></para>
1202 <para>This controls the debug logs3 for subsystems (see S_* definitions).</para>
1203 <para><emphasis role="bold">/proc/sys/lnet/debug_path</emphasis></para>
1204 <para>This indicates the location where debugging symbols should be stored for gdb. The default is set to /r/tmp/lustre-log-localhost.localdomain.</para>
1205 <para>These values can also be set via sysctl -w lnet.debug={value}</para>
1206 <note><para>The above entries only exist when Lustre has already been loaded.</para></note>
1207 <para><emphasis role="bold">/proc/sys/lnet/panic_on_lbug</emphasis></para>
1208 <para>This causes Lustre to call ''panic'' when it detects an internal problem (an LBUG); panic crashes the node. This is particularly useful when a kernel crash dump utility is configured. The crash dump is triggered when the internal inconsistency is detected by Lustre.</para>
1209 <para><emphasis role="bold">/proc/sys/lnet/upcall</emphasis></para>
1210 <para>This allows you to specify the path to the binary which will be invoked when an LBUG is encountered. This binary is called with four parameters. The first one is the string ''LBUG''. The second one is the file where the LBUG occurred. The third one is the function name. The fourth one is the line number in the file.</para>
1211 <section remap="h3">
1212 <title>31.3.1 RPC Information for Other OBD Devices</title>
1213 <para>Some OBD devices maintain a count of the number of RPC events that they process. Sometimes these events are more specific to operations of the device, like llite, than actual raw RPC counts.</para>
1214 <screen>$ find /proc/fs/lustre/ -name stats
1215 /proc/fs/lustre/osc/lustre-OST0001-osc-ce63ca00/stats
1216 /proc/fs/lustre/osc/lustre-OST0000-osc-ce63ca00/stats
1217 /proc/fs/lustre/osc/lustre-OST0001-osc/stats
1218 /proc/fs/lustre/osc/lustre-OST0000-osc/stats
1219 /proc/fs/lustre/mdt/MDS/mds_readpage/stats
1220 /proc/fs/lustre/mdt/MDS/mds_setattr/stats
1221 /proc/fs/lustre/mdt/MDS/mds/stats
1222 /proc/fs/lustre/mds/lustre-MDT0000/exports/ab206805-0630-6647-8543-d24265c9\
1224 /proc/fs/lustre/mds/lustre-MDT0000/exports/08ac6584-6c4a-3536-2c6d-b36cf9cb\
1226 /proc/fs/lustre/mds/lustre-MDT0000/stats
1227 /proc/fs/lustre/ldlm/services/ldlm_canceld/stats
1228 /proc/fs/lustre/ldlm/services/ldlm_cbd/stats
1229 /proc/fs/lustre/llite/lustre-ce63ca00/stats
1231 <section remap="h4">
1232 <title>31.3.1.1 Interpreting OST Statistics</title>
1233 <note><para>See also <xref linkend="dbdoclet.50438219_84890"/> (llobdstat) and <xref linkend="dbdoclet.50438273_80593"/> (CollectL).</para></note>
1235 <para>The OST .../stats files can be used to track client statistics (client activity) for each OST. It is possible to get a periodic dump of values from these file (for example, every 10 seconds), that show the RPC rates (similar to iostat) by using the llstat.pl tool:</para>
1236 <screen># llstat /proc/fs/lustre/osc/lustre-OST0000-osc/stats
1237 /usr/bin/llstat: STATS on 09/14/07 /proc/fs/lustre/osc/lustre-OST0000-osc/s\
1238 tats on 192.168.10.34@tcp
1239 snapshot_time 1189732762.835363
1246 <para>To clear the statistics, give the -c option to llstat.pl. To specify how frequently the statistics should be cleared (in seconds), use an integer for the -i option. This is sample output with -c and -i10 options used, providing statistics every 10s):</para>
1247 <screen>$ llstat -c -i10 /proc/fs/lustre/ost/OSS/ost_io/stats
1249 /usr/bin/llstat: STATS on 06/06/07 /proc/fs/lustre/ost/OSS/ost_io/ stats on\
1251 snapshot_time 1181074093.276072
1253 /proc/fs/lustre/ost/OSS/ost_io/stats @ 1181074103.284895
1254 Name Cur.Count Cur.Rate #Events Unit \
1255 \last min avg max stddev
1256 req_waittime 8 0 8 [usec] \
1257 2078\ 34 259.75 868 317.49
1258 req_qdepth 8 0 8 [reqs] 1\ \
1260 req_active 8 0 8 [reqs] 11\ \
1262 reqbuf_avail 8 0 8 [bufs] \
1263 511\ 63 63.88 64 0.35
1264 ost_write 8 0 8 [bytes] 1697677\\
1265 72914 212209.62 387579 91874.29
1267 /proc/fs/lustre/ost/OSS/ost_io/stats @ 1181074113.290180
1268 Name Cur.Count Cur.Rate #Events Unit \
1269 \last min avg max stddev
1270 req_waittime 31 3 39 [usec] \
1271 30011\ 34 822.79 12245 2047.71
1272 req_qdepth 31 3 39 [reqs] 0\ \
1274 req_active 31 3 39 [reqs] 58\ \
1276 reqbuf_avail 31 3 39 [bufs] \
1277 1977\ 63 63.79 64 0.41
1278 ost_write 30 3 38 [bytes] 10284679\
1279 \ 15019 315325.16 910694 197776.51
1281 /proc/fs/lustre/ost/OSS/ost_io/stats @ 1181074123.325560
1282 Name Cur.Count Cur.Rate #Events Unit \
1283 \last min avg max stddev
1284 req_waittime 21 2 60 [usec] \
1285 14970\ 34 784.32 12245 1878.66
1286 req_qdepth 21 2 60 [reqs] 0\ \
1288 req_active 21 2 60 [reqs] 33\ \
1290 reqbuf_avail 21 2 60 [bufs] \
1291 1341\ 63 63.82 64 0.39
1292 ost_write 21 2 59 [bytes] 7648424\\
1293 15019 332725.08 910694 180397.87
1296 <informaltable frame="all">
1298 <colspec colname="c1" colwidth="50*"/>
1299 <colspec colname="c2" colwidth="50*"/>
1302 <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
1303 <entry><para><emphasis role="bold">Description</emphasis></para></entry>
1308 <entry><para> <emphasis role="bold">Cur. Count</emphasis></para></entry>
1309 <entry><para> Number of events of each type sent in the last interval (in this example, 10s)</para></entry>
1312 <entry><para> <emphasis role="bold">Cur. Rate</emphasis></para></entry>
1313 <entry><para> Number of events per second in the last interval</para></entry>
1316 <entry><para> <emphasis role="bold">#Events</emphasis></para></entry>
1317 <entry><para> Total number of such events since the system started</para></entry>
1320 <entry><para> <emphasis role="bold">Unit</emphasis></para></entry>
1321 <entry><para> Unit of measurement for that statistic (microseconds, requests, buffers)</para></entry>
1324 <entry><para> <emphasis role="bold">last</emphasis></para></entry>
1325 <entry><para> Average rate of these events (in units/event) for the last interval during which they arrived. For instance, in the above mentioned case of ost_destroy it took an average of 736 microseconds per destroy for the 400 object destroys in the previous 10 seconds.</para></entry>
1328 <entry><para> <emphasis role="bold">min</emphasis></para></entry>
1329 <entry><para> Minimum rate (in units/events) since the service started</para></entry>
1332 <entry><para> <emphasis role="bold">avg</emphasis></para></entry>
1333 <entry><para> Average rate</para></entry>
1336 <entry><para> <emphasis role="bold">max</emphasis></para></entry>
1337 <entry><para> Maximum rate</para></entry>
1340 <entry><para> <emphasis role="bold">stddev</emphasis></para></entry>
1341 <entry><para> Standard deviation (not measured in all cases)</para></entry>
1346 <para>The events common to all services are:</para>
1347 <informaltable frame="all">
1349 <colspec colname="c1" colwidth="50*"/>
1350 <colspec colname="c2" colwidth="50*"/>
1353 <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
1354 <entry><para><emphasis role="bold">Description</emphasis></para></entry>
1359 <entry><para> <emphasis role="bold">req_waittime</emphasis></para></entry>
1360 <entry><para> Amount of time a request waited in the queue before being handled by an available server thread.</para></entry>
1363 <entry><para> <emphasis role="bold">req_qdepth</emphasis></para></entry>
1364 <entry><para> Number of requests waiting to be handled in the queue for this service.</para></entry>
1367 <entry><para> <emphasis role="bold">req_active</emphasis></para></entry>
1368 <entry><para> Number of requests currently being handled.</para></entry>
1371 <entry><para> <emphasis role="bold">reqbuf_avail</emphasis></para></entry>
1372 <entry><para> Number of unsolicited lnet request buffers for this service.</para></entry>
1377 <para>Some service-specific events of interest are:</para>
1378 <informaltable frame="all">
1380 <colspec colname="c1" colwidth="50*"/>
1381 <colspec colname="c2" colwidth="50*"/>
1384 <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
1385 <entry><para><emphasis role="bold">Description</emphasis></para></entry>
1390 <entry><para> <emphasis role="bold">ldlm_enqueue</emphasis></para></entry>
1391 <entry><para> Time it takes to enqueue a lock (this includes file open on the MDS)</para></entry>
1394 <entry><para> <emphasis role="bold">mds_reint</emphasis></para></entry>
1395 <entry><para> Time it takes to process an MDS modification record (includes create, mkdir, unlink, rename and setattr)</para></entry>
1401 <section remap="h4">
1402 <title>31.3.1.2 Interpreting MDT Statistics</title>
1403 <note><para>See also <xref linkend="dbdoclet.50438219_84890"/> (llobdstat) and <xref linkend="dbdoclet.50438273_80593"/> (CollectL).</para></note>
1404 <para>The MDT .../stats files can be used to track MDT statistics for the MDS. Here is sample output for an MDT stats file:</para>
1405 <screen># cat /proc/fs/lustre/mds/*-MDT0000/stats
1406 snapshot_time 1244832003.676892 secs.usecs
1407 open 2 samples [reqs]
1408 close 1 samples [reqs]
1409 getxattr 3 samples [reqs]
1410 process_config 1 samples [reqs]
1411 connect 2 samples [reqs]
1412 disconnect 2 samples [reqs]
1413 statfs 3 samples [reqs]
1414 setattr 1 samples [reqs]
1415 getattr 3 samples [reqs]
1416 llog_init 6 samples [reqs]
1417 notify 16 samples [reqs]
git://git.whamcloud.com / doc / manual.git / blob