1 <?xml version='1.0' encoding='UTF-8'?>
2 <chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0"
3 xml:lang="en-US" xml:id="lustreproc">
4 <title xml:id="lustreproc.title">LustreProc</title>
5 <para>The <literal>/proc</literal> file system acts as an interface to internal data structures in
6 the kernel. This chapter describes entries in <literal>/proc</literal> that are useful for
7 tuning and monitoring aspects of a Lustre file system. It includes these sections:</para>
10 <para><xref linkend="dbdoclet.50438271_83523"/></para>
15 <title>Introduction to <literal>/proc</literal></title>
16 <para>The <literal>/proc</literal> directory provides an interface to internal data structures
17 in the kernel that enables monitoring and tuning of many aspects of Lustre file system and
18 application performance These data structures include settings and metrics for components such
19 as memory, networking, file systems, and kernel housekeeping routines, which are available
20 throughout the hierarchical file layout in <literal>/proc.</literal>
22 <para>Typically, metrics are accessed by reading from <literal>/proc</literal> files and
23 settings are changed by writing to <literal>/proc</literal> files. Some data is server-only,
24 some data is client-only, and some data is exported from the client to the server and is thus
25 duplicated in both locations.</para>
27 <para>In the examples in this chapter, <literal>#</literal> indicates a command is entered as
28 root. Servers are named according to the convention
29 <literal><replaceable>fsname</replaceable>-<replaceable>MDT|OSTnumber</replaceable></literal>.
30 The standard UNIX wildcard designation (*) is used.</para>
32 <para>In most cases, information is accessed using the <literal>lctl get_param</literal> command
33 and settings are changed using the <literal>lctl set_param</literal> command. Some examples
34 are shown below:</para>
37 <para> To obtain data from a Lustre client:</para>
38 <screen># lctl list_param osc.*
39 osc.testfs-OST0000-osc-ffff881071d5cc00
40 osc.testfs-OST0001-osc-ffff881071d5cc00
41 osc.testfs-OST0002-osc-ffff881071d5cc00
42 osc.testfs-OST0003-osc-ffff881071d5cc00
43 osc.testfs-OST0004-osc-ffff881071d5cc00
44 osc.testfs-OST0005-osc-ffff881071d5cc00
45 osc.testfs-OST0006-osc-ffff881071d5cc00
46 osc.testfs-OST0007-osc-ffff881071d5cc00
47 osc.testfs-OST0008-osc-ffff881071d5cc00</screen>
48 <para>In this example, information about OST connections available on a client is displayed
49 (indicated by "osc").</para>
54 <para> To see multiple levels of parameters, use multiple
55 wildcards:<screen># lctl list_param osc.*.*
56 osc.testfs-OST0000-osc-ffff881071d5cc00.active
57 osc.testfs-OST0000-osc-ffff881071d5cc00.blocksize
58 osc.testfs-OST0000-osc-ffff881071d5cc00.checksum_type
59 osc.testfs-OST0000-osc-ffff881071d5cc00.checksums
60 osc.testfs-OST0000-osc-ffff881071d5cc00.connect_flags
61 osc.testfs-OST0000-osc-ffff881071d5cc00.contention_seconds
62 osc.testfs-OST0000-osc-ffff881071d5cc00.cur_dirty_bytes
64 osc.testfs-OST0000-osc-ffff881071d5cc00.rpc_stats</screen></para>
69 <para> To view a specific file, use <literal>lctl get_param</literal>
70 :<screen># lctl get_param osc.lustre-OST0000-osc-ffff881071d5cc00.rpc_stats</screen></para>
73 <para>For more information about using <literal>lctl</literal>, see <xref
74 xmlns:xlink="http://www.w3.org/1999/xlink" linkend="dbdoclet.50438194_51490"/>.</para>
75 <para>Data can also be viewed using the <literal>cat</literal> command with the full path to the
76 file. The form of the <literal>cat</literal> command is similar to that of the <literal>lctl
77 get_param</literal> command with these differences. In the <literal>cat</literal> command: </para>
80 <para> Replace the dots in the path with slashes.</para>
83 <para> Prepend the path with the following as
84 appropriate:<screen>/proc/{fs,sys}/{lustre,lnet}</screen></para>
87 <para>For example, an <literal>lctl get_param</literal> command may look like
88 this:<screen># lctl get_param osc.*.uuid
89 osc.testfs-OST0000-osc-ffff881071d5cc00.uuid=594db456-0685-bd16-f59b-e72ee90e9819
90 osc.testfs-OST0001-osc-ffff881071d5cc00.uuid=594db456-0685-bd16-f59b-e72ee90e9819
92 <para>The equivalent <literal>cat</literal> command looks like
93 this:<screen># cat /proc/fs/lustre/osc/*/uuid
94 594db456-0685-bd16-f59b-e72ee90e9819
95 594db456-0685-bd16-f59b-e72ee90e9819
97 <para>The <literal>llstat</literal> utility can be used to monitor some Lustre file system I/O
98 activity over a specified time period. For more details, see <xref
99 xmlns:xlink="http://www.w3.org/1999/xlink" linkend="dbdoclet.50438219_23232"/></para>
100 <para>Some data is imported from attached clients and is available in a directory called
101 <literal>exports</literal> located in the corresponding per-service directory on a Lustre
103 example:<screen># ls /proc/fs/lustre/obdfilter/testfs-OST0000/exports/192.168.124.9\@o2ib1/
104 # hash ldlm_stats stats uuid</screen></para>
106 <title>Identifying Lustre File Systems and Servers</title>
107 <para>Several <literal>/proc</literal> files on the MGS list existing Lustre file systems and
108 file system servers. The examples below are for a Lustre file system called
109 <literal>testfs</literal> with one MDT and three OSTs.</para>
112 <para> To view all known Lustre file systems, enter:</para>
113 <screen>mgs# lctl get_param mgs.*.filesystems
117 <para> To view the names of the servers in a file system in which least one server is
119 enter:<screen>lctl get_param mgs.*.live.<replaceable><filesystem name></replaceable></screen></para>
120 <para>For example:</para>
121 <screen>mgs# lctl get_param mgs.*.live.testfs
129 Secure RPC Config Rules:
131 imperative_recovery_state:
135 notify_duration_total: 0.001000
136 notify_duation_max: 0.001000
137 notify_count: 4</screen>
140 <para>To view the names of all live servers in the file system as listed in
141 <literal>/proc/fs/lustre/devices</literal>, enter:</para>
142 <screen># lctl device_list
144 1 UP mgc MGC192.168.10.34@tcp 1f45bb57-d9be-2ddb-c0b0-5431a49226705
145 2 UP mdt MDS MDS_uuid 3
146 3 UP lov testfs-mdtlov testfs-mdtlov_UUID 4
147 4 UP mds testfs-MDT0000 testfs-MDT0000_UUID 7
148 5 UP osc testfs-OST0000-osc testfs-mdtlov_UUID 5
149 6 UP osc testfs-OST0001-osc testfs-mdtlov_UUID 5
150 7 UP lov testfs-clilov-ce63ca00 08ac6584-6c4a-3536-2c6d-b36cf9cbdaa04
151 8 UP mdc testfs-MDT0000-mdc-ce63ca00 08ac6584-6c4a-3536-2c6d-b36cf9cbdaa05
152 9 UP osc testfs-OST0000-osc-ce63ca00 08ac6584-6c4a-3536-2c6d-b36cf9cbdaa05
153 10 UP osc testfs-OST0001-osc-ce63ca00 08ac6584-6c4a-3536-2c6d-b36cf9cbdaa05</screen>
154 <para>The information provided on each line includes:</para>
155 <para> - Device number</para>
156 <para> - Device status (UP, INactive, or STopping) </para>
157 <para> - Device name</para>
158 <para> - Device UUID</para>
159 <para> - Reference count (how many users this device has)</para>
162 <para>To display the name of any server, view the device
163 label:<screen>mds# e2label /dev/sda
164 testfs-MDT0000</screen></para>
170 <title>Tuning Multi-Block Allocation (mballoc)</title>
171 <para>Capabilities supported by <literal>mballoc</literal> include:</para>
174 <para> Pre-allocation for single files to help to reduce fragmentation.</para>
177 <para> Pre-allocation for a group of files to enable packing of small files into large,
178 contiguous chunks.</para>
181 <para> Stream allocation to help decrease the seek rate.</para>
184 <para>The following <literal>mballoc</literal> tunables are available:</para>
185 <informaltable frame="all">
187 <colspec colname="c1" colwidth="30*"/>
188 <colspec colname="c2" colwidth="70*"/>
192 <para><emphasis role="bold">Field</emphasis></para>
195 <para><emphasis role="bold">Description</emphasis></para>
203 <literal>mb_max_to_scan</literal></para>
206 <para>Maximum number of free chunks that <literal>mballoc</literal> finds before a
207 final decision to avoid a livelock situation.</para>
213 <literal>mb_min_to_scan</literal></para>
216 <para>Minimum number of free chunks that <literal>mballoc</literal> searches before
217 picking the best chunk for allocation. This is useful for small requests to reduce
218 fragmentation of big free chunks.</para>
224 <literal>mb_order2_req</literal></para>
227 <para>For requests equal to 2^N, where N >= <literal>mb_order2_req</literal>, a
228 fast search is done using a base 2 buddy allocation service.</para>
234 <literal>mb_small_req</literal></para>
237 <para><literal>mb_small_req</literal> - Defines (in MB) the upper bound of "small
239 <para><literal>mb_large_req</literal> - Defines (in MB) the lower bound of "large
241 <para>Requests are handled differently based on size:<itemizedlist>
243 <para>< <literal>mb_small_req</literal> - Requests are packed together to
244 form large, aggregated requests.</para>
247 <para>> <literal>mb_small_req</literal> and < <literal>mb_large_req</literal>
248 - Requests are primarily allocated linearly.</para>
251 <para>> <literal>mb_large_req</literal> - Requests are allocated since hard disk
252 seek time is less of a concern in this case.</para>
254 </itemizedlist></para>
255 <para>In general, small requests are combined to create larger requests, which are
256 then placed close to one another to minimize the number of seeks required to access
263 <literal>mb_large_req</literal></para>
269 <literal>mb_prealloc_table</literal></para>
272 <para>A table of values used to preallocate space when a new request is received. By
273 default, the table looks like
274 this:<screen>prealloc_table
275 4 8 16 32 64 128 256 512 1024 2048 </screen></para>
276 <para>When a new request is received, space is preallocated at the next higher
277 increment specified in the table. For example, for requests of less than 4 file
278 system blocks, 4 blocks of space are preallocated; for requests between 4 and 8, 8
279 blocks are preallocated; and so forth</para>
280 <para>Although customized values can be entered in the table, the performance of
281 general usage file systems will not typically be improved by modifying the table (in
282 fact, in ext4 systems, the table values are fixed). However, for some specialized
283 workloads, tuning the <literal>prealloc_table</literal> values may result in smarter
284 preallocation decisions. </para>
290 <literal>mb_group_prealloc</literal></para>
293 <para>The amount of space (in kilobytes) preallocated for groups of small
300 <para>Buddy group cache information found in
301 <literal>/proc/fs/ldiskfs/<replaceable>disk_device</replaceable>/mb_groups</literal> may
302 be useful for assessing on-disk fragmentation. For
303 example:<screen>cat /proc/fs/ldiskfs/loop0/mb_groups
304 #group: free free frags first pa [ 2^0 2^1 2^2 2^3 2^4 2^5 2^6 2^7 2^8 2^9
306 #0 : 2936 2936 1 42 0 [ 0 0 0 1 1 1 1 2 0 1
307 2 0 0 0 ]</screen></para>
308 <para>In this example, the columns show:<itemizedlist>
310 <para>#group number</para>
313 <para>Available blocks in the group</para>
316 <para>Blocks free on a disk</para>
319 <para>Number of free fragments</para>
322 <para>First free block in the group</para>
325 <para>Number of preallocated chunks (not blocks)</para>
328 <para>A series of available chunks of different sizes</para>
330 </itemizedlist></para>
333 <title>Monitoring Lustre File System I/O</title>
334 <para>A number of system utilities are provided to enable collection of data related to I/O
335 activity in a Lustre file system. In general, the data collected describes:</para>
338 <para> Data transfer rates and throughput of inputs and outputs external to the Lustre file
339 system, such as network requests or disk I/O operations performed</para>
342 <para> Data about the throughput or transfer rates of internal Lustre file system data, such
343 as locks or allocations. </para>
347 <para>It is highly recommended that you complete baseline testing for your Lustre file system
348 to determine normal I/O activity for your hardware, network, and system workloads. Baseline
349 data will allow you to easily determine when performance becomes degraded in your system.
350 Two particularly useful baseline statistics are:</para>
353 <para><literal>brw_stats</literal> – Histogram data characterizing I/O requests to the
354 OSTs. For more details, see <xref xmlns:xlink="http://www.w3.org/1999/xlink"
355 linkend="dbdoclet.50438271_55057"/>.</para>
358 <para><literal>rpc_stats</literal> – Histogram data showing information about RPCs made by
359 clients. For more details, see <xref xmlns:xlink="http://www.w3.org/1999/xlink"
360 linkend="MonitoringClientRCPStream"/>.</para>
364 <section remap="h3" xml:id="MonitoringClientRCPStream">
366 <primary>proc</primary>
367 <secondary>watching RPC</secondary>
368 </indexterm>Monitoring the Client RPC Stream</title>
369 <para>The <literal>rpc_stats</literal> file contains histogram data showing information about
370 remote procedure calls (RPCs) that have been made since this file was last cleared. The
371 histogram data can be cleared by writing any value into the <literal>rpc_stats</literal>
373 <para><emphasis role="italic"><emphasis role="bold">Example:</emphasis></emphasis></para>
374 <screen># lctl get_param osc.testfs-OST0000-osc-ffff810058d2f800.rpc_stats
375 snapshot_time: 1372786692.389858 (secs.usecs)
376 read RPCs in flight: 0
377 write RPCs in flight: 1
378 dio read RPCs in flight: 0
379 dio write RPCs in flight: 0
380 pending write pages: 256
381 pending read pages: 0
384 pages per rpc rpcs % cum % | rpcs % cum %
393 256: 850 100 100 | 18346 99 100
396 rpcs in flight rpcs % cum % | rpcs % cum %
397 0: 691 81 81 | 1740 9 9
398 1: 48 5 86 | 938 5 14
399 2: 29 3 90 | 1059 5 20
400 3: 17 2 92 | 1052 5 26
401 4: 13 1 93 | 920 5 31
402 5: 12 1 95 | 425 2 33
403 6: 10 1 96 | 389 2 35
404 7: 30 3 100 | 11373 61 97
405 8: 0 0 100 | 460 2 100
408 offset rpcs % cum % | rpcs % cum %
409 0: 850 100 100 | 18347 99 99
417 128: 0 0 100 | 4 0 100
420 <para>The header information includes:</para>
423 <para><literal>snapshot_time</literal> - UNIX* epoch instant the file was read.</para>
426 <para><literal>read RPCs in flight</literal> - Number of read RPCs issued by the OSC, but
427 not complete at the time of the snapshot. This value should always be less than or equal
428 to <literal>max_rpcs_in_flight</literal>.</para>
431 <para><literal>write RPCs in flight</literal> - Number of write RPCs issued by the OSC,
432 but not complete at the time of the snapshot. This value should always be less than or
433 equal to <literal>max_rpcs_in_flight</literal>.</para>
436 <para><literal>dio read RPCs in flight</literal> - Direct I/O (as opposed to block I/O)
437 read RPCs issued but not completed at the time of the snapshot.</para>
440 <para><literal>dio write RPCs in flight</literal> - Direct I/O (as opposed to block I/O)
441 write RPCs issued but not completed at the time of the snapshot.</para>
444 <para><literal>pending write pages</literal> - Number of pending write pages that have
445 been queued for I/O in the OSC.</para>
448 <para><literal>pending read pages</literal> - Number of pending read pages that have been
449 queued for I/O in the OSC.</para>
452 <para>The tabular data is described in the table below. Each row in the table shows the number
453 of reads or writes (<literal>ios</literal>) occurring for the statistic, the relative
454 percentage (<literal>%</literal>) of total reads or writes, and the cumulative percentage
455 (<literal>cum %</literal>) to that point in the table for the statistic.</para>
456 <informaltable frame="all">
458 <colspec colname="c1" colwidth="40*"/>
459 <colspec colname="c2" colwidth="60*"/>
463 <para><emphasis role="bold">Field</emphasis></para>
466 <para><emphasis role="bold">Description</emphasis></para>
473 <para> pages per RPC</para>
476 <para>Shows cumulative RPC reads and writes organized according to the number of
477 pages in the RPC. A single page RPC increments the <literal>0:</literal>
483 <para> RPCs in flight</para>
486 <para> Shows the number of RPCs that are pending when an RPC is sent. When the first
487 RPC is sent, the <literal>0:</literal> row is incremented. If the first RPC is
488 sent while another RPC is pending, the <literal>1:</literal> row is incremented
497 <para> The page index of the first page read from or written to the object by the
504 <para><emphasis role="italic"><emphasis role="bold">Analysis:</emphasis></emphasis></para>
505 <para>This table provides a way to visualize the concurrency of the RPC stream. Ideally, you
506 will see a large clump around the <literal>max_rpcs_in_flight value</literal>, which shows
507 that the network is being kept busy.</para>
508 <para>For information about optimizing the client I/O RPC stream, see <xref
509 xmlns:xlink="http://www.w3.org/1999/xlink" linkend="TuningClientIORPCStream"/>.</para>
513 <primary>proc</primary>
514 <secondary>read/write survey</secondary>
515 </indexterm>Monitoring Client Read-Write Offset Statistics</title>
516 <para>The <literal>offset_stats</literal> parameter maintains statistics for occurrences of a
517 series of read or write calls from a process that did not access the next sequential
518 location. The <literal>OFFSET</literal> field is reset to 0 (zero) whenever a different file
519 is read or written.</para>
520 <para>Read/write offset statistics are "off" by default. The statistics can be activated by
521 writing anything into the <literal>offset_stats</literal> file.</para>
522 <para>The <literal>offset_stats</literal> file can be cleared by
523 entering:<screen>lctl set_param llite.*.offset_stats=0</screen></para>
524 <para><emphasis role="italic"><emphasis role="bold">Example:</emphasis></emphasis></para>
525 <screen># lctl get_param llite.testfs-f57dee0.offset_stats
526 snapshot_time: 1155748884.591028 (secs.usecs)
527 RANGE RANGE SMALLEST LARGEST
528 R/W PID START END EXTENT EXTENT OFFSET
529 R 8385 0 128 128 128 0
530 R 8385 0 224 224 224 -128
531 W 8385 0 250 50 100 0
532 W 8385 100 1110 10 500 -150
533 W 8384 0 5233 5233 5233 0
534 R 8385 500 600 100 100 -610</screen>
535 <para>In this example, <literal>snapshot_time</literal> is the UNIX epoch instant the file was
536 read. The tabular data is described in the table below.</para>
537 <informaltable frame="all">
539 <colspec colname="c1" colwidth="50*"/>
540 <colspec colname="c2" colwidth="50*"/>
544 <para><emphasis role="bold">Field</emphasis></para>
547 <para><emphasis role="bold">Description</emphasis></para>
557 <para>Indicates if the non-sequential call was a read or write</para>
565 <para>Process ID of the process that made the read/write call.</para>
570 <para>RANGE START/RANGE END</para>
573 <para>Range in which the read/write calls were sequential.</para>
578 <para>SMALLEST EXTENT </para>
581 <para>Smallest single read/write in the corresponding range (in bytes).</para>
586 <para>LARGEST EXTENT </para>
589 <para>Largest single read/write in the corresponding range (in bytes).</para>
597 <para>Difference between the previous range end and the current range start.</para>
603 <para><emphasis role="italic"><emphasis role="bold">Analysis:</emphasis></emphasis></para>
604 <para>This data provides an indication of how contiguous or fragmented the data is. For
605 example, the fourth entry in the example above shows the writes for this RPC were sequential
606 in the range 100 to 1110 with the minimum write 10 bytes and the maximum write 500 bytes.
607 The range started with an offset of -150 from the <literal>RANGE END</literal> of the
608 previous entry in the example.</para>
610 <section xml:id="lustreproc.clientstats" remap="h3">
612 <primary>proc</primary>
613 <secondary>client stats</secondary>
614 </indexterm>Monitoring Client Activity</title>
615 <para>The <literal>stats</literal> file maintains statistics accumulate during typical
616 operation of a client across the VFS interface of the Lustre file system. Only non-zero
617 parameters are displayed in the file. </para>
618 <para>Client statistics are enabled by default. The statistics can be cleared by echoing an
619 empty string into the <literal>stats</literal> file or by using the command:
620 <screen>lctl set_param llite.*.stats=0</screen></para>
622 <para>Statistics for all mounted file systems can be discovered by
623 entering:<screen>lctl get_param llite.*.stats</screen></para>
625 <para><emphasis role="italic"><emphasis role="bold">Example:</emphasis></emphasis></para>
626 <screen>client# lctl get_param llite.*.stats
627 snapshot_time 1308343279.169704 secs.usecs
628 dirty_pages_hits 14819716 samples [regs]
629 dirty_pages_misses 81473472 samples [regs]
630 read_bytes 36502963 samples [bytes] 1 26843582 55488794
631 write_bytes 22985001 samples [bytes] 0 125912 3379002
632 brw_read 2279 samples [pages] 1 1 2270
633 ioctl 186749 samples [regs]
634 open 3304805 samples [regs]
635 close 3331323 samples [regs]
636 seek 48222475 samples [regs]
637 fsync 963 samples [regs]
638 truncate 9073 samples [regs]
639 setxattr 19059 samples [regs]
640 getxattr 61169 samples [regs]
642 <para>The statistics displayed are described in the table below.</para>
643 <informaltable frame="all">
645 <colspec colname="c1" colwidth="3*"/>
646 <colspec colname="c2" colwidth="7*"/>
650 <para><emphasis role="bold">Entry</emphasis></para>
653 <para><emphasis role="bold">Description</emphasis></para>
661 <literal>snapshot_time</literal></para>
664 <para>UNIX epoch instant the stats file was read.</para>
670 <literal>dirty_page_hits</literal></para>
673 <para>The number of write operations that have been satisfied by the dirty page
674 cache. See <xref xmlns:xlink="http://www.w3.org/1999/xlink"
675 linkend="TuningClientIORPCStream"/> for more information about dirty cache
676 behavior in a Lustre file system.</para>
682 <literal>dirty_page_misses</literal></para>
685 <para>The number of write operations that were not satisfied by the dirty page
692 <literal>read_bytes</literal></para>
695 <para>The number of read operations that have occurred. Three additional parameters
696 are displayed:</para>
701 <para>The minimum number of bytes read in a single request since the counter
708 <para>The maximum number of bytes read in a single request since the counter
715 <para>The accumulated sum of bytes of all read requests since the counter was
725 <literal>write_bytes</literal></para>
728 <para>The number of write operations that have occurred. Three additional parameters
729 are displayed:</para>
734 <para>The minimum number of bytes written in a single request since the
735 counter was reset.</para>
741 <para>The maximum number of bytes written in a single request since the
742 counter was reset.</para>
748 <para>The accumulated sum of bytes of all write requests since the counter was
758 <literal>brw_read</literal></para>
761 <para>The number of pages that have been read. Three additional parameters are
767 <para>The minimum number of bytes read in a single block read/write
768 (<literal>brw</literal>) read request since the counter was reset.</para>
774 <para>The maximum number of bytes read in a single <literal>brw</literal> read
775 requests since the counter was reset.</para>
781 <para>The accumulated sum of bytes of all <literal>brw</literal> read requests
782 since the counter was reset.</para>
791 <literal>ioctl</literal></para>
794 <para>The number of combined file and directory <literal>ioctl</literal>
801 <literal>open</literal></para>
804 <para>The number of open operations that have succeeded.</para>
810 <literal>close</literal></para>
813 <para>The number of close operations that have succeeded.</para>
819 <literal>seek</literal></para>
822 <para>The number of times <literal>seek</literal> has been called.</para>
828 <literal>fsync</literal></para>
831 <para>The number of times <literal>fsync</literal> has been called.</para>
837 <literal>truncate</literal></para>
840 <para>The total number of calls to both locked and lockless
841 <literal>truncate</literal>.</para>
847 <literal>setxattr</literal></para>
850 <para>The number of times extended attributes have been set. </para>
856 <literal>getxattr</literal></para>
859 <para>The number of times value(s) of extended attributes have been fetched.</para>
865 <para><emphasis role="italic"><emphasis role="bold">Analysis:</emphasis></emphasis></para>
866 <para>Information is provided about the amount and type of I/O activity is taking place on the
871 <primary>proc</primary>
872 <secondary>read/write survey</secondary>
873 </indexterm>Monitoring Client Read-Write Extent Statistics</title>
874 <para>For in-depth troubleshooting, client read-write extent statistics can be accessed to
875 obtain more detail about read/write I/O extents for the file system or for a particular
878 <title>Client-Based I/O Extent Size Survey</title>
879 <para>The <literal>rw_extent_stats</literal> histogram in the <literal>llite</literal>
880 directory shows the statistics for the sizes of the read?write I/O extents. This file does
881 not maintain the per-process statistics. The file can be cleared by issuing the following
882 command:<screen># lctl set_param llite.testfs-*.extents_stats=0</screen></para>
883 <para><emphasis role="italic"><emphasis role="bold">Example:</emphasis></emphasis></para>
884 <screen># lctl get_param llite.testfs-*.extents_stats
885 snapshot_time: 1213828728.348516 (secs.usecs)
887 extents calls % cum% | calls % cum%
889 0K - 4K : 0 0 0 | 2 2 2
890 4K - 8K : 0 0 0 | 0 0 2
891 8K - 16K : 0 0 0 | 0 0 2
892 16K - 32K : 0 0 0 | 20 23 26
893 32K - 64K : 0 0 0 | 0 0 26
894 64K - 128K : 0 0 0 | 51 60 86
895 128K - 256K : 0 0 0 | 0 0 86
896 256K - 512K : 0 0 0 | 0 0 86
897 512K - 1024K : 0 0 0 | 0 0 86
898 1M - 2M : 0 0 0 | 11 13 100</screen>
899 <para>In this example, <literal>snapshot_time</literal> is the UNIX epoch instant the file
900 was read. The table shows cumulative extents organized according to size with statistics
901 provided separately for reads and writes. Each row in the table shows the number of RPCs
902 for reads and writes respectively (<literal>calls</literal>), the relative percentage of
903 total calls (<literal>%</literal>), and the cumulative percentage to that point in the
904 table of calls (<literal>cum %</literal>). </para>
907 <title>Per-Process Client I/O Statistics</title>
908 <para>The <literal>extents_stats_per_process</literal> file maintains the I/O extent size
909 statistics on a per-process basis.</para>
910 <para><emphasis role="italic"><emphasis role="bold">Example:</emphasis></emphasis></para>
911 <screen># lctl get_param llite.testfs-*.extents_stats_per_process
912 snapshot_time: 1213828762.204440 (secs.usecs)
914 extents calls % cum% | calls % cum%
917 0K - 4K : 0 0 0 | 0 0 0
918 4K - 8K : 0 0 0 | 0 0 0
919 8K - 16K : 0 0 0 | 0 0 0
920 16K - 32K : 0 0 0 | 0 0 0
921 32K - 64K : 0 0 0 | 0 0 0
922 64K - 128K : 0 0 0 | 0 0 0
923 128K - 256K : 0 0 0 | 0 0 0
924 256K - 512K : 0 0 0 | 0 0 0
925 512K - 1024K : 0 0 0 | 0 0 0
926 1M - 2M : 0 0 0 | 10 100 100
929 0K - 4K : 0 0 0 | 0 0 0
930 4K - 8K : 0 0 0 | 0 0 0
931 8K - 16K : 0 0 0 | 0 0 0
932 16K - 32K : 0 0 0 | 20 100 100
935 0K - 4K : 0 0 0 | 0 0 0
936 4K - 8K : 0 0 0 | 0 0 0
937 8K - 16K : 0 0 0 | 0 0 0
938 16K - 32K : 0 0 0 | 0 0 0
939 32K - 64K : 0 0 0 | 0 0 0
940 64K - 128K : 0 0 0 | 16 100 100
943 0K - 4K : 0 0 0 | 1 100 100
946 0K - 4K : 0 0 0 | 1 100 100
949 <para>This table shows cumulative extents organized according to size for each process ID
950 (PID) with statistics provided separately for reads and writes. Each row in the table
951 shows the number of RPCs for reads and writes respectively (<literal>calls</literal>), the
952 relative percentage of total calls (<literal>%</literal>), and the cumulative percentage
953 to that point in the table of calls (<literal>cum %</literal>). </para>
956 <section xml:id="dbdoclet.50438271_55057">
958 <primary>proc</primary>
959 <secondary>block I/O</secondary>
960 </indexterm>Monitoring the OST Block I/O Stream</title>
961 <para>The <literal>brw_stats</literal> file in the <literal>obdfilter</literal> directory
962 contains histogram data showing statistics for number of I/O requests sent to the disk,
963 their size, and whether they are contiguous on the disk or not.</para>
964 <para><emphasis role="italic"><emphasis role="bold">Example:</emphasis></emphasis></para>
965 <para>Enter on the OSS:</para>
966 <screen># lctl get_param obdfilter.testfs-OST0000.brw_stats
967 snapshot_time: 1372775039.769045 (secs.usecs)
969 pages per bulk r/w rpcs % cum % | rpcs % cum %
970 1: 108 100 100 | 39 0 0
977 128: 0 0 100 | 24 0 0
978 256: 0 0 100 | 23142 99 100
981 discontiguous pages rpcs % cum % | rpcs % cum %
982 0: 108 100 100 | 23245 100 100
985 discontiguous blocks rpcs % cum % | rpcs % cum %
986 0: 108 100 100 | 23243 99 99
990 disk fragmented I/Os ios % cum % | ios % cum %
992 1: 14 12 100 | 23243 99 99
996 disk I/Os in flight ios % cum % | ios % cum %
997 1: 14 100 100 | 20896 89 89
998 2: 0 0 100 | 1071 4 94
999 3: 0 0 100 | 573 2 96
1000 4: 0 0 100 | 300 1 98
1001 5: 0 0 100 | 166 0 98
1002 6: 0 0 100 | 108 0 99
1003 7: 0 0 100 | 81 0 99
1004 8: 0 0 100 | 47 0 99
1005 9: 0 0 100 | 5 0 100
1008 I/O time (1/1000s) ios % cum % | ios % cum %
1011 4: 14 12 100 | 27 0 0
1013 16: 0 0 100 | 31 0 0
1014 32: 0 0 100 | 38 0 0
1015 64: 0 0 100 | 18979 81 82
1016 128: 0 0 100 | 943 4 86
1017 256: 0 0 100 | 1233 5 91
1018 512: 0 0 100 | 1825 7 99
1019 1K: 0 0 100 | 99 0 99
1020 2K: 0 0 100 | 0 0 99
1021 4K: 0 0 100 | 0 0 99
1022 8K: 0 0 100 | 49 0 100
1025 disk I/O size ios % cum % | ios % cum %
1026 4K: 14 100 100 | 41 0 0
1028 16K: 0 0 100 | 1 0 0
1029 32K: 0 0 100 | 0 0 0
1030 64K: 0 0 100 | 4 0 0
1031 128K: 0 0 100 | 17 0 0
1032 256K: 0 0 100 | 12 0 0
1033 512K: 0 0 100 | 24 0 0
1034 1M: 0 0 100 | 23142 99 100
1036 <para>The tabular data is described in the table below. Each row in the table shows the number
1037 of reads and writes occurring for the statistic (<literal>ios</literal>), the relative
1038 percentage of total reads or writes (<literal>%</literal>), and the cumulative percentage to
1039 that point in the table for the statistic (<literal>cum %</literal>). </para>
1040 <informaltable frame="all">
1042 <colspec colname="c1" colwidth="40*"/>
1043 <colspec colname="c2" colwidth="60*"/>
1047 <para><emphasis role="bold">Field</emphasis></para>
1050 <para><emphasis role="bold">Description</emphasis></para>
1058 <literal>pages per bulk r/w</literal></para>
1061 <para>Number of pages per RPC request, which should match aggregate client
1062 <literal>rpc_stats</literal> (see <xref
1063 xmlns:xlink="http://www.w3.org/1999/xlink" linkend="MonitoringClientRCPStream"
1070 <literal>discontiguous pages</literal></para>
1073 <para>Number of discontinuities in the logical file offset of each page in a single
1080 <literal>discontiguous blocks</literal></para>
1083 <para>Number of discontinuities in the physical block allocation in the file system
1084 for a single RPC.</para>
1089 <para><literal>disk fragmented I/Os</literal></para>
1092 <para>Number of I/Os that were not written entirely sequentially.</para>
1097 <para><literal>disk I/Os in flight</literal></para>
1100 <para>Number of disk I/Os currently pending.</para>
1105 <para><literal>I/O time (1/1000s)</literal></para>
1108 <para>Amount of time for each I/O operation to complete.</para>
1113 <para><literal>disk I/O size</literal></para>
1116 <para>Size of each I/O operation.</para>
1122 <para><emphasis role="italic"><emphasis role="bold">Analysis:</emphasis></emphasis></para>
1123 <para>This data provides an indication of extent size and distribution in the file
1128 <title>Tuning Lustre File System I/O</title>
1129 <para>Each OSC has its own tree of tunables. For example:</para>
1130 <screen>$ ls -d /proc/fs/testfs/osc/OSC_client_ost1_MNT_client_2 /localhost
1131 /proc/fs/testfs/osc/OSC_uml0_ost1_MNT_localhost
1132 /proc/fs/testfs/osc/OSC_uml0_ost2_MNT_localhost
1133 /proc/fs/testfs/osc/OSC_uml0_ost3_MNT_localhost
1135 $ ls /proc/fs/testfs/osc/OSC_uml0_ost1_MNT_localhost
1136 blocksizefilesfree max_dirty_mb ost_server_uuid stats
1139 <para>The following sections describe some of the parameters that can be tuned in a Lustre file
1141 <section remap="h3" xml:id="TuningClientIORPCStream">
1143 <primary>proc</primary>
1144 <secondary>RPC tunables</secondary>
1145 </indexterm>Tuning the Client I/O RPC Stream</title>
1146 <para>Ideally, an optimal amount of data is packed into each I/O RPC and a consistent number
1147 of issued RPCs are in progress at any time. To help optimize the client I/O RPC stream,
1148 several tuning variables are provided to adjust behavior according to network conditions and
1149 cluster size. For information about monitoring the client I/O RPC stream, see <xref
1150 xmlns:xlink="http://www.w3.org/1999/xlink" linkend="MonitoringClientRCPStream"/>.</para>
1151 <para>RPC stream tunables include:</para>
1155 <para><literal>osc.<replaceable>osc_instance</replaceable>.max_dirty_mb</literal> -
1156 Controls how many MBs of dirty data can be written and queued up in the OSC. POSIX
1157 file writes that are cached contribute to this count. When the limit is reached,
1158 additional writes stall until previously-cached writes are written to the server. This
1159 may be changed by writing a single ASCII integer to the file. Only values between 0
1160 and 2048 or 1/4 of RAM are allowable. If 0 is specified, no writes are cached.
1161 Performance suffers noticeably unless you use large writes (1 MB or more).</para>
1162 <para>To maximize performance, the value for <literal>max_dirty_mb</literal> is
1163 recommended to be 4 * <literal>max_pages_per_rpc </literal>*
1164 <literal>max_rpcs_in_flight</literal>.</para>
1167 <para><literal>osc.<replaceable>osc_instance</replaceable>.cur_dirty_bytes</literal> - A
1168 read-only value that returns the current number of bytes written and cached on this
1172 <para><literal>osc.<replaceable>osc_instance</replaceable>.max_pages_per_rpc</literal> -
1173 The maximum number of pages that will undergo I/O in a single RPC to the OST. The
1174 minimum setting is a single page and the maximum setting is 1024 (for systems with a
1175 <literal>PAGE_SIZE</literal> of 4 KB), with the default maximum of 1 MB in the RPC.
1176 It is also possible to specify a units suffix (e.g. <literal>4M</literal>), so that
1177 the RPC size can be specified independently of the client
1178 <literal>PAGE_SIZE</literal>.</para>
1181 <para><literal>osc.<replaceable>osc_instance</replaceable>.max_rpcs_in_flight</literal>
1182 - The maximum number of concurrent RPCs in flight from an OSC to its OST. If the OSC
1183 tries to initiate an RPC but finds that it already has the same number of RPCs
1184 outstanding, it will wait to issue further RPCs until some complete. The minimum
1185 setting is 1 and maximum setting is 256. </para>
1186 <para>To improve small file I/O performance, increase the
1187 <literal>max_rpcs_in_flight</literal> value.</para>
1190 <para><literal>llite.<replaceable>fsname-instance</replaceable>/max_cache_mb</literal> -
1191 Maximum amount of inactive data cached by the client (default is 3/4 of RAM). For
1193 <screen># lctl get_param llite.testfs-ce63ca00.max_cached_mb
1199 <para>The value for <literal><replaceable>osc_instance</replaceable></literal> is typically
1200 <literal><replaceable>fsname</replaceable>-OST<replaceable>ost_index</replaceable>-osc-<replaceable>mountpoint_instance</replaceable></literal>,
1201 where the value for <literal><replaceable>mountpoint_instance</replaceable></literal> is
1202 unique to each mount point to allow associating osc, mdc, lov, lmv, and llite parameters
1203 with the same mount point. For
1204 example:<screen>lctl get_param osc.testfs-OST0000-osc-ffff88107412f400.rpc_stats
1205 osc.testfs-OST0000-osc-ffff88107412f400.rpc_stats=
1206 snapshot_time: 1375743284.337839 (secs.usecs)
1207 read RPCs in flight: 0
1208 write RPCs in flight: 0
1212 <section remap="h3">
1214 <primary>proc</primary>
1215 <secondary>readahead</secondary>
1216 </indexterm>Tuning File Readahead and Directory Statahead</title>
1217 <para>File readahead and directory statahead enable reading of data into memory before a
1218 process requests the data. File readahead reads file content data into memory and directory
1219 statahead reads metadata into memory. When readahead and statahead work well, a process that
1220 accesses data finds that the information it needs is available immediately when requested in
1221 memory without the delay of network I/O.</para>
1222 <para condition="l22">In Lustre release 2.2.0, the directory statahead feature was improved to
1223 enhance directory traversal performance. The improvements primarily addressed two
1227 <para>A race condition existed between the statahead thread and other VFS operations while
1228 processing asynchronous <literal>getattr</literal> RPC replies, causing duplicate
1229 entries in dcache. This issue was resolved by using statahead local dcache. </para>
1232 <para>File size/block attributes pre-fetching was not supported, so the traversing thread
1233 had to send synchronous glimpse size RPCs to OST(s). This issue was resolved by using
1234 asynchronous glimpse lock (AGL) RPCs to pre-fetch file size/block attributes from
1239 <section remap="h4">
1240 <title>Tuning File Readahead</title>
1241 <para>File readahead is triggered when two or more sequential reads by an application fail
1242 to be satisfied by data in the Linux buffer cache. The size of the initial readahead is 1
1243 MB. Additional readaheads grow linearly and increment until the readahead cache on the
1244 client is full at 40 MB.</para>
1245 <para>Readahead tunables include:</para>
1248 <para><literal>llite.<replaceable>fsname-instance</replaceable>.max_read_ahead_mb</literal>
1249 - Controls the maximum amount of data readahead on a file. Files are read ahead in
1250 RPC-sized chunks (1 MB or the size of the <literal>read()</literal> call, if larger)
1251 after the second sequential read on a file descriptor. Random reads are done at the
1252 size of the <literal>read()</literal> call only (no readahead). Reads to
1253 non-contiguous regions of the file reset the readahead algorithm, and readahead is not
1254 triggered again until sequential reads take place again. </para>
1255 <para>To disable readahead, set this tunable to 0. The default value is 40 MB.</para>
1258 <para><literal>llite.<replaceable>fsname-instance</replaceable>.max_read_ahead_whole_mb</literal>
1259 - Controls the maximum size of a file that is read in its entirety, regardless of the
1260 size of the <literal>read()</literal>.</para>
1265 <title>Tuning Directory Statahead and AGL</title>
1266 <para>Many system commands, such as <literal>ls –l</literal>, <literal>du</literal>, and
1267 <literal>find</literal>, traverse a directory sequentially. To make these commands run
1268 efficiently, the directory statahead and asynchronous glimpse lock (AGL) can be enabled to
1269 improve the performance of traversing.</para>
1270 <para>The statahead tunables are:</para>
1273 <para><literal>statahead_max</literal> - Controls whether directory statahead is enabled
1274 and the maximum statahead window size (i.e., how many files can be pre-fetched by the
1275 statahead thread). By default, statahead is enabled and the value of
1276 <literal>statahead_max</literal> is 32.</para>
1277 <para>To disable statahead, run:</para>
1278 <screen>lctl set_param llite.*.statahead_max=0</screen>
1279 <para>To set the maximum statahead window size (<replaceable>n</replaceable>),
1281 <screen>lctl set_param llite.*.statahead_max=<replaceable>n</replaceable></screen>
1282 <para>The maximum value of <replaceable>n</replaceable> is 8192.</para>
1283 <para>The AGL can be controlled by entering:</para>
1284 <screen>lctl set_param llite.*.statahead_agl=<replaceable>n</replaceable></screen>
1285 <para>The default value for <replaceable>n</replaceable> is 1, which enables the AGL. If
1286 <replaceable>n</replaceable> is 0, the AGL is disabled.</para>
1289 <para><literal>statahead_stats</literal> - A read-only interface that indicates the
1290 current statahead and AGL statistics, such as how many times statahead/AGL has been
1291 triggered since the last mount, how many statahead/AGL failures have occurred due to
1292 an incorrect prediction or other causes.</para>
1294 <para>The AGL is affected by statahead because the inodes processed by AGL are built
1295 by the statahead thread, which means the statahead thread is the input of the AGL
1296 pipeline. So if statahead is disabled, then the AGL is disabled by force.</para>
1302 <section remap="h3">
1304 <primary>proc</primary>
1305 <secondary>read cache</secondary>
1306 </indexterm>Tuning OSS Read Cache</title>
1307 <para>The OSS read cache feature provides read-only caching of data on an OSS. This
1308 functionality uses the Linux page cache to store the data and uses as much physical memory
1309 as is allocated.</para>
1310 <para>OSS read cache improves Lustre file system performance in these situations:</para>
1313 <para>Many clients are accessing the same data set (as in HPC applications or when
1314 diskless clients boot from the Lustre file system).</para>
1317 <para>One client is storing data while another client is reading it (i.e., clients are
1318 exchanging data via the OST).</para>
1321 <para>A client has very limited caching of its own.</para>
1324 <para>OSS read cache offers these benefits:</para>
1327 <para>Allows OSTs to cache read data more frequently.</para>
1330 <para>Improves repeated reads to match network speeds instead of disk speeds.</para>
1333 <para>Provides the building blocks for OST write cache (small-write aggregation).</para>
1336 <section remap="h4">
1337 <title>Using OSS Read Cache</title>
1338 <para>OSS read cache is implemented on the OSS, and does not require any special support on
1339 the client side. Since OSS read cache uses the memory available in the Linux page cache,
1340 the appropriate amount of memory for the cache should be determined based on I/O patterns;
1341 if the data is mostly reads, then more cache is required than would be needed for mostly
1343 <para>OSS read cache is managed using the following tunables:</para>
1346 <para><literal>read_cache_enable</literal> - Controls whether data read from disk during
1347 a read request is kept in memory and available for later read requests for the same
1348 data, without having to re-read it from disk. By default, read cache is enabled
1349 (<literal>read_cache_enable=1</literal>).</para>
1350 <para>When the OSS receives a read request from a client, it reads data from disk into
1351 its memory and sends the data as a reply to the request. If read cache is enabled,
1352 this data stays in memory after the request from the client has been fulfilled. When
1353 subsequent read requests for the same data are received, the OSS skips reading data
1354 from disk and the request is fulfilled from the cached data. The read cache is managed
1355 by the Linux kernel globally across all OSTs on that OSS so that the least recently
1356 used cache pages are dropped from memory when the amount of free memory is running
1358 <para>If read cache is disabled (<literal>read_cache_enable=0</literal>), the OSS
1359 discards the data after a read request from the client is serviced and, for subsequent
1360 read requests, the OSS again reads the data from disk.</para>
1361 <para>To disable read cache on all the OSTs of an OSS, run:</para>
1362 <screen>root@oss1# lctl set_param obdfilter.*.read_cache_enable=0</screen>
1363 <para>To re-enable read cache on one OST, run:</para>
1364 <screen>root@oss1# lctl set_param obdfilter.{OST_name}.read_cache_enable=1</screen>
1365 <para>To check if read cache is enabled on all OSTs on an OSS, run:</para>
1366 <screen>root@oss1# lctl get_param obdfilter.*.read_cache_enable</screen>
1369 <para><literal>writethrough_cache_enable</literal> - Controls whether data sent to the
1370 OSS as a write request is kept in the read cache and available for later reads, or if
1371 it is discarded from cache when the write is completed. By default, the writethrough
1372 cache is enabled (<literal>writethrough_cache_enable=1</literal>).</para>
1373 <para>When the OSS receives write requests from a client, it receives data from the
1374 client into its memory and writes the data to disk. If the writethrough cache is
1375 enabled, this data stays in memory after the write request is completed, allowing the
1376 OSS to skip reading this data from disk if a later read request, or partial-page write
1377 request, for the same data is received.</para>
1378 <para>If the writethrough cache is disabled
1379 (<literal>writethrough_cache_enabled=0</literal>), the OSS discards the data after
1380 the write request from the client is completed. For subsequent read requests, or
1381 partial-page write requests, the OSS must re-read the data from disk.</para>
1382 <para>Enabling writethrough cache is advisable if clients are doing small or unaligned
1383 writes that would cause partial-page updates, or if the files written by one node are
1384 immediately being accessed by other nodes. Some examples where enabling writethrough
1385 cache might be useful include producer-consumer I/O models or shared-file writes with
1386 a different node doing I/O not aligned on 4096-byte boundaries. </para>
1387 <para>Disabling the writethrough cache is advisable when files are mostly written to the
1388 file system but are not re-read within a short time period, or files are only written
1389 and re-read by the same node, regardless of whether the I/O is aligned or not.</para>
1390 <para>To disable the writethrough cache on all OSTs of an OSS, run:</para>
1391 <screen>root@oss1# lctl set_param obdfilter.*.writethrough_cache_enable=0</screen>
1392 <para>To re-enable the writethrough cache on one OST, run:</para>
1393 <screen>root@oss1# lctl set_param obdfilter.{OST_name}.writethrough_cache_enable=1</screen>
1394 <para>To check if the writethrough cache is enabled, run:</para>
1395 <screen>root@oss1# lctl set_param obdfilter.*.writethrough_cache_enable=1</screen>
1398 <para><literal>readcache_max_filesize</literal> - Controls the maximum size of a file
1399 that both the read cache and writethrough cache will try to keep in memory. Files
1400 larger than <literal>readcache_max_filesize</literal> will not be kept in cache for
1401 either reads or writes.</para>
1402 <para>Setting this tunable can be useful for workloads where relatively small files are
1403 repeatedly accessed by many clients, such as job startup files, executables, log
1404 files, etc., but large files are read or written only once. By not putting the larger
1405 files into the cache, it is much more likely that more of the smaller files will
1406 remain in cache for a longer time.</para>
1407 <para>When setting <literal>readcache_max_filesize</literal>, the input value can be
1408 specified in bytes, or can have a suffix to indicate other binary units such as
1409 <literal>K</literal> (kilobytes), <literal>M</literal> (megabytes),
1410 <literal>G</literal> (gigabytes), <literal>T</literal> (terabytes), or
1411 <literal>P</literal> (petabytes).</para>
1412 <para>To limit the maximum cached file size to 32 MB on all OSTs of an OSS, run:</para>
1413 <screen>root@oss1# lctl set_param obdfilter.*.readcache_max_filesize=32M</screen>
1414 <para>To disable the maximum cached file size on an OST, run:</para>
1415 <screen>root@oss1# lctl set_param obdfilter.{OST_name}.readcache_max_filesize=-1</screen>
1416 <para>To check the current maximum cached file size on all OSTs of an OSS, run:</para>
1417 <screen>root@oss1# lctl get_param obdfilter.*.readcache_max_filesize</screen>
1424 <primary>proc</primary>
1425 <secondary>OSS journal</secondary>
1426 </indexterm>Enabling OSS Asynchronous Journal Commit</title>
1427 <para>The OSS asynchronous journal commit feature asynchronously writes data to disk without
1428 forcing a journal flush. This reduces the number of seeks and significantly improves
1429 performance on some hardware.</para>
1431 <para>Asynchronous journal commit cannot work with direct I/O-originated writes
1432 (<literal>O_DIRECT</literal> flag set). In this case, a journal flush is forced. </para>
1434 <para>When the asynchronous journal commit feature is enabled, client nodes keep data in the
1435 page cache (a page reference). Lustre clients monitor the last committed transaction number
1436 (<literal>transno</literal>) in messages sent from the OSS to the clients. When a client
1437 sees that the last committed <literal>transno</literal> reported by the OSS is at least
1438 equal to the bulk write <literal>transno</literal>, it releases the reference on the
1439 corresponding pages. To avoid page references being held for too long on clients after a
1440 bulk write, a 7 second ping request is scheduled (the default OSS file system commit time
1441 interval is 5 seconds) after the bulk write reply is received, so the OSS has an opportunity
1442 to report the last committed <literal>transno</literal>.</para>
1443 <para>If the OSS crashes before the journal commit occurs, then intermediate data is lost.
1444 However, OSS recovery functionality incorporated into the asynchronous journal commit
1445 feature causes clients to replay their write requests and compensate for the missing disk
1446 updates by restoring the state of the file system.</para>
1447 <para>By default, <literal>sync_journal</literal> is enabled
1448 (<literal>sync_journal=1</literal>), so that journal entries are committed synchronously.
1449 To enable asynchronous journal commit, set the <literal>sync_journal</literal> parameter to
1450 <literal>0</literal> by entering: </para>
1451 <screen>$ lctl set_param obdfilter.*.sync_journal=0
1452 obdfilter.lol-OST0001.sync_journal=0</screen>
1453 <para>An associated <literal>sync-on-lock-cancel</literal> feature (enabled by default)
1454 addresses a data consistency issue that can result if an OSS crashes after multiple clients
1455 have written data into intersecting regions of an object, and then one of the clients also
1456 crashes. A condition is created in which the POSIX requirement for continuous writes is
1457 violated along with a potential for corrupted data. With
1458 <literal>sync-on-lock-cancel</literal> enabled, if a cancelled lock has any volatile
1459 writes attached to it, the OSS synchronously writes the journal to disk on lock
1460 cancellation. Disabling the <literal>sync-on-lock-cancel</literal> feature may enhance
1461 performance for concurrent write workloads, but it is recommended that you not disable this
1463 <para> The <literal>sync_on_lock_cancel</literal> parameter can be set to the following
1467 <para><literal>always</literal> - Always force a journal flush on lock cancellation
1468 (default when <literal>async_journal</literal> is enabled).</para>
1471 <para><literal>blocking</literal> - Force a journal flush only when the local cancellation
1472 is due to a blocking callback.</para>
1475 <para><literal>never</literal> - Do not force any journal flush (default when
1476 <literal>async_journal</literal> is disabled).</para>
1479 <para>For example, to set <literal>sync_on_lock_cancel</literal> to not to force a journal
1480 flush, use a command similar to:</para>
1481 <screen>$ lctl get_param obdfilter.*.sync_on_lock_cancel
1482 obdfilter.lol-OST0001.sync_on_lock_cancel=never</screen>
1486 <title>Configuring Timeouts in a Lustre File System</title>
1487 <para>In a Lustre file system, RPC timeouts are set using an adaptive timeouts mechanism, which
1488 is enabled by default. Servers track RPC completion times and then report back to clients
1489 estimates for completion times for future RPCs. Clients use these estimates to set RPC
1490 timeout values. If the processing of server requests slows down for any reason, the server
1491 estimates for RPC completion increase, and clients then revise RPC timeout values to allow
1492 more time for RPC completion.</para>
1493 <para>If the RPCs queued on the server approach the RPC timeout specified by the client, to
1494 avoid RPC timeouts and disconnect/reconnect cycles, the server sends an "early reply" to the
1495 client, telling the client to allow more time. Conversely, as server processing speeds up, RPC
1496 timeout values decrease, resulting in faster detection if the server becomes non-responsive
1497 and quicker connection to the failover partner of the server.</para>
1500 <primary>proc</primary>
1501 <secondary>configuring adaptive timeouts</secondary>
1502 </indexterm><indexterm>
1503 <primary>configuring</primary>
1504 <secondary>adaptive timeouts</secondary>
1505 </indexterm><indexterm>
1506 <primary>proc</primary>
1507 <secondary>adaptive timeouts</secondary>
1508 </indexterm>Configuring Adaptive Timeouts</title>
1509 <para>The adaptive timeout parameters in the table below can be set persistently system-wide
1510 using <literal>lctl conf_param</literal> on the MGS. For example, the following command sets
1511 the <literal>at_max</literal> value for all servers and clients associated with the file
1513 <literal>testfs</literal>:<screen>lctl conf_param testfs.sys.at_max=1500</screen></para>
1515 <para>Clients that access multiple Lustre file systems must use the same parameter values
1516 for all file systems.</para>
1518 <informaltable frame="all">
1520 <colspec colname="c1" colwidth="30*"/>
1521 <colspec colname="c2" colwidth="80*"/>
1525 <para><emphasis role="bold">Parameter</emphasis></para>
1528 <para><emphasis role="bold">Description</emphasis></para>
1536 <literal> at_min </literal></para>
1539 <para>Minimum adaptive timeout (in seconds). The default value is 0. The
1540 <literal>at_min</literal> parameter is the minimum processing time that a server
1541 will report. Ideally, <literal>at_min</literal> should be set to its default
1542 value. Clients base their timeouts on this value, but they do not use this value
1544 <para>If, for unknown reasons (usually due to temporary network outages), the
1545 adaptive timeout value is too short and clients time out their RPCs, you can
1546 increase the <literal>at_min</literal> value to compensate for this.</para>
1552 <literal> at_max </literal></para>
1555 <para>Maximum adaptive timeout (in seconds). The <literal>at_max</literal> parameter
1556 is an upper-limit on the service time estimate. If <literal>at_max</literal> is
1557 reached, an RPC request times out.</para>
1558 <para>Setting <literal>at_max</literal> to 0 causes adaptive timeouts to be disabled
1559 and a fixed timeout method to be used instead (see <xref
1560 xmlns:xlink="http://www.w3.org/1999/xlink" linkend="section_c24_nt5_dl"/></para>
1562 <para>If slow hardware causes the service estimate to increase beyond the default
1563 value of <literal>at_max</literal>, increase <literal>at_max</literal> to the
1564 maximum time you are willing to wait for an RPC completion.</para>
1571 <literal> at_history </literal></para>
1574 <para>Time period (in seconds) within which adaptive timeouts remember the slowest
1575 event that occurred. The default is 600.</para>
1581 <literal> at_early_margin </literal></para>
1584 <para>Amount of time before the Lustre server sends an early reply (in seconds).
1585 Default is 5.</para>
1591 <literal> at_extra </literal></para>
1594 <para>Incremental amount of time that a server requests with each early reply (in
1595 seconds). The server does not know how much time the RPC will take, so it asks for
1596 a fixed value. The default is 30, which provides a balance between sending too
1597 many early replies for the same RPC and overestimating the actual completion
1599 <para>When a server finds a queued request about to time out and needs to send an
1600 early reply out, the server adds the <literal>at_extra</literal> value. If the
1601 time expires, the Lustre server drops the request, and the client enters recovery
1602 status and reconnects to restore the connection to normal status.</para>
1603 <para>If you see multiple early replies for the same RPC asking for 30-second
1604 increases, change the <literal>at_extra</literal> value to a larger number to cut
1605 down on early replies sent and, therefore, network load.</para>
1611 <literal> ldlm_enqueue_min </literal></para>
1614 <para>Minimum lock enqueue time (in seconds). The default is 100. The time it takes
1615 to enqueue a lock, <literal>ldlm_enqueue</literal>, is the maximum of the measured
1616 enqueue estimate (influenced by <literal>at_min</literal> and
1617 <literal>at_max</literal> parameters), multiplied by a weighting factor and the
1618 value of <literal>ldlm_enqueue_min</literal>. </para>
1619 <para>Lustre Distributed Lock Manager (LDLM) lock enqueues have a dedicated minimum
1620 value for <literal>ldlm_enqueue_min</literal>. Lock enqueue timeouts increase as
1621 the measured enqueue times increase (similar to adaptive timeouts).</para>
1628 <title>Interpreting Adaptive Timeout Information</title>
1629 <para>Adaptive timeout information can be obtained from the <literal>timeouts</literal>
1630 files in <literal>/proc/fs/lustre/*/</literal> on each server and client using the
1631 <literal>lctl</literal> command. To read information from a <literal>timeouts</literal>
1632 file, enter a command similar to:</para>
1633 <screen># lctl get_param -n ost.*.ost_io.timeouts
1634 service : cur 33 worst 34 (at 1193427052, 0d0h26m40s ago) 1 1 33 2</screen>
1635 <para>In this example, the <literal>ost_io</literal> service on this node is currently
1636 reporting an estimated RPC service time of 33 seconds. The worst RPC service time was 34
1637 seconds, which occurred 26 minutes ago.</para>
1638 <para>The output also provides a history of service times. Four "bins" of adaptive
1639 timeout history are shown, with the maximum RPC time in each bin reported. In both the
1640 0-150s bin and the 150-300s bin, the maximum RPC time was 1. The 300-450s bin shows the
1641 worst (maximum) RPC time at 33 seconds, and the 450-600s bin shows a maximum of RPC time
1642 of 2 seconds. The estimated service time is the maximum value across the four bins (33
1643 seconds in this example).</para>
1644 <para>Service times (as reported by the servers) are also tracked in the client OBDs, as
1645 shown in this example:</para>
1646 <screen># lctl get_param osc.*.timeouts
1647 last reply : 1193428639, 0d0h00m00s ago
1648 network : cur 1 worst 2 (at 1193427053, 0d0h26m26s ago) 1 1 1 1
1649 portal 6 : cur 33 worst 34 (at 1193427052, 0d0h26m27s ago) 33 33 33 2
1650 portal 28 : cur 1 worst 1 (at 1193426141, 0d0h41m38s ago) 1 1 1 1
1651 portal 7 : cur 1 worst 1 (at 1193426141, 0d0h41m38s ago) 1 0 1 1
1652 portal 17 : cur 1 worst 1 (at 1193426177, 0d0h41m02s ago) 1 0 0 1
1654 <para>In this example, portal 6, the <literal>ost_io</literal> service portal, shows the
1655 history of service estimates reported by the portal.</para>
1656 <para>Server statistic files also show the range of estimates including min, max, sum, and
1657 sumsq. For example:</para>
1658 <screen># lctl get_param mdt.*.mdt.stats
1660 req_timeout 6 samples [sec] 1 10 15 105
1665 <section xml:id="section_c24_nt5_dl">
1666 <title>Setting Static Timeouts<indexterm>
1667 <primary>proc</primary>
1668 <secondary>static timeouts</secondary>
1669 </indexterm></title>
1670 <para>The Lustre software provides two sets of static (fixed) timeouts, LND timeouts and
1671 Lustre timeouts, which are used when adaptive timeouts are not enabled.</para>
1675 <para><emphasis role="italic"><emphasis role="bold">LND timeouts</emphasis></emphasis> -
1676 LND timeouts ensure that point-to-point communications across a network complete in a
1677 finite time in the presence of failures, such as packages lost or broken connections.
1678 LND timeout parameters are set for each individual LND.</para>
1679 <para>LND timeouts are logged with the <literal>S_LND</literal> flag set. They are not
1680 printed as console messages, so check the Lustre log for <literal>D_NETERROR</literal>
1681 messages or enable printing of <literal>D_NETERROR</literal> messages to the console
1682 using:<screen>lctl set_param printk=+neterror</screen></para>
1683 <para>Congested routers can be a source of spurious LND timeouts. To avoid this
1684 situation, increase the number of LNET router buffers to reduce back-pressure and/or
1685 increase LND timeouts on all nodes on all connected networks. Also consider increasing
1686 the total number of LNET router nodes in the system so that the aggregate router
1687 bandwidth matches the aggregate server bandwidth.</para>
1690 <para><emphasis role="italic"><emphasis role="bold">Lustre timeouts
1691 </emphasis></emphasis>- Lustre timeouts ensure that Lustre RPCs complete in a finite
1692 time in the presence of failures when adaptive timeouts are not enabled. Adaptive
1693 timeouts are enabled by default. To disable adaptive timeouts at run time, set
1694 <literal>at_max</literal> to 0 by running on the
1695 MGS:<screen># lctl conf_param <replaceable>fsname</replaceable>.sys.at_max=0</screen></para>
1697 <para>Changing the status of adaptive timeouts at runtime may cause a transient client
1698 timeout, recovery, and reconnection.</para>
1700 <para>Lustre timeouts are always printed as console messages. </para>
1701 <para>If Lustre timeouts are not accompanied by LND timeouts, increase the Lustre
1702 timeout on both servers and clients. Lustre timeouts are set using a command such as
1703 the following:<screen># lctl set_param timeout=30</screen></para>
1704 <para>Lustre timeout parameters are described in the table below.</para>
1707 <informaltable frame="all">
1709 <colspec colname="c1" colnum="1" colwidth="30*"/>
1710 <colspec colname="c2" colnum="2" colwidth="70*"/>
1713 <entry>Parameter</entry>
1714 <entry>Description</entry>
1719 <entry><literal>timeout</literal></entry>
1721 <para>The time that a client waits for a server to complete an RPC (default 100s).
1722 Servers wait half this time for a normal client RPC to complete and a quarter of
1723 this time for a single bulk request (read or write of up to 4 MB) to complete.
1724 The client pings recoverable targets (MDS and OSTs) at one quarter of the
1725 timeout, and the server waits one and a half times the timeout before evicting a
1726 client for being "stale."</para>
1727 <para>Lustre client sends periodic 'ping' messages to servers with which
1728 it has had no communication for the specified period of time. Any network
1729 activity between a client and a server in the file system also serves as a
1734 <entry><literal>ldlm_timeout</literal></entry>
1736 <para>The time that a server waits for a client to reply to an initial AST (lock
1737 cancellation request). The default is 20s for an OST and 6s for an MDS. If the
1738 client replies to the AST, the server will give it a normal timeout (half the
1739 client timeout) to flush any dirty data and release the lock.</para>
1743 <entry><literal>fail_loc</literal></entry>
1745 <para>An internal debugging failure hook. The default value of
1746 <literal>0</literal> means that no failure will be triggered or
1751 <entry><literal>dump_on_timeout</literal></entry>
1753 <para>Triggers a dump of the Lustre debug log when a timeout occurs. The default
1754 value of <literal>0</literal> (zero) means a dump of the Lustre debug log will
1755 not be triggered.</para>
1759 <entry><literal>dump_on_eviction</literal></entry>
1761 <para>Triggers a dump of the Lustre debug log when an eviction occurs. The default
1762 value of <literal>0</literal> (zero) means a dump of the Lustre debug log will
1763 not be triggered. </para>
1772 <section remap="h3">
1774 <primary>proc</primary>
1775 <secondary>LNET</secondary>
1776 </indexterm><indexterm>
1777 <primary>LNET</primary>
1778 <secondary>proc</secondary>
1779 </indexterm>Monitoring LNET</title>
1780 <para>LNET information is located in <literal>/proc/sys/lnet</literal> in these files:<itemizedlist>
1782 <para><literal>peers</literal> - Shows all NIDs known to this node and provides
1783 information on the queue state.</para>
1784 <para>Example:</para>
1785 <screen># lctl get_param peers
1786 nid refs state max rtr min tx min queue
1787 0@lo 1 ~rtr 0 0 0 0 0 0
1788 192.168.10.35@tcp 1 ~rtr 8 8 8 8 6 0
1789 192.168.10.36@tcp 1 ~rtr 8 8 8 8 6 0
1790 192.168.10.37@tcp 1 ~rtr 8 8 8 8 6 0</screen>
1791 <para>The fields are explained in the table below:</para>
1792 <informaltable frame="all">
1794 <colspec colname="c1" colwidth="30*"/>
1795 <colspec colname="c2" colwidth="80*"/>
1799 <para><emphasis role="bold">Field</emphasis></para>
1802 <para><emphasis role="bold">Description</emphasis></para>
1810 <literal>refs</literal>
1814 <para>A reference count. </para>
1820 <literal>state</literal>
1824 <para>If the node is a router, indicates the state of the router. Possible
1828 <para><literal>NA</literal> - Indicates the node is not a router.</para>
1831 <para><literal>up/down</literal>- Indicates if the node (router) is up or
1840 <literal>max </literal></para>
1843 <para>Maximum number of concurrent sends from this peer.</para>
1849 <literal>rtr </literal></para>
1852 <para>Number of routing buffer credits.</para>
1858 <literal>min </literal></para>
1861 <para>Minimum number of routing buffer credits seen.</para>
1867 <literal>tx </literal></para>
1870 <para>Number of send credits.</para>
1876 <literal>min </literal></para>
1879 <para>Minimum number of send credits seen.</para>
1885 <literal>queue </literal></para>
1888 <para>Total bytes in active/queued sends.</para>
1894 <para>Credits are initialized to allow a certain number of operations (in the example
1895 above the table, eight as shown in the <literal>max</literal> column. LNET keeps track
1896 of the minimum number of credits ever seen over time showing the peak congestion that
1897 has occurred during the time monitored. Fewer available credits indicates a more
1898 congested resource. </para>
1899 <para>The number of credits currently in flight (number of transmit credits) is shown in
1900 the <literal>tx</literal> column. The maximum number of send credits available is shown
1901 in the <literal>max</literal> column and never changes. The number of router buffers
1902 available for consumption by a peer is shown in the <literal>rtr</literal>
1904 <para>Therefore, <literal>rtr</literal> – <literal>tx</literal> is the number of transmits
1905 in flight. Typically, <literal>rtr == max</literal>, although a configuration can be set
1906 such that <literal>max >= rtr</literal>. The ratio of routing buffer credits to send
1907 credits (<literal>rtr/tx</literal>) that is less than <literal>max</literal> indicates
1908 operations are in progress. If the ratio <literal>rtr/tx</literal> is greater than
1909 <literal>max</literal>, operations are blocking.</para>
1910 <para>LNET also limits concurrent sends and number of router buffers allocated to a single
1911 peer so that no peer can occupy all these resources.</para>
1914 <para><literal>nis</literal> - Shows the current queue health on this node.</para>
1915 <para>Example:</para>
1916 <screen># lctl get_param nis
1917 nid refs peer max tx min
1919 192.168.10.34@tcp 4 8 256 256 252
1921 <para> The fields are explained in the table below.</para>
1922 <informaltable frame="all">
1924 <colspec colname="c1" colwidth="30*"/>
1925 <colspec colname="c2" colwidth="80*"/>
1929 <para><emphasis role="bold">Field</emphasis></para>
1932 <para><emphasis role="bold">Description</emphasis></para>
1940 <literal> nid </literal></para>
1943 <para>Network interface.</para>
1949 <literal> refs </literal></para>
1952 <para>Internal reference counter.</para>
1958 <literal> peer </literal></para>
1961 <para>Number of peer-to-peer send credits on this NID. Credits are used to size
1962 buffer pools.</para>
1968 <literal> max </literal></para>
1971 <para>Total number of send credits on this NID.</para>
1977 <literal> tx </literal></para>
1980 <para>Current number of send credits available on this NID.</para>
1986 <literal> min </literal></para>
1989 <para>Lowest number of send credits available on this NID.</para>
1995 <literal> queue </literal></para>
1998 <para>Total bytes in active/queued sends.</para>
2004 <para><emphasis role="bold"><emphasis role="italic">Analysis:</emphasis></emphasis></para>
2005 <para>Subtracting <literal>max</literal> from <literal>tx</literal>
2006 (<literal>max</literal> - <literal>tx</literal>) yields the number of sends currently
2007 active. A large or increasing number of active sends may indicate a problem.</para>
2009 </itemizedlist></para>
2011 <section remap="h3">
2013 <primary>proc</primary>
2014 <secondary>free space</secondary>
2015 </indexterm>Allocating Free Space on OSTs</title>
2016 <para>Free space is allocated using either a round-robin or a weighted algorithm. The allocation
2017 method is determined by the maximum amount of free-space imbalance between the OSTs. When free
2018 space is relatively balanced across OSTs, the faster round-robin allocator is used, which
2019 maximizes network balancing. The weighted allocator is used when any two OSTs are out of
2020 balance by more than a specified threshold.</para>
2021 <para>Free space distribution can be tuned using these two <literal>/proc</literal>
2025 <para><literal>qos_threshold_rr</literal> - The threshold at which the allocation method
2026 switches from round-robin to weighted is set in this file. The default is to switch to the
2027 weighted algorithm when any two OSTs are out of balance by more than 17 percent.</para>
2030 <para><literal>qos_prio_free</literal> - The weighting priority used by the weighted
2031 allocator can be adjusted in this file. Increasing the value of
2032 <literal>qos_prio_free</literal> puts more weighting on the amount of free space
2033 available on each OST and less on how stripes are distributed across OSTs. The default
2034 value is 91 percent. When the free space priority is set to 100, weighting is based
2035 entirely on free space and location is no longer used by the striping algorthm.</para>
2038 <para>For more information about monitoring and managing free space, see <xref
2039 xmlns:xlink="http://www.w3.org/1999/xlink" linkend="dbdoclet.50438209_10424"/>.</para>
2041 <section remap="h3">
2043 <primary>proc</primary>
2044 <secondary>locking</secondary>
2045 </indexterm>Configuring Locking</title>
2046 <para>The <literal>lru_size</literal> parameter is used to control the number of client-side
2047 locks in an LRU cached locks queue. LRU size is dynamic, based on load to optimize the number
2048 of locks available to nodes that have different workloads (e.g., login/build nodes vs. compute
2049 nodes vs. backup nodes).</para>
2050 <para>The total number of locks available is a function of the server RAM. The default limit is
2051 50 locks/1 MB of RAM. If memory pressure is too high, the LRU size is shrunk. The number of
2052 locks on the server is limited to <emphasis role="italic">the number of OSTs per
2053 server</emphasis> * <emphasis role="italic">the number of clients</emphasis> * <emphasis
2054 role="italic">the value of the</emphasis>
2055 <literal>lru_size</literal>
2056 <emphasis role="italic">setting on the client</emphasis> as follows: </para>
2059 <para>To enable automatic LRU sizing, set the <literal>lru_size</literal> parameter to 0. In
2060 this case, the <literal>lru_size</literal> parameter shows the current number of locks
2061 being used on the export. LRU sizing is enabled by default.</para>
2064 <para>To specify a maximum number of locks, set the <literal>lru_size</literal> parameter to
2065 a value other than zero but, normally, less than 100 * <emphasis role="italic">number of
2066 CPUs in client</emphasis>. It is recommended that you only increase the LRU size on a
2067 few login nodes where users access the file system interactively.</para>
2070 <para>To clear the LRU on a single client, and, as a result, flush client cache without changing
2071 the <literal>lru_size</literal> value, run:</para>
2072 <screen>$ lctl set_param ldlm.namespaces.<replaceable>osc_name|mdc_name</replaceable>.lru_size=clear</screen>
2073 <para>If the LRU size is set to be less than the number of existing unused locks, the unused
2074 locks are canceled immediately. Use <literal>echo clear</literal> to cancel all locks without
2075 changing the value.</para>
2077 <para>The <literal>lru_size</literal> parameter can only be set temporarily using
2078 <literal>lctl set_param</literal>; it cannot be set permanently.</para>
2080 <para>To disable LRU sizing, on the Lustre clients, run:</para>
2081 <screen>$ lctl set_param ldlm.namespaces.*osc*.lru_size=$((<replaceable>NR_CPU</replaceable>*100))</screen>
2082 <para>Replace <literal><replaceable>NR_CPU</replaceable></literal> with the number of CPUs on
2084 <para>To determine the number of locks being granted, run:</para>
2085 <screen>$ lctl get_param ldlm.namespaces.*.pool.limit</screen>
2087 <section xml:id="dbdoclet.50438271_87260">
2089 <primary>proc</primary>
2090 <secondary>thread counts</secondary>
2091 </indexterm>Setting MDS and OSS Thread Counts</title>
2092 <para>MDS and OSS thread counts tunable can be used to set the minimum and maximum thread counts
2093 or get the current number of running threads for the services listed in the table
2095 <informaltable frame="all">
2097 <colspec colname="c1" colwidth="50*"/>
2098 <colspec colname="c2" colwidth="50*"/>
2103 <emphasis role="bold">Service</emphasis></para>
2107 <emphasis role="bold">Description</emphasis></para>
2112 <literal> mds.MDS.mdt </literal>
2115 <para>Main metadata operations service</para>
2120 <literal> mds.MDS.mdt_readpage </literal>
2123 <para>Metadata <literal>readdir</literal> service</para>
2128 <literal> mds.MDS.mdt_setattr </literal>
2131 <para>Metadata <literal>setattr/close</literal> operations service </para>
2136 <literal> ost.OSS.ost </literal>
2139 <para>Main data operations service</para>
2144 <literal> ost.OSS.ost_io </literal>
2147 <para>Bulk data I/O services</para>
2152 <literal> ost.OSS.ost_create </literal>
2155 <para>OST object pre-creation service</para>
2160 <literal> ldlm.services.ldlm_canceld </literal>
2163 <para>DLM lock cancel service</para>
2168 <literal> ldlm.services.ldlm_cbd </literal>
2171 <para>DLM lock grant service</para>
2177 <para>For each service, an entry as shown below is
2178 created:<screen>/proc/fs/lustre/<replaceable>service</replaceable>/*/threads_<replaceable>min|max|started</replaceable></screen></para>
2181 <para>To temporarily set this tunable, run:</para>
2182 <screen># lctl <replaceable>get|set</replaceable>_param <replaceable>service</replaceable>.threads_<replaceable>min|max|started</replaceable> </screen>
2185 <para>To permanently set this tunable, run:</para>
2186 <screen># lctl conf_param <replaceable>obdname|fsname.obdtype</replaceable>.threads_<replaceable>min|max|started</replaceable> </screen>
2187 <para condition='l25'>For version 2.5 or later, run:
2188 <screen># lctl set_param -P <replaceable>service</replaceable>.threads_<replaceable>min|max|started</replaceable></screen></para>
2191 <para>The following examples show how to set thread counts and get the number of running threads
2192 for the service <literal>ost_io</literal> using the tunable
2193 <literal><replaceable>service</replaceable>.threads_<replaceable>min|max|started</replaceable></literal>.</para>
2196 <para>To get the number of running threads, run:</para>
2197 <screen># lctl get_param ost.OSS.ost_io.threads_started
2198 ost.OSS.ost_io.threads_started=128</screen>
2201 <para>To set the number of threads to the maximum value (512), run:</para>
2202 <screen># lctl get_param ost.OSS.ost_io.threads_max
2203 ost.OSS.ost_io.threads_max=512</screen>
2206 <para>To set the maximum thread count to 256 instead of 512 (to avoid overloading the
2207 storage or for an array with requests), run:</para>
2208 <screen># lctl set_param ost.OSS.ost_io.threads_max=256
2209 ost.OSS.ost_io.threads_max=256</screen>
2212 <para>To set the maximum thread count to 256 instead of 512 permanently, run:</para>
2213 <screen># lctl conf_param testfs.ost.ost_io.threads_max=256</screen>
2214 <para condition='l25'>For version 2.5 or later, run:
2215 <screen># lctl set_param -P ost.OSS.ost_io.threads_max=256
2216 ost.OSS.ost_io.threads_max=256 </screen> </para>
2219 <para> To check if the <literal>threads_max</literal> setting is active, run:</para>
2220 <screen># lctl get_param ost.OSS.ost_io.threads_max
2221 ost.OSS.ost_io.threads_max=256</screen>
2225 <para>If the number of service threads is changed while the file system is running, the change
2226 may not take effect until the file system is stopped and rest. If the number of service
2227 threads in use exceeds the new <literal>threads_max</literal> value setting, service threads
2228 that are already running will not be stopped.</para>
2230 <para>See also <xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="lustretuning"/></para>
2232 <section xml:id="dbdoclet.50438271_83523">
2234 <primary>proc</primary>
2235 <secondary>debug</secondary>
2236 </indexterm>Enabling and Interpreting Debugging Logs</title>
2237 <para>By default, a detailed log of all operations is generated to aid in debugging. Flags that
2238 control debugging are found in <literal>/proc/sys/lnet/debug</literal>. </para>
2239 <para>The overhead of debugging can affect the performance of Lustre file system. Therefore, to
2240 minimize the impact on performance, the debug level can be lowered, which affects the amount
2241 of debugging information kept in the internal log buffer but does not alter the amount of
2242 information to goes into syslog. You can raise the debug level when you need to collect logs
2243 to debug problems. </para>
2244 <para>The debugging mask can be set using "symbolic names". The symbolic format is
2245 shown in the examples below.<itemizedlist>
2247 <para>To verify the debug level used, examine the <literal>sysctl</literal> that controls
2248 debugging by running:</para>
2249 <screen># sysctl lnet.debug
2250 lnet.debug = ioctl neterror warning error emerg ha config console</screen>
2253 <para>To turn off debugging (except for network error debugging), run the following
2254 command on all nodes concerned:</para>
2255 <screen># sysctl -w lnet.debug="neterror"
2256 lnet.debug = neterror</screen>
2258 </itemizedlist><itemizedlist>
2260 <para>To turn off debugging completely, run the following command on all nodes
2262 <screen># sysctl -w lnet.debug=0
2263 lnet.debug = 0</screen>
2266 <para>To set an appropriate debug level for a production environment, run:</para>
2267 <screen># sysctl -w lnet.debug="warning dlmtrace error emerg ha rpctrace vfstrace"
2268 lnet.debug = warning dlmtrace error emerg ha rpctrace vfstrace</screen>
2269 <para>The flags shown in this example collect enough high-level information to aid
2270 debugging, but they do not cause any serious performance impact.</para>
2272 </itemizedlist><itemizedlist>
2274 <para>To clear all flags and set new flags, run:</para>
2275 <screen># sysctl -w lnet.debug="warning"
2276 lnet.debug = warning</screen>
2278 </itemizedlist><itemizedlist>
2280 <para>To add new flags to flags that have already been set, precede each one with a
2281 "<literal>+</literal>":</para>
2282 <screen># sysctl -w lnet.debug="+neterror +ha"
2283 lnet.debug = +neterror +ha
2285 lnet.debug = neterror warning ha</screen>
2288 <para>To remove individual flags, precede them with a
2289 "<literal>-</literal>":</para>
2290 <screen># sysctl -w lnet.debug="-ha"
2293 lnet.debug = neterror warning</screen>
2296 <para>To verify or change the debug level, run commands such as the following: :</para>
2297 <screen># lctl get_param debug
2300 # lctl set_param debug=+ha
2301 # lctl get_param debug
2304 # lctl set_param debug=-warning
2305 # lctl get_param debug
2307 neterror ha</screen>
2309 </itemizedlist></para>
2310 <para>Debugging parameters include:</para>
2313 <para><literal>subsystem_debug</literal> - Controls the debug logs for subsystems.</para>
2316 <para><literal>debug_path</literal> - Indicates the location where the debug log is dumped
2317 when triggered automatically or manually. The default path is
2318 <literal>/tmp/lustre-log</literal>.</para>
2321 <para>These parameters are also set using:<screen>sysctl -w lnet.debug={value}</screen></para>
2322 <para>Additional useful parameters: <itemizedlist>
2324 <para><literal>panic_on_lbug</literal> - Causes ''panic'' to be called
2325 when the Lustre software detects an internal problem (an <literal>LBUG</literal> log
2326 entry); panic crashes the node. This is particularly useful when a kernel crash dump
2327 utility is configured. The crash dump is triggered when the internal inconsistency is
2328 detected by the Lustre software. </para>
2331 <para><literal>upcall</literal> - Allows you to specify the path to the binary which will
2332 be invoked when an <literal>LBUG</literal> log entry is encountered. This binary is
2333 called with four parameters:</para>
2334 <para> - The string ''<literal>LBUG</literal>''.</para>
2335 <para> - The file where the <literal>LBUG</literal> occurred.</para>
2336 <para> - The function name.</para>
2337 <para> - The line number in the file</para>
2339 </itemizedlist></para>
2341 <title>Interpreting OST Statistics</title>
2343 <para>See also <xref linkend="dbdoclet.50438219_84890"/> (<literal>llobdstat</literal>) and
2344 <xref linkend="dbdoclet.50438273_80593"/> (<literal>collectl</literal>).</para>
2346 <para>OST <literal>stats</literal> files can be used to provide statistics showing activity
2347 for each OST. For example:</para>
2348 <screen># lctl get_param osc.testfs-OST0000-osc.stats
2349 snapshot_time 1189732762.835363
2354 obd_ping 212</screen>
2355 <para>Use the <literal>llstat</literal> utility to monitor statistics over time.</para>
2356 <para>To clear the statistics, use the <literal>-c</literal> option to
2357 <literal>llstat</literal>. To specify how frequently the statistics should be reported (in
2358 seconds), use the <literal>-i</literal> option. In the example below, the
2359 <literal>-c</literal> option clears the statistics and <literal>-i10</literal> option
2360 reports statistics every 10 seconds:</para>
2361 <screen role="smaller">$ llstat -c -i10 /proc/fs/lustre/ost/OSS/ost_io/stats
2363 /usr/bin/llstat: STATS on 06/06/07
2364 /proc/fs/lustre/ost/OSS/ost_io/ stats on 192.168.16.35@tcp
2365 snapshot_time 1181074093.276072
2367 /proc/fs/lustre/ost/OSS/ost_io/stats @ 1181074103.284895
2369 Count Rate Events Unit last min avg max stddev
2370 req_waittime 8 0 8 [usec] 2078 34 259.75 868 317.49
2371 req_qdepth 8 0 8 [reqs] 1 0 0.12 1 0.35
2372 req_active 8 0 8 [reqs] 11 1 1.38 2 0.52
2373 reqbuf_avail 8 0 8 [bufs] 511 63 63.88 64 0.35
2374 ost_write 8 0 8 [bytes] 169767 72914 212209.62 387579 91874.29
2376 /proc/fs/lustre/ost/OSS/ost_io/stats @ 1181074113.290180
2378 Count Rate Events Unit last min avg max stddev
2379 req_waittime 31 3 39 [usec] 30011 34 822.79 12245 2047.71
2380 req_qdepth 31 3 39 [reqs] 0 0 0.03 1 0.16
2381 req_active 31 3 39 [reqs] 58 1 1.77 3 0.74
2382 reqbuf_avail 31 3 39 [bufs] 1977 63 63.79 64 0.41
2383 ost_write 30 3 38 [bytes] 1028467 15019 315325.16 910694 197776.51
2385 /proc/fs/lustre/ost/OSS/ost_io/stats @ 1181074123.325560
2387 Count Rate Events Unit last min avg max stddev
2388 req_waittime 21 2 60 [usec] 14970 34 784.32 12245 1878.66
2389 req_qdepth 21 2 60 [reqs] 0 0 0.02 1 0.13
2390 req_active 21 2 60 [reqs] 33 1 1.70 3 0.70
2391 reqbuf_avail 21 2 60 [bufs] 1341 63 63.82 64 0.39
2392 ost_write 21 2 59 [bytes] 7648424 15019 332725.08 910694 180397.87
2394 <para>The columns in this example are described in the table below.</para>
2395 <informaltable frame="all">
2397 <colspec colname="c1" colwidth="50*"/>
2398 <colspec colname="c2" colwidth="50*"/>
2402 <para><emphasis role="bold">Parameter</emphasis></para>
2405 <para><emphasis role="bold">Description</emphasis></para>
2411 <entry><literal>Name</literal></entry>
2412 <entry>Name of the service event. See the tables below for descriptions of service
2413 events that are tracked.</entry>
2418 <literal>Cur. Count </literal></para>
2421 <para>Number of events of each type sent in the last interval.</para>
2427 <literal>Cur. Rate </literal></para>
2430 <para>Number of events per second in the last interval.</para>
2436 <literal> # Events </literal></para>
2439 <para>Total number of such events since the events have been cleared.</para>
2445 <literal> Unit </literal></para>
2448 <para>Unit of measurement for that statistic (microseconds, requests,
2455 <literal> last </literal></para>
2458 <para>Average rate of these events (in units/event) for the last interval during
2459 which they arrived. For instance, in the above mentioned case of
2460 <literal>ost_destroy</literal> it took an average of 736 microseconds per
2461 destroy for the 400 object destroys in the previous 10 seconds.</para>
2467 <literal> min </literal></para>
2470 <para>Minimum rate (in units/events) since the service started.</para>
2476 <literal> avg </literal></para>
2479 <para>Average rate.</para>
2485 <literal> max </literal></para>
2488 <para>Maximum rate.</para>
2494 <literal> stddev </literal></para>
2497 <para>Standard deviation (not measured in some cases)</para>
2503 <para>Events common to all services are shown in the table below.</para>
2504 <informaltable frame="all">
2506 <colspec colname="c1" colwidth="50*"/>
2507 <colspec colname="c2" colwidth="50*"/>
2511 <para><emphasis role="bold">Parameter</emphasis></para>
2514 <para><emphasis role="bold">Description</emphasis></para>
2522 <literal> req_waittime </literal></para>
2525 <para>Amount of time a request waited in the queue before being handled by an
2526 available server thread.</para>
2532 <literal> req_qdepth </literal></para>
2535 <para>Number of requests waiting to be handled in the queue for this service.</para>
2541 <literal> req_active </literal></para>
2544 <para>Number of requests currently being handled.</para>
2550 <literal> reqbuf_avail </literal></para>
2553 <para>Number of unsolicited lnet request buffers for this service.</para>
2559 <para>Some service-specific events of interest are described in the table below.</para>
2560 <informaltable frame="all">
2562 <colspec colname="c1" colwidth="50*"/>
2563 <colspec colname="c2" colwidth="50*"/>
2567 <para><emphasis role="bold">Parameter</emphasis></para>
2570 <para><emphasis role="bold">Description</emphasis></para>
2578 <literal> ldlm_enqueue </literal></para>
2581 <para>Time it takes to enqueue a lock (this includes file open on the MDS)</para>
2587 <literal> mds_reint </literal></para>
2590 <para>Time it takes to process an MDS modification record (includes
2591 <literal>create</literal>, <literal>mkdir</literal>, <literal>unlink</literal>,
2592 <literal>rename</literal> and <literal>setattr</literal>)</para>
2600 <title>Interpreting MDT Statistics</title>
2602 <para>See also <xref linkend="dbdoclet.50438219_84890"/> (<literal>llobdstat</literal>) and
2603 <xref linkend="dbdoclet.50438273_80593"/> (<literal>collectl</literal>).</para>
2605 <para>MDT <literal>stats</literal> files can be used to track MDT statistics for the MDS. The
2606 example below shows sample output from an MDT <literal>stats</literal> file.</para>
2607 <screen># lctl get_param mds.*-MDT0000.stats
2608 snapshot_time 1244832003.676892 secs.usecs
2609 open 2 samples [reqs]
2610 close 1 samples [reqs]
2611 getxattr 3 samples [reqs]
2612 process_config 1 samples [reqs]
2613 connect 2 samples [reqs]
2614 disconnect 2 samples [reqs]
2615 statfs 3 samples [reqs]
2616 setattr 1 samples [reqs]
2617 getattr 3 samples [reqs]
2618 llog_init 6 samples [reqs]
2619 notify 16 samples [reqs]</screen>
git://git.whamcloud.com / doc / manual.git / blob