</listitem>
<listitem>
<para>On the server side, view the statistics at:</para>
- <screen>/proc/fs/lustre/obdecho/<replaceable>echo_srv</replaceable>/stats</screen>
+ <screen>lctl get_param obdecho.<replaceable>echo_srv</replaceable>.stats</screen>
<para>where <literal><replaceable>echo_srv</replaceable></literal>
is the <literal>obdecho</literal> server created by the script.</para>
</listitem>
numbers of I/Os in flight.</para>
<para>It is also useful to monitor and record average disk I/O sizes
during each test using the 'disk io size' histogram in the
- file <literal>/proc/fs/lustre/obdfilter/*/brw_stats</literal>
+ file <literal>lctl get_param obdfilter.*.brw_stats</literal>
(see <xref linkend="dbdoclet.50438271_55057"/> for details). These
numbers help identify problems in the system when full-sized I/Os are
not submitted to the underlying disk. This may be caused by problems in
</screen>
<para>Global quota limits are stored in dedicated index files (there is one
such index per quota type) on the quota master target (aka QMT). The QMT
- runs on MDT0000 and exports the global indexes via /proc. The global
- indexes can thus be dumped via the following command:
+ runs on MDT0000 and exports the global indices via <replaceable>lctl
+ get_param</replaceable>. The global indices can thus be dumped via the
+ following command:
<screen>
# lctl get_param qmt.testfs-QMT0000.*.glb-*
</screen>The format of global indexes depends on the OSD type. The ldiskfs OSD
slave is disconnected, the index version is used to determine whether the
slave copy of the global index isn't up to date any more. If so, the slave
fetches the whole index again and updates the local copy. The slave copy of
- the global index is also exported via /proc and can be accessed via the
- following command:
+ the global index can also be accessed via the following command:
<screen>
lctl get_param osd-*.*.quota_slave.limit*
</screen></para>
<emphasis role="bold">Use the same user IDs (UID) and group IDs
(GID) on all clients.</emphasis>
</emphasis>If use of supplemental groups is required, see
- <xref linkend="dbdoclet.50438291_32926" />for information about
- supplementary user and group cache upcall (
- <code>identity_upcall</code>).</para>
+ <xref linkend="dbdoclet.identity_upcall" /> for information about
+ supplementary user and group cache upcall (<code>identity_upcall</code>).</para>
</listitem>
<listitem>
<para>
<section xml:id="dbdoclet.50438274_15874">
<title><indexterm><primary>debugging</primary></indexterm>
Diagnostic and Debugging Tools</title>
- <para>A variety of diagnostic and analysis tools are available to debug issues with the Lustre software. Some of these are provided in Linux distributions, while others have been developed and are made available by the Lustre project.</para>
+ <para>A variety of diagnostic and analysis tools are available to debug
+ issues with the Lustre software. Some of these are provided in Linux
+ distributions, while others have been developed and are made available
+ by the Lustre project.</para>
<section remap="h3" xml:id="section_dms_q21_kk">
<title><indexterm>
<primary>debugging</primary>
<secondary>tools</secondary>
</indexterm> Lustre Debugging Tools</title>
- <para>The following in-kernel debug mechanisms are incorporated into the Lustre
- software:</para>
+ <para>The following in-kernel debug mechanisms are incorporated into
+ the Lustre software:</para>
<itemizedlist>
<listitem>
- <para xml:id="para_fkj_rld_hk"><emphasis role="bold">Debug logs</emphasis> - A circular
- debug buffer to which Lustre internal debug messages are written (in contrast to error
- messages, which are printed to the syslog or console). Entries to the Lustre debug log
- are controlled by the mask set by <literal>/proc/sys/lnet/debug</literal>. The log size
- defaults to 5 MB per CPU but can be increased as a busy system will quickly overwrite 5
- MB. When the buffer fills, the oldest information is discarded.</para>
+ <para xml:id="para_fkj_rld_hk"><emphasis role="bold">Debug logs</emphasis>
+ - A circular debug buffer to which Lustre internal debug messages
+ are written (in contrast to error messages, which are printed to the
+ syslog or console). Entries in the Lustre debug log are controlled
+ by a mask set by <literal>lctl set_param debug=<replaceable>mask</replaceable></literal>.
+ The log size defaults to 5 MB per CPU but can be increased as a
+ busy system will quickly overwrite 5 MB. When the buffer fills,
+ the oldest log records are discarded.</para>
</listitem>
<listitem>
- <para><emphasis role="bold">Debug daemon</emphasis> - The debug daemon controls logging of
- debug messages.</para>
+ <para><emphasis role="bold">
+ <literal>lctl get_param debug</literal>
+ </emphasis> - This shows the current debug mask used to delimit
+ the debugging information written out to the kernel debug logs.
+ </para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">
+ <literal>lctl debug_kernel <replaceable>file</replaceable></literal>
+ </emphasis> - Dump the Lustre kernel debug log to the specified
+ file as ASCII text for further debugging and analysis.
+ </para>
</listitem>
<listitem>
<para><emphasis role="bold">
- <literal>/proc/sys/lnet/debug</literal>
- </emphasis> - This file contains a mask that can be used to delimit the debugging
- information written out to the kernel debug logs.</para>
+ <literal>lctl set_param debug_mb=<replaceable>size</replaceable></literal>
+ </emphasis> - This sets the maximum size of the in-kernel Lustre
+ debug buffer, in units of MiB.
+ </para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Debug daemon</emphasis>
+ - The debug daemon controls the continuous logging of debug
+ messages to a log file in userspace.</para>
</listitem>
</itemizedlist>
<para>The following tools are also provided with the Lustre software:</para>
<listitem>
<para><emphasis role="bold">
<literal>lctl</literal>
- </emphasis> - This tool is used with the debug_kernel option to manually dump the Lustre
- debugging log or post-process debugging logs that are dumped automatically. For more
- information about the lctl tool, see <xref linkend="dbdoclet.50438274_62472"/> and <xref
+ </emphasis> - This tool is used with the debug_kernel option to
+ manually dump the Lustre debugging log or post-process debugging
+ logs that are dumped automatically. For more information about the
+ lctl tool, see <xref linkend="dbdoclet.50438274_62472"/> and <xref
linkend="dbdoclet.50438219_38274"/>.</para>
</listitem>
<listitem>
<para> <emphasis role="bold">trace</emphasis></para>
</entry>
<entry>
- <para> Entry/Exit markers</para>
+ <para> Function entry/exit markers</para>
</entry>
</row>
<row>
<para> <emphasis role="bold">dlmtrace</emphasis></para>
</entry>
<entry>
- <para> Locking-related information</para>
+ <para> Distributed locking-related information</para>
</entry>
</row>
<row>
</row>
<row>
<entry>
- <para> <emphasis role="bold">ext2</emphasis></para>
+ <para> <emphasis role="bold">malloc</emphasis></para>
</entry>
<entry>
- <para> Anything from the ext2_debug</para>
+ <para> Memory allocation or free information</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">malloc</emphasis></para>
+ <para> <emphasis role="bold">cache</emphasis></para>
</entry>
<entry>
- <para> Print malloc or free information</para>
+ <para> Cache-related information</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">cache</emphasis></para>
+ <para> <emphasis role="bold">info</emphasis></para>
</entry>
<entry>
- <para> Cache-related information</para>
+ <para> Non-critical general information</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">info</emphasis></para>
+ <para> <emphasis role="bold">dentry</emphasis></para>
</entry>
<entry>
- <para> General information</para>
+ <para> kernel namespace cache handling</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">ioctl</emphasis></para>
+ <para> <emphasis role="bold">mmap</emphasis></para>
</entry>
<entry>
- <para> IOCTL-related information</para>
+ <para> Memory-mapped IO interface</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">blocks</emphasis></para>
+ <para> <emphasis role="bold">page</emphasis></para>
</entry>
<entry>
- <para> Ext2 block allocation information</para>
+ <para> Page cache and bulk data transfers</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">info</emphasis></para>
+ </entry>
+ <entry>
+ <para> Miscellaneous informational messages</para>
</entry>
</row>
<row>
<para> <emphasis role="bold">net</emphasis></para>
</entry>
<entry>
- <para> Networking</para>
+ <para> LNet network related debugging</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <emphasis role="bold">console</emphasis></para>
+ </entry>
+ <entry>
+ <para>Significant system events, printed to console</para>
</entry>
</row>
<row>
<para> <emphasis role="bold">warning</emphasis></para>
</entry>
<entry>
- <para>  </para>
+ <para>Significant but non-fatal exceptions, printed
+ to console</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">buffs</emphasis></para>
+ <para> <emphasis role="bold">error</emphasis></para>
</entry>
<entry>
- <para>  </para>
+ <para>Critical error messages, printed to console</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">other</emphasis></para>
+ <para> <emphasis role="bold">neterror</emphasis></para>
</entry>
<entry>
- <para>  </para>
+ <para>Significant LNet error messages</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">dentry</emphasis></para>
+ <para> <emphasis role="bold">emerg</emphasis></para>
</entry>
<entry>
- <para>  </para>
+ <para>Fatal system errors, printed to console</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">portals</emphasis></para>
+ <para><emphasis role="bold">config</emphasis></para>
</entry>
<entry>
- <para> Entry/Exit markers</para>
+ <para>Configuration and setup, enabled by default</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">page</emphasis></para>
+ <para><emphasis role="bold">ha</emphasis></para>
</entry>
<entry>
- <para> Bulk page handling</para>
+ <para>Failover and recovery-related information,
+ enabled by default</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">error</emphasis></para>
+ <para><emphasis role="bold">hsm</emphasis></para>
</entry>
<entry>
- <para> Error messages</para>
+ <para>Hierarchical space management/tiering</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">emerg</emphasis></para>
+ <para><emphasis role="bold">ioctl</emphasis></para>
</entry>
<entry>
- <para>  </para>
+ <para>IOCTL-related information, enabled by default</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><emphasis role="bold">layout</emphasis></para>
+ </entry>
+ <entry>
+ <para>File layout handling (PFL, FLR, DoM)</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><emphasis role="bold">lfsck</emphasis></para>
+ </entry>
+ <entry>
+ <para>Filesystem consistency checking, enabled by
+ default</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><emphasis role="bold">other</emphasis></para>
+ </entry>
+ <entry>
+ <para>Miscellaneious other debug messages</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><emphasis role="bold">quota</emphasis></para>
+ </entry>
+ <entry>
+ <para>Space accounting and management</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><emphasis role="bold">reada</emphasis></para>
+ </entry>
+ <entry>
+ <para>Client readahead management</para>
</entry>
</row>
<row>
<para> <emphasis role="bold">rpctrace</emphasis></para>
</entry>
<entry>
- <para> For distributed debugging</para>
+ <para>Remote request/reply tracing and debugging</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><emphasis role="bold">sec</emphasis></para>
+ </entry>
+ <entry>
+ <para>Security, Kerberos, Shared Secret Key handling</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><emphasis role="bold">snapshot</emphasis></para>
+ </entry>
+ <entry>
+ <para>Filesystem snapshot management</para>
</entry>
</row>
<row>
<entry>
- <para> <emphasis role="bold">ha</emphasis></para>
+ <para><emphasis role="bold">vfstrace</emphasis></para>
</entry>
<entry>
- <para> Failover and recovery-related information</para>
+ <para>Kernel VFS interface operations</para>
</entry>
</row>
</tbody>
<para>Buffers are culled from the service request buffer history if it has grown above <literal>req_buffer_history_max</literal> and its reqs are removed from the service request history.</para>
</listitem>
</orderedlist>
- <para>Request history is accessed and controlled using the following /proc files under the service directory:</para>
+ <para>Request history is accessed and controlled using the following parameters for each service:</para>
<itemizedlist>
<listitem>
<para><literal>req_buffer_history_len </literal></para>
<para>If UID 11002 or GID 11001 do not exist on the Lustre MDS or MGS,
create them in LDAP or other data sources, or trust clients by setting
<literal>identity_upcall</literal> to <literal>NONE</literal>. For more
- information, see <xref linkend="dbdoclet.50438291_32926"/>.</para>
+ information, see <xref linkend="dbdoclet.identity_upcall"/>.</para>
<para>Building a larger and more complex configuration is possible by
iterating through the <literal>lctl</literal> commands above. In
<section xml:id="verifyingsettings">
<title>Verifying Settings</title>
- <para>By using <literal>lctl nodemap_info all</literal>, existing
- nodemap configuration is listed for easy export. This command
- acts as a shortcut into the /proc interface for nodemap.
- Within /proc/fs/lustre/nodemap/ on the Lustre MGS, the
- file <literal>active</literal> contains a 1 if nodemap is active on the
- system. Each policy group creates a directory containing the
- following parameters:</para>
+ <para>By using <literal>lctl nodemap_info all</literal>, existing nodemap
+ configuration is listed for easy export. This command acts as a shortcut
+ into the configuration interface for nodemap. On the Lustre MGS, the
+ <literal>nodemap.active</literal> parameter contains a <literal>1</literal>
+ if nodemap is active on the system. Each policy group
+ creates a directory containing the following parameters:</para>
<itemizedlist>
<listitem>
- <para><literal>admin</literal> and
- <literal>trusted</literal> each contain a ‘1’ if the values
- are set, and a ‘0’ otherwise.</para>
+ <para><literal>admin</literal> and <literal>trusted</literal> each
+ contain a <literal>1</literal> if the values are set, and
+ <literal>0</literal> otherwise.</para>
</listitem>
<listitem>
<replaceable>new_parameters</replaceable>
</screen>
<para>The tunefs.lustre command can be used to set any parameter settable
- in a /proc/fs/lustre file and that has its own OBD device, so it can be
- specified as
+ via <literal>lctl conf_param</literal> and that has its own OBD device,
+ so it can be specified as
<literal>
<replaceable>obdname|fsname</replaceable>.
<replaceable>obdtype</replaceable>.
are active as long as the server or client is not shut down. Permanent
parameters live through server and client reboots.</para>
<note>
- <para>The lctl list_param command enables users to list all parameters
- that can be set. See
+ <para>The <literal>lctl list_param</literal> command enables users to
+ list all parameters that can be set. See
<xref linkend="dbdoclet.50438194_88217" />.</para>
</note>
<para>For more details about the
<literal>/proc/{fs,sys}/{lnet,lustre}</literal>. The
<literal>lctl set_param</literal> command uses this syntax:</para>
<screen>
-lctl set_param [-n]
+lctl set_param [-n] [-P]
<replaceable>obdtype</replaceable>.
<replaceable>obdname</replaceable>.
<replaceable>proc_file_name</replaceable>=
</section>
<section xml:id="dbdoclet.50438194_64195">
<title>Setting Permanent Parameters</title>
- <para>Use the
+ <para>Use <literal>lctl set_param -P</literal> or
<literal>lctl conf_param</literal> command to set permanent parameters.
In general, the
<literal>lctl conf_param</literal> command can be used to specify any
</section>
<section xml:id="dbdoclet.setparamp" condition='l25'>
<title>Setting Permanent Parameters with lctl set_param -P</title>
- <para>Use the
- <literal>lctl set_param -P</literal> to set parameters permanently. This
- command must be issued on the MGS. The given parameter is set on every
- host using
- <literal>lctl</literal> upcall. Parameters map to items in
- <literal>/proc/{fs,sys}/{lnet,lustre}</literal>. The
- <literal>lctl set_param</literal> command uses this syntax:</para>
+ <para>The <literal>lctl set_param -P</literal> command can also
+ set parameters permanently. This command must be issued on the MGS.
+ The given parameter is set on every host using
+ <literal>lctl</literal> upcall. Parameters map to items in
+ <literal>/proc/{fs,sys}/{lnet,lustre}</literal>. The
+ <literal>lctl set_param</literal> command uses this syntax:</para>
<screen>
lctl set_param -P
<replaceable>obdtype</replaceable>.
</para>
<para>Typically, metrics are accessed via <literal>lctl get_param</literal>
files and settings are changed by via <literal>lctl set_param</literal>.
+ While it is possible to access parameters in <literal>/proc</literal>
+ and <literal>/sys</literal> directly, the location of these parameters may
+ change between releases, so it is recommended to always use
+ <literal>lctl</literal> to access the parameters from userspace scripts.
Some data is server-only, some data is client-only, and some data is
exported from the client to the server and is thus duplicated in both
locations.</para>
osc.testfs-OST0007-osc-ffff881071d5cc00
osc.testfs-OST0008-osc-ffff881071d5cc00</screen>
<para>In this example, information about OST connections available
- on a client is displayed (indicated by "osc").</para>
+ on a client is displayed (indicated by "osc").</para>
</listitem>
</itemizedlist>
<itemizedlist>
</listitem>
<listitem>
<para>Prepend the path with the appropriate directory component:
- <screen>/{proc,sys}/{fs,sys}/{lustre,lnet}</screen></para>
+ <screen>/{proc,sys}/{fs,sys}/{lustre,lnet}</screen></para>
</listitem>
</itemizedlist>
<para>For example, an <literal>lctl get_param</literal> command may look like
# hash ldlm_stats stats uuid</screen></para>
<section remap="h3">
<title>Identifying Lustre File Systems and Servers</title>
- <para>Several <literal>/proc</literal> files on the MGS list existing
+ <para>Several parameter files on the MGS list existing
Lustre file systems and file system servers. The examples below are for
a Lustre file system called
<literal>testfs</literal> with one MDT and three OSTs.</para>
notify_count: 4</screen>
</listitem>
<listitem>
- <para>To view the names of all live servers in the file system as listed in
- <literal>/proc/fs/lustre/devices</literal>, enter:</para>
+ <para>To list all configured devices on the local node, enter:</para>
<screen># lctl device_list
0 UP mgs MGS MGS 11
1 UP mgc MGC192.168.10.34@tcp 1f45bb57-d9be-2ddb-c0b0-5431a49226705
<row>
<entry>
<para>
- <literal>mb_prealloc_table</literal></para>
+ <literal>prealloc_table</literal></para>
</entry>
<entry>
<para>A table of values used to preallocate space when a new request is received. By
</tgroup>
</informaltable>
<para>Buddy group cache information found in
- <literal>/proc/fs/ldiskfs/<replaceable>disk_device</replaceable>/mb_groups</literal> may
+ <literal>/sys/fs/ldiskfs/<replaceable>disk_device</replaceable>/mb_groups</literal> may
be useful for assessing on-disk fragmentation. For
example:<screen>cat /proc/fs/ldiskfs/loop0/mb_groups
#group: free free frags first pa [ 2^0 2^1 2^2 2^3 2^4 2^5 2^6 2^7 2^8 2^9
</informaltable>
<section>
<title>Interpreting Adaptive Timeout Information</title>
- <para>Adaptive timeout information can be obtained from the <literal>timeouts</literal>
- files in <literal>/proc/fs/lustre/*/</literal> on each server and client using the
- <literal>lctl</literal> command. To read information from a <literal>timeouts</literal>
- file, enter a command similar to:</para>
+ <para>Adaptive timeout information can be obtained via
+ <literal>lctl get_param {osc,mdc}.*.timeouts</literal> files on each
+ client and <literal>lctl get_param {ost,mds}.*.*.timeouts</literal>
+ on each server. To read information from a
+ <literal>timeouts</literal> file, enter a command similar to:</para>
<screen># lctl get_param -n ost.*.ost_io.timeouts
-service : cur 33 worst 34 (at 1193427052, 0d0h26m40s ago) 1 1 33 2</screen>
- <para>In this example, the <literal>ost_io</literal> service on this node is currently
- reporting an estimated RPC service time of 33 seconds. The worst RPC service time was 34
- seconds, which occurred 26 minutes ago.</para>
- <para>The output also provides a history of service times. Four "bins" of adaptive
- timeout history are shown, with the maximum RPC time in each bin reported. In both the
- 0-150s bin and the 150-300s bin, the maximum RPC time was 1. The 300-450s bin shows the
- worst (maximum) RPC time at 33 seconds, and the 450-600s bin shows a maximum of RPC time
- of 2 seconds. The estimated service time is the maximum value across the four bins (33
- seconds in this example).</para>
- <para>Service times (as reported by the servers) are also tracked in the client OBDs, as
- shown in this example:</para>
+service : cur 33 worst 34 (at 1193427052, 1600s ago) 1 1 33 2</screen>
+ <para>In this example, the <literal>ost_io</literal> service on this
+ node is currently reporting an estimated RPC service time of 33
+ seconds. The worst RPC service time was 34 seconds, which occurred
+ 26 minutes ago.</para>
+ <para>The output also provides a history of service times.
+ Four "bins" of adaptive timeout history are shown, with the
+ maximum RPC time in each bin reported. In both the 0-150s bin and the
+ 150-300s bin, the maximum RPC time was 1. The 300-450s bin shows the
+ worst (maximum) RPC time at 33 seconds, and the 450-600s bin shows a
+ maximum of RPC time of 2 seconds. The estimated service time is the
+ maximum value in the four bins (33 seconds in this example).</para>
+ <para>Service times (as reported by the servers) are also tracked in
+ the client OBDs, as shown in this example:</para>
<screen># lctl get_param osc.*.timeouts
last reply : 1193428639, 0d0h00m00s ago
network : cur 1 worst 2 (at 1193427053, 0d0h26m26s ago) 1 1 1 1
portal 7 : cur 1 worst 1 (at 1193426141, 0d0h41m38s ago) 1 0 1 1
portal 17 : cur 1 worst 1 (at 1193426177, 0d0h41m02s ago) 1 0 0 1
</screen>
- <para>In this example, portal 6, the <literal>ost_io</literal> service portal, shows the
- history of service estimates reported by the portal.</para>
- <para>Server statistic files also show the range of estimates including min, max, sum, and
- sumsq. For example:</para>
+ <para>In this example, portal 6, the <literal>ost_io</literal> service
+ portal, shows the history of service estimates reported by the portal.
+ </para>
+ <para>Server statistic files also show the range of estimates including
+ min, max, sum, and sum-squared. For example:</para>
<screen># lctl get_param mdt.*.mdt.stats
...
req_timeout 6 samples [sec] 1 10 15 105
<primary>LNet</primary>
<secondary>proc</secondary>
</indexterm>Monitoring LNet</title>
- <para>LNet information is located in <literal>/proc/sys/lnet</literal> in these files:<itemizedlist>
+ <para>LNet information is located via <literal>lctl get_param</literal>
+ in these parameters:
+ <itemizedlist>
<listitem>
- <para><literal>peers</literal> - Shows all NIDs known to this node and provides
- information on the queue state.</para>
+ <para><literal>peers</literal> - Shows all NIDs known to this node
+ and provides information on the queue state.</para>
<para>Example:</para>
<screen># lctl get_param peers
nid refs state max rtr min tx min queue
maximizes network balancing. The weighted allocator is used when any two
OSTs are out of balance by more than a specified threshold.</para>
<para>Free space distribution can be tuned using these two
- <literal>/proc</literal> tunables:</para>
+ tunable parameters:</para>
<itemizedlist>
<listitem>
- <para><literal>qos_threshold_rr</literal> - The threshold at which
+ <para><literal>lod.*.qos_threshold_rr</literal> - The threshold at which
the allocation method switches from round-robin to weighted is set
in this file. The default is to switch to the weighted algorithm when
any two OSTs are out of balance by more than 17 percent.</para>
</listitem>
<listitem>
- <para><literal>qos_prio_free</literal> - The weighting priority used
- by the weighted allocator can be adjusted in this file. Increasing the
- value of <literal>qos_prio_free</literal> puts more weighting on the
+ <para><literal>lod.*.qos_prio_free</literal> - The weighting priority
+ used by the weighted allocator can be adjusted in this file. Increasing
+ the value of <literal>qos_prio_free</literal> puts more weighting on the
amount of free space available on each OST and less on how stripes are
distributed across OSTs. The default value is 91 percent weighting for
free space rebalancing and 9 percent for OST balancing. When the
space and location is no longer used by the striping algorithm.</para>
</listitem>
<listitem>
- <para condition="l29"><literal>reserved_mb_low</literal> - The low
- watermark used to stop object allocation if available space is less
- than it. The default is 0.1 percent of total OST size.</para>
+ <para condition="l29"><literal>osp.*.reserved_mb_low</literal>
+ - The low watermark used to stop object allocation if available space
+ is less than this. The default is 0.1% of total OST size.</para>
</listitem>
<listitem>
- <para condition="l29"><literal>reserved_mb_high</literal> - The high watermark used to start
- object allocation if available space is more than it. The default is 0.2 percent of total
- OST size.</para>
+ <para condition="l29"><literal>osp.*.reserved_mb_high</literal>
+ - The high watermark used to start object allocation if available
+ space is more than this. The default is 0.2% of total OST size.</para>
</listitem>
</itemizedlist>
<para>For more information about monitoring and managing free space, see <xref
</tbody>
</tgroup>
</informaltable>
- <para>For each service, an entry as shown below is
- created:<screen>/proc/fs/lustre/<replaceable>service</replaceable>/*/threads_<replaceable>min|max|started</replaceable></screen></para>
+ <para>For each service, tunable parameters as shown below are available.
+ </para>
<itemizedlist>
<listitem>
- <para>To temporarily set this tunable, run:</para>
- <screen># lctl <replaceable>get|set</replaceable>_param <replaceable>service</replaceable>.threads_<replaceable>min|max|started</replaceable> </screen>
+ <para>To temporarily set these tunables, run:</para>
+ <screen># lctl set_param <replaceable>service</replaceable>.threads_<replaceable>min|max|started=num</replaceable> </screen>
</listitem>
<listitem>
<para>To permanently set this tunable, run:</para>
<listitem>
<para>To set the maximum thread count to 256 instead of 512 permanently, run:</para>
<screen># lctl conf_param testfs.ost.ost_io.threads_max=256</screen>
- <para condition='l25'>For version 2.5 or later, run:
- <screen># lctl set_param -P ost.OSS.ost_io.threads_max=256
+
<para condition='l25'>For version 2.5 or later, run:
+ <screen># lctl set_param -P ost.OSS.ost_io.threads_max=256
ost.OSS.ost_io.threads_max=256 </screen> </para>
</listitem>
<listitem>
<primary>proc</primary>
<secondary>debug</secondary>
</indexterm>Enabling and Interpreting Debugging Logs</title>
- <para>By default, a detailed log of all operations is generated to aid in debugging. Flags that
- control debugging are found in <literal>/proc/sys/lnet/debug</literal>. </para>
- <para>The overhead of debugging can affect the performance of Lustre file system. Therefore, to
- minimize the impact on performance, the debug level can be lowered, which affects the amount
- of debugging information kept in the internal log buffer but does not alter the amount of
- information to goes into syslog. You can raise the debug level when you need to collect logs
- to debug problems. </para>
- <para>The debugging mask can be set using "symbolic names". The symbolic format is
- shown in the examples below.<itemizedlist>
+ <para>By default, a detailed log of all operations is generated to aid in
+ debugging. Flags that control debugging are found via
+ <literal>lctl get_param debug</literal>.</para>
+ <para>The overhead of debugging can affect the performance of Lustre file
+ system. Therefore, to minimize the impact on performance, the debug level
+ can be lowered, which affects the amount of debugging information kept in
+ the internal log buffer but does not alter the amount of information to
+ goes into syslog. You can raise the debug level when you need to collect
+ logs to debug problems. </para>
+ <para>The debugging mask can be set using "symbolic names". The
+ symbolic format is shown in the examples below.
+ <itemizedlist>
<listitem>
- <para>To verify the debug level used, examine the <literal>sysctl</literal> that controls
- debugging by running:</para>
- <screen># sysctl lnet.debug
-lnet.debug = ioctl neterror warning error emerg ha config console</screen>
+ <para>To verify the debug level used, examine the parameter that
+ controls debugging by running:</para>
+ <screen># lctl get_param debug
+debug=
+ioctl neterror warning error emerg ha config console</screen>
</listitem>
<listitem>
- <para>To turn off debugging (except for network error debugging), run the following
- command on all nodes concerned:</para>
+ <para>To turn off debugging except for network error debugging, run
+ the following command on all nodes concerned:</para>
<screen># sysctl -w lnet.debug="neterror"
-lnet.debug = neterror</screen>
+debug=neterror</screen>
</listitem>
- </itemizedlist><itemizedlist>
+ </itemizedlist>
+ <itemizedlist>
<listitem>
- <para>To turn off debugging completely, run the following command on all nodes
+ <para>To turn off debugging completely (except for the minimum error
+ reporting to the console), run the following command on all nodes
concerned:</para>
- <screen># sysctl -w lnet.debug=0
-lnet.debug = 0</screen>
- </listitem>
- <listitem>
- <para>To set an appropriate debug level for a production environment, run:</para>
- <screen># sysctl -w lnet.debug="warning dlmtrace error emerg ha rpctrace vfstrace"
-lnet.debug = warning dlmtrace error emerg ha rpctrace vfstrace</screen>
- <para>The flags shown in this example collect enough high-level information to aid
- debugging, but they do not cause any serious performance impact.</para>
+ <screen># lctl set_param debug=0
+debug=0</screen>
</listitem>
- </itemizedlist><itemizedlist>
<listitem>
- <para>To clear all flags and set new flags, run:</para>
- <screen># sysctl -w lnet.debug="warning"
-lnet.debug = warning</screen>
+ <para>To set an appropriate debug level for a production environment,
+ run:</para>
+ <screen># lctl set_param debug="warning dlmtrace error emerg ha rpctrace vfstrace"
+debug=warning dlmtrace error emerg ha rpctrace vfstrace</screen>
+ <para>The flags shown in this example collect enough high-level
+ information to aid debugging, but they do not cause any serious
+ performance impact.</para>
</listitem>
- </itemizedlist><itemizedlist>
+ </itemizedlist>
+ <itemizedlist>
<listitem>
- <para>To add new flags to flags that have already been set, precede each one with a
- "<literal>+</literal>":</para>
- <screen># sysctl -w lnet.debug="+neterror +ha"
-lnet.debug = +neterror +ha
-# sysctl lnet.debug
-lnet.debug = neterror warning ha</screen>
+ <para>To add new flags to flags that have already been set,
+ precede each one with a "<literal>+</literal>":</para>
+ <screen># lctl set_param debug="+neterror +ha"
+debug=+neterror +ha
+# lctl get_param debug
+debug=neterror warning error emerg ha console</screen>
</listitem>
<listitem>
<para>To remove individual flags, precede them with a
"<literal>-</literal>":</para>
- <screen># sysctl -w lnet.debug="-ha"
-lnet.debug = -ha
-# sysctl lnet.debug
-lnet.debug = neterror warning</screen>
- </listitem>
- <listitem>
- <para>To verify or change the debug level, run commands such as the following: :</para>
- <screen># lctl get_param debug
-debug=
-neterror warning
-# lctl set_param debug=+ha
-# lctl get_param debug
-debug=
-neterror warning ha
-# lctl set_param debug=-warning
-# lctl get_param debug
-debug=
-neterror ha</screen>
+ <screen># lctl set_param debug="-ha"
+debug=-ha
+# lctl get_param debug
+debug=neterror warning error emerg console</screen>
</listitem>
- </itemizedlist></para>
+ </itemizedlist>
+ </para>
<para>Debugging parameters include:</para>
<itemizedlist>
<listitem>
<literal>/tmp/lustre-log</literal>.</para>
</listitem>
</itemizedlist>
- <para>These parameters are also set using:<screen>sysctl -w lnet.debug={value}</screen></para>
+ <para>These parameters can also be set using:<screen>sysctl -w lnet.debug={value}</screen></para>
<para>Additional useful parameters: <itemizedlist>
<listitem>
<para><literal>panic_on_lbug</literal> - Causes ''panic'' to be called
obd_ping 212</screen>
<para>Use the <literal>llstat</literal> utility to monitor statistics over time.</para>
<para>To clear the statistics, use the <literal>-c</literal> option to
- <literal>llstat</literal>. To specify how frequently the statistics should be reported (in
- seconds), use the <literal>-i</literal> option. In the example below, the
- <literal>-c</literal> option clears the statistics and <literal>-i10</literal> option
- reports statistics every 10 seconds:</para>
- <screen role="smaller">$ llstat -c -i10 /proc/fs/lustre/ost/OSS/ost_io/stats
+ <literal>llstat</literal>. To specify how frequently the statistics
+ should be reported (in seconds), use the <literal>-i</literal> option.
+ In the example below, the <literal>-c</literal> option clears the
+ statistics and <literal>-i10</literal> option reports statistics every
+ 10 seconds:</para>
+<screen role="smaller">$ llstat -c -i10 ost_io
/usr/bin/llstat: STATS on 06/06/07
/proc/fs/lustre/ost/OSS/ost_io/ stats on 192.168.16.35@tcp
<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustreprogramminginterfaces">
<title xml:id="lustreprogramminginterfaces.title">Programming Interfaces</title>
- <para>This chapter describes public programming interfaces to that can be used to control various
- aspects of a Lustre file system from userspace. This chapter includes the following
- sections:</para>
+ <para>This chapter describes public programming interfaces to that can be
+ used to control various aspects of a Lustre file system from userspace.
+ This chapter includes the following sections:</para>
<itemizedlist>
<listitem>
- <para><xref linkend="dbdoclet.50438291_32926"/></para>
+ <para><xref linkend="dbdoclet.identity_upcall"/></para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438291_73963"/></para>
+ <para><xref linkend="dbdoclet.perm_downcall_data"/></para>
</listitem>
</itemizedlist>
<note>
<para>Lustre programming interface man pages are found in the <literal>lustre/doc</literal> folder.</para>
</note>
- <section xml:id="dbdoclet.50438291_32926">
+ <section xml:id="dbdoclet.identity_upcall">
<title><indexterm>
<primary>programming</primary>
<secondary>upcall</secondary>
</indexterm>User/Group Upcall</title>
- <para>This section describes the supplementary user/group upcall, which allows the MDS to
- retrieve and verify the supplementary groups to which a particular user is assigned. This
- avoids the need to pass all the supplementary groups from the client to the MDS with every
- RPC.</para>
+ <para>This section describes the supplementary user/group upcall, which
+ allows the MDS to retrieve and verify the supplementary groups to which
+ a particular user is assigned. This avoids the need to pass all the
+ supplementary groups from the client to the MDS with every RPC.</para>
<note>
- <para>For information about universal UID/GID requirements in a Lustre file system
- environment, see <xref xmlns:xlink="http://www.w3.org/1999/xlink"
- linkend="section_rh2_d4w_gk"/>.</para>
+ <para>For information about universal UID/GID requirements in a Lustre
+ file system environment, see
+ <xref xmlns:xlink="http://www.w3.org/1999/xlink"
+ linkend="section_rh2_d4w_gk"/>.</para>
</note>
<section remap="h3">
<title>Synopsis</title>
- <para>The MDS uses the utility as specified by <literal>lctl get_param
- mdt.${FSNAME}-MDT{xxxx}.identity_upcall</literal> to look up the supplied UID in order to
- retrieve the user's supplementary group membership. The result is temporarily cached in the
- kernel (for five minutes, by default) to avoid the overhead of calling into userspace
- repeatedly.</para>
+ <para>The MDS uses the utility as specified by
+ <literal>lctl get_param mdt.${FSNAME}-MDT{xxxx}.identity_upcall</literal>
+ to look up the supplied UID in order to retrieve the user's supplementary
+ group membership. The result is temporarily cached in the kernel (for
+ five minutes, by default) to avoid the overhead of calling into
+ userspace repeatedly.</para>
</section>
<section remap="h3">
<title>Description</title>
- <para>The identity upcall file contains the path to an executable that is invoked to resolve a
- numeric UID to a group membership list. This utility opens
- <literal>/proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_info</literal> and fills in the
- related <literal>identity_downcall_data</literal> data structure (see <xref
- linkend="dbdoclet.50438291_33759"/>). The data is persisted with <literal>lctl set_param
- mdt.${FSNAME}-MDT{xxxx}.identity_info</literal>.</para>
- <para>For a sample upcall program, see <literal>lustre/utils/l_getidentity.c</literal> in the Lustre source distribution.</para>
+ <para>The <literal>identity_upcall</literal> parameter contains the path
+ to an executable that is run to map a numeric UID to a group membership
+ list. This upcall executable opens the
+ <literal>mdt.${FSNAME}-MDT{xxxx}.identity_info</literal> parameter file
+ and writes the related <literal>identity_downcall_data</literal> data
+ structure (see <xref linkend="dbdoclet.perm_downcall_data"/>). The
+ upcall is configured with
+ <literal>lctl set_param mdt.${FSNAME}-MDT{xxxx}.identity_upcall</literal>.</para>
+ <para>The default identity upcall program installed is
+ <literal>lustre/utils/l_getidentity.c</literal> in the Lustre source
+ distribution.</para>
<section remap="h4">
<title>Primary and Secondary Groups</title>
- <para>The mechanism for the primary/secondary group is as follows:</para>
+ <para>The mechanism for the primary/secondary group is as follows:
+ </para>
<itemizedlist>
<listitem>
- <para>The MDS issues an upcall (set per MDS) to map the numeric UID to the supplementary
- group(s).</para>
+ <para>The MDS issues an upcall (set per MDS) to map the numeric
+ UID to the supplementary group(s).</para>
</listitem>
<listitem>
- <para>If there is no upcall or if there is an upcall and it fails, one supplementary
- group at most will be added as supplied by the client.</para>
+ <para>If there is no upcall or if there is an upcall and it fails,
+ one supplementary group at most will be added as supplied by the
+ client.</para>
</listitem>
<listitem>
- <para>The default upcall is <literal>/usr/sbin/l_getidentity</literal>, which can
- interact with the user/group database to obtain UID/GID/suppgid. The user/group
- database depends on how authentication is configured, such as local
- <literal>/etc/passwd</literal>, Network Information Service (NIS), or Lightweight
- Directory Access Protocol (LDAP). If necessary, the administrator can use a parse
- utility to set
- <literal>/proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_upcall</literal>. If the
- upcall interface is set to NONE, then upcall is disabled. The MDS uses the
- UID/GID/suppgid supplied by the client.</para>
+ <para>The default upcall <literal>/usr/sbin/l_getidentity</literal>
+ can interact with the user/group database on the MDS to map the
+ UID to the GID and supplementary GID. The user/group database
+ depends on how authentication is configured on the MDS, such as
+ local <literal>/etc/passwd</literal>, Network Information Service
+ (NIS), Lightweight Directory Access Protocol (LDAP), or SMB
+ Domain services, as configured. If the upcall interface is set
+ to NONE, then upcall is disabled, and the MDS uses only the UID,
+ GID, and one supplementary GID supplied by the client.</para>
</listitem>
<listitem>
- <para>The default group upcall is set by <literal>mkfs.lustre</literal>. Use
- <literal>tunefs.lustre --param</literal> or <literal>lctl set_param
- mdt.${FSNAME}-MDT{xxxx}.identity_upcall={path}</literal></para>
+ <para>The MDS will wait a limited time for the group upcall program
+ to complete, to avoid MDS threads and clients hanging due to
+ errors accessing a remote service node. The upcall must finish
+ within 30s before the MDS will continue without the supplementary
+ data. The upcall timeout can be set on the MDS using:
+ <literal>lctl set_param mdt.*.identity_acquire_expire=<replaceable>seconds</replaceable></literal></para>
</listitem>
- </itemizedlist>
- <itemizedlist>
- <para>A Lustre file system administrator can specify permissions for a specific UID by
- configuring <literal>/etc/lustre/perm.conf</literal> on the MDS. The
- <literal>/usr/sbin/l_getidentity</literal> utility parses
- <literal>/etc/lustre/perm.conf</literal> to obtain the permission mask for a specified
- UID.</para>
- <para>The permission file format
- is:<screen>{<replaceable>nid</replaceable>} {<replaceable>uid</replaceable>} {<replaceable>perms</replaceable>}</screen>An
- asterisk (*) in the <replaceable>nid</replaceable> column or
- <replaceable>uid</replaceable> column matches any NID or UID respectively. When '*' is
- specified as the NID, it is used for the default permissions for all NIDS, unless
- permissions are specified for a particular NID. In this case the specified permissions
- take precedence for that particular NID. Valid values for
- <replaceable>perms</replaceable> are:<itemizedlist>
- <listitem>
- <para><literal>setuid/setgid/setgrp/<replaceable>XXX</replaceable></literal> -
- enables the corresponding <replaceable>perm</replaceable></para>
- </listitem>
- <listitem>
- <para><literal>nosetuid/nosetgid/nosetgrp/no<replaceable>XXX</replaceable></literal>
- - disables the corresponding <replaceable>perm</replaceable></para>
- </listitem>
- </itemizedlist>Permissions can be specified in a comma-separated list. When a
- <replaceable>perm</replaceable> and a <replaceable>noperm</replaceable> permission are
- listed in the same line, the <replaceable>noperm</replaceable> permission takes
- precedence. When they are listed on separate lines, the permission that appears later
- takes precedence.</para>
<listitem>
- <?oxy_custom_start type="oxy_content_highlight" color="255,255,0"?>
- <para><?oxy_custom_end?>The <literal>/usr/sbin/l_getidentity</literal> utility can parse
- <literal>/etc/lustre/perm.conf</literal> to obtain the permission mask for the
- specified UID.</para>
+ <para>The default group upcall is set permanently by
+ <literal>mkfs.lustre</literal>. To set a custom upcall for a
+ particular filesystem, use
+ <literal>tunefs.lustre --param</literal> or
+ <literal>lctl set_param -P mdt.<replaceable>FSNAME</replaceable>-MDT<replaceable>xxxx</replaceable>.identity_upcall=<replaceable>path</replaceable></literal></para>
</listitem>
<listitem>
- <para>To avoid repeated upcalls, the MDS caches supplemental group information. Use
- <literal>lctl set_param mdt.*.identity_expire=<seconds></literal> to set the
- cache time. The default cache time is 600 seconds. The kernel waits for the upcall to
- complete (at most, 5 seconds) and takes the "failure" behavior as described. </para>
- <para>Set the wait time using <literal>lctl set_param
- mdt.*.identity_acquire_expire=<seconds></literal> to change the length of time
- that the kernel waits for the upcall to finish. Note that the client process is
- blocked during this time. The default wait time is 15 seconds.</para>
- <para>Cached entries are flushed using <literal>lctl set_param
- mdt.${FSNAME}-MDT{xxxx}.identity_flush=0</literal>.</para>
+ <para>The group downcall data is cached by the kernel to avoid
+ repeated upcalls for the same user slowing down the MDS. This
+ cache is expired from the kernel after 1200s (20 minutes) by
+ default. The cache age can be set on the MDS using:
+ <literal>lctl set_param mdt.*.identity_expire=<replaceable>seconds</replaceable></literal></para>
</listitem>
</itemizedlist>
</section>
</section>
- <section remap="h3">
- <title>Parameters</title>
- <itemizedlist>
- <listitem>
- <para> Name of the MDS service</para>
- </listitem>
- <listitem>
- <para> Numeric UID</para>
- </listitem>
- </itemizedlist>
- </section>
- <section xml:id="dbdoclet.50438291_33759">
+ <section xml:id="dbdoclet.perm_downcall_data">
<title>Data Structures</title>
- <screen>struct perm_downcall_data{
+ <screen>struct perm_downcall_data {
__u64 pdd_nid;
__u32 pdd_perm;
__u32 pdd_padding;
:</screen>
</section>
</section>
- <section xml:id="dbdoclet.50438291_73963">
- <title><indexterm><primary>programming</primary><secondary>l_getidentity</secondary></indexterm><literal>l_getidentity</literal> Utility</title>
- <para>The <literal>l_getidentity</literal> utility handles the Lustre supplementary group upcall
- by default as described in the previous section.</para>
- <section remap="h5">
- <title>Synopsis</title>
- <screen>l_getidentity ${FSNAME}-MDT{xxxx} {uid}</screen>
- </section>
- <section remap="h5">
- <title>Description</title>
- <para>The identity upcall file contains the path to an executable that is invoked to resolve a
- numeric UID to a group membership list. This utility opens
- <literal>/proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_info</literal>, completes the
- <literal>identity_downcall_data</literal> data structure (see <xref
- linkend="dbdoclet.50438291_33759"/>) and writes the data to the
- <literal>/proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_info</literal> pseudo file. The
- data is persisted with <literal>lctl set_param
- mdt.${FSNAME}-MDT{xxxx}.identity_info</literal>.</para>
- <para><literal>l_getidentity</literal> is the reference implementation of the user/group cache
- upcall.</para>
- </section>
- <section remap="h5">
- <title>Files</title>
- <para>
- <screen>/proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_upcall</screen>
- <screen>/proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_info</screen>
- </para>
- </section>
- </section>
</chapter>
<para><xref linkend="dbdoclet.50438219_55923"/></para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438219_76969"/></para>
+ <para><xref linkend="dbdoclet.l_getidentity"/></para>
</listitem>
<listitem>
<para><xref linkend="dbdoclet.50438219_38274"/></para>
</informaltable>
</section>
</section>
- <section xml:id="dbdoclet.50438219_76969">
+ <section xml:id="dbdoclet.l_getidentity">
<title><indexterm><primary>l_getidentity</primary></indexterm>
l_getidentity</title>
- <para>The l_getidentity utility handles Lustre user / group cache upcall.</para>
+ <para>The l_getidentity tool normally handles Lustre user/group mapping
+ upcall.</para>
<section remap="h5">
<title>Synopsis</title>
- <screen>l_getidentity ${FSNAME}-MDT{xxxx} {uid}</screen>
+ <screen>l_getidentity { $FSNAME-MDT{xxxx}| -d} {uid}</screen>
</section>
<section remap="h5">
<title>Description</title>
- <para>The group upcall file contains the path to an executable file that is invoked to resolve
- a numeric UID to a group membership list. This utility opens
- <literal>/proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_info</literal> and writes the
- related <literal>identity_downcall_data</literal> structure (see <xref
- linkend="dbdoclet.50438291_33759"/>.) The data is persisted with <literal>lctl set_param
- mdt.${FSNAME}-MDT{xxxx}.identity_info</literal>.</para>
- <para>The l_getidentity utility is the reference implementation of the user or group cache upcall.</para>
+ <para>The <literal>l_getidentity</literal> utility is called from the
+ MDS to map a numeric UID value into the list of supplementary group
+ values for that UID, and writes this into the
+ <literal>mdt.*.identity_info</literal> parameter file. The list of
+ supplementary groups is cached in the kernel to avoid repeated
+ upcalls. See <xref linkend="dbdoclet.identity_upcall"/> for more
+ details.</para>
+ <para>The <literal>l_getidentity</literal> utility can also be run
+ directly for debugging purposes to ensure that the UID mapping for a
+ particular user is configured correctly, by using the
+ <literal>-d</literal> argument instead of the MDT name.
+ </para>
</section>
<section remap="h5">
<title>Options</title>
git://git.whamcloud.com / doc / manual.git / commitdiff