<?xml version='1.0' encoding='utf-8'?>
<chapter xmlns="http://docbook.org/ns/docbook"
-xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"
-xml:id="lustreoperations">
+ xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"
+ xml:id="lustreoperations">
<title xml:id="lustreoperations.title">Lustre Operations</title>
<para>Once you have the Lustre file system up and running, you can use the
procedures in this section to perform these basic Lustre administration
<listitem>
<para>Mount the MDT.</para>
<note>
- <para condition='l24'>Mount all MDTs if multiple MDTs are
- present.</para>
+ <para>Mount all MDTs if multiple MDTs are present.</para>
</note>
</listitem>
<listitem>
[154523.177714] Lustre: Unmounted testfs-client</screen></para>
</listitem>
<listitem><para>Unmount the MDT and MGT</para>
- <para>On the MGS and MDS node(s), use the <literal>umount</literal>
- command:</para>
+ <para>On the MGS and MDS node(s), run the
+ <literal>umount</literal> command:</para>
<para><literal>umount -a -t lustre</literal></para>
<para>The example below shows the unmount of the MDT and MGT for
- the <literal>testfs</literal> filesystem on a combined MGS/MDS:
- </para>
+ the <literal>testfs</literal> filesystem on a combined MGS/MDS:
+ </para>
<para><screen>[root@mds1 ~]# mount |grep lustre
/dev/sda on /mnt/mgt type lustre (ro)
/dev/sdb on /mnt/mdt type lustre (ro)
[155263.566230] Lustre: Failing over testfs-MDT0000
[155263.775355] Lustre: server umount testfs-MDT0000 complete
[155269.843862] Lustre: server umount MGS complete</screen></para>
- <para>For a seperate MGS and MDS, the same command is used, first on
+
<para>For a seperate MGS and MDS, the same command is used, first on
the MDS and then followed by the MGS.</para>
</listitem>
<listitem><para>Unmount all the OSTs</para>
</para>
<para><literal>umount -a -t lustre</literal></para>
<para>The example below shows the unmount of all OSTs for the
- <literal>testfs</literal> filesystem on server
- <literal>OSS1</literal>:
+ <literal>testfs</literal> filesystem on server
+ <literal>OSS1</literal>:
</para>
<para><screen>[root@oss1 ~]# mount |grep lustre
/dev/sda on /mnt/ost0 type lustre (ro)
client# mount -t lustre mgsnode@tcp0:/bar /mnt/bar
</screen>
</section>
- <section xml:id="dbdoclet.lfsmkdir" condition='l24'>
+ <section xml:id="dbdoclet.lfsmkdir">
<title>
<indexterm>
<primary>operations</primary>
<secondary>remote directory</secondary>
- </indexterm>Creating a sub-directory on a given MDT</title>
- <para>Lustre 2.4 enables individual sub-directories to be serviced by
- unique MDTs. An administrator can allocate a sub-directory to a given MDT
- using the command:</para>
+ </indexterm>Creating a sub-directory on a specific MDT</title>
+ <para>It is possible to create individual directories, along with its
+ files and sub-directories, to be stored on specific MDTs. To create
+ a sub-directory on a given MDT use the command:
+ </para>
<screen>
client# lfs mkdir –i
<replaceable>mdt_index</replaceable>
<warning>
<para>An administrator can allocate remote sub-directories to separate
MDTs. Creating remote sub-directories in parent directories not hosted on
- MDT0 is not recommended. This is because the failure of the parent MDT
+ MDT0000 is not recommended. This is because the failure of the parent MDT
will leave the namespace below it inaccessible. For this reason, by
- default it is only possible to create remote sub-directories off MDT0. To
- relax this restriction and enable remote sub-directories off any MDT, an
- administrator must issue the following command on the MGS:
+ default it is only possible to create remote sub-directories off MDT0000.
+ To relax this restriction and enable remote sub-directories off any MDT,
+ an administrator must issue the following command on the MGS:
<screen>mgs# lctl conf_param <replaceable>fsname</replaceable>.mdt.enable_remote_dir=1</screen>
For Lustre filesystem 'scratch', the command executed is:
<screen>mgs# lctl conf_param scratch.mdt.enable_remote_dir=1</screen>
<literal>enable_remote_dir_gid</literal>. For example, setting this
parameter to the 'wheel' or 'admin' group ID allows users with that GID
to create and delete remote and striped directories. Setting this
- parameter to <literal>-1</literal> on MDT0 to permanently allow any
+ parameter to <literal>-1</literal> on MDT0000 to permanently allow any
non-root users create and delete remote and striped directories.
On the MGS execute the following command:
<screen>mgs# lctl conf_param <replaceable>fsname</replaceable>.mdt.enable_remote_dir_gid=-1</screen>
<para>The Lustre 2.8 DNE feature enables individual files in a given
directory to store their metadata on separate MDTs (a <emphasis>striped
directory</emphasis>) once additional MDTs have been added to the
- filesystem, see <xref linkend="dbdoclet.adding_new_mdt"/>.
+ filesystem, see <xref linkend="lustremaint.adding_new_mdt"/>.
The result of this is that metadata requests for
files in a striped directory are serviced by multiple MDTs and metadata
service load is distributed over all the MDTs that service a given
<para>The striped directory feature is most useful for distributing
single large directories (50k entries or more) across multiple MDTs,
since it incurs more overhead than non-striped directories.</para>
+ <section xml:id="dbdoclet.lfsmkdirbyspace" condition='l2D'>
+ <title>Directory creation by space/inode usage</title>
+ <para>If the starting MDT is not specified when creating a new directory,
+ this directory and its stripes will be distributed on MDTs by space usage.
+ For example the following will create a directory and its stripes on MDTs
+ with balanced space usage:</para>
+ <screen>lfs mkdir -c 2 <dir1></screen>
+ <para>Alternatively, if a default directory stripe is set on a directory,
+ the subsequent syscall <literal>mkdir</literal> under
+ <literal><dir1></literal> will have the same effect:
+ <screen>lfs setdirstripe -D -c 2 <dir1></screen></para>
+ <para>The policy is:</para>
+ <itemizedlist>
+ <listitem><para>If free inodes/blocks on all MDT are almost the same,
+ i.e. <literal>max_inodes_avail * 84% < min_inodes_avail</literal> and
+ <literal>max_blocks_avail * 84% < min_blocks_avail</literal>, then
+ choose MDT roundrobin.</para></listitem>
+ <listitem><para>Otherwise, create more subdirectories on MDTs with more
+ free inodes/blocks.</para></listitem>
+ </itemizedlist>
+ </section>
</section>
<section xml:id="dbdoclet.50438194_88980">
<title>
<title>Setting Temporary Parameters</title>
<para>Use
<literal>lctl set_param</literal> to set temporary parameters on the
- node where it is run. These parameters map to items in
- <literal>/proc/{fs,sys}/{lnet,lustre}</literal>. The
+ node where it is run. These parameters internally map to corresponding
+ items in the kernel <literal>/proc/{fs,sys}/{lnet,lustre}</literal> and
+ <literal>/sys/{fs,kernel/debug}/lustre</literal> virtual filesystems.
+ However, since the mapping between a particular parameter name and the
+ underlying virtual pathname may change, it is <emphasis>not</emphasis>
+ recommended to access the virtual pathname directly. The
<literal>lctl set_param</literal> command uses this syntax:</para>
<screen>
lctl set_param [-n] [-P]
<literal>lctl conf_param</literal> command to set permanent parameters.
In general, the
<literal>lctl conf_param</literal> command can be used to specify any
- parameter settable in a
- <literal>/proc/fs/lustre</literal> file, with its own OBD device. The
- <literal>lctl conf_param</literal> command uses this syntax (same as the
-
- <literal>mkfs.lustre</literal> and
+ settable parameter with its own OBD device. The
+ <literal>lctl conf_param</literal> command uses the following syntax
+ (the same as the <literal>mkfs.lustre</literal> and
<literal>tunefs.lustre</literal> commands):</para>
<screen>
<replaceable>obdname|fsname</replaceable>.
<replaceable>proc_file_name</replaceable>=
<replaceable>value</replaceable>)
</screen>
+ <note><para>The <literal>lctl conf_param</literal> and
+ <literal>lctl set_param</literal> syntax is <emphasis>not</emphasis>
+ the same.</para></note>
<para>Here are a few examples of
<literal>lctl conf_param</literal> commands:</para>
<screen>
<section xml:id="dbdoclet.setparamp" condition='l25'>
<title>Setting Permanent Parameters with lctl set_param -P</title>
<para>The <literal>lctl set_param -P</literal> command can also
- set parameters permanently. This command must be issued on the MGS.
+ set parameters permanently using the same syntax as
+ <literal>lctl set_param</literal> and <literal>lctl
+ get_param</literal> commands. This command must be issued on the MGS.
The given parameter is set on every host using
- <literal>lctl</literal> upcall. Parameters map to items in
- <literal>/proc/{fs,sys}/{lnet,lustre}</literal>. The
- <literal>lctl set_param</literal> command uses this syntax:</para>
+ <literal>lctl</literal> upcall. The <literal>lctl set_param</literal>
+ command uses the following syntax:</para>
<screen>
lctl set_param -P
<replaceable>obdtype</replaceable>.
lctl set_param -P -d
<replaceable>obdtype</replaceable>.
<replaceable>obdname</replaceable>.
-<replaceable>proc_file_name</replaceable>
+<replaceable>parameter_name</replaceable>
</screen>
<para>For example:</para>
<screen>
# lctl set_param -P -d osc.*.max_dirty_mb
</screen>
+ <note condition='l2c'><para>Starting in Lustre 2.12, there is
+ <literal>lctl get_param</literal> command can provide
+ <emphasis>tab completion</emphasis> when using an interactive shell
+ with <literal>bash-completion</literal> installed. This simplifies
+ the use of <literal>get_param</literal> significantly, since it
+ provides an interactive list of available parameters.
+ </para></note>
</section>
<section xml:id="dbdoclet.50438194_88217">
<title>Listing Parameters</title>
<replaceable>obdname</replaceable>.
<replaceable>proc_file_name</replaceable>
</screen>
+ <note condition='l2c'><para>Starting in Lustre 2.12, there is
+ <literal>lctl get_param</literal> command can provide
+ <emphasis>tab completion</emphasis> when using an interactive shell
+ with <literal>bash-completion</literal> installed. This simplifies
+ the use of <literal>get_param</literal> significantly, since it
+ provides an interactive list of available parameters.
+ </para></note>
<para>This example reports data on RPC service times.</para>
<screen>
oss# lctl get_param -n ost.*.ost_io.timeouts
software tries the first one, and if that fails, it tries the second
one.)</para>
<para>Two options to
- <literal>mkfs.lustre</literal> can be used to specify failover nodes.
- Introduced in Lustre software release 2.0, the
+ <literal>mkfs.lustre</literal> can be used to specify failover nodes. The
<literal>--servicenode</literal> option is used to specify all service NIDs,
including those for primary nodes and failover nodes. When the
<literal>--servicenode</literal> option is used, the first service node to
load the target device becomes the primary service node, while nodes
corresponding to the other specified NIDs become failover locations for the
- target device. An older option,
- <literal>--failnode</literal>, specifies just the NIDS of failover nodes.
- For more information about the
+ target device. An older option, <literal>--failnode</literal>, specifies
+ just the NIDs of failover nodes. For more information about the
<literal>--servicenode</literal> and
<literal>--failnode</literal> options, see
<xref xmlns:xlink="http://www.w3.org/1999/xlink"
<xref linkend='dbdoclet.backup_device' />or
<xref linkend='backup_fs_level' />.
For more information on removing a MDT, see
- <xref linkend='dbdoclet.rmremotedir' />.</para>
+ <xref linkend='lustremaint.rmremotedir' />.</para>
</section>
<section xml:id="dbdoclet.50438194_30872">
<title>
</screen></para>
<para>The command output is:
<screen>
-debugfs 1.42.3.wc3 (15-Aug-2012)
+debugfs 1.45.6.wc1 (20-Mar-2020)
/dev/lustre/ost_test2: catastrophic mode - not reading inode or group bitmaps
Inode: 352365 Type: regular Mode: 0666 Flags: 0x80000
Generation: 2393149953 Version: 0x0000002a:00005f81
Extended attributes stored in inode body:
fid = "b9 da 24 00 00 00 00 00 6a fa 0d 3f 01 00 00 00 eb 5b 0b 00 00 00 0000
00 00 00 00 00 00 00 00 " (32)
- fid: objid=34976 seq=0 parent=[0x24dab9:0x3f0dfa6a:0x0] stripe=1
+ fid: objid=34976 seq=0 parent=[0x200000400:0x122:0x0] stripe=1
EXTENTS:
(0-64):4620544-4620607
</screen></para>
</listitem>
<listitem>
- <para>For Lustre software release 2.x file systems, the parent FID will
- be of the form [0x200000400:0x122:0x0] and can be resolved directly
- using the
- <literal>lfs fid2path [0x200000404:0x122:0x0]
- /mnt/lustre</literal> command on any Lustre client, and the process is
+ <para>The parent FID will be of the form
+ <literal>[0x200000400:0x122:0x0]</literal> and can be resolved directly
+ using the command <literal>lfs fid2path [0x200000404:0x122:0x0]
+ /mnt/lustre</literal> on any Lustre client, and the process is
complete.</para>
</listitem>
<listitem>
- <para>In this example the parent inode FID is an upgraded 1.x inode
- (due to the first part of the FID being below 0x200000400), the MDT
- inode number is
+ <para>In cases of an upgraded 1.x inode (if the first part of the
+ FID is below 0x200000400), the MDT inode number is
<literal>0x24dab9</literal> and generation
- <literal>0x3f0dfa6a</literal> and the pathname needs to be resolved
+ <literal>0x3f0dfa6a</literal> and the pathname can also be resolved
using
<literal>debugfs</literal>.</para>
</listitem>
</screen>
<para>Here is the command output:</para>
<screen>
-debugfs 1.42.3.wc2 (15-Aug-2012)
+debugfs 1.42.3.wc3 (15-Aug-2012)
/dev/lustre/mdt_test: catastrophic mode - not reading inode or group bitmap\
s
Inode Pathname
</note>
</section>
</chapter>
+<!--
+ vim:expandtab:shiftwidth=2:tabstop=8:
+ -->
git://git.whamcloud.com / doc / manual.git / blobdiff