<para><emphasis role="bold">Back up all file system data.</emphasis></para>
<screen>[oss]# tar czvf {backup file}.tgz --sparse .</screen>
<note>
- <para>In Lustre 1.6.7 and later, the <literal>--sparse</literal> option reduces the size of the backup file. Be sure to use it so the tar command does not mistakenly create an archive full of zeros.</para>
+ <para>The tar <literal>--sparse</literal> option reduces the size of the MDT backup file. Be sure to use it so the tar command does not mistakenly create an archive full of zeros.</para>
</note>
</listitem>
<listitem>
<para>When <literal>lfs quotacheck</literal> runs, it creates a quota file -- a sparse file with a size proportional to the highest UID in use and UID/GID distribution. As a general rule, if the highest UID in use is large, then the sparse file will be large, which may affect functions such as creating a snapshot.</para>
</note>
<note>
- <para>For Lustre 1.6 releases before version 1.6.5, and 1.4 releases before version 1.4.12, if the underlying <literal>ldiskfs</literal> file system has not unmounted gracefully (due to a crash, for example), re-run <literal>quotacheck</literal> to obtain accurate quota information. Lustre 1.6.5 and 1.4.12 use journaled quota, so it is not necessary to run <literal>quotacheck</literal> after an unclean shutdown.</para>
+ <para>Lustre 1.6.5 and later use journaled quota files, so it is not necessary to run <literal>quotacheck</literal> after an unclean shutdown.</para>
<para>In certain failure situations (e.g., when a broken Lustre installation or build is used), re-run <literal>quotacheck</literal> after checking the server kernel logs and fixing the root problem.</para>
</note>
<para>The <literal>lfs</literal> command includes several command options to work with quotas:</para>
</orderedlist>
<para>Because of granted cache, writes always overwrite quota limitations. For example, if you set a 400 GB quota on user A and use IOR to write for user A from a bundle of clients, you will write much more data than 400 GB, and cause an out-of-quota error (<literal>-EDQUOT</literal>).</para>
<note>
- <para>The effect of granted cache on quota limits can be mitigated, but not eradicated. Reduce the <literal>max_dirty_buffer</literal> in the clients (can be set from 0 to 512). To set <literal>max_dirty_buffer</literal> to 0:</para>
+ <para>The effect of granted cache on quota limits can be mitigated, but not eradicated. Reduce the maximum amount of dirty data on the clients (can be set from 1 to 512):</para>
<itemizedlist>
<listitem>
- <para>In releases after Lustre 1.6.5, <literal>lctl set_param osc.*.max_dirty_mb=0</literal>.</para>
- </listitem>
- <listitem>
- <para>In releases before Lustre 1.6.5, <literal>proc/fs/lustre/osc/*/max_dirty_mb; do echo 512 > $O</literal></para>
+ <para><literal>lctl set_param osc.*.max_dirty_mb=0</literal></para>
</listitem>
</itemizedlist>
</note>
</section>
- <section remap="h3">
- <title><indexterm><primary>Quotas</primary><secondary>limits</secondary></indexterm>Quota Limits</title>
- <para>Available quota limits depend on the Lustre version you are using.</para>
- <itemizedlist>
- <listitem>
- <para> Lustre version 1.4.11 and earlier (for 1.4.x releases) and Lustre version 1.6.4 and earlier (for 1.6.x releases) support quota limits less than 4 TB.</para>
- </listitem>
- <listitem>
- <para> Lustre versions 1.4.12, 1.6.5 and later support quota limits of 4 TB and greater in Lustre configurations with OST storage limits of 4 TB and less.</para>
- </listitem>
- <listitem>
- <para> Future Lustre versions are expected to support quota limits of 4 TB and greater with no OST storage limits.</para>
- </listitem>
- <listitem>
- <informaltable frame="all">
- <tgroup cols="3">
- <colspec colname="c1" colwidth="33*"/>
- <colspec colname="c2" colwidth="33*"/>
- <colspec colname="c3" colwidth="33*"/>
- <thead>
- <row>
- <entry>
- <para><emphasis role="bold">Lustre Version</emphasis></para>
- </entry>
- <entry>
- <para><emphasis role="bold">Quota Limit Per User/Per Group</emphasis></para>
- </entry>
- <entry>
- <para><emphasis role="bold">OST Storage Limit</emphasis></para>
- </entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry>
- <para> 1.4.11 and earlier</para>
- </entry>
- <entry>
- <para> < 4TB</para>
- </entry>
- <entry>
- <para> n/a</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> 1.4.12</para>
- </entry>
- <entry>
- <para> => 4TB</para>
- </entry>
- <entry>
- <para> <= 4TB of storage</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> 1.6.4 and earlier</para>
- </entry>
- <entry>
- <para> < 4TB</para>
- </entry>
- <entry>
- <para> n/a</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> 1.6.5</para>
- </entry>
- <entry>
- <para> => 4TB</para>
- </entry>
- <entry>
- <para> <= 4TB of storage</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> Future Lustre versions</para>
- </entry>
- <entry>
- <para> => 4TB</para>
- </entry>
- <entry>
- <para> No storage limit</para>
- </entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </listitem>
- </itemizedlist>
- </section>
<section xml:id="dbdoclet.50438217_66360">
<title><indexterm><primary>Quotas</primary><secondary>file formats</secondary></indexterm>Quota File Formats</title>
<para>Lustre 1.6.5 introduced the v2 file format for administrative quotas, with 64-bit limits that support large-limits handling. The old quota file format (v1), with 32-bit limits, is also supported. Lustre 1.6.6 introduced the v2 file format for operational quotas. A few notes regarding the current quota file formats:</para>
<glossterm>Mountconf
</glossterm>
<glossdef>
- <para>The Lustre configuration protocol (introduced in version 1.6) which formats disk file systems on servers with the mkfs.lustre program, and prepares them for automatic incorporation into a Lustre cluster.</para>
+ <para>The Lustre configuration protocol that formats disk file systems on servers with the mkfs.lustre program, and prepares them for automatic incorporation into a Lustre cluster. This allows clients to be configured and mounted with a simple <literal>mount</literal> command.</para>
</glossdef>
</glossentry>
</glossdiv>
</entry>
<entry>
<para>Sets the maximum adaptive timeout (in seconds). The <literal>at_max</literal> parameter is an upper-limit on the service time estimate, and is used as a 'failsafe' in case of rogue/bad/buggy code that would lead to never-ending estimate increases. If at_max is reached, an RPC request is considered 'broken' and should time out.</para>
- <para>Setting at_max to 0 causes adaptive timeouts to be disabled and the old fixed-timeout method (<literal>obd_timeout</literal>) to be used. This is the default value in Lustre 1.6.5.</para>
+ <para>Setting at_max to 0 causes adaptive timeouts to be disabled and the old fixed-timeout method (<literal>obd_timeout</literal>) to be used.</para>
<note>
<para>It is possible that slow hardware might validly cause the service estimate to increase beyond the default value of at_max. In this case, you should increase at_max to the maximum time you are willing to wait for an RPC completion.</para>
</note>
</tbody>
</tgroup>
</informaltable>
- <para>Most customers are probably interested in found/cr. If cr is 0 1 and found is less than 100, then <literal>mballoc</literal> is doing quite well.</para>
+ <para>Most users are probably interested in found/cr. If cr is 0 1 and found is less than 100, then <literal>mballoc</literal> is doing quite well.</para>
<para>Also, number-of-blocks-in-request (third number in the goal triple) can tell the number of blocks requested by the <literal>obdfilter</literal>. If the <literal>obdfilter</literal> is doing a lot of small requests (just few blocks), then either the client is processing input/output to a lot of small files, or something may be wrong with the client (because it is better if client sends large input/output requests). This can be investigated with the OSC <literal>rpc_stats</literal> or OST <literal>brw_stats</literal> mentioned above.</para>
<para>Number of groups scanned (<literal>grps</literal> column) should be small. If it reaches a few dozen often, then either your disk file system is pretty fragmented or <literal>mballoc</literal> is doing something wrong in the group selection part.</para>
</section>
<section remap="h3">
- <title><indexterm><primary>proc</primary><secondary>mballoc3 tunables</secondary></indexterm><literal>mballoc3</literal> Tunables</title>
- <para>Lustre version 1.6.1 and later includes <literal>mballoc3</literal>, which was built on top of <literal>mballoc2</literal>. By default, mballoc3 is enabled, and adds these features:</para>
+ <title><indexterm><primary>proc</primary><secondary>mballoc tunables</secondary></indexterm><literal>mballoc</literal> Tunables</title>
+ <para>Lustre ldiskfs includes a multi-block allocation for ldiskfs to improve the efficiency of space allocation in the OST storage. Multi-block allocation adds the following features:</para>
<itemizedlist>
<listitem>
<para> Pre-allocation for single files (helps to resist fragmentation)</para>
<para> Stream allocation (helps to decrease the seek rate)</para>
</listitem>
</itemizedlist>
- <para>The following <literal>mballoc3</literal> tunables are available:</para>
+ <para>The following <literal>mballoc</literal> tunables are available:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<para>The total number of locks available is a function of the server's RAM. The default limit is 50 locks/1 MB of RAM. If there is too much memory pressure, then the LRU size is shrunk. The number of locks on the server is limited to {number of OST/MDT on node} * {number of clients} * {client lru_size}.</para>
<itemizedlist>
<listitem>
- <para>To enable automatic LRU sizing, set the <literal>lru_size</literal> parameter to 0. In this case, the <literal>lru_size</literal> parameter shows the current number of locks being used on the export. (In Lustre 1.6.5.1 and later, LRU sizing is enabled, by default.)</para>
+ <para>To enable automatic LRU sizing, set the <literal>lru_size</literal> parameter to 0. In this case, the <literal>lru_size</literal> parameter shows the current number of locks being used on the export. LRU sizing is enabled by default starting with Lustre 1.6.5.1.</para>
</listitem>
<listitem>
<para>To specify a maximum number of locks, set the lru_size parameter to a value > 0 (former numbers are okay, 100 * CPU_NR). We recommend that you only increase the LRU size on a few login nodes where users access the file system interactively.</para>
</section>
<section remap="h5">
<title>Description</title>
- <para>tunefs.lustre is used to modify configuration information on a Lustre target disk. This includes upgrading old (pre-Lustre 1.6) disks. This does not reformat the disk or erase the target information, but modifying the configuration information can result in an unusable file system.</para>
+ <para>tunefs.lustre is used to modify configuration information on a Lustre target disk. This does not reformat the disk or erase the target information, but modifying the configuration information can result in an unusable file system.</para>
<caution>
<para>Changes made here affect a file system only when the target is mounted the next time.</para>
</caution>
<para><emphasis role="bold">lustre_req_history.sh</emphasis></para>
<para>The lustre_req_history.sh utility (run from a client), assembles as much Lustre RPC request history as possible from the local node and from the servers that were contacted, providing a better picture of the coordinated network activity.</para>
<para><emphasis role="bold">llstat.sh</emphasis></para>
- <para>The llstat.sh utility (improved in Lustre 1.6), handles a wider range of /proc files, and has command line switches to produce more graphable output.</para>
+ <para>The llstat.sh utility handles a wider range of statistics files, and has command line switches to produce more graphable output.</para>
<para><emphasis role="bold">plot-llstat.sh</emphasis></para>
<para>The plot-llstat.sh utility plots the output from llstat.sh using gnuplot.</para>
</section>
<para>The client offset_stats utility shows the read/write seek activity of a client by offsets and ranges.</para>
<screen>/proc/fs/lustre/llite/*/offset_stats
</screen>
- <para>Lustre 1.6 included per-client and improved MDT statistics:</para>
+ <para>Lustre includes per-client and improved MDT statistics:</para>
<itemizedlist>
<listitem>
<para> Per-client statistics tracked on the servers</para>
<?xml version='1.0' encoding='UTF-8'?>
<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="upgradinglustre">
<title xml:id="upgradinglustre.title">Upgrading Lustre</title>
- <para>This chapter describes Lustre interoperability and how to upgrade to Lustre 2.0, and includes the following sections:</para>
+ <para>This chapter describes Lustre interoperability and how to upgrade from Lustre 1.8 to Lustre 2.x, and includes the following sections:</para>
<itemizedlist>
<listitem>
<para><xref linkend="dbdoclet.50438205_82079"/>Lustre Interoperability</para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438205_51369"/>Upgrading Lustre 1.8.x to 2.0</para>
+ <para><xref linkend="dbdoclet.50438205_51369"/>Upgrading Lustre 1.8 to 2.x</para>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438205_82079">
<indexterm><primary>upgrading</primary></indexterm>
Lustre Interoperability</title>
- <para>Lustre 2.0 is built on a new architectural code base, which is different than the one used with Lustre 1.8. These architectural changes require existing Lustre 1.8.x users to follow a slightly different procedure to upgrade to Lustre 2.0 - requiring clients to be unmounted and the file system be shut down. Once the servers are upgraded and restarted, then the clients can be remounted. After the upgrade, Lustre 2.0 servers can interoperate with compatible 1.8 clients and servers. Lustre 2.0 does <emphasis>not</emphasis> support 2.0 clients interoperating with 1.8 servers.</para>
+ <para>Lustre 2.x is built on a new architectural code base which is different than the one used with Lustre 1.8. These architectural changes require existing Lustre 1.8 users to follow a slightly different procedure to upgrade to Lustre 2.x - requiring clients to be unmounted and the file system be shut down. Once the servers are upgraded and restarted, then the clients can be remounted. After the upgrade, Lustre 2.x servers can interoperate with compatible 1.8 clients and servers. Lustre 2.x does <emphasis>not</emphasis> support 2.x clients interoperating with 1.8 servers.</para>
<note>
- <para>Lustre 1.8 clients support a mix of 1.8 and 2.0 OSTs, not all OSSs need to be upgraded at the same time.</para>
+ <para>Lustre 1.8 clients can interoperate with 2.x servers, but the servers should all be upgraded at the same time.</para>
</note>
<note>
- <para>Lustre 2.0 is compatible with version 1.8.4 and above. If you are planning a heterogeneous environment (mixed 1.8 and 2.0 servers), make sure that version 1.8.4 is installed on the client and server nodes that are not upgraded to 2.0.</para>
+ <para>Lustre 2.x servers are compatible with clients 1.8.6 and later, though it is strongly recommended that the clients are upgraded to the latest version of Lustre 1.8 available. If you are planning a heterogeneous environment (mixed 1.8 and 2.x servers), make sure that version 1.8.6 or later is installed on the client nodes that are not upgraded to 2.x.</para>
</note>
</section>
<section xml:id="dbdoclet.50438205_51369">
- <title><indexterm><primary>upgrading</primary><secondary>1.8 to 2.x</secondary></indexterm>Upgrading Lustre 1.8.x to 2.0</title>
- <para>Upgrading to Lustre 2.0 involves shutting down the file system and upgrading servers, and optionally clients, all at the same time. Lustre 2.0 does not support a rolling upgrade in which the file system operates continuously while individual servers (or their failover partners) and clients are upgraded one at a time.</para>
+ <title><indexterm><primary>upgrading</primary><secondary>1.8 to 2.x</secondary></indexterm>Upgrading Lustre 1.8 to 2.x</title>
+ <para>Upgrading from 1.8 to Lustre 2.x involves shutting down the file system and upgrading servers, and optionally clients, all at the same time. This upgrade process does <emphasis>not</emphasis> support a rolling upgrade in which the file system operates continuously while individual servers (or their failover partners) and clients are upgraded one at a time.</para>
<note>
- <para>Although the Lustre 1.8 to 2.0 upgrade path has been tested, for best results we recommend performing a fresh Lustre 2.0 installation, rather than upgrading from 1.8 to 2.0.</para>
+ <para>Although the Lustre 1.8 to 2.x upgrade path has been tested, optimum performance will be seen with a freshly formatted 2.x filesystem.</para>
</note>
<section remap="h3">
<title><indexterm><primary>upgrading</primary><secondary>file system</secondary></indexterm>Performing a File System Upgrade</title>
- <para>This procedure describes a file system upgrade in which Lustre 2.0 packages are installed on multiple 1.8.x servers and, optionally, clients, requiring a file system shut down. You can choose to upgrade the entire Lustre file system to 2.0 or just upgrade the Lustre servers to 2.0 and leave the clients at 1.8.x. Lustre 2.0 servers can interoperate with compatible 1.8 clients and servers.</para>
+ <para>This procedure describes a file system upgrade in which Lustre 2.x packages are installed on multiple 1.8 servers and, optionally, clients, requiring a file system shutdown. You can choose to upgrade the entire Lustre file system to 2.x, or just upgrade the servers to Lustre 2.x and leave the clients running 1.8.6 or later.</para>
<tip>
- <para>In a Lustre upgrade, the package install and file system unmount steps are reversible; you can do either step first. To minimize downtime, this procedure first performs the 2.0 package installation, and then unmounts the file system.</para>
+ <para>In a Lustre upgrade, the package install/update can be done either before or after the filesystem is unmount. To minimize downtime, this procedure first performs the 2.x package installation, and then unmounts the file system.</para>
</tip>
<orderedlist>
<listitem>
- <para>Make a complete, restorable file system backup before upgrading Lustre.</para>
+ <para>Make a complete, restorable file system backup before upgrading Lustre. The Lustre 2.x on-disk format itself is compatible with the 1.8 on-disk format, but having a backup is always important. If it is not possible to backup the full filesystem, it is still valuable to have a full device-level backup of the MDT filesystem, as described in <xref linkend="backupandrestore"/>.</para>
</listitem>
<listitem>
- <para>If any Lustre nodes will not be upgraded to 2.0, make sure that these client and server nodes are at version 1.8.4.</para>
- <para>Lustre 2.0 is compatible with version 1.8.4 and above. If you are planning a heterogeneous environment (mixed 1.8 and 2.0 clients and servers), make sure that version 1.8.4 is installed on nodes that are not upgraded to 2.0.</para>
+ <para>If you are planning a heterogeneous environment (1.8 clients and 2.x servers), make sure that at least version 1.8.6 is installed on clients that are not upgraded to 2.x.</para>
</listitem>
<listitem>
- <para>Install the 2.0 packages on the Lustre servers and, optionally, the clients.</para>
- <para>Some or all servers can be upgraded. Some or all clients can be upgraded.</para>
+ <para>Install the 2.x packages on the Lustre servers and, optionally, the clients.</para>
+ <para>All servers must be upgraded from 1.8 to 2.x at the same time. Some or all clients can be upgraded to 2.x at this time.</para>
<para>For help determining where to install a specific package, see <xref linkend="installinglustre.tab.req"/>.</para>
<orderedlist>
<listitem>
<para>If a new <literal>e2fsprogs</literal> package is available, upgrade it. For example:</para>
<screen>$ rpm -Uvh e2fsprogs-<ver>
</screen>
- <para>Use e2fsprogs-1.41-10 or later, available at:</para>
+ <para>Use e2fsprogs-1.41.90-wc3 or later, available at:</para>
<para><link xl:href="http://downloads.whamcloud.com/public/e2fsprogs/latest/">http://downloads.whamcloud.com/public/e2fsprogs/latest/</link></para>
</listitem>
<listitem>
- <para>(Optional) If you want to add optional packages to your Lustre system, install them now.</para>
+ <para>If you want to add optional packages to your Lustre system, install them now.</para>
</listitem>
</orderedlist>
</listitem>
<orderedlist>
<listitem>
<para>Unmount the clients. On each client node, run:</para>
- <screen>umount <mount point></screen>
+ <screen>umount -a -t lustre</screen>
</listitem>
<listitem>
<para>Unmount the MDT. On the MDS node, run:</para>
- <screen>umount <mount point></screen>
+ <screen>umount -a -t lustre</screen>
</listitem>
<listitem>
<para>Unmount the OSTs (be sure to unmount all OSTs). On each OSS node, run:</para>
- <screen>umount <mount point></screen>
+ <screen>umount -a -t lustre</screen>
</listitem>
</orderedlist>
</listitem>
<listitem>
- <para>Unload the old Lustre modules by rebooting the node or manually removing the Lustre modules.</para>
- <para>Run <literal>lustre_rmmod</literal> several times and use <literal>lsmod</literal> to check the currently loaded modules.</para>
+ <para>Since the kernel will typically be upgraded with a 1.8 to 2.x upgrade, the nodes will need to be rebooted in order to use the new kernel.</para>
</listitem>
<listitem>
<para>Start the upgraded file system.</para>
<orderedlist>
<listitem>
<para>Mount the OSTs (be sure to mount all OSTs). On each OSS node, run:</para>
- <screen>mount -t lustre <block device name> <mount point></screen>
+ <screen>mount -a -t lustre</screen>
+ <para>This command assumes that all OSTs are listed in the /etc/fstab file. If the OSTs are not in the /etc/fstab file, they need to be mounted individually by running the mount command:</para>
+ <screen>mount -t lustre <block device name> <mount point> </screen>
</listitem>
<listitem>
<para>Mount the MDT. On the MDS node, run:</para>
- <screen>mount -t lustre <block device name> <mount point> </screen>
+ <screen>mount -a -t lustre</screen>
</listitem>
<listitem>
<para>Mount the file system on the clients. On each client node, run:</para>
- <screen>mount -t lustre <MGS node>:/<fsname> <mount point> </screen>
+ <screen>mount -a -t lustre</screen>
</listitem>
</orderedlist>
</listitem>
</orderedlist>
- <para>If you have a problem upgrading Lustre, contact us via the <link xl:href="https://jira.whamcloud.com">Whamcloud Jira</link> bug tracker.</para>
+ <para>If you have a problem upgrading Lustre, use the <link xl:href="https://groups.google.com/a/whamcloud.com/group/wc-discuss/">wc-discuss</link> mailing list, or file a ticket at the <link xl:href="https://jira.whamcloud.com">Whamcloud Jira</link> bug tracker.</para>
</section>
</section>
</chapter>
git://git.whamcloud.com / doc / manual.git / commitdiff