-<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='backupandrestore'>
- <info>
- <title xml:id='backupandrestore.title'>Backing Up and Restoring a File System</title>
- </info>
- <para>Lustre provides backups at the file system-level, device-level and file-level. This chapter describes how to backup and restore on Lustre, and includes the following sections:</para>
- <itemizedlist><listitem>
- <para><xref linkend="dbdoclet.50438207_56395"/></para>
- </listitem>
- <listitem>
- <para><xref linkend="dbdoclet.50438207_71633"/></para>
- </listitem>
- <listitem>
- <para><xref linkend="dbdoclet.50438207_21638"/></para>
- </listitem>
- <listitem>
- <para><xref linkend="dbdoclet.50438207_22325"/></para>
- </listitem>
- <listitem>
- <para><xref linkend="dbdoclet.50438207_31553"/></para>
- </listitem>
-</itemizedlist>
-
- <section xml:id="dbdoclet.50438207_56395">
- <title>17.1 Backing up a File System</title>
- <para>Backing up a complete file system gives you full control over the files to back up, and allows restoration of individual files as needed. File system-level backups are also the easiest to integrate into existing backup solutions.</para>
- <para>File system backups are performed from a Lustre client (or many clients working parallel in different directories) rather than on individual server nodes; this is no different than backing up any other file system.</para>
- <para>However, due to the large size of most Lustre file systems, it is not always possible to get a complete backup. We recommend that you back up subsets of a file system. This includes subdirectories of the entire file system, filesets for a single user, files incremented by date, and so on.</para>
-
- <note><para>In order to allow Lustre to scale the filesystem namespace for future applications, Lustre 2.x internally uses a 128-bit file identifier for all files. To interface with user applications, Lustre presents 64-bit inode numbers for the stat(), fstat(), and readdir() system calls on 64-bit applications, and 32-bit inode numbers to 32-bit applications.</para><para> Some 32-bit applications accessing Lustre filesystems (on both 32-bit and 64-bit CPUs) may experience problems with the stat(), fstat() or readdir() system calls under certain circumstances, though the Lustre client should return 32-bit inode numbers to these applications.</para><para> In particular, if the Lustre filesystem is exported from a 64-bit client via NFS to a 32-bit client, the Linux NFS server will export 64-bit inode numbers to applications running on the NFS client. If the 32-bit applications are not compiled with Large File Support (LFS), then they return EOVERFLOW errors when accessing the Lustre files. To avoid this problem, Linux NFS clients can use the kernel command-line option "nfs.enable_ino64=0" in order to force the NFS client to export 32-bit inode numbers to the client.</para><para><emphasis role="bold">Workaround</emphasis>: We very strongly recommend that backups using tar(1) and other utilities that depend on the inode number to uniquely identify an inode to be run on 64-bit clients. The 128-bit Lustre file identifiers cannot be uniquely mapped to a 32-bit inode number, and as a result these utilities may operate incorrectly on 32-bit clients.</para></note>
-
- <section remap="h3">
- <title>17.1.1 Lustre_rsync</title>
- <para>The lustre_rsync feature keeps the entire file system in sync on a backup by replicating the file system's changes to a second file system (the second file system need not be a Lustre file system, but it must be sufficiently large). Lustre_rsync uses Lustre changelogs to efficiently synchronize the file systems without having to scan (directory walk) the Lustre file system. This efficiency is critically important for large file systems, and distinguishes the Lustre lustre_rsync feature from other replication/backup solutions.</para>
- <section remap="h4">
- <title>17.1.1.1 Using Lustre_rsync</title>
- <para>The lustre_rsync feature works by periodically running lustre_rsync, a userspace program used to synchronize changes in the Lustre file system onto the target file system. The lustre_rsync utility keeps a status file, which enables it to be safely interrupted and restarted without losing synchronization between the file systems.</para>
- <para>The first time that lustre_rsync is run, the user must specify a set of parameters for the program to use. These parameters are described in the following table and in <link xl:href="SystemConfigurationUtilities.html#50438219_63667">lustre_rsync</link>. On subsequent runs, these parameters are stored in the the status file, and only the name of the status file needs to be passed to lustre_rsync.</para>
- <para>Before using lustre_rsync:</para>
- <itemizedlist><listitem>
- <para> Register the changelog user. For details, see the <xref linkend='systemconfigurationutilities'/> (changelog_register) parameter in the <xref linkend='systemconfigurationutilities'/> (lctl).</para>
- </listitem>
-</itemizedlist>
- <para>- AND -</para>
- <itemizedlist><listitem>
- <para> Verify that the Lustre file system (source) and the replica file system (target) are identical <emphasis>before</emphasis> registering the changelog user. If the file systems are discrepant, use a utility, e.g. regular rsync (not lustre_rsync), to make them identical.</para>
- </listitem>
-</itemizedlist>
- <para>The lustre_rsync utility uses the following parameters:</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="3*"/>
- <colspec colname="c2" colwidth="10*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Parameter</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> --source=<src></para></entry>
- <entry><para> The path to the root of the Lustre file system (source) which will be synchronized. This is a mandatory option if a valid status log created during a previous synchronization operation (--statuslog) is not specified.</para></entry>
- </row>
- <row>
- <entry><para> --target=<tgt></para></entry>
- <entry><para> The path to the root where the source file system will be synchronized (target). This is a mandatory option if the status log created during a previous synchronization operation (--statuslog) is not specified. This option can be repeated if multiple synchronization targets are desired.</para></entry>
- </row>
- <row>
- <entry><para> --mdt=<mdt></para></entry>
- <entry><para> The metadata device to be synchronized. A changelog user must be registered for this device. This is a mandatory option if a valid status log created during a previous synchronization operation (--statuslog) is not specified.</para></entry>
- </row>
- <row>
- <entry><para> --user=<user id></para></entry>
- <entry><para> The changelog user ID for the specified MDT. To use lustre_rsync, the changelog user must be registered. For details, see the changelog_register parameter in <xref linkend='systemconfigurationutilities'/> (lctl). This is a mandatory option if a valid status log created during a previous synchronization operation (--statuslog) is not specified.</para></entry>
- </row>
- <row>
- <entry><para> --statuslog=<log></para></entry>
- <entry><para> A log file to which synchronization status is saved. When the lustre_rsync utility starts, if the status log from a previous synchronization operation is specified, then the state is read from the log and otherwise mandatory --source, --target and --mdt options can be skipped. Specifying the --source, --target and/or --mdt options, in addition to the --statuslog option, causes the specified parameters in the status log to be overriden. Command line options take precedence over options in the status log.</para></entry>
- </row>
- <row>
- <entry><para> --xattr <yes|no></para></entry>
- <entry><para> Specifies whether extended attributes (xattrs) are synchronized or not. The default is to synchronize extended attributes.</para><para><note><para>Disabling xattrs causes Lustre striping information not to be synchronized.</para></note></para></entry>
- </row>
- <row>
- <entry><para> --verbose</para></entry>
- <entry><para> Produces verbose output.</para></entry>
- </row>
- <row>
- <entry><para> --dry-run</para></entry>
- <entry><para> Shows the output of lustre_rsync commands (copy, mkdir, etc.) on the target file system without actually executing them.</para></entry>
- </row>
- <row>
- <entry><para> --abort-on-err</para></entry>
- <entry><para> Stops processing the lustre_rsync operation if an error occurs. The default is to continue the operation.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h4">
- <title>17.1.1.2 lustre_rsync Examples</title>
- <para>Sample lustre_rsync commands are listed below.</para>
- <para>Register a changelog user for an MDT (e.g. lustre-MDT0000).</para>
- <screen># lctl --device lustre-MDT0000 changelog_register lustre-MDT0000 Registered\
- changelog userid 'cl1'
-</screen>
- <para>Synchronize a Lustre file system (/mnt/lustre) to a target file system (/mnt/target).</para>
- <screen>$ lustre_rsync --source=/mnt/lustre --target=/mnt/target --mdt=lustre-MDT00\
-00 --user=cl1 --statuslog sync.log --verbose
-Lustre filesystem: lustre
-MDT device: lustre-MDT0000
+<?xml version='1.0' encoding='UTF-8'?>
+<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="backupandrestore">
+ <title xml:id="backupandrestore.title">Backing Up and Restoring a File System</title>
+ <para>This chapter describes how to backup and restore at the file system-level, device-level and
+ file-level in a Lustre file system. Each backup approach is described in the the following
+ sections:</para>
+ <itemizedlist>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438207_56395"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438207_71633"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438207_21638"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438207_22325"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438207_31553"/></para>
+ </listitem>
+ </itemizedlist>
+ <section xml:id="dbdoclet.50438207_56395">
+ <title>
+ <indexterm><primary>backup</primary></indexterm>
+ <indexterm><primary>restoring</primary><see>backup</see></indexterm>
+ <indexterm><primary>LVM</primary><see>backup</see></indexterm>
+ <indexterm><primary>rsync</primary><see>backup</see></indexterm>
+ Backing up a File System</title>
+ <para>Backing up a complete file system gives you full control over the files to back up, and
+ allows restoration of individual files as needed. File system-level backups are also the
+ easiest to integrate into existing backup solutions.</para>
+ <para>File system backups are performed from a Lustre client (or many clients working parallel in different directories) rather than on individual server nodes; this is no different than backing up any other file system.</para>
+ <para>However, due to the large size of most Lustre file systems, it is not always possible to get a complete backup. We recommend that you back up subsets of a file system. This includes subdirectories of the entire file system, filesets for a single user, files incremented by date, and so on.</para>
+ <note>
+ <para>In order to allow the file system namespace to scale for future applications, Lustre
+ software release 2.x internally uses a 128-bit file identifier for all files. To interface
+ with user applications, the Lustre software presents 64-bit inode numbers for the
+ <literal>stat()</literal>, <literal>fstat()</literal>, and <literal>readdir()</literal>
+ system calls on 64-bit applications, and 32-bit inode numbers to 32-bit applications.</para>
+ <para>Some 32-bit applications accessing Lustre file systems (on both 32-bit and 64-bit CPUs)
+ may experience problems with the <literal>stat()</literal>, <literal>fstat()</literal>
+ or<literal> readdir()</literal> system calls under certain circumstances, though the
+ Lustre client should return 32-bit inode numbers to these applications.</para>
+ <para>In particular, if the Lustre file system is exported from a 64-bit client via NFS to a
+ 32-bit client, the Linux NFS server will export 64-bit inode numbers to applications running
+ on the NFS client. If the 32-bit applications are not compiled with Large File Support
+ (LFS), then they return <literal>EOVERFLOW</literal> errors when accessing the Lustre files.
+ To avoid this problem, Linux NFS clients can use the kernel command-line option
+ "<literal>nfs.enable_ino64=0</literal>" in order to force the NFS client to
+ export 32-bit inode numbers to the client.</para>
+ <para><emphasis role="bold">Workaround</emphasis>: We very strongly recommend that backups using <literal>tar(1)</literal> and other utilities that depend on the inode number to uniquely identify an inode to be run on 64-bit clients. The 128-bit Lustre file identifiers cannot be uniquely mapped to a 32-bit inode number, and as a result these utilities may operate incorrectly on 32-bit clients.</para>
+ </note>
+ <section remap="h3">
+ <title><indexterm><primary>backup</primary><secondary>rsync</secondary></indexterm>Lustre_rsync</title>
+ <para>The <literal>lustre_rsync </literal>feature keeps the entire file system in sync on a backup by replicating the file system's changes to a second file system (the second file system need not be a Lustre file system, but it must be sufficiently large). <literal>lustre_rsync </literal>uses Lustre changelogs to efficiently synchronize the file systems without having to scan (directory walk) the Lustre file system. This efficiency is critically important for large file systems, and distinguishes the Lustre <literal>lustre_rsync</literal> feature from other replication/backup solutions.</para>
+ <section remap="h4">
+ <title><indexterm><primary>backup</primary><secondary>rsync</secondary><tertiary>using</tertiary></indexterm>Using Lustre_rsync</title>
+ <para>The <literal>lustre_rsync</literal> feature works by periodically running <literal>lustre_rsync</literal>, a userspace program used to synchronize changes in the Lustre file system onto the target file system. The <literal>lustre_rsync</literal> utility keeps a status file, which enables it to be safely interrupted and restarted without losing synchronization between the file systems.</para>
+ <para>The first time that <literal>lustre_rsync</literal> is run, the user must specify a set of parameters for the program to use. These parameters are described in the following table and in <xref linkend="dbdoclet.50438219_63667"/>. On subsequent runs, these parameters are stored in the the status file, and only the name of the status file needs to be passed to <literal>lustre_rsync</literal>.</para>
+ <para>Before using <literal>lustre_rsync</literal>:</para>
+ <itemizedlist>
+ <listitem>
+ <para>Register the changelog user. For details, see the <xref linkend="systemconfigurationutilities"/> (<literal>changelog_register</literal>) parameter in the <xref linkend="systemconfigurationutilities"/> (<literal>lctl</literal>).</para>
+ </listitem>
+ </itemizedlist>
+ <para>- AND -</para>
+ <itemizedlist>
+ <listitem>
+ <para>Verify that the Lustre file system (source) and the replica file system (target) are identical <emphasis>before</emphasis> registering the changelog user. If the file systems are discrepant, use a utility, e.g. regular <literal>rsync</literal> (not <literal>lustre_rsync</literal>), to make them identical.</para>
+ </listitem>
+ </itemizedlist>
+ <para>The <literal>lustre_rsync</literal> utility uses the following parameters:</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="3*"/>
+ <colspec colname="c2" colwidth="10*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Parameter</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <literal>--source=<replaceable>src</replaceable></literal></para>
+ </entry>
+ <entry>
+ <para>The path to the root of the Lustre file system (source) which will be synchronized. This is a mandatory option if a valid status log created during a previous synchronization operation (<literal>--statuslog</literal>) is not specified.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>--target=<replaceable>tgt</replaceable></literal></para>
+ </entry>
+ <entry>
+ <para>The path to the root where the source file system will be synchronized (target). This is a mandatory option if the status log created during a previous synchronization operation (<literal>--statuslog</literal>) is not specified. This option can be repeated if multiple synchronization targets are desired.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>--mdt=<replaceable>mdt</replaceable></literal></para>
+ </entry>
+ <entry>
+ <para>The metadata device to be synchronized. A changelog user must be registered for this device. This is a mandatory option if a valid status log created during a previous synchronization operation (<literal>--statuslog</literal>) is not specified.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>--user=<replaceable>userid</replaceable></literal></para>
+ </entry>
+ <entry>
+ <para>The changelog user ID for the specified MDT. To use <literal>lustre_rsync</literal>, the changelog user must be registered. For details, see the <literal>changelog_register</literal> parameter in <xref linkend="systemconfigurationutilities"/> (<literal>lctl</literal>). This is a mandatory option if a valid status log created during a previous synchronization operation (<literal>--statuslog</literal>) is not specified.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>--statuslog=<replaceable>log</replaceable></literal></para>
+ </entry>
+ <entry>
+ <para>A log file to which synchronization status is saved. When the <literal>lustre_rsync</literal> utility starts, if the status log from a previous synchronization operation is specified, then the state is read from the log and otherwise mandatory <literal>--source</literal>, <literal>--target</literal> and <literal>--mdt</literal> options can be skipped. Specifying the <literal>--source</literal>, <literal>--target</literal> and/or <literal>--mdt</literal> options, in addition to the <literal>--statuslog</literal> option, causes the specified parameters in the status log to be overridden. Command line options take precedence over options in the status log.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <literal> --xattr <replaceable>yes|no</replaceable> </literal>
+ </entry>
+ <entry>
+ <para>Specifies whether extended attributes (<literal>xattrs</literal>) are synchronized or not. The default is to synchronize extended attributes.</para>
+ <para><note>
+ <para>Disabling xattrs causes Lustre striping information not to be synchronized.</para>
+ </note></para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>--verbose</literal></para>
+ </entry>
+ <entry>
+ <para>Produces verbose output.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>--dry-run</literal></para>
+ </entry>
+ <entry>
+ <para>Shows the output of <literal>lustre_rsync</literal> commands (<literal>copy</literal>, <literal>mkdir</literal>, etc.) on the target file system without actually executing them.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>--abort-on-err</literal></para>
+ </entry>
+ <entry>
+ <para>Stops processing the <literal>lustre_rsync</literal> operation if an error occurs. The default is to continue the operation.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ <section remap="h4">
+ <title><indexterm><primary>backup</primary><secondary>rsync</secondary><tertiary>examples</tertiary></indexterm><literal>lustre_rsync</literal> Examples</title>
+ <para>Sample <literal>lustre_rsync</literal> commands are listed below.</para>
+ <para>Register a changelog user for an MDT (e.g. <literal>testfs-MDT0000</literal>).</para>
+ <screen># lctl --device testfs-MDT0000 changelog_register testfs-MDT0000
+Registered changelog userid 'cl1'</screen>
+ <para>Synchronize a Lustre file system (<literal>/mnt/lustre</literal>) to a target file system (<literal>/mnt/target</literal>).</para>
+ <screen>$ lustre_rsync --source=/mnt/lustre --target=/mnt/target \
+ --mdt=testfs-MDT0000 --user=cl1 --statuslog sync.log --verbose
+Lustre filesystem: testfs
+MDT device: testfs-MDT0000