<?xml version='1.0' encoding='UTF-8'?>
<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="troubleshootingrecovery">
- <title xml:id="troubleshootingrecovery.title">Troubleshooting Recovery</title>
- <para>This chapter describes what to do if something goes wrong during recovery. It describes:</para>
- <itemizedlist>
- <listitem>
- <para><xref linkend="dbdoclet.50438225_71141"/></para>
- </listitem>
- <listitem>
- <para><xref linkend="dbdoclet.50438225_37365"/></para>
- </listitem>
- <listitem>
- <para><xref linkend="dbdoclet.50438225_12316"/></para>
- </listitem>
- </itemizedlist>
- <section xml:id="dbdoclet.50438225_71141">
- <title><indexterm><primary>recovery</primary><secondary>corruption of backing file system</secondary></indexterm>Recovering from Errors or Corruption on a Backing File System</title>
- <para>When an OSS, MDS, or MGS server crash occurs, it is not necessary to run e2fsck on the file system. <literal>ldiskfs</literal> journaling ensures that the file system remains coherent. The backing file systems are never accessed directly from the client, so client crashes are not relevant.</para>
- <para>The only time it is REQUIRED that <literal>e2fsck</literal> be run on a device is when an event causes problems that ldiskfs journaling is unable to handle, such as a hardware device failure or I/O error. If the ldiskfs kernel code detects corruption on the disk, it mounts the file system as read-only to prevent further corruption, but still allows read access to the device. This appears as error "-30" (<literal>EROFS</literal>) in the syslogs on the server, e.g.:</para>
- <screen>Dec 29 14:11:32 mookie kernel: LDISKFS-fs error (device sdz):
-ldiskfs_lookup: unlinked inode 5384166 in dir #145170469</screen>
- <para>Dec 29 14:11:32 mookie kernel: Remounting filesystem read-only</para>
- <para>In such a situation, it is normally required that e2fsck only be run on the bad device before placing the device back into service.</para>
- <para>In the vast majority of cases, Lustre can cope with any inconsistencies it finds on the disk and between other devices in the file system.</para>
- <note>
- <para><literal>lfsck</literal> is rarely required for Lustre operation.</para>
- </note>
- <para>For problem analysis, it is strongly recommended that <literal>e2fsck</literal> be run under a logger, like script, to record all of the output and changes that are made to the file system in case this information is needed later.</para>
- <para>If time permits, it is also a good idea to first run <literal>e2fsck</literal> in non-fixing mode (-n option) to assess the type and extent of damage to the file system. The drawback is that in this mode, <literal>e2fsck</literal> does not recover the file system journal, so there may appear to be file system corruption when none really exists.</para>
- <para>To address concern about whether corruption is real or only due to the journal not being replayed, you can briefly mount and unmount the <literal>ldiskfs</literal> filesystem directly on the node with Lustre stopped (NOT via Lustre), using a command similar to:</para>
- <screen>mount -t ldiskfs /dev/{ostdev} /mnt/ost; umount /mnt/ost</screen>
- <para>This causes the journal to be recovered.</para>
- <para>The <literal>e2fsck</literal> utility works well when fixing file system corruption (better than similar file system recovery tools and a primary reason why <literal>ldiskfs</literal> was chosen over other file systems for Lustre). However, it is often useful to identify the type of damage that has occurred so an <literal>ldiskfs</literal> expert can make intelligent decisions about what needs fixing, in place of <literal>e2fsck</literal>.</para>
- <screen>root# {stop lustre services for this device, if running}
-root# script /tmp/e2fsck.sda
-Script started, file is /tmp/e2fsck.sda
-root# mount -t ldiskfs /dev/sda /mnt/ost
-root# umount /mnt/ost
-root# e2fsck -fn /dev/sda # don't fix file system, just check for corruption
-:
-[e2fsck output]
-:
-root# e2fsck -fp /dev/sda # fix filesystem using "prudent" answers (usually 'y')
-</screen>
- <para>In addition, the <literal>e2fsprogs</literal> package contains the <literal>lfsck</literal> tool, which does distributed coherency checking for the Lustre file system after <literal>e2fsck</literal> has been run. Running <literal>lfsck</literal> is NOT required in a large majority of cases, at a small risk of having some leaked space in the file system. To avoid a lengthy downtime, it can be run (with care) after Lustre is started.</para>
- </section>
- <section xml:id="dbdoclet.50438225_37365">
- <title><indexterm><primary>recovery</primary><secondary>corruption of Lustre file system</secondary></indexterm>Recovering from Corruption in the Lustre File System</title>
- <para>In cases where the MDS or an OST becomes corrupt, you can run a distributed check on the file system to determine what sort of problems exist. Use <literal>lfsck</literal> to correct any defects found.</para>
- <orderedlist>
- <listitem>
- <para>Stop the Lustre file system.</para>
- </listitem>
- <listitem>
- <para>Run <literal>e2fsck -f</literal> on the individual MDS / OST that had problems to fix any local file system damage.</para>
- <para>We recommend running <literal>e2fsck</literal> under script, to create a log of changes made to the file system in case it is needed later. After <literal>e2fsck</literal> is run, bring up the file system, if necessary, to reduce the outage window.</para>
- </listitem>
- <listitem>
- <para>Run a full <literal>e2fsck</literal> of the MDS to create a database for <literal>lfsck</literal>. You <emphasis>must</emphasis> use the <literal>-n</literal> option for a mounted file system, otherwise you will corrupt the file system.</para>
- <screen>e2fsck -n -v --mdsdb /tmp/mdsdb /dev/{mdsdev}
-</screen>
- <para>The <literal>mds</literal>db file can grow fairly large, depending on the number of files in the file system (10 GB or more for millions of files, though the actual file size is larger because the file is sparse). It is quicker to write the file to a local file system due to seeking and small writes. Depending on the number of files, this step can take several hours to complete.</para>
- <para><emphasis role="bold">Example</emphasis></para>
- <screen>e2fsck -n -v --mdsdb /tmp/mdsdb /dev/sdb
-e2fsck 1.39.cfs1 (29-May-2006)
-Warning: skipping journal recovery because doing a read-only filesystem check.
-lustre-MDT0000 contains a file system with errors, check forced.
-Pass 1: Checking inodes, blocks, and sizes
-MDS: ost_idx 0 max_id 288
-MDS: got 8 bytes = 1 entries in lov_objids
-MDS: max_files = 13
-MDS: num_osts = 1
-mds info db file written
-Pass 2: Checking directory structure
-Pass 3: Checking directory connectivity
-Pass 4: Checking reference counts
-Pass 5: Checking group summary information
-Free blocks count wrong (656160, counted=656058).
-Fix? no
-
-Free inodes count wrong (786419, counted=786036).
-Fix? no
-
-Pass 6: Acquiring information for lfsck
-MDS: max_files = 13
-MDS: num_osts = 1
-MDS: 'lustre-MDT0000_UUID' mdt idx 0: compat 0x4 rocomp 0x1 incomp 0x4
-lustre-MDT0000: ******* WARNING: Filesystem still has errors *******
- 13 inodes used (0%)
- 2 non-contiguous inodes (15.4%)
- # of inodes with ind/dind/tind blocks: 0/0/0
-130272 blocks used (16%)
- 0 bad blocks
- 1 large file
- 296 regular files
- 91 directories
- 0 character device files
- 0 block device files
- 0 fifos
- 0 links
- 0 symbolic links (0 fast symbolic links)
- 0 sockets
- --------
- 387 files
-</screen>
- </listitem>
- <listitem>
- <para>Make this file accessible on all OSTs, either by using a shared file system or copying the file to the OSTs. The <literal>pdcp</literal> command is useful here.</para>
- <para>The <literal>pdcp</literal> command (installed with <literal>pdsh</literal>), can be used to copy files to groups of hosts. <literal>pdcp</literal> is available here:</para>
- <para><link xl:href="http://sourceforge.net/projects/pdsh">http://sourceforge.net/projects/pdsh</link></para>
- </listitem>
- <listitem>
- <para>Run a similar <literal>e2fsck</literal> step on the OSTs. The <literal>e2fsck --ostdb</literal> command can be run in parallel on all OSTs.</para>
- <screen>e2fsck -n -v --mdsdb /tmp/mdsdb --ostdb /tmp/{ostNdb} \/dev/{ostNdev}
-</screen>
- <para>The <literal>mdsdb</literal> file is read-only in this step; a single copy can be shared by all OSTs.</para>
+ <title xml:id="troubleshootingrecovery.title">Troubleshooting Recovery</title>
+ <para>This chapter describes what to do if something goes wrong during recovery. It describes:</para>
+ <itemizedlist>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438225_71141"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438225_37365"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438225_12316"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.lfsckadmin"/></para>
+ </listitem>
+ </itemizedlist>
+ <section xml:id="dbdoclet.50438225_71141">
+ <title><indexterm><primary>recovery</primary><secondary>corruption of backing file system</secondary></indexterm>Recovering from Errors or Corruption on a Backing File System</title>
+ <para>When an OSS, MDS, or MGS server crash occurs, it is not necessary to run e2fsck on the file system. <literal>ldiskfs</literal> journaling ensures that the file system remains coherent. The backing file systems are never accessed directly from the client, so client crashes are not relevant.</para>
+ <para>The only time it is REQUIRED that <literal>e2fsck</literal> be run on a device is when an event causes problems that ldiskfs journaling is unable to handle, such as a hardware device failure or I/O error. If the ldiskfs kernel code detects corruption on the disk, it mounts the file system as read-only to prevent further corruption, but still allows read access to the device. This appears as error "-30" (<literal>EROFS</literal>) in the syslogs on the server, e.g.:</para>
+ <screen>Dec 29 14:11:32 mookie kernel: LDISKFS-fs error (device sdz):
+ ldiskfs_lookup: unlinked inode 5384166 in dir #145170469</screen>
+ <para>Dec 29 14:11:32 mookie kernel: Remounting filesystem read-only</para>
+ <para>In such a situation, it is normally required that e2fsck only be run on the bad device before placing the device back into service.</para>
+ <para>In the vast majority of cases, Lustre can cope with any inconsistencies it finds on the disk and between other devices in the file system.</para>
<note>
- <para>If the OSTs do not have shared file system access to the MDS, a stub <literal>mdsdb</literal> file, <literal>{mdsdb}.mdshdr</literal>, is generated. This can be used instead of the full <literal>mdsdb</literal> file.</para>
+ <para><literal>lfsck</literal> is rarely required for Lustre operation.</para>
</note>
- <para><emphasis role="bold">Example:</emphasis></para>
- <screen>[root@oss161 ~]# e2fsck -n -v --mdsdb /tmp/mdsdb --ostdb \ /tmp/ostdb /dev/sda
-e2fsck 1.39.cfs1 (29-May-2006)
-Warning: skipping journal recovery because doing a read-only filesystem check.
-lustre-OST0000 contains a file system with errors, check forced.
-Pass 1: Checking inodes, blocks, and sizes
-Pass 2: Checking directory structure
-Pass 3: Checking directory connectivity
-Pass 4: Checking reference counts
-Pass 5: Checking group summary information
-Free blocks count wrong (989015, counted=817968).
-Fix? no
-
-Free inodes count wrong (262088, counted=261767).
-Fix? no
-
-Pass 6: Acquiring information for lfsck
-OST: 'lustre-OST0000_UUID' ost idx 0: compat 0x2 rocomp 0 incomp 0x2
-OST: num files = 321
-OST: last_id = 321
-
-lustre-OST0000: ******* WARNING: Filesystem still has errors *******
-
- 56 inodes used (0%)
- 27 non-contiguous inodes (48.2%)
- # of inodes with ind/dind/tind blocks: 13/0/0
-59561 blocks used (5%)
- 0 bad blocks
- 1 large file
- 329 regular files
- 39 directories
- 0 character device files
- 0 block device files
- 0 fifos
- 0 links
- 0 symbolic links (0 fast symbolic links)
- 0 sockets
- --------
- 368 files
- </screen>
- </listitem>
- <listitem>
- <para>Make the <literal>mdsdb</literal> file and all <literal>ostdb</literal> files available on a mounted client and run <literal>lfsck</literal> to examine the file system. Optionally, correct the defects found by <literal>lfsck</literal>.</para>
- <screen>script /root/lfsck.lustre.log
-lfsck -n -v --mdsdb /tmp/mdsdb --ostdb /tmp/{ost1db} /tmp/{ost2db} ... /lustre/mount/point\</screen>
- <para><emphasis role="bold">Example:</emphasis></para>
- <screen>script /root/lfsck.lustre.log
-lfsck -n -v --mdsdb /home/mdsdb --ostdb /home/{ost1db} /mnt/lustre/client/
-MDSDB: /home/mdsdb
-OSTDB[0]: /home/ostdb
-MOUNTPOINT: /mnt/lustre/client/
-MDS: max_id 288 OST: max_id 321
-lfsck: ost_idx 0: pass1: check for duplicate objects
-lfsck: ost_idx 0: pass1 OK (287 files total)
-lfsck: ost_idx 0: pass2: check for missing inode objects
-lfsck: ost_idx 0: pass2 OK (287 objects)
-lfsck: ost_idx 0: pass3: check for orphan objects
-[0] uuid lustre-OST0000_UUID
-[0] last_id 288
-[0] zero-length orphan objid 1
-lfsck: ost_idx 0: pass3 OK (321 files total)
-lfsck: pass4: check for duplicate object references
-lfsck: pass4 OK (no duplicates)
-lfsck: fixed 0 errors</screen>
- <para>By default, <literal>lfsck</literal> reports errors, but it does not repair any inconsistencies found. <literal>lfsck</literal> checks for three kinds of inconsistencies:</para>
- <itemizedlist>
- <listitem>
- <para>Inode exists but has missing objects (dangling inode). This normally happens if there was a problem with an OST.</para>
- </listitem>
- <listitem>
- <para>Inode is missing but OST has unreferenced objects (orphan object). Normally, this happens if there was a problem with the MDS.</para>
- </listitem>
- <listitem>
- <para>Multiple inodes reference the same objects. This can happen if the MDS is corrupted or if the MDS storage is cached and loses some, but not all, writes.</para>
- </listitem>
- </itemizedlist>
- <para>If the file system is in use and being modified while the <literal>--mdsdb</literal> and <literal>--ostdb</literal> steps are running, <literal>lfsck</literal> may report inconsistencies where none exist due to files and objects being created/removed after the database files were collected. Examine the <literal>lfsck</literal> results closely. You may want to re-run the test.</para>
- </listitem>
- </orderedlist>
- <section xml:id="dbdoclet.50438225_13916">
- <title><indexterm><primary>recovery</primary><secondary>orphaned objects</secondary></indexterm>Working with Orphaned Objects</title>
- <para>The easiest problem to resolve is that of orphaned objects. When the <literal>-l</literal> option for <literal>lfsck</literal> is used, these objects are linked to new files and put into <literal>lost+found</literal> in the Lustre file system, where they can be examined and saved or deleted as necessary. If you are certain the objects are not useful, run <literal>lfsck</literal> with the <literal>-d</literal> option to delete orphaned objects and free up any space they are using.</para>
- <para>To fix dangling inodes, use <literal>lfsck</literal> with the <literal>-c</literal> option to create new, zero-length objects on the OSTs. These files read back with binary zeros for stripes that had objects re-created. Even without <literal>lfsck</literal> repair, these files can be read by entering:</para>
- <screen>dd if=/lustre/bad/file of=/new/file bs=4k conv=sync,noerror</screen>
- <para>Because it is rarely useful to have files with large holes in them, most users delete these files after reading them (if useful) and/or restoring them from backup.</para>
- <note>
- <para>You cannot write to the holes of such files without having <literal>lfsck</literal> re-create the objects. Generally, it is easier to delete these files and restore them from backup.</para>
- </note>
- <para>To fix inodes with duplicate objects, use <literal>lfsck</literal> with the <literal>-c</literal> option to copy the duplicate object to a new object and assign it to a file. One file will be okay and the duplicate will likely contain garbage. By itself, <literal>lfsck</literal> cannot tell which file is the usable one.</para>
+ <para>For problem analysis, it is strongly recommended that <literal>e2fsck</literal> be run under a logger, like script, to record all of the output and changes that are made to the file system in case this information is needed later.</para>
+ <para>If time permits, it is also a good idea to first run <literal>e2fsck</literal> in non-fixing mode (-n option) to assess the type and extent of damage to the file system. The drawback is that in this mode, <literal>e2fsck</literal> does not recover the file system journal, so there may appear to be file system corruption when none really exists.</para>
+ <para>To address concern about whether corruption is real or only due to the journal not being replayed, you can briefly mount and unmount the <literal>ldiskfs</literal> filesystem directly on the node with Lustre stopped (NOT via Lustre), using a command similar to:</para>
+ <screen>mount -t ldiskfs /dev/{ostdev} /mnt/ost; umount /mnt/ost</screen>
+ <para>This causes the journal to be recovered.</para>
+ <para>The <literal>e2fsck</literal> utility works well when fixing file system corruption (better than similar file system recovery tools and a primary reason why <literal>ldiskfs</literal> was chosen over other file systems for Lustre). However, it is often useful to identify the type of damage that has occurred so an <literal>ldiskfs</literal> expert can make intelligent decisions about what needs fixing, in place of <literal>e2fsck</literal>.</para>
+ <screen>root# {stop lustre services for this device, if running}
+ root# script /tmp/e2fsck.sda
+ Script started, file is /tmp/e2fsck.sda
+ root# mount -t ldiskfs /dev/sda /mnt/ost
+ root# umount /mnt/ost
+ root# e2fsck -fn /dev/sda # don't fix file system, just check for corruption
+ :
+ [e2fsck output]
+ :
+ root# e2fsck -fp /dev/sda # fix filesystem using "prudent" answers (usually 'y')
+ </screen>
+ <para>In addition, the <literal>e2fsprogs</literal> package contains the <literal>lfsck</literal> tool, which does distributed coherency checking for the Lustre file system after <literal>e2fsck</literal> has been run. Running <literal>lfsck</literal> is NOT required in a large majority of cases, at a small risk of having some leaked space in the file system. To avoid a lengthy downtime, it can be run (with care) after Lustre is started.</para>
+ </section>
+ <section xml:id="dbdoclet.50438225_37365">
+ <title><indexterm><primary>recovery</primary><secondary>corruption of Lustre file system</secondary></indexterm>Recovering from Corruption in the Lustre File System</title>
+ <para>In cases where the MDS or an OST becomes corrupt, you can run a distributed check on the file system to determine what sort of problems exist. Use <literal>lfsck</literal> to correct any defects found.</para>
+ <orderedlist>
+ <listitem>
+ <para>Stop the Lustre file system.</para>
+ </listitem>
+ <listitem>
+ <para>Run <literal>e2fsck -f</literal> on the individual MDS / OST that had problems to fix any local file system damage.</para>
+ <para>We recommend running <literal>e2fsck</literal> under script, to create a log of changes made to the file system in case it is needed later. After <literal>e2fsck</literal> is run, bring up the file system, if necessary, to reduce the outage window.</para>
+ </listitem>
+ <listitem>
+ <para>Run a full <literal>e2fsck</literal> of the MDS to create a database for <literal>lfsck</literal>. You <emphasis>must</emphasis> use the <literal>-n</literal> option for a mounted file system, otherwise you will corrupt the file system.</para>
+ <screen>e2fsck -n -v --mdsdb /tmp/mdsdb /dev/{mdsdev}
+ </screen>
+ <para>The <literal>mds</literal>db file can grow fairly large, depending on the number of files in the file system (10 GB or more for millions of files, though the actual file size is larger because the file is sparse). It is quicker to write the file to a local file system due to seeking and small writes. Depending on the number of files, this step can take several hours to complete.</para>
+ <para><emphasis role="bold">Example</emphasis></para>
+ <screen>e2fsck -n -v --mdsdb /tmp/mdsdb /dev/sdb
+ e2fsck 1.39.cfs1 (29-May-2006)
+ Warning: skipping journal recovery because doing a read-only filesystem check.
+ lustre-MDT0000 contains a file system with errors, check forced.
+ Pass 1: Checking inodes, blocks, and sizes
+ MDS: ost_idx 0 max_id 288
+ MDS: got 8 bytes = 1 entries in lov_objids
+ MDS: max_files = 13
+ MDS: num_osts = 1
+ mds info db file written
+ Pass 2: Checking directory structure
+ Pass 3: Checking directory connectivity
+ Pass 4: Checking reference counts
+ Pass 5: Checking group summary information
+ Free blocks count wrong (656160, counted=656058).
+ Fix? no
+
+ Free inodes count wrong (786419, counted=786036).
+ Fix? no
+
+ Pass 6: Acquiring information for lfsck
+ MDS: max_files = 13
+ MDS: num_osts = 1
+ MDS: 'lustre-MDT0000_UUID' mdt idx 0: compat 0x4 rocomp 0x1 incomp 0x4
+ lustre-MDT0000: ******* WARNING: Filesystem still has errors *******
+ 13 inodes used (0%)
+ 2 non-contiguous inodes (15.4%)
+ # of inodes with ind/dind/tind blocks: 0/0/0
+ 130272 blocks used (16%)
+ 0 bad blocks
+ 1 large file
+ 296 regular files
+ 91 directories
+ 0 character device files
+ 0 block device files
+ 0 fifos
+ 0 links
+ 0 symbolic links (0 fast symbolic links)
+ 0 sockets
+ --------
+ 387 files
+ </screen>
+ </listitem>
+ <listitem>
+ <para>Make this file accessible on all OSTs, either by using a shared file system or copying the file to the OSTs. The <literal>pdcp</literal> command is useful here.</para>
+ <para>The <literal>pdcp</literal> command (installed with <literal>pdsh</literal>), can be used to copy files to groups of hosts. <literal>pdcp</literal> is available here:</para>
+ <para><link xl:href="http://sourceforge.net/projects/pdsh">http://sourceforge.net/projects/pdsh</link></para>
+ </listitem>
+ <listitem>
+ <para>Run a similar <literal>e2fsck</literal> step on the OSTs. The <literal>e2fsck --ostdb</literal> command can be run in parallel on all OSTs.</para>
+ <screen>e2fsck -n -v --mdsdb /tmp/mdsdb --ostdb /tmp/{ostNdb} \/dev/{ostNdev}
+ </screen>
+ <para>The <literal>mdsdb</literal> file is read-only in this step; a single copy can be shared by all OSTs.</para>
+ <note>
+ <para>If the OSTs do not have shared file system access to the MDS, a stub <literal>mdsdb</literal> file, <literal>{mdsdb}.mdshdr</literal>, is generated. This can be used instead of the full <literal>mdsdb</literal> file.</para>
+ </note>
+ <para><emphasis role="bold">Example:</emphasis></para>
+ <screen>[root@oss161 ~]# e2fsck -n -v --mdsdb /tmp/mdsdb --ostdb \ /tmp/ostdb /dev/sda
+ e2fsck 1.39.cfs1 (29-May-2006)
+ Warning: skipping journal recovery because doing a read-only filesystem check.
+ lustre-OST0000 contains a file system with errors, check forced.
+ Pass 1: Checking inodes, blocks, and sizes
+ Pass 2: Checking directory structure
+ Pass 3: Checking directory connectivity
+ Pass 4: Checking reference counts
+ Pass 5: Checking group summary information
+ Free blocks count wrong (989015, counted=817968).
+ Fix? no
+
+ Free inodes count wrong (262088, counted=261767).
+ Fix? no
+
+ Pass 6: Acquiring information for lfsck
+ OST: 'lustre-OST0000_UUID' ost idx 0: compat 0x2 rocomp 0 incomp 0x2
+ OST: num files = 321
+ OST: last_id = 321
+
+ lustre-OST0000: ******* WARNING: Filesystem still has errors *******
+
+ 56 inodes used (0%)
+ 27 non-contiguous inodes (48.2%)
+ # of inodes with ind/dind/tind blocks: 13/0/0
+ 59561 blocks used (5%)
+ 0 bad blocks
+ 1 large file
+ 329 regular files
+ 39 directories
+ 0 character device files
+ 0 block device files
+ 0 fifos
+ 0 links
+ 0 symbolic links (0 fast symbolic links)
+ 0 sockets
+ --------
+ 368 files
+ </screen>
+ </listitem>
+ <listitem>
+ <para>Make the <literal>mdsdb</literal> file and all <literal>ostdb</literal> files available on a mounted client and run <literal>lfsck</literal> to examine the file system. Optionally, correct the defects found by <literal>lfsck</literal>.</para>
+ <screen>script /root/lfsck.lustre.log
+ lfsck -n -v --mdsdb /tmp/mdsdb --ostdb /tmp/{ost1db} /tmp/{ost2db} ... /lustre/mount/point\</screen>
+ <para><emphasis role="bold">Example:</emphasis></para>
+ <screen>script /root/lfsck.lustre.log
+ lfsck -n -v --mdsdb /home/mdsdb --ostdb /home/{ost1db} /mnt/lustre/client/
+ MDSDB: /home/mdsdb
+ OSTDB[0]: /home/ostdb
+ MOUNTPOINT: /mnt/lustre/client/
+ MDS: max_id 288 OST: max_id 321
+ lfsck: ost_idx 0: pass1: check for duplicate objects
+ lfsck: ost_idx 0: pass1 OK (287 files total)
+ lfsck: ost_idx 0: pass2: check for missing inode objects
+ lfsck: ost_idx 0: pass2 OK (287 objects)
+ lfsck: ost_idx 0: pass3: check for orphan objects
+ [0] uuid lustre-OST0000_UUID
+ [0] last_id 288
+ [0] zero-length orphan objid 1
+ lfsck: ost_idx 0: pass3 OK (321 files total)
+ lfsck: pass4: check for duplicate object references
+ lfsck: pass4 OK (no duplicates)
+ lfsck: fixed 0 errors</screen>
+ <para>By default, <literal>lfsck</literal> reports errors, but it does not repair any inconsistencies found. <literal>lfsck</literal> checks for three kinds of inconsistencies:</para>
+ <itemizedlist>
+ <listitem>
+ <para>Inode exists but has missing objects (dangling inode). This normally happens if there was a problem with an OST.</para>
+ </listitem>
+ <listitem>
+ <para>Inode is missing but OST has unreferenced objects (orphan object). Normally, this happens if there was a problem with the MDS.</para>
+ </listitem>
+ <listitem>
+ <para>Multiple inodes reference the same objects. This can happen if the MDS is corrupted or if the MDS storage is cached and loses some, but not all, writes.</para>
+ </listitem>
+ </itemizedlist>
+ <para>If the file system is in use and being modified while the <literal>--mdsdb</literal> and <literal>--ostdb</literal> steps are running, <literal>lfsck</literal> may report inconsistencies where none exist due to files and objects being created/removed after the database files were collected. Examine the <literal>lfsck</literal> results closely. You may want to re-run the test.</para>
+ </listitem>
+ </orderedlist>
+ <section xml:id="dbdoclet.50438225_13916">
+ <title><indexterm><primary>recovery</primary><secondary>orphaned objects</secondary></indexterm>Working with Orphaned Objects</title>
+ <para>The easiest problem to resolve is that of orphaned objects. When the <literal>-l</literal> option for <literal>lfsck</literal> is used, these objects are linked to new files and put into <literal>lost+found</literal> in the Lustre file system, where they can be examined and saved or deleted as necessary. If you are certain the objects are not useful, run <literal>lfsck</literal> with the <literal>-d</literal> option to delete orphaned objects and free up any space they are using.</para>
+ <para>To fix dangling inodes, use <literal>lfsck</literal> with the <literal>-c</literal> option to create new, zero-length objects on the OSTs. These files read back with binary zeros for stripes that had objects re-created. Even without <literal>lfsck</literal> repair, these files can be read by entering:</para>
+ <screen>dd if=/lustre/bad/file of=/new/file bs=4k conv=sync,noerror</screen>
+ <para>Because it is rarely useful to have files with large holes in them, most users delete these files after reading them (if useful) and/or restoring them from backup.</para>
+ <note>
+ <para>You cannot write to the holes of such files without having <literal>lfsck</literal> re-create the objects. Generally, it is easier to delete these files and restore them from backup.</para>
+ </note>
+ <para>To fix inodes with duplicate objects, use <literal>lfsck</literal> with the <literal>-c</literal> option to copy the duplicate object to a new object and assign it to a file. One file will be okay and the duplicate will likely contain garbage. By itself, <literal>lfsck</literal> cannot tell which file is the usable one.</para>
+ </section>
+ </section>
+ <section xml:id="dbdoclet.50438225_12316">
+ <title><indexterm><primary>recovery</primary><secondary>unavailable OST</secondary></indexterm>Recovering from an Unavailable OST</title>
+ <para>One of the most common problems encountered in a Lustre environment is when an OST becomes unavailable, because of a network partition, OSS node crash, etc. When this happens, the OST's clients pause and wait for the OST to become available again, either on the primary OSS or a failover OSS. When the OST comes back online, Lustre starts a recovery process to enable clients to reconnect to the OST. Lustre servers put a limit on the time they will wait in recovery for clients to reconnect. The timeout length is determined by the <literal>obd_timeout</literal> parameter.</para>
+ <para>During recovery, clients reconnect and replay their requests serially, in the same order they were done originally. Until a client receives a confirmation that a given transaction has been written to stable storage, the client holds on to the transaction, in case it needs to be replayed. Periodically, a progress message prints to the log, stating how_many/expected clients have reconnected. If the recovery is aborted, this log shows how many clients managed to reconnect. When all clients have completed recovery, or if the recovery timeout is reached, the recovery period ends and the OST resumes normal request processing.</para>
+ <para>If some clients fail to replay their requests during the recovery period, this will not stop the recovery from completing. You may have a situation where the OST recovers, but some clients are not able to participate in recovery (e.g. network problems or client failure), so they are evicted and their requests are not replayed. This would result in any operations on the evicted clients failing, including in-progress writes, which would cause cached writes to be lost. This is a normal outcome; the recovery cannot wait indefinitely, or the file system would be hung any time a client failed. The lost transactions are an unfortunate result of the recovery process.</para>
+ <note>
+ <para>The version-based recovery (VBR) feature enables a failed client to be ''skipped'', so remaining clients can replay their requests, resulting in a more successful recovery from a downed OST. For more information about the VBR feature, see <xref linkend="lustrerecovery"/>(Version-based Recovery).</para>
+ </note>
+ </section>
+ <section xml:id="dbdoclet.lfsckadmin">
+ <title><indexterm><primary>recovery</primary><secondary>oiscrub</secondary></indexterm><indexterm><primary>recovery</primary><secondary>lfsck</secondary></indexterm>Checking the Lustre file system with LFSCK</title>
+ <para>LFSCK is an administrative tool that has been introduced in Lustre 2.3. Control and monitoring of LFSCK is through a user interface. LFSCK included in Lustre 2.3 contains functionality to repair the OI table (OI Scrub) which is useful in restoring from a file-level backup, see <xref linkend='dbdoclet.50438207_71633'/>.</para>
+ <para>The LFSCK user interface is implemented through LFSCK and <literal>procfs</literal> interfaces. LFSCK supports three types of interface: switch interfaces, status interfaces and adjustments interfaces. These interfaces are detailed below.</para>
+ </section>
+ <section>
+ <title>LFSCK switch interface</title>
+ <section>
+ <title>Manually Starting LFSCK</title>
+ <section>
+ <title>Synopsis</title>
+ <screen>lctl lfsck_start <-M | --device MDT_device> \
+ [-e | --error error_handle] \
+ [-h | --help] \
+ [-m | --method iteration_method] \
+ [-n | --dryrun switch] \
+ [-r | --reset] \
+ [-s | --speed speed_limit] \
+ [-t | --type lfsck_type[,lfsck_type...]]
+ </screen>
+ </section>
+ <section>
+ <title>Description</title>
+ <para>This is command is used by LFSCK after the MDT is mounted.</para>
+ </section>
+ <section>
+ <title>Options</title>
+ <para>The various <literal>lfsck_start</literal> options are listed and described below. For a complete list of available options, type <literal>lctl lfsck_start -h</literal>.</para>
+ <informaltable frame="all">
+ <tgroup cols="3">
+ <colspec colname="c1" colwidth="3*"/>
+ <colspec colname="c2" colwidth="7*"/>
+ <thead>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para><literal>-M | --device</literal> </para>
+ </entry>
+ <entry>
+ <para>The MDT device to start LFSCK on. This will be a requirement when multiple MDTs are supported.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><literal>-e | --error</literal> </para>
+ </entry>
+ <entry>
+ <para>Error handle, <literal>continue</literal> (default) or <literal>abort</literal>. Specify whether the LFSCK will stop or not if fail to repair something. If it is not specified, the saved value (when resuming from checkpoint) will be used if present. This option cannot be changed if LFSCK is running.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><literal>-h | --help</literal> </para>
+ </entry>
+ <entry>
+ <para>Operating help.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><literal>-m | --method</literal> </para>
+ </entry>
+ <entry>
+ <para>Method for scanning the MDT device. Currently only <literal>otable</literal> (object table based) iteration is supported. If it is not specified, the saved value (when resuming from checkpoint) will be used if present.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><literal>-n | --dryrun</literal> </para>
+ </entry>
+ <entry>
+ <para>Perform a trial without making any changes.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><literal>-r | --reset</literal> </para>
+ </entry>
+ <entry>
+ <para>Reset the start position for the object iteration to the beginning for the specified MDT. By default the iterator will resume scanning from the last checkpoint (saved periodically by LFSCK) provided it is available.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><literal>-s | --speed</literal> </para>
+ </entry>
+ <entry>
+ <para>Set the upper speed limit of LFSCK processing in objects per second. If it is not specified, the saved value (when resuming from checkpoint) or default value of 0 (0 = run as fast as possible) is used. Speed can be adjusted while LFSCK is running with the adjustment interface.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><literal>-t | --type</literal> </para>
+ </entry>
+ <entry>
+ <para>The type of checking/repairing that should be performed. The new LFSCK framework provides the uniform interfaces for kinds of system consistency checking/repairing, including 'layout' (MDT-OST consistency for phase II), and 'DNE' (MDT-MDT consistency for phase III). Additional consistency checks may be added in future. If type is specified the LFSCK component(s) which ran last time and not finished, or the component(s) corresponding to some known system inconsistency, will be started. The default type is OI Scrub.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ </section>
+ <section>
+ <title>Manually Stopping <literal>lfsck</literal></title>
+ <section>
+ <title>Synopsis</title>
+ <screen>lctl lfsck_stop <-M | --device MDT_device> \
+ [-h | --help]
+ </screen>
+ </section>
+ <section>
+ <title>Description</title>
+ <para>This is command is used by <literal>lfsck</literal> after the MDT is mounted.</para>
+ </section>
+ <section>
+ <title>Options</title>
+ <para>The various <literal>lfsck_stop</literal> options are listed and described below. For a complete list of available options, type <literal>lctl lfsck_stop h</literal>.</para>
+ <informaltable frame="all">
+ <tgroup cols="3">
+ <colspec colname="c1" colwidth="3*"/>
+ <colspec colname="c2" colwidth="7*"/>
+ <thead>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para><literal>-M | --device</literal> </para>
+ </entry>
+ <entry>
+ <para>The MDT device to start LFSCK on. This will be a requirement when multiple MDTs are supported.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para><literal>-h | --help</literal> </para>
+ </entry>
+ <entry>
+ <para>Operating help.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ </section>
+ </section>
+ <section>
+ <title><literal>lfsck</literal> status interface</title>
+ <section>
+ <title>LFSCK status via <literal>procfs</literal></title>
+ <section>
+ <title>Synopsis</title>
+ <screen>lctl get_param -n osd-ldisk.${FSNAME}-${MDT_device}.oi_scrub
+ </screen>
+ </section>
+ <section>
+ <title>Description</title>
+ <para>For each LFSCK component there is a dedicated procfs interface to trace corresponding LFSCK component status. For OI Scrub, the interface is the osd layer procfs interface, named <literal>oi_scrub</literal>. To show OI Scrub status, the standard <literal>lctl get_param</literal> command is used as described in the synopsis.</para>
+ </section>
+ <section>
+ <title>Output</title>
+ <informaltable frame="all">
+ <tgroup cols="3">
+ <colspec colname="c1" colwidth="3*"/>
+ <colspec colname="c2" colwidth="7*"/>
+ <thead>
+ <row>
+ <entry nameend="c2" namest="c1">
+ <para><emphasis role="bold">Information</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Detail</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para>General Information</para>
+ </entry>
+ <entry>
+ <itemizedlist>
+ <listitem><para>Name: OI scrub.</para></listitem>
+ <listitem><para>OI scrub magic id (an identifier unique to OI scrub).</para></listitem>
+ <listitem><para>OI files count.</para></listitem>
+ <listitem><para>Status: one of the status - 'init', 'scanning', 'completed', 'failed', 'stopped', 'paused', or 'crashed'.</para></listitem>
+ <listitem><para>Flags: including - 'recreated' (OI file(s) is/are removed/recreated), 'inconsistent' (restored from file-level backup), and 'auto' (triggered by non-UI mechanism).</para></listitem>
+ <listitem><para>Parameters: OI scrub parameters, like 'failout'.</para></listitem>
+ <listitem><para>Time Since Last Completed.</para></listitem>
+ <listitem><para>Time Since Latest Start.</para></listitem>
+ <listitem><para>Time Since Last Checkpoint.</para></listitem>
+ <listitem><para>Latest Start Position: the position for the latest scrub started from.</para></listitem>
+ <listitem><para>Last Checkpoint Position.</para></listitem>
+ <listitem><para>First Failure Position: the position for the first object to be repaired.</para></listitem>
+ <listitem><para>Current Position.</para></listitem>
+ <listitem><para></para></listitem>
+ <listitem><para></para></listitem>
+ <listitem><para></para></listitem>
+ </itemizedlist>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>Statistics</para>
+ </entry>
+ <entry>
+ <itemizedlist>
+ <listitem><para>Checked: how many objects have been scanned.</para></listitem>
+ <listitem><para>Updated: how many objects have been repaired.</para></listitem>
+ <listitem><para>Failed: how many objects failed to be repaired.</para></listitem>
+ <listitem><para>Prior Updated: how many objects have been repaired which are triggered by parallel RPC.</para></listitem>
+ <listitem><para>Success Count: how many completed scrub ran on the device.</para></listitem>
+ <listitem><para>Run Time: how long the scrub has run, tally from the time of scanning from the beginning of the specified MDT device, not include the paused/failure time among checkpoints.</para></listitem>
+ <listitem><para>Average Speed: calculated by checked / run_time.</para></listitem>
+ <listitem><para>Real-Time Speed: the speed since last checkpoint if the scrub is running.</para></listitem>
+ </itemizedlist>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ </section>
+ </section>
+ <section>
+ <title><literal>lfsck</literal> adjustment interface</title>
+ <section>
+ <title>Rate control</title>
+ <section>
+ <title>Synopsis</title>
+ <screen>lctl set_param mdt.${FSNAME}-${MDT_device}.lfsck_speed_limit=N
+ </screen>
+ </section>
+ <section>
+ <title>Description</title>
+ <para>Change the LFSCK upper speed limit.</para>
+ </section>
+ <section>
+ <title>Values</title>
+ <informaltable frame="all">
+ <tgroup cols="3">
+ <colspec colname="c1" colwidth="3*"/>
+ <colspec colname="c2" colwidth="7*"/>
+ <tbody>
+ <row>
+ <entry>
+ <para>0</para>
+ </entry>
+ <entry>
+ <para>No speed limit (run at maximum speed.)</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>positive integer</para>
+ </entry>
+ <entry>
+ <para>Maximum number of objects to scan per second.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ </section>
+ <section>
+ <title>Mount options</title>
+ <section>
+ <title>Synopsis</title>
+ <screen>lctl set_param osd_ldiskfs.${FSNAME}-${MDT_device}.auto_scrub=N
+ </screen>
+ </section>
+ <section>
+ <title>Description</title>
+ <para>Typically, the MDT will detect restoration from a file-level backup during mount. For convenience an mount option <literal>noscrub</literal> is provided for MDTs. <literal>noscrub</literal> prevents the OI Scrub starting automatically when the MDT is mounted. The administrator can start LFSCK manually after the MDT is mounted with <literal>lctl</literal>. Manually starting LFSCK after mounting provides finer control over the starting conditions.</para>
+ </section>
+ <section>
+ <title>Values</title>
+ <informaltable frame="all">
+ <tgroup cols="3">
+ <colspec colname="c1" colwidth="3*"/>
+ <colspec colname="c2" colwidth="7*"/>
+ <tbody>
+ <row>
+ <entry>
+ <para>0</para>
+ </entry>
+ <entry>
+ <para>Do not start OI Scrub automatically.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>positive integer</para>
+ </entry>
+ <entry>
+ <para>Manually start OI Scrub if needed.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ </section>
</section>
- </section>
- <section xml:id="dbdoclet.50438225_12316">
- <title><indexterm><primary>recovery</primary><secondary>unavailable OST</secondary></indexterm>Recovering from an Unavailable OST</title>
- <para>One of the most common problems encountered in a Lustre environment is when an OST becomes unavailable, because of a network partition, OSS node crash, etc. When this happens, the OST's clients pause and wait for the OST to become available again, either on the primary OSS or a failover OSS. When the OST comes back online, Lustre starts a recovery process to enable clients to reconnect to the OST. Lustre servers put a limit on the time they will wait in recovery for clients to reconnect. The timeout length is determined by the <literal>obd_timeout</literal> parameter.</para>
- <para>During recovery, clients reconnect and replay their requests serially, in the same order they were done originally. Until a client receives a confirmation that a given transaction has been written to stable storage, the client holds on to the transaction, in case it needs to be replayed. Periodically, a progress message prints to the log, stating how_many/expected clients have reconnected. If the recovery is aborted, this log shows how many clients managed to reconnect. When all clients have completed recovery, or if the recovery timeout is reached, the recovery period ends and the OST resumes normal request processing.</para>
- <para>If some clients fail to replay their requests during the recovery period, this will not stop the recovery from completing. You may have a situation where the OST recovers, but some clients are not able to participate in recovery (e.g. network problems or client failure), so they are evicted and their requests are not replayed. This would result in any operations on the evicted clients failing, including in-progress writes, which would cause cached writes to be lost. This is a normal outcome; the recovery cannot wait indefinitely, or the file system would be hung any time a client failed. The lost transactions are an unfortunate result of the recovery process.</para>
- <note>
- <para>The version-based recovery (VBR) feature enables a failed client to be ''skipped'', so remaining clients can replay their requests, resulting in a more successful recovery from a downed OST. For more information about the VBR feature, see <xref linkend="lustrerecovery"/>(Version-based Recovery).</para>
- </note>
- </section>
</chapter>
git://git.whamcloud.com / doc / manual.git / commitdiff