- <listitem>
- <para>Run a full <literal>e2fsck</literal> of the MDS to create a database for LFSCK. You <emphasis>must</emphasis> use the <literal>-n</literal> option for a mounted file system, otherwise you will corrupt the file system.</para>
- <screen>e2fsck -n -v --mdsdb /tmp/mdsdb /dev/{mdsdev}
- </screen>
- <para>The <literal>mds</literal>db file can grow fairly large, depending on the number of files in the file system (10 GB or more for millions of files, though the actual file size is larger because the file is sparse). It is quicker to write the file to a local file system due to seeking and small writes. Depending on the number of files, this step can take several hours to complete.</para>
- <para><emphasis role="bold">Example</emphasis></para>
- <screen>e2fsck -n -v --mdsdb /tmp/mdsdb /dev/sdb
-e2fsck 1.42.5.wc3 (15-Sep-2012)
-Warning: skipping journal recovery because doing a read-only filesystem check.
-lustre-MDT0000 contains a file system with errors, check forced.
-Pass 1: Checking inodes, blocks, and sizes
-MDS: ost_idx 0 max_id 288
-MDS: got 8 bytes = 1 entries in lov_objids
-MDS: max_files = 13
-MDS: num_osts = 1
-mds info db file written
-Pass 2: Checking directory structure
-Pass 3: Checking directory connectivity
-Pass 4: Checking reference counts
-Pass 5: Checking group summary information
-Free blocks count wrong (656160, counted=656058).
-Fix? no
-
-Free inodes count wrong (786419, counted=786036).
-Fix? no
-
-Pass 6: Acquiring information for lfsck
-MDS: max_files = 13
-MDS: num_osts = 1
-MDS: 'lustre-MDT0000_UUID' mdt idx 0: compat 0x4 rocomp 0x1 incomp 0x4
-lustre-MDT0000: ******* WARNING: Filesystem still has errors *******
-13 inodes used (0%)
-2 non-contiguous inodes (15.4%)
-# of inodes with ind/dind/tind blocks: 0/0/0
-130272 blocks used (16%)
-0 bad blocks
-1 large file
-296 regular files
-91 directories
-0 character device files
-0 block device files
-0 fifos
-0 links
-0 symbolic links (0 fast symbolic links)
-0 sockets
---------
-387 files
- </screen>
- </listitem>
- <listitem>
- <para>Make this file accessible on all OSTs, either by using a shared file system or copying the file to the OSTs. The <literal>pdcp</literal> command is useful here.</para>
- <para>The <literal>pdcp</literal> command (installed with <literal>pdsh</literal>), can be used to copy files to groups of hosts. <literal>pdcp</literal> is available here:</para>
- <para><link xl:href="http://sourceforge.net/projects/pdsh">http://sourceforge.net/projects/pdsh</link></para>
- </listitem>
- <listitem>
- <para>Run a similar <literal>e2fsck</literal> step on the OSTs. The <literal>e2fsck --ostdb</literal> command can be run in parallel on all OSTs.</para>
- <screen>e2fsck -n -v --mdsdb /tmp/mdsdb --ostdb /tmp/{ostNdb} \/dev/{ostNdev}
- </screen>
- <para>The <literal>mdsdb</literal> file is read-only in this step; a single copy can be shared by all OSTs.</para>
- <note>
- <para>If the OSTs do not have shared file system access to the MDS, a stub <literal>mdsdb</literal> file, <literal>{mdsdb}.mdshdr</literal>, is generated. This can be used instead of the full <literal>mdsdb</literal> file.</para>
- </note>
- <para><emphasis role="bold">Example:</emphasis></para>
- <screen>[root@oss161 ~]# e2fsck -n -v --mdsdb /tmp/mdsdb --ostdb \ /tmp/ostdb /dev/sda
-e2fsck 1.42.5.wc3 (15-Sep-2012)
-Warning: skipping journal recovery because doing a read-only filesystem check.
-lustre-OST0000 contains a file system with errors, check forced.
-Pass 1: Checking inodes, blocks, and sizes
-Pass 2: Checking directory structure
-Pass 3: Checking directory connectivity
-Pass 4: Checking reference counts
-Pass 5: Checking group summary information
-Free blocks count wrong (989015, counted=817968).
-Fix? no
-
-Free inodes count wrong (262088, counted=261767).
-Fix? no
-
-Pass 6: Acquiring information for lfsck
-OST: 'lustre-OST0000_UUID' ost idx 0: compat 0x2 rocomp 0 incomp 0x2
-OST: num files = 321
-OST: last_id = 321
-
-lustre-OST0000: ******* WARNING: Filesystem still has errors *******
-
-56 inodes used (0%)
-27 non-contiguous inodes (48.2%)
-# of inodes with ind/dind/tind blocks: 13/0/0
-59561 blocks used (5%)
-0 bad blocks
-1 large file
-329 regular files
-39 directories
-0 character device files
-0 block device files
-0 fifos
-0 links
-0 symbolic links (0 fast symbolic links)
-0 sockets
---------
-368 files
- </screen>
- </listitem>
- <listitem>
- <para>Make the <literal>mdsdb</literal> file and all <literal>ostdb</literal> files available on a mounted client and run LFSKC to examine the file system. Optionally, correct the defects found by LFSCK.</para>
- <screen>script /root/lfsck.lustre.log
- lfsck -n -v --mdsdb /tmp/mdsdb --ostdb /tmp/{ost1db} /tmp/{ost2db} ... /lustre/mount/point\</screen>
- <para><emphasis role="bold">Example:</emphasis></para>
- <screen>script /root/lfsck.lustre.log
-lfsck -n -v --mdsdb /home/mdsdb --ostdb /home/{ost1db} /mnt/lustre/client/
-MDSDB: /home/mdsdb
-OSTDB[0]: /home/ostdb
-MOUNTPOINT: /mnt/lustre/client/
-MDS: max_id 288 OST: max_id 321
-lfsck: ost_idx 0: pass1: check for duplicate objects
-lfsck: ost_idx 0: pass1 OK (287 files total)
-lfsck: ost_idx 0: pass2: check for missing inode objects
-lfsck: ost_idx 0: pass2 OK (287 objects)
-lfsck: ost_idx 0: pass3: check for orphan objects
-[0] uuid lustre-OST0000_UUID
-[0] last_id 288
-[0] zero-length orphan objid 1
-lfsck: ost_idx 0: pass3 OK (321 files total)
-lfsck: pass4: check for duplicate object references
-lfsck: pass4 OK (no duplicates)
-lfsck: fixed 0 errors</screen>
- <para>By default, LFSCK reports errors, but it does not repair any inconsistencies found. LFSCK checks for three kinds of inconsistencies:</para>
- <itemizedlist>
- <listitem>
- <para>Inode exists but has missing objects (dangling inode). This normally happens if there was a problem with an OST.</para>
- </listitem>
- <listitem>
- <para>Inode is missing but OST has unreferenced objects (orphan object). Normally, this happens if there was a problem with the MDS.</para>
- </listitem>
- <listitem>
- <para>Multiple inodes reference the same objects. This can happen if the MDS is corrupted or if the MDS storage is cached and loses some, but not all, writes.</para>
- </listitem>
- </itemizedlist>
- <para>If the file system is in use and being modified while the <literal>--mdsdb</literal> and <literal>--ostdb</literal> steps are running, LFSCK may report inconsistencies where none exist due to files and objects being created/removed after the database files were collected. Examine the LFSCK results closely. You may want to re-run the test.</para>
- </listitem>