TroubleShootingRecovery.xml

   1 <?xml version='1.0' encoding='utf-8'?>
   2 <chapter xmlns="http://docbook.org/ns/docbook"
   3 xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"
   4 xml:id="troubleshootingrecovery">
   5   <title xml:id="troubleshootingrecovery.title">Troubleshooting
   6   Recovery</title>
   7   <para>This chapter describes what to do if something goes wrong during
   8   recovery. It describes:</para>
   9   <itemizedlist>
  10     <listitem>
  11       <para>
  12         <xref linkend="dbdoclet.50438225_71141" />
  13       </para>
  14     </listitem>
  15     <listitem>
  16       <para>
  17         <xref linkend="dbdoclet.50438225_37365" />
  18       </para>
  19     </listitem>
  20     <listitem>
  21       <para>
  22         <xref linkend="dbdoclet.50438225_12316" />
  23       </para>
  24     </listitem>
  25     <listitem>
  26       <para>
  27         <xref linkend="dbdoclet.lfsckadmin" />
  28       </para>
  29     </listitem>
  30   </itemizedlist>
  31   <section xml:id="dbdoclet.50438225_71141">
  32     <title>
  33     <indexterm>
  34       <primary>recovery</primary>
  35       <secondary>corruption of backing ldiskfs file system</secondary>
  36     </indexterm>Recovering from Errors or Corruption on a Backing ldiskfs File
  37     System</title>
  38     <para>When an OSS, MDS, or MGS server crash occurs, it is not necessary to
  39     run e2fsck on the file system.
  40     <literal>ldiskfs</literal> journaling ensures that the file system remains
  41     consistent over a system crash. The backing file systems are never accessed
  42     directly from the client, so client crashes are not relevant for server
  43     file system consistency.</para>
  44     <para>The only time it is REQUIRED that
  45     <literal>e2fsck</literal> be run on a device is when an event causes
  46     problems that ldiskfs journaling is unable to handle, such as a hardware
  47     device failure or I/O error. If the ldiskfs kernel code detects corruption
  48     on the disk, it mounts the file system as read-only to prevent further
  49     corruption, but still allows read access to the device. This appears as
  50     error "-30" (
  51     <literal>EROFS</literal>) in the syslogs on the server, e.g.:</para>
  52     <screen>Dec 29 14:11:32 mookie kernel: LDISKFS-fs error (device sdz):
  53             ldiskfs_lookup: unlinked inode 5384166 in dir #145170469
  54 Dec 29 14:11:32 mookie kernel: Remounting filesystem read-only </screen>
  55     <para>In such a situation, it is normally required that e2fsck only be run
  56     on the bad device before placing the device back into service.</para>
  57     <para>In the vast majority of cases, the Lustre software can cope with any
  58     inconsistencies found on the disk and between other devices in the file
  59     system.</para>
  60     <note>
  61           <para>The legacy offline-LFSCK tool included with e2fsprogs is rarely
  62       required for Lustre file system operation. offline-LFSCK is not to be
  63       confused with LFSCK tool, which is part of Lustre and provides online
  64       consistency checking.</para>
  65     </note>
  66     <para>For problem analysis, it is strongly recommended that
  67     <literal>e2fsck</literal> be run under a logger, like script, to record all
  68     of the output and changes that are made to the file system in case this
  69     information is needed later.</para>
  70     <para>If time permits, it is also a good idea to first run
  71     <literal>e2fsck</literal> in non-fixing mode (-n option) to assess the type
  72     and extent of damage to the file system. The drawback is that in this mode,
  73     <literal>e2fsck</literal> does not recover the file system journal, so there
  74     may appear to be file system corruption when none really exists.</para>
  75     <para>To address concern about whether corruption is real or only due to
  76     the journal not being replayed, you can briefly mount and unmount the
  77     <literal>ldiskfs</literal> file system directly on the node with the Lustre
  78     file system stopped, using a command similar to:</para>
  79     <screen>mount -t ldiskfs /dev/{ostdev} /mnt/ost; umount /mnt/ost</screen>
  80     <para>This causes the journal to be recovered.</para>
  81     <para>The
  82     <literal>e2fsck</literal> utility works well when fixing file system
  83     corruption (better than similar file system recovery tools and a primary
  84     reason why
  85     <literal>ldiskfs</literal> was chosen over other file systems). However, it
  86     is often useful to identify the type of damage that has occurred so an
  87     <literal>ldiskfs</literal> expert can make intelligent decisions about what
  88     needs fixing, in place of
  89     <literal>e2fsck</literal>.</para>
  90     <screen>root# {stop lustre services for this device, if running}
  91 root# script /tmp/e2fsck.sda
  92 Script started, file is /tmp/e2fsck.sda
  93 root# mount -t ldiskfs /dev/sda /mnt/ost
  94 root# umount /mnt/ost
  95 root# e2fsck -fn /dev/sda   # don't fix file system, just check for corruption
  96 :
  97 [e2fsck output]
  98 :
  99 root# e2fsck -fp /dev/sda   # fix errors with prudent answers (usually <literal>yes</literal>)</screen>
 100   </section>
 101   <section xml:id="dbdoclet.50438225_37365">
 102     <title>
 103     <indexterm>
 104       <primary>recovery</primary>
 105       <secondary>corruption of Lustre file system</secondary>
 106     </indexterm>Recovering from Corruption in the Lustre File System</title>
 107     <para>In cases where an ldiskfs MDT or OST becomes corrupt, you need to run
 108     e2fsck to correct the local filesystem consistency, then use
 109     <literal>LFSCK</literal> to run a distributed check on the file system to
 110     resolve any inconsistencies between the MDTs and OSTs, or among MDTs.</para>
 111     <orderedlist>
 112       <listitem>
 113         <para>Stop the Lustre file system.</para>
 114       </listitem>
 115       <listitem>
 116         <para>Run
 117         <literal>e2fsck -f</literal> on the individual MDT/OST that had
 118         problems to fix any local file system damage.</para>
 119         <para>We recommend running
 120         <literal>e2fsck</literal> under script, to create a log of changes made
 121         to the file system in case it is needed later. After
 122         <literal>e2fsck</literal> is run, bring up the file system, if
 123         necessary, to reduce the outage window.</para>
 124       </listitem>
 125     </orderedlist>
 126     <section xml:id="dbdoclet.50438225_13916">
 127       <title>
 128       <indexterm>
 129         <primary>recovery</primary>
 130         <secondary>orphaned objects</secondary>
 131       </indexterm>Working with Orphaned Objects</title>
 132       <para>The simplest problem to resolve is that of orphaned objects. When
 133       the LFSCK layout check is run, these objects are linked to new files and
 134       put into
 135       <literal>.lustre/lost+found/MDT<replaceable>xxxx</replaceable></literal>
 136       in the Lustre file system
 137       (where MDTxxxx is the index of the MDT on which the orphan was found),
 138       where they can be examined and saved or deleted as necessary.</para>
 139       <para condition='l27'>With Lustre version 2.7 and later, LFSCK will
 140        identify and process orphan objects found on MDTs as well.</para>
 141     </section>
 142   </section>
 143   <section xml:id="dbdoclet.50438225_12316">
 144     <title>
 145     <indexterm>
 146       <primary>recovery</primary>
 147       <secondary>unavailable OST</secondary>
 148     </indexterm>Recovering from an Unavailable OST</title>
 149     <para>One problem encountered in a Lustre file system environment is when
 150     an OST becomes unavailable due to a network partition, OSS node crash, etc.
 151     When this happens, the OST's clients pause and wait for the OST to become
 152     available again, either on the primary OSS or a failover OSS. When the OST
 153     comes back online, the Lustre file system starts a recovery process to
 154     enable clients to reconnect to the OST. Lustre servers put a limit on the
 155     time they will wait in recovery for clients to reconnect.</para>
 156     <para>During recovery, clients reconnect and replay their requests
 157     serially, in the same order they were done originally. Until a client
 158     receives a confirmation that a given transaction has been written to stable
 159     storage, the client holds on to the transaction, in case it needs to be
 160     replayed. Periodically, a progress message prints to the log, stating
 161     how_many/expected clients have reconnected. If the recovery is aborted,
 162     this log shows how many clients managed to reconnect. When all clients have
 163     completed recovery, or if the recovery timeout is reached, the recovery
 164     period ends and the OST resumes normal request processing.</para>
 165     <para>If some clients fail to replay their requests during the recovery
 166     period, this will not stop the recovery from completing. You may have a
 167     situation where the OST recovers, but some clients are not able to
 168     participate in recovery (e.g. network problems or client failure), so they
 169     are evicted and their requests are not replayed. This would result in any
 170     operations on the evicted clients failing, including in-progress writes,
 171     which would cause cached writes to be lost. This is a normal outcome; the
 172     recovery cannot wait indefinitely, or the file system would be hung any
 173     time a client failed. The lost transactions are an unfortunate result of
 174     the recovery process.</para>
 175     <note>
 176       <para>The failure of client recovery does not indicate or lead to
 177       filesystem corruption. This is a normal event that is handled by the MDT
 178       and OST, and should not result in any inconsistencies between
 179       servers.</para>
 180     </note>
 181     <note>
 182       <para>The version-based recovery (VBR) feature enables a failed client to
 183       be ''skipped'', so remaining clients can replay their requests, resulting
 184       in a more successful recovery from a downed OST. For more information
 185       about the VBR feature, see
 186       <xref linkend="lustrerecovery" />(Version-based Recovery).</para>
 187     </note>
 188   </section>
 189   <section xml:id="dbdoclet.lfsckadmin" condition='l23'>
 190     <title>
 191     <indexterm>
 192       <primary>recovery</primary>
 193       <secondary>oiscrub</secondary>
 194     </indexterm>
 195     <indexterm>
 196       <primary>recovery</primary>
 197       <secondary>LFSCK</secondary>
 198     </indexterm>Checking the file system with LFSCK</title>
 199         <para condition='l23'>LFSCK is an administrative tool introduced in Lustre
 200     software release 2.3 for checking and repair of the attributes specific to a
 201     mounted Lustre file system. It is similar in concept to an offline fsck repair
 202     tool for a local filesystem, but LFSCK is implemented to run as part of the
 203     Lustre file system while the file system is mounted and in use. This allows
 204     consistency of checking and repair by the Lustre software without unnecessary
 205     downtime, and can be run on the largest Lustre file systems with negligible
 206     disruption to normal operations.</para>
 207     <para condition='l23'>Since Lustre software release 2.3, LFSCK can verify
 208     and repair the Object Index (OI) table that is used internally to map
 209     Lustre File Identifiers (FIDs) to MDT internal ldiskfs inode numbers, in
 210     an internal table called the OI Table. An OI Scrub traverses this the IO
 211     Table and makes corrections where necessary. An OI Scrub is required after
 212     restoring from a file-level MDT backup (
 213     <xref linkend="dbdoclet.50438207_71633" />), or in case the OI Table is
 214     otherwise corrupted. Later phases of LFSCK will add further checks to the
 215     Lustre distributed file system state.</para>
 216     <para condition='l24'>In Lustre software release 2.4, LFSCK namespace
 217     scanning can verify and repair the directory FID-in-Dirent and LinkEA
 218     consistency.</para>
 219     <para condition='l26'>In Lustre software release 2.6, LFSCK layout scanning
 220     can verify and repair MDT-OST file layout inconsistencies. File layout
 221     inconsistencies between MDT-objects and OST-objects that are checked and
 222     corrected include dangling reference, unreferenced OST-objects, mismatched
 223     references and multiple references.</para>
 224     <para condition='l27'>In Lustre software release 2.7, LFSCK layout scanning
 225     is enhanced to support verify and repair inconsistencies between multiple
 226     MDTs.</para>
 227     <para>Control and monitoring of LFSCK is through LFSCK and the
 228     <literal>/proc</literal> file system interfaces. LFSCK supports three types
 229     of interface: switch interface, status interface, and adjustment interface.
 230     These interfaces are detailed below.</para>
 231     <section>
 232       <title>LFSCK switch interface</title>
 233       <section>
 234         <title>Manually Starting LFSCK</title>
 235         <section>
 236           <title>Description</title>
 237           <para>LFSCK can be started after the MDT is mounted using the
 238           <literal>lctl lfsck_start</literal> command.</para>
 239         </section>
 240         <section>
 241           <title>Usage</title>
 242           <screen>lctl lfsck_start -M | --device
 243 <replaceable>[MDT,OST]_device</replaceable> \
 244                     [-A | --all] \
 245                     [-c | --create_ostobj
 246 <replaceable>[on | off]</replaceable>] \
 247                     [-C | --create_mdtobj
 248 <replaceable>[on | off]</replaceable>] \
 249                     [-e | --error
 250 <replaceable>{continue | abort}</replaceable>] \
 251                     [-h | --help] \
 252                     [-n | --dryrun
 253 <replaceable>[on | off]</replaceable>] \
 254                     [-o | --orphan] \
 255                     [-r | --reset] \
 256                     [-s | --speed
 257 <replaceable>ops_per_sec_limit</replaceable>] \
 258                     [-t | --type
 259 <replaceable>lfsck_type[,lfsck_type...]</replaceable>] \
 260                     [-w | --window_size
 261 <replaceable>size</replaceable>]</screen>
 262         </section>
 263         <section>
 264           <title>Options</title>
 265           <para>The various
 266           <literal>lfsck_start</literal> options are listed and described below.
 267           For a complete list of available options, type
 268           <literal>lctl lfsck_start -h</literal>.</para>
 269           <informaltable frame="all">
 270             <tgroup cols="2">
 271               <colspec colname="c1" colwidth="3*" />
 272               <colspec colname="c2" colwidth="7*" />
 273               <thead>
 274                 <row>
 275                   <entry>
 276                     <para>
 277                       <emphasis role="bold">Option</emphasis>
 278                     </para>
 279                   </entry>
 280                   <entry>
 281                     <para>
 282                       <emphasis role="bold">Description</emphasis>
 283                     </para>
 284                   </entry>
 285                 </row>
 286               </thead>
 287               <tbody>
 288                 <row>
 289                   <entry>
 290                     <para>
 291                       <literal>-M | --device</literal>
 292                     </para>
 293                   </entry>
 294                   <entry>
 295                     <para>The MDT or OST device to start LFSCK on.</para>
 296                   </entry>
 297                 </row>
 298                 <row>
 299                   <entry>
 300                     <para>
 301                       <literal>-A | --all</literal>
 302                     </para>
 303                   </entry>
 304                   <entry>
 305                     <para condition='l26'>Start LFSCK on all devices.
 306                     This applies to both layout and
 307                     namespace consistency checking and repair.</para>
 308                   </entry>
 309                 </row>
 310                 <row>
 311                   <entry>
 312                     <para>
 313                       <literal>-c | --create_ostobj</literal>
 314                     </para>
 315                   </entry>
 316                   <entry>
 317                     <para condition='l26'>Create the lost OST-object for
 318                     dangling LOV EA,
 319                     <literal>off</literal>(default) or
 320                     <literal>on</literal>. If not specified, then the default
 321                     behaviour is to keep the dangling LOV EA there without
 322                     creating the lost OST-object.</para>
 323                   </entry>
 324                 </row>
 325                 <row>
 326                   <entry>
 327                     <para>
 328                       <literal>-C | --create_mdtobj</literal>
 329                     </para>
 330                   </entry>
 331                   <entry>
 332                     <para condition='l27'>Create the lost MDT-object for
 333                     dangling name entry,
 334                     <literal>off</literal>(default) or
 335                     <literal>on</literal>. If not specified, then the default
 336                     behaviour is to keep the dangling name entry there without
 337                     creating the lost MDT-object.</para>
 338                   </entry>
 339                 </row>
 340                 <row>
 341                   <entry>
 342                     <para>
 343                       <literal>-e | --error</literal>
 344                     </para>
 345                   </entry>
 346                   <entry>
 347                     <para>Error handle,
 348                     <literal>continue</literal>(default) or
 349                     <literal>abort</literal>. Specify whether the LFSCK will
 350                     stop or not if fails to repair something. If it is not
 351                     specified, the saved value (when resuming from checkpoint)
 352                     will be used if present. This option cannot be changed
 353                     while LFSCK is running.</para>
 354                   </entry>
 355                 </row>
 356                 <row>
 357                   <entry>
 358                     <para>
 359                       <literal>-h | --help</literal>
 360                     </para>
 361                   </entry>
 362                   <entry>
 363                     <para>Operating help information.</para>
 364                   </entry>
 365                 </row>
 366                 <row>
 367                   <entry>
 368                     <para>
 369                       <literal>-n | --dryrun</literal>
 370                     </para>
 371                   </entry>
 372                   <entry>
 373                     <para>Perform a trial without making any changes.
 374                     <literal>off</literal>(default) or
 375                     <literal>on</literal>.</para>
 376                   </entry>
 377                 </row>
 378                 <row>
 379                   <entry>
 380                     <para>
 381                       <literal>-o | --orphan</literal>
 382                     </para>
 383                   </entry>
 384                   <entry>
 385                     <para condition='l26'>Repair orphan OST-objects for layout
 386                     LFSCK.</para>
 387                   </entry>
 388                 </row>
 389                 <row>
 390                   <entry>
 391                     <para>
 392                       <literal>-r | --reset</literal>
 393                     </para>
 394                   </entry>
 395                   <entry>
 396                     <para>Reset the start position for the object iteration to
 397                     the beginning for the specified MDT. By default the
 398                     iterator will resume scanning from the last checkpoint
 399                     (saved periodically by LFSCK) provided it is
 400                     available.</para>
 401                   </entry>
 402                 </row>
 403                 <row>
 404                   <entry>
 405                     <para>
 406                       <literal>-s | --speed</literal>
 407                     </para>
 408                   </entry>
 409                   <entry>
 410                     <para>Set the upper speed limit of LFSCK processing in
 411                     objects per second. If it is not specified, the saved value
 412                     (when resuming from checkpoint) or default value of 0 (0 =
 413                     run as fast as possible) is used. Speed can be adjusted
 414                     while LFSCK is running with the adjustment
 415                     interface.</para>
 416                   </entry>
 417                 </row>
 418                 <row>
 419                   <entry>
 420                     <para>
 421                       <literal>-t | --type</literal>
 422                     </para>
 423                   </entry>
 424                   <entry>
 425                     <para>The type of checking/repairing that should be
 426                     performed. The new LFSCK framework provides a single
 427                     interface for a variety of system consistency
 428                     checking/repairing operations including:</para>
 429                     <para>Without a specified option, the LFSCK component(s)
 430                     which ran last time and did not finish or the component(s)
 431                     corresponding to some known system inconsistency, will be
 432                     started. Anytime the LFSCK is triggered, the OI scrub will
 433                     run automatically, so there is no need to specify
 434                     OI_scrub in that case.</para>
 435                     <para condition='l24'>
 436                     <literal>namespace</literal>: check and repair
 437                     FID-in-Dirent and LinkEA consistency.</para>
 438                     <para condition='l27'> Lustre-2.7 enhances
 439                     namespace consistency verification under DNE mode.</para>
 440                     <para condition='l26'>
 441                     <literal>layout</literal>: check and repair MDT-OST
 442                     inconsistency.</para>
 443                   </entry>
 444                 </row>
 445                 <row>
 446                   <entry>
 447                     <para>
 448                       <literal>-w | --window_size</literal>
 449                     </para>
 450                   </entry>
 451                   <entry>
 452                     <para condition='l26'>The window size for the async request
 453                     pipeline. The LFSCK async request pipeline's input/output
 454                     may have quite different processing speeds, and there may
 455                     be too many requests in the pipeline as to cause abnormal
 456                     memory/network pressure. If not specified, then the default
 457                     window size for the async request pipeline is 1024.</para>
 458                   </entry>
 459                 </row>
 460               </tbody>
 461             </tgroup>
 462           </informaltable>
 463         </section>
 464       </section>
 465       <section>
 466         <title>Manually Stopping LFSCK</title>
 467         <section>
 468           <title>Description</title>
 469           <para>To stop LFSCK when the MDT is mounted, use the
 470           <literal>lctl lfsck_stop</literal> command.</para>
 471         </section>
 472         <section>
 473           <title>Usage</title>
 474           <screen>lctl lfsck_stop -M | --device
 475 <replaceable>[MDT,OST]_device</replaceable> \
 476                     [-A | --all] \
 477                     [-h | --help]</screen>
 478         </section>
 479         <section>
 480           <title>Options</title>
 481           <para>The various
 482           <literal>lfsck_stop</literal> options are listed and described below.
 483           For a complete list of available options, type
 484           <literal>lctl lfsck_stop -h</literal>.</para>
 485           <informaltable frame="all">
 486             <tgroup cols="2">
 487               <colspec colname="c1" colwidth="3*" />
 488               <colspec colname="c2" colwidth="7*" />
 489               <thead>
 490                 <row>
 491                   <entry>
 492                     <para>
 493                       <emphasis role="bold">Option</emphasis>
 494                     </para>
 495                   </entry>
 496                   <entry>
 497                     <para>
 498                       <emphasis role="bold">Description</emphasis>
 499                     </para>
 500                   </entry>
 501                 </row>
 502               </thead>
 503               <tbody>
 504                 <row>
 505                   <entry>
 506                     <para>
 507                       <literal>-M | --device</literal>
 508                     </para>
 509                   </entry>
 510                   <entry>
 511                     <para>The MDT or OST device to stop LFSCK on.</para>
 512                   </entry>
 513                 </row>
 514                 <row>
 515                   <entry>
 516                     <para>
 517                       <literal>-A | --all</literal>
 518                     </para>
 519                   </entry>
 520                   <entry>
 521                     <para>Stop LFSCK on all devices.</para>
 522                   </entry>
 523                 </row>
 524                 <row>
 525                   <entry>
 526                     <para>
 527                       <literal>-h | --help</literal>
 528                     </para>
 529                   </entry>
 530                   <entry>
 531                     <para>Operating help information.</para>
 532                   </entry>
 533                 </row>
 534               </tbody>
 535             </tgroup>
 536           </informaltable>
 537         </section>
 538       </section>
 539     </section>
 540     <section>
 541       <title>LFSCK status interface</title>
 542       <section>
 543         <title>LFSCK status of OI Scrub via
 544         <literal>procfs</literal></title>
 545         <section>
 546           <title>Description</title>
 547           <para>For each LFSCK component there is a dedicated procfs interface
 548           to trace the corresponding LFSCK component status. For OI Scrub, the
 549           interface is the OSD layer procfs interface, named
 550           <literal>oi_scrub</literal>. To display OI Scrub status, the standard
 551
 552           <literal>lctl get_param</literal> command is used as shown in the
 553           usage below.</para>
 554         </section>
 555         <section>
 556           <title>Usage</title>
 557           <screen>lctl get_param -n osd-ldiskfs.<replaceable>FSNAME</replaceable>-[<replaceable>MDT_device|OST_device</replaceable>].oi_scrub</screen>
 558         </section>
 559         <section>
 560           <title>Output</title>
 561           <informaltable frame="all">
 562             <tgroup cols="2">
 563               <colspec colname="c1" colwidth="3*" />
 564               <colspec colname="c2" colwidth="7*" />
 565               <thead>
 566                 <row>
 567                   <entry>
 568                     <para>
 569                       <emphasis role="bold">Information</emphasis>
 570                     </para>
 571                   </entry>
 572                   <entry>
 573                     <para>
 574                       <emphasis role="bold">Detail</emphasis>
 575                     </para>
 576                   </entry>
 577                 </row>
 578               </thead>
 579               <tbody>
 580                 <row>
 581                   <entry>
 582                     <para>General Information</para>
 583                   </entry>
 584                   <entry>
 585                     <itemizedlist>
 586                       <listitem>
 587                         <para>Name: OI_scrub.</para>
 588                       </listitem>
 589                       <listitem>
 590                         <para>OI scrub magic id (an identifier unique to OI
 591                         scrub).</para>
 592                       </listitem>
 593                       <listitem>
 594                         <para>OI files count.</para>
 595                       </listitem>
 596                       <listitem>
 597                         <para>Status: one of the status -
 598                         <literal>init</literal>,
 599                         <literal>scanning</literal>,
 600                         <literal>completed</literal>,
 601                         <literal>failed</literal>,
 602                         <literal>stopped</literal>,
 603                         <literal>paused</literal>, or
 604                         <literal>crashed</literal>.</para>
 605                       </listitem>
 606                       <listitem>
 607                         <para>Flags: including -
 608                         <literal>recreated</literal>(OI file(s) is/are
 609                         removed/recreated),
 610                         <literal>inconsistent</literal>(restored from
 611                         file-level backup),
 612                         <literal>auto</literal>(triggered by non-UI mechanism),
 613                         and
 614                         <literal>upgrade</literal>(from Lustre software release
 615                         1.8 IGIF format.)</para>
 616                       </listitem>
 617                       <listitem>
 618                         <para>Parameters: OI scrub parameters, like
 619                         <literal>failout</literal>.</para>
 620                       </listitem>
 621                       <listitem>
 622                         <para>Time Since Last Completed.</para>
 623                       </listitem>
 624                       <listitem>
 625                         <para>Time Since Latest Start.</para>
 626                       </listitem>
 627                       <listitem>
 628                         <para>Time Since Last Checkpoint.</para>
 629                       </listitem>
 630                       <listitem>
 631                         <para>Latest Start Position: the position for the
 632                         latest scrub started from.</para>
 633                       </listitem>
 634                       <listitem>
 635                         <para>Last Checkpoint Position.</para>
 636                       </listitem>
 637                       <listitem>
 638                         <para>First Failure Position: the position for the
 639                         first object to be repaired.</para>
 640                       </listitem>
 641                       <listitem>
 642                         <para>Current Position.</para>
 643                       </listitem>
 644                     </itemizedlist>
 645                   </entry>
 646                 </row>
 647                 <row>
 648                   <entry>
 649                     <para>Statistics</para>
 650                   </entry>
 651                   <entry>
 652                     <itemizedlist>
 653                       <listitem>
 654                         <para>
 655                         <literal>Checked</literal> total number of objects
 656                         scanned.</para>
 657                       </listitem>
 658                       <listitem>
 659                         <para>
 660                         <literal>Updated</literal> total number of objects
 661                         repaired.</para>
 662                       </listitem>
 663                       <listitem>
 664                         <para>
 665                         <literal>Failed</literal> total number of objects that
 666                         failed to be repaired.</para>
 667                       </listitem>
 668                       <listitem>
 669                         <para>
 670                         <literal>No Scrub</literal> total number of objects
 671                         marked
 672                         <literal>LDISKFS_STATE_LUSTRE_NOSCRUB and
 673                         skipped</literal>.</para>
 674                       </listitem>
 675                       <listitem>
 676                         <para>
 677                         <literal>IGIF</literal> total number of objects IGIF
 678                         scanned.</para>
 679                       </listitem>
 680                       <listitem>
 681                         <para>
 682                         <literal>Prior Updated</literal> how many objects have
 683                         been repaired which are triggered by parallel
 684                         RPC.</para>
 685                       </listitem>
 686                       <listitem>
 687                         <para>
 688                         <literal>Success Count</literal> total number of
 689                         completed OI_scrub runs on the device.</para>
 690                       </listitem>
 691                       <listitem>
 692                         <para>
 693                         <literal>Run Time</literal> how long the scrub has run,
 694                         tally from the time of scanning from the beginning of
 695                         the specified MDT device, not include the
 696                         paused/failure time among checkpoints.</para>
 697                       </listitem>
 698                       <listitem>
 699                         <para>
 700                         <literal>Average Speed</literal> calculated by dividing
 701                         <literal>Checked</literal> by
 702                         <literal>run_time</literal>.</para>
 703                       </listitem>
 704                       <listitem>
 705                         <para>
 706                         <literal>Real-Time Speed</literal> the speed since last
 707                         checkpoint if the OI_scrub is running.</para>
 708                       </listitem>
 709                       <listitem>
 710                         <para>
 711                         <literal>Scanned</literal> total number of objects under
 712                         /lost+found that have been scanned.</para>
 713                       </listitem>
 714                       <listitem>
 715                         <para>
 716                         <literal>Repaired</literal> total number of objects
 717                         under /lost+found that have been recovered.</para>
 718                       </listitem>
 719                       <listitem>
 720                         <para>
 721                         <literal>Failed</literal> total number of objects under
 722                         /lost+found failed to be scanned or failed to be
 723                         recovered.</para>
 724                       </listitem>
 725                     </itemizedlist>
 726                   </entry>
 727                 </row>
 728               </tbody>
 729             </tgroup>
 730           </informaltable>
 731         </section>
 732       </section>
 733       <section condition='l24'>
 734         <title>LFSCK status of namespace via
 735         <literal>procfs</literal></title>
 736         <section>
 737           <title>Description</title>
 738           <para>The
 739           <literal>namespace</literal> component is responsible for checks
 740           described in <xref linkend="dbdoclet.lfsckadmin" />. The
 741           <literal>procfs</literal> interface for this component is in the
 742           MDD layer, named
 743           <literal>lfsck_namespace</literal>. To show the status of this
 744           component,
 745           <literal>lctl get_param</literal> should be used as described in the
 746           usage below.</para>
 747         </section>
 748         <section>
 749           <title>Usage</title>
 750           <screen>lctl get_param -n mdd. <replaceable>FSNAME</replaceable>-<replaceable>MDT_device</replaceable>.lfsck_namespace</screen>
 751         </section>
 752         <section>
 753           <title>Output</title>
 754           <informaltable frame="all">
 755             <tgroup cols="2">
 756               <colspec colname="c1" colwidth="3*" />
 757               <colspec colname="c2" colwidth="7*" />
 758               <thead>
 759                 <row>
 760                   <entry>
 761                     <para>
 762                       <emphasis role="bold">Information</emphasis>
 763                     </para>
 764                   </entry>
 765                   <entry>
 766                     <para>
 767                       <emphasis role="bold">Detail</emphasis>
 768                     </para>
 769                   </entry>
 770                 </row>
 771               </thead>
 772               <tbody>
 773                 <row>
 774                   <entry>
 775                     <para>General Information</para>
 776                   </entry>
 777                   <entry>
 778                     <itemizedlist>
 779                       <listitem>
 780                         <para>Name:
 781                         <literal>lfsck_namespace</literal></para>
 782                       </listitem>
 783                       <listitem>
 784                         <para>LFSCK namespace magic.</para>
 785                       </listitem>
 786                       <listitem>
 787                         <para>LFSCK namespace version..</para>
 788                       </listitem>
 789                       <listitem>
 790                         <para>Status: one of the status -
 791                         <literal>init</literal>,
 792                         <literal>scanning-phase1</literal>,
 793                         <literal>scanning-phase2</literal>,
 794                         <literal>completed</literal>,
 795                         <literal>failed</literal>,
 796                         <literal>stopped</literal>,
 797                         <literal>paused</literal>,
 798                         <literal>partial</literal>,
 799                         <literal>co-failed</literal>,
 800                         <literal>co-stopped</literal> or
 801                         <literal>co-paused</literal>.</para>
 802                       </listitem>
 803                       <listitem>
 804                         <para>Flags: including -
 805                         <literal>scanned-once</literal>(the first cycle
 806                         scanning has been completed),
 807                         <literal>inconsistent</literal>(one or more
 808                         inconsistent FID-in-Dirent or LinkEA entries that have
 809                         been discovered),
 810                         <literal>upgrade</literal>(from Lustre software release
 811                         1.8 IGIF format.)</para>
 812                       </listitem>
 813                       <listitem>
 814                         <para>Parameters: including
 815                         <literal>dryrun</literal>,
 816                         <literal>all_targets</literal>,
 817                         <literal>failout</literal>,
 818                         <literal>broadcast</literal>,
 819                         <literal>orphan</literal>,
 820                         <literal>create_ostobj</literal> and
 821                         <literal>create_mdtobj</literal>.</para>
 822                       </listitem>
 823                       <listitem>
 824                         <para>Time Since Last Completed.</para>
 825                       </listitem>
 826                       <listitem>
 827                         <para>Time Since Latest Start.</para>
 828                       </listitem>
 829                       <listitem>
 830                         <para>Time Since Last Checkpoint.</para>
 831                       </listitem>
 832                       <listitem>
 833                         <para>Latest Start Position: the position the checking
 834                         began most recently.</para>
 835                       </listitem>
 836                       <listitem>
 837                         <para>Last Checkpoint Position.</para>
 838                       </listitem>
 839                       <listitem>
 840                         <para>First Failure Position: the position for the
 841                         first object to be repaired.</para>
 842                       </listitem>
 843                       <listitem>
 844                         <para>Current Position.</para>
 845                       </listitem>
 846                     </itemizedlist>
 847                   </entry>
 848                 </row>
 849                 <row>
 850                   <entry>
 851                     <para>Statistics</para>
 852                   </entry>
 853                   <entry>
 854                     <itemizedlist>
 855                       <listitem>
 856                         <para>
 857                         <literal>Checked Phase1</literal> total number of
 858                         objects scanned during
 859                         <literal>scanning-phase1</literal>.</para>
 860                       </listitem>
 861                       <listitem>
 862                         <para>
 863                         <literal>Checked Phase2</literal> total number of
 864                         objects scanned during
 865                         <literal>scanning-phase2</literal>.</para>
 866                       </listitem>
 867                       <listitem>
 868                         <para>
 869                         <literal>Updated Phase1</literal> total number of
 870                         objects repaired during
 871                         <literal>scanning-phase1</literal>.</para>
 872                       </listitem>
 873                       <listitem>
 874                         <para>
 875                         <literal>Updated Phase2</literal> total number of
 876                         objects repaired during
 877                         <literal>scanning-phase2</literal>.</para>
 878                       </listitem>
 879                       <listitem>
 880                         <para>
 881                         <literal>Failed Phase1</literal> total number of objets
 882                         that failed to be repaired during
 883                         <literal>scanning-phase1</literal>.</para>
 884                       </listitem>
 885                       <listitem>
 886                         <para>
 887                         <literal>Failed Phase2</literal> total number of objets
 888                         that failed to be repaired during
 889                         <literal>scanning-phase2</literal>.</para>
 890                       </listitem>
 891                       <listitem>
 892                         <para>
 893                         <literal>directories</literal> total number of
 894                         directories scanned.</para>
 895                       </listitem>
 896                       <listitem>
 897                         <para>
 898                         <literal>multiple_linked_checked</literal> total number
 899                         of multiple-linked objects that have been
 900                         scanned.</para>
 901                       </listitem>
 902                       <listitem>
 903                         <para>
 904                         <literal>dirent_repaired</literal> total number of
 905                         FID-in-dirent entries that have been repaired.</para>
 906                       </listitem>
 907                       <listitem>
 908                         <para>
 909                         <literal>linkea_repaired</literal> total number of
 910                         linkEA entries that have been repaired.</para>
 911                       </listitem>
 912                       <listitem>
 913                         <para>
 914                         <literal>unknown_inconsistency</literal> total number of
 915                         undefined inconsistencies found in
 916                         scanning-phase2.</para>
 917                       </listitem>
 918                       <listitem>
 919                         <para>
 920                         <literal>unmatched_pairs_repaired</literal> total number
 921                         of unmatched pairs that have been repaired.</para>
 922                       </listitem>
 923                       <listitem>
 924                         <para>
 925                         <literal>dangling_repaired</literal> total number of
 926                         dangling name entries that have been
 927                         found/repaired.</para>
 928                       </listitem>
 929                       <listitem>
 930                         <para>
 931                         <literal>multi_referenced_repaired</literal> total
 932                         number of multiple referenced name entries that have
 933                         been found/repaired.</para>
 934                       </listitem>
 935                       <listitem>
 936                         <para>
 937                         <literal>bad_file_type_repaired</literal> total number
 938                         of name entries with bad file type that have been
 939                         repaired.</para>
 940                       </listitem>
 941                       <listitem>
 942                         <para>
 943                         <literal>lost_dirent_repaired</literal> total number of
 944                         lost name entries that have been re-inserted.</para>
 945                       </listitem>
 946                       <listitem>
 947                         <para>
 948                         <literal>striped_dirs_scanned</literal> total number of
 949                         striped directories (master) that have been
 950                         scanned.</para>
 951                       </listitem>
 952                       <listitem>
 953                         <para>
 954                         <literal>striped_dirs_repaired</literal> total number of
 955                         striped directories (master) that have been
 956                         repaired.</para>
 957                       </listitem>
 958                       <listitem>
 959                         <para>
 960                         <literal>striped_dirs_failed</literal> total number of
 961                         striped directories (master) that have failed to be
 962                         verified.</para>
 963                       </listitem>
 964                       <listitem>
 965                         <para>
 966                         <literal>striped_dirs_disabled</literal> total number of
 967                         striped directories (master) that have been
 968                         disabled.</para>
 969                       </listitem>
 970                       <listitem>
 971                         <para>
 972                         <literal>striped_dirs_skipped</literal> total number of
 973                         striped directories (master) that have been skipped
 974                         (for shards verification) because of lost master LMV
 975                         EA.</para>
 976                       </listitem>
 977                       <listitem>
 978                         <para>
 979                         <literal>striped_shards_scanned</literal> total number
 980                         of striped directory shards (slave) that have been
 981                         scanned.</para>
 982                       </listitem>
 983                       <listitem>
 984                         <para>
 985                         <literal>striped_shards_repaired</literal> total number
 986                         of striped directory shards (slave) that have been
 987                         repaired.</para>
 988                       </listitem>
 989                       <listitem>
 990                         <para>
 991                         <literal>striped_shards_failed</literal> total number of
 992                         striped directory shards (slave) that have failed to be
 993                         verified.</para>
 994                       </listitem>
 995                       <listitem>
 996                         <para>
 997                         <literal>striped_shards_skipped</literal> total number
 998                         of striped directory shards (slave) that have been
 999                         skipped (for name hash verification) because LFSCK does
1000                         not know whether the slave LMV EA is valid or
1001                         not.</para>
1002                       </listitem>
1003                       <listitem>
1004                         <para>
1005                         <literal>name_hash_repaired</literal> total number of
1006                         name entries under striped directory with bad name hash
1007                         that have been repaired.</para>
1008                       </listitem>
1009                       <listitem>
1010                         <para>
1011                         <literal>nlinks_repaired</literal> total number of
1012                         objects with nlink fixed.</para>
1013                       </listitem>
1014                       <listitem>
1015                         <para>
1016                         <literal>mul_linked_repaired</literal> total number of
1017                         multiple-linked objects that have been repaired.</para>
1018                       </listitem>
1019                       <listitem>
1020                         <para>
1021                         <literal>local_lost_found_scanned</literal> total number
1022                         of objects under /lost+found that have been
1023                         scanned.</para>
1024                       </listitem>
1025                       <listitem>
1026                         <para>
1027                         <literal>local_lost_found_moved</literal> total number
1028                         of objects under /lost+found that have been moved to
1029                         namespace visible directory.</para>
1030                       </listitem>
1031                       <listitem>
1032                         <para>
1033                         <literal>local_lost_found_skipped</literal> total number
1034                         of objects under /lost+found that have been
1035                         skipped.</para>
1036                       </listitem>
1037                       <listitem>
1038                         <para>
1039                         <literal>local_lost_found_failed</literal> total number
1040                         of objects under /lost+found that have failed to be
1041                         processed.</para>
1042                       </listitem>
1043                       <listitem>
1044                         <para>
1045                         <literal>Success Count</literal> the total number of
1046                         completed LFSCK runs on the device.</para>
1047                       </listitem>
1048                       <listitem>
1049                         <para>
1050                         <literal>Run Time Phase1</literal> the duration of the
1051                         LFSCK run during
1052                         <literal>scanning-phase1</literal>. Excluding the time
1053                         spent paused between checkpoints.</para>
1054                       </listitem>
1055                       <listitem>
1056                         <para>
1057                         <literal>Run Time Phase2</literal> the duration of the
1058                         LFSCK run during
1059                         <literal>scanning-phase2</literal>. Excluding the time
1060                         spent paused between checkpoints.</para>
1061                       </listitem>
1062                       <listitem>
1063                         <para>
1064                         <literal>Average Speed Phase1</literal> calculated by
1065                         dividing
1066                         <literal>checked_phase1</literal> by
1067                         <literal>run_time_phase1</literal>.</para>
1068                       </listitem>
1069                       <listitem>
1070                         <para>
1071                         <literal>Average Speed Phase2</literal> calculated by
1072                         dividing
1073                         <literal>checked_phase2</literal> by
1074                         <literal>run_time_phase1</literal>.</para>
1075                       </listitem>
1076                       <listitem>
1077                         <para>
1078                         <literal>Real-Time Speed Phase1</literal> the speed
1079                         since the last checkpoint if the LFSCK is running
1080                         <literal>scanning-phase1</literal>.</para>
1081                       </listitem>
1082                       <listitem>
1083                         <para>
1084                         <literal>Real-Time Speed Phase2</literal> the speed
1085                         since the last checkpoint if the LFSCK is running
1086                         <literal>scanning-phase2</literal>.</para>
1087                       </listitem>
1088                     </itemizedlist>
1089                   </entry>
1090                 </row>
1091               </tbody>
1092             </tgroup>
1093           </informaltable>
1094         </section>
1095       </section>
1096       <section condition='l26'>
1097         <title>LFSCK status of layout via
1098         <literal>procfs</literal></title>
1099         <section>
1100           <title>Description</title>
1101           <para>The
1102           <literal>layout</literal> component is responsible for checking and
1103           repairing MDT-OST inconsistency. The
1104           <literal>procfs</literal> interface for this component is in the MDD
1105           layer, named
1106           <literal>lfsck_layout</literal>, and in the OBD layer, named
1107           <literal>lfsck_layout</literal>. To show the status of this component
1108           <literal>lctl get_param</literal> should be used as described in the
1109           usage below.</para>
1110         </section>
1111         <section>
1112           <title>Usage</title>
1113           <screen>lctl get_param -n mdd.
1114 <replaceable>FSNAME</replaceable>-
1115 <replaceable>MDT_device</replaceable>.lfsck_layout
1116 lctl get_param -n obdfilter.
1117 <replaceable>FSNAME</replaceable>-
1118 <replaceable>OST_device</replaceable>.lfsck_layout</screen>
1119         </section>
1120         <section>
1121           <title>Output</title>
1122           <informaltable frame="all">
1123             <tgroup cols="2">
1124               <colspec colname="c1" colwidth="3*" />
1125               <colspec colname="c2" colwidth="7*" />
1126               <thead>
1127                 <row>
1128                   <entry>
1129                     <para>
1130                       <emphasis role="bold">Information</emphasis>
1131                     </para>
1132                   </entry>
1133                   <entry>
1134                     <para>
1135                       <emphasis role="bold">Detail</emphasis>
1136                     </para>
1137                   </entry>
1138                 </row>
1139               </thead>
1140               <tbody>
1141                 <row>
1142                   <entry>
1143                     <para>General Information</para>
1144                   </entry>
1145                   <entry>
1146                     <itemizedlist>
1147                       <listitem>
1148                         <para>Name:
1149                         <literal>lfsck_layout</literal></para>
1150                       </listitem>
1151                       <listitem>
1152                         <para>LFSCK namespace magic.</para>
1153                       </listitem>
1154                       <listitem>
1155                         <para>LFSCK namespace version..</para>
1156                       </listitem>
1157                       <listitem>
1158                         <para>Status: one of the status -
1159                         <literal>init</literal>,
1160                         <literal>scanning-phase1</literal>,
1161                         <literal>scanning-phase2</literal>,
1162                         <literal>completed</literal>,
1163                         <literal>failed</literal>,
1164                         <literal>stopped</literal>,
1165                         <literal>paused</literal>,
1166                         <literal>crashed</literal>,
1167                         <literal>partial</literal>,
1168                         <literal>co-failed</literal>,
1169                         <literal>co-stopped</literal>, or
1170                         <literal>co-paused</literal>.</para>
1171                       </listitem>
1172                       <listitem>
1173                         <para>Flags: including -
1174                         <literal>scanned-once</literal>(the first cycle
1175                         scanning has been completed),
1176                         <literal>inconsistent</literal>(one or more MDT-OST
1177                         inconsistencies have been discovered),
1178                         <literal>incomplete</literal>(some MDT or OST did not
1179                         participate in the LFSCK or failed to finish the LFSCK)
1180                         or
1181                         <literal>crashed_lastid</literal>(the lastid files on
1182                         the OST crashed and needs to be rebuilt).</para>
1183                       </listitem>
1184                       <listitem>
1185                         <para>Parameters: including
1186                         <literal>dryrun</literal>,
1187                         <literal>all_targets</literal> and
1188                         <literal>failout</literal>.</para>
1189                       </listitem>
1190                       <listitem>
1191                         <para>Time Since Last Completed.</para>
1192                       </listitem>
1193                       <listitem>
1194                         <para>Time Since Latest Start.</para>
1195                       </listitem>
1196                       <listitem>
1197                         <para>Time Since Last Checkpoint.</para>
1198                       </listitem>
1199                       <listitem>
1200                         <para>Latest Start Position: the position the checking
1201                         began most recently.</para>
1202                       </listitem>
1203                       <listitem>
1204                         <para>Last Checkpoint Position.</para>
1205                       </listitem>
1206                       <listitem>
1207                         <para>First Failure Position: the position for the
1208                         first object to be repaired.</para>
1209                       </listitem>
1210                       <listitem>
1211                         <para>Current Position.</para>
1212                       </listitem>
1213                     </itemizedlist>
1214                   </entry>
1215                 </row>
1216                 <row>
1217                   <entry>
1218                     <para>Statistics</para>
1219                   </entry>
1220                   <entry>
1221                     <itemizedlist>
1222                       <listitem>
1223                         <para>
1224                         <literal>Success Count:</literal> the total number of
1225                         completed LFSCK runs on the device.</para>
1226                       </listitem>
1227                       <listitem>
1228                         <para>
1229                         <literal>Repaired Dangling:</literal> total number of
1230                         MDT-objects with dangling reference have been repaired
1231                         in the scanning-phase1.</para>
1232                       </listitem>
1233                       <listitem>
1234                         <para>
1235                         <literal>Repaired Unmatched Pairs</literal> total number
1236                         of unmatched MDT and OST-object paris have been
1237                         repaired in the scanning-phase1</para>
1238                       </listitem>
1239                       <listitem>
1240                         <para>
1241                         <literal>Repaired Multiple Referenced</literal> total
1242                         number of OST-objects with multiple reference have been
1243                         repaired in the scanning-phase1.</para>
1244                       </listitem>
1245                       <listitem>
1246                         <para>
1247                         <literal>Repaired Orphan</literal> total number of
1248                         orphan OST-objects have been repaired in the
1249                         scanning-phase2.</para>
1250                       </listitem>
1251                       <listitem>
1252                         <para>
1253                         <literal>Repaired Inconsistent Owner</literal> total
1254                         number.of OST-objects with incorrect owner information
1255                         have been repaired in the scanning-phase1.</para>
1256                       </listitem>
1257                       <listitem>
1258                         <para>
1259                         <literal>Repaired Others</literal> total number of.other
1260                         inconsistency repaired in the scanning phases.</para>
1261                       </listitem>
1262                       <listitem>
1263                         <para>
1264                         <literal>Skipped</literal> Number of skipped
1265                         objects.</para>
1266                       </listitem>
1267                       <listitem>
1268                         <para>
1269                         <literal>Failed Phase1</literal> total number of objects
1270                         that failed to be repaired during
1271                         <literal>scanning-phase1</literal>.</para>
1272                       </listitem>
1273                       <listitem>
1274                         <para>
1275                         <literal>Failed Phase2</literal> total number of objects
1276                         that failed to be repaired during
1277                         <literal>scanning-phase2</literal>.</para>
1278                       </listitem>
1279                       <listitem>
1280                         <para>
1281                         <literal>Checked Phase1</literal> total number of
1282                         objects scanned during
1283                         <literal>scanning-phase1</literal>.</para>
1284                       </listitem>
1285                       <listitem>
1286                         <para>
1287                         <literal>Checked Phase2</literal> total number of
1288                         objects scanned during
1289                         <literal>scanning-phase2</literal>.</para>
1290                       </listitem>
1291                       <listitem>
1292                         <para>
1293                         <literal>Run Time Phase1</literal> the duration of the
1294                         LFSCK run during
1295                         <literal>scanning-phase1</literal>. Excluding the time
1296                         spent paused between checkpoints.</para>
1297                       </listitem>
1298                       <listitem>
1299                         <para>
1300                         <literal>Run Time Phase2</literal> the duration of the
1301                         LFSCK run during
1302                         <literal>scanning-phase2</literal>. Excluding the time
1303                         spent paused between checkpoints.</para>
1304                       </listitem>
1305                       <listitem>
1306                         <para>
1307                         <literal>Average Speed Phase1</literal> calculated by
1308                         dividing
1309                         <literal>checked_phase1</literal> by
1310                         <literal>run_time_phase1</literal>.</para>
1311                       </listitem>
1312                       <listitem>
1313                         <para>
1314                         <literal>Average Speed Phase2</literal> calculated by
1315                         dividing
1316                         <literal>checked_phase2</literal> by
1317                         <literal>run_time_phase1</literal>.</para>
1318                       </listitem>
1319                       <listitem>
1320                         <para>
1321                         <literal>Real-Time Speed Phase1</literal> the speed
1322                         since the last checkpoint if the LFSCK is running
1323                         <literal>scanning-phase1</literal>.</para>
1324                       </listitem>
1325                       <listitem>
1326                         <para>
1327                         <literal>Real-Time Speed Phase2</literal> the speed
1328                         since the last checkpoint if the LFSCK is running
1329                         <literal>scanning-phase2</literal>.</para>
1330                       </listitem>
1331                     </itemizedlist>
1332                   </entry>
1333                 </row>
1334               </tbody>
1335             </tgroup>
1336           </informaltable>
1337         </section>
1338       </section>
1339     </section>
1340     <section>
1341       <title>LFSCK adjustment interface</title>
1342       <section condition='l26'>
1343         <title>Rate control</title>
1344         <section>
1345           <title>Description</title>
1346           <para>The LFSCK upper speed limit can be changed using
1347           <literal>lctl set_param</literal> as shown in the usage below.</para>
1348         </section>
1349         <section>
1350           <title>Usage</title>
1351           <screen>lctl set_param mdd.${FSNAME}-${MDT_device}.lfsck_speed_limit=
1352 <replaceable>N</replaceable>
1353 lctl set_param obdfilter.${FSNAME}-${OST_device}.lfsck_speed_limit=
1354 <replaceable>N</replaceable></screen>
1355         </section>
1356         <section>
1357           <title>Values</title>
1358           <informaltable frame="all">
1359             <tgroup cols="2">
1360               <colspec colname="c1" colwidth="3*" />
1361               <colspec colname="c2" colwidth="7*" />
1362               <tbody>
1363                 <row>
1364                   <entry>
1365                     <para>0</para>
1366                   </entry>
1367                   <entry>
1368                     <para>No speed limit (run at maximum speed.)</para>
1369                   </entry>
1370                 </row>
1371                 <row>
1372                   <entry>
1373                     <para>positive integer</para>
1374                   </entry>
1375                   <entry>
1376                     <para>Maximum number of objects to scan per second.</para>
1377                   </entry>
1378                 </row>
1379               </tbody>
1380             </tgroup>
1381           </informaltable>
1382         </section>
1383       </section>
1384       <section xml:id="dbdoclet.lfsck_auto_scrub">
1385         <title>Auto scrub</title>
1386         <section>
1387           <title>Description</title>
1388           <para>The
1389           <literal>auto_scrub</literal> parameter controls whether OI scrub will
1390           be triggered when an inconsistency is detected during OI lookup. It
1391           can be set as described in the usage and values sections
1392           below.</para>
1393           <para>There is also a
1394           <literal>noscrub</literal> mount option (see
1395           <xref linkend="dbdoclet.50438219_12635" />) which can be used to
1396           disable automatic OI scrub upon detection of a file-level backup at
1397           mount time. If the
1398           <literal>noscrub</literal> mount option is specified,
1399           <literal>auto_scrub</literal> will also be disabled, so OI scrub will
1400           not be triggered when an OI inconsistency is detected. Auto scrub can
1401           be renabled after the mount using the command shown in the usage.
1402           Manually starting LFSCK after mounting provides finer control over
1403           the starting conditions.</para>
1404         </section>
1405         <section>
1406           <title>Usage</title>
1407           <screen>lctl set_param osd_ldiskfs.${FSNAME}-${MDT_device}.auto_scrub=<replaceable>N</replaceable></screen>
1408           <para>where
1409           <replaceable>N</replaceable>is an integer as described below.</para>
1410           <note condition='l25'><para>Lustre software 2.5 and later supports
1411           <literal>-P</literal> option that makes the
1412           <literal>set_param</literal> permanent.</para></note>
1413         </section>
1414         <section>
1415           <title>Values</title>
1416           <informaltable frame="all">
1417             <tgroup cols="2">
1418               <colspec colname="c1" colwidth="3*" />
1419               <colspec colname="c2" colwidth="7*" />
1420               <tbody>
1421                 <row>
1422                   <entry>
1423                     <para>0</para>
1424                   </entry>
1425                   <entry>
1426                     <para>Do not start OI Scrub automatically.</para>
1427                   </entry>
1428                 </row>
1429                 <row>
1430                   <entry>
1431                     <para>positive integer</para>
1432                   </entry>
1433                   <entry>
1434                     <para>Automatically start OI Scrub if inconsistency is
1435                     detected during OI lookup.</para>
1436                   </entry>
1437                 </row>
1438               </tbody>
1439             </tgroup>
1440           </informaltable>
1441         </section>
1442       </section>
1443     </section>
1444   </section>
1445 </chapter>