LustreDebugging.xml

   1 <?xml version='1.0' encoding='UTF-8'?>
   2 <chapter xmlns="http://docbook.org/ns/docbook"
   3  xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"
   4  xml:id="lustredebugging">
   5   <title xml:id="lustredebugging.title">Debugging a Lustre File System</title>
   6   <para>This chapter describes tips and information to debug a Lustre file system, and includes the
   7     following sections:</para>
   8   <itemizedlist>
   9     <listitem>
  10       <para><xref linkend="debugging_tools"/></para>
  11     </listitem>
  12     <listitem>
  13       <para><xref linkend="debugging_procedures"/></para>
  14     </listitem>
  15     <listitem>
  16       <para><xref linkend="debugging_for_devs"/></para>
  17     </listitem>
  18   </itemizedlist>
  19   <section xml:id="debugging_tools">
  20     <title><indexterm><primary>debugging</primary></indexterm>
  21 Diagnostic and Debugging Tools</title>
  22     <para>A variety of diagnostic and analysis tools are available to debug
  23       issues with the Lustre software. Some of these are provided in Linux
  24       distributions, while others have been developed and are made available
  25       by the Lustre project.</para>
  26     <section remap="h3" xml:id="section_dms_q21_kk">
  27       <title><indexterm>
  28           <primary>debugging</primary>
  29           <secondary>tools</secondary>
  30         </indexterm> Lustre Debugging Tools</title>
  31       <para>The following in-kernel debug mechanisms are incorporated into
  32         the Lustre software:</para>
  33       <itemizedlist>
  34         <listitem>
  35           <para xml:id="para_fkj_rld_hk"><emphasis role="bold">Debug logs</emphasis>
  36             - A circular debug buffer to which Lustre internal debug messages
  37             are written (in contrast to error messages, which are printed to the
  38             syslog or console). Entries in the Lustre debug log are controlled
  39             by a mask set by <literal>lctl set_param debug=<replaceable>mask</replaceable></literal>.
  40             The log size defaults to 5 MB per CPU but can be increased as a
  41             busy system will quickly overwrite 5 MB. When the buffer fills,
  42             the oldest log records are discarded.</para>
  43         </listitem>
  44         <listitem>
  45           <para><emphasis role="bold">
  46               <literal>lctl get_param debug</literal>
  47             </emphasis> - This shows the current debug mask used to delimit
  48             the debugging information written out to the kernel debug logs.
  49           </para>
  50         </listitem>
  51         <listitem>
  52           <para><emphasis role="bold">
  53               <literal>lctl debug_kernel <replaceable>file</replaceable></literal>
  54             </emphasis> - Dump the Lustre kernel debug log to the specified
  55             file as ASCII text for further debugging and analysis.
  56           </para>
  57         </listitem>
  58         <listitem>
  59           <para><emphasis role="bold">
  60               <literal>lctl set_param debug_mb=<replaceable>size</replaceable></literal>
  61             </emphasis> - This sets the maximum size of the in-kernel Lustre
  62             debug buffer, in units of MiB.
  63           </para>
  64         </listitem>
  65         <listitem>
  66           <para><emphasis role="bold">Debug daemon</emphasis>
  67             - The debug daemon controls the continuous logging of debug
  68             messages to a log file in userspace.</para>
  69         </listitem>
  70       </itemizedlist>
  71       <para>The following tools are also provided with the Lustre software:</para>
  72       <itemizedlist>
  73         <listitem>
  74           <para><emphasis role="bold">
  75               <literal>lctl</literal>
  76             </emphasis> - This tool is used with the debug_kernel option to
  77             manually dump the Lustre debugging log or post-process debugging
  78             logs that are dumped automatically. For more information about the
  79             lctl tool, see <xref linkend="using_lctl_tool"/> and <xref
  80               linkend="lctl"/>.</para>
  81         </listitem>
  82         <listitem>
  83           <para>Lustre subsystem asserts - A panic-style assertion (LBUG) in the kernel causes the
  84             Lustre file system to dump the debug log to the file
  85                 <literal>/tmp/lustre-log.<replaceable>timestamp</replaceable></literal> where it can
  86             be retrieved after a reboot. For more information, see <xref
  87               linkend="troubleshooting"/>.</para>
  88         </listitem>
  89         <listitem>
  90           <para>
  91             <literal>
  92               <replaceable>lfs</replaceable>
  93             </literal> - This utility provides access to the layout of
  94             of a Lustre file, along with other information relevant to users.
  95             For more information about lfs, see <xref
  96               linkend="userutilities.lfs"/>.</para>
  97         </listitem>
  98       </itemizedlist>
  99     </section>
 100     <section remap="h3">
 101       <title><indexterm><primary>debugging</primary><secondary>external tools</secondary></indexterm>External Debugging Tools</title>
 102       <para>The tools described in this section are provided in the Linux
 103         kernel or are available at an external website. For information about
 104         using some of these tools for Lustre debugging, see
 105         <xref linkend="debugging_procedures"/> and
 106         <xref linkend="debugging_for_devs"/>.
 107       </para>
 108       <section remap="h4">
 109         <title><indexterm><primary>debugging</primary><secondary>admin tools</secondary></indexterm>Tools for Administrators and Developers</title>
 110         <para>Some general debugging tools provided as a part of the standard Linux distribution
 111           are:</para>
 112         <itemizedlist>
 113           <listitem>
 114             <para><emphasis role="bold">
 115                 <literal>strace</literal>
 116               </emphasis> . This tool allows a system call to be traced.</para>
 117           </listitem>
 118           <listitem>
 119             <para><emphasis role="bold">
 120                 <literal>/var/log/messages</literal>
 121               </emphasis> . <literal>syslogd</literal> prints fatal or serious messages at this log.</para>
 122           </listitem>
 123           <listitem>
 124             <para><emphasis role="bold">Crash dumps</emphasis> . On crash-dump enabled kernels,
 125               sysrq c produces a crash dump. The Lustre software enhances this crash dump with a log
 126               dump (the last 64 KB of the log) to the console.</para>
 127           </listitem>
 128           <listitem>
 129             <para><emphasis role="bold">
 130                 <literal>debugfs</literal>
 131               </emphasis>. Interactive file system debugger.</para>
 132           </listitem>
 133         </itemizedlist>
 134         <para>The following logging and data collection tools can be used to collect information for debugging Lustre kernel issues:</para>
 135         <itemizedlist>
 136           <listitem>
 137             <para><emphasis role="bold"><literal>kdump</literal></emphasis>.
 138             A Linux kernel crash utility useful for debugging a system running
 139             Red Hat Enterprise Linux. For more information about
 140             <literal>kdump</literal>, see the Red Hat knowledge base article
 141             <link xl:href="https://access.redhat.com/solutions/6038">How to
 142             troubleshoot kernel crashes, hangs, or reboots with kdump on
 143             Red Hat Enterprise Linux</link>. To download
 144             <literal>kdump</literal>, install the RPM package via
 145             <literal>yum install kexec-tools</literal>.
 146             </para>
 147           </listitem>
 148           <listitem>
 149             <para><emphasis role="bold"><literal>netconsole</literal></emphasis>.
 150             Enables kernel-level network logging over UDP. A system requires
 151             (SysRq) allows users to collect relevant data through
 152             <literal>netconsole</literal>.</para>
 153           </listitem>
 154           <listitem>
 155             <para><emphasis><literal>wireshark</literal> </emphasis>. A network
 156             packet inspection tool that allows debugging of information that was
 157             sent between the various Lustre nodes. This tool is built on top of
 158             <literal>tcpdump</literal> and can read packet dumps generated by
 159             it.  There are plug-ins available to dissassemble the LNet and
 160             Lustre protocols.  They are included with
 161             <literal>wireshark</literal> since version 2.6.0.  See also the
 162             <link xl:href="https://www.wireshark.org/">Wireshark Website</link>
 163             for more details.</para>
 164           </listitem>
 165         </itemizedlist>
 166       </section>
 167       <section remap="h4">
 168         <title><indexterm><primary>debugging</primary><secondary>developer tools</secondary></indexterm>Tools for Developers</title>
 169         <para>The tools described in this section may be useful for debugging a Lustre file system
 170           in a development environment.</para>
 171         <para>Of general interest is:</para>
 172         <itemizedlist>
 173           <listitem>
 174             <para><literal>
 175                 <replaceable>leak_finder.pl</replaceable>
 176               </literal> . This program provided with the Lustre software is useful for finding
 177               memory leaks in the code.</para>
 178           </listitem>
 179         </itemizedlist>
 180         <para>A virtual machine is often used to create an isolated development and test environment. Some commonly-used virtual machines are:</para>
 181         <itemizedlist>
 182           <listitem>
 183             <para>
 184             <emphasis role="bold">VirtualBox Open Source Edition</emphasis>.
 185             Provides enterprise-class virtualization capability for all major
 186             platforms and is available free at
 187             <link xl:href="https://www.virtualbox.org/wiki/Downloads">
 188             https://www.virtualbox.org/wiki/Downloads</link>.</para>
 189           </listitem>
 190           <listitem>
 191             <para><emphasis role="bold">VMware Server</emphasis>.
 192             Virtualization platform available as free introductory software at
 193             <link xl:href="https://my.vmware.com/web/vmware/downloads/">
 194             https://my.vmware.com/web/vmware/downloads/</link>.</para>
 195           </listitem>
 196           <listitem>
 197             <para><emphasis role="bold">Xen</emphasis>. A para-virtualized
 198             environment with virtualization capabilities similar to VMware
 199             Server and Virtual Box. However, Xen allows the use of modified
 200             kernels to provide near-native performance and the ability to
 201             emulate shared storage. For more information, go to
 202             <link xl:href="https://xen.org/">https://xen.org/</link>.</para>
 203           </listitem>
 204         </itemizedlist>
 205         <para>A variety of debuggers and analysis tools are available including:</para>
 206         <itemizedlist>
 207           <listitem>
 208             <para><emphasis role="bold">
 209                 <literal>kgdb</literal>
 210               </emphasis> . The Linux Kernel Source Level Debugger kgdb is
 211               used in conjunction with the GNU Debugger <literal>gdb</literal>
 212               for debugging the Linux kernel. For more information about using
 213               <literal>kgdb</literal> with <literal>gdb</literal>, see
 214               <link xl:href="https://www.linuxtopia.org/online_books/redhat_linux_debugging_with_gdb/running.html">
 215               Chapter 6. Running Programs Under gdb</link> in the
 216               <emphasis>Red Hat Linux 4 Debugging with GDB</emphasis> guide.
 217             </para>
 218           </listitem>
 219           <listitem>
 220             <para><emphasis role="bold">
 221                 <literal>crash</literal>
 222               </emphasis> . Used to analyze saved crash dump data when a system had panicked or locked up or appears unresponsive. For more information about using crash to analyze a crash dump, see:</para>
 223             <itemizedlist>
 224               <listitem>
 225                 <para> Overview on how to use crash by the author:
 226                 <link xl:href="https://crash-utility.github.io/crash_whitepaper.html">
 227                 White Paper: Red Hat Crash Utility</link>
 228                 </para>
 229               </listitem>
 230             </itemizedlist>
 231           </listitem>
 232         </itemizedlist>
 233       </section>
 234     </section>
 235   </section>
 236   <section xml:id="debugging_procedures">
 237     <title><indexterm><primary>debugging</primary><secondary>procedure</secondary></indexterm>Lustre Debugging Procedures</title>
 238     <para>The procedures below may be useful to administrators or developers
 239       debugging a Lustre files system.</para>
 240     <section remap="h3">
 241       <title><indexterm><primary>debugging</primary><secondary>message format</secondary></indexterm>Understanding the Lustre Debug Messaging Format</title>
 242       <para>Lustre debug messages are categorized by originating subsystem,
 243         message type, and location in the source code. For a list of subsystems
 244         and message types, see <xref linkend="lctl"/>.
 245       </para>
 246       <note>
 247         <para>For a current list of subsystems and debug message types, see
 248             <literal>libcfs/include/libcfs/libcfs_debug.h</literal> in the Lustre software
 249           tree</para>
 250       </note>
 251       <para>The elements of a Lustre debug message are described in <xref linkend="lustre_msg_format"/> Format of Lustre Debug Messages.</para>
 252       <section xml:id="lustre_debug_msg">
 253         <title>Lustre Debug Messages</title>
 254         <para>Each Lustre debug message has the tag of the subsystem it originated in, the message
 255           type, and the location in the source code. The subsystems and debug types used are as
 256           follows:</para>
 257         <itemizedlist>
 258           <listitem>
 259             <para>  Standard Subsystems:</para>
 260             <para> mdc, mds, osc, ost, obdclass, obdfilter, llite, ptlrpc, portals, lnd, ldlm, lov</para>
 261           </listitem>
 262         </itemizedlist>
 263         <itemizedlist>
 264           <listitem>
 265             <para>  Debug Types:</para>
 266           </listitem>
 267           <listitem>
 268             <para><informaltable frame="all">
 269                 <tgroup cols="2">
 270                   <colspec colname="c1" colwidth="50*"/>
 271                   <colspec colname="c2" colwidth="50*"/>
 272                   <thead>
 273                     <row>
 274                       <entry>
 275                         <para><emphasis role="bold">Types</emphasis></para>
 276                       </entry>
 277                       <entry>
 278                         <para><emphasis role="bold">Description</emphasis></para>
 279                       </entry>
 280                     </row>
 281                   </thead>
 282                   <tbody>
 283                     <row>
 284                       <entry>
 285                         <para> <emphasis role="bold">trace</emphasis></para>
 286                       </entry>
 287                       <entry>
 288                         <para> Function entry/exit markers</para>
 289                       </entry>
 290                     </row>
 291                     <row>
 292                       <entry>
 293                         <para> <emphasis role="bold">dlmtrace</emphasis></para>
 294                       </entry>
 295                       <entry>
 296                         <para> Distributed locking-related information</para>
 297                       </entry>
 298                     </row>
 299                     <row>
 300                       <entry>
 301                         <para> <emphasis role="bold">inode</emphasis></para>
 302                       </entry>
 303                       <entry>
 304                         <para> &#160;</para>
 305                       </entry>
 306                     </row>
 307                     <row>
 308                       <entry>
 309                         <para> <emphasis role="bold">super</emphasis></para>
 310                       </entry>
 311                       <entry>
 312                         <para> &#160;</para>
 313                       </entry>
 314                     </row>
 315                     <row>
 316                       <entry>
 317                         <para> <emphasis role="bold">malloc</emphasis></para>
 318                       </entry>
 319                       <entry>
 320                         <para> Memory allocation or free information</para>
 321                       </entry>
 322                     </row>
 323                     <row>
 324                       <entry>
 325                         <para> <emphasis role="bold">cache</emphasis></para>
 326                       </entry>
 327                       <entry>
 328                         <para> Cache-related information</para>
 329                       </entry>
 330                     </row>
 331                     <row>
 332                       <entry>
 333                         <para> <emphasis role="bold">info</emphasis></para>
 334                       </entry>
 335                       <entry>
 336                         <para> Non-critical general information</para>
 337                       </entry>
 338                     </row>
 339                     <row>
 340                       <entry>
 341                         <para> <emphasis role="bold">dentry</emphasis></para>
 342                       </entry>
 343                       <entry>
 344                         <para> kernel namespace cache handling</para>
 345                       </entry>
 346                     </row>
 347                     <row>
 348                       <entry>
 349                         <para> <emphasis role="bold">mmap</emphasis></para>
 350                       </entry>
 351                       <entry>
 352                         <para> Memory-mapped IO interface</para>
 353                       </entry>
 354                     </row>
 355                     <row>
 356                       <entry>
 357                         <para> <emphasis role="bold">page</emphasis></para>
 358                       </entry>
 359                       <entry>
 360                         <para> Page cache and bulk data transfers</para>
 361                       </entry>
 362                     </row>
 363                     <row>
 364                       <entry>
 365                         <para> <emphasis role="bold">info</emphasis></para>
 366                       </entry>
 367                       <entry>
 368                         <para> Miscellaneous informational messages</para>
 369                       </entry>
 370                     </row>
 371                     <row>
 372                       <entry>
 373                         <para> <emphasis role="bold">net</emphasis></para>
 374                       </entry>
 375                       <entry>
 376                         <para> LNet network related debugging</para>
 377                       </entry>
 378                     </row>
 379                     <row>
 380                       <entry>
 381                         <para> <emphasis role="bold">console</emphasis></para>
 382                       </entry>
 383                       <entry>
 384                         <para>Significant system events, printed to console</para>
 385                       </entry>
 386                     </row>
 387                     <row>
 388                       <entry>
 389                         <para> <emphasis role="bold">warning</emphasis></para>
 390                       </entry>
 391                       <entry>
 392                         <para>Significant but non-fatal exceptions, printed
 393                           to console</para>
 394                       </entry>
 395                     </row>
 396                     <row>
 397                       <entry>
 398                         <para> <emphasis role="bold">error</emphasis></para>
 399                       </entry>
 400                       <entry>
 401                         <para>Critical error messages, printed to console</para>
 402                       </entry>
 403                     </row>
 404                     <row>
 405                       <entry>
 406                         <para> <emphasis role="bold">neterror</emphasis></para>
 407                       </entry>
 408                       <entry>
 409                         <para>Significant LNet error messages</para>
 410                       </entry>
 411                     </row>
 412                     <row>
 413                       <entry>
 414                         <para> <emphasis role="bold">emerg</emphasis></para>
 415                       </entry>
 416                       <entry>
 417                         <para>Fatal system errors, printed to console</para>
 418                       </entry>
 419                     </row>
 420                     <row>
 421                       <entry>
 422                         <para><emphasis role="bold">config</emphasis></para>
 423                       </entry>
 424                       <entry>
 425                         <para>Configuration and setup, enabled by default</para>
 426                       </entry>
 427                     </row>
 428                     <row>
 429                       <entry>
 430                         <para><emphasis role="bold">ha</emphasis></para>
 431                       </entry>
 432                       <entry>
 433                         <para>Failover and recovery-related information,
 434                         enabled by default</para>
 435                       </entry>
 436                     </row>
 437                     <row>
 438                       <entry>
 439                         <para><emphasis role="bold">hsm</emphasis></para>
 440                       </entry>
 441                       <entry>
 442                         <para>Hierarchical space management/tiering</para>
 443                       </entry>
 444                     </row>
 445                     <row>
 446                       <entry>
 447                         <para><emphasis role="bold">ioctl</emphasis></para>
 448                       </entry>
 449                       <entry>
 450                         <para>IOCTL-related information, enabled by default</para>
 451                       </entry>
 452                     </row>
 453                     <row>
 454                       <entry>
 455                         <para><emphasis role="bold">layout</emphasis></para>
 456                       </entry>
 457                       <entry>
 458                         <para>File layout handling (PFL, FLR, DoM)</para>
 459                       </entry>
 460                     </row>
 461                     <row>
 462                       <entry>
 463                         <para><emphasis role="bold">lfsck</emphasis></para>
 464                       </entry>
 465                       <entry>
 466                         <para>Filesystem consistency checking, enabled by
 467                         default</para>
 468                       </entry>
 469                     </row>
 470                     <row>
 471                       <entry>
 472                         <para><emphasis role="bold">other</emphasis></para>
 473                       </entry>
 474                       <entry>
 475                         <para>Miscellaneious other debug messages</para>
 476                       </entry>
 477                     </row>
 478                     <row>
 479                       <entry>
 480                         <para><emphasis role="bold">quota</emphasis></para>
 481                       </entry>
 482                       <entry>
 483                         <para>Space accounting and management</para>
 484                       </entry>
 485                     </row>
 486                     <row>
 487                       <entry>
 488                         <para><emphasis role="bold">reada</emphasis></para>
 489                       </entry>
 490                       <entry>
 491                         <para>Client readahead management</para>
 492                       </entry>
 493                     </row>
 494                     <row>
 495                       <entry>
 496                         <para> <emphasis role="bold">rpctrace</emphasis></para>
 497                       </entry>
 498                       <entry>
 499                         <para>Remote request/reply tracing and debugging</para>
 500                       </entry>
 501                     </row>
 502                     <row>
 503                       <entry>
 504                         <para><emphasis role="bold">sec</emphasis></para>
 505                       </entry>
 506                       <entry>
 507                         <para>Security, Kerberos, Shared Secret Key handling</para>
 508                       </entry>
 509                     </row>
 510                     <row>
 511                       <entry>
 512                         <para><emphasis role="bold">snapshot</emphasis></para>
 513                       </entry>
 514                       <entry>
 515                         <para>Filesystem snapshot management</para>
 516                       </entry>
 517                     </row>
 518                     <row>
 519                       <entry>
 520                         <para><emphasis role="bold">vfstrace</emphasis></para>
 521                       </entry>
 522                       <entry>
 523                         <para>Kernel VFS interface operations</para>
 524                       </entry>
 525                     </row>
 526                   </tbody>
 527                 </tgroup>
 528               </informaltable>
 529 </para>
 530           </listitem>
 531         </itemizedlist>
 532       </section>
 533       <section xml:id="lustre_msg_format">
 534         <title>Format of Lustre Debug Messages</title>
 535         <para>The Lustre software uses the <literal>CDEBUG()</literal> and
 536             <literal>CERROR()</literal> macros to print the debug or error messages. To print the
 537           message, the <literal>CDEBUG()</literal> macro uses the function
 538             <literal>libcfs_debug_msg()</literal> (<literal>libcfs/libcfs/tracefile.c</literal>).
 539           The message format is described below, along with an example.</para>
 540         <informaltable frame="all">
 541           <tgroup cols="2">
 542             <colspec colname="c1" colwidth="50*"/>
 543             <colspec colname="c2" colwidth="50*"/>
 544             <thead>
 545               <row>
 546                 <entry>
 547                   <para><emphasis role="bold">Description</emphasis></para>
 548                 </entry>
 549                 <entry>
 550                   <para><emphasis role="bold">Parameter</emphasis></para>
 551                 </entry>
 552               </row>
 553             </thead>
 554             <tbody>
 555               <row>
 556                 <entry>
 557                   <para> <emphasis role="bold">subsystem</emphasis></para>
 558                 </entry>
 559                 <entry>
 560                   <para> 800000</para>
 561                 </entry>
 562               </row>
 563               <row>
 564                 <entry>
 565                   <para> <emphasis role="bold">debug mask</emphasis></para>
 566                 </entry>
 567                 <entry>
 568                   <para> 000010</para>
 569                 </entry>
 570               </row>
 571               <row>
 572                 <entry>
 573                   <para> <emphasis role="bold">smp_processor_id</emphasis></para>
 574                 </entry>
 575                 <entry>
 576                   <para> 0</para>
 577                 </entry>
 578               </row>
 579               <row>
 580                 <entry>
 581                   <para> <emphasis role="bold">seconds.microseconds</emphasis></para>
 582                 </entry>
 583                 <entry>
 584                   <para> 1081880847.677302</para>
 585                 </entry>
 586               </row>
 587               <row>
 588                 <entry>
 589                   <para> <emphasis role="bold">stack size</emphasis></para>
 590                 </entry>
 591                 <entry>
 592                   <para> 1204</para>
 593                 </entry>
 594               </row>
 595               <row>
 596                 <entry>
 597                   <para> <emphasis role="bold">pid</emphasis></para>
 598                 </entry>
 599                 <entry>
 600                   <para> 2973</para>
 601                 </entry>
 602               </row>
 603               <row>
 604                 <entry>
 605                   <para> <emphasis role="bold">host pid (UML only) or zero</emphasis></para>
 606                 </entry>
 607                 <entry>
 608                   <para> 31070</para>
 609                 </entry>
 610               </row>
 611               <row>
 612                 <entry>
 613                   <para> <emphasis role="bold">(file:line #:function_name())</emphasis></para>
 614                 </entry>
 615                 <entry>
 616                   <para> (obd_mount.c:2089:lustre_fill_super())</para>
 617                 </entry>
 618               </row>
 619               <row>
 620                 <entry>
 621                   <para> <emphasis role="bold">debug message</emphasis></para>
 622                 </entry>
 623                 <entry>
 624                   <para> kmalloced &apos;*obj&apos;: 24 at a375571c (tot 17447717)</para>
 625                 </entry>
 626               </row>
 627             </tbody>
 628           </tgroup>
 629         </informaltable>
 630       </section>
 631       <section remap="h4">
 632         <title>Lustre Debug Messages Buffer</title>
 633         <para>Lustre debug messages are maintained in a buffer, with the maximum buffer size specified (in MBs) by the <literal>debug_mb</literal> parameter (<literal>lctl get_param debug_mb</literal>). The buffer is circular, so debug messages are kept until the allocated buffer limit is reached, and then the first messages are overwritten.</para>
 634       </section>
 635     </section>
 636     <section xml:id="using_lctl_tool">
 637       <title><indexterm><primary>debugging</primary><secondary>using lctl</secondary></indexterm>Using the lctl Tool to View Debug Messages</title>
 638       <para>The <literal>lctl</literal> tool allows debug messages to be
 639         filtered based on subsystems and message types to extract information
 640         useful for troubleshooting from a kernel debug log. For a command
 641         reference, see <xref linkend="lctl"/>.</para>
 642       <para>You can use <literal>lctl</literal> to:</para>
 643       <itemizedlist>
 644         <listitem>
 645           <para>Obtain a list of all the types and subsystems:</para>
 646           <screen>lctl &gt; debug_list <replaceable>subsystems|types</replaceable></screen>
 647         </listitem>
 648       </itemizedlist>
 649       <itemizedlist>
 650         <listitem>
 651           <para>Filter the debug log:</para>
 652           <screen>lctl &gt; filter <replaceable>subsystem_name|debug_type</replaceable></screen>
 653         </listitem>
 654       </itemizedlist>
 655       <note>
 656         <para>When <literal>lctl</literal> filters, it removes unwanted lines from the displayed output. This does not affect the contents of the debug log in the kernel&apos;s memory. As a result, you can print the log many times with different filtering levels without worrying about losing data.</para>
 657       </note>
 658       <itemizedlist>
 659         <listitem>
 660           <para>Show debug messages belonging to certain subsystem or type:</para>
 661           <screen>lctl &gt; show <replaceable>subsystem_name|debug_type</replaceable></screen>
 662           <para><literal>debug_kernel</literal> pulls the data from the kernel logs, filters it appropriately, and displays or saves it as per the specified options</para>
 663           <screen>lctl &gt; debug_kernel [<replaceable>output filename</replaceable>]</screen>
 664           <para>If the debugging is being done on User Mode Linux (UML), it might be useful to save the logs on the host machine so that they can be used at a later time.</para>
 665         </listitem>
 666       </itemizedlist>
 667       <itemizedlist>
 668         <listitem>
 669           <para>Filter a log on disk, if you already have a debug log saved to disk (likely from a crash):</para>
 670           <screen>lctl &gt; debug_file <replaceable>input_file</replaceable> [<replaceable>output_file</replaceable>] </screen>
 671           <para>During the debug session, you can add markers or breaks to the log for any reason:</para>
 672           <screen>lctl &gt; mark [marker text] </screen>
 673           <para>The marker text defaults to the current date and time in the debug log (similar to the example shown below):</para>
 674           <screen>DEBUG MARKER: Tue Mar 5 16:06:44 EST 2002
 675 </screen>
 676         </listitem>
 677       </itemizedlist>
 678       <itemizedlist>
 679         <listitem>
 680           <para>Completely flush the kernel debug buffer:</para>
 681           <screen>lctl &gt; clear
 682 </screen>
 683         </listitem>
 684       </itemizedlist>
 685       <note>
 686         <para>Debug messages displayed with <literal>lctl</literal> are also subject to the kernel debug masks; the filters are additive.</para>
 687       </note>
 688       <section remap="h4">
 689         <title><indexterm><primary>debugging</primary><secondary>lctl example</secondary></indexterm>Sample <literal>lctl</literal> Run</title>
 690         <para>Below is a sample run using the <literal>lctl</literal> command.</para>
 691         <screen>bash-2.04# ./lctl
 692 lctl &gt; debug_kernel /tmp/lustre_logs/log_all
 693 Debug log: 324 lines, 324 kept, 0 dropped.
 694 lctl &gt; filter trace
 695 Disabling output of type &quot;trace&quot;
 696 lctl &gt; debug_kernel /tmp/lustre_logs/log_notrace
 697 Debug log: 324 lines, 282 kept, 42 dropped.
 698 lctl &gt; show trace
 699 Enabling output of type &quot;trace&quot;
 700 lctl &gt; filter portals
 701 Disabling output from subsystem &quot;portals&quot;
 702 lctl &gt; debug_kernel /tmp/lustre_logs/log_noportals
 703 Debug log: 324 lines, 258 kept, 66 dropped.
 704 </screen>
 705       </section>
 706     </section>
 707     <section remap="h3">
 708       <title>Dumping the Buffer to a File (<literal>debug_daemon</literal>)</title>
 709       <para>The <literal>lctl debug_daemon</literal> command is used to continuously dump the <literal>debug_kernel</literal> buffer to a user-specified file. This functionality uses a kernel thread to continuously dump the messages from the kernel debug log, so that much larger debug logs can be saved over a longer time than would fit in the kernel ringbuffer.</para>
 710       <para>The <literal>debug_daemon</literal> is highly dependent on file system write speed. File system write operations may not be fast enough to flush out all of the <literal>debug_buffer</literal> if the Lustre file system is under heavy system load and continues to log debug messages to the <literal>debug_buffer</literal>. The <literal>debug_daemon</literal> will write the message <literal>DEBUG MARKER: Trace buffer full</literal> into the <literal>debug_buffer</literal> to indicate the <literal>debug_buffer</literal> contents are overlapping before the <literal>debug_daemon</literal> flushes data to a file.</para>
 711       <para>Users can use the <literal>lctl debug_daemon</literal> command to start or stop the Lustre daemon from dumping the <literal>debug_buffer</literal> to a file.</para>
 712       <section remap="h4">
 713         <title><literal>lctl debug_daemon</literal> Commands</title>
 714         <para>To initiate the <literal>debug_daemon</literal> to start dumping the <literal>debug_buffer</literal> into a file, run as the root user:</para>
 715         <screen>lctl debug_daemon start <replaceable>filename</replaceable> [<replaceable>megabytes</replaceable>]</screen>
 716         <para>The debug log will be written to the specified filename from the kernel.  The file will be limited to the optionally specified number of megabytes.</para>
 717         <para>The daemon wraps around and dumps data to the beginning of the file when the output file size is over the limit of the user-specified file size. To decode the dumped file to ASCII and sort the log entries by time, run:</para>
 718         <screen>lctl debug_file <replaceable>filename</replaceable> &gt; <replaceable>newfile</replaceable></screen>
 719         <para>The output is internally sorted by the <literal>lctl</literal> command.</para>
 720         <para>To stop the <literal>debug_daemon</literal> operation and flush the file output, run:</para>
 721         <screen>lctl debug_daemon stop</screen>
 722         <para>Otherwise, <literal>debug_daemon</literal> is shut down as part of the Lustre file system shutdown process. Users can restart <literal>debug_daemon</literal> by using start command after each stop command issued.</para>
 723         <para>This is an example using <literal>debug_daemon</literal> with the interactive mode of <literal>lctl</literal> to dump debug logs to a 40 MB file.</para>
 724         <screen>lctl</screen>
 725         <screen>lctl &gt; debug_daemon start /var/log/lustre.40.bin 40 </screen>
 726         <screen><replaceable>run filesystem operations to debug</replaceable></screen>
 727         <screen>lctl &gt; debug_daemon stop </screen>
 728         <screen>lctl &gt; debug_file /var/log/lustre.bin /var/log/lustre.log</screen>
 729         <para>To start another daemon with an unlimited file size, run:</para>
 730         <screen>lctl &gt; debug_daemon start /var/log/lustre.bin </screen>
 731         <para>The text message <literal>*** End of debug_daemon trace log ***</literal> appears at the end of each output file.</para>
 732       </section>
 733     </section>
 734     <section remap="h3">
 735       <title><indexterm><primary>debugging</primary><secondary>kernel debug log</secondary></indexterm>Controlling Information Written to the Kernel Debug Log</title>
 736       <para>The <literal>lctl set_param subsystem_debug=<replaceable>subsystem_mask</replaceable></literal> and <literal>lctl set_param debug=<replaceable>debug_mask</replaceable></literal> are used to determine which information is written to the debug log. The subsystem_debug mask determines the information written to the log based on the functional area of the code (such as lnet, osc, or ldlm). The debug mask controls information based on the message type (such as info, error, trace, or malloc). For a complete list of possible debug masks use the <literal>lctl debug_list types</literal> command.</para>
 737       <para>To turn off Lustre debugging completely:</para>
 738       <screen>lctl set_param debug=0 </screen>
 739       <para>To turn on full Lustre debugging:</para>
 740       <screen>lctl set_param debug=-1 </screen>
 741       <para>To list all possible debug masks:</para>
 742       <screen>lctl debug_list types</screen>
 743       <para>To log only messages related to network communications:</para>
 744       <screen>lctl set_param debug=net </screen>
 745       <para>To turn on logging of messages related to network communications and existing debug flags:</para>
 746       <screen>lctl set_param debug=+net </screen>
 747       <para>To turn off network logging with changing existing flags:</para>
 748       <screen>lctl set_param debug=-net </screen>
 749       <para>The various options available to print to kernel debug logs are listed in <literal>libcfs/include/libcfs/libcfs.h</literal></para>
 750     </section>
 751     <section remap="h3">
 752       <title><indexterm><primary>debugging</primary><secondary>using strace</secondary></indexterm>Troubleshooting with <literal>strace</literal></title>
 753       <para>The <literal>strace</literal> utility provided with the Linux distribution enables system calls to be traced by intercepting all the system calls made by a process and recording the system call name, arguments, and return values.</para>
 754       <para>To invoke <literal>strace</literal> on a program, enter:</para>
 755       <screen>$ strace <replaceable>program</replaceable> <replaceable>[arguments]</replaceable> </screen>
 756       <para>Sometimes, a system call may fork child processes. In this situation, use the <literal>-f</literal> option of <literal>strace</literal> to trace the child processes:</para>
 757       <screen>$ strace -f <replaceable>program</replaceable> <replaceable>[arguments]</replaceable> </screen>
 758       <para>To redirect the <literal>strace</literal> output to a file, enter:</para>
 759       <screen>$ strace -o <replaceable>filename</replaceable> <replaceable>program</replaceable> <replaceable>[arguments]</replaceable> </screen>
 760       <para>Use the <literal>-ff</literal> option, along with <literal>-o</literal>, to save the trace output in <literal>filename.pid</literal>, where <literal>pid</literal> is the process ID of the process being traced. Use the <literal>-ttt</literal> option to timestamp all lines in the strace output, so they can be correlated to operations in the lustre kernel debug log.</para>
 761     </section>
 762     <section remap="h3">
 763       <title><indexterm><primary>debugging</primary><secondary>disk contents</secondary></indexterm>Looking at Disk Content</title>
 764       <para>In a Lustre file system, the inodes on the metadata server contain extended attributes
 765         (EAs) that store information about file striping. EAs contain a list of all object IDs and
 766         their locations (that is, the OST that stores them). The <literal>lfs</literal> tool can be
 767         used to obtain this information for a given file using the <literal>getstripe</literal>
 768         subcommand. Use a corresponding <literal>lfs setstripe</literal> command to specify striping
 769         attributes for a new file or directory.</para>
 770       <para>The <literal>lfs getstripe</literal> command takes a Lustre filename as input and lists
 771         all the objects that form a part of this file. To obtain this information for the file
 772           <literal>/mnt/testfs/frog</literal> in a Lustre file system, run:</para>
 773       <screen>$ lfs getstripe /mnt/testfs/frog
 774 lmm_stripe_count:   2
 775 lmm_stripe_size:    1048576
 776 lmm_pattern:        1
 777 lmm_layout_gen:     0
 778 lmm_stripe_offset:  2
 779         obdidx           objid          objid           group
 780              2          818855        0xc7ea7               0
 781              0          873123        0xd52a3               0
 782         </screen>
 783       <para>The <literal>debugfs</literal> tool is provided in the
 784           <literal>e2fsprogs</literal> package. It can be used for interactive
 785           debugging of an <literal>ldiskfs</literal> file system. The
 786           <literal>debugfs</literal> tool can either be used to check status or
 787           modify information in the file system. In a Lustre file system, all
 788           objects that belong to a file are stored in an underlying
 789           <literal>ldiskfs</literal> file system on the OSTs. The file system
 790           uses the object IDs as the file names. Once the object IDs are known,
 791           use the <literal>debugfs</literal> tool to obtain the attributes of
 792           all objects from different OSTs.</para>
 793       <para>A sample run for the <literal>/mnt/testfs/frog</literal> file used
 794           in the above example is shown here:</para>
 795       <screen>$ debugfs -c -R "stat O/0/d$((818855 % 32))/818855" /dev/vgmyth/lvmythost2
 796
 797 debugfs 1.41.90.wc3 (28-May-2011)
 798 /dev/vgmyth/lvmythost2: catastrophic mode - not reading inode or group bitmaps
 799 Inode: 227649   Type: regular    Mode:  0666   Flags: 0x80000
 800 Generation: 1375019198    Version: 0x0000002f:0000728f
 801 User:  1000   Group:  1000   Size: 2800
 802 File ACL: 0    Directory ACL: 0
 803 Links: 1   Blockcount: 8
 804 Fragment:  Address: 0    Number: 0    Size: 0
 805  ctime: 0x4e177fe5:00000000 -- Fri Jul  8 16:08:37 2011
 806  atime: 0x4d2e2397:00000000 -- Wed Jan 12 14:56:39 2011
 807  mtime: 0x4e177fe5:00000000 -- Fri Jul  8 16:08:37 2011
 808 crtime: 0x4c3b5820:a364117c -- Mon Jul 12 12:00:00 2010
 809 Size of extra inode fields: 28
 810 Extended attributes stored in inode body:
 811   fid = "08 80 24 00 00 00 00 00 28 8a e7 fc 00 00 00 00 a7 7e 0c 00 00 00 00 00
 812  00 00 00 00 00 00 00 00 " (32)
 813   fid: objid=818855 seq=0 parent=[0x248008:0xfce78a28:0x0] stripe=0
 814 EXTENTS:
 815 (0):63331288
 816       </screen>
 817     </section>
 818     <section remap="h3">
 819       <title>Finding the Lustre UUID of an OST</title>
 820       <para>To determine the Lustre UUID of an OST disk (for example, if you mix up the cables on your OST devices or the SCSI bus numbering suddenly changes and the SCSI devices get new names), it is possible to extract this from the last_rcvd file using debugfs:</para>
 821       <screen>debugfs -c -R "dump last_rcvd /tmp/last_rcvd" /dev/sdc
 822 strings /tmp/last_rcvd | head -1
 823 myth-OST0004_UUID
 824       </screen>
 825       <para>It is also possible (and easier) to extract this from the file system label using the
 826           <literal>dumpe2fs</literal> command:</para>
 827       <screen>dumpe2fs -h /dev/sdc | grep volume
 828 dumpe2fs 1.41.90.wc3 (28-May-2011)
 829 Filesystem volume name:   myth-OST0004
 830       </screen>
 831       <para>The debugfs and dumpe2fs commands are well documented in the <literal>debugfs(8)</literal> and <literal>dumpe2fs(8)</literal> manual pages.</para>
 832     </section>
 833     <section remap="h3">
 834       <title>Printing Debug Messages to the Console</title>
 835       <para>To dump debug messages to the console (<literal>/var/log/messages</literal>), set the corresponding debug mask in the <literal>printk</literal> flag:</para>
 836       <screen>lctl set_param printk=-1 </screen>
 837       <para>This slows down the system dramatically. It is also possible to selectively enable or disable this capability for particular flags using:<literal>lctl set_param printk=+vfstrace</literal> and <literal>lctl set_param printk=-vfstrace </literal>.</para>
 838       <para>It is possible to disable warning, error, and console messages, though it is strongly recommended to have something like <literal>lctl debug_daemon</literal> running to capture this data to a local file system for failure detection purposes.</para>
 839     </section>
 840     <section remap="h3">
 841       <title>Tracing Lock Traffic</title>
 842       <para>The Lustre software provides a specific debug type category for tracing lock traffic.
 843         Use:</para>
 844       <screen>lctl&gt; filter all_types
 845 lctl&gt; show dlmtrace
 846 lctl&gt; debug_kernel [<replaceable>filename</replaceable>] </screen>
 847     </section>
 848     <section remap="h3">
 849       <title>Controlling Console Message Rate Limiting</title>
 850       <para>Some console messages which are printed by Lustre are rate limited.  When such messages are printed, they may be followed by a message saying "Skipped N previous similar message(s)," where N is the number of messages skipped.  This rate limiting can be completely disabled by a libcfs module parameter called <literal>libcfs_console_ratelimit</literal>.  To disable console message rate limiting, add this line to <literal>/etc/modprobe.d/lustre.conf</literal> and then reload Lustre modules.</para>
 851       <screen>options libcfs libcfs_console_ratelimit=0</screen>
 852       <para>It is also possible to set the minimum and maximum delays between rate-limited console messages using the module parameters <literal>libcfs_console_max_delay</literal> and <literal>libcfs_console_min_delay</literal>.  Set these in <literal>/etc/modprobe.d/lustre.conf</literal> and then reload Lustre modules.  Additional information on libcfs module parameters is available via <literal>modinfo</literal>:</para>
 853       <screen>modinfo libcfs</screen>
 854     </section>
 855   </section>
 856   <section xml:id="debugging_for_devs">
 857     <title><indexterm><primary>debugging</primary><secondary>developers tools</secondary></indexterm>Lustre Debugging for Developers</title>
 858     <para>The procedures in this section may be useful to developers debugging Lustre source
 859       code.</para>
 860     <section remap="h3">
 861       <title>Adding Debugging to the Lustre Source Code</title>
 862       <para>The debugging infrastructure provides a number of macros that can be used in Lustre source code to aid in debugging or reporting serious errors.</para>
 863       <para>To use these macros, you will need to set the <literal>DEBUG_SUBSYSTEM</literal> variable at the top of the file as shown below:</para>
 864       <screen>#define DEBUG_SUBSYSTEM S_PORTALS</screen>
 865       <para>A list of available macros with descriptions is provided in the table below.</para>
 866       <informaltable frame="all">
 867         <tgroup cols="2">
 868           <colspec colname="c1" colwidth="50*"/>
 869           <colspec colname="c2" colwidth="50*"/>
 870           <thead>
 871             <row>
 872               <entry>
 873                 <para><emphasis role="bold">Macro</emphasis></para>
 874               </entry>
 875               <entry>
 876                 <para><emphasis role="bold">Description</emphasis></para>
 877               </entry>
 878             </row>
 879           </thead>
 880           <tbody>
 881             <row>
 882               <entry>
 883                 <para> <emphasis role="bold">
 884                     <literal>LBUG()</literal>
 885                   </emphasis></para>
 886               </entry>
 887               <entry>
 888                 <para>A panic-style assertion in the kernel which causes the Lustre file system to
 889                   dump its circular log to the <literal>/tmp/lustre-log</literal> file. This file
 890                   can be retrieved after a reboot. <literal>LBUG()</literal> freezes the thread to
 891                   allow capture of the panic stack. A system reboot is needed to clear the
 892                   thread.</para>
 893               </entry>
 894             </row>
 895             <row>
 896               <entry>
 897                 <para> <emphasis role="bold">
 898                     <literal>LASSERT()</literal>
 899                   </emphasis></para>
 900               </entry>
 901               <entry>
 902                 <para>Validates a given expression as true, otherwise calls LBUG(). The failed expression is printed on the console, although the values that make up the expression are not printed.</para>
 903               </entry>
 904             </row>
 905             <row>
 906               <entry>
 907                 <para> <emphasis role="bold">
 908                     <literal>LASSERTF()</literal>
 909                   </emphasis></para>
 910               </entry>
 911               <entry>
 912                 <para>Similar to <literal>LASSERT()</literal> but allows a free-format message to be printed, like <literal>printf/printk</literal>.</para>
 913               </entry>
 914             </row>
 915             <row>
 916               <entry>
 917                 <para> <emphasis role="bold">
 918                     <literal>CDEBUG()</literal>
 919                   </emphasis></para>
 920               </entry>
 921               <entry>
 922                 <para>The basic, most commonly used debug macro that takes just one more argument than standard <literal>printf()</literal> - the debug type. This message adds to the debug log with the debug mask set accordingly. Later, when a user retrieves the log for troubleshooting, they can filter based on this type.</para>
 923                 <para><literal>CDEBUG(D_INFO, &quot;debug message: rc=%d\n&quot;, number);</literal></para>
 924               </entry>
 925             </row>
 926             <row>
 927               <entry>
 928                 <para> <emphasis role="bold">
 929                     <literal>CDEBUG_LIMIT()</literal>
 930                   </emphasis></para>
 931               </entry>
 932               <entry>
 933                 <para> Behaves similarly to <literal>CDEBUG()</literal>, but rate limits this message when printing to the console (for <literal>D_WARN</literal>, <literal>D_ERROR</literal>, and <literal>D_CONSOLE</literal> message types. This is useful for messages that use a variable debug mask:</para>
 934                 <para><literal>CDEBUG(mask, &quot;maybe bad: rc=%d\n&quot;, rc);</literal></para>
 935               </entry>
 936             </row>
 937             <row>
 938               <entry>
 939                 <para> <emphasis role="bold">
 940                     <literal>CERROR()</literal>
 941                   </emphasis></para>
 942               </entry>
 943               <entry>
 944                 <para>Internally using <literal>CDEBUG_LIMIT(D_ERROR, ...)</literal>, which unconditionally prints the message in the debug log and to the console. This is appropriate for serious errors or fatal conditions.  Messages printed to the console are prefixed with <literal>LustreError:</literal>, and are rate-limited, to avoid flooding the console with duplicates.</para>
 945                 <para><literal>CERROR(&quot;Something bad happened: rc=%d\n&quot;, rc);</literal></para>
 946               </entry>
 947             </row>
 948             <row>
 949               <entry>
 950                 <para> <emphasis role="bold">
 951                     <literal>CWARN()</literal>
 952                   </emphasis></para>
 953               </entry>
 954               <entry>
 955                 <para> Behaves similarly to <literal>CERROR()</literal>, but prefixes the messages with <literal>Lustre:</literal>. This is appropriate for important, but not fatal conditions.  Messages printed to the console are rate-limited.</para>
 956               </entry>
 957             </row>
 958             <row>
 959               <entry>
 960                 <para> <emphasis role="bold">
 961                     <literal>CNETERR()</literal>
 962                   </emphasis></para>
 963               </entry>
 964               <entry>
 965                 <para> Behaves similarly to <literal>CERROR()</literal>, but prints error messages for LNet if <literal>D_NETERR</literal> is set in the <literal>debug</literal> mask. This is appropriate for serious networking errors. Messages printed to the console are rate-limited.</para>
 966               </entry>
 967             </row>
 968             <row>
 969               <entry>
 970                 <para> <emphasis role="bold">
 971                     <literal>DEBUG_REQ()</literal>
 972                   </emphasis></para>
 973               </entry>
 974               <entry>
 975                 <para>Prints information about the given <literal>ptlrpc_request</literal> structure.</para>
 976                 <para><literal>DEBUG_REQ(D_RPCTRACE, req, &quot;"Handled RPC: rc=%d\n&quot;, rc);</literal></para>
 977               </entry>
 978             </row>
 979             <row>
 980               <entry>
 981                 <para> <emphasis role="bold">
 982                     <literal>ENTRY</literal>
 983                   </emphasis></para>
 984               </entry>
 985               <entry>
 986                 <para> Add messages to the entry of a function to aid in call tracing (takes no arguments). When using these macros, cover all exit conditions with a single <literal>EXIT</literal>, <literal>GOTO()</literal>, or <literal>RETURN()</literal> macro to avoid confusion when the debug log reports that a function was entered, but never exited.</para>
 987               </entry>
 988             </row>
 989             <row>
 990               <entry>
 991                 <para> <emphasis role="bold">
 992                     <literal>EXIT</literal>
 993                   </emphasis></para>
 994               </entry>
 995               <entry>
 996                 <para> Mark the exit of a function, to match <literal>ENTRY</literal> (takes no arguments).</para>
 997               </entry>
 998             </row>
 999             <row>
1000               <entry>
1001                 <para> <emphasis role="bold">
1002                     <literal>GOTO()</literal>
1003                   </emphasis></para>
1004               </entry>
1005               <entry>
1006                 <para> Mark when code jumps via <literal>goto</literal> to the end of a function, to match <literal>ENTRY</literal>, and prints out the goto label and function return code in signed and unsigned decimal, and hexadecimal format.</para>
1007               </entry>
1008             </row>
1009             <row>
1010               <entry>
1011                 <para> <emphasis role="bold">
1012                     <literal>RETURN()</literal>
1013                   </emphasis></para>
1014               </entry>
1015               <entry>
1016                 <para> Mark the exit of a function, to match <literal>ENTRY</literal>, and prints out the function return code in signed and unsigned decimal, and hexadecimal format.</para>
1017               </entry>
1018             </row>
1019             <row>
1020               <entry>
1021                 <para> <emphasis role="bold">
1022                     <literal>LDLM_DEBUG()</literal>
1023                   </emphasis></para>
1024                 <para> <emphasis role="bold">
1025                     <literal>LDLM_DEBUG_NOLOCK()</literal>
1026                   </emphasis></para>
1027               </entry>
1028               <entry>
1029                 <para>Used when tracing LDLM locking operations. These macros build a thin trace that shows the locking requests on a node, and can also be linked across the client and server node using the printed lock handles.</para>
1030               </entry>
1031             </row>
1032             <row>
1033               <entry>
1034                 <para> <emphasis role="bold">
1035                     <literal>OBD_FAIL_CHECK()</literal>
1036                   </emphasis></para>
1037               </entry>
1038               <entry>
1039                 <para>Allows insertion of failure points into the Lustre source code. This is useful
1040                   to generate regression tests that can hit a very specific sequence of events. This
1041                   works in conjunction with &quot;<literal>lctl set_param
1042                       fail_loc=<replaceable>fail_loc</replaceable></literal>&quot; to set a specific
1043                   failure point for which a given <literal>OBD_FAIL_CHECK()</literal> will
1044                   test.</para>
1045               </entry>
1046             </row>
1047             <row>
1048               <entry>
1049                 <para> <emphasis role="bold">
1050                     <literal>OBD_FAIL_TIMEOUT()</literal>
1051                   </emphasis></para>
1052               </entry>
1053               <entry>
1054                 <para>Similar to <literal>OBD_FAIL_CHECK()</literal>. Useful to simulate hung, blocked or busy processes or network devices. If the given <literal>fail_loc</literal> is hit, <literal>OBD_FAIL_TIMEOUT()</literal> waits for the specified number of seconds.</para>
1055               </entry>
1056             </row>
1057             <row>
1058               <entry>
1059                 <para> <emphasis role="bold">
1060                     <literal>OBD_RACE()</literal>
1061                   </emphasis></para>
1062               </entry>
1063               <entry>
1064                 <para>Similar to <literal>OBD_FAIL_CHECK()</literal>. Useful to have multiple processes execute the same code concurrently to provoke locking races. The first process to hit <literal>OBD_RACE()</literal> sleeps until a second process hits <literal>OBD_RACE()</literal>, then both processes continue.</para>
1065               </entry>
1066             </row>
1067             <row>
1068               <entry>
1069                 <para> <emphasis role="bold">
1070                     <literal>OBD_FAIL_ONCE</literal>
1071                   </emphasis></para>
1072               </entry>
1073               <entry>
1074                 <para>A flag set on a <literal>fail_loc</literal> breakpoint to cause the <literal>OBD_FAIL_CHECK()</literal> condition to be hit only one time. Otherwise, a <literal>fail_loc</literal> is permanent until it is cleared with &quot;<literal>lctl set_param fail_loc=0</literal>&quot;.</para>
1075               </entry>
1076             </row>
1077             <row>
1078               <entry>
1079                 <para> <emphasis role="bold">
1080                     <literal>OBD_FAIL_RAND</literal>
1081                   </emphasis></para>
1082               </entry>
1083               <entry>
1084                 <para>A flag set on a <literal>fail_loc</literal> breakpoint to cause <literal>OBD_FAIL_CHECK()</literal> to fail randomly; on average every (1 / fail_val) times.</para>
1085               </entry>
1086             </row>
1087             <row>
1088               <entry>
1089                 <para> <emphasis role="bold">
1090                     <literal>OBD_FAIL_SKIP</literal>
1091                   </emphasis></para>
1092               </entry>
1093               <entry>
1094                 <para>A flag set on a <literal>fail_loc</literal> breakpoint to cause <literal>OBD_FAIL_CHECK()</literal> to succeed <literal>fail_val</literal> times, and then fail permanently or once with <literal>OBD_FAIL_ONCE</literal>.</para>
1095               </entry>
1096             </row>
1097             <row>
1098               <entry>
1099                 <para> <emphasis role="bold">
1100                     <literal>OBD_FAIL_SOME</literal>
1101                   </emphasis></para>
1102               </entry>
1103               <entry>
1104                 <para>A flag set on <literal>fail_loc</literal> breakpoint to cause <literal>OBD_FAIL_CHECK</literal> to fail <literal>fail_val</literal> times, and then succeed.</para>
1105               </entry>
1106             </row>
1107           </tbody>
1108         </tgroup>
1109       </informaltable>
1110     </section>
1111     <section remap="h3">
1112       <title>Accessing the <literal>ptlrpc</literal> Request History</title>
1113       <para>Each service maintains a request history, which can be useful for first occurrence troubleshooting.</para>
1114       <para><literal>ptlrpc</literal> is an RPC protocol layered on LNet that deals with stateful servers and has semantics and built-in support for recovery.</para>
1115       <para>The ptlrpc request history works as follows:</para>
1116       <orderedlist>
1117         <listitem>
1118           <para><literal>request_in_callback()</literal> adds the new request to the service&apos;s request history.</para>
1119         </listitem>
1120         <listitem>
1121           <para>When a request buffer becomes idle, it is added to the service&apos;s request buffer history list.</para>
1122         </listitem>
1123         <listitem>
1124           <para>Buffers are culled from the service request buffer history if it has grown above <literal>req_buffer_history_max</literal> and its reqs are removed from the service request history.</para>
1125         </listitem>
1126       </orderedlist>
1127       <para>Request history is accessed and controlled using the following parameters for each service:</para>
1128       <itemizedlist>
1129         <listitem>
1130           <para><literal>req_buffer_history_len </literal></para>
1131           <para>Number of request buffers currently in the history</para>
1132         </listitem>
1133       </itemizedlist>
1134       <itemizedlist>
1135         <listitem>
1136           <para><literal>req_buffer_history_max </literal></para>
1137           <para>Maximum number of request buffers to keep</para>
1138         </listitem>
1139       </itemizedlist>
1140       <itemizedlist>
1141         <listitem>
1142           <para><literal>req_history </literal> </para>
1143           <para>The request history</para>
1144         </listitem>
1145       </itemizedlist>
1146       <para>Requests in the history include &quot;live&quot; requests that are currently being handled. Each line in <literal>req_history</literal> looks like:</para>
1147       <screen><replaceable>sequence</replaceable>:<replaceable>target_NID</replaceable>:<replaceable>client_NID</replaceable>:<replaceable>cliet_xid</replaceable>:<replaceable>request_length</replaceable>:<replaceable>rpc_phase</replaceable> <replaceable>service_specific_data</replaceable> </screen>
1148       <informaltable frame="all">
1149         <tgroup cols="2">
1150           <colspec colname="c1" colwidth="50*"/>
1151           <colspec colname="c2" colwidth="50*"/>
1152           <thead>
1153             <row>
1154               <entry>
1155                 <para><emphasis role="bold">Parameter</emphasis></para>
1156               </entry>
1157               <entry>
1158                 <para><emphasis role="bold">Description</emphasis></para>
1159               </entry>
1160             </row>
1161           </thead>
1162           <tbody>
1163             <row>
1164               <entry>
1165                 <para> <emphasis role="bold">
1166                     <literal>seq</literal>
1167                   </emphasis></para>
1168               </entry>
1169               <entry>
1170                 <para> Request sequence number</para>
1171               </entry>
1172             </row>
1173             <row>
1174               <entry>
1175                 <para> <literal>
1176                     <replaceable>target NID</replaceable>
1177                   </literal></para>
1178               </entry>
1179               <entry>
1180                 <para> Destination <literal>NID</literal> of the incoming request</para>
1181               </entry>
1182             </row>
1183             <row>
1184               <entry>
1185                 <para> <literal>
1186                     <replaceable>client ID</replaceable>
1187                   </literal></para>
1188               </entry>
1189               <entry>
1190                 <para> Client <literal>PID</literal> and <literal>NID</literal></para>
1191               </entry>
1192             </row>
1193             <row>
1194               <entry>
1195                 <para>
1196                         <literal>
1197                     <replaceable>xid</replaceable>
1198                   </literal>
1199                   </para>
1200               </entry>
1201               <entry>
1202                 <para> <literal>rq_xid</literal></para>
1203               </entry>
1204             </row>
1205             <row>
1206               <entry>
1207                 <para> <emphasis role="bold">
1208                     <literal>length</literal>
1209                   </emphasis></para>
1210               </entry>
1211               <entry>
1212                 <para> Size of the request message</para>
1213               </entry>
1214             </row>
1215             <row>
1216               <entry>
1217                 <para> <emphasis role="bold">
1218                     <literal>phase</literal>
1219                   </emphasis></para>
1220               </entry>
1221               <entry>
1222                 <para><itemizedlist>
1223                     <listitem>
1224                       <para>New (waiting to be handled or could not be unpacked)</para>
1225                     </listitem>
1226                     <listitem>
1227                       <para>Interpret (unpacked or being handled)</para>
1228                     </listitem>
1229                     <listitem>
1230                       <para>Complete (handled)</para>
1231                     </listitem>
1232                   </itemizedlist></para>
1233               </entry>
1234             </row>
1235             <row>
1236               <entry>
1237                 <para> <emphasis role="bold">
1238                     <literal>svc specific</literal>
1239                   </emphasis></para>
1240               </entry>
1241               <entry>
1242                 <para>Service-specific request printout. Currently, the only service that does this is the OST (which prints the opcode if the message has been unpacked successfully</para>
1243               </entry>
1244             </row>
1245           </tbody>
1246         </tgroup>
1247       </informaltable>
1248     </section>
1249     <section remap="h3">
1250       <title><indexterm><primary>debugging</primary><secondary>memory leaks</secondary></indexterm>Finding Memory Leaks Using <literal>leak_finder.pl</literal></title>
1251       <para>Memory leaks can occur in code when memory has been allocated and then not freed once it is no longer required. The <literal>leak_finder.pl</literal> program provides a way to find memory leaks.</para>
1252       <para>Before running this program, you must turn on debugging to collect all <literal>malloc</literal> and free entries. Run:</para>
1253       <screen>lctl set_param debug=+malloc </screen>
1254       <para>Then complete the following steps:</para>
1255       <orderedlist>
1256         <listitem>
1257           <para>Dump the log into a user-specified log file using lctl
1258             (see <xref linkend="using_lctl_tool"/>).
1259           </para>
1260         </listitem>
1261         <listitem>
1262           <para>Run the leak finder on the newly-created log dump:</para>
1263           <screen>perl leak_finder.pl <replaceable>ascii-logname</replaceable></screen>
1264         </listitem>
1265       </orderedlist>
1266       <para>The output is:</para>
1267       <screen>malloced 8bytes at a3116744 (called pathcopy)
1268 (lprocfs_status.c:lprocfs_add_vars:80)
1269 freed 8bytes at a3116744 (called pathcopy)
1270 (lprocfs_status.c:lprocfs_add_vars:80)
1271 </screen>
1272       <para>The tool displays the following output to show the leaks found:</para>
1273       <screen>Leak:32bytes allocated at a23a8fc(service.c:ptlrpc_init_svc:144,debug file line 241)</screen>
1274     </section>
1275   </section>
1276 </chapter>
1277 <!--
1278   vim:expandtab:shiftwidth=2:tabstop=8:
1279   -->