LustreDebugging.xml

   1 <?xml version="1.0" encoding="UTF-8"?>
   2 <chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='lustredebugging'>
   3   <info>
   4     <title xml:id='lustredebugging.title'>Lustre Debugging</title>
   5   </info>
   6
   7   <para><anchor xml:id="dbdoclet.50438274_pgfId-1295651" xreflabel=""/>This chapter describes tips and information to debug Lustre, and includes the following sections:</para>
   8   <itemizedlist><listitem>
   9       <para><xref linkend="dbdoclet.50438274_15874"/></para>
  10     </listitem>
  11
  12 <listitem>
  13       <para><xref linkend="dbdoclet.50438274_23607"/></para>
  14     </listitem>
  15
  16 <listitem>
  17       <para><xref linkend="dbdoclet.50438274_80443"/></para>
  18     </listitem>
  19
  20 </itemizedlist>
  21
  22     <section xml:id="dbdoclet.50438274_15874">
  23       <title>28.1 Diagnostic and Debugging Tools</title>
  24       <para><anchor xml:id="dbdoclet.50438274_pgfId-1295666" xreflabel=""/>A variety of diagnostic and analysis tools are available to debug issues with the Lustre software. Some of these are provided in Linux distributions, while others have been developed and are made available by the Lustre project.</para>
  25       <section remap="h3">
  26         <title><anchor xml:id="dbdoclet.50438274_pgfId-1295667" xreflabel=""/>28.1.1 Lustre Debugging Tools</title>
  27         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295668" xreflabel=""/>The following in-kernel debug mechanisms are incorporated into the Lustre software:</para>
  28         <itemizedlist><listitem>
  29             <para><anchor xml:id="dbdoclet.50438274_pgfId-1295669" xreflabel=""/><emphasis role="bold">Debug logs</emphasis>  - A circular debug buffer to which Lustre internal debug messages are written (in contrast to error messages, which are printed to the syslog or console). Entries to the Lustre debug log are controlled by the mask set by /proc/sys/lnet/debug. The log size defaults to 5 MB per CPU but can be increased as a busy system will quickly overwrite 5 MB. When the buffer fills, the oldest information is discarded.</para>
  30           </listitem>
  31
  32 <listitem>
  33             <para><anchor xml:id="dbdoclet.50438274_pgfId-1295670" xreflabel=""/><emphasis role="bold">Debug daemon</emphasis>  - The debug daemon controls logging of debug messages.</para>
  34           </listitem>
  35
  36 <listitem>
  37             <para><anchor xml:id="dbdoclet.50438274_pgfId-1295671" xreflabel=""/><emphasis role="bold">/proc/sys/lnet/debug</emphasis>  - This file contains a mask that can be used to delimit the debugging information written out to the kernel debug logs.</para>
  38           </listitem>
  39
  40 </itemizedlist>
  41         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295672" xreflabel=""/>The following tools are also provided with the Lustre software:</para>
  42         <itemizedlist><listitem>
  43                 <para><anchor xml:id="dbdoclet.50438274_pgfId-1295673" xreflabel=""/><emphasis role="bold">lctl</emphasis>  - This tool is used with the debug_kernel option to manually dump the Lustre debugging log or post-process debugging logs that are dumped automatically. For more information about the lctl tool, see <xref linkend="dbdoclet.50438274_62472"/> and <xref linkend="systemconfigurationutilities"/>(lctl).</para>
  44           </listitem>
  45
  46 <listitem>
  47             <para><anchor xml:id="dbdoclet.50438274_pgfId-1295680" xreflabel=""/><emphasis role="bold">Lustre subsystem asserts</emphasis>  - A panic-style assertion (LBUG) in the kernel causes Lustre to dump the debug log to the file /tmp/lustre-log.<emphasis>&lt;timestamp&gt;</emphasis> where it can be retrieved after a reboot. For more information, see <link xl:href="LustreTroubleshooting.html#50438198_40669">Viewing Error Messages</link>.</para>
  48           </listitem>
  49
  50 <listitem>
  51             <para><anchor xml:id="dbdoclet.50438274_pgfId-1295684" xreflabel=""/><emphasis role="bold">lfs</emphasis>  - This utility provides access to the extended attributes (EAs) of a Lustre file (along with other information). For more inforamtion about lfs, see <link xl:href="UserUtilities.html#50438206_94597">lfs</link>.</para>
  52           </listitem>
  53
  54 </itemizedlist>
  55       </section>
  56       <section remap="h3">
  57         <title><anchor xml:id="dbdoclet.50438274_pgfId-1295688" xreflabel=""/>28.1.2 External Debugging Tools</title>
  58         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295689" xreflabel=""/>The tools described in this section are provided in the Linux kernel or are available at an external website. For information about using some of these tools for Lustre debugging, see <link xl:href="LustreDebugging.html#50438274_23607">Lustre Debugging Procedures</link> and <link xl:href="LustreDebugging.html#50438274_80443">Lustre Debugging for Developers</link>.</para>
  59         <section remap="h4">
  60           <title><anchor xml:id="dbdoclet.50438274_pgfId-1295696" xreflabel=""/>28.1.2.1 Tools for Administrators and Developers</title>
  61           <para><anchor xml:id="dbdoclet.50438274_pgfId-1295697" xreflabel=""/>Some general debugging tools provided as a part of the standard Linux distro are:</para>
  62           <itemizedlist><listitem>
  63               <para><anchor xml:id="dbdoclet.50438274_pgfId-1295698" xreflabel=""/><emphasis role="bold">strace</emphasis> . This tool allows a system call to be traced.</para>
  64             </listitem>
  65
  66 <listitem>
  67               <para><anchor xml:id="dbdoclet.50438274_pgfId-1295699" xreflabel=""/><emphasis role="bold">/var/log/messages</emphasis> . syslogd prints fatal or serious messages at this log.</para>
  68             </listitem>
  69
  70 <listitem>
  71               <para><anchor xml:id="dbdoclet.50438274_pgfId-1295700" xreflabel=""/><emphasis role="bold">Crash dumps</emphasis> . On crash-dump enabled kernels, sysrq c produces a crash dump. Lustre enhances this crash dump with a log dump (the last 64 KB of the log) to the console.</para>
  72             </listitem>
  73
  74 <listitem>
  75               <para><anchor xml:id="dbdoclet.50438274_pgfId-1295701" xreflabel=""/><emphasis role="bold">debugfs</emphasis> . Interactive file system debugger.</para>
  76             </listitem>
  77
  78 </itemizedlist>
  79           <para><anchor xml:id="dbdoclet.50438274_pgfId-1295702" xreflabel=""/>The following logging and data collection tools can be used to collect information for debugging Lustre kernel issues:</para>
  80           <itemizedlist><listitem>
  81               <para><anchor xml:id="dbdoclet.50438274_pgfId-1295703" xreflabel=""/><emphasis role="bold">kdump</emphasis> . A Linux kernel crash utility useful for debugging a system running Red Hat Enterprise Linux. For more information about kdump, see the Red Hat knowledge base article <link xl:href="http://kbase.redhat.com/faq/docs/DOC-6039">How do I configure kexec/kdump on Red Hat Enterprise Linux 5?</link>. To download kdump, go to the <link xl:href="http://fedoraproject.org/wiki/SystemConfig/kdump#Download">Fedora Project Download</link> site.</para>
  82             </listitem>
  83
  84 <listitem>
  85               <para><anchor xml:id="dbdoclet.50438274_pgfId-1295706" xreflabel=""/><emphasis role="bold">netconsole</emphasis> . Enables kernel-level network logging over UDP. A system requires (SysRq) allows users to collect relevant data through netconsole.</para>
  86             </listitem>
  87
  88 <listitem>
  89               <para><anchor xml:id="dbdoclet.50438274_pgfId-1295707" xreflabel=""/><emphasis role="bold">netdump</emphasis> . A crash dump utility from Red Hat that allows memory images to be dumped over a network to a central server for analysis. The netdump utility was replaced by kdump in RHEL 5. For more information about netdump, see <link xl:href="http://www.redhat.com/support/wpapers/redhat/netdump/">Red Hat, Inc.&apos;s Network Console and Crash Dump Facility</link>.</para>
  90             </listitem>
  91
  92 </itemizedlist>
  93         </section>
  94         <section remap="h4">
  95           <title><anchor xml:id="dbdoclet.50438274_pgfId-1295709" xreflabel=""/>28.1.2.2 Tools for Developers</title>
  96           <para><anchor xml:id="dbdoclet.50438274_pgfId-1295710" xreflabel=""/>The tools described in this section may be useful for debugging Lustre in a development environment.</para>
  97           <para><anchor xml:id="dbdoclet.50438274_pgfId-1295711" xreflabel=""/>Of general interest is:</para>
  98           <itemizedlist><listitem>
  99               <para><anchor xml:id="dbdoclet.50438274_pgfId-1295712" xreflabel=""/><emphasis role="bold">leak_finder.pl</emphasis> . This program provided with Lustre is useful for finding memory leaks in the code.</para>
 100             </listitem>
 101
 102 </itemizedlist>
 103           <para><anchor xml:id="dbdoclet.50438274_pgfId-1295713" xreflabel=""/>A virtual machine is often used to create an isolated development and test environment. Some commonly-used virtual machines are:</para>
 104           <itemizedlist><listitem>
 105               <para><anchor xml:id="dbdoclet.50438274_pgfId-1295714" xreflabel=""/><emphasis role="bold">VirtualBox Open Source Edition</emphasis> . Provides enterprise-class virtualization capability for all major platforms and is available free at <link xl:href="http://www.sun.com/software/products/virtualbox/get.jsp?intcmp=2945">Get Sun VirtualBox</link>.</para>
 106             </listitem>
 107
 108 <listitem>
 109               <para><anchor xml:id="dbdoclet.50438274_pgfId-1295716" xreflabel=""/><emphasis role="bold">VMware Server</emphasis> . Virtualization platform available as free introductory software at <link xl:href="http://downloads.vmware.com/d/info/datacenter_downloads/vmware_server/2_0">Download VMware Server</link>.</para>
 110             </listitem>
 111
 112 <listitem>
 113               <para><anchor xml:id="dbdoclet.50438274_pgfId-1295718" xreflabel=""/><emphasis role="bold">Xen</emphasis> . A para-virtualized environment with virtualization capabilities similar to VMware Server and Virtual Box. However, Xen allows the use of modified kernels to provide near-native performance and the ability to emulate shared storage. For more information, go to <link xl:href="http://xen.org/">xen.org</link>.</para>
 114             </listitem>
 115
 116 </itemizedlist>
 117           <para><anchor xml:id="dbdoclet.50438274_pgfId-1295720" xreflabel=""/>A variety of debuggers and analysis tools are available including:</para>
 118           <itemizedlist><listitem>
 119               <para><anchor xml:id="dbdoclet.50438274_pgfId-1295721" xreflabel=""/><emphasis role="bold">kgdb</emphasis> . The Linux Kernel Source Level Debugger kgdb is used in conjunction with the GNU Debugger gdb for debugging the Linux kernel. For more information about using kgdb with gdb, see <link xl:href="http://www.linuxtopia.org/online_books/redhat_linux_debugging_with_gdb/running.html">Chapter 6. Running Programs Under gdb</link> in the <emphasis>Red Hat Linux 4 Debugging with GDB</emphasis> guide.</para>
 120             </listitem>
 121
 122 <listitem>
 123               <para><anchor xml:id="dbdoclet.50438274_pgfId-1295723" xreflabel=""/><emphasis role="bold">crash</emphasis> . Used to analyze saved crash dump data when a system had panicked or locked up or appears unresponsive. For more information about using crash to analyze a crash dump, see:</para>
 124               <itemizedlist><listitem>
 125                   <para><anchor xml:id="dbdoclet.50438274_pgfId-1295725" xreflabel=""/> Red Hat Magazine article: <link xl:href="http://magazine.redhat.com/2007/08/15/a-quick-overview-of-linux-kernel-crash-dump-analysis/">A quick overview of Linux kernel crash dump analysis</link></para>
 126                 </listitem>
 127
 128 <listitem>
 129                   <para><anchor xml:id="dbdoclet.50438274_pgfId-1295727" xreflabel=""/><link xl:href="http://people.redhat.com/anderson/crash_whitepaper/#EXAMPLES">Crash Usage: A Case Study</link>  from the white paper <emphasis>Red Hat Crash Utility</emphasis> by David Anderson</para>
 130                 </listitem>
 131
 132 <listitem>
 133                   <para><anchor xml:id="dbdoclet.50438274_pgfId-1295729" xreflabel=""/> Kernel Trap forum entry: <link xl:href="http://kerneltrap.org/node/5758">Linux: Kernel Crash Dumps</link></para>
 134                 </listitem>
 135
 136 <listitem>
 137                   <para><anchor xml:id="dbdoclet.50438274_pgfId-1295731" xreflabel=""/> White paper: <link xl:href="http://www.google.com/url?sa=t&amp;source=web&amp;ct=res&amp;cd=8&amp;ved=0CCUQFjAH&amp;url=http%3A%2F%2Fwww.kernel.sg%2Fpapers%2Fcrash-dump-analysis.pdf&amp;rct=j&amp;q=redhat+crash+dump&amp;ei=6aQBS-ifK4T8tAPcjdiHCw&amp;usg=AFQjCNEk03E3GDtAsawG3gfpwc1gGNELAg">A Quick Overview of Linux Kernel Crash Dump Analysis</link></para>
 138                 </listitem>
 139
 140 </itemizedlist>
 141             </listitem>
 142 </itemizedlist>
 143         </section>
 144       </section>
 145     </section>
 146     <section xml:id="dbdoclet.50438274_23607">
 147       <title>28.2 Lustre Debugging Procedures</title>
 148       <para><anchor xml:id="dbdoclet.50438274_pgfId-1295734" xreflabel=""/>The procedures below may be useful to administrators or developers debugging a Lustre files system.</para>
 149       <section remap="h3">
 150         <title><anchor xml:id="dbdoclet.50438274_pgfId-1295735" xreflabel=""/>28.2.1 Understanding the Lustre Debug Messaging Format</title>
 151         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295736" xreflabel=""/>Lustre debug messages are categorized by originating sybsystem, message type, and locaton in the source code. For a list of subsystems and message types, see <xref linkend="dbdoclet.50438274_57603"/>.</para>
 152                 <note><para>For a current list of subsystems and debug message types, see lnet/include/libcfs/libcfs.h in the Lustre tree</para></note>
 153                 <para><anchor xml:id="dbdoclet.50438274_pgfId-1295743" xreflabel=""/>The elements of a Lustre debug message are described in <xref linkend="dbdoclet.50438274_57177"/>Format of Lustre Debug Messages.</para>
 154         <section remap="h4">
 155           <title><anchor xml:id="dbdoclet.50438274_pgfId-1295747" xreflabel=""/>28.2.1.1 <anchor xml:id="dbdoclet.50438274_57603" xreflabel=""/>Lustre <anchor xml:id="dbdoclet.50438274_marker-1295746" xreflabel=""/>Debug Messages</title>
 156           <para><anchor xml:id="dbdoclet.50438274_pgfId-1295748" xreflabel=""/>Each Lustre debug message has the tag of the subsystem it originated in, the message type, and the location in the source code. The subsystems and debug types used in Lustre are as follows:</para>
 157           <itemizedlist><listitem>
 158               <para><anchor xml:id="dbdoclet.50438274_pgfId-1295749" xreflabel=""/>  Standard Subsystems:</para>
 159             </listitem>
 160
 161 </itemizedlist>
 162           <para><anchor xml:id="dbdoclet.50438274_pgfId-1295750" xreflabel=""/> mdc, mds, osc, ost, obdclass, obdfilter, llite, ptlrpc, portals, lnd, ldlm, lov</para>
 163           <itemizedlist><listitem>
 164               <para><anchor xml:id="dbdoclet.50438274_pgfId-1295840" xreflabel=""/>  Debug Types:</para>
 165             </listitem>
 166 <listitem>
 167               <para><informaltable frame="all">
 168                   <tgroup cols="2">
 169                     <colspec colname="c1" colwidth="50*"/>
 170                     <colspec colname="c2" colwidth="50*"/>
 171                     <thead>
 172                       <row>
 173                         <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438274_pgfId-1295753" xreflabel=""/>Types</emphasis></para></entry>
 174                         <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438274_pgfId-1295755" xreflabel=""/>Description</emphasis></para></entry>
 175                       </row>
 176                     </thead>
 177                     <tbody>
 178                       <row>
 179                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295757" xreflabel=""/><emphasis role="bold">trace</emphasis></para></entry>
 180                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295759" xreflabel=""/>Entry/Exit markers</para></entry>
 181                       </row>
 182                       <row>
 183                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295761" xreflabel=""/><emphasis role="bold">dlmtrace</emphasis></para></entry>
 184                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295763" xreflabel=""/>Locking-related information</para></entry>
 185                       </row>
 186                       <row>
 187                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295765" xreflabel=""/><emphasis role="bold">inode</emphasis></para></entry>
 188                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295767" xreflabel=""/> </para></entry>
 189                       </row>
 190                       <row>
 191                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295769" xreflabel=""/><emphasis role="bold">super</emphasis></para></entry>
 192                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295771" xreflabel=""/> </para></entry>
 193                       </row>
 194                       <row>
 195                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295773" xreflabel=""/><emphasis role="bold">ext2</emphasis></para></entry>
 196                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295775" xreflabel=""/>Anything from the ext2_debug</para></entry>
 197                       </row>
 198                       <row>
 199                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295777" xreflabel=""/><emphasis role="bold">malloc</emphasis></para></entry>
 200                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295779" xreflabel=""/>Print malloc or free information</para></entry>
 201                       </row>
 202                       <row>
 203                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295781" xreflabel=""/><emphasis role="bold">cache</emphasis></para></entry>
 204                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295783" xreflabel=""/>Cache-related information</para></entry>
 205                       </row>
 206                       <row>
 207                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295785" xreflabel=""/><emphasis role="bold">info</emphasis></para></entry>
 208                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295787" xreflabel=""/>General information</para></entry>
 209                       </row>
 210                       <row>
 211                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295789" xreflabel=""/><emphasis role="bold">ioctl</emphasis></para></entry>
 212                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295791" xreflabel=""/>IOCTL-related information</para></entry>
 213                       </row>
 214                       <row>
 215                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295793" xreflabel=""/><emphasis role="bold">blocks</emphasis></para></entry>
 216                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295795" xreflabel=""/>Ext2 block allocation information</para></entry>
 217                       </row>
 218                       <row>
 219                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295797" xreflabel=""/><emphasis role="bold">net</emphasis></para></entry>
 220                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295799" xreflabel=""/>Networking</para></entry>
 221                       </row>
 222                       <row>
 223                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295801" xreflabel=""/><emphasis role="bold">warning</emphasis></para></entry>
 224                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295803" xreflabel=""/> </para></entry>
 225                       </row>
 226                       <row>
 227                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295805" xreflabel=""/><emphasis role="bold">buffs</emphasis></para></entry>
 228                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295807" xreflabel=""/> </para></entry>
 229                       </row>
 230                       <row>
 231                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295809" xreflabel=""/><emphasis role="bold">other</emphasis></para></entry>
 232                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295811" xreflabel=""/> </para></entry>
 233                       </row>
 234                       <row>
 235                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295813" xreflabel=""/><emphasis role="bold">dentry</emphasis></para></entry>
 236                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295815" xreflabel=""/> </para></entry>
 237                       </row>
 238                       <row>
 239                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295817" xreflabel=""/><emphasis role="bold">portals</emphasis></para></entry>
 240                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295819" xreflabel=""/>Entry/Exit markers</para></entry>
 241                       </row>
 242                       <row>
 243                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295821" xreflabel=""/><emphasis role="bold">page</emphasis></para></entry>
 244                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295823" xreflabel=""/>Bulk page handling</para></entry>
 245                       </row>
 246                       <row>
 247                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295825" xreflabel=""/><emphasis role="bold">error</emphasis></para></entry>
 248                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295827" xreflabel=""/>Error messages</para></entry>
 249                       </row>
 250                       <row>
 251                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295829" xreflabel=""/><emphasis role="bold">emerg</emphasis></para></entry>
 252                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295831" xreflabel=""/> </para></entry>
 253                       </row>
 254                       <row>
 255                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295833" xreflabel=""/><emphasis role="bold">rpctrace</emphasis></para></entry>
 256                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295835" xreflabel=""/>For distributed debugging</para></entry>
 257                       </row>
 258                       <row>
 259                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295837" xreflabel=""/><emphasis role="bold">ha</emphasis></para></entry>
 260                         <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295839" xreflabel=""/>Failover and recovery-related information</para></entry>
 261                       </row>
 262                     </tbody>
 263                   </tgroup>
 264                 </informaltable>
 265 </para>
 266             </listitem>
 267 </itemizedlist>
 268         </section>
 269         <section remap="h4">
 270           <title><anchor xml:id="dbdoclet.50438274_pgfId-1295842" xreflabel=""/>28.2.1.2 <anchor xml:id="dbdoclet.50438274_57177" xreflabel=""/>Format of Lustre Debug Messages</title>
 271           <para><anchor xml:id="dbdoclet.50438274_pgfId-1295843" xreflabel=""/>Lustre uses the CDEBUG and CERROR macros to print the debug or error messages. To print the message, the CDEBUG macro uses portals_debug_msg (portals/linux/oslib/debug.c). The message format is described below, along with an example.</para>
 272           <informaltable frame="all">
 273             <tgroup cols="2">
 274               <colspec colname="c1" colwidth="50*"/>
 275               <colspec colname="c2" colwidth="50*"/>
 276               <thead>
 277                 <row>
 278                   <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438274_pgfId-1295846" xreflabel=""/>Parameter</emphasis></para></entry>
 279                   <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438274_pgfId-1295848" xreflabel=""/>Description</emphasis></para></entry>
 280                 </row>
 281               </thead>
 282               <tbody>
 283                 <row>
 284                   <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295850" xreflabel=""/><emphasis role="bold">subsystem</emphasis></para></entry>
 285                   <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295852" xreflabel=""/>800000</para></entry>
 286                 </row>
 287                 <row>
 288                   <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295854" xreflabel=""/><emphasis role="bold">debug mask</emphasis></para></entry>
 289                   <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295856" xreflabel=""/>000010</para></entry>
 290                 </row>
 291                 <row>
 292                   <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295858" xreflabel=""/><emphasis role="bold">smp_processor_id</emphasis></para></entry>
 293                   <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295860" xreflabel=""/>0</para></entry>
 294                 </row>
 295                 <row>
 296                   <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295862" xreflabel=""/><emphasis role="bold">sec.used</emphasis></para></entry>
 297                   <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295864" xreflabel=""/>10818808</para><para> 47.677302</para></entry>
 298                 </row>
 299                 <row>
 300                   <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295866" xreflabel=""/><emphasis role="bold">stack size</emphasis></para></entry>
 301                   <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295868" xreflabel=""/>1204:</para></entry>
 302                 </row>
 303                 <row>
 304                   <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295870" xreflabel=""/><emphasis role="bold">pid</emphasis></para></entry>
 305                   <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295872" xreflabel=""/>2973:</para></entry>
 306                 </row>
 307                 <row>
 308                   <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295874" xreflabel=""/><emphasis role="bold">host pid (if uml) or zero</emphasis></para></entry>
 309                   <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295876" xreflabel=""/>31070:</para></entry>
 310                 </row>
 311                 <row>
 312                   <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295878" xreflabel=""/><emphasis role="bold">(file:line #:functional())</emphasis></para></entry>
 313                   <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295880" xreflabel=""/>(as_dev.c:144:create_write_buffers())</para></entry>
 314                 </row>
 315                 <row>
 316                   <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295882" xreflabel=""/><emphasis role="bold">debug message</emphasis></para></entry>
 317                   <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1295884" xreflabel=""/>kmalloced &apos;*obj&apos;: 24 at a375571c (tot 17447717)</para></entry>
 318                 </row>
 319               </tbody>
 320             </tgroup>
 321           </informaltable>
 322         </section>
 323         <section remap="h4">
 324           <title><anchor xml:id="dbdoclet.50438274_pgfId-1295886" xreflabel=""/>28.2.1.3 Lustre <anchor xml:id="dbdoclet.50438274_marker-1295885" xreflabel=""/>Debug Messages Buffer</title>
 325           <para><anchor xml:id="dbdoclet.50438274_pgfId-1295887" xreflabel=""/>Lustre debug messages are maintained in a buffer, with the maximum buffer size specified (in MBs) by the debug_mb parameter (/proc/sys/lnet/debug_mb). The buffer is circular, so debug messages are kept until the allocated buffer limit is reached, and then the first messages are overwritten.</para>
 326         </section>
 327       </section>
 328       <section remap="h3">
 329         <title><anchor xml:id="dbdoclet.50438274_pgfId-1295889" xreflabel=""/>28.2.2 <anchor xml:id="dbdoclet.50438274_62472" xreflabel=""/>Using the lctl Tool to View Debug Messages</title>
 330         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295890" xreflabel=""/>The lctl tool allows debug messages to be filtered based on subsystems and message types to extract information useful for troubleshooting from a kernel debug log. For a command reference, see <link xl:href="SystemConfigurationUtilities.html#50438219_38274">lctl</link>.</para>
 331         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295894" xreflabel=""/>You can use lctl to:</para>
 332         <itemizedlist><listitem>
 333             <para><anchor xml:id="dbdoclet.50438274_pgfId-1295895" xreflabel=""/> Obtain a list of all the types and subsystems:</para>
 334           </listitem>
 335
 336 </itemizedlist>
 337         <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295896" xreflabel=""/>lctl &gt; debug_list <emphasis>&lt;subs | types&gt;</emphasis></screen>
 338         <itemizedlist><listitem>
 339             <para><anchor xml:id="dbdoclet.50438274_pgfId-1295897" xreflabel=""/> Filter the debug log:</para>
 340           </listitem>
 341
 342 </itemizedlist>
 343         <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295898" xreflabel=""/>lctl &gt; filter <emphasis>&lt;subsystem name | debug type&gt;</emphasis></screen>
 344                 <note><para>When lctl filters, it removes unwanted lines from the displayed output. This does not affect the contents of the debug log in the kernel&apos;s memory. As a result, you can print the log many times with different filtering levels without worrying about losing data.</para></note>
 345
 346         <itemizedlist><listitem>
 347             <para><anchor xml:id="dbdoclet.50438274_pgfId-1295900" xreflabel=""/> Show debug messages belonging to certain subsystem or type:</para>
 348           </listitem>
 349
 350 </itemizedlist>
 351         <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295901" xreflabel=""/>lctl &gt; show <emphasis>&lt;subsystem name | debug type&gt;</emphasis></screen>
 352         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295902" xreflabel=""/>debug_kernel pulls the data from the kernel logs, filters it appropriately, and displays or saves it as per the specified options</para>
 353         <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295903" xreflabel=""/>lctl &gt; debug_kernel [<emphasis>output filename</emphasis>]
 354 </screen>
 355         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295904" xreflabel=""/>If the debugging is being done on User Mode Linux (UML), it might be useful to save the logs on the host machine so that they can be used at a later time.</para>
 356         <itemizedlist><listitem>
 357             <para><anchor xml:id="dbdoclet.50438274_pgfId-1295905" xreflabel=""/> Filter a log on disk, if you already have a debug log saved to disk (likely from a crash):</para>
 358           </listitem>
 359
 360 </itemizedlist>
 361         <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295906" xreflabel=""/>lctl &gt; debug_file <emphasis>&lt;input filename&gt;</emphasis> [<emphasis>output filename</emphasis>]
 362 </screen>
 363         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295907" xreflabel=""/>During the debug session, you can add markers or breaks to the log for any reason:</para>
 364         <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295908" xreflabel=""/>lctl &gt; mark [marker text]
 365 </screen>
 366         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295909" xreflabel=""/>The marker text defaults to the current date and time in the debug log (similar to the example shown below):</para>
 367         <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295910" xreflabel=""/>DEBUG MARKER: Tue Mar 5 16:06:44 EST 2002
 368 </screen>
 369         <itemizedlist><listitem>
 370             <para><anchor xml:id="dbdoclet.50438274_pgfId-1295911" xreflabel=""/> Completely flush the kernel debug buffer:</para>
 371           </listitem>
 372
 373 </itemizedlist>
 374         <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295912" xreflabel=""/>lctl &gt; clear
 375 </screen>
 376                 <note><para>Debug messages displayed with lctl are also subject to the kernel debug masks; the filters are additive.</para></note>
 377         <section remap="h4">
 378           <title><anchor xml:id="dbdoclet.50438274_pgfId-1295915" xreflabel=""/>28.2.2.1 Sample lctl<anchor xml:id="dbdoclet.50438274_marker-1295914" xreflabel=""/>Run</title>
 379           <para><anchor xml:id="dbdoclet.50438274_pgfId-1295916" xreflabel=""/>Below is a sample run using the lctl command.</para>
 380           <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295917" xreflabel=""/>bash-2.04# ./lctl
 381 <anchor xml:id="dbdoclet.50438274_pgfId-1295918" xreflabel=""/>lctl &gt; debug_kernel /tmp/lustre_logs/log_all
 382 <anchor xml:id="dbdoclet.50438274_pgfId-1295919" xreflabel=""/>Debug log: 324 lines, 324 kept, 0 dropped.
 383 <anchor xml:id="dbdoclet.50438274_pgfId-1295920" xreflabel=""/>lctl &gt; filter trace
 384 <anchor xml:id="dbdoclet.50438274_pgfId-1295921" xreflabel=""/>Disabling output of type &quot;trace&quot;
 385 <anchor xml:id="dbdoclet.50438274_pgfId-1295922" xreflabel=""/>lctl &gt; debug_kernel /tmp/lustre_logs/log_notrace
 386 <anchor xml:id="dbdoclet.50438274_pgfId-1295923" xreflabel=""/>Debug log: 324 lines, 282 kept, 42 dropped.
 387 <anchor xml:id="dbdoclet.50438274_pgfId-1295924" xreflabel=""/>lctl &gt; show trace
 388 <anchor xml:id="dbdoclet.50438274_pgfId-1295925" xreflabel=""/>Enabling output of type &quot;trace&quot;
 389 <anchor xml:id="dbdoclet.50438274_pgfId-1295926" xreflabel=""/>lctl &gt; filter portals
 390 <anchor xml:id="dbdoclet.50438274_pgfId-1295927" xreflabel=""/>Disabling output from subsystem &quot;portals&quot;
 391 <anchor xml:id="dbdoclet.50438274_pgfId-1295928" xreflabel=""/>lctl &gt; debug_kernel /tmp/lustre_logs/log_noportals
 392 <anchor xml:id="dbdoclet.50438274_pgfId-1295929" xreflabel=""/>Debug log: 324 lines, 258 kept, 66 dropped.
 393 </screen>
 394         </section>
 395       </section>
 396       <section remap="h3">
 397         <title><anchor xml:id="dbdoclet.50438274_pgfId-1295930" xreflabel=""/>28.2.3 Dumping the Buffer to a File (debug_daemon)</title>
 398         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295931" xreflabel=""/>The debug_daemon option is used by lctl to control the dumping of the debug_kernel buffer to a user-specified file. This functionality uses a kernel thread on top of debug_kernel, which works in parallel with the debug_daemon command.</para>
 399         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295932" xreflabel=""/>The debug_daemon is highly dependent on file system write speed. File system write operations may not be fast enough to flush out all of the debug_buffer if the Lustre file system is under heavy system load and continues to CDEBUG to the debug_buffer. The debug_daemon will write the message DEBUG MARKER: Trace buffer full into the debug_buffer to indicate the debug_buffer contents are overlapping before the debug_daemon flushes data to a file.</para>
 400         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295933" xreflabel=""/>Users can use lctlcontrol to start or stop the Lustre daemon from dumping the debug_buffer to a file. Users can also temporarily hold daemon from dumping the file. Use of the debug_daemon sub-command to lctl can provide the same function.</para>
 401         <section remap="h4">
 402           <title><anchor xml:id="dbdoclet.50438274_pgfId-1295934" xreflabel=""/>28.2.3.1 lctldebug_daemon Commands</title>
 403           <para><anchor xml:id="dbdoclet.50438274_pgfId-1295935" xreflabel=""/>This section describes lctldebug_daemon commands.</para>
 404           <para><anchor xml:id="dbdoclet.50438274_pgfId-1295936" xreflabel=""/>To initiate the debug_daemon to start dumping debug_buffer into a file., enter</para>
 405           <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295937" xreflabel=""/>$ lctl debug_daemon start [{file} {megabytes}]
 406 </screen>
 407           <para><anchor xml:id="dbdoclet.50438274_pgfId-1295938" xreflabel=""/>The file can be a system default file, as shown in /proc/sys/lnet/debug_path. After Lustre starts, the default path is /tmp/lustre-log-$HOSTNAME. Users can specify a new filename for debug_daemon to output debug_buffer. The new file name shows up in /proc/sys/lnet/debug_path. Megabytes is the limitation of the file size in MBs.</para>
 408           <para><anchor xml:id="dbdoclet.50438274_pgfId-1295939" xreflabel=""/>The daemon wraps around and dumps data to the beginning of the file when the output file size is over the limit of the user-specified file size. To decode the dumped file to ASCII and order the log entries by time, run:</para>
 409           <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295940" xreflabel=""/>lctl debug_file {file} &gt; {newfile}
 410 </screen>
 411           <para><anchor xml:id="dbdoclet.50438274_pgfId-1295941" xreflabel=""/>The output is internally sorted by the lctl command using quicksort.</para>
 412           <para><anchor xml:id="dbdoclet.50438274_pgfId-1295942" xreflabel=""/>To completely shut down the debug_daemon operation and flush the file output, enter:</para>
 413           <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295943" xreflabel=""/>debug_daemon stop
 414 </screen>
 415           <para><anchor xml:id="dbdoclet.50438274_pgfId-1295944" xreflabel=""/>Otherwise, debug_daemon is shut down as part of the Lustre file system shutdown process. Users can restart debug_daemon by using start command after each stop command issued.</para>
 416           <para><anchor xml:id="dbdoclet.50438274_pgfId-1295945" xreflabel=""/>This is an example using debug_daemon with the interactive mode of lctl to dump debug logs to a 10 MB file.</para>
 417           <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295946" xreflabel=""/>#~/utils/lctl
 418 </screen>
 419           <para><anchor xml:id="dbdoclet.50438274_pgfId-1295947" xreflabel=""/>To start the daemon to dump debug_buffer into a 40 MB /tmp/dump file, enter:</para>
 420           <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295948" xreflabel=""/>lctl &gt; debug_daemon start /trace/log 40
 421 </screen>
 422           <para><anchor xml:id="dbdoclet.50438274_pgfId-1295949" xreflabel=""/>To completely shut down the daemon, enter:</para>
 423           <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295950" xreflabel=""/>lctl &gt; debug_daemon stop
 424 </screen>
 425           <para><anchor xml:id="dbdoclet.50438274_pgfId-1295951" xreflabel=""/>To start another daemon with an unlimited file size, enter:</para>
 426           <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295952" xreflabel=""/>lctl &gt; debug_daemon start /tmp/unlimited
 427 </screen>
 428           <para><anchor xml:id="dbdoclet.50438274_pgfId-1295953" xreflabel=""/>The text message *** End of debug_daemon trace log *** appears at the end of each output file.</para>
 429         </section>
 430       </section>
 431       <section remap="h3">
 432         <title><anchor xml:id="dbdoclet.50438274_pgfId-1295954" xreflabel=""/>28.2.4 Controlling Information Written to the Kernel <anchor xml:id="dbdoclet.50438274_marker-1295955" xreflabel=""/>Debug Log</title>
 433         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295956" xreflabel=""/>Masks are provided in /proc/sys/lnet/subsystem_debug and /proc/sys/lnet/debug to be used with the systctl command to determine what information is to be written to the debug log. The subsystem_debug mask determines the information written to the log based on the subsystem (such as iobdfilter, net, portals, or OSC). The debug mask controls information based on debug type (such as info, error, trace, or alloc).</para>
 434         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295957" xreflabel=""/>To turn off Lustre debugging completely:</para>
 435         <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295958" xreflabel=""/>sysctl -w lnet.debug=0
 436 </screen>
 437         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295959" xreflabel=""/>To turn on full Lustre debugging:</para>
 438         <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295960" xreflabel=""/>sysctl -w lnet.debug=-1
 439 </screen>
 440         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295961" xreflabel=""/>To turn on logging of messages related to network communications:</para>
 441         <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295962" xreflabel=""/>sysctl -w lnet.debug=net
 442 </screen>
 443         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295963" xreflabel=""/>To turn on logging of messages related to network communications and existing debug flags:</para>
 444         <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295964" xreflabel=""/>sysctl -w lnet.debug=+net
 445 </screen>
 446         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295965" xreflabel=""/>To turn off network logging with changing existing flags:</para>
 447         <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295966" xreflabel=""/>sysctl -w lnet.debug=-net
 448 </screen>
 449         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295967" xreflabel=""/>The various options available to print to kernel debug logs are listed in lnet/include/libcfs/libcfs.h</para>
 450       </section>
 451       <section remap="h3">
 452         <title><anchor xml:id="dbdoclet.50438274_pgfId-1295970" xreflabel=""/>28.2.5 <anchor xml:id="dbdoclet.50438274_26909" xreflabel=""/>Troubleshooting with strace<anchor xml:id="dbdoclet.50438274_marker-1295969" xreflabel=""/></title>
 453         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295971" xreflabel=""/>The strace utility provided with the Linux distribution enables system calls to be traced by intercepting all the system calls made by a process and recording the system call name, aruguments, and return values.</para>
 454         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295972" xreflabel=""/>To invoke strace on a program, enter:</para>
 455         <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295973" xreflabel=""/>$ strace <emphasis>&lt;program&gt; &lt;args&gt;</emphasis>
 456 </screen>
 457         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295974" xreflabel=""/>Sometimes, a system call may fork child processes. In this situation, use the -f option of strace to trace the child processes:</para>
 458         <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295975" xreflabel=""/>$ strace -f <emphasis>&lt;program&gt; &lt;args&gt;</emphasis>
 459 </screen>
 460         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295976" xreflabel=""/>To redirect the strace output to a file, enter:</para>
 461         <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295977" xreflabel=""/>$ strace -o <emphasis>&lt;filename&gt; &lt;program&gt; &lt;args&gt;</emphasis>
 462 </screen>
 463         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295978" xreflabel=""/>Use the -ff option, along with -o, to save the trace output in filename.pid, where pid is the process ID of the process being traced. Use the -ttt option to timestamp all lines in the strace output, so they can be correlated to operations in the lustre kernel debug log.</para>
 464         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295979" xreflabel=""/>If the debugging is done in UML, save the traces on the host machine. In this example, hostfs is mounted on /r:</para>
 465         <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295980" xreflabel=""/>$ strace -o /r/tmp/vi.strace
 466 </screen>
 467       </section>
 468       <section remap="h3">
 469         <title><anchor xml:id="dbdoclet.50438274_pgfId-1295983" xreflabel=""/>28.2.6 <anchor xml:id="dbdoclet.50438274_54455" xreflabel=""/>Looking at Disk <anchor xml:id="dbdoclet.50438274_marker-1295982" xreflabel=""/>Content</title>
 470         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295984" xreflabel=""/>In Lustre, the inodes on the metadata server contain extended attributes (EAs) that store information about file striping. EAs contain a list of all object IDs and their locations (that is, the OST that stores them). The lfs tool can be used to obtain this information for a given file using the getstripe subcommand. Use a corresponding lfs setstripe command to specify striping attributes for a new file or directory.</para>
 471         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295985" xreflabel=""/>The lfsgetstripe utility is written in C; it takes a Lustre filename as input and lists all the objects that form a part of this file. To obtain this information for the file /mnt/lustre/frog in Lustre file system, run:</para>
 472         <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295986" xreflabel=""/>$ lfs getstripe /mnt/lustre/frog
 473 <anchor xml:id="dbdoclet.50438274_pgfId-1295987" xreflabel=""/>$
 474 <anchor xml:id="dbdoclet.50438274_pgfId-1295988" xreflabel=""/>   obdix                           objid
 475 <anchor xml:id="dbdoclet.50438274_pgfId-1295989" xreflabel=""/>   0                               17
 476 <anchor xml:id="dbdoclet.50438274_pgfId-1295990" xreflabel=""/>   1                               4
 477 </screen>
 478         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295991" xreflabel=""/>The debugfs tool is provided in the e2fsprogs package. It can be used for interactive debugging of an ldiskfs file system. The debugfs tool can either be used to check status or modify information in the file system. In Lustre, all objects that belong to a file are stored in an underlying ldiskfs file system on the OSTs. The file system uses the object IDs as the file names. Once the object IDs are known, use the debugfs tool to obtain the attributes of all objects from different OSTs.</para>
 479         <para><anchor xml:id="dbdoclet.50438274_pgfId-1295992" xreflabel=""/>A sample run for the /mnt/lustre/frog file used in the above example is shown here:</para>
 480         <screen><anchor xml:id="dbdoclet.50438274_pgfId-1295993" xreflabel=""/>     $ debugfs -c /tmp/ost1
 481 <anchor xml:id="dbdoclet.50438274_pgfId-1295994" xreflabel=""/>   debugfs: cd O
 482 <anchor xml:id="dbdoclet.50438274_pgfId-1295995" xreflabel=""/>   debugfs: cd 0                                   /* for files in group 0 \
 483 */
 484 <anchor xml:id="dbdoclet.50438274_pgfId-1295996" xreflabel=""/>   debugfs: cd d&lt;objid % 32&gt;
 485 <anchor xml:id="dbdoclet.50438274_pgfId-1295997" xreflabel=""/>   debugfs: stat &lt;objid&gt;                             /* for getattr on object\
 486  */
 487 <anchor xml:id="dbdoclet.50438274_pgfId-1295998" xreflabel=""/>   debugfs: quit
 488 <anchor xml:id="dbdoclet.50438274_pgfId-1295999" xreflabel=""/>## Suppose object id is 36, then follow the steps below:
 489 <anchor xml:id="dbdoclet.50438274_pgfId-1296000" xreflabel=""/>   $ debugfs /tmp/ost1
 490 <anchor xml:id="dbdoclet.50438274_pgfId-1296001" xreflabel=""/>   debugfs: cd O
 491 <anchor xml:id="dbdoclet.50438274_pgfId-1296002" xreflabel=""/>   debugfs: cd 0
 492 <anchor xml:id="dbdoclet.50438274_pgfId-1296003" xreflabel=""/>   debugfs: cd d4                                  /* objid % 32 */
 493 <anchor xml:id="dbdoclet.50438274_pgfId-1296004" xreflabel=""/>   debugfs: stat 36                                /* for getattr on obj 4*\
 494 /
 495 <anchor xml:id="dbdoclet.50438274_pgfId-1296005" xreflabel=""/>   debugfs: dump 36 /tmp/obj.36                    /* dump contents of obj \
 496 4 */
 497 <anchor xml:id="dbdoclet.50438274_pgfId-1296006" xreflabel=""/>   debugfs: quit
 498 </screen>
 499       </section>
 500       <section remap="h3">
 501         <title><anchor xml:id="dbdoclet.50438274_pgfId-1296008" xreflabel=""/>28.2.7 Finding the Lustre <anchor xml:id="dbdoclet.50438274_marker-1296007" xreflabel=""/>UUID of an OST</title>
 502         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296009" xreflabel=""/>To determine the Lustre UUID of an obdfilter disk (for example, if you mix up the cables on your OST devices or the SCSI bus numbering suddenly changes and the SCSI devices get new names), use debugfs to get the last_rcvd file.</para>
 503       </section>
 504       <section remap="h3">
 505         <title><anchor xml:id="dbdoclet.50438274_pgfId-1296010" xreflabel=""/>28.2.8 Printing Debug Messages to the Console</title>
 506         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296011" xreflabel=""/>To dump debug messages to the console (/var/log/messages), set the corresponding debug mask in the printk flag:</para>
 507         <screen><anchor xml:id="dbdoclet.50438274_pgfId-1296012" xreflabel=""/>sysctl -w lnet.printk=-1
 508 </screen>
 509         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296013" xreflabel=""/>This slows down the system dramatically. It is also possible to selectively enable or disable this capability for particular flags using:</para>
 510         <screen><anchor xml:id="dbdoclet.50438274_pgfId-1296014" xreflabel=""/>sysctl -w lnet.printk=+vfstrace
 511 <anchor xml:id="dbdoclet.50438274_pgfId-1296015" xreflabel=""/>sysctl -w lnet.printk=-vfstrace
 512 </screen>
 513         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296016" xreflabel=""/>It is possible to disable warning, error , and console messages, though it is strongly recommended to have something like lctldebug_daemon runing to capture this data to a local file system for debugging purposes.</para>
 514       </section>
 515       <section remap="h3">
 516         <title><anchor xml:id="dbdoclet.50438274_pgfId-1296018" xreflabel=""/>28.2.9 Tracing <anchor xml:id="dbdoclet.50438274_marker-1296017" xreflabel=""/>Lock Traffic</title>
 517         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296019" xreflabel=""/>Lustre has a specific debug type category for tracing lock traffic. Use:</para>
 518         <screen><anchor xml:id="dbdoclet.50438274_pgfId-1296020" xreflabel=""/>lctl&gt; filter all_types
 519 <anchor xml:id="dbdoclet.50438274_pgfId-1296021" xreflabel=""/>lctl&gt; show dlmtrace
 520 <anchor xml:id="dbdoclet.50438274_pgfId-1296022" xreflabel=""/>lctl&gt; debug_kernel [filename]
 521 </screen>
 522       </section>
 523     </section>
 524     <section xml:id="dbdoclet.50438274_80443">
 525       <title>28.3 Lustre Debugging for Developers</title>
 526       <para><anchor xml:id="dbdoclet.50438274_pgfId-1296025" xreflabel=""/>The procedures in this section may be useful to developers debugging Lustre code.</para>
 527       <section remap="h3">
 528         <title><anchor xml:id="dbdoclet.50438274_pgfId-1296027" xreflabel=""/>28.3.1 Adding Debugging to the <anchor xml:id="dbdoclet.50438274_marker-1296026" xreflabel=""/>Lustre Source Code</title>
 529         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296028" xreflabel=""/>The debugging infrastructure provides a number of macros that can be used in Lustre source code to aid in debugging or reporting serious errors.</para>
 530         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296029" xreflabel=""/>To use these macros, you will need to set the DEBUG_SUBSYSTEM variable at the top of the file as shown below:</para>
 531         <screen><anchor xml:id="dbdoclet.50438274_pgfId-1296030" xreflabel=""/>#define DEBUG_SUBSYSTEM S_PORTALS
 532 </screen>
 533         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296098" xreflabel=""/>A list of available macros with descritions is provided in the table below.</para>
 534         <informaltable frame="all">
 535           <tgroup cols="2">
 536             <colspec colname="c1" colwidth="50*"/>
 537             <colspec colname="c2" colwidth="50*"/>
 538             <thead>
 539               <row>
 540                 <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438274_pgfId-1296033" xreflabel=""/>Macro</emphasis></para></entry>
 541                 <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438274_pgfId-1296035" xreflabel=""/>Description</emphasis></para></entry>
 542               </row>
 543             </thead>
 544             <tbody>
 545               <row>
 546                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296037" xreflabel=""/><emphasis role="bold">LBUG</emphasis></para></entry>
 547                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296039" xreflabel=""/>A panic-style assertion in the kernel which causes Lustre to dump its circular log to the /tmp/lustre-log file. This file can be retrieved after a reboot. LBUG freezes the thread to allow capture of the panic stack. A system reboot is needed to clear the thread.</para></entry>
 548               </row>
 549               <row>
 550                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296041" xreflabel=""/><emphasis role="bold">LASSERT</emphasis></para></entry>
 551                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296043" xreflabel=""/>Validates a given expression as true, otherwise calls LBUG. The failed expression is printed on the console, although the values that make up the expression are not printed.</para></entry>
 552               </row>
 553               <row>
 554                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296045" xreflabel=""/><emphasis role="bold">LASSERTF</emphasis></para></entry>
 555                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296047" xreflabel=""/>Similar to LASSERT but allows a free-format message to be printed, like printf/printk.</para></entry>
 556               </row>
 557               <row>
 558                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296049" xreflabel=""/><emphasis role="bold">CDEBUG</emphasis></para></entry>
 559                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296051" xreflabel=""/>The basic, most commonly used debug macro that takes just one more argument than standard printf - the debug type. This message adds to the debug log with the debug mask set accordingly. Later, when a user retrieves the log for troubleshooting, they can filter based on this type.</para><para><anchor xml:id="dbdoclet.50438274_pgfId-1296052" xreflabel=""/>CDEBUG(D_INFO, &quot;This is my debug message: the number is %d\n&quot;, number).</para></entry>
 560               </row>
 561               <row>
 562                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296054" xreflabel=""/><emphasis role="bold">CERROR</emphasis></para></entry>
 563                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296056" xreflabel=""/>Behaves similarly to CDEBUG, but unconditionally prints the message in the debug log and to the console. This is appropriate for serious errors or fatal conditions:</para><para><anchor xml:id="dbdoclet.50438274_pgfId-1296057" xreflabel=""/>CERROR(&quot;Something very bad has happened, and the return code is %d.\n&quot;, rc);</para></entry>
 564               </row>
 565               <row>
 566                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296059" xreflabel=""/><emphasis role="bold">ENTRY and EXIT</emphasis></para></entry>
 567                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296061" xreflabel=""/>Add messages to aid in call tracing (takes no arguments). When using these macros, cover all exit conditions to avoid confusion when the debug log reports that a function was entered, but never exited.</para></entry>
 568               </row>
 569               <row>
 570                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296063" xreflabel=""/><emphasis role="bold">LDLM_DEBUG and LDLM_DEBUG_NOLOCK</emphasis></para></entry>
 571                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296065" xreflabel=""/>Used when tracing MDS and VFS operations for locking. These macros build a thin trace that shows the protocol exchanges between nodes.</para></entry>
 572               </row>
 573               <row>
 574                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296067" xreflabel=""/><emphasis role="bold">DEBUG_REQ</emphasis></para></entry>
 575                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296069" xreflabel=""/>Prints information about the given ptlrpc_request structure.</para></entry>
 576               </row>
 577               <row>
 578                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296071" xreflabel=""/><emphasis role="bold">OBD_FAIL_CHECK</emphasis></para></entry>
 579                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296073" xreflabel=""/>Allows insertion of failure points into the Lustre code. This is useful to generate regression tests that can hit a very specific sequence of events. This works in conjunction with &quot;sysctl -w lustre.fail_loc={fail_loc}&quot; to set a specific failure point for which a given OBD_FAIL_CHECK will test.</para></entry>
 580               </row>
 581               <row>
 582                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296075" xreflabel=""/><emphasis role="bold">OBD_FAIL_TIMEOUT</emphasis></para></entry>
 583                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296077" xreflabel=""/>Similar to OBD_FAIL_CHECK. Useful to simulate hung, blocked or busy processes or network devices. If the given fail_loc is hit, OBD_FAIL_TIMEOUT waits for the specified number of seconds.</para></entry>
 584               </row>
 585               <row>
 586                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296079" xreflabel=""/><emphasis role="bold">OBD_RACE</emphasis></para></entry>
 587                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296081" xreflabel=""/>Similar to OBD_FAIL_CHECK. Useful to have multiple processes execute the same code concurrently to provoke locking races. The first process to hit OBD_RACE sleeps until a second process hits OBD_RACE, then both processes continue.</para></entry>
 588               </row>
 589               <row>
 590                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296083" xreflabel=""/><emphasis role="bold">OBD_FAIL_ONCE</emphasis></para></entry>
 591                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296085" xreflabel=""/>A flag set on a lustre.fail_loc breakpoint to cause the OBD_FAIL_CHECK condition to be hit only one time. Otherwise, a fail_loc is permanent until it is cleared with &quot;sysctl -w lustre.fail_loc=0&quot;.</para></entry>
 592               </row>
 593               <row>
 594                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296087" xreflabel=""/><emphasis role="bold">OBD_FAIL_RAND</emphasis></para></entry>
 595                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296089" xreflabel=""/>Has OBD_FAIL_CHECK fail randomly; on average every (1 / lustre.fail_val) times.</para></entry>
 596               </row>
 597               <row>
 598                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296091" xreflabel=""/><emphasis role="bold">OBD_FAIL_SKIP</emphasis></para></entry>
 599                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296093" xreflabel=""/>Has OBD_FAIL_CHECK succeed lustre.fail_val times, and then fail permanently or once with OBD_FAIL_ONCE.</para></entry>
 600               </row>
 601               <row>
 602                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296095" xreflabel=""/><emphasis role="bold">OBD_FAIL_SOME</emphasis></para></entry>
 603                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296097" xreflabel=""/>Has OBD_FAIL_CHECK fail lustre.fail_val times, and then succeed.</para></entry>
 604               </row>
 605             </tbody>
 606           </tgroup>
 607         </informaltable>
 608       </section>
 609       <section remap="h3">
 610         <title><anchor xml:id="dbdoclet.50438274_pgfId-1296100" xreflabel=""/>28.3.2 Accessing a Ptlrpc <anchor xml:id="dbdoclet.50438274_marker-1296099" xreflabel=""/>Request History</title>
 611         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296101" xreflabel=""/>Each service maintains a request history, which can be useful for first occurrence troubleshooting.</para>
 612         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296102" xreflabel=""/>Ptlrpc is an RPC protocol layered on LNET that deals with stateful servers and has semantics and built-in support for recovery.</para>
 613         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296103" xreflabel=""/>A prlrpc request history works as follows:</para>
 614         <orderedlist><listitem>
 615         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296104" xreflabel=""/>Request_in_callback() adds the new request to the service&apos;s request history.</para>
 616     </listitem><listitem>
 617         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296105" xreflabel=""/>When a request buffer becomes idle, it is added to the service&apos;s request buffer history list.</para>
 618     </listitem><listitem>
 619         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296106" xreflabel=""/>Buffers are culled from the service&apos;s request buffer history if it has grown above</para>
 620         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296107" xreflabel=""/>req_buffer_history_max and its reqs are removed from the service&apos;s request history.</para>
 621     </listitem></orderedlist>
 622         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296108" xreflabel=""/>Request history is accessed and controlled using the following /proc files under the service directory:</para>
 623         <itemizedlist><listitem>
 624             <para><anchor xml:id="dbdoclet.50438274_pgfId-1296109" xreflabel=""/>req_buffer_history_len</para>
 625           </listitem>
 626
 627 </itemizedlist>
 628         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296110" xreflabel=""/>Number of request buffers currently in the history</para>
 629         <itemizedlist><listitem>
 630             <para><anchor xml:id="dbdoclet.50438274_pgfId-1296111" xreflabel=""/>req_buffer_history_max</para>
 631           </listitem>
 632
 633 </itemizedlist>
 634         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296112" xreflabel=""/>Maximum number of request buffers to keep</para>
 635         <itemizedlist><listitem>
 636             <para><anchor xml:id="dbdoclet.50438274_pgfId-1296113" xreflabel=""/>req_history</para>
 637           </listitem>
 638
 639 </itemizedlist>
 640         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296114" xreflabel=""/>The request history</para>
 641         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296115" xreflabel=""/>Requests in the history include &quot;live&quot; requests that are currently being handled. Each line in req_history looks like:</para>
 642         <screen><anchor xml:id="dbdoclet.50438274_pgfId-1296151" xreflabel=""/>&lt;seq&gt;:&lt;target NID&gt;:&lt;client ID&gt;:&lt;xid&gt;:&lt;length&gt;:&lt;phase&gt; &lt;svc specific&gt;
 643 </screen>
 644         <informaltable frame="all">
 645           <tgroup cols="2">
 646             <colspec colname="c1" colwidth="50*"/>
 647             <colspec colname="c2" colwidth="50*"/>
 648             <thead>
 649               <row>
 650                 <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438274_pgfId-1296118" xreflabel=""/>Parameter</emphasis></para></entry>
 651                 <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438274_pgfId-1296120" xreflabel=""/>Description</emphasis></para></entry>
 652               </row>
 653             </thead>
 654             <tbody>
 655               <row>
 656                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296122" xreflabel=""/><emphasis role="bold">seq</emphasis></para></entry>
 657                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296124" xreflabel=""/>Request sequence number</para></entry>
 658               </row>
 659               <row>
 660                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296126" xreflabel=""/><emphasis role="bold">target NID</emphasis></para></entry>
 661                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296128" xreflabel=""/>Destination NID of the incoming request</para></entry>
 662               </row>
 663               <row>
 664                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296130" xreflabel=""/><emphasis role="bold">client ID</emphasis></para></entry>
 665                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296132" xreflabel=""/>Client PID and NID</para></entry>
 666               </row>
 667               <row>
 668                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296134" xreflabel=""/><emphasis role="bold">xid</emphasis></para></entry>
 669                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296136" xreflabel=""/>rq_xid</para></entry>
 670               </row>
 671               <row>
 672                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296138" xreflabel=""/><emphasis role="bold">length</emphasis></para></entry>
 673                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296140" xreflabel=""/>Size of the request message</para></entry>
 674               </row>
 675               <row>
 676                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296142" xreflabel=""/><emphasis role="bold">phase</emphasis></para></entry>
 677                 <entry><para><itemizedlist><listitem>
 678                         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296144" xreflabel=""/> New (waiting to be handled or could not be unpacked)</para>
 679                       </listitem>
 680 <listitem>
 681                         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296145" xreflabel=""/> Interpret (unpacked or being handled)</para>
 682                       </listitem>
 683 <listitem>
 684                         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296146" xreflabel=""/> Complete (handled)</para>
 685                       </listitem>
 686 </itemizedlist></para></entry>
 687               </row>
 688               <row>
 689                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296148" xreflabel=""/><emphasis role="bold">svc specific</emphasis></para></entry>
 690                 <entry><para> <anchor xml:id="dbdoclet.50438274_pgfId-1296150" xreflabel=""/>Service-specific request printout. Currently, the only service that does this is the OST (which prints the opcode if the message has been unpacked successfully</para></entry>
 691               </row>
 692             </tbody>
 693           </tgroup>
 694         </informaltable>
 695       </section>
 696       <section remap="h3">
 697         <title><anchor xml:id="dbdoclet.50438274_pgfId-1296154" xreflabel=""/>28.3.3 Finding Memory <anchor xml:id="dbdoclet.50438274_marker-1296153" xreflabel=""/>Leaks Using leak_finder.pl</title>
 698         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296155" xreflabel=""/>Memory leaks can occur in code when memory has been allocated and then not freed once it is no longer required. The leak_finder.pl program provides a way to find memory leaks.</para>
 699         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296156" xreflabel=""/>Before running this program, you must turn on debugging to collect all malloc and free entries. Run:</para>
 700         <screen><anchor xml:id="dbdoclet.50438274_pgfId-1296157" xreflabel=""/>sysctl -w lnet.debug=+malloc
 701 </screen>
 702         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296158" xreflabel=""/>Then complete the following steps:</para>
 703         <orderedlist><listitem>
 704         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296161" xreflabel=""/> 1. Dump the log into a user-specified log file using lctl (see <link xl:href="LustreDebugging.html#50438274_62472">Using the lctl Tool to View Debug Messages</link>).</para>
 705         </listitem><listitem>
 706         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296163" xreflabel=""/> 2. Run the leak finder on the newly-created log dump:</para>
 707         <screen><anchor xml:id="dbdoclet.50438274_pgfId-1296164" xreflabel=""/>perl leak_finder.pl &lt;ascii-logname&gt;
 708 </screen>
 709         </listitem></orderedlist>
 710         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296165" xreflabel=""/>The output is:</para>
 711         <screen><anchor xml:id="dbdoclet.50438274_pgfId-1296166" xreflabel=""/>malloced 8bytes at a3116744 (called pathcopy)
 712 <anchor xml:id="dbdoclet.50438274_pgfId-1296167" xreflabel=""/>(lprocfs_status.c:lprocfs_add_vars:80)
 713 <anchor xml:id="dbdoclet.50438274_pgfId-1296168" xreflabel=""/>freed 8bytes at a3116744 (called pathcopy)
 714 <anchor xml:id="dbdoclet.50438274_pgfId-1296169" xreflabel=""/>(lprocfs_status.c:lprocfs_add_vars:80)
 715 </screen>
 716         <para><anchor xml:id="dbdoclet.50438274_pgfId-1296170" xreflabel=""/>The tool displays the following output to show the leaks found:</para>
 717         <screen><anchor xml:id="dbdoclet.50438274_pgfId-1296171" xreflabel=""/>Leak:32bytes allocated at a23a8fc(service.c:ptlrpc_init_svc:144,debug file \
 718 line 241)
 719 </screen>
 720       </section>
 721   </section>
 722 </chapter>