LNETSelfTest.xml

   1 <?xml version='1.0' encoding='UTF-8'?>
   2 <!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lnetselftest">
   3   <title xml:id="lnetselftest.title">Testing Lustre Network Performance (LNET Self-Test)</title>
   4   <para><anchor xml:id="dbdoclet.50438223_36273" xreflabel=""/>This chapter describes the LNET self-test, which is used by site administrators to confirm that Lustre Networking (LNET) has been properly installed and configured, and that underlying network software and hardware are performing according to expectations. The chapter includes:</para>
   5   <itemizedlist>
   6     <listitem>
   7       <para><xref linkend="dbdoclet.50438223_91742"/></para>
   8     </listitem>
   9     <listitem>
  10       <para><xref linkend="dbdoclet.50438223_48138"/></para>
  11     </listitem>
  12     <listitem>
  13       <para><xref linkend="dbdoclet.50438223_27277"/></para>
  14     </listitem>
  15   </itemizedlist>
  16   <section xml:id="dbdoclet.50438223_91742">
  17     <title><indexterm><primary>LNET</primary><secondary>self-test</secondary></indexterm>
  18 LNET Self-Test Overview</title>
  19     <para>LNET self-test is a kernel module that runs over LNET and the Lustre network drivers (LNDs. It is designed to:</para>
  20     <itemizedlist>
  21       <listitem>
  22         <para>Test the connection ability of the Lustre network</para>
  23       </listitem>
  24       <listitem>
  25         <para>Run regression tests of the Lustre network</para>
  26       </listitem>
  27       <listitem>
  28         <para>Test performance of the Lustre network</para>
  29       </listitem>
  30     </itemizedlist>
  31     <para>After you have obtained performance results for your Lustre network, refer to <xref linkend="lustretuning"/> for information about parameters that can be used to tune LNET for optimum performance.</para>
  32     <note>
  33       <para>Apart from the performance impact, LNET self-test is invisible to Lustre.</para>
  34     </note>
  35     <para>An LNET self-test cluster includes two types of nodes:</para>
  36     <itemizedlist>
  37       <listitem>
  38         <para><emphasis role="bold">Console node</emphasis>  - A node used to control and monitor an LNET self-test cluster. The console node serves as the user interface of the LNET self-test system and can be any node in the test cluster. All self-test commands are entered from the console node. From the console node, a user can control and monitor the status of the entire LNET self-test cluster (session). The console node is exclusive in that a user cannot control two different sessions from one console node.</para>
  39       </listitem>
  40       <listitem>
  41         <para><emphasis role="bold">Test nodes</emphasis>  - The nodes on which the tests are run. Test nodes are controlled by the user from the console node; the user does not need to log into them directly.</para>
  42       </listitem>
  43     </itemizedlist>
  44     <para>LNET self-test has two user utilities:</para>
  45     <itemizedlist>
  46       <listitem>
  47         <para><emphasis role="bold">
  48             <literal>lst</literal>
  49           </emphasis>  - The user interface for the self-test console (run on the console node). It provides a list of commands to control the entire test system, including commands to create a session, create test groups, etc.</para>
  50       </listitem>
  51       <listitem>
  52         <para><emphasis role="bold">
  53             <literal>lstclient</literal>
  54           </emphasis>  - The userspace LNET self-test program (run on a test node). The lstclient utility is linked with userspace LNDs and LNET. This utility is not needed if only kernel space LNET and LNDs are used.</para>
  55       </listitem>
  56     </itemizedlist>
  57     <note>
  58       <para>Test nodes can be in either kernel or userspace. A console user can invite a kernel test node to join the test session by running lstadd_groupNID, but the console user cannot actively add a userspace test node to the test-session. However, the console user can passively accept a test node to the test session while the test node is running lstclient to connect to the console.</para>
  59     </note>
  60     <section remap="h3">
  61       <title>Prerequisites</title>
  62       <para>To run LNET self-test, these modules must be loaded:</para>
  63       <itemizedlist>
  64         <listitem>
  65           <para><literal>libcfs</literal></para>
  66         </listitem>
  67         <listitem>
  68           <para><literal>net</literal></para>
  69         </listitem>
  70         <listitem>
  71           <para><literal>lnet_selftest</literal></para>
  72         </listitem>
  73         <listitem>
  74           <para> One of the <literal>klnds</literal> (i.e, <literal>ksocklnd</literal>, <literal>ko2iblnd</literal>...) as needed by your network configuration</para>
  75         </listitem>
  76       </itemizedlist>
  77       <para>To load the required modules, run:</para>
  78       <screen>modprobe lnet_selftest </screen>
  79       <para>This command recursively loads the modules on which LNET self-test depends.</para>
  80       <note>
  81         <para>While the console and test nodes require all the prerequisite modules to be loaded, userspace test nodes do not require these modules.</para>
  82       </note>
  83     </section>
  84   </section>
  85   <section xml:id="dbdoclet.50438223_48138">
  86     <title>Using LNET Self-Test</title>
  87     <para>This section describes how to create and run an LNET self-test. The examples shown are for a test that simulates the traffic pattern of a set of Lustre servers on a TCP network accessed by Lustre clients on an InfiniBand network connected via LNET routers. In this example, half the clients are reading and half the clients are writing.</para>
  88     <section remap="h3">
  89       <title>Creating a Session</title>
  90       <para>A <emphasis>session</emphasis> is a set of processes that run on a test node. Only one session can be run at a time on a test node to ensure that the session has exclusive use of the node. The console node is used to create, change or destroy a session (new_session, end_session, show_session). For more about session parameters, see <xref linkend="dbdoclet.50438223_91247"/>.</para>
  91       <para>Almost all operations should be performed within the context of a session. From the console node, a user can only operate nodes in his own session. If a session ends, the session context in all test nodes is stopped.</para>
  92       <para>The following commands set the LST_SESSION environment variable to identify the session on the console node and create a session called read_write:</para>
  93       <screen>export LST_SESSION=$$
  94 lst new_session read_write</screen>
  95     </section>
  96     <section remap="h3">
  97       <title>Setting Up Groups</title>
  98       <para>A <emphasis>group</emphasis> is a named collection of nodes. Any number of groups can exist in a single LNET self-test session. Group membership is not restricted in that a node can be included in any number of groups.</para>
  99       <para>Each node in a group has a rank, determined by the order in which it was added to the group. The rank is used to establish test traffic patterns.</para>
 100       <para>A user can only control nodes in his/her session. To allocate nodes to the session, the user needs to add nodes to a group (of the session). All nodes in a group can be referenced by the group name. A node can be allocated to multiple groups of a session.</para>
 101       <para>In the following example, three groups are established:</para>
 102       <screen>lst add_group servers 192.168.10.[8,10,12-16]@tcp
 103 lst add_group readers 192.168.1.[1-253/2]@o2ib
 104 lst add_group writers 192.168.1.[2-254/2]@o2ib</screen>
 105       <para>These three groups include:</para>
 106       <itemizedlist>
 107         <listitem>
 108           <para>Nodes that will function as &apos;servers&apos; to be accessed by &apos;clients&apos; during the LNET self-test session</para>
 109         </listitem>
 110         <listitem>
 111           <para>Nodes that will function as &apos;clients&apos; that will simulate <emphasis>reading</emphasis> data from the &apos;servers&apos;</para>
 112         </listitem>
 113         <listitem>
 114           <para>Nodes that will function as &apos;clients&apos; that will simulate <emphasis>writing</emphasis> data to the &apos;servers&apos;</para>
 115         </listitem>
 116       </itemizedlist>
 117       <note>
 118         <para>A console user can associate kernel space test nodes with the session by running lst <literal>add_group</literal> <literal>NIDs</literal>, but a userspace test node cannot be actively added to the session. However, the console user can passively &quot;accept&quot; a test node to associate with a test session while the test node running <literal>lstclient</literal> connects to the console node, i.e: <literal>lstclient --sesid CONSOLE_NID --group NAME</literal>).</para>
 119       </note>
 120     </section>
 121     <section xml:id="dbdoclet.50438223_42848">
 122       <title>Defining and Running the Tests</title>
 123       <para>A <emphasis>test</emphasis> generates a network load between two groups of nodes, a source group identified using the <literal>--from</literal> parameter and a target group identified using the <literal>--to</literal> parameter. When a test is running, each node in the <literal>--from<replaceable>&lt;group&gt;</replaceable></literal> simulates a client by sending requests to nodes in the <literal>--to<replaceable>&lt;group&gt;</replaceable></literal>, which are simulating a set of servers, and then receives responses in return. This activity is designed to mimic Lustre RPC traffic.</para>
 124       <para>A <emphasis>batch</emphasis> is a collection of tests that are started and stopped together and run in parallel. A test must always be run as part of a batch, even if it is just a single test. Users can only run or stop a test batch, not individual tests.</para>
 125       <para>Tests in a batch are non-destructive to the file system, and can be run in a normal Lustre environment (provided the performance impact is acceptable).</para>
 126       <para>A simple batch might contain a single test, for example, to determine whether the network bandwidth presents an I/O bottleneck. In this example, the <literal>--to<replaceable>&lt;group&gt;</replaceable></literal> could be comprised of Lustre OSSs and <literal>--from<replaceable>&lt;group&gt;</replaceable></literal> the compute nodes. A second test could be added to perform pings from a login node to the MDS to see how checkpointing affects the <literal>ls -l</literal> process.</para>
 127       <para>Two types of tests are available:</para>
 128       <itemizedlist>
 129         <listitem>
 130           <para><emphasis role="bold"><literal>ping</literal> -</emphasis>  A <literal>ping</literal> generates a short request message, which results in a short response. Pings are useful to determine latency and small message overhead and to simulate Lustre metadata traffic.</para>
 131         </listitem>
 132         <listitem>
 133           <para><emphasis role="bold"><literal>brw</literal> -</emphasis>  In a <literal>brw</literal> (&apos;bulk read write&apos;) test, data is transferred from the target to the source (<literal>brwread</literal>) or data is transferred from the source to the target (<literal>brwwrite</literal>). The size of the bulk transfer is set using the <literal>size</literal> parameter. A brw test is useful to determine network bandwidth and to simulate Lustre I/O traffic.</para>
 134         </listitem>
 135       </itemizedlist>
 136       <para>In the example below, a batch is created called <literal>bulk_rw</literal>. Then two <literal>brw</literal> tests are added. In the first test, 1M of data is sent from the servers to the clients as a simulated read operation with a simple data validation check. In the second test, 4K of data is sent from the clients to the servers as a simulated write operation with a full data validation check.</para>
 137       <screen>lst add_batch bulk_rw
 138 lst add_test --batch bulk_rw --from readers --to servers \
 139   brw read check=simple size=1M
 140 lst add_test --batch bulk_rw --from writers --to servers \
 141   brw write check=full size=4K</screen>
 142       <para>The traffic pattern and test intensity is determined by several properties such as test type, distribution of test nodes, concurrency of test, and RDMA operation type. For more details, see <xref linkend="dbdoclet.50438223_36860"/>.</para>
 143     </section>
 144     <section remap="h3">
 145       <title>Sample Script</title>
 146       <para>This sample LNET self-test script simulates the traffic pattern of a set of Lustre servers on a TCP network, accessed by Lustre clients on an InfiniBand network (connected via LNET routers). In this example, half the clients are reading and half the clients are writing.</para>
 147       <para>Run this script on the console node:</para>
 148       <screen>#!/bin/bash
 149 export LST_SESSION=$$
 150 lst new_session read/write
 151 lst add_group servers 192.168.10.[8,10,12-16]@tcp
 152 lst add_group readers 192.168.1.[1-253/2]@o2ib
 153 lst add_group writers 192.168.1.[2-254/2]@o2ib
 154 lst add_batch bulk_rw
 155 lst add_test --batch bulk_rw --from readers --to servers \
 156 brw read check=simple size=1M
 157 lst add_test --batch bulk_rw --from writers --to servers \
 158 brw write check=full size=4K
 159 # start running
 160 lst run bulk_rw
 161 # display server stats for 30 seconds
 162 lst stat servers &amp; sleep 30; kill $!
 163 # tear down
 164 lst end_session</screen>
 165       <note>
 166         <para>This script can be easily adapted to pass the group NIDs by shell variables or command line arguments (making it good for general-purpose use).</para>
 167       </note>
 168     </section>
 169   </section>
 170   <section xml:id="dbdoclet.50438223_27277">
 171     <title>LNET Self-Test Command Reference</title>
 172     <para>The LNET self-test (<literal>lst</literal>) utility is used to issue LNET self-test commands. The <literal>lst</literal> utility takes a number of command line arguments. The first argument is the command name and subsequent arguments are command-specific.</para>
 173     <section xml:id="dbdoclet.50438223_91247">
 174       <title>Session Commands</title>
 175       <para>This section describes <literal>lst</literal> session commands.</para>
 176       <para><emphasis role="bold">
 177           <literal>LST_SESSION</literal>
 178         </emphasis></para>
 179       <para>The <literal>lst</literal> utility uses the <literal>LST_SESSION</literal> environmental variable to identify the session locally on the self-test console node. This should be a numeric value that uniquely identifies all session processes on the node. It is convenient to set this to the process ID of the shell both for interactive use and in shell scripts. Almost all <literal>lst</literal> commands require <literal>LST_SESSION</literal> to be set.</para>
 180       <para>Example:</para>
 181       <screen>export LST_SESSION=$$</screen>
 182       <para><emphasis role="bold">
 183           <literal>new_session [--timeout SECONDS] [--force] NAME</literal>
 184         </emphasis></para>
 185       <para>Creates a new session.</para>
 186       <informaltable frame="all">
 187         <tgroup cols="2">
 188           <colspec colname="c1" colwidth="50*"/>
 189           <colspec colname="c2" colwidth="50*"/>
 190           <thead>
 191             <row>
 192               <entry>
 193                 <para><emphasis role="bold">Parameter</emphasis></para>
 194               </entry>
 195               <entry>
 196                 <para><emphasis role="bold">Description</emphasis></para>
 197               </entry>
 198             </row>
 199           </thead>
 200           <tbody>
 201             <row>
 202               <entry>
 203                 <para><literal>--timeout<replaceable>&lt;seconds&gt;</replaceable></literal></para>
 204               </entry>
 205               <entry>
 206                 <para>Console timeout value of the session. The session ends automatically if it remains idle (i.e., no commands are issued) for this period.</para>
 207               </entry>
 208             </row>
 209             <row>
 210               <entry>
 211                 <para><literal>--force</literal></para>
 212               </entry>
 213               <entry>
 214                 <para>Ends conflicting sessions. This determines who &apos;wins&apos; when one session conflicts with another. For example, if there is already an active session on this node, then the attempt to create a new session fails unless the -force flag is specified. If the -force flag is specified, then the active session is ended. Similarly, if a session attempts to add a node that is already &apos;owned&apos; by another session, the -force flag allows this session to &apos;steal&apos; the node.</para>
 215               </entry>
 216             </row>
 217             <row>
 218               <entry>
 219                 <para> <emphasis>
 220                     <literal>&lt;name&gt;</literal>
 221                   </emphasis></para>
 222               </entry>
 223               <entry>
 224                 <para>A human-readable string to print when listing sessions or reporting session conflicts.</para>
 225               </entry>
 226             </row>
 227           </tbody>
 228         </tgroup>
 229       </informaltable>
 230       <para><emphasis role="bold">Example:</emphasis></para>
 231       <screen>$ lst new_session --force read_write</screen>
 232       <para><literal>
 233           <replaceable role="bold">end_session</replaceable>
 234         </literal></para>
 235       <para>Stops all operations and tests in the current session and clears the session&apos;s status.</para>
 236       <screen>$ lst end_session</screen>
 237       <para><literal>
 238           <replaceable role="bold">show_session</replaceable>
 239         </literal></para>
 240       <para>Shows the session information. This command prints information about the current session. It does not require LST_SESSION to be defined in the process environment.</para>
 241       <screen>$ lst show_session</screen>
 242     </section>
 243     <section remap="h3">
 244       <title>Group Commands</title>
 245       <para>This section describes <literal>lst</literal> group commands.</para>
 246       <para><literal>
 247           <replaceable role="bold">add_group</replaceable>
 248           <replaceable> &lt;name&gt; &lt;NIDS&gt; [&lt;NIDs&gt;...]</replaceable>
 249         </literal></para>
 250       <para>Creates the group and adds a list of test nodes to the group.</para>
 251       <informaltable frame="all">
 252         <tgroup cols="2">
 253           <colspec colname="c1" colwidth="50*"/>
 254           <colspec colname="c2" colwidth="50*"/>
 255           <thead>
 256             <row>
 257               <entry>
 258                 <para><emphasis role="bold">Parameter</emphasis></para>
 259               </entry>
 260               <entry>
 261                 <para><emphasis role="bold">Description</emphasis></para>
 262               </entry>
 263             </row>
 264           </thead>
 265           <tbody>
 266             <row>
 267               <entry>
 268                 <para> <literal>
 269                     <replaceable>&lt;name&gt;</replaceable>
 270                   </literal></para>
 271               </entry>
 272               <entry>
 273                 <para>Name of the group.</para>
 274               </entry>
 275             </row>
 276             <row>
 277               <entry>
 278                 <para> <literal>
 279                     <replaceable>&lt;NIDs&gt;</replaceable>
 280                   </literal></para>
 281               </entry>
 282               <entry>
 283                 <para>A string that may be expanded to include one or more LNET NIDs.</para>
 284               </entry>
 285             </row>
 286           </tbody>
 287         </tgroup>
 288       </informaltable>
 289       <para><emphasis role="bold">Example:</emphasis></para>
 290       <screen>$ lst add_group servers 192.168.10.[35,40-45]@tcp
 291 $ lst add_group clients 192.168.1.[10-100]@tcp 192.168.[2,4].\
 292   [10-20]@tcp</screen>
 293       <para><literal>
 294           <replaceable role="bold">update_group</replaceable>
 295           <replaceable>&lt;name&gt;</replaceable>
 296           <replaceable role="bold">[--refresh] [--clean</replaceable>
 297           <replaceable>&lt;status&gt;</replaceable>
 298           <replaceable role="bold">] [--remove</replaceable>
 299           <replaceable>&lt;NIDs&gt;</replaceable>
 300           <replaceable role="bold">]</replaceable>
 301         </literal></para>
 302       <para>Updates the state of nodes in a group or adjusts a group&apos;s membership. This command is useful if some nodes have crashed and should be excluded from the group.</para>
 303       <informaltable frame="all">
 304         <tgroup cols="3">
 305           <colspec colname="c1" colwidth="33*"/>
 306           <colspec colname="c2" colwidth="33*"/>
 307           <colspec colname="c3" colwidth="33*"/>
 308           <thead>
 309             <row>
 310               <entry>
 311                 <para><emphasis role="bold">Parameter</emphasis></para>
 312               </entry>
 313               <entry nameend="c3" namest="c2">
 314                 <para><emphasis role="bold">Description</emphasis></para>
 315               </entry>
 316             </row>
 317           </thead>
 318           <tbody>
 319             <row>
 320               <entry>
 321                 <para>
 322                 <literal> --refresh </literal>
 323             </para>
 324               </entry>
 325               <entry nameend="c3" namest="c2">
 326                 <para> Refreshes the state of all inactive nodes in the group.</para>
 327               </entry>
 328             </row>
 329             <row>
 330               <entry>
 331                 <para> <literal>--clean<replaceable>&lt;status&gt;</replaceable></literal></para>
 332               </entry>
 333               <entry nameend="c3" namest="c2">
 334                 <para> Removes nodes with a specified status from the group. Status may be:</para>
 335               </entry>
 336             </row>
 337             <row>
 338               <entry>
 339                 <para> </para>
 340               </entry>
 341               <entry>
 342                 <para> active</para>
 343               </entry>
 344               <entry>
 345                 <para> The node is in the current session.</para>
 346               </entry>
 347             </row>
 348             <row>
 349               <entry>
 350                 <para> </para>
 351               </entry>
 352               <entry>
 353                 <para> busy</para>
 354               </entry>
 355               <entry>
 356                 <para> The node is now owned by another session.</para>
 357               </entry>
 358             </row>
 359             <row>
 360               <entry>
 361                 <para> </para>
 362               </entry>
 363               <entry>
 364                 <para> down</para>
 365               </entry>
 366               <entry>
 367                 <para> The node has been marked down.</para>
 368               </entry>
 369             </row>
 370             <row>
 371               <entry>
 372                 <para> </para>
 373               </entry>
 374               <entry>
 375                 <para> unknown</para>
 376               </entry>
 377               <entry>
 378                 <para> The node&apos;™s status has yet to be determined.</para>
 379               </entry>
 380             </row>
 381             <row>
 382               <entry>
 383                 <para> </para>
 384               </entry>
 385               <entry>
 386                 <para> invalid</para>
 387               </entry>
 388               <entry>
 389                 <para> Any state but active.</para>
 390               </entry>
 391             </row>
 392             <row>
 393               <entry>
 394                 <para> <literal>--remove<replaceable>&lt;NIDs&gt;</replaceable></literal></para>
 395               </entry>
 396               <entry nameend="c3" namest="c2">
 397                 <para> Removes specified nodes from the group.</para>
 398               </entry>
 399             </row>
 400           </tbody>
 401         </tgroup>
 402       </informaltable>
 403       <para><emphasis role="bold">Example:</emphasis></para>
 404       <screen>$ lst update_group clients --refresh
 405 $ lst update_group clients --clean busy
 406 $ lst update_group clients --clean invalid // \
 407   invalid == busy || down || unknown
 408 $ lst update_group clients --remove \192.168.1.[10-20]@tcp</screen>
 409       <para><literal>
 410           <replaceable role="bold">list_group [</replaceable>
 411           <replaceable>&lt;name&gt;</replaceable>
 412           <replaceable role="bold">] [--active] [--busy] [--down] [--unknown] [--all]</replaceable>
 413         </literal></para>
 414       <para>Prints information about a group or lists all groups in the current session if no group is specified.</para>
 415       <informaltable frame="all">
 416         <tgroup cols="2">
 417           <colspec colname="c1" colwidth="50*"/>
 418           <colspec colname="c2" colwidth="50*"/>
 419           <thead>
 420             <row>
 421               <entry>
 422                 <para><emphasis role="bold">Parameter</emphasis></para>
 423               </entry>
 424               <entry>
 425                 <para><emphasis role="bold">Description</emphasis></para>
 426               </entry>
 427             </row>
 428           </thead>
 429           <tbody>
 430             <row>
 431               <entry>
 432                 <para><literal>
 433                     <replaceable>&lt;name&gt;</replaceable>
 434                   </literal></para>
 435               </entry>
 436               <entry>
 437                 <para> The name of the group.</para>
 438               </entry>
 439             </row>
 440             <row>
 441               <entry>
 442                 <para>
 443                 <literal> --active </literal>
 444             </para>
 445               </entry>
 446               <entry>
 447                 <para> Lists the active nodes.</para>
 448               </entry>
 449             </row>
 450             <row>
 451               <entry>
 452                 <para>
 453                 <literal> --busy </literal>
 454             </para>
 455               </entry>
 456               <entry>
 457                 <para> Lists the busy nodes.</para>
 458               </entry>
 459             </row>
 460             <row>
 461               <entry>
 462                 <literal> --down </literal>
 463               </entry>
 464               <entry>
 465                 <para> Lists the down nodes.</para>
 466               </entry>
 467             </row>
 468             <row>
 469               <entry>
 470                 <literal> --unknown </literal>
 471               </entry>
 472               <entry>
 473                 <para> Lists unknown nodes.</para>
 474               </entry>
 475             </row>
 476             <row>
 477               <entry>
 478                 <literal> --all </literal>
 479               </entry>
 480               <entry>
 481                 <para> Lists all nodes.</para>
 482               </entry>
 483             </row>
 484           </tbody>
 485         </tgroup>
 486       </informaltable>
 487       <para>Example:</para>
 488       <screen>$ lst list_group
 489 1) clients
 490 2) servers
 491 Total 2 groups
 492 $ lst list_group clients
 493 ACTIVE BUSY DOWN UNKNOWN TOTAL
 494 3 1 2 0 6
 495 $ lst list_group clients --all
 496 192.168.1.10@tcp Active
 497 192.168.1.11@tcp Active
 498 192.168.1.12@tcp Busy
 499 192.168.1.13@tcp Active
 500 192.168.1.14@tcp DOWN
 501 192.168.1.15@tcp DOWN
 502 Total 6 nodes
 503 $ lst list_group clients --busy
 504 192.168.1.12@tcp Busy
 505 Total 1 node</screen>
 506       <para><literal>
 507           <replaceable role="bold">del_group</replaceable>
 508           <replaceable> &lt;name&gt;</replaceable>
 509         </literal></para>
 510       <para>Removes a group from the session. If the group is referred to by any test, then the operation fails. If nodes in the group are referred to only by this group, then they are kicked out from the current session; otherwise, they are still in the current session.</para>
 511       <screen>$ lst del_group clients</screen>
 512       <para><literal><replaceable role="bold">lstclient --sesid</replaceable><replaceable> &lt;NID&gt; </replaceable><replaceable role="bold">--group</replaceable><replaceable> &lt;name&gt;</replaceable> [--server_mode]</literal></para>
 513       <para>Use <literal>lstclient</literal> to run the userland self-test client. The <literal>lstclient</literal> command should be executed after creating a session on the console. There are only two mandatory options for <literal>lstclient</literal>:</para>
 514       <informaltable frame="all">
 515         <tgroup cols="2">
 516           <colspec colname="c1" colwidth="50*"/>
 517           <colspec colname="c2" colwidth="50*"/>
 518           <thead>
 519             <row>
 520               <entry>
 521                 <para><emphasis role="bold">Parameter</emphasis></para>
 522               </entry>
 523               <entry>
 524                 <para><emphasis role="bold">Description</emphasis></para>
 525               </entry>
 526             </row>
 527           </thead>
 528           <tbody>
 529             <row>
 530               <entry>
 531                 <literal> --sesid<replaceable>&lt;NID&gt;</replaceable></literal>
 532               </entry>
 533               <entry>
 534                 <para> The first console&apos;s NID.</para>
 535               </entry>
 536             </row>
 537             <row>
 538               <entry>
 539                 <literal> --group<replaceable>&lt;name&gt;</replaceable></literal>
 540               </entry>
 541               <entry>
 542                 <para> The test group to join.</para>
 543               </entry>
 544             </row>
 545             <row>
 546               <entry>
 547                 <literal> --server_mode </literal>
 548               </entry>
 549               <entry>
 550                 <para> When included, forces LNET to behave as a server, such as starting an acceptor if the underlying NID needs it or using privileged ports. Only root is allowed to use the <literal>--server_mode</literal> option.</para>
 551               </entry>
 552             </row>
 553           </tbody>
 554         </tgroup>
 555       </informaltable>
 556       <para><emphasis role="bold">Example:</emphasis></para>
 557       <screen>Console $ lst new_session testsession
 558 Client1 $ lstclient --sesid 192.168.1.52@tcp --group clients</screen>
 559       <para><emphasis role="bold">Example:</emphasis></para>
 560       <screen>Client1 $ lstclient --sesid 192.168.1.52@tcp |--group clients --server_mode</screen>
 561     </section>
 562     <section xml:id="dbdoclet.50438223_36860">
 563       <title>Batch and Test Commands</title>
 564       <para>This section describes <literal>lst</literal> batch and test commands.</para>
 565       <para><literal>
 566           <replaceable role="bold">add_batch NAME</replaceable>
 567         </literal></para>
 568       <para>A default batch test set named batch is created when the session is started. You can specify a batch name by using add_batch:</para>
 569       <screen>$ lst add_batch bulkperf</screen>
 570       <para>Creates a batch test called <literal>bulkperf</literal>.</para>
 571       <screen>
 572           <replaceable role="bold">add_test --batch <replaceable>&lt;<replaceable role="italic">batchname</replaceable>&gt;</replaceable> [--loop&lt;#&gt;] <replaceable role="bold">[--concurrency</replaceable><replaceable>&lt;#&gt;</replaceable><replaceable role="bold">] [--distribute</replaceable><replaceable>&lt;#:#&gt;</replaceable><replaceable role="bold">]</replaceable> \ --from<replaceable> &lt;<replaceable role="italic">group</replaceable>&gt;</replaceable> --to <replaceable>&lt;<replaceable role="italic">group</replaceable>&gt;</replaceable> {brw|ping} &lt;test options&gt;</replaceable>
 573         </screen>
 574       <para>Adds a test to a batch. The parameters are described below.</para>
 575       <informaltable frame="all">
 576         <tgroup cols="3">
 577           <colspec colname="c1" colwidth="33*"/>
 578           <colspec colname="c2" colwidth="33*"/>
 579           <colspec colname="c3" colwidth="33*"/>
 580           <thead>
 581             <row>
 582               <entry>
 583                 <para><emphasis role="bold">Parameter</emphasis></para>
 584               </entry>
 585               <entry nameend="c3" namest="c2">
 586                 <para><emphasis role="bold">Description</emphasis></para>
 587               </entry>
 588             </row>
 589           </thead>
 590           <tbody>
 591             <row>
 592               <entry>
 593                 <para> <literal>--batch<replaceable>&lt;batchname&gt;</replaceable></literal></para>
 594               </entry>
 595               <entry nameend="c3" namest="c2">
 596                 <para>Names a group of tests for later execution.</para>
 597               </entry>
 598             </row>
 599             <row>
 600               <entry>
 601                 <literal> --loop<replaceable>&lt;#&gt;</replaceable></literal>
 602               </entry>
 603               <entry nameend="c3" namest="c2">
 604                 <para>Number of times to run the test.</para>
 605               </entry>
 606             </row>
 607             <row>
 608               <entry>
 609                 <literal> --concurrency<replaceable>&lt;#&gt;</replaceable></literal>
 610               </entry>
 611               <entry nameend="c3" namest="c2">
 612                 <para>The number of requests that are active at one time.</para>
 613               </entry>
 614             </row>
 615             <row>
 616               <entry>
 617                 <literal> --distribute<replaceable>&lt;#:#&gt;</replaceable></literal>
 618               </entry>
 619               <entry nameend="c3" namest="c2">
 620                 <para>Determines the ratio of client nodes to server nodes for the specified test. This allows you to specify a wide range of topologies, including one-to-one and all-to-all. Distribution divides the source group into subsets, which are paired with equivalent subsets from the target group so only nodes in matching subsets communicate.</para>
 621               </entry>
 622             </row>
 623             <row>
 624               <entry>
 625                 <literal> --from<replaceable>&lt;group&gt;</replaceable></literal>
 626               </entry>
 627               <entry nameend="c3" namest="c2">
 628                 <para>The source group (test client).</para>
 629               </entry>
 630             </row>
 631             <row>
 632               <entry>
 633                 <literal> --to<replaceable>&lt;group&gt;</replaceable></literal>
 634               </entry>
 635               <entry nameend="c3" namest="c2">
 636                 <para>The target group (test server).</para>
 637               </entry>
 638             </row>
 639             <row>
 640               <entry>
 641                 <literal>ping</literal>
 642               </entry>
 643               <entry nameend="c3" namest="c2">
 644                 <para>Sends a small request message, resulting in a small reply message. For more details, see <xref linkend="dbdoclet.50438223_42848"/></para>
 645               </entry>
 646             </row>
 647             <row>
 648               <entry>
 649                 <literal>brw</literal>
 650               </entry>
 651               <entry nameend="c3" namest="c2">
 652                 <para>Sends a small request message followed by a bulk data transfer, resulting in a small reply message. <xref linkend="dbdoclet.50438223_42848"/>. Options are:</para>
 653               </entry>
 654             </row>
 655             <row>
 656               <entry>
 657                 <para> </para>
 658               </entry>
 659               <entry>
 660                 <literal> read | write </literal>
 661               </entry>
 662               <entry>
 663                 <para>Read or write. The default is read.</para>
 664               </entry>
 665             </row>
 666             <row>
 667               <entry/>
 668               <entry>
 669                 <literal> size=&lt;#&gt;| &lt;#&gt;K | &lt;#&gt;M </literal>
 670               </entry>
 671               <entry>
 672                 <para>I/O size in bytes, KB or MB (i.e., <literal>size=1024</literal>, <literal>size=4</literal>K, <literal>size=1M</literal>). The default is 4K bytes.</para>
 673               </entry>
 674             </row>
 675             <row>
 676               <entry/>
 677               <entry>
 678                 <literal> check=full|simple </literal>
 679               </entry>
 680               <entry>
 681                 <para>A data validation check (checksum of data). The default is that no check is done.</para>
 682               </entry>
 683             </row>
 684           </tbody>
 685         </tgroup>
 686       </informaltable>
 687       <para><emphasis role="bold">Examples showing use of the distribute parameter:</emphasis></para>
 688       <screen>Clients: (C1, C2, C3, C4, C5, C6)
 689 Server: (S1, S2, S3)
 690 --distribute 1:1 (C1-&gt;S1), (C2-&gt;S2), (C3-&gt;S3), (C4-&gt;S1), (C5-&gt;S2),
 691 \(C6-&gt;S3) /* -&gt; means test conversation */ --distribute 2:1 (C1,C2-&gt;S1), (C3,C4-&gt;S2), (C5,C6-&gt;S3)
 692 --distribute 3:1 (C1,C2,C3-&gt;S1), (C4,C5,C6-&gt;S2), (NULL-&gt;S3)
 693 --distribute 3:2 (C1,C2,C3-&gt;S1,S2), (C4,C5,C6-&gt;S3,S1)
 694 --distribute 4:1 (C1,C2,C3,C4-&gt;S1), (C5,C6-&gt;S2), (NULL-&gt;S3)
 695 --distribute 4:2 (C1,C2,C3,C4-&gt;S1,S2), (C5, C6-&gt;S3, S1)
 696 --distribute 6:3 (C1,C2,C3,C4,C5,C6-&gt;S1,S2,S3)</screen>
 697       <para>The setting <literal>--distribute 1:1</literal> is the default setting where each source node communicates with one target node.</para>
 698       <para>When the setting <literal>--distribute 1:<replaceable>&lt;n&gt;</replaceable></literal> (where <literal>
 699           <replaceable>&lt;n&gt;</replaceable>
 700         </literal>  is the size of the target group) is used, each source node communicates with every node in the target group.</para>
 701       <para>Note that if there are more source nodes than target nodes, some source nodes may share the same target nodes. Also, if there are more target nodes than source nodes, some higher-ranked target nodes will be idle.</para>
 702       <para><emphasis role="bold">Example showing a <literal>brw</literal> test:</emphasis></para>
 703       <screen>$ lst add_group clients 192.168.1.[10-17]@tcp
 704 $ lst add_group servers 192.168.10.[100-103]@tcp
 705 $ lst add_batch bulkperf
 706 $ lst add_test --batch bulkperf --loop 100 --concurrency 4 \
 707   --distribute 4:2 --from clients brw WRITE size=16K</screen>
 708       <para>In the example above, a batch test called bulkperf that will do a 16 kbyte bulk write request. In this test, two groups of four clients (sources) write to each of four servers (targets) as shown below:</para>
 709       <itemizedlist>
 710         <listitem>
 711           <para> <literal>192.168.1.[10-13]</literal> will write to <literal>192.168.10.[100,101]</literal></para>
 712         </listitem>
 713         <listitem>
 714           <para> <literal>192.168.1.[14-17]</literal> will write to <literal>192.168.10.[102,103]</literal></para>
 715         </listitem>
 716       </itemizedlist>
 717       <para><emphasis role="bold">
 718           <literal><replaceable role="bold">list_batch [</replaceable><replaceable><replaceable role="italic">&lt;name</replaceable>&gt;</replaceable><replaceable role="bold">] [--test</replaceable><replaceable> &lt;<replaceable role="italic">index</replaceable>&gt;</replaceable><replaceable role="bold">] [--active] [--invalid]</replaceable> [--server | client<replaceable role="bold">]</replaceable></literal>
 719         </emphasis></para>
 720       <para>Lists batches in the current session or lists client and server nodes in a batch or a test.</para>
 721       <informaltable frame="all">
 722         <tgroup cols="3">
 723           <colspec colname="c1" colwidth="33*"/>
 724           <colspec colname="c2" colwidth="33*"/>
 725           <colspec colname="c3" colwidth="33*"/>
 726           <thead>
 727             <row>
 728               <entry>
 729                 <para><emphasis role="bold">Parameter</emphasis></para>
 730               </entry>
 731               <entry nameend="c3" namest="c2">
 732                 <para><emphasis role="bold">Description</emphasis></para>
 733               </entry>
 734             </row>
 735           </thead>
 736           <tbody>
 737             <row>
 738               <entry>
 739                 <literal> --test<replaceable>&lt;index&gt;</replaceable></literal>
 740               </entry>
 741               <entry nameend="c3" namest="c2">
 742                 <para>Lists tests in a batch. If no option is used, all tests in the batch are listed. IIf one of these options are used, only specified tests in the batch are listed:</para>
 743               </entry>
 744             </row>
 745             <row>
 746               <entry>
 747                 <para> </para>
 748               </entry>
 749               <entry>
 750                 <para><literal>active</literal></para>
 751               </entry>
 752               <entry>
 753                 <para>Lists only active batch tests.</para>
 754               </entry>
 755             </row>
 756             <row>
 757               <entry>
 758                 <para> </para>
 759               </entry>
 760               <entry>
 761                 <para><literal>invalid</literal></para>
 762               </entry>
 763               <entry>
 764                 <para>Lists only invalid batch tests.</para>
 765               </entry>
 766             </row>
 767             <row>
 768               <entry>
 769                 <para> </para>
 770               </entry>
 771               <entry>
 772                 <literal> server | client </literal>
 773               </entry>
 774               <entry>
 775                 <para>Lists client and server nodes in a batch test.</para>
 776               </entry>
 777             </row>
 778           </tbody>
 779         </tgroup>
 780       </informaltable>
 781       <para><emphasis role="bold">Example:</emphasis></para>
 782       <screen>$ lst list_batchbulkperf
 783 $ lst list_batch bulkperf
 784 Batch: bulkperf Tests: 1 State: Idle
 785 ACTIVE BUSY DOWN UNKNOWN TOTAL
 786 client 8 0 0 0 8
 787 server 4 0 0 0 4
 788 Test 1(brw) (loop: 100, concurrency: 4)
 789 ACTIVE BUSY DOWN UNKNOWN TOTAL
 790 client 8 0 0 0 8
 791 server 4 0 0 0 4
 792 $ lst list_batch bulkperf --server --active
 793 192.168.10.100@tcp Active
 794 192.168.10.101@tcp Active
 795 192.168.10.102@tcp Active
 796 192.168.10.103@tcp Active</screen>
 797       <para><literal>
 798           <replaceable role="bold">run</replaceable>
 799           <replaceable role="bold">
 800             <replaceable> &lt;<replaceable role="italic">name</replaceable>&gt;</replaceable>
 801           </replaceable>
 802         </literal></para>
 803       <para>Runs the batch.</para>
 804       <screen>$ lst run bulkperf</screen>
 805       <para><literal>
 806           <replaceable role="bold">
 807             <replaceable role="bold">stop </replaceable>
 808             <replaceable>&lt;<replaceable role="italic">name</replaceable>&gt;</replaceable>
 809           </replaceable>
 810         </literal></para>
 811       <para>Stops the batch.</para>
 812       <screen>$ lst stop bulkperf</screen>
 813       <para><emphasis role="bold">
 814           <literal>
 815             <replaceable role="bold">query </replaceable>
 816             <replaceable>&lt;<replaceable role="italic">name</replaceable>&gt; </replaceable>
 817             <replaceable role="bold">[--test</replaceable>
 818             <replaceable> &lt;<replaceable role="italic">index</replaceable>&gt;</replaceable>
 819             <replaceable role="bold">] [--timeout</replaceable>
 820             <replaceable> &lt;<replaceable role="italic">seconds</replaceable>&gt;</replaceable>
 821             <replaceable role="bold">] [--loop</replaceable>
 822             <replaceable>&lt;#&gt;</replaceable>
 823             <replaceable role="bold">] [--delay</replaceable>
 824             <replaceable>&lt;<replaceable role="italic">seconds</replaceable>&gt;</replaceable>
 825             <replaceable role="bold">] [--all]</replaceable>
 826           </literal>
 827         </emphasis></para>
 828       <para>Queries the batch status.</para>
 829       <informaltable frame="all">
 830         <tgroup cols="2">
 831           <colspec colname="c1" colwidth="50*"/>
 832           <colspec colname="c2" colwidth="50*"/>
 833           <thead>
 834             <row>
 835               <entry>
 836                 <para><emphasis role="bold">Parameter</emphasis></para>
 837               </entry>
 838               <entry>
 839                 <para><emphasis role="bold">Description</emphasis></para>
 840               </entry>
 841             </row>
 842           </thead>
 843           <tbody>
 844             <row>
 845               <entry>
 846                 <literal> --test<replaceable>&lt;index&gt;</replaceable></literal>
 847               </entry>
 848               <entry>
 849                 <para> Only queries the specified test. The test index starts from 1.</para>
 850               </entry>
 851             </row>
 852             <row>
 853               <entry>
 854                 <literal> --timeout<replaceable>&lt;seconds&gt;</replaceable></literal>
 855               </entry>
 856               <entry>
 857                 <para> The timeout value to wait for RPC. The default is 5 seconds.</para>
 858               </entry>
 859             </row>
 860             <row>
 861               <entry>
 862                 <literal> --loop<replaceable>&lt;#&gt;</replaceable></literal>
 863               </entry>
 864               <entry>
 865                 <para> The loop count of the query.</para>
 866               </entry>
 867             </row>
 868             <row>
 869               <entry>
 870                 <literal> --delay<replaceable>&lt;seconds&gt;</replaceable></literal>
 871               </entry>
 872               <entry>
 873                 <para> The interval of each query. The default is 5 seconds.</para>
 874               </entry>
 875             </row>
 876             <row>
 877               <entry>
 878                 <literal> --all </literal>
 879               </entry>
 880               <entry>
 881                 <para> The list status of all nodes in a batch or a test.</para>
 882               </entry>
 883             </row>
 884           </tbody>
 885         </tgroup>
 886       </informaltable>
 887       <para><emphasis role="bold">Example:</emphasis></para>
 888       <screen>$ lst run bulkperf
 889 $ lst query bulkperf --loop 5 --delay 3
 890 Batch is running
 891 Batch is running
 892 Batch is running
 893 Batch is running
 894 Batch is running
 895 $ lst query bulkperf --all
 896 192.168.1.10@tcp Running
 897 192.168.1.11@tcp Running
 898 192.168.1.12@tcp Running
 899 192.168.1.13@tcp Running
 900 192.168.1.14@tcp Running
 901 192.168.1.15@tcp Running
 902 192.168.1.16@tcp Running
 903 192.168.1.17@tcp Running
 904 $ lst stop bulkperf
 905 $ lst query bulkperf
 906 Batch is idle</screen>
 907     </section>
 908     <section remap="h3">
 909       <title>Other Commands</title>
 910       <para>This section describes other <literal>lst</literal> commands.</para>
 911       <para><literal>
 912           <replaceable role="bold">
 913             <replaceable role="bold">ping [-session] [--group</replaceable>
 914             <replaceable>&lt;<replaceable role="italic">name</replaceable>&gt;</replaceable>
 915             <replaceable role="bold">] [--nodes</replaceable>
 916             <replaceable> &lt;NIDs&gt;</replaceable>
 917             <replaceable role="bold">] [--batch</replaceable>
 918             <replaceable> &lt;<replaceable role="italic">name</replaceable>&gt;</replaceable>
 919             <replaceable role="bold">] [--server] [--timeout</replaceable>
 920             <replaceable> &lt;<replaceable role="italic">seconds</replaceable>&gt;</replaceable>
 921             <replaceable role="bold">]</replaceable>
 922           </replaceable>
 923         </literal></para>
 924       <para>Sends a &apos;hello&apos; query to the nodes.</para>
 925       <informaltable frame="all">
 926         <tgroup cols="2">
 927           <colspec colname="c1" colwidth="50*"/>
 928           <colspec colname="c2" colwidth="50*"/>
 929           <thead>
 930             <row>
 931               <entry>
 932                 <para><emphasis role="bold">Parameter</emphasis></para>
 933               </entry>
 934               <entry>
 935                 <para><emphasis role="bold">Description</emphasis></para>
 936               </entry>
 937             </row>
 938           </thead>
 939           <tbody>
 940             <row>
 941               <entry>
 942                 <literal> --session </literal>
 943               </entry>
 944               <entry>
 945                 <para>Pings all nodes in the current session.</para>
 946               </entry>
 947             </row>
 948             <row>
 949               <entry>
 950                 <literal> --group<replaceable>&lt;name&gt;</replaceable></literal>
 951               </entry>
 952               <entry>
 953                 <para>Pings all nodes in a specified group.</para>
 954               </entry>
 955             </row>
 956             <row>
 957               <entry>
 958                 <literal> --nodes<replaceable>&lt;NIDs&gt;</replaceable></literal>
 959               </entry>
 960               <entry>
 961                 <para>Pings all specified nodes.</para>
 962               </entry>
 963             </row>
 964             <row>
 965               <entry>
 966                 <literal> --batch<replaceable>&lt;name&gt;</replaceable></literal>
 967               </entry>
 968               <entry>
 969                 <para>Pings all client nodes in a batch.</para>
 970               </entry>
 971             </row>
 972             <row>
 973               <entry>
 974                 <literal> --server </literal>
 975               </entry>
 976               <entry>
 977                 <para>Sends RPC to all server nodes instead of client nodes. This option is only used with <literal>--batch<replaceable>&lt;name&gt;</replaceable></literal>.</para>
 978               </entry>
 979             </row>
 980             <row>
 981               <entry>
 982                 <literal> --timeout<replaceable>&lt;seconds&gt;</replaceable></literal>
 983               </entry>
 984               <entry>
 985                 <para>The RPC timeout value.</para>
 986               </entry>
 987             </row>
 988           </tbody>
 989         </tgroup>
 990       </informaltable>
 991       <para><emphasis role="bold">Example:</emphasis></para>
 992       <screen>$ lst ping 192.168.10.[15-20]@tcp
 993 192.168.1.15@tcp Active [session: liang id: 192.168.1.3@tcp]
 994 192.168.1.16@tcp Active [session: liang id: 192.168.1.3@tcp]
 995 192.168.1.17@tcp Active [session: liang id: 192.168.1.3@tcp]
 996 192.168.1.18@tcp Busy [session: Isaac id: 192.168.10.10@tcp]
 997 192.168.1.19@tcp Down [session: &lt;NULL&gt; id: LNET_NID_ANY]
 998 192.168.1.20@tcp Down [session: &lt;NULL&gt; id: LNET_NID_ANY]</screen>
 999       <para><literal>
1000           <replaceable role="bold">
1001             <replaceable role="bold">stat [--bw] [--rate] [--read] [--write] [--max] [--min] [--avg] &quot; &quot; [--timeout</replaceable>
1002             <replaceable> &lt;<replaceable role="italic">seconds</replaceable>&gt;</replaceable>
1003             <replaceable role="bold">] [--delay </replaceable>
1004             <replaceable>&lt;<replaceable role="italic">seconds</replaceable>&gt;</replaceable>
1005             <replaceable role="bold">] </replaceable>
1006             <replaceable>&lt;<replaceable role="italic">group</replaceable>&gt;</replaceable>
1007             <replaceable role="bold">|&lt;</replaceable>
1008             <replaceable><replaceable role="italic">NIDs</replaceable>&gt; </replaceable>
1009             <replaceable role="bold">[</replaceable>
1010             <replaceable>&lt;<replaceable role="italic">group</replaceable>&gt;</replaceable>
1011             <replaceable role="bold">|</replaceable>
1012             <replaceable>&lt;<replaceable role="italic">NIDs</replaceable>&gt;</replaceable>
1013             <replaceable role="bold">]</replaceable>
1014           </replaceable>
1015         </literal></para>
1016       <para>The collection performance and RPC statistics of one or more nodes.</para>
1017       <informaltable frame="all">
1018         <tgroup cols="2">
1019           <colspec colname="c1" colwidth="50*"/>
1020           <colspec colname="c2" colwidth="50*"/>
1021           <thead>
1022             <row>
1023               <entry>
1024                 <para><emphasis role="bold">Parameter</emphasis></para>
1025               </entry>
1026               <entry>
1027                 <para><emphasis role="bold">Description</emphasis></para>
1028               </entry>
1029             </row>
1030           </thead>
1031           <tbody>
1032             <row>
1033               <entry>
1034                 <literal> --bw </literal>
1035               </entry>
1036               <entry>
1037                 <para>Displays the bandwidth of the specified group/nodes.</para>
1038               </entry>
1039             </row>
1040             <row>
1041               <entry>
1042                 <literal> --rate </literal>
1043               </entry>
1044               <entry>
1045                 <para>Displays the rate of RPCs of the specified group/nodes.</para>
1046               </entry>
1047             </row>
1048             <row>
1049               <entry>
1050                 <literal> --read </literal>
1051               </entry>
1052               <entry>
1053                 <para>Displays the read statistics of the specified group/nodes.</para>
1054               </entry>
1055             </row>
1056             <row>
1057               <entry>
1058                 <literal> --write </literal>
1059               </entry>
1060               <entry>
1061                 <para>Displays the write statistics of the specified group/nodes.</para>
1062               </entry>
1063             </row>
1064             <row>
1065               <entry>
1066                 <literal> --max </literal>
1067               </entry>
1068               <entry>
1069                 <para>Displays the maximum value of the statistics.</para>
1070               </entry>
1071             </row>
1072             <row>
1073               <entry>
1074                 <literal> --min </literal>
1075               </entry>
1076               <entry>
1077                 <para>Displays the minimum value of the statistics.</para>
1078               </entry>
1079             </row>
1080             <row>
1081               <entry>
1082                 <literal> --avg </literal>
1083               </entry>
1084               <entry>
1085                 <para>Displays the average of the statistics.</para>
1086               </entry>
1087             </row>
1088             <row>
1089               <entry>
1090                 <literal> --timeout<replaceable>&lt;seconds&gt;</replaceable></literal>
1091               </entry>
1092               <entry>
1093                 <para>The timeout of the statistics RPC. The default is 5 seconds.</para>
1094               </entry>
1095             </row>
1096             <row>
1097               <entry>
1098                 <literal> --delay<replaceable>&lt;seconds&gt;</replaceable></literal>
1099               </entry>
1100               <entry>
1101                 <para>The interval of the statistics (in seconds).</para>
1102               </entry>
1103             </row>
1104           </tbody>
1105         </tgroup>
1106       </informaltable>
1107       <para><emphasis role="bold">Example:</emphasis></para>
1108       <screen>$ lst run bulkperf
1109 $ lst stat clients
1110 [LNet Rates of clients]
1111 [W] Avg: 1108 RPC/s Min: 1060 RPC/s Max: 1155 RPC/s
1112 [R] Avg: 2215 RPC/s Min: 2121 RPC/s Max: 2310 RPC/s
1113 [LNet Bandwidth of clients]
1114 [W] Avg: 16.60 MB/s Min: 16.10 MB/s Max: 17.1 MB/s
1115 [R] Avg: 40.49 MB/s Min: 40.30 MB/s Max: 40.68 MB/s</screen>
1116       <para>Specifying a group name (<emphasis>
1117           <literal>&lt;group&gt;</literal>
1118         </emphasis>) causes statistics to be gathered for all nodes in a test group. For example:</para>
1119       <screen>$ lst stat servers
1120 </screen>
1121       <para>where servers is the name of a test group created by <literal>lst add_group</literal></para>
1122       <para>Specifying a <literal>NID</literal> range (<emphasis>&lt;<literal>NIDs</literal>&gt;</emphasis>) causes statistics to be gathered for selected nodes. For example:</para>
1123       <screen>$ lst stat 192.168.0.[1-100/2]@tcp</screen>
1124       <para>Only LNET performance statistics are available. By default, all statistics
1125 information is displayed. Users can specify additional information with these options.</para>
1126       <para><literal>
1127           <replaceable role="bold">show_error [--session] [&lt;group&gt;|&lt;NIDs&gt;]...</replaceable>
1128         </literal></para>
1129       <para>Lists the number of failed RPCs on test nodes.</para>
1130       <informaltable frame="all">
1131         <tgroup cols="2">
1132           <colspec colname="c1" colwidth="50*"/>
1133           <colspec colname="c2" colwidth="50*"/>
1134           <thead>
1135             <row>
1136               <entry>
1137                 <para><emphasis role="bold">Parameter</emphasis></para>
1138               </entry>
1139               <entry>
1140                 <para><emphasis role="bold">Description</emphasis></para>
1141               </entry>
1142             </row>
1143           </thead>
1144           <tbody>
1145             <row>
1146               <entry>
1147                 <literal> --session </literal>
1148               </entry>
1149               <entry>
1150                 <para>Lists errors in the current test session. With this option, historical RPC errors are not listed.</para>
1151               </entry>
1152             </row>
1153           </tbody>
1154         </tgroup>
1155       </informaltable>
1156       <para><emphasis role="bold">Example:</emphasis></para>
1157       <screen>$ lst show_error client
1158 sclients
1159 12345-192.168.1.15@tcp: [Session: 1 brw errors, 0 ping errors] \
1160   [RPC: 20 errors, 0 dropped,
1161 12345-192.168.1.16@tcp: [Session: 0 brw errors, 0 ping errors] \
1162   [RPC: 1 errors, 0 dropped, Total 2 error nodes in clients
1163 $ lst show_error --session clients
1164 clients
1165 12345-192.168.1.15@tcp: [Session: 1 brw errors, 0 ping errors]
1166 Total 1 error nodes in clients</screen>
1167     </section>
1168   </section>
1169 </chapter>