ConfiguringLustre.xml

   1 <?xml version='1.0' encoding='utf-8'?>
   2 <chapter xmlns="http://docbook.org/ns/docbook"
   3 xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"
   4 xml:id="configuringlustre">
   5   <title xml:id="configuringlustre.title">Configuring a Lustre File
   6   System</title>
   7   <para>This chapter shows how to configure a simple Lustre file system
   8   comprised of a combined MGS/MDT, an OST and a client. It includes:</para>
   9   <itemizedlist>
  10     <listitem>
  11       <para>
  12         <xref linkend="dbdoclet.50438267_50692" />
  13       </para>
  14     </listitem>
  15     <listitem>
  16       <para>
  17         <xref linkend="dbdoclet.50438267_76752" />
  18       </para>
  19     </listitem>
  20   </itemizedlist>
  21   <section xml:id="dbdoclet.50438267_50692">
  22     <title>
  23     <indexterm>
  24       <primary>Lustre</primary>
  25       <secondary>configuring</secondary>
  26     </indexterm>Configuring a Simple Lustre File System</title>
  27     <para>A Lustre file system can be set up in a variety of configurations by
  28     using the administrative utilities provided with the Lustre software. The
  29     procedure below shows how to configure a simple Lustre file system
  30     consisting of a combined MGS/MDS, one OSS with two OSTs, and a client. For
  31     an overview of the entire Lustre installation procedure, see
  32     <xref linkend="installoverview" />.</para>
  33     <para>This configuration procedure assumes you have completed the
  34     following:</para>
  35     <itemizedlist>
  36       <listitem>
  37         <para>
  38         <emphasis>
  39           <emphasis role="bold">Set up and configured your hardware</emphasis>
  40         </emphasis>. For more information about hardware requirements, see
  41         <xref linkend="settinguplustresystem" />.</para>
  42       </listitem>
  43       <listitem>
  44         <para>
  45         <emphasis role="bold">Downloaded and installed the Lustre
  46         software.</emphasis>For more information about preparing for and
  47         installing the Lustre software, see
  48         <xref linkend="installinglustre" />.</para>
  49       </listitem>
  50     </itemizedlist>
  51     <para>The following optional steps should also be completed, if needed,
  52     before the Lustre software is configured:</para>
  53     <itemizedlist>
  54       <listitem>
  55         <para>
  56         <emphasis>Set up a hardware or software RAID on block devices to be
  57         used as OSTs or MDTs.</emphasis>For information about setting up RAID,
  58         see the documentation for your RAID controller or
  59         <xref linkend="configuringstorage" />.</para>
  60       </listitem>
  61       <listitem>
  62         <para>
  63         <emphasis>Set up network interface bonding on Ethernet
  64         interfaces.</emphasis>For information about setting up network
  65         interface bonding, see
  66         <xref linkend="settingupbonding" />.</para>
  67       </listitem>
  68       <listitem>
  69         <para>
  70         <emphasis>Set</emphasis> lnet
  71         <emphasis>module parameters to specify how Lustre Networking (LNet) is
  72         to be configured to work with a Lustre file system and test the LNet
  73         configuration.</emphasis>LNet will, by default, use the first TCP/IP
  74         interface it discovers on a system. If this network configuration is
  75         sufficient, you do not need to configure LNet. LNet configuration is
  76         required if you are using InfiniBand or multiple Ethernet
  77         interfaces.</para>
  78       </listitem>
  79     </itemizedlist>
  80     <para>For information about configuring LNet, see
  81     <xref linkend="configuringlnet" />. For information about testing LNet, see
  82
  83     <xref linkend="lnetselftest" />.</para>
  84     <itemizedlist>
  85       <listitem>
  86         <para>
  87         <emphasis>Run the benchmark script
  88         <literal>sgpdd-survey</literal> to determine baseline performance of
  89         your hardware.</emphasis>Benchmarking your hardware will simplify
  90         debugging performance issues that are unrelated to the Lustre software
  91         and ensure you are getting the best possible performance with your
  92         installation. For information about running
  93         <literal>sgpdd-survey</literal>, see
  94         <xref linkend="benchmarkingtests" />.</para>
  95       </listitem>
  96     </itemizedlist>
  97     <note>
  98       <para>The
  99       <literal>sgpdd-survey</literal> script overwrites the device being tested
 100       so it must be run before the OSTs are configured.</para>
 101     </note>
 102     <para>To configure a simple Lustre file system, complete these
 103     steps:</para>
 104     <orderedlist>
 105       <listitem>
 106         <para>Create a combined MGS/MDT file system on a block device. On the
 107         MDS node, run:</para>
 108         <screen>
 109 mkfs.lustre --fsname=
 110 <replaceable>fsname</replaceable> --mgs --mdt --index=0
 111 <replaceable>/dev/block_device</replaceable>
 112 </screen>
 113         <para>The default file system name (
 114         <literal>fsname</literal>) is
 115         <literal>lustre</literal>.</para>
 116         <note>
 117           <para>If you plan to create multiple file systems, the MGS should be
 118           created separately on its own dedicated block device, by
 119           running:</para>
 120           <screen>
 121 mkfs.lustre --fsname=
 122 <replaceable>fsname</replaceable> --mgs
 123 <replaceable>/dev/block_device</replaceable>
 124 </screen>
 125           <para>See
 126           <xref linkend="dbdoclet.50438194_88063" />for more details.</para>
 127         </note>
 128       </listitem>
 129       <listitem xml:id="dbdoclet.addmdtindex">
 130         <para>Optional for Lustre software release 2.4 and later.
 131         Add in additional MDTs.</para>
 132         <screen>
 133 mkfs.lustre --fsname=
 134 <replaceable>fsname</replaceable> --mgsnode=
 135 <replaceable>nid</replaceable> --mdt --index=1
 136 <replaceable>/dev/block_device</replaceable>
 137 </screen>
 138         <note>
 139           <para>Up to 4095 additional MDTs can be added.</para>
 140         </note>
 141       </listitem>
 142       <listitem>
 143         <para>Mount the combined MGS/MDT file system on the block device. On
 144         the MDS node, run:</para>
 145         <screen>
 146 mount -t lustre
 147 <replaceable>/dev/block_device</replaceable>
 148 <replaceable>/mount_point</replaceable>
 149 </screen>
 150         <note>
 151           <para>If you have created an MGS and an MDT on separate block
 152           devices, mount them both.</para>
 153         </note>
 154       </listitem>
 155       <listitem xml:id="dbdoclet.format_ost">
 156         <para>Create the OST. On the OSS node, run:</para>
 157         <screen>
 158 mkfs.lustre --fsname=
 159 <replaceable>fsname</replaceable> --mgsnode=
 160 <replaceable>MGS_NID</replaceable> --ost --index=
 161 <replaceable>OST_index</replaceable>
 162 <replaceable>/dev/block_device</replaceable>
 163 </screen>
 164         <para>When you create an OST, you are formatting a
 165         <literal>ldiskfs</literal> or
 166         <literal>ZFS</literal> file system on a block storage device like you
 167         would with any local file system.</para>
 168         <para>You can have as many OSTs per OSS as the hardware or drivers
 169         allow. For more information about storage and memory requirements for a
 170         Lustre file system, see
 171         <xref linkend="settinguplustresystem" />.</para>
 172         <para>You can only configure one OST per block device. You should
 173         create an OST that uses the raw block device and does not use
 174         partitioning.</para>
 175         <para>You should specify the OST index number at format time in order
 176         to simplify translating the OST number in error messages or file
 177         striping to the OSS node and block device later on.</para>
 178         <para>If you are using block devices that are accessible from multiple
 179         OSS nodes, ensure that you mount the OSTs from only one OSS node at at
 180         time. It is strongly recommended that multiple-mount protection be
 181         enabled for such devices to prevent serious data corruption. For more
 182         information about multiple-mount protection, see
 183         <xref linkend="managingfailover" />.</para>
 184         <note>
 185           <para>The Lustre software currently supports block devices up to 128
 186           TB on Red Hat Enterprise Linux 5 and 6 (up to 8 TB on other
 187           distributions). If the device size is only slightly larger that 16
 188           TB, it is recommended that you limit the file system size to 16 TB at
 189           format time. We recommend that you not place DOS partitions on top of
 190           RAID 5/6 block devices due to negative impacts on performance, but
 191           instead format the whole disk for the file system.</para>
 192         </note>
 193       </listitem>
 194       <listitem xml:id="dbdoclet.mount_ost">
 195         <para>Mount the OST. On the OSS node where the OST was created,
 196         run:</para>
 197         <screen>
 198 mount -t lustre
 199 <replaceable>/dev/block_device</replaceable>
 200 <replaceable>/mount_point</replaceable>
 201 </screen>
 202         <note>
 203           <para>To create additional OSTs, repeat Step
 204           <xref linkend="dbdoclet.format_ost" />and Step
 205           <xref linkend="dbdoclet.mount_ost" />, specifying the
 206           next higher OST index number.</para>
 207         </note>
 208       </listitem>
 209       <listitem xml:id="dbdoclet.mount_on_client">
 210         <para>Mount the Lustre file system on the client. On the client node,
 211         run:</para>
 212         <screen>
 213 mount -t lustre
 214 <replaceable>MGS_node</replaceable>:/
 215 <replaceable>fsname</replaceable>
 216 <replaceable>/mount_point</replaceable>
 217 </screen>
 218         <note>
 219           <para>To mount the filesystem on additional clients, repeat Step
 220           <xref linkend="dbdoclet.mount_on_client" />.</para>
 221         </note>
 222         <note>
 223           <para>If you have a problem mounting the file system, check the
 224           syslogs on the client and all the servers for errors and also check
 225           the network settings. A common issue with newly-installed systems is
 226           that
 227           <literal>hosts.deny</literal> or firewall rules may prevent
 228           connections on port 988.</para>
 229         </note>
 230       </listitem>
 231       <listitem>
 232         <para>Verify that the file system started and is working correctly. Do
 233         this by running
 234         <literal>lfs df</literal>,
 235         <literal>dd</literal> and
 236         <literal>ls</literal> commands on the client node.</para>
 237       </listitem>
 238       <listitem>
 239         <para>
 240         <emphasis>(Optional)</emphasis>Run benchmarking tools to validate the
 241         performance of hardware and software layers in the cluster. Available
 242         tools include:</para>
 243         <itemizedlist>
 244           <listitem>
 245             <para>
 246             <literal>obdfilter-survey</literal>- Characterizes the storage
 247             performance of a Lustre file system. For details, see
 248             <xref linkend="dbdoclet.50438212_26516" />.</para>
 249           </listitem>
 250           <listitem>
 251             <para>
 252             <literal>ost-survey</literal>- Performs I/O against OSTs to detect
 253             anomalies between otherwise identical disk subsystems. For details,
 254             see
 255             <xref linkend="dbdoclet.50438212_85136" />.</para>
 256           </listitem>
 257         </itemizedlist>
 258       </listitem>
 259     </orderedlist>
 260     <section remap="h3">
 261       <title>
 262       <indexterm>
 263         <primary>Lustre</primary>
 264         <secondary>configuring</secondary>
 265         <tertiary>simple example</tertiary>
 266       </indexterm>Simple Lustre Configuration Example</title>
 267       <para>To see the steps to complete for a simple Lustre file system
 268       configuration, follow this example in which a combined MGS/MDT and two
 269       OSTs are created to form a file system called
 270       <literal>temp</literal>. Three block devices are used, one for the
 271       combined MGS/MDS node and one for each OSS node. Common parameters used
 272       in the example are listed below, along with individual node
 273       parameters.</para>
 274       <informaltable frame="all">
 275         <tgroup cols="4">
 276           <colspec colname="c1" colwidth="2*" />
 277           <colspec colname="c2" colwidth="25*" />
 278           <colspec colname="c3" colwidth="25*" />
 279           <colspec colname="c4" colwidth="25*" />
 280           <thead>
 281             <row>
 282               <entry nameend="c2" namest="c1">
 283                 <para>
 284                   <emphasis role="bold">Common Parameters</emphasis>
 285                 </para>
 286               </entry>
 287               <entry>
 288                 <para>
 289                   <emphasis role="bold">Value</emphasis>
 290                 </para>
 291               </entry>
 292               <entry>
 293                 <para>
 294                   <emphasis role="bold">Description</emphasis>
 295                 </para>
 296               </entry>
 297             </row>
 298           </thead>
 299           <tbody>
 300             <row>
 301               <entry>
 302                 <para>&#160;</para>
 303               </entry>
 304               <entry>
 305                 <para>
 306                   <emphasis role="bold">MGS node</emphasis>
 307                 </para>
 308               </entry>
 309               <entry>
 310                 <para>
 311                   <literal>10.2.0.1@tcp0</literal>
 312                 </para>
 313               </entry>
 314               <entry>
 315                 <para>Node for the combined MGS/MDS</para>
 316               </entry>
 317             </row>
 318             <row>
 319               <entry>
 320                 <para>&#160;</para>
 321               </entry>
 322               <entry>
 323                 <para>
 324                   <emphasis role="bold">file system</emphasis>
 325                 </para>
 326               </entry>
 327               <entry>
 328                 <para>
 329                   <literal>temp</literal>
 330                 </para>
 331               </entry>
 332               <entry>
 333                 <para>Name of the Lustre file system</para>
 334               </entry>
 335             </row>
 336             <row>
 337               <entry>
 338                 <para>&#160;</para>
 339               </entry>
 340               <entry>
 341                 <para>
 342                   <emphasis role="bold">network type</emphasis>
 343                 </para>
 344               </entry>
 345               <entry>
 346                 <para>
 347                   <literal>TCP/IP</literal>
 348                 </para>
 349               </entry>
 350               <entry>
 351                 <para>Network type used for Lustre file system
 352                 <literal>temp</literal></para>
 353               </entry>
 354             </row>
 355           </tbody>
 356         </tgroup>
 357       </informaltable>
 358       <informaltable frame="all">
 359         <tgroup cols="4">
 360           <colspec colname="c1" colwidth="25*" />
 361           <colspec colname="c2" colwidth="25*" />
 362           <colspec colname="c3" colwidth="25*" />
 363           <colspec colname="c4" colwidth="25*" />
 364           <thead>
 365             <row>
 366               <entry nameend="c2" namest="c1">
 367                 <para>
 368                   <emphasis role="bold">Node Parameters</emphasis>
 369                 </para>
 370               </entry>
 371               <entry>
 372                 <para>
 373                   <emphasis role="bold">Value</emphasis>
 374                 </para>
 375               </entry>
 376               <entry>
 377                 <para>
 378                   <emphasis role="bold">Description</emphasis>
 379                 </para>
 380               </entry>
 381             </row>
 382           </thead>
 383           <tbody>
 384             <row>
 385               <entry nameend="c4" namest="c1">
 386                 <para>MGS/MDS node</para>
 387               </entry>
 388             </row>
 389             <row>
 390               <entry>
 391                 <para>&#160;</para>
 392               </entry>
 393               <entry>
 394                 <para>
 395                   <emphasis role="bold">MGS/MDS node</emphasis>
 396                 </para>
 397               </entry>
 398               <entry>
 399                 <para>
 400                   <literal>mdt0</literal>
 401                 </para>
 402               </entry>
 403               <entry>
 404                 <para>MDS in Lustre file system
 405                 <literal>temp</literal></para>
 406               </entry>
 407             </row>
 408             <row>
 409               <entry>
 410                 <para>&#160;</para>
 411               </entry>
 412               <entry>
 413                 <para>
 414                   <emphasis role="bold">block device</emphasis>
 415                 </para>
 416               </entry>
 417               <entry>
 418                 <para>
 419                   <literal>/dev/sdb</literal>
 420                 </para>
 421               </entry>
 422               <entry>
 423                 <para>Block device for the combined MGS/MDS node</para>
 424               </entry>
 425             </row>
 426             <row>
 427               <entry>
 428                 <para>&#160;</para>
 429               </entry>
 430               <entry>
 431                 <para>
 432                   <emphasis role="bold">mount point</emphasis>
 433                 </para>
 434               </entry>
 435               <entry>
 436                 <para>
 437                   <literal>/mnt/mdt</literal>
 438                 </para>
 439               </entry>
 440               <entry>
 441                 <para>Mount point for the
 442                 <literal>mdt0</literal> block device (
 443                 <literal>/dev/sdb</literal>) on the MGS/MDS node</para>
 444               </entry>
 445             </row>
 446             <row>
 447               <entry nameend="c4" namest="c1">
 448                 <para>First OSS node</para>
 449               </entry>
 450             </row>
 451             <row>
 452               <entry>
 453                 <para>&#160;</para>
 454               </entry>
 455               <entry>
 456                 <para>
 457                   <emphasis role="bold">OSS node</emphasis>
 458                 </para>
 459               </entry>
 460               <entry>
 461                 <para>
 462                   <literal>oss0</literal>
 463                 </para>
 464               </entry>
 465               <entry>
 466                 <para>First OSS node in Lustre file system
 467                 <literal>temp</literal></para>
 468               </entry>
 469             </row>
 470             <row>
 471               <entry>
 472                 <para>&#160;</para>
 473               </entry>
 474               <entry>
 475                 <para>
 476                   <emphasis role="bold">OST</emphasis>
 477                 </para>
 478               </entry>
 479               <entry>
 480                 <para>
 481                   <literal>ost0</literal>
 482                 </para>
 483               </entry>
 484               <entry>
 485                 <para>First OST in Lustre file system
 486                 <literal>temp</literal></para>
 487               </entry>
 488             </row>
 489             <row>
 490               <entry>
 491                 <para>&#160;</para>
 492               </entry>
 493               <entry>
 494                 <para>
 495                   <emphasis role="bold">block device</emphasis>
 496                 </para>
 497               </entry>
 498               <entry>
 499                 <para>
 500                   <literal>/dev/sdc</literal>
 501                 </para>
 502               </entry>
 503               <entry>
 504                 <para>Block device for the first OSS node (
 505                 <literal>oss0</literal>)</para>
 506               </entry>
 507             </row>
 508             <row>
 509               <entry>
 510                 <para>&#160;</para>
 511               </entry>
 512               <entry>
 513                 <para>
 514                   <emphasis role="bold">mount point</emphasis>
 515                 </para>
 516               </entry>
 517               <entry>
 518                 <para>
 519                   <literal>/mnt/ost0</literal>
 520                 </para>
 521               </entry>
 522               <entry>
 523                 <para>Mount point for the
 524                 <literal>ost0</literal> block device (
 525                 <literal>/dev/sdc</literal>) on the
 526                 <literal>oss1</literal> node</para>
 527               </entry>
 528             </row>
 529             <row>
 530               <entry nameend="c4" namest="c1">
 531                 <para>Second OSS node</para>
 532               </entry>
 533             </row>
 534             <row>
 535               <entry>
 536                 <para></para>
 537               </entry>
 538               <entry>
 539                 <para>
 540                   <emphasis role="bold">OSS node</emphasis>
 541                 </para>
 542               </entry>
 543               <entry>
 544                 <para>
 545                   <literal>oss1</literal>
 546                 </para>
 547               </entry>
 548               <entry>
 549                 <para>Second OSS node in Lustre file system
 550                 <literal>temp</literal></para>
 551               </entry>
 552             </row>
 553             <row>
 554               <entry>
 555                 <para></para>
 556               </entry>
 557               <entry>
 558                 <para>
 559                   <emphasis role="bold">OST</emphasis>
 560                 </para>
 561               </entry>
 562               <entry>
 563                 <para>
 564                   <literal>ost1</literal>
 565                 </para>
 566               </entry>
 567               <entry>
 568                 <para>Second OST in Lustre file system
 569                 <literal>temp</literal></para>
 570               </entry>
 571             </row>
 572             <row>
 573               <entry />
 574               <entry>
 575                 <para>
 576                   <emphasis role="bold">block device</emphasis>
 577                 </para>
 578               </entry>
 579               <entry>
 580                 <para>
 581                   <literal>/dev/sdd</literal>
 582                 </para>
 583               </entry>
 584               <entry>
 585                 <para>Block device for the second OSS node (oss1)</para>
 586               </entry>
 587             </row>
 588             <row>
 589               <entry>
 590                 <para></para>
 591               </entry>
 592               <entry>
 593                 <para>
 594                   <emphasis role="bold">mount point</emphasis>
 595                 </para>
 596               </entry>
 597               <entry>
 598                 <para>
 599                   <literal>/mnt/ost1</literal>
 600                 </para>
 601               </entry>
 602               <entry>
 603                 <para>Mount point for the
 604                 <literal>ost1</literal> block device (
 605                 <literal>/dev/sdd</literal>) on the
 606                 <literal>oss1</literal> node</para>
 607               </entry>
 608             </row>
 609             <row>
 610               <entry nameend="c4" namest="c1">
 611                 <para>Client node</para>
 612               </entry>
 613             </row>
 614             <row>
 615               <entry>
 616                 <para></para>
 617               </entry>
 618               <entry>
 619                 <para>
 620                   <emphasis role="bold">client node</emphasis>
 621                 </para>
 622               </entry>
 623               <entry>
 624                 <para>
 625                   <literal>client1</literal>
 626                 </para>
 627               </entry>
 628               <entry>
 629                 <para>Client in Lustre file system
 630                 <literal>temp</literal></para>
 631               </entry>
 632             </row>
 633             <row>
 634               <entry>
 635                 <para></para>
 636               </entry>
 637               <entry>
 638                 <para>
 639                   <emphasis role="bold">mount point</emphasis>
 640                 </para>
 641               </entry>
 642               <entry>
 643                 <para>
 644                   <literal>/lustre</literal>
 645                 </para>
 646               </entry>
 647               <entry>
 648                 <para>Mount point for Lustre file system
 649                 <literal>temp</literal> on the
 650                 <literal>client1</literal> node</para>
 651               </entry>
 652             </row>
 653           </tbody>
 654         </tgroup>
 655       </informaltable>
 656       <note>
 657         <para>We recommend that you use 'dotted-quad' notation for IP addresses
 658         rather than host names to make it easier to read debug logs and debug
 659         configurations with multiple interfaces.</para>
 660       </note>
 661       <para>For this example, complete the steps below:</para>
 662       <orderedlist>
 663         <listitem>
 664           <para>Create a combined MGS/MDT file system on the block device. On
 665           the MDS node, run:</para>
 666           <screen>
 667 [root@mds /]# mkfs.lustre --fsname=temp --mgs --mdt --index=0 /dev/sdb
 668 </screen>
 669           <para>This command generates this output:</para>
 670           <screen>
 671     Permanent disk data:
 672 Target:            temp-MDT0000
 673 Index:             0
 674 Lustre FS: temp
 675 Mount type:        ldiskfs
 676 Flags:             0x75
 677    (MDT MGS first_time update )
 678 Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
 679 Parameters: mdt.identity_upcall=/usr/sbin/l_getidentity
 680
 681 checking for existing Lustre data: not found
 682 device size = 16MB
 683 2 6 18
 684 formatting backing filesystem ldiskfs on /dev/sdb
 685    target name             temp-MDTffff
 686    4k blocks               0
 687    options                 -i 4096 -I 512 -q -O dir_index,uninit_groups -F
 688 mkfs_cmd = mkfs.ext2 -j -b 4096 -L temp-MDTffff  -i 4096 -I 512 -q -O
 689 dir_index,uninit_groups -F /dev/sdb
 690 Writing CONFIGS/mountdata
 691 </screen>
 692         </listitem>
 693         <listitem>
 694           <para>Mount the combined MGS/MDT file system on the block device. On
 695           the MDS node, run:</para>
 696           <screen>
 697 [root@mds /]# mount -t lustre /dev/sdb /mnt/mdt
 698 </screen>
 699           <para>This command generates this output:</para>
 700           <screen>
 701 Lustre: temp-MDT0000: new disk, initializing
 702 Lustre: 3009:0:(lproc_mds.c:262:lprocfs_wr_identity_upcall()) temp-MDT0000:
 703 group upcall set to /usr/sbin/l_getidentity
 704 Lustre: temp-MDT0000.mdt: set parameter identity_upcall=/usr/sbin/l_getidentity
 705 Lustre: Server temp-MDT0000 on device /dev/sdb has started
 706 </screen>
 707         </listitem>
 708         <listitem xml:id="dbdoclet.create_and_mount_ost">
 709           <para>Create and mount
 710           <literal>ost0</literal>.</para>
 711           <para>In this example, the OSTs (
 712           <literal>ost0</literal> and
 713           <literal>ost1</literal>) are being created on different OSS nodes (
 714           <literal>oss0</literal> and
 715           <literal>oss1</literal> respectively).</para>
 716           <orderedlist>
 717             <listitem>
 718               <para>Create
 719               <literal>ost0</literal>. On
 720               <literal>oss0</literal> node, run:</para>
 721               <screen>
 722 [root@oss0 /]# mkfs.lustre --fsname=temp --mgsnode=10.2.0.1@tcp0 --ost
 723 --index=0 /dev/sdc
 724 </screen>
 725               <para>The command generates this output:</para>
 726               <screen>
 727     Permanent disk data:
 728 Target:            temp-OST0000
 729 Index:             0
 730 Lustre FS: temp
 731 Mount type:        ldiskfs
 732 Flags:             0x72
 733 (OST first_time update)
 734 Persistent mount opts: errors=remount-ro,extents,mballoc
 735 Parameters: mgsnode=10.2.0.1@tcp
 736
 737 checking for existing Lustre data: not found
 738 device size = 16MB
 739 2 6 18
 740 formatting backing filesystem ldiskfs on /dev/sdc
 741    target name             temp-OST0000
 742    4k blocks               0
 743    options                 -I 256 -q -O dir_index,uninit_groups -F
 744 mkfs_cmd = mkfs.ext2 -j -b 4096 -L temp-OST0000  -I 256 -q -O
 745 dir_index,uninit_groups -F /dev/sdc
 746 Writing CONFIGS/mountdata
 747 </screen>
 748             </listitem>
 749             <listitem>
 750               <para>Mount ost0 on the OSS on which it was created. On
 751               <literal>oss0</literal> node, run:</para>
 752               <screen>
 753 root@oss0 /] mount -t lustre /dev/sdc /mnt/ost0
 754 </screen>
 755               <para>The command generates this output:</para>
 756               <screen>
 757 LDISKFS-fs: file extents enabled
 758 LDISKFS-fs: mballoc enabled
 759 Lustre: temp-OST0000: new disk, initializing
 760 Lustre: Server temp-OST0000 on device /dev/sdb has started
 761 </screen>
 762               <para>Shortly afterwards, this output appears:</para>
 763               <screen>
 764 Lustre: temp-OST0000: received MDS connection from 10.2.0.1@tcp0
 765 Lustre: MDS temp-MDT0000: temp-OST0000_UUID now active, resetting orphans
 766 </screen>
 767             </listitem>
 768           </orderedlist>
 769         </listitem>
 770         <listitem>
 771           <para>Create and mount
 772           <literal>ost1</literal>.</para>
 773           <orderedlist>
 774             <listitem>
 775               <para>Create ost1. On
 776               <literal>oss1</literal> node, run:</para>
 777               <screen>
 778 [root@oss1 /]# mkfs.lustre --fsname=temp --mgsnode=10.2.0.1@tcp0 \
 779            --ost --index=1 /dev/sdd
 780 </screen>
 781               <para>The command generates this output:</para>
 782               <screen>
 783     Permanent disk data:
 784 Target:            temp-OST0001
 785 Index:             1
 786 Lustre FS: temp
 787 Mount type:        ldiskfs
 788 Flags:             0x72
 789 (OST first_time update)
 790 Persistent mount opts: errors=remount-ro,extents,mballoc
 791 Parameters: mgsnode=10.2.0.1@tcp
 792
 793 checking for existing Lustre data: not found
 794 device size = 16MB
 795 2 6 18
 796 formatting backing filesystem ldiskfs on /dev/sdd
 797    target name             temp-OST0001
 798    4k blocks               0
 799    options                 -I 256 -q -O dir_index,uninit_groups -F
 800 mkfs_cmd = mkfs.ext2 -j -b 4096 -L temp-OST0001  -I 256 -q -O
 801 dir_index,uninit_groups -F /dev/sdc
 802 Writing CONFIGS/mountdata
 803 </screen>
 804             </listitem>
 805             <listitem>
 806               <para>Mount ost1 on the OSS on which it was created. On
 807               <literal>oss1</literal> node, run:</para>
 808               <screen>
 809 root@oss1 /] mount -t lustre /dev/sdd /mnt/ost1
 810 </screen>
 811               <para>The command generates this output:</para>
 812               <screen>
 813 LDISKFS-fs: file extents enabled
 814 LDISKFS-fs: mballoc enabled
 815 Lustre: temp-OST0001: new disk, initializing
 816 Lustre: Server temp-OST0001 on device /dev/sdb has started
 817 </screen>
 818               <para>Shortly afterwards, this output appears:</para>
 819               <screen>
 820 Lustre: temp-OST0001: received MDS connection from 10.2.0.1@tcp0
 821 Lustre: MDS temp-MDT0000: temp-OST0001_UUID now active, resetting orphans
 822 </screen>
 823             </listitem>
 824           </orderedlist>
 825         </listitem>
 826         <listitem>
 827           <para>Mount the Lustre file system on the client. On the client node,
 828           run:</para>
 829           <screen>
 830 root@client1 /] mount -t lustre 10.2.0.1@tcp0:/temp /lustre
 831 </screen>
 832           <para>This command generates this output:</para>
 833           <screen>
 834 Lustre: Client temp-client has started
 835 </screen>
 836         </listitem>
 837         <listitem>
 838           <para>Verify that the file system started and is working by running
 839           the
 840           <literal>df</literal>,
 841           <literal>dd</literal> and
 842           <literal>ls</literal> commands on the client node.</para>
 843           <orderedlist>
 844             <listitem>
 845               <para>Run the
 846               <literal>lfs df -h</literal> command:</para>
 847               <screen>
 848 [root@client1 /] lfs df -h
 849 </screen>
 850               <para>The
 851               <literal>lfs df -h</literal> command lists space usage per OST and
 852               the MDT in human-readable format. This command generates output
 853               similar to this:</para>
 854               <screen>
 855 UUID               bytes      Used      Available   Use%    Mounted on
 856 temp-MDT0000_UUID  8.0G      400.0M       7.6G        0%      /lustre[MDT:0]
 857 temp-OST0000_UUID  800.0G    400.0M     799.6G        0%      /lustre[OST:0]
 858 temp-OST0001_UUID  800.0G    400.0M     799.6G        0%      /lustre[OST:1]
 859 filesystem summary:  1.6T    800.0M       1.6T        0%      /lustre
 860 </screen>
 861             </listitem>
 862             <listitem>
 863               <para>Run the
 864               <literal>lfs df -ih</literal> command.</para>
 865               <screen>
 866 [root@client1 /] lfs df -ih
 867 </screen>
 868               <para>The
 869               <literal>lfs df -ih</literal> command lists inode usage per OST
 870               and the MDT. This command generates output similar to
 871               this:</para>
 872               <screen>
 873 UUID              Inodes      IUsed       IFree   IUse%     Mounted on
 874 temp-MDT0000_UUID   2.5M        32         2.5M      0%       /lustre[MDT:0]
 875 temp-OST0000_UUID   5.5M        54         5.5M      0%       /lustre[OST:0]
 876 temp-OST0001_UUID   5.5M        54         5.5M      0%       /lustre[OST:1]
 877 filesystem summary: 2.5M        32         2.5M      0%       /lustre
 878 </screen>
 879             </listitem>
 880             <listitem>
 881               <para>Run the
 882               <literal>dd</literal> command:</para>
 883               <screen>
 884 [root@client1 /] cd /lustre
 885 [root@client1 /lustre] dd if=/dev/zero of=/lustre/zero.dat bs=4M count=2
 886 </screen>
 887               <para>The
 888               <literal>dd</literal> command verifies write functionality by
 889               creating a file containing all zeros (
 890               <literal>0</literal>s). In this command, an 8 MB file is created.
 891               This command generates output similar to this:</para>
 892               <screen>
 893 2+0 records in
 894 2+0 records out
 895 8388608 bytes (8.4 MB) copied, 0.159628 seconds, 52.6 MB/s
 896 </screen>
 897             </listitem>
 898             <listitem>
 899               <para>Run the
 900               <literal>ls</literal> command:</para>
 901               <screen>
 902 [root@client1 /lustre] ls -lsah
 903 </screen>
 904               <para>The
 905               <literal>ls -lsah</literal> command lists files and directories in
 906               the current working directory. This command generates output
 907               similar to this:</para>
 908               <screen>
 909 total 8.0M
 910 4.0K drwxr-xr-x  2 root root 4.0K Oct 16 15:27 .
 911 8.0K drwxr-xr-x 25 root root 4.0K Oct 16 15:27 ..
 912 8.0M -rw-r--r--  1 root root 8.0M Oct 16 15:27 zero.dat
 913
 914 </screen>
 915             </listitem>
 916           </orderedlist>
 917         </listitem>
 918       </orderedlist>
 919       <para>Once the Lustre file system is configured, it is ready for
 920       use.</para>
 921     </section>
 922   </section>
 923   <section xml:id="dbdoclet.50438267_76752">
 924     <title>
 925     <indexterm>
 926       <primary>Lustre</primary>
 927       <secondary>configuring</secondary>
 928       <tertiary>additional options</tertiary>
 929     </indexterm>Additional Configuration Options</title>
 930     <para>This section describes how to scale the Lustre file system or make
 931     configuration changes using the Lustre configuration utilities.</para>
 932     <section remap="h3">
 933       <title>
 934       <indexterm>
 935         <primary>Lustre</primary>
 936         <secondary>configuring</secondary>
 937         <tertiary>for scale</tertiary>
 938       </indexterm>Scaling the Lustre File System</title>
 939       <para>A Lustre file system can be scaled by adding OSTs or clients. For
 940       instructions on creating additional OSTs repeat Step
 941       <xref linkend="dbdoclet.create_and_mount_ost" />and Step
 942       <xref linkend="dbdoclet.mount_ost" />above. For mounting
 943       additional clients, repeat Step
 944       <xref linkend="dbdoclet.mount_on_client" />for each client.</para>
 945     </section>
 946     <section remap="h3">
 947       <title>
 948       <indexterm>
 949         <primary>Lustre</primary>
 950         <secondary>configuring</secondary>
 951         <tertiary>striping</tertiary>
 952       </indexterm>Changing Striping Defaults</title>
 953       <para>The default settings for the file layout stripe pattern are shown
 954       in
 955       <xref linkend="configuringlustre.tab.stripe" />.</para>
 956       <table frame="none" xml:id="configuringlustre.tab.stripe">
 957         <title>Default stripe pattern</title>
 958         <tgroup cols="3">
 959           <colspec colname="c1" colwidth="13*" />
 960           <colspec colname="c2" colwidth="13*" />
 961           <colspec colname="c3" colwidth="13*" />
 962           <tbody>
 963             <row>
 964               <entry>
 965                 <para>
 966                   <emphasis role="bold">File Layout Parameter</emphasis>
 967                 </para>
 968               </entry>
 969               <entry>
 970                 <para>
 971                   <emphasis role="bold">Default</emphasis>
 972                 </para>
 973               </entry>
 974               <entry>
 975                 <para>
 976                   <emphasis role="bold">Description</emphasis>
 977                 </para>
 978               </entry>
 979             </row>
 980             <row>
 981               <entry>
 982                 <para>
 983                   <literal>stripe_size</literal>
 984                 </para>
 985               </entry>
 986               <entry>
 987                 <para>1 MB</para>
 988               </entry>
 989               <entry>
 990                 <para>Amount of data to write to one OST before moving to the
 991                 next OST.</para>
 992               </entry>
 993             </row>
 994             <row>
 995               <entry>
 996                 <para>
 997                   <literal>stripe_count</literal>
 998                 </para>
 999               </entry>
1000               <entry>
1001                 <para>1</para>
1002               </entry>
1003               <entry>
1004                 <para>The number of OSTs to use for a single file.</para>
1005               </entry>
1006             </row>
1007             <row>
1008               <entry>
1009                 <para>
1010                   <literal>start_ost</literal>
1011                 </para>
1012               </entry>
1013               <entry>
1014                 <para>-1</para>
1015               </entry>
1016               <entry>
1017                 <para>The first OST where objects are created for each file.
1018                 The default -1 allows the MDS to choose the starting index
1019                 based on available space and load balancing.
1020                 <emphasis>It's strongly recommended not to change the default
1021                 for this parameter to a value other than -1.</emphasis></para>
1022               </entry>
1023             </row>
1024           </tbody>
1025         </tgroup>
1026       </table>
1027       <para>Use the
1028       <literal>lfs setstripe</literal> command described in
1029       <xref linkend="managingstripingfreespace" />to change the file layout
1030       configuration.</para>
1031     </section>
1032     <section remap="h3">
1033       <title>
1034       <indexterm>
1035         <primary>Lustre</primary>
1036         <secondary>configuring</secondary>
1037         <tertiary>utilities</tertiary>
1038       </indexterm>Using the Lustre Configuration Utilities</title>
1039       <para>If additional configuration is necessary, several configuration
1040       utilities are available:</para>
1041       <itemizedlist>
1042         <listitem>
1043           <para>
1044           <literal>mkfs.lustre</literal>- Use to format a disk for a Lustre
1045           service.</para>
1046         </listitem>
1047         <listitem>
1048           <para>
1049           <literal>tunefs.lustre</literal>- Use to modify configuration
1050           information on a Lustre target disk.</para>
1051         </listitem>
1052         <listitem>
1053           <para>
1054           <literal>lctl</literal>- Use to directly control Lustre features via
1055           an
1056           <literal>ioctl</literal> interface, allowing various configuration,
1057           maintenance and debugging features to be accessed.</para>
1058         </listitem>
1059         <listitem>
1060           <para>
1061           <literal>mount.lustre</literal>- Use to start a Lustre client or
1062           target service.</para>
1063         </listitem>
1064       </itemizedlist>
1065       <para>For examples using these utilities, see the topic
1066       <xref linkend="systemconfigurationutilities" /></para>
1067       <para>The
1068       <literal>lfs</literal> utility is useful for configuring and querying a
1069       variety of options related to files. For more information, see
1070       <xref linkend="userutilities" />.</para>
1071       <note>
1072         <para>Some sample scripts are included in the directory where the
1073         Lustre software is installed. If you have installed the Lustre source
1074         code, the scripts are located in the
1075         <literal>lustre/tests</literal> sub-directory. These scripts enable
1076         quick setup of some simple standard Lustre configurations.</para>
1077       </note>
1078     </section>
1079   </section>
1080 </chapter>