<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink">
+<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='settinguplustresystem'>
<info>
- <title>Setting Up a Lustre File System</title>
+ <title xml:id='settinguplustresystem.title'>Setting Up a Lustre File System</title>
</info>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1290293" xreflabel=""/>This chapter describes hardware configuration requirements for a Lustre file system including:</para>
- <itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1290573" xreflabel=""/><link xl:href="SettingUpLustreSystem.html#50438256_49017">Hardware Considerations</link></para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1291160" xreflabel=""/><link xl:href="SettingUpLustreSystem.html#50438256_31079">Determining Space Requirements</link></para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292400" xreflabel=""/><link xl:href="SettingUpLustreSystem.html#50438256_84701">Setting File System Formatting Options</link></para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1294481" xreflabel=""/><link xl:href="SettingUpLustreSystem.html#50438256_26456">Determining Memory Requirements</link></para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1291164" xreflabel=""/><link xl:href="SettingUpLustreSystem.html#50438256_78272">Implementing Networks To Be Used by Lustre</link></para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-</itemizedlist>
- <section remap="h2">
- <title><anchor xml:id="dbdoclet.50438256_pgfId-1290296" xreflabel=""/></title>
- <section remap="h2">
- <title>5.1 <anchor xml:id="dbdoclet.50438256_49017" xreflabel=""/>Hardware Considerations</title>
+
+
+ <itemizedlist>
+ <listitem>
+ <para>
+ <link linkend='dbdoclet.50438256_49017'/>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <link linkend='dbdoclet.50438256_31079'/>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <link linkend='dbdoclet.50438256_84701'/>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <link linkend='dbdoclet.50438256_26456'/>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <link linkend='dbdoclet.50438256_78272'/>
+ </para>
+ </listitem>
+ </itemizedlist>
+
+
+
+ <section xml:id="dbdoclet.50438256_49017">
+ <title>5.1 Hardware Considerations</title>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1292812" xreflabel=""/>Lustre can work with any kind of block storage device such as single disks, software RAID, hardware RAID, or a logical volume manager. In contrast to some networked file systems, the block devices are only attached to the MDS and OSS nodes in Lustre and are not accessed by the clients directly.</para>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1292817" xreflabel=""/>Since the block devices are accessed by only one or two server nodes, a storage area network (SAN) that is accessible from all the servers is not required. Expensive switches are not needed because point-to-point connections between the servers and the storage arrays normally provide the simplest and best attachments. (If failover capability is desired, the storage must be attached to multiple servers.)</para>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1290297" xreflabel=""/>For a production environment, it is preferable that the MGS have separate storage to allow future expansion to multiple file systems. However, it is possible to run the MDS and MGS on the same machine and have them share the same storage device.</para>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1292472" xreflabel=""/>The storage attached to the servers typically uses RAID to provide fault tolerance and can optionally be organized with logical volume management (LVM). It is then formatted by Lustre as a file system. Lustre OSS and MDS servers read, write and modify data in the format imposed by the file system.</para>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1292545" xreflabel=""/>Lustre uses journaling file system technology on both the MDTs and OSTs. For a MDT, as much as a 20 percent performance gain can be obtained by placing the journal on a separate device.</para>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1292546" xreflabel=""/>The MDS can effectively utilize a lot of CPU cycles. A minimium of four processor cores are recommended. More are advisable for files systems with many clients.</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438256_pgfId-1292627" xreflabel=""/>Lustre clients running on architectures with different endianness are supported. One limitation is that the PAGE_SIZE kernel macro on the client must be as large as the PAGE_SIZE of the server. In particular, ia64 or PPC clients with large pages (up to 64kB pages) can run with x86 servers (4kB pages). If you are running x86 clients with ia64 or PPC servers, you must compile the ia64 kernel with a 4kB PAGE_SIZE (so the server page size is not larger than the client page size). <anchor xml:id="dbdoclet.50438256_51943" xreflabel=""/></para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+ <note>
+ <para>Lustre clients running on architectures with different endianness are supported. One limitation is that the PAGE_SIZE kernel macro on the client must be as large as the PAGE_SIZE of the server. In particular, ia64 or PPC clients with large pages (up to 64kB pages) can run with x86 servers (4kB pages). If you are running x86 clients with ia64 or PPC servers, you must compile the ia64 kernel with a 4kB PAGE_SIZE (so the server page size is not larger than the client page size). <anchor xml:id="dbdoclet.50438256_51943" xreflabel=""/></para>
+ </note>
+
<section remap="h3">
<title><anchor xml:id="dbdoclet.50438256_pgfId-1293428" xreflabel=""/>5.1.1 MDT Storage Hardware Considerations</title>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1293438" xreflabel=""/>The data access pattern for MDS storage is a database-like access pattern with many seeks and read-and-writes of small amounts of data. High throughput to MDS storage is not important. Storage types that provide much lower seek times, such as high-RPM SAS or SSD drives can be used for the MDT.</para>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1295314" xreflabel=""/>For maximum performance, the MDT should be configured as RAID1 with an internal journal and two disks from different controllers.</para>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1295315" xreflabel=""/>If you need a larger MDT, create multiple RAID1 devices from pairs of disks, and then make a RAID0 array of the RAID1 devices. This ensures maximum reliability because multiple disk failures only have a small chance of hitting both disks in the same RAID1 device.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1295316" xreflabel=""/> Doing the opposite (RAID1 of a pair of RAID0 devices) has a 50% chance that even two disk failures can cause the loss of the whole MDT device. The first failure disables an entire half of the mirror and the second failure has a 50% chance of disabling the remaining mirror.</para>
+ <para><anchor xml:id="dbdoclet.50438256_pgfId-1295316" xreflabel=""/>Doing the opposite (RAID1 of a pair of RAID0 devices) has a 50% chance that even two disk failures can cause the loss of the whole MDT device. The first failure disables an entire half of the mirror and the second failure has a 50% chance of disabling the remaining mirror.</para>
</section>
<section remap="h3">
<title><anchor xml:id="dbdoclet.50438256_pgfId-1295312" xreflabel=""/>5.1.2 OST Storage Hardware Considerations</title>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1293431" xreflabel=""/>Lustre file system capacity is the sum of the capacities provided by the targets. For example, 64 OSSs, each with two 8 TB targets, provide a file system with a capacity of nearly 1 PB. If each OST uses ten 1 TB SATA disks (8 data disks plus 2 parity disks in a RAID 6 configuration), it may be possible to get 50 MB/sec from each drive, providing up to 400 MB/sec of disk bandwidth per OST. If this system is used as storage backend with a system network like InfiniBand that provides a similar bandwidth, then each OSS could provide 800 MB/sec of end-to-end I/O throughput. (Although the architectural constraints described here are simple, in practice it takes careful hardware selection, benchmarking and integration to obtain such results.)</para>
</section>
</section>
- <section remap="h2">
- <title>5.2 <anchor xml:id="dbdoclet.50438256_31079" xreflabel=""/>Determining Space Requirements</title>
+ <section xml:id="dbdoclet.50438256_31079" >
+ <title>5.2 Determining Space Requirements</title>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1293293" xreflabel=""/>The desired performance characteristics of the backing file systems on the MDT and OSTs are independent of one another. The size of the MDT backing file system depends on the number of inodes needed in the total Lustre file system, while the aggregate OST space depends on the total amount of data stored on the file system.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1293404" xreflabel=""/>Each time a file is created on a Lustre file system, it consumes one inode on the MDT and one inode for each OST object over which the file is striped. Normally, each file’s stripe count is based on the system-wide default stripe count. However, this can be changed for individual files using the lfssetstripe option. For more details,see <link xl:href="ManagingStripingFreeSpace.html#50438209_78664">Setting the File Layout/Striping Configuration (lfs setstripe)</link>.</para>
+ <para><anchor xml:id="dbdoclet.50438256_pgfId-1293404" xreflabel=""/>Each time a file is created on a Lustre file system, it consumes one inode on the MDT and one inode for each OST object over which the file is striped. Normally, each file’s stripe count is based on the system-wide default stripe count. However, this can be changed for individual files using the lfssetstripe option. For more details, see <xref linkend='managingstripingfreespace'/>.</para>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1293400" xreflabel=""/>In a Lustre ldiskfs file system, all the inodes are allocated on the MDT and OSTs when the file system is first formatted. The total number of inodes on a formatted MDT or OST cannot be easily changed, although it is possible to add OSTs with additional space and corresponding inodes. Thus, the number of inodes created at format time should be generous enough to anticipate future expansion.</para>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1293818" xreflabel=""/>When the file system is in use and a file is created, the metadata associated with that file is stored in one of the pre-allocated inodes and does not consume any of the free space used to store file data.</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438256_pgfId-1293451" xreflabel=""/>By default, the ldiskfs file system used by Lustre servers to store user-data objects and system data reserves 5% of space that cannot be used by Lustre. Additionally, Lustre reserves up to 400 MB on each OST for journal use and a small amount of space outside the journal to store accounting data for Lustre. This reserved space is unusable for general storage. Thus, at least 400 MB of space is used on each OST before any file object data is saved.<anchor xml:id="dbdoclet.50438256_74070" xreflabel=""/></para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+ <note>
+<para>By default, the ldiskfs file system used by Lustre servers to store user-data objects and system data reserves 5% of space that cannot be used by Lustre. Additionally, Lustre reserves up to 400 MB on each OST for journal use and a small amount of space outside the journal to store accounting data for Lustre. This reserved space is unusable for general storage. Thus, at least 400 MB of space is used on each OST before any file object data is saved.<anchor xml:id="dbdoclet.50438256_74070" xreflabel=""/></para>
+</note>
+
<section remap="h3">
<title><anchor xml:id="dbdoclet.50438256_pgfId-1292700" xreflabel=""/>5.2.1 <anchor xml:id="dbdoclet.50438256_87676" xreflabel=""/>Determining MDS/MDT Space Requirements</title>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1293355" xreflabel=""/>When calculating the MDT size, the important factor to consider is the number of files to be stored in the file system. This determines the number of inodes needed, which drives the MDT sizing. To be on the safe side, plan for 4 KB per inode on the MDT, which is the default value. Attached storage required for Lustre metadata is typically 1-2 percent of the file system capacity depending upon file size.</para>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1293527" xreflabel=""/>We recommend that you use at least twice the minimum number of inodes to allow for future expansion and allow for an average file size smaller than expected. Thus, the required space is:</para>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1293545" xreflabel=""/>4 KB/inode * 40 million inodes = 160 GB</para>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1293135" xreflabel=""/>If the average file size is small, 4 KB for example, Lustre is not very efficient as the MDT uses as much space as the OSTs. However, this is not a common configuration for Lustre.</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438256_pgfId-1293142" xreflabel=""/>If the MDT is too small, this can cause all the space on the OSTs to be unusable. Be sure to determine the appropriate size of the MDT needed to support the file system before formatting the file system. It is difficult to increase the number of inodes after the file system is formatted.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+ <note>
+<para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438256_pgfId-1293142" xreflabel=""/>If the MDT is too small, this can cause all the space on the OSTs to be unusable. Be sure to determine the appropriate size of the MDT needed to support the file system before formatting the file system. It is difficult to increase the number of inodes after the file system is formatted.</para>
+</note>
</section>
+
<section remap="h3">
<title><anchor xml:id="dbdoclet.50438256_pgfId-1293490" xreflabel=""/>5.2.2 Determining OSS/OST Space Requirements</title>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1293491" xreflabel=""/>For the OST, the amount of space taken by each object depends on the usage pattern of the users/applications running on the system. Lustre defaults to a conservative estimate for the object size (16 KB per object). If you are confident that the average file size for your applications will be larger than this, you can specify a larger average file size (fewer total inodes) to reduce file system overhead and minimize file system check time. See <emphasis><link xl:href="SettingUpLustreSystem.html#50438256_53886">Setting the Number of Inodes for an OST</link></emphasis> for more details.</para>
+ <para><anchor xml:id="dbdoclet.50438256_pgfId-1293491" xreflabel=""/>For the OST, the amount of space taken by each object depends on the usage pattern of the users/applications running on the system. Lustre defaults to a conservative estimate for the object size (16 KB per object). If you are confident that the average file size for your applications will be larger than this, you can specify a larger average file size (fewer total inodes) to reduce file system overhead and minimize file system check time. See <xref linkend="dbdoclet.50438256_53886"/> for more details.</para>
</section>
</section>
- <section remap="h2">
- <title>5.3 <anchor xml:id="dbdoclet.50438256_84701" xreflabel=""/>Setting File System Formatting Options</title>
+ <section xml:id="dbdoclet.50438256_84701" >
+ <title>5.3 Setting File System Formatting Options</title>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1293227" xreflabel=""/>To override the default formatting options for any of the Lustre backing file systems, use this argument to mkfs.lustre to pass formatting options to the backing mkfs:</para>
<screen><anchor xml:id="dbdoclet.50438256_pgfId-1293982" xreflabel=""/>--mkfsoptions='backing fs options'</screen>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1293480" xreflabel=""/>For other options to format backing ldiskfs filesystems, see the Linux man page for mke2fs(8).</para>
<para> Alternately, if you know the average file size, then you can specify the OST inode count for the OST file systems using:</para>
<screen><anchor xml:id="dbdoclet.50438256_pgfId-1294260" xreflabel=""/>-i <emphasis><average_file_size</emphasis> / (<emphasis>number_of_stripes</emphasis> * 4)<emphasis>></emphasis></screen>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1294261" xreflabel=""/>For example, if the average file size is 16 MB and there are, by default 4 stripes per file, then --mkfsoptions='-i 1048576' would be appropriate.</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438256_pgfId-1293240" xreflabel=""/>In addition to the number of inodes, file system check time on OSTs is affected by a number of other variables: size of the file system, number of allocated blocks, distribution of allocated blocks on the disk, disk speed, CPU speed, and amount of RAM on the server. Reasonable file system check times (without serious file system problems), are expected to take five and thirty minutes per TB.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1293243" xreflabel=""/>For more details on formatting MDT and OST file systems, see <link xl:href="ConfiguringStorage.html#50438208_51921">Formatting Options for RAID Devices</link>.</para>
+ <note>
+<para>In addition to the number of inodes, file system check time on OSTs is affected by a number of other variables: size of the file system, number of allocated blocks, distribution of allocated blocks on the disk, disk speed, CPU speed, and amount of RAM on the server. Reasonable file system check times (without serious file system problems), are expected to take five and thirty minutes per TB.</para>
+</note>
+
+ <para><anchor xml:id="dbdoclet.50438256_pgfId-1293243" xreflabel=""/>For more details on formatting MDT and OST file systems, see <xref linkend='dbdoclet.50438208_51921'/>.</para>
</section>
<section remap="h3">
<title><anchor xml:id="dbdoclet.50438256_pgfId-1290698" xreflabel=""/>5.3.4 File and File System Limits</title>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1290699" xreflabel=""/><link xl:href="SettingUpLustreSystem.html#50438256_27068">TABLE 5-1</link> describes file and file system size limits. These limits are imposed by either the Lustre architecture or the Linux virtual file system (VFS) and virtual memory subsystems. In a few cases, a limit is defined within the code and can be changed by re-compiling Lustre (see <link xl:href="InstallingLustrefrSourceCode.html#50438210_75891">Chapter 29</link>: <link xl:href="InstallingLustrefrSourceCode.html#50438210_67391">Installing Lustre from Source Code</link>). In these cases, the indicated limit was used for Lustre testing. <anchor xml:id="dbdoclet.50438256_27479" xreflabel=""/><anchor xml:id="dbdoclet.50438256_46859" xreflabel=""/></para>
+ <para><xref linkend='settinguplustresystem.tab1'/> describes file and file system size limits. These limits are imposed by either the Lustre architecture or the Linux virtual file system (VFS) and virtual memory subsystems. In a few cases, a limit is defined within the code and can be changed by re-compiling Lustre (see <xref linkend='installinglustrefromsourcecode'/><link xl:href="InstallingLustrefrSourceCode.html#50438210_75891">Chapter 29 : Installing Lustre from Source Code</link>). In these cases, the indicated limit was used for Lustre testing. </para>
+
<table frame="all">
- <title><anchor xml:id="dbdoclet.50438256_pgfId-1290702" xreflabel=""/> TABLE 5-1 <anchor xml:id="dbdoclet.50438256_27068" xreflabel=""/> File and file system limits</title>
+ <title xml:id='settinguplustresystem.tab1'>File and file system limits</title>
<tgroup cols="3">
- <colspec colname="c1" colwidth="33*"/>
- <colspec colname="c2" colwidth="33*"/>
- <colspec colname="c3" colwidth="33*"/>
+ <colspec colname="c1" colwidth="3*"/>
+ <colspec colname="c2" colwidth="2*"/>
+ <colspec colname="c3" colwidth="4*"/>
<thead>
<row>
<entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438256_pgfId-1290708" xreflabel=""/>Limit</emphasis></para></entry>
<row>
<entry><para> <anchor xml:id="dbdoclet.50438256_pgfId-1290779" xreflabel=""/>Maximum number of files in the file system</para></entry>
<entry><para> <anchor xml:id="dbdoclet.50438256_pgfId-1290781" xreflabel=""/>4 billion</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438256_pgfId-1290783" xreflabel=""/>The ldiskfs file system imposes an upper limit of 4 billion inodes. By default, the MDS file system is formatted with 4 KB of space per inode, meaning 512 million inodes per file system of 2 TB.</para><para><anchor xml:id="dbdoclet.50438256_pgfId-1291983" xreflabel=""/>This can be increased initially, at the time of MDS file system creation. For more information, see <link xl:href="SettingUpLustreSystem.html#50438256_84701">Setting File System Formatting Options</link>.</para></entry>
+ <entry><para> <anchor xml:id="dbdoclet.50438256_pgfId-1290783" xreflabel=""/>The ldiskfs file system imposes an upper limit of 4 billion inodes. By default, the MDS file system is formatted with 4 KB of space per inode, meaning 512 million inodes per file system of 2 TB.</para><para><anchor xml:id="dbdoclet.50438256_pgfId-1291983" xreflabel=""/>This can be increased initially, at the time of MDS file system creation. For more information, see <xref linkend='settinguplustresystem'/><link xl:href="SettingUpLustreSystem.html#50438256_84701">Setting File System Formatting Options</link>.</para></entry>
</row>
<row>
<entry><para> <anchor xml:id="dbdoclet.50438256_pgfId-1290792" xreflabel=""/>Maximum length of a <anchor xml:id="dbdoclet.50438256_marker-1290791" xreflabel=""/>filename</para></entry>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1290676" xreflabel=""/> </para>
</section>
</section>
- <section remap="h2">
- <title>5.4
<anchor xml:id="dbdoclet.50438256_26456" xreflabel=""/>Determining Memory<anchor xml:id="dbdoclet.50438256_marker-1292212" xreflabel=""/> Requirements</title>
+ <section xml:id="dbdoclet.50438256_26456">
+ <title>5.4 Determining Memory<anchor xml:id="dbdoclet.50438256_marker-1292212" xreflabel=""/> Requirements</title>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1292214" xreflabel=""/>This section describes the memory requirements for each Lustre file system component.</para>
<section remap="h3">
<title><anchor xml:id="dbdoclet.50438256_pgfId-1292221" xreflabel=""/>5.4.1 Client Memory Requirements</title>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1292226" xreflabel=""/> Number of clients</para>
</listitem>
<listitem>
- <para> </para>
- </listitem>
-<listitem>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1292227" xreflabel=""/> Size of the directories</para>
</listitem>
<listitem>
- <para> </para>
- </listitem>
-<listitem>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1292228" xreflabel=""/> Load placed on server</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
</itemizedlist>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1292229" xreflabel=""/>The amount of memory used by the MDS is a function of how many clients are on the system, and how many files they are using in their working set. This is driven, primarily, by the number of locks a client can hold at one time. The number of locks held by clients varies by load and memory availability on the server. Interactive clients can hold in excess of 10,000 locks at times. On the MDS, memory usage is approximately 2 KB per file, including the Lustre distributed lock manager (DLM) lock and kernel data structures for the files currently in use. Having file data in cache can improve metadata performance by a factor of 10x or more compared to reading it from disk.</para>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1292230" xreflabel=""/>MDS memory requirements include:</para>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1294632" xreflabel=""/><emphasis role="bold">File system metadata</emphasis> : A reasonable amount of RAM needs to be available for file system metadata. While no hard limit can be placed on the amount of file system metadata, if more RAM is available, then the disk I/O is needed less often to retrieve the metadata.</para>
</listitem>
<listitem>
- <para> </para>
- </listitem>
-<listitem>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1294633" xreflabel=""/><emphasis role="bold">Network transport</emphasis> : If you are using TCP or other network transport that uses system memory for send/receive buffers, this memory requirement must also be taken into consideration.</para>
</listitem>
<listitem>
- <para> </para>
- </listitem>
-<listitem>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1294399" xreflabel=""/><emphasis role="bold">Journal size</emphasis> : By default, the journal size is 400 MB for each Lustre ldiskfs file system. This can pin up to an equal amount of RAM on the MDS node per file system.</para>
</listitem>
<listitem>
- <para> </para>
- </listitem>
-<listitem>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1292234" xreflabel=""/><emphasis role="bold">Failover configuration</emphasis> : If the MDS node will be used for failover from another node, then the RAM for each journal should be doubled, so the backup server can handle the additional load if the primary server fails.</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
</itemizedlist>
<section remap="h4">
<title><anchor xml:id="dbdoclet.50438256_pgfId-1292235" xreflabel=""/>5.4.2.1 Calculating MDS Memory Requirements</title>
</section>
</section>
<section remap="h3">
- <title><anchor xml:id="dbdoclet.50438256_pgfId-1292246" xreflabel=""/>5.4.3 <anchor xml:id="dbdoclet.50438256_84553" xreflabel=""/>OSS Memory <anchor xml:id="dbdoclet.50438256_marker-1292245" xreflabel=""/>Requirements</title>
+ <title><anchor xml:id="dbdoclet.50438256_pgfId-1292246" xreflabel=""/>5.4.3 <anchor xml:id="dbdoclet.50438256_84553" xreflabel=""/>OSS Memory Requirements</title>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1292248" xreflabel=""/>When planning the hardware for an OSS node, consider the memory usage of several components in the Lustre system (i.e., journal, service threads, file system metadata, etc.). Also, consider the effect of the OSS read cache feature, which consumes memory as it caches data on the OSS node.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1292249" xreflabel=""/>In addition to the MDS memory requirements mentioned in <link xl:href="SettingUpLustreSystem.html#50438256_87676">Determining MDS/MDT Space Requirements</link>, the OSS requirements include:</para>
+ <para><anchor xml:id="dbdoclet.50438256_pgfId-1292249" xreflabel=""/>In addition to the MDS memory requirements mentioned in <xref linkend="dbdoclet.50438256_87676"/>, the OSS requirements include:</para>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1292250" xreflabel=""/><emphasis role="bold">Service threads</emphasis> : The service threads on the OSS node pre-allocate a 1 MB I/O buffer for each ost_io service thread, so these buffers do not need to be allocated and freed for each I/O request.</para>
</listitem>
<listitem>
- <para> </para>
- </listitem>
-<listitem>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1292251" xreflabel=""/><emphasis role="bold">OSS read cache</emphasis> : OSS read cache provides read-only caching of data on an OSS, using the regular Linux page cache to store the data. Just like caching from a regular file system in Linux, OSS read cache uses as much physical memory as is available.</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
</itemizedlist>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1292252" xreflabel=""/>The same calculation applies to files accessed from the OSS as for the MDS, but the load is distributed over many more OSSs nodes, so the amount of memory required for locks, inode cache, etc. listed under MDS is spread out over the OSS nodes. as shown in 5.2.3.1.</para>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1292253" xreflabel=""/>Because of these memory requirements, the following calculations should be taken as determining the absolute minimum RAM required in an OSS node.</para>
</section>
</section>
</section>
- <section remap="h2">
- <title>5.5 <anchor xml:id="dbdoclet.50438256_78272" xreflabel=""/>Implementing Networks To Be Used by Lustre</title>
+ <section xml:id="dbdoclet.50438256_78272">
+ <title>5.5 Implementing Networks To Be Used by Lustre</title>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1290491" xreflabel=""/>As a high performance file system, Lustre places heavy loads on networks. Thus, a network interface in each Lustre server and client is commonly dedicated to Lustre traffic. This is often a dedicated TCP/IP subnet, although other network hardware can also be used.</para>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1290944" xreflabel=""/>A typical Lustre implementation may include the following:</para>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1290945" xreflabel=""/> A high-performance backend network for the Lustre servers, typically an InfiniBand (IB) network.</para>
</listitem>
<listitem>
- <para> </para>
- </listitem>
-<listitem>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1290946" xreflabel=""/> A larger client network.</para>
</listitem>
<listitem>
- <para> </para>
- </listitem>
-<listitem>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1290947" xreflabel=""/> Lustre routers to connect the two networks.</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
</itemizedlist>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1291003" xreflabel=""/>Lustre networks and routing are configured and managed by specifying parameters to the Lustre Networking (lnet) module in /etc/modprobe.conf or /etc/modprobe.conf.local (depending on your Linux distribution).</para>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1290498" xreflabel=""/>To prepare to configure Lustre Networking, complete the following steps:</para>
+ <orderedlist>
+ <listitem>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1290499" xreflabel=""/> 1. Identify all machines that will be running Lustre and the network interfaces they will use to run Lustre traffic. These machines will form the Lustre network.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1290500" xreflabel=""/>A network is a group of nodes that communicate directly with one another. Lustre includes Lustre network drivers (LNDs) to support a variety of network types and hardware (see <link xl:href="UnderstandingLustreNetworking.html#50438191_53094">Chapter 2</link>: <link xl:href="UnderstandingLustreNetworking.html#50438191_40026">Understanding Lustre Networking (LNET)</link> for a complete list). The standard rules for specifying networks applies to Lustre networks. For example, two TCP networks on two different subnets (tcp0 and tcp1) are considered to be two different Lustre networks.</para>
+ <para><anchor xml:id="dbdoclet.50438256_pgfId-1290500" xreflabel=""/>A network is a group of nodes that communicate directly with one another. Lustre includes Lustre network drivers (LNDs) to support a variety of network types and hardware (see <xref linkend='understandinglustrenetworking'/> for a complete list). The standard rules for specifying networks applies to Lustre networks. For example, two TCP networks on two different subnets (tcp0 and tcp1) are considered to be two different Lustre networks.</para>
+ </listitem>
+ <listitem>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1290501" xreflabel=""/> 2. If routing is needed, identify the nodes to be used to route traffic between networks.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1291115" xreflabel=""/>If you are using multiple network types, then you’ll need a router. Any node with appropriate interfaces can route Lustre Networking (LNET) traffic between different network hardware types or topologies --the node may be a server, a client, or a standalone router. LNET can route messages between different network types (such as TCP-to-InfiniBand) or across different topologies (such as bridging two InfiniBand or TCP/IP networks). Routing will be configured in <link xl:href="ConfiguringLNET.html#50438216_64580">Chapter 9</link>: <link xl:href="ConfiguringLNET.html#50438216_29256">Configuring Lustre Networking (LNET)</link>.</para>
+ <para><anchor xml:id="dbdoclet.50438256_pgfId-1291115" xreflabel=""/>If you are using multiple network types, then you’ll need a router. Any node with appropriate interfaces can route Lustre Networking (LNET) traffic between different network hardware types or topologies --the node may be a server, a client, or a standalone router. LNET can route messages between different network types (such as TCP-to-InfiniBand) or across different topologies (such as bridging two InfiniBand or TCP/IP networks). Routing will be configured in <xref linkend='configuringlnet'/>.</para>
+ </listitem>
+ <listitem>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1290503" xreflabel=""/> 3. Identify the network interfaces to include in or exclude from LNET.</para>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1290504" xreflabel=""/>If not explicitly specified, LNET uses either the first available interface or a pre-defined default for a given network type. Interfaces that LNET should not use (such as an administrative network or IP-over-IB), can be excluded.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1291053" xreflabel=""/>Network interfaces to be used or excluded will be specified using the lnet kernel module parameters networks and ip2netsas described in <link xl:href="ConfiguringLNET.html#50438216_64580">Chapter 9</link>: <link xl:href="ConfiguringLNET.html#50438216_29256">Configuring Lustre Networking (LNET)</link>.</para>
+ <para><anchor xml:id="dbdoclet.50438256_pgfId-1291053" xreflabel=""/>Network interfaces to be used or excluded will be specified using the lnet kernel module parameters networks and ip2netsas described in <xref linkend='configuringlnet'/>.</para>
+ </listitem>
+ <listitem>
<para><anchor xml:id="dbdoclet.50438256_pgfId-1290505" xreflabel=""/> 4. To ease the setup of networks with complex network configurations, determine a cluster-wide module configuration.</para>
- <para><anchor xml:id="dbdoclet.50438256_pgfId-1290506" xreflabel=""/>For large clusters, you can configure the networking setup for all nodes by using a single, unified set of parameters in the modprobe.conf file on each node. Cluster-wide configuration is described in <link xl:href="ConfiguringLNET.html#50438216_64580">Chapter 9</link>: <link xl:href="ConfiguringLNET.html#50438216_29256">Configuring Lustre Networking (LNET)</link>.</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438256_pgfId-1291030" xreflabel=""/>We recommend that you use “dotted-quad†notation for IP addresses rather than host names to make it easier to read debug logs and debug configurations with multiple interfaces.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+ <para><anchor xml:id="dbdoclet.50438256_pgfId-1290506" xreflabel=""/>For large clusters, you can configure the networking setup for all nodes by using a single, unified set of parameters in the modprobe.conf file on each node. Cluster-wide configuration is described in <xref linkend='configuringlnet'/>.</para>
+ </listitem>
+ </orderedlist>
+
+ <note>
+ <para><anchor xml:id="dbdoclet.50438256_pgfId-1291030" xreflabel=""/>We recommend that you use “dotted-quad†notation for IP addresses rather than host names to make it easier to read debug logs and debug configurations with multiple interfaces.</para>
+ </note>
+
</section>
- </section>
</chapter>
git://git.whamcloud.com / doc / manual.git / commitdiff