1 <?xml version='1.0' encoding='UTF-8'?>
2 <chapter xmlns="http://docbook.org/ns/docbook"
3 xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"
4 xml:id="settinguplustresystem">
5 <title xml:id="settinguplustresystem.title">Determining Hardware Configuration Requirements and
6 Formatting Options</title>
7 <para>This chapter describes hardware configuration requirements for a Lustre file system
12 <xref linkend="storage_hardware_considerations"/>
17 <xref linkend="space_requirements"/>
22 <xref linkend="ldiskfs_mkfs_opts"/>
27 <xref linkend="mds_oss_memory"/>
32 <xref linkend="network_considerations"/>
36 <section xml:id="storage_hardware_considerations">
37 <title><indexterm><primary>setup</primary></indexterm>
38 <indexterm><primary>setup</primary><secondary>hardware</secondary></indexterm>
39 <indexterm><primary>design</primary><see>setup</see></indexterm>
40 Hardware Considerations</title>
41 <para>A Lustre file system can utilize any kind of block storage device such as single disks,
42 software RAID, hardware RAID, or a logical volume manager. In contrast to some networked file
43 systems, the block devices are only attached to the MDS and OSS nodes in a Lustre file system
44 and are not accessed by the clients directly.</para>
45 <para>Since the block devices are accessed by only one or two server nodes, a storage area network (SAN) that is accessible from all the servers is not required. Expensive switches are not needed because point-to-point connections between the servers and the storage arrays normally provide the simplest and best attachments. (If failover capability is desired, the storage must be attached to multiple servers.)</para>
46 <para>For a production environment, it is preferable that the MGS have separate storage to allow future expansion to multiple file systems. However, it is possible to run the MDS and MGS on the same machine and have them share the same storage device.</para>
47 <para>For best performance in a production environment, dedicated clients are required. For a non-production Lustre environment or for testing, a Lustre client and server can run on the same machine. However, dedicated clients are the only supported configuration.</para>
48 <warning><para>Performance and recovery issues can occur if you put a client on an MDS or OSS:</para>
51 <para>Running the OSS and a client on the same machine can cause issues with low memory and memory pressure. If the client consumes all the memory and then tries to write data to the file system, the OSS will need to allocate pages to receive data from the client but will not be able to perform this operation due to low memory. This can cause the client to hang.</para>
54 <para>Running the MDS and a client on the same machine can cause recovery and deadlock issues and impact the performance of other Lustre clients.</para>
58 <para>Only servers running on 64-bit CPUs are tested and supported.
59 64-bit CPU clients are typically used for testing to match expected
60 customer usage and avoid limitations due to the 4 GB limit for RAM
61 size, 1 GB low-memory limitation, and 16 TB file size limit of 32-bit
62 CPUs. Also, due to kernel API limitations, performing backups of Lustre
63 filesystems on 32-bit clients may cause backup tools to confuse files
64 that report the same 32-bit inode number, if the backup tools depend
65 on the inode number for correct operation.</para>
66 <para>The storage attached to the servers typically uses RAID to provide fault tolerance and can
67 optionally be organized with logical volume management (LVM), which is then formatted as a
68 Lustre file system. Lustre OSS and MDS servers read, write and modify data in the format
69 imposed by the file system.</para>
70 <para>The Lustre file system uses journaling file system technology on both the MDTs and OSTs.
71 For a MDT, as much as a 20 percent performance gain can be obtained by placing the journal on
72 a separate device.</para>
73 <para>The MDS can effectively utilize a lot of CPU cycles. A minimum of four processor cores are recommended. More are advisable for files systems with many clients.</para>
75 <para>Lustre clients running on different CPU architectures is supported.
76 One limitation is that the PAGE_SIZE kernel macro on the client must be
77 as large as the PAGE_SIZE of the server. In particular, ARM or PPC
78 clients with large pages (up to 64kB pages) can run with x86 servers
83 <primary>setup</primary>
84 <secondary>MDT</secondary>
85 </indexterm> MGT and MDT Storage Hardware Considerations</title>
86 <para>MGT storage requirements are small (less than 100 MB even in the
87 largest Lustre file systems), and the data on an MGT is only accessed
88 on a server/client mount, so disk performance is not a consideration.
89 However, this data is vital for file system access, so
90 the MGT should be reliable storage, preferably mirrored RAID1.</para>
91 <para>MDS storage is accessed in a database-like access pattern with
92 many seeks and read-and-writes of small amounts of data.
93 Storage types that provide much lower seek times, such as SSD or NVMe
94 is strongly preferred for the MDT, and high-RPM SAS is acceptable.</para>
95 <para>For maximum performance, the MDT should be configured as RAID1 with
96 an internal journal and two disks from different controllers.</para>
97 <para>If you need a larger MDT, create multiple RAID1 devices from pairs
98 of disks, and then make a RAID0 array of the RAID1 devices. For ZFS,
99 use <literal>mirror</literal> VDEVs for the MDT. This ensures
100 maximum reliability because multiple disk failures only have a small
101 chance of hitting both disks in the same RAID1 device.</para>
102 <para>Doing the opposite (RAID1 of a pair of RAID0 devices) has a 50%
103 chance that even two disk failures can cause the loss of the whole MDT
104 device. The first failure disables an entire half of the mirror and the
105 second failure has a 50% chance of disabling the remaining mirror.</para>
106 <para>If multiple MDTs are going to be present in the
107 system, each MDT should be specified for the anticipated usage and load.
108 For details on how to add additional MDTs to the filesystem, see
109 <xref linkend="lustremaint.adding_new_mdt"/>.</para>
110 <warning><para>MDT0000 contains the root of the Lustre file system. If
111 MDT0000 is unavailable for any reason, the file system cannot be used.
113 <note><para>Using the DNE feature it is possible to dedicate additional
114 MDTs to sub-directories off the file system root directory stored on
115 MDT0000, or arbitrarily for lower-level subdirectories, using the
116 <literal>lfs mkdir -i <replaceable>mdt_index</replaceable></literal>
117 command. If an MDT serving a subdirectory becomes unavailable, any
118 subdirectories on that MDT and all directories beneath it will also
119 become inaccessible. This is typically useful for top-level directories
120 to assign different users or projects to separate MDTs, or to distribute
121 other large working sets of files to multiple MDTs.</para></note>
122 <note condition='l28'><para>Starting in the 2.8 release it is possible
123 to spread a single large directory across multiple MDTs using the DNE
124 striped directory feature by specifying multiple stripes (or shards)
125 at creation time using the
126 <literal>lfs mkdir -c <replaceable>stripe_count</replaceable></literal>
127 command, where <replaceable>stripe_count</replaceable> is often the
128 number of MDTs in the filesystem. Striped directories should typically
129 not be used for all directories in the filesystem, since this incurs
130 extra overhead compared to non-striped directories, but is useful for
131 larger directories (over 50k entries) where many output files are being
136 <title><indexterm><primary>setup</primary><secondary>OST</secondary></indexterm>OST Storage Hardware Considerations</title>
137 <para>The data access pattern for the OSS storage is a streaming I/O
138 pattern that is dependent on the access patterns of applications being
139 used. Each OSS can manage multiple object storage targets (OSTs), one
140 for each volume with I/O traffic load-balanced between servers and
141 targets. An OSS should be configured to have a balance between the
142 network bandwidth and the attached storage bandwidth to prevent
143 bottlenecks in the I/O path. Depending on the server hardware, an OSS
144 typically serves between 2 and 8 targets, with each target between
145 24-48TB, but may be up to 256 terabytes (TBs) in size.</para>
146 <para>Lustre file system capacity is the sum of the capacities provided
147 by the targets. For example, 64 OSSs, each with two 8 TB OSTs,
148 provide a file system with a capacity of nearly 1 PB. If each OST uses
149 ten 1 TB SATA disks (8 data disks plus 2 parity disks in a RAID-6
150 configuration), it may be possible to get 50 MB/sec from each drive,
151 providing up to 400 MB/sec of disk bandwidth per OST. If this system
152 is used as storage backend with a system network, such as the InfiniBand
153 network, that provides a similar bandwidth, then each OSS could provide
154 800 MB/sec of end-to-end I/O throughput. (Although the architectural
155 constraints described here are simple, in practice it takes careful
156 hardware selection, benchmarking and integration to obtain such
160 <section xml:id="space_requirements">
161 <title><indexterm><primary>setup</primary><secondary>space</secondary></indexterm>
162 <indexterm><primary>space</primary><secondary>determining requirements</secondary></indexterm>
163 Determining Space Requirements</title>
164 <para>The desired performance characteristics of the backing file systems
165 on the MDT and OSTs are independent of one another. The size of the MDT
166 backing file system depends on the number of inodes needed in the total
167 Lustre file system, while the aggregate OST space depends on the total
168 amount of data stored on the file system. If MGS data is to be stored
169 on the MDT device (co-located MGT and MDT), add 100 MB to the required
170 size estimate for the MDT.</para>
171 <para>Each time a file is created on a Lustre file system, it consumes
172 one inode on the MDT and one OST object over which the file is striped.
173 Normally, each file's stripe count is based on the system-wide
174 default stripe count. However, this can be changed for individual files
175 using the <literal>lfs setstripe</literal> option. For more details,
176 see <xref linkend="managingstripingfreespace"/>.</para>
177 <para>In a Lustre ldiskfs file system, all the MDT inodes and OST
178 objects are allocated when the file system is first formatted. When
179 the file system is in use and a file is created, metadata associated
180 with that file is stored in one of the pre-allocated inodes and does
181 not consume any of the free space used to store file data. The total
182 number of inodes on a formatted ldiskfs MDT or OST cannot be easily
183 changed. Thus, the number of inodes created at format time should be
184 generous enough to anticipate near term expected usage, with some room
185 for growth without the effort of additional storage.</para>
186 <para>By default, the ldiskfs file system used by Lustre servers to store
187 user-data objects and system data reserves 5% of space that cannot be used
188 by the Lustre file system. Additionally, an ldiskfs Lustre file system
189 reserves up to 400 MB on each OST, and up to 4GB on each MDT for journal
190 use and a small amount of space outside the journal to store accounting
191 data. This reserved space is unusable for general storage. Thus, at least
192 this much space will be used per OST before any file object data is saved.
194 <para>With a ZFS backing filesystem for the MDT or OST,
195 the space allocation for inodes and file data is dynamic, and inodes are
196 allocated as needed. A minimum of 4kB of usable space (before mirroring)
197 is needed for each inode, exclusive of other overhead such as directories,
198 internal log files, extended attributes, ACLs, etc. ZFS also reserves
199 approximately 3% of the total storage space for internal and redundant
200 metadata, which is not usable by Lustre.
201 Since the size of extended attributes and ACLs is highly dependent on
202 kernel versions and site-specific policies, it is best to over-estimate
203 the amount of space needed for the desired number of inodes, and any
204 excess space will be utilized to store more inodes.
208 <primary>setup</primary>
209 <secondary>MGT</secondary>
212 <primary>space</primary>
213 <secondary>determining MGT requirements</secondary>
214 </indexterm> Determining MGT Space Requirements</title>
215 <para>Less than 100 MB of space is typically required for the MGT.
216 The size is determined by the total number of servers in the Lustre
217 file system cluster(s) that are managed by the MGS.</para>
219 <section xml:id="mdt_space_requirements">
221 <primary>setup</primary>
222 <secondary>MDT</secondary>
225 <primary>space</primary>
226 <secondary>determining MDT requirements</secondary>
227 </indexterm> Determining MDT Space Requirements</title>
228 <para>When calculating the MDT size, the important factor to consider
229 is the number of files to be stored in the file system, which depends on
230 at least 2 KiB per inode of usable space on the MDT. Since MDTs typically
231 use RAID-1+0 mirroring, the total storage needed will be double this.
233 <para>Please note that the actual used space per MDT depends on the number
234 of files per directory, the number of stripes per file, whether files
235 have ACLs or user xattrs, and the number of hard links per file. The
236 storage required for Lustre file system metadata is typically 1-2
237 percent of the total file system capacity depending upon file size.
238 If the <xref linkend="dataonmdt"/> feature is in use for Lustre
239 2.11 or later, MDT space should typically be 5 percent or more of the
240 total space, depending on the distribution of small files within the
241 filesystem and the <literal>lod.*.dom_stripesize</literal> limit on
242 the MDT and file layout used.</para>
243 <para>For ZFS-based MDT filesystems, the number of inodes created on
244 the MDT and OST is dynamic, so there is less need to determine the
245 number of inodes in advance, though there still needs to be some thought
246 given to the total MDT space compared to the total filesystem size.</para>
247 <para>For example, if the average file size is 5 MiB and you have
248 100 TiB of usable OST space, then you can calculate the
249 <emphasis>minimum</emphasis> total number of inodes for MDTs and OSTs
252 <para>(500 TB * 1000000 MB/TB) / 5 MB/inode = 100M inodes</para>
254 <para>It is recommended that the MDT(s) have at least twice the minimum
255 number of inodes to allow for future expansion and allow for an average
256 file size smaller than expected. Thus, the minimum space for ldiskfs
257 MDT(s) should be approximately:
260 <para>2 KiB/inode x 100 million inodes x 2 = 400 GiB ldiskfs MDT</para>
262 <para>For details about formatting options for ldiskfs MDT and OST file
263 systems, see <xref linkend="ldiskfs_mdt_mkfs"/>.</para>
265 <para>If the median file size is very small, 4 KB for example, the
266 MDT would use as much space for each file as the space used on the OST,
267 so the use of Data-on-MDT is strongly recommended in that case.
268 The MDT space per inode should be increased correspondingly to
269 account for the extra data space usage for each inode:
271 <para>6 KiB/inode x 100 million inodes x 2 = 1200 GiB ldiskfs MDT</para>
276 <para>If the MDT has too few inodes, this can cause the space on the
277 OSTs to be inaccessible since no new files can be created. In this
278 case, the <literal>lfs df -i</literal> and <literal>df -i</literal>
279 commands will limit the number of available inodes reported for the
280 filesystem to match the total number of available objects on the OSTs.
281 Be sure to determine the appropriate MDT size needed to support the
282 filesystem before formatting. It is possible to increase the
283 number of inodes after the file system is formatted, depending on the
284 storage. For ldiskfs MDT filesystems the <literal>resize2fs</literal>
285 tool can be used if the underlying block device is on a LVM logical
286 volume and the underlying logical volume size can be increased.
287 For ZFS new (mirrored) VDEVs can be added to the MDT pool to increase
288 the total space available for inode storage.
289 Inodes will be added approximately in proportion to space added.
293 <para>Note that the number of total and free inodes reported by
294 <literal>lfs df -i</literal> for ZFS MDTs and OSTs is estimated based
295 on the current average space used per inode. When a ZFS filesystem is
296 first formatted, this free inode estimate will be very conservative
297 (low) due to the high ratio of directories to regular files created for
298 internal Lustre metadata storage, but this estimate will improve as
299 more files are created by regular users and the average file size will
300 better reflect actual site usage.
304 <para>Using the DNE remote directory feature
305 it is possible to increase the total number of inodes of a Lustre
306 filesystem, as well as increasing the aggregate metadata performance,
307 by configuring additional MDTs into the filesystem, see
308 <xref linkend="lustremaint.adding_new_mdt"/> for details.
314 <primary>setup</primary>
315 <secondary>OST</secondary>
318 <primary>space</primary>
319 <secondary>determining OST requirements</secondary>
320 </indexterm> Determining OST Space Requirements</title>
321 <para>For the OST, the amount of space taken by each object depends on
322 the usage pattern of the users/applications running on the system. The
323 Lustre software defaults to a conservative estimate for the average
324 object size (between 64 KiB per object for 10 GiB OSTs, and 1 MiB per
325 object for 16 TiB and larger OSTs). If you are confident that the average
326 file size for your applications will be different than this, you can
327 specify a different average file size (number of total inodes for a given
328 OST size) to reduce file system overhead and minimize file system check
330 See <xref linkend="ldiskfs_ost_mkfs"/> for more details.</para>
333 <section xml:id="ldiskfs_mkfs_opts">
336 <primary>ldiskfs</primary>
337 <secondary>formatting options</secondary>
340 <primary>setup</primary>
341 <secondary>ldiskfs</secondary>
343 Setting ldiskfs File System Formatting Options
345 <para>By default, the <literal>mkfs.lustre</literal> utility applies these
346 options to the Lustre backing file system used to store data and metadata
347 in order to enhance Lustre file system performance and scalability. These
348 options include:</para>
351 <para><literal>flex_bg</literal> - When the flag is set to enable
352 this flexible-block-groups feature, block and inode bitmaps for
353 multiple groups are aggregated to minimize seeking when bitmaps
354 are read or written and to reduce read/modify/write operations
355 on typical RAID storage (with 1 MiB RAID stripe widths). This flag
356 is enabled on both OST and MDT file systems. On MDT file systems
357 the <literal>flex_bg</literal> factor is left at the default value
358 of 16. On OSTs, the <literal>flex_bg</literal> factor is set
359 to 256 to allow all of the block or inode bitmaps in a single
360 <literal>flex_bg</literal> to be read or written in a single
361 1MiB I/O typical for RAID storage.</para>
364 <para><literal>huge_file</literal> - Setting this flag allows
365 files on OSTs to be larger than 2 TiB in size.</para>
368 <para><literal>lazy_journal_init</literal> - This extended option
369 is enabled to prevent a full overwrite to zero out the large
370 journal that is allocated by default in a Lustre file system
371 (up to 400 MiB for OSTs, up to 4GiB for MDTs), to reduce the
372 formatting time.</para>
375 <para>To override the default formatting options, use arguments to
376 <literal>mkfs.lustre</literal> to pass formatting options to the backing file system:</para>
377 <screen>--mkfsoptions='backing fs options'</screen>
378 <para>For other <literal>mkfs.lustre</literal> options, see the Linux man page for
379 <literal>mke2fs(8)</literal>.</para>
380 <section xml:id="ldiskfs_mdt_mkfs">
382 <primary>inodes</primary>
383 <secondary>MDS</secondary>
384 </indexterm><indexterm>
385 <primary>setup</primary>
386 <secondary>inodes</secondary>
387 </indexterm>Setting Formatting Options for an ldiskfs MDT</title>
388 <para>The number of inodes on the MDT is determined at format time
389 based on the total size of the file system to be created. The default
390 <emphasis role="italic">bytes-per-inode</emphasis> ratio ("inode ratio")
391 for an ldiskfs MDT is optimized at one inode for every 2560 bytes of file
393 <para>This setting takes into account the space needed for additional
394 ldiskfs filesystem-wide metadata, such as the journal (up to 4 GB),
395 bitmaps, and directories, as well as files that Lustre uses internally
396 to maintain cluster consistency. There is additional per-file metadata
397 such as file layout for files with a large number of stripes, Access
398 Control Lists (ACLs), and user extended attributes.</para>
399 <para condition="l2B"> Starting in Lustre 2.11, the <xref linkend=
400 "dataonmdt.title"/> (DoM) feature allows storing small files on the MDT
401 to take advantage of high-performance flash storage, as well as reduce
402 space and network overhead. If you are planning to use the DoM feature
403 with an ldiskfs MDT, it is recommended to <emphasis>increase</emphasis>
404 the bytes-per-inode ratio to have enough space on the MDT for small files,
407 <para>It is possible to change the recommended default of 2560 bytes
408 per inode for an ldiskfs MDT when it is first formatted by adding the
409 <literal>--mkfsoptions="-i bytes-per-inode"</literal> option to
410 <literal>mkfs.lustre</literal>. Decreasing the inode ratio tunable
411 <literal>bytes-per-inode</literal> will create more inodes for a given
412 MDT size, but will leave less space for extra per-file metadata and is
413 not recommended. The inode ratio must always be strictly larger than
414 the MDT inode size, which is 1024 bytes by default. It is recommended
415 to use an inode ratio at least 1536 bytes larger than the inode size to
416 ensure the MDT does not run out of space. Increasing the inode ratio
417 with enough space for the most commonly file size (e.g. 5632 or 66560
418 bytes if 4KB or 64KB files are widely used) is recommended for DoM.</para>
419 <para>The size of the inode may be changed at format time by adding the
420 <literal>--stripe-count-hint=N</literal> to have
421 <literal>mkfs.lustre</literal> automatically calculate a reasonable
422 inode size based on the default stripe count that will be used by the
423 filesystem, or directly by specifying the
424 <literal>--mkfsoptions="-I inode-size"</literal> option. Increasing
425 the inode size will provide more space in the inode for a larger Lustre
426 file layout, ACLs, user and system extended attributes, SELinux and
427 other security labels, and other internal metadata and DoM data. However,
428 if these features or other in-inode xattrs are not needed, a larger inode
429 size may hurt metadata performance as 2x, 4x, or 8x as much data would be
430 read or written for each MDT inode access.
433 <section xml:id="ldiskfs_ost_mkfs">
435 <primary>inodes</primary>
436 <secondary>OST</secondary>
437 </indexterm>Setting Formatting Options for an ldiskfs OST</title>
438 <para>When formatting an OST file system, it can be beneficial
439 to take local file system usage into account, for example by running
440 <literal>df</literal> and <literal>df -i</literal> on a current filesystem
441 to get the used bytes and used inodes respectively, then computing the
442 average bytes-per-inode value. When deciding on the ratio for a new
443 filesystem, try to avoid having too many inodes on each OST, while keeping
444 enough margin to allow for future usage of smaller files. This helps
445 reduce the format and e2fsck time and makes more space available for data.
447 <para>The table below shows the default
448 <emphasis role="italic">bytes-per-inode</emphasis> ratio ("inode ratio")
449 used for OSTs of various sizes when they are formatted.</para>
451 <table frame="all" xml:id="settinguplustresystem.tab1">
452 <title>Default Inode Ratios Used for Newly Formatted OSTs</title>
454 <colspec colname="c1" colwidth="3*"/>
455 <colspec colname="c2" colwidth="2*"/>
456 <colspec colname="c3" colwidth="4*"/>
460 <para><emphasis role="bold">LUN/OST size</emphasis></para>
463 <para><emphasis role="bold">Default Inode ratio</emphasis></para>
466 <para><emphasis role="bold">Total inodes</emphasis></para>
473 <para>under 10GiB </para>
476 <para>1 inode/16KiB </para>
479 <para>640 - 655k </para>
484 <para>10GiB - 1TiB </para>
487 <para>1 inode/68KiB </para>
490 <para>153k - 15.7M </para>
495 <para>1TiB - 8TiB </para>
498 <para>1 inode/256KiB </para>
501 <para>4.2M - 33.6M </para>
506 <para>over 8TiB </para>
509 <para>1 inode/1MiB </para>
512 <para>8.4M - 268M </para>
519 <para>In environments with few small files, the default inode ratio
520 may result in far too many inodes for the average file size. In this
521 case, performance can be improved by increasing the number of
522 <emphasis role="italic">bytes-per-inode</emphasis>. To set the inode
523 ratio, use the <literal>--mkfsoptions="-i <replaceable>bytes-per-inode</replaceable>"</literal>
524 argument to <literal>mkfs.lustre</literal> to specify the expected
525 average (mean) size of OST objects. For example, to create an OST
526 with an expected average object size of 8 MiB run:
527 <screen>[oss#] mkfs.lustre --ost --mkfsoptions="-i $((8192 * 1024))" ...</screen>
530 <para>OSTs formatted with ldiskfs should preferably have fewer than
531 320 million objects per MDT, and up to a maximum of 4 billion inodes.
532 Specifying a very small bytes-per-inode ratio for a large OST that
533 exceeds this limit can cause either premature out-of-space errors and
534 prevent the full OST space from being used, or will waste space and
535 slow down e2fsck more than necessary. The default inode ratios are
536 chosen to ensure the total number of inodes remain below this limit.
540 <para>File system check time on OSTs is affected by a number of
541 variables in addition to the number of inodes, including the size of
542 the file system, the number of allocated blocks, the distribution of
543 allocated blocks on the disk, disk speed, CPU speed, and the amount
544 of RAM on the server. Reasonable file system check times for valid
545 filesystems are 5-30 minutes per TiB, but may increase significantly
546 if substantial errors are detected and need to be repaired.</para>
548 <para>For further details about optimizing MDT and OST file systems,
549 see <xref linkend="ldiskfs_raid_opts"/>.</para>
554 <primary>setup</primary>
555 <secondary>limits</secondary>
556 </indexterm><indexterm xmlns:xi="http://www.w3.org/2001/XInclude">
557 <primary>wide striping</primary>
558 </indexterm><indexterm xmlns:xi="http://www.w3.org/2001/XInclude">
559 <primary>xattr</primary>
560 <secondary><emphasis role="italic">See</emphasis> wide striping</secondary>
561 </indexterm><indexterm>
562 <primary>large_xattr</primary>
563 <secondary>ea_inode</secondary>
564 </indexterm><indexterm>
565 <primary>wide striping</primary>
566 <secondary>large_xattr</secondary>
567 <tertiary>ea_inode</tertiary>
568 </indexterm>File and File System Limits</title>
570 <para><xref linkend="settinguplustresystem.tab2"/> describes
571 current known limits of Lustre. These limits may be imposed by either
572 the Lustre architecture or the Linux virtual file system (VFS) and
573 virtual memory subsystems. In a few cases, a limit is defined within
574 the code Lustre based on tested values and could be changed by editing
575 and re-compiling the Lustre software. In these cases, the indicated
576 limit was used for testing of the Lustre software.</para>
578 <table frame="all" xml:id="settinguplustresystem.tab2">
579 <title>File and file system limits</title>
581 <colspec colname="c1" colwidth="3*"/>
582 <colspec colname="c2" colwidth="2*"/>
583 <colspec colname="c3" colwidth="4*"/>
587 <para><emphasis role="bold">Limit</emphasis></para>
590 <para><emphasis role="bold">Value</emphasis></para>
593 <para><emphasis role="bold">Description</emphasis></para>
600 <para><anchor xml:id="max_mdt_count" xreflabel=""/>Maximum number of MDTs</para>
606 <para>A single MDS can host one or more MDTs, either for separate
607 filesystems, or aggregated into a single namespace. Each
608 filesystem requires a separate MDT for the filesystem root
610 Up to 255 more MDTs can be added to the filesystem and are
611 attached into the filesystem namespace with creation of DNE
612 remote or striped directories.</para>
617 <para><anchor xml:id="max_ost_count" xreflabel=""/>Maximum number of OSTs</para>
623 <para>The maximum number of OSTs is a constant that can be
624 changed at compile time. Lustre file systems with up to 4000
625 OSTs have been configured in the past. Multiple OST targets
626 can be configured on a single OSS node.</para>
631 <para><anchor xml:id="max_ost_size" xreflabel=""/>Maximum OST size</para>
634 <para>1024TiB (ldiskfs), 1024TiB (ZFS)</para>
637 <para>This is not a <emphasis>hard</emphasis> limit. Larger
638 OSTs are possible, but most production systems do not
639 typically go beyond the stated limit per OST because Lustre
640 can add capacity and performance with additional OSTs, and
641 having more OSTs improves aggregate I/O performance,
642 minimizes contention, and allows parallel recovery (e2fsck
643 for ldiskfs OSTs, scrub for ZFS OSTs).
646 With 32-bit kernels, due to page cache limits, 16TB is the
647 maximum block device size, which in turn applies to the
648 size of OST. It is <emphasis>strongly</emphasis> recommended
649 to run Lustre clients and servers with 64-bit kernels.</para>
654 <para><anchor xml:id="max_client_count" xreflabel=""/>Maximum number of clients</para>
660 <para>The maximum number of clients is a constant that can
661 be changed at compile time. Up to 30000 clients have been
662 used in production accessing a single filesystem.</para>
667 <para><anchor xml:id="max_filesysem_size" xreflabel=""/>Maximum size of a single file system</para>
670 <para>2EiB or larger</para>
673 <para>Each OST can have a file system up to the "Maximum OST
674 size" limit, and the Maximum number of OSTs can be combined
675 into a single filesystem.
681 <para><anchor xml:id="max_stripe_count" xreflabel=""/>Maximum stripe count</para>
687 <para>This limit is imposed by the size of the layout that
688 needs to be stored on disk and sent in RPC requests, but is
689 not a hard limit of the protocol. The number of OSTs in the
690 filesystem can exceed the stripe count, but this is the maximum
691 number of OSTs on which a <emphasis>single file</emphasis>
692 can be striped.</para>
693 <note condition='l2D'><para>Before 2.13, the default for ldiskfs
694 MDTs the maximum stripe count for a
695 <emphasis>single file</emphasis> is limited to 160 OSTs. In order to
696 increase the maximum file stripe count, use
697 <literal>--mkfsoptions="-O ea_inode"</literal> when formatting the MDT,
698 or use <literal>tune2fs -O ea_inode</literal> to enable it after the
699 MDT has been formatted.</para>
705 <para><anchor xml:id="max_stripe_size" xreflabel=""/>Maximum stripe size</para>
708 <para>< 4 GiB</para>
711 <para>The amount of data written to each object before moving
712 on to next object.</para>
717 <para><anchor xml:id="min_stripe_size" xreflabel=""/>Minimum stripe size</para>
723 <para>Due to the use of 64 KiB PAGE_SIZE on some CPU
724 architectures such as ARM and POWER, the minimum stripe
725 size is 64 KiB so that a single page is not split over
726 multiple servers. This is also the minimum Data-on-MDT
727 component size that can be specified.</para>
732 <para><anchor xml:id="max_object_size" xreflabel=""/>Maximum single object size</para>
735 <para>16TiB (ldiskfs), 256TiB (ZFS)</para>
738 <para>The amount of data that can be stored in a single object.
739 An object corresponds to a stripe. The ldiskfs limit of 16 TB
740 for a single object applies. For ZFS the limit is the size of
741 the underlying OST. Files can consist of up to 2000 stripes,
742 each stripe can be up to the maximum object size. </para>
747 <para><anchor xml:id="max_file_size" xreflabel=""/>Maximum file size</para>
750 <para>16 TiB on 32-bit systems</para>
752 <para>31.25 PiB on 64-bit ldiskfs systems,
753 8EiB on 64-bit ZFS systems</para>
756 <para>Individual files have a hard limit of nearly 16 TiB on
757 32-bit systems imposed by the kernel memory subsystem. On
758 64-bit systems this limit does not exist. Hence, files can
759 be 2^63 bits (8EiB) in size if the backing filesystem can
760 support large enough objects and/or the files are sparse.</para>
761 <para>A single file can have a maximum of 2000 stripes, which
762 gives an upper single file data capacity of 31.25 PiB for 64-bit
763 ldiskfs systems. The actual amount of data that can be stored
764 in a file depends upon the amount of free space in each OST
765 on which the file is striped.</para>
770 <para><anchor xml:id="max_directory_size" xreflabel=""/>Maximum number of files or subdirectories in a single directory</para>
773 <para>600M-3.8B files (ldiskfs), 16T (ZFS)</para>
776 <para>The Lustre software uses the ldiskfs hashed directory
777 code, which has a limit of at least 600 million files, depending
778 on the length of the file name. The limit on subdirectories
779 is the same as the limit on regular files.</para>
780 <note condition='l28'><para>Starting in the 2.8 release it is
781 possible to exceed this limit by striping a single directory
782 over multiple MDTs with the <literal>lfs mkdir -c</literal>
783 command, which increases the single directory limit by a
784 factor of the number of directory stripes used.</para></note>
785 <note condition='l2E'><para>Starting in the 2.14 release, the
786 <literal>large_dir</literal> feature of ldiskfs is enabled by
787 default to allow directories with more than 10M entries. In
788 the 2.12 release, the <literal>large_dir</literal> feature was
789 present but not enabled by default.</para></note>
794 <para><anchor xml:id="max_file_count" xreflabel=""/>Maximum number of files in the file system</para>
797 <para>4 billion (ldiskfs), 256 trillion (ZFS) <emphasis>per MDT</emphasis></para>
800 <para>The ldiskfs filesystem imposes an upper limit of
801 4 billion inodes per filesystem. By default, the MDT
802 filesystem is formatted with one inode per 2KB of space,
803 meaning 512 million inodes per TiB of MDT space. This can be
804 increased initially at the time of MDT filesystem creation.
805 For more information, see
806 <xref linkend="settinguplustresystem"/>.</para>
807 <para>The ZFS filesystem dynamically allocates
808 inodes and does not have a fixed ratio of inodes per unit of MDT
809 space, but consumes approximately 4KiB of mirrored space per
810 inode, depending on the configuration.</para>
811 <para>Each additional MDT can hold up to the
812 above maximum number of additional files, depending on
813 available space and the distribution directories and files
814 in the filesystem.</para>
819 <para><anchor xml:id="max_filename_size" xreflabel=""/>Maximum length of a filename</para>
822 <para>255 bytes (filename)</para>
825 <para>This limit is 255 bytes for a single filename, the
826 same as the limit in the underlying filesystems.</para>
831 <para><anchor xml:id="max_pathname_size" xreflabel=""/>Maximum length of a pathname</para>
834 <para>4096 bytes (pathname)</para>
837 <para>The Linux VFS imposes a full pathname length of 4096 bytes.</para>
842 <para><anchor xml:id="max_open_files" xreflabel=""/>Maximum number of open files for a Lustre file system</para>
845 <para>No limit</para>
848 <para>The Lustre software does not impose a maximum for the number
849 of open files, but the practical limit depends on the amount of
850 RAM on the MDS. No "tables" for open files exist on the
851 MDS, as they are only linked in a list to a given client's
852 export. Each client process has a limit of several
853 thousands of open files which depends on its ulimit.</para>
861 <section xml:id="mds_oss_memory">
862 <title><indexterm><primary>setup</primary><secondary>memory</secondary></indexterm>Determining Memory Requirements</title>
863 <para>This section describes the memory requirements for each Lustre file system component.</para>
866 <indexterm><primary>setup</primary><secondary>memory</secondary><tertiary>client</tertiary></indexterm>
867 Client Memory Requirements</title>
868 <para>A minimum of 2 GB RAM is recommended for clients.</para>
871 <title><indexterm><primary>setup</primary><secondary>memory</secondary><tertiary>MDS</tertiary></indexterm>MDS Memory Requirements</title>
872 <para>MDS memory requirements are determined by the following factors:</para>
875 <para>Number of clients</para>
878 <para>Size of the directories</para>
881 <para>Load placed on server</para>
884 <para>The amount of memory used by the MDS is a function of how many clients are on
885 the system, and how many files they are using in their working set. This is driven,
886 primarily, by the number of locks a client can hold at one time. The number of locks
887 held by clients varies by load and memory availability on the server. Interactive
888 clients can hold in excess of 10,000 locks at times. On the MDS, memory usage is
889 approximately 2 KB per file, including the Lustre distributed lock manager (LDLM)
890 lock and kernel data structures for the files currently in use. Having file data
891 in cache can improve metadata performance by a factor of 10x or more compared to
892 reading it from storage.</para>
893 <para>MDS memory requirements include:</para>
896 <para><emphasis role="bold">File system metadata</emphasis>:
897 A reasonable amount of RAM needs to be available for file system metadata.
898 While no hard limit can be placed on the amount of file system metadata,
899 if more RAM is available, then the disk I/O is needed less often to retrieve
903 <para><emphasis role="bold">Network transport</emphasis>:
904 If you are using TCP or other network transport that uses system memory for
905 send/receive buffers, this memory requirement must also be taken into
906 consideration.</para>
909 <para><emphasis role="bold">Journal size</emphasis>:
910 By default, the journal size is 4096 MB for each MDT ldiskfs file system.
911 This can pin up to an equal amount of RAM on the MDS node per file system.</para>
914 <para><emphasis role="bold">Failover configuration</emphasis>:
915 If the MDS node will be used for failover from another node, then the RAM
916 for each journal should be doubled, so the backup server can handle the
917 additional load if the primary server fails.</para>
921 <title><indexterm><primary>setup</primary><secondary>memory</secondary><tertiary>MDS</tertiary></indexterm>Calculating MDS Memory Requirements</title>
922 <para>By default, 4096 MB are used for the ldiskfs filesystem journal.
923 Additional RAM is used for caching file data for the larger working
924 set, which is not actively in use by clients but should be kept
925 "hot" for improved access times. Approximately 1.5 KB per
926 file is needed to keep a file in cache without a lock.</para>
927 <para>For example, for a single MDT on an MDS with 1,024 compute nodes,
928 12 interactive login nodes, and a 20 million file working set (of
929 which 9 million files are cached on the clients at one time):</para>
931 <para>Operating system overhead = 4096 MB (RHEL8)</para>
932 <para>File system journal = 4096 MB</para>
933 <para>1024 * 32-core clients * 256 files/core * 2KB = 16384 MB</para>
934 <para>12 interactive clients * 100,000 files * 2KB = 2400 MB</para>
935 <para>20 million file working set * 1.5KB/file = 30720 MB</para>
937 <para>Thus, a reasonable MDS configuration for this workload is
938 at least 60 GB of RAM. For active-active DNE MDT failover pairs,
939 each MDS should have at least 96 GB of RAM. The additional memory
940 can be used during normal operation to allow more metadata and locks
941 to be cached and improve performance, depending on the workload.
943 <para>For directories containing 1 million or more files, more memory
944 can provide a significant benefit. For example, in an environment
945 where clients randomly a single directory with 10 million files can
946 consume as much as 35GB of RAM on the MDS.</para>
950 <title><indexterm><primary>setup</primary><secondary>memory</secondary><tertiary>OSS</tertiary></indexterm>OSS Memory Requirements</title>
951 <para>When planning the hardware for an OSS node, consider the memory
952 usage of several components in the Lustre file system (i.e., journal,
953 service threads, file system metadata, etc.). Also, consider the
954 effect of the OSS read cache feature, which consumes memory as it
955 caches data on the OSS node.</para>
956 <para>In addition to the MDS memory requirements mentioned above,
957 the OSS requirements also include:</para>
960 <para><emphasis role="bold">Service threads</emphasis>:
961 The service threads on the OSS node pre-allocate an RPC-sized MB
962 I/O buffer for each <literal>ost_io</literal> service thread, so
963 these large buffers do not need to be allocated and freed for
964 each I/O request.</para>
967 <para><emphasis role="bold">OSS read cache</emphasis>:
968 OSS read cache provides read-only caching of data on an HDD-based
969 OSS, using the regular Linux page cache to store the data. Just
970 like caching from a regular file system in the Linux operating
971 system, OSS read cache uses as much physical memory as is available.
975 <para>The same calculation applies to files accessed from the OSS as for
976 the MDS, but the load is typically distributed over more OSS nodes, so
977 the amount of memory required for locks, inode cache, etc. listed for
978 the MDS is spread out over the OSS nodes.</para>
979 <para>Because of these memory requirements, the following calculations
980 should be taken as determining the minimum RAM required in an OSS node.
983 <title><indexterm><primary>setup</primary><secondary>memory</secondary><tertiary>OSS</tertiary></indexterm>Calculating OSS Memory Requirements</title>
984 <para>The minimum recommended RAM size for an OSS with eight OSTs,
985 handling objects for 1/4 of the active files for the MDS:</para>
987 <para>Linux kernel and userspace daemon memory = 4096 MB</para>
988 <para>Network send/receive buffers (16 MB * 512 threads) = 8192 MB</para>
989 <para>1024 MB ldiskfs journal size * 8 OST devices = 8192 MB</para>
990 <para>16 MB read/write buffer per OST IO thread * 512 threads = 8192 MB</para>
991 <para>2048 MB file system read cache * 8 OSTs = 16384 MB</para>
992 <para>1024 * 32-core clients * 64 objects/core * 2KB/object = 4096 MB</para>
993 <para>12 interactive clients * 25,000 objects * 2KB/object = 600 MB</para>
994 <para>5 million object working set * 1.5KB/object = 7500 MB</para>
996 <para> For a non-failover configuration, the minimum RAM would be about
997 60 GB for an OSS node with eight OSTs. Additional memory on the OSS
998 will improve the performance of reading smaller, frequently-accessed
1000 <para>For a failover configuration, the minimum RAM would be about
1001 90 GB, as some of the memory is per-node. When the OSS is not handling
1002 any failed-over OSTs the extra RAM will be used as a read cache.
1004 <para>As a reasonable rule of thumb, about 24 GB of base memory plus
1005 4 GB per OST can be used. In failover configurations, about 8 GB per
1006 primary OST is needed.</para>
1010 <section xml:id="network_considerations">
1012 <primary>setup</primary>
1013 <secondary>network</secondary>
1014 </indexterm>Implementing Networks To Be Used by the Lustre File System</title>
1015 <para>As a high performance file system, the Lustre file system places heavy loads on networks.
1016 Thus, a network interface in each Lustre server and client is commonly dedicated to Lustre
1017 file system traffic. This is often a dedicated TCP/IP subnet, although other network hardware
1018 can also be used.</para>
1019 <para>A typical Lustre file system implementation may include the following:</para>
1022 <para>A high-performance backend network for the Lustre servers, typically an InfiniBand (IB) network.</para>
1025 <para>A larger client network.</para>
1028 <para>Lustre routers to connect the two networks.</para>
1031 <para>Lustre networks and routing are configured and managed by specifying parameters to the
1032 Lustre Networking (<literal>lnet</literal>) module in
1033 <literal>/etc/modprobe.d/lustre.conf</literal>.</para>
1034 <para>To prepare to configure Lustre networking, complete the following steps:</para>
1037 <para><emphasis role="bold">Identify all machines that will be running Lustre software and
1038 the network interfaces they will use to run Lustre file system traffic. These machines
1039 will form the Lustre network .</emphasis></para>
1040 <para>A network is a group of nodes that communicate directly with one another. The Lustre
1041 software includes Lustre network drivers (LNDs) to support a variety of network types and
1042 hardware (see <xref linkend="understandinglustrenetworking"/> for a complete list). The
1043 standard rules for specifying networks applies to Lustre networks. For example, two TCP
1044 networks on two different subnets (<literal>tcp0</literal> and <literal>tcp1</literal>)
1045 are considered to be two different Lustre networks.</para>
1048 <para><emphasis role="bold">If routing is needed, identify the nodes to be used to route traffic between networks.</emphasis></para>
1049 <para>If you are using multiple network types, then you will need a router. Any node with
1050 appropriate interfaces can route Lustre networking (LNet) traffic between different
1051 network hardware types or topologies --the node may be a server, a client, or a standalone
1052 router. LNet can route messages between different network types (such as
1053 TCP-to-InfiniBand) or across different topologies (such as bridging two InfiniBand or
1054 TCP/IP networks). Routing will be configured in <xref linkend="configuringlnet"/>.</para>
1057 <para><emphasis role="bold">Identify the network interfaces to include
1058 in or exclude from LNet.</emphasis></para>
1059 <para>If not explicitly specified, LNet uses either the first available
1060 interface or a pre-defined default for a given network type. Interfaces
1061 that LNet should not use (such as an administrative network or
1062 IP-over-IB), can be excluded.</para>
1063 <para>Network interfaces to be used or excluded will be specified using
1064 the lnet kernel module parameters <literal>networks</literal> and
1065 <literal>ip2nets</literal> as described in
1066 <xref linkend="configuringlnet"/>.</para>
1069 <para><emphasis role="bold">To ease the setup of networks with complex
1070 network configurations, determine a cluster-wide module configuration.
1072 <para>For large clusters, you can configure the networking setup for
1073 all nodes by using a single, unified set of parameters in the
1074 <literal>lustre.conf</literal> file on each node. Cluster-wide
1075 configuration is described in <xref linkend="configuringlnet"/>.</para>
1079 <para>We recommend that you use 'dotted-quad' notation for IP addresses rather than host names to make it easier to read debug logs and debug configurations with multiple interfaces.</para>
1084 vim:expandtab:shiftwidth=2:tabstop=8: