1 <?xml version='1.0' encoding='utf-8'?>
2 <chapter xmlns="http://docbook.org/ns/docbook"
3 xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"
4 xml:id="lustreoperations">
5 <title xml:id="lustreoperations.title">Lustre Operations</title>
6 <para>Once you have the Lustre file system up and running, you can use the
7 procedures in this section to perform these basic Lustre administration
9 <section xml:id="dbdoclet.50438194_42877">
12 <primary>operations</primary>
15 <primary>operations</primary>
16 <secondary>mounting by label</secondary>
17 </indexterm>Mounting by Label</title>
18 <para>The file system name is limited to 8 characters. We have encoded the
19 file system and target information in the disk label, so you can mount by
20 label. This allows system administrators to move disks around without
21 worrying about issues such as SCSI disk reordering or getting the
22 <literal>/dev/device</literal> wrong for a shared target. Soon, file system
23 naming will be made as fail-safe as possible. Currently, Linux disk labels
24 are limited to 16 characters. To identify the target within the file
25 system, 8 characters are reserved, leaving 8 characters for the file system
28 <replaceable>fsname</replaceable>-MDT0000 or
29 <replaceable>fsname</replaceable>-OST0a19
31 <para>To mount by label, use this command:</para>
34 <replaceable>file_system_label</replaceable>
35 <replaceable>/mount_point</replaceable>
37 <para>This is an example of mount-by-label:</para>
39 mds# mount -t lustre -L testfs-MDT0000 /mnt/mdt
42 <para>Mount-by-label should NOT be used in a multi-path environment or
43 when snapshots are being created of the device, since multiple block
44 devices will have the same label.</para>
46 <para>Although the file system name is internally limited to 8 characters,
47 you can mount the clients at any mount point, so file system users are not
48 subjected to short names. Here is an example:</para>
50 client# mount -t lustre mds0@tcp0:/short
51 <replaceable>/dev/long_mountpoint_name</replaceable>
54 <section xml:id="dbdoclet.50438194_24122">
57 <primary>operations</primary>
58 <secondary>starting</secondary>
59 </indexterm>Starting Lustre</title>
60 <para>On the first start of a Lustre file system, the components must be
61 started in the following order:</para>
64 <para>Mount the MGT.</para>
66 <para>If a combined MGT/MDT is present, Lustre will correctly mount
67 the MGT and MDT automatically.</para>
71 <para>Mount the MDT.</para>
73 <para condition='l24'>Mount all MDTs if multiple MDTs are
78 <para>Mount the OST(s).</para>
81 <para>Mount the client(s).</para>
85 <section xml:id="dbdoclet.50438194_84876">
88 <primary>operations</primary>
89 <secondary>mounting</secondary>
90 </indexterm>Mounting a Server</title>
91 <para>Starting a Lustre server is straightforward and only involves the
92 mount command. Lustre servers can be added to
93 <literal>/etc/fstab</literal>:</para>
97 <para>The mount command generates output similar to this:</para>
99 /dev/sda1 on /mnt/test/mdt type lustre (rw)
100 /dev/sda2 on /mnt/test/ost0 type lustre (rw)
101 192.168.0.21@tcp:/testfs on /mnt/testfs type lustre (rw)
103 <para>In this example, the MDT, an OST (ost0) and file system (testfs) are
106 LABEL=testfs-MDT0000 /mnt/test/mdt lustre defaults,_netdev,noauto 0 0
107 LABEL=testfs-OST0000 /mnt/test/ost0 lustre defaults,_netdev,noauto 0 0
109 <para>In general, it is wise to specify noauto and let your
110 high-availability (HA) package manage when to mount the device. If you are
111 not using failover, make sure that networking has been started before
112 mounting a Lustre server. If you are running Red Hat Enterprise Linux, SUSE
113 Linux Enterprise Server, Debian operating system (and perhaps others), use
115 <literal>_netdev</literal> flag to ensure that these disks are mounted after
116 the network is up.</para>
117 <para>We are mounting by disk label here. The label of a device can be read
119 <literal>e2label</literal>. The label of a newly-formatted Lustre server
121 <literal>FFFF</literal> if the
122 <literal>--index</literal> option is not specified to
123 <literal>mkfs.lustre</literal>, meaning that it has yet to be assigned. The
124 assignment takes place when the server is first started, and the disk label
125 is updated. It is recommended that the
126 <literal>--index</literal> option always be used, which will also ensure
127 that the label is set at format time.</para>
129 <para>Do not do this when the client and OSS are on the same node, as
130 memory pressure between the client and OSS can lead to deadlocks.</para>
133 <para>Mount-by-label should NOT be used in a multi-path
137 <section xml:id="dbdoclet.50438194_69255">
140 <primary>operations</primary>
141 <secondary>unmounting</secondary>
142 </indexterm>Unmounting a Server</title>
143 <para>To stop a Lustre server, use the
145 <replaceable>/mount</replaceable>
146 <replaceable>point</replaceable></literal> command.</para>
147 <para>For example, to stop
148 <literal>ost0</literal> on mount point
149 <literal>/mnt/test</literal>, run:</para>
153 <para>Gracefully stopping a server with the
154 <literal>umount</literal> command preserves the state of the connected
155 clients. The next time the server is started, it waits for clients to
156 reconnect, and then goes through the recovery procedure.</para>
158 <literal>-f</literal>) flag is used, then the server evicts all clients and
159 stops WITHOUT recovery. Upon restart, the server does not wait for
160 recovery. Any currently connected clients receive I/O errors until they
163 <para>If you are using loopback devices, use the
164 <literal>-d</literal> flag. This flag cleans up loop devices and can
165 always be safely specified.</para>
168 <section xml:id="dbdoclet.50438194_57420">
171 <primary>operations</primary>
172 <secondary>failover</secondary>
173 </indexterm>Specifying Failout/Failover Mode for OSTs</title>
174 <para>In a Lustre file system, an OST that has become unreachable because
175 it fails, is taken off the network, or is unmounted can be handled in one
180 <literal>failout</literal> mode, Lustre clients immediately receive
181 errors (EIOs) after a timeout, instead of waiting for the OST to
186 <literal>failover</literal> mode, Lustre clients wait for the OST to
190 <para>By default, the Lustre file system uses
191 <literal>failover</literal> mode for OSTs. To specify
192 <literal>failout</literal> mode instead, use the
193 <literal>--param="failover.mode=failout"</literal> option as shown below
194 (entered on one line):</para>
196 oss# mkfs.lustre --fsname=
197 <replaceable>fsname</replaceable> --mgsnode=
198 <replaceable>mgs_NID</replaceable> --param=failover.mode=failout
200 <replaceable>ost_index</replaceable>
201 <replaceable>/dev/ost_block_device</replaceable>
203 <para>In the example below,
204 <literal>failout</literal> mode is specified for the OSTs on the MGS
205 <literal>mds0</literal> in the file system
206 <literal>testfs</literal>(entered on one line).</para>
208 oss# mkfs.lustre --fsname=testfs --mgsnode=mds0 --param=failover.mode=failout
209 --ost --index=3 /dev/sdb
212 <para>Before running this command, unmount all OSTs that will be affected
214 <literal>failover</literal>/
215 <literal>failout</literal> mode.</para>
218 <para>After initial file system configuration, use the
219 <literal>tunefs.lustre</literal> utility to change the mode. For example,
221 <literal>failout</literal> mode, run:</para>
224 $ tunefs.lustre --param failover.mode=failout
225 <replaceable>/dev/ost_device</replaceable>
230 <section xml:id="dbdoclet.50438194_54138">
233 <primary>operations</primary>
234 <secondary>degraded OST RAID</secondary>
235 </indexterm>Handling Degraded OST RAID Arrays</title>
236 <para>Lustre includes functionality that notifies Lustre if an external
237 RAID array has degraded performance (resulting in reduced overall file
238 system performance), either because a disk has failed and not been
239 replaced, or because a disk was replaced and is undergoing a rebuild. To
240 avoid a global performance slowdown due to a degraded OST, the MDS can
241 avoid the OST for new object allocation if it is notified of the degraded
243 <para>A parameter for each OST, called
244 <literal>degraded</literal>, specifies whether the OST is running in
245 degraded mode or not.</para>
246 <para>To mark the OST as degraded, use:</para>
248 lctl set_param obdfilter.{OST_name}.degraded=1
250 <para>To mark that the OST is back in normal operation, use:</para>
252 lctl set_param obdfilter.{OST_name}.degraded=0
254 <para>To determine if OSTs are currently in degraded mode, use:</para>
256 lctl get_param obdfilter.*.degraded
258 <para>If the OST is remounted due to a reboot or other condition, the flag
260 <literal>0</literal>.</para>
261 <para>It is recommended that this be implemented by an automated script
262 that monitors the status of individual RAID devices.</para>
264 <section xml:id="dbdoclet.50438194_88063">
267 <primary>operations</primary>
268 <secondary>multiple file systems</secondary>
269 </indexterm>Running Multiple Lustre File Systems</title>
270 <para>Lustre supports multiple file systems provided the combination of
271 <literal>NID:fsname</literal> is unique. Each file system must be allocated
272 a unique name during creation with the
273 <literal>--fsname</literal> parameter. Unique names for file systems are
274 enforced if a single MGS is present. If multiple MGSs are present (for
275 example if you have an MGS on every MDS) the administrator is responsible
276 for ensuring file system names are unique. A single MGS and unique file
277 system names provides a single point of administration and allows commands
278 to be issued against the file system even if it is not mounted.</para>
279 <para>Lustre supports multiple file systems on a single MGS. With a single
280 MGS fsnames are guaranteed to be unique. Lustre also allows multiple MGSs
281 to co-exist. For example, multiple MGSs will be necessary if multiple file
282 systems on different Lustre software versions are to be concurrently
283 available. With multiple MGSs additional care must be taken to ensure file
284 system names are unique. Each file system should have a unique fsname among
285 all systems that may interoperate in the future.</para>
286 <para>By default, the
287 <literal>mkfs.lustre</literal> command creates a file system named
288 <literal>lustre</literal>. To specify a different file system name (limited
289 to 8 characters) at format time, use the
290 <literal>--fsname</literal> option:</para>
293 mkfs.lustre --fsname=
294 <replaceable>file_system_name</replaceable>
298 <para>The MDT, OSTs and clients in the new file system must use the same
299 file system name (prepended to the device name). For example, for a new
301 <literal>foo</literal>, the MDT and two OSTs would be named
302 <literal>foo-MDT0000</literal>,
303 <literal>foo-OST0000</literal>, and
304 <literal>foo-OST0001</literal>.</para>
306 <para>To mount a client on the file system, run:</para>
308 client# mount -t lustre
309 <replaceable>mgsnode</replaceable>:
310 <replaceable>/new_fsname</replaceable>
311 <replaceable>/mount_point</replaceable>
313 <para>For example, to mount a client on file system foo at mount point
314 /mnt/foo, run:</para>
316 client# mount -t lustre mgsnode:/foo /mnt/foo
319 <para>If a client(s) will be mounted on several file systems, add the
321 <literal>/etc/xattr.conf</literal> file to avoid problems when files are
322 moved between the file systems:
323 <literal>lustre.* skip</literal></para>
326 <para>To ensure that a new MDT is added to an existing MGS create the MDT
328 <literal>--mdt --mgsnode=
329 <replaceable>mgs_NID</replaceable></literal>.</para>
331 <para>A Lustre installation with two file systems (
332 <literal>foo</literal> and
333 <literal>bar</literal>) could look like this, where the MGS node is
334 <literal>mgsnode@tcp0</literal> and the mount points are
335 <literal>/mnt/foo</literal> and
336 <literal>/mnt/bar</literal>.</para>
338 mgsnode# mkfs.lustre --mgs /dev/sda
339 mdtfoonode# mkfs.lustre --fsname=foo --mgsnode=mgsnode@tcp0 --mdt --index=0
341 ossfoonode# mkfs.lustre --fsname=foo --mgsnode=mgsnode@tcp0 --ost --index=0
343 ossfoonode# mkfs.lustre --fsname=foo --mgsnode=mgsnode@tcp0 --ost --index=1
345 mdtbarnode# mkfs.lustre --fsname=bar --mgsnode=mgsnode@tcp0 --mdt --index=0
347 ossbarnode# mkfs.lustre --fsname=bar --mgsnode=mgsnode@tcp0 --ost --index=0
349 ossbarnode# mkfs.lustre --fsname=bar --mgsnode=mgsnode@tcp0 --ost --index=1
352 <para>To mount a client on file system foo at mount point
353 <literal>/mnt/foo</literal>, run:</para>
355 client# mount -t lustre mgsnode@tcp0:/foo /mnt/foo
357 <para>To mount a client on file system bar at mount point
358 <literal>/mnt/bar</literal>, run:</para>
360 client# mount -t lustre mgsnode@tcp0:/bar /mnt/bar
363 <section xml:id="dbdoclet.lfsmkdir" condition='l24'>
366 <primary>operations</primary>
367 <secondary>remote directory</secondary>
368 </indexterm>Creating a sub-directory on a given MDT</title>
369 <para>Lustre 2.4 enables individual sub-directories to be serviced by
370 unique MDTs. An administrator can allocate a sub-directory to a given MDT
371 using the command:</para>
373 client# lfs mkdir –i
374 <replaceable>mdt_index</replaceable>
375 <replaceable>/mount_point/remote_dir</replaceable>
377 <para>This command will allocate the sub-directory
378 <literal>remote_dir</literal> onto the MDT of index
379 <literal>mdtindex</literal>. For more information on adding additional MDTs
381 <literal>mdtindex</literal> see
382 <xref linkend='dbdoclet.addmdtindex' />.</para>
384 <para>An administrator can allocate remote sub-directories to separate
385 MDTs. Creating remote sub-directories in parent directories not hosted on
386 MDT0 is not recommended. This is because the failure of the parent MDT
387 will leave the namespace below it inaccessible. For this reason, by
388 default it is only possible to create remote sub-directories off MDT0. To
389 relax this restriction and enable remote sub-directories off any MDT, an
390 administrator must issue the command
391 <literal>lctl set_param mdt.*.enable_remote_dir=1</literal>.</para>
393 <para condition='l28'>With Lustre software version 2.8, a new
394 tunable is available to allow users with a specific group ID to create
395 and delete remote and striped directories. This tunable is
396 <literal>enable_remote_dir_gid</literal>. For example, setting this
397 parameter to the 'wheel' or 'admin' group ID allows users with that GID
398 to create and delete remote and striped directories. Setting this
399 parameter to <literal>-1</literal> on MDT0 to permanently allow any
400 non-root users create and delete remote and striped directories. For
402 <screen>lctl set_param -P mdt.*.enable_remote_dir_gid=-1</screen>
405 <section xml:id="dbdoclet.lfsmkdirdne2" condition='l28'>
408 <primary>operations</primary>
409 <secondary>striped directory</secondary>
412 <primary>operations</primary>
413 <secondary>mkdir</secondary>
416 <primary>operations</primary>
417 <secondary>setdirstripe</secondary>
420 <primary>striping</primary>
421 <secondary>metadata</secondary>
422 </indexterm>Creating a directory striped across multiple MDTs</title>
423 <para>Lustre 2.8 enables individual files in a given directory to
424 record their metadata on separate MDTs (a <emphasis>striped
425 directory</emphasis>). The result of this is that metadata requests for
426 files in a striped directory are serviced by multiple MDTs and metadata
427 service load is distributed over all the MDTs that service a given
428 directory. By distributing metadata service load over multiple MDTs,
429 performance can be improved beyond the limit of single MDT
430 performance. Prior to the development of this feature all files in a
431 directory must record their metadata on a single MDT.</para>
432 <para>This command to stripe a directory over
433 <replaceable>mdt_count</replaceable> MDTs is:
436 client# lfs setdirstripe -c
437 <replaceable>mdt_count</replaceable>
438 <replaceable>/mount_point/new_directory</replaceable>
441 <section xml:id="dbdoclet.50438194_88980">
444 <primary>operations</primary>
445 <secondary>parameters</secondary>
446 </indexterm>Setting and Retrieving Lustre Parameters</title>
447 <para>Several options are available for setting parameters in
451 <para>When creating a file system, use mkfs.lustre. See
452 <xref linkend="dbdoclet.50438194_17237" />below.</para>
455 <para>When a server is stopped, use tunefs.lustre. See
456 <xref linkend="dbdoclet.50438194_55253" />below.</para>
459 <para>When the file system is running, use lctl to set or retrieve
460 Lustre parameters. See
461 <xref linkend="dbdoclet.50438194_51490" />and
462 <xref linkend="dbdoclet.50438194_63247" />below.</para>
465 <section xml:id="dbdoclet.50438194_17237">
466 <title>Setting Tunable Parameters with
467 <literal>mkfs.lustre</literal></title>
468 <para>When the file system is first formatted, parameters can simply be
470 <literal>--param</literal> option to the
471 <literal>mkfs.lustre</literal> command. For example:</para>
473 mds# mkfs.lustre --mdt --param="sys.timeout=50" /dev/sda
475 <para>For more details about creating a file system,see
476 <xref linkend="configuringlustre" />. For more details about
477 <literal>mkfs.lustre</literal>, see
478 <xref linkend="systemconfigurationutilities" />.</para>
480 <section xml:id="dbdoclet.50438194_55253">
481 <title>Setting Parameters with
482 <literal>tunefs.lustre</literal></title>
483 <para>If a server (OSS or MDS) is stopped, parameters can be added to an
484 existing file system using the
485 <literal>--param</literal> option to the
486 <literal>tunefs.lustre</literal> command. For example:</para>
488 oss# tunefs.lustre --param=failover.node=192.168.0.13@tcp0 /dev/sda
491 <literal>tunefs.lustre</literal>, parameters are
492 <emphasis>additive</emphasis>-- new parameters are specified in addition
493 to old parameters, they do not replace them. To erase all old
494 <literal>tunefs.lustre</literal> parameters and just use newly-specified
495 parameters, run:</para>
497 mds# tunefs.lustre --erase-params --param=
498 <replaceable>new_parameters</replaceable>
500 <para>The tunefs.lustre command can be used to set any parameter settable
501 in a /proc/fs/lustre file and that has its own OBD device, so it can be
504 <replaceable>obdname|fsname</replaceable>.
505 <replaceable>obdtype</replaceable>.
506 <replaceable>proc_file_name</replaceable>=
507 <replaceable>value</replaceable></literal>. For example:</para>
509 mds# tunefs.lustre --param mdt.identity_upcall=NONE /dev/sda1
511 <para>For more details about
512 <literal>tunefs.lustre</literal>, see
513 <xref linkend="systemconfigurationutilities" />.</para>
515 <section xml:id="dbdoclet.50438194_51490">
516 <title>Setting Parameters with
517 <literal>lctl</literal></title>
518 <para>When the file system is running, the
519 <literal>lctl</literal> command can be used to set parameters (temporary
520 or permanent) and report current parameter values. Temporary parameters
521 are active as long as the server or client is not shut down. Permanent
522 parameters live through server and client reboots.</para>
524 <para>The lctl list_param command enables users to list all parameters
526 <xref linkend="dbdoclet.50438194_88217" />.</para>
528 <para>For more details about the
529 <literal>lctl</literal> command, see the examples in the sections below
531 <xref linkend="systemconfigurationutilities" />.</para>
533 <title>Setting Temporary Parameters</title>
535 <literal>lctl set_param</literal> to set temporary parameters on the
536 node where it is run. These parameters map to items in
537 <literal>/proc/{fs,sys}/{lnet,lustre}</literal>. The
538 <literal>lctl set_param</literal> command uses this syntax:</para>
541 <replaceable>obdtype</replaceable>.
542 <replaceable>obdname</replaceable>.
543 <replaceable>proc_file_name</replaceable>=
544 <replaceable>value</replaceable>
546 <para>For example:</para>
548 # lctl set_param osc.*.max_dirty_mb=1024
549 osc.myth-OST0000-osc.max_dirty_mb=32
550 osc.myth-OST0001-osc.max_dirty_mb=32
551 osc.myth-OST0002-osc.max_dirty_mb=32
552 osc.myth-OST0003-osc.max_dirty_mb=32
553 osc.myth-OST0004-osc.max_dirty_mb=32
556 <section xml:id="dbdoclet.50438194_64195">
557 <title>Setting Permanent Parameters</title>
559 <literal>lctl conf_param</literal> command to set permanent parameters.
561 <literal>lctl conf_param</literal> command can be used to specify any
562 parameter settable in a
563 <literal>/proc/fs/lustre</literal> file, with its own OBD device. The
564 <literal>lctl conf_param</literal> command uses this syntax (same as the
566 <literal>mkfs.lustre</literal> and
567 <literal>tunefs.lustre</literal> commands):</para>
569 <replaceable>obdname|fsname</replaceable>.
570 <replaceable>obdtype</replaceable>.
571 <replaceable>proc_file_name</replaceable>=
572 <replaceable>value</replaceable>)
574 <para>Here are a few examples of
575 <literal>lctl conf_param</literal> commands:</para>
577 mgs# lctl conf_param testfs-MDT0000.sys.timeout=40
578 $ lctl conf_param testfs-MDT0000.mdt.identity_upcall=NONE
579 $ lctl conf_param testfs.llite.max_read_ahead_mb=16
580 $ lctl conf_param testfs-MDT0000.lov.stripesize=2M
581 $ lctl conf_param testfs-OST0000.osc.max_dirty_mb=29.15
582 $ lctl conf_param testfs-OST0000.ost.client_cache_seconds=15
583 $ lctl conf_param testfs.sys.timeout=40
586 <para>Parameters specified with the
587 <literal>lctl conf_param</literal> command are set permanently in the
588 file system's configuration file on the MGS.</para>
591 <section xml:id="dbdoclet.setparamp" condition='l25'>
592 <title>Setting Permanent Parameters with lctl set_param -P</title>
594 <literal>lctl set_param -P</literal> to set parameters permanently. This
595 command must be issued on the MGS. The given parameter is set on every
597 <literal>lctl</literal> upcall. Parameters map to items in
598 <literal>/proc/{fs,sys}/{lnet,lustre}</literal>. The
599 <literal>lctl set_param</literal> command uses this syntax:</para>
602 <replaceable>obdtype</replaceable>.
603 <replaceable>obdname</replaceable>.
604 <replaceable>proc_file_name</replaceable>=
605 <replaceable>value</replaceable>
607 <para>For example:</para>
609 # lctl set_param -P osc.*.max_dirty_mb=1024
610 osc.myth-OST0000-osc.max_dirty_mb=32
611 osc.myth-OST0001-osc.max_dirty_mb=32
612 osc.myth-OST0002-osc.max_dirty_mb=32
613 osc.myth-OST0003-osc.max_dirty_mb=32
614 osc.myth-OST0004-osc.max_dirty_mb=32
617 <literal>-d</literal>(only with -P) option to delete permanent
618 parameter. Syntax:</para>
621 <replaceable>obdtype</replaceable>.
622 <replaceable>obdname</replaceable>.
623 <replaceable>proc_file_name</replaceable>
625 <para>For example:</para>
627 # lctl set_param -P -d osc.*.max_dirty_mb
630 <section xml:id="dbdoclet.50438194_88217">
631 <title>Listing Parameters</title>
632 <para>To list Lustre or LNET parameters that are available to set, use
634 <literal>lctl list_param</literal> command. For example:</para>
636 lctl list_param [-FR]
637 <replaceable>obdtype</replaceable>.
638 <replaceable>obdname</replaceable>
640 <para>The following arguments are available for the
641 <literal>lctl list_param</literal> command.</para>
643 <literal>-F</literal> Add '
644 <literal>/</literal>', '
645 <literal>@</literal>' or '
646 <literal>=</literal>' for directories, symlinks and writeable files,
649 <literal>-R</literal> Recursively lists all parameters under the
650 specified path</para>
651 <para>For example:</para>
653 oss# lctl list_param obdfilter.lustre-OST0000
656 <section xml:id="dbdoclet.50438194_63247">
657 <title>Reporting Current Parameter Values</title>
658 <para>To report current Lustre parameter values, use the
659 <literal>lctl get_param</literal> command with this syntax:</para>
662 <replaceable>obdtype</replaceable>.
663 <replaceable>obdname</replaceable>.
664 <replaceable>proc_file_name</replaceable>
666 <para>This example reports data on RPC service times.</para>
668 oss# lctl get_param -n ost.*.ost_io.timeouts
669 service : cur 1 worst 30 (at 1257150393, 85d23h58m54s ago) 1 1 1 1
671 <para>This example reports the amount of space this client has reserved
672 for writeback cache with each OST:</para>
674 client# lctl get_param osc.*.cur_grant_bytes
675 osc.myth-OST0000-osc-ffff8800376bdc00.cur_grant_bytes=2097152
676 osc.myth-OST0001-osc-ffff8800376bdc00.cur_grant_bytes=33890304
677 osc.myth-OST0002-osc-ffff8800376bdc00.cur_grant_bytes=35418112
678 osc.myth-OST0003-osc-ffff8800376bdc00.cur_grant_bytes=2097152
679 osc.myth-OST0004-osc-ffff8800376bdc00.cur_grant_bytes=33808384
684 <section xml:id="dbdoclet.50438194_41817">
687 <primary>operations</primary>
688 <secondary>failover</secondary>
689 </indexterm>Specifying NIDs and Failover</title>
690 <para>If a node has multiple network interfaces, it may have multiple NIDs,
691 which must all be identified so other nodes can choose the NID that is
692 appropriate for their network interfaces. Typically, NIDs are specified in
693 a list delimited by commas (
694 <literal>,</literal>). However, when failover nodes are specified, the NIDs
695 are delimited by a colon (
696 <literal>:</literal>) or by repeating a keyword such as
697 <literal>--mgsnode=</literal> or
698 <literal>--servicenode=</literal>).</para>
699 <para>To display the NIDs of all servers in networks configured to work
700 with the Lustre file system, run (while LNET is running):</para>
704 <para>In the example below,
705 <literal>mds0</literal> and
706 <literal>mds1</literal> are configured as a combined MGS/MDT failover pair
708 <literal>oss0</literal> and
709 <literal>oss1</literal> are configured as an OST failover pair. The Ethernet
711 <literal>mds0</literal> is 192.168.10.1, and for
712 <literal>mds1</literal> is 192.168.10.2. The Ethernet addresses for
713 <literal>oss0</literal> and
714 <literal>oss1</literal> are 192.168.10.20 and 192.168.10.21
717 mds0# mkfs.lustre --fsname=testfs --mdt --mgs \
718 --servicenode=192.168.10.2@tcp0 \
719 -–servicenode=192.168.10.1@tcp0 /dev/sda1
720 mds0# mount -t lustre /dev/sda1 /mnt/test/mdt
721 oss0# mkfs.lustre --fsname=testfs --servicenode=192.168.10.20@tcp0 \
722 --servicenode=192.168.10.21 --ost --index=0 \
723 --mgsnode=192.168.10.1@tcp0 --mgsnode=192.168.10.2@tcp0 \
725 oss0# mount -t lustre /dev/sdb /mnt/test/ost0
726 client# mount -t lustre 192.168.10.1@tcp0:192.168.10.2@tcp0:/testfs \
728 mds0# umount /mnt/mdt
729 mds1# mount -t lustre /dev/sda1 /mnt/test/mdt
730 mds1# cat /proc/fs/lustre/mds/testfs-MDT0000/recovery_status
732 <para>Where multiple NIDs are specified separated by commas (for example,
733 <literal>10.67.73.200@tcp,192.168.10.1@tcp</literal>), the two NIDs refer
734 to the same host, and the Lustre software chooses the
735 <emphasis>best</emphasis>one for communication. When a pair of NIDs is
736 separated by a colon (for example,
737 <literal>10.67.73.200@tcp:10.67.73.201@tcp</literal>), the two NIDs refer
738 to two different hosts and are treated as a failover pair (the Lustre
739 software tries the first one, and if that fails, it tries the second
742 <literal>mkfs.lustre</literal> can be used to specify failover nodes.
743 Introduced in Lustre software release 2.0, the
744 <literal>--servicenode</literal> option is used to specify all service NIDs,
745 including those for primary nodes and failover nodes. When the
746 <literal>--servicenode</literal> option is used, the first service node to
747 load the target device becomes the primary service node, while nodes
748 corresponding to the other specified NIDs become failover locations for the
749 target device. An older option,
750 <literal>--failnode</literal>, specifies just the NIDS of failover nodes.
751 For more information about the
752 <literal>--servicenode</literal> and
753 <literal>--failnode</literal> options, see
754 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
755 linkend="configuringfailover" />.</para>
757 <section xml:id="dbdoclet.50438194_70905">
760 <primary>operations</primary>
761 <secondary>erasing a file system</secondary>
762 </indexterm>Erasing a File System</title>
763 <para>If you want to erase a file system and permanently delete all the
764 data in the file system, run this command on your targets:</para>
766 $ "mkfs.lustre --reformat"
768 <para>If you are using a separate MGS and want to keep other file systems
769 defined on that MGS, then set the
770 <literal>writeconf</literal> flag on the MDT for that file system. The
771 <literal>writeconf</literal> flag causes the configuration logs to be
772 erased; they are regenerated the next time the servers start.</para>
774 <literal>writeconf</literal> flag on the MDT:</para>
777 <para>Unmount all clients/servers using this file system, run:</para>
783 <para>Permanently erase the file system and, presumably, replace it
784 with another file system, run:</para>
786 $ mkfs.lustre --reformat --fsname spfs --mgs --mdt --index=0 /dev/
787 <emphasis>{mdsdev}</emphasis>
791 <para>If you have a separate MGS (that you do not want to reformat),
793 <literal>--writeconf</literal> flag to
794 <literal>mkfs.lustre</literal> on the MDT, run:</para>
796 $ mkfs.lustre --reformat --writeconf --fsname spfs --mgsnode=
797 <replaceable>mgs_nid</replaceable> --mdt --index=0
798 <replaceable>/dev/mds_device</replaceable>
803 <para>If you have a combined MGS/MDT, reformatting the MDT reformats the
804 MGS as well, causing all configuration information to be lost; you can
805 start building your new file system. Nothing needs to be done with old
806 disks that will not be part of the new file system, just do not mount
810 <section xml:id="dbdoclet.50438194_16954">
813 <primary>operations</primary>
814 <secondary>reclaiming space</secondary>
815 </indexterm>Reclaiming Reserved Disk Space</title>
816 <para>All current Lustre installations run the ldiskfs file system
817 internally on service nodes. By default, ldiskfs reserves 5% of the disk
818 space to avoid file system fragmentation. In order to reclaim this space,
819 run the following command on your OSS for each OST in the file
822 tune2fs [-m reserved_blocks_percent] /dev/
823 <emphasis>{ostdev}</emphasis>
825 <para>You do not need to shut down Lustre before running this command or
826 restart it afterwards.</para>
828 <para>Reducing the space reservation can cause severe performance
829 degradation as the OST file system becomes more than 95% full, due to
830 difficulty in locating large areas of contiguous free space. This
831 performance degradation may persist even if the space usage drops below
832 95% again. It is recommended NOT to reduce the reserved disk space below
836 <section xml:id="dbdoclet.50438194_69998">
839 <primary>operations</primary>
840 <secondary>replacing an OST or MDS</secondary>
841 </indexterm>Replacing an Existing OST or MDT</title>
842 <para>To copy the contents of an existing OST to a new OST (or an old MDT
843 to a new MDT), follow the process for either OST/MDT backups in
844 <xref linkend='dbdoclet.50438207_71633' />or
845 <xref linkend='dbdoclet.50438207_21638' />. For more information on
847 <xref linkend='dbdoclet.rmremotedir' />.</para>
849 <section xml:id="dbdoclet.50438194_30872">
852 <primary>operations</primary>
853 <secondary>identifying OSTs</secondary>
854 </indexterm>Identifying To Which Lustre File an OST Object Belongs</title>
855 <para>Use this procedure to identify the file containing a given object on
859 <para>On the OST (as root), run
860 <literal>debugfs</literal> to display the file identifier (
861 <literal>FID</literal>) of the file associated with the object.</para>
862 <para>For example, if the object is
863 <literal>34976</literal> on
864 <literal>/dev/lustre/ost_test2</literal>, the debug command is:
866 # debugfs -c -R "stat /O/0/d$((34976 % 32))/34976" /dev/lustre/ost_test2
868 <para>The command output is:
870 debugfs 1.42.3.wc3 (15-Aug-2012)
871 /dev/lustre/ost_test2: catastrophic mode - not reading inode or group bitmaps
872 Inode: 352365 Type: regular Mode: 0666 Flags: 0x80000
873 Generation: 2393149953 Version: 0x0000002a:00005f81
874 User: 1000 Group: 1000 Size: 260096
875 File ACL: 0 Directory ACL: 0
876 Links: 1 Blockcount: 512
877 Fragment: Address: 0 Number: 0 Size: 0
878 ctime: 0x4a216b48:00000000 -- Sat May 30 13:22:16 2009
879 atime: 0x4a216b48:00000000 -- Sat May 30 13:22:16 2009
880 mtime: 0x4a216b48:00000000 -- Sat May 30 13:22:16 2009
881 crtime: 0x4a216b3c:975870dc -- Sat May 30 13:22:04 2009
882 Size of extra inode fields: 24
883 Extended attributes stored in inode body:
884 fid = "b9 da 24 00 00 00 00 00 6a fa 0d 3f 01 00 00 00 eb 5b 0b 00 00 00 0000
885 00 00 00 00 00 00 00 00 " (32)
886 fid: objid=34976 seq=0 parent=[0x24dab9:0x3f0dfa6a:0x0] stripe=1
888 (0-64):4620544-4620607
892 <para>For Lustre software release 2.x file systems, the parent FID will
893 be of the form [0x200000400:0x122:0x0] and can be resolved directly
895 <literal>lfs fid2path [0x200000404:0x122:0x0]
896 /mnt/lustre</literal> command on any Lustre client, and the process is
900 <para>In this example the parent inode FID is an upgraded 1.x inode
901 (due to the first part of the FID being below 0x200000400), the MDT
903 <literal>0x24dab9</literal> and generation
904 <literal>0x3f0dfa6a</literal> and the pathname needs to be resolved
906 <literal>debugfs</literal>.</para>
909 <para>On the MDS (as root), use
910 <literal>debugfs</literal> to find the file associated with the
913 # debugfs -c -R "ncheck 0x24dab9" /dev/lustre/mdt_test
915 <para>Here is the command output:</para>
917 debugfs 1.42.3.wc2 (15-Aug-2012)
918 /dev/lustre/mdt_test: catastrophic mode - not reading inode or group bitmap\
921 2415289 /ROOT/brian-laptop-guest/clients/client11/~dmtmp/PWRPNT/ZD16.BMP
925 <para>The command lists the inode and pathname associated with the
929 <literal>Debugfs</literal>' ''ncheck'' is a brute-force search that may
930 take a long time to complete.</para>
933 <para>To find the Lustre file from a disk LBA, follow the steps listed in
934 the document at this URL:
935 <link xl:href="http://smartmontools.sourceforge.net/badblockhowto.html">
936 http://smartmontools.sourceforge.net/badblockhowto.html</link>. Then,
937 follow the steps above to resolve the Lustre filename.</para>