1 <?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustremaintenance">
2 <title xml:id="lustremaintenance.title">Lustre Maintenance</title>
3 <para>Once you have the Lustre file system up and running, you can use the procedures in this section to perform these basic Lustre maintenance tasks:</para>
6 <para><xref linkend="lustremaint.inactiveOST"/></para>
9 <para><xref linkend="lustremaint.findingNodes"/></para>
12 <para><xref linkend="lustremaint.mountingServerWithoutLustre"/></para>
15 <para><xref linkend="lustremaint.regenerateConfigLogs"/></para>
18 <para><xref linkend="lustremaint.changingservernid"/></para>
21 <para><xref linkend="lustremaint.clear_conf"/></para>
24 <para><xref linkend="lustremaint.adding_new_mdt"/></para>
27 <para><xref linkend="lustremaint.adding_new_ost"/></para>
30 <para><xref linkend="lustremaint.deactivating_mdt_ost"/></para>
33 <para><xref linkend="lustremaint.rmremotedir"/></para>
36 <para><xref linkend="lustremaint.inactivemdt"/></para>
39 <para><xref linkend="lustremaint.remove_ost"/></para>
42 <para><xref linkend="lustremaint.ydg_pgt_tl"/></para>
45 <para><xref linkend="lustremaint.restore_ost"/></para>
48 <para><xref linkend="lustremaint.ucf_qgt_tl"/></para>
51 <para><xref linkend="lustremaint.abortRecovery"/></para>
54 <para><xref linkend="lustremaint.determineOST"/></para>
57 <para><xref linkend="lustremaint.ChangeAddrFailoverNode"/></para>
60 <para><xref linkend="lustremaint.seperateCombinedMGSMDT"/></para>
63 <section xml:id="lustremaint.inactiveOST">
65 <indexterm><primary>maintenance</primary></indexterm>
66 <indexterm><primary>maintenance</primary><secondary>inactive OSTs</secondary></indexterm>
67 Working with Inactive OSTs</title>
68 <para>To mount a client or an MDT with one or more inactive OSTs, run commands similar to this:</para>
69 <screen>client# mount -o exclude=testfs-OST0000 -t lustre \
70 uml1:/testfs /mnt/testfs
71 client# lctl get_param lov.testfs-clilov-*.target_obd</screen>
72 <para>To activate an inactive OST on a live client or MDT, use the
73 <literal>lctl activate</literal> command on the OSC device. For example:</para>
74 <screen>lctl --device 7 activate</screen>
76 <para>A colon-separated list can also be specified. For example,
77 <literal>exclude=testfs-OST0000:testfs-OST0001</literal>.</para>
80 <section xml:id="lustremaint.findingNodes">
81 <title><indexterm><primary>maintenance</primary><secondary>finding nodes</secondary></indexterm>
82 Finding Nodes in the Lustre File System</title>
83 <para>There may be situations in which you need to find all nodes in
84 your Lustre file system or get the names of all OSTs.</para>
85 <para>To get a list of all Lustre nodes, run this command on the MGS:</para>
86 <screen># lctl get_param mgs.MGS.live.*</screen>
88 <para>This command must be run on the MGS.</para>
90 <para>In this example, file system <literal>testfs</literal> has three
91 nodes, <literal>testfs-MDT0000</literal>,
92 <literal>testfs-OST0000</literal>, and
93 <literal>testfs-OST0001</literal>.</para>
94 <screen>mgs:/root# lctl get_param mgs.MGS.live.*
99 testfs-OST0001 </screen>
100 <para>To get the names of all OSTs, run this command on the MDS:</para>
101 <screen>mds:/root# lctl get_param lov.*-mdtlov.target_obd </screen>
103 <para>This command must be run on the MDS.</para>
105 <para>In this example, there are two OSTs, testfs-OST0000 and
106 testfs-OST0001, which are both active.</para>
107 <screen>mgs:/root# lctl get_param lov.testfs-mdtlov.target_obd
108 0: testfs-OST0000_UUID ACTIVE
109 1: testfs-OST0001_UUID ACTIVE </screen>
111 <section xml:id="lustremaint.mountingServerWithoutLustre">
112 <title><indexterm><primary>maintenance</primary><secondary>mounting a server</secondary></indexterm>
113 Mounting a Server Without Lustre Service</title>
114 <para>If you are using a combined MGS/MDT, but you only want to start the MGS and not the MDT, run this command:</para>
115 <screen>mount -t lustre <replaceable>/dev/mdt_partition</replaceable> -o nosvc <replaceable>/mount_point</replaceable></screen>
116 <para>The <literal><replaceable>mdt_partition</replaceable></literal> variable is the combined MGS/MDT block device.</para>
117 <para>In this example, the combined MGS/MDT is <literal>testfs-MDT0000</literal> and the mount point is <literal>/mnt/test/mdt</literal>.</para>
118 <screen>$ mount -t lustre -L testfs-MDT0000 -o nosvc /mnt/test/mdt</screen>
120 <section xml:id="lustremaint.regenerateConfigLogs">
121 <title><indexterm><primary>maintenance</primary><secondary>regenerating config logs</secondary></indexterm>
122 Regenerating Lustre Configuration Logs</title>
123 <para>If the Lustre file system configuration logs are in a state where
124 the file system cannot be started, use the
125 <literal>tunefs.lustre --writeconf</literal> command to regenerate them.
126 After the <literal>writeconf</literal> command is run and the servers
127 restart, the configuration logs are re-generated and stored on the MGS
128 (as with a new file system).</para>
129 <para>You should only use the <literal>writeconf</literal> command if:</para>
132 <para>The configuration logs are in a state where the file system cannot start</para>
135 <para>A server NID is being changed</para>
138 <para>The <literal>writeconf</literal> command is destructive to some
139 configuration items (e.g. OST pools information and tunables set via
140 <literal>conf_param</literal>), and should be used with caution.</para>
142 <para>The OST pools feature enables a group of OSTs to be named for
143 file striping purposes. If you use OST pools, be aware that running
144 the <literal>writeconf</literal> command erases
145 <emphasis role="bold">all</emphasis> pools information (as well as
146 any other parameters set via <literal>lctl conf_param</literal>).
147 We recommend that the pools definitions (and
148 <literal>conf_param</literal> settings) be executed via a script,
149 so they can be regenerated easily after <literal>writeconf</literal>
150 is performed. However, tunables saved with <literal>lctl set_param
151 -P</literal> are <emphasis>not</emphasis> erased in this case.</para>
154 <para>If the MGS still holds any configuration logs, it may be
155 possible to dump these logs to save any parameters stored with
156 <literal>lctl conf_param</literal> by dumping the config logs on
157 the MGS and saving the output:</para>
159 mgs# lctl --device MGS llog_print <replaceable>fsname</replaceable>-client
160 mgs# lctl --device MGS llog_print <replaceable>fsname</replaceable>-MDT0000
161 mgs# lctl --device MGS llog_print <replaceable>fsname</replaceable>-OST0000
164 <para>To regenerate Lustre file system configuration logs:</para>
167 <para>Stop the file system services in the following order before
168 running the <literal>tunefs.lustre --writeconf</literal> command:
172 <para>Unmount the clients.</para>
175 <para>Unmount the MDT(s).</para>
178 <para>Unmount the OST(s).</para>
181 <para>If the MGS is separate from the MDT it can remain mounted
182 during this process.</para>
187 <para>Make sure the MDT and OST devices are available.</para>
190 <para>Run the <literal>tunefs.lustre --writeconf</literal> command
191 on all target devices.</para>
192 <para>Run writeconf on the MDT(s) first, and then the OST(s).</para>
195 <para>On each MDS, for each MDT run:</para>
196 <screen>mds# tunefs.lustre --writeconf <replaceable>/dev/mdt_device</replaceable></screen>
199 <para> On each OSS, for each OST run:
200 <screen>oss# tunefs.lustre --writeconf <replaceable>/dev/ost_device</replaceable></screen>
206 <para>Restart the file system in the following order:</para>
209 <para>Mount the separate MGT, if it is not already mounted.</para>
212 <para>Mount the MDT(s) in order, starting with MDT0000.</para>
215 <para>Mount the OSTs in order, starting with OST0000.</para>
218 <para>Mount the clients.</para>
223 <para>After the <literal>tunefs.lustre --writeconf</literal> command is
224 run, the configuration logs are re-generated as servers connect to the
227 <section xml:id="lustremaint.changingservernid">
228 <title><indexterm><primary>maintenance</primary><secondary>changing a NID</secondary></indexterm>
229 Changing a Server NID</title>
230 <para>In Lustre software release 2.3 or earlier, the <literal>tunefs.lustre
231 --writeconf</literal> command is used to rewrite all of the configuration files.</para>
232 <para condition="l24">If you need to change the NID on the MDT or OST, a new
233 <literal>replace_nids</literal> command was added in Lustre software release 2.4 to simplify
234 this process. The <literal>replace_nids</literal> command differs from <literal>tunefs.lustre
235 --writeconf</literal> in that it does not erase the entire configuration log, precluding the
236 need the need to execute the <literal>writeconf</literal> command on all servers and
237 re-specify all permanent parameter settings. However, the <literal>writeconf</literal> command
238 can still be used if desired.</para>
239 <para>Change a server NID in these situations:</para>
242 <para>New server hardware is added to the file system, and the MDS or an OSS is being moved to the new machine.</para>
245 <para>New network card is installed in the server.</para>
248 <para>You want to reassign IP addresses.</para>
251 <para>To change a server NID:</para>
254 <para>Update the LNet configuration in the <literal>/etc/modprobe.conf</literal> file so the list of server NIDs is correct. Use <literal>lctl list_nids</literal> to view the list of server NIDS.</para>
255 <para>The <literal>lctl list_nids</literal> command indicates which network(s) are
256 configured to work with the Lustre file system.</para>
259 <para>Shut down the file system in this order:</para>
262 <para>Unmount the clients.</para>
265 <para>Unmount the MDT.</para>
268 <para>Unmount all OSTs.</para>
273 <para>If the MGS and MDS share a partition, start the MGS only:</para>
274 <screen>mount -t lustre <replaceable>MDT partition</replaceable> -o nosvc <replaceable>mount_point</replaceable></screen>
277 <para>Run the <literal>replace_nids</literal> command on the MGS:</para>
278 <screen>lctl replace_nids <replaceable>devicename</replaceable> <replaceable>nid1</replaceable>[,nid2,nid3 ...]</screen>
279 <para>where <replaceable>devicename</replaceable> is the Lustre target name, e.g.
280 <literal>testfs-OST0013</literal></para>
283 <para>If the MGS and MDS share a partition, stop the MGS:</para>
284 <screen>umount <replaceable>mount_point</replaceable></screen>
287 <note><para>The <literal>replace_nids</literal> command also cleans
288 all old, invalidated records out of the configuration log, while
289 preserving all other current settings.</para></note>
290 <note><para>The previous configuration log is backed up on the MGS
291 disk with the suffix <literal>'.bak'</literal>.</para></note>
293 <section xml:id="lustremaint.clear_conf" condition="l2B">
295 <primary>maintenance</primary>
296 <secondary>Clearing a config</secondary>
297 </indexterm> Clearing configuration</title>
299 This command runs on MGS node having the MGS device mounted with
300 <literal>-o nosvc.</literal> It cleans up configuration files
301 stored in the CONFIGS/ directory of any records marked SKIP.
302 If the device name is given, then the specific logs for that
303 filesystem (e.g. testfs-MDT0000) are processed. Otherwise, if a
304 filesystem name is given then all configuration files are cleared.
305 The previous configuration log is backed up on the MGS disk with
306 the suffix 'config.timestamp.bak'. Eg: Lustre-MDT0000-1476454535.bak.
308 <para> To clear a configuration:</para>
311 <para>Shut down the file system in this order:</para>
314 <para>Unmount the clients.</para>
317 <para>Unmount the MDT.</para>
320 <para>Unmount all OSTs.</para>
326 If the MGS and MDS share a partition, start the MGS only
327 using "nosvc" option.
329 <screen>mount -t lustre <replaceable>MDT partition</replaceable> -o nosvc <replaceable>mount_point</replaceable></screen>
332 <para>Run the <literal>clear_conf</literal> command on the MGS:
334 <screen>lctl clear_conf <replaceable>config</replaceable></screen>
336 Example: To clear the configuration for
337 <literal>MDT0000</literal> on a filesystem named
338 <literal>testfs</literal>
340 <screen>mgs# lctl clear_conf testfs-MDT0000</screen>
344 <section xml:id="lustremaint.adding_new_mdt" condition='l24'>
346 <primary>maintenance</primary>
347 <secondary>adding an MDT</secondary>
348 </indexterm>Adding a New MDT to a Lustre File System</title>
349 <para>Additional MDTs can be added using the DNE feature to serve one
350 or more remote sub-directories within a filesystem, in order to
351 increase the total number of files that can be created in the
352 filesystem, to increase aggregate metadata performance, or to isolate
353 user or application workloads from other users of the filesystem. It
354 is possible to have multiple remote sub-directories reference the
355 same MDT. However, the root directory will always be located on
356 MDT0. To add a new MDT into the file system:</para>
359 <para>Discover the maximum MDT index. Each MDT must have unique index.</para>
361 client$ lctl dl | grep mdc
362 36 UP mdc testfs-MDT0000-mdc-ffff88004edf3c00 4c8be054-144f-9359-b063-8477566eb84e 5
363 37 UP mdc testfs-MDT0001-mdc-ffff88004edf3c00 4c8be054-144f-9359-b063-8477566eb84e 5
364 38 UP mdc testfs-MDT0002-mdc-ffff88004edf3c00 4c8be054-144f-9359-b063-8477566eb84e 5
365 39 UP mdc testfs-MDT0003-mdc-ffff88004edf3c00 4c8be054-144f-9359-b063-8477566eb84e 5
369 <para>Add the new block device as a new MDT at the next available
370 index. In this example, the next available index is 4.</para>
372 mds# mkfs.lustre --reformat --fsname=<replaceable>testfs</replaceable> --mdt --mgsnode=<replaceable>mgsnode</replaceable> --index 4 <replaceable>/dev/mdt4_device</replaceable>
376 <para>Mount the MDTs.</para>
378 mds# mount –t lustre <replaceable>/dev/mdt4_blockdevice</replaceable> /mnt/mdt4
382 <para>In order to start creating new files and directories on the
383 new MDT(s) they need to be attached into the namespace at one or
384 more subdirectories using the <literal>lfs mkdir</literal> command.
385 All files and directories below those created with
386 <literal>lfs mkdir</literal> will also be created on the same MDT
387 unless otherwise specified.
390 client# lfs mkdir -i 3 /mnt/testfs/new_dir_on_mdt3
391 client# lfs mkdir -i 4 /mnt/testfs/new_dir_on_mdt4
392 client# lfs mkdir -c 4 /mnt/testfs/new_directory_striped_across_4_mdts
397 <section xml:id="lustremaint.adding_new_ost">
398 <title><indexterm><primary>maintenance</primary><secondary>adding a OST</secondary></indexterm>
399 Adding a New OST to a Lustre File System</title>
400 <para>A new OST can be added to existing Lustre file system on either
401 an existing OSS node or on a new OSS node. In order to keep client IO
402 load balanced across OSS nodes for maximum aggregate performance, it is
403 not recommended to configure different numbers of OSTs to each OSS node.
407 <para> Add a new OST by using <literal>mkfs.lustre</literal> as when
408 the filesystem was first formatted, see
409 <xref linkend="dbdoclet.format_ost" /> for details. Each new OST
410 must have a unique index number, use <literal>lctl dl</literal> to
411 see a list of all OSTs. For example, to add a new OST at index 12
412 to the <literal>testfs</literal> filesystem run following commands
413 should be run on the OSS:</para>
414 <screen>oss# mkfs.lustre --fsname=testfs --mgsnode=mds16@tcp0 --ost --index=12 /dev/sda
415 oss# mkdir -p /mnt/testfs/ost12
416 oss# mount -t lustre /dev/sda /mnt/testfs/ost12</screen>
419 <para>Balance OST space usage (possibly).</para>
420 <para>The file system can be quite unbalanced when new empty OSTs
421 are added to a relatively full filesystem. New file creations are
422 automatically balanced to favour the new OSTs. If this is a scratch
423 file system or files are pruned at regular intervals, then no further
424 work may be needed to balance the OST space usage as new files being
425 created will preferentially be placed on the less full OST(s). As old
426 files are deleted, they will release space on the old OST(s).</para>
427 <para>Files existing prior to the expansion can optionally be
428 rebalanced using the <literal>lfs_migrate</literal> utility.
429 This redistributes file data over the entire set of OSTs.</para>
430 <para>For example, to rebalance all files within the directory
431 <literal>/mnt/lustre/dir</literal>, enter:</para>
432 <screen>client# lfs_migrate /mnt/lustre/dir</screen>
433 <para>To migrate files within the <literal>/test</literal> file
434 system on <literal>OST0004</literal> that are larger than 4GB in
435 size to other OSTs, enter:</para>
436 <screen>client# lfs find /test --ost test-OST0004 -size +4G | lfs_migrate -y</screen>
437 <para>See <xref linkend="dbdoclet.lfs_migrate"/> for details.</para>
441 <section xml:id="lustremaint.deactivating_mdt_ost">
442 <title><indexterm><primary>maintenance</primary><secondary>restoring an OST</secondary></indexterm>
443 <indexterm><primary>maintenance</primary><secondary>removing an OST</secondary></indexterm>
444 Removing and Restoring MDTs and OSTs</title>
445 <para>OSTs and DNE MDTs can be removed from and restored to a Lustre
446 filesystem. Deactivating an OST means that it is temporarily or
447 permanently marked unavailable. Deactivating an OST on the MDS means
448 it will not try to allocate new objects there or perform OST recovery,
449 while deactivating an OST the client means it will not wait for OST
450 recovery if it cannot contact the OST and will instead return an IO
451 error to the application immediately if files on the OST are accessed.
452 An OST may be permanently deactivated from the file system,
453 depending on the situation and commands used.</para>
454 <note><para>A permanently deactivated MDT or OST still appears in the
455 filesystem configuration until the configuration is regenerated with
456 <literal>writeconf</literal> or it is replaced with a new MDT or OST
457 at the same index and permanently reactivated. A deactivated OST
458 will not be listed by <literal>lfs df</literal>.
460 <para>You may want to temporarily deactivate an OST on the MDS to
461 prevent new files from being written to it in several situations:</para>
464 <para>A hard drive has failed and a RAID resync/rebuild is underway,
465 though the OST can also be marked <emphasis>degraded</emphasis> by
466 the RAID system to avoid allocating new files on the slow OST which
467 can reduce performance, see <xref linkend='dbdoclet.degraded_ost' />
472 <para>OST is nearing its space capacity, though the MDS will already
473 try to avoid allocating new files on overly-full OSTs if possible,
474 see <xref linkend='dbdoclet.balancing_free_space' /> for details.
478 <para>MDT/OST storage or MDS/OSS node has failed, and will not
479 be available for some time (or forever), but there is still a
480 desire to continue using the filesystem before it is repaired.</para>
483 <section condition="l24" xml:id="lustremaint.rmremotedir">
484 <title><indexterm><primary>maintenance</primary><secondary>removing an MDT</secondary></indexterm>Removing an MDT from the File System</title>
485 <para>If the MDT is permanently inaccessible,
486 <literal>lfs rm_entry {directory}</literal> can be used to delete the
487 directory entry for the unavailable MDT. Using <literal>rmdir</literal>
488 would otherwise report an IO error due to the remote MDT being inactive.
489 Please note that if the MDT <emphasis>is</emphasis> available, standard
490 <literal>rm -r</literal> should be used to delete the remote directory.
491 After the remote directory has been removed, the administrator should
492 mark the MDT as permanently inactive with:</para>
493 <screen>lctl conf_param {MDT name}.mdc.active=0</screen>
494 <para>A user can identify which MDT holds a remote sub-directory using
495 the <literal>lfs</literal> utility. For example:</para>
496 <screen>client$ lfs getstripe --mdt-index /mnt/lustre/remote_dir1
498 client$ mkdir /mnt/lustre/local_dir0
499 client$ lfs getstripe --mdt-index /mnt/lustre/local_dir0
502 <para>The <literal>lfs getstripe --mdt-index</literal> command
503 returns the index of the MDT that is serving the given directory.</para>
505 <section xml:id="lustremaint.inactivemdt" condition='l24'>
507 <indexterm><primary>maintenance</primary></indexterm>
508 <indexterm><primary>maintenance</primary><secondary>inactive MDTs</secondary></indexterm>Working with Inactive MDTs</title>
509 <para>Files located on or below an inactive MDT are inaccessible until
510 the MDT is activated again. Clients accessing an inactive MDT will receive
513 <section remap="h3" xml:id="lustremaint.remove_ost">
515 <primary>maintenance</primary>
516 <secondary>removing an OST</secondary>
517 </indexterm>Removing an OST from the File System</title>
518 <para>When deactivating an OST, note that the client and MDS each have
519 an OSC device that handles communication with the corresponding OST.
520 To remove an OST from the file system:</para>
523 <para>If the OST is functional, and there are files located on
524 the OST that need to be migrated off of the OST, the file creation
525 for that OST should be temporarily deactivated on the MDS (each MDS
526 if running with multiple MDS nodes in DNE mode).
530 <para condition="l29">With Lustre 2.9 and later, the MDS should be
531 set to only disable file creation on that OST by setting
532 <literal>max_create_count</literal> to zero:
533 <screen>mds# lctl set_param osp.<replaceable>osc_name</replaceable>.max_create_count=0</screen>
534 This ensures that files deleted or migrated off of the OST
535 will have their corresponding OST objects destroyed, and the space
536 will be freed. For example, to disable <literal>OST0000</literal>
537 in the filesystem <literal>testfs</literal>, run:
538 <screen>mds# lctl set_param osp.testfs-OST0000-osc-MDT*.max_create_count=0</screen>
539 on each MDS in the <literal>testfs</literal> filesystem.</para>
542 <para>With older versions of Lustre, to deactivate the OSC on the
544 <screen>mds# lctl set_param osp.<replaceable>osc_name</replaceable>.active=0</screen>
545 This will prevent the MDS from attempting any communication with
546 that OST, including destroying objects located thereon. This is
547 fine if the OST will be removed permanently, if the OST is not
548 stable in operation, or if it is in a read-only state. Otherwise,
549 the free space and objects on the OST will not decrease when
550 files are deleted, and object destruction will be deferred until
551 the MDS reconnects to the OST.</para>
552 <para>For example, to deactivate <literal>OST0000</literal> in
553 the filesystem <literal>testfs</literal>, run:
554 <screen>mds# lctl set_param osp.testfs-OST0000-osc-MDT*.active=0</screen>
555 Deactivating the OST on the <emphasis>MDS</emphasis> does not
556 prevent use of existing objects for read/write by a client.</para>
558 <para>If migrating files from a working OST, do not deactivate
559 the OST on clients. This causes IO errors when accessing files
560 located there, and migrating files on the OST would fail.</para>
563 <para>Do not use <literal>lctl conf_param</literal> to
564 deactivate the OST if it is still working, as this immediately
565 and permanently deactivates it in the file system configuration
566 on both the MDS and all clients.</para>
572 <para>Discover all files that have objects residing on the
573 deactivated OST. Depending on whether the deactivated OST is
574 available or not, the data from that OST may be migrated to
575 other OSTs, or may need to be restored from backup.</para>
578 <para>If the OST is still online and available, find all
579 files with objects on the deactivated OST, and copy them
580 to other OSTs in the file system to: </para>
581 <screen>client# lfs find --ost <replaceable>ost_name</replaceable> <replaceable>/mount/point</replaceable> | lfs_migrate -y</screen>
582 <para>Note that if multiple OSTs are being deactivated at one
583 time, the <literal>lfs find</literal> command can take multiple
584 <literal>--ost</literal> arguments, and will return files that
585 are located on <emphasis>any</emphasis> of the specified OSTs.
589 <para>If the OST is no longer available, delete the files
590 on that OST and restore them from backup:
591 <screen>client# lfs find --ost <replaceable>ost_uuid</replaceable> -print0 <replaceable>/mount/point</replaceable> |
592 tee /tmp/files_to_restore | xargs -0 -n 1 unlink</screen>
593 The list of files that need to be restored from backup is
594 stored in <literal>/tmp/files_to_restore</literal>. Restoring
595 these files is beyond the scope of this document.</para>
600 <para>Deactivate the OST.</para>
604 If there is expected to be a replacement OST in some short
605 time (a few days), the OST can temporarily be deactivated on
607 <screen>client# lctl set_param osc.<replaceable>fsname</replaceable>-OST<replaceable>number</replaceable>-*.active=0</screen>
608 <note><para>This setting is only temporary and will be reset
609 if the clients are remounted or rebooted. It needs to be run
610 on all clients.</para>
615 <para>If there is not expected to be a replacement for this OST in
616 the near future, permanently deactivate it on all clients and
617 the MDS by running the following command on the MGS:
618 <screen>mgs# lctl conf_param <replaceable>ost_name</replaceable>.osc.active=0</screen></para>
619 <note><para>A deactivated OST still appears in the file system
620 configuration, though a replacement OST can be created using the
621 <literal>mkfs.lustre --replace</literal> option, see
622 <xref linkend="lustremaint.restore_ost"/>.
629 <section remap="h3" xml:id="lustremaint.ydg_pgt_tl">
631 <primary>maintenance</primary>
632 <secondary>backing up OST config</secondary>
635 <primary>backup</primary>
636 <secondary>OST config</secondary>
637 </indexterm> Backing Up OST Configuration Files</title>
638 <para>If the OST device is still accessible, then the Lustre
639 configuration files on the OST should be backed up and saved for
640 future use in order to avoid difficulties when a replacement OST is
641 returned to service. These files rarely change, so they can and
642 should be backed up while the OST is functional and accessible. If
643 the deactivated OST is still available to mount (i.e. has not
644 permanently failed or is unmountable due to severe corruption), an
645 effort should be made to preserve these files. </para>
648 <para>Mount the OST file system.
649 <screen>oss# mkdir -p /mnt/ost
650 oss# mount -t ldiskfs <replaceable>/dev/ost_device</replaceable> /mnt/ost</screen>
654 <para>Back up the OST configuration files.
655 <screen>oss# tar cvf <replaceable>ost_name</replaceable>.tar -C /mnt/ost last_rcvd \
656 CONFIGS/ O/0/LAST_ID</screen>
660 <para> Unmount the OST file system. <screen>oss# umount /mnt/ost</screen>
665 <section xml:id="lustremaint.restore_ost">
667 <primary>maintenance</primary>
668 <secondary>restoring OST config</secondary>
671 <primary>backup</primary>
672 <secondary>restoring OST config</secondary>
673 </indexterm> Restoring OST Configuration Files</title>
674 <para>If the original OST is still available, it is best to follow the
675 OST backup and restore procedure given in either
676 <xref linkend="dbdoclet.backup_device"/>, or
677 <xref linkend="backup_fs_level"/> and
678 <xref linkend="backup_fs_level.restore"/>.</para>
679 <para>To replace an OST that was removed from service due to corruption
680 or hardware failure, the replacement OST needs to be formatted using
681 <literal>mkfs.lustre</literal>, and the Lustre file system configuration
682 should be restored, if available. Any objects stored on the OST will
683 be permanently lost, and files using the OST should be deleted and/or
684 restored from backup.</para>
685 <para condition="l25">With Lustre 2.5 and later, it is possible to
686 replace an OST to the same index without restoring the configuration
687 files, using the <literal>--replace</literal> option at format time.
688 <screen>oss# mkfs.lustre --ost --reformat --replace --index=<replaceable>old_ost_index</replaceable> \
689 <replaceable>other_options</replaceable> <replaceable>/dev/new_ost_dev</replaceable></screen>
690 The MDS and OSS will negotiate the <literal>LAST_ID</literal> value
691 for the replacement OST.
693 <para>If the OST configuration files were not backed up, due to the
694 OST file system being completely inaccessible, it is still possible to
695 replace the failed OST with a new one at the same OST index. </para>
698 <para>For older versions, format the OST file system without the
699 <literal>--replace</literal> option and restore the saved
701 <screen>oss# mkfs.lustre --ost --reformat --index=<replaceable>old_ost_index</replaceable> \
702 <replaceable>other_options</replaceable> <replaceable>/dev/new_ost_dev</replaceable></screen>
706 <para> Mount the OST file system.
707 <screen>oss# mkdir /mnt/ost
708 oss# mount -t ldiskfs <replaceable>/dev/new_ost_dev</replaceable> <replaceable>/mnt/ost</replaceable></screen>
712 <para>Restore the OST configuration files, if available.
713 <screen>oss# tar xvf <replaceable>ost_name</replaceable>.tar -C /mnt/ost</screen></para>
716 <para>Recreate the OST configuration files, if unavailable. </para>
717 <para>Follow the procedure in
718 <xref linkend="dbdoclet.repair_ost_lastid"/> to recreate the LAST_ID
719 file for this OST index. The <literal>last_rcvd</literal> file
720 will be recreated when the OST is first mounted using the default
721 parameters, which are normally correct for all file systems. The
722 <literal>CONFIGS/mountdata</literal> file is created by
723 <literal>mkfs.lustre</literal> at format time, but has flags set
724 that request it to register itself with the MGS. It is possible to
725 copy the flags from another working OST (which should be the same):
726 <screen>oss1# debugfs -c -R "dump CONFIGS/mountdata /tmp" <replaceable>/dev/other_osdev</replaceable>
727 oss1# scp /tmp/mountdata oss0:/tmp/mountdata
728 oss0# dd if=/tmp/mountdata of=/mnt/ost/CONFIGS/mountdata bs=4 count=1 seek=5 skip=5 conv=notrunc</screen></para>
731 <para> Unmount the OST file system.
732 <screen>oss# umount /mnt/ost</screen>
737 <section xml:id="lustremaint.ucf_qgt_tl">
739 <primary>maintenance</primary>
740 <secondary>reintroducing an OSTs</secondary>
741 </indexterm>Returning a Deactivated OST to Service</title>
742 <para>If the OST was permanently deactivated, it needs to be
743 reactivated in the MGS configuration.
744 <screen>mgs# lctl conf_param <replaceable>ost_name</replaceable>.osc.active=1</screen>
745 If the OST was temporarily deactivated, it needs to be reactivated on
747 <screen>mds# lctl set_param osp.<replaceable>fsname</replaceable>-OST<replaceable>number</replaceable>-*.active=1
748 client# lctl set_param osc.<replaceable>fsname</replaceable>-OST<replaceable>number</replaceable>-*.active=1</screen></para>
751 <section xml:id="lustremaint.abortRecovery">
752 <title><indexterm><primary>maintenance</primary><secondary>aborting recovery</secondary></indexterm>
753 <indexterm><primary>backup</primary><secondary>aborting recovery</secondary></indexterm>
754 Aborting Recovery</title>
755 <para>You can abort recovery with either the <literal>lctl</literal> utility or by mounting the target with the <literal>abort_recov</literal> option (<literal>mount -o abort_recov</literal>). When starting a target, run: <screen>mds# mount -t lustre -L <replaceable>mdt_name</replaceable> -o abort_recov <replaceable>/mount_point</replaceable></screen></para>
757 <para>The recovery process is blocked until all OSTs are available. </para>
760 <section xml:id="lustremaint.determineOST">
761 <title><indexterm><primary>maintenance</primary><secondary>identifying OST host</secondary></indexterm>
762 Determining Which Machine is Serving an OST </title>
763 <para>In the course of administering a Lustre file system, you may need to determine which
764 machine is serving a specific OST. It is not as simple as identifying the machine’s IP
765 address, as IP is only one of several networking protocols that the Lustre software uses and,
766 as such, LNet does not use IP addresses as node identifiers, but NIDs instead. To identify the
767 NID that is serving a specific OST, run one of the following commands on a client (you do not
768 need to be a root user):
769 <screen>client$ lctl get_param osc.<replaceable>fsname</replaceable>-OST<replaceable>number</replaceable>*.ost_conn_uuid</screen>
771 <screen>client$ lctl get_param osc.*-OST0000*.ost_conn_uuid
772 osc.testfs-OST0000-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp</screen>-
774 <screen>client$ lctl get_param osc.*.ost_conn_uuid
775 osc.testfs-OST0000-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
776 osc.testfs-OST0001-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
777 osc.testfs-OST0002-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
778 osc.testfs-OST0003-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp
779 osc.testfs-OST0004-osc-f1579000.ost_conn_uuid=192.168.20.1@tcp</screen></para>
781 <section xml:id="lustremaint.ChangeAddrFailoverNode">
782 <title><indexterm><primary>maintenance</primary><secondary>changing failover node address</secondary></indexterm>
783 Changing the Address of a Failover Node</title>
784 <para>To change the address of a failover node (e.g, to use node X instead of node Y), run
785 this command on the OSS/OST partition (depending on which option was used to originally
787 <screen>oss# tunefs.lustre --erase-params --servicenode=<replaceable>NID</replaceable> <replaceable>/dev/ost_device</replaceable></screen>
789 <screen>oss# tunefs.lustre --erase-params --failnode=<replaceable>NID</replaceable> <replaceable>/dev/ost_device</replaceable></screen>
790 For more information about the <literal>--servicenode</literal> and
791 <literal>--failnode</literal> options, see <xref xmlns:xlink="http://www.w3.org/1999/xlink"
792 linkend="configuringfailover"/>.</para>
794 <section xml:id="lustremaint.seperateCombinedMGSMDT">
795 <title><indexterm><primary>maintenance</primary><secondary>separate a
796 combined MGS/MDT</secondary></indexterm>
797 Separate a combined MGS/MDT</title>
798 <para>These instructions assume the MGS node will be the same as the MDS
799 node. For instructions on how to move MGS to a different node, see
800 <xref linkend="lustremaint.changingservernid"/>.</para>
801 <para>These instructions are for doing the split without shutting down
802 other servers and clients.</para>
805 <para>Stop the MDS.</para>
806 <para>Unmount the MDT</para>
807 <screen>umount -f <replaceable>/dev/mdt_device</replaceable> </screen>
810 <para>Create the MGS.</para>
811 <screen>mds# mkfs.lustre --mgs --device-size=<replaceable>size</replaceable> <replaceable>/dev/mgs_device</replaceable></screen>
814 <para>Copy the configuration data from MDT disk to the new MGS disk.</para>
815 <screen>mds# mount -t ldiskfs -o ro <replaceable>/dev/mdt_device</replaceable> <replaceable>/mdt_mount_point</replaceable></screen>
816 <screen>mds# mount -t ldiskfs -o rw <replaceable>/dev/mgs_device</replaceable> <replaceable>/mgs_mount_point</replaceable> </screen>
817 <screen>mds# cp -r <replaceable>/mdt_mount_point</replaceable>/CONFIGS/<replaceable>filesystem_name</replaceable>-* <replaceable>/mgs_mount_point</replaceable>/CONFIGS/. </screen>
818 <screen>mds# umount <replaceable>/mgs_mount_point</replaceable></screen>
819 <screen>mds# umount <replaceable>/mdt_mount_point</replaceable></screen>
820 <para>See <xref linkend="lustremaint.regenerateConfigLogs"/> for alternative method.</para>
823 <para>Start the MGS.</para>
824 <screen>mgs# mount -t lustre <replaceable>/dev/mgs_device</replaceable> <replaceable>/mgs_mount_point</replaceable></screen>
825 <para>Check to make sure it knows about all your file system</para>
826 <screen>mgs:/root# lctl get_param mgs.MGS.filesystems</screen>
829 <para>Remove the MGS option from the MDT, and set the new MGS nid.</para>
830 <screen>mds# tunefs.lustre --nomgs --mgsnode=<replaceable>new_mgs_nid</replaceable> <replaceable>/dev/mdt-device</replaceable></screen>
833 <para>Start the MDT.</para>
834 <screen>mds# mount -t lustre <replaceable>/dev/mdt_device /mdt_mount_point</replaceable></screen>
835 <para>Check to make sure the MGS configuration looks right:</para>
836 <screen>mgs# lctl get_param mgs.MGS.live.<replaceable>filesystem_name</replaceable></screen>