1 <?xml version='1.0' encoding='utf-8'?>
2 <chapter xmlns="http://docbook.org/ns/docbook"
3 xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"
4 xml:id="upgradinglustre">
5 <title xml:id="upgradinglustre.title">Upgrading a Lustre File System</title>
6 <para>This chapter describes interoperability between Lustre software
7 releases. It also provides procedures for upgrading from Lustre software
8 release 1.8 to Lustre software release 2.x , from a Lustre software release
9 2.x to a more recent Lustre software release 2.x (major release upgrade), and
10 from a a Lustre software release 2.x.y to a more recent Lustre software
11 release 2.x.y (minor release upgrade). It includes the following
16 <xref linkend="dbdoclet.50438205_82079" />
21 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
22 linkend="Upgrading_2.x" />
27 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
28 linkend="Upgrading_2.x.x" />
32 <section xml:id="dbdoclet.50438205_82079">
35 <primary>Lustre</primary>
36 <secondary>upgrading</secondary>
40 <primary>upgrading</primary>
41 </indexterm>Release Interoperability and Upgrade Requirements</title>
43 <emphasis role="italic">
44 <emphasis role="bold">Lustre software release 2.x (major)
49 <para>All servers must be upgraded at the same time, while some or
50 all clients may be upgraded independently of the servers.</para>
53 <para>All servers must be be upgraded to a Linux kernel supported by
54 the Lustre software. See the Lustre Release Notes for your Lustre
55 version for a list of tested Linux distributions.</para>
58 <para>Clients to be upgraded must be running a compatible Linux
59 distribution as described in the Release Notes.</para>
64 <emphasis role="italic">
65 <emphasis role="bold">Lustre software release 2.x.y release (minor)
71 <para>All servers must be upgraded at the same time, while some or all
72 clients may be upgraded.</para>
75 <para>Rolling upgrades are supported for minor releases allowing
76 individual servers and clients to be upgraded without stopping the
77 Lustre file system.</para>
81 <section xml:id="Upgrading_2.x">
84 <primary>upgrading</primary>
85 <secondary>major release (2.x to 2.x)</secondary>
88 <primary>wide striping</primary>
91 <primary>MDT</primary>
92 <secondary>multiple MDSs</secondary>
95 <primary>large_xattr</primary>
96 <secondary>ea_inode</secondary>
99 <primary>wide striping</primary>
100 <secondary>large_xattr</secondary>
101 <tertiary>ea_inode</tertiary>
102 </indexterm>Upgrading to Lustre Software Release 2.x (Major
104 <para>The procedure for upgrading from a Lustre software release 2.x to a
105 more recent 2.x release of the Lustre software is described in this
108 <para>This procedure can also be used to upgrade Lustre software release
109 1.8.6-wc1 or later to any Lustre software release 2.x. To upgrade other
110 versions of Lustre software release 1.8.x, contact your support
114 <para>In Lustre software release 2.2, a feature has been added for
115 ldiskfs-based MDTs that allows striping a single file across up to 2000
116 OSTs. By default, this "wide striping" feature is disabled. It is
117 activated by setting the <literal>ea_inode</literal> option on the MDT
118 using either <literal>mkfs.lustre</literal> or <literal>tune2fs</literal>.
119 For example after upgrading an existing file system to Lustre software
120 release 2.2 or later, wide striping can be enabled by running the
121 following command on the MDT device before mounting it:
122 <screen>tune2fs -O large_xattr</screen>
123 Once the wide striping feature is enabled and in use on the MDT, it is
124 not possible to directly downgrade the MDT file system to an earlier
125 version of the Lustre software that does not support wide striping. To
126 disable wide striping:
129 <para>Delete all wide-striped files, <emphasis>OR</emphasis>
130 use <literal>lfs_migrate -c 160</literal> (or fewer stripes)
131 to migrate the files to use fewer OSTs. This does not affect the
132 total number of OSTs that the whole filesystem can access.</para>
135 <para>Unmount the MDT.</para>
138 <para>Run the following command to turn off the
139 <literal>large_xattr</literal> option:
140 <screen>tune2fs -O ^large_xattr</screen></para>
142 </orderedlist>Using either
143 <literal>mkfs.lustre</literal> or
144 <literal>tune2fs</literal> with
145 <literal>large_xattr</literal> or
146 <literal>ea_inode</literal> option reseults in
147 <literal>ea_inode</literal> in the file system feature list.</para>
149 <note condition="l23">
150 <para>To generate a list of all files with more than 160 stripes use
151 <literal>lfs find</literal> with the
152 <literal>--stripe-count</literal> option:
153 <screen>lfs find ${mountpoint} --stripe-count=+160</screen></para>
155 <note condition="l24">
156 <para>In Lustre software release 2.4, a new feature allows using multiple
157 MDTs, which can each serve one or more remote sub-directories in the file
159 <literal>root</literal> directory is always located on MDT0.</para>
160 <para>Note that clients running a release prior to the Lustre software
161 release 2.4 can only see the namespace hosted by MDT0 and will return an
162 IO error if an attempt is made to access a directory on another
165 <para>To upgrade a Lustre software release 2.x to a more recent major
166 release, complete these steps:</para>
169 <para>Create a complete, restorable file system backup.</para>
171 <para>Before installing the Lustre software, back up ALL data. The
172 Lustre software contains kernel modifications that interact with
173 storage devices and may introduce security issues and data loss if
174 not installed, configured, or administered properly. If a full backup
175 of the file system is not practical, a device-level backup of the MDT
176 file system is recommended. See
177 <xref linkend="backupandrestore" /> for a procedure.</para>
181 <para>Shut down the entire filesystem by following
182 <xref linkend="dbdoclet.shutdownLustre"/></para>
185 <para>Upgrade the Linux operating system on all servers to a compatible
186 (tested) Linux distribution and reboot.</para>
189 <para>Upgrade the Linux operating system on all clients to Red Hat
190 Enterprise Linux 6 or other compatible (tested) distribution and
194 <para>Download the Lustre server RPMs for your platform from the
195 <link xl:href="https://wiki.whamcloud.com/display/PUB/Lustre+Releases">
196 Lustre Releases</link>repository. See
197 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
198 linkend="table_cnh_5m3_gk" />for a list of required packages.</para>
201 <para>Install the Lustre server packages on all Lustre servers (MGS,
202 MDSs, and OSSs).</para>
203 <orderedlist numeration="loweralpha">
205 <para>Log onto a Lustre server as the
206 <literal>root</literal> user</para>
210 <literal>yum</literal> command to install the packages:</para>
212 <screen># yum --nogpgcheck install pkg1.rpm pkg2.rpm ... </screen>
216 <para>Verify the packages are installed correctly:</para>
218 <screen>rpm -qa|egrep "lustre|wc"</screen>
222 <para>Repeat these steps on each Lustre server.</para>
227 <para>Download the Lustre client RPMs for your platform from the
228 <link xl:href="https://wiki.whamcloud.com/display/PUB/Lustre+Releases">
229 Lustre Releases</link>repository. See
230 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
231 linkend="table_j3r_ym3_gk" />for a list of required packages.</para>
233 <para>The version of the kernel running on a Lustre client must be
234 the same as the version of the
235 <literal>lustre-client-modules-</literal>
236 <replaceable>ver</replaceable>package being installed. If not, a
237 compatible kernel must be installed on the client before the Lustre
238 client packages are installed.</para>
242 <para>Install the Lustre client packages on each of the Lustre clients
243 to be upgraded.</para>
244 <orderedlist numeration="loweralpha">
246 <para>Log onto a Lustre client as the
247 <literal>root</literal> user.</para>
251 <literal>yum</literal> command to install the packages:</para>
253 <screen># yum --nogpgcheck install pkg1.rpm pkg2.rpm ... </screen>
257 <para>Verify the packages were installed correctly:</para>
259 <screen># rpm -qa|egrep "lustre|kernel"</screen>
263 <para>Repeat these steps on each Lustre client.</para>
268 <para>(Optional) For upgrades to Lustre software release 2.2 or higher,
269 to enable wide striping on an existing MDT, run the following command
271 <screen>tune2fs -O ea_inode /dev/<replaceable>mdtdev</replaceable></screen>
273 <para>For more information about wide striping, see
274 <xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="wide_striping" />.</para>
277 <para>(Optional) For upgrades to Lustre software release 2.4 or higher,
278 to format an additional MDT, complete these steps:
279 <orderedlist numeration="loweralpha">
281 <para>Determine the index used for the first MDT (each MDT must
282 have unique index). Enter:
283 <screen>client$ lctl dl | grep mdc
284 36 UP mdc lustre-MDT0000-mdc-ffff88004edf3c00
285 4c8be054-144f-9359-b063-8477566eb84e 5</screen></para>
286 <para>In this example, the next available index is 1.</para>
289 <para>Add the new block device as a new MDT at the next available
290 index by entering (on one line):
291 <screen>mds# mkfs.lustre --reformat --fsname=<replaceable>filesystem_name</replaceable> --mdt \
292 --mgsnode=<replaceable>mgsnode</replaceable> --index <replaceable>1</replaceable>
293 <replaceable>/dev/mdt1_device</replaceable></screen></para>
295 </orderedlist></para>
298 <para>(Optional) If you are upgrading to Lustre software release 2.3 or
299 higher from Lustre software release 2.2 or earlier and want to enable
300 the quota feature, complete these steps:
301 <orderedlist numeration="loweralpha">
303 <para>Before setting up the file system, enter on both the MDS and
305 <screen>tunefs.lustre --quota</screen></para>
308 <para>(Optional) If you are upgrading before Lustre software release
309 2.10, to enable the project quota feature enter the following on every
310 ldiskfs backend target:
311 <screen>tune2fs –O project /dev/<replaceable>dev</replaceable></screen>
313 <note><para>Enabling the <literal>project</literal> feature will prevent
314 the filesystem from being used by older versions of ldiskfs, so it
315 should only be enabled if the project quota feature is required and/or
316 after it is known that the upgraded release does not need to be
317 downgraded.</para></note>
320 <para>When setting up the file system, enter:
321 <screen>conf_param $FSNAME.quota.mdt=$QUOTA_TYPE
322 conf_param $FSNAME.quota.ost=$QUOTA_TYPE</screen></para>
324 </orderedlist></para>
327 <para>(Optional) If you are upgrading from Lustre software release 1.8,
328 you must manually enable the FID-in-dirent feature. On the MDS, enter:
329 <screen>tune2fs –O dirdata /dev/<replaceable>mdtdev</replaceable></screen></para>
331 <para>This step is not reversible. Do not complete this step until
332 you are sure you will not be downgrading the Lustre software.</para>
334 <para condition='l24'>This step only enables FID-in-dirent for newly
335 created files. If you are upgrading to Lustre software release 2.4,
336 you can use namespace LFSCK to enable FID-in-dirent for the existing
337 files. For the case of upgrading from Lustre software release 1.8, it is
338 important to note that if you do NOT enable <literal>dirdata</literal> via
339 the <literal>tune2fs</literal> command above, the namespace LFSCK will NOT
340 generate FID-in-dirent for the existing files. For more information about
341 FID-in-dirent and related functionalities in LFSCK, see
342 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
343 linkend="understandinglustre.storageio" />.</para>
346 <para>Start the Lustre file system by starting the components in the
347 order shown in the following steps:</para>
348 <orderedlist numeration="loweralpha">
350 <para>Mount the MGT. On the MGS, run
351 <screen>mgs# mount -a -t lustre</screen></para>
354 <para>Mount the MDT(s). On each MDT, run:
355 <screen>mds# mount -a -t lustre</screen></para>
358 <para>Mount all the OSTs. On each OSS node, run:</para>
359 <screen>oss# mount -a -t lustre</screen>
361 <para>This command assumes that all the OSTs are listed in the
362 <literal>/etc/fstab</literal> file. OSTs that are not listed in
364 <literal>/etc/fstab</literal> file, must be mounted individually
365 by running the mount command:</para>
366 <screen>mount -t lustre <replaceable>/dev/block_device</replaceable><replaceable>/mount_point</replaceable></screen>
370 <para>Mount the file system on the clients. On each client node,
372 <screen>client# mount -a -t lustre</screen>
378 <para>The mounting order described in the steps above must be followed
379 for the initial mount and registration of a Lustre file system after an
380 upgrade. For a normal start of a Lustre file system, the mounting order
381 is MGT, OSTs, MDT(s), clients.</para>
383 <para>If you have a problem upgrading a Lustre file system, see
384 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
385 linkend="dbdoclet.50438198_30989" />for some ways to get help.</para>
387 <section xml:id="Upgrading_2.x.x">
390 <primary>upgrading</primary>
391 <secondary>2.X.y to 2.X.y (minor release)</secondary>
392 </indexterm>Upgrading to Lustre Software Release 2.x.y (Minor
394 <para>Rolling upgrades are supported for upgrading from any Lustre software
395 release 2.x.y to a more recent Lustre software release 2.X.y. This allows
396 the Lustre file system to continue to run while individual servers (or
397 their failover partners) and clients are upgraded one at a time. The
398 procedure for upgrading a Lustre software release 2.x.y to a more recent
399 minor release is described in this section.</para>
400 <para>To upgrade Lustre software release 2.x.y to a more recent minor
401 release, complete these steps:</para>
404 <para>Create a complete, restorable file system backup.</para>
406 <para>Before installing the Lustre software, back up ALL data. The
407 Lustre software contains kernel modifications that interact with
408 storage devices and may introduce security issues and data loss if
409 not installed, configured, or administered properly. If a full backup
410 of the file system is not practical, a device-level backup of the MDT
411 file system is recommended. See
412 <xref linkend="backupandrestore" /> for a procedure.</para>
416 <para>Download the Lustre server RPMs for your platform from the
417 <link xl:href="https://wiki.whamcloud.com/display/PUB/Lustre+Releases">
418 Lustre Releases</link>repository. See
419 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
420 linkend="table_cnh_5m3_gk" />for a list of required packages.</para>
423 <para>For a rolling upgrade, complete any procedures required to keep
424 the Lustre file system running while the server to be upgraded is
425 offline, such as failing over a primary server to its secondary
429 <para>Unmount the Lustre server to be upgraded (MGS, MDS, or
433 <para>Install the Lustre server packages on the Lustre server.</para>
434 <orderedlist numeration="loweralpha">
436 <para>Log onto the Lustre server as the
437 <literal>root</literal> user</para>
441 <literal>yum</literal> command to install the packages:</para>
443 <screen># yum --nogpgcheck install pkg1.rpm pkg2.rpm ... </screen>
447 <para>Verify the packages are installed correctly:</para>
449 <screen>rpm -qa|egrep "lustre|wc"</screen>
453 <para>Mount the Lustre server to restart the Lustre software on the
455 <screen>server# mount -a -t lustre</screen></para>
458 <para>Repeat these steps on each Lustre server.</para>
463 <para>Download the Lustre client RPMs for your platform from the
464 <link xl:href="https://wiki.whamcloud.com/display/PUB/Lustre+Releases">
465 Lustre Releases</link>repository. See
466 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
467 linkend="table_j3r_ym3_gk" />for a list of required packages.</para>
470 <para>Install the Lustre client packages on each of the Lustre clients
471 to be upgraded.</para>
472 <orderedlist numeration="loweralpha">
474 <para>Log onto a Lustre client as the
475 <literal>root</literal> user.</para>
479 <literal>yum</literal> command to install the packages:</para>
481 <screen># yum --nogpgcheck install pkg1.rpm pkg2.rpm ... </screen>
485 <para>Verify the packages were installed correctly:</para>
487 <screen># rpm -qa|egrep "lustre|kernel"</screen>
491 <para>Mount the Lustre client to restart the Lustre software on the
493 <screen>client# mount -a -t lustre</screen></para>
496 <para>Repeat these steps on each Lustre client.</para>
501 <para>If you have a problem upgrading a Lustre file system, see
502 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
503 linkend="dbdoclet.50438198_30989" />for some suggestions for how to get