1 <?xml version='1.0' encoding='utf-8'?>
2 <chapter xmlns="http://docbook.org/ns/docbook"
3 xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"
4 xml:id="upgradinglustre">
5 <title xml:id="upgradinglustre.title">Upgrading a Lustre File System</title>
6 <para>This chapter describes interoperability between Lustre software
7 releases. It also provides procedures for upgrading from Lustre software
8 release 1.8 to Lustre software release 2.x , from a Lustre software release
9 2.x to a more recent Lustre software release 2.x (major release upgrade), and
10 from a a Lustre software release 2.x.y to a more recent Lustre software
11 release 2.x.y (minor release upgrade). It includes the following
16 <xref linkend="dbdoclet.50438205_82079" />
21 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
22 linkend="Upgrading_2.x" />
27 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
28 linkend="Upgrading_2.x.x" />
32 <section xml:id="dbdoclet.50438205_82079">
35 <primary>Lustre</primary>
36 <secondary>upgrading</secondary>
40 <primary>upgrading</primary>
41 </indexterm>Release Interoperability and Upgrade Requirements</title>
43 <emphasis role="italic">
44 <emphasis role="bold">Lustre software release 2.x (major)
49 <para>All servers must be upgraded at the same time, while some or
50 all clients may be upgraded.</para>
53 <para>All servers must be be upgraded to a Linux kernel supported by
54 the Lustre software. See the Linux Test Matrix at
55 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
56 linkend="LustreTestMatrixTable" />for a list of tested Lustre
60 <para>Clients to be upgraded to the Lustre software release 2.4 or
61 higher must be running a compatible Linux distribution. See the Linux
63 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
64 linkend="LustreTestMatrixTable" />for a list of tested Linux
70 <emphasis role="italic">
71 <emphasis role="bold">Lustre software release 2.x.y release (minor)
77 <para>All servers must be upgraded at the same time, while some or all
78 clients may be upgraded.</para>
81 <para>Rolling upgrades are supported for minor releases allowing
82 individual servers and clients to be upgraded without stopping the
83 Lustre file system.</para>
87 <section xml:id="Upgrading_2.x">
90 <primary>upgrading</primary>
91 <secondary>major release (2.x to 2.x)</secondary>
94 <primary>wide striping</primary>
97 <primary>MDT</primary>
98 <secondary>multiple MDSs</secondary>
101 <primary>large_xattr</primary>
102 <secondary>ea_inode</secondary>
105 <primary>wide striping</primary>
106 <secondary>large_xattr</secondary>
107 <tertiary>ea_inode</tertiary>
108 </indexterm>Upgrading to Lustre Software Release 2.x (Major
110 <para>The procedure for upgrading from a Lustre software release 2.x to a
111 more recent 2.x release of the Lustre software is described in this
114 <para>This procedure can also be used to upgrade Lustre software release
115 1.8.6-wc1 or later to any Lustre software release 2.x. To upgrade other
116 versions of Lustre software release 1.8.x, contact your support
120 <para condition="l22">In Lustre software release 2.2, a feature has been
121 added that allows striping across up to 2000 OSTs. By default, this "wide
122 striping" feature is disabled. It is activated by setting the
123 <literal>large_xattr</literal> or
124 <literal>ea_inode</literal> option on the MDT using either
125 <literal>mkfs.lustre</literal> or
126 <literal>tune2fs</literal>. For example after upgrading an existing file
127 system to Lustre software release 2.2 or later, wide striping can be
128 enabled by running the following command on the MDT device before
130 <screen>tune2fs -O large_xattr</screen>
131 Once the wide striping feature is enabled and in use on the MDT, it is
132 not possible to directly downgrade the MDT file system to an earlier
133 version of the Lustre software that does not support wide striping. To
134 disable wide striping:
137 <para>Delete all wide-striped files.</para>
140 <literal>lfs_migrate</literal> with the option
141 <literal>-c</literal>
142 <replaceable>stripe_count</replaceable>(set
143 <replaceable>stripe_count</replaceable>to 160) to move the files to
144 another location.</para>
147 <para>Unmount the MDT.</para>
150 <para>Run the following command to turn off the
151 <literal>large_xattr</literal> option:
152 <screen>tune2fs -O ^large_xattr</screen></para>
154 </orderedlist>Using either
155 <literal>mkfs.lustre</literal> or
156 <literal>tune2fs</literal> with
157 <literal>large_xattr</literal> or
158 <literal>ea_inode</literal> option reseults in
159 <literal>ea_inode</literal> in the file system feature list.</para>
161 <note condition="l23">
162 <para>To generate a list of all files with more than 160 stripes use
163 <literal>lfs find</literal> with the
164 <literal>--stripe-count</literal> option:
165 <screen>lfs find ${mountpoint} --stripe-count=+160</screen></para>
167 <note condition="l24">
168 <para>In Lustre software release 2.4, a new feature allows using multiple
169 MDTs, which can each serve one or more remote sub-directories in the file
171 <literal>root</literal> directory is always located on MDT0.</para>
172 <para>Note that clients running a release prior to the Lustre software
173 release 2.4 can only see the namespace hosted by MDT0 and will return an
174 IO error if an attempt is made to access a directory on another
177 <para>To upgrade a Lustre software release 2.x to a more recent major
178 release, complete these steps:</para>
181 <para>Create a complete, restorable file system backup.</para>
183 <para>Before installing the Lustre software, back up ALL data. The
184 Lustre software contains kernel modifications that interact with
185 storage devices and may introduce security issues and data loss if
186 not installed, configured, or administered properly. If a full backup
187 of the file system is not practical, a device-level backup of the MDT
188 file system is recommended. See
189 <xref linkend="backupandrestore" /> for a procedure.</para>
193 <para>Shut down the file system by unmounting all clients and servers
194 in the order shown below (unmounting a block device causes the Lustre
195 software to be shut down on that node):</para>
196 <orderedlist numeration="loweralpha">
198 <para>Unmount the clients. On each client node, run:</para>
199 <screen>umount -a -t lustre</screen>
202 <para>Unmount the MDT. On the MDS node, run:</para>
203 <screen>umount -a -t lustre</screen>
206 <para>Unmount all the OSTs. On each OSS node, run:</para>
207 <screen>umount -a -t lustre</screen>
212 <para>Upgrade the Linux operating system on all servers to a compatible
213 (tested) Linux distribution and reboot. See the Linux Test Matrix at
214 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
215 linkend="LustreTestMatrixTable" />.</para>
218 <para>Upgrade the Linux operating system on all clients to Red Hat
219 Enterprise Linux 6 or other compatible (tested) distribution and
220 reboot. See the Linux Test Matrix at
221 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
222 linkend="LustreTestMatrixTable" />.</para>
225 <para>Download the Lustre server RPMs for your platform from the
226 <link xl:href="https://wiki.hpdd.intel.com/display/PUB/Lustre+Releases">
227 Lustre Releases</link>repository. See
228 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
229 linkend="table_cnh_5m3_gk" />for a list of required packages.</para>
232 <para>Install the Lustre server packages on all Lustre servers (MGS,
233 MDSs, and OSSs).</para>
234 <orderedlist numeration="loweralpha">
236 <para>Log onto a Lustre server as the
237 <literal>root</literal> user</para>
241 <literal>yum</literal> command to install the packages:</para>
243 <screen># yum --nogpgcheck install pkg1.rpm pkg2.rpm ... </screen>
247 <para>Verify the packages are installed correctly:</para>
249 <screen>rpm -qa|egrep "lustre|wc"</screen>
253 <para>Repeat these steps on each Lustre server.</para>
258 <para>Download the Lustre client RPMs for your platform from the
259 <link xl:href="https://wiki.hpdd.intel.com/display/PUB/Lustre+Releases">
260 Lustre Releases</link>repository. See
261 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
262 linkend="table_j3r_ym3_gk" />for a list of required packages.</para>
264 <para>The version of the kernel running on a Lustre client must be
265 the same as the version of the
266 <literal>lustre-client-modules-</literal>
267 <replaceable>ver</replaceable>package being installed. If not, a
268 compatible kernel must be installed on the client before the Lustre
269 client packages are installed.</para>
273 <para>Install the Lustre client packages on each of the Lustre clients
274 to be upgraded.</para>
275 <orderedlist numeration="loweralpha">
277 <para>Log onto a Lustre client as the
278 <literal>root</literal> user.</para>
282 <literal>yum</literal> command to install the packages:</para>
284 <screen># yum --nogpgcheck install pkg1.rpm pkg2.rpm ... </screen>
288 <para>Verify the packages were installed correctly:</para>
290 <screen># rpm -qa|egrep "lustre|kernel"</screen>
294 <para>Repeat these steps on each Lustre client.</para>
299 <para>(Optional) For upgrades to Lustre software release 2.2 or higher,
300 to enable wide striping on an existing MDT, run the following command
302 <screen>mdt# tune2fs -O large_xattr <replaceable>device</replaceable></screen></para>
303 <para>For more information about wide striping, see
304 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
305 linkend="section_syy_gcl_qk" />.</para>
308 <para>(Optional) For upgrades to Lustre software release 2.4 or higher,
309 to format an additional MDT, complete these steps:
310 <orderedlist numeration="loweralpha">
312 <para>Determine the index used for the first MDT (each MDT must
313 have unique index). Enter:
314 <screen>client$ lctl dl | grep mdc
315 36 UP mdc lustre-MDT0000-mdc-ffff88004edf3c00
316 4c8be054-144f-9359-b063-8477566eb84e 5</screen></para>
317 <para>In this example, the next available index is 1.</para>
320 <para>Add the new block device as a new MDT at the next available
321 index by entering (on one line):
322 <screen>mds# mkfs.lustre --reformat --fsname=<replaceable>filesystem_name</replaceable> --mdt \
323 --mgsnode=<replaceable>mgsnode</replaceable> --index <replaceable>1</replaceable>
324 <replaceable>/dev/mdt1_device</replaceable></screen></para>
326 </orderedlist></para>
329 <para>(Optional) If you are upgrading to Lustre software release 2.3 or
330 higher from Lustre software release 2.2 or earlier and want to enable
331 the quota feature, complete these steps:
332 <orderedlist numeration="loweralpha">
334 <para>Before setting up the file system, enter on both the MDS and
336 <screen>tunefs.lustre --quota</screen></para>
339 <para>When setting up the file system, enter:
340 <screen>conf_param $FSNAME.quota.mdt=$QUOTA_TYPE
341 conf_param $FSNAME.quota.ost=$QUOTA_TYPE</screen></para>
343 </orderedlist></para>
346 <para>(Optional) If you are upgrading from Lustre software release 1.8,
347 you must manually enable the FID-in-dirent feature. On the MDS, enter:
348 <screen>tune2fs –O dirdata /dev/<replaceable>mdtdev</replaceable></screen></para>
350 <para>This step is not reversible. Do not complete this step until
351 you are sure you will not be downgrading the Lustre software.</para>
353 <para condition='l24'>This step only enables FID-in-dirent for newly
354 created files. If you are upgrading to Lustre software release 2.4,
355 you can use LFSCK to enable FID-in-dirent for existing files. For
356 more information about FID-in-dirent and related functionalities in
357 LFSCK, see <xref xmlns:xlink="http://www.w3.org/1999/xlink"
358 linkend="understandinglustre.storageio" />.</para>
361 <para>Start the Lustre file system by starting the components in the
362 order shown in the following steps:</para>
363 <orderedlist numeration="loweralpha">
365 <para>Mount the MGT. On the MGS, run
366 <screen>mgs# mount -a -t lustre</screen></para>
369 <para>Mount the MDT(s). On each MDT, run:
370 <screen>mds# mount -a -t lustre</screen></para>
373 <para>Mount all the OSTs. On each OSS node, run:</para>
374 <screen>oss# mount -a -t lustre</screen>
376 <para>This command assumes that all the OSTs are listed in the
377 <literal>/etc/fstab</literal> file. OSTs that are not listed in
379 <literal>/etc/fstab</literal> file, must be mounted individually
380 by running the mount command:</para>
381 <screen>mount -t lustre <replaceable>/dev/block_device</replaceable><replaceable>/mount_point</replaceable></screen>
385 <para>Mount the file system on the clients. On each client node,
387 <screen>client# mount -a -t lustre</screen>
393 <para>The mounting order described in the steps above must be followed
394 for the initial mount and registration of a Lustre file system after an
395 upgrade. For a normal start of a Lustre file system, the mounting order
396 is MGT, OSTs, MDT(s), clients.</para>
398 <para>If you have a problem upgrading a Lustre file system, see
399 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
400 linkend="dbdoclet.50438198_30989" />for some ways to get help.</para>
402 <section xml:id="Upgrading_2.x.x">
405 <primary>upgrading</primary>
406 <secondary>2.X.y to 2.X.y (minor release)</secondary>
407 </indexterm>Upgrading to Lustre Software Release 2.x.y (Minor
409 <para>Rolling upgrades are supported for upgrading from any Lustre software
410 release 2.x.y to a more recent Lustre software release 2.X.y. This allows
411 the Lustre file system to continue to run while individual servers (or
412 their failover partners) and clients are upgraded one at a time. The
413 procedure for upgrading a Lustre software release 2.x.y to a more recent
414 minor release is described in this section.</para>
415 <para>To upgrade Lustre software release 2.x.y to a more recent minor
416 release, complete these steps:</para>
419 <para>Create a complete, restorable file system backup.</para>
421 <para>Before installing the Lustre software, back up ALL data. The
422 Lustre software contains kernel modifications that interact with
423 storage devices and may introduce security issues and data loss if
424 not installed, configured, or administered properly. If a full backup
425 of the file system is not practical, a device-level backup of the MDT
426 file system is recommended. See
427 <xref linkend="backupandrestore" /> for a procedure.</para>
431 <para>Download the Lustre server RPMs for your platform from the
432 <link xl:href="https://wiki.hpdd.intel.com/display/PUB/Lustre+Releases">
433 Lustre Releases</link>repository. See
434 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
435 linkend="table_cnh_5m3_gk" />for a list of required packages.</para>
438 <para>For a rolling upgrade, complete any procedures required to keep
439 the Lustre file system running while the server to be upgraded is
440 offline, such as failing over a primary server to its secondary
444 <para>Unmount the Lustre server to be upgraded (MGS, MDS, or
448 <para>Install the Lustre server packages on the Lustre server.</para>
449 <orderedlist numeration="loweralpha">
451 <para>Log onto the Lustre server as the
452 <literal>root</literal> user</para>
456 <literal>yum</literal> command to install the packages:</para>
458 <screen># yum --nogpgcheck install pkg1.rpm pkg2.rpm ... </screen>
462 <para>Verify the packages are installed correctly:</para>
464 <screen>rpm -qa|egrep "lustre|wc"</screen>
468 <para>Mount the Lustre server to restart the Lustre software on the
470 <screen>server# mount -a -t lustre</screen></para>
473 <para>Repeat these steps on each Lustre server.</para>
478 <para>Download the Lustre client RPMs for your platform from the
479 <link xl:href="https://wiki.hpdd.intel.com/display/PUB/Lustre+Releases">
480 Lustre Releases</link>repository. See
481 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
482 linkend="table_j3r_ym3_gk" />for a list of required packages.</para>
485 <para>Install the Lustre client packages on each of the Lustre clients
486 to be upgraded.</para>
487 <orderedlist numeration="loweralpha">
489 <para>Log onto a Lustre client as the
490 <literal>root</literal> user.</para>
494 <literal>yum</literal> command to install the packages:</para>
496 <screen># yum --nogpgcheck install pkg1.rpm pkg2.rpm ... </screen>
500 <para>Verify the packages were installed correctly:</para>
502 <screen># rpm -qa|egrep "lustre|kernel"</screen>
506 <para>Mount the Lustre client to restart the Lustre software on the
508 <screen>client# mount -a -t lustre</screen></para>
511 <para>Repeat these steps on each Lustre client.</para>
516 <para>If you have a problem upgrading a Lustre file system, see
517 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
518 linkend="dbdoclet.50438198_30989" />for some suggestions for how to get