1 <?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="installinglustrefromsourcecode">
2 <title xml:id="installinglustrefromsourcecode.title">Installing a Lustre File System from Source
4 <para>This chapter describes how to create a customized Lustre server kernel from source code.
5 Sections included are:</para>
8 <para><xref linkend="dbdoclet.50438210_69313"/></para>
11 <para><xref linkend="dbdoclet.50438210_65411"/></para>
14 <para><xref linkend="dbdoclet.50438210_47529"/></para>
17 <para><xref linkend="section_g4d_yby_lk"/></para>
21 <para>It is recommended that you install from prebuild RPMs of the Lustre software unless you
22 need to customize the Lustre server kernel or will be using an Linux kernel that has not been
23 tested with the Lustre software. Instructions on installing pre-built RPMs are in <xref linkend='installinglustre.title'/></para>
24 <para>For a list of supported Linux distributions and architectures, see <xref
25 xmlns:xlink="http://www.w3.org/1999/xlink" linkend="LustreTestMatrixTable"/>. Prebuild RPMs
26 are available in the <link xl:href="https://wiki.hpdd.intel.com/display/PUB/Lustre+Releases">Lustre Releases</link>
27 repository. For information about installing Lustre RPMs, see <xref
28 xmlns:xlink="http://www.w3.org/1999/xlink" linkend="installinglustre"/>.</para>
30 <section xml:id="dbdoclet.50438210_69313">
32 <primary>installing</primary>
33 <secondary>from source</secondary>
34 <tertiary>prerequisites</tertiary>
35 </indexterm>Overview and Prerequisites</title>
36 <para>To install the Lustre software from source code, the following are required:</para>
39 <para>A x86_64 machine with a fresh installation of a Linux operating system.</para>
40 <note><para>This section of the Manual only describes the specific procedure against building on Red Hat
41 Enterprise Linux 6 and derivatives. For a complete set of supported platforms please
42 visit <xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="LustreTestMatrixTable"
43 />.The server kernel should be one of the supported kernel versions but it is not
44 necessary to run a vendor distribution to build the Lustre software.</para></note>
47 <para>Access to the Lustre software git repository at <link xl:href="http://git.hpdd.intel.com/">http://git.hpdd.intel.com/</link>.</para>
50 <para>(Recommended) Access to a recent version of <link xmlns:xlink="http://www.w3.org/1999/xlink"
51 xlink:href="http://fedoraproject.org/wiki/EPEL">EPEL</link> containing the
52 <literal>quilt</literal> utility used for managing a series of patches. </para>
53 <note><para>The use of <literal>quilt</literal> is optional for building the Lustre software, and is
54 detected at configure time if it is installed. Otherwise the <literal>patch</literal>
55 utility is used to apply <literal>ldiskfs</literal> patches to the kernel source. Quilt
56 is useful if you intend to modify the Lustre kernel patches.</para></note>
59 <para><emphasis role="italic">(Recommended)</emphasis> At least 1 GB memory on the machine
60 used for the build.</para>
63 <para><emphasis role="italic">(Recommended)</emphasis> At least 20 GB hard disk space on the
64 machine used for the build. </para>
67 <para>Security-Enhanced Linux (SELinux) <emphasis role="italic">disabled</emphasis> on all
68 Lustre servers and clients. The Lustre software does not support SELinux.</para>
71 <para>The installation procedure includes several steps:</para>
74 <para> Patching the core kernel.</para>
77 <para> Building Lustre RPMs.</para>
80 <para> Installing and testing the Lustre file system.</para>
83 <para>These steps are described in the following sections.</para>
85 <section xml:id="dbdoclet.50438210_65411">
86 <title><indexterm><primary>installing</primary><secondary>from source</secondary><tertiary>patching the kernel</tertiary></indexterm>Patching the Kernel</title>
87 <para>This section first describes how to prepare your machine to serve as a development
88 environment, including build tools, the Lustre source, and the Linux kernel source. It then
89 describes how to apply Lustre patches to the Linux kernel.</para>
91 <para>A patched Linux kernel is <emphasis>NOT</emphasis> required in order to build the
92 Lustre client code. The steps below for building the Linux kernel are only
93 required if a new kernel is needed for a Lustre server.</para>
97 <primary>quilt</primary>
98 </indexterm>Provisioning the Build Machine and Installing Dependencies</title>
100 <para>This example is specific for building a modified Red Hat Enteprise Linux 6 kernel for
101 the Lustre file system server. It is assumed that the operating system has been freshly
102 installed on a machine with the hostname <literal>rhel6-master</literal>.</para>
104 <para>To provision the build machine and install dependencies, complete these steps.</para>
107 <para>Log in as <systemitem>root</systemitem>.</para>
110 <para>Install the kernel development tools:</para>
112 <screen># yum -y groupinstall "Development Tools"</screen>
115 <para>If the Development Tools group is not available, you will need to satisfy the dependencies for the build manually.</para>
119 <para>Install additional dependencies:</para>
121 <screen># yum -y install xmlto asciidoc elfutils-libelf-devel
122 zlib-devel binutils-devel newt-devel python-devel hmaccalc
123 perl-ExtUtils-Embed</screen>
127 <para>(Optional) Install EPEL:</para>
130 http://download.fedoraproject.org/pub/epel/6/x86_64/
131 epel-release-6-8.noarch.rpm
132 # rpm -ivh ./epel-release-5-4.noarch.rpm
137 <para>(Optional) Install quilt:</para>
139 <screen># yum -y install quilt
144 <package>newt-devel</package> may not be available for Red Hat Enterprise Linux 6. One
145 option is to download the <link xmlns:xlink="http://www.w3.org/1999/xlink"
146 xlink:href="http://mirror.centos.org/centos/6/os/x86_64/Packages/newt-devel-0.52.11-3.el6.x86_64.rpm"
147 >newt-devel</link>, <link xmlns:xlink="http://www.w3.org/1999/xlink"
148 xlink:href="http://mirror.centos.org/centos/6/os/x86_64/Packages/slang-devel-2.2.1-1.el6.x86_64.rpm"
149 >slang-devel</link>, and <link xmlns:xlink="http://www.w3.org/1999/xlink"
150 xlink:href="http://mirror.centos.org/centos/6/os/i386/Packages/asciidoc-8.4.5-4.1.el6.noarch.rpm"
151 >asciidoc</link> RPMs from the CentOS Enterprise Linux Distribution and install
154 <screen>yum --nogpgcheck localinstall
155 ./newt-devel-0.52.11-3.el6.x86_64.rpm
156 ./slang-devel-2.2.1-1.el6.x86_64.rpm
157 ./asciidoc-8.4.5-4.1.el6.noarch.rpm
164 <section remap="h3" xml:id="PreparingLustreSource">
165 <title>Preparing the Lustre Source</title>
166 <para>To prepare the Lustre source, complete these steps.</para>
169 <para>Create a user called <systemitem>build</systemitem> with a home directory
170 <systemitem>/home/build</systemitem>:</para>
172 <screen># useradd -m build
177 <para>Switch to the user called <systemitem>build</systemitem> and change the directory to
178 the users home directory:</para>
186 <para>Get the MASTER branch of the Lustre software from the Lustre software git
189 <screen># git clone git://git.hpdd.intel.com/fs/lustre-release.git
197 <screen>sh ./autogen.sh
202 <para>Resolve any outstanding dependencies. </para>
203 <para><screen># sh ./autogen.sh</screen>When <package>autogen.sh</package> completes
204 successfully, a response similar to the following is displayed: </para>
206 <screen>Checking for a complete tree...
207 checking forautomake-1.9>= 1.9... found 1.9.6
210 configure.ac:10: installing `./config.sub'
211 configure.ac:12: installing `./install-sh'
212 configure.ac:12: installing `./missing'
213 Running autoconf</screen>
219 <title>Preparing the Kernel Source</title>
220 <para>To build the kernel using <literal>rpmbuild</literal> (a tool specific to RPM-based
221 distributions), complete these steps.</para>
224 <para>Get the kernel source: </para>
227 # mkdir -p kernel/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
229 # echo '%_topdir %(echo $HOME)/kernel/rpmbuild'> ~/.rpmmacros
234 <para>Install the kernel source (enter on one line):</para>
236 <screen># rpm -ivh http://ftp.redhat.com/pub/redhat/linux/enterprise/
237 6Server/en/os/SRPMS/kernel-2.6.32-358.14.1.el6.src.rpm
238 2>&1 | grep -v mockb</screen>
241 <para>It is intended that the Lustre software master branch be kept up-to-date with the
242 most recent kernel distributions. However, a delay may occur between a periodic update
243 to a kernel distribution and a corresponding update of the master branch. The most
244 recent supported kernel in the master branch can be found in the source directory in
245 <systemitem>lustre/kernel_patches/which_patch</systemitem>. If the master branch is
246 not current with the latest distribution, download the most recent kernel RPMs from
247 the vendor's download site.</para>
251 <para>Prepare the source using <literal>rpmbuild</literal>:</para>
253 <screen># cd ~/kernel/rpmbuild
254 # rpmbuild -bp --target=`uname -m` ./SPECS/kernel.spec
257 <para>The text displayed ends with the following:</para>
260 gpg: Total number processed: 1
262 + gpg --homedir . --export --keyring ./kernel.pub Red
263 gpg: WARNING: unsafe permissions on homedir `.'
264 + gcc -o scripts/bin2c scripts/bin2c.c
265 + scripts/bin2c ksign_def_public_key __initdata
271 <para>The kernel source with the Red Hat Enterprise Linux patches applied is now residing in
273 <systemitem>/home/build/kernel/rpmbuild/BUILD/kernel-2.6.32-358.14.1.el6.x86_64/</systemitem></para>
276 <title>Patching the Kernel Source with the Lustre Code</title>
277 <para>To patch the kernel source with the Lustre code, complete these steps.</para>
280 <para>Go to the directory containing the kernel source:</para>
282 <screen>#cd ~/kernel/rpmbuild/BUILD/kernel-2.6.32-358.14.1.el6/
283 linux-2.6.32-358.14.1.el6.x86_64/</screen>
287 <para>Edit the <filename>Makefile</filename> file in this directory to add a unique build
288 id to be able to ascertain that the kernel is booted. Modify line 4 as shown
291 <screen>EXTRAVERSION = -358.14.1.el6_lustre.x86_64</screen>
295 <para>Overwrite the <filename>.config</filename> file in this directory with the Lustre
296 <filename>.config</filename> file:</para>
298 <screen># cp ~/lustre-release/lustre/kernel_patches/kernel_configs/
299 kernel-2.6.32-2.6-rhel6-x86_64.config ./.config</screen>
303 <para>Link the Lustre series and patches:</para>
305 <screen># ln -s ~/lustre-release/lustre/kernel_patches/series/
306 2.6-rhel6.series series
307 # ln -s ~/lustre-release/lustre/kernel_patches/patches patches</screen>
311 <para>Apply the patches to the kernel source using <package>quilt</package>:</para>
313 <screen># quilt push -av</screen>
315 <para>The following is displayed:</para>
318 patching file fs/jbd2/transaction.c
319 Hunk #3succeeded at 1222(offset 3lines).
320 Hunk #4succeeded at 1357(offset 3lines).
321 Now at patch patches/jbd2-jcberr-2.6-rhel6.patch</screen>
322 <note><para>If quilt is not installed, the following steps would be equivalent:
323 <screen>for PATCH in $(cat series); do patch -p1 < patches/$PATCH; done</screen></para></note>
328 <section xml:id="dbdoclet.50438210_47529">
331 <primary>installing</primary>
332 <secondary>from source</secondary>
333 <tertiary>installing Lustre RPMs</tertiary>
334 </indexterm> Building the Lustre RPMs</title>
335 <para>This section describes how to configure the patched kernel to work with the Lustre
336 software and how to create and install the Lustre packages (RPMs).</para>
338 <title>Building a New Kernel</title>
339 <para>To build a new kernel as an RPM, complete these steps.</para>
342 <para>In the kernel source directory, build a kernel rpm:</para>
343 <screen># cd ~/kernel/rpmbuild/BUILD/kernel-2.6.32-131.2.1.el6/
344 linux-2.6.32-131.2.1.el6.x86_64/
345 # make oldconfig || make menuconfig
347 # make include/linux/version.h
348 # make SUBDIRS=scripts
349 # make include/linux/utsrelease.h
351 <para>A successful build returns text similar to the following:</para>
354 Wrote: /home/build/kernel/rpmbuild/SRPMS/
355 kernel-2.6.32-358.14.1.el6_lustre.x86_64-1.src.rpm
356 Wrote: /home/build/kernel/rpmbuild/RPMS/x86_64/
357 kernel-2.6.32-358.14.1.el6_lustre.x86_64-1.x86_64.rpm
358 Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.f73m1V
359 + umask 022+ cd /home/build/kernel/rpmbuild/BUILD
360 + cd kernel-2.6.32-358.14.1.el6_lustre.x86_64
361 + rm -rf /home/build/kernel/rpmbuild/BUILDROOT/
362 kernel-2.6.32-358.14.1.el6_lustre.x86_64-1.x86_64
364 rm ../kernel-2.6.3-358.14.1.el6_lustre.x86_64.tar.gz</screen>
366 <para>If a request to generate more entropy appears, some disk or keyboard I/O needs to
367 be generated. You can generate entropy by entering the following on another
368 terminal:<screen># grep -Ri 'any_text' /usr</screen></para>
372 <para>A fresh kernel RPM can now be found at
373 <systemitem>~/kernel/rpmbuild/RPMS/x86_64/kernel-2.6.32-358.14.1.el6_lustre.x86_64-1.x86_64.rpm</systemitem>.</para>
376 <title>Configuring and Building Lustre RPMs</title>
377 <para>To configure and build a set of Lustre RPMs, complete these steps.<orderedlist>
379 <para>Configure the Lustre source:</para>
381 <screen># cd ~/lustre-release/
382 # ./configure --with-linux=/home/build/kernel/rpmbuild/BUILD/
383 kernel-2.6.32-358.14.1.el6_lustre.x86_64/</screen>
385 <para>Text similar to the following is displayed:</para>
389 LLCPPFLAGS: -D__arch_lib__ -D_LARGEFILE64_SOURCE=1
390 CFLAGS: -g -O2 -Werror
391 EXTRA_KCFLAGS: -include /home/build/lustre-release/config.h -g
392 -I/home/build/lustre-release
393 /libcfs/include -I/home/build/lustre-release/lnet/include
394 -I/home/build/lustre-release/
396 LLCFLAGS: -g -Wall -fPIC -D_GNU_SOURCE
397 Type 'make' to build Lustre.</screen>
401 <para>Create the RPMs:</para>
403 <screen># make rpms</screen>
405 <para>Text similar to the following is displayed:</para>
409 Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.TsLWpD
411 + cd /home/build/kernel/rpmbuild/BUILD
413 + rm -rf /home/build/kernel/rpmbuild/BUILDROOT/
414 lustre-2.0.61-2.6.32-358.14.1.el6_lustre.x86_64_g0533e7b.x86_64
416 make[1]: Leaving directory `/home/build/lustre-release'</screen>
419 </orderedlist></para>
420 <para>The resulting RPMs are in
421 <systemitem>~build/kernel/rpmbuild/RPMS/x86_64/</systemitem>.<screen>kernel-2.6.3-358.14.1.el6_lustre.x86_64-1.x86_64.rpm
422 lustre-2.0.61-2.6.32-358.14.1.el6_lustre.x86_64_g0533e7b.x86_64.rpm
423 lustre-debuginfo-2.0.61-2.6.32-358.14.1.el6_lustre.x86_64_g0533e7b.x86_64.rpm
424 lustre-ldiskfs-3.3.0-2.6.32-358.14.1.el6_lustre.x86_64_g0533e7b.x86_64.rpm
425 lustre-ldiskfs-debuginfo-3.3.0-2.6.32-358.14.1.el6_lustre.x86_64_g0533e7b.x86_64.rpm
426 lustre-modules-2.0.61-2.6.32-358.14.1.el6_lustre.x86_64_g0533e7b.x86_64.rpm
427 lustre-source-2.0.61-2.6.32-358.14.1.el6_lustre.x86_64_g0533e7b.x86_64.rpm
428 lustre-tests-2.0.61-2.6.32-358.14.1.el6_lustre.x86_64_g0533e7b.x86_64.rpm</screen></para>
430 <section xml:id="InstallingLustreKernel">
432 <primary>installing</primary>
433 <secondary>from source</secondary>
434 <tertiary>installing the Lustre kernel</tertiary>
435 </indexterm>Installing the Lustre Kernel</title>
436 <para>To install the Lustre kernel, complete these steps.<orderedlist>
438 <para>As <systemitem>root</systemitem>, install the kernel:</para>
440 <screen># rpm -ivh ~build/kernel/rpmbuild/RPMS/x86_64/
441 kernel-2.6.32-358.14.1.el6_lustre.x86_64-1.x86_64.rpm</screen>
445 <para>Create <literal>initrd</literal> using <literal>dracut</literal>:</para>
447 <screen># /sbin/new-kernel-pkg --package kernel --mkinitrd --dracut
448 --depmod --install 2.6.32-358.14.1.el6_lustre.x86_64</screen>
452 <para>Reboot the system:</para>
454 <screen>reboot</screen>
456 <para>A login prompt such as that shown below indicates
457 success:<screen>Red Hat Enterprise Linux Server release 6.0(Santiago)
458 Kernel 2.6.3-358.14.1.el6_lustre.x86_64 on an x86_64
463 </orderedlist></para>
467 <primary>installing</primary>
468 <secondary>from source</secondary>
469 <tertiary>building Lustre with OFED</tertiary>
470 </indexterm>Building a Lustre File System with a Third-party Network Stack on OFED
472 <para>When using third-party network hardware, you must follow a specific process to install
473 and recompile the Lustre software. This section provides an installation example that shows
474 how to build Lustre software release 2.4.x when using the OpenFabrics Enterprise
475 Distribution (OFED*) version 3.5-1 open-source software. The same process can be used for
476 other third-party network stacks by replacing the OFED installation procedure and using the
477 appropriate <literal>--with</literal> option when configuring the Lustre source code.<orderedlist>
479 <para>Install the Lustre kernel (see <xref xmlns:xlink="http://www.w3.org/1999/xlink"
480 linkend="InstallingLustreKernel"/>).</para>
483 <para>Compile and install the OFED stack. As root, enter:</para>
484 <screen># cd /usr/src
485 # tar –zxvf OFED-3.5-1.tgz
487 # ./install.pl</screen>
488 <para>A directory <literal>/usr/src/compat-rdma</literal> is created.</para>
491 <para>Configure and build Lustre source code:</para>
492 <orderedlist numeration="loweralpha">
494 <para>Install the Lustre source. This can be done via RPM or git (see <xref
495 xmlns:xlink="http://www.w3.org/1999/xlink" linkend="PreparingLustreSource"
499 <para>Configure and build the Lustre source code</para>
500 <screen>$ cd ~/lustre-release
501 $ sh ./autogen.sh</screen>
505 </orderedlist></para>
506 <para>The <literal>./configure --help</literal> command shows a list of all the
507 <literal>--with</literal> options. All third-party network stacks are built in this
508 manner.<screen>$ ./configure --with-linux=/usr/src/kernels/$(uname –r) --with-o2ib=/usr/src/compat-rdma
510 $ make rpms</screen></para>
511 <para>The <literal>make rpms</literal> command output shows the location of the generated
515 <section xml:id="section_g4d_yby_lk">
517 <primary>installing</primary>
518 <secondary>from source</secondary>
519 <tertiary>testing a Lustre file system</tertiary>
520 </indexterm>Installing and Testing a Lustre File System</title>
521 <para>This section describes how to install the Lustre RPMs and run the Lustre test
524 <title>Installing <package>e2fsprogs</package></title>
525 <para>The <package>e2fsprogs</package> package is required on all Lustre file system server nodes. To download
526 and install <literal>e2fsprogs</literal>, complete these steps.<orderedlist>
528 <para>Download <literal>e2fsprogs</literal> from the <link
529 xmlns:xlink="http://www.w3.org/1999/xlink"
530 xlink:href="https://wiki.hpdd.intel.com/display/PUB/Lustre+Releases">Lustre
531 Releases</link> repository.</para>
534 <para>Install the <literal>e2fsprogs</literal> package:</para>
537 ./e2fsprogs-1.42.7.wc2-7.el6.x86_64.rpm
538 ./e2fsprogs-libs-1.42.7.wc2-7.el6.x86_64.rpm
543 </orderedlist></para>
546 <title>Installing the Lustre RPMs</title>
547 <para>To install the Lustre RPMs, complete these steps as <systemitem>root</systemitem>:<orderedlist>
549 <para>Make the directory containing the Lustre RPMs your current directory:</para>
551 <screen># cd /home/build/kernel/rpmbuild/RPMS/x86_64/
556 <para>Install the RPMs:</para>
558 <screen># rpm -ivh lustre-ldiskfs-3.3.0-2.6.32-358.14.1.el6_lustre.x86_64*
559 # rpm -ivh lustre-modules-2.0.61-2.6.32-358.14.1.el6_lustre.x86_64*
560 # rpm -ivh lustre-2.0.61-2.6.32-358.14.1.el6_lustre.x86_64_*
561 # rpm -ivh lustre-tests-*</screen>
564 </orderedlist></para>
567 <title>Running the Test Suite</title>
568 <para>To test a single node Lustre file system installation, complete these steps.<orderedlist>
570 <para>Mount the Lustre file system.</para>
572 <screen># /usr/lib64/lustre/tests/llmount.sh</screen>
574 <para>Text similar to the following will be
575 displayed:<screen>Loading modules from /usr/lib64/lustre/tests/..
577 subsystem_debug=0xffb7e3ff
578 gss/krb5 is not supported
579 Formatting mgs, mds, osts
580 Format mds1: /tmp/lustre-mdt1
581 Format ost1: /tmp/lustre-ost1
582 Format ost2: /tmp/lustre-ost2
583 Checking servers environments
584 Checking clients rhel6-master environments
585 Loading modules from /usr/lib64/lustre/tests/..
587 subsystem_debug=0xffb7e3ff
588 gss/krb5 is not supported
590 Starting mds1: -o loop,user_xattr,acl
591 /tmp/lustre-mdt1 /mnt/mds1
593 subsystem_debug=0xffb7e3ff
595 Started lustre-MDT0000
596 Starting ost1: -o loop /tmp/lustre-ost1 /mnt/ost1
598 subsystem_debug=0xffb7e3ff
600 Started lustre-OST0000
601 Starting ost2: -o loop /tmp/lustre-ost2 /mnt/ost2
603 subsystem_debug=0xffb7e3ff
605 Started lustre-OST0001
606 Starting client: rhel5-build: -o user_xattr,acl,flock
607 rhel6-master@tcp:/lustre /mnt/lustre
609 subsystem_debug=0xffb7e3ff
612 disable quota as required</screen></para>
613 <para>The Lustre file system is now available at
614 <systemitem>/mnt/lustre</systemitem>.</para>
615 <warning><para>The Lustre file system available at <literal>/mnt/lustre</literal> is a test file system built from small loopback devices in <literal>/tmp</literal> and not for production use.</para></warning>
617 <para>If you see the error below, associate the IP address of a non-loopback interface
618 with the name of your machine in the file
619 <filename>/etc/hosts</filename>.<screen>mkfs.lustre: Can't parse NID 'rhel6-master@tcp'</screen></para>
622 </orderedlist></para>