1 <?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="installinglustrefromsourcecode">
2 <title xml:id="installinglustrefromsourcecode.title">Installing a Lustre* File System from Source
4 <para>This chapter describes how to create a customized Lustre* server kernel from source code.
5 Sections included are:</para>
8 <para><xref linkend="dbdoclet.50438210_69313"/></para>
11 <para><xref linkend="dbdoclet.50438210_65411"/></para>
14 <para><xref linkend="dbdoclet.50438210_47529"/></para>
17 <para><xref linkend="section_g4d_yby_lk"/></para>
21 <para>It is recommended that you install from prebuild RPMs of the Lustre software unless you
22 need to customize the Lustre server kernel or will be using an Linux kernel that has not been
23 tested with the Lustre software. Instructions on installing pre-built RPMs are in <xref linkend='installinglustre.title'/></para>
24 <para>For a list of supported Linux distributions and architectures, see <xref
25 xmlns:xlink="http://www.w3.org/1999/xlink" linkend="LustreTestMatrixTable"/>. Prebuild RPMs
26 are available in the <link xl:href="https://wiki.hpdd.intel.com/display/PUB/Lustre+Releases">Lustre Releases</link>
27 repository. For information about installing Lustre RPMs, see <xref
28 xmlns:xlink="http://www.w3.org/1999/xlink" linkend="installinglustre"/>.</para>
30 <section xml:id="dbdoclet.50438210_69313">
32 <indexterm><primary>installing</primary><secondary>from source</secondary></indexterm>
33 Overview and Prerequisites</title>
34 <para>To install Lustre from source code, the following are required:</para>
37 <para>A x86_64 machine with a fresh installation of a Linux operating system.</para>
38 <note><para>This section of the Manual only describes the specific procedure against building on Red Hat* 6 Linux and derivatives. For a complete set of supported platforms please visit <xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="LustreTestMatrixTable"/>.The server kernel should be one of the supported kernel versions but it is not necessary to run a vendor distribution to build the Lustre software.</para></note>
41 <para>Access to the Lustre software git repository at <link xl:href="http://git.hpdd.intel.com/">http://git.hpdd.intel.com/</link>.</para>
44 <para>(Recommended) Access to a recent version of <link xmlns:xlink="http://www.w3.org/1999/xlink"
45 xlink:href="http://fedoraproject.org/wiki/EPEL">EPEL</link> containing the
46 <literal>quilt</literal> utility used for managing a series of patches. </para>
47 <note><para>The use of <literal>quilt</literal> is optional for building Lustre, and is detected at configure time if it is installed. Otherwise the <literal>patch</literal> utility is used to apply <literal>ldiskfs</literal> patches to the kernel source. Quilt is useful if you intend to modify the Lustre kernel patches.</para></note>
50 <para><emphasis role="italic">(Recommended)</emphasis> At least 1 GB memory on the machine
51 used for the build.</para>
54 <para><emphasis role="italic">(Recommended)</emphasis> At least 20 GB hard disk space on the
55 machine used for the build. </para>
58 <para>Security-Enhanced Linux (SELinux) <emphasis role="italic">disabled</emphasis> on all
59 Lustre servers and clients. The Lustre software does not support SELinux.</para>
62 <para>The installation procedure includes several steps:</para>
65 <para> Patching the core kernel.</para>
68 <para> Building Lustre RPMs.</para>
71 <para> Installing and testing the Lustre file system.</para>
74 <para>These steps are described in the following sections.</para>
76 <section xml:id="dbdoclet.50438210_65411">
77 <title><indexterm><primary>installing</primary><secondary>from source</secondary><tertiary>patching the kernel</tertiary></indexterm>Patching the Kernel</title>
78 <para>This section first describes how to prepare your machine to serve as a development
79 environment, including build tools, the Lustre source, and the Linux kernel source. It then
80 describes how to apply Lustre patches to the Linux kernel.</para>
82 <para>A patched Linux kernel is <emphasis>NOT</emphasis> required in order to build the
83 Lustre client code. The steps below for building the Linux kernel are only
84 required if a new kernel is needed for a Lustre server.</para>
88 <primary>quilt</primary>
89 </indexterm>Provisioning the Build Machine and Installing Dependencies</title>
91 <para>This example is specific for building a modified Red Hat* Enteprise Linux* 6 kernel for the Lustre file system server. It is assumed that the operating system has been
92 freshly installed on a machine with the hostname <literal>rhel6-master</literal>.</para>
94 <para>To provision the build machine and install dependencies, complete these steps.</para>
97 <para>Log in as <systemitem>root</systemitem>.</para>
100 <para>Install the kernel development tools:</para>
102 <screen># yum -y groupinstall "Development Tools"</screen>
105 <para>If the Development Tools group is not available, you will need to satisfy the dependencies for the build manually.</para>
109 <para>Install additional dependencies:</para>
111 <screen># yum -y install xmlto asciidoc elfutils-libelf-devel
112 zlib-devel binutils-devel newt-devel python-devel hmaccalc
113 perl-ExtUtils-Embed</screen>
117 <para>(Optional) Install EPEL:</para>
120 http://download.fedoraproject.org/pub/epel/6/x86_64/
121 epel-release-6-8.noarch.rpm
122 # rpm -ivh ./epel-release-5-4.noarch.rpm
127 <para>(Optional) Install quilt:</para>
129 <screen># yum -y install quilt
134 <package>newt-devel</package> may not be available for RHEL6. One option is to
135 download the <link xmlns:xlink="http://www.w3.org/1999/xlink"
136 xlink:href="http://mirror.centos.org/centos/6/os/x86_64/Packages/newt-devel-0.52.11-3.el6.x86_64.rpm"
137 >newt-devel</link>, <link xmlns:xlink="http://www.w3.org/1999/xlink"
138 xlink:href="http://mirror.centos.org/centos/6/os/x86_64/Packages/slang-devel-2.2.1-1.el6.x86_64.rpm"
139 >slang-devel</link>, and <link xmlns:xlink="http://www.w3.org/1999/xlink"
140 xlink:href="http://mirror.centos.org/centos/6/os/i386/Packages/asciidoc-8.4.5-4.1.el6.noarch.rpm"
141 >asciidoc</link> RPMs from CentOS and install using:</para>
143 <screen>yum --nogpgcheck localinstall
144 ./newt-devel-0.52.11-3.el6.x86_64.rpm
145 ./slang-devel-2.2.1-1.el6.x86_64.rpm
146 ./asciidoc-8.4.5-4.1.el6.noarch.rpm
154 <title>Preparing the Lustre Source</title>
155 <para>To prepare the Lustre source, complete these steps.</para>
158 <para>Create a user called <systemitem>build</systemitem> with a home directory
159 <systemitem>/home/build</systemitem>:</para>
161 <screen># useradd -m build
166 <para>Switch to the user called <systemitem>build</systemitem> and change the directory to the users home directory:</para>
174 <para>Get the MASTER branch of the Lustre software from the Lustre software git repository:</para>
176 <screen># git clone git://git.hpdd.intel.com/fs/lustre-release.git
184 <screen>sh ./autogen.sh
189 <para>Resolve any outstanding dependencies. </para>
190 <para><screen># sh ./autogen.sh</screen>When <package>autogen.sh</package> completes
191 successfully, a response similar to the following is displayed: </para>
193 <screen>Checking for a complete tree...
194 checking forautomake-1.9>= 1.9... found 1.9.6
197 configure.ac:10: installing `./config.sub'
198 configure.ac:12: installing `./install-sh'
199 configure.ac:12: installing `./missing'
200 Running autoconf</screen>
206 <title>Preparing the Kernel Source</title>
207 <para>To build the kernel using <literal>rpmbuild</literal> (a tool specific to RPM-based
208 distributions), complete these steps.</para>
211 <para>Get the kernel source: </para>
214 # mkdir -p kernel/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
216 # echo '%_topdir %(echo $HOME)/kernel/rpmbuild'> ~/.rpmmacros
221 <para>Install the kernel source (enter on one line):</para>
223 <screen># rpm -ivh http://ftp.redhat.com/pub/redhat/linux/enterprise/
224 6Server/en/os/SRPMS/kernel-2.6.32-358.14.1.el6.src.rpm
225 2>&1 | grep -v mockb</screen>
228 <para>It is intended that the Lustre software master branch be kept up-to-date with the
229 most recent kernel distributions. However, a delay may occur between a periodic update
230 to a kernel distribution and a corresponding update of the master branch. The most
231 recent supported kernel in the master branch can be found in the source directory in
232 <systemitem>lustre/kernel_patches/which_patch</systemitem>. If the master branch is
233 not current with the latest distribution, download the most recent kernel RPMs from
234 the vendor's download site.</para>
238 <para>Prepare the source using <literal>rpmbuild</literal>:</para>
240 <screen># cd ~/kernel/rpmbuild
241 # rpmbuild -bp --target=`uname -m` ./SPECS/kernel.spec
244 <para>The text displayed ends with the following:</para>
247 gpg: Total number processed: 1
249 + gpg --homedir . --export --keyring ./kernel.pub Red
250 gpg: WARNING: unsafe permissions on homedir `.'
251 + gcc -o scripts/bin2c scripts/bin2c.c
252 + scripts/bin2c ksign_def_public_key __initdata
258 <para>The kernel source with the Red Hat Enterprise Linux patches applied is now residing in
260 <systemitem>/home/build/kernel/rpmbuild/BUILD/kernel-2.6.32-358.14.1.el6.x86_64/</systemitem></para>
263 <title>Patching the Kernel Source with the Lustre Code</title>
264 <para>To patch the kernel source with the Lustre code, complete these steps.</para>
267 <para>Go to the directory containing the kernel source:</para>
269 <screen>#cd ~/kernel/rpmbuild/BUILD/kernel-2.6.32-358.14.1.el6/
270 linux-2.6.32-358.14.1.el6.x86_64/</screen>
274 <para>Edit the <filename>Makefile</filename> file in this directory to add a unique build
275 id to be able to ascertain that the kernel is booted. Modify line 4 as shown
278 <screen>EXTRAVERSION = -358.14.1.el6_lustre.x86_64</screen>
282 <para>Overwrite the <filename>.config</filename> file in this directory with the Lustre
283 <filename>.config</filename> file:</para>
285 <screen># cp ~/lustre-release/lustre/kernel_patches/kernel_configs/
286 kernel-2.6.32-2.6-rhel6-x86_64.config ./.config</screen>
290 <para>Link the Lustre series and patches:</para>
292 <screen># ln -s ~/lustre-release/lustre/kernel_patches/series/
293 2.6-rhel6.series series
294 # ln -s ~/lustre-release/lustre/kernel_patches/patches patches</screen>
298 <para>Apply the patches to the kernel source using <package>quilt</package>:</para>
300 <screen># quilt push -av</screen>
302 <para>The following is displayed:</para>
305 patching file fs/jbd2/transaction.c
306 Hunk #3succeeded at 1222(offset 3lines).
307 Hunk #4succeeded at 1357(offset 3lines).
308 Now at patch patches/jbd2-jcberr-2.6-rhel6.patch</screen>
309 <note><para>If quilt is not installed, the following steps would be equivalent:
310 <screen>for PATCH in $(cat series); do patch -p1 < patches/$PATCH; done</screen></para></note>
315 <section xml:id="dbdoclet.50438210_47529">
318 <primary>installing</primary>
319 <secondary>from source</secondary>
320 <tertiary>packaging</tertiary>
321 </indexterm> Building the Lustre RPMs</title>
322 <para>This section describes how to configure the patched kernel to work with the Lustre
323 software and how to create and install the Lustre packages (RPMs).</para>
325 <title>Building a New Kernel</title>
326 <para>To build a new kernel as an RPM, complete these steps.</para>
329 <para>In the kernel source directory, build a kernel rpm:</para>
330 <screen># cd ~/kernel/rpmbuild/BUILD/kernel-2.6.32-131.2.1.el6/
331 linux-2.6.32-131.2.1.el6.x86_64/
332 # make oldconfig || make menuconfig
334 # make include/linux/version.h
335 # make SUBDIRS=scripts
336 # make include/linux/utsrelease.h
338 <para>A successful build returns text similar to the following:</para>
341 Wrote: /home/build/kernel/rpmbuild/SRPMS/
342 kernel-2.6.32-358.14.1.el6_lustre.x86_64-1.src.rpm
343 Wrote: /home/build/kernel/rpmbuild/RPMS/x86_64/
344 kernel-2.6.32-358.14.1.el6_lustre.x86_64-1.x86_64.rpm
345 Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.f73m1V
346 + umask 022+ cd /home/build/kernel/rpmbuild/BUILD
347 + cd kernel-2.6.32-358.14.1.el6_lustre.x86_64
348 + rm -rf /home/build/kernel/rpmbuild/BUILDROOT/
349 kernel-2.6.32-358.14.1.el6_lustre.x86_64-1.x86_64
351 rm ../kernel-2.6.3-358.14.1.el6_lustre.x86_64.tar.gz</screen>
353 <para>If a request to generate more entropy appears, some disk or keyboard I/O needs to
354 be generated. You can generate entropy by entering the following on another
355 terminal:<screen># grep -Ri 'any_text' /usr</screen></para>
359 <para>A fresh kernel RPM can now be found at
360 <systemitem>~/kernel/rpmbuild/RPMS/x86_64/kernel-2.6.32-358.14.1.el6_lustre.x86_64-1.x86_64.rpm</systemitem>.</para>
363 <title>Configuring and Building Lustre RPMs</title>
364 <para>To configure and build a set of Lustre RPMs, complete these steps.<orderedlist>
366 <para>Configure the Lustre source:</para>
368 <screen># cd ~/lustre-release/
369 # ./configure --with-linux=/home/build/kernel/rpmbuild/BUILD/
370 kernel-2.6.32-358.14.1.el6_lustre.x86_64/</screen>
372 <para>Text similar to the following is displayed:</para>
376 LLCPPFLAGS: -D__arch_lib__ -D_LARGEFILE64_SOURCE=1
377 CFLAGS: -g -O2 -Werror
378 EXTRA_KCFLAGS: -include /home/build/lustre-release/config.h -g
379 -I/home/build/lustre-release
380 /libcfs/include -I/home/build/lustre-release/lnet/include
381 -I/home/build/lustre-release/
383 LLCFLAGS: -g -Wall -fPIC -D_GNU_SOURCE
384 Type 'make' to build Lustre.</screen>
388 <para>Create the RPMs:</para>
390 <screen># make rpms</screen>
392 <para>Text similar to the following is displayed:</para>
396 Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.TsLWpD
398 + cd /home/build/kernel/rpmbuild/BUILD
400 + rm -rf /home/build/kernel/rpmbuild/BUILDROOT/
401 lustre-2.0.61-2.6.32-358.14.1.el6_lustre.x86_64_g0533e7b.x86_64
403 make[1]: Leaving directory `/home/build/lustre-release'</screen>
406 </orderedlist></para>
407 <para>The resulting RPMs are in
408 <systemitem>~build/kernel/rpmbuild/RPMS/x86_64/</systemitem>.<screen>kernel-2.6.3-358.14.1.el6_lustre.x86_64-1.x86_64.rpm
409 lustre-2.0.61-2.6.32-358.14.1.el6_lustre.x86_64_g0533e7b.x86_64.rpm
410 lustre-debuginfo-2.0.61-2.6.32-358.14.1.el6_lustre.x86_64_g0533e7b.x86_64.rpm
411 lustre-ldiskfs-3.3.0-2.6.32-358.14.1.el6_lustre.x86_64_g0533e7b.x86_64.rpm
412 lustre-ldiskfs-debuginfo-3.3.0-2.6.32-358.14.1.el6_lustre.x86_64_g0533e7b.x86_64.rpm
413 lustre-modules-2.0.61-2.6.32-358.14.1.el6_lustre.x86_64_g0533e7b.x86_64.rpm
414 lustre-source-2.0.61-2.6.32-358.14.1.el6_lustre.x86_64_g0533e7b.x86_64.rpm
415 lustre-tests-2.0.61-2.6.32-358.14.1.el6_lustre.x86_64_g0533e7b.x86_64.rpm</screen></para>
418 <title>Installing the Lustre Kernel</title>
419 <para>To install the Lustre kernel, complete these steps.<orderedlist>
421 <para>As <systemitem>root</systemitem>, install the kernel:</para>
423 <screen># rpm -ivh ~build/kernel/rpmbuild/RPMS/x86_64/
424 kernel-2.6.32-358.14.1.el6_lustre.x86_64-1.x86_64.rpm</screen>
428 <para>Create <literal>initrd</literal> using <literal>dracut</literal>:</para>
430 <screen># /sbin/new-kernel-pkg --package kernel --mkinitrd --dracut
431 --depmod --install 2.6.32-358.14.1.el6_lustre.x86_64</screen>
435 <para>Reboot the system:</para>
437 <screen>reboot</screen>
439 <para>A login prompt such as that shown below indicates
440 success:<screen>Red Hat Enterprise Linux Server release 6.0(Santiago)
441 Kernel 2.6.3-358.14.1.el6_lustre.x86_64 on an x86_64
446 </orderedlist></para>
449 <section xml:id="section_g4d_yby_lk">
450 <title>Installing and Testing a Lustre File System</title>
451 <para>This section describes how to install the Lustre RPMs and run the Lustre test
454 <title>Installing <package>e2fsprogs</package></title>
455 <para>The <package>e2fsprogs</package> package is required on all Lustre file system server nodes. To download
456 and install <literal>e2fsprogs</literal>, complete these steps.<orderedlist>
458 <para>Download <literal>e2fsprogs</literal> from the <link
459 xmlns:xlink="http://www.w3.org/1999/xlink"
460 xlink:href="https://wiki.hpdd.intel.com/display/PUB/Lustre+Releases">Lustre
461 Releases</link> repository.</para>
464 <para>Install the <literal>e2fsprogs</literal> package:</para>
467 ./e2fsprogs-1.42.7.wc2-7.el6.x86_64.rpm
468 ./e2fsprogs-libs-1.42.7.wc2-7.el6.x86_64.rpm
473 </orderedlist></para>
476 <title>Installing the Lustre RPMs</title>
477 <para>To install the Lustre RPMs, complete these steps as <systemitem>root</systemitem>:<orderedlist>
479 <para>Make the directory containing the Lustre RPMs your current directory:</para>
481 <screen># cd /home/build/kernel/rpmbuild/RPMS/x86_64/
486 <para>Install the RPMs:</para>
488 <screen># rpm -ivh lustre-ldiskfs-3.3.0-2.6.32-358.14.1.el6_lustre.x86_64*
489 # rpm -ivh lustre-modules-2.0.61-2.6.32-358.14.1.el6_lustre.x86_64*
490 # rpm -ivh lustre-2.0.61-2.6.32-358.14.1.el6_lustre.x86_64_*
491 # rpm -ivh lustre-tests-*</screen>
494 </orderedlist></para>
497 <title>Running the Test Suite</title>
498 <para>To test a single node Lustre file system installation, complete these steps.<orderedlist>
500 <para>Mount the Lustre file system.</para>
502 <screen># /usr/lib64/lustre/tests/llmount.sh</screen>
504 <para>Text similar to the following will be
505 displayed:<screen>Loading modules from /usr/lib64/lustre/tests/..
507 subsystem_debug=0xffb7e3ff
508 gss/krb5 is not supported
509 Formatting mgs, mds, osts
510 Format mds1: /tmp/lustre-mdt1
511 Format ost1: /tmp/lustre-ost1
512 Format ost2: /tmp/lustre-ost2
513 Checking servers environments
514 Checking clients rhel6-master environments
515 Loading modules from /usr/lib64/lustre/tests/..
517 subsystem_debug=0xffb7e3ff
518 gss/krb5 is not supported
520 Starting mds1: -o loop,user_xattr,acl
521 /tmp/lustre-mdt1 /mnt/mds1
523 subsystem_debug=0xffb7e3ff
525 Started lustre-MDT0000
526 Starting ost1: -o loop /tmp/lustre-ost1 /mnt/ost1
528 subsystem_debug=0xffb7e3ff
530 Started lustre-OST0000
531 Starting ost2: -o loop /tmp/lustre-ost2 /mnt/ost2
533 subsystem_debug=0xffb7e3ff
535 Started lustre-OST0001
536 Starting client: rhel5-build: -o user_xattr,acl,flock
537 rhel6-master@tcp:/lustre /mnt/lustre
539 subsystem_debug=0xffb7e3ff
542 disable quota as required</screen></para>
543 <para>The Lustre file system is now available at
544 <systemitem>/mnt/lustre</systemitem>.</para>
545 <warning><para>The Lustre file system available at <literal>/mnt/lustre</literal> is a test file system built from small loopback devices in <literal>/tmp</literal> and not for production use.</para></warning>
547 <para>If you see the error below, associate the IP address of a non-loopback interface
548 with the name of your machine in the file
549 <filename>/etc/hosts</filename>.<screen>mkfs.lustre: Can't parse NID 'rhel6-master@tcp'</screen></para>
552 </orderedlist></para>