X-Git-Url: https://git.whamcloud.com/?a=blobdiff_plain;f=InstallingLustre.xml;h=7a7a63bb89f13f4c4e732065cc005df33853ca71;hb=5ea7ffbf061474770d38a85f7a85791c884309af;hp=a0294be3dab1a843203f81ddcb0dc8dc984d2fb7;hpb=a91a21ad3f10c834eb788ced7724cf4986949fa3;p=doc%2Fmanual.git diff --git a/InstallingLustre.xml b/InstallingLustre.xml index a0294be..7a7a63b 100644 --- a/InstallingLustre.xml +++ b/InstallingLustre.xml @@ -1,335 +1,490 @@ - - - - Installing the Lustre Software - - - This chapter describes how to install the Lustre software. It includes: - + + + Installing the Lustre Software + This chapter describes how to install the Lustre software from RPM + packages. It includes: - + + + + + + + + + + + + For hardware and system requirements and hardware configuration + information, see + . +
+ + <indexterm> + <primary>installing</primary> + <secondary>preparation</secondary> + </indexterm>Preparing to Install the Lustre Software + You can install the Lustre software from downloaded packages (RPMs) + or directly from the source code. This chapter describes how to install the + Lustre RPM packages. Instructions to install from source code are beyond + the scope of this document, and can be found elsewhere online. + The Lustre RPM packages are tested on current versions of Linux + enterprise distributions at the time they are created. See the release + notes for each version for specific details. +
+ Software Requirements + To install the Lustre software from RPMs, the following are + required: + + + + + Lustre server packages + . The required packages for Lustre 2.9 EL7 servers are + listed in the table below, where + ver refers to the Lustre release and + kernel version (e.g., 2.9.0-1.el7) and + arch refers to the processor architecture + (e.g., x86_64). These packages are available in the + + Lustre Releases repository, and may differ depending on + your distro and version. - + + Packages Installed on Lustre Servers + + + + + + Package Name + Description + + + + + + kernel-ver_lustre.arch + + Linux kernel with Lustre software patches (often + referred to as "patched kernel") + + + + lustre-ver.arch + + Lustre software command line tools + + + + kmod-lustre-ver.arch + + Lustre-patched kernel modules + + + + kmod-lustre-osd-ldiskfs-ver.arch + + Lustre back-end file system tools for ldiskfs-based + servers. + + + + + lustre-osd-ldiskfs-mount-ver.arch + + Helper library for mount.lustre + and mkfs.lustre for ldiskfs-based servers. + + + + + kmod-lustre-osd-zfs-ver.arch + + Lustre back-end file system tools for ZFS. This is + an alternative to + lustre-osd-ldiskfs (kmod-spl and + kmod-zfs available separately). + + + + lustre-osd-zfs-mount-ver.arch + + Helper library for mount.lustre + and mkfs.lustre for ZFS-based servers + (zfs utilities available separately). + + + + + e2fsprogs + + Utilities to maintain Lustre ldiskfs back-end file + system(s) + + + + + lustre-tests-ver_lustre.arch + + Lustre I/O Kit benchmarking tools + (Included in Lustre software as of + release 2.2) + + + +
- - + + + + + Lustre client packages + . The required packages for Lustre 2.9 EL7 clients are + listed in the table below, where + ver refers to the Linux distribution (e.g., + 3.6.18-348.1.1.el5). These packages are available in the + + Lustre Releases repository. - + + Packages Installed on Lustre Clients + + + + + + Package Name + Description + + + + + + kmod-lustre-client-ver.arch + + Patchless kernel modules for client + + + + lustre-client-ver.arch + + Client command line tools + + + + lustre-client-dkms-ver.arch + + Alternate client RPM to kmod-lustre-client with + Dynamic Kernel Module Support (DKMS) installation. This + avoids the need to install a new RPM for each kernel update, + but requires a full build environment on the client. + + + +
- - - - - - - For hardware/system requirements, see . -
- 8.1 Preparing to Install the Lustre Software - If you are using a supported Linux distribution and architecture, you can install Lustre from downloaded packages (RPMs). For a list of supported configurations, see the Lustre 2.0 topic on the Lustre wiki. - A list of supported Lustre LNET drivers can be found in . - If you are not using a supported configuration, you can install Lustre directly from the source code. For more information on this installation method, see . -
- <anchor xml:id="dbdoclet.50438261_pgfId-1292590" xreflabel=""/>8.1.1 <anchor xml:id="dbdoclet.50438261_37079" xreflabel=""/>Required Software - To install Lustre, the following are required: - - (On Linux servers only) Linux kernel patched with Lustre-specific patches for your platform and architecture. A Linux patched kernel can be installed on a client if it is desireable to use the same kernel on all nodes, but this is not required. + + The version of the kernel running on a Lustre client must be + the same as the version of the + kmod-lustre-client-ver + package being installed, unless the DKMS package is installed. + If the kernel running on the client is not compatible, a kernel + that is compatible must be installed on the client before the + Lustre file system software is used. + + + + + + Lustre LNet network driver (LND) + . The Lustre LNDs provided with the Lustre software are + listed in the table below. For more information about Lustre LNet, + see + . + + Network Types Supported by Lustre LNDs + + + + + + Supported Network Types + Notes + + + + + TCP + Any network carrying TCP traffic, including GigE, + 10GigE, and IPoIB + + + InfiniBand network + OpenFabrics OFED (o2ib) + + + gni + Gemini (Cray) + + + +
+ + + + + The InfiniBand and TCP Lustre LNDs are routinely tested during + release cycles. The other LNDs are maintained by their respective + owners + + + + + + High availability software + . If needed, install third party high-availability + software. For more information, see + . - - Lustre modules compiled for the Linux kernel (see for which packages are required for servers and clients in your configuration). + + + + Optional packages. + Optional packages provided in the + + Lustre Releases repository may include the following + (depending on the operating system and platform): + + + + kernel-debuginfo, + kernel-debuginfo-common, + lustre-debuginfo, + lustre-osd-ldiskfs-debuginfo- Versions of required + packages with debugging symbols and other debugging options + enabled for use in troubleshooting. + + + + kernel-devel, - Portions of the kernel tree needed + to compile third party modules, such as network drivers. + + + + kernel-firmware- Standard Red Hat Enterprise Linux + distribution that has been recompiled to work with the Lustre + kernel. + + + + kernel-headers- Header files installed under + /user/include and used when compiling user-space, + kernel-related code. + + + + lustre-source- Lustre software source code. + + + + (Recommended) + perf, + perf-debuginfo, + python-perf, + python-perf-debuginfo- Linux performance analysis + tools that have been compiled to match the Lustre kernel + version. + + - - Lustre utilities required for configuring Lustre (see for which packages are required for servers and clients in your configuration). + + +
+
+ Environmental Requirements + Before installing the Lustre software, make sure the following + environmental requirements are met. + + + + (Required) + + Use the same user IDs (UID) and group IDs + (GID) on all clients. + If use of supplemental groups is required, see + for information about + supplementary user and group cache upcall (identity_upcall). + + + + (Recommended) + + Provide remote shell access to + clients. + It is recommended that all cluster nodes have remote shell + client access to facilitate the use of Lustre configuration and + monitoring scripts. Parallel Distributed SHell (pdsh) is preferable, + although Secure SHell (SSH) is acceptable. + + + + (Recommended) + + Ensure client clocks are + synchronized. + The Lustre file system uses client clocks for timestamps. + If clocks are out of sync between clients, files will appear with + different time stamps when accessed by different clients. Drifting + clocks can also cause problems by, for example, making it difficult + to debug multi-node issues or correlate logs, which depend on + timestamps. We recommend that you use Network Time Protocol (NTP) to + keep client and server clocks in sync with each other. For more + information about NTP, see: + http://www.ntp.org. + + + + (Recommended) + + Make sure security extensions + + (such as the Novell AppArmor *security + system) and + + network packet filtering tools + + (such as iptables) do not interfere with the Lustre software. + + +
+
+
+ Lustre Software Installation Procedure + + Before installing the Lustre software, back up ALL data. The Lustre + software contains kernel modifications that interact with storage devices + and may introduce security issues and data loss if not installed, + configured, or administered properly. + + To install the Lustre software from RPMs, complete the steps + below. + + + Verify that all Lustre installation requirements have been + met. + + + For hardware requirements, see + . - - (On Linux servers only)e2fsprogs package containing Lustre-specific tools (e2fsck and lfsck) used to repair a backing file system. This replaces the existing e2fsprogs package but provides complete e2fsprogs functionality + + For software and environmental requirements, see the section + above. - - (Optional) Network-specific kernel modules and libraries such as the Lustre-specific OFED package required for an InfiniBand network interconnect. + + + + Download the + e2fsprogs RPMs for your platform from the + + Lustre Releases repository. + + + Download the Lustre server RPMs for your platform from the + + Lustre Releases repository. See + for a list of required packages. + + + Install the Lustre server and + e2fsprogs packages on all Lustre servers (MGS, MDSs, + and OSSs). + + + Log onto a Lustre server as the + root user - -At least one Lustre RPM must be installed on each server and on each client in a Lustre file system. lists required Lustre packages and indicates where they are to be installed. Some Lustre packages are installed on Lustre servers (MGS, MDS, and OSSs), some are installed on Lustre clients, and some are installed on all Lustre nodes. - . - - - Lustre required packages, descriptions and installation guidance - - - - - - - - - Lustre Package - Description - Install on servers - Installing a patched kernel on a client node is not required. However, if a client node will be used as both a client and a server, or if you want to install the same kernel on all nodes for any reason, install the server packages designated with an asterisk (*) on the client node. - Install on clients - - - - - Lustre patched kernel RPMs: -   -   - - -   - kernel-<ver>_lustre.<ver> - For OEL 5 or RHEL 5 server platform - X* -   - - -   - kernel-ib-<ver> - Lustre OFED package. Required only if the network interconnect is InfiniBand. - X* - X - - - Lustre module RPMs: -   -   -   - - -   - lustre-modules-<ver> - For Lustre-patched kernel. - X* -   - - -   - lustre-client-modules-<ver> - For clients. -   - X - - - Lustre utilities: -   -   -   - - -   - lustre-<ver> - Lustre utilities package. This includes userspace utilities to configure and run Lustre. -   X* -    - - -   - lustre-client-<ver> - Lustre utilities for clients. -   - X - - -   - lustre-ldiskfs-<ver> - Lustre-patched backing file system kernel module package for the ldiskfs file system. -  X -   - - -   - e2fsprogs-<ver> - Utilities package used to maintain the ldiskfs backing file system. -  X -   - - - -
- In all supported Lustre installations, a patched kernel must be run on each server, including the the MGS, the MDS, and all OSSs. Running a patched kernel on a Lustre client is only required if the client will be used for multiple purposes, such as running as both a client and an OST or if you want to use the same kernel on all nodes. - Lustre RPM packages are available on the Lustre download site. They must be installed in the order described in the Lustre Installation Procedure. -
- <anchor xml:id="dbdoclet.50438261_pgfId-1292891" xreflabel=""/>8.1.1.1 Network-specific Kernel Modules and Libraries - Network-specific kernel modules and libraries may be needed such as the Lustre-specific OFED package required for an InfiniBand network interconnect. -
-
- <anchor xml:id="dbdoclet.50438261_pgfId-1293747" xreflabel=""/>8.1.1.2 Lustre-Specific Tools <anchor xml:id="dbdoclet.50438261_marker-1292890" xreflabel=""/>and Utilities - Several third-party utilities are must be installed on servers: - - e2fsprogs : Lustre requires a recent version of e2fsprogs that understands extents. Use e2fsprogs-1.41-10-sun2 or later, available at: - http://downloads.lustre.org/public/tools/e2fsprogs/ - A quilt patchset of all changes to the vanilla e2fsprogs is available in e2fsprogs-{version}-patches.tgz. - The Lustre-patched e2fsprogs utility is only required on machines that mount backend (ldiskfs) file systems, such as the OSS, MDS and MGS nodes. It does not need to be loaded on clients. - - - - Perl - Various userspace utilities are written in Perl. Any recent version of Perl will work with Lustre. - - - -
-
- <anchor xml:id="dbdoclet.50438261_pgfId-1292903" xreflabel=""/>8.1.1.3 <anchor xml:id="dbdoclet.50438261_29838" xreflabel=""/>(Optional) <anchor xml:id="dbdoclet.50438261_marker-1292902" xreflabel=""/>High-Availability Software - If you plan to enable failover server functionality with Lustre (either on an OSS or the MDS), you must add high-availability (HA) software to your cluster software. For more information, see Setting up High-Availability (HA) Software with Lustre. -
-
- <anchor xml:id="dbdoclet.50438261_pgfId-1292911" xreflabel=""/>8.1.1.4 (Optional) Debugging Tools and Other Optional Packages - A variety of optional packages are provided on the Whamcloud download site. These include debugging tools, test programs and scripts, Linux kernel and Lustre source code, and other packages. - For more information about debugging tools, see the topic Debugging Lustre on the Lustre wiki. -
-
-
- <anchor xml:id="dbdoclet.50438261_pgfId-1292915" xreflabel=""/>8.1.2 <anchor xml:id="dbdoclet.50438261_19503" xreflabel=""/>Environmental <anchor xml:id="dbdoclet.50438261_marker-1292914" xreflabel=""/>Requirements - Make sure the following environmental requirements are met before installing Lustre: - - (Required)Disable Security-Enhanced Linux (SELinux) on servers and clients . Lustre does not support SELinux. Therefore, disable the SELinux system extension on all Lustre nodes and make sure other security extensions, like Novell AppArmorand network packet filtering tools (such as iptables) do not interfere with Lustre. See Step 3 in the Lustre Installation Procedure below. + Use the + yum command to install the packages: + + +# yum --nogpgcheck install pkg1.rpm pkg2.rpm ... + + - - (Required)Maintain uniform user and group databases on all cluster nodes . Use the same user IDs (UID) and group IDs (GID) on all clients. If use of supplemental groups is required, verify that the group_upcall requirements have been met. See User/Group Cache Upcall. + + Verify the packages are installed correctly: + + +rpm -qa|egrep "lustre|wc"|sort + + - - (Recommended)Provide remote shell access to clients . Although not strictly required to run Lustre, we recommend that all cluster nodes have remote shell client access to facilitate the use of Lustre configuration and monitoring scripts. Parallel Distributed SHell (pdsh) is preferable, although Secure SHell (SSH) is acceptable. + + Reboot the server. - - (Recommended)Ensure client clocks are synchronized . Lustre uses client clocks for timestamps. If clocks are out of sync between clients, files will appear with different time stamps when accessed by different clients. Drifting clocks can also cause problems, for example, by making it difficult to debug multi-node issues or correlate logs, which depend on timestamps. We recommend that you use Network Time Protocol (NTP) to keep client and server clocks in sync with each other. For more information about NTP, see: http://www.ntp.org. + + Repeat these steps on each Lustre server. - -
-
-
- 8.2 Lustre Installation Procedure - - -Before installing Lustre, back up ALL data. Lustre contains kernel modifications which interact with storage devices and may introduce security issues and data loss if not installed, configured or administered properly. - - - Use this procedure to install Lustre from RPMs. - - - Verify that all Lustre installation requirements have been met. - For more information on these prerequisites, see: - - Hardware requirements in - - - Software and environmental requirements in - - - - - Download the Lustre RPMs. - - - On the Lustre download site, select your platform. - The files required to install Lustre (kernels, modules and utilities RPMs) are listed for the selected platform. - - - Download the required files. - - - - - Install the Lustre packages on the servers. - - - Refer to to determine which Lustre packages are to be installed on servers for your platform and architecture. - Some Lustre packages are installed on the Lustre servers (MGS, MDS, and OSSs). Others are installed on Lustre clients. - Lustre packages must be installed in the order specified in the following steps. - - - Install the kernel, modules and ldiskfs packages. - Use the rpm -ivh command to install the kernel, module and ldiskfs packages. - - - It is not recommended that you use the rpm -Uvh command to install a kernel, because this may leave you with an unbootable system if the new kernel doesn't work for some reason. - - - For example, the command in the following example would install required packages on a server with Infiniband networking - $ rpm -ivh kernel-<ver>_lustre-<ver> kernel-ib-<ver> \ -lustre-modules-<ver> lustre-ldiskfs-<ver> - - - - Update the bootloader (grub.conf or lilo.conf) configuration file as needed. - - - Verify that the bootloader configuration file has been updated with an entry for the patched kernel. - Before you can boot to a new distribution or kernel, there must be an entry for it in the bootloader configuration file. Often this is added automatically when the kernel RPM is installed. - - - Disable security-enhanced (SE) Linux on servers and clients by including an entry in the bootloader configuration file as shown below: - selinux=0 - - - - - - Install the utilities/userspace packages. - Use the rpm -ivh command to install the utilities packages. For example: - $ rpm -ivh lustre-<ver> - - - - Install the e2fsprogs package. - Use the rpm -ivh command to install the e2fsprogs package. For example: - $ rpm -ivh e2fsprogs-<ver> - - - If e2fsprogs is already installed on your Linux system, install the Lustre-specific e2fsprogs version by using rpm -Uvh to upgrade the existing e2fsprogs package. For example: - $ rpm -Uvh e2fsprogs-<ver> - - - - - - Note -The rpm command options --force or --nodeps should not be used to install or update the Lustre-specific e2fsprogs package. If errors are reported, file a bug (for instructions see the topic Reporting Bugs on the Lustre wiki). - - - - - - - (Optional) To add optional packages to your Lustre file system, install them now. - Optional packages include file system creation and repair tools, debugging tools, test programs and scripts, Linux kernel and Lustre source code, and other packages. A complete list of optional packages for your platform is provided on the Whamcloud wiki. - - - - - Install the Lustre packages on the clients. - Refer to to determine which Lustre packages are to be installed on clients for your platform and architecture. - Install the module packages for clients. - Use the rpm -ivh command to install the lustre-client and lustre-client-modules-<ver> packages. For example: - $ rpm -ivh lustre-client-modules-<ver> kernel-ib-<ver> - - Install the utilities/userspace packages for clients. - Use the rpm -ivh command to install the utilities packages. For example: - $ rpm -ivh lustre-client + + + + Download the Lustre client RPMs for your platform from the + + Lustre Releases repository. See + for a list of required packages. + + + Install the Lustre client packages on all Lustre clients. + + The version of the kernel running on a Lustre client must be + the same as the version of the + lustre-client-modules- + ver package being installed. If not, a + compatible kernel must be installed on the client before the Lustre + client packages are installed. + + + + Log onto a Lustre client as the root user. + + + Use the + yum command to install the packages: + + +# yum --nogpgcheck install pkg1.rpm pkg2.rpm ... - Update the bootloader (grub.conf or lilo.conf) configuration file as needed. - - - Verify that the bootloader configuration file has been updated with an entry for the patched kernel. - Before you can boot to a new distribution or kernel, there must be an entry for it in the bootloader configuration file. Often this is added automatically when the kernel RPM is installed. - - - Disable security-enhanced (SE) Linux on servers and clients by including an entry in the bootloader configuration file as shown below: - selinux=0 + + + + Verify the packages were installed correctly: + + +# rpm -qa|egrep "lustre|kernel"|sort - - - - - Reboot the patched clients and the servers. - - - If you applied the patched kernel to any clients, reboot them. - Unpatched clients do not need to be rebooted. - - - Reboot the servers. - - - To configure LNET, go next to . If default settings will be used for LNET, go to . - - -
+ + + + Reboot the client. + + + Repeat these steps on each Lustre client. + + + + + To configure LNet, go to + . If default settings will be used for + LNet, go to + . +