+ information, in case SELinux is enabled locally, the
+ <literal>send_sepol</literal> ptlrpc kernel module's parameter has to be
+ set to a non-zero value. <literal>send_sepol</literal> accepts various
+ values:</para>
+ <itemizedlist>
+ <listitem>
+ <para>0: do not send SELinux policy info;</para>
+ </listitem>
+ <listitem>
+ <para>-1: fetch SELinux policy info for every request;</para>
+ </listitem>
+ <listitem>
+ <para>N > 0: only fetch SELinux policy info every N seconds. Use
+ <literal>N = 2^31-1</literal> to have SELinux policy info
+ fetched only at mount time.</para>
+ </listitem>
+ </itemizedlist>
+ <para>Clients that are part of a nodemap on which
+ <literal>sepol</literal> is defined must send SELinux status info.
+ And the SELinux policy they enforce must match the representation
+ stored into the nodemap. Otherwise they will be denied access to the
+ Lustre file system.</para>
+ </section>
+ </section>
+ <section xml:id="managingSecurity.clientencryption" condition='l2E'>
+ <title><indexterm><primary>Client-side encryption</primary></indexterm>
+ Encrypting files and directories</title>
+ <para>The purpose that client-side encryption wants to serve is to be able
+ to provide a special directory for each user, to safely store sensitive
+ files. The goals are to protect data in transit between clients and
+ servers, and protect data at rest.</para>
+ <para>This feature is implemented directly at the Lustre client level.
+ Lustre client-side encryption relies on kernel <literal>fscrypt</literal>.
+ <literal>fscrypt</literal> is a library which filesystems can hook into to
+ support transparent encryption of files and directories. As a consequence,
+ the key points described below are extracted from
+ <literal>fscrypt</literal> documentation.</para>
+ <para>For full details, please refer to documentation available with the
+ Lustre sources, under the
+ <literal>Documentation/client_side_encryption</literal> directory.
+ </para>
+ <note><para>The client-side encryption feature is available natively on
+ Lustre clients running a Linux distribution with at least kernel 5.4. It
+ is also available thanks to an additional kernel library provided by
+ Lustre, on clients that run a Linux distribution with basic support for
+ encryption, including:</para>
+ <itemizedlist>
+ <listitem><para>CentOS/RHEL 8.1 and later;</para></listitem>
+ <listitem><para>Ubuntu 18.04 and later;</para></listitem>
+ <listitem><para>SLES 15 SP2 and later.</para></listitem>
+ </itemizedlist>
+ </note>
+ <section xml:id="managingSecurity.clientencryption.semantics" remap="h3">
+ <title><indexterm><primary>encryption access semantics</primary>
+ </indexterm>Client-side encryption access semantics</title>
+ <para>Only Lustre clients need access to encryption master keys. Keys are
+ added to the filesystem-level encryption keyring on the Lustre client.
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">With the key</emphasis></para>
+ <para>With the encryption key, encrypted regular files, directories,
+ and symlinks behave very similarly to their unencrypted
+ counterparts --- after all, the encryption is intended to be
+ transparent. However, astute users may notice some differences in
+ behavior:</para>
+ <itemizedlist>
+ <listitem>
+ <para>Unencrypted files, or files encrypted with a different
+ encryption policy (i.e. different key, modes, or flags),
+ cannot be renamed or linked into an encrypted directory.
+ However, encrypted files can be renamed within an encrypted
+ directory, or into an unencrypted directory.</para>
+ <note><para>"moving" an unencrypted file into an encrypted
+ directory, e.g. with the <literal>mv</literal> program, is
+ implemented in userspace by a copy followed by a delete. Be
+ aware the original unencrypted data may remain recoverable
+ from free space on the disk; it is best to keep all files
+ encrypted from the very beginning.</para></note>
+ </listitem>
+ <listitem><para>On Lustre, Direct I/O is supported for encrypted
+ files.</para>
+ </listitem>
+ <listitem><para>The <literal>fallocate()</literal> operations
+ <literal>FALLOC_FL_COLLAPSE_RANGE</literal>,
+ <literal>FALLOC_FL_INSERT_RANGE</literal>, and
+ <literal>FALLOC_FL_ZERO_RANGE</literal> are not
+ supported on encrypted files and will fail with
+ <literal>EOPNOTSUPP</literal>.
+ </para>
+ </listitem>
+ <listitem><para>DAX (Direct Access) is not supported on encrypted
+ files.</para>
+ </listitem>
+ <listitem><para condition='l2F'>The st_size of an encrypted
+ symlink will not necessarily give the length of the symlink
+ target as required by POSIX. It will actually give the length of
+ the ciphertext, which will be slightly longer than the plaintext
+ due to NUL-padding and an extra 2-byte overhead.</para>
+ </listitem>
+ <listitem><para condition='l2F'>The maximum length of an encrypted
+ symlink is 2 bytes shorter than the maximum length of an
+ unencrypted symlink.</para>
+ </listitem>
+ <listitem><para><literal>mmap</literal> is supported. This is
+ possible because the pagecache for an encrypted file contains
+ the plaintext, not the ciphertext.</para>
+ </listitem>
+ </itemizedlist>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Without the key</emphasis></para>
+ <para>Some filesystem operations may be performed on encrypted
+ regular files, directories, and symlinks even before their
+ encryption key has been added, or after their encryption key has
+ been removed:</para>
+ <itemizedlist>
+ <listitem>
+ <para>File metadata may be read, e.g. using
+ <literal>stat()</literal>.</para>
+ </listitem>
+ <listitem>
+ <para condition='l2F'>Directories may be listed, in which case
+ the filenames will be listed in an encoded form derived from
+ their ciphertext. The algorithm is subject to change but it is
+ guaranteed that the presented filenames will be no longer than
+ NAME_MAX bytes, will not contain the <literal>/</literal> or
+ <literal>\0</literal> characters, and will uniquely identify
+ directory entries. The <literal>.</literal> and
+ <literal>..</literal> directory entries are special. They are
+ always present and are not encrypted or encoded.</para>
+ </listitem>
+ <listitem>
+ <para>Files may be deleted. That is, nondirectory files may be
+ deleted with <literal>unlink()</literal> as usual, and empty
+ directories may be deleted with <literal>rmdir()</literal> as
+ usual. Therefore, <literal>rm</literal> and
+ <literal>rm -r</literal> will work as expected.</para>
+ </listitem>
+ <listitem>
+ <para>Symlink targets may be read and followed, but they will
+ be presented in encrypted form, similar to filenames in
+ directories. Hence, they are unlikely to point to anywhere
+ useful.</para>
+ </listitem>
+ </itemizedlist>
+ <para>Without the key, regular files cannot be opened or truncated.
+ Attempts to do so will fail with <literal>ENOKEY</literal>. This
+ implies that any regular file operations that require a file
+ descriptor, such as <literal>read()</literal>,
+ <literal>write()</literal>, <literal>mmap()</literal>,
+ <literal>fallocate()</literal>, and <literal>ioctl()</literal>,
+ are also forbidden.</para>
+ <para>Also without the key, files of any type (including
+ directories) cannot be created or linked into an encrypted
+ directory, nor can a name in an encrypted directory be the source
+ or target of a rename, nor can an <literal>O_TMPFILE</literal>
+ temporary file be created in an encrypted directory. All such
+ operations will fail with <literal>ENOKEY</literal>.</para>
+ <para>It is not currently possible to backup and restore encrypted
+ files without the encryption key. This would require special
+ APIs which have not yet been implemented.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Encryption policy enforcement
+ </emphasis></para>
+ <para>After an encryption policy has been set on a directory, all
+ regular files, directories, and symbolic links created in that
+ directory (recursively) will inherit that encryption policy.
+ Special files --- that is, named pipes, device nodes, and UNIX
+ domain sockets --- will not be encrypted.</para>
+ <para>Except for those special files, it is forbidden to have
+ unencrypted files, or files encrypted with a different encryption
+ policy, in an encrypted directory tree.</para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ </section>
+ <section xml:id="managingSecurity.clientencryption.keyhierarchy" remap="h3">
+ <title><indexterm><primary>encryption key hierarchy</primary>
+ </indexterm>Client-side encryption key hierarchy</title>
+ <para>Each encrypted directory tree is protected by a master key.</para>
+ <para>To "unlock" an encrypted directory tree, userspace must provide the
+ appropriate master key. There can be any number of master keys, each
+ of which protects any number of directory trees on any number of
+ filesystems.</para>
+ </section>
+ <section xml:id="managingSecurity.clientencryption.modes" remap="h3">
+ <title><indexterm><primary>encryption modes usage</primary>
+ </indexterm>Client-side encryption modes and usage</title>
+ <para><literal>fscrypt</literal> allows one encryption mode to be
+ specified for file contents and one encryption mode to be specified for
+ filenames. Different directory trees are permitted to use different
+ encryption modes. Currently, the following pairs of encryption modes are
+ supported:</para>
+ <itemizedlist>
+ <listitem>
+ <para>AES-256-XTS for contents and AES-256-CTS-CBC for filenames
+ </para>
+ </listitem>
+ <listitem>
+ <para>AES-128-CBC for contents and AES-128-CTS-CBC for filenames
+ </para>
+ </listitem>
+ </itemizedlist>
+ <para>If unsure, you should use the (AES-256-XTS, AES-256-CTS-CBC) pair.
+ </para>
+ <warning><para>In Lustre 2.14, client-side encryption only supports
+ content encryption, and not filename encryption. As a consequence, only
+ content encryption mode will be taken into account, and filename
+ encryption mode will be ignored to leave filenames in clear text.</para>
+ </warning>
+ <warning><para condition='l2F'>In Lustre 2.15, filename encryption mode
+ will be taken into account for new files and directories, if they are
+ under a parent encrypted directory created with Lustre 2.15. This means
+ new files and directories under a parent encrypted directory created with
+ Lustre 2.14 will not have their names encrypted.
+ Also, because files created with Lustre 2.14 did not have their names
+ encrypted, they will remain so after upgrade to 2.15.</para>
+ </warning>
+ </section>
+ <section xml:id="managingSecurity.clientencryption.threatmodel" remap="h3">
+ <title><indexterm><primary>encryption threat model</primary>
+ </indexterm>Client-side encryption threat model</title>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">Offline attacks</emphasis></para>
+ <para>For the Lustre case, block devices are Lustre targets attached
+ to the Lustre servers. Manipulating the filesystem offline means
+ accessing the filesystem on these targets while Lustre is offline.
+ </para>
+ <para>Provided that a strong encryption key is chosen,
+ <literal>fscrypt</literal> protects the confidentiality of file
+ contents in the event of a single point-in-time permanent offline
+ compromise of the block device content.
+ Lustre client-side encryption does not protect the confidentiality
+ of metadata, e.g. file names, file sizes, file permissions, file
+ timestamps, and extended attributes. Also, the existence and
+ location of holes (unallocated blocks which logically contain all
+ zeroes) in files is not protected.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Online attacks</emphasis></para>
+ <itemizedlist>
+ <listitem>
+ <para>On Lustre client</para>
+ <para>After an encryption key has been added,
+ <literal>fscrypt</literal> does not hide the plaintext file
+ contents or filenames from other users on the same node.
+ Instead, existing access control mechanisms such as file mode
+ bits, POSIX ACLs, LSMs, or namespaces should be used for this
+ purpose.</para>
+ <para>For the Lustre case, it means plaintext file contents or
+ filenames are not hidden from other users on the same Lustre
+ client.</para>
+ <para>An attacker who compromises the system enough to read from
+ arbitrary memory, e.g. by exploiting a kernel security
+ vulnerability, can compromise all encryption keys that are
+ currently in use.
+ However, <literal>fscrypt</literal> allows encryption keys to
+ be removed from the kernel, which may protect them from later
+ compromise. Key removal can be carried out by non-root users.
+ In more detail, the key removal will wipe the master encryption
+ key from kernel memory. Moreover, it will try to evict all
+ cached inodes which had been "unlocked" using the key, thereby
+ wiping their per-file keys and making them once again appear
+ "locked", i.e. in ciphertext or encrypted form.</para>
+ </listitem>
+ <listitem>
+ <para>On Lustre server</para>
+ <para>An attacker on a Lustre server who compromises the system
+ enough to read arbitrary memory, e.g. by exploiting a kernel
+ security vulnerability, cannot compromise Lustre files content.
+ Indeed, encryption keys are not forwarded to the Lustre servers,
+ and servers do not carry out decryption or encryption.
+ Moreover, bulk RPCs received by servers contain encrypted data,
+ which is written as-is to the underlying filesystem.</para>
+ </listitem>
+ </itemizedlist>
+ </listitem>
+ </itemizedlist>
+ </section>
+ <section xml:id="managingSecurity.clientencryption.fscrypt" remap="h3">
+ <title><indexterm><primary>encryption fscrypt policy</primary>
+ </indexterm>Manage encryption on directories</title>
+ <para>By default, Lustre client-side encryption is enabled, letting users
+ define encryption policies on a per-directory basis.</para>
+ <note><para>Administrators can decide to prevent a Lustre client
+ mount-point from using encryption by specifying the
+ <literal>noencrypt</literal> client mount option. This can be also
+ enforced from server side thanks to the
+ <literal>forbid_encryption</literal> property on nodemaps. See
+ <xref linkend="alteringproperties"/> for how to manage nodemaps.
+ </para></note>
+ <para><literal>fscrypt</literal> userspace tool can be used to manage
+ encryption policies. See https://github.com/google/fscrypt for
+ comprehensive explanations. Below are examples on how to use this tool
+ with Lustre. If not told otherwise, commands must be run on Lustre
+ client side.</para>
+ <itemizedlist>
+ <listitem>
+ <para>Two preliminary steps are required before actually deciding
+ which directories to encrypt, and this is the only
+ functionality which requires root privileges. Administrator has to
+ run:</para>
+ <screen># fscrypt setup
+Customizing passphrase hashing difficulty for this system...
+Created global config file at "/etc/fscrypt.conf".
+Metadata directories created at "/.fscrypt".</screen>
+ <para>This first command has to be run on all clients that want to use
+ encryption, as it sets up global fscrypt parameters outside of
+ Lustre.</para>
+ <screen># fscrypt setup /mnt/lustre
+Metadata directories created at "/mnt/lustre/.fscrypt"</screen>
+ <para>This second command has to be run on just one Lustre
+ client.</para>
+ <note><para>The file <literal>/etc/fscrypt.conf</literal> can be
+ edited. It is strongly recommended to set
+ <literal>policy_version</literal> to 2, so that
+ <literal>fscrypt</literal> wipes files from memory when the
+ encryption key is removed.</para></note>
+ </listitem>
+ <listitem>
+ <para>Now a regular user is able to select a directory to
+ encrypt:</para>
+ <screen>$ fscrypt encrypt /mnt/lustre/vault
+The following protector sources are available:
+1 - Your login passphrase (pam_passphrase)
+2 - A custom passphrase (custom_passphrase)
+3 - A raw 256-bit key (raw_key)
+Enter the source number for the new protector [2 - custom_passphrase]: 2
+Enter a name for the new protector: shield
+Enter custom passphrase for protector "shield":
+Confirm passphrase:
+"/mnt/lustre/vault" is now encrypted, unlocked, and ready for use.</screen>
+ <para>Starting from here, all files and directories created under
+ <literal>/mnt/lustre/vault</literal> will be encrypted, according
+ to the policy defined at the previsous step.</para>
+ <note><para>The encryption policy is inherited by all subdirectories.
+ It is not possible to change the policy for a subdirectory.</para>
+ </note>
+ </listitem>
+ <listitem>
+ <para>Another user can decide to encrypt a different directory with
+ its own protector:</para>
+ <screen>$ fscrypt encrypt /mnt/lustre/private
+Should we create a new protector? [y/N] Y
+The following protector sources are available:
+1 - Your login passphrase (pam_passphrase)
+2 - A custom passphrase (custom_passphrase)
+3 - A raw 256-bit key (raw_key)
+Enter the source number for the new protector [2 - custom_passphrase]: 2
+Enter a name for the new protector: armor
+Enter custom passphrase for protector "armor":
+Confirm passphrase:
+"/mnt/lustre/private" is now encrypted, unlocked, and ready for use.</screen>
+ </listitem>
+ <listitem>
+ <para>Users can decide to lock an encrypted directory at any
+ time:</para>
+ <screen>$ fscrypt lock /mnt/lustre/vault
+"/mnt/lustre/vault" is now locked.</screen>
+ <para>This action prevents access to encrypted content, and by
+ removing the key from memory, it also wipes files from memory if
+ they are not still open.</para>
+ </listitem>
+ <listitem>
+ <para>Users regain access to the encrypted directory with the command:
+ </para>
+ <screen>$ fscrypt unlock /mnt/lustre/vault
+Enter custom passphrase for protector "shield":
+"/mnt/lustre/vault" is now unlocked and ready for use.</screen>
+ </listitem>
+ <listitem>
+ <para>Actually, <literal>fscrypt</literal> does not give direct access
+ to master keys, but to protectors that are used to encrypt them.
+ This mechanism gives the ability to change a passphrase:</para>
+ <screen>$ fscrypt status /mnt/lustre
+lustre filesystem "/mnt/lustre" has 2 protectors and 2 policies
+
+PROTECTOR LINKED DESCRIPTION
+deacab807bf0e788 No custom protector "shield"
+e691ae7a1990fc2a No custom protector "armor"
+
+POLICY UNLOCKED PROTECTORS
+52b2b5aff0e59d8e0d58f962e715862e No deacab807bf0e788
+374e8944e4294b527e50363d86fc9411 No e691ae7a1990fc2a
+
+$ fscrypt metadata change-passphrase --protector=/mnt/lustre:deacab807bf0e788
+Enter old custom passphrase for protector "shield":
+Enter new custom passphrase for protector "shield":
+Confirm passphrase:
+Passphrase for protector deacab807bf0e788 successfully changed.</screen>
+ <para>It makes also possible to have multiple protectors for the same
+ policy. This is really useful when several users share an encrypted
+ directory, because it avoids the need to share any secret between
+ them.</para>
+ <screen>$ fscrypt status /mnt/lustre/vault
+"/mnt/lustre/vault" is encrypted with fscrypt.
+
+Policy: 52b2b5aff0e59d8e0d58f962e715862e
+Options: padding:32 contents:AES_256_XTS filenames:AES_256_CTS policy_version:2
+Unlocked: No
+
+Protected with 1 protector:
+PROTECTOR LINKED DESCRIPTION
+deacab807bf0e788 No custom protector "shield"
+
+$ fscrypt metadata create protector /mnt/lustre
+Create new protector on "/mnt/lustre" [Y/n] Y
+The following protector sources are available:
+1 - Your login passphrase (pam_passphrase)
+2 - A custom passphrase (custom_passphrase)
+3 - A raw 256-bit key (raw_key)
+Enter the source number for the new protector [2 - custom_passphrase]: 2
+Enter a name for the new protector: bunker
+Enter custom passphrase for protector "bunker":
+Confirm passphrase:
+Protector f3cc1b5cf9b8f41c created on filesystem "/mnt/lustre".
+
+$ fscrypt metadata add-protector-to-policy
+ --protector=/mnt/lustre:f3cc1b5cf9b8f41c
+ --policy=/mnt/lustre:52b2b5aff0e59d8e0d58f962e715862e
+WARNING: All files using this policy will be accessible with this protector!!
+Protect policy 52b2b5aff0e59d8e0d58f962e715862e with protector f3cc1b5cf9b8f41c? [Y/n] Y
+Enter custom passphrase for protector "bunker":
+Enter custom passphrase for protector "shield":
+Protector f3cc1b5cf9b8f41c now protecting policy 52b2b5aff0e59d8e0d58f962e715862e.
+
+$ fscrypt status /mnt/lustre/vault
+"/mnt/lustre/vault" is encrypted with fscrypt.
+
+Policy: 52b2b5aff0e59d8e0d58f962e715862e
+Options: padding:32 contents:AES_256_XTS filenames:AES_256_CTS policy_version:2
+Unlocked: No
+
+Protected with 2 protectors:
+PROTECTOR LINKED DESCRIPTION
+deacab807bf0e788 No custom protector "shield"
+f3cc1b5cf9b8f41c No custom protector "bunker"</screen>
+ </listitem>
+ </itemizedlist>
+ </section>
+ </section>
+ <section xml:id="managingSecurity.kerberos">
+ <title><indexterm><primary>Kerberos</primary></indexterm>
+ Configuring Kerberos (KRB) Security</title>
+ <para>This chapter describes how to use Kerberos with Lustre.</para>
+ <section xml:id="managingSecurity.kerberos.whatisit">
+ <title>What Is Kerberos?</title>
+ <para>Kerberos is a mechanism for authenticating all entities (such as
+ users and servers) on an "unsafe" network. Each of these
+ entities, known as "principals", negotiate a runtime key with
+ the Kerberos server. This key enables principals to verify that messages
+ from the Kerberos server are authentic. By trusting the Kerberos server,
+ users and services can authenticate one another.</para>
+ <para>Setting up Lustre with Kerberos can provide advanced security
+ protections for the Lustre network. Broadly, Kerberos offers three types
+ of benefit:</para>
+ <itemizedlist>
+ <listitem>
+ <para>Allows Lustre connection peers (MDS, OSS and clients) to
+ authenticate one another.</para>
+ </listitem>
+ <listitem>
+ <para>Protects the integrity of PTLRPC messages from being modified
+ during network transfer.</para>
+ </listitem>
+ <listitem>
+ <para>Protects the privacy of the PTLRPC message from being
+ eavesdropped during network transfer.</para>
+ </listitem>
+ </itemizedlist>
+ <para>Kerberos uses the “kernel keyring” client upcall mechanism.</para>
+ </section>
+ <section xml:id="managingSecurity.kerberos.securityflavor">
+ <title>Security Flavor</title>
+ <para>
+ A security flavor is a string to describe what kind authentication
+ and data transformation be performed upon a PTLRPC connection. It
+ covers both RPC message and BULK data.
+ </para>
+ <para>
+ The supported flavors are described in following table:
+ </para>
+ <informaltable>
+ <tgroup cols="5">
+ <colspec align="left" />
+ <colspec align="left" />
+ <colspec align="left" />
+ <colspec align="left" />
+ <colspec align="left" />
+ <thead>
+ <row>
+ <entry>
+ Base Flavor
+ </entry>
+ <entry>
+ Authentication
+ </entry>
+ <entry>
+ RPC Message Protection
+ </entry>
+ <entry>
+ Bulk Data Protection
+ </entry>
+ <entry>
+ Notes
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <emphasis><emphasis role="strong">null</emphasis></emphasis>
+ </entry>
+ <entry>
+ N/A
+ </entry>
+ <entry>
+ N/A
+ </entry>
+ <entry>
+ N/A
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <emphasis><emphasis role="strong">krb5n</emphasis></emphasis>
+ </entry>
+ <entry>
+ GSS/Kerberos5
+ </entry>
+ <entry>
+ null
+ </entry>
+ <entry>
+ checksum
+ </entry>
+ <entry>
+ No protection of RPC message, checksum protection
+ of bulk data, light performance overhead.
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <emphasis><emphasis role="strong">krb5a</emphasis></emphasis>
+ </entry>
+ <entry>
+ GSS/Kerberos5
+ </entry>
+ <entry>
+ partial integrity (krb5)
+ </entry>
+ <entry>
+ checksum
+ </entry>
+ <entry>
+ Only header of RPC message is integrity protected, and
+ checksum protection of bulk data, more performance
+ overhead compare to krb5n.
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <emphasis><emphasis role="strong">krb5i</emphasis></emphasis>
+ </entry>
+ <entry>
+ GSS/Kerberos5
+ </entry>
+ <entry>
+ integrity (krb5)
+ </entry>
+ <entry>
+ integrity (krb5)
+ </entry>
+ <entry>
+ transformation algorithm is determined by actual Kerberos
+ algorithms enforced by KDC and principals; heavy performance
+ penalty.
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <emphasis><emphasis role="strong">krb5p</emphasis></emphasis>
+ </entry>
+ <entry>
+ GSS/Kerberos5
+ </entry>
+ <entry>
+ privacy (krb5)
+ </entry>
+ <entry>
+ privacy (krb5)
+ </entry>
+ <entry>
+ transformation privacy protection algorithm is determined
+ by actual Kerberos algorithms enforced by KDC and principals;
+ the heaviest performance penalty.
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ <section xml:id="managingSecurity.kerberos.kerberossetup">
+ <title>Kerberos Setup</title>
+ <section xml:id="managingSecurity.kerberos.kerberossetup.distribution">
+ <title>Distribution</title>
+ <para>We only support MIT Kerberos 5, from version 1.3.</para>
+ <para>For environmental requirements in general, and clock
+ synchronization in particular, please refer to section
+ <xref linkend="section_rh2_d4w_gk"/>.</para>
+ </section>
+ <section xml:id="managingSecurity.kerberos.kerberossetup.configuration">
+ <title>Principals Configuration</title>
+ <itemizedlist>
+ <listitem>
+ <para>Configure client nodes:</para>
+ <itemizedlist>
+ <listitem>
+ <para>
+ For each client node, create a <literal>lustre_root</literal>
+ principal and generate keytab.
+ </para>
+ <screen>kadmin> addprinc -randkey lustre_root/client_host.domain@REALM</screen>
+ <screen>kadmin> ktadd lustre_root/client_host.domain@REALM</screen>
+ </listitem>
+ <listitem>
+ <para>
+ Install the keytab on the client node.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </listitem>
+ <listitem>
+ <para>Configure MGS nodes:</para>
+ <itemizedlist>
+ <listitem>
+ <para>
+ For each MGS node, create a <literal>lustre_mgs</literal>
+ principal and generate keytab.
+ </para>
+ <screen>kadmin> addprinc -randkey lustre_mgs/mgs_host.domain@REALM</screen>
+ <screen>kadmin> ktadd lustre_mds/mgs_host.domain@REALM</screen>
+ </listitem>
+ <listitem>
+ <para>
+ Install the keytab on the MGS nodes.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </listitem>
+ <listitem>
+ <para>Configure MDS nodes:</para>
+ <itemizedlist>
+ <listitem>
+ <para>
+ For each MDS node, create a <literal>lustre_mds</literal>
+ principal and generate keytab.
+ </para>
+ <screen>kadmin> addprinc -randkey lustre_mds/mds_host.domain@REALM</screen>
+ <screen>kadmin> ktadd lustre_mds/mds_host.domain@REALM</screen>
+ </listitem>
+ <listitem>
+ <para>
+ Install the keytab on the MDS nodes.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </listitem>
+ <listitem>
+ <para>Configure OSS nodes:</para>
+ <itemizedlist>
+ <listitem>
+ <para>
+ For each OSS node, create a <literal>lustre_oss</literal>
+ principal and generate keytab.
+ </para>
+ <screen>kadmin> addprinc -randkey lustre_oss/oss_host.domain@REALM</screen>
+ <screen>kadmin> ktadd lustre_oss/oss_host.domain@REALM</screen>
+ </listitem>
+ <listitem>
+ <para>
+ Install the keytab on the client node.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </listitem>
+ </itemizedlist>
+ <note>
+ <itemizedlist>
+ <listitem>
+ <para>The <emphasis>host.domain</emphasis> should be the FQDN in
+ your network, otherwise server might not recognize any GSS
+ request.</para>
+ </listitem>
+ <listitem>
+ <para>
+ As an alternative for the client keytab, if you want to save
+ the trouble of assigning unique keytab for each client node,
+ you can create a general lustre_root principal and its
+ keytab, and install the same keytab on as many client nodes
+ as you want. <emphasis role="strong">Be aware that in
+ this way one compromised client means all clients are
+ insecure</emphasis>.
+ </para>
+ <screen>kadmin> addprinc -randkey lustre_root@REALM</screen>
+ <screen>kadmin> ktadd lustre_root@REALM</screen>
+ </listitem>
+ <listitem>
+ <para>
+ Lustre support following <emphasis>enctypes</emphasis> for
+ MIT Kerberos 5 version 1.3 or higher:
+ </para>
+ <itemizedlist>
+ <listitem>
+ <para>
+ <emphasis>aes128-cts</emphasis>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <emphasis>aes256-cts</emphasis>
+ </para>
+ </listitem>
+ </itemizedlist>
+ </listitem>
+ </itemizedlist>
+ </note>
+ </section>
+ </section>
+ <section xml:id="managingSecurity.kerberos.network">
+ <title>Networking</title>
+ <para>On networks for which name resolution to IP address is possible,
+ like TCP or InfiniBand, the names used in the principals must be the
+ ones that resolve to the IP addresses used by the Lustre NIDs.</para>
+ <para>If you are using a network which is
+ <emphasis role="strong">NOT</emphasis> TCP or InfiniBand (e.g.
+ PTL4LND), you need to have a <literal>/etc/lustre/nid2hostname</literal>
+ script on <emphasis role="strong">each</emphasis> node, which purpose is
+ to translate NID into hostname.
+ Following is a possible example for PTL4LND:</para>
+ <screen>#!/bin/bash
+set -x
+
+# convert a NID for a LND to a hostname
+
+# called with thre arguments: lnd netid nid
+# $lnd is the string "PTL4LND", etc.
+# $netid is the network identifier in hex string format
+# $nid is the NID in hex format
+# output the corresponding hostname,
+# or error message leaded by a '@' for error logging.
+
+lnd=$1
+netid=$2
+# convert hex NID number to decimal
+nid=$((0x$3))
+
+case $lnd in
+ PTL4LND) # simply add 'node' at the beginning
+ echo "node$nid"
+ ;;
+ *)
+ echo "@unknown LND: $lnd"
+ ;;
+esac</screen>
+ </section>
+ <section xml:id="managingSecurity.kerberos.requiredpackages">
+ <title>Required packages</title>
+ <para>
+ Every node should have following packages installed:
+ </para>
+ <itemizedlist>
+ <listitem>
+ <para>krb5-workstation</para>
+ </listitem>
+ <listitem>
+ <para>krb5-libs</para>
+ </listitem>
+ <listitem>
+ <para>keyutils</para>
+ </listitem>
+ <listitem>
+ <para>keyutils-libs</para>
+ </listitem>
+ </itemizedlist>
+ <para>On the node used to build Lustre with GSS support, following
+ packages should be installed:</para>
+ <itemizedlist>
+ <listitem>
+ <para>krb5-devel</para>
+ </listitem>
+ <listitem>
+ <para>keyutils-libs-devel</para>
+ </listitem>
+ </itemizedlist>
+ </section>
+ <section xml:id="managingSecurity.kerberos.buildlustre">
+ <title>Build Lustre</title>
+ <para>
+ Enable GSS at configuration time:
+ </para>
+ <screen>./configure --enable-gss --other-options</screen>
+ </section>
+ <section xml:id="managingSecurity.kerberos.running">
+ <title>Running</title>
+ <section xml:id="managingSecurity.kerberos.running.gssdaemons">
+ <title>GSS Daemons</title>
+ <para>
+ Make sure to start the daemon process
+ <literal>lsvcgssd</literal> on each server node (MGS, MDS and OSS)
+ before starting Lustre. The command syntax is:
+ </para>
+ <screen>lsvcgssd [-f] [-v] [-g] [-m] [-o] -k</screen>
+ <itemizedlist>
+ <listitem>
+ <para>-f: run in foreground, instead of as daemon</para>
+ </listitem>
+ <listitem>
+ <para>-v: increase verbosity by 1. For example, to set the verbose
+ level to 3, run 'lsvcgssd -vvv'. Verbose logging can help you make
+ sure Kerberos is set up correctly.
+ </para>
+ </listitem>
+ <listitem>
+ <para>-g: service MGS</para>
+ </listitem>
+ <listitem>
+ <para>-m: service MDS</para>
+ </listitem>
+ <listitem>
+ <para>-o: service OSS</para>
+ </listitem>
+ <listitem>
+ <para>-k: enable kerberos support</para>
+ </listitem>
+ </itemizedlist>
+ </section>
+ <section xml:id="managingSecurity.kerberos.running.settingsecurityflavors">
+ <title>Setting Security Flavors</title>
+ <para>
+ Security flavors can be set by defining sptlrpc rules on the MGS.
+ These rules are persistent, and are in the form:
+ <literal><spec>=<flavor></literal>
+ </para>
+ <itemizedlist>
+ <listitem>
+ <para>To add a rule:</para>
+ <screen>mgs> lctl conf_param <spec>=<flavor></screen>
+ <para>
+ If there is an existing rule on <spec>, it will be
+ overwritten.</para>
+ </listitem>
+ <listitem>
+ <para>To delete a rule:</para>
+ <screen>mgs> lctl conf_param -d <spec></screen>
+ </listitem>
+ <listitem>
+ <para>To list existing rules:</para>
+ <screen>msg> lctl get_param mgs.MGS.live.<fs-name> | grep "srpc.flavor"</screen>
+ </listitem>
+ </itemizedlist>
+ <note>
+ <itemizedlist>
+ <listitem>
+ <para>If nothing is specified, by default all RPC connections will
+ use <literal>null</literal> flavor, which means no security.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ After you change a rule, it usually takes a few minutes to apply
+ the new rule to all nodes, depending on global system load.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Before you change a rule, make sure affected nodes are ready
+ for the new security flavor. E.g. if you change flavor from
+ <literal>null</literal> to <literal>krb5p</literal>
+ but GSS/Kerberos environment is not properly configured on
+ affected nodes, those nodes might be evicted because they cannot
+ communicate with each other.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </note>
+ </section>
+ <section xml:id="managingSecurity.kerberos.running.rulessyntaxexamples">
+ <title>Rules Syntax & Examples</title>
+ <para>
+ The general syntax is:
+ <literal>
+ <target>.srpc.flavor.<network>[.<direction>]=flavor
+ </literal></para>
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal><target></literal> can be filesystem name, or
+ specific MDT/OST device name. For example
+ <literal>testfs</literal>,
+ <literal>testfs-MDT0000</literal>,
+ <literal>testfs-OST0001</literal>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal><network></literal> is the LNet network name, for
+ example <literal>tcp0</literal>, <literal>o2ib0</literal>, or
+ <literal>default</literal> to not filter on LNet network.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal><direction></literal> can be one of
+ <emphasis>cli2mdt</emphasis>, <emphasis>cli2ost</emphasis>,
+ <emphasis>mdt2mdt</emphasis>, <emphasis>mdt2ost</emphasis>.
+ Direction is optional.
+ </para>
+ </listitem>
+ </itemizedlist>
+ <para>
+ Examples:
+ </para>
+ <itemizedlist>
+ <listitem>
+ <para>
+ Apply <literal>krb5i</literal> on
+ <emphasis role="strong">ALL</emphasis> connections for file system
+ <literal>testfs</literal>:
+ </para>
+ </listitem>
+ </itemizedlist>
+ <screen>mgs> lctl conf_param testfs.srpc.flavor.default=krb5i</screen>
+ <itemizedlist>
+ <listitem>
+ <para>
+ Nodes in network <literal>tcp0</literal> use
+ <literal>krb5p</literal>; all other nodes use
+ <literal>null</literal>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ <screen>mgs> lctl conf_param testfs.srpc.flavor.tcp0=krb5p
+mgs> lctl conf_param testfs.srpc.flavor.default=null</screen>
+ <itemizedlist>
+ <listitem>
+ <para>
+ Nodes in network <literal>tcp0</literal> use
+ <literal>krb5p</literal>; nodes in
+ <literal>o2ib0</literal> use <literal>krb5n</literal>;
+ among other nodes, clients use <literal>krb5i</literal>
+ to MDT/OST, MDTs use <literal>null</literal> to other MDTs,
+ MDTs use <literal>krb5a</literal> to OSTs.
+ </para>
+ </listitem>
+ </itemizedlist>
+ <screen>mgs> lctl conf_param testfs.srpc.flavor.tcp0=krb5p
+mgs> lctl conf_param testfs.srpc.flavor.o2ib0=krb5n
+mgs> lctl conf_param testfs.srpc.flavor.default.cli2mdt=krb5i
+mgs> lctl conf_param testfs.srpc.flavor.default.cli2ost=krb5i
+mgs> lctl conf_param testfs.srpc.flavor.default.mdt2mdt=null
+mgs> lctl conf_param testfs.srpc.flavor.default.mdt2ost=krb5a</screen>
+ </section>
+ <section xml:id="managingSecurity.kerberos.running.authenticatenormalusers">
+ <title>Regular Users Authentication</title>
+ <para>
+ On client nodes, non-root users need to issue
+ <literal>kinit</literal> before accessing Lustre, just like other
+ Kerberized applications.
+ </para>
+ <itemizedlist>
+ <listitem>
+ <para>
+ Required by kerberos, the user's principal
+ (<literal>username@REALM</literal>) should be added to the KDC.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Client and MDT nodes should have the same user database
+ used for name and uid/gid translation.
+ </para>
+ </listitem>
+ </itemizedlist>
+ <para>
+ Regular users can destroy the established security contexts before
+ logging out, by issuing:
+ </para>
+ <screen>lfs flushctx -k -r <mount point></screen>
+ <para>
+ Here <literal>-k</literal> is to destroy the on-disk Kerberos
+ credential cache, similar to <literal>kdestroy</literal>, and
+ <literal>-r</literal> is to reap the revoked keys from the keyring
+ when flushing the GSS context. Otherwise it only destroys established
+ contexts in kernel memory.
+ </para>
+ </section>
+ </section>
+ <section xml:id="managingSecurity.kerberos.securemgsconnection">
+ <title>Secure MGS connection</title>
+ <para>
+ Each node can specify which flavor to use to connect to the MGS, by
+ using the <literal>mgssec=flavor</literal> mount option.
+ Once a flavor is chosen, it cannot be changed until re-mount.
+ </para>
+ <para>
+ Because a Lustre node only has one connection to the MGS, if there is
+ more than one target or client on the node, they necessarily use the
+ same security flavor to the MGS, being the one enforced when the first
+ connection to the MGS was established.
+ </para>
+ <para>
+ By default, the MGS accepts RPCs with any flavor. But it is possible to
+ configure the MGS to only accept a given flavor. The syntax is identical
+ to what is explained in paragraph
+ <xref linkend="managingSecurity.kerberos.running.rulessyntaxexamples"/>,
+ but with special target <literal>_mgs</literal>:
+ </para>
+ <screen>mgs> lctl conf_param _mgs.srpc.flavor.<network>=<flavor></screen>