+ </itemizedlist>
+ </note>
+ <para>The root squash settings can also be changed temporarily with
+ <literal>lctl set_param</literal> or persistently with
+ <literal>lctl set_param -P</literal>. For example:</para>
+ <screen>mgs# lctl set_param mdt.testfs-MDT0000.root_squash="1:0"
+mgs# lctl set_param -P mdt.testfs-MDT0000.root_squash="1:0"</screen>
+ <para>The <literal>nosquash_nids</literal> list can be cleared with:</para>
+ <screen>mgs# lctl conf_param testfs.mdt.nosquash_nids="NONE"</screen>
+ <para>- OR -</para>
+ <screen>mgs# lctl conf_param testfs.mdt.nosquash_nids="clear"</screen>
+ <para>If the <literal>nosquash_nids</literal> value consists of several
+ NID ranges (e.g. <literal>0@elan</literal>, <literal>1@elan1</literal>),
+ the list of NID ranges must be quoted with single (') or double
+ ('') quotation marks. List elements must be separated with a
+ space. For example:</para>
+ <screen>mds# mkfs.lustre ... --param "mdt.nosquash_nids='0@elan1 1@elan2'" /dev/sda1
+lctl conf_param testfs.mdt.nosquash_nids="24@elan 15@elan1"</screen>
+ <para>These are examples of incorrect syntax:</para>
+ <screen>mds# mkfs.lustre ... --param "mdt.nosquash_nids=0@elan1 1@elan2" /dev/sda1
+lctl conf_param testfs.mdt.nosquash_nids=24@elan 15@elan1</screen>
+ <para>To check root squash parameters, use the lctl get_param command:
+ </para>
+ <screen>mds# lctl get_param mdt.testfs-MDT0000.root_squash
+lctl get_param mdt.*.nosquash_nids</screen>
+ <note>
+ <para>An empty nosquash_nids list is reported as NONE.</para>
+ </note>
+ </section>
+ <section xml:id="managingSecurity.root_squash.tips" remap="h3">
+ <title><indexterm>
+ <primary>root squash</primary>
+ <secondary>tips</secondary>
+ </indexterm>Tips on Using Root Squash</title>
+ <para>Lustre configuration management limits root squash in several ways.
+ </para>
+ <itemizedlist>
+ <listitem>
+ <para>The <literal>lctl conf_param</literal> value overwrites the
+ parameter's previous value. If the new value uses an incorrect
+ syntax, then the system continues with the old parameters and the
+ previously-correct value is lost on remount. That is, be careful
+ doing root squash tuning.</para>
+ </listitem>
+ <listitem>
+ <para><literal>mkfs.lustre</literal> and
+ <literal>tunefs.lustre</literal> do not perform parameter syntax
+ checking. If the root squash parameters are incorrect, they are
+ ignored on mount and the default values are used instead.</para>
+ </listitem>
+ <listitem>
+ <para>Root squash parameters are parsed with rigorous syntax checking.
+ The root_squash parameter should be specified as
+ <literal><decnum>:<decnum></literal>. The
+ <literal>nosquash_nids</literal> parameter should follow LNet NID
+ range list syntax.</para>
+ </listitem>
+ </itemizedlist>
+ <para>LNet NID range syntax:</para>
+ <screen><nidlist> :== <nidrange> [ ' ' <nidrange> ]
+<nidrange> :== <addrrange> '@' <net>
+<addrrange> :== '*' |
+ <ipaddr_range> |
+ <numaddr_range>
+<ipaddr_range> :==
+<numaddr_range>.<numaddr_range>.<numaddr_range>.<numaddr_range>
+<numaddr_range> :== <number> |
+ <expr_list>
+<expr_list> :== '[' <range_expr> [ ',' <range_expr>] ']'
+<range_expr> :== <number> |
+ <number> '-' <number> |
+ <number> '-' <number> '/' <number>
+<net> :== <netname> | <netname><number>
+<netname> :== "lo" | "tcp" | "o2ib"
+ | "ra" | "elan"
+<number> :== <nonnegative decimal> | <hexadecimal></screen>
+ <note>
+ <para>For networks using numeric addresses (e.g. elan), the address
+ range must be specified in the
+ <literal><numaddr_range></literal> syntax. For networks using
+ IP addresses, the address range must be in the
+ <literal><ipaddr_range></literal>. For example, if elan is using
+ numeric addresses, <literal>1.2.3.4@elan</literal> is incorrect.
+ </para>
+ </note>
+ </section>
+ </section>
+ <section xml:id="managingSecurity.isolation">
+ <title><indexterm><primary>Isolation</primary></indexterm>
+ Isolating Clients to a Sub-directory Tree</title>
+ <para>Isolation is the Lustre implementation of the generic concept of
+ multi-tenancy, which aims at providing separated namespaces from a single
+ filesystem. Lustre Isolation enables different populations of users on
+ the same file system beyond normal Unix permissions/ACLs, even when users
+ on the clients may have root access. Those tenants share the same file
+ system, but they are isolated from each other: they cannot access or even
+ see each other’s files, and are not aware that they are sharing common
+ file system resources.</para>
+ <para>Lustre Isolation leverages the Fileset feature
+ (<xref linkend="SystemConfigurationUtilities.fileset" />)
+ to mount only a subdirectory of the filesystem rather than the root
+ directory.
+ In order to achieve isolation, the subdirectory mount, which presents to
+ tenants only their own fileset, has to be imposed to the clients. To that
+ extent, we make use of the nodemap feature
+ (<xref linkend="lustrenodemap.title" />). We group all clients used by a
+ tenant under a common nodemap entry, and we assign to this nodemap entry
+ the fileset to which the tenant is restricted.</para>
+ <section xml:id="managingSecurity.isolation.clientid" remap="h3">
+ <title><indexterm><primary>Isolation</primary><secondary>
+ client identification</secondary></indexterm>Identifying Clients</title>
+ <para>Enforcing multi-tenancy on Lustre relies on the ability to properly
+ identify the client nodes used by a tenant, and trust those identities.
+ This can be achieved by having physical hardware and/or network
+ security, so that client nodes have well-known NIDs. It is also possible
+ to make use of strong authentication with Kerberos or Shared-Secret Key
+ (see <xref linkend="lustressk" />).
+ Kerberos prevents NID spoofing, as every client needs its own
+ credentials, based on its NID, in order to connect to the servers.
+ Shared-Secret Key also prevents tenant impersonation, because keys
+ can be linked to a specific nodemap. See
+ <xref linkend="ssknodemaprole" /> for detailed explanations.
+</para>
+ </section>
+ <section xml:id="managingSecurity.isolation.configuring" remap="h3">
+ <title><indexterm><primary>Isolation</primary><secondary>
+ configuring</secondary></indexterm>Configuring Isolation</title>
+ <para>Isolation on Lustre can be achieved by setting the
+ <literal>fileset</literal> parameter on a nodemap entry. All clients
+ belonging to this nodemap entry will automatically mount this fileset
+ instead of the root directory. For example:</para>
+ <screen>mgs# lctl nodemap_set_fileset --name tenant1 --fileset '/dir1'</screen>
+ <para>So all clients matching the <literal>tenant1</literal> nodemap will
+ be automatically presented the fileset <literal>/dir1</literal> when
+ mounting. This means these clients are doing an implicit subdirectory
+ mount on the subdirectory <literal>/dir1</literal>.
+ </para>
+ <note>
+ <para>
+ If subdirectory defined as fileset does not exist on the file system,
+ it will prevent any client belonging to the nodemap from mounting
+ Lustre.
+ </para>
+ </note>
+ <para>To delete the fileset parameter, just set it to an empty string:
+ </para>
+ <screen>mgs# lctl nodemap_set_fileset --name tenant1 --fileset ''</screen>
+ </section>
+ <section xml:id="managingSecurity.isolation.permanent" remap="h3">
+ <title><indexterm><primary>Isolation</primary><secondary>
+ making permanent</secondary></indexterm>Making Isolation Permanent
+ </title>
+ <para>In order to make isolation permanent, the fileset parameter on the
+ nodemap has to be set with <literal>lctl set_param</literal> with the
+ <literal>-P</literal> option.</para>
+ <screen>mgs# lctl set_param nodemap.tenant1.fileset=/dir1
+mgs# lctl set_param -P nodemap.tenant1.fileset=/dir1</screen>
+ <para>This way the fileset parameter will be stored in the Lustre config
+ logs, letting the servers retrieve the information after a restart.
+ </para>
+ </section>
+ </section>
+ <section xml:id="managingSecurity.sepol" condition='l2D'>
+ <title><indexterm><primary>selinux policy check</primary></indexterm>
+ Checking SELinux Policy Enforced by Lustre Clients</title>
+ <para>SELinux provides a mechanism in Linux for supporting Mandatory Access
+ Control (MAC) policies. When a MAC policy is enforced, the operating
+ system’s (OS) kernel defines application rights, firewalling applications
+ from compromising the entire system. Regular users do not have the ability to
+ override the policy.</para>
+ <para>One purpose of SELinux is to protect the
+ <emphasis role="bold">OS</emphasis> from privilege escalation. To that
+ extent, SELinux defines confined and unconfined domains for processes and
+ users. Each process, user, file is assigned a security context, and
+ rules define the allowed operations by processes and users on files.
+ </para>
+ <para>Another purpose of SELinux can be to protect
+ <emphasis role="bold">data</emphasis> sensitivity, thanks to Multi-Level
+ Security (MLS). MLS works on top of SELinux, by defining the concept of
+ security levels in addition to domains. Each process, user and file is
+ assigned a security level, and the model states that processes and users
+ can read the same or lower security level, but can only write to their own
+ or higher security level.
+ </para>
+ <para>From a file system perspective, the security context of files must be
+ stored permanently. Lustre makes use of the
+ <literal>security.selinux</literal> extended attributes on files to hold
+ this information. Lustre supports SELinux on the client side. All you have
+ to do to have MAC and MLS on Lustre is to enforce the appropriate SELinux
+ policy (as provided by the Linux distribution) on all Lustre clients. No
+ SELinux is required on Lustre servers.
+ </para>
+ <para>Because Lustre is a distributed file system, the specificity when
+ using MLS is that Lustre really needs to make sure data is always accessed
+ by nodes with the SELinux MLS policy properly enforced. Otherwise, data is
+ not protected. This means Lustre has to check that SELinux is properly
+ enforced on client side, with the right, unaltered policy. And if SELinux
+ is not enforced as expected on a client, the server denies its access to
+ Lustre.
+ </para>
+ <section xml:id="managingSecurity.sepol.determining" remap="h3">
+ <title><indexterm><primary>selinux policy check</primary><secondary>
+ determining</secondary></indexterm>Determining SELinux Policy Info
+ </title>
+ <para>A string that represents the SELinux Status info will be used by
+ servers as a reference, to check if clients are enforcing SELinux
+ properly. This reference string can be obtained on a client node known
+ to enforce the right SELinux policy, by calling the
+ <literal>l_getsepol</literal> command line utility:</para>
+ <screen>client# l_getsepol
+SELinux status info: 1:mls:31:40afb76d077c441b69af58cccaaa2ca63641ed6e21b0a887dc21a684f508b78f</screen>
+ <para>The string describing the SELinux policy has the following
+ syntax:</para>
+ <para><literal>mode:name:version:hash</literal></para>
+ <para>where:</para>
+ <itemizedlist>
+ <listitem>
+ <para><literal>mode</literal> is a digit telling if SELinux is in
+ Permissive mode (0) or Enforcing mode (1)</para>
+ </listitem>
+ <listitem>
+ <para><literal>name</literal> is the name of the SELinux policy
+ </para>
+ </listitem>
+ <listitem>
+ <para><literal>version</literal> is the version of the SELinux
+ policy</para>
+ </listitem>
+ <listitem>
+ <para><literal>hash</literal> is the computed hash of the binary
+ representation of the policy, as exported in
+ /etc/selinux/<literal>name</literal>/policy/policy.
+ <literal>version</literal></para>
+ </listitem>
+ </itemizedlist>
+ </section>
+ <section xml:id="managingSecurity.sepol.configuring" remap="h3">
+ <title><indexterm><primary>selinux policy check</primary><secondary>
+ enforcing</secondary></indexterm>Enforcing SELinux Policy Check</title>
+ <para>SELinux policy check can be enforced by setting the
+ <literal>sepol</literal> parameter on a nodemap entry. All clients
+ belonging to this nodemap entry must enforce the SELinux policy
+ described by this parameter, otherwise they are denied access to the
+ Lustre file system. For example:</para>
+ <screen>mgs# lctl nodemap_set_sepol --name restricted
+ --sepol '1:mls:31:40afb76d077c441b69af58cccaaa2ca63641ed6e21b0a887dc21a684f508b78f'</screen>
+ <para>So all clients matching the <literal>restricted</literal> nodemap
+ must enforce the SELinux policy which description matches
+ <literal>1:mls:31:40afb76d077c441b69af58cccaaa2ca63641ed6e21b0a887dc21a684f508b78f</literal>.
+ If not, they will get Permission Denied when trying to mount or access
+ files on the Lustre file system.</para>
+ <para>To delete the <literal>sepol</literal> parameter, just set it to an
+ empty string:</para>
+ <screen>mgs# lctl nodemap_set_sepol --name restricted --sepol ''</screen>
+ <para>See <xref linkend="lustrenodemap.title" /> for more details about
+ the Nodemap feature.</para>
+ </section>
+ <section xml:id="managingSecurity.sepol.permanent" remap="h3">
+ <title><indexterm><primary>selinux policy check</primary><secondary>
+ making permanent</secondary></indexterm>Making SELinux Policy Check
+ Permanent</title>
+ <para>In order to make SELinux Policy check permanent, the sepol parameter
+ on the nodemap has to be set with <literal>lctl set_param</literal> with
+ the <literal>-P</literal> option.</para>
+ <screen>mgs# lctl set_param nodemap.restricted.sepol=1:mls:31:40afb76d077c441b69af58cccaaa2ca63641ed6e21b0a887dc21a684f508b78f
+mgs# lctl set_param -P nodemap.restricted.sepol=1:mls:31:40afb76d077c441b69af58cccaaa2ca63641ed6e21b0a887dc21a684f508b78f</screen>
+ <para>This way the sepol parameter will be stored in the Lustre config
+ logs, letting the servers retrieve the information after a restart.
+ </para>
+ </section>
+ <section xml:id="managingSecurity.sepol.client" remap="h3">
+ <title><indexterm><primary>selinux policy check</primary><secondary>
+ sending client</secondary></indexterm>Sending SELinux Status Info from
+ Clients</title>
+ <para>In order for Lustre clients to send their SELinux status
+ information, in case SELinux is enabled locally, the
+ <literal>send_sepol</literal> ptlrpc kernel module's parameter has to be
+ set to a non-zero value. <literal>send_sepol</literal> accepts various
+ values:</para>
+ <itemizedlist>
+ <listitem>
+ <para>0: do not send SELinux policy info;</para>
+ </listitem>
+ <listitem>
+ <para>-1: fetch SELinux policy info for every request;</para>
+ </listitem>
+ <listitem>
+ <para>N > 0: only fetch SELinux policy info every N seconds. Use
+ <literal>N = 2^31-1</literal> to have SELinux policy info
+ fetched only at mount time.</para>
+ </listitem>
+ </itemizedlist>
+ <para>Clients that are part of a nodemap on which
+ <literal>sepol</literal> is defined must send SELinux status info.
+ And the SELinux policy they enforce must match the representation
+ stored into the nodemap. Otherwise they will be denied access to the
+ Lustre file system.</para>
+ </section>
+ </section>
+ <section xml:id="managingSecurity.clientencryption" condition='l2E'>
+ <title><indexterm><primary>Client-side encryption</primary></indexterm>
+ Encrypting files and directories</title>
+ <para>The purpose that client-side encryption wants to serve is to be able
+ to provide a special directory for each user, to safely store sensitive
+ files. The goals are to protect data in transit between clients and
+ servers, and protect data at rest.</para>
+ <para>This feature is implemented directly at the Lustre client level.
+ Lustre client-side encryption relies on kernel <literal>fscrypt</literal>.
+ <literal>fscrypt</literal> is a library which filesystems can hook into to
+ support transparent encryption of files and directories. As a consequence,
+ the key points described below are extracted from
+ <literal>fscrypt</literal> documentation.</para>
+ <para>For full details, please refer to documentation available with the
+ Lustre sources, under the
+ <literal>Documentation/client_side_encryption</literal> directory.
+ </para>
+ <note><para>The client-side encryption feature is available natively on
+ Lustre clients running a Linux distribution with at least kernel 5.4. It
+ is also available thanks to an additional kernel library provided by
+ Lustre, on clients that run a Linux distribution with basic support for
+ encryption, including:</para>
+ <itemizedlist>
+ <listitem><para>CentOS/RHEL 8.1 and later;</para></listitem>
+ <listitem><para>Ubuntu 18.04 and later;</para></listitem>
+ <listitem><para>SLES 15 SP2 and later.</para></listitem>
+ </itemizedlist>
+ </note>
+ <section xml:id="managingSecurity.clientencryption.semantics" remap="h3">
+ <title><indexterm><primary>encryption access semantics</primary>
+ </indexterm>Client-side encryption access semantics</title>
+ <para>Only Lustre clients need access to encryption master keys. Keys are
+ added to the filesystem-level encryption keyring on the Lustre client.
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">With the key</emphasis></para>
+ <para>With the encryption key, encrypted regular files, directories,
+ and symlinks behave very similarly to their unencrypted
+ counterparts --- after all, the encryption is intended to be
+ transparent. However, astute users may notice some differences in
+ behavior:</para>
+ <itemizedlist>
+ <listitem>
+ <para>Unencrypted files, or files encrypted with a different
+ encryption policy (i.e. different key, modes, or flags),
+ cannot be renamed or linked into an encrypted directory.
+ However, encrypted files can be renamed within an encrypted
+ directory, or into an unencrypted directory.</para>
+ <note><para>"moving" an unencrypted file into an encrypted
+ directory, e.g. with the <literal>mv</literal> program, is
+ implemented in userspace by a copy followed by a delete. Be
+ aware the original unencrypted data may remain recoverable
+ from free space on the disk; it is best to keep all files
+ encrypted from the very beginning.</para></note>
+ </listitem>
+ <listitem><para>On Lustre, Direct I/O is supported for encrypted
+ files.</para>
+ </listitem>
+ <listitem><para>The <literal>fallocate()</literal> operations
+ <literal>FALLOC_FL_COLLAPSE_RANGE</literal>,
+ <literal>FALLOC_FL_INSERT_RANGE</literal>, and
+ <literal>FALLOC_FL_ZERO_RANGE</literal> are not
+ supported on encrypted files and will fail with
+ <literal>EOPNOTSUPP</literal>.
+ </para>
+ </listitem>
+ <listitem><para>DAX (Direct Access) is not supported on encrypted
+ files.</para>
+ </listitem>
+ <listitem><para condition='l2F'>The st_size of an encrypted
+ symlink will not necessarily give the length of the symlink
+ target as required by POSIX. It will actually give the length of
+ the ciphertext, which will be slightly longer than the plaintext
+ due to NUL-padding and an extra 2-byte overhead.</para>
+ </listitem>
+ <listitem><para condition='l2F'>The maximum length of an encrypted
+ symlink is 2 bytes shorter than the maximum length of an
+ unencrypted symlink.</para>
+ </listitem>
+ <listitem><para><literal>mmap</literal> is supported. This is
+ possible because the pagecache for an encrypted file contains
+ the plaintext, not the ciphertext.</para>
+ </listitem>
+ </itemizedlist>