+ <listitem>
+ <para><xref linkend="managingSecurity.clientencryption"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="managingSecurity.kerberos"/></para>
+ </listitem>
+ </itemizedlist>
+ <section xml:id="managingSecurity.acl">
+ <title><indexterm><primary>Access Control List (ACL)</primary></indexterm>
+ Using ACLs</title>
+ <para>An access control list (ACL), is a set of data that informs an
+ operating system about permissions or access rights that each user or
+ group has to specific system objects, such as directories or files. Each
+ object has a unique security attribute that identifies users who have
+ access to it. The ACL lists each object and user access privileges such as
+ read, write or execute.</para>
+ <section xml:id="managingSecurity.acl.howItWorks" remap="h3">
+ <title><indexterm><primary>Access Control List (ACL)</primary><secondary>
+ how they work</secondary></indexterm>How ACLs Work</title>
+ <para>Implementing ACLs varies between operating systems. Systems that
+ support the Portable Operating System Interface (POSIX) family of
+ standards share a simple yet powerful file system permission model,
+ which should be well-known to the Linux/UNIX administrator. ACLs add
+ finer-grained permissions to this model, allowing for more complicated
+ permission schemes. For a detailed explanation of ACLs on a Linux
+ operating system, refer to the SUSE Labs article
+ <link xl:href="https://www.usenix.org/legacyurl/posix-access-control-lists-linux">
+ Posix Access Control Lists on Linux</link>.</para>
+ <para>We have implemented ACLs according to this model. The Lustre
+ software works with the standard Linux ACL tools, setfacl, getfacl, and
+ the historical chacl, normally installed with the ACL package.</para>
+ <note>
+ <para>ACL support is a system-range feature, meaning that all clients
+ have ACL enabled or not. You cannot specify which clients should
+ enable ACL.</para>
+ </note>
+ </section>
+ <section xml:id="managingSecurity.acl.using" remap="h3">
+ <title><indexterm>
+ <primary>Access Control List (ACL)</primary>
+ <secondary>using</secondary>
+ </indexterm>Using ACLs with the Lustre Software</title>
+ <para>POSIX Access Control Lists (ACLs) can be used with the Lustre
+ software. An ACL consists of file entries representing permissions based
+ on standard POSIX file system object permissions that define three
+ classes of user (owner, group and other). Each class is associated with
+ a set of permissions [read (r), write (w) and execute (x)].</para>
+ <itemizedlist>
+ <listitem>
+ <para>Owner class permissions define access privileges of the file
+ owner.</para>
+ </listitem>
+ <listitem>
+ <para>Group class permissions define access privileges of the owning
+ group.</para>
+ </listitem>
+ <listitem>
+ <para>Other class permissions define access privileges of all users
+ not in the owner or group class.</para>
+ </listitem>
+ </itemizedlist>
+ <para>The <literal>ls -l</literal> command displays the owner, group, and
+ other class permissions in the first column of its output (for example,
+ <literal>-rw-r- --</literal> for a regular file with read and write
+ access for the owner class, read access for the group class, and no
+ access for others).</para>
+ <para>Minimal ACLs have three entries. Extended ACLs have more than the
+ three entries. Extended ACLs also contain a mask entry and may contain
+ any number of named user and named group entries.</para>
+ <para>The MDS needs to be configured to enable ACLs. Use
+ <literal>--mountfsoptions</literal> to enable ACLs when creating your
+ configuration:</para>
+ <screen>$ mkfs.lustre --fsname spfs --mountfsoptions=acl --mdt -mgs /dev/sda</screen>
+ <para>Alternately, you can enable ACLs at run time by using the
+ <literal>--acl</literal> option with <literal>mkfs.lustre</literal>:
+ </para>
+ <screen>$ mount -t lustre -o acl /dev/sda /mnt/mdt</screen>
+ <para>To check ACLs on the MDS:</para>
+ <screen>$ lctl get_param -n mdc.home-MDT0000-mdc-*.connect_flags | grep acl acl</screen>
+ <para>To mount the client with no ACLs:</para>
+ <screen>$ mount -t lustre -o noacl ibmds2@o2ib:/home /home</screen>
+ <para>ACLs are enabled in a Lustre file system on a system-wide basis;
+ either all clients enable ACLs or none do. Activating ACLs is controlled
+ by MDS mount options <literal>acl</literal> / <literal>noacl</literal>
+ (enable/disable ACLs). Client-side mount options acl/noacl are ignored.
+ You do not need to change the client configuration, and the
+ 'acl' string will not appear in the client /etc/mtab. The
+ client acl mount option is no longer needed. If a client is mounted with
+ that option, then this message appears in the MDS syslog:</para>
+ <screen>...MDS requires ACL support but client does not</screen>
+ <para>The message is harmless but indicates a configuration issue, which
+ should be corrected.</para>
+ <para>If ACLs are not enabled on the MDS, then any attempts to reference
+ an ACL on a client return an Operation not supported error.</para>
+ </section>
+ <section xml:id="managingSecurity.acl.examples" remap="h3">
+ <title><indexterm>
+ <primary>Access Control List (ACL)</primary>
+ <secondary>examples</secondary>
+ </indexterm>Examples</title>
+ <para>These examples are taken directly from the POSIX paper referenced
+ above. ACLs on a Lustre file system work exactly like ACLs on any Linux
+ file system. They are manipulated with the standard tools in the
+ standard manner. Below, we create a directory and allow a specific user
+ access.</para>
+ <screen>[root@client lustre]# umask 027
+[root@client lustre]# mkdir rain
+[root@client lustre]# ls -ld rain
+drwxr-x--- 2 root root 4096 Feb 20 06:50 rain
+[root@client lustre]# getfacl rain
+# file: rain
+# owner: root
+# group: root
+user::rwx
+group::r-x
+other::---
+
+[root@client lustre]# setfacl -m user:chirag:rwx rain
+[root@client lustre]# ls -ld rain
+drwxrwx---+ 2 root root 4096 Feb 20 06:50 rain
+[root@client lustre]# getfacl --omit-header rain
+user::rwx
+user:chirag:rwx
+group::r-x
+mask::rwx
+other::---</screen>
+ </section>
+ </section>
+ <section xml:id="managingSecurity.root_squash">
+ <title><indexterm>
+ <primary>root squash</primary>
+ </indexterm>Using Root Squash</title>
+ <para>Root squash is a security feature which restricts super-user access
+ rights to a Lustre file system. Without the root squash feature enabled,
+ Lustre file system users on untrusted clients could access or modify files
+ owned by root on the file system, including deleting them. Using the root
+ squash feature restricts file access/modifications as the root user to
+ only the specified clients. Note, however, that this does
+ <emphasis>not</emphasis> prevent users on insecure clients from accessing
+ files owned by <emphasis>other</emphasis> users.</para>
+ <para>The root squash feature works by re-mapping the user ID (UID) and
+ group ID (GID) of the root user to a UID and GID specified by the system
+ administrator, via the Lustre configuration management server (MGS). The
+ root squash feature also enables the Lustre file system administrator to
+ specify a set of client for which UID/GID re-mapping does not apply.
+ </para>
+ <note><para>Nodemaps (<xref linkend="lustrenodemap.title" />) are an
+ alternative to root squash, since it also allows root squash on a per-client
+ basis. With UID maps, the clients can even have a local root UID without
+ actually having root access to the filesystem itself.</para></note>
+ <section xml:id="managingSecurity.root_squash.config" remap="h3">
+ <title><indexterm>
+ <primary>root squash</primary>
+ <secondary>configuring</secondary>
+ </indexterm>Configuring Root Squash</title>
+ <para>Root squash functionality is managed by two configuration
+ parameters, <literal>root_squash</literal> and
+ <literal>nosquash_nids</literal>.</para>
+ <itemizedlist>
+ <listitem>
+ <para>The <literal>root_squash</literal> parameter specifies the UID
+ and GID with which the root user accesses the Lustre file system.
+ </para>
+ </listitem>
+ <listitem>
+ <para>The <literal>nosquash_nids</literal> parameter specifies the set
+ of clients to which root squash does not apply. LNet NID range
+ syntax is used for this parameter (see the NID range syntax rules
+ described in <xref linkend="managingSecurity.root_squash"/>). For
+ example:</para>
+ </listitem>
+ </itemizedlist>
+ <screen>nosquash_nids=172.16.245.[0-255/2]@tcp</screen>
+ <para>In this example, root squash does not apply to TCP clients on subnet
+ 172.16.245.0 that have an even number as the last component of their IP
+ address.</para>
+ </section>
+ <section xml:id="managingSecurity.root_squash.tuning">
+ <title><indexterm>
+ <primary>root squash</primary><secondary>enabling</secondary>
+ </indexterm>Enabling and Tuning Root Squash</title>
+ <para>The default value for <literal>nosquash_nids</literal> is NULL,
+ which means that root squashing applies to all clients. Setting the root
+ squash UID and GID to 0 turns root squash off.</para>
+ <para>Root squash parameters can be set when the MDT is created
+ (<literal>mkfs.lustre --mdt</literal>). For example:</para>
+ <screen>mds# mkfs.lustre --reformat --fsname=testfs --mdt --mgs \
+ --param "mdt.root_squash=500:501" \
+ --param "mdt.nosquash_nids='0@elan1 192.168.1.[10,11]'" /dev/sda1</screen>
+ <para>Root squash parameters can also be changed on an unmounted device
+ with <literal>tunefs.lustre</literal>. For example:</para>
+ <screen>tunefs.lustre --param "mdt.root_squash=65534:65534" \
+--param "mdt.nosquash_nids=192.168.0.13@tcp0" /dev/sda1
+</screen>
+ <para>Root squash parameters can also be changed with the
+ <literal>lctl conf_param</literal> command. For example:</para>
+ <screen>mgs# lctl conf_param testfs.mdt.root_squash="1000:101"
+mgs# lctl conf_param testfs.mdt.nosquash_nids="*@tcp"</screen>
+ <para>To retrieve the current root squash parameter settings, the
+ following <literal>lctl get_param</literal> commands can be used:</para>
+ <screen>mgs# lctl get_param mdt.*.root_squash
+mgs# lctl get_param mdt.*.nosquash_nids</screen>
+ <note>
+ <para>When using the lctl conf_param command, keep in mind:</para>
+ <itemizedlist>
+ <listitem>
+ <para><literal>lctl conf_param</literal> must be run on a live MGS
+ </para>
+ </listitem>
+ <listitem>
+ <para><literal>lctl conf_param</literal> causes the parameter to
+ change on all MDSs</para>
+ </listitem>
+ <listitem>
+ <para><literal>lctl conf_param</literal> is to be used once per a
+ parameter</para>
+ </listitem>
+ </itemizedlist>
+ </note>
+ <para>The root squash settings can also be changed temporarily with
+ <literal>lctl set_param</literal> or persistently with
+ <literal>lctl set_param -P</literal>. For example:</para>
+ <screen>mgs# lctl set_param mdt.testfs-MDT0000.root_squash="1:0"
+mgs# lctl set_param -P mdt.testfs-MDT0000.root_squash="1:0"</screen>
+ <para>The <literal>nosquash_nids</literal> list can be cleared with:</para>
+ <screen>mgs# lctl conf_param testfs.mdt.nosquash_nids="NONE"</screen>
+ <para>- OR -</para>
+ <screen>mgs# lctl conf_param testfs.mdt.nosquash_nids="clear"</screen>
+ <para>If the <literal>nosquash_nids</literal> value consists of several
+ NID ranges (e.g. <literal>0@elan</literal>, <literal>1@elan1</literal>),
+ the list of NID ranges must be quoted with single (') or double
+ ('') quotation marks. List elements must be separated with a
+ space. For example:</para>
+ <screen>mds# mkfs.lustre ... --param "mdt.nosquash_nids='0@elan1 1@elan2'" /dev/sda1
+lctl conf_param testfs.mdt.nosquash_nids="24@elan 15@elan1"</screen>
+ <para>These are examples of incorrect syntax:</para>
+ <screen>mds# mkfs.lustre ... --param "mdt.nosquash_nids=0@elan1 1@elan2" /dev/sda1
+lctl conf_param testfs.mdt.nosquash_nids=24@elan 15@elan1</screen>
+ <para>To check root squash parameters, use the lctl get_param command:
+ </para>
+ <screen>mds# lctl get_param mdt.testfs-MDT0000.root_squash
+lctl get_param mdt.*.nosquash_nids</screen>
+ <note>
+ <para>An empty nosquash_nids list is reported as NONE.</para>
+ </note>
+ </section>
+ <section xml:id="managingSecurity.root_squash.tips" remap="h3">
+ <title><indexterm>
+ <primary>root squash</primary>
+ <secondary>tips</secondary>
+ </indexterm>Tips on Using Root Squash</title>
+ <para>Lustre configuration management limits root squash in several ways.
+ </para>
+ <itemizedlist>
+ <listitem>
+ <para>The <literal>lctl conf_param</literal> value overwrites the
+ parameter's previous value. If the new value uses an incorrect
+ syntax, then the system continues with the old parameters and the
+ previously-correct value is lost on remount. That is, be careful
+ doing root squash tuning.</para>
+ </listitem>
+ <listitem>
+ <para><literal>mkfs.lustre</literal> and
+ <literal>tunefs.lustre</literal> do not perform parameter syntax
+ checking. If the root squash parameters are incorrect, they are
+ ignored on mount and the default values are used instead.</para>
+ </listitem>
+ <listitem>
+ <para>Root squash parameters are parsed with rigorous syntax checking.
+ The root_squash parameter should be specified as
+ <literal><decnum>:<decnum></literal>. The
+ <literal>nosquash_nids</literal> parameter should follow LNet NID
+ range list syntax.</para>
+ </listitem>
+ </itemizedlist>
+ <para>LNet NID range syntax:</para>
+ <screen><nidlist> :== <nidrange> [ ' ' <nidrange> ]
+<nidrange> :== <addrrange> '@' <net>
+<addrrange> :== '*' |
+ <ipaddr_range> |
+ <numaddr_range>
+<ipaddr_range> :==
+<numaddr_range>.<numaddr_range>.<numaddr_range>.<numaddr_range>
+<numaddr_range> :== <number> |
+ <expr_list>
+<expr_list> :== '[' <range_expr> [ ',' <range_expr>] ']'
+<range_expr> :== <number> |
+ <number> '-' <number> |
+ <number> '-' <number> '/' <number>
+<net> :== <netname> | <netname><number>
+<netname> :== "lo" | "tcp" | "o2ib"
+ | "ra" | "elan"
+<number> :== <nonnegative decimal> | <hexadecimal></screen>
+ <note>
+ <para>For networks using numeric addresses (e.g. elan), the address
+ range must be specified in the
+ <literal><numaddr_range></literal> syntax. For networks using
+ IP addresses, the address range must be in the
+ <literal><ipaddr_range></literal>. For example, if elan is using
+ numeric addresses, <literal>1.2.3.4@elan</literal> is incorrect.
+ </para>
+ </note>
+ </section>
+ </section>
+ <section xml:id="managingSecurity.isolation">
+ <title><indexterm><primary>Isolation</primary></indexterm>
+ Isolating Clients to a Sub-directory Tree</title>
+ <para>Isolation is the Lustre implementation of the generic concept of
+ multi-tenancy, which aims at providing separated namespaces from a single
+ filesystem. Lustre Isolation enables different populations of users on
+ the same file system beyond normal Unix permissions/ACLs, even when users
+ on the clients may have root access. Those tenants share the same file
+ system, but they are isolated from each other: they cannot access or even
+ see each other’s files, and are not aware that they are sharing common
+ file system resources.</para>
+ <para>Lustre Isolation leverages the Fileset feature
+ (<xref linkend="SystemConfigurationUtilities.fileset" />)
+ to mount only a subdirectory of the filesystem rather than the root
+ directory.
+ In order to achieve isolation, the subdirectory mount, which presents to
+ tenants only their own fileset, has to be imposed to the clients. To that
+ extent, we make use of the nodemap feature
+ (<xref linkend="lustrenodemap.title" />). We group all clients used by a
+ tenant under a common nodemap entry, and we assign to this nodemap entry
+ the fileset to which the tenant is restricted.</para>
+ <section xml:id="managingSecurity.isolation.clientid" remap="h3">
+ <title><indexterm><primary>Isolation</primary><secondary>
+ client identification</secondary></indexterm>Identifying Clients</title>
+ <para>Enforcing multi-tenancy on Lustre relies on the ability to properly
+ identify the client nodes used by a tenant, and trust those identities.
+ This can be achieved by having physical hardware and/or network
+ security, so that client nodes have well-known NIDs. It is also possible
+ to make use of strong authentication with Kerberos or Shared-Secret Key
+ (see <xref linkend="lustressk" />).
+ Kerberos prevents NID spoofing, as every client needs its own
+ credentials, based on its NID, in order to connect to the servers.
+ Shared-Secret Key also prevents tenant impersonation, because keys
+ can be linked to a specific nodemap. See
+ <xref linkend="ssknodemaprole" /> for detailed explanations.
+</para>
+ </section>
+ <section xml:id="managingSecurity.isolation.configuring" remap="h3">
+ <title><indexterm><primary>Isolation</primary><secondary>
+ configuring</secondary></indexterm>Configuring Isolation</title>
+ <para>Isolation on Lustre can be achieved by setting the
+ <literal>fileset</literal> parameter on a nodemap entry. All clients
+ belonging to this nodemap entry will automatically mount this fileset
+ instead of the root directory. For example:</para>
+ <screen>mgs# lctl nodemap_set_fileset --name tenant1 --fileset '/dir1'</screen>
+ <para>So all clients matching the <literal>tenant1</literal> nodemap will
+ be automatically presented the fileset <literal>/dir1</literal> when
+ mounting. This means these clients are doing an implicit subdirectory
+ mount on the subdirectory <literal>/dir1</literal>.
+ </para>
+ <note>
+ <para>
+ If subdirectory defined as fileset does not exist on the file system,
+ it will prevent any client belonging to the nodemap from mounting
+ Lustre.
+ </para>
+ </note>
+ <para>To delete the fileset parameter, just set it to an empty string:
+ </para>
+ <screen>mgs# lctl nodemap_set_fileset --name tenant1 --fileset ''</screen>
+ </section>
+ <section xml:id="managingSecurity.isolation.permanent" remap="h3">
+ <title><indexterm><primary>Isolation</primary><secondary>
+ making permanent</secondary></indexterm>Making Isolation Permanent
+ </title>
+ <para>In order to make isolation permanent, the fileset parameter on the
+ nodemap has to be set with <literal>lctl set_param</literal> with the
+ <literal>-P</literal> option.</para>
+ <screen>mgs# lctl set_param nodemap.tenant1.fileset=/dir1
+mgs# lctl set_param -P nodemap.tenant1.fileset=/dir1</screen>
+ <para>This way the fileset parameter will be stored in the Lustre config
+ logs, letting the servers retrieve the information after a restart.
+ </para>
+ </section>
+ </section>
+ <section xml:id="managingSecurity.sepol" condition='l2D'>
+ <title><indexterm><primary>selinux policy check</primary></indexterm>
+ Checking SELinux Policy Enforced by Lustre Clients</title>
+ <para>SELinux provides a mechanism in Linux for supporting Mandatory Access
+ Control (MAC) policies. When a MAC policy is enforced, the operating
+ system’s (OS) kernel defines application rights, firewalling applications
+ from compromising the entire system. Regular users do not have the ability to
+ override the policy.</para>
+ <para>One purpose of SELinux is to protect the
+ <emphasis role="bold">OS</emphasis> from privilege escalation. To that
+ extent, SELinux defines confined and unconfined domains for processes and
+ users. Each process, user, file is assigned a security context, and
+ rules define the allowed operations by processes and users on files.
+ </para>
+ <para>Another purpose of SELinux can be to protect
+ <emphasis role="bold">data</emphasis> sensitivity, thanks to Multi-Level
+ Security (MLS). MLS works on top of SELinux, by defining the concept of
+ security levels in addition to domains. Each process, user and file is
+ assigned a security level, and the model states that processes and users
+ can read the same or lower security level, but can only write to their own
+ or higher security level.
+ </para>
+ <para>From a file system perspective, the security context of files must be
+ stored permanently. Lustre makes use of the
+ <literal>security.selinux</literal> extended attributes on files to hold
+ this information. Lustre supports SELinux on the client side. All you have
+ to do to have MAC and MLS on Lustre is to enforce the appropriate SELinux
+ policy (as provided by the Linux distribution) on all Lustre clients. No
+ SELinux is required on Lustre servers.
+ </para>
+ <para>Because Lustre is a distributed file system, the specificity when
+ using MLS is that Lustre really needs to make sure data is always accessed
+ by nodes with the SELinux MLS policy properly enforced. Otherwise, data is
+ not protected. This means Lustre has to check that SELinux is properly
+ enforced on client side, with the right, unaltered policy. And if SELinux
+ is not enforced as expected on a client, the server denies its access to
+ Lustre.
+ </para>
+ <section xml:id="managingSecurity.sepol.determining" remap="h3">
+ <title><indexterm><primary>selinux policy check</primary><secondary>
+ determining</secondary></indexterm>Determining SELinux Policy Info
+ </title>
+ <para>A string that represents the SELinux Status info will be used by
+ servers as a reference, to check if clients are enforcing SELinux
+ properly. This reference string can be obtained on a client node known
+ to enforce the right SELinux policy, by calling the
+ <literal>l_getsepol</literal> command line utility:</para>
+ <screen>client# l_getsepol
+SELinux status info: 1:mls:31:40afb76d077c441b69af58cccaaa2ca63641ed6e21b0a887dc21a684f508b78f</screen>
+ <para>The string describing the SELinux policy has the following
+ syntax:</para>
+ <para><literal>mode:name:version:hash</literal></para>
+ <para>where:</para>
+ <itemizedlist>
+ <listitem>
+ <para><literal>mode</literal> is a digit telling if SELinux is in
+ Permissive mode (0) or Enforcing mode (1)</para>
+ </listitem>
+ <listitem>
+ <para><literal>name</literal> is the name of the SELinux policy
+ </para>
+ </listitem>
+ <listitem>
+ <para><literal>version</literal> is the version of the SELinux
+ policy</para>
+ </listitem>
+ <listitem>
+ <para><literal>hash</literal> is the computed hash of the binary
+ representation of the policy, as exported in
+ /etc/selinux/<literal>name</literal>/policy/policy.
+ <literal>version</literal></para>
+ </listitem>
+ </itemizedlist>
+ </section>
+ <section xml:id="managingSecurity.sepol.configuring" remap="h3">
+ <title><indexterm><primary>selinux policy check</primary><secondary>
+ enforcing</secondary></indexterm>Enforcing SELinux Policy Check</title>
+ <para>SELinux policy check can be enforced by setting the
+ <literal>sepol</literal> parameter on a nodemap entry. All clients
+ belonging to this nodemap entry must enforce the SELinux policy
+ described by this parameter, otherwise they are denied access to the
+ Lustre file system. For example:</para>
+ <screen>mgs# lctl nodemap_set_sepol --name restricted
+ --sepol '1:mls:31:40afb76d077c441b69af58cccaaa2ca63641ed6e21b0a887dc21a684f508b78f'</screen>
+ <para>So all clients matching the <literal>restricted</literal> nodemap
+ must enforce the SELinux policy which description matches
+ <literal>1:mls:31:40afb76d077c441b69af58cccaaa2ca63641ed6e21b0a887dc21a684f508b78f</literal>.
+ If not, they will get Permission Denied when trying to mount or access
+ files on the Lustre file system.</para>
+ <para>To delete the <literal>sepol</literal> parameter, just set it to an
+ empty string:</para>
+ <screen>mgs# lctl nodemap_set_sepol --name restricted --sepol ''</screen>
+ <para>See <xref linkend="lustrenodemap.title" /> for more details about
+ the Nodemap feature.</para>
+ </section>
+ <section xml:id="managingSecurity.sepol.permanent" remap="h3">
+ <title><indexterm><primary>selinux policy check</primary><secondary>
+ making permanent</secondary></indexterm>Making SELinux Policy Check
+ Permanent</title>
+ <para>In order to make SELinux Policy check permanent, the sepol parameter
+ on the nodemap has to be set with <literal>lctl set_param</literal> with
+ the <literal>-P</literal> option.</para>
+ <screen>mgs# lctl set_param nodemap.restricted.sepol=1:mls:31:40afb76d077c441b69af58cccaaa2ca63641ed6e21b0a887dc21a684f508b78f
+mgs# lctl set_param -P nodemap.restricted.sepol=1:mls:31:40afb76d077c441b69af58cccaaa2ca63641ed6e21b0a887dc21a684f508b78f</screen>
+ <para>This way the sepol parameter will be stored in the Lustre config
+ logs, letting the servers retrieve the information after a restart.
+ </para>
+ </section>
+ <section xml:id="managingSecurity.sepol.client" remap="h3">
+ <title><indexterm><primary>selinux policy check</primary><secondary>
+ sending client</secondary></indexterm>Sending SELinux Status Info from
+ Clients</title>
+ <para>In order for Lustre clients to send their SELinux status
+ information, in case SELinux is enabled locally, the
+ <literal>send_sepol</literal> ptlrpc kernel module's parameter has to be
+ set to a non-zero value. <literal>send_sepol</literal> accepts various
+ values:</para>
+ <itemizedlist>
+ <listitem>
+ <para>0: do not send SELinux policy info;</para>
+ </listitem>
+ <listitem>
+ <para>-1: fetch SELinux policy info for every request;</para>
+ </listitem>
+ <listitem>
+ <para>N > 0: only fetch SELinux policy info every N seconds. Use
+ <literal>N = 2^31-1</literal> to have SELinux policy info
+ fetched only at mount time.</para>
+ </listitem>
+ </itemizedlist>
+ <para>Clients that are part of a nodemap on which
+ <literal>sepol</literal> is defined must send SELinux status info.
+ And the SELinux policy they enforce must match the representation
+ stored into the nodemap. Otherwise they will be denied access to the
+ Lustre file system.</para>
+ </section>
+ </section>
+ <section xml:id="managingSecurity.clientencryption" condition='l2E'>
+ <title><indexterm><primary>Client-side encryption</primary></indexterm>
+ Encrypting files and directories</title>
+ <para>The purpose that client-side encryption wants to serve is to be able
+ to provide a special directory for each user, to safely store sensitive
+ files. The goals are to protect data in transit between clients and
+ servers, and protect data at rest.</para>
+ <para>This feature is implemented directly at the Lustre client level.
+ Lustre client-side encryption relies on kernel <literal>fscrypt</literal>.
+ <literal>fscrypt</literal> is a library which filesystems can hook into to
+ support transparent encryption of files and directories. As a consequence,
+ the key points described below are extracted from
+ <literal>fscrypt</literal> documentation.</para>
+ <para>For full details, please refer to documentation available with the
+ Lustre sources, under the
+ <literal>Documentation/client_side_encryption</literal> directory.
+ </para>
+ <note><para>The client-side encryption feature is available natively on
+ Lustre clients running a Linux distribution with at least kernel 5.4. It
+ is also available thanks to an additional kernel library provided by
+ Lustre, on clients that run a Linux distribution with basic support for
+ encryption, including:</para>
+ <itemizedlist>
+ <listitem><para>CentOS/RHEL 8.1 and later;</para></listitem>
+ <listitem><para>Ubuntu 18.04 and later;</para></listitem>
+ <listitem><para>SLES 15 SP2 and later.</para></listitem>
+ </itemizedlist>
+ </note>
+ <section xml:id="managingSecurity.clientencryption.semantics" remap="h3">
+ <title><indexterm><primary>encryption access semantics</primary>
+ </indexterm>Client-side encryption access semantics</title>
+ <para>Only Lustre clients need access to encryption master keys. Keys are
+ added to the filesystem-level encryption keyring on the Lustre client.
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">With the key</emphasis></para>
+ <para>With the encryption key, encrypted regular files, directories,
+ and symlinks behave very similarly to their unencrypted
+ counterparts --- after all, the encryption is intended to be
+ transparent. However, astute users may notice some differences in
+ behavior:</para>
+ <itemizedlist>
+ <listitem>
+ <para>Unencrypted files, or files encrypted with a different
+ encryption policy (i.e. different key, modes, or flags),
+ cannot be renamed or linked into an encrypted directory.
+ However, encrypted files can be renamed within an encrypted
+ directory, or into an unencrypted directory.</para>
+ <note><para>"moving" an unencrypted file into an encrypted
+ directory, e.g. with the <literal>mv</literal> program, is
+ implemented in userspace by a copy followed by a delete. Be
+ aware the original unencrypted data may remain recoverable
+ from free space on the disk; it is best to keep all files
+ encrypted from the very beginning.</para></note>
+ </listitem>
+ <listitem><para>On Lustre, Direct I/O is supported for encrypted
+ files.</para>
+ </listitem>
+ <listitem><para>The <literal>fallocate()</literal> operations
+ <literal>FALLOC_FL_COLLAPSE_RANGE</literal>,
+ <literal>FALLOC_FL_INSERT_RANGE</literal>, and
+ <literal>FALLOC_FL_ZERO_RANGE</literal> are not
+ supported on encrypted files and will fail with
+ <literal>EOPNOTSUPP</literal>.
+ </para>
+ </listitem>
+ <listitem><para>DAX (Direct Access) is not supported on encrypted
+ files.</para>
+ </listitem>
+ <listitem><para condition='l2F'>The st_size of an encrypted
+ symlink will not necessarily give the length of the symlink
+ target as required by POSIX. It will actually give the length of
+ the ciphertext, which will be slightly longer than the plaintext
+ due to NUL-padding and an extra 2-byte overhead.</para>
+ </listitem>
+ <listitem><para condition='l2F'>The maximum length of an encrypted
+ symlink is 2 bytes shorter than the maximum length of an
+ unencrypted symlink.</para>
+ </listitem>
+ <listitem><para><literal>mmap</literal> is supported. This is
+ possible because the pagecache for an encrypted file contains
+ the plaintext, not the ciphertext.</para>
+ </listitem>
+ </itemizedlist>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Without the key</emphasis></para>
+ <para>Some filesystem operations may be performed on encrypted
+ regular files, directories, and symlinks even before their
+ encryption key has been added, or after their encryption key has
+ been removed:</para>
+ <itemizedlist>
+ <listitem>
+ <para>File metadata may be read, e.g. using
+ <literal>stat()</literal>.</para>
+ </listitem>
+ <listitem>
+ <para condition='l2F'>Directories may be listed, in which case
+ the filenames will be listed in an encoded form derived from
+ their ciphertext. The algorithm is subject to change but it is
+ guaranteed that the presented filenames will be no longer than
+ NAME_MAX bytes, will not contain the <literal>/</literal> or
+ <literal>\0</literal> characters, and will uniquely identify
+ directory entries. The <literal>.</literal> and
+ <literal>..</literal> directory entries are special. They are
+ always present and are not encrypted or encoded.</para>
+ </listitem>
+ <listitem>
+ <para>Files may be deleted. That is, nondirectory files may be
+ deleted with <literal>unlink()</literal> as usual, and empty
+ directories may be deleted with <literal>rmdir()</literal> as
+ usual. Therefore, <literal>rm</literal> and
+ <literal>rm -r</literal> will work as expected.</para>
+ </listitem>
+ <listitem>
+ <para>Symlink targets may be read and followed, but they will
+ be presented in encrypted form, similar to filenames in
+ directories. Hence, they are unlikely to point to anywhere
+ useful.</para>
+ </listitem>
+ </itemizedlist>
+ <para>Without the key, regular files cannot be opened or truncated.
+ Attempts to do so will fail with <literal>ENOKEY</literal>. This
+ implies that any regular file operations that require a file
+ descriptor, such as <literal>read()</literal>,
+ <literal>write()</literal>, <literal>mmap()</literal>,
+ <literal>fallocate()</literal>, and <literal>ioctl()</literal>,
+ are also forbidden.</para>
+ <para>Also without the key, files of any type (including
+ directories) cannot be created or linked into an encrypted
+ directory, nor can a name in an encrypted directory be the source
+ or target of a rename, nor can an <literal>O_TMPFILE</literal>
+ temporary file be created in an encrypted directory. All such
+ operations will fail with <literal>ENOKEY</literal>.</para>
+ <para>It is not currently possible to backup and restore encrypted
+ files without the encryption key. This would require special
+ APIs which have not yet been implemented.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Encryption policy enforcement
+ </emphasis></para>
+ <para>After an encryption policy has been set on a directory, all
+ regular files, directories, and symbolic links created in that
+ directory (recursively) will inherit that encryption policy.
+ Special files --- that is, named pipes, device nodes, and UNIX
+ domain sockets --- will not be encrypted.</para>
+ <para>Except for those special files, it is forbidden to have
+ unencrypted files, or files encrypted with a different encryption
+ policy, in an encrypted directory tree.</para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ </section>
+ <section xml:id="managingSecurity.clientencryption.keyhierarchy" remap="h3">
+ <title><indexterm><primary>encryption key hierarchy</primary>
+ </indexterm>Client-side encryption key hierarchy</title>
+ <para>Each encrypted directory tree is protected by a master key.</para>
+ <para>To "unlock" an encrypted directory tree, userspace must provide the
+ appropriate master key. There can be any number of master keys, each
+ of which protects any number of directory trees on any number of
+ filesystems.</para>
+ </section>
+ <section xml:id="managingSecurity.clientencryption.modes" remap="h3">
+ <title><indexterm><primary>encryption modes usage</primary>
+ </indexterm>Client-side encryption modes and usage</title>
+ <para><literal>fscrypt</literal> allows one encryption mode to be
+ specified for file contents and one encryption mode to be specified for
+ filenames. Different directory trees are permitted to use different
+ encryption modes. Currently, the following pairs of encryption modes are
+ supported:</para>
+ <itemizedlist>
+ <listitem>
+ <para>AES-256-XTS for contents and AES-256-CTS-CBC for filenames
+ </para>
+ </listitem>
+ <listitem>
+ <para>AES-128-CBC for contents and AES-128-CTS-CBC for filenames
+ </para>
+ </listitem>
+ </itemizedlist>
+ <para>If unsure, you should use the (AES-256-XTS, AES-256-CTS-CBC) pair.
+ </para>
+ <warning><para>In Lustre 2.14, client-side encryption only supports
+ content encryption, and not filename encryption. As a consequence, only
+ content encryption mode will be taken into account, and filename
+ encryption mode will be ignored to leave filenames in clear text.</para>
+ </warning>
+ <warning><para condition='l2F'>In Lustre 2.15, filename encryption mode
+ will be taken into account for new files and directories, if they are
+ under a parent encrypted directory created with Lustre 2.15. This means
+ new files and directories under a parent encrypted directory created with
+ Lustre 2.14 will not have their names encrypted.
+ Also, because files created with Lustre 2.14 did not have their names
+ encrypted, they will remain so after upgrade to 2.15.</para>
+ </warning>
+ </section>
+ <section xml:id="managingSecurity.clientencryption.threatmodel" remap="h3">
+ <title><indexterm><primary>encryption threat model</primary>
+ </indexterm>Client-side encryption threat model</title>
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="bold">Offline attacks</emphasis></para>
+ <para>For the Lustre case, block devices are Lustre targets attached
+ to the Lustre servers. Manipulating the filesystem offline means
+ accessing the filesystem on these targets while Lustre is offline.
+ </para>
+ <para>Provided that a strong encryption key is chosen,
+ <literal>fscrypt</literal> protects the confidentiality of file
+ contents in the event of a single point-in-time permanent offline
+ compromise of the block device content.
+ Lustre client-side encryption does not protect the confidentiality
+ of metadata, e.g. file names, file sizes, file permissions, file
+ timestamps, and extended attributes. Also, the existence and
+ location of holes (unallocated blocks which logically contain all
+ zeroes) in files is not protected.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">Online attacks</emphasis></para>
+ <itemizedlist>
+ <listitem>
+ <para>On Lustre client</para>
+ <para>After an encryption key has been added,
+ <literal>fscrypt</literal> does not hide the plaintext file
+ contents or filenames from other users on the same node.
+ Instead, existing access control mechanisms such as file mode
+ bits, POSIX ACLs, LSMs, or namespaces should be used for this
+ purpose.</para>
+ <para>For the Lustre case, it means plaintext file contents or
+ filenames are not hidden from other users on the same Lustre
+ client.</para>
+ <para>An attacker who compromises the system enough to read from
+ arbitrary memory, e.g. by exploiting a kernel security
+ vulnerability, can compromise all encryption keys that are
+ currently in use.
+ However, <literal>fscrypt</literal> allows encryption keys to
+ be removed from the kernel, which may protect them from later
+ compromise. Key removal can be carried out by non-root users.
+ In more detail, the key removal will wipe the master encryption
+ key from kernel memory. Moreover, it will try to evict all
+ cached inodes which had been "unlocked" using the key, thereby
+ wiping their per-file keys and making them once again appear
+ "locked", i.e. in ciphertext or encrypted form.</para>
+ </listitem>
+ <listitem>
+ <para>On Lustre server</para>
+ <para>An attacker on a Lustre server who compromises the system
+ enough to read arbitrary memory, e.g. by exploiting a kernel
+ security vulnerability, cannot compromise Lustre files content.
+ Indeed, encryption keys are not forwarded to the Lustre servers,
+ and servers do not carry out decryption or encryption.
+ Moreover, bulk RPCs received by servers contain encrypted data,
+ which is written as-is to the underlying filesystem.</para>
+ </listitem>
+ </itemizedlist>
+ </listitem>
+ </itemizedlist>
+ </section>
+ <section xml:id="managingSecurity.clientencryption.fscrypt" remap="h3">
+ <title><indexterm><primary>encryption fscrypt policy</primary>
+ </indexterm>Manage encryption on directories</title>
+ <para>By default, Lustre client-side encryption is enabled, letting users
+ define encryption policies on a per-directory basis.</para>
+ <note><para>Administrators can decide to prevent a Lustre client
+ mount-point from using encryption by specifying the
+ <literal>noencrypt</literal> client mount option. This can be also
+ enforced from server side thanks to the
+ <literal>forbid_encryption</literal> property on nodemaps. See
+ <xref linkend="alteringproperties"/> for how to manage nodemaps.
+ </para></note>
+ <para><literal>fscrypt</literal> userspace tool can be used to manage
+ encryption policies. See https://github.com/google/fscrypt for
+ comprehensive explanations. Below are examples on how to use this tool
+ with Lustre. If not told otherwise, commands must be run on Lustre
+ client side.</para>
+ <itemizedlist>
+ <listitem>
+ <para>Two preliminary steps are required before actually deciding
+ which directories to encrypt, and this is the only
+ functionality which requires root privileges. Administrator has to
+ run:</para>
+ <screen># fscrypt setup
+Customizing passphrase hashing difficulty for this system...
+Created global config file at "/etc/fscrypt.conf".
+Metadata directories created at "/.fscrypt".</screen>
+ <para>This first command has to be run on all clients that want to use
+ encryption, as it sets up global fscrypt parameters outside of
+ Lustre.</para>
+ <screen># fscrypt setup /mnt/lustre
+Metadata directories created at "/mnt/lustre/.fscrypt"</screen>
+ <para>This second command has to be run on just one Lustre
+ client.</para>
+ <note><para>The file <literal>/etc/fscrypt.conf</literal> can be
+ edited. It is strongly recommended to set
+ <literal>policy_version</literal> to 2, so that
+ <literal>fscrypt</literal> wipes files from memory when the
+ encryption key is removed.</para></note>
+ </listitem>
+ <listitem>
+ <para>Now a regular user is able to select a directory to
+ encrypt:</para>
+ <screen>$ fscrypt encrypt /mnt/lustre/vault
+The following protector sources are available:
+1 - Your login passphrase (pam_passphrase)
+2 - A custom passphrase (custom_passphrase)
+3 - A raw 256-bit key (raw_key)
+Enter the source number for the new protector [2 - custom_passphrase]: 2
+Enter a name for the new protector: shield
+Enter custom passphrase for protector "shield":
+Confirm passphrase:
+"/mnt/lustre/vault" is now encrypted, unlocked, and ready for use.</screen>
+ <para>Starting from here, all files and directories created under
+ <literal>/mnt/lustre/vault</literal> will be encrypted, according
+ to the policy defined at the previsous step.</para>
+ <note><para>The encryption policy is inherited by all subdirectories.
+ It is not possible to change the policy for a subdirectory.</para>
+ </note>
+ </listitem>
+ <listitem>
+ <para>Another user can decide to encrypt a different directory with
+ its own protector:</para>
+ <screen>$ fscrypt encrypt /mnt/lustre/private
+Should we create a new protector? [y/N] Y
+The following protector sources are available:
+1 - Your login passphrase (pam_passphrase)
+2 - A custom passphrase (custom_passphrase)
+3 - A raw 256-bit key (raw_key)
+Enter the source number for the new protector [2 - custom_passphrase]: 2
+Enter a name for the new protector: armor
+Enter custom passphrase for protector "armor":
+Confirm passphrase:
+"/mnt/lustre/private" is now encrypted, unlocked, and ready for use.</screen>
+ </listitem>
+ <listitem>
+ <para>Users can decide to lock an encrypted directory at any
+ time:</para>
+ <screen>$ fscrypt lock /mnt/lustre/vault
+"/mnt/lustre/vault" is now locked.</screen>
+ <para>This action prevents access to encrypted content, and by
+ removing the key from memory, it also wipes files from memory if
+ they are not still open.</para>
+ </listitem>
+ <listitem>
+ <para>Users regain access to the encrypted directory with the command:
+ </para>
+ <screen>$ fscrypt unlock /mnt/lustre/vault
+Enter custom passphrase for protector "shield":
+"/mnt/lustre/vault" is now unlocked and ready for use.</screen>
+ </listitem>
+ <listitem>
+ <para>Actually, <literal>fscrypt</literal> does not give direct access
+ to master keys, but to protectors that are used to encrypt them.
+ This mechanism gives the ability to change a passphrase:</para>
+ <screen>$ fscrypt status /mnt/lustre
+lustre filesystem "/mnt/lustre" has 2 protectors and 2 policies