From 2c82bcd0e7356e9c228a95800789febec5fdc6b7 Mon Sep 17 00:00:00 2001 From: Sebastien Buisson Date: Fri, 5 Feb 2021 14:32:10 +0100 Subject: [PATCH] LUDOC-322 sec: reinstate Kerberos doc This patch adds documentation for Kerberos support in Lustre. This is adapted and updated from the original Lustre user manual. This doc is added under the Managing Security in a Lustre File System section. Signed-off-by: Sebastien Buisson Change-Id: I430b99d2d11e034347021337b00e063158d7ef4f Reviewed-on: https://review.whamcloud.com/41425 Tested-by: jenkins Reviewed-by: Andreas Dilger --- ManagingSecurity.xml | 583 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 583 insertions(+) diff --git a/ManagingSecurity.xml b/ManagingSecurity.xml index 95ecd24..4d2b16e 100644 --- a/ManagingSecurity.xml +++ b/ManagingSecurity.xml @@ -21,6 +21,9 @@ + + +
<indexterm><primary>Access Control List (ACL)</primary></indexterm> @@ -926,6 +929,586 @@ f3cc1b5cf9b8f41c No custom protector "bunker"</screen> </itemizedlist> </section> </section> + <section xml:id="managingSecurity.kerberos"> + <title><indexterm><primary>Kerberos</primary></indexterm> + Configuring Kerberos (KRB) Security + This chapter describes how to use Kerberos with Lustre. +
+ What Is Kerberos? + Kerberos is a mechanism for authenticating all entities (such as + users and servers) on an "unsafe" network. Each of these + entities, known as "principals", negotiate a runtime key with + the Kerberos server. This key enables principals to verify that messages + from the Kerberos server are authentic. By trusting the Kerberos server, + users and services can authenticate one another. + Setting up Lustre with Kerberos can provide advanced security + protections for the Lustre network. Broadly, Kerberos offers three types + of benefit: + + + Allows Lustre connection peers (MDS, OSS and clients) to + authenticate one another. + + + Protects the integrity of PTLRPC messages from being modified + during network transfer. + + + Protects the privacy of the PTLRPC message from being + eavesdropped during network transfer. + + + Kerberos uses the “kernel keyring” client upcall mechanism. +
+
+ Security Flavor + + A security flavor is a string to describe what kind authentication + and data transformation be performed upon a PTLRPC connection. It + covers both RPC message and BULK data. + + + The supported flavors are described in following table: + + + + + + + + + + + + Base Flavor + + + Authentication + + + RPC Message Protection + + + Bulk Data Protection + + + Notes + + + + + + + null + + + N/A + + + N/A + + + N/A + + + + + krb5n + + + GSS/Kerberos5 + + + null + + + checksum + + + No protection of RPC message, checksum protection + of bulk data, light performance overhead. + + + + + krb5a + + + GSS/Kerberos5 + + + partial integrity (krb5) + + + checksum + + + Only header of RPC message is integrity protected, and + checksum protection of bulk data, more performance + overhead compare to krb5n. + + + + + krb5i + + + GSS/Kerberos5 + + + integrity (krb5) + + + integrity (krb5) + + + transformation algorithm is determined by actual Kerberos + algorithms enforced by KDC and principals; heavy performance + penalty. + + + + + krb5p + + + GSS/Kerberos5 + + + privacy (krb5) + + + privacy (krb5) + + + transformation privacy protection algorithm is determined + by actual Kerberos algorithms enforced by KDC and principals; + the heaviest performance penalty. + + + + + +
+
+ Kerberos Setup +
+ Distribution + We only support MIT Kerberos 5, from version 1.3. + For environmental requirements in general, and clock + synchronization in particular, please refer to section + . +
+
+ Principals Configuration + + + Configure client nodes: + + + + For each client node, create a lustre_root + principal and generate keytab. + + kadmin> addprinc -randkey lustre_root/client_host.domain@REALM + kadmin> ktadd lustre_root/client_host.domain@REALM + + + + Install the keytab on the client node. + + + + + + Configure MGS nodes: + + + + For each MGS node, create a lustre_mgs + principal and generate keytab. + + kadmin> addprinc -randkey lustre_mgs/mgs_host.domain@REALM + kadmin> ktadd lustre_mds/mgs_host.domain@REALM + + + + Install the keytab on the MGS nodes. + + + + + + Configure MDS nodes: + + + + For each MDS node, create a lustre_mds + principal and generate keytab. + + kadmin> addprinc -randkey lustre_mds/mds_host.domain@REALM + kadmin> ktadd lustre_mds/mds_host.domain@REALM + + + + Install the keytab on the MDS nodes. + + + + + + Configure OSS nodes: + + + + For each OSS node, create a lustre_oss + principal and generate keytab. + + kadmin> addprinc -randkey lustre_oss/oss_host.domain@REALM + kadmin> ktadd lustre_oss/oss_host.domain@REALM + + + + Install the keytab on the client node. + + + + + + + + + The host.domain should be the FQDN in + your network, otherwise server might not recognize any GSS + request. + + + + As an alternative for the client keytab, if you want to save + the trouble of assigning unique keytab for each client node, + you can create a general lustre_root principal and its + keytab, and install the same keytab on as many client nodes + as you want. Be aware that in + this way one compromised client means all clients are + insecure. + + kadmin> addprinc -randkey lustre_root@REALM + kadmin> ktadd lustre_root@REALM + + + + Lustre support following enctypes for + MIT Kerberos 5 version 1.3 or higher: + + + + + aes128-cts + + + + + aes256-cts + + + + + + +
+
+
+ Networking + On networks for which name resolution to IP address is possible, + like TCP or InfiniBand, the names used in the principals must be the + ones that resolve to the IP addresses used by the Lustre NIDs. + If you are using a network which is + NOT TCP or InfiniBand (e.g. + PTL4LND), you need to have a /etc/lustre/nid2hostname + script on each node, which purpose is + to translate NID into hostname. + Following is a possible example for PTL4LND: + #!/bin/bash +set -x + +# convert a NID for a LND to a hostname + +# called with thre arguments: lnd netid nid +# $lnd is the string "PTL4LND", etc. +# $netid is the network identifier in hex string format +# $nid is the NID in hex format +# output the corresponding hostname, +# or error message leaded by a '@' for error logging. + +lnd=$1 +netid=$2 +# convert hex NID number to decimal +nid=$((0x$3)) + +case $lnd in + PTL4LND) # simply add 'node' at the beginning + echo "node$nid" + ;; + *) + echo "@unknown LND: $lnd" + ;; +esac +
+
+ Required packages + + Every node should have following packages installed: + + + + krb5-workstation + + + krb5-libs + + + keyutils + + + keyutils-libs + + + On the node used to build Lustre with GSS support, following + packages should be installed: + + + krb5-devel + + + keyutils-libs-devel + + +
+
+ Build Lustre + + Enable GSS at configuration time: + + ./configure --enable-gss --other-options +
+
+ Running +
+ GSS Daemons + + Make sure to start the daemon process + lsvcgssd on each server node (MGS, MDS and OSS) + before starting Lustre. The command syntax is: + + lsvcgssd [-f] [-v] [-g] [-m] [-o] -k + + + -f: run in foreground, instead of as daemon + + + -v: increase verbosity by 1. For example, to set the verbose + level to 3, run 'lsvcgssd -vvv'. Verbose logging can help you make + sure Kerberos is set up correctly. + + + + -g: service MGS + + + -m: service MDS + + + -o: service OSS + + + -k: enable kerberos support + + +
+
+ Setting Security Flavors + + Security flavors can be set by defining sptlrpc rules on the MGS. + These rules are persistent, and are in the form: + <spec>=<flavor> + + + + To add a rule: + mgs> lctl conf_param <spec>=<flavor> + + If there is an existing rule on <spec>, it will be + overwritten. + + + To delete a rule: + mgs> lctl conf_param -d <spec> + + + To list existing rules: + msg> lctl get_param mgs.MGS.live.<fs-name> | grep "srpc.flavor" + + + + + + If nothing is specified, by default all RPC connections will + use null flavor, which means no security. + + + + + After you change a rule, it usually takes a few minutes to apply + the new rule to all nodes, depending on global system load. + + + + + Before you change a rule, make sure affected nodes are ready + for the new security flavor. E.g. if you change flavor from + null to krb5p + but GSS/Kerberos environment is not properly configured on + affected nodes, those nodes might be evicted because they cannot + communicate with each other. + + + + +
+
+ Rules Syntax & Examples + + The general syntax is: + + <target>.srpc.flavor.<network>[.<direction>]=flavor + + + + + <target> can be filesystem name, or + specific MDT/OST device name. For example + testfs, + testfs-MDT0000, + testfs-OST0001. + + + + + <network> is the LNet network name, for + example tcp0, o2ib0, or + default to not filter on LNet network. + + + + + <direction> can be one of + cli2mdt, cli2ost, + mdt2mdt, mdt2ost. + Direction is optional. + + + + + Examples: + + + + + Apply krb5i on + ALL connections for file system + testfs: + + + + mgs> lctl conf_param testfs.srpc.flavor.default=krb5i + + + + Nodes in network tcp0 use + krb5p; all other nodes use + null. + + + + mgs> lctl conf_param testfs.srpc.flavor.tcp0=krb5p +mgs> lctl conf_param testfs.srpc.flavor.default=null + + + + Nodes in network tcp0 use + krb5p; nodes in + o2ib0 use krb5n; + among other nodes, clients use krb5i + to MDT/OST, MDTs use null to other MDTs, + MDTs use krb5a to OSTs. + + + + mgs> lctl conf_param testfs.srpc.flavor.tcp0=krb5p +mgs> lctl conf_param testfs.srpc.flavor.o2ib0=krb5n +mgs> lctl conf_param testfs.srpc.flavor.default.cli2mdt=krb5i +mgs> lctl conf_param testfs.srpc.flavor.default.cli2ost=krb5i +mgs> lctl conf_param testfs.srpc.flavor.default.mdt2mdt=null +mgs> lctl conf_param testfs.srpc.flavor.default.mdt2ost=krb5a +
+
+ Regular Users Authentication + + On client nodes, non-root users need to issue + kinit before accessing Lustre, just like other + Kerberized applications. + + + + + Required by kerberos, the user's principal + (username@REALM) should be added to the KDC. + + + + + Client and MDT nodes should have the same user database + used for name and uid/gid translation. + + + + + Regular users can destroy the established security contexts before + logging out, by issuing: + + lfs flushctx -k -r <mount point> + + Here -k is to destroy the on-disk Kerberos + credential cache, similar to kdestroy, and + -r is to reap the revoked keys from the keyring + when flushing the GSS context. Otherwise it only destroys established + contexts in kernel memory. + +
+
+
+ Secure MGS connection + + Each node can specify which flavor to use to connect to the MGS, by + using the mgssec=flavor mount option. + Once a flavor is chosen, it cannot be changed until re-mount. + + + Because a Lustre node only has one connection to the MGS, if there is + more than one target or client on the node, they necessarily use the + same security flavor to the MGS, being the one enforced when the first + connection to the MGS was established. + + + By default, the MGS accepts RPCs with any flavor. But it is possible to + configure the MGS to only accept a given flavor. The syntax is identical + to what is explained in paragraph + , + but with special target _mgs: + + mgs> lctl conf_param _mgs.srpc.flavor.<network>=<flavor> +
+