1 <?xml version="1.0" encoding="UTF-8"?>
2 <chapter version="5.0" xml:id="lustressk" xml:lang="en-US"
4 xmlns="http://docbook.org/ns/docbook">
5 <title xml:id="lustressk.title">Configuring Shared-Secret Key
7 <para>This chapter describes how to configure Shared-Secret Key security
8 and includes the following sections:</para>
11 <para><xref linkend="sskoverview"/></para>
14 <para><xref linkend="ssksecurityflavors"/></para>
17 <para><xref linkend="sskkeyfiles"/></para>
20 <para><xref linkend="ssklustregsskeyring"/></para>
23 <para><xref linkend="ssknodemaprole"/></para>
26 <para><xref linkend="sskexamples"/></para>
29 <para><xref linkend="ssksptlrpcctx"/></para>
32 <section xml:id="sskoverview">
33 <title>SSK Security Overview</title>
34 <para>The SSK feature ensures integrity and data protection for Lustre
35 PtlRPC traffic. Key files containing a shared secret and session-specific
36 attributes are distributed to Lustre hosts. This authorizes Lustre hosts
37 to mount the file system and optionally enables secure data transport,
38 depending on which security flavor is configured. The administrator handles
39 the generation, distribution, and installation of SSK key files, see
40 <xref linkend="sskkeyfilemanagement"/>.</para>
42 <title>Key features</title>
43 <para>SSK provides the following key features:</para>
46 <para>Host-based authentication</para>
49 <para>Data Transport Privacy</para>
52 <para>Encrypts Lustre RPCs</para>
55 <para>Prevents eavesdropping</para>
60 <para>Data Transport Integrity - Keyed-Hashing Message
61 Authentication Code (HMAC)</para>
64 <para>Prevents man-in-the-middle attacks</para>
67 <para>Ensures RPCs cannot be altered undetected</para>
74 <section xml:id="ssksecurityflavors">
75 <title>SSK Security Flavors</title>
76 <para>SSK is implemented as a Generic Security Services (GSS) mechanism
77 through Lustre's support of the GSS Application Program Interface (GSSAPI).
78 The SSK GSS mechanism supports five flavors that offer varying levels of
80 <para>Flavors provided:</para>
83 <para><literal>skn</literal> - SSK Null (Authentication)</para>
86 <para><literal>ska</literal> - SSK Authentication and Integrity for
90 <para><literal>ski</literal> - SSK Authentication and Integrity</para>
93 <para><literal>skpi</literal> - SSK Authentication, Privacy, and
97 <para><literal>gssnull</literal> - Provides no protection. Used for
98 testing purposes only</para>
101 <para>The table below describes the security characteristics of each
103 <table xml:id="ex.sskflavortable" frame='all'>
104 <title>SSK Security Flavor Protections</title>
105 <tgroup cols='5' align='center' colsep='1' rowsep='1'>
106 <colspec colname='c1' align='left' colwidth='3*'/>
107 <colspec colname='c2' colwidth='3*'/>
108 <colspec colname='c3' colwidth='3*'/>
109 <colspec colname='c4' colwidth='3*'/>
110 <colspec colname='c5' colwidth='3*'/>
133 <para>Required to mount file system</para>
135 <entry><?dbhtml bgcolor="#00cc00" ?><?dbfo bgcolor="#00cc00" ?>
138 <entry><?dbhtml bgcolor="#00cc00" ?><?dbfo bgcolor="#00cc00" ?>
141 <entry><?dbhtml bgcolor="#00cc00" ?><?dbfo bgcolor="#00cc00" ?>
144 <entry><?dbhtml bgcolor="#00cc00" ?><?dbfo bgcolor="#00cc00" ?>
150 <para>Provides RPC Integrity</para>
152 <entry><?dbhtml bgcolor="#ff3300" ?><?dbfo bgcolor="#ff3300" ?>
155 <entry><?dbhtml bgcolor="#00cc00" ?><?dbfo bgcolor="#00cc00" ?>
158 <entry><?dbhtml bgcolor="#00cc00" ?><?dbfo bgcolor="#00cc00" ?>
161 <entry><?dbhtml bgcolor="#00cc00" ?><?dbfo bgcolor="#00cc00" ?>
167 <para>Provides RPC Privacy</para>
169 <entry><?dbhtml bgcolor="#ff3300" ?><?dbfo bgcolor="#ff3300" ?>
172 <entry><?dbhtml bgcolor="#ff3300" ?><?dbfo bgcolor="#ff3300" ?>
175 <entry><?dbhtml bgcolor="#ff3300" ?><?dbfo bgcolor="#ff3300" ?>
178 <entry><?dbhtml bgcolor="#00cc00" ?><?dbfo bgcolor="#00cc00" ?>
184 <para>Provides Bulk RPC Integrity</para>
186 <entry><?dbhtml bgcolor="#ff3300" ?><?dbfo bgcolor="#ff3300" ?>
189 <entry><?dbhtml bgcolor="#ff3300" ?><?dbfo bgcolor="#ff3300" ?>
192 <entry><?dbhtml bgcolor="#00cc00" ?><?dbfo bgcolor="#00cc00" ?>
195 <entry><?dbhtml bgcolor="#00cc00" ?><?dbfo bgcolor="#00cc00" ?>
201 <para>Provides Bulk RPC Privacy</para>
203 <entry><?dbhtml bgcolor="#ff3300" ?><?dbfo bgcolor="#ff3300" ?>
206 <entry><?dbhtml bgcolor="#ff3300" ?><?dbfo bgcolor="#ff3300" ?>
209 <entry><?dbhtml bgcolor="#ff3300" ?><?dbfo bgcolor="#ff3300" ?>
212 <entry><?dbhtml bgcolor="#00cc00" ?><?dbfo bgcolor="#00cc00" ?>
219 <para>Valid non-GSS flavors include:</para>
220 <para><literal>null</literal> - Provides no protection. This is the
221 default flavor.</para>
222 <para><literal>plain</literal> - Plaintext with a hash on each RPC.</para>
224 <title>Secure RPC Rules</title>
225 <para>Secure RPC configuration rules are written to the Lustre log
226 (llog) with the <literal>lctl</literal> command. Rules are processed
227 with the llog and dictate the security flavor that is used for a
228 particular Lustre network and direction.</para>
230 <para>Rules take affect in a matter of seconds and impact both existing
231 and new connections.</para>
233 <para>Rule format:</para>
234 <para><replaceable>target</replaceable>.srpc.flavor.<replaceable>network</replaceable>[.<replaceable>direction</replaceable>]=<replaceable>flavor</replaceable></para>
237 <para><replaceable>target</replaceable> - This could be the
238 file system name or a specific
239 MDT/OST device name.</para>
242 <para><replaceable>network</replaceable> - LNet network
243 name of the RPC initiator. For example
244 <literal>tcp1</literal> or <literal>o2ib0</literal>. This can also
245 be the keyword <literal>default</literal> that applies to all
246 networks otherwise specified.</para>
249 <para><replaceable>direction</replaceable> - Direction is
250 optional. This could be one of
251 <literal>mdt2mdt</literal>, <literal>mdt2ost</literal>,<literal>
252 cli2mdt</literal>, or <literal>cli2ost</literal>.</para>
256 <para>To secure the connection to the MGS use the
257 <literal>mgssec=</literal><replaceable>flavor</replaceable>
258 mount option. This is required because security rules are unknown to
259 the initiator until after the MGS connection has been established.
262 <para>The examples below are for a test Lustre file system named
263 <emphasis>testfs</emphasis>.</para>
265 <title>Defining Rules</title>
266 <para>Rules can be defined and deleted in any order. The rule with the
267 greatest specificity for a given connection is applied. The
268 <replaceable>fsname</replaceable><literal>.srpc.flavor.default</literal>
269 rule is the broadest rule as it applies to all non-MGS connections for
270 the file system in the absence of a more specific rule. You may tailor
271 SSK security to your needs by further specifying a specific
272 <literal>target</literal>, <literal>network</literal>, and/or
273 <literal>direction</literal>.</para>
274 <para>The following example illustrates an approach to configuring SSK
275 security for an environment consisting of three LNet networks. The
276 requirements for this example are:</para>
279 <para>All non-MGS connections must be authenticated.</para>
282 <para>PtlRPC traffic on LNet network <literal>tcp0</literal> must
286 <para>LNet networks <literal>tcp1</literal> and
287 <literal>o2ib0</literal> are local physically secure networks that
288 require high performance. Do not encrypt PtlRPC traffic on these
294 <para>Ensure that all non-MGS connections are authenticated and
295 encrypted by default.</para>
296 <screen>mgs# lctl conf_param testfs.srpc.flavor.default=skpi</screen>
299 <para>Override the file system default security flavor on LNet
300 networks <literal>tcp1</literal> and <literal>o2ib0</literal> with
301 <literal>ska</literal>. Security flavor <literal>ska</literal>
302 provides authentication but without the performance impact of
303 encryption and bulk RPC integrity.</para>
304 <screen>mgs# lctl conf_param testfs.srpc.flavor.tcp1=ska
305 mgs# lctl conf_param testfs.srpc.flavor.o2ib0=ska</screen>
309 <para>Currently the "<literal>lctl set_param -P</literal>" format does
310 not work with sptlrpc.</para>
314 <title>Listing Rules</title>
315 <para>To view the Secure RPC Config Rules, enter:</para>
316 <screen>mgs# lctl get_param mgs.*.live.testfs
318 Secure RPC Config Rules:
319 testfs.srpc.flavor.tcp.cli2mdt=skpi
320 testfs.srpc.flavor.tcp.cli2ost=skpi
321 testfs.srpc.flavor.o2ib=ski
325 <title>Deleting Rules</title>
326 <para>To delete a security flavor for an LNet network use the
327 <literal>conf_param -d</literal> command to delete the flavor for that
329 <para>For example, to delete the
330 <literal>testfs.srpc.flavor.o2ib1=ski</literal> rule, enter:</para>
331 <screen>mgs# lctl conf_param -d testfs.srpc.flavor.o2ib1</screen>
335 <section xml:id="sskkeyfiles">
336 <title>SSK Key Files</title>
337 <para>SSK key files are a collection of attributes formatted as fixed
338 length values and stored in a file, which are distributed by the
339 administrator to client and server nodes. Attributes include:</para>
342 <para><emphasis role='bold'>Version</emphasis> - Key file schema
343 version number. Not user-defined.</para>
346 <para><emphasis role='bold'>Type</emphasis> - A mandatory attribute
347 that denotes the Lustre role of the key file consumer. Valid key types
351 <para><emphasis role='bold'>mgs</emphasis> - for MGS when the
352 <literal>mgssec</literal> <literal>mount.lustre</literal> option
356 <para><emphasis role='bold'>server</emphasis> - for MDS and OSS
360 <para><emphasis role='bold'>client</emphasis> - for clients as
361 well as servers who communicate with other servers in a client
362 context (e.g. MDS communication with OSTs).</para>
368 <para><emphasis role='bold'>HMAC algorithm</emphasis> - The Keyed-Hash
369 Message Authentication Code algorithm used for integrity. Valid
370 algorithms are (Default: SHA256):
382 <para><emphasis role='bold'>Cryptographic algorithm</emphasis> - Cipher
383 for encryption. Valid algorithms are (Default: AES-256-CTR).</para>
386 <para>AES-256-CTR</para>
391 <para><emphasis role='bold'>Session security context expiration</emphasis> -
392 Seconds before session contexts generated from key expire and are
393 regenerated (Default: 604800 seconds (7 days)).</para>
396 <para><emphasis role='bold'>Shared key length</emphasis> - Shared key
397 length in bits (Default: 256).</para>
400 <para><emphasis role='bold'>Prime length</emphasis> - Length of prime
401 (p) in bits used for the Diffie-Hellman Key Exchange (DHKE).
402 (Default: 2048). This is generated only for client keys and can take a
403 while to generate. This value also sets the minimum prime length that
404 servers and MGS will accept from a client. Clients attempting to
405 connect with a prime length less than the minimum will be rejected.
406 In this way servers can guarantee the minimum encryption level that
407 will be permitted.</para>
410 <para><emphasis role='bold'>File system name</emphasis> - Lustre File
411 system name for key.</para>
414 <para><emphasis role='bold'>MGS NIDs</emphasis> - Comma-separated list
415 of MGS NIDs. Only required when <literal>mgssec</literal> is used
416 (Default: "").</para>
419 <para><emphasis role='bold'>Nodemap name</emphasis> - Nodemap name for
420 key (Default: "default"). See <xref linkend="ssknodemaprole"/></para>
423 <para><emphasis role='bold'>Shared key</emphasis> - Shared secret used
424 by all SSK flavors to provide authentication.</para>
427 <para><emphasis role='bold'>Prime (p)</emphasis> - Prime used for the
428 DHKE. This is only used for keys with <literal>Type=client</literal>.
433 <para>Key files provide a means to authenticate Lustre connections;
434 always store and transfer key files securely. Key files must not be
435 world writable or they will fail to load.</para>
437 <section xml:id="sskkeyfilemanagement" remap="h3">
438 <title>Key File Management</title>
439 <para>The <literal>lgss_sk</literal> utility is used to write, modify,
440 and read SSK key files. <literal>lgss_sk</literal> can be used to load
441 key files singularly into the kernel keyring. <literal>lgss_sk</literal>
442 options include:</para>
443 <table xml:id="table.lgss_sk_params" frame='all'>
444 <title>lgss_sk Parameters</title>
445 <tgroup cols='3' align='left' colsep='1' rowsep='1'>
446 <colspec colname='c1' colwidth='2*'/>
447 <colspec colname='c2' colwidth='1*'/>
448 <colspec colname='c3' colwidth='3*'/>
452 <para>Parameter</para>
458 <para>Description</para>
465 <para><literal>-l|--load</literal></para>
468 <para><replaceable>filename</replaceable></para>
471 <para>Install key from file into user's session keyring. Must
472 be executed by <emphasis>root</emphasis>.</para>
477 <para><literal>-m|--modify</literal></para>
480 <para><replaceable>filename</replaceable></para>
483 <para>Modify a file's key attributes</para>
488 <para><literal>-r|--read</literal></para>
491 <para><replaceable>filename</replaceable></para>
494 <para>Show file's key attributes</para>
499 <para><literal>-w|--write</literal></para>
502 <para><replaceable>filename</replaceable></para>
505 <para>Generate key file</para>
510 <para><literal>-c|--crypt</literal></para>
513 <para><replaceable>cipher</replaceable></para>
516 <para>Cipher for encryption (Default: AES Counter mode)</para>
517 <para>AES-256-CTR</para>
522 <para><literal>-i|--hmac</literal></para>
525 <para><replaceable>hash</replaceable></para>
528 <para>Hash algorithm for intregrity (Default: SHA256)</para>
529 <para>SHA256 or SHA512</para>
534 <para><literal>-e|--expire</literal></para>
537 <para><replaceable>seconds</replaceable></para>
540 <para>Seconds before contexts from key expire (Default: 604800
546 <para><literal>-f|--fsname</literal></para>
549 <para><replaceable>name</replaceable></para>
552 <para>File system name for key</para>
557 <para><literal>-g|--mgsnids</literal></para>
560 <para><replaceable>NID(s)</replaceable></para>
563 <para>Comma separated list of MGS NID(s). Only required when
564 mgssec is used (Default: "")</para>
569 <para><literal>-n|--nodemap</literal></para>
572 <para><replaceable>map</replaceable></para>
575 <para>Nodemap name for key (Default: "default")</para>
580 <para><literal>-p|--prime-bits</literal></para>
583 <para><replaceable>length</replaceable></para>
586 <para>Prime length (p) for DHKE in bits (Default: 2048)</para>
591 <para><literal>-t|--type</literal></para>
594 <para><replaceable>type</replaceable></para>
597 <para>Key type (mgs, server, client)</para>
602 <para><literal>-k|--key-bits</literal></para>
605 <para><replaceable>length</replaceable></para>
608 <para>Shared key length in bits (Default: 256)</para>
613 <para><literal>-d|--data</literal></para>
616 <para><replaceable>file</replaceable></para>
619 <para>Shared key random data source (Default: /dev/random)</para>
624 <para><literal>-v|--verbose</literal></para>
630 <para>Increase verbosity for errors</para>
637 <title>Writing Key Files</title>
638 <para>Key files are generated by the <literal>lgss_sk</literal>
639 utility. Parameters are specified on the command line followed by the
640 <literal>--write</literal> parameter and the filename to write
641 to. The <literal>lgss_sk</literal> utility will not overwrite files
642 so the filename must be unique. Mandatory parameters for generating
643 key files are <literal>--type</literal>, either
644 <literal>--fsname</literal> or <literal>--mgsnids</literal>, and
645 <literal>--write</literal>; all other parameters
647 <para><literal>lgss_sk</literal> uses <literal>/dev/random</literal>
648 as the default entropy data source; you may override this with the
649 <literal>--data</literal> parameter. When no hardware random
650 number generator is available on the system where
651 <literal>lgss_sk</literal> is executing, you may need to press keys on
652 the keyboard or move the mouse (if directly attached to the system)
653 or cause disk IO (if system is remote), in order to generate entropy
654 for the shared key. It is possible to use
655 <literal>/dev/urandom</literal> for testing purposes but this may
656 provide less security in some cases.</para>
657 <para>Example:</para>
658 <para>To create a <emphasis>server</emphasis> type key file for the
659 <emphasis>testfs</emphasis> Lustre file system for clients in the
660 <emphasis>biology</emphasis> nodemap, enter:</para>
661 <screen>server# lgss_sk -t server -f testfs -n biology \
662 -w testfs.server.biology.key</screen>
665 <title>Modifying Key Files</title>
666 <para>Like writing key files you modify them by specifying the
667 paramaters on the command line that you want to change. Only key file
668 attributes associated with the parameters provided are changed; all
669 other attributes remain unchanged.</para>
670 <para>To modify a key file's <emphasis>Type</emphasis> to
671 <emphasis>client</emphasis> and populate the
672 <emphasis>Prime (p)</emphasis> key attribute, if it is missing,
674 <screen>client# lgss_sk -t client -m testfs.client.biology.key</screen>
675 <para>To add MGS NIDs <literal>192.168.1.101@tcp,10.10.0.101@o2ib</literal>
676 to server key file <literal>testfs.server.biology.key</literal> and
677 client key file <literal>testfs.client.biology.key</literal>, enter
679 <screen>server# lgss_sk -g 192.168.1.101@tcp,10.10.0.101@o2ib \
680 -m testfs.server.biology.key
682 client# lgss_sk -g 192.168.1.101@tcp,10.10.0.101@o2ib \
683 -m testfs.client.biology.key</screen>
684 <para>To modify the <literal>testfs.server.biology.key</literal> on
685 the MGS to support MGS connections from <emphasis>biology</emphasis>
686 clients, modify the key file's <emphasis>Type</emphasis> to include
687 <emphasis>mgs</emphasis> in addition to <emphasis>server</emphasis>,
689 <screen>mgs# lgss_sk -t mgs,server -m testfs.server.biology.key</screen>
692 <title>Reading Key Files</title>
693 <para>Read key files with the <literal>lgss_sk</literal> utility and
694 <literal>--read</literal> parameter. Read the keys modified in the
695 previous examples:</para>
696 <screen>mgs# lgss_sk -r testfs.server.biology.key
700 Crypt alg: AES-256-CTR
701 Ctx Expiration: 604800 seconds
702 Shared keylen: 256 bits
703 Prime length: 2048 bits
705 MGS NIDs: 192.168.1.101@tcp 10.10.0.101@o2ib
706 Nodemap name: biology
708 0000: 84d2 561f 37b0 4a58 de62 8387 217d c30a ..V.7.JX.b..!}..
709 0010: 1caa d39c b89f ee6c 2885 92e7 0765 c917 .......l(....e..
711 client# lgss_sk -r testfs.client.biology.key
715 Crypt alg: AES-256-CTR
716 Ctx Expiration: 604800 seconds
717 Shared keylen: 256 bits
718 Prime length: 2048 bits
720 MGS NIDs: 192.168.1.101@tcp 10.10.0.101@o2ib
721 Nodemap name: biology
723 0000: 84d2 561f 37b0 4a58 de62 8387 217d c30a ..V.7.JX.b..!}..
724 0010: 1caa d39c b89f ee6c 2885 92e7 0765 c917 .......l(....e..
726 0000: 8870 c3e3 09a5 7091 ae03 f877 f064 c7b5 .p....p....w.d..
727 0010: 14d9 bc54 75f8 80d3 22f9 2640 0215 6404 ...Tu...".&@..d.
728 0020: 1c53 ba84 1267 bea2 fb05 37a4 ed2d 5d90 .S...g....7..-].
729 0030: 84e3 1a67 67f0 47c7 0c68 5635 f50e 9cf0 ...gg.G..hV5....
730 0040: e622 6f53 2627 6af6 9598 eeed 6290 9b1e ."oS&'j.....b...
731 0050: 2ec5 df04 884a ea12 9f24 cadc e4b6 e91d .....J...$......
732 0060: 362f a239 0a6d 0141 b5e0 5c56 9145 6237 6/.9.m.A..\V.Eb7
733 0070: 59ed 3463 90d7 1cbe 28d5 a15d 30f7 528b Y.4c....(..]0.R.
734 0080: 76a3 2557 e585 a1be c741 2a81 0af0 2181 v.%W.....A*...!.
735 0090: 93cc a17a 7e27 6128 5ebd e0a4 3335 db63 ...z~'a(^...35.c
736 00a0: c086 8d0d 89c1 c203 3298 2336 59d8 d7e7 ........2.#6Y...
737 00b0: e52a b00c 088f 71c3 5109 ef14 3910 fcf6 .*....q.Q...9...
738 00c0: 0fa0 7db7 4637 bb95 75f4 eb59 b0cd 4077 ..}.F7..u..Y..@w
739 00d0: 8f6a 2ebd f815 a9eb 1b77 c197 5100 84c0 .j.......w..Q...
740 00e0: 3dc0 d75d 40b3 6be5 a843 751a b09c 1b20 =..]@.k..Cu....
741 00f0: 8126 4817 e657 b004 06b6 86fb 0e08 6a53 .&H..W........jS</screen>
744 <title>Loading Key Files</title>
745 <para>Key files can be loaded into the kernel keyring with the
746 <literal>lgss_sk</literal> utility or at mount time with the
747 <literal>skpath</literal> mount option. The <literal>skpath</literal>
748 method has the advantage that it accepts a directory path and loads
749 all key files within the directory into the keyring. The
750 <literal>lgss_sk</literal> utility loads a single key file into the
751 keyring with each invocation. Key files must not be world writable or
752 they will fail to load.</para>
753 <para>Third party tools can also load the keys if desired. The only
754 caveat is that the key must be available when the request_key upcall
755 to userspace is made and they use the correct key descriptions for a
756 key so that it can be found during the upcall
757 (see Key Descriptions).</para>
758 <para>Examples:</para>
759 <para>Load the <literal>testfs.server.biology.key</literal> key file
760 using <literal>lgss_sk</literal>,
762 <screen>server# lgss_sk -l testfs.server.biology.key</screen>
763 <para>Use the <literal>skpath</literal> mount option to load all of
764 the key files in the <literal>/secure_directory</literal> directory
765 when mounting a storage target, enter:</para>
766 <screen>server# mount -t lustre -o skpath=/secure_directory \
767 /storage/target /mount/point</screen>
768 <para>Use the <literal>skpath</literal> mount option to load key files
769 into the keyring on a client, enter:</para>
770 <screen>client# mount -t lustre -o skpath=/secure_directory \
771 mgsnode:/testfs /mnt/testfs</screen>
775 <section xml:id="ssklustregsskeyring">
776 <title>Lustre GSS Keyring</title>
777 <para>The Lustre GSS Keyring binary <literal>lgss_keyring</literal> is
778 used by SSK to handle the upcall from kernel space into user space via
779 <literal>request-key</literal>. The purpose of
780 <literal>lgss_keyring</literal> is to create a token that is passed as
781 part of the security context initialization RPC (SEC_CTX_INIT)</para>
784 <para>The Lustre GSS keyring types of flavors utilize the Linux kernel
785 keyring infrastructure to maintain keys as well as to perform the
786 upcall from kernel space to userspace for key
787 negotiation/establishment. The GSS keyring establishes a key type
788 (see “request-key(8)”) named <literal>lgssc</literal> when the Lustre
789 <literal>ptlrpc_gss</literal> kernel module is loaded. When a security
790 context must be established it creates a key and uses the
791 <literal>request-key</literal> binary in an upcall to establish the
792 key. This key will look for the configuration file in
793 <literal>/etc/request-key.d</literal> with the name
794 <replaceable>keytype</replaceable>.conf, for Lustre this is
795 <literal>lgssc.conf</literal>.</para>
796 <para>Each node participating in SSK Security must have a
797 <literal>/etc/request-key.d/lgssc.conf</literal> file that contains the
798 following single line:</para>
800 create lgssc * * /usr/sbin/lgss_keyring %o %k %t %d %c %u %g %T %P %S
803 <para>The <literal>request-key</literal> binary will call
804 <literal>lgss_keyring</literal> with the arguments following it with
805 their substituted values (see <literal>request-key.conf(5)</literal>).
809 <title>Server Setup</title>
810 <para>Lustre servers do not use the Linux
811 <literal>request-key</literal> mechanism as clients do. Instead
812 servers run a daemon that uses a pipefs with the kernel to trigger
813 events based on read/write to a file descriptor. The server-side
814 binary is <literal>lsvcgssd</literal>. It can be executed in the
815 foreground or as a daemon. Below are the parameters for the
816 <literal>lsvcgssd</literal> binary which requires various security
817 flavors (<literal>gssnull, krb5, sk</literal>) to be enabled
818 explicitly. This ensures that only required functionality is
820 <table xml:id="table.lsvcgssdparameters">
821 <title>lsvcgssd Parameters</title>
822 <tgroup cols="2" align='left' colsep='1' rowsep='1'>
823 <colspec colname='c1' colwidth='1*'/>
824 <colspec colname='c2' colwidth='1*'/>
828 <para>Parameter</para>
831 <para>Description</para>
838 <para><literal>-f</literal></para>
841 <para>Run in foreground</para>
846 <para><literal>-n</literal></para>
849 <para>Do not establish Kerberos credentials</para>
854 <para><literal>-v</literal></para>
857 <para>Verbosity</para>
862 <para><literal>-m</literal></para>
865 <para>Service MDS</para>
870 <para><literal>-o</literal></para>
873 <para>Service OSS</para>
878 <para><literal>-g</literal></para>
881 <para>Service MGS</para>
886 <para><literal>-k</literal></para>
889 <para>Enable Kerberos support</para>
894 <para><literal>-s</literal></para>
897 <para>Enable Shared Key support</para>
902 <para><literal>-z</literal></para>
905 <para>Enable <literal>gssnull</literal> support</para>
911 <para>A SysV style init script is installed for starting and stopping
912 the <literal>lsvcgssd</literal> daemon. The init script checks the
913 <literal>LSVCGSSARGS</literal> variable in the
914 <literal>/etc/sysconfig/lsvcgss</literal> configuration file for
915 startup parameters.</para>
916 <para>Keys during the upcall on the client and handling of an RPC on
917 the server are found by using a specific key description for each key
918 in the kernel keyring.</para>
919 <para>For each MGS NID there must be a separate key loaded. The format
920 of the key description should be:</para>
921 <table xml:id="table.sskkeydescription">
922 <title>Key Descriptions</title>
923 <tgroup cols='3' align='left' colsep='1' rowsep='1'>
924 <colspec colname='c1' colwidth='1*'/>
925 <colspec colname='c2' colwidth='3*'/>
926 <colspec colname='c3' colwidth='3*'/>
933 <para>Key Description</para>
946 <para>lustre:MGC<replaceable>NID</replaceable></para>
949 <para><literal>lustre:MGC192.168.1.10@tcp</literal></para>
954 <para>MDC/OSC/OSP/LWP</para>
957 <para>lustre:<replaceable>fsname</replaceable></para>
960 <para><literal>lustre:testfs</literal></para>
968 <para>lustre:<replaceable>fsname</replaceable>:<replaceable>NodemapName</replaceable></para>
971 <para><literal>lustre:testfs:biology</literal></para>
979 <para>lustre:<replaceable>fsname</replaceable>:<replaceable>NodemapName</replaceable></para>
982 <para><literal>lustre:testfs:biology</literal></para>
990 <para>lustre:MGS</para>
993 <para><literal>lustre:MGS</literal></para>
999 <para>All keys for Lustre use the <literal>user</literal> type for
1000 keys and are attached to the user’s keyring. This is not
1001 configurable. Below is an example showing how to list the user’s
1002 keyring, load a key file, read the key, and clear the key from the
1003 kernel keyring.</para>
1004 <screen><emphasis role='bold'>client#</emphasis> keyctl show
1006 17053352 --alswrv 0 0 keyring: _ses
1007 773000099 --alswrv 0 65534 \_ keyring: _uid.0
1009 <emphasis role='bold'>client#</emphasis> lgss_sk -l /secure_directory/testfs.client.key
1011 <emphasis role='bold'>client#</emphasis> keyctl show
1013 17053352 --alswrv 0 0 keyring: _ses
1014 773000099 --alswrv 0 65534 \_ keyring: _uid.0
1015 1028795127 --alswrv 0 0 \_ user: lustre:testfs
1017 <emphasis role='bold'>client#</emphasis> keyctl pipe 1028795127 | lgss_sk -r -
1021 Crypt alg: AES-256-CTR
1022 Ctx Expiration: 604800 seconds
1023 Shared keylen: 256 bits
1024 Prime length: 2048 bits
1027 Nodemap name: default
1029 0000: faaf 85da 93d0 6ffc f38c a5c6 f3a6 0408 ......o.........
1030 0010: 1e94 9b69 cf82 d0b9 880b f173 c3ea 787a ...i.......s..xz
1032 0000: 9c12 ed95 7b9d 275a 229e 8083 9280 94a0 ....{.'Z".......
1033 0010: 8593 16b2 a537 aa6f 8b16 5210 3dd5 4c0c .....7.o..R.=.L.
1034 0020: 6fae 2729 fcea 4979 9435 f989 5b6e 1b8a o.')..Iy.5..[n..
1035 0030: 5039 8db2 3a23 31f0 540c 33cb 3b8e 6136 P9..:#1.T.3.;.a6
1036 0040: ac18 1eba f79f c8dd 883d b4d2 056c 0501 .........=...l..
1037 0050: ac17 a4ab 9027 4930 1d19 7850 2401 7ac4 .....'I0..xP$.z.
1038 0060: 92b4 2151 8837 ba23 94cf 22af 72b3 e567 ..!Q.7.#..".r..g
1039 0070: 30eb 0cd4 3525 8128 b0ff 935d 0ba3 0fc0 0...5%.(...]....
1040 0080: 9afa 5da7 0329 3ce9 e636 8a7d c782 6203 ..]..)<..6.}..b.
1041 0090: bb88 012e 61e7 5594 4512 4e37 e01d bdfc ....a.U.E.N7....
1042 00a0: cb1d 6bd2 6159 4c3a 1f4f 1167 0e26 9e5e ..k.aYL:.O.g.&.^
1043 00b0: 3cdc 4a93 63f6 24b1 e0f1 ed77 930b 9490 <.J.c.$....w....
1044 00c0: 25ef 4718 bff5 033e 11ba e769 4969 8a73 %.G....>...iIi.s
1045 00d0: 9f5f b7bb 9fa0 7671 79a4 0d28 8a80 1ea1 ._....vqy..(....
1046 00e0: a4df 98d6 e20e fe10 8190 5680 0d95 7c83 ..........V...|.
1047 00f0: 6e21 abb3 a303 ff55 0aa8 ad89 b8bf 7723 n!.....U......w#
1049 <emphasis role='bold'>client#</emphasis> keyctl clear @u
1051 <emphasis role='bold'>client#</emphasis> keyctl show
1053 17053352 --alswrv 0 0 keyring: _ses
1054 773000099 --alswrv 0 65534 \_ keyring: _uid.0
1057 <section remap='h3'>
1058 <title>Debugging GSS Keyring</title>
1059 <para>Lustre client and server support several debug levels, which
1060 can be seen below.</para>
1061 <para>Debug levels:</para>
1064 <para>0 - Error</para>
1067 <para>1 - Warn</para>
1070 <para>2 - Info</para>
1073 <para>3 - Debug</para>
1076 <para>4 - Trace</para>
1079 <para>To set the debug level on the client use the Lustre
1081 <para><literal>sptlrpc.gss.lgss_keyring.debug_level</literal></para>
1082 <para>For example to set the debug level to trace, enter:</para>
1083 <screen>client# lctl set_param sptlrpc.gss.lgss_keyring.debug_level=4</screen>
1084 <para>Server-side verbosity is increased by adding additional verbose
1085 flags (<literal>-v</literal>) to the command line arguments for the
1086 daemon. The following command runs the <literal>lsvcgssd</literal>
1087 daemon in the foreground with debug verbosity supporting gssnull and
1089 <screen>server# lsvcgssd -f -vvv -z -s</screen>
1090 <para><literal>lgss_keyring</literal> is called as part of the
1091 <literal>request-key</literal> upcall which has no standard output;
1092 therefore logging is done through syslog. The server-side logging with
1093 <literal>lsvcgssd</literal> is written to standard output when
1094 executing in the foreground and to syslog in daemon mode.</para>
1096 <section remap='h3'>
1097 <title>Revoking Keys</title>
1098 <para>The keys discussed above with <literal>lgss_sk</literal> and the
1099 <literal>skpath</literal> mount options are not revoked. They are only
1100 used to create valid contexts for client connections. Instead of
1101 revoking them they can be invalidated in one of two ways.</para>
1104 <para>Unloading the key from the user keyring on the server will
1105 cause new client connections to fail. If no longer necessary it
1106 can be deleted.</para>
1109 <para>Changing the nodemap name for the clients on the servers.
1110 Since the nodemap is an integral part of the shared key context
1111 instantiation, renaming the nodemap a group of NIDs belongs to
1112 will prevent any new contexts.</para>
1115 <para>There currently does not exist a mechanism to flush contexts
1116 from Lustre. Targets could be unmounted from the servers to purge
1117 contexts. Alternatively shorter context expiration could be used
1118 when the key is created so that contexts need to be refreshed more
1119 frequently than the default. 3600 seconds could be reasonable
1120 depending on the use case so that contexts will have to be
1121 renegotiated every hour.</para>
1124 <section xml:id='ssknodemaprole'>
1125 <title>Role of Nodemap in SSK</title>
1126 <para>SSK uses Nodemap (See <xref linkend="lustrenodemap"/>)
1127 policy group names and their associated NID range(s) as a mechanism to
1128 prevent key file forgery, and to control the range of NIDs on which a
1129 given key file can be used.</para>
1130 <para>Clients assume they are in the nodemap specified in the key file
1131 they use. When clients instantiate security contexts an upcall is
1132 triggered that specifies information about the context that triggers it.
1133 From this context information <literal>request-key</literal> calls
1134 <literal>lgss_keyring</literal>, which in turn looks up the key with
1135 description lustre:<replaceable>fsname</replaceable> or
1136 lustre:<replaceable>target_name</replaceable> for the MGC. Using
1137 the key found in the user keyring matching the description, the nodemap
1138 name is read from the key, hashed with SHA256, and sent to the server.
1140 <para>Servers look up the client’s NID to determine which nodemap the NID
1141 is associated with and sends the nodemap name to
1142 <literal>lsvcgssd</literal>. The <literal>lsvcgssd</literal> daemon
1143 verifies whether the HMAC equals the nodemap value sent by the client.
1144 This prevents forgery and invalidates the key when a client’s NID is not
1145 associated with the nodemap name defined on the servers.</para>
1146 <para>It is not required to activate the Nodemap feature in order for SSK
1147 to perform client NID to nodemap name lookups.</para>
1149 <section xml:id='sskexamples'>
1150 <title>SSK Examples</title>
1151 <para>The examples in this section use 1 MGS/MDS (NID 172.16.0.1@tcp),
1152 1 OSS (NID 172.16.0.3@tcp), and 2 clients. The Lustre file system name is
1153 <emphasis>testfs</emphasis>.</para>
1154 <section remap='h3'>
1155 <title>Securing Client to Server Communications</title>
1156 <para>This example illustrates how to configure SSK to apply Privacy and
1157 Integrity protections to client-to-server PtlRPC traffic on the
1158 <literal>tcp</literal> network. Rules that specify a direction,
1159 specifically <literal>cli2mdt</literal> and <literal>cli2ost</literal>,
1160 are used. This permits server-to-server communications to continue using
1161 <literal>null</literal> which is the <emphasis>default</emphasis> flavor
1162 for all Lustre connections. This arrangement provides no server-to-server
1163 protections, see <xref linkend="ssks2sexample"/>.</para>
1166 <para>Create secure directory for storing SSK key files.</para>
1167 <screen>mds# mkdir /secure_directory
1168 mds# chmod 600 /secure_directory
1169 oss# mkdir /secure_directory
1170 oss# chmod 600 /secure_directory
1171 cli1# mkdir /secure_directory
1172 cli1# chmod 600 /secure_directory
1173 cli2# mkdir /secure_directory
1174 cli2# chmod 600 /secure_directory</screen>
1177 <para>Generate a key file for the MDS and OSS servers. Run:</para>
1178 <screen>mds# lgss_sk -t server -f testfs -w \
1179 /secure_directory/testfs.server.key</screen>
1182 <para>Securely copy the /secure_directory/testfs.server.key key
1183 file to the OSS.</para>
1184 <screen>mds# scp /secure_directory/testfs.server.key \
1185 oss:/secure_directory/</screen>
1188 <para>Securely copy the
1189 <literal>/secure_directory/testfs.server.key</literal> key file to
1190 <literal>/secure_directory/testfs.client.key</literal> on
1191 <emphasis>client1</emphasis>.</para>
1192 <screen>mds# scp /secure_directory/testfs.server.key \
1193 client1:/secure_directory/testfs.<emphasis role="bold">client</emphasis>.key</screen>
1196 <para>Modify the key file type to <literal>client</literal> on
1197 <emphasis>client1</emphasis>. This operation also generates a prime
1198 number of <literal>Prime length</literal> to populate the
1199 <literal>Prime (p)</literal> attribute. Run:</para>
1200 <screen>client1# lgss_sk -t client \
1201 -m /secure_directory/testfs.client.key</screen>
1204 <para>Create a <literal>/etc/request-key.d/lgssc.conf</literal> file
1205 on all nodes that contains this line
1206 '<literal>create lgssc * * /usr/sbin/lgss_keyring %o %k %t %d %c %u
1207 %g %T %P %S</literal>' without the single quotes. Run:</para>
1208 <screen>mds# echo create lgssc \* \* /usr/sbin/lgss_keyring %o %k %t %d %c %u %g %T %P %S > /etc/request-key.d/lgssc.conf
1209 oss# echo create lgssc \* \* /usr/sbin/lgss_keyring %o %k %t %d %c %u %g %T %P %S > /etc/request-key.d/lgssc.conf
1210 client1# echo create lgssc \* \* /usr/sbin/lgss_keyring %o %k %t %d %c %u %g %T %P %S > /etc/request-key.d/lgssc.conf
1211 client2# echo create lgssc \* \* /usr/sbin/lgss_keyring %o %k %t %d %c %u %g %T %P %S > /etc/request-key.d/lgssc.conf</screen>
1214 <para>Configure the <literal>lsvcgss</literal> daemon on the MDS and
1215 OSS. Set the <literal>LSVCGSSDARGS</literal> variable in
1216 <literal>/etc/sysconfig/lsvcgss</literal> on the MDS to
1217 <literal>‘-s -m’</literal>. On the OSS, set the
1218 <literal>LSVCGSSDARGS</literal> variable in
1219 <literal>/etc/sysconfig/lsvcgss</literal> to
1220 <literal>‘-s -o’</literal></para>
1223 <para>Start the <literal>lsvcgssd</literal> daemon on the MDS and
1225 <screen>mds# systemctl start lsvcgss.service
1226 oss# systemctl start lsvcgss.service</screen>
1229 <para>Mount the MDT and OST with the
1230 <literal>-o skpath=/secure_directory</literal> mount option. The
1231 <literal>skpath</literal> option loads all SSK key files found in the
1232 directory into the kernel keyring.</para>
1235 <para> Set client to MDT and client to OST security flavor to SSK
1236 Privacy and Integrity, <literal>skpi</literal>:</para>
1237 <screen>mds# lctl conf_param testfs.srpc.flavor.tcp.cli2mdt=skpi
1238 mds# lctl conf_param testfs.srpc.flavor.tcp.cli2ost=skpi</screen>
1241 <para>Mount the testfs file system on client1 and client2:</para>
1242 <screen>client1# mount -t lustre -o skpath=/secure_directory 172.16.0.1@tcp:/testfs /mnt/testfs
1243 client2# mount -t lustre -o skpath=/secure_directory 172.16.0.1@tcp:/testfs /mnt/testfs
1244 mount.lustre: mount 172.16.0.1@tcp:/testfs at /mnt/testfs failed: Connection refused</screen>
1247 <para><emphasis>client2</emphasis> failed to authenticate because it
1248 does not have a valid key file. Repeat steps 4 and 5, substitute
1249 client1 for client2, then mount the testfs file system on
1251 <screen>client2# mount -t lustre -o skpath=/secure_directory 172.16.0.1@tcp:/testfs /mnt/testfs</screen>
1254 <para>Verify that the <literal>mdc</literal> and
1255 <literal>osc</literal> connections are using the SSK mechanism and
1256 that <literal>rpc</literal> and <literal>bulk</literal> security
1257 flavors are <literal>skpi</literal>. See
1258 <xref linkend="ssksptlrpcctx"/>.</para>
1259 <para>Notice the <literal>mgc</literal> connection to the MGS has no
1260 secure PtlRPC security context. This is because
1261 <literal>skpi</literal> security was only specified for
1262 client-to-MDT and client-to-OST connections in step 10. The
1263 following example details the steps necessary to secure the
1264 connection to the MGS.</para>
1268 <section remap='h3'>
1269 <title>Securing MGS Communications</title>
1270 <para>This example builds on the previous example.</para>
1273 <para>Enable <literal>lsvcgss</literal> MGS service support on MGS.
1274 Edit <literal>/etc/sysconfig/lsvcgss</literal> on the MGS and add
1275 the (<literal>-g</literal>) parameter to the
1276 <literal>LSVCGSSDARGS</literal> variable. Restart the
1277 <literal>lsvcgss</literal> service.</para>
1280 <para>Add <emphasis>mgs</emphasis> key type and
1281 <emphasis>MGS NIDs</emphasis> to
1282 <literal>/secure_directory/testfs.server.key</literal>
1284 <screen>mgs# lgss_sk -t mgs,server -g 172.16.0.1@tcp,172.16.0.2@tcp -m /secure_directory/testfs.server.key</screen>
1287 <para>Load the modified key file on the MGS. Run:</para>
1288 <screen>mgs# lgss_sk -l /secure_directory/testfs.server.key</screen>
1291 <para>Add <emphasis>MGS NIDs</emphasis> to
1292 <literal>/secure_directory/testfs.client.key</literal>
1293 on client, client1.</para>
1294 <screen>client1# lgss_sk -g 172.16.0.1@tcp,172.16.0.2@tcp -m /secure_directory/testfs.client.key</screen>
1297 <para>Unmount the testfs file system on client1, then mount with the
1298 <literal>mgssec=skpi</literal> mount option:</para>
1299 <screen>cli1# mount -t lustre -o mgssec=skpi,skpath=/secure_directory 172.16.0.1@tcp:/testfs /mnt/testfs</screen>
1302 <para>Verify that client1’s MGC connection is using the SSK mechanism
1303 and <literal>skpi</literal> security flavor. See
1304 <xref linkend="ssksptlrpcctx"/>.</para>
1308 <section xml:id='ssks2sexample' remap='h3'>
1309 <title>Securing Server to Server Communications</title>
1310 <para>This example illustrates how to configure SSK to apply
1311 <emphasis>Integrity</emphasis> protection, <literal>ski</literal> flavor,
1312 to MDT-to-OST PtlRPC traffic on the <literal>tcp</literal> network.</para>
1313 <para>This example builds on the previous example.</para>
1316 <para>Create a Nodemap policy group named
1317 <literal>LustreServers</literal> on the MGS for the Lustre Servers,
1319 <screen>mgs# lctl nodemap_add LustreServers</screen>
1322 <para>Add MDS and OSS NIDs to the LustreServers nodemap, enter:</para>
1323 <screen>mgs# lctl nodemap_add_range --name LustreServers --range 172.16.0.[1-3]@tcp</screen>
1326 <para>Create key file of type <literal>mgs,server</literal> for use
1327 with nodes in the <emphasis>LustreServers</emphasis> Nodemap range.
1329 <screen>mds# lgss_sk -t mgs,server -f testfs -g \
1330 172.16.0.1@tcp,172.16.0.2@tcp -n LustreServers -w \
1331 /secure_directory/testfs.LustreServers.key</screen>
1334 <para>Securely copy the
1335 <literal>/secure_directory/testfs.LustreServers.key</literal> key
1336 file to the OSS.</para>
1337 <screen>mds# scp /secure_directory/testfs.LustreServers.key oss:/secure_directory/</screen>
1340 <para>On the MDS and OSS, copy
1341 <literal>/secure_directory/testfs.LustreServers.key</literal> to
1342 <literal>/secure_directory/testfs.LustreServers.client.key</literal>.
1346 <para>On each server modify the key file type of
1347 <literal>/secure_directory/testfs.LustreServers.client.key</literal>
1348 to be of type client. This operation also generates a prime number
1349 of <emphasis>Prime length</emphasis> to populate the
1350 <emphasis>Prime (p)</emphasis> attribute. Run:</para>
1351 <screen>mds# lgss_sk -t client -m \
1352 /secure_directory/testfs.LustreServers.client.key
1353 oss# lgss_sk -t client -m \
1354 /secure_directory/testfs.LustreServers.client.key</screen>
1358 <literal>/secure_directory/testfs.LustreServers.key</literal> and
1359 <literal>/secure_directory/testfs.LustreServers.client.key</literal>
1360 key files into the keyring on the MDS and OSS, enter:</para>
1361 <screen>mds# lgss_sk -l /secure_directory/testfs.LustreServers.key
1362 mds# lgss_sk -l /secure_directory/testfs.LustreServers.client.key
1363 oss# lgss_sk -l /secure_directory/testfs.LustreServers.key
1364 oss# lgss_sk -l /secure_directory/testfs.LustreServers.client.key</screen>
1367 <para>Set MDT to OST security flavor to SSK Integrity,
1368 <literal>ski</literal>:</para>
1369 <screen>mds# lctl conf_param testfs.srpc.flavor.tcp.mdt2ost=ski</screen>
1372 <para>Verify that the <literal>osc</literal> and
1373 <literal>osp</literal> connections to the OST have a secure
1374 <literal>ski</literal> security context. See
1375 <xref linkend="ssksptlrpcctx"/>.</para>
1380 <section xml:id='ssksptlrpcctx'>
1381 <title>Viewing Secure PtlRPC Contexts</title>
1382 <para>From the client (or servers which have mgc, osc, mdc contexts) you
1383 can view info regarding all users’ contexts and the flavor in use for an
1384 import. For user’s contexts (srpc_context), SSK and gssnull only support
1385 a single root UID so there should only be one context. The other file in
1386 the import (srpc_info) has additional sptlrpc details. The
1387 <literal>rpc</literal> and <literal>bulk</literal> flavors allow you to
1388 verify which security flavor is in use.</para>
1389 <screen>client1# lctl get_param *.*.srpc_*
1390 mdc.testfs-MDT0000-mdc-ffff8800da9f0800.srpc_contexts=
1391 ffff8800da9600c0: uid 0, ref 2, expire 1478531769(+604695), fl uptodate,cached,, seq 7, win 2048, key 27a24430(ref 1), hdl 0xf2020f47cbffa93d:0xc23f4df4bcfb7be7, <emphasis role="bold">mech: sk</emphasis>
1392 mdc.testfs-MDT0000-mdc-ffff8800da9f0800.srpc_info=
1393 <emphasis role="bold">rpc flavor: skpi</emphasis>
1394 <emphasis role="bold">bulk flavor: skpi</emphasis>
1395 flags: rootonly,udesc,
1401 mgc.MGC172.16.0.1@tcp.srpc_contexts=
1402 ffff8800dbb09b40: uid 0, ref 2, expire 1478531769(+604695), fl uptodate,cached,, seq 18, win 2048, key 3e3f709f(ref 1), hdl 0xf2020f47cbffa93b:0xc23f4df4bcfb7be6, <emphasis role="bold">mech: sk</emphasis>
1403 mgc.MGC172.16.0.1@tcp.srpc_info=
1404 <emphasis role="bold">rpc flavor: skpi</emphasis>
1405 <emphasis role="bold">bulk flavor: skpi</emphasis>
1412 osc.testfs-OST0000-osc-ffff8800da9f0800.srpc_contexts=
1413 ffff8800db9e5600: uid 0, ref 2, expire 1478531770(+604696), fl uptodate,cached,, seq 3, win 2048, key 3f7c1d70(ref 1), hdl 0xf93e61c64b6b415d:0xc23f4df4bcfb7bea, <emphasis role="bold">mech: sk</emphasis>
1414 osc.testfs-OST0000-osc-ffff8800da9f0800.srpc_info=
1415 <emphasis role="bold">rpc flavor: skpi</emphasis>
1416 <emphasis role="bold">bulk flavor: skpi</emphasis>
1417 flags: rootonly,bulk,
1422 gc next 3505</screen>