1 <?xml version="1.0" encoding="UTF-8"?>
2 <chapter version="5.0" xml:id="lustrenodemap" xml:lang="en-US"
4 xmlns="http://docbook.org/ns/docbook">
5 <title xml:id="lustrenodemap.title">Mapping UIDs and GIDs with
8 <para>This chapter describes how to map UID and GIDs across a Lustre file
9 system using the nodemap feature, and includes the following
14 <para><xref linkend="settingamapping"/></para>
18 <para><xref linkend="alteringproperties"/></para>
22 <para><xref linkend="enablingthefeature"/></para>
26 <para><xref linkend="defaultNodemap"/></para>
30 <para><xref linkend="verifyingsettings"/></para>
34 <para><xref linkend="ensuringconsistency"/></para>
38 <section xml:id="settingamapping">
39 <title>Setting a Mapping</title>
41 <para>The nodemap feature supported in Lustre 2.9 was first
42 introduced in Lustre 2.7 as a technology preview. It allows UIDs and GIDs
43 from remote systems to be mapped to local sets of UIDs and GIDs while
44 retaining POSIX ownership, permissions and quota information. As a result,
45 multiple sites with conflicting user and group identifiers can operate on
46 a single Lustre file system without creating collisions in UID or GID
50 <title>Defining Terms</title>
52 <para>When the nodemap feature is enabled, client file system access to
53 a Lustre system is filtered through the nodemap identity mapping policy
54 engine. Lustre connectivity is governed by network identifiers, or
55 <emphasis>NIDs</emphasis>, such as
56 <literal>192.168.7.121@tcp</literal>. When an operation is made from a
57 NID, Lustre decides if that NID is part of a
58 <emphasis>nodemap</emphasis>, a policy group consisting of one or
59 more NID ranges. If no policy group exists for that NID, access is
60 squashed to user <literal>nobody</literal> by default. Each policy group
61 also has several <emphasis>properties</emphasis>, such as
62 <literal>trusted</literal>
63 and <literal>admin</literal>, which determine access conditions.
64 A collection of identity maps or
65 <emphasis>idmaps</emphasis> are kept for each policy group. These
66 idmaps determine how UIDs and GIDs on the client are translated into the
67 canonical user space of the local Lustre file system.</para>
69 <para>In order for nodemap to function properly, the MGS, MDS, and OSS
70 systems must all have a version of Lustre which supports nodemap.
71 Clients operate transparently and do not require special
72 configuration or knowledge of the nodemap setup.</para>
76 <title>Deciding on NID Ranges</title>
78 <para>NIDs can be described as either a singleton address or a range of
79 addresses. A single address is described in standard Lustre NID format,
80 such as <literal>10.10.6.120@tcp</literal>. A range
81 is described using a dash to separate the range, for example,
82 <literal>192.168.20.[0-255]@tcp</literal>.</para>
84 <para>The range must be contiguous. The full LNet definition for a
85 nidlist is as follows:</para>
88 <nidlist> :== <nidrange> [ ' ' <nidrange> ]
89 <nidrange> :== <addrrange> '@' <net>
90 <addrrange> :== '*' |
91 <ipaddr_range> |
93 <ipaddr_range> :==
94 <numaddr_range>.<numaddr_range>.<numaddr_range>.<numaddr_range>
95 <numaddr_range> :== <number> |
97 <expr_list> :== '[' <range_expr> [ ',' <range_expr>] ']'
98 <range_expr> :== <number> |
99 <number> '-' <number> |
100 <number> '-' <number> '/' <number>
101 <net> :== <netname> | <netname><number>
102 <netname> :== "lo" | "tcp" | "o2ib" | "gni"
103 <number> :== <nonnegative decimal> | <hexadecimal></screen>
107 <title>Describing and Deploying a Sample Mapping</title>
109 <para>Deploy nodemap by first considering which users need to be
110 mapped, and what sets of network addresses or ranges are involved.
111 Issues of visibility between users must be examined as well.</para>
113 <para>Consider a deployment where researchers are working on data
114 relating to birds. The researchers use a computing system which mounts
115 Lustre from a single IPv4 address, <literal>192.168.0.100</literal>.
116 Name this policy group <literal>BirdResearchSite</literal>. The IP
117 address forms the NID <literal>192.168.0.100@tcp</literal>. Create the
118 policy group and add the NID to that group on the MGS
119 using the <literal>lctl</literal> command:</para>
121 <screen>mgs# lctl nodemap_add <replaceable>BirdResearchSite</replaceable>
122 mgs# lctl nodemap_add_range --name <replaceable>BirdResearchSite</replaceable> --range 192.168.0.100@tcp</screen>
125 <para>A NID cannot be in more than one policy group. Assign a NID to
126 a new policy group by first removing it from the existing group.</para>
129 <para>The researchers use the following identifiers on their host system:</para>
133 <para><literal>swan</literal> (UID 530) member of group
134 <literal>wetlands</literal> (GID 600)</para>
138 <para><literal>duck</literal> (UID 531) member of group
139 <literal>wetlands</literal> (GID 600)</para>
143 <para><literal>hawk</literal> (UID 532) member of group
144 <literal>raptor</literal> (GID 601)</para>
148 <para><literal>merlin</literal> (UID 533) member of group
149 <literal>raptor</literal> (GID 601)</para>
153 <para>Assign a set of six idmaps to this policy group, with four for UIDs,
154 and two for GIDs. Pick a starting point, e.g. UID 11000, with room for
155 additional UIDs and GIDs to be added as the configuration grows.
156 Use the <literal>lctl</literal> command to set up the idmaps:</para>
158 <screen>mgs# lctl nodemap_add_idmap --name <replaceable>BirdResearchSite</replaceable> --idtype uid --idmap <replaceable>530:11000</replaceable>
159 mgs# lctl nodemap_add_idmap --name <replaceable>BirdResearchSite</replaceable> --idtype uid --idmap <replaceable>531:11001</replaceable>
160 mgs# lctl nodemap_add_idmap --name <replaceable>BirdResearchSite</replaceable> --idtype uid --idmap <replaceable>532:11002</replaceable>
161 mgs# lctl nodemap_add_idmap --name <replaceable>BirdResearchSite</replaceable> --idtype uid --idmap <replaceable>533:11003</replaceable>
162 mgs# lctl nodemap_add_idmap --name <replaceable>BirdResearchSite</replaceable> --idtype gid --idmap <replaceable>600:11000</replaceable>
163 mgs# lctl nodemap_add_idmap --name <replaceable>BirdResearchSite</replaceable> --idtype gid --idmap <replaceable>601:11001</replaceable></screen>
165 <para>The parameter <literal>530:11000</literal> assigns a client UID,
166 for example UID 530, to a single canonical UID,
167 such as UID 11000. Each assignment is made individually. There is no
168 method to specify a range <literal>530-533:11000-11003</literal>.
169 UID and GID idmaps are assigned separately. There is no implied
170 relationship between the two.</para>
172 <para>Files created on the Lustre file system from the
173 <literal>192.168.0.100@tcp</literal> NID using UID
174 <literal>duck</literal> and GID <literal>wetlands</literal> are stored
175 in the Lustre file system using the canonical identifiers, in this case
176 UID 11001 and GID 11000. A different NID, if not part of the same policy
177 group, sees its own view of the same file space.</para>
179 <para>Suppose a previously created project directory exists owned by UID
180 11002/GID 11001, with mode 770. When users <literal>hawk</literal> and
181 <literal>merlin</literal> at 192.168.0.100 place files named
182 <literal>hawk-file</literal> and <literal>merlin-file</literal> into the
183 directory, the contents from the 192.168.0.100 client appear as:</para>
185 <screen>[merlin@192.168.0.100 projectsite]$ ls -la
187 drwxrwx--- 2 hawk raptor 4096 Jul 23 09:06 .
188 drwxr-xr-x 3 nobody nobody 4096 Jul 23 09:02 ..
189 -rw-r--r-- 1 hawk raptor 10240000 Jul 23 09:05 hawk-file
190 -rw-r--r-- 1 merlin raptor 25100288 Jul 23 09:06 merlin-file</screen>
192 <para>From a privileged view, the canonical owners are displayed:</para>
194 <screen>[root@trustedSite projectsite]# ls -la
196 drwxrwx--- 2 11002 11001 4096 Jul 23 09:06 .
197 drwxr-xr-x 3 root root 4096 Jul 23 09:02 ..
198 -rw-r--r-- 1 11002 11001 10240000 Jul 23 09:05 hawk-file
199 -rw-r--r-- 1 11003 11001 25100288 Jul 23 09:06 merlin-file</screen>
201 <para>If UID 11002 or GID 11001 do not exist on the Lustre MDS or MGS,
202 create them in LDAP or other data sources, or trust clients by setting
203 <literal>identity_upcall</literal> to <literal>NONE</literal>. For more
204 information, see <xref linkend="dbdoclet.identity_upcall"/>.</para>
206 <para>Building a larger and more complex configuration is possible by
207 iterating through the <literal>lctl</literal> commands above. In
212 <para>Create a name for the policy group.</para>
216 <para>Create a set of NID ranges used by the
221 <para>Define which UID and GID translations need to occur for the
228 <section xml:id="alteringproperties">
229 <title>Altering Properties</title>
231 <para>Privileged users access mapped systems with rights dependent on
232 certain properties, described below. By default, root access is squashed
233 to user <literal>nobody</literal>, which interferes with most
234 administrative actions.</para>
237 <title>Managing the Properties</title>
239 <para>Several properties exist, off by default, which change
240 client behavior: <literal>admin</literal>,
241 <literal>trusted</literal>, <literal>squash_uid</literal>,
242 <literal>squash_gid</literal>, and <literal>deny_unknown</literal>.
247 <para>The <literal>trusted</literal> property permits members
248 of a policy group to see the file system's canonical identifiers.
249 In the above example, UID 11002 and GID 11001 will be seen without
250 translation. This can be utilized when local UID and GID sets
251 already map directly to the specified users.</para>
255 <para>The property <literal>admin</literal> defines whether
256 root is squashed on the policy group. By default, it is
257 squashed, unless this property is enabled. Coupled with the
258 <literal>trusted</literal> property, this will allow unmapped
259 access for backup nodes, transfer points, or other administrative
264 <para>The property <literal>deny_unknown</literal> denies all access
265 to users not mapped in a particular nodemap. This is useful if a site
266 is concerned about unmapped users accessing the file system in order to
267 satisfy security requirements.
272 <para>The properties <literal>squash_uid</literal> and <literal>
273 squash_gid</literal> define the default UID and GID that users will
274 be squashed to if unmapped, unless the deny_unknown flag is set, in
275 which case access will still be denied.
280 <para>Alter values to either true (1) or false (0) on the MGS:</para>
282 <screen>mgs# lctl nodemap_modify --name <replaceable>BirdAdminSite</replaceable> --property trusted --value 1
283 mgs# lctl nodemap_modify --name <replaceable>BirdAdminSite</replaceable> --property admin --value 1
284 mgs# lctl nodemap_modify --name <replaceable>BirdAdminSite</replaceable> --property deny_unknown --value 1</screen>
286 <para>Change values during system downtime to minimize the chance of any
287 ownership or permissions problems if the policy group is active.
288 Although changes can be made live, client caching of data may interfere
289 with modification as there are a few seconds of lead time before the
290 change is distributed.</para>
294 <title>Mixing Properties</title>
296 <para>With both <literal>admin</literal> and <literal>trusted</literal>
297 properties set, the policy group has full access, as if nodemap was
298 turned off, to the Lustre file system. The administrative site for the
299 Lustre file system needs at least one group with both properties in
300 order to perform maintenance or to perform administrative tasks. </para>
303 <para>MDS systems <emphasis role="bold">must</emphasis> be in a policy
304 group with both these properties set to 1. It is recommended to put the
305 MDS in a policy group labeled “TrustedSystems” or some identifier that
306 makes the association clear.</para>
309 <para>If a policy group has the <literal>admin</literal>
310 property set, but does not have the property
311 <literal>trusted</literal> set, root is mapped directly to
312 root, any explicitly specified UID and GID idmaps are honored, and
313 other access is squashed. If root alters ownership to UIDs or GIDs
314 which are locally known from that host but not part of an idmap, root
315 effectively changes ownership of those files to the default
316 squashed UID and GID.</para>
318 <para>If <literal>trusted</literal> is set but <literal>admin</literal>
319 is not, the policy group has full access to the canonical UID and GID
320 sets of the Lustre file system, and root is squashed.</para>
322 <para>The deny_unknown property, once enabled, prevents unmapped users
323 from accessing the file system. Root access also is denied, if the
324 <literal>admin</literal> property is off, and root is not part of any
327 <para>When nodemaps are modified, the change events are queued and
328 distributed across the cluster. Under normal conditions, these changes
329 can take around ten seconds to propagate. During this distribution
330 window, file access could be made via the old or new nodemap settings.
331 Therefore, it is recommended to save changes for a maintenance window
332 or to deploy them while the mapped nodes are not actively writing to the
338 <section xml:id="enablingthefeature">
339 <title>Enabling the Feature</title>
341 <para>The nodemap feature is simple to enable:</para>
343 <screen>mgs# lctl nodemap_activate 1</screen>
345 <para>Passing the parameter 0 instead of 1 disables the feature again.
346 After deploying the feature, validate the mappings are intact before
347 offering the file system to be mounted by clients.</para>
349 <para condition='l28'>So far, changes have been made on the MGS. Prior to
350 Lustre 2.9, changes must also be manually set on MDS systems as well.
351 Also, changes must be manually deployed to OSS servers if quota
352 is enforced, utilizing <literal>lctl set_param</literal>
353 instead of <literal>lctl</literal>. Prior to 2.9,
354 the configuration is not persistent, requiring a script
355 which generates the mapping to be saved and deployed after every Lustre
356 restart. As an example, use this style to deploy settings on the
359 <screen>oss# lctl set_param nodemap.add_nodemap=<replaceable>SiteName</replaceable>
360 oss# lctl set_param nodemap.add_nodemap_range='<replaceable>SiteName 192.168.0.15@tcp</replaceable>'
361 oss# lctl set_param nodemap.add_nodemap_idmap='<replaceable>SiteName</replaceable> uid <replaceable>510:1700</replaceable>'
362 oss# lctl set_param nodemap.add_nodemap_idmap='<replaceable>SiteName</replaceable> gid <replaceable>612:1702</replaceable>'</screen>
364 In Lustre 2.9 and later, nodemap
365 configuration is saved on the MGS and distributed automatically to
366 MGS, MDS, and OSS nodes, a process which takes approximately
367 ten seconds in normal circumstances.</para>
370 <section xml:id="defaultNodemap">
371 <title><literal>default</literal> Nodemap</title>
373 <para>There is a special nodemap called <literal>default</literal>. As the
374 name suggests, it is created by default and cannot be removed. It is like
375 a fallback nodemap, setting the behaviour for Lustre clients that do not
376 match any other nodemap.</para>
377 <para>Because of its special role, only some parameters can be set on the
378 <literal>default</literal> nodemap:</para>
381 <para><literal>admin</literal></para>
384 <para><literal>trusted</literal></para>
387 <para><literal>squash_uid</literal></para>
390 <para><literal>squash_gid</literal></para>
393 <para><literal>fileset</literal></para>
396 <para><literal>audit_mode</literal></para>
399 <para>In particular, no UID/GID mapping can be defined on the
400 <literal>default</literal> nodemap.</para>
402 <para>Be careful when altering the <literal>admin</literal> and
403 <literal>trusted</literal> properties of the <literal>default</literal>
404 nodemap, especially if your Lustre servers fall into this nodemap.
409 <section xml:id="verifyingsettings">
410 <title>Verifying Settings</title>
412 <para>By using <literal>lctl nodemap_info all</literal>, existing nodemap
413 configuration is listed for easy export. This command acts as a shortcut
414 into the configuration interface for nodemap. On the Lustre MGS, the
415 <literal>nodemap.active</literal> parameter contains a <literal>1</literal>
416 if nodemap is active on the system. Each policy group
417 creates a directory containing the following parameters:</para>
421 <para><literal>admin</literal> and <literal>trusted</literal> each
422 contain a <literal>1</literal> if the values are set, and
423 <literal>0</literal> otherwise.</para>
427 <para><literal>idmap</literal> contains a list of the idmaps for the
428 policy group, while <literal>ranges</literal> contains a list of
429 NIDs for the group.</para>
433 <para><literal>squash_uid</literal> and <literal>squash_gid</literal>
434 determine what UID and GID users are squashed to if needed.</para>
438 <para>The expected outputs for the BirdResearchSite in the example above
441 <screen>mgs# lctl get_param nodemap.BirdResearchSite.idmap
444 { idtype: uid, client_id: 530, fs_id: 11000 },
445 { idtype: uid, client_id: 531, fs_id: 11001 },
446 { idtype: uid, client_id: 532, fs_id: 11002 },
447 { idtype: uid, client_id: 533, fs_id: 11003 },
448 { idtype: gid, client_id: 600, fs_id: 11000 },
449 { idtype: gid, client_id: 601, fs_id: 11001 }
452 mgs# lctl get_param nodemap.BirdResearchSite.ranges
454 { id: 11, start_nid: 192.168.0.100@tcp, end_nid: 192.168.0.100@tcp }
458 <section xml:id="ensuringconsistency">
459 <title>Ensuring Consistency</title>
461 <para>Consistency issues may arise in a nodemap enabled configuration when
462 Lustre clients mount from an unknown NID range, new UIDs and GIDs that
463 were not part of a known map are added, or there are misconfigurations in
464 the rules. Keep in mind the following when activating nodemap
465 on a production system:</para>
469 <para>Creating new policy groups or idmaps on a production system
470 is allowed, but reserve a maintenance window to alter the <literal>
471 trusted</literal> property to avoid metadata problems.</para>
475 <para>To perform administrative tasks, access the Lustre file system
476 via a policy group with <literal>trusted</literal>
477 and <literal>admin</literal> properties set. This prevents
478 the creation of orphaned and squashed files. Granting the
479 <literal>admin</literal> property without the
480 <literal>trusted</literal> property
481 is dangerous. The root user on the client may know of UIDs
482 and GIDs that are not present in any idmap. If root alters ownership
483 to those identifiers, the ownership is squashed as a result. For
484 example, tar file extracts may be flipped from an expected UID
485 such as UID 500 to <literal>nobody</literal>, normally UID 99.</para>
489 <para>To map distinct UIDs at two or more sites onto a single UID or GID
490 on the Lustre file system, create overlapping idmaps and place each site
491 in its own policy group. Each distinct UID may have its own mapping onto
492 the target UID or GID.</para>
496 <para condition='l28'>In Lustre 2.8, changes must be manually kept in a
497 script file to be re-applied after a Lustre reload, and changes must be
498 made on each OSS, MDS, and MGS nodes, as there is no automatic
499 synchronization between the nodes.</para>
503 <para>If <literal>deny_unknown</literal> is in effect, it is possible
504 for unmapped users to see dentries which were viewed by a mapped user.
505 This is a result of client caching, and unmapped users will not be able
506 to view any file contents.</para>
510 <para>Nodemap activation status can be checked with
511 <literal>lctl nodemap_info</literal>,
512 but extra validation is possible. One way of ensuring valid
513 deployment on a production system is to create a fingerprint of known
514 files with specific UIDs and GIDs mapped to a test
515 client. After bringing the Lustre system online after maintenance, the
516 test client can validate the UIDs and GIDs map correctly before the
517 system is mounted in user space.</para>