1 <?xml version="1.0" encoding="UTF-8"?>
2 <chapter version="5.0" xml:id="lustrenodemap" xml:lang="en-US"
4 xmlns="http://docbook.org/ns/docbook">
5 <title xml:id="lustrenodemap.title">Mapping UIDs and GIDs with
8 <para>This chapter describes how to map UID and GIDs across a Lustre file
9 system using the nodemap feature, and includes the following
14 <para><xref linkend="settingamapping"/></para>
18 <para><xref linkend="alteringproperties"/></para>
22 <para><xref linkend="enablingthefeature"/></para>
26 <para><xref linkend="verifyingsettings"/></para>
30 <para><xref linkend="ensuringconsistency"/></para>
34 <section xml:id="settingamapping">
35 <title>Setting a Mapping</title>
37 <para>The nodemap feature supported in Lustre 2.9 was first
38 introduced in Lustre 2.7 as a technology preview. It allows UIDs and GIDs
39 from remote systems to be mapped to local sets of UIDs and GIDs while
40 retaining POSIX ownership, permissions and quota information. As a result,
41 multiple sites with conflicting user and group identifiers can operate on
42 a single Lustre file system without creating collisions in UID or GID
46 <title>Defining Terms</title>
48 <para>When the nodemap feature is enabled, client file system access to
49 a Lustre system is filtered through the nodemap identity mapping policy
50 engine. Lustre connectivity is governed by network identifiers, or
51 <emphasis>NIDs</emphasis>, such as
52 <literal>192.168.7.121@tcp</literal>. When an operation is made from a
53 NID, Lustre decides if that NID is part of a
54 <emphasis>nodemap</emphasis>, a policy group consisting of one or
55 more NID ranges. If no policy group exists for that NID, access is
56 squashed to user <literal>nobody</literal> by default. Each policy group
57 also has several <emphasis>properties</emphasis>, such as
58 <literal>trusted</literal>
59 and <literal>admin</literal>, which determine access conditions.
60 A collection of identity maps or
61 <emphasis>idmaps</emphasis> are kept for each policy group. These
62 idmaps determine how UIDs and GIDs on the client are translated into the
63 canonical user space of the local Lustre file system.</para>
65 <para>In order for nodemap to function properly, the MGS, MDS, and OSS
66 systems must all have a version of Lustre which supports nodemap.
67 Clients operate transparently and do not require special
68 configuration or knowledge of the nodemap setup.</para>
72 <title>Deciding on NID Ranges</title>
74 <para>NIDs can be described as either a singleton address or a range of
75 addresses. A single address is described in standard Lustre NID format,
76 such as <literal>10.10.6.120@tcp</literal>. A range
77 is described using a dash to separate the range, for example,
78 <literal>192.168.20.[0-255]@tcp</literal>.</para>
80 <para>The range must be contiguous. The full LNET definition for a
81 nidlist is as follows:</para>
84 <nidlist> :== <nidrange> [ ' ' <nidrange> ]
85 <nidrange> :== <addrrange> '@' <net>
86 <addrrange> :== '*' |
87 <ipaddr_range> |
89 <ipaddr_range> :==
90 <numaddr_range>.<numaddr_range>.<numaddr_range>.<numaddr_range>
91 <numaddr_range> :== <number> |
93 <expr_list> :== '[' <range_expr> [ ',' <range_expr>] ']'
94 <range_expr> :== <number> |
95 <number> '-' <number> |
96 <number> '-' <number> '/' <number>
97 <net> :== <netname> | <netname><number>
98 <netname> :== "lo" | "tcp" | "o2ib" | "gni"
99 <number> :== <nonnegative decimal> | <hexadecimal></screen>
103 <title>Describing and Deploying a Sample Mapping</title>
105 <para>Deploy nodemap by first considering which users need to be
106 mapped, and what sets of network addresses or ranges are involved.
107 Issues of visibility between users must be examined as well.</para>
109 <para>Consider a deployment where researchers are working on data
110 relating to birds. The researchers use a computing system which mounts
111 Lustre from a single IPv4 address, <literal>192.168.0.100</literal>.
112 Name this policy group <literal>BirdResearchSite</literal>. The IP
113 address forms the NID <literal>192.168.0.100@tcp</literal>. Create the
114 policy group and add the NID to that group on the MGS
115 using the <literal>lctl</literal> command:</para>
117 <screen>mgs# lctl nodemap_add <replaceable>BirdResearchSite</replaceable>
118 mgs# lctl nodemap_add_range --name <replaceable>BirdResearchSite</replaceable> --range 192.168.0.100@tcp</screen>
121 <para>A NID cannot be in more than one policy group. Assign a NID to
122 a new policy group by first removing it from the existing group.</para>
125 <para>The researchers use the following identifiers on their host system:</para>
129 <para><literal>swan</literal> (UID 530) member of group
130 <literal>wetlands</literal> (GID 600)</para>
134 <para><literal>duck</literal> (UID 531) member of group
135 <literal>wetlands</literal> (GID 600)</para>
139 <para><literal>hawk</literal> (UID 532) member of group
140 <literal>raptor</literal> (GID 601)</para>
144 <para><literal>merlin</literal> (UID 533) member of group
145 <literal>raptor</literal> (GID 601)</para>
149 <para>Assign a set of six idmaps to this policy group, with four for UIDs,
150 and two for GIDs. Pick a starting point, e.g. UID 11000, with room for
151 additional UIDs and GIDs to be added as the configuration grows.
152 Use the <literal>lctl</literal> command to set up the idmaps:</para>
154 <screen>mgs# lctl nodemap_add_idmap --name <replaceable>BirdResearchSite</replaceable> --idtype uid --idmap <replaceable>530:11000</replaceable>
155 mgs# lctl nodemap_add_idmap --name <replaceable>BirdResearchSite</replaceable> --idtype uid --idmap <replaceable>531:11001</replaceable>
156 mgs# lctl nodemap_add_idmap --name <replaceable>BirdResearchSite</replaceable> --idtype uid --idmap <replaceable>532:11002</replaceable>
157 mgs# lctl nodemap_add_idmap --name <replaceable>BirdResearchSite</replaceable> --idtype uid --idmap <replaceable>533:11003</replaceable>
158 mgs# lctl nodemap_add_idmap --name <replaceable>BirdResearchSite</replaceable> --idtype gid --idmap <replaceable>600:11000</replaceable>
159 mgs# lctl nodemap_add_idmap --name <replaceable>BirdResearchSite</replaceable> --idtype gid --idmap <replaceable>601:11001</replaceable></screen>
161 <para>The parameter <literal>530:11000</literal> assigns a client UID,
162 for example UID 530, to a single canonical UID,
163 such as UID 11000. Each assignment is made individually. There is no
164 method to specify a range <literal>530-533:11000-11003</literal>.
165 UID and GID idmaps are assigned separately. There is no implied
166 relationship between the two.</para>
168 <para>Files created on the Lustre file system from the
169 <literal>192.168.0.100@tcp</literal> NID using UID
170 <literal>duck</literal> and GID <literal>wetlands</literal> are stored
171 in the Lustre file system using the canonical identifiers, in this case
172 UID 11001 and GID 11000. A different NID, if not part of the same policy
173 group, sees its own view of the same file space.</para>
175 <para>Suppose a previously created project directory exists owned by UID
176 11002/GID 11001, with mode 770. When users <literal>hawk</literal> and
177 <literal>merlin</literal> at 192.168.0.100 place files named
178 <literal>hawk-file</literal> and <literal>merlin-file</literal> into the
179 directory, the contents from the 192.168.0.100 client appear as:</para>
181 <screen>[merlin@192.168.0.100 projectsite]$ ls -la
183 drwxrwx--- 2 hawk raptor 4096 Jul 23 09:06 .
184 drwxr-xr-x 3 nobody nobody 4096 Jul 23 09:02 ..
185 -rw-r--r-- 1 hawk raptor 10240000 Jul 23 09:05 hawk-file
186 -rw-r--r-- 1 merlin raptor 25100288 Jul 23 09:06 merlin-file</screen>
188 <para>From a privileged view, the canonical owners are displayed:</para>
190 <screen>[root@trustedSite projectsite]# ls -la
192 drwxrwx--- 2 11002 11001 4096 Jul 23 09:06 .
193 drwxr-xr-x 3 root root 4096 Jul 23 09:02 ..
194 -rw-r--r-- 1 11002 11001 10240000 Jul 23 09:05 hawk-file
195 -rw-r--r-- 1 11003 11001 25100288 Jul 23 09:06 merlin-file</screen>
197 <para>If UID 11002 or GID 11001 do not exist on the Lustre MDS or MGS,
198 create them in LDAP or other data sources, or trust clients by setting
199 <literal>identity_upcall</literal> to <literal>NONE</literal>. For more
200 information, see <xref linkend="dbdoclet.50438291_32926"/>.</para>
202 <para>Building a larger and more complex configuration is possible by
203 iterating through the <literal>lctl</literal> commands above. In
208 <para>Create a name for the policy group.</para>
212 <para>Create a set of NID ranges used by the
217 <para>Define which UID and GID translations need to occur for the
224 <section xml:id="alteringproperties">
225 <title>Altering Properties</title>
227 <para>Privileged users access mapped systems with rights dependent on
228 certain properties, described below. By default, root access is squashed
229 to user <literal>nobody</literal>, which interferes with most
230 administrative actions.</para>
233 <title>Managing the Properties</title>
235 <para>Several properties exist, off by default, which change
236 client behavior: <literal>admin</literal>,
237 <literal>trusted</literal>, <literal>squash_uid</literal>,
238 <literal>squash_gid</literal>, and <literal>deny_unknown</literal>.
243 <para>The <literal>trusted</literal> property permits members
244 of a policy group to see the file system's canonical identifiers.
245 In the above example, UID 11002 and GID 11001 will be seen without
246 translation. This can be utilized when local UID and GID sets
247 already map directly to the specified users.</para>
251 <para>The property <literal>admin</literal> defines whether
252 root is squashed on the policy group. By default, it is
253 squashed, unless this property is enabled. Coupled with the
254 <literal>trusted</literal> property, this will allow unmapped
255 access for backup nodes, transfer points, or other administrative
260 <para>The property <literal>deny_unknown</literal> denies all access
261 to users not mapped in a particular nodemap. This is useful if a site
262 is concerned about unmapped users accessing the file system in order to
263 satisfy security requirements.
268 <para>The properties <literal>squash_uid</literal> and <literal>
269 squash_gid</literal> define the default UID and GID that users will
270 be squashed to if unmapped, unless the deny_unknown flag is set, in
271 which case access will still be denied.
276 <para>Alter values to either true (1) or false (0) on the MGS:</para>
278 <screen>mgs# lctl nodemap_modify --name <replaceable>BirdAdminSite</replaceable> --property trusted --value 1
279 mgs# lctl nodemap_modify --name <replaceable>BirdAdminSite</replaceable> --property admin --value 1
280 mgs# lctl nodemap_modify --name <replaceable>BirdAdminSite</replaceable> --property deny_unknown --value 1</screen>
282 <para>Change values during system downtime to minimize the chance of any
283 ownership or permissions problems if the policy group is active.
284 Although changes can be made live, client caching of data may interfere
285 with modification as there are a few seconds of lead time before the
286 change is distributed.</para>
290 <title>Mixing Properties</title>
292 <para>With both <literal>admin</literal> and <literal>trusted</literal>
293 properties set, the policy group has full access, as if nodemap was
294 turned off, to the Lustre file system. The administrative site for the
295 Lustre file system needs at least one group with both properties in
296 order to perform maintenance or to perform administrative tasks. </para>
299 <para>MDS systems <emphasis role="bold">must</emphasis> be in a policy
300 group with both these properties set to 1. It is recommended to put the
301 MDS in a policy group labeled “TrustedSystems” or some identifier that
302 makes the association clear.</para>
305 <para>If a policy group has the <literal>admin</literal>
306 property set, but does not have the property
307 <literal>trusted</literal> set, root is mapped directly to
308 root, any explicitly specified UID and GID idmaps are honored, and
309 other access is squashed. If root alters ownership to UIDs or GIDs
310 which are locally known from that host but not part of an idmap, root
311 effectively changes ownership of those files to the default
312 squashed UID and GID.</para>
314 <para>If <literal>trusted</literal> is set but <literal>admin</literal>
315 is not, the policy group has full access to the canonical UID and GID
316 sets of the Lustre file system, and root is squashed.</para>
318 <para>The deny_unknown property, once enabled, prevents unmapped users
319 from accessing the file system. Root access also is denied, if the
320 <literal>admin</literal> property is off, and root is not part of any
323 <para>When nodemaps are modified, the change events are queued and
324 distributed across the cluster. Under normal conditions, these changes
325 can take around ten seconds to propagate. During this distribution
326 window, file access could be made via the old or new nodemap settings.
327 Therefore, it is recommended to save changes for a maintenance window
328 or to deploy them while the mapped nodes are not actively writing to the
334 <section xml:id="enablingthefeature">
335 <title>Enabling the Feature</title>
337 <para>The nodemap feature is simple to enable:</para>
339 <screen>mgs# lctl nodemap_activate 1</screen>
341 <para>Passing the parameter 0 instead of 1 disables the feature again.
342 After deploying the feature, validate the mappings are intact before
343 offering the file system to be mounted by clients.</para>
345 <para condition='l28'>So far, changes have been made on the MGS. Prior to
346 Lustre 2.9, changes must also be manually set on MDS systems as well.
347 Also, changes must be manually deployed to OSS servers if quota
348 is enforced, utilizing <literal>lctl set_param</literal>
349 instead of <literal>lctl</literal>. Prior to 2.9,
350 the configuration is not persistent, requiring a script
351 which generates the mapping to be saved and deployed after every Lustre
352 restart. As an example, use this style to deploy settings on the
355 <screen>oss# lctl set_param nodemap.add_nodemap=<replaceable>SiteName</replaceable>
356 oss# lctl set_param nodemap.add_nodemap_range='<replaceable>SiteName 192.168.0.15@tcp</replaceable>'
357 oss# lctl set_param nodemap.add_nodemap_idmap='<replaceable>SiteName</replaceable> uid <replaceable>510:1700</replaceable>'
358 oss# lctl set_param nodemap.add_nodemap_idmap='<replaceable>SiteName</replaceable> gid <replaceable>612:1702</replaceable>'</screen>
360 In Lustre 2.9 and later, nodemap
361 configuration is saved on the MGS and distributed automatically to
362 MGS, MDS, and OSS nodes, a process which takes approximately
363 ten seconds in normal circumstances.</para>
366 <section xml:id="verifyingsettings">
367 <title>Verifying Settings</title>
369 <para>By using <literal>lctl nodemap_info all</literal>, existing
370 nodemap configuration is listed for easy export. This command
371 acts as a shortcut into the /proc interface for nodemap.
372 Within /proc/fs/lustre/nodemap/ on the Lustre MGS, the
373 file <literal>active</literal> contains a 1 if nodemap is active on the
374 system. Each policy group creates a directory containing the
375 following parameters:</para>
379 <para><literal>admin</literal> and
380 <literal>trusted</literal> each contain a ‘1’ if the values
381 are set, and a ‘0’ otherwise.</para>
385 <para><literal>idmap</literal> contains a list of the idmaps for the
386 policy group, while <literal>ranges</literal> contains a list of
387 NIDs for the group.</para>
391 <para><literal>squash_uid</literal> and <literal>squash_gid</literal>
392 determine what UID and GID users are squashed to if needed.</para>
396 <para>The expected outputs for the BirdResearchSite in the example above
399 <screen>mgs# lctl get_param nodemap.BirdResearchSite.idmap
402 { idtype: uid, client_id: 530, fs_id: 11000 },
403 { idtype: uid, client_id: 531, fs_id: 11001 },
404 { idtype: uid, client_id: 532, fs_id: 11002 },
405 { idtype: uid, client_id: 533, fs_id: 11003 },
406 { idtype: gid, client_id: 600, fs_id: 11000 },
407 { idtype: gid, client_id: 601, fs_id: 11001 }
410 mgs# lctl get_param nodemap.BirdResearchSite.ranges
412 { id: 11, start_nid: 192.168.0.100@tcp, end_nid: 192.168.0.100@tcp }
416 <section xml:id="ensuringconsistency">
417 <title>Ensuring Consistency</title>
419 <para>Consistency issues may arise in a nodemap enabled configuration when
420 Lustre clients mount from an unknown NID range, new UIDs and GIDs that
421 were not part of a known map are added, or there are misconfigurations in
422 the rules. Keep in mind the following when activating nodemap
423 on a production system:</para>
427 <para>Creating new policy groups or idmaps on a production system
428 is allowed, but reserve a maintenance window to alter the <literal>
429 trusted</literal> property to avoid metadata problems.</para>
433 <para>To perform administrative tasks, access the Lustre file system
434 via a policy group with <literal>trusted</literal>
435 and <literal>admin</literal> properties set. This prevents
436 the creation of orphaned and squashed files. Granting the
437 <literal>admin</literal> property without the
438 <literal>trusted</literal> property
439 is dangerous. The root user on the client may know of UIDs
440 and GIDs that are not present in any idmap. If root alters ownership
441 to those identifiers, the ownership is squashed as a result. For
442 example, tar file extracts may be flipped from an expected UID
443 such as UID 500 to <literal>nobody</literal>, normally UID 99.</para>
447 <para>To map distinct UIDs at two or more sites onto a single UID or GID
448 on the Lustre file system, create overlapping idmaps and place each site
449 in its own policy group. Each distinct UID may have its own mapping onto
450 the target UID or GID.</para>
454 <para condition='l28'>In Lustre 2.8, changes must be manually kept in a
455 script file to be re-applied after a Lustre reload, and changes must be
456 made on each OSS, MDS, and MGS nodes, as there is no automatic
457 synchronization between the nodes.</para>
461 <para>If <literal>deny_unknown</literal> is in effect, it is possible
462 for unmapped users to see dentries which were viewed by a mapped user.
463 This is a result of client caching, and unmapped users will not be able
464 to view any file contents.</para>
468 <para>Nodemap activation status can be checked with
469 <literal>lctl nodemap_info</literal>,
470 but extra validation is possible. One way of ensuring valid
471 deployment on a production system is to create a fingerprint of known
472 files with specific UIDs and GIDs mapped to a test
473 client. After bringing the Lustre system online after maintenance, the
474 test client can validate the UIDs and GIDs map correctly before the
475 system is mounted in user space.</para>