1 <?xml version="1.0" encoding="UTF-8"?>
2 <chapter xmlns="http://docbook.org/ns/docbook"
3 xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"
4 xml:id="lustrenodemap" condition='l29'>
5 <title xml:id="lustrenodemap.title">Mapping UIDs and GIDs with
8 <para>This chapter describes how to map UID and GIDs across a Lustre file
9 system using the nodemap feature, and includes the following
14 <para><xref linkend="settingamapping"/></para>
18 <para><xref linkend="alteringproperties"/></para>
22 <para><xref linkend="enablingthefeature"/></para>
26 <para><xref linkend="defaultNodemap"/></para>
30 <para><xref linkend="verifyingsettings"/></para>
34 <para><xref linkend="ensuringconsistency"/></para>
38 <section xml:id="settingamapping">
39 <title>Setting a Mapping</title>
41 <para>The nodemap feature supported in Lustre 2.9 was first
42 introduced in Lustre 2.7 as a technology preview. It allows UIDs and GIDs
43 from remote systems to be mapped to local sets of UIDs and GIDs while
44 retaining POSIX ownership, permissions and quota information. As a result,
45 multiple sites with conflicting user and group identifiers can operate on
46 a single Lustre file system without creating collisions in UID or GID
50 <title>Defining Terms</title>
52 <para>When the nodemap feature is enabled, client file system access to
53 a Lustre system is filtered through the nodemap identity mapping policy
54 engine. Lustre connectivity is governed by network identifiers, or
55 <emphasis>NIDs</emphasis>, such as
56 <literal>192.168.7.121@tcp</literal>. When an operation is made from a
57 NID, Lustre decides if that NID is part of a
58 <emphasis>nodemap</emphasis>, a policy group consisting of one or
59 more NID ranges. If no policy group exists for that NID, access is
60 squashed to user <literal>nobody</literal> by default. Each policy group
61 also has several <emphasis>properties</emphasis>, such as
62 <literal>trusted</literal>
63 and <literal>admin</literal>, which determine access conditions.
64 A collection of identity maps or
65 <emphasis>idmaps</emphasis> are kept for each policy group. These
66 idmaps determine how UIDs and GIDs on the client are translated into the
67 canonical user space of the local Lustre file system.</para>
69 <para>In order for nodemap to function properly, the MGS, MDS, and OSS
70 systems must all have a version of Lustre which supports nodemap.
71 Clients operate transparently and do not require special
72 configuration or knowledge of the nodemap setup.</para>
76 <title>Deciding on NID Ranges</title>
78 <para>NIDs can be described as either a singleton address or a range of
79 addresses. A single address is described in standard Lustre NID format,
80 such as <literal>10.10.6.120@tcp</literal>. A range
81 is described using a dash to separate the range, for example,
82 <literal>192.168.20.[0-255]@tcp</literal>.</para>
84 <para>The range must be contiguous. The full LNet definition for a
85 nidlist is as follows:</para>
88 <nidlist> :== <nidrange> [ ' ' <nidrange> ]
89 <nidrange> :== <addrrange> '@' <net>
90 <addrrange> :== '*' |
91 <ipaddr_range> |
93 <ipaddr_range> :==
94 <numaddr_range>.<numaddr_range>.<numaddr_range>.<numaddr_range>
95 <numaddr_range> :== <number> |
97 <expr_list> :== '[' <range_expr> [ ',' <range_expr>] ']'
98 <range_expr> :== <number> |
99 <number> '-' <number> |
100 <number> '-' <number> '/' <number>
101 <net> :== <netname> | <netname><number>
102 <netname> :== "lo" | "tcp" | "o2ib" | "gni"
103 <number> :== <nonnegative decimal> | <hexadecimal></screen>
107 <title>Defining a Servers Specific Group</title>
109 <para>For proper operations, the Lustre file system
110 <emphasis role="bold">requires</emphasis> to have a privileged group that
111 covers all Lustre server nodes. So the very first step when working with
112 nodemaps is to create such a group with both properties
113 <literal>admin</literal> and <literal>trusted</literal> set. It is
114 recommended to give this group an explicit label such as “TrustedSystems”
115 or some identifier that makes the association clear.</para>
117 <para>Let's consider a deployment where the server nodes are in the NID
118 range <literal>192.168.0.[1-10]@tcp</literal>. Create the policy group,
119 add the NID range to that group, and set the properties accordingly using
120 the <literal>lctl</literal> command on the MGS:</para>
122 <screen>mgs# lctl nodemap_add <replaceable>TrustedSystems</replaceable>
123 mgs# lctl nodemap_add_range --name <replaceable>TrustedSystems</replaceable> --range 192.168.0.[1-10]@tcp
124 mgs# lctl nodemap_modify --name <replaceable>TrustedSystems</replaceable> --property admin --value 1
125 mgs# lctl nodemap_modify --name <replaceable>TrustedSystems</replaceable> --property trusted --value 1
131 <title>Describing and Deploying a Sample Mapping</title>
133 <para>Deploy nodemap by first considering which users need to be
134 mapped, and what sets of network addresses or ranges are involved.
135 Issues of visibility between users must be examined as well.</para>
137 <para>Consider a deployment where researchers are working on data
138 relating to birds. The researchers use a computing system which mounts
139 Lustre from a single IPv4 address, <literal>192.168.0.100</literal>.
140 Name this policy group <literal>BirdResearchSite</literal>. The IP
141 address forms the NID <literal>192.168.0.100@tcp</literal>. Create the
142 policy group and add the NID to that group on the MGS
143 using the <literal>lctl</literal> command:</para>
145 <screen>mgs# lctl nodemap_add <replaceable>BirdResearchSite</replaceable>
146 mgs# lctl nodemap_add_range --name <replaceable>BirdResearchSite</replaceable> --range 192.168.0.100@tcp</screen>
149 <para>A NID cannot be in more than one policy group. Assign a NID to
150 a new policy group by first removing it from the existing group.</para>
153 <para>The researchers use the following identifiers on their host system:</para>
157 <para><literal>swan</literal> (UID 530) member of group
158 <literal>wetlands</literal> (GID 600)</para>
162 <para><literal>duck</literal> (UID 531) member of group
163 <literal>wetlands</literal> (GID 600)</para>
167 <para><literal>hawk</literal> (UID 532) member of group
168 <literal>raptor</literal> (GID 601)</para>
172 <para><literal>merlin</literal> (UID 533) member of group
173 <literal>raptor</literal> (GID 601)</para>
177 <para>Assign a set of six idmaps to this policy group, with four for UIDs,
178 and two for GIDs. Pick a starting point, e.g. UID 11000, with room for
179 additional UIDs and GIDs to be added as the configuration grows.
180 Use the <literal>lctl</literal> command to set up the idmaps:</para>
182 <screen>mgs# lctl nodemap_add_idmap --name <replaceable>BirdResearchSite</replaceable> --idtype uid --idmap <replaceable>530:11000</replaceable>
183 mgs# lctl nodemap_add_idmap --name <replaceable>BirdResearchSite</replaceable> --idtype uid --idmap <replaceable>531:11001</replaceable>
184 mgs# lctl nodemap_add_idmap --name <replaceable>BirdResearchSite</replaceable> --idtype uid --idmap <replaceable>532:11002</replaceable>
185 mgs# lctl nodemap_add_idmap --name <replaceable>BirdResearchSite</replaceable> --idtype uid --idmap <replaceable>533:11003</replaceable>
186 mgs# lctl nodemap_add_idmap --name <replaceable>BirdResearchSite</replaceable> --idtype gid --idmap <replaceable>600:11000</replaceable>
187 mgs# lctl nodemap_add_idmap --name <replaceable>BirdResearchSite</replaceable> --idtype gid --idmap <replaceable>601:11001</replaceable></screen>
189 <para>The parameter <literal>530:11000</literal> assigns a client UID,
190 for example UID 530, to a single canonical UID,
191 such as UID 11000. Each assignment is made individually. There is no
192 method to specify a range <literal>530-533:11000-11003</literal>.
193 UID and GID idmaps are assigned separately. There is no implied
194 relationship between the two.</para>
196 <para>Files created on the Lustre file system from the
197 <literal>192.168.0.100@tcp</literal> NID using UID
198 <literal>duck</literal> and GID <literal>wetlands</literal> are stored
199 in the Lustre file system using the canonical identifiers, in this case
200 UID 11001 and GID 11000. A different NID, if not part of the same policy
201 group, sees its own view of the same file space.</para>
203 <para>Suppose a previously created project directory exists owned by UID
204 11002/GID 11001, with mode 770. When users <literal>hawk</literal> and
205 <literal>merlin</literal> at 192.168.0.100 place files named
206 <literal>hawk-file</literal> and <literal>merlin-file</literal> into the
207 directory, the contents from the 192.168.0.100 client appear as:</para>
209 <screen>[merlin@192.168.0.100 projectsite]$ ls -la
211 drwxrwx--- 2 hawk raptor 4096 Jul 23 09:06 .
212 drwxr-xr-x 3 nobody nobody 4096 Jul 23 09:02 ..
213 -rw-r--r-- 1 hawk raptor 10240000 Jul 23 09:05 hawk-file
214 -rw-r--r-- 1 merlin raptor 25100288 Jul 23 09:06 merlin-file</screen>
216 <para>From a privileged view, the canonical owners are displayed:</para>
218 <screen>[root@trustedSite projectsite]# ls -la
220 drwxrwx--- 2 11002 11001 4096 Jul 23 09:06 .
221 drwxr-xr-x 3 root root 4096 Jul 23 09:02 ..
222 -rw-r--r-- 1 11002 11001 10240000 Jul 23 09:05 hawk-file
223 -rw-r--r-- 1 11003 11001 25100288 Jul 23 09:06 merlin-file</screen>
225 <para>If UID 11002 or GID 11001 do not exist on the Lustre MDS or MGS,
226 create them in LDAP or other data sources, or trust clients by setting
227 <literal>identity_upcall</literal> to <literal>NONE</literal>. For more
228 information, see <xref linkend="identity_upcall"/>.</para>
230 <para>Building a larger and more complex configuration is possible by
231 iterating through the <literal>lctl</literal> commands above. In
236 <para>Create a name for the policy group.</para>
240 <para>Create a set of NID ranges used by the
245 <para>Define which UID and GID translations need to occur for the
250 <section remap="h3" condition='l2F'>
251 <title>Mapping Project IDs</title>
252 <para>Like UIDs and GIDs, PROJIDs can be mapped via nodemaps, from client
253 to file system IDs and conversely. To declare a PROJID mapping, use the
254 <literal>projid</literal> type:</para>
255 <screen>mgs# lctl nodemap_add_idmap --name <replaceable>BirdResearchSite</replaceable> --idtype projid --idmap <replaceable>33:1</replaceable></screen>
259 <section xml:id="alteringproperties">
260 <title>Altering Properties</title>
262 <para>Privileged users access mapped systems with rights dependent on
263 certain properties, described below. By default, root access is squashed
264 to user <literal>nobody</literal>, which interferes with most
265 administrative actions.</para>
267 <para>For proper operations, the Lustre file system
268 <emphasis role="bold">requires</emphasis> a group that covers all Lustre
269 server nodes, with both properties <literal>admin</literal> and
270 <literal>trusted</literal> set. It is recommended to give this group an
271 explicit label such as “TrustedSystems” or some identifier that makes the
272 association clear.</para>
274 <section xml:id="lustrenodemap.alteringproperties.managing" remap="h3">
275 <title>Managing the Properties</title>
277 <para>Several properties exist, off by default, which change
278 client behavior: <literal>admin</literal>,
279 <literal>trusted</literal>, <literal>map_mode</literal>,
280 <literal>squash_uid</literal>, <literal>squash_gid</literal>,
281 <literal>squash_projid</literal>, <literal>deny_unknown</literal>,
282 <literal>audit_mode</literal> and <literal>forbid_encryption</literal>.
287 <para>The property <literal>admin</literal> defines whether
288 root is squashed on the policy group. By default, it is
289 squashed, unless this property is enabled. Coupled with the
290 <literal>trusted</literal> property, this will allow unmapped
291 access for backup nodes, transfer points, or other administrative
296 <para>The <literal>trusted</literal> property permits members
297 of a policy group to see the file system's canonical identifiers.
298 In the above example, UID 11002 and GID 11001 will be seen without
299 translation. This can be utilized when local UID and GID sets
300 already map directly to the specified users.</para>
304 <para condition='l2A'>The <literal>map_mode</literal> property lets
305 control the way mapping is carried out. By default it is set to
306 <literal>all</literal> which means the nodemap will map UIDs, GIDs,
307 and PROJIDs. If set to <literal>uid_only</literal> or just
308 <literal>uid</literal>, only UIDs will be mapped. If set to
309 <literal>gid_only</literal> or just <literal>gid</literal>, only GIDs
310 will be mapped. If set to <literal>projid_only</literal> or just
311 <literal>projid</literal>, only PROJIDs will be mapped. If set to
312 <literal>both</literal>, both UIDs and GIDs will be mapped. Multiple
313 values can be specified, comma separated.</para>
317 <para>The properties <literal>squash_uid</literal>, <literal>
318 squash_gid</literal> and <literal>squash_projid</literal> define the
319 default UID, GID and PROJID respectively that users will be squashed
320 to if unmapped, unless the deny_unknown flag is set, in which case
321 access will still be denied.
324 <para>The <literal>squash_projid</literal> property was introduced
325 in Lustre 2.15</para>
330 <para>The property <literal>deny_unknown</literal> denies all access
331 to users not mapped in a particular nodemap. This is useful if a site
332 is concerned about unmapped users accessing the file system in order to
333 satisfy security requirements.
338 <para condition='l2B'>The property <literal>audit_mode</literal> lets
339 control which Lustre client nodes can trigger the recording of file
340 system access events to the Changelogs. When this flag is set to 1,
341 clients will be able to record file system access events to the
342 Changelogs, if Changelogs are otherwise activated. When set to 0,
343 events are not logged into the Changelogs, no matter if Changelogs are
344 activated or not. By default, this flag is set to 1 in newly created
345 nodemap entries. And it is also set to 1 in 'default' nodemap.</para>
349 <para condition='l2E'>The property
350 <literal>forbid_encryption</literal> prevents clients from using
355 <para condition='l2G'>The property
356 <literal>readonly_mount</literal> forces clients to a read-only
357 mount if not specified explicitly. By default it is set to 0 which
358 means clients are allowed to mount in read-write mode. Set it to 1
359 to force read-only mount.</para>
363 <para condition='l2G'>The property <literal>rbac</literal> defines
364 different Role-Based Admin Control mechanisms:
367 <para><literal>byfid_ops</literal>, to allow operations by FID
368 (e.g. 'lfs rmfid').</para>
371 <para><literal>chlg_ops</literal>, to allow access to Lustre
375 <para><literal>dne_ops</literal>, to allow operations related to
376 DNE (e.g. 'lfs mkdir').</para>
379 <para><literal>file_perms</literal>, to allow modifications of
380 file permissions and owners.</para>
383 <para><literal>fscrypt_admin</literal>, to allow fscrypt related
384 admin tasks (create or modify protectors/policies). Note that even
385 without this role, it is still possible to lock or unlock
386 encrypted directories, as these operations only need read access
387 to fscrypt metadata.</para>
390 <para><literal>quota_ops</literal>, to allow quota modifications.
394 The default value for this property is <literal>all</literal>,
395 which means all roles are allowed. Multiple values among those listed
396 above can be specified, comma separated. Apart from all, any role not
397 explicitly specified is forbidden. And to forbid all roles, use
398 <literal>none</literal> value.</para>
402 <para>Alter values to either true (1) or false (0) on the MGS:</para>
404 <screen>mgs# lctl nodemap_modify --name <replaceable>BirdAdminSite</replaceable> --property trusted --value 1
405 mgs# lctl nodemap_modify --name <replaceable>BirdAdminSite</replaceable> --property admin --value 1
406 mgs# lctl nodemap_modify --name <replaceable>BirdAdminSite</replaceable> --property deny_unknown --value 1</screen>
408 <para>Change values during system downtime to minimize the chance of any
409 ownership or permissions problems if the policy group is active.
410 Although changes can be made live, client caching of data may interfere
411 with modification as there are a few seconds of lead time before the
412 change is distributed.</para>
415 <section xml:id="lustrenodemap.alteringproperties.mixing" remap="h3">
416 <title>Mixing Properties</title>
418 <para>With both <literal>admin</literal> and <literal>trusted</literal>
419 properties set, the policy group has full access, as if nodemap was
420 turned off, to the Lustre file system. The administrative site for the
421 Lustre file system needs at least one group with both properties in
422 order to perform maintenance or to perform administrative tasks. </para>
425 <para>Lustre server nodes <emphasis role="bold">must</emphasis> be in a
426 policy group with both these properties set to 1. It is recommended to
427 use a policy group labeled “TrustedSystems” or some identifier that
428 makes the association clear.</para>
431 <para>If a policy group has the <literal>admin</literal>
432 property set, but does not have the property
433 <literal>trusted</literal> set, root is mapped directly to
434 root, any explicitly specified UID and GID idmaps are honored, and
435 other access is squashed. If root alters ownership to UIDs or GIDs
436 which are locally known from that host but not part of an idmap, root
437 effectively changes ownership of those files to the default
438 squashed UID and GID.</para>
440 <para>If <literal>trusted</literal> is set but <literal>admin</literal>
441 is not, the policy group has full access to the canonical UID and GID
442 sets of the Lustre file system, and root is squashed.</para>
444 <para>The deny_unknown property, once enabled, prevents unmapped users
445 from accessing the file system. Root access also is denied, if the
446 <literal>admin</literal> property is off, and root is not part of any
449 <para condition='l2F'>To prevent a client from changing quota settings
450 for a project other than the one assigned to the fileset it is restricted
451 to, you should map the PROJID to itself, set <literal>map_mode</literal>
452 to <literal>projid</literal>, and then <literal>trusted</literal> to
453 0 and <literal>deny_unknown</literal> to 1. This way, only operations on
454 the designated PROJID will be allowed.
457 <para>When nodemaps are modified, the change events are queued and
458 distributed across the cluster. Under normal conditions, these changes
459 can take around ten seconds to propagate. During this distribution
460 window, file access could be made via the old or new nodemap settings.
461 Therefore, it is recommended to save changes for a maintenance window
462 or to deploy them while the mapped nodes are not actively writing to the
468 <section xml:id="enablingthefeature">
469 <title>Enabling the Feature</title>
471 <para>The nodemap feature is simple to enable:</para>
473 <screen>mgs# lctl nodemap_activate 1</screen>
475 <para>Passing the parameter 0 instead of 1 disables the feature again.
476 After deploying the feature, validate the mappings are intact before
477 offering the file system to be mounted by clients.</para>
479 <para condition='l28'>So far, changes have been made on the MGS. Prior to
480 Lustre 2.9, changes must also be manually set on MDS systems as well.
481 Also, changes must be manually deployed to OSS servers if quota
482 is enforced, utilizing <literal>lctl set_param</literal>
483 instead of <literal>lctl</literal>. Prior to 2.9,
484 the configuration is not persistent, requiring a script
485 which generates the mapping to be saved and deployed after every Lustre
486 restart. As an example, use this style to deploy settings on the
489 <screen>oss# lctl set_param nodemap.add_nodemap=<replaceable>SiteName</replaceable>
490 oss# lctl set_param nodemap.add_nodemap_range='<replaceable>SiteName 192.168.0.15@tcp</replaceable>'
491 oss# lctl set_param nodemap.add_nodemap_idmap='<replaceable>SiteName</replaceable> uid <replaceable>510:1700</replaceable>'
492 oss# lctl set_param nodemap.add_nodemap_idmap='<replaceable>SiteName</replaceable> gid <replaceable>612:1702</replaceable>'</screen>
494 In Lustre 2.9 and later, nodemap
495 configuration is saved on the MGS and distributed automatically to
496 MGS, MDS, and OSS nodes, a process which takes approximately
497 ten seconds in normal circumstances.</para>
500 <section xml:id="defaultNodemap">
501 <title><literal>default</literal> Nodemap</title>
503 <para>There is a special nodemap called <literal>default</literal>. As the
504 name suggests, it is created by default and cannot be removed. It is like
505 a fallback nodemap, setting the behaviour for Lustre clients that do not
506 match any other nodemap.</para>
507 <para>Because of its special role, only some parameters can be set on the
508 <literal>default</literal> nodemap:</para>
511 <para><literal>admin</literal></para>
514 <para><literal>trusted</literal></para>
517 <para><literal>squash_uid</literal></para>
520 <para><literal>squash_gid</literal></para>
523 <para><literal>fileset</literal></para>
526 <para><literal>audit_mode</literal></para>
529 <para>In particular, no UID/GID mapping can be defined on the
530 <literal>default</literal> nodemap.</para>
532 <para>Be careful when altering the <literal>admin</literal> and
533 <literal>trusted</literal> properties of the <literal>default</literal>
534 nodemap, especially if your Lustre servers fall into this nodemap.
539 <section xml:id="verifyingsettings">
540 <title>Verifying Settings</title>
542 <para>By using <literal>lctl nodemap_info all</literal>, existing nodemap
543 configuration is listed for easy export. This command acts as a shortcut
544 into the configuration interface for nodemap. On the Lustre MGS, the
545 <literal>nodemap.active</literal> parameter contains a <literal>1</literal>
546 if nodemap is active on the system. Each policy group
547 creates a directory containing the following parameters:</para>
551 <para><literal>admin</literal> and <literal>trusted</literal> each
552 contain a <literal>1</literal> if the values are set, and
553 <literal>0</literal> otherwise.</para>
557 <para><literal>idmap</literal> contains a list of the idmaps for the
558 policy group, while <literal>ranges</literal> contains a list of
559 NIDs for the group.</para>
563 <para><literal>squash_uid</literal> and <literal>squash_gid</literal>
564 determine what UID and GID users are squashed to if needed.</para>
568 <para>The expected outputs for the BirdResearchSite in the example above
571 <screen>mgs# lctl get_param nodemap.BirdResearchSite.idmap
574 { idtype: uid, client_id: 530, fs_id: 11000 },
575 { idtype: uid, client_id: 531, fs_id: 11001 },
576 { idtype: uid, client_id: 532, fs_id: 11002 },
577 { idtype: uid, client_id: 533, fs_id: 11003 },
578 { idtype: gid, client_id: 600, fs_id: 11000 },
579 { idtype: gid, client_id: 601, fs_id: 11001 }
582 mgs# lctl get_param nodemap.BirdResearchSite.ranges
584 { id: 11, start_nid: 192.168.0.100@tcp, end_nid: 192.168.0.100@tcp }
588 <section xml:id="ensuringconsistency">
589 <title>Ensuring Consistency</title>
591 <para>Consistency issues may arise in a nodemap enabled configuration when
592 Lustre clients mount from an unknown NID range, new UIDs and GIDs that
593 were not part of a known map are added, or there are misconfigurations in
594 the rules. Keep in mind the following when activating nodemap
595 on a production system:</para>
599 <para>Creating new policy groups or idmaps on a production system
600 is allowed, but reserve a maintenance window to alter the <literal>
601 trusted</literal> property to avoid metadata problems.</para>
605 <para>To perform administrative tasks, access the Lustre file system
606 via a policy group with <literal>trusted</literal>
607 and <literal>admin</literal> properties set. This prevents
608 the creation of orphaned and squashed files. Granting the
609 <literal>admin</literal> property without the
610 <literal>trusted</literal> property
611 is dangerous. The root user on the client may know of UIDs
612 and GIDs that are not present in any idmap. If root alters ownership
613 to those identifiers, the ownership is squashed as a result. For
614 example, tar file extracts may be flipped from an expected UID
615 such as UID 500 to <literal>nobody</literal>, normally UID 99.</para>
619 <para>To map distinct UIDs at two or more sites onto a single UID or GID
620 on the Lustre file system, create overlapping idmaps and place each site
621 in its own policy group. Each distinct UID may have its own mapping onto
622 the target UID or GID.</para>
626 <para condition='l28'>In Lustre 2.8, changes must be manually kept in a
627 script file to be re-applied after a Lustre reload, and changes must be
628 made on each OSS, MDS, and MGS nodes, as there is no automatic
629 synchronization between the nodes.</para>
633 <para>If <literal>deny_unknown</literal> is in effect, it is possible
634 for unmapped users to see dentries which were viewed by a mapped user.
635 This is a result of client caching, and unmapped users will not be able
636 to view any file contents.</para>
640 <para>Nodemap activation status can be checked with
641 <literal>lctl nodemap_info</literal>,
642 but extra validation is possible. One way of ensuring valid
643 deployment on a production system is to create a fingerprint of known
644 files with specific UIDs and GIDs mapped to a test
645 client. After bringing the Lustre system online after maintenance, the
646 test client can validate the UIDs and GIDs map correctly before the
647 system is mounted in user space.</para>
653 vim:expandtab:shiftwidth=2:tabstop=8: