1 <?xml version='1.0' encoding='UTF-8'?>
2 <chapter xmlns="http://docbook.org/ns/docbook"
3 xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"
4 xml:id="pcc" condition="l2D">
5 <title xml:id="pcc.title">Persistent Client Cache (PCC)</title>
6 <para>This chapter describes Persistent Client Cache (PCC).</para>
7 <section xml:id="pcc.intro">
8 <title>Introduction</title>
9 <para> Flash-based SSDs help to (partly) close the ever-increasing
10 performance gap between magnetic disks and CPUs. SSDs build a new level
11 in the storage hierarchy, both in terms of price and performance. The
12 large size of data sets stored in Lustre, ranging up to hundreds of PiB
13 in the largest centers, makes it more cost-effective to store most of the
14 data on HDDs and only an active subset of data on SSDs.</para>
15 <para>The PCC mechanism allows clients equipped with internal SSDs to
16 deliver additional performance for both read and write intensive
17 applications that have node-local I/O patterns without losing the benefits
18 of the global Lustre namespace. PCC is combined with Lustre HSM and layout
19 lock mechanisms to provide persistent caching services using the local SSD
20 storage, while allowing migration of individual files between local and
21 shared storage. This enables I/O intensive applications to read and write
22 data on client nodes without losing the benefits of the global Lustre
24 <para>The main advantages to use this cache on the Lustre clients is
25 that the I/O stack is much simpler for the cached data, as there is no
26 interference with I/Os from other clients, which enables performance
27 optimizations. There are no special requirements on the hardware of
28 the client nodes. Any Linux filesystem, such as ext4 on an NVMe device,
29 can be used as PCC cache. Local file caching reduces the pressure on the
30 object storage targets (OSTs), as small or random I/Os can be aggregated
31 to large sequential I/Os and temporary files do not even need to be
32 flushed to OSTs.</para>
34 <section xml:id="pcc.design">
36 <section xml:id="pcc.design.rwpcc">
37 <title>Lustre Read-Write PCC Caching</title>
38 <figure xml:id="pcc.rwpccarch.fig">
39 <title>Overview of PCC-RW Architecture</title>
42 <imagedata scalefit="1" width="50%"
43 fileref="figures/pccarch.png" />
46 <phrase>Overview of PCC-RW Architecture</phrase>
50 <para>Lustre typically uses its integrated HSM mechanism to interface
51 with larger and slower archival storage using tapes or other media.
52 PCC-RW, on the contrary, is actually an HSM backend storage system which
53 provides a group of high-speed local caches on Lustre clients.
54 <xref linkend="pcc.rwpccarch.fig"/> shows the PCC-RW architecture. Each
55 client uses its own local storage, usually in the form of NVMe,
56 formatted as a local file system for the local cache. Cached I/Os are
57 directed to files in the local file system, while normal I/O are
58 directed to OSTs.</para>
59 <para>PCC-RW uses Lustre's HSM mechanism for data synchronization. Each
60 PCC node is actually an HSM agent and has a copy tool instance running
61 on it. The Lustre HSM copytool is used to restore files from the local
62 cache to Lustre OSTs. Any remote access for a PCC cached file from
63 another Lustre client triggers this data synchronization. If a PCC
64 client goes offline, the cached data becomes temporarily inaccessible
65 to other clients. The data will be accessible again after the PCC
66 client reboots, mounts the Lustre filesystem, and restarts the
68 <para>Currently, PCC clients cache entire files on their local
69 filesystems. A file has to be attached to PCC before I/O can be directed
70 to a client cache. The Lustre layout lock feature is used to ensure that
71 the caching services are consistent with the global file system state.
72 The file data can be written/read directly to/from the local PCC cache
73 after a successful attach operation. If the attach has not been
74 successful, the client will simply fall back to the normal I/O path and
75 direct I/Os to OSTs. PCC-RW cached files are automatically restored to
76 the global filesystem when a process on another client tries to read or
77 modify them. The corresponding I/O will be blocked, waiting for the
78 released file to be restored. This is transparent to the application.
80 <para>The revocation of the layout lock can automatically detach the file
81 from the PCC cache at any time. The PCC-RW cached file can be
82 manually detached by the <literal>lfs pcc detach</literal> command. After
83 the cached file is detached from the cache and restored to OSTs, it
84 will be removed from the PCC filesystem.</para>
85 <para>Failed PCC-RW operations usually return corresponding error codes.
86 There is a special case when the space of the local PCC file system is
87 exhausted. In this case, PCC-RW can fall back to the normal I/O path
88 automatically since the capacity of the Lustre file system is much
89 larger than the capacity of the PCC device.</para>
91 <section xml:id="pcc.design.rules">
92 <title>Rule-based Persistent Client Cache</title>
93 <para>PCC includes a rule-based, configurable caching infrastructure that
94 enables it to achieve various objectives, such as customizing I/O
95 caching and providing performance isolation and QoS guarantees.</para>
96 <para>For PCC-RW, when a file is being created, a rule-based policy is
97 used to determine whether it will be cached. It supports rules for
98 different users, groups, projects, or filenames extensions.</para>
99 <para>Rule-based PCC-RW caching of newly created files can determine
100 which file can use a cache on PCC directly without administrator's
104 <section xml:id="pcc.operations">
105 <title>PCC Command Line Tools</title>
106 <para>Lustre provides <literal>lfs</literal> and <literal>lctl</literal>
107 command line tools for users to interact with PCC feature.</para>
108 <section xml:id="pcc.operations.add">
109 <title>Add a PCC backend on a client</title>
110 <para><emphasis role="strong">Command:</emphasis></para>
111 <screen>client# lctl pcc add <replaceable>mountpoint</replaceable> <replaceable>pccpath</replaceable> [--param|-p <replaceable>cfgparam</replaceable>]</screen>
112 <para>The above command will add a PCC backend to the Lustre client.</para>
115 <colspec align="left" colwidth="1*"/>
116 <colspec align="left" colwidth="2*"/>
119 <entry>Option</entry>
120 <entry>Description</entry>
125 <entry>mountpoint</entry>
127 <para>The Lustre client mount point.
132 <entry>pccpath</entry>
134 <para>The directory path on local filesystem for PCC cache. The
135 whole filesystem does not need to be exclusively dedicated to
136 the PCC cache, but the directory should not be accessible to
137 regular users.</para>
142 <para>cfgparam</para>
145 <para>A string in the form of name-value pairs to config the
146 PCC backend such as read-write attach id (archive ID),
147 and auto caching rule, etc.</para>
153 <para><emphasis role="strong">Note:</emphasis> when a client node has
154 more than one Lustre mount point or Lustre filesystem instance, the
155 parameter <replaceable>mountpoint</replaceable> makes sure that only
156 the PCC backend on specified Lustre filesystem instance or Lustre
157 mount point is configured. This Lustre mount point must be the same as
158 the HSM (lhsmtool_posix) configuration, if the PCC backend is used as
159 PCC-RW caching. Also, the parameter <replaceable>pccpath</replaceable>
160 should be the same as the HSM root parameter of the POSIX copytool
161 (lhsmtool_posix).</para>
162 <para>PCC-RW uses Lustre's HSM mechanism for data synchronization.
163 Before using PCC-RW on a client, it is still necessary to setup HSM on
164 the MDTs and the PCC client nodes.</para>
165 <para>First, a coordinator must be activated on each of the filesystem
166 MDTs. This can be achieved with the command:</para>
167 <screen>mds# lctl set_param mdt.<replaceable>$FSNAME-MDT0000</replaceable>.hsm_control=enabled
168 mdt.lustre-MDT0000.hsm_control=enabled</screen>
169 <para>Next, launch the copytool on each agent node (PCC client node)
170 to connect to your HSM storage. This command will be of the form:</para>
171 <screen>client# lhsmtool_posix --daemon --hsm-root <replaceable>$PCCPATH</replaceable> --archive=<replaceable>$ARCHIVE_ID</replaceable> <replaceable>$LUSTREPATH</replaceable></screen>
172 <para><emphasis role="strong">Examples:</emphasis></para>
173 <para>The following command adds a PCC backend on a client:</para>
174 <screen>client# lctl pcc add /mnt/lustre /mnt/pcc --param "projid={500,1000}&fname={*.h5},uid=1001 rwid=2"</screen>
175 <para>The first substring of the config parameter is the auto-cache rule,
176 where "&" represents the logical AND operator while "," represents
177 the logical OR operator. The example rule means that new files are only
178 auto cached if either of the following conditions are satisfied:</para>
181 <para>The project ID is either 500 or 1000 and the suffix of the file
185 <para>The user ID is 1001;</para>
188 <para>The currently supported name-value pairs for PCC backend
189 configuration are listing as follows:</para>
192 <para><literal>rwid</literal> PCC-RW attach ID which
193 is same as the archive ID of the copytool agent running on this PCC
197 <para><literal>auto_attach</literal> <literal>"auto_attach=1"
198 </literal> enables auto attach at the next open or during I/O.
199 Enabling this option should cause automatic attaching of valid
200 PCC-cached files which were detached due to the manual <literal>
201 lfs pcc detach</literal> command or revocation of layout lock (i.e.
202 LRU lock shrinking). <literal>"auto_attach=0"</literal> means that
203 auto file attach is disabled and is the default mode.
208 <section xml:id="pcc.operations.del">
209 <title>Delete a PCC backend from a client</title>
210 <para><emphasis role="strong">Command:</emphasis></para>
211 <screen>lctl pcc del <mountpoint> <pccpath></screen>
212 <para>The above command will delete a PCC backend from a Lustre client.
216 <colspec align="left" colwidth="1*"/>
217 <colspec align="left" colwidth="2*"/>
220 <entry>Option</entry>
221 <entry>Description</entry>
226 <entry>mountpoint</entry>
228 <para>The Lustre client mount point.
233 <entry>pccpath</entry>
235 <para>A PCC backend is specified by this path. Please refer to
236 <literal>lctl pcc add</literal> for details.</para>
242 <para><emphasis role="strong">Examples:</emphasis></para>
243 <para>The following command will delete a PCC backend referenced by
244 <replaceable>"/mnt/pcc"</replaceable> on a client with the mount point
245 of <replaceable>"/mnt/lustre"</replaceable>.</para>
246 <screen>client# lctl pcc del /mnt/lustre /mnt/pcc</screen>
248 <section xml:id="pcc.operations.clear">
249 <title>Remove all PCC backends on a client</title>
250 <para><emphasis role="strong">Command:</emphasis></para>
251 <screen>lctl pcc clear <mountpoint></screen>
252 <para>The above command will remove all PCC backends on a Lustre client.
256 <colspec align="left" colwidth="1*"/>
257 <colspec align="left" colwidth="2*"/>
260 <entry>Option</entry>
261 <entry>Description</entry>
266 <entry>mountpoint</entry>
268 <para>The Lustre client mount point.
275 <para><emphasis role="strong">Examples:</emphasis></para>
276 <para>The following command will remove all PCC backends from a client
277 with the mount point of <replaceable>"/mnt/lustre"</replaceable>.
279 <screen>client# lctl pcc clear /mnt/lustre</screen>
281 <section xml:id="pcc.operations.list">
282 <title>List all PCC backends on a client</title>
283 <para><emphasis role="strong">Command:</emphasis></para>
284 <screen>lctl pcc list <mountpoint></screen>
285 <para>The above command will list all PCC backends on a Lustre client.
289 <colspec align="left" colwidth="1*"/>
290 <colspec align="left" colwidth="2*"/>
293 <entry>Option</entry>
294 <entry>Description</entry>
299 <entry>mountpoint</entry>
301 <para>The Lustre client mount point.
308 <para><emphasis role="strong">Examples:</emphasis></para>
309 <para>The following command will list all PCC backends on a client with
310 the mount point of <replaceable>"/mnt/lustre"</replaceable>.</para>
311 <screen>client# lctl pcc list /mnt/lustre</screen>
313 <section xml:id="pcc.operations.attach">
314 <title>Attach given files into PCC</title>
315 <para><emphasis role="strong">Command:</emphasis></para>
316 <screen>lfs pcc attach --id|-i <NUM> <file...></screen>
317 <para>The above command will attach the given files onto PCC.</para>
320 <colspec align="left" colwidth="1*"/>
321 <colspec align="left" colwidth="2*"/>
324 <entry>Option</entry>
325 <entry>Description</entry>
330 <entry>--id|-i <NUM></entry>
332 <para>Attach ID to select which PCC backend to use.
339 <para><emphasis role="strong">Examples:</emphasis></para>
340 <para>The following command will attach the file referenced by
341 <replaceable>/mnt/lustre/test</replaceable> onto the PCC backend with
342 PCC-RW attach ID that equals 2.</para>
343 <screen>client# lfs pcc attach -i 2 /mnt/lustre/test</screen>
345 <section xml:id="pcc.operations.attach_fid">
346 <title>Attach given files into PCC by FID(s)</title>
347 <para><emphasis role="strong">Command:</emphasis></para>
348 <screen>lfs pcc attach_fid --id|-i <NUM> --mnt|-m <mountpoint> <fid...></screen>
349 <para>The above command will attach the given files referenced by their
350 FIDs into PCC.</para>
353 <colspec align="left" colwidth="1*"/>
354 <colspec align="left" colwidth="2*"/>
357 <entry>Option</entry>
358 <entry>Description</entry>
363 <entry>--id|-i <NUM></entry>
365 <para>Attach ID to select which PCC backend to use.
370 <entry>--mnt|-m <mountpoint></entry>
372 <para>The Lustre mount point.</para>
378 <para><emphasis role="strong">Examples:</emphasis></para>
379 <para>The following command will attach the file referenced by FID
380 <replaceable>0x200000401:0x1:0x0</replaceable> onto the PCC backend
381 with PCC-RW attach ID that equals 2.</para>
382 <screen>client# lfs pcc attach_fid -i 2 -m /mnt/lustre 0x200000401:0x1:0x0</screen>
384 <section xml:id="pcc.operations.detach">
385 <title>Detach given files from PCC</title>
386 <para><emphasis role="strong">Command:</emphasis></para>
387 <screen>lfs pcc detach [--keep|-k] <file...></screen>
388 <para>The above command will detach given files from PCC.</para>
391 <colspec align="left" colwidth="1*"/>
392 <colspec align="left" colwidth="2*"/>
395 <entry>Option</entry>
396 <entry>Description</entry>
401 <entry>--keep|-k</entry>
403 <para>By default, the <literal>detach</literal> command will
404 detach the file from PCC permanently and remove the PCC copy
405 after detach. This option will only detach the file, but keep
406 the PCC copy in cache. It allows the detached file to be
407 attached automatically at the next open if the cached copy of
408 the file is still valid.</para>
414 <para><emphasis role="strong">Examples:</emphasis></para>
415 <para>The following command will detach the file referenced by
416 <replaceable>/mnt/lustre/test</replaceable> from PCC permanently and
417 remove the corresponding cached file on PCC.
419 <screen>client# lfs pcc detach /mnt/lustre/test</screen>
420 <para>The following command will detach the file referenced by
421 <replaceable>/mnt/lustre/test</replaceable> from PCC, but allow the file
422 to be attached automatically at the next open.</para>
423 <screen>client# lfs pcc detach -k /mnt/lustre/test</screen>
425 <section xml:id="pcc.operations.detach_fid">
426 <title>Detach given files from PCC by FID(s)</title>
427 <para><emphasis role="strong">Command:</emphasis></para>
428 <screen>lfs pcc detach_fid [--keep|-k] <mountpoint> <fid...></screen>
429 <para>The above command will detach the given files from PCC by FID(s).
433 <colspec align="left" colwidth="1*"/>
434 <colspec align="left" colwidth="2*"/>
437 <entry>Option</entry>
438 <entry>Description</entry>
443 <entry>--keep|-k</entry>
445 <para>Please refer to the command <literal>lfs pcc detach
446 </literal> for details</para>
452 <para><emphasis role="strong">Examples:</emphasis></para>
453 <para>The following command will detach the file referenced by FID
454 <replaceable>0x200000401:0x1:0x0</replaceable> from PCC permanently and
455 remove the corresponding cached file on PCC.</para>
456 <screen>client# lfs pcc detach_fid /mnt/lustre 0x200000401:0x1:0x0</screen>
457 <para>The following command will detach the file referenced by FID
458 <replaceable>0x200000401:0x1:0x0</replaceable> from PCC, but allow the
459 file to be attached automatically at the next open.</para>
460 <screen>client# lfs pcc detach_fid -k /mnt/lustre 0x200000401:0x1:0x0</screen>
462 <section xml:id="pcc.operations.state">
463 <title>Display the PCC state for given files</title>
464 <para><emphasis role="strong">Command:</emphasis></para>
465 <screen>lfs pcc state <file...></screen>
466 <para>The above command will display the PCC state for given files.</para>
467 <para><emphasis role="strong">Examples:</emphasis></para>
468 <para>The following command will display the PCC state of the file
469 referenced by <replaceable>/mnt/lustre/test</replaceable>.</para>
470 <screen>client# lfs pcc state /mnt/lustre/test
471 file: /mnt/lustre/test, type: readwrite, PCC file: /mnt/pcc/0004/0000/0bd1/0000/0002/0000/0x200000bd1:0x4:0x0, user number: 1, flags: 4</screen>
472 <para>If the file "/mnt/lustre/test" is not cached on PCC, the output of
473 its PCC state is as follow:</para>
474 <screen>client# lfs pcc state /mnt/lustre/test
475 file: /mnt/lustre/test, type: none</screen>
478 <section xml:id="pcc.examples">
479 <title>PCC Configuration Example</title>
482 <para>Setup HSM on MDT</para>
483 <screen>mds# lctl set_param mdt.lustre-MDT0000.hsm_control=enabled</screen>
486 <para>Setup PCC on the clients</para>
487 <screen>client1# lhsmtool_posix --daemon --hsm-root /mnt/pcc --archive=1 /mnt/lustre < /dev/null > /tmp/copytool_log 2>&1
488 client1# lctl pcc add /mnt/lustre /mnt/pcc "projid={1000},uid={500} rwid=1"</screen>
489 <screen>client2# lhsmtool_posix --daemon --hsm-root /mnt/pcc --archive=2 /mnt/lustre < /dev/null > /tmp/copytool_log 2>&1
490 client2# lctl pcc add /mnt/lustre /mnt/pcc "projid={1000}&gid={500} rwid=2"</screen>
493 <para>Execute PCC commands on the clients</para>
494 <screen>client1# echo "QQQQQ" > /mnt/lustre/test
496 client2# lfs pcc attach -i 2 /mnt/lustre/test
498 client2# lfs pcc state /mnt/lustre/test
499 file: /mnt/lustre/test, type: readwrite, PCC file: /mnt/pcc/0004/0000/0bd1/0000/0002/0000/0x200000bd1:0x4:0x0, user number: 1, flags: 6
501 client2# lfs pcc detach /mnt/lustre/test</screen>
507 vim:expandtab:shiftwidth=2:tabstop=8: