1 <?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="dataonmdt"
3 <title xml:id="dataonmdt.title">Data on MDT (DoM)</title>
4 <para>This chapter describes Data on MDT (DoM).</para>
5 <section xml:id="dbdoclet.domintro">
11 <primary>dom</primary>
12 <secondary>intro</secondary>
13 </indexterm>Introduction to Data on MDT (DoM)</title>
14 <para>The Lustre Data on MDT (DoM) feature improves small file IO by
15 placing small files directly on the MDT, and also improves large file IO
16 by avoiding the OST being affected by small random IO that can cause
17 device seeking and hurt the streaming IO performance. Therefore, users
18 can expect more consistent performance for both small file IO and mixed IO
20 <para>The layout of a DoM file is stored on disk as a composite layout
21 and is a special case of Progressive File Layout (PFL). Please see
22 <xref linkend="pfl" /> for more information on PFL. For DoM files, the
23 file layout is composed of the component of the file, which is placed on
24 an MDT, and the rest of components are placed on OSTs, if needed. The
25 first component is placed on the MDT in the MDT object data blocks.
26 This component always has one stripe with size equal to the component
27 size. Such a component with an MDT layout can be only the first component
28 in composite layout. The rest of components are placed over OSTs as usual
29 with a RAID0 layout. The OST components are not instantiated until
30 a client writes or truncates the file beyond the size of the MDT
33 <section xml:id="dbdoclet.usercommands">
36 <primary>dom</primary>
39 <primary>dom</primary>
40 <secondary>usercommands</secondary>
41 </indexterm>User Commands</title>
42 <para>Lustre provides the <literal>lfs setstripe</literal> command for
43 users to create DoM files. Also, as usual,
44 <literal>lfs getstripe</literal> command can be used to list the
45 striping/component information for a given file, while
46 <literal>lfs find</literal> command can be used to search the directory
47 tree rooted at the given directory or file name for the files that match
48 the given DoM component parameters, e.g. layout type.</para>
49 <section xml:id="dbdoclet.lfssetstripe">
52 <primary>dom</primary>
53 <secondary>lfssetstripe</secondary>
54 </indexterm>lfs setstripe for DoM files</title>
55 <para>The <literal>lfs setstripe</literal> command is used to create
57 <section><title>Command</title>
58 <para><screen>lfs setstripe --component-end|-E end1 --layout|-L \
59 mdt [--component-end|-E end2 [STRIPE_OPTIONS] ...] <filename>
61 The command above creates a file with the special composite
62 layout, which defines the first component as an MDT component. The
63 MDT component must start from offset 0 and ends at
64 <replaceable>end1</replaceable>. The
65 <replaceable>end1</replaceable> is also the stripe size of this
66 component, and is limited by the
67 <literal>lod.*.dom_stripesize</literal> of the MDT the file is
68 created on. No other options are required. The rest of the
69 components use the normal syntax for composite files creation.
71 <note><para>If the next component doesn't specify striping, such
73 <screen>lfs setstripe -E 1M -L mdt -E EOF <filename></screen>
74 Then that component get its settings from the default filesystem
75 striping.</para></note>
77 <section><title>Example</title>
78 <para>The command below creates a file with a DoM layout. The first
79 component has an <literal>mdt</literal> layout and is placed on the
80 MDT, covering [0, 1M). The second component covers [1M, EOF) and is
81 striped over all available OSTs.</para>
82 <para><screen>client$ lfs setstripe -E 1M -L mdt -E -1 -S 4M -c -1 \
83 /mnt/lustre/domfile</screen></para>
84 <para>The resulting layout is illustrated by
85 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
86 linkend="dataonmdt.fig.layout1" />.</para>
87 <figure xml:id="dataonmdt.fig.layout1">
88 <title>Resulting file layout</title>
91 <imagedata scalefit="1" width="50%"
92 fileref="./figures/DoM_Layout1.png" />
95 <phrase>Resulting file layout</phrase>
99 <para>The resulting can also be checked with
100 <literal>lfs getstripe</literal> as shown below:</para>
101 <screen>client$ lfs getstripe /mnt/lustre/domfile
108 lcme_extent.e_start: 0
109 lcme_extent.e_end: 1048576
111 lmm_stripe_size: 1048576
119 lcme_extent.e_start: 1048576
120 lcme_extent.e_end: EOF
122 lmm_stripe_size: 4194304
124 lmm_layout_gen: 65535
125 lmm_stripe_offset: -1</screen>
126 <para>The output above shows that the first component has size 1MB and
127 pattern is 'mdt'. The second component is not instantiated yet, which
128 is seen by <literal>lcme_flags: 0</literal>.</para>
129 <para>If more than 1MB of data is written to the file, then
130 <literal>lfs getstripe</literal> output is changed accordingly:</para>
131 <screen>client$ lfs getstripe /mnt/lustre/domfile
138 lcme_extent.e_start: 0
139 lcme_extent.e_end: 1048576
141 lmm_stripe_size: 1048576
149 lcme_extent.e_start: 1048576
150 lcme_extent.e_end: EOF
152 lmm_stripe_size: 4194304
157 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }
158 - 1: { l_ost_idx: 1, l_fid: [0x100010000:0x2:0x0] }</screen>
159 <para>The output above shows that the second component now has objects
160 on OSTs with a 4MB stripe.</para>
163 <section><title>Setting a default DoM layout to an existing directory
165 <para>A DoM layout can be set on an existing directory as well. When
166 set, all the files created after that will inherit this layout by
168 <section><title>Command</title>
169 <screen>lfs setstripe --component-end|-E end1 --layout|-L mdt \
170 [--component-end|-E end2 [STRIPE_OPTIONS] ...] <dirname></screen>
172 <section><title>Example</title>
173 <screen>client$ mkdir /mnt/lustre/domdir
174 client$ touch /mnt/lustre/domdir/normfile
175 client$ lfs setstripe -E 1M -L mdt -E -1 /mnt/lustre/domdir/
176 client$ lfs getstripe -d /mnt/lustre/domdir
182 lcme_extent.e_start: 0
183 lcme_extent.e_end: 1048576
184 stripe_count: 0 stripe_size: 1048576 \
185 pattern: mdt stripe_offset: -1
189 lcme_extent.e_start: 1048576
190 lcme_extent.e_end: EOF
191 stripe_count: 1 stripe_size: 1048576 \
192 pattern: raid0 stripe_offset: -1
194 <para>In the output above, it can be seen that the directory has
195 a default layout with a DoM component.</para>
196 <para>The following example will check layouts of files in that
198 <screen>client$ touch /mnt/lustre/domdir/domfile
199 client$ lfs getstripe /mnt/lustre/domdir/normfile
200 /mnt/lustre/domdir/normfile
202 lmm_stripe_size: 1048576
206 obdidx objid objid group
210 client$ lfs getstripe /mnt/lustre/domdir/domfile
211 /mnt/lustre/domdir/domfile
217 lcme_extent.e_start: 0
218 lcme_extent.e_end: 1048576
220 lmm_stripe_size: 1048576
228 lcme_extent.e_start: 1048576
229 lcme_extent.e_end: EOF
231 lmm_stripe_size: 1048576
233 lmm_layout_gen: 65535
234 lmm_stripe_offset: -1</screen>
235 <para>We can see that first file
236 <emphasis role="bold">normfile</emphasis> in that directory has an
237 ordinary layout, whereas the file <emphasis role="bold">domfile
238 </emphasis> inherits the directory default layout and is a DoM
240 <note><para>The directory default layout setting will be inherited
241 by new files even if the server DoM size limit will be set to a
242 lower value.</para></note>
245 <section xml:id="dbdoclet.domstripesize">
248 <primary>dom</primary>
251 <primary>dom</primary>
252 <secondary>domstripesize</secondary>
253 </indexterm>DoM Stripe Size Restrictions</title>
254 <para>The maximum size of a DoM component is restricted in several
255 ways to protect the MDT from being eventually filled with large files.
257 <section><title>LFS limits for DoM component size</title>
258 <para><literal>lfs setstripe</literal> allows for setting the
259 component size for MDT layouts up to 1GB, however, the size must
260 be also aligned by 64KB. This is the maximum possible size on MDT.
261 Meanwhile, there is another limit which is checked by
262 <literal>lfs setstripe</literal> is provided by MDT server itself.
265 <section><title>MDT Server Limits</title>
266 <para>The LOD parameter <literal>dom_stripesize</literal> is used
267 to control the per-server maximum size for a DoM component. It is
268 1MB by default and can be changed with the
269 <literal>lctl</literal> tool. For more information on setting
270 <literal>dom_stripesize</literal> please see
271 <xref linkend="dbdoclet.dom_stripesize" />.</para>
274 <section xml:id="dbdoclet.domlfsgetstripe">
277 <primary>dom</primary>
280 <primary>dom</primary>
281 <secondary>lfsgetstripe</secondary>
282 </indexterm>lfs getstripe for DoM files</title>
283 <para>The <literal>lfs getstripe</literal> command is used to list
284 the striping/component information for a given file. For DoM files, it
285 can be used to check its layout and size.</para>
286 <section><title>Command</title>
287 <para><screen>lfs getstripe [--component-id|-I [comp_id]] [--layout|-L] \
288 [--stripe-size|-S] <dirname|filename></screen></para>
290 <section><title>Examples</title>
291 <screen>client$ lfs getstripe -I1 /mnt/lustre/domfile
298 lcme_extent.e_start: 0
299 lcme_extent.e_end: 1048576
301 lmm_stripe_size: 1048576
305 lmm_objects:</screen>
306 <para>Short info about the layout and size of DoM component can
307 be obtained with the use of the <literal>-L</literal> option
308 along with <literal>-S</literal> or <literal>-E</literal> options:
309 <screen>client$ lfs getstripe -I1 -L -S /mnt/lustre/domfile
310 lmm_stripe_size: 1048576
312 client$ lfs getstripe -I1 -L -E /mnt/lustre/domfile
313 lcme_extent.e_end: 1048576
314 lmm_pattern: mdt</screen>
315 Both commands return layout type and its size. The stripe size is
316 equal to the extent size of component in case of DoM files, so
317 both can be used to get size on the MDT.</para>
320 <section xml:id="dbdoclet.domlfsfind">
323 <primary>dom</primary>
326 <primary>dom</primary>
327 <secondary>lfsfind</secondary>
328 </indexterm>lfs find for DoM files</title>
329 <para>The <literal>lfs find</literal> command can be used to search
330 the directory tree rooted at the given directory or file name for the
331 files that match the given parameters. The command below shows the new
332 parameters for DoM files and their usages are similar to the
333 <literal>lfs getstripe</literal> command.</para>
334 <section><title>Command</title>
335 <para><screen>lfs find <directory|filename> [--layout|-L] [...]
338 <section><title>Examples</title>
339 <para>Find all files with DoM layout under directory
340 <literal>/mnt/lustre</literal>:
341 <screen>client$ lfs find -L mdt /mnt/lustre
344 /mnt/lustre/domdir/domfile
346 client$ lfs find -L mdt -type f /mnt/lustre
348 /mnt/lustre/domdir/domfile
350 client$ lfs find -L mdt -type d /mnt/lustre
351 /mnt/lustre/domdir</screen>
352 By using this command you can find all DoM objects, only DoM
353 files, or only directories with default DoM layout.</para>
354 <para>Find the DoM files/dirs with a particular stripe size:
355 <screen>client$ lfs find -L mdt -S -1200K -type f /mnt/lustre
357 /mnt/lustre/domdir/domfile
359 client$ lfs find -L mdt -S +200K -type f /mnt/lustre
361 /mnt/lustre/domdir/domfile</screen>
362 The first command finds all DoM files with stripe size less
363 than 1200KB. The second command above does the same for files
364 with a stripe size greater than 200KB. In both cases, all DoM
365 files are found because their DoM size is 1MB.</para>
368 <section xml:id="dbdoclet.dom_stripesize">
371 <primary>dom</primary>
374 <primary>dom</primary>
375 <secondary>dom_stripesize</secondary>
376 </indexterm>The dom_stripesize parameter</title>
377 <para>The MDT controls the default maximum DoM size on the server via
378 the parameter <literal>dom_stripesize</literal> in the LOD device.
379 The <literal>dom_stripesize</literal> can be set differently for each
380 MDT, if necessary. The default value of the parameter is 1MB and can
381 be changed with <literal>lclt</literal> tool.</para>
382 <section><title>Get Command</title>
383 <para><screen>lctl get_param lod.*MDT<index>*.dom_stripesize
386 <section><title>Get Examples</title>
387 <para>The commands below get the maximum allowed DoM size on the
388 server. The final command is an attempt to create a file with a
389 larger size than the parameter setting and correctly fails.
390 <screen>mds# lctl get_param lod.*MDT0000*.dom_stripesize
391 lod.lustre-MDT0000-mdtlov.dom_stripesize=1048576
393 mds# lctl get_param -n lod.*MDT0000*.dom_stripesize
396 client$ lfs setstripe -E 2M -L mdt /mnt/lustre/dom2mb
397 Create composite file /mnt/lustre/dom2mb failed. Invalid argument
398 error: setstripe: create composite file '/mnt/lustre/dom2mb' failed:
399 Invalid argument</screen></para>
401 <section><title>Temporary Set Command</title>
402 <para>To temporarily set the value of the parameter, the
403 <literal>lctl set_param</literal> is used:
404 <screen>lctl set_param lod.*MDT<index>*.dom_stripesize=<value>
407 <section><title>Temporary Set Examples</title>
408 <para>The example below shows a change to the default DoM limit on
409 the server to 32KB and try to create a file with 1MB DoM size
411 <screen>mds# lctl set_param -n lod.*MDT0000*.dom_stripesize=32K
412 mds# lctl get_param -n lod.*MDT0000*.dom_stripesize
415 client$ lfs setstripe -E 1M -L mdt /mnt/lustre/dom
416 Create composite file /mnt/lustre/dom failed. Invalid argument
417 error: setstripe: create composite file '/mnt/lustre/dom' failed:
418 Invalid argument</screen></para>
420 <section><title>Persistent Set Command</title>
421 <para>To persistently set the value of the parameter, the
422 <literal>lctl conf_param</literal> command is used:
423 <screen>lctl conf_param <fsname>-MDT<index>.lod.dom_stripesize=<value>
426 <section><title>Persistent Set Examples</title>
427 <para>The new value of the parameter is saved in config log
429 <screen>mgs# lctl conf_param lustre-MDT0000.lod.dom_stripesize=512K
430 mds# lctl get_param -n lod.*MDT0000*.dom_stripesize
432 New settings are applied in few seconds and saved persistently in
437 <section xml:id="dbdoclet.disabledom">
440 <primary>dom</primary>
443 <primary>dom</primary>
444 <secondary>disabledom</secondary>
445 </indexterm>Disable DoM</title>
446 <para>When <literal>lclt set_param</literal> or
447 <literal>lctl conf_param</literal> sets
448 <literal>dom_stripesize</literal> to <literal>0</literal>, DoM
449 file creation will be prohibited on the selected server.</para>
450 <note><para>DoM files can still be created in existing directories
451 with the default DoM layout</para></note>