1 <?xml version='1.0' encoding='UTF-8'?>
2 <chapter xmlns="http://docbook.org/ns/docbook"
3 xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"
4 xml:id="dataonmdt" condition="l2B">
5 <title xml:id="dataonmdt.title">Data on MDT (DoM)</title>
6 <para>This chapter describes Data on MDT (DoM).</para>
7 <section xml:id="dbdoclet.domintro">
10 <primary>dom</primary>
13 <primary>dom</primary>
14 <secondary>intro</secondary>
15 </indexterm>Introduction to Data on MDT (DoM)</title>
16 <para>The Lustre Data on MDT (DoM) feature improves small file IO by
17 placing small files directly on the MDT, and also improves large file IO
18 by avoiding the OST being affected by small random IO that can cause
19 device seeking and hurt the streaming IO performance. Therefore, users
20 can expect more consistent performance for both small file IO and mixed IO
22 <para>The layout of a DoM file is stored on disk as a composite layout
23 and is a special case of Progressive File Layout (PFL). Please see
24 <xref linkend="pfl" /> for more information on PFL. For DoM files, the
25 file layout is composed of the component of the file, which is placed on
26 an MDT, and the rest of components are placed on OSTs, if needed. The
27 first component is placed on the MDT in the MDT object data blocks.
28 This component always has one stripe with size equal to the component
29 size. Such a component with an MDT layout can be only the first component
30 in composite layout. The rest of components are placed over OSTs as usual
31 with a RAID0 layout. The OST components are not instantiated until
32 a client writes or truncates the file beyond the size of the MDT
35 <section xml:id="dbdoclet.usercommands">
38 <primary>dom</primary>
41 <primary>dom</primary>
42 <secondary>usercommands</secondary>
43 </indexterm>User Commands</title>
44 <para>Lustre provides the <literal>lfs setstripe</literal> command for
45 users to create DoM files. Also, as usual,
46 <literal>lfs getstripe</literal> command can be used to list the
47 striping/component information for a given file, while
48 <literal>lfs find</literal> command can be used to search the directory
49 tree rooted at the given directory or file name for the files that match
50 the given DoM component parameters, e.g. layout type.</para>
51 <section xml:id="dbdoclet.lfssetstripe">
54 <primary>dom</primary>
55 <secondary>lfssetstripe</secondary>
56 </indexterm>lfs setstripe for DoM files</title>
57 <para>The <literal>lfs setstripe</literal> command is used to create
59 <section><title>Command</title>
61 lfs setstripe --component-end|-E end1 --layout|-L mdt \
62 [--component-end|-E end2 [STRIPE_OPTIONS] ...] <filename>
64 The command above creates a file with the special composite
65 layout, which defines the first component as an MDT component. The
66 MDT component must start from offset 0 and ends at
67 <replaceable>end1</replaceable>. The
68 <replaceable>end1</replaceable> is also the stripe size of this
69 component, and is limited by the
70 <literal>lod.*.dom_stripesize</literal> of the MDT the file is
71 created on. No other options are required for this component.
72 The rest of the components use the normal syntax for composite
75 <note><para>If the next component doesn't specify striping, such
77 <screen>lfs setstripe -E 1M -L mdt -E EOF <filename></screen>
78 Then that component get its settings from the default filesystem
79 striping.</para></note>
81 <section><title>Example</title>
82 <para>The command below creates a file with a DoM layout. The first
83 component has an <literal>mdt</literal> layout and is placed on the
84 MDT, covering [0, 1M). The second component covers [1M, EOF) and is
85 striped over all available OSTs.</para>
86 <para><screen>client$ lfs setstripe -E 1M -L mdt -E -1 -S 4M -c -1 \
87 /mnt/lustre/domfile</screen></para>
88 <para>The resulting layout is illustrated by
89 <xref xmlns:xlink="http://www.w3.org/1999/xlink"
90 linkend="dataonmdt.fig.layout1" />.</para>
91 <figure xml:id="dataonmdt.fig.layout1">
92 <title>Resulting file layout</title>
95 <imagedata scalefit="1" width="50%"
96 fileref="./figures/DoM_Layout1.png" />
99 <phrase>Resulting file layout</phrase>
103 <para>The resulting can also be checked with
104 <literal>lfs getstripe</literal> as shown below:</para>
105 <screen>client$ lfs getstripe /mnt/lustre/domfile
112 lcme_extent.e_start: 0
113 lcme_extent.e_end: 1048576
115 lmm_stripe_size: 1048576
123 lcme_extent.e_start: 1048576
124 lcme_extent.e_end: EOF
126 lmm_stripe_size: 4194304
128 lmm_layout_gen: 65535
129 lmm_stripe_offset: -1</screen>
130 <para>The output above shows that the first component has size 1MB and
131 pattern is 'mdt'. The second component is not instantiated yet, which
132 is seen by <literal>lcme_flags: 0</literal>.</para>
133 <para>If more than 1MB of data is written to the file, then
134 <literal>lfs getstripe</literal> output is changed accordingly:</para>
135 <screen>client$ lfs getstripe /mnt/lustre/domfile
142 lcme_extent.e_start: 0
143 lcme_extent.e_end: 1048576
145 lmm_stripe_size: 1048576
153 lcme_extent.e_start: 1048576
154 lcme_extent.e_end: EOF
156 lmm_stripe_size: 4194304
161 - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }
162 - 1: { l_ost_idx: 1, l_fid: [0x100010000:0x2:0x0] }</screen>
163 <para>The output above shows that the second component now has objects
164 on OSTs with a 4MB stripe.</para>
167 <section><title>Setting a default DoM layout to an existing directory
169 <para>A DoM layout can be set on an existing directory as well. When
170 set, all the files created after that will inherit this layout by
172 <section><title>Command</title>
173 <screen>lfs setstripe --component-end|-E end1 --layout|-L mdt \
174 [--component-end|-E end2 [STRIPE_OPTIONS] ...] <dirname></screen>
176 <section><title>Example</title>
177 <screen>client$ mkdir /mnt/lustre/domdir
178 client$ touch /mnt/lustre/domdir/normfile
179 client$ lfs setstripe -E 1M -L mdt -E -1 /mnt/lustre/domdir/
180 client$ lfs getstripe -d /mnt/lustre/domdir
186 lcme_extent.e_start: 0
187 lcme_extent.e_end: 1048576
188 stripe_count: 0 stripe_size: 1048576 \
189 pattern: mdt stripe_offset: -1
193 lcme_extent.e_start: 1048576
194 lcme_extent.e_end: EOF
195 stripe_count: 1 stripe_size: 1048576 \
196 pattern: raid0 stripe_offset: -1
198 <para>In the output above, it can be seen that the directory has
199 a default layout with a DoM component.</para>
200 <para>The following example will check layouts of files in that
202 <screen>client$ touch /mnt/lustre/domdir/domfile
203 client$ lfs getstripe /mnt/lustre/domdir/normfile
204 /mnt/lustre/domdir/normfile
206 lmm_stripe_size: 1048576
210 obdidx objid objid group
214 client$ lfs getstripe /mnt/lustre/domdir/domfile
215 /mnt/lustre/domdir/domfile
221 lcme_extent.e_start: 0
222 lcme_extent.e_end: 1048576
224 lmm_stripe_size: 1048576
232 lcme_extent.e_start: 1048576
233 lcme_extent.e_end: EOF
235 lmm_stripe_size: 1048576
237 lmm_layout_gen: 65535
238 lmm_stripe_offset: -1</screen>
239 <para>We can see that first file
240 <emphasis role="bold">normfile</emphasis> in that directory has an
241 ordinary layout, whereas the file <emphasis role="bold">domfile
242 </emphasis> inherits the directory default layout and is a DoM
244 <note><para>The directory default layout setting will be inherited
245 by new files even if the server DoM size limit will be set to a
246 lower value.</para></note>
249 <section xml:id="dbdoclet.domstripesize">
252 <primary>dom</primary>
255 <primary>dom</primary>
256 <secondary>domstripesize</secondary>
257 </indexterm>DoM Stripe Size Restrictions</title>
258 <para>The maximum size of a DoM component is restricted in several
259 ways to protect the MDT from being eventually filled with large files.
261 <section><title>LFS limits for DoM component size</title>
262 <para><literal>lfs setstripe</literal> allows for setting the
263 component size for MDT layouts up to 1GB (this is a compile-time
264 limit to avoid improper configuration), however, the size must
265 also be aligned by 64KB due to the minimum stripe size in Lustre
266 (see <xref linkend="settinguplustresystem.tab2"/>
267 <literal>Minimum stripe size</literal>). There is also a limit
268 imposed on each file by <literal>lfs setstripe -E end</literal>
269 that may be smaller than the MDT-imposed limit if this is better
270 for a particular usage.</para>
272 <section><title>MDT Server Limits</title>
273 <para>The <literal>lod.$fsname-MDTxxxx.dom_stripesize</literal>
274 is used to control the per-MDT maximum size for a DoM component.
275 Larger DoM components specified by the user will be truncated to
276 the MDT-specified limit, and as such may be different on each
277 MDT to balance DoM space usage on each MDT separately, if needed.
278 It is 1MB by default and can be changed with the
279 <literal>lctl</literal> tool. For more information on setting
280 <literal>dom_stripesize</literal> please see
281 <xref linkend="dbdoclet.dom_stripesize" />.</para>
284 <section xml:id="dbdoclet.domlfsgetstripe">
287 <primary>dom</primary>
290 <primary>dom</primary>
291 <secondary>lfsgetstripe</secondary>
292 </indexterm>lfs getstripe for DoM files</title>
293 <para>The <literal>lfs getstripe</literal> command is used to list
294 the striping/component information for a given file. For DoM files, it
295 can be used to check its layout and size.</para>
296 <section><title>Command</title>
297 <para><screen>lfs getstripe [--component-id|-I [comp_id]] [--layout|-L] \
298 [--stripe-size|-S] <dirname|filename></screen></para>
300 <section><title>Examples</title>
301 <screen>client$ lfs getstripe -I1 /mnt/lustre/domfile
308 lcme_extent.e_start: 0
309 lcme_extent.e_end: 1048576
311 lmm_stripe_size: 1048576
315 lmm_objects:</screen>
316 <para>Short info about the layout and size of DoM component can
317 be obtained with the use of the <literal>-L</literal> option
318 along with <literal>-S</literal> or <literal>-E</literal> options:
319 <screen>client$ lfs getstripe -I1 -L -S /mnt/lustre/domfile
320 lmm_stripe_size: 1048576
322 client$ lfs getstripe -I1 -L -E /mnt/lustre/domfile
323 lcme_extent.e_end: 1048576
324 lmm_pattern: mdt</screen>
325 Both commands return layout type and its size. The stripe size is
326 equal to the extent size of component in case of DoM files, so
327 both can be used to get size on the MDT.</para>
330 <section xml:id="dbdoclet.domlfsfind">
333 <primary>dom</primary>
336 <primary>dom</primary>
337 <secondary>lfsfind</secondary>
338 </indexterm>lfs find for DoM files</title>
339 <para>The <literal>lfs find</literal> command can be used to search
340 the directory tree rooted at the given directory or file name for the
341 files that match the given parameters. The command below shows the new
342 parameters for DoM files and their usages are similar to the
343 <literal>lfs getstripe</literal> command.</para>
344 <section><title>Command</title>
345 <para><screen>lfs find <directory|filename> [--layout|-L] [...]
348 <section><title>Examples</title>
349 <para>Find all files with DoM layout under directory
350 <literal>/mnt/lustre</literal>:
351 <screen>client$ lfs find -L mdt /mnt/lustre
354 /mnt/lustre/domdir/domfile
356 client$ lfs find -L mdt -type f /mnt/lustre
358 /mnt/lustre/domdir/domfile
360 client$ lfs find -L mdt -type d /mnt/lustre
361 /mnt/lustre/domdir</screen>
362 By using this command you can find all DoM objects, only DoM
363 files, or only directories with default DoM layout.</para>
364 <para>Find the DoM files/dirs with a particular stripe size:
365 <screen>client$ lfs find -L mdt -S -1200K -type f /mnt/lustre
367 /mnt/lustre/domdir/domfile
369 client$ lfs find -L mdt -S +200K -type f /mnt/lustre
371 /mnt/lustre/domdir/domfile</screen>
372 The first command finds all DoM files with stripe size less
373 than 1200KB. The second command above does the same for files
374 with a stripe size greater than 200KB. In both cases, all DoM
375 files are found because their DoM size is 1MB.</para>
378 <section xml:id="dbdoclet.dom_stripesize">
381 <primary>dom</primary>
384 <primary>dom</primary>
385 <secondary>dom_stripesize</secondary>
386 </indexterm>The dom_stripesize parameter</title>
387 <para>The MDT controls the default maximum DoM size on the server via
388 the parameter <literal>dom_stripesize</literal> in the LOD device.
389 The <literal>dom_stripesize</literal> can be set differently for each
390 MDT, if necessary. The default value of the parameter is 1MB and can
391 be changed with <literal>lctl</literal> tool.</para>
392 <section><title>Get Command</title>
393 <para><screen>lctl get_param lod.*MDT<index>*.dom_stripesize
396 <section><title>Get Examples</title>
397 <para>The commands below get the maximum allowed DoM size on the
398 server. The final command is an attempt to create a file with a
399 larger size than the parameter setting and correctly fails.
400 <screen>mds# lctl get_param lod.*MDT0000*.dom_stripesize
401 lod.lustre-MDT0000-mdtlov.dom_stripesize=1048576
403 mds# lctl get_param -n lod.*MDT0000*.dom_stripesize
406 client$ lfs setstripe -E 2M -L mdt /mnt/lustre/dom2mb
407 Create composite file /mnt/lustre/dom2mb failed. Invalid argument
408 error: setstripe: create composite file '/mnt/lustre/dom2mb' failed:
409 Invalid argument</screen></para>
411 <section><title>Temporary Set Command</title>
412 <para>To temporarily set the value of the parameter, the
413 <literal>lctl set_param</literal> is used:
414 <screen>lctl set_param lod.*MDT<index>*.dom_stripesize=<value>
417 <section><title>Temporary Set Examples</title>
418 <para>The example below shows a change to the default DoM limit on
419 the server to 64KB and try to create a file with 1MB DoM size
421 <screen>mds# lctl set_param -n lod.*MDT0000*.dom_stripesize=64K
422 mds# lctl get_param -n lod.*MDT0000*.dom_stripesize
425 client$ lfs setstripe -E 1M -L mdt /mnt/lustre/dom
426 Create composite file /mnt/lustre/dom failed. Invalid argument
427 error: setstripe: create composite file '/mnt/lustre/dom' failed:
428 Invalid argument</screen></para>
430 <section><title>Persistent Set Command</title>
431 <para>To persistently set the value of the parameter, the
432 <literal>lctl conf_param</literal> command is used:
433 <screen>lctl conf_param <fsname>-MDT<index>.lod.dom_stripesize=<value>
436 <section><title>Persistent Set Examples</title>
437 <para>The new value of the parameter is saved in config log
439 <screen>mgs# lctl conf_param lustre-MDT0000.lod.dom_stripesize=512K
440 mds# lctl get_param -n lod.*MDT0000*.dom_stripesize
442 New settings are applied in few seconds and saved persistently in
447 <section xml:id="dbdoclet.disabledom">
450 <primary>dom</primary>
453 <primary>dom</primary>
454 <secondary>disabledom</secondary>
455 </indexterm>Disable DoM</title>
456 <para>When <literal>lctl set_param</literal> or
457 <literal>lctl conf_param</literal> sets
458 <literal>dom_stripesize</literal> to <literal>0</literal>, DoM
459 component creation will be disabled on the selected server, and
460 any <emphasis>new</emphasis> layouts with a specified DoM component
461 will have that component removed from the file layout. Existing
462 files and layouts with DoM components on that MDT are not changed.
464 <note><para>DoM files can still be created in existing directories
465 with a default DoM layout.</para></note>
470 vim:expandtab:shiftwidth=2:tabstop=8: