<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink">
+<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='configuringquotas'>
<info>
- <title>Configuring and Managing Quotas</title>
+ <title xml:id='configuringquotas.title'>Configuring and Managing Quotas</title>
</info>
+
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290092" xreflabel=""/>This chapter describes how to configure quotas and includes the following sections:</para>
+
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290096" xreflabel=""/><link xl:href="ConfiguringQuotas.html#50438217_54945">Working with Quotas</link></para>
+ <para><xref linkend="dbdoclet.50438217_54945"/></para>
</listitem>
<listitem>
- <para> </para>
+ <para><xref linkend="dbdoclet.50438217_31982"/></para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290100" xreflabel=""/><link xl:href="ConfiguringQuotas.html#50438217_31982">Enabling Disk Quotas</link></para>
+ <para><xref linkend="dbdoclet.50438217_49939"/></para>
</listitem>
<listitem>
- <para> </para>
+ <para><xref linkend="dbdoclet.50438217_15106"/></para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290104" xreflabel=""/><link xl:href="ConfiguringQuotas.html#50438217_49939">Creating Quota Files and Quota Administration</link></para>
+ <para><xref linkend="dbdoclet.50438217_27895"/></para>
</listitem>
<listitem>
- <para> </para>
- </listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290108" xreflabel=""/><link xl:href="ConfiguringQuotas.html#50438217_15106">Quota Allocation</link></para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290112" xreflabel=""/><link xl:href="ConfiguringQuotas.html#50438217_27895">Known Issues with Quotas</link></para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290116" xreflabel=""/><link xl:href="ConfiguringQuotas.html#50438217_20772">Lustre Quota Statistics</link></para>
- </listitem>
-<listitem>
- <para> </para>
+ <para><xref linkend="dbdoclet.50438217_20772"/></para>
</listitem>
</itemizedlist>
- <section remap="h2">
- <title><anchor xml:id="dbdoclet.50438217_pgfId-1290119" xreflabel=""/></title>
- <section remap="h2">
- <title>21.1 <anchor xml:id="dbdoclet.50438217_54945" xreflabel=""/>Working with <anchor xml:id="dbdoclet.50438217_marker-1290118" xreflabel=""/>Quotas</title>
+
+ <section xml:id="dbdoclet.50438217_54945">
+ <title>21.1 Working with <anchor xml:id="dbdoclet.50438217_marker-1290118" xreflabel=""/>Quotas</title>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290120" xreflabel=""/>Quotas allow a system administrator to limit the amount of disk space a user or group can use in a directory. Quotas are set by root, and can be specified for individual users and/or groups. Before a file is written to a partition where quotas are set, the quota of the creator's group is checked. If a quota exists, then the file size counts towards the group's quota. If no quota exists, then the owner's user quota is checked before the file is written. Similarly, inode usage for specific functions can be controlled if a user over-uses the allocated space.</para>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290121" xreflabel=""/>Lustre quota enforcement differs from standard Linux quota enforcement in several ways:</para>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290122" xreflabel=""/> Quotas are administered via the lfs command (post-mount).</para>
</listitem>
<listitem>
- <para> </para>
- </listitem>
-<listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290123" xreflabel=""/> Quotas are distributed (as Lustre is a distributed file system), which has several ramifications.</para>
</listitem>
<listitem>
- <para> </para>
- </listitem>
-<listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290124" xreflabel=""/> Quotas are allocated and consumed in a quantized fashion.</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
<listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290125" xreflabel=""/> Client does not set the usrquota or grpquota options to mount. When quota is enabled, it is enabled for all clients of the file system; started automatically using quota_type or started manually with lfs quotaon.</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
</itemizedlist>
- <informaltable frame="none">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="5*"/>
- <colspec colname="c2" colwidth="95*"/>
-
-
- <tbody>
- <row>
- <entry><para><inlinemediaobject><imageobject role="html">
- <imagedata fileref="./shared/caution.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/caution.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></para></entry>
- <entry><para><emphasis role="bold">Caution -</emphasis><anchor xml:id="dbdoclet.50438217_pgfId-1290126" xreflabel=""/>Although quotas are available in Lustre, root quotas are NOT enforced.</para><para>lfs setquota -u root (limits are not enforced)</para><para>lfs quota -u root (usage includes internal Lustre data that is dynamic in size and does not accurately reflect mount point visible block and inode usage).</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+ <caution><para>Although quotas are available in Lustre, root quotas are NOT enforced.</para><para>lfs setquota -u root (limits are not enforced)</para><para>lfs quota -u root (usage includes internal Lustre data that is dynamic in size and does not accurately reflect mount point visible block and inode usage).</para></caution>
+
</section>
- <section remap="h2">
- <title>21.2
<anchor xml:id="dbdoclet.50438217_31982" xreflabel=""/>Enabling <anchor xml:id="dbdoclet.50438217_marker-1290128" xreflabel=""/>Disk Quotas</title>
+ <section xml:id="dbdoclet.50438217_31982">
+ <title>21.2 Enabling <anchor xml:id="dbdoclet.50438217_marker-1290128" xreflabel=""/>Disk Quotas</title>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290130" xreflabel=""/>Use this procedure to enable (configure) disk quotas in Lustre.</para>
+ <orderedlist><listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290131" xreflabel=""/> 1. If you have re-complied your Linux kernel, be sure that CONFIG_QUOTA and CONFIG_QUOTACTL are enabled. Also, verify that CONFIG_QFMT_V1 and/or CONFIG_QFMT_V2 are enabled.</para>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290132" xreflabel=""/>Quota is enabled in all Linux 2.6 kernels supplied for Lustre.</para>
+
+ </listitem><listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290133" xreflabel=""/> 2. Start the server.</para>
+ </listitem><listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290134" xreflabel=""/> 3. Mount the Lustre file system on the client and verify that the lquota module has loaded properly by using the lsmod command.</para>
<screen><anchor xml:id="dbdoclet.50438217_pgfId-1290135" xreflabel=""/>$ lsmod
<anchor xml:id="dbdoclet.50438217_pgfId-1290136" xreflabel=""/>[root@oss161 ~]# lsmod
<anchor xml:id="dbdoclet.50438217_pgfId-1290146" xreflabel=""/>mdc 95016 1 lustre
<anchor xml:id="dbdoclet.50438217_pgfId-1290147" xreflabel=""/>ksocklnd 111812 1
</screen>
+ </listitem></orderedlist>
+
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290148" xreflabel=""/>The Lustre mount command no longer recognizes the usrquota and grpquota options. If they were previously specified, remove them from /etc/fstab.</para>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290149" xreflabel=""/>When quota is enabled, it is enabled for all file system clients (started automatically using quota_type or manually with lfs quotaon).</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438217_pgfId-1290150" xreflabel=""/>Lustre with the Linux kernel 2.4 does <emphasis>not</emphasis> support quotas.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+
+ <note><para>Lustre with the Linux kernel 2.4 does <emphasis>not</emphasis> support quotas.</para></note>
+
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290151" xreflabel=""/>To enable quotas automatically when the file system is started, you must set the mdt.quota_type and ost.quota_type parameters, respectively, on the MDT and OSTs. The parameters can be set to the string u (user), g (group) or ug for both users and groups.</para>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290152" xreflabel=""/>You can enable quotas at mkfs time (mkfs.lustre --param mdt.quota_type=ug) or with tunefs.lustre. As an example:</para>
<screen><anchor xml:id="dbdoclet.50438217_pgfId-1290153" xreflabel=""/>tunefs.lustre --param ost.quota_type=ug $ost_dev
</screen>
- <informaltable frame="none">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="5*"/>
- <colspec colname="c2" colwidth="95*"/>
-
-
- <tbody>
- <row>
- <entry><para><inlinemediaobject><imageobject role="html">
- <imagedata fileref="./shared/caution.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/caution.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></para></entry>
- <entry><para><emphasis role="bold">Caution -</emphasis><anchor xml:id="dbdoclet.50438217_pgfId-1290154" xreflabel=""/>If you are using mkfs.lustre --param mdt.quota_type=ug or tunefs.lustre --param ost.quota_type=ug, be sure to run the command on all OSTs and the MDT. Otherwise, abnormal results may occur.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+ <caution><para>If you are using mkfs.lustre --param mdt.quota_type=ug or tunefs.lustre --param ost.quota_type=ug, be sure to run the command on all OSTs and the MDT. Otherwise, abnormal results may occur.</para></caution>
+
<section remap="h4">
<title><anchor xml:id="dbdoclet.50438217_pgfId-1290155" xreflabel=""/>21.2.0.1 Administrative and Operational Quotas</title>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290156" xreflabel=""/>Lustre has two kinds of quota files:</para>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290157" xreflabel=""/> Administrative quotas (for the MDT), which contain limits for users/groups for the entire cluster.</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
<listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290158" xreflabel=""/> Operational quotas (for the MDT and OSTs), which contain quota information dedicated to a cluster node.</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
</itemizedlist>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290159" xreflabel=""/>Lustre 1.6.5 introduced the v2 file format for administrative quota files, with continued support for the old file format (v1). The mdt.quota_type parameter also handles ‘1’ and ‘2’ options, to specify the Lustre quota versions that will be used. For example:</para>
<screen><anchor xml:id="dbdoclet.50438217_pgfId-1290160" xreflabel=""/>--param mdt.quota_type=ug1
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290168" xreflabel=""/>For more information about the v1 and v2 formats, see <link xl:href="ConfiguringQuotas.html#50438217_66360">Quota File Formats</link>.</para>
</section>
</section>
- <section remap="h2">
- <title>21.3
<anchor xml:id="dbdoclet.50438217_49939" xreflabel=""/>Creating Quota <anchor xml:id="dbdoclet.50438217_marker-1290170" xreflabel=""/>Files and Quota Administration</title>
+ <section xml:id="dbdoclet.50438217_49939">
+ <title>21.3 Creating Quota <anchor xml:id="dbdoclet.50438217_marker-1290170" xreflabel=""/>Files and Quota Administration</title>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290172" xreflabel=""/>Once each quota-enabled file system is remounted, it is capable of working with disk quotas. However, the file system is not yet ready to support quotas. If umount has been done regularly, run the lfs command with the quotaon option. If umount has not been done, perform these steps:</para>
+ <orderedlist><listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290173" xreflabel=""/> 1. Take Lustre ''offline''.</para>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1291329" xreflabel=""/>That is, verify that no write operations (append, write, truncate, create or delete) are being performed (preparing to run lfs quotacheck). Operations that do not change Lustre files (such as read or mount) are okay to run.</para>
- <informaltable frame="none">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="5*"/>
- <colspec colname="c2" colwidth="95*"/>
-
-
- <tbody>
- <row>
- <entry><para><inlinemediaobject><imageobject role="html">
- <imagedata fileref="./shared/caution.gif" scalefit="1"/>
- </imageobject>
-<imageobject role="fo">
- <imagedata contentdepth="100%" contentwidth="" depth="" fileref="./shared/caution.gif" scalefit="1" width="100%"/>
- </imageobject>
-</inlinemediaobject></para></entry>
- <entry><para><emphasis role="bold">Caution -</emphasis><anchor xml:id="dbdoclet.50438217_pgfId-1290174" xreflabel=""/>When lfsquotacheck is run, Lustre must NOT be performing any write operations. Failure to follow this caution may cause the statistic information of quota to be inaccurate. For example, the number of blocks used by OSTs for users or groups will be inaccurate, which can cause unexpected quota problems.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+
+ <caution><para>When lfsquotacheck is run, Lustre must NOT be performing any write operations. Failure to follow this caution may cause the statistic information of quota to be inaccurate. For example, the number of blocks used by OSTs for users or groups will be inaccurate, which can cause unexpected quota problems.</para></caution>
+
+ </listitem><listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290175" xreflabel=""/> 2. Run the <emphasis role="bold">lfs</emphasis> command with the <emphasis role="bold">quotacheck</emphasis> option:</para>
<screen><anchor xml:id="dbdoclet.50438217_pgfId-1290176" xreflabel=""/># lfs quotacheck -ug /mnt/lustre
</screen>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290178" xreflabel=""/>u -- checks the user disk quota information</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
<listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290179" xreflabel=""/>g -- checks the group disk quota information</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
</itemizedlist>
+ </listitem></orderedlist>
+
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290180" xreflabel=""/>The lfsquotacheck command checks all objects on all OSTs and the MDS to sum up for every UID/GID. It reads all Lustre metadata and re-computes the number of blocks/inodes that each UID/GID has used. If there are many files in Lustre, it may take a long time to complete.</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438217_pgfId-1290181" xreflabel=""/>User and group quotas are separate. If either quota limit is reached, a process with the corresponding UID/GID cannot allocate more space on the file system.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438217_pgfId-1290182" xreflabel=""/>When lfsquotacheck runs, it creates a quota file -- a sparse file with a size proportional to the highest UID in use and UID/GID distribution. As a general rule, if the highest UID in use is large, then the sparse file will be large, which may affect functions such as creating a snapshot.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438217_pgfId-1290183" xreflabel=""/>For Lustre 1.6 releases before version 1.6.5, and 1.4 releases before version 1.4.12, if the underlying ldiskfs file system has not unmounted gracefully (due to a crash, for example), re-run quotacheck to obtain accurate quota information. Lustre 1.6.5 and 1.4.12 use journaled quota, so it is not necessary to run quotacheck after an unclean shutdown.</para><para> In certain failure situations (e.g., when a broken Lustre installation or build is used), re-run quotacheck after checking the server kernel logs and fixing the root problem.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+
+ <note><para>User and group quotas are separate. If either quota limit is reached, a process with the corresponding UID/GID cannot allocate more space on the file system.</para></note>
+
+ <note><para>When lfsquotacheck runs, it creates a quota file -- a sparse file with a size proportional to the highest UID in use and UID/GID distribution. As a general rule, if the highest UID in use is large, then the sparse file will be large, which may affect functions such as creating a snapshot.</para></note>
+
+ <note><para>For Lustre 1.6 releases before version 1.6.5, and 1.4 releases before version 1.4.12, if the underlying ldiskfs file system has not unmounted gracefully (due to a crash, for example), re-run quotacheck to obtain accurate quota information. Lustre 1.6.5 and 1.4.12 use journaled quota, so it is not necessary to run quotacheck after an unclean shutdown.</para><para> In certain failure situations (e.g., when a broken Lustre installation or build is used), re-run quotacheck after checking the server kernel logs and fixing the root problem.</para></note>
+
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290184" xreflabel=""/>The lfs command includes several command options to work with quotas:</para>
<itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290185" xreflabel=""/>quotaon -- enables disk quotas on the specified file system. The file system quota files must be present in the root directory of the file system.</para>
- </listitem>
-<listitem>
- <para> </para>
+ <para><anchor xml:id="dbdoclet.50438217_pgfId-1290185" xreflabel=""/><varname>quotaon</varname> -- enables disk quotas on the specified file system. The file system quota files must be present in the root directory of the file system.</para>
</listitem>
+
<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290186" xreflabel=""/>quotaoff -- disables disk quotas on the specified file system.</para>
+ <para><anchor xml:id="dbdoclet.50438217_pgfId-1290186" xreflabel=""/><varname>quotaoff</varname> -- disables disk quotas on the specified file system.</para>
</listitem>
+
<listitem>
- <para> </para>
+ <para><anchor xml:id="dbdoclet.50438217_pgfId-1290187" xreflabel=""/><varname>quota</varname> -- displays general quota information (disk usage and limits)</para>
</listitem>
+
<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290187" xreflabel=""/>quota -- displays general quota information (disk usage and limits)</para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290188" xreflabel=""/>setquota -- specifies quota limits and tunes the grace period. By default, the grace period is one week.</para>
- </listitem>
-<listitem>
- <para> </para>
+ <para><anchor xml:id="dbdoclet.50438217_pgfId-1290188" xreflabel=""/><varname>setquota</varname> -- specifies quota limits and tunes the grace period. By default, the grace period is one week.</para>
</listitem>
+
</itemizedlist>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290189" xreflabel=""/> Usage:</para>
<screen><anchor xml:id="dbdoclet.50438217_pgfId-1290190" xreflabel=""/>lfs quotaon [-ugf] <filesystem>
<screen><anchor xml:id="dbdoclet.50438217_pgfId-1290212" xreflabel=""/>$ lfs setquota -u bob 307200 309200 10000 11000 /mnt/lustre
</screen>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290213" xreflabel=""/>In this example, the quota for user "bob" is set to 300 MB (309200*1024) and the hard limit is 11,000 files. Therefore, the inode hard limit should be 11000.</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438217_pgfId-1290214" xreflabel=""/>For the Lustre command $lfssetquota/quota ... the qunit for block is KB (1024) and the qunit for inode is 1.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+
+ <note><para>For the Lustre command $lfssetquota/quota ... the qunit for block is KB (1024) and the qunit for inode is 1.</para></note>
+
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290215" xreflabel=""/>The quota command displays the quota allocated and consumed for each Lustre device. Using the previous setquota example, running this lfs quota command:</para>
<screen><anchor xml:id="dbdoclet.50438217_pgfId-1290216" xreflabel=""/>$ lfs quota -u bob -v /mnt/lustre
</screen>
- 0 - 0 -
</screen>
</section>
- <section remap="h2">
- <title>21.4
<anchor xml:id="dbdoclet.50438217_15106" xreflabel=""/>Quota<anchor xml:id="dbdoclet.50438217_marker-1290226" xreflabel=""/> Allocation</title>
+ <section xml:id="dbdoclet.50438217_15106">
+ <title>21.4 Quota<anchor xml:id="dbdoclet.50438217_marker-1290226" xreflabel=""/> Allocation</title>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290228" xreflabel=""/>In Lustre, quota must be properly allocated or users may experience unnecessary failures. The file system block quota is divided up among the OSTs within the file system. Each OST requests an allocation which is increased up to the quota limit. The quota allocation is then quantized to reduce the number of quota-related request traffic. By default, Lustre supports both user and group quotas to limit disk usage and file counts.</para>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290229" xreflabel=""/>The quota system in Lustre is completely compatible with the quota systems used on other file systems. The Lustre quota system distributes quotas from the quota master. Generally, the MDS is the quota master for both inodes and blocks. All OSTs and the MDS are quota slaves to the OSS nodes. To reduce quota requests and get reasonably accurate quota distribution, the transfer quota unit (qunit) between quota master and quota slaves is changed dynamically by the lquota module. The default minimum value of qunit is 1 MB for blocks and 2 for inodes. The proc entries to set these values are: /proc/fs/lustre/mds/lustre-MDT*/quota_least_bunit and /proc/fs/lustre/mds/lustre-MDT*/quota_least_iunit. The default maximum value of qunit is 128 MB for blocks and 5120 for inodes. The proc entries to set these values are quota_bunit_sz and quota_iunit_sz in the MDT and OSTs.</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438217_pgfId-1290230" xreflabel=""/>In general, the quota_bunit_sz value should be larger than 1 MB. For testing purposes, it can be set to 4 KB, if necessary.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+ <note><para>In general, the quota_bunit_sz value should be larger than 1 MB. For testing purposes, it can be set to 4 KB, if necessary.</para></note>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290231" xreflabel=""/>The file system block quota is divided up among the OSTs and the MDS within the file system. Only the MDS uses the file system inode quota.</para>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290232" xreflabel=""/>This means that the minimum quota for block is 1 MB* (the number of OSTs + the number of MDSs), which is 1 MB* (number of OSTs + 1). If you attempt to assign a smaller quota, users maybe not be able to create files. As noted, the default minimum quota for inodes is 2. The default is established at file system creation time, but can be tuned via /proc values (described below). The inode quota is also allocated in a quantized manner on the MDS.</para>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290233" xreflabel=""/>If we look at the setquota example again, running this lfsquota command:</para>
- - - -
</screen>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290242" xreflabel=""/>The total quota limit of 30,920 is allotted to user bob, which is further distributed to two OSTs and one MDS.</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438217_pgfId-1290243" xreflabel=""/>Values appended with “*†show the limit that has been over-used (exceeding the quota), and receives this message Disk quota exceeded. For example:</para><para> \</para><para>$ cp: writing `/mnt/lustre/var/cache/fontconfig/ beeeeb3dfe132a8a0633a017c99ce0-x86.cache’: Disk quota exceeded.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+ <note><para>Values appended with '*' show the limit that has been over-used (exceeding the quota), and receives this message Disk quota exceeded. For example:</para><para> \</para><para>$ cp: writing `/mnt/lustre/var/cache/fontconfig/ beeeeb3dfe132a8a0633a017c99ce0-x86.cache': Disk quota exceeded.</para></note>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290244" xreflabel=""/>The requested quota of 300 MB is divided across the OSTs.</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438217_pgfId-1290245" xreflabel=""/>It is very important to note that the block quota is consumed per OST and the MDS per block and inode (there is only one MDS for inodes). Therefore, when the quota is consumed on one OST, the client may not be able to create files regardless of the quota available on other OSTs.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+ <note><para>It is very important to note that the block quota is consumed per OST and the MDS per block and inode (there is only one MDS for inodes). Therefore, when the quota is consumed on one OST, the client may not be able to create files regardless of the quota available on other OSTs.</para></note>
<section remap="h5">
<title><anchor xml:id="dbdoclet.50438217_pgfId-1290246" xreflabel=""/>Additional information:</title>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290247" xreflabel=""/><emphasis role="bold">Grace period</emphasis> -- The period of time (in seconds) within which users are allowed to exceed their soft limit. There are four types of grace periods:</para>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290248" xreflabel=""/> user block soft limit</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
<listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290249" xreflabel=""/> user inode soft limit</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
<listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290250" xreflabel=""/> group block soft limit</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
<listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290251" xreflabel=""/> group inode soft limit</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
</itemizedlist>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290252" xreflabel=""/>The grace periods are applied to all users. The user block soft limit is for all users who are using a blocks quota.</para>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290253" xreflabel=""/><emphasis role="bold">Soft limit</emphasis> -- Once you are beyond the soft limit, the quota module begins to time, but you still can write block and inode. When you are always beyond the soft limit and use up your grace time, you get the same result as the hard limit. For inodes and blocks, it is the same. Usually, the soft limit MUST be less than the hard limit; if not, the quota module never triggers the timing. If the soft limit is not needed, leave it as zero (0).</para>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290264" xreflabel=""/>The quota_bunit_sz parameter displays bytes, however lfs setquota uses KBs. The quota_bunit_sz parameter must be a multiple of 1024. A proper minimum KB size for lfs setquota can be calculated as:</para>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290265" xreflabel=""/><emphasis role="bold">Size in KBs = minimum_quota_bunit_sz * (number of OSTS + 1) = 1024 * (number of OSTs +1)</emphasis></para>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290266" xreflabel=""/>We add one (1) to the number of OSTs as the MDS also consumes KBs. As inodes are only consumed on the MDS, the minimum inode size for lfs setquota is equal to quota_iunit_sz.</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438217_pgfId-1290267" xreflabel=""/>Setting the quota below this limit may prevent the user from all file creation.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+ <note><para>Setting the quota below this limit may prevent the user from all file creation.</para></note>
</section>
</section>
- <section remap="h2">
- <title>21.5
<anchor xml:id="dbdoclet.50438217_27895" xreflabel=""/>Known Issues <anchor xml:id="dbdoclet.50438217_marker-1290269" xreflabel=""/>with Quotas</title>
+ <section xml:id="dbdoclet.50438217_27895">
+ <title>21.5 Known Issues <anchor xml:id="dbdoclet.50438217_marker-1290269" xreflabel=""/>with Quotas</title>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290271" xreflabel=""/>Using quotas in Lustre can be complex and there are several known issues.</para>
<section remap="h3">
<title><anchor xml:id="dbdoclet.50438217_pgfId-1290273" xreflabel=""/>21.5.1 Granted<anchor xml:id="dbdoclet.50438217_marker-1290272" xreflabel=""/> Cache and Quota Limits</title>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290274" xreflabel=""/>In Lustre, granted cache does not respect quota limits. In this situation, OSTs grant cache to Lustre client to accelerate I/O. Granting cache causes writes to be successful in OSTs, even if they exceed the quota limits, and will overwrite them.</para>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290275" xreflabel=""/>The sequence is:</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290276" xreflabel=""/> 1. A user writes files to Lustre.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290277" xreflabel=""/> 2. If the Lustre client has enough granted cache, then it returns ‘success’ to users and arranges the writes to the OSTs.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290278" xreflabel=""/> 3. Because Lustre clients have delivered success to users, the OSTs cannot fail these writes.</para>
+ <orderedlist><listitem>
+ <para><anchor xml:id="dbdoclet.50438217_pgfId-1290276" xreflabel=""/>A user writes files to Lustre.</para>
+ </listitem><listitem>
+ <para><anchor xml:id="dbdoclet.50438217_pgfId-1290277" xreflabel=""/>If the Lustre client has enough granted cache, then it returns ‘success’ to users and arranges the writes to the OSTs.</para>
+ </listitem><listitem>
+ <para><anchor xml:id="dbdoclet.50438217_pgfId-1290278" xreflabel=""/>Because Lustre clients have delivered success to users, the OSTs cannot fail these writes.</para>
+ </listitem></orderedlist>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290279" xreflabel=""/>Because of granted cache, writes always overwrite quota limitations. For example, if you set a 400 GB quota on user A and use IOR to write for user A from a bundle of clients, you will write much more data than 400 GB, and cause an out-of-quota error (-EDQUOT).</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438217_pgfId-1290280" xreflabel=""/>The effect of granted cache on quota limits can be mitigated, but not eradicated. Reduce the max_dirty_buffer in the clients (can be set from 0 to 512). To set max_dirty_buffer to 0:</para><para> * In releases after Lustre 1.6.5, lctl set_param osc.*.max_dirty_mb=0.</para><para> * In releases before Lustre 1.6.5, proc/fs/lustre/osc/*/max_dirty_mb; do echo 512 > $O</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+ <note><para>The effect of granted cache on quota limits can be mitigated, but not eradicated. Reduce the max_dirty_buffer in the clients (can be set from 0 to 512). To set max_dirty_buffer to 0:</para><para> * In releases after Lustre 1.6.5, lctl set_param osc.*.max_dirty_mb=0.</para><para> * In releases before Lustre 1.6.5, proc/fs/lustre/osc/*/max_dirty_mb; do echo 512 > $O</para></note>
</section>
<section remap="h3">
<title><anchor xml:id="dbdoclet.50438217_pgfId-1290283" xreflabel=""/>21.5.2 <anchor xml:id="dbdoclet.50438217_50442" xreflabel=""/>Quota <anchor xml:id="dbdoclet.50438217_marker-1290282" xreflabel=""/>Limits</title>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290285" xreflabel=""/> Lustre version 1.4.11 and earlier (for 1.4.x releases) and Lustre version 1.6.4 and earlier (for 1.6.x releases) support quota limits less than 4 TB.</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
<listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290286" xreflabel=""/> Lustre versions 1.4.12, 1.6.5 and later support quota limits of 4 TB and greater in Lustre configurations with OST storage limits of 4 TB and less.</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
<listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290287" xreflabel=""/> Future Lustre versions are expected to support quota limits of 4 TB and greater with no OST storage limits.</para>
</listitem>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290330" xreflabel=""/> For the v2 quota file format, (OBJECTS/admin_quotafile_v2.{usr,grp})</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
<listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290331" xreflabel=""/> For the v1 quota file format, (OBJECTS/admin_quotafile.{usr,grp})</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
</itemizedlist>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290332" xreflabel=""/>Lustre 1.6.6 and later use ost.quota_type to force a specific operational quota version (v2 or v1).</para>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290333" xreflabel=""/> For the v2 quota file format, (lquota_v2.{user,group})</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
<listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290334" xreflabel=""/> For the v1 quota file format, (lquota.{user,group})</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
</itemizedlist>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290335" xreflabel=""/>The quota_type specifier can be used to set different combinations of administrative/operational quota file versions on a Lustre node:</para>
<itemizedlist><listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290336" xreflabel=""/> "1" - v1 (32-bit) administrative quota file, v1 (32-bit) operational quota file (default in releases before Lustre 1.6.5)</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
<listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290337" xreflabel=""/> "2" - v2 (64-bit) administrative quota file, v1 (32-bit) operational quota file (default in Lustre 1.6.5)</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
<listitem>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290338" xreflabel=""/> "3" - v2 (64-bit) administrative quota file, v2 (64-bit) operational quota file (default in releases after Lustre 1.6.5)</para>
</listitem>
-<listitem>
- <para> </para>
- </listitem>
+
</itemizedlist>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290339" xreflabel=""/>If quotas do not exist or look broken, then quotacheck creates quota files of a required name and format.</para>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290340" xreflabel=""/>If Lustre is using the v2 quota file format when only v1 quota files exist, then quotacheck converts old v1 quota files to new v2 quota files. This conversion is triggered automatically, and is transparent to users. If an old quota file does not exist or looks broken, then the new v2 quota file will be empty. In case of an error, details can be found in the kernel log of the corresponding MDS/OST. During conversion of a v1 quota file to a v2 quota file, the v2 quota file is marked as broken, to avoid it being used if a crash occurs. The quota module does not use broken quota files (keeping quota off).</para>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290341" xreflabel=""/>In most situations, Lustre administrators do not need to set specific versioning options. Upgrading Lustre without using quota_type to force specific quota file versions results in quota files being upgraded automatically to the latest version. The option ensures backward compatibility, preventing a quota file upgrade to a version which is not supported by earlier Lustre versions.</para>
</section>
</section>
- <section remap="h2">
- <title>21.6
<anchor xml:id="dbdoclet.50438217_20772" xreflabel=""/>Lustre <anchor xml:id="dbdoclet.50438217_marker-1290343" xreflabel=""/>Quota Statistics</title>
+ <section xml:id="dbdoclet.50438217_20772">
+ <title>21.6 Lustre <anchor xml:id="dbdoclet.50438217_marker-1290343" xreflabel=""/>Quota Statistics</title>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290345" xreflabel=""/>Lustre includes statistics that monitor quota activity, such as the kinds of quota RPCs sent during a specific period, the average time to complete the RPCs, etc. These statistics are useful to measure performance of a Lustre file system.</para>
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290346" xreflabel=""/>Each quota statistic consists of a quota event and min_time, max_time and sum_time values for the event.</para>
<informaltable frame="all">
<para><anchor xml:id="dbdoclet.50438217_pgfId-1290422" xreflabel=""/>In the fifth line, the quota_ctl event occurs four times. The min_time, max_time and sum_time statistics for this event are 80, 3470 and 4293, respectively. The unit is microseconds (s).</para>
</section>
</section>
- </section>
</chapter>
git://git.whamcloud.com / doc / manual.git / commitdiff