- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290242" xreflabel=""/>The total quota limit of 30,920 is allotted to user bob, which is further distributed to two OSTs and one MDS.</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438217_pgfId-1290243" xreflabel=""/>Values appended with “*†show the limit that has been over-used (exceeding the quota), and receives this message Disk quota exceeded. For example:</para><para> \</para><para>$ cp: writing `/mnt/lustre/var/cache/fontconfig/ beeeeb3dfe132a8a0633a017c99ce0-x86.cache’: Disk quota exceeded.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290244" xreflabel=""/>The requested quota of 300 MB is divided across the OSTs.</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438217_pgfId-1290245" xreflabel=""/>It is very important to note that the block quota is consumed per OST and the MDS per block and inode (there is only one MDS for inodes). Therefore, when the quota is consumed on one OST, the client may not be able to create files regardless of the quota available on other OSTs.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <section remap="h5">
- <title><anchor xml:id="dbdoclet.50438217_pgfId-1290246" xreflabel=""/>Additional information:</title>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290247" xreflabel=""/><emphasis role="bold">Grace period</emphasis> -- The period of time (in seconds) within which users are allowed to exceed their soft limit. There are four types of grace periods:</para>
- <itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290248" xreflabel=""/> user block soft limit</para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290249" xreflabel=""/> user inode soft limit</para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290250" xreflabel=""/> group block soft limit</para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290251" xreflabel=""/> group inode soft limit</para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290252" xreflabel=""/>The grace periods are applied to all users. The user block soft limit is for all users who are using a blocks quota.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290253" xreflabel=""/><emphasis role="bold">Soft limit</emphasis> -- Once you are beyond the soft limit, the quota module begins to time, but you still can write block and inode. When you are always beyond the soft limit and use up your grace time, you get the same result as the hard limit. For inodes and blocks, it is the same. Usually, the soft limit MUST be less than the hard limit; if not, the quota module never triggers the timing. If the soft limit is not needed, leave it as zero (0).</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290254" xreflabel=""/><emphasis role="bold">Hard limit</emphasis> -- When you are beyond the hard limit, you get -EQUOTA and cannot write inode/block any more. The hard limit is the absolute limit. When a grace period is set, you can exceed the soft limit within the grace period if are under the hard limits.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290255" xreflabel=""/>Lustre quota allocation is controlled by two variables, quota_bunit_sz and quota_iunit_sz referring to KBs and inodes, respectively. These values can be accessed on the MDS as /proc/fs/lustre/mds/*/quota_* and on the OST as /proc/fs/lustre/obdfilter/*/quota_*. The quota_bunit_sz and quota_iunit_sz variables are the maximum qunit values for blocks and inodes, respectively. At any time, module lquota chooses a reasonable qunit between the minimum and maximum values.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290256" xreflabel=""/>The /proc values are bounded by two other variables quota_btune_sz and quota_itune_sz. By default, the *tune_sz variables are set at 1/2 the *unit_sz variables, and you cannot set *tune_sz larger than *unit_sz. You must set bunit_sz first if it is increasing by more than 2x, and btune_sz first if it is decreasing by more than 2x.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290257" xreflabel=""/><emphasis role="bold">Total number of inodes</emphasis> -- To determine the total number of inodes, use lfsdf-i (and also /proc/fs/lustre/*/*/filestotal). For more information on using the lfsdf-i command and the command output, see <link xl:href="ManagingStripingFreeSpace.html#50438209_35838">Checking File System Free Space</link>.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290261" xreflabel=""/>Unfortunately, the statfs interface does not report the free inode count directly, but instead reports the total inode and used inode counts. The free inode count is calculated for df from (total inodes - used inodes).</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290262" xreflabel=""/>It is not critical to know a file system’s total inode count. Instead, you should know (accurately), the free inode count and the used inode count for a file system. Lustre manipulates the total inode count in order to accurately report the other two values.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290263" xreflabel=""/>The values set for the MDS must match the values set on the OSTs.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290264" xreflabel=""/>The quota_bunit_sz parameter displays bytes, however lfs setquota uses KBs. The quota_bunit_sz parameter must be a multiple of 1024. A proper minimum KB size for lfs setquota can be calculated as:</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290265" xreflabel=""/><emphasis role="bold">Size in KBs = minimum_quota_bunit_sz * (number of OSTS + 1) = 1024 * (number of OSTs +1)</emphasis></para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290266" xreflabel=""/>We add one (1) to the number of OSTs as the MDS also consumes KBs. As inodes are only consumed on the MDS, the minimum inode size for lfs setquota is equal to quota_iunit_sz.</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438217_pgfId-1290267" xreflabel=""/>Setting the quota below this limit may prevent the user from all file creation.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- </section>
- <section remap="h2">
- <title>21.5 <anchor xml:id="dbdoclet.50438217_27895" xreflabel=""/>Known Issues <anchor xml:id="dbdoclet.50438217_marker-1290269" xreflabel=""/>with Quotas</title>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290271" xreflabel=""/>Using quotas in Lustre can be complex and there are several known issues.</para>
- <section remap="h3">
- <title><anchor xml:id="dbdoclet.50438217_pgfId-1290273" xreflabel=""/>21.5.1 Granted<anchor xml:id="dbdoclet.50438217_marker-1290272" xreflabel=""/> Cache and Quota Limits</title>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290274" xreflabel=""/>In Lustre, granted cache does not respect quota limits. In this situation, OSTs grant cache to Lustre client to accelerate I/O. Granting cache causes writes to be successful in OSTs, even if they exceed the quota limits, and will overwrite them.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290275" xreflabel=""/>The sequence is:</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290276" xreflabel=""/> 1. A user writes files to Lustre.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290277" xreflabel=""/> 2. If the Lustre client has enough granted cache, then it returns ‘success’ to users and arranges the writes to the OSTs.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290278" xreflabel=""/> 3. Because Lustre clients have delivered success to users, the OSTs cannot fail these writes.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290279" xreflabel=""/>Because of granted cache, writes always overwrite quota limitations. For example, if you set a 400 GB quota on user A and use IOR to write for user A from a bundle of clients, you will write much more data than 400 GB, and cause an out-of-quota error (-EDQUOT).</para>
- <informaltable frame="none">
- <tgroup cols="1">
- <colspec colname="c1" colwidth="100*"/>
- <tbody>
- <row>
- <entry><para><emphasis role="bold">Note -</emphasis><anchor xml:id="dbdoclet.50438217_pgfId-1290280" xreflabel=""/>The effect of granted cache on quota limits can be mitigated, but not eradicated. Reduce the max_dirty_buffer in the clients (can be set from 0 to 512). To set max_dirty_buffer to 0:</para><para> * In releases after Lustre 1.6.5, lctl set_param osc.*.max_dirty_mb=0.</para><para> * In releases before Lustre 1.6.5, proc/fs/lustre/osc/*/max_dirty_mb; do echo 512 > $O</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h3">
- <title><anchor xml:id="dbdoclet.50438217_pgfId-1290283" xreflabel=""/>21.5.2 <anchor xml:id="dbdoclet.50438217_50442" xreflabel=""/>Quota <anchor xml:id="dbdoclet.50438217_marker-1290282" xreflabel=""/>Limits</title>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290284" xreflabel=""/>Available quota limits depend on the Lustre version you are using.</para>
- <itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290285" xreflabel=""/> Lustre version 1.4.11 and earlier (for 1.4.x releases) and Lustre version 1.6.4 and earlier (for 1.6.x releases) support quota limits less than 4 TB.</para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290286" xreflabel=""/> Lustre versions 1.4.12, 1.6.5 and later support quota limits of 4 TB and greater in Lustre configurations with OST storage limits of 4 TB and less.</para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290287" xreflabel=""/> Future Lustre versions are expected to support quota limits of 4 TB and greater with no OST storage limits.</para>
- </listitem>
-<listitem>
- <informaltable frame="all">
- <tgroup cols="3">
- <colspec colname="c1" colwidth="33*"/>
- <colspec colname="c2" colwidth="33*"/>
- <colspec colname="c3" colwidth="33*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438217_pgfId-1290290" xreflabel=""/>Lustre Version</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438217_pgfId-1290292" xreflabel=""/>Quota Limit Per User/Per Group</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438217_pgfId-1290294" xreflabel=""/>OST Storage Limit</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290296" xreflabel=""/>1.4.11 and earlier</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290298" xreflabel=""/>< 4TB</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290300" xreflabel=""/>n/a</para></entry>
- </row>
- <row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290302" xreflabel=""/>1.4.12</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290304" xreflabel=""/>=> 4TB</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290306" xreflabel=""/><= 4TB of storage</para></entry>
- </row>
- <row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290308" xreflabel=""/>1.6.4 and earlier</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290310" xreflabel=""/>< 4TB</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290312" xreflabel=""/>n/a</para></entry>
- </row>
- <row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290314" xreflabel=""/>1.6.5</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290316" xreflabel=""/>=> 4TB</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290318" xreflabel=""/><= 4TB of storage</para></entry>
- </row>
- <row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290320" xreflabel=""/>Future Lustre versions</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290322" xreflabel=""/>=> 4TB</para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290324" xreflabel=""/>No storage limit</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </listitem>
-</itemizedlist>
- </section>
- <section remap="h3">
- <title><anchor xml:id="dbdoclet.50438217_pgfId-1290327" xreflabel=""/>21.5.3 <anchor xml:id="dbdoclet.50438217_66360" xreflabel=""/>Quota <anchor xml:id="dbdoclet.50438217_marker-1290326" xreflabel=""/>File Formats</title>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290328" xreflabel=""/>Lustre 1.6.5 introduced the v2 file format for administrative quotas, with 64-bit limits that support large-limits handling. The old quota file format (v1), with 32-bit limits, is also supported. Lustre 1.6.6 introduced the v2 file format for operational quotas. A few notes regarding the current quota file formats:</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290329" xreflabel=""/>Lustre 1.6.5 and later use mdt.quota_type to force a specific administrative quota version (v2 or v1).</para>
- <itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290330" xreflabel=""/> For the v2 quota file format, (OBJECTS/admin_quotafile_v2.{usr,grp})</para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290331" xreflabel=""/> For the v1 quota file format, (OBJECTS/admin_quotafile.{usr,grp})</para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290332" xreflabel=""/>Lustre 1.6.6 and later use ost.quota_type to force a specific operational quota version (v2 or v1).</para>
- <itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290333" xreflabel=""/> For the v2 quota file format, (lquota_v2.{user,group})</para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290334" xreflabel=""/> For the v1 quota file format, (lquota.{user,group})</para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290335" xreflabel=""/>The quota_type specifier can be used to set different combinations of administrative/operational quota file versions on a Lustre node:</para>
- <itemizedlist><listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290336" xreflabel=""/> "1" - v1 (32-bit) administrative quota file, v1 (32-bit) operational quota file (default in releases before Lustre 1.6.5)</para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290337" xreflabel=""/> "2" - v2 (64-bit) administrative quota file, v1 (32-bit) operational quota file (default in Lustre 1.6.5)</para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-<listitem>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290338" xreflabel=""/> "3" - v2 (64-bit) administrative quota file, v2 (64-bit) operational quota file (default in releases after Lustre 1.6.5)</para>
- </listitem>
-<listitem>
- <para> </para>
- </listitem>
-</itemizedlist>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290339" xreflabel=""/>If quotas do not exist or look broken, then quotacheck creates quota files of a required name and format.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290340" xreflabel=""/>If Lustre is using the v2 quota file format when only v1 quota files exist, then quotacheck converts old v1 quota files to new v2 quota files. This conversion is triggered automatically, and is transparent to users. If an old quota file does not exist or looks broken, then the new v2 quota file will be empty. In case of an error, details can be found in the kernel log of the corresponding MDS/OST. During conversion of a v1 quota file to a v2 quota file, the v2 quota file is marked as broken, to avoid it being used if a crash occurs. The quota module does not use broken quota files (keeping quota off).</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290341" xreflabel=""/>In most situations, Lustre administrators do not need to set specific versioning options. Upgrading Lustre without using quota_type to force specific quota file versions results in quota files being upgraded automatically to the latest version. The option ensures backward compatibility, preventing a quota file upgrade to a version which is not supported by earlier Lustre versions.</para>
- </section>
- </section>
- <section remap="h2">
- <title>21.6 <anchor xml:id="dbdoclet.50438217_20772" xreflabel=""/>Lustre <anchor xml:id="dbdoclet.50438217_marker-1290343" xreflabel=""/>Quota Statistics</title>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290345" xreflabel=""/>Lustre includes statistics that monitor quota activity, such as the kinds of quota RPCs sent during a specific period, the average time to complete the RPCs, etc. These statistics are useful to measure performance of a Lustre file system.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290346" xreflabel=""/>Each quota statistic consists of a quota event and min_time, max_time and sum_time values for the event.</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438217_pgfId-1290349" xreflabel=""/>Quota Event</emphasis></para></entry>
- <entry><para><emphasis role="bold"><anchor xml:id="dbdoclet.50438217_pgfId-1290351" xreflabel=""/>Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290353" xreflabel=""/><emphasis role="bold">sync_acq_req</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290355" xreflabel=""/>Quota slaves send a acquiring_quota request and wait for its return.</para></entry>
- </row>
- <row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290357" xreflabel=""/><emphasis role="bold">sync_rel_req</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290359" xreflabel=""/>Quota slaves send a releasing_quota request and wait for its return.</para></entry>
- </row>
- <row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290361" xreflabel=""/><emphasis role="bold">async_acq_req</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290363" xreflabel=""/>Quota slaves send an acquiring_quota request and do not wait for its return.</para></entry>
- </row>
- <row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290365" xreflabel=""/><emphasis role="bold">async_rel_req</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290367" xreflabel=""/>Quota slaves send a releasing_quota request and do not wait for its return.</para></entry>
- </row>
- <row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290369" xreflabel=""/><emphasis role="bold">wait_for_blk_quota (lquota_chkquota)</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290371" xreflabel=""/>Before data is written to OSTs, the OSTs check if the remaining block quota is sufficient. This is done in the lquota_chkquota function.</para></entry>
- </row>
- <row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290373" xreflabel=""/><emphasis role="bold">wait_for_ino_quota (lquota_chkquota)</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290375" xreflabel=""/>Before files are created on the MDS, the MDS checks if the remaining inode quota is sufficient. This is done in the lquota_chkquota function.</para></entry>
- </row>
- <row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290377" xreflabel=""/><emphasis role="bold">wait_for_blk_quota (lquota_pending_commit)</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290379" xreflabel=""/>After blocks are written to OSTs, relative quota information is updated. This is done in the lquota_pending_commit function.</para></entry>
- </row>
- <row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290381" xreflabel=""/><emphasis role="bold">wait_for_ino_quota (lquota_pending_commit)</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290383" xreflabel=""/>After files are created, relative quota information is updated. This is done in the lquota_pending_commit function.</para></entry>
- </row>
- <row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290385" xreflabel=""/><emphasis role="bold">wait_for_pending_blk_quota_req (qctxt_wait_pending_dqacq)</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290387" xreflabel=""/>On the MDS or OSTs, there is one thread sending a quota request for a specific UID/GID for block quota at any time. At that time, if other threads need to do this too, they should wait. This is done in the qctxt_wait_pending_dqacq function.</para></entry>
- </row>
- <row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290389" xreflabel=""/><emphasis role="bold">wait_for_pending_ino_quota_req (qctxt_wait_pending_dqacq)</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290391" xreflabel=""/>On the MDS, there is one thread sending a quota request for a specific UID/GID for inode quota at any time. If other threads need to do this too, they should wait. This is done in the qctxt_wait_pending_dqacq function.</para></entry>
- </row>
- <row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290393" xreflabel=""/><emphasis role="bold">nowait_for_pending_blk_quota_req (qctxt_wait_pending_dqacq)</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290395" xreflabel=""/>On the MDS or OSTs, there is one thread sending a quota request for a specific UID/GID for block quota at any time. When threads enter qctxt_wait_pending_dqacq, they do not need to wait. This is done in the qctxt_wait_pending_dqacq function.</para></entry>
- </row>
- <row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290397" xreflabel=""/><emphasis role="bold">nowait_for_pending_ino_quota_req (qctxt_wait_pending_dqacq)</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290399" xreflabel=""/>On the MDS, there is one thread sending a quota request for a specific UID/GID for inode quota at any time. When threads enter qctxt_wait_pending_dqacq, they do not need to wait. This is done in the qctxt_wait_pending_dqacq function.</para></entry>
- </row>
- <row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290401" xreflabel=""/><emphasis role="bold">quota_ctl</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290403" xreflabel=""/>The quota_ctl statistic is generated when lfs setquota, lfs quota and so on, are issued.</para></entry>
- </row>
- <row>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290405" xreflabel=""/><emphasis role="bold">adjust_qunit</emphasis></para></entry>
- <entry><para> <anchor xml:id="dbdoclet.50438217_pgfId-1290407" xreflabel=""/>Each time qunit is adjusted, it is counted.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <section remap="h3">
- <title><anchor xml:id="dbdoclet.50438217_pgfId-1290408" xreflabel=""/>21.6.1 Interpreting Quota Statistics</title>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290409" xreflabel=""/>Quota statistics are an important measure of a Lustre file system’s performance. Interpreting these statistics correctly can help you diagnose problems with quotas, and may indicate adjustments to improve system performance.</para>
- <para><anchor xml:id="dbdoclet.50438217_pgfId-1290410" xreflabel=""/>For example, if you run this command on the OSTs:</para>
- <screen><anchor xml:id="dbdoclet.50438217_pgfId-1290411" xreflabel=""/>cat /proc/fs/lustre/lquota/lustre-OST0000/stats
+ </para>
+ <note>
+ <para>It is very important to note that the block quota is consumed per
+ OST and the inode quota per MDS. Therefore, when the quota is consumed on
+ one OST (resp. MDT), the client may not be able to create files
+ regardless of the quota available on other OSTs (resp. MDTs).</para>
+ <para>Setting the quota limit below the minimal qunit size may prevent
+ the user/group from all file creation. It is thus recommended to use
+ soft/hard limits which are a multiple of the number of OSTs * the minimal
+ qunit size.</para>
+ </note>
+ <para>To determine the total number of inodes, use
+ <literal>lfs df -i</literal>(and also
+ <literal>lctl get_param *.*.filestotal</literal>). For more information on
+ using the
+ <literal>lfs df -i</literal> command and the command output, see
+ <xref linkend="dbdoclet.50438209_35838" />.</para>
+ <para>Unfortunately, the
+ <literal>statfs</literal> interface does not report the free inode count
+ directly, but instead reports the total inode and used inode counts. The
+ free inode count is calculated for
+ <literal>df</literal> from (total inodes - used inodes). It is not critical
+ to know the total inode count for a file system. Instead, you should know
+ (accurately), the free inode count and the used inode count for a file
+ system. The Lustre software manipulates the total inode count in order to
+ accurately report the other two values.</para>
+ </section>
+ <section xml:id="quota_interoperability">
+ <title>
+ <indexterm>
+ <primary>Quotas</primary>
+ <secondary>Interoperability</secondary>
+ </indexterm>Quotas and Version Interoperability</title>
+ <para>The new quota protocol introduced in Lustre software release 2.4.0
+ <emphasis role="bold">is not compatible</emphasis> with previous
+ versions. As a consequence,
+ <emphasis role="bold">all Lustre servers must be upgraded to release 2.4.0
+ for quota to be functional</emphasis>. Quota limits set on the Lustre file
+ system prior to the upgrade will be automatically migrated to the new quota
+ index format. As for accounting information with ldiskfs backend, they will
+ be regenerated by running
+ <literal>tunefs.lustre --quota</literal> against all targets. It is worth
+ noting that running
+ <literal>tunefs.lustre --quota</literal> is
+ <emphasis role="bold">mandatory</emphasis> for all targets formatted with a
+ Lustre software release older than release 2.4.0, otherwise quota
+ enforcement as well as accounting won't be functional.</para>
+ <para>Besides, the quota protocol in release 2.4 takes for granted that the
+ Lustre client supports the
+ <literal>OBD_CONNECT_EINPROGRESS</literal> connect flag. Clients supporting
+ this flag will retry indefinitely when the server returns
+ <literal>EINPROGRESS</literal> in a reply. Here is the list of Lustre client
+ version which are compatible with release 2.4:</para>
+ <itemizedlist>
+ <listitem>
+ <para>Release 2.3-based clients and later</para>
+ </listitem>
+ <listitem>
+ <para>Release 1.8 clients newer or equal to release 1.8.9-wc1</para>
+ </listitem>
+ <listitem>
+ <para>Release 2.1 clients newer or equal to release 2.1.4</para>
+ </listitem>
+ </itemizedlist>
+ <para condition="l2A">To use the project quota functionality introduced in
+ Lustre 2.10, <emphasis role="bold">all Lustre servers and clients must be
+ upgraded to Lustre release 2.10 or later for project quota to work
+ correctly</emphasis>. Otherwise, project quota will be inaccessible on
+ clients and not be accounted for on OSTs.</para>
+ </section>
+ <section xml:id="granted_cache_and_quota_limits">
+ <title>
+ <indexterm>
+ <primary>Quotas</primary>
+ <secondary>known issues</secondary>
+ </indexterm>Granted Cache and Quota Limits</title>
+ <para>In a Lustre file system, granted cache does not respect quota limits.
+ In this situation, OSTs grant cache to a Lustre client to accelerate I/O.
+ Granting cache causes writes to be successful in OSTs, even if they exceed
+ the quota limits, and will overwrite them.</para>
+ <para>The sequence is:</para>
+ <orderedlist>
+ <listitem>
+ <para>A user writes files to the Lustre file system.</para>
+ </listitem>
+ <listitem>
+ <para>If the Lustre client has enough granted cache, then it returns
+ 'success' to users and arranges the writes to the OSTs.</para>
+ </listitem>
+ <listitem>
+ <para>Because Lustre clients have delivered success to users, the OSTs
+ cannot fail these writes.</para>
+ </listitem>
+ </orderedlist>
+ <para>Because of granted cache, writes always overwrite quota limitations.
+ For example, if you set a 400 GB quota on user A and use IOR to write for
+ user A from a bundle of clients, you will write much more data than 400 GB,
+ and cause an out-of-quota error (
+ <literal>EDQUOT</literal>).</para>
+ <note>
+ <para>The effect of granted cache on quota limits can be mitigated, but
+ not eradicated. Reduce the maximum amount of dirty data on the clients
+ (minimal value is 1MB):</para>
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>lctl set_param osc.*.max_dirty_mb=8</literal>
+ </para>
+ </listitem>
+ </itemizedlist>
+ </note>
+ </section>
+ <section xml:id="lustre_quota_statistics">
+ <title>
+ <indexterm>
+ <primary>Quotas</primary>
+ <secondary>statistics</secondary>
+ </indexterm>Lustre Quota Statistics</title>
+ <para>The Lustre software includes statistics that monitor quota activity,
+ such as the kinds of quota RPCs sent during a specific period, the average
+ time to complete the RPCs, etc. These statistics are useful to measure
+ performance of a Lustre file system.</para>
+ <para>Each quota statistic consists of a quota event and
+ <literal>min_time</literal>,
+ <literal>max_time</literal> and
+ <literal>sum_time</literal> values for the event.</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*" />
+ <colspec colname="c2" colwidth="50*" />
+ <thead>
+ <row>
+ <entry>
+ <para>
+ <emphasis role="bold">Quota Event</emphasis>
+ </para>
+ </entry>
+ <entry>
+ <para>
+ <emphasis role="bold">Description</emphasis>
+ </para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para>
+ <emphasis role="bold">sync_acq_req</emphasis>
+ </para>
+ </entry>
+ <entry>
+ <para>Quota slaves send a acquiring_quota request and wait for
+ its return.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>
+ <emphasis role="bold">sync_rel_req</emphasis>
+ </para>
+ </entry>
+ <entry>
+ <para>Quota slaves send a releasing_quota request and wait for
+ its return.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>
+ <emphasis role="bold">async_acq_req</emphasis>
+ </para>
+ </entry>
+ <entry>
+ <para>Quota slaves send an acquiring_quota request and do not
+ wait for its return.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>
+ <emphasis role="bold">async_rel_req</emphasis>
+ </para>
+ </entry>
+ <entry>
+ <para>Quota slaves send a releasing_quota request and do not wait
+ for its return.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>
+ <emphasis role="bold">wait_for_blk_quota
+ (lquota_chkquota)</emphasis>
+ </para>
+ </entry>
+ <entry>
+ <para>Before data is written to OSTs, the OSTs check if the
+ remaining block quota is sufficient. This is done in the
+ lquota_chkquota function.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>
+ <emphasis role="bold">wait_for_ino_quota
+ (lquota_chkquota)</emphasis>
+ </para>
+ </entry>
+ <entry>
+ <para>Before files are created on the MDS, the MDS checks if the
+ remaining inode quota is sufficient. This is done in the
+ lquota_chkquota function.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>
+ <emphasis role="bold">wait_for_blk_quota
+ (lquota_pending_commit)</emphasis>
+ </para>
+ </entry>
+ <entry>
+ <para>After blocks are written to OSTs, relative quota
+ information is updated. This is done in the lquota_pending_commit
+ function.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>
+ <emphasis role="bold">wait_for_ino_quota
+ (lquota_pending_commit)</emphasis>
+ </para>
+ </entry>
+ <entry>
+ <para>After files are created, relative quota information is
+ updated. This is done in the lquota_pending_commit
+ function.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>
+ <emphasis role="bold">wait_for_pending_blk_quota_req
+ (qctxt_wait_pending_dqacq)</emphasis>
+ </para>
+ </entry>
+ <entry>
+ <para>On the MDS or OSTs, there is one thread sending a quota
+ request for a specific UID/GID for block quota at any time. At
+ that time, if other threads need to do this too, they should
+ wait. This is done in the qctxt_wait_pending_dqacq
+ function.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>
+ <emphasis role="bold">wait_for_pending_ino_quota_req
+ (qctxt_wait_pending_dqacq)</emphasis>
+ </para>
+ </entry>
+ <entry>
+ <para>On the MDS, there is one thread sending a quota request for
+ a specific UID/GID for inode quota at any time. If other threads
+ need to do this too, they should wait. This is done in the
+ qctxt_wait_pending_dqacq function.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>
+ <emphasis role="bold">nowait_for_pending_blk_quota_req
+ (qctxt_wait_pending_dqacq)</emphasis>
+ </para>
+ </entry>
+ <entry>
+ <para>On the MDS or OSTs, there is one thread sending a quota
+ request for a specific UID/GID for block quota at any time. When
+ threads enter qctxt_wait_pending_dqacq, they do not need to wait.
+ This is done in the qctxt_wait_pending_dqacq function.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>
+ <emphasis role="bold">nowait_for_pending_ino_quota_req
+ (qctxt_wait_pending_dqacq)</emphasis>
+ </para>
+ </entry>
+ <entry>
+ <para>On the MDS, there is one thread sending a quota request for
+ a specific UID/GID for inode quota at any time. When threads
+ enter qctxt_wait_pending_dqacq, they do not need to wait. This is
+ done in the qctxt_wait_pending_dqacq function.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>
+ <emphasis role="bold">quota_ctl</emphasis>
+ </para>
+ </entry>
+ <entry>
+ <para>The quota_ctl statistic is generated when lfs
+ <literal>setquota</literal>,
+ <literal>lfs quota</literal> and so on, are issued.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para>
+ <emphasis role="bold">adjust_qunit</emphasis>
+ </para>
+ </entry>
+ <entry>
+ <para>Each time qunit is adjusted, it is counted.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <section remap="h3">
+ <title>Interpreting Quota Statistics</title>
+ <para>Quota statistics are an important measure of the performance of a
+ Lustre file system. Interpreting these statistics correctly can help you
+ diagnose problems with quotas, and may indicate adjustments to improve
+ system performance.</para>
+ <para>For example, if you run this command on the OSTs:</para>
+ <screen>
+lctl get_param lquota.testfs-OST0000.stats