LUDOC-100 monitoring: jobstats manual

Update manual for jobstats.

Signed-off-by: Niu Yawei <niu@whamcloud.com>
Change-Id: Icfd85f788c1657b0601d06f2890db03068ea5c06
Reviewed-on: http://review.whamcloud.com/4500
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Richard Henwood <richard.henwood@intel.com>

diff --git a/LustreMonitoring.xml b/LustreMonitoring.xml
index 912c1c8..54c03b0 100644 (file)
--- a/LustreMonitoring.xml
+++ b/LustreMonitoring.xml
@@ -7,6 +7,9 @@
       <para><xref linkend="dbdoclet.50438273_18711"/>Lustre Changelogs</para>
     </listitem>
     <listitem>
+      <para><xref linkend="dbdoclet.jobstats"/>Lustre Jobstats</para>
+    </listitem>
+    <listitem>
       <para><xref linkend="dbdoclet.50438273_81684"/>Lustre Monitoring Tool</para>
     </listitem>
     <listitem>
@@ -322,6 +325,178 @@ $ ln /mnt/lustre/mydir/foo/file /mnt/lustre/mydir/myhardlink
       </section>
     </section>
   </section>
+  <section xml:id="dbdoclet.jobstats">
+      <title><indexterm><primary>jobstats</primary><see>monitoring</see></indexterm>
+<indexterm><primary>monitoring</primary></indexterm>
+<indexterm><primary>monitoring</primary><secondary>jobstats</secondary></indexterm>
+
+Lustre Jobstats</title>
+    <para>The Lustre Jobstats feature is available starting in Lustre version 2.3. It collects filesystem operation statistics for the jobs running on Lustre clients, and exposes them via procfs on the server. Job schedulers known to be able to work with jobstats include: SLURM, SGE, LSF, Loadleveler, PBS and Maui/MOAB.</para>
+    <para>Since Jobstats is implemented in a scheduler-agnostic manner, it is likely that it will be able to work with other schedulers also.</para>
+    <section remap="h3">
+      <title><indexterm><primary>monitoring</primary><secondary>jobstats</secondary></indexterm>
+Enable/Disable Jobstats</title>
+      <para>Jobstats are disabled by default, the current state of jobstats can be verified by checking <literal>lctl get_param jobid_var</literal> on client:</para>
+      <screen>
+$ lctl get_param jobid_var
+jobid_var=disable
+      </screen>
+      <para>The Lustre Jobstats code extracts the job identifier from an environment variable set by the scheduler when the job is started. To enable jobstats, specify the <literal>jobid_var</literal> to name the environment variable set by the scheduler. For example, SLURM sets the <literal>SLURM_JOB_ID</literal> environment variable with the unique job ID on each client. To permanently enable Jobstats on the <literal>testfs</literal> filesystem:</para>
+      <screen>$ lctl conf_param testfs.sys.jobid_var=SLURM_JOB_ID</screen>
+      <para>The value of <literal>jobid_var</literal> can be:</para>
+    <informaltable frame="all">
+      <tgroup cols="2">
+        <colspec colname="c1" colwidth="50*"/>
+        <colspec colname="c2" colwidth="50*"/>
+        <thead>
+          <row>
+            <entry>
+              <para><emphasis role="bold">Value</emphasis></para>
+            </entry>
+            <entry>
+              <para><emphasis role="bold">Job Scheduler</emphasis></para>
+            </entry>
+          </row>
+        </thead>
+        <tbody>
+          <row>
+            <entry>
+              <para>SLURM_JOB_ID</para>
+            </entry>
+            <entry>
+              <para>Simple Linux Utility for Resource Management (SLURM)</para>
+            </entry>
+          </row>
+          <row>
+            <entry>
+              <para>JOB_ID</para>
+            </entry>
+            <entry>
+              <para>Sun Grid Engine (SGE)</para>
+            </entry>
+          </row>
+          <row>
+            <entry>
+              <para>LSB_JOBID</para>
+            </entry>
+            <entry>
+              <para>Load Sharing Facility (LSF)</para>
+            </entry>
+          </row>
+          <row>
+            <entry>
+              <para>LOADL_STEP_ID</para>
+            </entry>
+            <entry>
+              <para>Loadleveler</para>
+            </entry>
+          </row>
+          <row>
+            <entry>
+              <para>PBS_JOBID</para>
+            </entry>
+            <entry>
+              <para>Portable Batch Scheduler (PBS)/MAUI</para>
+            </entry>
+          </row>
+          <row>
+            <entry>
+              <para>procname_uid</para>
+            </entry>
+            <entry>
+              <para>process name and user ID (for debugging, or if no job scheduler is in use)</para>
+            </entry>
+          </row>
+          <row>
+            <entry>
+              <para>disable</para>
+            </entry>
+            <entry>
+              <para>disable jobstats</para>
+            </entry>
+          </row>
+        </tbody>
+      </tgroup>
+    </informaltable>
+    <para>To disable jobstats specify the <literal>jobid_var</literal> as <literal>disable</literal>:</para>
+    <screen>$ lctl conf_param testfs.sys.jobid_var=disable</screen>
+    </section>
+    <section remap="h3">
+      <title><indexterm><primary>monitoring</primary><secondary>jobstats</secondary></indexterm>
+Check Job Stats</title>
+    <para>Metadata operation statistics are collected on MDTs. These statistics can be accessed for all filesystems and all jobs on the MDT via the <literal>lctl get_param mdt.*.job_stats</literal>.  For example, clients running with <literal>jobid_var=procname_uid</literal>:</para>
+    <screen>
+$ lctl get_param mdt.*.job_stats
+job_stats:
+- job_id:          bash.0
+  snapshot_time:   1352084992
+  open:            { samples:           2, unit:  reqs }
+  close:           { samples:           2, unit:  reqs }
+  mknod:           { samples:           0, unit:  reqs }
+  link:            { samples:           0, unit:  reqs }
+  unlink:          { samples:           0, unit:  reqs }
+  mkdir:           { samples:           0, unit:  reqs }
+  rmdir:           { samples:           0, unit:  reqs }
+  rename:          { samples:           0, unit:  reqs }
+  getattr:         { samples:           3, unit:  reqs }
+  setattr:         { samples:           0, unit:  reqs }
+  getxattr:        { samples:           0, unit:  reqs }
+  setxattr:        { samples:           0, unit:  reqs }
+  statfs:          { samples:           0, unit:  reqs }
+  sync:            { samples:           0, unit:  reqs }
+  samedir_rename:  { samples:           0, unit:  reqs }
+  crossdir_rename: { samples:           0, unit:  reqs }
+- job_id:          dd.0
+  snapshot_time:   1352085037
+  open:            { samples:           1, unit:  reqs }
+  close:           { samples:           1, unit:  reqs }
+  mknod:           { samples:           0, unit:  reqs }
+  link:            { samples:           0, unit:  reqs }
+  unlink:          { samples:           0, unit:  reqs }
+  mkdir:           { samples:           0, unit:  reqs }
+  rmdir:           { samples:           0, unit:  reqs }
+  rename:          { samples:           0, unit:  reqs }
+  getattr:         { samples:           0, unit:  reqs }
+  setattr:         { samples:           0, unit:  reqs }
+  getxattr:        { samples:           0, unit:  reqs }
+  setxattr:        { samples:           0, unit:  reqs }
+  statfs:          { samples:           0, unit:  reqs }
+  sync:            { samples:           2, unit:  reqs }
+  samedir_rename:  { samples:           0, unit:  reqs }
+  crossdir_rename: { samples:           0, unit:  reqs }
+    </screen>
+    <para>Data operation statistics are collected on OSTs. Data operations statistics can be accessed via <literal>lctl get_param obdfilter.*.job_stats</literal>, for example:</para>
+    <screen>
+$ lctl get_param obdfilter.*.job_stats
+job_stats:
+- job_id:          bash.0
+  snapshot_time:   1352085025
+  read:            { samples:           0, unit: bytes, min:       0, max:       0, sum:               0 }
+  write:           { samples:           1, unit: bytes, min:       4, max:       4, sum:               4 }
+  setattr:         { samples:           0, unit:  reqs }
+  punch:           { samples:           0, unit:  reqs }
+  sync:            { samples:           0, unit:  reqs }
+    </screen>
+    </section>
+    <section remap="h3">
+      <title><indexterm><primary>monitoring</primary><secondary>jobstats</secondary></indexterm>
+Clear Job Stats</title>
+    <para>Accumulated job statistics can be reset by writing proc file <literal>job_stats</literal>.</para>
+    <para>Clear statistics for all jobs on the local node:</para>
+    <screen>$ lctl set_param obdfilter.*.job_stats=clear</screen>
+    <para>Clear statistics for job 'dd.0' on lustre-MDT0000:</para>
+    <screen>$ lctl set_param mdt.lustre-MDT0000.job_stats=clear</screen>
+    </section>
+    <section remap="h3">
+      <title><indexterm><primary>monitoring</primary><secondary>jobstats</secondary></indexterm>
+Configure Auto-cleanup Interval</title>
+    <para>By default, if a job is inactive for 600 seconds (10 minutes) statistics for this job will be dropped. This expiration value can be changed temporarily via:</para>
+    <screen>$ lctl set_param *.*.job_cleanup_interval={max_age}</screen>
+    <para>It can also be changed permanently, for example to 700 seconds via:</para>
+    <screen>$ lctl conf_param testfs.mdt.job_cleanup_interval=700</screen>
+    <para>The <literal>job_cleanup_interval</literal> can be set as 0 to disable the auto-cleanup. Note that if auto-cleanup of Jobstats is disabled, then all statistics will be kept in memory forever, which may eventually consume all memory on the servers. In this case, any monitoring tool should explicitly clear individual job statistics as they are processed, as shown above.</para>
+    </section>
+  </section>
   <section xml:id="dbdoclet.50438273_81684">
     <title><indexterm><primary>monitoring</primary><secondary>Lustre Monitoring Tool</secondary></indexterm>
 Lustre Monitoring Tool</title>