-FIGURES = figures/mkdir1.png
+FIGURES = figures/ost-setattr-generic.png \
+ figures/ost-punch-generic.png \
+ figures/mds-reint-setattr-generic.png \
+ figures/mds-getxattr-generic.png \
+ figures/ldlm-enqueue-generic.png \
+ figures/ldlm-enqueue-intent-layout-generic.png \
+ figures/ldlm-bl-callback-generic.png \
+ figures/ldlm-cp-callback-generic.png \
+ figures/ldlm-cancel-generic.png \
+ figures/chmod_rpcs.png \
+ figures/mds-reint-setattr-request.png \
+ figures/mds-reint-setattr-reply.png \
+ figures/touch_rpcs.png \
+ figures/ldlm-enqueue-intent-layout-request.png \
+ figures/ldlm-enqueue-intent-layout-reply.png \
+ figures/mds-getxattr-request.png \
+ figures/mds-getxattr-reply.png \
+ figures/ost-setattr-request.png \
+ figures/ost-setattr-reply.png \
+ figures/truncate_rpcs.png \
+ figures/ldlm-enqueue-extent-lvb-reply.png \
+ figures/ldlm-bl-callback-request.png \
+ figures/ldlm-bl-callback-reply.png \
+ figures/ldlm-cp-callback-request.png \
+ figures/ldlm-cp-callback-reply.png \
+ figures/ldlm-cancel-request.png \
+ figures/ldlm-cancel-reply.png \
+ figures/ost-punch-request.png \
+ figures/ost-punch-reply.png
+
TEXT = protocol.txt \
introduction.txt \
data_types.txt \
+ lustre_file_ids.txt \
+ lustre_handle.txt \
+ mdt_structs.txt \
+ mdt_body.txt \
+ mds_reint_structs.txt \
+ mdt_rec_reint.txt \
+ mdt_rec_setattr.txt \
+ ost_setattr_structs.txt \
connection.txt \
timeouts.txt \
file_id.txt \
ldlm.txt \
+ layout_intent.txt \
+ ldlm_resource_id.txt \
+ ldlm_intent.txt \
+ ldlm_resource_desc.txt \
+ ldlm_lock_desc.txt \
+ ldlm_request.txt \
+ ldlm_reply.txt \
+ ost_lvb.txt \
+ early_lock_cancellation.txt \
llog.txt \
recovery.txt \
security.txt \
lustre_messages.txt \
lustre_operations.txt \
+ ost_setattr.txt \
+ ost_punch.txt \
+ mds_reint.txt \
+ mds_getxattr.txt \
+ ldlm_enqueue.txt \
+ ldlm_bl_callback.txt \
+ ldlm_cp_callback.txt \
+ ldlm_cancel.txt \
file_system_operations.txt \
+ setattr.txt \
glossary.txt
-.SUFFIXES : .gnuplot .gv .pdf .png
+.SUFFIXES : .gnuplot .gv .pdf .png .fig
.PHONY: all
all: protocol.html protocol.pdf
.gv.png:
dot -Tpng $< -o $@
+.fig.png:
+ fig2dev -L png $< $@
+
.PHONY : clean
clean:
rm -f *.xml *.html *.pdf figures/*.png
requests that do not modify the file system the 'pb_transno' field in
the 'ptlrpc_body' is just set to 0.
-Structured Data Types
-~~~~~~~~~~~~~~~~~~~~~
+Miscellaneous Structured Data Types
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Extended Attributes
^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^
A 'lustre_capa' structure conveys details about the capabilities
-supported (or requested) between a client and a given target.
+supported (or requested) between a client and a given target. I am
+told that this is deprecated in later version of Lustre.
----
#define CAPA_HMAC_MAX_LEN 64
}
----
-MDT Data
-^^^^^^^^
+include::lustre_file_ids.txt[]
-An 'mdt_body' structure holds details about a given MDT.
-
-----
-struct mdt_body {
- struct lu_fid fid1;
- struct lu_fid fid2;
- struct lustre_handle handle;
- __u64 valid;
- __u64 size;
- obd_time mtime;
- obd_time atime;
- obd_time ctime;
- __u64 blocks;
- __u64 ioepoch;
- __u64 t_state;
- __u32 fsuid;
- __u32 fsgid;
- __u32 capability;
- __u32 mode;
- __u32 uid;
- __u32 gid;
- __u32 flags;
- __u32 rdev;
- __u32 nlink;
- __u32 unused2;
- __u32 suppgid;
- __u32 eadatasize;
- __u32 aclsize;
- __u32 max_mdsize;
- __u32 max_cookiesize;
- __u32 uid_h;
- __u32 gid_h;
- __u32 padding_5;
- __u64 padding_6;
- __u64 padding_7;
- __u64 padding_8;
- __u64 padding_9;
- __u64 padding_10;
-}; /* 216 */
-----
MGS Configuration Reference
^^^^^^^^^^^^^^^^^^^^^^^^^^^
The 'mgs_config_res' structure returns information about the Lustre
file system.
-Lustre Handle
-^^^^^^^^^^^^^
-
-----
-struct lustre_handle {
- __u64 cookie;
-};
-----
-
-A Lustre handle is a reference to an import or an export. Those
-objects maintain state about the connection between a given client
-and a given target. The import is on the client and the corresponding
-export is on the server.
+include::lustre_handle.txt[]
Lustre Message Header
^^^^^^^^^^^^^^^^^^^^^
};
----
+include::mdt_structs.txt[]
+
+include::mds_reint_structs.txt[]
+
+include::ost_setattr_structs.txt[]
--- /dev/null
+Early Lock Cancellation
+-----------------------
+[[early-lock-cancellation]]
+
+In some circumstances (see <<mds-reint-setattr-rpc>>) a client holding
+a lock "knows" that the operation it is initiating will necessitate
+canceling a lock that it holds. In such a circumstance the lock is
+said to be "in conflict" with the operation. For example, a side
+effect of the GNU 'fileutils' is that it always issues a 'stat' prior
+to other update operations, so when issuing an update to a resource
+attribute on the MDT a) the client will already have a read lock on
+the resource, and b) the MDT will then need a lock on that same
+resource in order to perform the update. Rather than waiting for an
+additional round of RPCs to carry out the lock cancellation (on the
+client) the RPC requesting the update can include a cancellation of
+the lock as part of its request. This proactive lock cancellation is
+called "early lock cancellation" or sometimes ELC.
+
--- /dev/null
+Lustre Protocol Documentation - setattr
+=======================================
+Andrew Uselton <andrew.c.uselton@intel.com>
+v0.0, May 2015
+:author: Andrew Uselton
+:doctype: article
+:Author Initials: ACU
+:toc:
+:icons:
+:numbered:
+:imagesdir: ./figures
+:website: http://lustre.org/
+:keywords: PtlRPC, Lustre, Protocol
+
+
+include::setattr.txt[]
+
+:numbered!:
+
+[appendix]
+License
+-------
+
+Copyright (C) Intel 2015
+
+This work is licensed under a Creative Commons Attribution-ShareAlike
+4.0 International License (CC BY-SA 4.0). See
+<https://creativecommons.org/licenses/by-sa/4.0/> for more detail.
+
--- /dev/null
+Lustre Protocol Documentation - setattr
+=======================================
+Andrew Uselton <andrew.c.uselton@intel.com>
+v0.0, May 2015
+:author: Andrew Uselton
+:doctype: article
+:Author Initials: ACU
+:toc:
+:icons:
+:numbered:
+:imagesdir: ./figures
+:website: http://lustre.org/
+:keywords: PtlRPC, Lustre, Protocol
+
+include::setattr_terms.txt[]
+
+File System Operations
+----------------------
+
+include::setattr.txt[]
+
+//////////////////////////////////////////////////////////////////////
+This is back up one level (starting with ----):
+//////////////////////////////////////////////////////////////////////
+
+include::early_lock_cancellation.txt[]
+
+RPCs
+----
+[[rpcs]]
+
+include::ost_setattr.txt[]
+
+include::ost_punch.txt[]
+
+include::mds_reint.txt[]
+
+include::mds_getxattr.txt[]
+
+include::ldlm_enqueue.txt[]
+
+include::ldlm_bl_callback.txt[]
+
+include::ldlm_cp_callback.txt[]
+
+Data Structures and Defines
+---------------------------
+[[data-structs]]
+
+Note that 'lustre_msg' and 'ptlrpc_body' are described in the main
+protocol document. Those details are not repeated in this brief
+extract. Only those structures that are particular to the 'setattr'
+activity are included here. This extract will be incorporated back in
+the protocol document.
+
+Miscellaneous Structured Data Types
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+include::lustre_file_ids.txt[]
+
+include::lustre_handle.txt[]
+
+LDLM Structures
+~~~~~~~~~~~~~~~
+
+include::layout_intent.txt[]
+
+include::ldlm_resource_id.txt[]
+
+include::ldlm_intent.txt[]
+
+include::ldlm_resource_desc.txt[]
+
+include::ldlm_lock_desc.txt[]
+
+include::ldlm_request.txt[]
+
+include::ldlm_reply.txt[]
+
+include::ost_lvb.txt[]
+
+//////////////////////////////////////////////////////////////////////
+These are back up one level (starting with ~~~~):
+//////////////////////////////////////////////////////////////////////
+
+include::mds_reint_structs.txt[]
+
+include::ost_setattr_structs.txt[]
+
+:numbered!:
+
+[appendix]
+License
+-------
+
+Copyright (C) Intel 2015
+
+This work is licensed under a Creative Commons Attribution-ShareAlike
+4.0 International License (CC BY-SA 4.0). See
+<https://creativecommons.org/licenses/by-sa/4.0/> for more detail.
+
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+6 1350 1275 1800 1725
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 1561 1514 181 181 1561 1514 1636 1679
+4 0 0 50 -1 0 12 0.0000 4 150 105 1531 1574 1\001
+-6
+6 4200 1725 4650 2175
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 4411 1936 181 181 4411 1936 4486 2101
+4 0 0 50 -1 0 12 0.0000 4 150 105 4381 2011 2\001
+-6
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
+ 1200 1200 1200 1350
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 150 1575 1125 1575
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 600 600 1800 600 1800 1200 600 1200 600 600
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 1275 1725 4725 1725
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 4215 600 5415 600 5415 1200 4215 1200 4215 600
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
+ 4815 1185 4815 1335
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 7770 615 8970 615 8970 1215 7770 1215 7770 615
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
+ 8370 1215 8370 1365
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 4740 2100 1290 2100
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 4740 1335 4875 1335 4875 2700 4740 2700 4740 1335
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 1125 1350 1275 1350 1275 2700 1125 2700 1125 1350
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 8295 1365 8475 1365 8475 2700 8295 2700 8295 1365
+4 0 0 50 -1 0 12 0.0000 4 150 615 885 960 Client1\001
+4 0 0 50 -1 0 12 0.0000 4 195 705 285 1470 setattr()\001
+4 0 0 50 -1 0 12 0.0000 4 150 480 4515 960 MDT\001
+4 0 0 50 -1 0 12 0.0000 4 150 405 8070 990 OST\001
+4 0 0 50 -1 0 12 0.0000 4 195 1905 1875 1650 MDS_REINT request\001
+4 0 0 50 -1 0 12 0.0000 4 195 1695 2400 2025 MDS_REINT reply\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3000 900 3000 975 3000 1050 3000 1125 3000 1200 3000 1275
+ 3000 1350 3000 1425 3000 1500
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 2025 900 4950 900 4950 1500 1200 1500 1200 1050
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 1815 1875 3120 1860 3120 2475 1215 2475 1215 2025
+4 0 0 50 -1 16 18 0.0000 4 255 930 1050 1005 request\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1350 1305 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 615 1140 1950 reply\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1365 2250 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 1620 3165 1320 ldlm_request\001
+4 0 0 50 -1 16 18 0.0000 4 270 3000 1050 675 LDLM_BL_CALLBACK\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+6 1050 780 3045 1530
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 1725 915 3030 900 3030 1515 1125 1515 1125 1065
+4 0 0 50 -1 16 18 0.0000 4 270 615 1050 990 reply\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1275 1290 ptlrpc_body\001
+-6
+4 0 0 50 -1 16 18 0.0000 4 270 3000 1050 675 LDLM_BL_CALLBACK\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3000 900 3000 975 3000 1050 3000 1125 3000 1200 3000 1275
+ 3000 1350 3000 1425 3000 1500
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 2025 900 4995 900 5010 1500 1200 1500 1200 1050
+4 0 0 50 -1 16 18 0.0000 4 255 930 1050 1005 request\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1350 1305 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 1620 3165 1320 ldlm_request\001
+4 0 0 50 -1 16 18 0.0000 4 270 3000 1050 675 LDLM_BL_CALLBACK\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3000 900 3000 975 3000 1050 3000 1125 3000 1200 3000 1275
+ 3000 1350 3000 1425 3000 1500
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 2025 900 4995 900 5010 1500 1200 1500 1200 1050
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 1815 1875 3225 1875 3225 2475 1215 2475 1215 2025
+4 0 0 50 -1 16 18 0.0000 4 255 930 1050 1005 request\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1350 1305 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 1620 3165 1320 ldlm_request\001
+4 0 0 50 -1 16 18 0.0000 4 270 2130 1050 675 LDLM_CANCEL\001
+4 0 0 50 -1 16 18 0.0000 4 270 615 1140 1950 reply\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1365 2250 ptlrpc_body\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+6 1050 825 3225 1650
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 1740 975 3150 975 3150 1575 1140 1575 1140 1125
+4 0 0 50 -1 16 18 0.0000 4 270 615 1065 1050 reply\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1290 1350 ptlrpc_body\001
+-6
+4 0 0 50 -1 16 18 0.0000 4 270 2130 1050 675 LDLM_CANCEL\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3000 900 3000 975 3000 1050 3000 1125 3000 1200 3000 1275
+ 3000 1350 3000 1425 3000 1500
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 2025 900 4995 900 5010 1500 1200 1500 1200 1050
+4 0 0 50 -1 16 18 0.0000 4 255 930 1050 1005 request\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1350 1305 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 1620 3165 1320 ldlm_request\001
+4 0 0 50 -1 16 18 0.0000 4 270 2130 1050 675 LDLM_CANCEL\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3000 900 3000 975 3000 1050 3000 1125 3000 1200 3000 1275
+ 3000 1350 3000 1425 3000 1500
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 2025 900 6150 900 6150 1500 1200 1500 1200 1050
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 1815 1875 3120 1860 3120 2475 1215 2475 1215 2025
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 4950 900 4950 975 4950 1050 4950 1125 4950 1200 4950 1275
+ 4950 1350 4950 1425 4950 1500
+4 0 0 50 -1 16 18 0.0000 4 255 930 1050 1005 request\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1350 1305 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 615 1140 1950 reply\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1365 2250 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 1620 3165 1320 ldlm_request\001
+4 0 0 50 -1 16 18 0.0000 4 270 930 5040 1320 ost_lvb\001
+4 0 0 50 -1 16 18 0.0000 4 270 3045 1050 675 LDLM_CP_CALLBACK\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+6 1050 780 3045 1530
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 1725 915 3030 900 3030 1515 1125 1515 1125 1065
+4 0 0 50 -1 16 18 0.0000 4 270 615 1050 990 reply\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1275 1290 ptlrpc_body\001
+-6
+4 0 0 50 -1 16 18 0.0000 4 270 3045 1050 675 LDLM_CP_CALLBACK\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3000 900 3000 975 3000 1050 3000 1125 3000 1200 3000 1275
+ 3000 1350 3000 1425 3000 1500
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 2025 900 6150 900 6150 1500 1200 1500 1200 1050
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 4950 900 4950 975 4950 1050 4950 1125 4950 1200 4950 1275
+ 4950 1350 4950 1425 4950 1500
+4 0 0 50 -1 16 18 0.0000 4 255 930 1050 1005 request\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1350 1305 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 1620 3165 1320 ldlm_request\001
+4 0 0 50 -1 16 18 0.0000 4 270 930 5040 1320 ost_lvb\001
+4 0 0 50 -1 16 18 0.0000 4 270 3045 1050 675 LDLM_CP_CALLBACK\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 2940 900 5940 900 5940 1500 1140 1515 1140 1065
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 2940 915 2940 990 2940 1065 2940 1140 2940 1215 2940 1290
+ 2940 1365 2940 1440 2940 1515
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 4515 900 4515 975 4515 1050 4515 1125 4515 1200 4515 1275
+ 4515 1350 4515 1425 4515 1500
+4 0 0 50 -1 16 18 0.0000 4 270 2385 1050 675 LDLM_ENQUEUE\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1290 1290 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 1305 3075 1320 ldlm_reply\001
+4 0 0 50 -1 16 18 0.0000 4 270 930 4650 1335 ost_lvb\001
+4 0 0 50 -1 16 18 0.0000 4 270 1920 1065 990 extent lvb reply\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+6 1140 1740 4605 2490
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 1815 1875 4590 1875 4590 2475 1215 2475 1215 2025
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3015 1875 3015 1950 3015 2025 3015 2100 3015 2175 3015 2250
+ 3015 2325 3015 2400 3015 2475
+4 0 0 50 -1 16 18 0.0000 4 270 615 1140 1950 reply\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1365 2250 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 1305 3150 2280 ldlm_reply\001
+-6
+6 1125 2775 6135 3525
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 2910 2880 6120 2865 6120 3495 1200 3510 1200 3060
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3000 2910 3000 2985 3000 3060 3000 3135 3000 3210 3000 3285
+ 3000 3360 3000 3435 3000 3510
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 4650 2910 4650 2985 4650 3060 4650 3135 4650 3210 4650 3285
+ 4650 3360 4650 3435 4650 3510
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1350 3285 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 1920 1125 2985 extent lvb reply\001
+4 0 0 50 -1 16 18 0.0000 4 270 1305 3135 3315 ldlm_reply\001
+4 0 0 50 -1 0 18 0.0000 4 255 810 4860 3315 ost_lvb\001
+-6
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3000 900 3000 975 3000 1050 3000 1125 3000 1200 3000 1275
+ 3000 1350 3000 1425 3000 1500
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 2025 900 4995 900 5010 1500 1200 1500 1200 1050
+4 0 0 50 -1 16 18 0.0000 4 255 930 1050 1005 request\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1350 1305 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 2385 1050 675 LDLM_ENQUEUE\001
+4 0 0 50 -1 16 18 0.0000 4 270 1620 3165 1320 ldlm_request\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3000 900 3000 975 3000 1050 3000 1125 3000 1200 3000 1275
+ 3000 1350 3000 1425 3000 1500
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 4890 915 4890 990 4890 1065 4890 1140 4890 1215 4890 1290
+ 4890 1365 4890 1440 4890 1515
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 8370 930 8370 1005 8370 1080 8370 1155 8370 1230 8370 1305
+ 8370 1380 8370 1455 8370 1530
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 3615 975 10365 960 10350 1575 1140 1575 1140 1125
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 6540 990 6540 1065 6540 1140 6540 1215 6540 1290 6540 1365
+ 6540 1440 6540 1515 6540 1590
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 3420 1905 6630 1890 6630 2490 1185 2505 1185 2055
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 2985 1905 2985 1980 2985 2055 2985 2130 2985 2205 2985 2280
+ 2985 2355 2985 2430 2985 2505
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 4560 1890 4560 1965 4560 2040 4560 2115 4560 2190 4560 2265
+ 4560 2340 4560 2415 4560 2490
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1350 1305 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 2385 1050 675 LDLM_ENQUEUE\001
+4 0 0 50 -1 16 18 0.0000 4 270 1620 3165 1320 ldlm_request\001
+4 0 0 50 -1 16 18 0.0000 4 270 2550 1050 1005 intent:layout request\001
+4 0 0 50 -1 16 18 0.0000 4 270 1380 4980 1335 ldlm_intent\001
+4 0 0 50 -1 16 18 0.0000 4 270 1635 6600 1335 layout_intent\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1335 2280 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 1305 3120 2310 ldlm_reply\001
+4 0 0 50 -1 16 18 0.0000 4 270 2235 1110 1980 intent:layout reply\001
+4 0 0 50 -1 16 18 0.0000 4 270 1650 4695 2295 lov_mds_md\001
+4 0 0 50 -1 16 18 0.0000 4 270 1650 8475 1365 lov_mds_md\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 3360 975 6525 975 6525 1575 1125 1575 1125 1125
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 2925 975 2925 1050 2925 1125 2925 1200 2925 1275 2925 1350
+ 2925 1425 2925 1500 2925 1575
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 4500 960 4500 1035 4500 1110 4500 1185 4500 1260 4500 1335
+ 4500 1410 4500 1485 4500 1560
+4 0 0 50 -1 16 18 0.0000 4 270 2385 1050 675 LDLM_ENQUEUE\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1275 1350 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 1305 3060 1380 ldlm_reply\001
+4 0 0 50 -1 16 18 0.0000 4 270 2235 1050 1050 intent:layout reply\001
+4 0 0 50 -1 16 18 0.0000 4 270 1650 4635 1365 lov_mds_md\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3000 900 3000 975 3000 1050 3000 1125 3000 1200 3000 1275
+ 3000 1350 3000 1425 3000 1500
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 4890 915 4890 990 4890 1065 4890 1140 4890 1215 4890 1290
+ 4890 1365 4890 1440 4890 1515
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 3615 975 8400 975 8400 1575 1140 1575 1140 1125
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 6540 990 6540 1065 6540 1140 6540 1215 6540 1290 6540 1365
+ 6540 1440 6540 1515 6540 1590
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1350 1305 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 2385 1050 675 LDLM_ENQUEUE\001
+4 0 0 50 -1 16 18 0.0000 4 270 1620 3165 1320 ldlm_request\001
+4 0 0 50 -1 16 18 0.0000 4 270 2550 1050 1005 intent:layout request\001
+4 0 0 50 -1 16 18 0.0000 4 270 1380 4980 1335 ldlm_intent\001
+4 0 0 50 -1 16 18 0.0000 4 270 1635 6600 1335 layout_intent\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+6 990 780 4455 1530
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 1665 915 4440 915 4440 1515 1065 1515 1065 1065
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 2865 915 2865 990 2865 1065 2865 1140 2865 1215 2865 1290
+ 2865 1365 2865 1440 2865 1515
+4 0 0 50 -1 16 18 0.0000 4 270 615 990 990 reply\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1215 1290 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 1305 3000 1320 ldlm_reply\001
+-6
+4 0 0 50 -1 16 18 0.0000 4 270 2385 1050 675 LDLM_ENQUEUE\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3000 900 3000 975 3000 1050 3000 1125 3000 1200 3000 1275
+ 3000 1350 3000 1425 3000 1500
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 2025 900 4995 900 5010 1500 1200 1500 1200 1050
+4 0 0 50 -1 16 18 0.0000 4 255 930 1050 1005 request\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1350 1305 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 2385 1050 675 LDLM_ENQUEUE\001
+4 0 0 50 -1 16 18 0.0000 4 270 1620 3165 1320 ldlm_request\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3000 900 3000 975 3000 1050 3000 1125 3000 1200 3000 1275
+ 3000 1350 3000 1425 3000 1500
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 1815 1875 5745 1860 5730 2460 1215 2475 1215 2025
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3015 1875 3015 1950 3015 2025 3015 2100 3015 2175 3015 2250
+ 3015 2325 3015 2400 3015 2475
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 4470 1890 4470 1965 4470 2040 4470 2115 4470 2190 4470 2265
+ 4470 2340 4470 2415 4470 2490
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 4545 915 4545 990 4545 1065 4545 1140 4545 1215 4545 1290
+ 4545 1365 4545 1440 4545 1515
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 2025 900 8385 885 8385 1485 1200 1500 1200 1050
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 6225 930 6225 1005 6225 1080 6225 1155 6225 1230 6225 1305
+ 6225 1380 6225 1455 6225 1530
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 7155 915 7155 990 7155 1065 7155 1140 7155 1215 7155 1290
+ 7155 1365 7155 1440 7155 1515
+4 0 0 50 -1 16 18 0.0000 4 255 930 1050 1005 request\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1350 1305 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 615 1140 1950 reply\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1365 2250 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 2370 1050 675 MDS_GETXATTR\001
+4 0 0 50 -1 16 18 0.0000 4 270 1305 3105 1320 mdt_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 1485 4680 1305 lustre_capa\001
+4 0 0 50 -1 16 18 0.0000 4 150 705 6345 1320 name\001
+4 0 0 50 -1 16 18 0.0000 4 210 900 7275 1320 eadata\001
+4 0 0 50 -1 16 18 0.0000 4 270 1305 3090 2265 mdt_body\001
+4 0 0 50 -1 16 18 0.0000 4 210 900 4620 2250 eadata\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+6 1050 780 5670 1530
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 1725 915 5655 900 5640 1500 1125 1515 1125 1065
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 2925 915 2925 990 2925 1065 2925 1140 2925 1215 2925 1290
+ 2925 1365 2925 1440 2925 1515
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 4380 930 4380 1005 4380 1080 4380 1155 4380 1230 4380 1305
+ 4380 1380 4380 1455 4380 1530
+4 0 0 50 -1 16 18 0.0000 4 270 615 1050 990 reply\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1275 1290 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 1305 3000 1305 mdt_body\001
+4 0 0 50 -1 16 18 0.0000 4 210 900 4530 1290 eadata\001
+-6
+4 0 0 50 -1 16 18 0.0000 4 270 2370 1050 675 MDS_GETXATTR\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3000 900 3000 975 3000 1050 3000 1125 3000 1200 3000 1275
+ 3000 1350 3000 1425 3000 1500
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 4545 915 4545 990 4545 1065 4545 1140 4545 1215 4545 1290
+ 4545 1365 4545 1440 4545 1515
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 2025 900 5505 900 5505 1485 1200 1500 1200 1050
+4 0 0 50 -1 16 18 0.0000 4 255 930 1050 1005 request\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1350 1305 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 2370 1050 675 MDS_GETXATTR\001
+4 0 0 50 -1 16 18 0.0000 4 270 1305 3105 1320 mdt_body\001
+4 0 0 50 -1 16 18 0.0000 4 150 705 4725 1320 name\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3000 900 3000 975 3000 1050 3000 1125 3000 1200 3000 1275
+ 3000 1350 3000 1425 3000 1500
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 5325 900 5325 975 5325 1050 5325 1125 5325 1200 5325 1275
+ 5325 1350 5325 1425 5325 1500
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 2025 900 7200 900 7200 1500 1200 1500 1200 1050
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 1875 2025 4650 2025 4650 2625 1275 2625 1275 2175
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3075 2025 3075 2100 3075 2175 3075 2250 3075 2325 3075 2400
+ 3075 2475 3075 2550 3075 2625
+4 0 0 50 -1 16 18 0.0000 4 270 4155 1050 675 MDS_REINT:REINT_SETATTR\001
+4 0 0 50 -1 16 18 0.0000 4 255 930 1050 1005 request\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1350 1305 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 2025 3150 1305 mdt_rec_setattr\001
+4 0 0 50 -1 16 18 0.0000 4 270 1620 5400 1320 ldlm_request\001
+4 0 0 50 -1 16 18 0.0000 4 270 615 1200 2100 reply\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1425 2400 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 1305 3225 2400 mdt_body\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3000 900 3000 975 3000 1050 3000 1125 3000 1200 3000 1275
+ 3000 1350 3000 1425 3000 1500
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 5325 900 5325 975 5325 1050 5325 1125 5325 1200 5325 1275
+ 5325 1350 5325 1425 5325 1500
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 2025 900 7200 900 7200 1500 1200 1500 1200 1050
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 1815 1875 4590 1875 4590 2475 1215 2475 1215 2025
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3015 1875 3015 1950 3015 2025 3015 2100 3015 2175 3015 2250
+ 3015 2325 3015 2400 3015 2475
+4 0 0 50 -1 16 18 0.0000 4 270 4155 1050 675 MDS_REINT:REINT_SETATTR\001
+4 0 0 50 -1 16 18 0.0000 4 255 930 1050 1005 request\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1350 1305 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 2025 3150 1305 mdt_rec_setattr\001
+4 0 0 50 -1 16 18 0.0000 4 270 1620 5400 1320 ldlm_request\001
+4 0 0 50 -1 16 18 0.0000 4 270 615 1140 1950 reply\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1365 2250 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 1305 3165 2250 mdt_body\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 1875 975 4650 975 4650 1575 1275 1575 1275 1125
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3075 975 3075 1050 3075 1125 3075 1200 3075 1275 3075 1350
+ 3075 1425 3075 1500 3075 1575
+4 0 0 50 -1 16 18 0.0000 4 270 4155 1050 675 MDS_REINT:REINT_SETATTR\001
+4 0 0 50 -1 16 18 0.0000 4 270 615 1200 1050 reply\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1425 1350 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 1305 3225 1350 mdt_body\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3000 900 3000 975 3000 1050 3000 1125 3000 1200 3000 1275
+ 3000 1350 3000 1425 3000 1500
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 5325 900 5325 975 5325 1050 5325 1125 5325 1200 5325 1275
+ 5325 1350 5325 1425 5325 1500
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 2025 900 7200 900 7200 1500 1200 1500 1200 1050
+4 0 0 50 -1 16 18 0.0000 4 270 4155 1050 675 MDS_REINT:REINT_SETATTR\001
+4 0 0 50 -1 16 18 0.0000 4 255 930 1050 1005 request\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1350 1305 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 2025 3150 1305 mdt_rec_setattr\001
+4 0 0 50 -1 16 18 0.0000 4 270 1620 5400 1320 ldlm_request\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3000 900 3000 975 3000 1050 3000 1125 3000 1200 3000 1275
+ 3000 1350 3000 1425 3000 1500
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 1815 1875 4425 1875 4425 2475 1215 2475 1215 2025
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3015 1875 3015 1950 3015 2025 3015 2100 3015 2175 3015 2250
+ 3015 2325 3015 2400 3015 2475
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 4545 915 4545 990 4545 1065 4545 1140 4545 1215 4545 1290
+ 4545 1365 4545 1440 4545 1515
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 2025 900 6300 900 6300 1500 1200 1500 1200 1050
+4 0 0 50 -1 16 18 0.0000 4 255 930 1050 1005 request\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1350 1305 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 615 1140 1950 reply\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1365 2250 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 1485 4680 1305 lustre_capa\001
+4 0 0 50 -1 16 18 0.0000 4 270 1800 1050 675 OST_PUNCH\001
+4 0 0 50 -1 16 18 0.0000 4 270 1200 3105 1320 ost_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 1200 3090 2265 ost_body\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+6 1050 750 4425 1575
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 1740 900 4350 900 4350 1500 1140 1500 1140 1050
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 2940 900 2940 975 2940 1050 2940 1125 2940 1200 2940 1275
+ 2940 1350 2940 1425 2940 1500
+4 0 0 50 -1 16 18 0.0000 4 270 615 1065 975 reply\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1290 1275 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 1200 3015 1290 ost_body\001
+-6
+4 0 0 50 -1 16 18 0.0000 4 270 1800 1050 675 OST_PUNCH\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3000 900 3000 975 3000 1050 3000 1125 3000 1200 3000 1275
+ 3000 1350 3000 1425 3000 1500
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 2025 900 4425 900 4425 1500 1200 1500 1200 1050
+4 0 0 50 -1 16 18 0.0000 4 255 930 1050 1005 request\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1350 1305 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 1800 1050 675 OST_PUNCH\001
+4 0 0 50 -1 16 18 0.0000 4 270 1200 3105 1320 ost_body\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3000 900 3000 975 3000 1050 3000 1125 3000 1200 3000 1275
+ 3000 1350 3000 1425 3000 1500
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 1815 1875 4425 1875 4425 2475 1215 2475 1215 2025
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3015 1875 3015 1950 3015 2025 3015 2100 3015 2175 3015 2250
+ 3015 2325 3015 2400 3015 2475
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 2025 900 4575 900 4575 1500 1200 1500 1200 1050
+4 0 0 50 -1 16 18 0.0000 4 255 930 1050 1005 request\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1350 1305 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 615 1140 1950 reply\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1365 2250 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 1200 3105 1320 ost_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 1200 3090 2265 ost_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 2100 1050 675 OST_SETATTR\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 1875 975 4650 975 4650 1575 1275 1575 1275 1125
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3075 975 3075 1050 3075 1125 3075 1200 3075 1275 3075 1350
+ 3075 1425 3075 1500 3075 1575
+4 0 0 50 -1 16 18 0.0000 4 270 615 1200 1050 reply\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1425 1350 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 2100 1050 675 OST_SETATTR\001
+4 0 0 50 -1 16 18 0.0000 4 270 1200 3225 1350 ost_body\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3000 900 3000 975 3000 1050 3000 1125 3000 1200 3000 1275
+ 3000 1350 3000 1425 3000 1500
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 2025 900 4350 900 4350 1500 1200 1500 1200 1050
+4 0 0 50 -1 16 18 0.0000 4 255 930 1050 1005 request\001
+4 0 0 50 -1 16 18 0.0000 4 270 1530 1350 1305 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 2100 1050 675 OST_SETATTR\001
+4 0 0 50 -1 16 18 0.0000 4 270 1200 3105 1320 ost_body\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 6044 4831 181 181 6044 4831 6119 4996
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 3706 4291 181 181 3706 4291 3781 4456
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 2281 3766 181 181 2281 3766 2356 3931
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 3976 3241 181 181 3976 3241 4051 3406
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 3959 2309 181 181 3959 2309 4034 2474
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 2221 2699 181 181 2221 2699 2296 2864
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 2686 1936 181 181 2686 1936 2761 2101
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 3511 1514 181 181 3511 1514 3586 1679
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
+ 1200 1200 1200 1350
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 150 1575 1125 1575
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 600 600 1800 600 1800 1200 600 1200 600 600
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 1275 1725 4725 1725
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 4725 2925 1275 2925
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 4215 600 5415 600 5415 1200 4215 1200 4215 600
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
+ 4815 1185 4815 1335
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 7770 615 8970 615 8970 1215 7770 1215 7770 615
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
+ 8370 1215 8370 1365
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 1275 3450 4725 3450
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 1275 4500 8325 4500
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 8250 5025 1275 5025
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 4740 2100 1290 2100
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 1290 2490 4740 2490
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 1125 1350 1275 1350 1275 5400 1125 5400 1125 1350
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 4740 1335 4875 1335 4875 5400 4740 5400 4740 1335
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 8295 1365 8475 1365 8475 5400 8295 5400 8295 1365
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 7 0 0 2
+ 4725 3975 1275 3975
+4 0 0 50 -1 0 12 0.0000 4 150 615 885 960 Client1\001
+4 0 0 50 -1 0 12 0.0000 4 195 705 285 1470 setattr()\001
+4 0 0 50 -1 0 12 0.0000 4 150 480 4515 960 MDT\001
+4 0 0 50 -1 0 12 0.0000 4 150 405 8070 990 OST\001
+4 0 0 50 -1 0 12 0.0000 4 195 1695 2865 2040 MDS_REINT reply\001
+4 0 0 50 -1 0 12 0.0000 4 195 1905 1380 1665 MDS_REINT request\001
+4 0 0 50 -1 0 12 0.0000 4 195 2430 1335 2415 LDLM_ENQUEUE request\001
+4 0 0 50 -1 0 12 0.0000 4 195 2220 2415 2835 LDLM_ENQUEUE reply\001
+4 0 0 50 -1 0 12 0.0000 4 195 2415 1365 3375 MDS_GETXATTR request\001
+4 0 0 50 -1 0 12 0.0000 4 195 2205 2460 3855 MDS_GETXATTR reply\001
+4 0 0 50 -1 0 12 0.0000 4 195 2160 1350 4425 OST_SETATTR request\001
+4 0 0 50 -1 0 12 0.0000 4 195 1950 6255 4920 OST_SETATTR reply\001
+4 0 0 50 -1 0 12 0.0000 4 150 105 5999 4921 8\001
+4 0 0 50 -1 0 12 0.0000 4 150 105 3661 4381 7\001
+4 0 0 50 -1 0 12 0.0000 4 150 105 2251 3841 6\001
+4 0 0 50 -1 0 12 0.0000 4 150 105 3931 3316 5\001
+4 0 0 50 -1 0 12 0.0000 4 150 105 3914 2384 3\001
+4 0 0 50 -1 0 12 0.0000 4 150 105 2176 2774 4\001
+4 0 0 50 -1 0 12 0.0000 4 150 105 2656 2011 2\001
+4 0 0 50 -1 0 12 0.0000 4 150 105 3481 1574 1\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+6 7725 3525 8100 3975
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 7906 3766 181 181 7906 3766 7981 3931
+4 0 0 50 -1 0 12 0.0000 4 150 105 7876 3841 6\001
+-6
+6 1350 3000 1800 3450
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 1576 3241 181 181 1576 3241 1651 3406
+4 0 0 50 -1 0 12 0.0000 4 150 105 1531 3316 5\001
+-6
+6 1350 1275 1800 1725
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 1561 1514 181 181 1561 1514 1636 1679
+4 0 0 50 -1 0 12 0.0000 4 150 105 1531 1574 1\001
+-6
+6 4275 1725 4725 2175
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 4486 1936 181 181 4486 1936 4561 2101
+4 0 0 50 -1 0 12 0.0000 4 150 105 4456 2011 2\001
+-6
+6 1275 2100 1725 2550
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 1484 2309 181 181 1484 2309 1559 2474
+4 0 0 50 -1 0 12 0.0000 4 150 105 1439 2384 3\001
+-6
+6 4200 2475 4650 2925
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 4396 2699 181 181 4396 2699 4471 2864
+4 0 0 50 -1 0 12 0.0000 4 150 105 4351 2774 4\001
+-6
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
+ 1200 1200 1200 1350
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 150 1575 1125 1575
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 600 600 1800 600 1800 1200 600 1200 600 600
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 1275 1725 4725 1725
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 4725 2925 1275 2925
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 4215 600 5415 600 5415 1200 4215 1200 4215 600
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
+ 4815 1185 4815 1335
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 7770 615 8970 615 8970 1215 7770 1215 7770 615
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
+ 8370 1215 8370 1365
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 1275 3450 8325 3450
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 8250 4050 1275 4050
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 1290 2475 4740 2475
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 4740 2100 1290 2100
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 1125 1350 1275 1350 1275 4500 1125 4500 1125 1350
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 4740 1335 4875 1335 4875 4500 4740 4500 4740 1335
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 8295 1365 8475 1365 8475 4500 8295 4500 8295 1365
+4 0 0 50 -1 0 12 0.0000 4 150 615 885 960 Client1\001
+4 0 0 50 -1 0 12 0.0000 4 195 705 285 1470 setattr()\001
+4 0 0 50 -1 0 12 0.0000 4 150 480 4515 960 MDT\001
+4 0 0 50 -1 0 12 0.0000 4 150 405 8070 990 OST\001
+4 0 0 50 -1 0 12 0.0000 4 195 1950 5700 3975 OST_SETATTR reply\001
+4 0 0 50 -1 0 12 0.0000 4 195 2160 1950 3375 OST_SETATTR request\001
+4 0 0 50 -1 0 12 0.0000 4 195 1905 1875 1650 MDS_REINT request\001
+4 0 0 50 -1 0 12 0.0000 4 195 1695 2550 2025 MDS_REINT reply\001
+4 0 0 50 -1 0 12 0.0000 4 195 2430 1725 2400 LDLM_ENQUEUE request\001
+4 0 0 50 -1 0 12 0.0000 4 195 2220 1950 2850 LDLM_ENQUEUE reply\001
--- /dev/null
+#FIG 3.2 Produced by xfig version 3.2.5b
+Landscape
+Center
+Inches
+Letter
+100.00
+Single
+-2
+1200 2
+6 8025 1800 8475 2250
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 8235 2055 181 181 8235 2055 8310 2220
+4 0 0 50 -1 0 12 0.0000 4 150 105 8191 2099 4\001
+-6
+6 9600 3225 10050 3675
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 9825 3450 181 181 9825 3450 9900 3615
+4 0 0 50 -1 0 12 0.0000 4 150 105 9750 3525 5\001
+-6
+6 7200 4800 7650 5250
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 7410 4995 181 181 7410 4995 7485 5160
+4 0 0 50 -1 0 12 0.0000 4 150 105 7364 5056 8\001
+-6
+6 1350 5325 1800 5775
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 1575 5550 181 181 1575 5550 1650 5715
+4 0 0 50 -1 0 12 0.0000 4 150 105 1500 5625 9\001
+-6
+6 1350 5850 1800 6300
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 1590 6105 181 181 1590 6105 1665 6270
+4 0 0 50 -1 0 12 0.0000 4 150 210 1485 6195 10\001
+-6
+6 7875 6000 8325 6450
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 8115 6255 181 181 8115 6255 8190 6420
+4 0 0 50 -1 0 12 0.0000 4 150 210 8010 6345 11\001
+-6
+6 10200 4425 10650 4875
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 10395 4635 181 181 10395 4635 10470 4800
+4 0 0 50 -1 0 12 0.0000 4 150 105 10366 4741 7\001
+-6
+6 7200 3000 7650 3450
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 7410 3225 181 181 7410 3225 7485 3390
+4 0 0 50 -1 0 12 0.0000 4 150 105 7366 3301 6\001
+-6
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 2686 1936 181 181 2686 1936 2761 2101
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 3511 1514 181 181 3511 1514 3586 1679
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 3960 2295 181 181 3960 2295 4035 2460
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 7290 7080 181 181 7290 7080 7365 7245
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
+ 1200 1200 1200 1350
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 150 1575 1125 1575
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 600 600 1800 600 1800 1200 600 1200 600 600
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 1275 1725 4725 1725
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 4215 600 5415 600 5415 1200 4215 1200 4215 600
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
+ 4815 1185 4815 1335
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 4740 2100 1290 2100
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 10275 660 11475 660 11475 1260 10275 1260 10275 660
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
+ 10815 1290 10815 1440
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 2
+ 7815 1215 7815 1365
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 7215 585 8415 585 8415 1185 7215 1185 7215 585
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 1305 2445 7710 2445
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 7950 2610 10784 2614
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 10770 3000 7920 3000
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 4740 1335 4875 1335 4875 7500 4740 7500 4740 1335
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 7725 1365 7905 1365 7905 7500 7725 7500 7725 1365
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 10770 1455 10950 1455 10950 7500 10770 7500 10770 1455
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 1290 5730 7740 5730
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 1290 6345 7710 6345
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 7725 6835 1290 6840
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 1125 1350 1290 1350 1290 7500 1125 7500 1125 1350
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 7725 5250 1290 5250
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 7875 6524 10709 6528
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 7710 3825 1275 3825
+2 1 0 1 0 0 50 -1 41 0.000 0 0 -1 1 0 2
+ 0 0 1.00 60.00 120.00
+ 10770 4125 7920 4125
+4 0 0 50 -1 0 12 0.0000 4 195 705 285 1470 setattr()\001
+4 0 0 50 -1 0 12 0.0000 4 150 480 4515 960 MDT\001
+4 0 0 50 -1 0 12 0.0000 4 195 1695 2865 2040 MDS_REINT reply\001
+4 0 0 50 -1 0 12 0.0000 4 195 1905 1380 1665 MDS_REINT request\001
+4 0 0 50 -1 0 12 0.0000 4 195 2430 1335 2415 LDLM_ENQUEUE request\001
+4 0 0 50 -1 0 12 0.0000 4 150 105 3914 2384 3\001
+4 0 0 50 -1 0 12 0.0000 4 150 105 2656 2011 2\001
+4 0 0 50 -1 0 12 0.0000 4 150 105 3481 1574 1\001
+4 0 0 50 -1 0 12 0.0000 4 150 615 885 960 Client1\001
+4 0 0 50 -1 0 12 0.0000 4 150 615 10560 1035 Client2\001
+4 0 0 50 -1 0 12 0.0000 4 150 405 7605 1005 OST\001
+4 0 0 50 -1 0 12 0.0000 4 195 2280 8025 2475 LDLM_BL_CALLBACK\001
+4 0 0 50 -1 0 12 0.0000 4 165 645 8175 2850 request\001
+4 0 0 50 -1 0 12 0.0000 4 195 435 10050 3525 reply\001
+4 0 0 50 -1 0 12 0.0000 4 195 2280 8475 3300 LDLM_BL_CALLBACK\001
+4 0 0 50 -1 0 12 0.0000 4 195 2280 1785 5670 LDLM_CP_CALLBACK\001
+4 0 0 50 -1 0 12 0.0000 4 195 2280 4950 5145 LDLM_CP_CALLBACK\001
+4 0 0 50 -1 0 12 0.0000 4 165 645 6480 5430 request\001
+4 0 0 50 -1 0 12 0.0000 4 195 435 1800 5940 reply\001
+4 0 0 50 -1 0 12 0.0000 4 195 1245 1785 6255 OST_PUNCH\001
+4 0 0 50 -1 0 12 0.0000 4 165 645 3135 6255 request\001
+4 0 0 50 -1 0 12 0.0000 4 195 1245 5685 6765 OST_PUNCH\001
+4 0 0 50 -1 0 12 0.0000 4 195 435 7050 6750 reply\001
+4 0 0 50 -1 0 12 0.0000 4 150 210 7185 7170 12\001
+4 0 0 50 -1 0 12 0.0000 4 195 2100 8325 6450 LDLM_CANCEL reply\001
+4 0 0 50 -1 0 12 0.0000 4 195 2220 5475 3675 LDLM_ENQUEUE reply\001
+4 0 0 50 -1 0 12 0.0000 4 195 2310 8325 4350 LDLM_CANCEL request\001
~~~~~~
Further discussion of the 'creat()' system call.
+
+include::setattr.txt[]
--- /dev/null
+Layout Intent
+^^^^^^^^^^^^^
+[[struct-layout-intent]]
+
+An LDLM_ENQUEUE RPC with a layout intent uses the 'layout_intent'
+structure to specify the desired use for the layout.
+
+----
+struct layout_intent {
+ __u32 li_opc;
+ __u32 li_flags;
+ __u64 li_start;
+ __u64 li_end;
+};
+----
+
+The uses for the layout that the 'li_opc' field can specify are:
+
+----
+enum {
+ LAYOUT_INTENT_ACCESS = 0,
+ LAYOUT_INTENT_READ = 1,
+ LAYOUT_INTENT_WRITE = 2,
+ LAYOUT_INTENT_GLIMPSE = 3,
+ LAYOUT_INTENT_TRUNC = 4,
+ LAYOUT_INTENT_RELEASE = 5,
+ LAYOUT_INTENT_RESTORE = 6
+};
+----
+
+The other fields - li_flags, li_start, adn li_end - are reserved for
+future use, but do not currently play arole.
-----------------------------------
[[ldlm]]
-The discussion of the LDLM is deferred for now. We'll get into it soon
-enough.
+Details about the operation of the LDLM will be introduced as they
+become relevant to the discussion of various file system operations.
LDLM Structures
~~~~~~~~~~~~~~~
-Lock Modes
-^^^^^^^^^^
+include::layout_intent.txt[]
-----
-typedef enum {
- LCK_MINMODE = 0,
- LCK_EX = 1,
- LCK_PW = 2,
- LCK_PR = 4,
- LCK_CW = 8,
- LCK_CR = 16,
- LCK_NL = 32,
- LCK_GROUP = 64,
- LCK_COS = 128,
- LCK_MAXMODE
-} ldlm_mode_t;
-----
+include::ldlm_resource_id.txt[]
-LDLM Extent
-^^^^^^^^^^^
+include::ldlm_intent.txt[]
-----
-struct ldlm_extent {
- __u64 start;
- __u64 end;
- __u64 gid;
-};
-----
+include::ldlm_resource_desc.txt[]
-LDLM Flock Wire
-^^^^^^^^^^^^^^^
+include::ldlm_lock_desc.txt[]
-----
-struct ldlm_flock_wire {
- __u64 lfw_start;
- __u64 lfw_end;
- __u64 lfw_owner;
- __u32 lfw_padding;
- __u32 lfw_pid;
-};
-----
+include::ldlm_request.txt[]
-LDLM Inode Bits
-^^^^^^^^^^^^^^^
+include::ldlm_reply.txt[]
-----
-struct ldlm_inodebits {
- __u64 bits;
-};
-----
+include::ost_lvb.txt[]
-LDLM Wire Policy Data
-^^^^^^^^^^^^^^^^^^^^^
-
-----
-typedef union {
- struct ldlm_extent l_extent;
- struct ldlm_flock_wire l_flock;
- struct ldlm_inodebits l_inodebits;
-} ldlm_wire_policy_data_t;
-----
-
-Resource Descriptor
-^^^^^^^^^^^^^^^^^^^
-
-----
-struct ldlm_resource_desc {
- ldlm_type_t lr_type;
- __u32 lr_padding; /* also fix lustre_swab_ldlm_resource_desc */
- struct ldlm_res_id lr_name;
-};
-----
-
-The 'ldlm_type_t' is given by one of these values:
-----
-typedef enum {
- LDLM_PLAIN = 10,
- LDLM_EXTENT = 11,
- LDLM_FLOCK = 12,
- LDLM_IBITS = 13
-} ldlm_type_t;
-----
-
-Lock Descriptor
-^^^^^^^^^^^^^^^
-
-----
-struct ldlm_lock_desc {
- struct ldlm_resource_desc l_resource;
- ldlm_mode_t l_req_mode;
- ldlm_mode_t l_granted_mode;
- ldlm_wire_policy_data_t l_policy_data;
-};
-----
-
-Lock Request
-^^^^^^^^^^^^
-
-----
-#define LDLM_LOCKREQ_HANDLES 2
-struct ldlm_request {
- __u32 lock_flags;
- __u32 lock_count;
- struct ldlm_lock_desc lock_desc;
- struct lustre_handle lock_handle[LDLM_LOCKREQ_HANDLES];
-};
-----
-
-Lock Reply
-^^^^^^^^^^
-
-----
-struct ldlm_reply {
- __u32 lock_flags;
- __u32 lock_padding; /* also fix lustre_swab_ldlm_reply */
- struct ldlm_lock_desc lock_desc;
- struct lustre_handle lock_handle;
- __u64 lock_policy_res1;
- __u64 lock_policy_res2;
-};
-----
-
-Lock Value Block
-^^^^^^^^^^^^^^^^
-
-A lock value block is part of reply messages from servers when an
-LDLM_ENQUEUE command has been issued. There are two varieties. Which
-is chosen depends on the target.
-
-----
-struct ost_lvb_v1 {
- __u64 lvb_size;
- obd_time lvb_mtime;
- obd_time lvb_atime;
- obd_time lvb_ctime;
- __u64 lvb_blocks;
-};
-struct ost_lvb {
- __u64 lvb_size;
- obd_time lvb_mtime;
- obd_time lvb_atime;
- obd_time lvb_ctime;
- __u64 lvb_blocks;
- __u32 lvb_mtime_ns;
- __u32 lvb_atime_ns;
- __u32 lvb_ctime_ns;
- __u32 lvb_padding;
-};
-----
+include::lov_mds_md.txt[]
--- /dev/null
+Command 104: LDLM_BL_CALLBACK
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+[[ldlm-bl-callback-rpc]]
+
+An RPC that assists with getting a lock back from an entity that has
+it.
+
+.LDLM_BL_CALLBACK Generic Packet Structure
+image::ldlm-bl-callback-generic.png["LDLM_BL_CALLBACK Generic Packet Structure",height=100]
+
+//////////////////////////////////////////////////////////////////////
+The ldlm-bl-callback-generic.png diagram resembles this text
+art:
+
+ LDLM_BL_CALLBACK:
+ --request---------------------
+ | ptlrpc_body | ldlm_request |
+ ------------------------------
+ --reply--------
+ | ptlrpc_body |
+ ---------------
+//////////////////////////////////////////////////////////////////////
+
+The request RPC resembles the simplest LDLM_ENQUEUE RPC, but only
+identifies the relevant resource that the destination entity already
+had a lock on. It notifies the recipient that the lock should be
+returned once and dirty write data has been flushed. The Reply is
+just an acknowledgment of receipt, and doesn't carry any further
+information about the lock or the resource.
+
+'ptlrpc_body'::
+RPC descriptor.
+
+'ldlm_request'::
+Description of the lock being requested. Which resource is the target,
+what lock is current, and what lock desired.
+
--- /dev/null
+Command 103: LDLM_CANCEL
+~~~~~~~~~~~~~~~~~~~~~~~~
+[[ldlm-cancel-rpc]]
+
+An RPC that notifies an entity that a requested lock is no longer needed.
+
+.LDLM_CANCEL Generic Packet Structure
+image::ldlm-cancel-generic.png["LDLM_CANCEL Generic Packet Structure",height=100]
+
+//////////////////////////////////////////////////////////////////////
+The ldlm-cancel-generic.png diagram resembles this text
+art:
+
+ LDLM_CANCEL:
+ --request---------------------
+ | ptlrpc_body | ldlm_request |
+ ------------------------------
+ --reply--------
+ | ptlrpc_body |
+ ---------------
+//////////////////////////////////////////////////////////////////////
+
+'ptlrpc_body'::
+RPC descriptor.
+
+'ldlm_request'::
+The request RPC identifies the lock being canceled. Only the first
+'lock_handle' field is used. The rest of the 'ldlm_request' sturcture
+is not used.
+
--- /dev/null
+Command 105: LDLM_CP_CALLBACK
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+[[ldlm-cp-callback-rpc]]
+
+An RPC that notifies an entity that a requested lock is now available.
+
+.LDLM_CP_CALLBACK Generic Packet Structure
+image::ldlm-cp-callback-generic.png["LDLM_CP_CALLBACK Generic Packet Structure",height=100]
+
+//////////////////////////////////////////////////////////////////////
+The ldlm-cp-callback-generic.png diagram resembles this text
+art:
+
+ LDLM_CP_CALLBACK:
+ --request-------------------------------
+ | ptlrpc_body | ldlm_request | ost_lvb |
+ ----------------------------------------
+ --reply--------
+ | ptlrpc_body |
+ ---------------
+//////////////////////////////////////////////////////////////////////
+
+The request RPC identifies the relevant resource that already had a
+request pending. It notifies the original requester that it now has
+the stated lock, along with attribute information the about the
+resource. The reply is just an acknowledgment of receipt, and doesn't
+carry any further information about the lock or the resource.
+
+'ptlrpc_body'::
+RPC descriptor.
+
+'ldlm_request'::
+Description of the lock being requested. Which resource is the target,
+what lock is current, and what lock desired.
+
+'ost_lvb'::
+Attribute data associated with a resource on an OST.
+
--- /dev/null
+Command 101: LDLM ENQUEUE - Enqueue a lock request
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+[[ldlm-enqueue-rpc]]
+
+An RPC that implements distributed locking across the servers and
+clients. In its simplest form it just exchanges 'ldlm_request' and
+'ldlm_reply' descriptors of the desired and granted locks.
+
+.LDLM_ENQUEUE Generic Packet Structure
+image::ldlm-enqueue-generic.png["LDLM_ENQUEUE Generic Packet Structure",height=150]
+
+//////////////////////////////////////////////////////////////////////
+The ldlm-enqueue-generic.png diagram resembles this text
+art:
+
+ LDLM_ENQUEUE:
+ --request---------------------
+ | ptlrpc_body | ldlm_request |
+ ------------------------------
+ --reply---------------------
+ | ptlrpc_body | ldlm_reply |
+ ----------------------------
+ --extent lvb reply--------------------
+ | ptlrpc_body | ldlm_reply | ost_lvb |
+ --------------------------------------
+//////////////////////////////////////////////////////////////////////
+
+However, there are many variants to this RPCs. A lock request may signal
+an intention to carry out an operation once a lock has been
+granted. In the following example, the RPCs are fostering the 'intent'
+to look at how a resource is mapped to the available targets, so
+called 'layout' information.
+
+.LDLM_ENQUEUE Intent:Layout Generic Packet Structure
+image::ldlm-enqueue-intent-layout-generic.png["LDLM_ENQUEUE Intent:Layout Generic Packet Structure",height=100]
+
+//////////////////////////////////////////////////////////////////////
+The ldlm-enqueue-intent-layout-generic.png diagram resembles this text
+art:
+
+ LDLM_ENQUEUE:
+ --intent:layout request------------------------------------
+ | ptlrpc_body | ldlm_request |ldlm_intent | layout_intent |
+ lov_mds_md |
+ -----------------------------------------------------------
+ --intent:layout reply--------------------
+ | ptlrpc_body | ldlm_reply | lov_mds_md |
+ -----------------------------------------
+//////////////////////////////////////////////////////////////////////
+
+'ptlrpc_body'::
+RPC descriptor.
+
+'ldlm_request'::
+Description of the lock being requested. Which resource is the target,
+what lock is current, and what lock desired.
+
+'ldlm_intent'::
+Description of the intent being included with the lock request.
+
+'layout_intent'::
+Description of the layout information that is the subject of a layout
+intent.
+
+'ldlm_reply'::
+Resembling the 'ldlm_request', but in this case indicating what the
+LDLM actually granted as well as relevant policy data.
+
+Lock Value Block::
+A Lock Value Block (LVB) is included in the LDLM_ENQUEUE reply message
+when one of three things needs to be communicated back to the
+requester. The three alternatives are captured by the
+'ost_lvb'structure, the 'lov_mds_md' structure, and one other related
+to quotas (and not yet incorporated into this document). LDLM_ENQUEUE
+reply RPCs may include a zero length instance of an LVB.
+
+'ost_lvb'::
+An LVB to communicate attribute data for an extent associated with a
+resource on a lock. It is returned from an OST to a client requesting
+an extent lock.
+
+'lov_mds_md'::
+An LVB to communicate layout data associated with a resource. It is
+returned from an MDT to a client requesting a lock a lock with a
+layout intent. In an intent request (as opposed to a reply and as yet
+unimplemanted) it will modify the layout. It will not be included
+(zero length) in requests in current releases.
--- /dev/null
+LDLM Intent
+^^^^^^^^^^^
+[[struct-ldlm-intent]]
+
+A lock request can include an 'intent' operation. Which operation is
+encoded in the 'ldlm_intent' 'opc'.
+
+----
+struct ldlm_intent {
+ __u64 opc;
+};
+----
+
+The available operations are:
+
+----
+#define IT_OPEN (1 << 0)
+#define IT_CREAT (1 << 1)
+#define IT_READDIR (1 << 2)
+#define IT_GETATTR (1 << 3)
+#define IT_LOOKUP (1 << 4)
+#define IT_UNLINK (1 << 5)
+#define IT_TRUNC (1 << 6)
+#define IT_GETXATTR (1 << 7)
+#define IT_EXEC (1 << 8)
+#define IT_PIN (1 << 9)
+#define IT_LAYOUT (1 << 10)
+#define IT_QUOTA_DQACQ (1 << 11)
+#define IT_QUOTA_CONN (1 << 12)
+#define IT_SETXATTR (1 << 13)
+----
+
--- /dev/null
+LDLM Lock Descriptor
+^^^^^^^^^^^^^^^^^^^^
+[[struct-ldlm-lock-desc]]
+
+The lock descriptor conveys the specific details about a particular
+lock being requested or granted. It appears in
+<<struct-ldlm-request>>.
+
+----
+struct ldlm_lock_desc {
+ struct ldlm_resource_desc l_resource;
+ ldlm_mode_t l_req_mode;
+ ldlm_mode_t l_granted_mode;
+ ldlm_wire_policy_data_t l_policy_data;
+};
+----
+
+The 'l_resource' field identifies the resource upon which the lock is
+being requested or granted. See the description of
+<<struct-ldlm-resource-desc>>.
+
+The 'l_req_mode' and 'l_granted_mode' fields give the kind of lock
+being requested and the kind of lock that has been granted. The field
+values are:
+
+----
+typedef enum {
+ LCK_EX = 1, /* exclusive */
+ LCK_PW = 2, /* privileged write */
+ LCK_PR = 4, /* privileged read */
+ LCK_CW = 8, /* concurrent write */
+ LCK_CR = 16, /* concurrent read */
+ LCK_NL = 32, /* */
+ LCK_GROUP = 64, /* */
+ LCK_COS = 128, /* */
+} ldlm_mode_t;
+----
+
+Despite the fact that the lock modes are not overlapping, these lock
+modes are exclusive. In addition the mode value 0 is the MINMODE,
+i.e. no lock at all.
+
+In a request 'l_req_mode' is the value actually being requested and
+'l_granted_mode' is the value that currently is in place on for the
+requester. In a reply the 'l_req_mode' may be modified if more or
+fewer privileges were granted than requested, and the
+'l_granted_mode' is what has, in fact, been granted.
+
+The 'l_policy_data' field gives the kind of resource being
+requested/granted. It is a union of these struct definitions:
+[[ldlm-wire-policy-data-t]]
+
+----
+typedef union {
+ struct ldlm_extent l_extent;
+ struct ldlm_flock_wire l_flock;
+ struct ldlm_inodebits l_inodebits;
+} ldlm_wire_policy_data_t;
+----
+
+----
+struct ldlm_extent {
+ __u64 start;
+ __u64 end;
+ __u64 gid;
+};
+struct ldlm_flock_wire {
+ __u64 lfw_start;
+ __u64 lfw_end;
+ __u64 lfw_owner;
+ __u32 lfw_padding;
+ __u32 lfw_pid;
+};
+struct ldlm_inodebits {
+ __u64 bits;
+};
+----
+
+Thus the lock may be on an 'extent', a contiguous sequence of bytes
+in a regular file; an 'flock wire', whatever to heck that is; or a
+portion of an inode. For a "plain" lock (or one with no type at all)
+the 'l_policy_data' field has zero length.
+
+
--- /dev/null
+LDLM Reply
+^^^^^^^^^^
+[[struct-ldlm-reply]]
+
+The 'ldlm_reply' structure is the reciprocal of the 'ldlm_request'.
+
+----
+struct ldlm_reply {
+ __u32 lock_flags;
+ __u32 lock_padding; /* also fix lustre_swab_ldlm_reply */
+ struct ldlm_lock_desc lock_desc;
+ struct lustre_handle lock_handle;
+ __u64 lock_policy_res1;
+ __u64 lock_policy_res2;
+};
+----
+
+It closely resembles the 'lock_request' in its structure. In this case
+it is reporting back to the initiator of the request (in the
+'lock_desc') both the lock that was actually granted, if any, and lock
+server's (possibly) updated value for what was being requested.
+
--- /dev/null
+
+LDLM Request
+^^^^^^^^^^^^
+[[struct-ldlm-request]]
+
+The 'ldlm_request' structure describes the details of a lock request.
+
+----
+struct ldlm_request {
+ __u32 lock_flags;
+ __u32 lock_count;
+ struct ldlm_lock_desc lock_desc;
+ struct lustre_handle lock_handle[2];
+};
+----
+
+The 'lock_flags' field governs how the lock request is to be
+interpreted. The flags are:
+
+----
+#define LDLM_FL_LOCK_CHANGED 0x0000000000000001ULL // bit 0
+#define LDLM_FL_BLOCK_GRANTED 0x0000000000000002ULL // bit 1
+#define LDLM_FL_BLOCK_CONV 0x0000000000000004ULL // bit 2
+#define LDLM_FL_BLOCK_WAIT 0x0000000000000008ULL // bit 3
+#define LDLM_FL_AST_SENT 0x0000000000000020ULL // bit 5
+#define LDLM_FL_REPLAY 0x0000000000000100ULL // bit 8
+#define LDLM_FL_INTENT_ONLY 0x0000000000000200ULL // bit 9
+#define LDLM_FL_HAS_INTENT 0x0000000000001000ULL // bit 12
+#define LDLM_FL_FLOCK_DEADLOCK 0x0000000000008000ULL // bit 15
+#define LDLM_FL_DISCARD_DATA 0x0000000000010000ULL // bit 16
+#define LDLM_FL_NO_TIMEOUT 0x0000000000020000ULL // bit 17
+#define LDLM_FL_BLOCK_NOWAIT 0x0000000000040000ULL // bit 18
+#define LDLM_FL_TEST_LOCK 0x0000000000080000ULL // bit 19
+#define LDLM_FL_CANCEL_ON_BLOCK 0x0000000000800000ULL // bit 23
+#define LDLM_FL_DENY_ON_CONTENTION 0x0000000040000000ULL // bit 30
+#define LDLM_FL_AST_DISCARD_DATA 0x0000000080000000ULL // bit 31
+#define LDLM_FL_FAIL_LOC 0x0000000100000000ULL // bit 32
+#define LDLM_FL_SKIPPED 0x0000000200000000ULL // bit 33
+#define LDLM_FL_CBPENDING 0x0000000400000000ULL // bit 34
+#define LDLM_FL_WAIT_NOREPROC 0x0000000800000000ULL // bit 35
+#define LDLM_FL_CANCEL 0x0000001000000000ULL // bit 36
+#define LDLM_FL_LOCAL_ONLY 0x0000002000000000ULL // bit 37
+#define LDLM_FL_FAILED 0x0000004000000000ULL // bit 38
+#define LDLM_FL_CANCELING 0x0000008000000000ULL // bit 39
+#define LDLM_FL_LOCAL 0x0000010000000000ULL // bit 40
+#define LDLM_FL_LVB_READY 0x0000020000000000ULL // bit 41
+#define LDLM_FL_KMS_IGNORE 0x0000040000000000ULL // bit 42
+#define LDLM_FL_CP_REQD 0x0000080000000000ULL // bit 43
+#define LDLM_FL_CLEANED 0x0000100000000000ULL // bit 44
+#define LDLM_FL_ATOMIC_CB 0x0000200000000000ULL // bit 45
+#define LDLM_FL_BL_AST 0x0000400000000000ULL // bit 46
+#define LDLM_FL_BL_DONE 0x0000800000000000ULL // bit 47
+#define LDLM_FL_NO_LRU 0x0001000000000000ULL // bit 48
+#define LDLM_FL_FAIL_NOTIFIED 0x0002000000000000ULL // bit 49
+#define LDLM_FL_DESTROYED 0x0004000000000000ULL // bit 50
+#define LDLM_FL_SERVER_LOCK 0x0008000000000000ULL // bit 51
+#define LDLM_FL_RES_LOCKED 0x0010000000000000ULL // bit 52
+#define LDLM_FL_WAITED 0x0020000000000000ULL // bit 53
+#define LDLM_FL_NS_SRV 0x0040000000000000ULL // bit 54
+#define LDLM_FL_EXCL 0x0080000000000000ULL // bit 55
+----
+
+The 'lock_count' field represents how many requests are queued on this
+resource.
+
+The 'lock_desc' field holds the lock descriptor as described in
+<<struct-ldlm-lock-desc>.
+
+The 'lock_handle' array's first element holds the handle for the lock
+manager (see the description of <<struct-lustre-handle>>) involved in
+the operation. There is only one lock manager involved in any given
+RPC. The second handle is set to zero except in the rare case that
+there is also an early lock cancellation. The latter case will be
+discussed elsewhere.
+
--- /dev/null
+LDLM Resource Descriptor
+^^^^^^^^^^^^^^^^^^^^^^^^
+[[struct-ldlm-resource-desc]]
+
+The resource descriptor identifies the individual resource that is
+being locked, along with what sort of thing it is.
+
+----
+struct ldlm_resource_desc {
+ ldlm_type_t lr_type;
+ __u32 lr_padding; /* also fix lustre_swab_ldlm_resource_desc */
+ struct ldlm_res_id lr_name;
+};
+----
+
+The 'lr_type' field identifies one of the four types of lock that
+might be placed on a resource. A "plain" lock type just locks a
+particular resource. An "extent" lock type only locks a contiguous
+sequence of byte offsets within a regular file. An "flock" lock type
+represents an application layer advisory lock from the 'flock()'
+system call. While Lustre manages "flock" types locks on behalf of the
+application, they do not affect Lustre operation. An "ibits" lock
+type allows fine grained locking of different parts of a single
+resource. A single lock request or cancellation may operate on one or
+more lock bits, or individual lock bits may be granted on the same
+resource separately. See also <<ldlm-wire-policy-data-t>>. A lock
+descriptor may also have no type at all, in which case the 'lr_type'
+field is 0, meaning "no lock".
+
+The 'lr_name' field identifies (by name) the resource(s) that are the
+objects of the locking operation. See the discussion of
+<<struct-ldlm-res-id>>.
+
+----
+typedef enum {
+ LDLM_PLAIN = 10,
+ LDLM_EXTENT = 11,
+ LDLM_FLOCK = 12,
+ LDLM_IBITS = 13,
+} ldlm_type_t;
+----
+
--- /dev/null
+LDLM Resource ID
+^^^^^^^^^^^^^^^^
+[[struct-ldlm-res-id]]
+
+This structure gets used in <<struct-ldlm-resource-desc>>.
+
+----
+struct ldlm_res_id {
+ __u64 name[4];
+};
+----
+
+The 'name' array holds identifiers for the resource in question. Those
+identifiers may be the elements of a 'struct lu_fid' file ID, or they
+may be other uniquely identifying values for the resource. See <<file-id>>.
+
--- /dev/null
+MDS Lock Value Block
+^^^^^^^^^^^^^^^^^^^^
+[[struct-lov-mds-md]]
+
+The 'lov_mds_md' structure is a Lock Value Block (LVB) for layout
+locks. In replies to lock requests and other situations requiring
+layout information from an MDT the 'lov_mds_md' information provides
+details about the layout of a file across the OSTs.
+
+----
+struct lov_mds_md {
+ __u32 lmm_magic;
+ __u32 lmm_pattern;
+ struct ost_id lmm_oi;
+ __u32 lmm_stripe_size;
+ __u16 lmm_stripe_count;
+ __u16 lmm_layout_gen;
+ struct lov_ost_data_v1 lmm_objects[0];
+};
+----
+
+The 'lmm_magic' field is filled in with the value LOV_MAGIC
+(0x0BD10BD0) when the structure is in use. If the structure is in a
+buffer of an RPC without the magic number in place, then the rest of
+the structure is ignored.
+
+The 'lmm_pattern' field is only ever set to LOV_PATTERN_RAID0
+(0x001).
+
+The 'lmm_oi' field gives the LOV object ID for the first OST of the
+layout. This is the OST where striping will begin.
+
+The 'lmm_stripe_size' field give the stripe size for the object. This
+is how many bytes will be on a particular OST before going to the next
+stripe.
+
+The 'lmm_stripe_count' field gives how many OSTs the file is striped
+over.
+
+The 'lmm_layout_gen' field ges updated as the layout of the obeject is
+updated. This way out-of-date references to the layout can be
+recognized.
+
+The 'lmm_objects' array gives per-stripe data for more complex
+(non-uniform) layouts.
+
+
--- /dev/null
+Lustre File IDs
+^^^^^^^^^^^^^^^
+[[file-id]]
+
+Each resource stored on a target is assigned an identifier that is
+unique to that resource.
+
+----
+struct lu_fid {
+ __u64 f_seq;
+ __u32 f_oid;
+ __u32 f_ver;
+};
+----
+
+The 'f_seq' field identifies the target. That is, all the resources
+with a common 'f_seq' will be on the same target. A target can have
+more than one 'f_seq' value assigned to it.
+
+The 'f_oid' gives the specific value for a given resource that is
+unique to that resource on that target.
+
+The 'f_ver' value identifies which version of a resource is being
+identified, in the event that the resource is being updated, and
+different hosts might be referring to different versions of the same
+resource.
--- /dev/null
+Lustre Handle
+^^^^^^^^^^^^^
+[[struct-lustre-handle]]
+
+A Lustre handle is an opaque cookie that identifies some local object
+(e.g. connection, open file, DLM lock, etc) to another node or another
+layer in the software stack.
+
+----
+struct lustre_handle {
+ __u64 cookie;
+};
+----
+
message's format, the reverse is not true. A given format may be used
in many different messages.
-The "Empty" Message
-~~~~~~~~~~~~~~~~~~~
-
-'empty'
-^^^^^^^
-
-.example
-[options="header"]
-|====
-| pb_opc | pb_type
-| MDS_STATFS | PTL_RPC_MSG_REQUEST
-|====
-
-
-An 'empty' message is one that consists only of the Lustre message
-preamble 'ptlrpc_body'. It occurs as either the request of the reply
-(or both) in a variety of Lustre operations.
-
-.format
-[options="header"]
-|====
-| structure | meaning
-| ptlrpc_body | message preamble
-|====
-
-The LDLM Enqueue Client Message
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-'ldlm_enqueue_client'
-^^^^^^^^^^^^^^^^^^^^^
-
-.example
-[options="header"]
-|====
-| pb_opc | pb_type
-| LDLM_ENQUEUE | PTL_RPC_MSG_REQUEST
-|====
-
-
-An 'ldlm_enqueue_client' message format is used to acquire a lock.
-
-.format
-[options="header"]
-|====
-| structure | meaning
-| ptlrpc_body | message preamble
-| ldlm_request | details of the lock request
-|====
-
-An 'ldlm_enqueue_client' message concatenates two data elements into a
-single byte-stream. The two elements correspond to structures
-detailed in the <<data-structs,Data Structures and Defines Section>>.
-
-The LDLM Enqueue Server Message
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-'ldlm_enqueue_server'
-^^^^^^^^^^^^^^^^^^^^^
-
-.example
-[options="header"]
-|====
-| pb_opc | pb_type
-| LDLM_ENQUEUE | PTL_RPC_MSG_REPLY
-|====
-
-
-An 'ldlm_enqueue_server' message format is used to inform a client
-about the status of its request for a lock.
-
-.format
-[options="header"]
-|====
-| structure | meaning
-| ptlrpc_body | message preamble
-| ldlm_reply | details of the lock request
-|====
-
-An 'ldlm_enqueue_server' message concatenates two data elements
-into a single byte-stream. The three elements correspond to structures
-detailed in the <<data-structs,Data Structures and Defines Section>>.
-
-The LDLM Enqueue Server Message with LVB
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-'ldlm_enqueue_lvb_server'
-^^^^^^^^^^^^^^^^^^^^^^^^^
-
-.example
-[options="header"]
-|====
-| pb_opc | pb_type
-| LDLM_ENQUEUE | PTL_RPC_MSG_REPLY
-|====
-
-
-An 'ldlm_enqueue_lvb_server' message format is used to inform a client
-about the status of its request for a lock.
-
-.format
-[options="header"]
-|====
-| structure | meaning
-| ptlrpc_body | message preamble
-| ldlm_reply | details of the lock request
-| ost_lvb | lock value block
-|====
-
-An 'ldlm_enqueue_lvb_server' message concatenates three data elements
-into a single byte-stream. It closely resembles the
-'ldlm_enqueue_server' message with the addition of a lock value
-block. The three elements correspond to structures detailed in the
-<<data-structs,Data Structures and Defines Section>>.
+N.B. As each of the following "message formats" is introduced in the
+context of describing "Lustre commands", the corresponding section
+here will be deleted. These brief entries just state the name of the
+format and its structures. That information is better included in the
+section on the command itself. -ACU
The LLog Origin Handle Create Client
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
and <<recovery>> for how the client is given confirmation that a
request has been completed.
+include::ost_setattr.txt[]
+
Command 8: OST CONNECT - Client connection to an OST
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The information exchanged in a DISCONNECT message is that normally
conveyed in the mesage preamble.
+include::ost_punch.txt[]
+
Command 33: MDS GETATTR - Get MDS Attributes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|====
+include::mds_reint.txt[]
Command 38: MDS CONNECT - Client connection to an MDS
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The MDS_GETSTATUS command targets a specific MDT. If there are several,
the client will need to send a separate message for each.
-.MDS_GETSTATUS (41)
+.MDS_GETSTATUS (40)
[options="header"]
|====
| request | reply
In the reply there is, in addition to the 'ptlrpc_body', data relevant
to a 'statfs' system call.
-Command 101: LDLM ENQUEUE - Enqueue a lock request
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-.LDLM_ENQUEUE (101)
-[options="header"]
-|====
-| request | reply
-| ldlm_enqueue_client | ldlm_enqueue_lvb_server
-|====
-
-In order to access data on disk at a target the client needs to
-establish a lock (either concurrent for reads or exclusive for
-writes). The client sends the 'ldlm_enqueue_client' message to the
-server holding the target, and the server will reply with an
-'ldlm_enqueue_server' message. (N.B. It actuallity it is an
-'ldlm_enqueue_lvb_server' message with the length of the 'struct
-ost_lvb' portion set to zero, which ammounts to the same thing).
-
-The target UUID is just "MGS", and the client UUID is set to the
-32byte string it gets from ... where?
-
-The 'struct lustre_handle' (the fourth buffer in the message) has its
-cookie set to .. what? It is set, but where does it come from?
-
-The 'ocd_connect_flags' field is set to (fixme: what?) reflecting the
-capabilities appropriate to the client. The 'ocd_brw_size' is set to the
-largest value for the size of an RPC that the client can handle. The
-'ocd_ibits_known' and 'ocd_checksum_types' values are set to what the client
-considers appropriate. Other fields in the preamble and
-'obd_connect_data' structures are zero.
+include::mds_getxattr.txt[]
-Once the server receives the 'obd_connect_client' message on behalf of
-the given target it replies with an 'obd_connect_server' message. In
-that message the server sends the 'pb__handle' to uniquely
-identify the connection for subsequent communication. The client notes
-that handle in its import for the given target.
+include::ldlm_enqueue.txt[]
-fixme: Are there circumstances that could lead to the 'status'
-value in the reply being non-zero? What would lead to that and what
-error values would result?
+include::ldlm_cancel.txt[]
-The target maintains the last committed transaction for a client in
-its export for that client. If this is the first connection, then that
-last transactiion value would just be zero. If there were previous
-transactions for the client, then the transaction number for the last
-such committed transaction is put in the 'pb_last_committed' field.
+include::ldlm_bl_callback.txt[]
-In a connection request the operation is not file system modifying, so
-the 'pb_transno' value will be zero in the reply as well.
+include::ldlm_cp_callback.txt[]
Command 250: MGS CONNECT - Client connection to an MGS
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
--- /dev/null
+Command 49: MDS_GETXATTR
+~~~~~~~~~~~~~~~~~~~~~~~~
+[[mds-getxattr-rpc]]
+
+An RPC that provides information about extended attributes for a
+resource.
+
+.MDS_GETXATTR Generic Packet Structure
+image::mds-getxattr-generic.png["MDS_GETXATTR Generic Packet Structure",height=100]
+
+//////////////////////////////////////////////////////////////////////
+The mds-getxattr-generic.png diagram resembles this text art:
+
+ MDS_GETXATTR:
+ --request-----------------------------------------------
+ | ptlrpc_body | mdt_body | lustre_capa | name | eadata |
+ --------------------------------------------------------
+ --reply----------------------------
+ | ptlrpc_body | mdt_body | eadata |
+ -----------------------------------
+//////////////////////////////////////////////////////////////////////
+
+'ptlrpc_body'::
+RPC descriptor.
+
+'mdt_body'::
+Metadata about the resource.
+
+'lustre_capa'::
+So called "capabilities" structure. This is deprecated in recent
+versions of Lustre, and commonly appears in the packet header as a zero
+length buffer.
+
+'name'::
+A text field supplying the name of the desired resource.
+
+'eadata'::
+Information about extended attributes. This buffer is optional and will
+appear as zero length in some packets.
+
--- /dev/null
+Command 36: MDS_REINT
+~~~~~~~~~~~~~~~~~~~~~
+[[mds-reint-rpm]]
+
+An RPC that implements an operation that will change the information
+on an MDT.
+
+REINT_SETATTR
+^^^^^^^^^^^^^
+[[mds-reint-setattr-rpc]]
+
+An RPC that implements the 'setattr' sub-command of the MDS_REINT.
+
+.MDS_REINT:REINT_SETATTR Generic Packet Structure
+image::mds-reint-setattr-generic.png["MDS_REINT:REINT_SETATTR Generic Packet Structure",height=100]
+
+//////////////////////////////////////////////////////////////////////
+The mds-reint-setattr-generic.png diagram resembles this text art:
+
+ MDS_REINT:
+ --REINT_SETATTR-request-------------------------------------
+ | ptlrpc_body | mdt_rec_setattr | lustre_capa | mdt_ioepoc |
+ eadata | llog_cookie | ldlm_request |
+ ------------------------------------------------------------
+
+ --REINT_SETATTR-reply----------------------------------
+ | ptlrpc_body | mdt_body | mdt_md | acl | lustre_capa |
+ lustre_capa |
+ -------------------------------------------------------
+//////////////////////////////////////////////////////////////////////
+
+'ptlrpc_body'::
+RPC descriptor.
+
+'mdt_rec_setattr'::
+Information pertinent to setting attributes on the MDT.
+
+'lustre_capa'::
+So called "capabilities" structure. This is deprecated in recent
+versions of Lustre, and commonly appears in the packet header as a zero
+length buffer.
+
+'mdt_ioepoch'::
+Identifying "epoch" information. This buffer is optional and will
+appear as zero length in some packets.
+
+'eadata'::
+Information about extended attributes. This buffer is optional and will
+appear as zero length in some packets.
+
+'llog_cookie'::
+A log handle. This buffer is optional and will appear as zero length
+in some packets.
+
+'ldlm_request'::
+A structure specifying a lock that can be the subject of early lock
+cancellation. See <<early-lock-cancellation>>.
+
+'mdt_body'::
+Metadata about the resource.
+
+'mdt_md'::
+Layout data for the resource. This buffer is optional and will appear
+as zero length in some packets.
+
+'acl'::
+Access control list data. This buffer is optional and will appear as
+zero length in some packets.
+
--- /dev/null
+MDS_REINT Structures
+~~~~~~~~~~~~~~~~~~~~
+[[mds-reint-structs]]
+
+MDS_REINT RPCs are those that get issued from a client to request an
+operation on an MDT that will modify the MDT, so the MDT will initiate
+a transaction to carry out the operation. The transactional nature of
+the operation allows it to be replayed in the event of failure. Note:
+The only other MDT-modifying operation is MDS_CLOSE.
+
+include::mdt_rec_reint.txt[]
+
+include::mdt_rec_setattr.txt[]
--- /dev/null
+MDT Body
+^^^^^^^^
+[[struct-mdt-body]]
+
+An 'mdt_body' structure conveys information about the metadata for a
+resource.
+
+----
+struct mdt_body {
+ struct lu_fid mbo_fid1; /* OBD_MD_FLID */
+ struct lu_fid mbo_fid2; /* OBD_MD_FLID */
+ struct lustre_handle mbo_handle;
+ __u64 mbo_valid;
+ __u64 mbo_size; /* OBD_MD_FLSIZE */
+ __s64 mbo_mtime; /* OBD_MD_FLATIME */
+ __s64 mbo_atime; /* OBD_MD_FLMTIME */
+ __s64 mbo_ctime; /* OBD_MD_FLCTIME */
+ __u64 mbo_blocks; /* OBD_MD_FLBLOCKS */
+ __u64 mbo_ioepoch;
+ __u64 mbo_t_state; /* OBD_MD_TSTATE */
+ __u32 mbo_fsuid;
+ __u32 mbo_fsgid;
+ __u32 mbo_capability;
+ __u32 mbo_mode; /* OBD_MD_FLMODE */
+ __u32 mbo_uid; /* OBD_MD_FLUID */
+ __u32 mbo_gid; /* OBD_MD_FLGID */
+ __u32 mbo_flags; /* OBD_MD_FLFLAGS */
+ __u32 mbo_rdev; /* OBD_MD_FLRDEV */
+ __u32 mbo_nlink; /* OBD_MD_FLNLINK */
+ __u32 mbo_unused2;
+ __u32 mbo_suppgid;
+ __u32 mbo_eadatasize; /* OBD_MD_FLEASIZE */
+ __u32 mbo_aclsize; /* OBD_MD_FLACL */
+ __u32 mbo_max_mdsize; /* OBD_MD_FLMODEASIZE */
+ __u32 mbo_max_cookiesize; /* OBD_MD_FLMODEASIZE */
+ __u32 mbo_uid_h;
+ __u32 mbo_gid_h;
+ __u32 mbo_padding_5;
+ __u64 mbo_padding_6;
+ __u64 mbo_padding_7;
+ __u64 mbo_padding_8;
+ __u64 mbo_padding_9;
+ __u64 mbo_padding_10;
+}; /* 216 */
+----
+
+For an operation that involves two resources both the 'mbo_fid1' and
+'mbo_fid2' fields will be filled in. If the 'mdt_body' is part of an
+RPC affecting or involving only a single resource then 'mbo_fid1' will
+designate that resource and 'mbo_fid2' will be cleared (see
+<<file-id>>).
+
+The 'mbo_handle' field indicates the identity of an open file related
+to the operation, if any. If there is no lock then it is just 0.
+
+The 'mbo_valid' field identifies which of the remaining fields are
+actually in force. The flags in 'mbo_valid' are:
+
+----
+#define OBD_MD_FLID (0x00000001ULL)
+#define OBD_MD_FLATIME (0x00000002ULL)
+#define OBD_MD_FLMTIME (0x00000004ULL)
+#define OBD_MD_FLCTIME (0x00000008ULL)
+#define OBD_MD_FLSIZE (0x00000010ULL)
+#define OBD_MD_FLBLOCKS (0x00000020ULL)
+#define OBD_MD_FLMODE (0x00000080ULL)
+#define OBD_MD_FLUID (0x00000200ULL)
+#define OBD_MD_FLGID (0x00000400ULL)
+#define OBD_MD_FLFLAGS (0x00000800ULL)
+#define OBD_MD_FLNLINK (0x00002000ULL)
+#define OBD_MD_FLRDEV (0x00010000ULL)
+#define OBD_MD_FLEASIZE (0x00020000ULL)
+#define OBD_MD_FLMODEASIZE (0x80000000ULL)
+#define OBD_MD_TSTATE (0x0000000800000000ULL)
+#define OBD_MD_FLACL (0x0000008000000000ULL)
+----
+
+Which flag is associated with which field is indicated in the comments
+beside the fields in the struct definition above. The fields without
+corresponding flags in the 'mbo_valid' field should be considered
+valid, though I'm not sure how much they actually get used.
+
+Many of the fields have names and meanings that correspond directly to
+those in 'struct mdt_rec_setattr' (ex. 'atime', 'size', 'mode',
+etc). Mainly these are the conventional metadata attributes, and
+should be self-explanatory. The ones that are a little less obvious are
+discussed here.
+
+The 'mbo_ioepoch' field is no longer in use.
+
+The 'mbo_t_state' field indicates if there is a "transient state", as
+when an HSM "restore" is in operation.
+
+The 'mbo_rdev' field holds the special device number for a block or
+character device associated with the resource, if there is one. It will
+be 0 for a regular file.
+
+The 'mbo_nlink' field gives the number of hard links to the resource.
+
+The 'mbo_eadatasize' field gives the size of the extended attributes,
+which hold such information as the resource's layout.
+
+The 'mbo_aclsize' field supports POSIX access control lists (ACLs).
+
+The 'mbo_max_mdsize' field
+
+The 'mbo_max_cookiesize' field is unused.
+
--- /dev/null
+Generic MDS_REINT
+^^^^^^^^^^^^^^^^^
+
+An 'mdt_rec_reint' structure specifies the generic form for MDS_REINT
+requests. Each sub-operation, as defned by the 'rr_opcode' field, has
+its own variant of this structure. Each variant has the same size as
+the generic 'mdt_rec_reint', but interprets its fields slightly
+differently. Note that in order for swabbing to take place correctly
+the sequence of field sizes must the same in every variant as it is in
+the generic version (not just the overal size of the sturcture).
+
+----
+struct mdt_rec_reint {
+ __u32 rr_opcode;
+ __u32 rr_cap;
+ __u32 rr_fsuid;
+ __u32 rr_fsuid_h;
+ __u32 rr_fsgid;
+ __u32 rr_fsgid_h;
+ __u32 rr_suppgid1;
+ __u32 rr_suppgid1_h;
+ __u32 rr_suppgid2;
+ __u32 rr_suppgid2_h;
+ struct lu_fid rr_fid1;
+ struct lu_fid rr_fid2;
+ obd_time rr_mtime;
+ obd_time rr_atime;
+ obd_time rr_ctime;
+ __u64 rr_size;
+ __u64 rr_blocks;
+ __u32 rr_bias;
+ __u32 rr_mode;
+ __u32 rr_flags;
+ __u32 rr_flags_h;
+ __u32 rr_umask;
+ __u32 rr_padding_4; /* also fix lustre_swab_mdt_rec_reint */
+};
+----
+
+The 'rr_opcode' field defines one among the several sub-commands for
+MDS REINT RPCs. Those opcodes are:
+
+----
+typedef enum {
+ REINT_SETATTR = 1,
+ REINT_CREATE = 2,
+ REINT_LINK = 3,
+ REINT_UNLINK = 4,
+ REINT_RENAME = 5,
+ REINT_OPEN = 6,
+ REINT_SETXATTR = 7,
+} mds_reint_t;
+----
+
+Based on that opcode one of the variants of the structure will
+actually be used. See <<mds-reint-setattr-rpc>> for one example.
+
+The 'rr_cap' field is not used (and has been dropped in more recent
+code updates).
+
+The 'rr_fsuid' field gives the UID of the file system mount point.
+
+The 'rr_fsuid_h' field gives the high bytes of a the file system mount
+point UID in an environment where UIDs are 64-bit. The same holds for
+the other '_h' fields.
+
+The 'rr_fsgid' field gives the GID of the file system mount point.
+For each opcode there is a variant of the 'mdt_rec_reint' that has
+identical byte fields, but slightly modified semantics. The following
+is for the REINT_SETATTR sub-operation.
+
+The 'rr_suppgid1' and 'rr_suppgid2' fields are supplementary GID
+information for kernels that have it.
+
+The 'rr_fid1' and 'rr_fid2' fields specify the file IDs for the
+resources being operated upon. If only one resource is being acted
+upon then 'rr_fid2' is not used.
+
+The 'rr_mtime' 'rr_atime' and 'rr_ctime' fields give the values of the
+time attributes. The 'obd_time' type is also a '__u64'.
+
+The 'rr_size' field gives the value of the size attribute.
+
+The 'rr_blocks' field gives the value of the number-of-blocks
+attribute.
+
+The 'rr_bias' field adds additional optional information to the
+REINT. The possible values are:
+
+----
+enum mds_op_bias {
+ MDS_CHECK_SPLIT = 1 << 0,
+ MDS_CROSS_REF = 1 << 1,
+ MDS_VTX_BYPASS = 1 << 2,
+ MDS_PERM_BYPASS = 1 << 3,
+ MDS_SOM = 1 << 4,
+ MDS_QUOTA_IGNORE = 1 << 5,
+ MDS_KEEP_ORPHAN = 1 << 7,
+ MDS_RECOV_OPEN = 1 << 8,
+ MDS_DATA_MODIFIED = 1 << 9,
+ MDS_CREATE_VOLATILE = 1 << 10,
+ MDS_OWNEROVERRIDE = 1 << 11,
+ MDS_HSM_RELEASE = 1 << 12,
+};
+----
+
+For example, MDS_DATA_MODIFIED signals to the HSM system that the MDT
+should set the corresponding HSM extended attribute. We'll return to
+this topic with more details about the individual flags at a later
+date.
+
+The 'rr_mode' field gives the value of the mode attribute.
+
+The 'rr_flags' and 'rr_flags_h' fields give the value (and the high
+order bytes of the value, respectively, if there are high order bytes)
+of the flags attribute.
+
+The 'rr_umask' gives the value of the 'umask' to apply in, for
+example, a REINT_CREATE.
--- /dev/null
+REINT_SETATTR
+^^^^^^^^^^^^^
+[[mdt-rec-setattr]]
+
+The variant of the 'mdt_rec_reint' for the 'setattr' operation is:
+
+----
+struct mdt_rec_setattr {
+ __u32 sa_opcode;
+ __u32 sa_cap;
+ __u32 sa_fsuid;
+ __u32 sa_fsuid_h;
+ __u32 sa_fsgid;
+ __u32 sa_fsgid_h;
+ __u32 sa_suppgid;
+ __u32 sa_suppgid_h;
+ __u32 sa_padding_1;
+ __u32 sa_padding_1_h;
+ struct lu_fid sa_fid;
+ __u64 sa_valid;
+ __u32 sa_uid;
+ __u32 sa_gid;
+ __u64 sa_size;
+ __u64 sa_blocks;
+ obd_time sa_mtime;
+ obd_time sa_atime;
+ obd_time sa_ctime;
+ __u32 sa_attr_flags;
+ __u32 sa_mode;
+ __u32 sa_bias; /* some operation flags */
+ __u32 sa_padding_3;
+ __u32 sa_padding_4;
+ __u32 sa_padding_5;
+};
+----
+
+The 'setattr' variant modifies the semantics of the generic REINT
+fields as follows:
+
+There is only one FID to be operated upon, and the 'sa_valid',
+'sa_uid', and 'sa_gid' values take the place of the second 'struct
+lu_fid'.
+
+The 'sa_valid' field identifies which of the other fields in the
+structure are to be honored. If the corresponding flag bit is not set
+then the value of the corresponding field is to be ignored. The flags
+are:
+
+----
+#define MDS_ATTR_MODE 0x1ULL /* = 1 */
+#define MDS_ATTR_UID 0x2ULL /* = 2 */
+#define MDS_ATTR_GID 0x4ULL /* = 4 */
+#define MDS_ATTR_SIZE 0x8ULL /* = 8 */
+#define MDS_ATTR_ATIME 0x10ULL /* = 16 */
+#define MDS_ATTR_MTIME 0x20ULL /* = 32 */
+#define MDS_ATTR_CTIME 0x40ULL /* = 64 */
+#define MDS_ATTR_ATIME_SET 0x80ULL /* = 128 */
+#define MDS_ATTR_MTIME_SET 0x100ULL /* = 256 */
+#define MDS_ATTR_FORCE 0x200ULL /* = 512 */
+#define MDS_ATTR_ATTR_FLAG 0x400ULL /* = 1024 */
+#define MDS_ATTR_KILL_SUID 0x800ULL /* = 2048 */
+#define MDS_ATTR_KILL_SGID 0x1000ULL /* = 4096 */
+#define MDS_ATTR_CTIME_SET 0x2000ULL /* = 8192 */
+#define MDS_ATTR_FROM_OPEN 0x4000ULL /* = 16384 */
+#define MDS_ATTR_BLOCKS 0x8000ULL /* = 32768 */
+----
+
+The 'sa_uid' and 'sa_gid' fields give the values of the UID and GID
+attributes, respectively, to be set on the target resource.
+
+There is no umask for the 'setattr' variant, and no high order bytes
+for the flags attribute. Otherwise, the order of the attributes is
+different in the structure, but their semantics remains the same as for
+the generic case.
+
+
--- /dev/null
+MDT Structures
+~~~~~~~~~~~~~~
+[[mdt-structs]]
+
+These structures convey information to or from an MDT concerning the
+metadata about a resource.
+
+include::mdt_body.txt[]
--- /dev/null
+OST Lock Value Block
+^^^^^^^^^^^^^^^^^^^^
+[[struct-ost-lvb]]
+
+The 'ost_lvb' structure is a "lock value block", and encompasses
+attribute data for resources on the OST. It is an optional part of an
+LDLM_ENQUEUE reply RPC .
+
+----
+struct ost_lvb {
+ __u64 lvb_size;
+ __s64 lvb_mtime;
+ __s64 lvb_atime;
+ __s64 lvb_ctime;
+ __u64 lvb_blocks;
+ __u32 lvb_mtime_ns;
+ __u32 lvb_atime_ns;
+ __u32 lvb_ctime_ns;
+ __u32 lvb_padding;
+};
+----
--- /dev/null
+Command 10: OST_PUNCH
+~~~~~~~~~~~~~~~~~~~~~
+[[ost-punch-rpc]]
+
+An RPC that modifies the size attribute of a resource on an OST.
+
+.OST_PUNCH Generic Packet Structure
+image::ost-punch-generic.png["OST_PUNCH Generic Packet Structure",height=100]
+
+//////////////////////////////////////////////////////////////////////
+The ost-punch-generic.png diagram resembles this text art:
+
+ OST_PUNCH:
+ --request-------------------------------
+ | ptlrpc_body | ost_body | lustre_capa |
+ ----------------------------------------
+ --reply-------------------
+ | ptlrpc_body | ost_body |
+ --------------------------
+//////////////////////////////////////////////////////////////////////
+
+'ptlrpc_body'::
+RPC descriptor.
+
+'ost_body'::
+Metadata about the resource, as it appears on the OST.
+
+'lustre_capa'::
+So called "capabilities" structure. This is deprecated in recent
+versions of Lustre, and commonly appears in the packet header as a zero
+length buffer.
+
--- /dev/null
+Command 2: OST_SETATTR
+~~~~~~~~~~~~~~~~~~~~~~
+[[ost-setattr-rpc]]
+
+An RPC that sets resource attributes.
+
+.OST_SETATTR Generic Packet Structure
+image::ost-setattr-generic.png["OST_SETATTR Generic Packet Structure",height=100]
+
+//////////////////////////////////////////////////////////////////////
+The ost-setattr-generic.png diagram resembles this text art:
+
+ OST_SETATTR:
+ --request-----------------
+ | ptlrpc_body | ost_body |
+ --------------------------
+ --reply-------------------
+ | ptlrpc_body | ost_body |
+ --------------------------
+//////////////////////////////////////////////////////////////////////
+
+'ptlrpc_body'::
+RPC descriptor.
+
+'ost_body'::
+Metadata about the resource, as it appears on the OST.
+
--- /dev/null
+OST_SETATTR Structures
+~~~~~~~~~~~~~~~~~~~~~~
+[[ost-setattr]]
+
+OST Body
+^^^^^^^^
+[[struct-ost-body]]
+
+The 'ost_body' structure just hold a 'struct 'obdo', which is where
+all the actual information is conveyed.
+
+----
+struct ost_body {
+ struct obdo oa;
+};
+----
+
+Obdo
+^^^^
+[[struct-obdo]]
+
+The 'obdo' structure conveys metadata about a resource on an OST.
+
+----
+struct obdo {
+ __u64 o_valid;
+ struct ost_id o_oi; /* OBD_MD_FLID */
+ __u64 o_parent_seq; /* OBD_MD_FLFID */
+ __u64 o_size; /* OBD_MD_FLSIZE */
+ __s64 o_mtime; /* OBD_MD_FLMTIME */
+ __s64 o_atime; /* OBD_MD_FLMTIME */
+ __s64 o_ctime; /* OBD_MD_FLCTIME */
+ __u64 o_blocks; /* OBD_MD_FLBLOCKS */
+ __u64 o_grant; /* OBD_MD_FLGRANT */
+ __u32 o_blksize; /* OBD_MD_FLBLKSZ */
+ __u32 o_mode; /* OBD_MD_FLMODE */
+ __u32 o_uid; /* OBD_MD_FLUID */
+ __u32 o_gid; /* OBD_MD_FLGID */
+ __u32 o_flags; /* OBD_MD_FLFLAGS */
+ __u32 o_nlink; /* OBD_MD_FLNLINK */
+ __u32 o_parent_oid;
+ __u32 o_misc;
+ __u64 o_ioepoch; /* OBD_MD_ */
+ __u32 o_stripe_idx;
+ __u32 o_parent_ver; /* OBD_MD_FLFID */
+ struct lustre_handle o_handle; /* OBD_MD_FLHANDLE */
+ struct llog_cookie o_lcookie; /* OBD_MD_FLCOOKIE */
+ __u32 o_uid_h;
+ __u32 o_gid_h;
+ __u64 o_data_version; /* OBD_MD_FLDATAVERSION */
+ __u64 o_padding_4;
+ __u64 o_padding_5;
+ __u64 o_padding_6;
+};
+----
+
+The 'o_valid' field identifies which other fields in the structure are
+to be interpreted. The flags are the same set (with additions) used
+for the 'mdt_body' field 'mbo_valid' (see <<struct-mdt-body>>).
+
+----
+#define OBD_MD_FLID (0x00000001ULL)
+#define OBD_MD_FLATIME (0x00000002ULL)
+#define OBD_MD_FLMTIME (0x00000004ULL)
+#define OBD_MD_FLCTIME (0x00000008ULL)
+#define OBD_MD_FLSIZE (0x00000010ULL)
+#define OBD_MD_FLBLOCKS (0x00000020ULL)
+#define OBD_MD_FLBLKSZ (0x00000040ULL)
+#define OBD_MD_FLMODE (0x00000080ULL)
+#define OBD_MD_FLTYPE (0x00000100ULL)
+#define OBD_MD_FLUID (0x00000200ULL)
+#define OBD_MD_FLGID (0x00000400ULL)
+#define OBD_MD_FLFLAGS (0x00000800ULL)
+#define OBD_MD_FLNLINK (0x00002000ULL)
+#define OBD_MD_FLGENER (0x00004000ULL)
+/*#define OBD_MD_FLINLINE (0x00008000ULL)
+#define OBD_MD_FLRDEV (0x00010000ULL)
+#define OBD_MD_FLEASIZE (0x00020000ULL)
+#define OBD_MD_LINKNAME (0x00040000ULL)
+#define OBD_MD_FLHANDLE (0x00080000ULL)
+#define OBD_MD_FLCKSUM (0x00100000ULL)
+#define OBD_MD_FLQOS (0x00200000ULL)
+/*#define OBD_MD_FLOSCOPQ (0x00400000ULL)
+#define OBD_MD_FLCOOKIE (0x00800000ULL)
+#define OBD_MD_FLGROUP (0x01000000ULL)
+#define OBD_MD_FLFID (0x02000000ULL)
+#define OBD_MD_FLEPOCH (0x04000000ULL)
+#define OBD_MD_FLGRANT (0x08000000ULL)
+#define OBD_MD_FLDIREA (0x10000000ULL)
+#define OBD_MD_FLUSRQUOTA (0x20000000ULL)
+#define OBD_MD_FLGRPQUOTA (0x40000000ULL)
+#define OBD_MD_FLMODEASIZE (0x80000000ULL)
+#define OBD_MD_MDS (0x0000000100000000ULL)
+#define OBD_MD_REINT (0x0000000200000000ULL)
+#define OBD_MD_MEA (0x0000000400000000ULL)
+#define OBD_MD_TSTATE (0x0000000800000000ULL)
+#define OBD_MD_FLXATTR (0x0000001000000000ULL)
+#define OBD_MD_FLXATTRLS (0x0000002000000000ULL)
+#define OBD_MD_FLXATTRRM (0x0000004000000000ULL)
+#define OBD_MD_FLACL (0x0000008000000000ULL)
+#define OBD_MD_FLRMTPERM (0x0000010000000000ULL)
+#define OBD_MD_FLMDSCAPA (0x0000020000000000ULL)
+#define OBD_MD_FLOSSCAPA (0x0000040000000000ULL)
+#define OBD_MD_FLCKSPLIT (0x0000080000000000ULL)
+#define OBD_MD_FLCROSSREF (0x0000100000000000ULL)
+#define OBD_MD_FLGETATTRLOCK (0x0000200000000000ULL)
+#define OBD_MD_FLOBJCOUNT (0x0000400000000000ULL)
+#define OBD_MD_FLRMTLSETFACL (0x0001000000000000ULL)
+#define OBD_MD_FLRMTLGETFACL (0x0002000000000000ULL)
+#define OBD_MD_FLRMTRSETFACL (0x0004000000000000ULL)
+#define OBD_MD_FLRMTRGETFACL (0x0008000000000000ULL)
+#define OBD_MD_FLDATAVERSION (0x0010000000000000ULL)
+#define OBD_MD_FLRELEASED (0x0020000000000000ULL)
+#define OBD_MD_DEFAULT_MEA (0x0040000000000000ULL)
+----
+
+The two fields here that are specific to the 'ost_body' case are the
+'o_oi' and the 'o_stripe_idx', which give the identity of the OST and
+the stripe index assigned to it.
+
+OST ID
+^^^^^^
+[[struct-ost-id]]
+
+The 'ost_id' identifies an object on a particular OST.
+
+----
+struct ost_id {
+ union {
+ struct {
+ __u64 oi_id;
+ __u64 oi_seq;
+ } oi;
+ struct lu_fid oi_fid;
+ };
+};
+----
+
include::ldlm.txt[]
+include::early_lock_cancellation.txt[]
+
include::llog.txt[]
include::recovery.txt[]
--- /dev/null
+Setattr
+~~~~~~~
+
+The 'setattr' VFS method is used to modify the attributes associated
+with a resource (it is an inode operation). The attributes are the
+same ones returned by a 'stat' operation: mode, uid, guid, size,
+atime, ctime, and mtime.
+
+Changing the File Mode Attribute
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If only the file 'mode' is being modified (a 'chmod' command, for
+instance) then the interaction is relatively simple as shown in
+<<chmod-rpcs>>.
+
+.Setattr RPCs for Changing the Resource's Mode
+[[chmod-rpcs]]
+image::chmod_rpcs.png["setattr RPCs for changing mode",height=100]
+
+//////////////////////////////////////////////////////////////////////
+The chmod_rpcs.png diagram resembles this text art:
+
+Time
+Step Client MDT OST
+ ------- ------- -------
+1 MDS_REINT------->
+2 <-------MDS_REINT
+//////////////////////////////////////////////////////////////////////
+
+1 - Client1 issues an MDS_REINT with the REINT_SETATTR sub-operation.
+
+In addition to the 'ptlrpc_body' (RPC message header), the MDS_REINT
+request RPC from the client has the REINT structure 'mdt_rec_setattr', and a
+lock request 'ldlm_request'. For a detailed discussion of all the fields in
+the 'mdt_rec_setattr' and 'ldlm_request' refer to <<mdt-rec-setattr>>
+and <<struct-ldlm-request>>.
+
+.MDS_REINT:REINT_SETATTR Request Packet Structure
+image::mds-reint-setattr-request.png["MDS_REINT:REINT_SETATTR Request Packet Structure",height=50]
+
+//////////////////////////////////////////////////////////////////////
+The mds-reint-setattr-request.png diagram resembles this text art:
+
+ MDS_REINT:
+ --REINT_SETATTR-request-------------------------
+ | ptlrpc_body | mdt_rec_setattr | ldlm_request |
+ ------------------------------------------------
+//////////////////////////////////////////////////////////////////////
+
+In this case the 'setattr' wants to set the mode attribute on
+the resource. The 'mdt_rec_setattr' identifies the resource with the
+'sa_fid' field, and the 'sa_valid' field is set to 0x2041:
+
+.Flags for 'sa_valid' field of 'struct mdt_rec_setattr'
+[options="header"]
+|====
+| Flag | Meaning
+| MDS_ATTR_MODE | mode attribute
+| MDS_ATTR_CTIME | ctime attribute
+| MDS_ATTR_CTIME_SET | ctime is being set
+|====
+
+So the 'ctime' is also updated on the MDT. The mode and time values
+are put in the corresponding fields of the 'mdt_rec_setattr', and the
+other attribute fields will be ignored.
+
+The 'ldlm_request' structure encompasses an early lock cancellation
+(see <<early-lock-cancellation>>) on the lock that the client had
+previously acquired for the target resource. The lock handle
+identifies this lock. Only lock_count and lock_handle are used, and
+the rest of the ldlm_request is cleared, i.e. all fields set to zero.
+
+2 - The MDS_REINT reply acknowledges the updated attributes.
+
+In addition to the 'ptlrpc_body' (RPC message header), the MDS_REINT
+reply RPC to the client has the 'mdt_body' structure. For a detailed
+discussion of the fields in the 'mdt_body' refer to <<struct-mdt-body>>.
+
+.MDS_REINT:REINT_SETATTR Reply Packet Structure
+image::mds-reint-setattr-reply.png["MDS_REINT:REINT_SETATTR Reply Packet Structure",height=50]
+
+//////////////////////////////////////////////////////////////////////
+The mds-reint-setattr-reply.png diagram resembles this text art:
+
+ --REINT_SETATTR-reply-----
+ | ptlrpc_body | mdt_body |
+ --------------------------
+//////////////////////////////////////////////////////////////////////
+
+The reply from the MDT after the setattr operation has these valid
+'mdt_body' fields:
+
+.Flags for 'mbo_valid' field of 'struct mdt_body'
+[options="header"]
+|====
+| Flag | Meaning
+| OBD_MD_FLID | FID
+| OBD_MD_FLMTIME | mtime attribute
+| OBD_MD_FLSIZE | size attribute
+| OBD_MD_FLBLOCKS | blocks attribute
+| OBD_MD_BLKSZ | block size attribute
+| OBD_MD_FLTYPE | type attribute
+| OBD_MD_FLNLINK | number of links attribute
+| OBD_MD_FLRDEV | device attribute
+|====
+
+So the client is updated with any other information the MDT has after
+the attributes were set at the client's request.
+
+Changing the File Time Attributes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The RPC(s) that get sent for the 'setattr' depend on specifically what
+values are being set. If the time values are being set (as in a
+"touch" command) then there are RPCs in addition to the MDS_REINT,
+with the REINT_SETATTR sub-operation, that update the time vales on
+the MDT. That operation is followed by an OST_SETATTR that sets the
+time values on the OST (or OSTs if there are several). But in order to
+know what OSTs to contact the client must first get the layout of the
+resource. Then it can send the OST_SETATTR RPC to the appropriate OSTs
+and update the time attributes.
+
+.Setattr RPCs for Changing the Resource's Time Attributes
+[[touch-rpcs]]
+image::touch_rpcs.png["setattr RPCs for the time attributes",height=200]
+
+//////////////////////////////////////////////////////////////////////
+The touch_rpcs.png diagram resembles this text art:
+
+Time
+Step Client MDT OST
+ ------- ------- -------
+1 MDS_REINT------->
+2 <-------MDS_REINT
+3 LDLM_ENQUEUE---->
+4 <-------LDLM_ENQUEUE
+5 OST_SETATTR------------------>
+6 <--------------------OST_SETATTR
+//////////////////////////////////////////////////////////////////////
+
+
+1 - The client issues an MDS_REINT with the REINT_SETATTR sub-operation.
+
+The MDS_REINT request RPC closely resembles the one described above,
+but in this case the 'setattr' wants to set the time attributes on the
+resource. The 'mdt_rec_setattr' again identifies the resource with the
+'sa_fid' field, and the 'sa_valid' field is set to 0x21f0:
+
+.Flags for 'sa_valid' field of 'struct mdt_rec_setattr'
+[options="header"]
+|====
+| Flag | Meaning
+| MDS_ATTR_ATIME | atime attribute
+| MDS_ATTR_MTIME | mtime attribute
+| MDS_ATTR_CTIME | ctime attribute
+| MDS_ATTR_ATIME_SET | atime is being set
+| MDS_ATTR_MTIME_SET | mtime is being set
+| MDS_ATTR_CTIME_SET | ctime is being set
+|====
+
+The time values are put in the corresponding fields of the
+'mdt_rec_setattr', and the other attribute fields will be ignored.
+
+There is again an early lock cancellation, since the client knows it
+no longer need to have a lock on the MDT resource attributes.
+
+2 - The MDS_REINT reply acknowledges the updated times.
+
+The MDS_REINT reply is identical to the previous case in every way,
+including which valid attributes it echoes back.
+
+3 - The client asks for a intent lock on the layout data for the
+resource.
+
+Before communicating with the OSTs the client needs to know which ones
+are involved with this resource, and before it can ask for that
+'layout' information it must acquire a 'layout lock'. The
+LDLM_ENQUEUE RPC in this case has (in addition to the 'ptlrpc_body'
+structure) an 'ldlm_request', an 'ldlm_intent', and a 'layout_intent'.
+
+.LDLM_ENQUEUE Intent:Layout Request Packet Structure
+image::ldlm-enqueue-intent-layout-request.png["LDLM_ENQUEUE Intent:Layout request Packet Structure",height=50]
+
+//////////////////////////////////////////////////////////////////////
+The ldlm-enqueue-intent-layout-request.png diagram resembles this text
+art:
+
+ LDLM_ENQUEUE:
+ --intent:layout request------------------------------------
+ | ptlrpc_body | ldlm_request |ldlm_intent | layout_intent |
+ -----------------------------------------------------------
+//////////////////////////////////////////////////////////////////////
+
+The 'ldlm_request' asks for a read lock on the resource and has its
+intent flag set. The 'ldlm_intent' has the intent opcode is 0x800:
+IT_LAYOUT. The 'layout_intent' has the 'li_opc' value 0:
+LAYOUT_INTENT_ACCESS.
+
+4 - The MDS replies with a read lock on the layout.
+
+The LDLM_ENQUEUE reply that the MDS sends back grants the read lock on
+the layout and provides a Lock Value Block (LVB) describing the
+layout of the resource. That layout is from the extended attribute
+'trusted.lov' and has the structure 'lov_mds_md'.
+
+.LDLM_ENQUEUE Intent:Layout Reply Packet Structure
+image::ldlm-enqueue-intent-layout-reply.png["LDLM_ENQUEUE Intent:Layout reply Packet Structure",height=50]
+
+//////////////////////////////////////////////////////////////////////
+The ldlm-enqueue-intent-layout-reply.png diagram resembles this text
+art:
+
+ LDLM_ENQUEUE:
+ --intent:layout reply--------------------
+ | ptlrpc_body | ldlm_reply | lov_mds_md |
+ -----------------------------------------
+//////////////////////////////////////////////////////////////////////
+
+5 - The client issues an OST_SETATTR with the updated times, which are
+maintained on the OST.
+
+At last the client can send an update to the OST. The OST_SETATTR RPC
+has an 'ost_body' structure.
+
+.OST_SETATTR Request Packet Structure
+image::ost-setattr-request.png["OST_SETATTR Request Packet Structure",height=50]
+
+//////////////////////////////////////////////////////////////////////
+The ost-setattr-request.png diagram resembles this text art:
+
+ OST_SETATTR:
+ --request-----------------
+ | ptlrpc_body | ost_body |
+ --------------------------
+//////////////////////////////////////////////////////////////////////
+
+The 'ost_body' structure is documented in <<struct-ost-body>>. In
+this case the 'o_valid' field is 0x300400f, so the valid fields are
+given by:
+
+.Flags for 'o_valid' field of 'struct os_body'
+[options="header"]
+|====
+| Flag | Meaning
+| OBD_MD_FLID | FID
+| OBD_MD_FLATIME | atime attribute
+| OBD_MD_FLMTIME | mtime attribute
+| OBD_MD_FLCTIME | ctime attribute
+| OBD_MD_FLGENER | generation
+| OBD_MD_FLGROUP | group
+| OBD_MD_FLFID |
+|====
+
+6 - The OST acknowledges the update.
+
+The reply RPC for the OST_SETATTR operation has the same form as the
+request.
+
+.OST_SETATTR Reply Packet Structure
+image::ost-setattr-reply.png["OST_SETATTR Reply Packet Structure",height=50]
+
+//////////////////////////////////////////////////////////////////////
+The ost-setattr-reply.png diagram resembles this text art:
+
+ OST_SETATTR:
+ --reply-------------------
+ | ptlrpc_body | ost_body |
+ --------------------------
+//////////////////////////////////////////////////////////////////////
+
+The OST_SETATTR reply acknowledges the update and sends back an
+'o_valid' of 0x10007bf, which indicates the fields:
+
+.Flags for 'o_valid' field of 'struct os_body'
+[options="header"]
+|====
+| Flag | Meaning
+| OBD_MD_FLID | OST ID
+| OBD_MD_FLATIME | atime attribute
+| OBD_MD_FLMTIME | mtime attribute
+| OBD_MD_FLCTIME | ctime attribute
+| OBD_MD_FLSIZE | size attribute
+| OBD_MD_FLBLOCKS | blocks attribute
+| OBD_MD_FLMODE | mode attribute
+| OBD_MD_FLTYPE | type attribute
+| OBD_MD_FLUID | UID attribute
+| OBD_MD_FLGID | GID attribute
+| OBD_MD_FLGROUP | group
+|====
+
+Changing the Size Attribute
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If the size is being set (as in a "truncate" command) then the client
+(Client1) will issue an LDLM_ENQUEUE to the OST for a write lock on
+the extent attributes of the resource. If another client (Client2) had
+a lock on the resource, then before the OST can grant the lock to
+Client1 it has to interact with Client2. The OST sends an
+LDLM_BL_CALLBACK request to Client2 asking Client 2 to finish up with
+the lock it has. Client2 replies with a simple acknowledgment. When
+Client2 is no longer using the lock it will send an LDLM_CANEL RPC to
+the OST. At that point the OST grants the original request sending an
+LDLM_CP_CALLBACK request to Client1 to notify it. With that taken care
+of Client1 is finally able to issue the OST_PUNCH request that
+actually modifies the size attribute of the affected
+resources. Meanwhile, the OST also replies to Client2 acknowledging
+its LDLM_CANCEL.
+
+.Setattr RPCs for Changing the Resource's Size Attribute
+[[truncate-rpcs]]
+image::truncate_rpcs.png["setattr RPCs for the size attribute",height=250]
+
+//////////////////////////////////////////////////////////////////////
+The truncate_rpcs.png diagram resembles this text art:
+
+Time
+Step Client1 MDT OST Client2
+ ------- ------- ------- -------
+1 MDS_REINT------->
+2 <-------MDS_REINT
+3 LDLM_ENQUEUE----------------->
+4 LDLM_BL_CALLBACK---->
+5 <----LDLM_BL_CALLBACK
+6 <-----------------LDLM_ENQUEUE
+7 <--------LDLM_CANCEL
+8 <-----------LDLM_CP_CALLBACK
+9 LDLM_CP_CALLBACK----------->
+10 OST_PUNCH-------------------->
+11 LDLM_CANCEL------>
+12 <--------------------OST_PUNCH
+//////////////////////////////////////////////////////////////////////
+
+1 - The client issues an MDS_REINT with the REINT_SETATTR sub-operation.
+
+The MDS_REINT request RPC closely resembles the one described above,
+but in this case the 'setattr' wants to modify the size attribute on the
+resource. The 'mdt_rec_setattr' again identifies the resource with the
+'sa_fid' field, and the 'sa_valid' field is set to 0x2002168:
+
+.Flags for 'sa_valid' field of 'struct mdt_rec_setattr'
+[options="header"]
+|====
+| Flag | Meaning
+| MDS_ATTR_SIZE | size attribute
+| MDS_ATTR_MTIME | mtime attribute
+| MDS_ATTR_CTIME | ctime attribute
+| MDS_ATTR_MTIME_SET | mtime being set
+| MDS_ATTR_CTIME_SET | ctime being set
+|====
+
+The size and time values are put in the corresponding fields of the
+'mdt_rec_setattr', and the other attribute fields will be ignored.
+
+There is again an 'ldlm_request' structure in the RPC, but in this
+case it is empty (all fields set to zero), so no early lock
+cancellation.
+
+2 - The MDS_REINT reply acknowledges the updated times.
+
+The MDS_REINT reply is identical to the previous cases in every way,
+including which valid attributes it echoes back.
+
+3 - The client asks the OST for a write lock of type LDLM_EXTENT.
+
+The 'ldlm_request' asks for a write lock with the lock descriptor
+resource's type set to LDLM_EXTENT, the policy data covering the whole
+file, and the lock handle set to identify this request. The rest of
+the lock request is blank (zeroes). The RPC resembles the simplest
+request form in <<ldlm-enqueue-rpc>>.
+
+4 - The OST contacts Client2 to ask for the return of the lock.
+
+The LDLM_BL_CALLBACK is initiated by the OST and sent to the client,
+identifying the resource in question. The content of the ldlm_request
+is otherwise identical to the one sent from Client1 to the OST
+('l_req_mode' == LCK_PW, 'l_granted_mode' == LCK_MINMODE).
+
+.LDLM_BL_CALLBACK Request Packet Structure
+image::ldlm-bl-callback-request.png["LDLM_BL_CALLBACK Request Packet Structure", height=50]
+
+//////////////////////////////////////////////////////////////////////
+The ldlm-bl-callback-request.png diagram resembles this text
+art:
+
+ LDLM_BL_CALLBACK:
+ --request---------------------
+ | ptlrpc_body | ldlm_request |
+ ------------------------------
+//////////////////////////////////////////////////////////////////////
+
+5 - Client2 acknowledges the request and returns the lock.
+
+The LDLM_BL_CALLBACK is an "empty" RPC in that it only has the
+LDLM_BL_CALLBACK opcode and no other content beyond the
+'ptlrpc_body'.
+
+.LDLM_BL_CALLBACK Reply Packet Structure
+image::ldlm-bl-callback-reply.png["LDLM_BL_CALLBACK Reply Packet Structure",height=50]
+
+//////////////////////////////////////////////////////////////////////
+The ldlm-bl-callback-reply.png diagram resembles this text
+art:
+
+ LDLM_BL_CALLBACK:
+ --reply--------
+ | ptlrpc_body |
+ ---------------
+//////////////////////////////////////////////////////////////////////
+
+Its effect is to notify the OST that the lock has been returned.
+
+6 - The OST replies acknowleging the lock request.
+
+The ldlm_reply's lock descriptor acknowledges the request for an
+extent write lock without granting it ('l_req_mode' == LCK_PW,
+'l_granted_mode' == LCK_MINMODE, 'lock_flags' == 0x2 ==
+LDLM_FL_BLOCK_GRANTED, it is not granted because it is
+blocked). Additional attribute data accompanies the LDLM_ENQUEUE reply
+to tell the client about the resource attributes on the OST.
+
+.LDLM_ENQUEUE Extent LVB Reply Packet Structure
+image::ldlm-enqueue-extent-lvb-reply.png["LDLM_ENQUEUE Extent LVB reply Packet Structure",height=50]
+
+//////////////////////////////////////////////////////////////////////
+The ldlm-enqueue-intent-lvb-reply.png diagram resembles this text
+art:
+
+ LDLM_ENQUEUE:
+ --extent lvb reply--------------------
+ | ptlrpc_body | ldlm_reply | ost_lvb |
+ --------------------------------------
+//////////////////////////////////////////////////////////////////////
+
+7 - Client2 cancels its lock
+
+Having received an LDLM_BL_CALLBACK Client2 must finish up with its
+lock. Once it does it sends an LDLM_CANCEL request to the OST to
+signal that it is done.
+
+.LDLM_CANCEL Request Packet Structure
+image::ldlm-cancel-request.png["LDLM_CANCEL Request Packet Structure",height=50]
+
+//////////////////////////////////////////////////////////////////////
+The ldlm-cancel-request.png diagram resembles this text art:
+
+ LDLM_CANCEL:
+ --request---------------------
+ | ptlrpc_body | ldlm_request |
+ ------------------------------
+//////////////////////////////////////////////////////////////////////
+
+The 'ldlm_request' indicates which lock is being canceled in its
+(first) 'lock_handle' field. The OST then looks for anyone else
+waiting on that lock, which it finds is Client1. It waits to reply to
+Client2 with an LDLM_CANCEL reply until after it has notified Client1.
+
+8 - The OST notifies Client1 that it now has the lock.
+
+The 'ldlm_request' structure now has the granted mode set to protected
+write. It also sends along any updated attributes as, for example, if
+Client1 had flushed its dirty write cache.
+
+.LDLM_CP_CALLBACK Request Packet Structure
+image::ldlm-cp-callback-request.png["LDLM_CP_CALLBACK Request Packet Structure",height=50]
+
+//////////////////////////////////////////////////////////////////////
+The ldlm-cp-callback-request.png diagram resembles this text
+art:
+
+ LDLM_CP_CALLBACK:
+ --request-------------------------------
+ | ptlrpc_body | ldlm_request | ost_lvb |
+ ----------------------------------------
+//////////////////////////////////////////////////////////////////////
+
+9 - Client1 acknowledges the lock update.
+
+The reply is "empty" in this case as well. The opcode in the
+'ptlrpc_body' is sufficient to inform the OST that Client1 got its
+lock update.
+
+.LDLM_CP_CALLBACK Reply Packet Structure
+image::ldlm-cp-callback-reply.png["LDLM_CP_CALLBACK Reply Packet Structure",height=50]
+
+//////////////////////////////////////////////////////////////////////
+The ldlm-cp-callback-reply.png diagram resembles this text
+art:
+
+ LDLM_CP_CALLBACK:
+ --reply--------
+ | ptlrpc_body |
+ ---------------
+//////////////////////////////////////////////////////////////////////
+
+10 - Client1 issues an OST_PUNCH request.
+
+As with the OST_SETATTR RPC there is an 'ost_body' structure.
+
+.OST_PUNCH Request Packet Structure
+image::ost-punch-request.png["OST_PUNCH Request Packet Structure",height=50]
+
+//////////////////////////////////////////////////////////////////////
+The ost-punch-request.png diagram resembles this text art:
+
+ OST_PUNCH:
+ --request-----------------
+ | ptlrpc_body | ost_body |
+ --------------------------
+//////////////////////////////////////////////////////////////////////
+
+In this case the 'o_valid' field is 0x30403d:
+
+.Flags for 'o_valid' field of 'struct os_body'
+[options="header"]
+|====
+| Flag | Meaning
+| OBD_MD_FLID | OST ID
+| OBD_MD_FLMTIME | mtime attribute
+| OBD_MD_FLCTIME | ctime attribute
+| OBD_MD_FLSIZE | size attribute
+| OBD_MD_FLBLOCKS | blocks attribute
+| OBD_MD_FLGENER | generation
+| OBD_MD_FLCKSUM | checksukm
+| OBD_MD_FLQOS | quality of service
+|====
+
+11 - The OST acknowledges the LDLM_CANCEL (step 7) from Client2
+
+The OST finishes up with the lock cancel (after having notified
+Client1) by replying to Clietn2. This happens asynchronously with the
+arrival of the OST_PUNCH request, and in <<truncate-rpcs>> it is shown
+occuring after the OST_PUNCH, but that is not required.
+
+.LDLM_CANCEL Reply Packet Structure
+image::ldlm-cancel-reply.png["LDLM_CANCEL Reply Packet Structure",height=50]
+
+//////////////////////////////////////////////////////////////////////
+The ldlm-cancel-reply.png diagram resembles this text art:
+
+ LDLM_CANCEL:
+ --reply--------
+ | ptlrpc_body |
+ ---------------
+//////////////////////////////////////////////////////////////////////
+
+The LDLM_CANCEL reply is a so-called "empty" RPC. Its only purpose is
+to acknowldge receipt of the LDLM_CANCEL request.
+
+12 - The OST an OST_PUNCH reply.
+
+The OST_PUNCH reply also resembles the OST_SETATTR reply:
+
+.OST_PUNCH Reply Packet Structure
+image::ost-punch-reply.png["OST_PUNCH Reply Packet Structure",height=50]
+
+//////////////////////////////////////////////////////////////////////
+The ost-punch-reply.png diagram resembles this text art:
+
+ OST_PUNCH:
+ --reply-------------------
+ | ptlrpc_body | ost_body |
+ --------------------------
+//////////////////////////////////////////////////////////////////////
+
+The 'o_valid' field is 0x1, so only the 'o_id' field is
+interpreted. It just acknowledges the requested change has been
+made.
+
--- /dev/null
+Terms and Symbols
+-----------------
+
+These are some terms and symbols used in this document. Many are
+explained in more detail later in the document.
+
+Client, Client1, Client2, MDT, OST, MDS, OSS::
+The entities exchanging messages. Clients are hosts mounting the
+Lustre file system. The MDT and OST are services being provided by the
+Lustre servers MDS and OSS.
+
+resource::
+A file or directory that is the object of a file system operation,
+either on the namespace or on its data.
+
+RPC request::
+RPCs are exchanged in pairs (with a very few exceptions). The request
+initiates the operation being carried out by the RPC pair.
+
+RPC reply::
+The matching reply that completes an operation implemented by an RPC
+pair.
+
+MDS_REINT::
+An RPC pair that implements an operation that will change the
+information on an MDT. See <<mds-reint-setattr-rpc>>.
+
+LDLM_ENQUEUE::
+An RPC pair that implements distributed locking across the servers and
+clients. See <<ldlm-enqueue-rpc>>.
+
+MDS_GETXATTR::
+An RPC pair that provides information about extended attributes for a
+resource. See <<mds-getxattr-rpc>>.
+
+OST_SETATTR::
+An RPC pair that sets resource attributes. See <<ost-setattr-rpc>>.
+
+LDLM_BL_CALLBACK::
+An RPC pair that assists with getting a lock back from an entity that
+has it. See <<ldlm-bl-callback-rpc>>.
+
+LDLM_CP_CALLBACK::
+An RPC pair that notifies an entity that a requested lock is now
+available. See <<ldlm-cp-callback-rpc>>.
+
+OST_PUNCH::
+An RPC pair that modifies the size attribute of a resource on an
+OST. See <<ost-punch-rpc>>.
+
+'ptlrpc_body'::
+The first section (structure) of each RPC. It identifies which RPC
+operation (opcode) the RPC conveys.
+