[[early-lock-cancellation]]
In some circumstances (see for example <<mds-reint-setattr-rpc>>) a
-client holding
-a lock "knows" that the operation it is initiating will necessitate
-canceling a lock that it holds. In such a circumstance the lock is
-said to be "in conflict" with the operation. For example, a side
-effect of GNU 'fileutils' is that it often issues a 'stat()' on a file
-prior to other update operations such as 'unlink()', so when issuing
-an update to a resource attribute on the MDT
-a) the client will already have a read lock on the resource, and
-b) the MDT will then need a lock on that same
-resource in order to perform the update. Rather than waiting for an
-additional round of RPCs to carry out the lock cancellation (on the
-client) the RPC requesting the update can include a cancellation of
-one or more locks as part of its request. This proactive lock cancellation
-is called "early lock cancellation" or sometimes ELC.
+client holding a lock "knows" that the operation it is initiating will
+necessitate canceling a lock that it holds. In such a circumstance the
+lock is said to be "in conflict" with the operation. For example, a
+side effect of GNU 'fileutils' is that it often issues a 'stat()' on a
+file prior to other update operations such as 'unlink()', so when
+issuing an update to a resource attribute on the MDT a) the client
+will already have a read lock on the resource, and b) the MDT will
+then need a lock on that same resource in order to perform the
+update. Rather than waiting for an additional round of RPCs to carry
+out the lock cancellation (on the client) the RPC requesting the
+update can include a cancellation of one or more locks as part of its
+request. This proactive lock cancellation is called "early lock
+cancellation" or sometimes ELC.
ELC can also proactively cancel unused cached locks on the client
from the lock LRU list to avoid the need for later lock callbacks
+++ /dev/null
-#FIG 3.2 Produced by xfig version 3.2.5b
-Landscape
-Center
-Inches
-Letter
-100.00
-Single
--2
-1200 2
-2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
- 3000 900 3000 975 3000 1050 3000 1125 3000 1200 3000 1275
- 3000 1350 3000 1425 3000 1500
-2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
- 5325 900 5325 975 5325 1050 5325 1125 5325 1200 5325 1275
- 5325 1350 5325 1425 5325 1500
-2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
- 2025 900 7200 900 7200 1500 1200 1500 1200 1050
-2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
- 1875 2025 4650 2025 4650 2625 1275 2625 1275 2175
-2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
- 3075 2025 3075 2100 3075 2175 3075 2250 3075 2325 3075 2400
- 3075 2475 3075 2550 3075 2625
-4 0 0 50 -1 16 18 0.0000 4 270 4155 1050 675 MDS_REINT:REINT_SETATTR\001
-4 0 0 50 -1 16 18 0.0000 4 255 930 1050 1005 request\001
-4 0 0 50 -1 16 18 0.0000 4 270 1530 1350 1305 ptlrpc_body\001
-4 0 0 50 -1 16 18 0.0000 4 270 2025 3150 1305 mdt_rec_setattr\001
-4 0 0 50 -1 16 18 0.0000 4 270 1620 5400 1320 ldlm_request\001
-4 0 0 50 -1 16 18 0.0000 4 270 615 1200 2100 reply\001
-4 0 0 50 -1 16 18 0.0000 4 270 1530 1425 2400 ptlrpc_body\001
-4 0 0 50 -1 16 18 0.0000 4 270 1305 3225 2400 mdt_body\001
Single
-2
1200 2
-2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
- 1875 975 4650 975 4650 1575 1275 1575 1275 1125
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
3075 975 3075 1050 3075 1125 3075 1200 3075 1275 3075 1350
3075 1425 3075 1500 3075 1575
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 1875 900 9525 900 9525 1500 1275 1500 1275 1050
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 4575 900 4575 975 4575 1050 4575 1125 4575 1200 4575 1275
+ 4575 1350 4575 1425 4575 1500
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 5850 900 5850 975 5850 1050 5850 1125 5850 1200 5850 1275
+ 5850 1350 5850 1425 5850 1500
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 6450 900 6450 975 6450 1050 6450 1125 6450 1200 6450 1275
+ 6450 1350 6450 1425 6450 1500
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 7950 900 7950 975 7950 1050 7950 1125 7950 1200 7950 1275
+ 7950 1350 7950 1425 7950 1500
4 0 0 50 -1 16 18 0.0000 4 270 4155 1050 675 MDS_REINT:REINT_SETATTR\001
4 0 0 50 -1 16 18 0.0000 4 270 615 1200 1050 reply\001
4 0 0 50 -1 16 18 0.0000 4 270 1530 1425 1350 ptlrpc_body\001
4 0 0 50 -1 16 18 0.0000 4 270 1305 3225 1350 mdt_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 1065 4650 1350 mdt_md\001
+4 0 0 50 -1 16 18 0.0000 4 210 375 6000 1350 acl\001
+4 0 0 50 -1 16 18 0.0000 4 270 1275 6600 1350 fid1_capa\001
+4 0 0 50 -1 16 18 0.0000 4 270 1275 8025 1350 fid2_capa\001
5325 900 5325 975 5325 1050 5325 1125 5325 1200 5325 1275
5325 1350 5325 1425 5325 1500
2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
- 2025 900 7200 900 7200 1500 1200 1500 1200 1050
+ 2025 900 8775 900 8775 1500 1200 1500 1200 1050
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 4
+ 5700 1500 5700 2100 1200 2100 1200 1500
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 6900 900 6900 975 6900 1050 6900 1125 6900 1200 6900 1275
+ 6900 1350 6900 1425 6900 1500
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 2325 1500 2325 1575 2325 1650 2325 1725 2325 1800 2325 1875
+ 2325 1950 2325 2025 2325 2100
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 3975 1500 3975 1575 3975 1650 3975 1725 3975 1800 3975 1875
+ 3975 1950 3975 2025 3975 2100
4 0 0 50 -1 16 18 0.0000 4 270 4155 1050 675 MDS_REINT:REINT_SETATTR\001
4 0 0 50 -1 16 18 0.0000 4 255 930 1050 1005 request\001
4 0 0 50 -1 16 18 0.0000 4 270 1530 1350 1305 ptlrpc_body\001
4 0 0 50 -1 16 18 0.0000 4 270 2025 3150 1305 mdt_rec_setattr\001
-4 0 0 50 -1 16 18 0.0000 4 270 1620 5400 1320 ldlm_request\001
+4 0 0 50 -1 16 18 0.0000 4 210 900 1350 1875 eadata\001
+4 0 0 50 -1 16 18 0.0000 4 270 1440 2475 1875 llog_cookie\001
+4 0 0 50 -1 16 18 0.0000 4 270 1620 4050 1875 ldlm_request\001
+4 0 0 50 -1 16 18 0.0000 4 270 1485 5400 1275 lustre_capa\001
+4 0 0 50 -1 16 18 0.0000 4 270 1665 6975 1275 mdt_ioepoch\001
Single
-2
1200 2
-2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
- 1875 975 4650 975 4650 1575 1275 1575 1275 1125
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
3075 975 3075 1050 3075 1125 3075 1200 3075 1275 3075 1350
3075 1425 3075 1500 3075 1575
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 1875 900 9525 900 9525 1500 1275 1500 1275 1050
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 4575 900 4575 975 4575 1050 4575 1125 4575 1200 4575 1275
+ 4575 1350 4575 1425 4575 1500
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 5850 900 5850 975 5850 1050 5850 1125 5850 1200 5850 1275
+ 5850 1350 5850 1425 5850 1500
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 6450 900 6450 975 6450 1050 6450 1125 6450 1200 6450 1275
+ 6450 1350 6450 1425 6450 1500
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 7950 900 7950 975 7950 1050 7950 1125 7950 1200 7950 1275
+ 7950 1350 7950 1425 7950 1500
4 0 0 50 -1 16 18 0.0000 4 270 615 1200 1050 reply\001
4 0 0 50 -1 16 18 0.0000 4 270 1530 1425 1350 ptlrpc_body\001
4 0 0 50 -1 16 18 0.0000 4 270 1305 3225 1350 mdt_body\001
+4 0 0 50 -1 16 18 0.0000 4 270 1065 4650 1350 mdt_md\001
+4 0 0 50 -1 16 18 0.0000 4 210 375 6000 1350 acl\001
+4 0 0 50 -1 16 18 0.0000 4 270 1275 6600 1350 fid1_capa\001
+4 0 0 50 -1 16 18 0.0000 4 270 1275 8025 1350 fid2_capa\001
4 0 0 50 -1 16 18 0.0000 4 270 4350 1050 675 MDS_REINT:REINT_SETXATTR\001
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
3000 900 3000 975 3000 1050 3000 1125 3000 1200 3000 1275
3000 1350 3000 1425 3000 1500
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 5325 900 5325 975 5325 1050 5325 1125 5325 1200 5325 1275
+ 5325 1350 5325 1425 5325 1500
2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 5
- 2025 900 7200 900 7200 1500 1200 1500 1200 1050
+ 2025 900 8775 900 8775 1500 1200 1500 1200 1050
+2 1 0 2 0 7 50 -1 -1 0.000 0 0 7 0 0 4
+ 5700 1500 5700 2100 1200 2100 1200 1500
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 6900 900 6900 975 6900 1050 6900 1125 6900 1200 6900 1275
+ 6900 1350 6900 1425 6900 1500
+2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
+ 2325 1500 2325 1575 2325 1650 2325 1725 2325 1800 2325 1875
+ 2325 1950 2325 2025 2325 2100
2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 9
- 5400 900 5400 975 5400 1050 5400 1125 5400 1200 5400 1275
- 5400 1350 5400 1425 5400 1500
+ 3975 1500 3975 1575 3975 1650 3975 1725 3975 1800 3975 1875
+ 3975 1950 3975 2025 3975 2100
4 0 0 50 -1 16 18 0.0000 4 255 930 1050 1005 request\001
4 0 0 50 -1 16 18 0.0000 4 270 1530 1350 1305 ptlrpc_body\001
+4 0 0 50 -1 16 18 0.0000 4 210 900 1350 1875 eadata\001
+4 0 0 50 -1 16 18 0.0000 4 270 1440 2475 1875 llog_cookie\001
+4 0 0 50 -1 16 18 0.0000 4 270 1620 4050 1875 ldlm_request\001
+4 0 0 50 -1 16 18 0.0000 4 270 1485 5400 1275 lustre_capa\001
+4 0 0 50 -1 16 18 0.0000 4 270 1665 6975 1275 mdt_ioepoch\001
4 0 0 50 -1 16 18 0.0000 4 270 4350 1050 675 MDS_REINT:REINT_SETXATTR\001
4 0 0 50 -1 16 18 0.0000 4 270 2175 3150 1305 mdt_rec_setxattr\001
-4 0 0 50 -1 16 18 0.0000 4 270 1620 5475 1275 ldlm_request\001
early reply::
A partial message reply returned in response to a request message
-that allows the target to indicate to the requestor to increase its
+that allows the target to indicate to the requester to increase its
timeout value without otherwise indicating the request has progressed.
export::
granted.
.LDLM_ENQUEUE Request Packet Structure
-image::ldlm-enqueue-request.png["LDLM_ENQUEUE Request Packet Structure",height=75]
+image::ldlm-enqueue-request.png["LDLM_ENQUEUE Request Packet Structure",height=50]
//////////////////////////////////////////////////////////////////////
The ldlm-enqueue-request.png diagram resembles this text
Here is another example of an intent, in this case the 'getxattr' intent.
.LDLM_ENQUEUE Intent:Getxattr Request Packet Structure
-image::ldlm-enqueue-intent-getxattr-request.png["LDLM_ENQUEUE Intent:Getxattr Request Packet Structure",height=50]
+image::ldlm-enqueue-intent-getxattr-request.png["LDLM_ENQUEUE Intent:Getxattr Request Packet Structure",height=75]
//////////////////////////////////////////////////////////////////////
The ldlm-enqueue-intent-getxattr-request.png diagram resembles this text
-----------
[[lustre-rpcs]]
-Lustre RPC operations are implemented as a pair of messages, a request
-and its reply. The 'pb_type' field is set to PTLRPC_MSG_REQUEST for
-requests initiating the operation, and normally PTLRPC_MSG_REPLY for
-replies unless the message encountered a fatal error before it could
-be processed, in which case it will contain PTLRPC_MSG_ERR.
+The Lustre protocol consists of a collection of remote procedure calls
+(RPCs) collectively known as PtlRPCs. Each Lustre RPC is an operation
+that consists of a pair of messages, a request and its reply. The
+'pb_type' field is set to PTLRPC_MSG_REQUEST for requests initiating
+the operation, and normally PTLRPC_MSG_REPLY for replies unless the
+message encountered a fatal error before it could be processed, in
+which case it will contain PTLRPC_MSG_ERR.
The type of operation requested is denoted by the 'pb_opc' op-code
field of the RPC request. Note that as a general matter, the receipt
MGS.
'client_uuid'::
-A string with the client's own UUID. This is also a
+A string with the client's own UUID. This is also an
<<struct-obd-uuid>>. The target sets the 'exp_client_uuid' field of
its 'eport' structure to this value.
'lustre_handle'::
See <<struct-lustre-handle>>. This 'lustre_handle' is distinct from
the 'pb_handle' field in the 'ptlrpc_body'. This 'lustre_handle'
-provied a unique 'cookie' to identify this client for this connection
+provides a unique 'cookie' to identify this client for this connection
attempt. After a disconnect, a subsequent connect RPC will have a
different value. The target preserves this cookie in the 'exp_handle'
field of its 'obd_export' structure for this client. In that way it
'ptlrpc_body'::
RPC descriptor. See <<struct-ptlrpc-body>>. The 'pb_handle' field has
-the value, $C_t$, uniquely identifying that target. The value $C_t$ is
+the value, C~t~, uniquely identifying that target. The value C~t~ is
preserved in the 'imp_remote_handle' field of the client's import
structure for this target. See <<struct-obd-import>>.
'mdt_body'::
Metadata about the resource. See <<struct-mdt-body>>.
-MDS_MD::
+MDT_MD::
Needs more detail.
ACL::
~~~~~~~~~~~~~~~~~
[[mds-reint-rpc]]
-An RPC that implements an operation that will change the state of
-an object on an MDT. There are a variety of operations all gathered
-under the MDS_REINT 'opcode'.
-
+An MDS_REINT RPC ('pb_opc' = 36) is one that implements an operation
+that will change the state of an object on an MDT. There are a variety
+of operations all gathered under the MDS_REINT 'opcode'. The possible
+values for the 'mdt_rec_reint' struct's 'rr_opcode' field give types
+of operations that are encoded as MDS_REINT RPCs:
+
+.MDS Reint Opcodes ('rr_opcode')
+****
+[source,c]
----
typedef enum {
REINT_SETATTR = 1,
REINT_OPEN = 6,
REINT_SETXATTR = 7,
REINT_RMENTRY = 8,
- REINT_MIGRATE = 9,
- REINT_MAX
-} mds_reint_t, mdt_reint_t;
+ REINT_MIGRATE = 9
+};
----
+****
include::struct_mdt_rec_reint.txt[]
^^^^^^^^^^^^^^^^^
[[mds-reint-setattr-rpc]]
-An RPC that implements the 'setattr' sub-command of the MDS_REINT.
+The REINT_SETATTR is an RPC that implements the 'setattr' sub-command
+of the MDS_REINT.
.MDS_REINT:REINT_SETATTR Request Packet Structure
-image::mds-reint-setattr-request.png["MDS_REINT:REINT_SETATTR Request Packet Structure",height=50]
+image::mds-reint-setattr-request.png["MDS_REINT:REINT_SETATTR Request Packet Structure",height=75]
//////////////////////////////////////////////////////////////////////
The mds-reint-setattr-request.png diagram resembles this text art:
include::struct_mdt_rec_setattr.txt[]
+After the 'mdt_rec_setattr' there are several more buffers defined for
+the RPC, but those before the 'ldlm_request' are often unused and
+appear as zero-length buffers in the header.
+
'lustre_capa'::
A "capabilities" structure. See <<struct-lustre-capa>>.
'mdt_md'::
Layout data for the resource. This buffer is optional and will appear
-as zero length in some packets.
+as zero length in some packets. This needs to be further explained.
'acl'::
Access control list data. This buffer is optional and will appear as
An RPC that implements the 'setxattr' sub-command of the MDS_REINT.
.MDS_REINT:REINT_SETXATTR Request Packet Structure
-image::mds-reint-setxattr-request.png["MDS_REINT:REINT_SETXATTR Request Packet Structure",height=50]
+image::mds-reint-setxattr-request.png["MDS_REINT:REINT_SETXATTR Request Packet Structure",height=75]
//////////////////////////////////////////////////////////////////////
The mds-reint-setxattr-request.png diagram resembles this text art:
include::struct_mdt_rec_setxattr.txt[]
+Retruning to the remaining buffers in the REINT_SETXATTR RPC we again
+have several optional buffers followed by the 'ldlm_request'.
+
'lustre_capa'::
-A "capabilities" structure. See <<struct-lustre-capa>>.
+A "capabilities" structure. See <<struct-lustre-capa>>. This buffer is
+optional and will appear as zero length in some packets.
'mdt_ioepoch'::
Identifying "epoch" information. This buffer is optional and will
MGS.
'client_uuid'::
-A string with the client's own UUID. This is also a
+A string with the client's own UUID. This is also an
<<struct-obd-uuid>>. The target sets the 'exp_client_uuid' field of
-its 'eport' structure to this value.
+its 'export' structure to this value.
'lustre_handle'::
See <<struct-lustre-handle>>. This 'lustre_handle' is distinct from
the 'pb_handle' field in the 'ptlrpc_body'. This 'lustre_handle'
-provied a unique 'cookie' to identify this client for this connection
+provides a unique 'cookie' to identify this client for this connection
attempt. After a disconnect, a subsequent connect RPC will have a
different value. The target preserves this cookie in the 'exp_handle'
field of its 'obd_export' structure for this client. In that way it
'ptlrpc_body'::
RPC descriptor. See <<struct-ptlrpc-body>>. The 'pb_handle' field has
-the value, $C_t$, uniquely identifying that target. The value $C_t$ is
+the value, C~t~, uniquely identifying that target. The value C~t~ is
preserved in the 'imp_remote_handle' field of the client's import
structure for this target. See <<struct-obd-import>>.
include::struct_ptlrpc_body.txt[]
+[NOTE]
+The RPC descriptor just described is the first buffer in the
+OST_SETATTR request message. It will also appear as the first buffer
+of every other RPC request or reply sent between entities. Refer back
+to the above discussion when examining other RPC messages. The next
+section describes the 'struct ost_body' which is the second (and
+final) buffer in an OST_SETATTR request message.
+
include::struct_ost_body.txt[]
+The above discussion described the structure of the OST_SETATTR
+request message. In this case the reply message sturcture looks much
+the same. It uses the same to buffers, and oly changes their contents
+to reflect that the message is a reply rather than a requsest.
+
.OST_SETATTR Reply Packet Structure
image::ost-setattr-reply.png["OST_SETATTR Reply Packet Structure",height=50]
//////////////////////////////////////////////////////////////////////
'ptlrpc_body'::
-RPC descriptor. See <<struct-ptlrpc-body>>.
+The RPC descriptor. See <<struct-ptlrpc-body>>.
'ost_body'::
-Metadata about the resource. See <struct-ost-body>>.
+Metadata about the resource. See <<struct-ost-body>>.
A file identifier ('FID') is a 128-bit number that uniquely identifies a
resource on an MDT or OST. The FID for the resource on an MDT
identifies the file as a whole. The FID for a resource on an OST
-identifiess the fragment of a file assigned to that OST.
+identifies the fragment of a file assigned to that OST.
-The 'f_seq' field holds the sequence number, or SEQ, and is used in
-conjunction with the 'FID location database' (FLDB) to
-determine on which target the resource is located. All resources with the
-same 'f_seq' value will be on the same target. A target can have more
-than one 'f_seq' value assigned to it.
+The 'f_seq' field holds the sequence number, or SEQ (see below), and
+is used in conjunction with the 'FID location database' (FLDB) to
+determine on which target the resource is located. All resources with
+the same 'f_seq' value will be on the same target. A target can have
+more than one 'f_seq' value assigned to it.
-The 'f_oid' field holds the unique 'object identifier' (OID) within the
-sequence to identify a particular object.
+The 'f_oid' field holds the unique 'object identifier' (OID) to
+identify a particular object.
The 'f_ver' value identifies which version of a resource is being
identified in the event that the resource is being updated, and
----
****
-There are several reserved ranges of FID sequence values, to allow for
-interoperability with older Lustre filesystems, to identify "well known"
-objects for internal or external use, as well as for future expansion.
+There are several reserved ranges of FID sequence values
+(summarized in the list above), to allow for interoperability with
+older Lustre filesystems, to identify "well known" objects for
+internal or external use, as well as for future expansion.
The 'FID_SEQ_OST_MDT0' (0x0) range is reserved for OST objects created
by MDT0 in non-DNE filesystems. Since all such OST objects used an
The 'FID_SEQ_LLOG_NAME' (0x10) range is used for named llog files such
as configuration logs and the ChangeLog.
-The 'FID_SEQ_IGIF' (0xb-0xffffffff) range is reserved for 'inode and
-generation in FID' (IGIF) inodes allocated by MDSes before Lustre 2.0.
+The 'FID_SEQ_IGIF' (0xb-0xffffffff) range is reserved for 'inode
+generation in FID' (IGIF) inodes allocated by MDSs before Lustre 2.0.
This corresponds to the 4 billion maximum inode number that could be
allocated for such filesystems. The 'f_oid' field for IGIF FIDs
contains the inode version number, and as such there is normally only
~~~~~~~~~~~~~~~~~~~~~
[[struct-lustre-msg]]
-Every message has an initial header that informs the receiver about
-the number of buffers and their size for the rest of the message to
-follow, along with other important information about the request or
-reply message.
+Every RPC has a header that informs the receiver about the number and
+sizes of the buffers to follow. It also holds a few generic pieces of
+information about the message.
[source,c]
----
The 'lm_bufcount' field holds the number of buffers that will follow
the header. The header and sequence of buffers constitutes one
-message. Each of the buffers is a sequence of bytes whose contents
-corresponds to one of the structures described in this section. Each
-message will always have at least one buffer, and no message can have
-more than thirty-one buffers.
+message. Each message will always have at least one buffer, and no
+message can have more than thirty-one buffers.
The 'lm_secflvr' field gives an indication of whether any sort of
cyptographic encoding of the subsequent buffers will be in force. The
-value is zero if there is no "crypto" and gives a code identifying the
-"flavor" of crypto if it is employed. Further, if crypto is employed
-there will only be one buffer following (i.e. 'lm_bufcount' = 1), and
-that buffer holds an encoding of what would otherwise have been the
-sequence of buffers normally following the header. Cryptography will
-be discussed in a separate chapter.
+value is zero if there is no "crypto". Otherwise, the value gives a
+code identifying the "flavor" of crypto. Further, if crypto is
+employed there will only be one buffer following (i.e. 'lm_bufcount' =
+1), and that buffer holds an encoding of what would otherwise have
+been the sequence of buffers normally following the header.
+Cryptography will be discussed in a separate chapter. See
+<<security>>.
-The 'lm_magic' field is a "magic" value (LUSTRE_MSG_MAGIC_V2 = 0x0BD00BD3,
-'OBD' for 'object based device') that is
-checked in order to positively identify that the message is intended
-for the use to which it is being put. That is, we are indeed dealing
-with a Lustre message, and not, for example, corrupted memory or a bad
-pointer.
+The 'lm_magic' field is a "magic" value (LUSTRE_MSG_MAGIC_V2 =
+0x0BD00BD3, 'OBD' for 'object based device') that is checked in order
+to positively identify that the message is intended for the use to
+which it is being put. That is, we are indeed dealing with a Lustre
+message, and not, for example, corrupted memory or a bad pointer.
The 'lm_repsize' field in a request indicates the maximum available
space that has been reserved for any reply to the request. A reply
to allow the receiver to verify that the message is intact. This is
used to verify that an 'early reply' has not been overwritten by the
actual reply message. If the 'MSGHDR_CKSUM_INCOMPAT18' flag is set
-in requests since Lustre 1.8
-(the server will send early reply messages with the appropriate 'lm_cksum'
-if it understands the flag
-and is mandatory in Lustre 2.8 and later.
+(in requests since Lustre 1.8) and the server understands the flag,
+then the server will send early reply messages with the appropriate
+'lm_cksum'. The 'lm_cksum' is mandatory in Lustre 2.8 and later.
-The 'lm_flags' field contains flags that affect the low-level RPC
-protocol. The 'MSGHDR_AT_SUPPORT' (0x1) bit indicates that the sender
-understands adaptive timeouts and can receive 'early reply' messages
-to extend its waiting period rather than timing out. This flag was
-introduced in Lustre 1.6. The 'MSGHDR_CKSUM_INCOMPAT18' (0x2) bit
+The 'lm_flags' field contains flags that affect the protocol. The
+'MSGHDR_AT_SUPPORT' (0x1) bit indicates that the sender understands
+adaptive timeouts and can receive 'early reply' messages. This flag
+was introduced in Lustre 1.6. The 'MSGHDR_CKSUM_INCOMPAT18' (0x2) bit
indicates that 'lm_cksum' is computed on the full 'ptlrpc_body'
message buffer rather than on the original 'ptlrpc_body_v2' structure
-size (88 bytes). It was introduced in Lustre 1.8 and is mandatory
-for all requests in Lustre 2.8 and later.
+size (88 bytes). It was introduced in Lustre 1.8 and is mandatory for
+all requests in Lustre 2.8 and later.
The 'lm_padding*' fields are reserved for future use.
include::struct_lu_fid.txt[]
-The 'mbo_handle' field indicates the identity of an open file related
-to the operation, if any. If there is no lock then it is just 0.
+*Returning to the 'mdt_body' structure the remaining fields are as
+follows.*
+
+The 'mbo_handle' field indicates the identity of the previously
+acquired lock for an open file, if any, that is the subject of the
+operation. If there is no lock then it is just 0.
The 'mbo_valid' field identifies which of the remaining fields are
actually in force. The flags in 'mbo_valid' are:
Which flag is associated with which field is indicated in the comments
beside the fields in the struct definition above. The fields without
-corresponding flags in the 'mbo_valid' field should be considered
-valid, though I'm not sure how much they actually get used.
+corresponding flags in the 'mbo_valid' field are also 'valid'.
Many of the fields have names and meanings that correspond directly to
those in 'struct mdt_rec_setattr' (ex. 'atime', 'size', 'mode',
----
The 'rr_opcode' field defines one among the several sub-commands for
-MDS REINT RPCs. Those opcodes are:
-
-.MDS Reint Type
-****
-[source,c]
-----
-typedef enum {
- REINT_SETATTR = 1,
- REINT_CREATE = 2,
- REINT_LINK = 3,
- REINT_UNLINK = 4,
- REINT_RENAME = 5,
- REINT_OPEN = 6,
- REINT_SETXATTR = 7,
-} mds_reint_t;
-----
-****
-
-Based on that opcode one of the variants of the structure will
-actually be used. See <<mds-reint-setattr-rpc>> for one example.
+MDS REINT RPCs. See the list above. Based on that opcode one of the
+variants of the structure will actually be used. See
+<<mds-reint-setattr-rpc>> for one example.
The 'rr_cap' field is not used (and has been dropped in more recent
code updates).
the other '_h' fields.
The 'rr_fsgid' field gives the GID of the file system mount point.
-For each opcode there is a variant of the 'mdt_rec_reint' that has
-identical byte fields, but slightly modified semantics. The following
-is for the REINT_SETATTR sub-operation.
The 'rr_suppgid1' and 'rr_suppgid2' fields are supplementary GID
information for kernels that have it.
<<struct-lu-fid>>) for the resources being operated upon. If only one
resource is being acted upon then 'rr_fid2' is not used.
-The 'rr_mtime' 'rr_atime' and 'rr_ctime' fields give the object
-timestamps in seconds for the last modification time, the last
+The 'rr_mtime', 'rr_atime', and 'rr_ctime' fields give the object
+timestamps in microseconds for the last modification time, the last
access time, and the last metadata change time, respectively.
The 'rr_size' field gives the value of the size attribute.
There is only one FID to be operated upon, and the 'sa_valid',
'sa_uid', and 'sa_gid' values take the place of the second 'struct
-lu_fid' in the generic <<struct-mdt-rec-reint>>. See also
-<<struct-lu-fid>>.
+lu_fid' in the generic 'mdt_rec_reint'. See also <<struct-lu-fid>>.
The 'sa_valid' field identifies which of the other fields in the
structure are to be honored. If the corresponding flag bit is not set
The 'os_bfree' field is the number of blocks not currently in use.
The 'os_bavail' is the number of blocks available to be allocated to
-new files. The number of available blocks is typically lower than
-the number of free blocks due to <<grant>> and blocks reserved for
-target internal usage such as metadata.
+new files. The number of available blocks is typically lower than the
+number of free blocks due to outstanding "grants" (see <<grant>>) and
+blocks reserved for target internal usage such as metadata.
The 'os_files' field is the total number of files on the target, both
allocated and free. For some OSD types this is a static number, and
set. If the file system has been set to read-only, either manually at
mount or automatcially due to detected corruption of the underlying
target filesystem, then 'os_state' is returned with OS_STATE_READONLY (0x2)
-set, for example if it was explicitly mounted read-only, or
-corruption has been detected at runtime in the backing filesystem.
+set.
The 'os_fprecreated' field counts the number of pre-created objects
available on an OST. The 'os_fprecreated' value counts as "used"
The 'o_handle' field is a 'lustre_handle' (See
<<struct-lustre-handle>>). In this context it represents a lock. In
-most cases it will be 0. In an OST_PUNCH RPC (See <<truncate-rpcs>>) it
-may refer to a lock on the resource being modified that was previously
-acquired, though a lock is not required for that operation. The effect
-is to extend the timeout for that lock.
+most cases it will be 0. In an OST_PUNCH RPC (See <<truncate-rpcs>>)
+it may have a (non-zero) value. if so that 'lustre_handle' refers to a
+lock on the resource being modified, and that had been previously
+acquired (a lock is not required for that operation). The effect is to
+extend the timeout for that lock.
****
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[[struct-ptlrpc-body]]
-Every Lustre message starts with a header (See <<struct-lustre-msg>>)
-describing a few generic details about the RPC, the most important of
-which is the number and sizes of additional "buffers" that will
-follow. The buffers just organize subsections of the RPC, and each
-comprises a set of fields. The fields of a buffer are presented here
-as 'struct' definitions because that is how they are actually
-organized, and it provides a convenient way to refer to groups of
-fields that appear together in many different RPCs. The first buffer
-in every RPC is given by the 'ptlrpc_body' structure, also known as
-the RPC descriptor. This descriptor identifies the type of the RPC via
-a Lustre operation code. The value of that 'opcode', as well as
-whether it is an RPC 'request' or 'reply', determines what other
-buffers will be in the message following the descriptor.
+Every Lustre message starts with a header (not shown - see
+<<struct-lustre-msg>>) describing a few generic details about the RPC,
+the most important of which is the number and sizes of additional
+"buffers" that will follow. The buffers just organize subsections of
+the RPC, and each comprises a set of fields. The fields of given a buffer
+are presented together as 'struct' definitions in order to show how they
+are organized, and it provides a convenient way to refer to
+groups of fields that appear together in many different RPCs. The
+first buffer in every RPC is given by the 'ptlrpc_body' structure,
+also known as the RPC descriptor. This descriptor identifies the type
+of the RPC via a Lustre operation code. The value of that 'opcode', as
+well as whether it is an RPC 'request' or 'reply', determines what
+other buffers will be in the message following the descriptor.
[source,c]
----
connection request 'pb_handle' is set to a new 'lustre_handle' that
uniquely identifies the target (indeed, it uniquely identifies the
'export', since each client gets a unique 'lustre_handle' for a given
-target). Subsequent request messages sent this client to this target
+target). Subsequent request messages sent from this client to this target
will use that 'lustre_handle' (the 'pb_handle' field will be set to
that value) to to gain access to their shared state. In subsequent RPC
-reply messages (after teh *_CONNECT reply) the 'pb_handle' field is
+reply messages (after the *_CONNECT reply) the 'pb_handle' field is
0. The 'lustre_handle' is persistent across client reconnects to the
same instance of the server, but if the client unmounts the filesystem
or is evicted then it must re-connect as a new client, with a new
and is ready to finish recovery.
The 'pb_op_flags' values are specific to a particular 'pb_opc', but
-are currently only used by the *_CONNECT RPCs.The 'pb_op_flags' value
+are currently only used by the *_CONNECT RPCs. The 'pb_op_flags' value
for connect operations governs the client connection status state
machine.
git://git.whamcloud.com / doc / protocol.git / commitdiff