there is an import on the client for each target it connects to. On
the server this connection state is referred to as an 'export', and
again the server has an export for each client that has connected to
-it. There a separate export for each client for each target.
+it.
+
+*1 - Client issues an MGS_CONNECT to the MGS.*
+
+.MGS_CONNECT Request Packet Structure
+image::mgs-connect-request.png["MGS_CONNECT Request Packet Structure",height=50]
+
+//////////////////////////////////////////////////////////////////////
+The mgs-connect-request.png diagram resembles this text art:
+
+ MGS_CONNECT:
+ --request--------------------------------------------
+ | ptlrpc_body | obd_uuid | obd_uuid | lustre_handle |
+ -----------------------------------------------------
+ | obd_connect_data |
+ ---------------------
+//////////////////////////////////////////////////////////////////////
The client begins by carrying out the <<mgs-connect-rpc,MGS_CONNECT>>
-Lustre operation, which establishes the connection (creates the
+Lustre RPC, which establishes the connection (creates the
import and the export) between the client and the MGS. The connect
message from the client includes a 'lustre_handle' to uniquely
identify itself. Subsequent request messages to the MGS will refer
| OBD_CONNECT_PINGLESS
|====
+*2 - The MGS sends an MGS_CONNECT reply to the client.*
+
+.MGS_CONNECT Reply Packet Structure
+image::mgs-connect-reply.png["MGS_CONNECT Reply Packet Structure",height=50]
+
+//////////////////////////////////////////////////////////////////////
+The mgs-connect-reply.png diagram resembles this text art:
+
+ MGS_CONNECT:
+ --reply---------------------------
+ | ptlrpc_body | obd_connect_data |
+ ----------------------------------
+//////////////////////////////////////////////////////////////////////
+
The MGS's reply to the connection request will include the handle that
the server and client will both use to identify this connection in
subsequent messages. This is the 'connection-handle' (as opposed to
Once the connection is established the client gets configuration
information for the file system from the MGS in four stages. First,
the two exchange messages establishing the file system wide security
-policy that will be followed in all subsequent communications. Second,
-the client gets a configuration <<llog>> starting with a bitmap
-instructing it as to which among the
-configuration records on the MGS it needs. Third, reading those
-records from the MGS gives the client the list of all the servers and
-targets it will need to communicate with. Fourth, the client reads
-the cluster wide configuration data (the sort that might be set at the
-client command line with a 'lctl conf_param' command). The following
-paragraphs go into these four stages in more detail.
+policy that will be followed in all subsequent communications.
Each time the client is going to read information from server storage
it needs to first acquire the appropriate lock. Since the client is
sort of modification to the MGS data then the lock exchange might
result in a delay while the client waits. More details about the
behavior of the <<ldlm,Distributed Lock Manager>> are in that
-section. For now, let's assume the locks are granted for each of these
-four operations. The first LLOG_ORIGIN_HANDLE_CREATE operation (the
-client is creating its own local handle not the target's file) asks
+section.
+
+*3 - The Client sends an LDLM_ENQUEUE to the MGS.*
+
+.LDLM_ENQUEUE Request Packet Structure
+image::ldlm-enqueue-request.png["LDLM_ENQUEUE Request Packet Structure",height=50]
+
+//////////////////////////////////////////////////////////////////////
+The ldlm-enqueue-request.png diagram resembles this text art:
+
+ LDLM_ENQUEUE:
+ --request---------------------
+ | ptlrpc_body | ldlm_request |
+ ------------------------------
+//////////////////////////////////////////////////////////////////////
+
+The client seeks a 'concurrent read' lock on the MGS.
+
+*4 - The MGS sends an LDLM_ENQUEUE reply to the client.*
+
+.LDLM_ENQUEUE Reply Packet Structure
+image::ldlm-enqueue-reply.png["LDLM_ENQUEUE Reply Packet Structure",height=50]
+
+//////////////////////////////////////////////////////////////////////
+The ldlm-enqueue-reply.png diagram resembles this text
+art:
+
+ LDLM_ENQUEUE:
+ --reply---------------------
+ | ptlrpc_body | ldlm_reply |
+ ----------------------------
+//////////////////////////////////////////////////////////////////////
+
+The MGS grants that lock.
+
+*5 - The Client sends an LLOG_ORIGIN_HANDLE_CREATE to the MGS.*
+
+
+.LLOG_ORIGIN_HANDLE_CREATE Request Packet Structure
+image::llog-origin-handle-create-request.png["LLOG_ORIGIN_HANDLE_CREATE Request Packet Structure",height=50]
+
+//////////////////////////////////////////////////////////////////////
+The llog-origin-handle-create-request.png diagram resembles this text
+art:
+
+ LLOG_ORIGIN_HANDLE_CREATE:
+ --request----------------------------
+ | ptlrpc_body | llogd_body | string |
+ -------------------------------------
+//////////////////////////////////////////////////////////////////////
+
+The first LLOG_ORIGIN_HANDLE_CREATE operation asks
for the security configuration file ("lfs-sptlrpc"). <<security>>
-discusses security, and for now let's assume there is nothing to be
-done for security. That is, subsequent messages will all use an "empty
-security flavor" and no encryption will take place. In this case the
-MGS's reply ('pb_status' == -2, ENOENT) indicated that there was no
-such file, so nothing actually gets read.
-
-Another LDLM_ENQUEUE and LLOG_ORIGIN_HANDLE_CREATE pair of operations
-identifies the configuration client data ("lfs-client") file, and in
-this case there is data to read. The LLOG_ORIGIN_HANDLE_CREATE reply
-identifies the actual object of interest on the MGS via the
-'llog_logid' field in the 'struct llogd_body'. The MGS stores
+discusses security.
+
+*6 - The MGS sends an LLOG_ORIGIN_HANDLE_CREATE reply to the client.*
+
+
+.LLOG_ORIGIN_HANDLE_CREATE Reply Packet Structure
+image::llog-origin-handle-create-reply.png["LLOG_ORIGIN_HANDLE_CREATE Reply Packet Structure",height=50]
+
+//////////////////////////////////////////////////////////////////////
+The llog-origin-handle-create-reply.png diagram resembles this text
+art:
+
+ LLOG_ORIGIN_HANDLE_CREATE:
+ --reply---------------------
+ | ptlrpc_body | llogd_body |
+ ----------------------------
+//////////////////////////////////////////////////////////////////////
+
+In this case there is nothing to be done for security. That is,
+subsequent messages will all use an "empty security flavor" and no
+encryption will take place. In this case the MGS's reply ('pb_status'
+== -2, ENOENT) indicated that there was no such file, so nothing
+actually gets read.
+
+Next, the client gets a configuration <<llog>> starting with a bitmap
+instructing it as to which among the configuration records on the MGS
+it needs. Another LDLM_ENQUEUE and LLOG_ORIGIN_HANDLE_CREATE pair of
+operations identifies the configuration client data ("lfs-client")
+file.
+
+*7 - The Client sends an LDLM_ENQUEUE to the MGS.*
+
+The client again seeks a 'concurrent read' lock on the MGS.
+
+*8 - The MGS sends an LDLM_ENQUEUE reply to the client.*
+
+The MGS grants that lock.
+
+*9 - The Client sends an LLOG_ORIGIN_HANDLE_CREATE to the MGS.*
+
+The client asks for the 'lfs-client' log file, which holds a bitmap
+indicating the available configuration records.
+
+*10 - The MGS sends an LLOG_ORIGIN_HANDLE_CREATE reply to the client.*
+
+In this case there is data to read. The LLOG_ORIGIN_HANDLE_CREATE
+reply identifies the actual object of interest on the MGS via the
+'llog_logid' field in the 'struct llogd_body'.
+
+*11 - The Client sends an LLOG_ORIGIN_HANDLE_READ_HEADER to the MGS.*
+
+.LLOG_ORIGIN_HANDLE_READ_HEADER Request Packet Structure
+image::llog-origin-handle-read-header-request.png["LLOG_ORIGIN_HANDLE_READ_HEADER Request Packet Structure",height=50]
+
+//////////////////////////////////////////////////////////////////////
+The llog-origin-handle-read-header-request.png diagram resembles this
+text art:
+
+ LLOG_ORIGIN_HANDLE_READ_HEADER:
+ --request-------------------
+ | ptlrpc_body | llogd_body |
+ ----------------------------
+//////////////////////////////////////////////////////////////////////
+
+The client asks for that log file's header. The MGS stores
configuration data in log records. A header at the beginning of
"lfs-client" uses a bitmap to identify the log records that are
actually needed. The header includes both which records to retrieve
and how large those records are. The LLOG_ORIGIN_HANDLE_READ_HEADER
request uses the 'llog_logid' to identify desired log file, and the
reply provides the bitmap and size information identifying the
-records that are actually needed. The
-LLOG_ORIGIN_HANDLE_NEXT_BLOCK operations retrieves the data thus
-identified.
+records that are actually needed.
+
+*12 - The MGS sends an LLOG_ORIGIN_HANDLE_READ_HEADER reply to the client.*
+
+.LLOG_ORIGIN_HANDLE_READ_HEADER Reply Packet Structure
+image::llog-origin-handle-read-header-reply.png["LLOG_ORIGIN_HANDLE_READ_HEADER Reply Packet Structure",height=50]
+
+//////////////////////////////////////////////////////////////////////
+The llog-origin-handle-read-header-reply.png diagram resembles this
+text art:
+
+ LLOG_ORIGIN_HANDLE_READ_HEADER:
+ --reply--------------------------
+ | ptlrpc_body | llog_log_header |
+ ---------------------------------
+//////////////////////////////////////////////////////////////////////
+
+The MGs responds with the header, which holds the actual bitmap.
+
+*13 - The Client sends an LLOG_ORIGIN_HANDLE_NEXT_BLOCK to the MGS.*
+
+.LLOG_ORIGIN_HANDLE_NEXT_BLOCK Request Packet Structure
+image::llog-origin-handle-next-block-request.png["LLOG_ORIGIN_HANDLE_NEXT_BLOCK Request Packet Structure",height=50]
+
+//////////////////////////////////////////////////////////////////////
+The llog-origin-handle-next-block-request.png diagram resembles this
+text art:
+
+ LLOG_ORIGIN_HANDLE_NEXT_BLOCK:
+ --request-------------------
+ | ptlrpc_body | llogd_body |
+ ----------------------------
+//////////////////////////////////////////////////////////////////////
+
+The client asks for the configuration record.
+
+*14 - The MGS sends an LLOG_ORIGIN_HANDLE_NEXT_BLOCK reply to the client.*
+
+.LLOG_ORIGIN_HANDLE_NEXT_BLOCK Reply Packet Structure
+image::llog-origin-handle-next-block-reply.png["LLOG_ORIGIN_HANDLE_NEXT_BLOCK Reply Packet Structure",height=50]
+
+//////////////////////////////////////////////////////////////////////
+The llog-origin-handle-next-block-reply.png diagram resembles this
+text art:
+
+ LLOG_ORIGIN_HANDLE_NEXT_BLOCK:
+ --reply------------------------------
+ | ptlrpc_body | llogd_body | eadata |
+ -------------------------------------
+//////////////////////////////////////////////////////////////////////
+
+The MGS replies with the details of where in the configuration data to
+find the desired details.
Knowing the specific configuration records it wants, the client then
proceeds to retrieve them. This requires another LDLM_ENQUEUE
get the UUIDs for the servers and targets from the configuration log
("lfs-cliir").
-A final LDLM_ENQUEUE, LLOG_ORIGIN_HANDLE_CREATE, and
-LLOG_ORIGIN_HANDLE_READ_HEADER then retrieve the cluster wide
-configuration data ("params").
+*15 - The Client sends an LDLM_ENQUEUE to the MGS.*
+
+The client again seeks a 'concurrent read' lock on the MGS.
+
+*16 - The MGS sends an LDLM_ENQUEUE reply to the client.*
+
+The MGS grants that lock.
+
+*17 - The Client sends an MGS_CONFIG_READ to the MGS.*
+
+The client identifies the desired record in the 'lfs-cliir' file,
+which contains the current details of the configuration for this file
+system.
+
+*18 - The MGS sends an MGS_CONFIG_READ reply to the client.*
+
+The MGS responds with the actual configuration data. This gives the
+client the list of all the servers and targets it will need to
+communicate with.
+
+Finally, the client reads the cluster wide configuration data (the
+sort that might be set at the client command line with a 'lctl
+conf_param' command). The following paragraphs go into these four
+stages in more detail.
+
+*19 - The Client sends an LDLM_ENQUEUE to the MGS.*
+
+The client again seeks a 'concurrent read' lock on the MGS.
+
+*20 - The MGS sends an LDLM_ENQUEUE reply to the client.*
+
+The MGS grants that lock.
+
+*21 - The Client sends an LLOG_ORIGIN_HANDLE_CREATE to the MGS.*
+
+The client asks for the 'params' log file.
+
+*22 - The MGS sends an LLOG_ORIGIN_HANDLE_CREATE reply to the client.*
+
+The MGS responds that the log file is available.
+
+*23 - The Client sends an LLOG_ORIGIN_HANDLE_READ_HEADER to the MGS.*
+
+The client asks for that log file's header.
+
+*24 - The MGS sends an LLOG_ORIGIN_HANDLE_READ_HEADER reply to the client.*
+
+The MGs responds with the header, which holds the actual bitmap. In
+this case there are no 'params' to report.
Messages Between the Client and the MDSs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
and a specific target (MDT) on an MDS. Thus, if an MDS has multiple
targets, there is a separate MDS_CONNECT operation for each. This
creates an import for the target on the client and an export for the
-client and target on the MDS. As with the connect operation for the
-MGS, the connect message from the client includes a UUID to uniquely
-identify this connection, and subsequent messages to the lock manager
-on the server will refer to that UUID. The connection data from the
-client also proposes the set of <<connect-flags,connection flags>>
-appropriate to connecting to an MDS. The following are the flags
-always included.
+client and target on the MDS.
+
+*1 - Client issues an MDS_CONNECT to each MDT.*
+
+.MDS_CONNECT Request Packet Structure
+image::mds-connect-request.png["MDS_CONNECT Request Packet Structure",height=50]
+
+//////////////////////////////////////////////////////////////////////
+The mds-connect-request.png diagram resembles this text art:
+
+ MDS_CONNECT:
+ --request--------------------------------------------
+ | ptlrpc_body | obd_uuid | obd_uuid | lustre_handle |
+ -----------------------------------------------------
+ | obd_connect_data |
+ ---------------------
+//////////////////////////////////////////////////////////////////////
+
+As with the connect operation for the MGS, the connect message from
+the client includes a UUID to uniquely identify this connection, and
+subsequent messages to the lock manager on the server will refer to
+that UUID. The connection data from the client also proposes the set
+of <<connect-flags,connection flags>> appropriate to connecting to an
+MDS. The following are the flags always included.
.Always included flags for the client connection to an MDS
[options="header"]
| OBD_CONNECT_RMT_CLIENT_FORCE
|====
+*2 - The MDS sends an MDS_CONNECT reply to the client.*
+
+.MDS_CONNECT Reply Packet Structure
+image::mds-connect-reply.png["MDS_CONNECT Reply Packet Structure",height=50]
+
+//////////////////////////////////////////////////////////////////////
+The mds-connect-reply.png diagram resembles this text art:
+
+ MDS_CONNECT:
+ --reply---------------------------
+ | ptlrpc_body | obd_connect_data |
+ ----------------------------------
+//////////////////////////////////////////////////////////////////////
+
The MDS replies to the connect message with a subset of the flags
proposed by the client, and the client notes those values in its
import. The MDS's reply to the connection request will include a UUID
that the server and client will both use to identify this connection
in subsequent messages.
+*3 - The Client sends an MDS_STATFS to the MDS.*
+
The client next uses an MDS_STATFS operation to request 'statfs'
information from the target, and that data is returned in the reply
message. The actual fields closely resemble the results of a 'statfs'
system call. See the 'obd_statfs' structure in the <<data-structs,Data
Structures and Defines Section>>.
+*4 - The MDS sends an MDS_STATFS reply to the client.*
+
+The MDS replies witht eh 'statfs' information.
+
+*5 - The Client sends an MDS_GETSTATUS to the MDS.*
+
The client uses the MDS_GETSTATUS operation to request information
-about the mount point of the file system. fixme: Does MDS_GETSTATUS
-only ask about the root (so it would seem)? The server reply contains
-the 'fid' of the root directory of the file system being mounted. If
-there is a security policy the capabilities of that security policy
-are included in the reply.
+about the mount point of the file system.
+
+*6 - The MDS sends an MDS_GETSTATUS reply to the client.*
+
+The server reply contains the 'fid' of the root directory of the file
+system being mounted. If there is a security policy the capabilities
+of that security policy are included in the reply.
+
+*7 - The Client sends an MDS_GETATTR to the MDS.*
The client then uses the MDS_GETATTR operation to get get further
information about the root directory of the file system. The request
-message includes the above fid. It will also include the security
-capability (if appropriate). The reply also holds the same fid, and in
-this case the 'mdt_body' has several additional fields filled
-in. These include the mtime, atime, ctime, mode, uid, and gid. It also
-includes the size of the extended attributes and the size of the ACL
-information. The reply message also includes the extended attributes
-and the ACL. From the extended attributes the client can find out
-about striping information for the root, if any.
+message includes the above fid.
+
+*8 - The MDS sends an MDS_GETATTR reply to the client.*
+
+It will also include the security capability (if appropriate). The
+reply also holds the same fid, and in this case the 'mdt_body' has
+several additional fields filled in. These include the mtime, atime,
+ctime, mode, uid, and gid. It also includes the size of the extended
+attributes and the size of the ACL information. The reply message also
+includes the extended attributes and the ACL. From the extended
+attributes the client can find out about striping information for the
+root, if any.
Messages Between the Client and the OSSs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2 <-----------------------------------OST_CONNECT
//////////////////////////////////////////////////////////////////////
+The OST_CONNECT operation establishes a connection between the client
+and a specific target (OST) on an OST. Thus, if an OST has multiple
+targets, there is a separate OST_CONNECT operation for each. This
+creates an import for the target on the client and an export for the
+client and target on the OST.
+
+*1 - Client issues an OST_CONNECT to each OST.*
+
+.OST_CONNECT Request Packet Structure
+image::ost-connect-request.png["OST_CONNECT Request Packet Structure",height=50]
+
+//////////////////////////////////////////////////////////////////////
+The ost-connect-request.png diagram resembles this text art:
+
+ OST_CONNECT:
+ --request--------------------------------------------
+ | ptlrpc_body | obd_uuid | obd_uuid | lustre_handle |
+ -----------------------------------------------------
+ | obd_connect_data |
+ ---------------------
+//////////////////////////////////////////////////////////////////////
+
+As with the connect operations for the MGS and MDTs, the connect
+message from the client includes a UUID to uniquely identify this
+connection, and subsequent messages to the lock manager on the server
+will refer to that UUID. The connection data from the client also
+proposes the set of <<connect-flags,connection flags>> appropriate to
+connecting to an OST. The following are the flags always included.
+
+.Flags for the client connection to an OST
+[options="header"]
+|====
+| obd_connect_data->ocd_connect_flags
+| OBD_CONNECT_GRANT
+| OBD_CONNECT_SRVLOCK
+| OBD_CONNECT_VERSION
+| OBD_CONNECT_REQPORTAL
+| OBD_CONNECT_TRUNCLOCK
+| OBD_CONNECT_RMT_CLIENT
+| OBD_CONNECT_BRW_SIZE
+| OBD_CONNECT_OSS_CAPA
+| OBD_CONNECT_CANCELSET
+| OBD_CONNECT_AT
+| OBD_CONNECT_LRU_RESIZE
+| OBD_CONNECT_CKSUM
+| OBD_CONNECT_FID
+| OBD_CONNECT_VBR
+| OBD_CONNECT_FULL20
+| OBD_CONNECT_LAYOUTLOCK
+| OBD_CONNECT_64BITHASH
+| OBD_CONNECT_MAXBYTES
+| OBD_CONNECT_JOBSTATS
+| OBD_CONNECT_EINPROGRESS
+| OBD_CONNECT_LVB_TYPE
+| OBD_CONNECT_PINGLESS
+|====
+
+*2 - The OST sends an OST_CONNECT reply to the client.*
+
+.OST_CONNECT Reply Packet Structure
+image::ost-connect-reply.png["OST_CONNECT Reply Packet Structure",height=50]
+
+//////////////////////////////////////////////////////////////////////
+The ost-connect-reply.png diagram resembles this text art:
+
+ OST_CONNECT:
+ --reply---------------------------
+ | ptlrpc_body | obd_connect_data |
+ ----------------------------------
+//////////////////////////////////////////////////////////////////////
+
+The OST replies to the connect message with a subset of the flags
+proposed by the client, and the client notes those values in its
+import. The OST's reply to the connection request will include a UUID
+that the server and client will both use to identify this connection
+in subsequent messages. The flags that the server doesn't include are:
+OBD_CONNECT_RMT_CLIENT, OBD_CONNECT_OSS_CAPA, and
+OBD_CONNECT_PINGLESS.
git://git.whamcloud.com / doc / protocol.git / blobdiff