--- /dev/null
+Mount
+~~~~~
+
+Before any other interaction can take place between a client and a
+Lustre file system the client must 'mount' the file system, and Lustre
+services must already be in place (on the servers). A file system
+mount may be initiated at the Linux shell command line, which in turn
+invokes the 'mount()' system call. Kernel modules for Lustre
+exchange a series of messages with the servers, beginning with
+messages that retrieve details about the file system from the
+management server (MGS). This provides the client with the identities
+of all the metadata servers (MDSs) and targets (MDTs) as well as all
+the object storage servers (OSSs) and targets (OSTs). The client then
+sequences through each of the targets exchanging additional messages
+to initiate the connections with them. The following sections present
+the details of the Lustre operations that accomplish the file system
+mount.
+
+Messages Between the Client and the MGS
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In order to be able to mount the Lustre file system the client needs
+to know the identities of the various servers and targets so that it
+can initiate connections to them. The following sequence of operations
+accomplishes this.
+
+.Connect RPCs Between the Client and the MGS
+[[client-mgs-connect-rpcs]
+image::client_mgs_connect_rpcs.png["Connect RPCs Between the Client and the MGS", height=600]
+
+//////////////////////////////////////////////////////////////////////
+The client_mgs_connect_rpcs.png diagram resembles this text art:
+
+Time
+Step Client MGS MDT OST
+ ------- ------- --- ---
+1 MGS_CONNECT-------------------->
+2 <--------------------MGS_CONNECT
+3 LDLM_ENQUEUE------------------->
+4 <-------------------LDLM_ENQUEUE
+5 LLOG_ORIGIN_HANDLE_CREATE------>
+6 <------LLOG_ORIGIN_HANDLE_CREATE
+7 LDLM_ENQUEUE------------------->
+8 <-------------------LDLM_ENQUEUE
+9 LLOG_ORIGIN_HANDLE_CREATE------>
+10 <------LLOG_ORIGIN_HANDLE_CREATE
+11 LLOG_ORIGIN_HANDLE_READ_HEADER->
+12 <-LLOG_ORIGIN_HANDLE_READ_HEADER
+13 LLOG_ORIGIN_HANDLE_NEXT_BLOCK->
+14 <-LLOG_ORIGIN_HANDLE_NEXT_BLOCK
+15 LDLM_ENQUEUE------------------->
+16 <-------------------LDLM_ENQUEUE
+17 MGS_CONFIG_READ---------------->
+18 <----------------MGS_CONFIG_READ
+19 LDLM_ENQUEUE------------------->
+20 <-------------------LDLM_ENQUEUE
+21 LLOG_ORIGIN_HANDLE_CREATE------>
+22 <------LLOG_ORIGIN_HANDLE_CREATE
+23 LLOG_ORIGIN_HANDLE_READ_HEADER->
+24 <-LLOG_ORIGIN_HANDLE_READ_HEADER
+//////////////////////////////////////////////////////////////////////
+
+Prior to any other interaction between a client and a Lustre server
+(or between two servers) the client must establish a 'connection'. The
+connection establishes shared state between the two hosts. On the
+client this connection state information is called an 'import', and
+there is an import on the client for each target it connects to. On
+the server this connection state is referred to as an 'export', and
+again the server has an export for each client that has connected to
+it. There a separate export for each client for each target.
+
+The client begins by carrying out the MGS_CONNECT Lustre operation,
+which establishes the connection (creates the import and the export)
+between the client and the MGS. The connect message from the client
+includes a 'handle' to uniquely identify itself (subsequent messages
+to the LDLM will refer to that client-handle). The connection data
+from the client also proposes the set of <<connect-flags,connection
+flags>> appropriate to connecting to an MGS.
+
+.Flags for the client connection to an MGS
+[options="header"]
+|====
+| obd_connect_data->ocd_connect_flags
+| OBD_CONNECT_VERSION
+| OBD_CONNECT_AT
+| OBD_CONNECT_FULL20
+| OBD_CONNECT_IMP_RECOV
+| OBD_CONNECT_MNE_SWAB
+| OBD_CONNECT_PINGLESS
+|====
+
+The MGS's reply to the connection request will include the handle that
+the server and client will both use to identify this connection in
+subsequent messages. This is the 'connection-handle' (as opposed to
+the client-handle mentioned a moment ago). The MGS also replies with
+the same set of connection flags.
+
+Once the connection is established the client gets configuration
+information for the file system from the MGS in four stages. First,
+the two exchange messages establishing the file system wide security
+policy that will be followed in all subsequent communications. Second,
+the client gets a bitmap instructing it as to which among the
+configuration records on the MGS it needs. Third, reading those
+records from the MGS gives the client the list of all the servers and
+targets it will need to communicate with. Fourth, the client reads
+cluster wide configuration data (the sort that might be set at the
+client command line with a 'lctl conf_param' command). The following
+paragraphs go into these four stages in more detail.
+
+Each time the client is going to read information from server storage
+it needs to first acquire the appropriate lock. Since the client is
+only reading data, the locks will be 'concurrent read' locks. The
+LDLM_ENQUEUE command communicates this lock request to the MGS
+target. The request identifies the target via the connection-handle
+from the connection reply, and identifies the client (itself) with the
+client-handle from its original connection request. The MGS's reply
+grants that lock, if appropriate. If other clients were making some
+sort of modification to the MGS data then the lock exchange might
+result in a delay while the client waits. More details about the
+behavior of the <<ldlm,Distributed Lock Manager>> are in that
+section. For now, let's assume the locks are granted for each of these
+four operations. The first LLOG_ORIGIN_HANDLE_CREATE operation (the
+client is creating its own local handle not the target's file) asks
+for the security configuration file ("lfs-sptlrpc"). <<security>>
+discusses security, and for now let's assume there is nothing to be
+done for security. That is, subsequent messages will all use an "empty
+security flavor" and no encryption will take place. In this case the
+MGS's reply ('pb_status' == -2, ENOENT) indicated that there was no
+such file, so nothing actually gets read.
+
+Another LDLM_ENQUEUE and LLOG_ORIGIN_HANDLE_CREATE pair of operations
+identifies the configuration client data ("lfs-client") file, and in
+this case there is data to read. The LLOG_ORIGIN_HANDLE_CREATE reply
+identifies the actual object of interest on the MGS via the
+'llog_logid' field in the 'struct llogd_body'. The MGS stores
+configuration data in log records. A header at the beginning of
+"lfs-client" uses a bitmap to identify the log records that are
+actually needed. The header includes both which records to retrieve
+and how large those records are. The LLOG_ORIGIN_HANDLE_READ_HEADER
+request uses the 'llog_logid' to identify desired log file, and the
+reply provides the bitmap and size information identifying the
+records that are actually needed. The
+LLOG_ORIGIN_HANDLE_NEXT_BLOCK operations retrieves the data thus
+identified.
+
+Knowing the specific configuration records it wants, the client then
+proceeds to retrieve them. This requires another LDLM_ENQUEUE
+operation, followed this time by the MGS_CONFIG_READ operation, which
+get the UUIDs for the servers and targets from the configuration log
+("lfs-cliir").
+
+A final LDLM_ENQUEUE, LLOG_ORIGIN_HANDLE_CREATE, and
+LLOG_ORIGIN_HANDLE_READ_HEADER then retrieve the cluster wide
+configuration data ("params").
+
+Messages Between the Client and the MDSs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+After the foregoing interaction with the MGS the client has a list of
+the MDSs and MDTs in the file system. Next, the client invokes four
+Lustre operations with each MDT on the list.
+
+.Connect RPCs Between the Client and each MDT
+[[client-mdt-connect-rpcs]
+image::client_mdt_connect_rpcs.png["Connect RPCs Between the Client and each MDT", height=200]
+
+//////////////////////////////////////////////////////////////////////
+The client_mdt_connect_rpcs.png diagram resembles this text art:
+
+Time
+Step Client MGS MDT OST
+ ------- ------- -------- ------
+1 MDS_CONNECT-------------------->
+2 <--------------------MDS_CONNECT
+3 MDS_STATFS--------------------->
+4 <---------------------MDS_STATFS
+5 MDS_GETSTATUS------------------>
+6 <------------------MDS_GETSTATUS
+7 MDS_GETATTR-------------------->
+8 <--------------------MDS_GETATTR
+//////////////////////////////////////////////////////////////////////
+
+The MDS_CONNECT operation establishes a connection between the client
+and a specific target (MDT) on an MDS. Thus, if an MDS has multiple
+targets, there is a separate MDS_CONNECT operation for each. This
+creates an import for the target on the client and an export for the
+client and target on the MDS. As with the connect operation for the
+MGS, the connect message from the client includes a UUID to uniquely
+identify this connection, and subsequent messages to the lock manager
+on the server will refer to that UUID. The connection data from the
+client also proposes the set of <<connect-flags,connection flags>>
+appropriate to connecting to an MDS. The following are the flags
+always included.
+
+.Always included flags for the client connection to an MDS
+[options="header"]
+|====
+| obd_connect_data->ocd_connect_flags
+| OBD_CONNECT_RDONLY
+| OBD_CONNECT_VERSION
+| OBD_CONNECT_ACL
+| OBD_CONNECT_XATTR
+| OBD_CONNECT_IBITS
+| OBD_CONNECT_NODEVOH
+| OBD_CONNECT_ATTRFID
+| OBD_CONNECT_CANCELSET
+| OBD_CONNECT_AT
+| OBD_CONNECT_RMT_CLIENT
+| OBD_CONNECT_RMT_CLIENT_FORCE
+| OBD_CONNECT_BRW_SIZE
+| OBD_CONNECT_MDS_CAPA
+| OBD_CONNECT_OSS_CAPA
+| OBD_CONNECT_MDS_MDS
+| OBD_CONNECT_FID
+| LRU_RESIZE_CONNECT_FLAG
+| OBD_CONNECT_VBR
+| OBD_CONNECT_LOV_V3
+| OBD_CONNECT_SOM
+| OBD_CONNECT_FULL20
+| OBD_CONNECT_64BITHASH
+| OBD_CONNECT_JOBSTATS
+| OBD_CONNECT_EINPROGRESS
+| OBD_CONNECT_LIGHTWEIGHT
+| OBD_CONNECT_UMASK
+| OBD_CONNECT_LVB_TYPE
+| OBD_CONNECT_LAYOUTLOCK
+| OBD_CONNECT_PINGLESS
+| OBD_CONNECT_MAX_EASIZE
+| OBD_CONNECT_FLOCK_DEAD
+| OBD_CONNECT_DISP_STRIPE
+| OBD_CONNECT_LFSCK
+| OBD_CONNECT_OPEN_BY_FID
+| OBD_CONNECT_DIR_STRIPE
+|====
+
+.Optional flags for the client connection to an MDS
+[options="header"]
+|====
+| obd_connect_data->ocd_connect_flags
+| OBD_CONNECT_SOM
+| OBD_CONNECT_LRU_RESIZE
+| OBD_CONNECT_ACL
+| OBD_CONNECT_UMASK
+| OBD_CONNECT_RDONLY
+| OBD_CONNECT_XATTR
+| OBD_CONNECT_XATTR
+| OBD_CONNECT_RMT_CLIENT_FORCE
+|====
+
+The MDS replies to the connect message with a subset of the flags
+proposed by the client, and the client notes those values in its
+import. The MDS's reply to the connection request will include a UUID
+that the server and client will both use to identify this connection
+in subsequent messages.
+
+The client next uses an MDS_STATFS operation to request 'statfs'
+information from the target, and that data is returned in the reply
+message. The actual fields closely resemble the results of a 'statfs'
+system call. See the 'obd_statfs' structure in the <<data-structs,Data
+Structures and Defines Section>>.
+
+The client uses the MDS_GETSTATUS operation to request information
+about the mount point of the file system. fixme: Does MDS_GETSTATUS
+only ask about the root (so it would seem)? The server reply contains
+the 'fid' of the root directory of the file system being mounted. If
+there is a security policy the capabilities of that security policy
+are included in the reply.
+
+The client then uses the MDS_GETATTR operation to get get further
+information about the root directory of the file system. The request
+message includes the above fid. It will also include the security
+capability (if appropriate). The reply also holds the same fid, and in
+this case the 'mdt_body' has several additional fields filled
+in. These include the mtime, atime, ctime, mode, uid, and gid. It also
+includes the size of the extended attributes and the size of the ACL
+information. The reply message also includes the extended attributes
+and the ACL. From the extended attributes the client can find out
+about striping information for the root, if any.
+
+Messages Between the Client and the OSSs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Additional CONNECT messages flow between the client and each OST
+enumerated by the MGS.
+
+.Connect RPCs Between the Client and each OST
+[[client-ost-connect-rpcs]
+image::client_ost_connect_rpcs.png["Connect RPCs Between the Client and each OST", height=75]
+
+//////////////////////////////////////////////////////////////////////
+The client_ost_connect_rpcs.png diagram resembles this text art:
+
+Time
+Step Client MGS MDT OST
+ ------- ------- -------- ------
+1 OST_CONNECT----------------------------------->
+2 <-----------------------------------OST_CONNECT
+//////////////////////////////////////////////////////////////////////
+
+
git://git.whamcloud.com / doc / protocol.git / blobdiff