OBD Connect Data
^^^^^^^^^^^^^^^^
-[[obd-connect-data]]
+[[struct-obd-connect-data]]
An 'obd_connect_data' structure accompanies every connect operation in
-both the request message and in the reply message.
+both the request message and in the reply message. It contains the
+mutually supported features that are negotiated between the client and
+server at *_CONNECT time.
+
+At *_CONNECT time the client sends in its request 'ocd_connect_flags'
+the flags for all features that it understands and intends to use (for
+example 'OBD_CONNECT_RDONLY' is optional depending on client mount
+options). The request also contains other fields that are only valid
+if the matching flag is set. The server replies in 'ocd_connect_flags'
+with the subset of feature flags that it understands and intends to honour.
+The server may set fields in the reply for mutually-understood features.
----
struct obd_connect_data {
capabilities of a connection between client and target. Several of
those flags (noted in comments above and the discussion below)
actually control whether the remaining fields of 'obd_connect_data'
-get used. The [[connect-flags]] flags are:
+get used. The [[obd-connect-flags]] flags are:
----
#define OBD_CONNECT_RDONLY 0x1ULL /*client has read-only access*/
#define OBD_CONNECT_REQPORTAL 0x40ULL /*Separate non-IO req portal */
#define OBD_CONNECT_ACL 0x80ULL /*access control lists */
#define OBD_CONNECT_XATTR 0x100ULL /*client use extended attr */
-#define OBD_CONNECT_CROW 0x200ULL /*MDS+OST create obj on write*/
#define OBD_CONNECT_TRUNCLOCK 0x400ULL /*locks on server for punch */
#define OBD_CONNECT_TRANSNO 0x800ULL /*replay sends init transno */
#define OBD_CONNECT_IBITS 0x1000ULL /*support for inodebits locks*/
-#define OBD_CONNECT_JOIN 0x2000ULL /*files can be concatenated.
- *We do not support JOIN FILE
- *anymore, reserve this flags
- *just for preventing such bit
- *to be reused.*/
#define OBD_CONNECT_ATTRFID 0x4000ULL /*Server can GetAttr By Fid*/
#define OBD_CONNECT_NODEVOH 0x8000ULL /*No open hndl on specl nodes*/
#define OBD_CONNECT_RMT_CLIENT 0x10000ULL /*Remote client */
#define OBD_CONNECT_RMT_CLIENT_FORCE 0x20000ULL /*Remote client by force */
#define OBD_CONNECT_BRW_SIZE 0x40000ULL /*Max bytes per rpc */
#define OBD_CONNECT_QUOTA64 0x80000ULL /*Not used since 2.4 */
-#define OBD_CONNECT_MDS_CAPA 0x100000ULL /*MDS capability */
-#define OBD_CONNECT_OSS_CAPA 0x200000ULL /*OSS capability */
#define OBD_CONNECT_CANCELSET 0x400000ULL /*Early batched cancels. */
#define OBD_CONNECT_SOM 0x800000ULL /*Size on MDS */
#define OBD_CONNECT_AT 0x1000000ULL /*client uses AT */
Each flag corresponds to a particular capability that the client and
target together will honor. A client will send a message including
some subset of these capabilities during a connection request to a
-specific target. It tells the server what capabilities it has. The
+specific target. This tells the server what capabilities it has. The
server then replies with the subset of those capabilities it agrees to
honor (for the given target).
-If the OBD_CONNECT_VERSION flag is set then the 'ocd_version' field is
-honored. The 'ocd_version' gives an encoding of the Lustre
-version. For example, Version 2.7.32 would be hexadecimal number
-0x02073200.
+If the 'OBD_CONNECT_VERSION' flag is set then the 'ocd_version' field is
+valid. The 'ocd_version' gives an encoding of the Lustre
+version. For example, Version 2.7.55 would be hexadecimal number
+0x02075500.
If the OBD_CONNECT_GRANT flag is set then the 'ocd_grant' field is
-honored. The 'ocd_grant' value in a reply (to a connection request)
+valid. The 'ocd_grant' value in a reply (to a connection request)
sets the client's grant.
If the OBD_CONNECT_INDEX flag is set then the 'ocd_index' field is
-honored. The 'ocd_index' value is set in a reply to a connection
-request. It holds the LOV index of the target.
+valid. The 'ocd_index' value is set in a request to hold the LOV
+index of the OST or the LMV index of the MDT. The server should
+refuse connections to targets for which the 'ocd_index' does not
+match the actual target index.
If the OBD_CONNECT_BRW_SIZE flag is set then the 'ocd_brw_size' field
-is honored. The 'ocd_brw_size' value sets the size of the maximum
-supported RPC. The client proposes a value in its connection request,
-and the server's reply will either agree or further limit the size.
+is valid. The 'ocd_brw_size' value sets the maximum supported bulk
+transfer size. The client proposes a value in its connection request,
+and the server's reply will either accept the requested size or
+further limit the size. The server will not increase the client's
+requested maximum bulk transfer size.
If the OBD_CONNECT_IBITS flag is set then the 'ocd_ibits_known' field
-is honored. The 'ocd_ibits_known' value determines the handling of
-locks on inodes. See the discussion of inodes and extended attributes.
+is valid. The 'ocd_ibits_known' flags determine the handling of
+locks on MDS inodes. The OBD_CONNECT_IBITS flag was introduced in
+Lustre 1.4 and is mandatory for MDS_CONNECT RPCs. See also the discussion
+of inodes and extended attributes.
+[[mds-inode-bits-locks]]
+
+----
+#define MDS_INODELOCK_LOOKUP 0x000001 /* For namespace, dentry etc, and also
+ * was used to protect permission (mode,
+ * owner, group etc) before 2.4. */
+#define MDS_INODELOCK_UPDATE 0x000002 /* size, links, timestamps */
+#define MDS_INODELOCK_OPEN 0x000004 /* For opened files */
+#define MDS_INODELOCK_LAYOUT 0x000008 /* for layout */
+#define MDS_INODELOCK_PERM 0x000010 /* separate permission bits */
+#define MDS_INODELOCK_XATTR 0x000020 /* extended attributes */
+----
+
+//////////////////////////////////////////////////////////////////////
+/* The PERM bit is added int 2.4, and it is used to protect permission(mode,
+ * owner, group, acl etc), so to separate the permission from LOOKUP lock.
+ * Because for remote directories(in DNE), these locks will be granted by
+ * different MDTs(different ldlm namespace).
+ *
+ * For local directory, MDT will always grant UPDATE_LOCK|PERM_LOCK together.
+ * For Remote directory, the master MDT, where the remote directory is, will
+ * grant UPDATE_LOCK|PERM_LOCK, and the remote MDT, where the name entry is,
+ * will grant LOOKUP_LOCK. */
+#define MDS_INODELOCK_PERM 0x000010
+#define MDS_INODELOCK_XATTR 0x000020 /* extended attributes */
+//////////////////////////////////////////////////////////////////////
If the OBD_CONNECT_GRANT_PARAM flag is set then the 'ocd_blocksize',
'ocd_inodespace', and 'ocd_grant_extent' fields are honored. A server
replay.
If the OBD_CONNECT_MDS flag is set then the 'ocd_group' field is
-honored. When an MDT connects to an OST the 'ocd_group' field informs
+valid. When an MDT connects to an OST the 'ocd_group' field informs
the OSS of the MDT's index. Objects on that OST for that MDT will be
in a common namespace served by that MDT.
If the OBD_CONNECT_CKSUM flag is set then the 'ocd_cksum_types' field
-is honored. The client uses the 'ocd_checksum_types' field to propose
-to the server the client's available (presumably hardware assisted)
+is valid. The client uses the 'ocd_cksum_types' field to propose
+to the server the client's available (possibly hardware assisted)
checksum mechanisms. The server replies with the checksum types it has
-available. Finally, the client will employ the fastest of the agreed
-mechanisms.
+available and that are most efficient on the server. The client may
+employ any of the replied checksum algorithms for a given bulk transfer,
+but will typically select the fastest of the agreed algorithms.
If the OBD_CONNECT_MAX_EASIZE flag is set then the 'ocd_max_easize'
-field is honored. The server uses 'ocd_max_easize' to inform the
+field is valid. The server uses 'ocd_max_easize' to inform the
client about the amount of space that can be allocated in each inode
for extended attributes. The 'ocd_max_easize' specifically refers to
-the space used for striping information. This allows the client to
+the space used for layout information. This allows the client to
determine the maximum layout size (and hence stripe count) that can be
stored on the MDT.
If the OBD_CONNECT_MAXBYTES flag is set then the 'ocd_maxbytes' field
is honored. An OSS uses the 'ocd_maxbytes' value to inform the client
-of the maximum OST object size for this target. A stripe on any OST
-for a multi-striped file cannot be larger than the minimum maxbytes
-value.
+of the maximum OST object size for this target. A file that is striped
+uniformly across multiple OST objects (RAID-0) cannot be larger than the
+number of stripes times the minimum 'ocd_maxbytes' value from any of its
+consituent objects.
The additional space in the 'obd_connect_data' structure is unused and
reserved for future use.
If the OBD_CONNECT_SRVLOCK flag is set then the client and server
support lockless IO. The server will take locks for client IO requests
with the OBD_BRW_SRVLOCK flag in the 'niobuf_remote' structure
-flags. This is used for Direct IO. The client takes no LDLM lock and
+flags. This is used for Direct IO or when there is significant lock
+contention on a single OST object. The client takes no LDLM lock and
delegates locking to the server.
If the OBD_CONNECT_ACL flag is set then the server supports the ACL
-mount option for its filesystem. The client supports this mount option
-as well, in that case.
+mount option for its filesystem. If the server is mounted with ACL
+support but the client does not pass OBD_CONNECT_ACL then the client
+mount is refused.
If the OBD_CONNECT_XATTR flag is set then the server supports user
-extended attributes. This is defined by the mount options of the
-servers of the backend file systems and is reflected on the client
-side by the same mount option but for the Lustre file system itself.
+extended attributes. This is requested by the client if mounted
+with the appropriate mount option, but is enabled or disabled by the
+mount options of the backend file system of MDT0000.
If the OBD_CONNECT_TRUNCLOCK flag is set then the client and the
server support lockless truncate. This is realized in an OST_PUNCH RPC
-by setting the 'obdo' sturcture's 'o_flag' field to include the
+by setting the 'obdo' structure's 'o_flag' field to include the
OBD_FL_SRVLOCK. In that circumstance the client takes no lock, and the
-server must take a lock on the resource.
+server must take a lock on the resource while performing the truncate.
If the OBD_CONNECT_ATTRFID flag is set then the server supports
getattr requests by FID of file instead of name. This reduces
-unnecessary RPCs for DNE.
+unnecessary RPCs for DNE and for file attribute revalidation after
+a lock cancellation.
If the OBD_CONNECT_NODEVOH flag is set then the server provides no
-open handle for special inodes.
+open handle for block and character special inodes.
If the OBD_CONNECT_RMT_CLIENT is set then the client is set as
'remote' with respect to the server. The client is considered as
'local' if the user/group database on the client is identical to that
on the server, otherwise the client is set as 'remote'. This
terminology is part of Lustre Kerberos feature which is not supported
-now.
+as of Lustre 2.7.
If the OBD_CONNECT_RMT_CLIENT_FORCE is set then client is set as
remote client forcefully. If the server security level doesn't support
remote clients then this client connect reply will return an -EACCESS
error.
-If the OBD_CONNECT_MDS_CAPA is set then MDS supports capability.
-Capabilities are part of Lustre Kerberos. The MDS prepares the
-capability when a file is opened and sends it to a client. A client
-has to present a capability when it wishes to perform an operation on
-that object.
-
-If the OBD_CONNECT_OSS_CAPA is set then OSS supports capability.
-Capabilities are part of Lustre Kerberos. When the clients request the
-OSS to perform a modification operations on objects the capability
-authorizes these operations.
-
If the OBD_CONNECT_CANCELSET is set then early batched cancels are
enabled. The ELC (Early Lock Cancel) feature allows client locks to
be cancelled prior the cancellation callback if it is clear that lock
is not needed anymore, for example after rename, after removing file
or directory, link, etc. This can reduce amount of RPCs significantly.
-If the OBD_CONNECT_AT is set then client and server use Adaptive
-Timeout while request processing. Servers keep track of the RPCs
+If the OBD_CONNECT_AT is set then client and server use 'Adaptive
+Timeouts' during request processing. Servers keep track of the RPC
processing time and report this information back to clients to
estimate the time needed for future requests and set appropriate RPC
timeouts.
If the OBD_CONNECT_LRU_RESIZE is set then the LRU self-adjusting is
-enabled. This is set by the Lustre configurable option
---enable-lru-resize, and is enabled by default.
+enabled.
-If the OBD_CONNECT_FID is set then FID support is required by
-server. This compatibility flag was introduced in Lustre 2.0. All
-servers and clients are using FIDs nowadays. This flag is always set
-on server and used to filter out clients without FID support.
+If the OBD_CONNECT_FID is set then FID support is understood by the
+client and server. This flag was introduced in Lustre 2.0 and
+is required when connecting to any 2.0 or later server. It is
+understood by Lustre 1.8 clients.
If the OBD_CONNECT_VBR is set then version based recovery is used on
-server. The VBR uses an object version to track its changes on the
-server and to decide if the replay can be applied during recovery
-based on that version. This helps to complete recovery even if some
-clients were missed or evicted. That flag is always set on server
-since Lustre 1.8 and is used just to notify the server if client
-doesn't support VBR.
-
-If the OBD_CONNECT_LOV_V3 is set then the client supports LOV vs
-EA. This type of the LOV extended attribute was introduced along with
-OST pools support and changed the internal structure of that EA. The
-OBD_CONNECT_LOV_V3 flag notifies a server if client doesn't support
+the server. VBR stores the most recent transaction in which each
+object was modified to track its changes on the server and to decide
+if a replayed RPC can be applied during recovery or not. This helps
+to complete recovery even if some clients were missed or evicted.
+That flag is always set by clients since Lustre 1.8 and is used to
+notify the server if client supports VBR.
+
+If the OBD_CONNECT_LOV_V3 is set if the client supports LOV_MAGIC_V3
+(0x0BD30BD0) style layouts. This type of the layout was introduced
+along with OST pools support and added the 'lov_mds_md_v3' layout. The
+OBD_CONNECT_LOV_V3 flag notifies a server if client supports
this type of LOV EA to handle requests from it properly.
If the OBD_CONNECT_GRANT_SHRINK is set then the client can release
grant space when idle.
If the OBD_CONNECT_SKIP_ORPHAN is set then OST doesn't reuse orphan
-object ids after recovery. This connection flag is used between MDS
+object IDs after recovery. This connection flag is used between MDS
and OST to agree about an object pre-creation policy after MDS
-recovery. If some of precreated objects weren't used but an MDT was
-restarted then an OST may re-use not used objects for new pre-create
-request or may not. The latter is preferred and is used by default
-nowadays.
+recovery. If set, then if some of precreated objects weren't used
+when an MDT was restarted then an OST must destroy any unused objects
+rather than re-use those objects.
If the OBD_CONNECT_FULL20 is set then the client is Lustre 2.x client.
Clients that are using old 1.8 format protocol conventions are not
-allowed to connect. This flag should be set on all connections since
-2.0, it is no longer affects behaviour and will be disabled completely
-once Lustre interoperation with old clients is no longer needed.
+allowed to connect to 2.x servers. This flag should be set on all
+connections since Lustre 1.8.5.
If the OBD_CONNECT_LAYOUTLOCK is set then the client supports layout
-lock. The server will not grant a layout lock to the old clients
-having no such flag.
+lock, which allows the server to revoke the layout of a file from
+a client if the layout has changed (e.g. migration between OSTs or
+changes in replication state). The server will not grant a layout
+lock to the old clients that do not support this feature.
If the OBD_CONNECT_64BITHASH is set then the client supports 64-bit
-directory hash. The server will also use 64-bit hash mode while
-working with ldiskfs.
+directory readdir cookie. The server will also use 64-bit hash mode
+while working with ldiskfs.
If the OBD_CONNECT_JOBSTATS is set then the client fills jobid in
-'ptlrpc_body' so server can provide extended statistics per jobid.
+'ptlrpc_body' so server can provide extended RPC statistics per jobid.
If the OBD_CONNECT_UMASK is set then create uses client umask. This is
-default flag for MDS but not for OST.
+default flag for MDS but is not relevant for OSTs.
-If the OBD_CONNECT_LVB_TYPE is set then the variable type of LVB is
-supported by a client. This flag was introduced along with DNE to
-recognize DNE-aware clients.
+If the OBD_CONNECT_LVB_TYPE is set then the variable type of lock
+value block (LVB) is supported by a client. This flag was introduced
+to allow the MDS to return data with an IBITS lock, in addition to
+the OST object attributes returned with an EXTENT lock.
If the OBD_CONNECT_LIGHTWEIGHT is set then this connection is the
'lightweight' one. A lightweight connection has no entry in last_rcvd
-file, so no recovery is possible, at the same time a lightweight
+file, so no recovery is possible. A new lightweight
connection can be set up while the target is in recovery, locks can
still be acquired through this connection, although they won't be
replayed. Such type of connection is used by services like quota
optimize a recovery of open requests.
If the OBD_CONNECT_OPEN_BY_FID is set then an open by FID won't pack
-the name in a request. This is used by DNE.
+the name in a request. This is used by HSM or other ChangeLog consumers
+for accessing objects by their FID via .lustre/fid/ instead of by name.
-If the OBD_CONNECT_MDS_MDS is set then the current connection is a
-MDS-MDS one. Such connections are distinguished because they provide
-more functionality specific to MDS-MDS interoperation.
+If the OBD_CONNECT_MDS_MDS is set then the current connection is an
+intra-MDS one. Such connections are distinguished because they provide
+more functionality specific to MDS-MDS interoperation such as OUT RPCs.
If the OBD_CONNECT_IMP_RECOV is set then the Imperative Recovery is
supported. Imperative recovery means the clients are notified
-explicitly when and where a failed target has restarted.
+explicitly when and where a target has restarted after failure.
The OBD_CONNECT_REQPORTAL was used to specify that client may use
OST_REQUEST_PORTAL for requests to don't interfere with IO portal,
and this flag does nothing though it is still set on client side
during connection process.
-The OBD_CONNECT_CROW flag was used for create-on-write functionality
-on OST, when data objects were created upon first write from the
-client. This wasn't implemented because of complex recovery problems.
-
The OBD_CONNECT_SOM flag was used to signal that MDS is capable to
store file size in file attributes, so client may get it directly from
MDS avoiding glimpse request to OSTs. This feature was implemented as
-demo feature and wasn't enabled by default. Finally it was disabled in
+demo feature and wasn't enabled by default. Finally it was removed in
Lustre 2.7 because it causes quite complex recovery cases to handle
with relatevely small benefits.
-The OBD_CONNECT_JOIN flag was used for the 'join files' feature, which
-allowed files to be concatenated. Lustre no longer supports that
-feature.
-
The OBD_CONNECT_QUOTA64 was used prior Lustre 2.4 for quota purposes,
it is obsoleted due to new quota design.
client to distinguish real connection from local connections between
layers.
-The OBD_CONNECT_CHANGE_QS was used prior Lustre 2.4 for quota needs
-and it is obsoleted now due to new quota design.
+The OBD_CONNECT_CHANGE_QS was used prior Lustre 2.4 for quota, but
+it is obsoleted now due to new quota design.
If the OBD_CONNECT_EINPROGRESS is set then client handles -EINPROGRESS
RPC error properly. The quota design requires that client must resend
is not checked anywhere, so does nothing.
If the OBD_CONNECT_NANOSEC_TIME is set then nanosecond timestamps are
-enabled. This flag is not used nowadays, but reserved for future use.
+enabled. This flag is not used yet, but reserved for future use.
If the OBD_CONNECT_SHORTIO is set then short IO feature is enabled on
server. The server will avoid bulk IO for small amount of data but
-data is incapsulated into ptlrpc request/reply. This flag is just
-reserved for future use and does nothing nowadays.
+data is encapsulated into ptlrpc request/reply. This flag is reserved
+for future use and does nothing yet.
git://git.whamcloud.com / doc / protocol.git / blobdiff