LU-3363 api: HSM import uses new released pattern

[fs/lustre-release.git] / lustre / doc / osd-api.txt

               ****************************************************
               * Overview of the Lustre Object Storage Device API *
               ****************************************************

Original Authors:
=================
Alex    Zhuravlev <alexey.zhuravlev@intel.com>
Andreas Dilger    <andreas.dilger@intel.com>
Johann  Lombardi  <johann.lombardi@intel.com>
Li      Wei       <wei.g.li@intel.com>
Niu     Yawei     <yawei.niu@intel.com>

Last Updated: October 9, 2012

Copyright © 2012 Intel, Corp.

This file is released under the GPLv2.

Topics
======

I.   Introduction
        1. What OSD API is
        2. What OSD API is Not
        3. Layering
        4. Audience/Goal
II.  Backend Storage Subsystem Requirements
        1. Atomicity of Updates
        2. Object Attributes
                i.  Standard POSIX Attributes
                ii. Extended Attributes
        3. Efficient Index
        4. Commit Callbacks
        5. Space Accounting
III. OSD & LU Infrastructure
        1. Devices
                i.   Device Overview
                ii.  Device Type & Operations
                iii. Device Operations
                iv.  OBD Methods
        2. Objects
                i.   Object Overview
                ii.  Object Lifecycle
                iii. Special Objects
                iv.  Object Operations
        3. Lustre Environment
IV.  Data (DT) API
        1. Data Device
        2. Data Objects
                i.   Common Storage Operations
                ii.  Data Object Operations
                iii. Indice Operations
        3. Transactions
                i.   Description
                ii.  Lifetime
                iii. Methods
        4. Locking
                i.   Description
                ii.  Methods
V.   Quota Enforcement
        1. Overview
        2. QSD API
Appendix 1. A brief note on Lustre configuration.
Appendix 2. Sample Code

===================
= I. Introduction =
===================

1. What OSD API is
==================

OSD API is the interface to access and modify data that is supposed to be stored
persistently. This API layer is the interface to code that bridges individual
file systems such as ext4 or ZFS to Lustre.
The API is a generic interface to transaction and journaling based file systems
so many backend file systems can be supported in a Lustre implementation.
Data can be cached within the OSD or backend target and could be destroyed
before hitting storage, but in general the final target is a persistent storage.
This API creates many possibilities, including using object-storage devices or
other new persistent storage technologies.

2. What OSD API is Not
======================

OSD API should not be used to control in-core-only state (like ldlm locking),
configuration, etc. The upper layers of the IO/metadata stack should not be
involved with the underlying layout or allocation in the OSD storage.

3. Layering
===========

Lustre is composed of different kernel modules, each implementing different
layers in the software stack in an object-oriented approach. Generally, each
layer builds (or stacks) upon another, and each object is a child of the
generic LU object class. Hence the term "LU stack" is often used to reference
this hierarchy of lustre modules and objects.

Each layer (i.e. mdt/mdd/lod/osp/ofd/osd) defines its own generic item
(lu_object/lu_device) which are thus gathered in a compound item (lu_site/
lu_object_layer) representing the multi-layered stacks. Different classes of
operations can then be implemented by each layer, depending on its natures.

As a result, each OSD is expected to implement:
- the generic LU API used to manage the device stack and objects (see chapter
  III)
- the DT API (most commonly called OSD API) used to manipulate on-disk
  structures (see chapter IV).

4. Audience/Goal
================

The goal of this document is to provide the reader with the information
necessary to accurately construct a new Object Storage Device (OSD) module
interface layer for Lustre in order to use a new backend file system with
Lustre 2.4 and greater.

==============================================
= II. Backend Storage Subsystem Requirements =
==============================================

The purpose of this section is to gather the requirements for the storage
subsystems below the OSD API.

1. Atomicity of Updates
=======================

The underlying OSD storage must be able to provide some form of atomic commit
for multiple arbitrary updates to OSD storage within a single transaction.
It will always know in advance of the transaction starting which objects will
be modified, and how they will be modified.

If any of the updates associated with a transaction are stored persistently
(i.e. some state in the OSD is modified), then all of the updates in that
transaction must also be stored persistently (Atomic). If the OSD should fail
in some manner that prevents all the updates of a transaction from being
completed, then none of the updates shall be completed (Consistent).
Once the updates have been reported committed to the caller (i.e. commit
callbacks have been run), they cannot be rolled back for any reason (Durable).

2. Object Attributes
====================

i. Standard POSIX Attributes
----------------------------
The OSD object should be able to store normal POSIX attributes on each object
as specified by Lustre:
- user ID (32 bits)
- group ID (32 bits)
- object type (16 bits)
- access mode (16 bits)
- metadata change time (96 bits, 64-bit seconds, 32-bit nanoseconds)
- data modification time (96 bits, 64-bit seconds, 32-bit nanoseconds)
- data access time (96 bits, 64-bit seconds, 32-bit nanoseconds)
- creation time (96 bits, 64-bit seconds, 32-bit nanoseconds, optional)
- object size (64 bits)
- link count (32 bits)
- flags (32 bits)
- object version (64 bits)

The OSD object shall not modify these attributes itself.

In addition, it is desirable track the object allocation size (“blocks”), which
the OSD manages itself. Lustre will query the object allocation size, but will
never modify it. If these attributes are not managed by the OSD natively as part
of the object itself, they can be stored in an extended attribute associated
with the object.

ii. Extended Attributes
------------------------
The OSD should have an efficient mechanism for storing small extended attributes
with each object. This implies that the extended attributes can be accessed at
the same time as the object (without extra seek/read operations). There is also
a requirement to store larger extended attributes in some cases (over 1kB in
size), but the performance of such attributes can be slower proportional to the
attribute size.

3. Efficient Index
==================

The OSD must provide a mechanism for efficient key=value retrieval, for both
fixed-length and variable length keys and values. It is expected that an index
may hold tens of millions of keys, and must be able to do random key lookups
in an efficient manner. It must also provide a mechanism for iterating over all
of the keys in a particular index and returning these to the caller in a
consistent order across multiple calls. It must be able to provide a cookie that
defines the current index at which the iteration is positioned, and must be able
to continue iteration at this index at a later time.

4. Commit Callbacks
===================

The OSD must provide some mechanism to register multiple arbitrary callback
functions for each transaction, and call these functions after the transaction
with which they are associated has committed to persistent storage.
It is not required that they be called immediately at transaction commit time,
but they cannot be delayed an arbitrarily long time, or other parts of the
system may suffer resource exhaustion. If this mechanism is not implemented by
the underlying storage, then it needs to be provided in some manner by the OSD
implementation itself.

5. Space Accounting
===================

In order to provide quota functionality for the OSD, it must be able to track
the object allocation size against at least two different keys (typically User
ID and Group ID). The actual mechanism of tracking this allocation is internal
to the OSD. Lustre will specify the owners of the object against which to track
this space. Space accounting information will be accessed by Lustre via the
index API on special objects dedicated to space allocation management.

================================
= III. OSD & LU Infrastructure =
================================

As a member of the LU stack, each OSD module is expected to implement the
generic LU API used to manage devices and objects.

1. Devices
==========

i. Device Overview
------------------
Each layer in the stack is represented by a lu_device structure which holds
the very basic data like reference counter, a reference to the site (Lustre
object collection in-core, very similar to inode cache), a reference to
struct lu_type which in turn describe this specific type of devices
(type name, operations etc).

OSD device is created and initialized at mount time to let configuration
component access data it needs before the whole Lustre stack is ready.
OSD device is destroyed when all the devices using that are destroyed too.
Usually this happen when the server stack shuts down at unmount time.

There might be few OSD devices of the given type (say, few zfs-osd and
ldiskfs-osd). The type stores method common for all OSD instances of given type
(below they start with ldto_ prefix). Then every instance of OSD device can get
few specific methods (below the start with ldo_ prefix).

To connect devices into a stack, ->o_connect() method is used (see struct
obd_ops). Currently OSD should implement this method to track all it’s users.
Then to disconnect ->o_disconnect() method is used. OSD should implement this
method, track remaining users and once no users left, call
class_manual_cleanup() function which initiate removal of OSD.

As the stack involves many devices and there may be cross-references between
them, it’s easier to break the whole shutdown procedure into the two steps and
do not set a specific order in which different devices shutdown: at the first
step the devices should release all the resources they use internally
(so-called pre-cleanup procedure), at the second step they are actually
destroyed.

ii. Device Type & Operations
----------------------------
The first thing to do when developing a new OSD is to define a lu_device_type
structure to define and register the new OSD type. The following fields of the
lu_device_type needs to be filled appropriately:
ldt_tags
        is the type of device, typically data, metadata or client (see
        lu_device_tag). An OSD device is of data type and should always
        registers as such by setting this field to LU_DEVICE_DT.
ldt_name
        is the name associated with the new OSD type.
        See LUSTRE_OSD_{LDISKFS,ZFS}_NAME for reference.
ldt_ops
        is the vector of lu_device_type operations, please see below for
        further details
ldt_ctxt_type
        is the lu_context_tag to be used for operations.
        This should be set to LCT_LOCAL for OSDs.

In the original 2.0 MDS stack the devices were built from the top down and OSD
was the final device to setup. This schema does not work very well when you have
to access on-disk data early and when you have OSD shared among few services
(e.g. MDS + MGS on a same storage). So the schema has changed to a reverse one:
mount procedure sets up correct OSD, then the stack is built from the bottom up.
And instead of introducing another set of methods we decided to use existing
obd_connect() and obd_disconnect() given that many existing devices have been
already configured this way by the configuration component. Notice also that
configuration profiles are organized in this order (LOV/LOD go first, then MDT).
Given that device “below” is ready at every step, there is no point in calling
separate init method.

Due to complexity in other modules, when the device itself can be referenced by
number of entities like exports, RPCs, transactions, callbacks, access via
procfs, the notion of precleanup was introduced to be able all the activity
safely before the actual cleanup takes place. Similarly ->ldto_device_fini()
and ->ldto_device_free() were introduced. So, the former should be used to break
any interaction with the outside, the latter - to actually free the device.

So, the configuration component meets SETUP command in the configuration profile
(see Appendix 1), finds appropriate device and calls ->ldto_device_alloc() to
set up it as an LU device.

The prototypes of device type operations are the following:

struct lu_device *(*ldto_device_alloc)(const struct lu_env *,
                                       struct lu_device_type *,
                                       struct lustre_cfg *);
struct lu_device *(*ldto_device_free)(const struct lu_env *,
                                      struct lu_device *);
int  (*ldto_device_init)(const struct lu_env *, struct lu_device *,
                         const char *, struct lu_device *);
struct lu_device *(*ldto_device_fini)(const struct lu_env *env, struct lu_device *);
int  (*ldto_init)(struct lu_device_type *t);
void (*ldto_fini)(struct lu_device_type *t);
void (*ldto_start)(struct lu_device_type *t);
void (*ldto_stop)(struct lu_device_type *t);

ldto_device_alloc
        The method is called by configuration component (in case of disk file
        system OSD, this is lustre/obdclass/obd_mount.c) to allocate device.
        Notice generic struct lu_device does not hold a pointer to private data.
        Instead OSD should embed struct lu_device into own structure (like
        struct osd_device) and return address of lu_device in that structure.
ldto_device_fini
        The method is called when OSD is about to release. OSD should detach
        from resources like disk file system, procfs, release objects it holds
        internally, etc. This is so-called precleanup procedure.
ldto_device_free
        The method is called to actually release memory allocated in
        ->ldto_device_alloc().
ldto_device_ini
        The method is not used by OSD currently.
ldto_init
        The method is called when specific type of OSD is registered in the
        system. Currently the method is used to register OSD-specific data for
        environments (see Lustre environment in section 3).
        See LU_TYPE_INIT_FINI() macro as an example.
ldto_fini
        The method is called when specific type of OSD unregisters.
        Currently used to unregister OSD-specific data from environment.
ldto_start
        The method is called when the first device of this type is being
        instantiated. Currently used to fill existing environments with
        OSD-specific data.
ldto_stop
        This method is called when the last instance of specific OSD has gone.
        Currently used to release OSD-specific data from environments.

iii. Device Operations
----------------------
Now that the osd device can be set up, we need to export methods to handle
device-level operation. All those methods are listed in the lu_device_operations
structure, this includes:

struct lu_object *(*ldo_object_alloc)(const struct lu_env *,
                                      const struct lu_object_header *,
                                      struct lu_device *);
int (*ldo_process_config)(const struct lu_env *, struct lu_device *,
                          struct lustre_cfg *);
int (*ldo_recovery_complete)(const struct lu_env *, struct lu_device *);
int (*ldo_prepare)(const struct lu_env *, struct lu_device *,
                   struct lu_device *);

ldo_object_alloc
        The method is called when a high-level service wants to access an
        object not found in local lustre cache (see struct lu_site).
        OSD should allocate a structure, initialize object’s methods and return
        a pointer to struct lu_device which is embedded into OSD object
        structure.
ldo_process_config
        The method is called in case of configuration changes. Mostly used by
        high-level services to update local tunables. It’s also possible to let
        MGS store OSD tunables and set them properly on every server mount or
        when tunables change run-time.
ldto_recovery_complete
        The method is called when recovery procedure between a server and
        clients is completed. This method is used by high-level devices mostly
        (like OSP to cleanup OST orphans, MDD to cleanup open unlinked files
        left by missing client, etc).
ldo_prepare
        The method is called when all the devices belonging to the stack are
        configured and setup properly. At this point the server becomes ready
        to handle RPCs and start recovery procedure.
        In current implementation OSD uses this method to initialize local quota
        management.

iv.  OBD Methods
----------------
Although the LU infrastructure aims at replacing the storage operations of the
legacy OBD API (see struct obd_ops in lustre/include/obd.h). The OBD API is
still used in several places for device configuration and on the Lustre client
(e.g. it’s still used on the client for LDLM locking). The OBD API storage
operations are not needed for server components, and should be ignored.

As far as the OSD layer is concerned, upper layers still connect/disconnect
to/from the OSD instance via obd_ops::o_connect/disconnect. As a result, each
OSD should implement those two operations:

int (*o_connect)(const struct lu_env *, struct obd_export **,
                 struct obd_device *, struct obd_uuid *,
                 struct obd_connect_data *, void *);
int (*o_disconnect)(struct obd_export *);

o_connect
        The method should track number of connections made (i.e. number of
        active users of this OSD) and call class_connect() and return a struct
        obd_export via class_conn2export(), see osd_obd_connect(). The structure
        holds a reference on the device, preventing it from early release.
o_disconnect
        The method is called then some one using this OSD does not need its
        service any more (i.e. at unmount). For every passed struct export the
        method should call class_disconnect(export). Once the last user has
        gone, OSD should call class_manual_cleanup() to schedule the device
        removal.

2. Objects
==========

i. Object Overview
------------------
Lustre identifies objects in the underlying OSD storage by a unique 128-bit
File IDentifier (FID) that is specified by Lustre and is the only identifier
that Lustre is aware of for this object. The FID is known to Lustre before any
access to the object is done (even before it is created), using
lu_object_find(). Since Lustre only uses the FID to identify an object, if the
underlying OSD storage cannot directly use the Lustre-specified FID to retrieve
the object at a later time, it must create a table or index object (normally
called the Object Index (OI)) to map Lustre FIDs to an internal object
identifier. Lustre does not need to understand the format or value of the
internal object identifier at any time outside of the OSD.

The FID itself is composed of 3 members:
struct lu_fid {
        __u64   f_seq;
        __u32   f_oid;
        __u32   f_ver;
};

While the OSD itself should typically not interpret the FID, it may be possible
to optimize the OSD performance by understanding the properties of a FID.

The f_seq (sequence) component is allocated in piecewise (though not contiguous)
manner to different nodes, and each sequence forms a “group” of related objects.
The sequence number may be any value in the range [1, 263], but there are
typically not a huge number of sequences in use at one time (typically less than
one million at the maximum). Within a single sequence, it is likely that tens to
thousands (and less commonly millions) of mostly-sequential f_oid values will be
allocated. In order to efficiently map FIDs into objects, it is desirable to
also be able to associate the OSD-internal index with key-value pairs.

Every object is represented with a header (struct lu_header) and so-called slice
on every layer of the stack. Core Lustre code maintains a cache of objects
(so-called lu-site, see struct lu_site). which is very similar to Linux inode
cache.

ii. Object Lifecycle
--------------------
In-core object is created when high-level service needs it to process RPC or
perform some background job like LFSCK. FID of the object is supposed to be
known before the object is created. FID can come from RPC or from a disk.
Having the FID lu_object_find() function is called, it search for the object in
the cache (see struct lu_site) and if the object is not found, creates it
using ->ldo_device_alloc(), ->loo_object_init() and ->loo_object_start()
methods.

Objects are referenced and tracked by Lustre core. If object is not in use,
it’s put on LRU list and at some point (subject to internal caching policy or
memory pressure callbacks from the kernel) Lustre schedules such an object for
removal from the cache. To do so Lustre core marks the object is going out and
calls ->loo_object_release() and ->loo_object_free() iterating over all the
layers involved.

iii. Special Objects
--------------------
Lustre uses a set of special objects using the FID_SEQ_LOCAL_FILE sequence.
All the objects are listed in the local_oid enum, which includes:
- OTABLE_OT_OID which is an index object providing list of all existing
  objects on this storage. The key is an opaque string and the record is FID.
  This object is used by high-level components like LFSCK to iterate over
  objects.
- ACCT_USER_OID/ACCT_GROUP_OID are used for accessing space accounting
  information for respectively users and groups.
- LAST_RECV_OID is the last_rcvd file for respectively
  the MDT and OST.

iv. Object Operations
---------------------
Object management methods are called by Lustre to manipulate OSD-specific
(private) data associated with a specific object during the lifetime of an
object. All the object operations are described in struct lu_object_operations:

int (*loo_object_init)(const struct lu_env *, struct lu_object *,
                       const struct lu_object_conf *);
int (*loo_object_start)(const struct lu_env *, struct lu_object *);
void (*loo_object_delete)(const struct lu_env *, struct lu_object *);
void (*loo_object_free)(const struct lu_env *, struct lu_object *);
void (*loo_object_release)(const struct lu_env *, struct lu_object *);
int (*loo_object_print)(const struct lu_env *, void *, lu_printer_t,
                        const struct lu_object *);
int (*loo_object_invariant)(const struct lu_object *);

loo_object_init
        This method is called when a new object is being created (see
        lu_object_alloc(), it’s purpose is to initialize object’s internals,
        usually file system lookups object on a disk (notice a header storing
        FID is already created by a top device) using Object Index mapping FID
        to local object id like dnode. LOC_F_NEW can be passed to the method
        when the caller knows the object is new and OSD can skip OI lookup to
        improve performance. If the object exists, then the LOHA_FLAG flag in
        loh_flags (struct lu_object_header) is set.
loo_object_start
        The method is called when all the structures and the header are
        initialized. Currently user by high-level service to as a post-init
        procedure (i.e. to setup own methods depending on object type which is
        brought into the header by OSD’s ->loo_object_init())
loo_object_delete
        is called to let OSD release resources behind an object (except memory
        allocated for an object), like release file system’s inode.
        It’s separated from ->loo_object_free() to be able to iterate over
        still-existing objects. the main purpose to separate
        ->loo_object_delete() and ->loo_object_free() is to avoid recursion
        during potentially stack consuming resource release.
loo_object_free
        is called to actually release memory allocated by ->ldo->object_alloc()
loo_object_release
        is called when object last it’s last user and moves onto LRU list of
        unused objects. implementation of this method is optional to OSD.
loo_object_print
        is used for debugging purpose, it should output details of an object in
        human-readable format. Details usually include information like address
        of an object, local object number (dnode/inode), type of an object, etc.
loo_object_invariant
        another optional method for debugging purposes which is called to verify
        internal consistency of object.

3. Lustre Environment
=====================

There is a notion of an environment represented by struct lu_env in many
functions and methods. Literally this is a Thread Local Storage (TLS), which is
bound to every service thread and used by that thread exclusively, there is no
need to serialize access to the data stored here.
The original purpose of the environment was to workaround small Linux stack
(4-8K). A component (like device or library) can register its own descriptor
(see LU_KEY_INIT macro) and then every new thread will be populating the
environment with buffers described.

=====================
= IV. Data (DT) API =
=====================

The previous section listed all the methods that have to be provided by an OSD
module in order to fit in the LU stack. In addition to those generic functions,
each layer should implement a different class of operations depending on its
natures. There are currently 3 classes of devices:
- LU_DEVICE_DT: DaTa device (e.g. lod, osp, osd, ofd),
- LU_DEVICE_MD: MetaData device (e.g. mdt, mdd),
- LU_DEVICE_CL: CLient I/O device (e.g. vvp, lov, lovsub, osc).

The purpose of this section is to document the DT API (used for devices and
objects) which has to be implemented by each OSD module. The DT API is most
commonly called the OSD API.

1. Data Device
==============

To access disk file system, Lustre defines a new device type called dt_device
which is a sub-class of generic lu_device. It includes a new operation vector
(namely dt_device_operations structure) defining all the actions that can be
performed against a dt_device. Here are the operation prototypes:

int   (*dt_statfs)(const struct lu_env *, struct dt_device *,
                   struct obd_statfs *);
struct thandle *(*dt_trans_create)(const struct lu_env *, struct dt_device *);
int   (*dt_trans_start)(const struct lu_env *, struct dt_device *,
                        struct thandle *th);
int   (*dt_trans_stop)(const struct lu_env *, struct thandle *);
int   (*dt_trans_cb_add)(struct thandle *, struct dt_txn_commit_cb *);
int   (*dt_root_get)(const struct lu_env *, struct dt_device *,
                     struct lu_fid *);
void  (*dt_conf_get)(const struct lu_env *, const struct dt_device *,
                     struct dt_device_param *);
int   (*dt_sync)(const struct lu_env *, struct dt_device *);
int   (*dt_ro)(const struct lu_env *, struct dt_device *);
int   (*dt_commit_async)(const struct lu_env *, struct dt_device *);
int   (*dt_init_capa_ctxt)(const struct lu_env *, struct dt_device *, int,
                           unsigned long, __u32, struct lustre_capa_key *);

dt_trans_create
dt_trans_start
dt_trans_stop
dt_trans_cb_add
        please refer to IV.3
dt_statfs
        called to report current file system usage information: all, free and
        available blocks/objects.
dt_root_get
        called to get FID of the root object. Used to follow backend filesystem
        rules and support backend file system in a state where users can mount
        it directly (with ldiskfs/zfs/etc).
dt_sync
        called to flush all complete but not written transactions. Should block
        until the flush is completed.
dt_ro
        called to turn backend into read-only mode.
        Used by testing infrastructure to simulate recovery cases.
dt_commit_async
        called to notify OSD/backend that higher level need transaction to be
        flushed as soon as possible. Used by Commit-on-Share feature.
        Should return immediately and not block for long.
dt_init_capa_ctxt
        called to initialize context for capabilities. not in use currently.

2. Data Objects
===============

There are two types of DT objects:
1) regular objects, storing unstructured data (e.g. flat files, OST objects,
   llog objects)
2) index objects, storing key=value pairs (e.g. directories, quota indexes,
   FLDB)

As a result, there are 3 sets of methods that should be implemented by the OSD
layer:
- core methods used to create/destroy/manipulate attributes of objects
- data methods used to access the object body as a flat address space
  (read/write/truncate/punch) for regular objects
- index operations to access index objects as a key-value association

A data object is represented by the dt_object structure which is defined as
a sub-class of lu_object, plus operation vectors for the core, data and index
methods as listed above.

i. Common Storage Operations
----------------------------
The core methods are defined in dt_object_operations as follows:

void (*do_read_lock)(const struct lu_env *, struct dt_object *, unsigned);
void (*do_write_lock)(const struct lu_env *, struct dt_object *, unsigned);
void (*do_read_unlock)(const struct lu_env *, struct dt_object *);
void (*do_write_unlock)(const struct lu_env *, struct dt_object *);
int  (*do_write_locked)(const struct lu_env *, struct dt_object *);
int  (*do_attr_get)(const struct lu_env *, struct dt_object *,
                     struct lu_attr *, struct lustre_capa *);
int  (*do_declare_attr_set)(const struct lu_env *, struct dt_object *,
                            const struct lu_attr *, struct thandle *);
int  (*do_attr_set)(const struct lu_env *, struct dt_object *,
                    const struct lu_attr *, struct thandle *,
                    struct lustre_capa *);
int  (*do_xattr_get)(const struct lu_env *, struct dt_object *,
                      struct lu_buf *, const char *, struct lustre_capa *);
int  (*do_declare_xattr_set)(const struct lu_env *, struct dt_object *,
                             const struct lu_buf *, const char *, int,
                             struct thandle *);
int  (*do_xattr_set)(const struct lu_env *, struct dt_object *,
                      const struct lu_buf *, const char *, int,
                      struct thandle *, struct lustre_capa *);
int  (*do_declare_xattr_del)(const struct lu_env *, struct dt_object *,
                              const char *, struct thandle *);
int  (*do_xattr_del)(const struct lu_env *, struct dt_object *, const char *,
                      struct thandle *, struct lustre_capa *);
int  (*do_xattr_list)(const struct lu_env *, struct dt_object *,
                       struct lu_buf *, struct lustre_capa *);
void (*do_ah_init)(const struct lu_env *, struct dt_allocation_hint *,
                    struct dt_object *, struct dt_object *, cfs_umode_t);
int  (*do_declare_create)(const struct lu_env *, struct dt_object *,
                           struct lu_attr *, struct dt_allocation_hint *,
                           struct dt_object_format *, struct thandle *);
int  (*do_create)(const struct lu_env *, struct dt_object *, struct lu_attr *,
                   struct dt_allocation_hint *, struct dt_object_format *,
                   struct thandle *);
int  (*do_declare_destroy)(const struct lu_env *, struct dt_object *,
                           struct thandle *);
int  (*do_destroy)(const struct lu_env *, struct dt_object *, struct thandle *);
int  (*do_index_try)(const struct lu_env *, struct dt_object *, 
                     const struct dt_index_features *);
int  (*do_declare_ref_add)(const struct lu_env *, struct dt_object *,
                           struct thandle *);
int  (*do_ref_add)(const struct lu_env *, struct dt_object *, struct thandle *);
int  (*do_declare_ref_del)(const struct lu_env *, struct dt_object *,
                           struct thandle *);
int  (*do_ref_del)(const struct lu_env *, struct dt_object *, struct thandle *);
struct obd_capa *(*do_capa_get)(const struct lu_env *, struct dt_object *,
                                struct lustre_capa *, __u64);
int  (*do_object_sync)(const struct lu_env *, struct dt_object *);

do_read_lock
do_write_lock
do_read_unlock
do_write_unlock
do_write_locked
        please refer to IV.4
do_attr_get
        The method is called to get regular attributes an object stores.
        The lu_attr fields maps the usual unix file attributes, like ownership
        or size. The object must exist.
do_declare_attr_set
        the method is called to notify OSD the caller is going to modify regular
        attributes of an object in specified transaction. OSD should use this
        method to reserve resources needed to change attributes. Can be called
        on an non-existing object.
do_attr_set
        the method is called to change attributes of an object. The object
        must exist. If the fl argument has LU_XATTR_CREATE, the extended
        argument must not exist, otherwise -EEXIST should be returned.
        If the fl argument has LU_XATTR_REPLACE, the extended argument must
        exist, otherwise -ENODATA should be returned. The object must exist.
        The maximum size of extended attribute supported by OSD should be
        present in struct dt_device_param the caller can get with
        ->dt_conf_get() method.
do_xattr_get
        called when the caller needs to get an extended attribute with a
        specified name. If the struct lu_buf argument has a null lb_buf, the
        size of the extended attribute should be returned. If the requested
        extended attribute does not exist, -ENODATA should be returned.
        The object must exist. If buffer space (specified in lu_buf.lb_len) is
        not enough to fit the value, then return -ERANGE.
do_declare_xattr_set
        called to notify OSD the caller is going to set/change an extended
        attribute on an object. OSD should use this method to reserve resources
        needed to change an attribute.
do_xattr_set
        called when the caller needs to change an extended attribute with
        specified name.
do_declare_xattr_del
        called to notify OSD the caller is going to remove an extended attribute
        with a specified name
do_xattr_del
        called when the caller needs to remove an extended attribute with a
        specified name. Deleting an nonexistent extended attribute is allowed.
        The object must exist. The method called on a non-existing attribute
        returns 0.
do_xattr_list
        called when the caller needs to get a list of existing extended
        attributes (only names of attributes are returned). The size of the list
        is returned, including the string terminator. If the lu_buf argument has
        a null lb_buf, how many bytes the list would require is returned to help
        the caller to allocate a buffer of an appropriate size.
        The object must exist.
do_ah_init
        called to let OSD to prepare allocation hint which stores information
        about object locality, type. later this allocation hint is passed to
        ->do_create() method and use OSD can use this information to optimize
        on-disk object location. allocation hint is opaque for the caller and
        can contain OSD-specific information.
do_declare_create
        called to notify OSD the caller is going to create a new object in a
        specified transaction.
do_create
        called to create an object on the OSD in a specified transaction.
        For index objects the caller can request a set of index properties (like
        key/value size). If OSD can not support requested properties, it should
        return an error. The object shouldn't exist already (i.e.
        dt_object_exist() should return false).
do_declare_destroy
        called to notify OSD the caller is going to destroy an object in a
        specified transaction.
do_destroy
        called to destroy an object in a specified transaction. Semantically,
        it’s dual to object creation and does not care about on-disk reference
        to the object (in contrast with POSIX unlink operation).
        The object must exist (i.e. dt_object_exist() must return true).
do_index_try
        called when the caller needs to use an object as an index (the object
        should be created as an index before). Also the caller specify a set of
        properties she expect the index should support.
do_declare_ref_add
        called to notify OSD the caller is going to increment nlink attribute
        in a specified transaction.
do_ref_add
        called to increment nlink attribute in a specified transaction.
        The object must exist.
do_declare_ref_del
        called to notify OSD the caller is going to decrement nlink attribute
        in a specified transaction.
do_ref_del
        called to decrement nlink attribute in a specified transaction.
        This is typically done on an object when a record referring to it is
        deleted from an index object. The object must exist.
do_capa_get
        called to get a capability for a specified object. not used currently.
do_object_sync
        called to flush a given object on-disk. It’s a fine grained version of
        ->do_sync() method which should make sure an object is stored on-disk.
        OSD (or backend file system) can track a status of every object and if
        an object is already flushed, then just the method can return
        immediately. The method is used on OSS now, but can also be used on MDS
        at some point to improve performance of COS.
do_data_get
        the method is not used any more and planned for removal.

ii. Data Object Operations
--------------------------
Set of methods described in struct dt_body_operations which should be used with
regular objects storing unstructured data:

ssize_t (*dbo_read)(const struct lu_env *, struct dt_object *, struct lu_buf *,
                    loff_t *pos, struct lustre_capa *);
ssize_t (*dbo_declare_write)(const struct lu_env *, struct dt_object *,
                             const loff_t, loff_t, struct thandle *);
ssize_t (*dbo_write)(const struct lu_env , struct dt_object *,
                     const struct lu_buf *, loff_t *, struct thandle *,
                     struct lustre_capa *, int);
int (*dbo_bufs_get)(const struct lu_env *, struct dt_object *, loff_t,
                    ssize_t, struct niobuf_local *, int, struct lustre_capa *);
int (*dbo_bufs_put)(const struct lu_env *, struct dt_object *,
                    struct niobuf_local *, int);
int (*dbo_write_prep)(const struct lu_env *, struct dt_object *,
                      struct niobuf_local *, int);
int (*dbo_declare_write_commit)(const struct lu_env *, struct dt_object *,
                                struct niobuf_local *,int, struct thandle *);
int (*dbo_write_commit)(const struct lu_env *, struct dt_object *,
                        struct niobuf_local *, int, struct thandle *);
int (*dbo_read_prep)(const struct lu_env *, struct dt_object *,
                     struct niobuf_local *, int);
int (*dbo_fiemap_get)(const struct lu_env *, struct dt_object *,
                      struct ll_user_fiemap *);
int (*dbo_declare_punch)(const struct lu_env*, struct dt_object *, __u64,
                          __u64,struct thandle *);
int (*dbo_punch)(const struct lu_env *, struct dt_object *, __u64, __u64,
                struct thandle *, struct lustre_capa *);

dbo_read
        is called to read raw unstructured data from a specified range of an
        object. It returns number of bytes read or an error. Usually OSD
        implements this method using internal buffering (to be able to put data
        at non-aligned address). So this method should not be used to move a
        lot of data. Lustre services use it to read to read small internal data
        like last_rcvd file, llog files. It's also used to fetch body symlinks.
dbo_declare_write
        is called to notify OSD the caller will be writing data to a specific
        range of an object in a specified transaction.
dbo_write
        is called to write raw unstructured data to a specified range of an
        object in a specified transaction. data should be written atomically
        with another change in the transaction. The method is used by Lustre
        services to update small portions on a disk. OSD should maintain size
        attribute consistent with data written.
dbo_bufs_get
        is called to fill memory with buffer descriptors (see struct
        niobuf_local) for a specified range of an object. memory for the set is
        provided by the caller, no concurrent access to this memory is allowed.
        OSD can fill all fields of the descriptor except lnb_grant_used.
        The caller specify whether buffers will be user to read or write data.
        This method is used to access file system's internal buffers for
        zero-copy IO. Internal buffers referenced by descriptors are supposed to
        be pinned in memory
dbo_bufs_put
        is called to unpin/release internal buffers referenced by the
        descriptors dbo_bufs_get returns. After this point pointers in the
        descriptors are not valid.
dbo_write_prep
        is called to fill internal buffers with actual data. this is required
        for buffers which do not match filesystem blocksize, as later the buffer
        is supposed to be written as a whole. for example, ldiskfs uses 4k
        blocks, but the caller wants to update just a half of that. to prevent
        data corruption, this method is called OSD compares range to be written
        with 4k, if they do not match, then OSD fetches data from a disk.
        If they do match, then all the data will be overwritten and there is no
        need to fetch data from a disk.
dbo_declare_write_commit
        is called to notify OSD the caller is going to write internal buffers
        and OSD needs to reserve enough resource in a transaction.
dbo_write_commit
        is called to actually make data in internal buffers part of a specified
        transaction. Data is supposed to be written by the moment the
        transaction is considered committed. This is slightly different from
        generic transaction model because in this case it's allowed to have
        data written, but not have transaction committed.
        If no dbo_write_commit is called, then dbo_bufs_put should discard
        internal buffers and possible changes made to internal buffers should
        not be visible.
dbo_read_prep
        is called to fill all internal buffers referenced by descriptors with
        actual data. buffers may already contain valid data (be cached), so OSD
        can just verify the data is valid and return immediately.
dbo_fiemap_get
        is called to map logical range of an object to physical blocks where
        corresponded range of data is actually stored.
dbo_declare_punch
        is called to notify OSD the caller is going to punch (deallocate)
        specified range in a transaction.
dbo_punch
        is called to punch (deallocate) specified range of data in a
        transaction. this method is allowed to use few disk file system
        transactions (within the same lustre transaction handle).
        Currently Lustre calls the method in form of truncate only where the end
        offset is EOF always.

iii. Indice Operations
----------------------
In contrast with raw unstructured data they are collection of key=value pairs.
OSD should provide with few methods to lookup, insert, delete and scan pairs.
Indices may have different properties like key/value size, string/binary keys,
etc. When user need to use an index, it needs to check whether the index has
required properties with a special method. indices are used by Lustre services
to maintain user-visible namespace, FLD, index of unlinked files, etc.

The method prototypes are defined in dt_index_operations as follows:

int (*dio_lookup)(const struct lu_env *, struct dt_object *, struct dt_rec *,
                  const struct dt_key *, struct lustre_capa *);
int (*dio_declare_insert)(const struct lu_env *, struct dt_object *,
                          const struct dt_rec *, const struct dt_key *,
                          struct thandle *);
int (*dio_insert)(const struct lu_env *, struct dt_object *,
                  const struct dt_rec *, const struct dt_key *,
                  struct thandle *, struct lustre_capa *, int);
int (*dio_declare_delete)(const struct lu_env *, struct dt_object *,
                          const struct dt_key *, struct thandle *);
int (*dio_delete)(const struct lu_env *, struct dt_object *,
                  const struct dt_key *, struct thandle *,
                  struct lustre_capa *);

dio_lookup
        is called to lookup exact key=value pair. A value is copied into a
        buffer provided by the caller. so the caller should make sure the
        buffer's size is big enough. this should be done with ->do_index_try()
        method.
dio_declare_insert
        is called to notify OSD the caller is going to insert key=value pair in
        a transaction. exact key is specified by a caller so OSD can use this to
        make reservation better (i.e. smaller).
dio_insert
        is called to insert key/value pair into an index object. it's up to OSD
        whether to allow concurrent inserts or not. the caller is not required
        to serialize access to an index
dio_declare_delete
        is called to notify OSD the caller is going to remove a specified key
        in a transaction. exact key is specified by a caller so OSD can use this
        to make reservation better.
dio_delete
        is called to remove a key/value pair specified by a caller.

To iterate over all key=value pair stored in an index, OSD should provide the
following set of methods:

struct dt_it *(*init)(const struct lu_env *, struct dt_object *, __u32,
                      struct lustre_capa *);
void  (*fini)(const struct lu_env *, struct dt_it *);
int   (*get)(const struct lu_env *, struct dt_it *, const struct dt_key *);
void  (*put)(const struct lu_env *, struct dt_it *);
int   (*next)(const struct lu_env *, struct dt_it *);
struct dt_key *(*key)(const struct lu_env *, const struct dt_it *);
int   (*key_size)(const struct lu_env *, const struct dt_it *);
int   (*rec)(const struct lu_env *, const struct dt_it *, struct dt_rec *,
             __u32);
__u64 (*store)(const struct lu_env *, const struct dt_it *);
int   (*load)(const struct lu_env *, const struct dt_it *, __u64);
int   (*key_rec)(const struct lu_env *, const struct dt_it *, void *);

init
        is called to allocate and initialize an instance of "iterator" which
        subsequent methods will be passed in. the structure is not accessed by
        Lustre and its content is totally internal to OSD. Usually it contains a
        reference to index, current position in an index.
        It may contain prefetched key/value pairs. It's not required to maintain
        this cache up-to-date, if index changes this is not required to be
        reflected by an already initialized iterator. In the extreme case
        ->init() can prefetch all existing pairs to be returned by subsequent
        calls to an iterator.
fini
        is called to release an iterator and all its resources.
        For example, iterator can unpin an index, free prefetched pairs, etc.
get
        is called to move an iterator to a specified key. if key does not exist
        then it should be the closest position from the beginning of iteration.
put
        is called to release an iterator.
next
        is called to move an iterator to a next item
key
        is called to fill specified buffer with a key at a current position of
        an iterator. it’s the caller responsibility to pass big enough buffer.
        In turn OSD should not exceed sizes negotiated with ->do_index_try()
        method
key_size
        is called to learn size of a key at current position of an iterator
rec
        is called to fill specified buffer with a value at a current position of
        an iterator. it’s the caller responsibility to pass big enough buffer.
        in turn OSD should not exceed sizes negotiated with ->do_index_try()
        method.
store
        is called to get a 64bit cookie of a current position of an iterator.
load
        is called to reset current position of an iterator to match 64bit
        cookie ->store() method returns. these two methods allow to implement
        functionality like POSIX readdir where current position is stored as an
        integer.
key_rec
        is not used currently

3. Transactions
===============

i. Description
--------------
Transactions are used by Lustre to implement recovery protocol and support
failover. The main purpose of transactions is to atomically update backend file
system. This include as regular changes (file creation, for example) as special
Lustre changes (last_rcvd file, lastid, llogs). OSD is supposed to provide the
transactional mechanism and let Lustre to control what specific updates to put
into transactions.

Lustre relies on the following rule for transactions order: if transaction T1
starts before transaction T2 starts, then the commit of T2 means that T1 is
committed at the same time or earlier. Notice that the creation of a transaction
does not imply the immediate start of the updates on storage, do not confuse
creation of a transaction with start of a transaction.

It’s up to OSD and backend file system to group few transactions for better
performance given it still follow the rule above.

Transactions are identified in the OSD API by an opaque transaction handle,
which is a pointer to an OSD-private data structure that it can use to track
(and optionally verify) the updates done within that transaction. This handle is
returned by the OSD to the caller when the transaction is first created.
Any potential updates (modifications to the underlying storage) must be declared
as part of a transaction, after the transaction has been created, and before the
transaction is started. The transaction handle is passed when declaring all
updates. If any part of the declaration should fail, the transaction is aborted
without having modified the storage.

After all updates have been declared, and have completed successfully, the
handle is passed to the transaction start. After the transaction has started,
the handle will be passed to every update that is done as part of that
transaction. All updates done under the transaction must previously have been
declared. Once the transaction has started, it is not permitted to add new
updates to the transaction, nor is it possible to roll back the transaction
after this point. Should some update to the storage fail, the caller will try
to undo the previous updates within the context of the transaction itself, to
ensure that the resulting OSD state is correct.

Any update that was not previously declared is an implementation error in the
caller. Not all declared updates need to be executed, as they form a worst-case
superset of the possible updates that may be required in order to complete the
desired operation in a consistent manner.

OSD should let a caller to register callback function(s) to be called on
transaction commit to a disk. Also OSD should be able to call a special of
transaction hooks on all the stages (creation, start, stop, commit) on
per-devices basis so that high-level services (like MDT) which are not involved
directly into controlling transactions still can be involved.
Every commit callback gets a result of transaction commit, if disk filesystem was
not able to commit the transaction, then an appropriate error code will be passed.

It’s important to note that OSD and disk file system should use asynchronous IO
to implement transactions, otherwise the performance is expected to be bad.

The maximum number of updates that make up a single transaction is OSD-specific,
but is expected to be at least in the tens of updates to multiple objects in the
OSD (extending writes of multiple MB of data, modifying or adding attributes,
extended attributes, references, etc). For example, in ext4, each update to the
filesystem will modify one or more blocks of storage. Since one transaction is
limited to one quarter of the journal size, if the caller declares a series of
updates that modify more than this number of blocks, the declaration must fail
or it could not be committed atomically.
In general, every constraint must be checked here to ensure that all changes
that must commit atomically can complete successfully.

ii. Lifetime
------------
From Lustre point of view a transaction goes through the following steps:
1. creation
2. declaration of all possible changes planned in transaction
3. transaction start
4. execution of planned and declared changes
5. transaction stop
6. commit callback(s)

iii. Methods
------------
OSD should implement the following methods to let Lustre control transactions:

struct thandle *(*dt_trans_create)(const struct lu_env *, struct dt_device *);
int (*dt_trans_start)(const struct lu_env *, struct dt_device *,
                      struct thandle *);
int   (*dt_trans_stop)(const struct lu_env *, struct thandle *);
int   (*dt_trans_cb_add)(struct thandle *, struct dt_txn_commit_cb *);

dt_trans_create
        is called to allocate and initialize transaction handle (see struct
        thandle). This structure has no pointer to a private data so, it should
        be embedded into private representation of transaction at OSD layer.
        This method can block.
dt_trans_start
        is called to notify OSD a specified transaction has got all the
        declarations and now OSD should tell whether it has enough resources to
        proceed with declared changes or to return an error to a caller.
        This method can block. OSD should call dt_txn_hook_start() function
        before underlying file system’s transaction starts to support per-device
        transaction hooks. If OSD (or disk files system) can not start
        transaction, then an error is returned and transaction handle is
        destroyed, no commit callbacks are called.
dt_trans_stop
        is called to notify OSD a specified transaction has been executed and no
        more changes are expected in a context of that. Usually this mean that at
        this point OSD is free to start writeout preserving notion
        all-or-nothing. This method can block.
        If th_sync flag is set at this point, then OSD should start to commit
        this transaction and block until the transaction is committed. the order
        of unblock event and transaction’s commit callback functions is not
        defined by the API. OSD should call dt_txn_hook_stop() functions once
        underlying file system’s transaction is stopped to support per-device
        transaction hooks.
dt_trans_cb_add
        is called to register commit callback function(s), which OSD will be
        calling up on transaction commit to a storage. when all the callback
        functions are processed, transaction handle can be freed by OSD.
        There are no constraints on how many callback functions can be running
        concurrently. They should not be running in an interrupt context.
        Usually this method should not block and use spinlocks. As part of
        commit callback functions processing dt_txn_hook_commit() function
        should be called to support per-device transaction hooks.

The callback mechanism let layers not commanding transactions be involved.
For example, MDT registers its set and now every transaction happening on
corresponded OSD will be seen by MDT, which adds recovery information to the
transactions: generate transaction number, puts it into a special file -- all
this happen within the context of the transaction, so atomically.
Similarly VBR functionality in MDT updates objects versions.

4. Locking
==========

i. Description
--------------
OSD is expected to maintain internal consistency of the file system and its
object on its own, requiring no additional locking or serialization from higher
levels. This let OSD to control how fine the locking is depending on the
internal structuring of a specific file system. If few update conflict then the
result is not defined by OSD API and left to OSD.

OSD should provide the caller with few methods to serialize access to an object
in shared and exclusive mode. It’s up to caller how to use them, to define order
of locking. In general the locks provided by OSD are used to group complex
updates so that other threads do not see intermediate result of operations.

ii. Methods
-----------
Methods to lock/unlock object
The set of methods exported by each OSD to manage locking is the following:
void (*do_read_lock)(const struct lu_env *, struct dt_object *, unsigned);
void (*do_write_lock)(const struct lu_env *, struct dt_object *, unsigned);
void (*do_read_unlock)(const struct lu_env *, struct dt_object *);
void (*do_write_unlock)(const struct lu_env *, struct dt_object *);
int  (*do_write_locked)(const struct lu_env *, struct dt_object *);

do_read_lock
        get a shared lock on the object, this is a blocking lock.
do_write_lock
        get an exclusive lock on the object, this is a blocking lock.
do_read_unlock
        release a shared lock on an object, this is a blocking lock.
do_write_unlock
        release an exclusive lock on an object, this is a blocking lock.
do_write_locked
        check whether an object is exclusive-locked.

It is highly desirable that an OSD object can be accessed and modified by
multiple threads concurrently.

For regular objects, the preferred implementation allows an object to be read
concurrently at overlapping offsets, and written by multiple threads at
non-overlapping offsets with the minimum amount of contention possible, or any
combination of concurrent read/write operations. Lustre will not itself perform
concurrent overlapping writes to a single region of the object, due to
serialization at a higher level.

For index objects, the preferred implementation allows key/value pair to be
looked up concurrently, allows non-conflicting keys to be inserted or removed
concurrently, or any combination of concurrent lookup, insertion, or removal.
Lustre does not require the storage of multiple identical keys. Operations on
the same key should be serialized.

========================
= V. Quota Enforcement =
========================

1. Overview
===========

The OSD layer is in charge of setting up a Quota Slave Device (aka QSD) to
manage quota enforcement for a specific OSD device. The QSD is implemented under
the form of a library. Each OSD device should create a QSD instance which will
be used to manage quota enforcement for this device. This implies:
- completing the reintegration procedure with the quota master (aka QMT) to
  to retrieve the latest quota settings and quota space distribution for each
  UID/GID.
- managing quota locks in order to be notified of configuration changes.
- acquiring space from the QMT when quota space for a given user/group is
  close to exhaustion.
- allocating quota space to service threads for local request processing.

The reintegration procedure allows a disconnected slave to re-synchronize with
the quota master, which means:
- re-acquiring quota locks,
- fetching up-to-date quota settings (e.g. list of UIDs with quota enforced),
- reporting space usage to master for newly (e.g. setquota was run while the
  slave wasn't connected) enforced UID/GID,
- adjusting spare quota space (e.g. slave hold a large amount of unused quota
  space for a user which ran out of quota space on the master while the slave
  was disconnected).

The latter two actions are known as reconciliation.

2. QSD API
==========

The QSD API is defined in lustre/include/lustre_quota.h as follows:

struct qsd_instance *qsd_init(const struct lu_env *, char *, struct dt_device *,
                              cfs_proc_dir_entry_t *);
int qsd_prepare(const struct lu_env *, struct qsd_instance *);
int qsd_start(const struct lu_env *, struct qsd_instance *);
void qsd_fini(const struct lu_env *, struct qsd_instance *);
int qsd_op_begin(const struct lu_env *, struct qsd_instance *,
                 struct lquota_trans *, struct lquota_id_info *, int *);
void qsd_op_end(const struct lu_env *, struct qsd_instance *,
                struct lquota_trans *);
void qsd_op_adjust(const struct lu_env *, struct qsd_instance *,
                   union lquota_id *, int);

qsd_init
        The OSD module should first allocate a qsd instance via qsd_init.
        This creates all required structures to manage quota enforcement for
        this target and performs all low-level initialization which does not
        involve any lustre object. qsd_init should typically be called when
        the OSD is being set up.

qsd_prepare
        This sets up on-disk objects associated with the quota slave feature
        and initiates the quota reintegration procedure if needed.
        qsd_prepare should typically be called when ->ldo_prepare is invoked.

qsd_start
        a qsd instance should be started once recovery is completed (i.e. when
        ->ldo_recovery_complete is called). This is used to notify the qsd layer
        that quota should now be enforced again via the qsd_op_begin/end
        functions. The last step of the reintegration procedure (namely usage
        reconciliation) will be completed during start.

qsd_fini
        is used to release a qsd_instance structure allocated with qsd_init.
        This releases all quota slave objects and frees the structures
        associated with the qsd_instance.

qsd_op_begin
        is used to enforce quota, it must be called in the declaration of each
        operation. qsd_op_end should then be invoked later once all operations
        have been completed in order to release/adjust the quota space.
        Running qsd_op_begin before qsd_start isn't fatal and will return
        success. Once qsd_start has been run, qsd_op_begin will block until the
        reintegration procedure is completed.

qsd_op_end
        performs the post operation quota processing. This must be called after
        the operation transaction stopped. While qsd_op_begin must be invoked
        each time a new operation is declared, qsd_op_end should be called only
        once for the whole transaction.

qsd_op_adjust
        Trigger pre-acquire/release if necessary, it's only used for ldiskfs osd
        so far. When unlink a file in ldiskfs, the quota accounting isn't
        updated when the transaction stopped. Instead, it'll be updated on the
        final iput, so qsd_op_adjust() will be called then (in
        osd_object_delete()) to trigger quota release if necessary.

Appendix 1. A brief note on Lustre configuration.
=================================================

In the current versions (1.8, 2.x) MGS is used to store configuration of the
servers, so called profile. The profile stores configuration commands and
arguments to setup specific stack. To see how it looks exactly you can fetch
MDT profile with debugfs -R "dump /CONFIGS/lustre-MDT0000 <tempfile>", then
parse it with: llog_reader <tempfile>. Here is a short extract:

#02 (136)attach    0:lustre-MDT0000-mdtlov  1:lov  2:lustre-MDT0000-mdtlov_UUID
#03 (176)lov_setup 0:lustre-MDT0000-mdtlov  1:(struct lov_desc)
                uuid=lustre-MDT0000-mdtlov_UUID  stripe:cnt=1 size=1048576 offset=18446744073709551615 pattern=0x1
#06 (120)attach    0:lustre-MDT0000  1:mdt  2:lustre-MDT0000_UUID
#07 (112)mount_option 0:  1:lustre-MDT0000  2:lustre-MDT0000-mdtlov
#08 (160)setup     0:lustre-MDT0000  1:lustre-MDT0000_UUID  2:0  3:lustre-MDT0000-mdtlov  4:f
#23 (080)add_uuid  nid=10.0.2.15@tcp(0x200000a00020f)  0:  1:10.0.2.15@tcp
#24 (144)attach    0:lustre-OST0000-osc-MDT0000  1:osc  2:lustre-MDT0000-mdtlov_UUID
#25 (144)setup     0:lustre-OST0000-osc-MDT0000  1:lustre-OST0000_UUID  2:10.0.2.15@tcp
#26 (136)lov_modify_tgts add 0:lustre-MDT0000-mdtlov  1:lustre-OST0000_UUID  2:0  3:1
#32 (080)add_uuid  nid=10.0.2.15@tcp(0x200000a00020f)  0:  1:10.0.2.15@tcp
#33 (144)attach    0:lustre-OST0001-osc-MDT0000  1:osc  2:lustre-MDT0000-mdtlov_UUID
#34 (144)setup     0:lustre-OST0001-osc-MDT0000  1:lustre-OST0001_UUID  2:10.0.2.15@tcp
#35 (136)lov_modify_tgts add 0:lustre-MDT0000-mdtlov  1:lustre-OST0001_UUID  2:1  3:1
#41 (120)param 0:  1:sys.jobid_var=procname_uid  2:procname_uid
#44 (080)set_timeout=20
#48 (112)param 0:lustre-MDT0000-mdtlov  1:lov.stripesize=1048576
#51 (112)param 0:lustre-MDT0000-mdtlov  1:lov.stripecount=-1
#54 (160)param 0:lustre-MDT0000  1:mdt.identity_upcall=/work/lustre/head/lustre-release/lustre/utils/l_getidentity

Every line starts with a specific command (attach, lov_setup, set, etc) to do
specific configuration action. Then arguments follow. Often the first argument
is a device name. For example,
#02 (136)attach    0:lustre-MDT0000-mdtlov  1:lov  2:lustre-MDT0000-mdtlov_UUID

This command will be setting up device “lustre-MDT0000-mdtlov” of type “lov”
with additional argument “lustre-MDT0000-mdtlov_UUID”. All these arguments are
packed into lustre configuration buffers ( struct lustre_cfg).

Another commands will be attaching device into the stack (like setup and
lov_modify_tgts).

Appendix 2. Sample Code
=======================

Lustre currently has 2 different OSD implementations:
- ldiskfs OSD under lustre/osd-ldiskfs
  http://git.whamcloud.com/?p=fs/lustre-release.git;a=tree;f=lustre/osd-ldiskfs;hb=HEAD
- ZFS OSD under lustre/zfs-osd
  http://git.whamcloud.com/?p=fs/lustre-release.git;a=tree;f=lustre/osd-zfs;hb=HEAD