+Lustre Protocol Documentation
+=============================
+Andrew Uselton <andrew.c.uselton@intel.com>
+v0.0, January 2015
+:author: Andrew Uselton
+:doctype: article
+:Author Initials: ACU
+:toc:
+:icons:
+:numbered:
+:imagesdir: ./figures
+:website: http://lustre.org/
+:keywords: PtlRPC, Lustre, Protocol
+
+
+:numbered!: [abstract] Abstract -------- The Lustre parallel file
+system <<lustre>> provides a global POSIX <<POSIX>> namespace for the
+computing resources of a data center. Lustre runs on Linux-based hosts
+via kernel modules, and delegates block storage management to the
+back-end servers while providing object-based storage to its
+clients. Servers are responsible for both data objects (the contents
+of actual files) and index objects (for directory information). Data
+objects are gathered on Object Storage Servers (OSSs), and index
+objects are stored on MetaData Storage Servers (MDSs). Each back-end
+storage volume is a target with Object Storage Targets (OSTs) on OSSs,
+and MetaData Storage Targets (MDTs) on MDSs. Clients assemble the
+data from the MDT(s) and OST(s) to present a single coherent
+POSIX-compliant file system. The clients and servers communicate and
+coordinate among themselves via network protocols. A low-level
+protocol, LNet, abstracts the details of the underlying networking
+hardware and presents a uniform interface, originally based on Sandia
+Portals <<PORTALS>>, to Lustre clients and servers. Lustre, in turn,
+layers its own protocol PtlRPC atop LNet. This document describes the
+Lustre protocols, including how they are implemeted via PtlRPC and the
+Lustre Distributed Lock Manager (based on the VAX/VMS Distributed Lock
+Manager <<VAX_DLM>>). This document does not describe Lustre itself in
+any detail, except where doing so explains concepts that allow this
+document to be self-contained.
+
+:numbered:
+
+Overview
+--------
+
+'Content to be provided'
+
+Messages
+--------
+
+These are the messages that traverse the network using PTLRPC.
+
+[NOTE]
+This initial list combines some actual message names or types with the
+POSIX semantic operations they are being used to implement, as well as
+a few other underlying mechanisms (cf. "grant"). A subsequent
+refinement will separate the various items and relate them to one
+another.
+
+Client-MDS RPCs for POSIX namespace operations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+'Content to be provided'
+
+=== mount ===
+
+'Content to be provided'
+
+=== unmount ===
+
+'Content to be provided'
+
+=== create ===
+
+'Content to be provided'
+
+=== open ===
+
+'Content to be provided'
+
+=== close ===
+
+'Content to be provided'
+
+=== unlink ===
+
+'Content to be provided'
+
+=== mkdir ===
+
+image:mkdir1.png[mkdir]
+
+=== rmdir ===
+
+'Content to be provided'
+
+=== rename ===
+
+'Content to be provided'
+
+=== link ===
+
+'Content to be provided'
+
+=== symlink ===
+
+'Content to be provided'
+
+=== getattr ===
+
+'Content to be provided'
+
+=== setattr ===
+
+'Content to be provided'
+
+=== statfs ===
+
+'Content to be provided'
+
+=== ... ===
+
+'Content to be provided'
+
+
+Client-MDS RPCs for internal state management
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+'Content to be provided'
+
+=== connect ===
+
+'Content to be provided'
+
+=== disconnect ===
+
+'Content to be provided'
+
+=== FLD ===
+
+'Content to be provided'
+
+=== SEQ ===
+
+'Content to be provided'
+
+=== PING ===
+
+'Content to be provided'
+
+=== LDLM ===
+
+'Content to be provided'
+
+=== ... ===
+
+'Content to be provided'
+
+Client-OSS RPCs for IO Operations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+'Content to be provided'
+
+=== read ===
+
+'Content to be provided'
+
+=== write ===
+
+'Content to be provided'
+
+=== truncate ===
+
+'Content to be provided'
+
+=== setattr ===
+
+'Content to be provided'
+
+=== grant ===
+
+'Content to be provided'
+
+=== ... ===
+
+'Content to be provided'
+
+MDS-OSS RPCs for internal state management
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+'Content to be provided'
+
+=== object precreation ===
+
+'Content to be provided'
+
+=== orphan recovery ===
+
+'Content to be provided'
+
+=== UID/GID change ===
+
+'Content to be provided'
+
+=== unlink ===
+
+'Content to be provided'
+
+=== ... ===
+
+'Content to be provided'
+
+MDS-OSS RPCs for quota management
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+'Content to be provided'
+
+
+MDS-OSS OUT RPCs for distributed updates
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+'Content to be provided'
+
+=== DNE1 remote directories ===
+
+'Content to be provided'
+
+=== DNE2 striped directories ===
+
+'Content to be provided'
+
+=== LFSCK2/3 verification and repair ===
+
+'Content to be provided'
+
+Message Flows
+-------------
+
+ Each file operation (in Lustre) generates a set of messages in a
+ particular sequence. There is one sequence for any particular
+ concrete operation, but under varying circumstances the same file
+ operation may generate a different sequence.
+
+State Machines
+--------------
+
+ For each File operation, the collection of possible sequences of
+ messages is governed by a state machine.
+
+:numbered!:
+
+[glossary]
+Glossary
+--------
+Here are some common terms used in discussing Lustre, POSIX semantics,
+and the prtocols used to implement them.
+
+[glossary]
+Object Storage Server::
+ An object storage server (OSS) is a computer responsible for
+ running Lustre kernel services in support of managing bulk data
+ objects on the underlying storage. There can be multiple OSSs in a
+ Lustre file system.
+
+MetaData Server::
+ A metadata server (MDS) is a computer responsible for running the
+ Lustre kernel services in support of managing the POSIX-compliant
+ name space and the indices associating files in that name space with
+ the locations of their corresponding objects. As of v2.4 there can
+ be multiple MDSs in a Lustre file system.
+
+Object Storage Target::
+ An object storage target (OST) is the service provided by an OSS
+ that mediates the placement of data objects on the specific
+ underlying file system hardware. There can be multiple OSTs on a
+ given OSS.
+
+MetaData Target::
+ A metadata target (MDT) is the service provided by an MDS that
+ mediates the management of name space indices on the underlying file
+ system hardware. As of v2.4 there can be multiple MDTs on an MDS.
+
+server::
+ A computer providing a service, such as an OSS or an MDS
+
+target::
+ Storage available to be served, such as an OST or an MDT. Also the
+ service being provided.
+
+protocol::
+ An agreed upon formalism for communicating between two entities,
+ such as between two servers or between a client and a server.
+
+client::
+ A computer taking advantage of a service provided by a server, such
+ as a Lustre client using MDS(s) and OSS(s) to assemble a
+ POSIX-compliant file system with its namespace and data storage
+ capabilities.
+
+PtlRPC::
+ The protocol (or set of protocols) implemented via RPCs that is
+ (are) employed by Lustre to communicate between its clients and
+ servers.
+
+Remote Procedure Call::
+ A mechanism for implementing operations involving one computer
+ acting on the behalf of another (RPC).
+
+LNet::
+ A lower level protocol employed by PtlRPC to abstract the mechanisms
+ provided by various hardware centric protocols, such as TCP or
+ Infiniband.
+
+[appendix]
+Concepts
+--------
+
+'Content to be provided'
+
+[appendix]
+License
+-------
+
+Copyright (C) Intel 2015
+
+This work is licensed under a Creative Commons Attribution-ShareAlike
+4.0 International License (CC BY-SA 4.0). See
+<https://creativecommons.org/licenses/by-sa/4.0/> for more detail.
+
+[bibliography]
+Bibliography
+------------
+Here is a selected list of references, including those cited in the
+foregoing text.
+
+[bibliography]
+- [[[lustre]]] 'Lustre'. http://lustre.opensfs.org
+- [[[POSIX]]] 'POSIX'. http://pubs.opengroup.org/onlinepubs/9699919799/
+- [[[PORTALS]]] 'The Portals 3.0 Message Passing
+ Interface Revision 1.1.'. Ron Brightwell, Trammel
+ Hudson, Rolf Riesen, and Arthur B. Maccabe. Technical
+ report, December 1999.
+- [[[VAX_DLM]]] 'The VAX/VMS Distributed Lock Manager'. W Snaman and
+ D Thiel. Digital Technical Journal, September 1987.
+- [[[Barton_Dilger]]] 'Lustre'. Eric Barton and Andreas Dilger. A book
+ on parallel file systems. Chapter 8. High
+ Performance Parallel I/O, Prabhat and Quincey
+ Koziol, Chapman and Hall/CRC Press, 2014, ISBN:
+ 978-1466582347.
+
+
git://git.whamcloud.com / doc / protocol.git / commitdiff