Introduction ------------ [NOTE] I am leaving the introductory content here for now, but it is the last thing that should be written. The following is just a very early sketch, and will be revised entirely once the rest of the content has begun to shape up. The Lustre parallel file system provides a global POSIX namespace for the computing resources of a data center. Lustre runs on Linux-based hosts via kernel modules, and delegates block storage management to the back-end servers while providing object-based storage to its clients. Servers are responsible for both data objects (the contents of actual files) and index objects (for directory information). Data objects are gathered on Object Storage Servers (OSSs), and index objects are stored on Metadata Servers (MDSs). Each back-end storage volume is a target with Object Storage Targets (OSTs) on OSSs, and Metadata Targets (MDTs) on MDSs. Clients assemble the data from the MDTs and OSTs to present a single coherent POSIX-compliant file system. The clients and servers communicate and coordinate among themselves via network protocols. A low-level protocol, LNet, abstracts the details of the underlying networking hardware and presents a uniform interface, originally based on Sandia Portals <>, to Lustre clients and servers. Lustre, in turn, layers its own protocol atop LNet. This document describes the Lustre protocol. Lustre runs across multiple hosts, coordinating the activities among those hosts via the exchange of messages over a network. On each host, Lustre is implemented via a collection of Linux processes (often called "threads"). This discussion will refer to a more formalized notion of 'processes' that abstract some of the thread-level details. The description of the activities on each host comprise a collection of 'abstract processes'. Each abstract process may be thought of as a state machine, or automaton, following a fixed set of rules for how it consumes messages, changes state, and produces other messages. We speak of the 'behavior' of a process as shorthand for the management of its state and the rules governing what messages it can consume and produce. Processes communicate with each other via messages. The Lustre protocol is the collection of messages the processes exchange along with the rules governing the behavior of those processes. include::connection.txt[] include::file_id.txt[] include::path_lookup.txt[] include::ldlm.txt[] include::llog.txt[] include::security.txt[]