-A cl_lock is a state machine. This requires some clarification. One of the
-goals of CLIO is to make IO path non-blocking, or at least to make it easier to
-make it non-blocking in the future. Here `non-blocking' means that when a
-system call (read, write, truncate) reaches a situation where it has to wait
-for a communication with the server, it should--instead of waiting--remember
-its current state and switch to some other work. That is, instead of waiting
-for a lock enqueue, the client should proceed doing IO on the next stripe, etc.
-Obviously this is rather radical redesign, and it is not planned to be fully
-implemented at this time. Instead we are putting some infrastructure in place
-that would make it easier to do asynchronous non-blocking IO in the future.
-Specifically, where the old locking code goes to sleep (waiting for enqueue,
-for example), the new code returns cl_lock_transition::CLO_WAIT. When the
-enqueue reply comes, its completion handler signals that the lock state-machine
-is ready to move to the next state. There is some generic code in cl_lock.c
-that sleeps, waiting for these signals. As a result, for users of this
-cl_lock.c code, it looks like locking is done in the normal blocking fashion,
-and at the same time it is possible to switch to the non-blocking locking
-(simply by returning cl_lock_transition::CLO_WAIT from cl_lock.c functions).
-
-For a description of state machine states and transitions see enum
-cl_lock_state.
-
-There are two ways to restrict a set of states which a lock might move to:
-
-- Placing a "hold" on a lock guarantees that the lock will not be moved into
- cl_lock_state::CLS_FREEING state until the hold is released. A hold can only
- be acquired on a lock that is not in cl_lock_state::CLS_FREEING. All holds on
- a lock are counted in cl_lock::cll_holds. A hold protects the lock from
- cancellation and destruction. Requests to cancel and destroy a lock on hold
- will be recorded, but only honoured when the last hold on a lock is released;
-
-- Placing a "user" on a lock guarantees that lock will not leave the set of
- states cl_lock_state::CLS_NEW, cl_lock_state::CLS_QUEUING,
- cl_lock_state::CLS_ENQUEUED and cl_lock_state::CLS_HELD, once it enters this
- set. That is, if a user is added onto a lock in a state not from this set, it
- doesn't immediately force the lock to move to this set, but once the lock
- enters this set it will remain there until all users are removed. Lock users
- are counted in cl_lock::cll_users.
-
- A user is used to assure that the lock is not cancelled or destroyed while it
- is being enqueued or actively used by some IO.
-
- Currently, a user always comes with a hold (cl_lock_invariant() checks that a
- number of holds is not less than a number of users).
-
-Lock "users" are used by the top-level IO code to guarantee that a lock is not
-cancelled when IO it protects is going on. Lock "holds" are used by a top-lock
-(LOV code) to guarantee that its sub-locks are in an expected state.
-
-5.4. Lock Concurrency
-=====================
-
-The following describes how the lock state-machine operates. The fields of
-struct cl_lock are protected by the cl_lock::cll_guard mutex.
-
-- The mutex is taken, and cl_lock::cll_state is examined.
-
-- For every state there are possible target states which the lock can move
- into. They are tried in order. Attempts to move into the next state are
- done by _try() functions in cl_lock.c:cl_{enqueue,unlock,wait}_try().
-
-- If the transition can be performed immediately, the state is changed and the
- mutex is released.
-
-- If the transition requires blocking, the _try() function returns
- cl_lock_transition::CLO_WAIT. The caller unlocks the mutex and goes to sleep,
- waiting for the possibility of a lock state change. It is woken up when some
- event occurs that makes lock state change possible (e.g., the reception of
- the reply from the server), and repeats the loop.
-
-Top-lock and sub-lock have separate mutices and the latter has to be taken
-first to avoid deadlock.
-
-To see an example of interaction of all these issues, take a look at the
-lov_cl.c:lov_lock_enqueue() function. It is called as part of cl_enqueue_try(),
-and tries to advance top-lock to the ENQUEUED state by advancing the
-state-machines of its sub-locks (lov_lock_enqueue_one()). Note also that it
-uses trylock to take the sub-lock mutex to avoid deadlock. It also has to
-handle CEF_ASYNC enqueue, when sub-locks enqueues have to be done in parallel
-(this is used for glimpse locks which cannot deadlock).
-
- +------------------>NEW
- | |
- | | cl_enqueue_try()
- | |
- | cl_unuse_try() V
- | +--------------QUEUING (*)
- | | |
- | | | cl_enqueue_try()
- | | |
- | | cl_unuse_try() V
- sub-lock | +-------------ENQUEUED (*)
- cancelled | | |
- | | | cl_wait_try()
- | | |
- | | V
- | | HELD<---------+
- | | | |
- | | | |
- | | cl_unuse_try() | |
- | | | |
- | | V | cached
- | +------------>UNLOCKING (*) | lock found
- | | |
- | cl_unuse_try() | |
- | | |
- | | | cl_use_try()
- | V |
- +------------------CACHED---------+
- |
- (Cancelled)
- |
- V
- FREEING
-
-5.5. Shared Sub-locks
-=====================