#LyX 1.3 created this file. For more info see http://www.lyx.org/
\lyxformat 221
\textclass article
\begin_preamble
\usepackage{listings}
\usepackage{color}
\usepackage{courier}

%\usepackage{pt}
%\usepackage{units}
%\usepackage{coz}

%\usepackage{epsf}
%\usepackage{lncsexample}
%\conttrue
%\usepackage{times}
%\usepackage{top}
%\usepackage{graphicx}
\newcommand{\lst}[3] {
        \noindent\rule[-0.3mm]{\textwidth}{0.3mm}\vspace{-0.3mm}
        \lstinputlisting[caption={#2},
        label={#1},
        showstringspaces=false, 
        numbers=left, 
        stepnumber=1,
        frame=bottomline,
        language=C,
        extendedchars=true,
        basicstyle=\small\tt,
        numberstyle=\tiny,
        keywordstyle=\color{red},
        commentstyle=\color{cyan},
        stringstyle=\color{green},
        directivestyle=\color{magenta},
        emph={1, 2, 3, 4, 5, 6, 7, 8, 9, 0, NULL, lustre, CFS},
        emphstyle=\color{blue},
        breaklines=true,
        #3]
        {#1}\vspace{0.3mm}
}
\end_preamble
\language english
\inputencoding auto
\fontscheme times
\graphics default
\paperfontsize default
\spacing single 
\papersize Default
\paperpackage a4
\use_geometry 1
\use_amsmath 0
\use_natbib 0
\use_numerical_citations 0
\paperorientation portrait
\leftmargin 1in
\topmargin 0.5in
\rightmargin 0.5in
\bottommargin 0.5in
\secnumdepth 4
\tocdepth 4
\paragraph_separation skip
\defskip medskip
\quotes_language english
\quotes_times 2
\papercolumns 1
\papersides 1
\paperpagestyle default

\layout Title


\noun on 
Storage Management File System
\layout Date

14 February 2005
\layout Author

Peter Braam, Mike Pershin
\layout Standard


\begin_inset LatexCommand \tableofcontents{}

\end_inset 


\layout Section

Requirements
\layout Standard

In this document SMFS (Storage Management File System) is described.
 It is required to do the following:
\layout Itemize

works as transparent pseudo-filesystem on the top of real filesystem
\layout Itemize

have access to Lustre's API like llog and btree 
\layout Itemize

be portable to other file systems than ext3
\layout Itemize

be able to call side-functions from other modules and plugins using own
 API.
\layout Standard

All features can exist together and should not affect bottom filesystem
 in any way.
\layout Section

Architectural overview
\layout Standard

In order to achieve requirements SMFS is expected to have the following
 major functionality divisions:
\layout Description

Backstore\SpecialChar ~
FS - main module that is placed in Lustre stack on top of backing
 FS.
 It is pseudo filesystem that just pass all operations to real one and can
 invoke side-functions for each filesystem operation.
 There is plugin API that can be used by special modules (plugins) to register
 list of functions for various filesystem operations.
\layout Description

Plugins - major reason for SMFS existence is possibility to call side-functions
 for any filesystem operation.
 Let's call module with such functionality as SMFS plugin.
 Each plugin is initialized via options when SMFS is mounted and corresponds
 to certain method tables.
 Plugins should register in SMFS using plugins API after that SMFS will
 call their functions for specified filesystem operations.
 
\layout Description

Upcalls - very similar with plugins but more generic.
 Modules can use this API to place own hooks in SMFS.
 It can be needed for notification or taking control from SMFS by upper-level.
 
\layout Section
\pagebreak_top 
High Level Design
\layout Subsection


\noun on 
Functional Specification
\layout Subsubsection

Backstore FS operations
\layout Standard

SMFS creates copy of backstore FS objects when needed but setup own super/inode/
file operations and save pointer to original one.
 SMFS should not affect the Lustre IO path.
 Backstore FS should remains operational in case of SMFS error as long as
 it possible.
\layout Subsubsection

SMFS plugins/upcalls managment
\layout Standard

SMFS can registers/deregisters plugin.
 Plugins are activated after SMFS initialization and can register own functions
 in SMFS using special API.
 
\layout Standard

All plugins are linked in the list and their functions are invoked for several
 filesystem operations if defined.
 
\layout Standard

Deactivated process starts when SMFS uninitialization occur or ioctl is
 received from user.
 Ioctl interface or procfs (sysfs later) can be used also for plugin management.
\layout Standard

Upcall are similar to plugins but more generic.
 There is upcall API in SMFS.
 Upcalls are also organized in linked list and called one by one where needed.
 
\layout Subsubsection

SMFS plugin API
\layout Standard

Plugins should be able to register and deregister with SMFS.
 For these purposes SMFS has two methods
\layout Standard


\noun on 
Prototype
\noun default 
:
\layout LyX-Code

int smfs_register_plugin(struct smfs_plugin);
\layout LyX-Code

int smfs_deregister_plugin(int type);
\layout Standard

Before calling this function plugin should fill struct smfs_plugin with
 own data.
 Registration process will check it and add to the list if possible.
 Deregistration takes type of plugin, search it in list and delete it.
\layout Standard

Plugins can insert own function hooks into SMFS.
 Two major functions that can be setted by plugins are pre- and post-hook.
 They will be called before filesystem operation and after it.
\layout Standard


\noun on 
Prototype:
\layout LyX-Code

/* this is prototype for plugin hook */
\layout LyX-Code

int smfs_hook_func (int hook_opcode, void * parameters);
\layout Standard

Here we pass to plugin hook_opcode and several parameters.
 For each opcode paramaters can be different.
\layout Standard


\emph on 
Hook_opcode
\emph default 
 is a number from enumerated list of all possible operations where hooks
 are used.
\layout Subsubsection

SMFS upcall API
\layout Standard

Plugin API is defined for using with filesystem operations specially.
 For cases where more generic functionality is needed upcall API can be
 used.
 
\layout Standard


\noun on 
Prototype:
\layout LyX-Code

int smfs_upcall (int type, void * arg);
\layout Standard

Here 
\emph on 
type
\emph default 
 is number from enumerated list of upcalls, 
\emph on 
arg
\emph default 
 is parameter or structure with parameters related to this upcall.
 Upcalls can be used for notification purposes.
 Upcalls are also widely used with plugins to get some data from they or
 do some action.
\layout Standard

Due to upcall origin there can be no some modules while SMFS already is
 loaded.
 Therefore it is possible that not all functionality presents at the moment.
 Upcall API should notify caller about this.
\layout Subsection


\noun on 
Use cases
\layout Standard

There are several plugins that use plugin API.
 They should fit in proposed API well.
\layout Subsubsection

KML
\layout Standard

KML module designed to support writeback caching.
 KML is quite simple plugin that receive notification about FS operations
 and write they to the log using llog API.
\layout Standard

KML use post_op function for logging, that are called after FS operations.
 It should have also pre_ops for transaction handling.
 The SMFS functionality is sufficient for it.
 
\layout Subsubsection

LRU
\layout Standard

LRU is designed for cache management in local MDS on client, but can be
 used also for HSM in future.
 LRU module defines both pre_op and post_op.
 Pre_op is quit simple and is used mostly to check do we need purge cache
 or not.
 Post_op does cache operations.
 There is special thread for purging cache, it can be awaken from pre_op
 but.
 SMFS plugin API functionality is sufficient for this module.
\layout Subsubsection

COW
\layout Standard

This module supports snapfs working over SMFS.
 This module use all functionality of SMFS plugin API - pre_op and post_op.
 Meanwhile it doesn't need extra functionality and fits in existent API.
\layout Subsection


\noun on 
Logic Specification
\layout Subsubsection

Backstore FS operations
\layout Standard

For succefull functioning SMFS should maintain copies of many filesystem
 objects such as superblock, inodes, files, etc.
 and replace their operations with own.
 Therefore each filesystem object in SMFS has its origin object in backstore
 FS and there is pointer on it.
 While operating SMFS should maintain own object in consistency with real
 objects.
\layout Standard

Example of superblock operation handling:
\layout LyX-Code

smfs_dirty/write_inode(struct inode * inode)
\layout LyX-Code

{
\layout LyX-Code

        /* getting saved backstore fs inode */
\layout LyX-Code

        backfs_inode = I2CI(inode);
\layout LyX-Code

        /* here can be some pre-work */
\layout LyX-Code

        ...
\layout LyX-Code

        /* call real operation */
\layout LyX-Code

        backfs_sb->s_op->dirty/write_inode(backfs_inode);
\layout LyX-Code

  
\layout LyX-Code

        /* postprocessing */
\layout LyX-Code

        duplicate_inode(inode, backfs_inode);
\layout LyX-Code

}
\layout Subsubsection

Fsfilt operations handlilng
\layout Standard

Fsfilt operations are handled like ordinary FS operations.
 SMFS store fsfilt operations that corresponds to backstore FS, fsfilt module
 just use operations declared for SMFS.
 SMFS calls real operation with some framework.
 There are several common steps to pass fsfilt operations to real FS:
\layout Itemize

get backfs sb and inode from SMFS struct inode
\layout Itemize

get backfs fsfilt_operations from superblock
\layout Itemize

call needed operation using backfs objects
\layout Standard

Transactions in SMFS are handled using fsfilter operation for backstore
 FS.
 SMFS make fs_start and fs_commit in all cases where it is needed.
 Therefore these all extra changes will be included into current transaction:
\layout LyX-Code

smfs_trans_start()
\layout LyX-Code

...
\layout LyX-Code

SMFS_HOOK()
\layout LyX-Code

...
\layout LyX-Code

smfs_trans_commit()
\layout Standard

Transaction handler are passed to plugin in hook and can be used by plugin
 to do proper transaction handling when it is needed.
\layout Subsubsection

SMFS plugin activation/deactivation
\layout Standard

Now all plugins are compiled in SMFS module and are initialized via options
 when SMFS is mounting.
 
\layout Standard

Options should be passed as mountfsoptions like this: 
\emph on 
...
 [kml|cache|snap...]
\layout Standard

SMFS parses module options and call initialization method for selected plugins.
 It is possible to activate/deactivate plugins using ioctl (procfs or sysfs)
 later.
 
\layout Standard

While initialization plugins should registers with SMFS:
\layout LyX-Code

smfs_register_plugin(parameters);
\layout Standard

Parameters are struct smfs_plugin filled with valid data:
\layout Itemize

type of plugin - to distinguish it from anothers,
\layout Itemize

pre_op function - function that will be called before fs operation,
\layout Itemize

post_op function - function that will be called after fs operation,
\layout Itemize

helper function - helper function.
 See Plugin API,
\layout Itemize

any private data - data which will be return to plugin in each call.
\layout Subsubsection

SMFS hook invokation
\layout Standard

SMFS place special wrapper in own filesystem operations to call plugins
 hook:
\layout LyX-Code

/* wrapper that calls all hooks walking through the list */
\layout LyX-Code

#define SMFS_HOOK(opcode,...)    
\backslash 

\layout LyX-Code

do {                                                          
\backslash 

\layout LyX-Code

       smfs_hooks_op * hops;                                  
\backslash 

\layout LyX-Code

       ...
                                                    
\backslash 

\layout LyX-Code

       list_for_each_entry(plugin_list, hops, smh_list) {     
\backslash 

\layout LyX-Code

              ...
                                             
\backslash 

\layout LyX-Code

              rc = hops->smh_hook_op(opcode, ...);            
\backslash 

\layout LyX-Code

       }                                                      
\backslash 

\layout LyX-Code

        ...
                                                   
\backslash 

\layout LyX-Code

} while(0)     
\layout Standard

Hooks are placed in SMFS methods before and after calling backstore FS operation
s:
\layout LyX-Code

/* this is how to SMFS uses hooks */
\layout LyX-Code

smfs_some_op()
\layout LyX-Code

{ 
\layout LyX-Code

        struct inode * backfs_inode = I2CI(inode);
\layout LyX-Code

        struct smfs_file_info *sfi;
\layout LyX-Code

       
\layout LyX-Code

        SMFS_HOOK(..., hook_opcode , ..., PRE_HOOK, ...);
\layout LyX-Code

        backfs_inode->i_fop->some_op(sfi->c_file, ...);
\layout LyX-Code

        SMFS_HOOK(..., hook_opcode, ..., POST_HOOK, ...);
\layout LyX-Code

} 
\layout Subsubsection

SMFS upcall usage
\layout Standard

Upcalls can be placed in any place in SMFS, where it is needed.
\layout LyX-Code

/* example of upcalls handling */
\layout LyX-Code

int smfs_upcall(int type, void * arg)
\layout LyX-Code

{
\layout LyX-Code

        list_for_each(upcall_list)
\layout LyX-Code

                entry->upcall(type, arg);
\layout LyX-Code

}
\layout LyX-Code

\layout LyX-Code

/* example of using upcall  */
\layout LyX-Code

int check_uninitialization()
\layout LyX-Code

{
\layout LyX-Code

        ...
\layout LyX-Code

        smfs_upcall(SMFS_UP_SMTH, &parameter);
\layout LyX-Code

        ...
\layout LyX-Code

}
\layout Section
\pagebreak_top 
Plugins API
\layout Subsection


\noun on 
Functional Specification
\layout Standard

Each plugin has own unique name and corresponding type.
 There is enumerated list of plugins types.
 Any new plugin should use next sequence number as type and name which not
 used yet by other plugins.
 SMFS exports two functions for plugins: 
\emph on 
smfs_register_plugin()
\emph default 
 and 
\emph on 
smfs_deregister_plugin().

\emph default 
 
\layout Standard

SMFS defines format of function that can be used by plugins to invoke own
 methods.
 Plugins should pass pointers on desired function while registration.
\layout Standard

After successfull registration plugin's function will be called when selected
 FS operations are occur.
 SMFS passes to plugin inode structre, dentry, hook opcode and fsfilt transactio
n handle if exists.
 
\layout Standard

Plugin should return zero value in case of successfull completion and error
 code otherwise.
 Plugin should decide how to handle error by itself.
 In case of critical error it can even stops operating or deregisters itself
 from SMFS.
 If plugin have no function for selected 
\emph on 
hook_opcode
\emph default 
 it has to return also 
\emph on 
0.
\layout Subsection


\noun on 
Logic Specificaion
\layout Paragraph*

Plugin identification
\layout Standard

There are defined types of plugins:
\layout LyX-Code

#define SMFS_PLG_KML 0x01
\layout LyX-Code

#define SMFS_PLG_LRU 0x02
\layout LyX-Code

...
\layout LyX-Code

\layout Paragraph*

Plugin data
\layout Standard

Main structure for communication between plugin and SMFS.
 
\layout LyX-Code

struct smfs_plugin {
\layout LyX-Code

        struct list_head plist;
\layout LyX-Code

        int type;
\layout LyX-Code

        smfs_hook_func * pre_op;
\layout LyX-Code

        smfs_hook_func * post_op;
\layout LyX-Code

        smfs_helper * helper;
\layout LyX-Code

        void * plugin_data;
\layout LyX-Code

}
\layout Standard

Where 
\layout Itemize


\emph on 
type
\emph default 
 - plugins type,
\layout Itemize


\emph on 
pre_hook_func
\emph default 
 - function to handle pre-operations, 
\layout Itemize


\emph on 
post_hook_func
\emph default 
 - function to handle post-hook operation,
\layout Itemize

helper - call to various helper functions needed by SMFS,
\layout Itemize

plugin_data - private plugin data.
\layout Paragraph*

Plugin registration
\layout Standard

SMFS provide following method for registration:
\layout LyX-Code

int smfs_register_plugin (struct smfs_plugin *);
\layout Standard

SMFS will return 0 in case of successfull registration or error code otherwise.
 
\layout Subsubsection*

Plugin deregistration
\layout Standard

SMFS provide following method for deregistration:
\layout LyX-Code

void * smfs_deregister_plugin (int type);
\layout Standard

SMFS will return private plugin data from struct smfs_plugin in case of
 successfull registration or NULL pointer otherwise.
 
\layout Paragraph*

Plugins operation
\layout Standard

Plugin's hook function must have the same type:
\layout LyX-Code

typedef int (*smfs_plg_hook) (int opcode, void * parameter, void * plg_private);
\layout Standard

Parameter can contains:
\layout Itemize

inode - SMFS inode
\layout Itemize

dentry - SMFS dentry
\layout Itemize

new_inode - used in rename
\layout Itemize

new_dentry - used in rename
\layout Itemize

opcode - hook code
\layout Itemize

handle - fsfilt transaction if exists
\layout Standard


\emph on 
Plg_private 
\emph default 
is private plugin data.
\layout Standard

There are opcodes for now:
\layout LyX-Code

#define HOOK_CREATE       1
\layout LyX-Code

#define HOOK_LOOKUP       2
\layout LyX-Code

#define HOOK_LINK         3
\layout LyX-Code

#define HOOK_UNLINK       4
\layout LyX-Code

#define HOOK_SYMLINK      5
\layout LyX-Code

#define HOOK_MKDIR        6
\layout LyX-Code

#define HOOK_RMDIR        7
\layout LyX-Code

#define HOOK_MKNOD        8
\layout LyX-Code

#define HOOK_RENAME       9
\layout LyX-Code

#define HOOK_SETATTR      10
\layout LyX-Code

#define HOOK_WRITE        11
\layout LyX-Code

#define HOOK_READDIR      12
\layout LyX-Code

#define HOOK_MAX          12
\layout Standard

Plugin may use opcode as index for array of some type.
 It can be table of functions for each hook type for example:
\layout LyX-Code

static  cache_hook_op cache_space_hook_ops[HOOK_MAX + 1] = {
\layout LyX-Code

        [HOOK_CREATE]     cache_space_hook_create,
\layout LyX-Code

        [HOOK_LOOKUP]     cache_space_hook_lookup,
\layout LyX-Code

        [HOOK_LINK]       cache_space_hook_link,
\layout LyX-Code

        [HOOK_UNLINK]     cache_space_hook_unlink,
\layout LyX-Code

        [HOOK_SYMLINK]    cache_space_hook_create,
\layout LyX-Code

        [HOOK_MKDIR]      cache_space_hook_mkdir,
\layout LyX-Code

        [HOOK_RMDIR]      cache_space_hook_rmdir,
\layout LyX-Code

        [HOOK_MKNOD]      cache_space_hook_create,
\layout LyX-Code

        [HOOK_RENAME]     cache_space_hook_rename,
\layout LyX-Code

        [HOOK_SETATTR]    NULL,
\layout LyX-Code

        [HOOK_WRITE]      NULL,
\layout LyX-Code

        [HOOK_READDIR]    NULL,
\layout LyX-Code

};
\layout Subsubsection*

Helpers
\layout Standard

Plugins can register helper function.
 There are several places where helpers are needed:
\layout Itemize

smfs uninitialization - after deleting plugin from list it can be usefull
 to invoke plugin deactivation
\layout Itemize

init_inode_info - several plugins can need to initialize own info on per-inode
 basis
\layout Itemize

calculate size for transaction - if plugin will participate into transaction
 it should return extra size value.
\layout Itemize

etc.
\layout LyX-Code

smfs_helper(int code, void * parameter, void * plg_private);
\layout Standard

Where 
\emph on 
code
\emph default 
 is type of helper:
\layout LyX-Code

#define SMFS_HLP_EXIT   0x01
\layout LyX-Code

#define SMFS_HLP_INODE  0x02
\layout LyX-Code

#define SMFS_HLP_TSIZE  0x03
\layout Subsection


\noun on 
Use cases
\layout Standard

SMFS returns zero value in case of successfull registration/deregistration
 or error code otherwise.
\layout Standard

Plugins will return zero value in case of successfull execution hook function
 or error code otherwise.
\layout Paragraph*

New plugin design example
\layout LyX-Code

//add new type
\layout LyX-Code

#define SMFS_PLG_NEW 0x03
\layout LyX-Code

\layout LyX-Code

//prepare hook functions
\layout LyX-Code

int new_plugin_pre_function(int opcode, void * parameter) 
\layout LyX-Code

{
\layout LyX-Code

        //let it be just statistic collector
\layout LyX-Code

        array[opcode]++;
\layout LyX-Code

}
\layout LyX-Code

\layout LyX-Code

int new_plugin_init ()
\layout LyX-Code

{
\layout LyX-Code

        struct smfs_plugin data;
\layout LyX-Code

        //prepare data
\layout LyX-Code

        data.type = SMFS_PLG_NEW;
\layout LyX-Code

        data.pre_op = &new_plugin_pre_function;
\layout LyX-Code

        data.post_op = NULL;
\layout LyX-Code

        //regisration here
\layout LyX-Code

        err = smfs_register_plugin(&data);
\layout LyX-Code

\layout LyX-Code

}
\layout Section
\pagebreak_top 
Upcall API
\layout Standard

SMFS provides the ability to place a call to other modules.
 This can be needed when some notification needed from SMFS for example.
\layout Standard

There are quite a few things to keep in mind:
\layout Enumerate

The module that causes the mount of smfs, like the low level OSS and MDS
 server will want to register methods for upcalls.
\layout Enumerate

Other modules that are not mounting need to be able to find smfs and register
 upcall methods.
\layout Enumerate

When these registration of upcall methods have not happened, we should get
 reasonable error handling.
 Oopses and hangs are not permitted.
\layout Subsection


\noun on 
Functional Specification
\layout Standard

We can use mechanism similar to plugin API but it is more generic.
 Upcall is invoked in particular place or when some conditions are occure.
 So each upcall has own type.
 There can be several upcalls with one type simultaneously.
 Therefore we can introduce set of types for various upcalls and make generic
 call function like 
\emph on 
smfs_upcall(int type, void * arg)
\emph default 
.
 Module should register upcall handler in SMFS with help of registering
 function.
 SMFS has list of upcalls and will call functions for each upcall type walking
 through list.
 
\layout Subsection


\noun on 
Logic Specification
\layout Standard

There are defined types of upcalls:
\layout LyX-Code

#define SMFS_UP_SMTH 0x01
\layout LyX-Code

#define SMFS_UP_...
  0x02
\layout LyX-Code

...
\layout LyX-Code

\layout Standard

SMFS provide following method for registration/deregistration:
\layout LyX-Code

int smfs_register_upcall (struct smfs_upcall *);
\layout LyX-Code

int smfs_deregister_upcall (struct smfs_upcall *);
\layout Standard

SMFS will return 0 in case of successfull registration or error code otherwise.
\layout Standard

SMFS will invoke function for each upcall:
\layout LyX-Code

typedef int smfs_upcall_func (int upcall_type, void * arg);
\layout Standard

Here:
\layout Itemize

upcall_type - upcall type.
 It is define where upcall was invoked and under what conditions;
\layout Itemize

arg - any desired arguments.
\layout Subsection


\noun on 
Use cases
\layout Standard

SMFS returns zero value in case of successfull registration/deregistration
 or error code otherwise.
\layout Standard

Modules should be aware about conditions under what upcall is called - locking,
 for example.
 
\the_end