From: tappro Date: Sat, 26 Feb 2005 08:56:34 +0000 (+0000) Subject: HLD document for SMFS(Storage Management Filesystem) X-Git-Tag: v1_7_100~1601 X-Git-Url: https://git.whamcloud.com/?a=commitdiff_plain;h=2d0ca68b3adb4d00c8466707f6f572e237cdb64a;p=fs%2Flustre-release.git HLD document for SMFS(Storage Management Filesystem) --- diff --git a/lustre/smfs/doc/hld.lyx b/lustre/smfs/doc/hld.lyx new file mode 100644 index 0000000..75a5215 --- /dev/null +++ b/lustre/smfs/doc/hld.lyx @@ -0,0 +1,1098 @@ +#LyX 1.3 created this file. For more info see http://www.lyx.org/ +\lyxformat 221 +\textclass article +\begin_preamble +\usepackage{listings} +\usepackage{color} +\usepackage{courier} + +%\usepackage{pt} +%\usepackage{units} +%\usepackage{coz} + +%\usepackage{epsf} +%\usepackage{lncsexample} +%\conttrue +%\usepackage{times} +%\usepackage{top} +%\usepackage{graphicx} +\newcommand{\lst}[3] { + \noindent\rule[-0.3mm]{\textwidth}{0.3mm}\vspace{-0.3mm} + \lstinputlisting[caption={#2}, + label={#1}, + showstringspaces=false, + numbers=left, + stepnumber=1, + frame=bottomline, + language=C, + extendedchars=true, + basicstyle=\small\tt, + numberstyle=\tiny, + keywordstyle=\color{red}, + commentstyle=\color{cyan}, + stringstyle=\color{green}, + directivestyle=\color{magenta}, + emph={1, 2, 3, 4, 5, 6, 7, 8, 9, 0, NULL, lustre, CFS}, + emphstyle=\color{blue}, + breaklines=true, + #3] + {#1}\vspace{0.3mm} +} +\end_preamble +\language english +\inputencoding auto +\fontscheme times +\graphics default +\paperfontsize default +\spacing single +\papersize Default +\paperpackage a4 +\use_geometry 1 +\use_amsmath 0 +\use_natbib 0 +\use_numerical_citations 0 +\paperorientation portrait +\leftmargin 1in +\topmargin 0.5in +\rightmargin 0.5in +\bottommargin 0.5in +\secnumdepth 4 +\tocdepth 4 +\paragraph_separation skip +\defskip medskip +\quotes_language english +\quotes_times 2 +\papercolumns 1 +\papersides 1 +\paperpagestyle default + +\layout Title + + +\noun on +Storage Management File System +\layout Date + +14 February 2005 +\layout Author + +Peter Braam, Mike Pershin +\layout Standard + + +\begin_inset LatexCommand \tableofcontents{} + +\end_inset + + +\layout Section + +Requirements +\layout Standard + +In this document SMFS (Storage Management File System) is described. + It is required to do the following: +\layout Itemize + +works as transparent pseudo-filesystem on the top of real filesystem +\layout Itemize + +have access to Lustre's API like llog and btree +\layout Itemize + +be portable to other file systems than ext3 +\layout Itemize + +be able to call side-functions from other modules and plugins using own + API. +\layout Standard + +All features can exist together and should not affect bottom filesystem + in any way. +\layout Section + +Architectural overview +\layout Standard + +In order to achieve requirements SMFS is expected to have the following + major functionality divisions: +\layout Description + +Backstore\SpecialChar ~ +FS - main module that is placed in Lustre stack on top of backing + FS. + It is pseudo filesystem that just pass all operations to real one and can + invoke side-functions for each filesystem operation. + There is plugin API that can be used by special modules (plugins) to register + list of functions for various filesystem operations. +\layout Description + +Plugins - major reason for SMFS existence is possibility to call side-functions + for any filesystem operation. + Let's call module with such functionality as SMFS plugin. + Each plugin is initialized via options when SMFS is mounted and corresponds + to certain method tables. + Plugins should register in SMFS using plugins API after that SMFS will + call their functions for specified filesystem operations. + +\layout Description + +Upcalls - very similar with plugins but more generic. + Modules can use this API to place own hooks in SMFS. + It can be needed for notification or taking control from SMFS by upper-level. + +\layout Section +\pagebreak_top +High Level Design +\layout Subsection + + +\noun on +Functional Specification +\layout Subsubsection + +Backstore FS operations +\layout Standard + +SMFS creates copy of backstore FS objects when needed but setup own super/inode/ +file operations and save pointer to original one. + SMFS should not affect the Lustre IO path. + Backstore FS should remains operational in case of SMFS error as long as + it possible. +\layout Subsubsection + +SMFS plugins/upcalls managment +\layout Standard + +SMFS can registers/deregisters plugin. + Plugins are activated after SMFS initialization and can register own functions + in SMFS using special API. + +\layout Standard + +All plugins are linked in the list and their functions are invoked for several + filesystem operations if defined. + +\layout Standard + +Deactivated process starts when SMFS uninitialization occur or ioctl is + received from user. + Ioctl interface or procfs (sysfs later) can be used also for plugin management. +\layout Standard + +Upcall are similar to plugins but more generic. + There is upcall API in SMFS. + Upcalls are also organized in linked list and called one by one where needed. + +\layout Subsubsection + +SMFS plugin API +\layout Standard + +Plugins should be able to register and deregister with SMFS. + For these purposes SMFS has two methods +\layout Standard + + +\noun on +Prototype +\noun default +: +\layout LyX-Code + +int smfs_register_plugin(struct smfs_plugin); +\layout LyX-Code + +int smfs_deregister_plugin(int type); +\layout Standard + +Before calling this function plugin should fill struct smfs_plugin with + own data. + Registration process will check it and add to the list if possible. + Deregistration takes type of plugin, search it in list and delete it. +\layout Standard + +Plugins can insert own function hooks into SMFS. + Two major functions that can be setted by plugins are pre- and post-hook. + They will be called before filesystem operation and after it. +\layout Standard + + +\noun on +Prototype: +\layout LyX-Code + +/* this is prototype for plugin hook */ +\layout LyX-Code + +int smfs_hook_func (int hook_opcode, void * parameters); +\layout Standard + +Here we pass to plugin hook_opcode and several parameters. + For each opcode paramaters can be different. +\layout Standard + + +\emph on +Hook_opcode +\emph default + is a number from enumerated list of all possible operations where hooks + are used. +\layout Subsubsection + +SMFS upcall API +\layout Standard + +Plugin API is defined for using with filesystem operations specially. + For cases where more generic functionality is needed upcall API can be + used. + +\layout Standard + + +\noun on +Prototype: +\layout LyX-Code + +int smfs_upcall (int type, void * arg); +\layout Standard + +Here +\emph on +type +\emph default + is number from enumerated list of upcalls, +\emph on +arg +\emph default + is parameter or structure with parameters related to this upcall. + Upcalls can be used for notification purposes. + Upcalls are also widely used with plugins to get some data from they or + do some action. +\layout Standard + +Due to upcall origin there can be no some modules while SMFS already is + loaded. + Therefore it is possible that not all functionality presents at the moment. + Upcall API should notify caller about this. +\layout Subsection + + +\noun on +Use cases +\layout Standard + +There are several plugins that use plugin API. + They should fit in proposed API well. +\layout Subsubsection + +KML +\layout Standard + +KML module designed to support writeback caching. + KML is quite simple plugin that receive notification about FS operations + and write they to the log using llog API. +\layout Standard + +KML use post_op function for logging, that are called after FS operations. + It should have also pre_ops for transaction handling. + The SMFS functionality is sufficient for it. + +\layout Subsubsection + +LRU +\layout Standard + +LRU is designed for cache management in local MDS on client, but can be + used also for HSM in future. + LRU module defines both pre_op and post_op. + Pre_op is quit simple and is used mostly to check do we need purge cache + or not. + Post_op does cache operations. + There is special thread for purging cache, it can be awaken from pre_op + but. + SMFS plugin API functionality is sufficient for this module. +\layout Subsubsection + +COW +\layout Standard + +This module supports snapfs working over SMFS. + This module use all functionality of SMFS plugin API - pre_op and post_op. + Meanwhile it doesn't need extra functionality and fits in existent API. +\layout Subsection + + +\noun on +Logic Specification +\layout Subsubsection + +Backstore FS operations +\layout Standard + +For succefull functioning SMFS should maintain copies of many filesystem + objects such as superblock, inodes, files, etc. + and replace their operations with own. + Therefore each filesystem object in SMFS has its origin object in backstore + FS and there is pointer on it. + While operating SMFS should maintain own object in consistency with real + objects. +\layout Standard + +Example of superblock operation handling: +\layout LyX-Code + +smfs_dirty/write_inode(struct inode * inode) +\layout LyX-Code + +{ +\layout LyX-Code + + /* getting saved backstore fs inode */ +\layout LyX-Code + + backfs_inode = I2CI(inode); +\layout LyX-Code + + /* here can be some pre-work */ +\layout LyX-Code + + ... +\layout LyX-Code + + /* call real operation */ +\layout LyX-Code + + backfs_sb->s_op->dirty/write_inode(backfs_inode); +\layout LyX-Code + + +\layout LyX-Code + + /* postprocessing */ +\layout LyX-Code + + duplicate_inode(inode, backfs_inode); +\layout LyX-Code + +} +\layout Subsubsection + +Fsfilt operations handlilng +\layout Standard + +Fsfilt operations are handled like ordinary FS operations. + SMFS store fsfilt operations that corresponds to backstore FS, fsfilt module + just use operations declared for SMFS. + SMFS calls real operation with some framework. + There are several common steps to pass fsfilt operations to real FS: +\layout Itemize + +get backfs sb and inode from SMFS struct inode +\layout Itemize + +get backfs fsfilt_operations from superblock +\layout Itemize + +call needed operation using backfs objects +\layout Standard + +Transactions in SMFS are handled using fsfilter operation for backstore + FS. + SMFS make fs_start and fs_commit in all cases where it is needed. + Therefore these all extra changes will be included into current transaction: +\layout LyX-Code + +smfs_trans_start() +\layout LyX-Code + +... +\layout LyX-Code + +SMFS_HOOK() +\layout LyX-Code + +... +\layout LyX-Code + +smfs_trans_commit() +\layout Standard + +Transaction handler are passed to plugin in hook and can be used by plugin + to do proper transaction handling when it is needed. +\layout Subsubsection + +SMFS plugin activation/deactivation +\layout Standard + +Now all plugins are compiled in SMFS module and are initialized via options + when SMFS is mounting. + +\layout Standard + +Options should be passed as mountfsoptions like this: +\emph on +... + [kml|cache|snap...] +\layout Standard + +SMFS parses module options and call initialization method for selected plugins. + It is possible to activate/deactivate plugins using ioctl (procfs or sysfs) + later. + +\layout Standard + +While initialization plugins should registers with SMFS: +\layout LyX-Code + +smfs_register_plugin(parameters); +\layout Standard + +Parameters are struct smfs_plugin filled with valid data: +\layout Itemize + +type of plugin - to distinguish it from anothers, +\layout Itemize + +pre_op function - function that will be called before fs operation, +\layout Itemize + +post_op function - function that will be called after fs operation, +\layout Itemize + +helper function - helper function. + See Plugin API, +\layout Itemize + +any private data - data which will be return to plugin in each call. +\layout Subsubsection + +SMFS hook invokation +\layout Standard + +SMFS place special wrapper in own filesystem operations to call plugins + hook: +\layout LyX-Code + +/* wrapper that calls all hooks walking through the list */ +\layout LyX-Code + +#define SMFS_HOOK(opcode,...) +\backslash + +\layout LyX-Code + +do { +\backslash + +\layout LyX-Code + + smfs_hooks_op * hops; +\backslash + +\layout LyX-Code + + ... + +\backslash + +\layout LyX-Code + + list_for_each_entry(plugin_list, hops, smh_list) { +\backslash + +\layout LyX-Code + + ... + +\backslash + +\layout LyX-Code + + rc = hops->smh_hook_op(opcode, ...); +\backslash + +\layout LyX-Code + + } +\backslash + +\layout LyX-Code + + ... + +\backslash + +\layout LyX-Code + +} while(0) +\layout Standard + +Hooks are placed in SMFS methods before and after calling backstore FS operation +s: +\layout LyX-Code + +/* this is how to SMFS uses hooks */ +\layout LyX-Code + +smfs_some_op() +\layout LyX-Code + +{ +\layout LyX-Code + + struct inode * backfs_inode = I2CI(inode); +\layout LyX-Code + + struct smfs_file_info *sfi; +\layout LyX-Code + + +\layout LyX-Code + + SMFS_HOOK(..., hook_opcode , ..., PRE_HOOK, ...); +\layout LyX-Code + + backfs_inode->i_fop->some_op(sfi->c_file, ...); +\layout LyX-Code + + SMFS_HOOK(..., hook_opcode, ..., POST_HOOK, ...); +\layout LyX-Code + +} +\layout Subsubsection + +SMFS upcall usage +\layout Standard + +Upcalls can be placed in any place in SMFS, where it is needed. +\layout LyX-Code + +/* example of upcalls handling */ +\layout LyX-Code + +int smfs_upcall(int type, void * arg) +\layout LyX-Code + +{ +\layout LyX-Code + + list_for_each(upcall_list) +\layout LyX-Code + + entry->upcall(type, arg); +\layout LyX-Code + +} +\layout LyX-Code + +\layout LyX-Code + +/* example of using upcall */ +\layout LyX-Code + +int check_uninitialization() +\layout LyX-Code + +{ +\layout LyX-Code + + ... +\layout LyX-Code + + smfs_upcall(SMFS_UP_SMTH, ¶meter); +\layout LyX-Code + + ... +\layout LyX-Code + +} +\layout Section +\pagebreak_top +Plugins API +\layout Subsection + + +\noun on +Functional Specification +\layout Standard + +Each plugin has own unique name and corresponding type. + There is enumerated list of plugins types. + Any new plugin should use next sequence number as type and name which not + used yet by other plugins. + SMFS exports two functions for plugins: +\emph on +smfs_register_plugin() +\emph default + and +\emph on +smfs_deregister_plugin(). + +\emph default + +\layout Standard + +SMFS defines format of function that can be used by plugins to invoke own + methods. + Plugins should pass pointers on desired function while registration. +\layout Standard + +After successfull registration plugin's function will be called when selected + FS operations are occur. + SMFS passes to plugin inode structre, dentry, hook opcode and fsfilt transactio +n handle if exists. + +\layout Standard + +Plugin should return zero value in case of successfull completion and error + code otherwise. + Plugin should decide how to handle error by itself. + In case of critical error it can even stops operating or deregisters itself + from SMFS. + If plugin have no function for selected +\emph on +hook_opcode +\emph default + it has to return also +\emph on +0. +\layout Subsection + + +\noun on +Logic Specificaion +\layout Paragraph* + +Plugin identification +\layout Standard + +There are defined types of plugins: +\layout LyX-Code + +#define SMFS_PLG_KML 0x01 +\layout LyX-Code + +#define SMFS_PLG_LRU 0x02 +\layout LyX-Code + +... +\layout LyX-Code + +\layout Paragraph* + +Plugin data +\layout Standard + +Main structure for communication between plugin and SMFS. + +\layout LyX-Code + +struct smfs_plugin { +\layout LyX-Code + + struct list_head plist; +\layout LyX-Code + + int type; +\layout LyX-Code + + smfs_hook_func * pre_op; +\layout LyX-Code + + smfs_hook_func * post_op; +\layout LyX-Code + + smfs_helper * helper; +\layout LyX-Code + + void * plugin_data; +\layout LyX-Code + +} +\layout Standard + +Where +\layout Itemize + + +\emph on +type +\emph default + - plugins type, +\layout Itemize + + +\emph on +pre_hook_func +\emph default + - function to handle pre-operations, +\layout Itemize + + +\emph on +post_hook_func +\emph default + - function to handle post-hook operation, +\layout Itemize + +helper - call to various helper functions needed by SMFS, +\layout Itemize + +plugin_data - private plugin data. +\layout Paragraph* + +Plugin registration +\layout Standard + +SMFS provide following method for registration: +\layout LyX-Code + +int smfs_register_plugin (struct smfs_plugin *); +\layout Standard + +SMFS will return 0 in case of successfull registration or error code otherwise. + +\layout Paragraph* + +Plugins operation +\layout Standard + +Plugin's hook function must have the same type: +\layout LyX-Code + +typedef int smfs_hook_func (int opcode, void * parameter); +\layout Standard + +Parameter can contains: +\layout Itemize + +inode - SMFS inode +\layout Itemize + +dentry - SMFS dentry +\layout Itemize + +new_inode - used in rename +\layout Itemize + +new_dentry - used in rename +\layout Itemize + +opcode - hook code +\layout Itemize + +handle - fsfilt transaction if exists +\layout Standard + +There are opcodes for now: +\layout LyX-Code + +#define HOOK_CREATE 1 +\layout LyX-Code + +#define HOOK_LOOKUP 2 +\layout LyX-Code + +#define HOOK_LINK 3 +\layout LyX-Code + +#define HOOK_UNLINK 4 +\layout LyX-Code + +#define HOOK_SYMLINK 5 +\layout LyX-Code + +#define HOOK_MKDIR 6 +\layout LyX-Code + +#define HOOK_RMDIR 7 +\layout LyX-Code + +#define HOOK_MKNOD 8 +\layout LyX-Code + +#define HOOK_RENAME 9 +\layout LyX-Code + +#define HOOK_SETATTR 10 +\layout LyX-Code + +#define HOOK_WRITE 11 +\layout LyX-Code + +#define HOOK_READDIR 12 +\layout LyX-Code + +#define HOOK_MAX 12 +\layout Standard + +Plugin may use opcode as index for array of some type. + It can be table of functions for each hook type for example: +\layout LyX-Code + +static cache_hook_op cache_space_hook_ops[HOOK_MAX + 1] = { +\layout LyX-Code + + [HOOK_CREATE] cache_space_hook_create, +\layout LyX-Code + + [HOOK_LOOKUP] cache_space_hook_lookup, +\layout LyX-Code + + [HOOK_LINK] cache_space_hook_link, +\layout LyX-Code + + [HOOK_UNLINK] cache_space_hook_unlink, +\layout LyX-Code + + [HOOK_SYMLINK] cache_space_hook_create, +\layout LyX-Code + + [HOOK_MKDIR] cache_space_hook_mkdir, +\layout LyX-Code + + [HOOK_RMDIR] cache_space_hook_rmdir, +\layout LyX-Code + + [HOOK_MKNOD] cache_space_hook_create, +\layout LyX-Code + + [HOOK_RENAME] cache_space_hook_rename, +\layout LyX-Code + + [HOOK_SETATTR] NULL, +\layout LyX-Code + + [HOOK_WRITE] NULL, +\layout LyX-Code + + [HOOK_READDIR] NULL, +\layout LyX-Code + +}; +\layout Subsubsection* + +Helpers +\layout Standard + +Plugins can register helper function. + There are several places where helpers are needed: +\layout Itemize + +smfs uninitialization - after deleting plugin from list it can be usefull + to invoke plugin deactivation +\layout Itemize + +init_inode_info - several plugins can need to initialize own info on per-inode + basis +\layout Itemize + +calculate size for transaction - if plugin will participate into transaction + it should return extra size value. +\layout Itemize + +etc. +\layout LyX-Code + +smfs_helper(int code, void * parameter); +\layout Standard + +Where +\emph on +code +\emph default + is type of helper: +\layout LyX-Code + +#define SMFS_HLP_EXIT 0x01 +\layout LyX-Code + +#define SMFS_HLP_INODE 0x02 +\layout LyX-Code + +#define SMFS_HLP_TSIZE 0x03 +\layout Subsection + + +\noun on +Use cases +\layout Standard + +SMFS returns zero value in case of successfull registration/deregistration + or error code otherwise. +\layout Standard + +Plugins will return zero value in case of successfull execution hook function + or error code otherwise. +\layout Paragraph* + +New plugin design example +\layout LyX-Code + +//add new type +\layout LyX-Code + +#define SMFS_PLG_NEW 0x03 +\layout LyX-Code + +\layout LyX-Code + +//prepare hook functions +\layout LyX-Code + +int new_plugin_pre_function(int opcode, void * parameter) +\layout LyX-Code + +{ +\layout LyX-Code + + //let it be just statistic collector +\layout LyX-Code + + array[opcode]++; +\layout LyX-Code + +} +\layout LyX-Code + +\layout LyX-Code + +int new_plugin_init () +\layout LyX-Code + +{ +\layout LyX-Code + + struct smfs_plugin data; +\layout LyX-Code + + //prepare data +\layout LyX-Code + + data.type = SMFS_PLG_NEW; +\layout LyX-Code + + data.pre_op = &new_plugin_pre_function; +\layout LyX-Code + + data.post_op = NULL; +\layout LyX-Code + + //regisration here +\layout LyX-Code + + err = smfs_register_plugin(&data); +\layout LyX-Code + +\layout LyX-Code + +} +\layout Section +\pagebreak_top +Upcall API +\layout Standard + +SMFS provides the ability to place a call to other modules. + This can be needed when some notification needed from SMFS for example. +\layout Standard + +There are quite a few things to keep in mind: +\layout Enumerate + +The module that causes the mount of smfs, like the low level OSS and MDS + server will want to register methods for upcalls. +\layout Enumerate + +Other modules that are not mounting need to be able to find smfs and register + upcall methods. +\layout Enumerate + +When these registration of upcall methods have not happened, we should get + reasonable error handling. + Oopses and hangs are not permitted. +\layout Subsection + + +\noun on +Functional Specification +\layout Standard + +We can use mechanism similar to plugin API but it is more generic. + Upcall is invoked in particular place or when some conditions are occure. + So each upcall has own type. + There can be several upcalls with one type simultaneously. + Therefore we can introduce set of types for various upcalls and make generic + call function like +\emph on +smfs_upcall(int type, void * arg) +\emph default +. + Module should register upcall handler in SMFS with help of registering + function. + SMFS has list of upcalls and will call functions for each upcall type walking + through list. + +\layout Subsection + + +\noun on +Logic Specification +\layout Standard + +There are defined types of upcalls: +\layout LyX-Code + +#define SMFS_UP_SMTH 0x01 +\layout LyX-Code + +#define SMFS_UP_... + 0x02 +\layout LyX-Code + +... +\layout LyX-Code + +\layout Standard + +SMFS provide following method for registration/deregistration: +\layout LyX-Code + +int smfs_register_upcall (struct smfs_upcall *); +\layout LyX-Code + +int smfs_deregister_upcall (struct smfs_upcall *); +\layout Standard + +SMFS will return 0 in case of successfull registration or error code otherwise. +\layout Standard + +SMFS will invoke function for each upcall: +\layout LyX-Code + +typedef int smfs_upcall_func (int upcall_type, void * arg); +\layout Standard + +Here: +\layout Itemize + +upcall_type - upcall type. + It is define where upcall was invoked and under what conditions; +\layout Itemize + +arg - any desired arguments. +\layout Subsection + + +\noun on +Use cases +\layout Standard + +SMFS returns zero value in case of successfull registration/deregistration + or error code otherwise. +\layout Standard + +Modules should be aware about conditions under what upcall is called - locking, + for example. + +\the_end