1 #LyX 1.3 created this file. For more info see http://www.lyx.org/
14 %\usepackage{lncsexample}
18 %\usepackage{graphicx}
19 \newcommand{\lst}[3] {
20 \noindent\rule[-0.3mm]{\textwidth}{0.3mm}\vspace{-0.3mm}
21 \lstinputlisting[caption={#2},
23 showstringspaces=false,
31 keywordstyle=\color{red},
32 commentstyle=\color{cyan},
33 stringstyle=\color{green},
34 directivestyle=\color{magenta},
35 emph={1, 2, 3, 4, 5, 6, 7, 8, 9, 0, NULL, lustre, CFS},
36 emphstyle=\color{blue},
46 \paperfontsize default
53 \use_numerical_citations 0
54 \paperorientation portrait
61 \paragraph_separation skip
63 \quotes_language english
67 \paperpagestyle default
73 Storage Management File System
79 Peter Braam, Mike Pershin
83 \begin_inset LatexCommand \tableofcontents{}
93 In this document SMFS (Storage Management File System) is described.
94 It is required to do the following:
97 works as transparent pseudo-filesystem on the top of real filesystem
100 have access to Lustre's API like llog and btree
103 be portable to other file systems than ext3
106 be able to call side-functions from other modules and plugins using own
110 All features can exist together and should not affect bottom filesystem
114 Architectural overview
117 In order to achieve requirements SMFS is expected to have the following
118 major functionality divisions:
121 Backstore\SpecialChar ~
122 FS - main module that is placed in Lustre stack on top of backing
124 It is pseudo filesystem that just pass all operations to real one and can
125 invoke side-functions for each filesystem operation.
126 There is plugin API that can be used by special modules (plugins) to register
127 list of functions for various filesystem operations.
130 Plugins - major reason for SMFS existence is possibility to call side-functions
131 for any filesystem operation.
132 Let's call module with such functionality as SMFS plugin.
133 Each plugin is initialized via options when SMFS is mounted and corresponds
134 to certain method tables.
135 Plugins should register in SMFS using plugins API after that SMFS will
136 call their functions for specified filesystem operations.
140 Upcalls - very similar with plugins but more generic.
141 Modules can use this API to place own hooks in SMFS.
142 It can be needed for notification or taking control from SMFS by upper-level.
151 Functional Specification
152 \layout Subsubsection
154 Backstore FS operations
157 SMFS creates copy of backstore FS objects when needed but setup own super/inode/
158 file operations and save pointer to original one.
159 SMFS should not affect the Lustre IO path.
160 Backstore FS should remains operational in case of SMFS error as long as
162 \layout Subsubsection
164 SMFS plugins/upcalls managment
167 SMFS can registers/deregisters plugin.
168 Plugins are activated after SMFS initialization and can register own functions
169 in SMFS using special API.
173 All plugins are linked in the list and their functions are invoked for several
174 filesystem operations if defined.
178 Deactivated process starts when SMFS uninitialization occur or ioctl is
180 Ioctl interface or procfs (sysfs later) can be used also for plugin management.
183 Upcall are similar to plugins but more generic.
184 There is upcall API in SMFS.
185 Upcalls are also organized in linked list and called one by one where needed.
187 \layout Subsubsection
192 Plugins should be able to register and deregister with SMFS.
193 For these purposes SMFS has two methods
203 int smfs_register_plugin(struct smfs_plugin);
206 int smfs_deregister_plugin(int type);
209 Before calling this function plugin should fill struct smfs_plugin with
211 Registration process will check it and add to the list if possible.
212 Deregistration takes type of plugin, search it in list and delete it.
215 Plugins can insert own function hooks into SMFS.
216 Two major functions that can be setted by plugins are pre- and post-hook.
217 They will be called before filesystem operation and after it.
225 /* this is prototype for plugin hook */
228 int smfs_hook_func (int hook_opcode, void * parameters);
231 Here we pass to plugin hook_opcode and several parameters.
232 For each opcode paramaters can be different.
239 is a number from enumerated list of all possible operations where hooks
241 \layout Subsubsection
246 Plugin API is defined for using with filesystem operations specially.
247 For cases where more generic functionality is needed upcall API can be
257 int smfs_upcall (int type, void * arg);
264 is number from enumerated list of upcalls,
268 is parameter or structure with parameters related to this upcall.
269 Upcalls can be used for notification purposes.
270 Upcalls are also widely used with plugins to get some data from they or
274 Due to upcall origin there can be no some modules while SMFS already is
276 Therefore it is possible that not all functionality presents at the moment.
277 Upcall API should notify caller about this.
285 There are several plugins that use plugin API.
286 They should fit in proposed API well.
287 \layout Subsubsection
292 KML module designed to support writeback caching.
293 KML is quite simple plugin that receive notification about FS operations
294 and write they to the log using llog API.
297 KML use post_op function for logging, that are called after FS operations.
298 It should have also pre_ops for transaction handling.
299 The SMFS functionality is sufficient for it.
301 \layout Subsubsection
306 LRU is designed for cache management in local MDS on client, but can be
307 used also for HSM in future.
308 LRU module defines both pre_op and post_op.
309 Pre_op is quit simple and is used mostly to check do we need purge cache
311 Post_op does cache operations.
312 There is special thread for purging cache, it can be awaken from pre_op
314 SMFS plugin API functionality is sufficient for this module.
315 \layout Subsubsection
320 This module supports snapfs working over SMFS.
321 This module use all functionality of SMFS plugin API - pre_op and post_op.
322 Meanwhile it doesn't need extra functionality and fits in existent API.
328 \layout Subsubsection
330 Backstore FS operations
333 For succefull functioning SMFS should maintain copies of many filesystem
334 objects such as superblock, inodes, files, etc.
335 and replace their operations with own.
336 Therefore each filesystem object in SMFS has its origin object in backstore
337 FS and there is pointer on it.
338 While operating SMFS should maintain own object in consistency with real
342 Example of superblock operation handling:
345 smfs_dirty/write_inode(struct inode * inode)
351 /* getting saved backstore fs inode */
354 backfs_inode = I2CI(inode);
357 /* here can be some pre-work */
363 /* call real operation */
366 backfs_sb->s_op->dirty/write_inode(backfs_inode);
375 duplicate_inode(inode, backfs_inode);
379 \layout Subsubsection
381 Fsfilt operations handlilng
384 Fsfilt operations are handled like ordinary FS operations.
385 SMFS store fsfilt operations that corresponds to backstore FS, fsfilt module
386 just use operations declared for SMFS.
387 SMFS calls real operation with some framework.
388 There are several common steps to pass fsfilt operations to real FS:
391 get backfs sb and inode from SMFS struct inode
394 get backfs fsfilt_operations from superblock
397 call needed operation using backfs objects
400 Transactions in SMFS are handled using fsfilter operation for backstore
402 SMFS make fs_start and fs_commit in all cases where it is needed.
403 Therefore these all extra changes will be included into current transaction:
421 Transaction handler are passed to plugin in hook and can be used by plugin
422 to do proper transaction handling when it is needed.
423 \layout Subsubsection
425 SMFS plugin activation/deactivation
428 Now all plugins are compiled in SMFS module and are initialized via options
429 when SMFS is mounting.
433 Options should be passed as mountfsoptions like this:
439 SMFS parses module options and call initialization method for selected plugins.
440 It is possible to activate/deactivate plugins using ioctl (procfs or sysfs)
445 While initialization plugins should registers with SMFS:
448 smfs_register_plugin(parameters);
451 Parameters are struct smfs_plugin filled with valid data:
454 type of plugin - to distinguish it from anothers,
457 pre_op function - function that will be called before fs operation,
460 post_op function - function that will be called after fs operation,
463 helper function - helper function.
467 any private data - data which will be return to plugin in each call.
468 \layout Subsubsection
473 SMFS place special wrapper in own filesystem operations to call plugins
477 /* wrapper that calls all hooks walking through the list */
480 #define SMFS_HOOK(opcode,...)
490 smfs_hooks_op * hops;
501 list_for_each_entry(plugin_list, hops, smh_list) {
512 rc = hops->smh_hook_op(opcode, ...);
531 Hooks are placed in SMFS methods before and after calling backstore FS operation
535 /* this is how to SMFS uses hooks */
544 struct inode * backfs_inode = I2CI(inode);
547 struct smfs_file_info *sfi;
553 SMFS_HOOK(..., hook_opcode , ..., PRE_HOOK, ...);
556 backfs_inode->i_fop->some_op(sfi->c_file, ...);
559 SMFS_HOOK(..., hook_opcode, ..., POST_HOOK, ...);
563 \layout Subsubsection
568 Upcalls can be placed in any place in SMFS, where it is needed.
571 /* example of upcalls handling */
574 int smfs_upcall(int type, void * arg)
580 list_for_each(upcall_list)
583 entry->upcall(type, arg);
591 /* example of using upcall */
594 int check_uninitialization()
603 smfs_upcall(SMFS_UP_SMTH, ¶meter);
617 Functional Specification
620 Each plugin has own unique name and corresponding type.
621 There is enumerated list of plugins types.
622 Any new plugin should use next sequence number as type and name which not
623 used yet by other plugins.
624 SMFS exports two functions for plugins:
626 smfs_register_plugin()
630 smfs_deregister_plugin().
636 SMFS defines format of function that can be used by plugins to invoke own
638 Plugins should pass pointers on desired function while registration.
641 After successfull registration plugin's function will be called when selected
642 FS operations are occur.
643 SMFS passes to plugin inode structre, dentry, hook opcode and fsfilt transactio
648 Plugin should return zero value in case of successfull completion and error
650 Plugin should decide how to handle error by itself.
651 In case of critical error it can even stops operating or deregisters itself
653 If plugin have no function for selected
657 it has to return also
667 Plugin identification
670 There are defined types of plugins:
673 #define SMFS_PLG_KML 0x01
676 #define SMFS_PLG_LRU 0x02
687 Main structure for communication between plugin and SMFS.
694 struct list_head plist;
700 smfs_hook_func * pre_op;
703 smfs_hook_func * post_op;
706 smfs_helper * helper;
729 - function to handle pre-operations,
736 - function to handle post-hook operation,
739 helper - call to various helper functions needed by SMFS,
742 plugin_data - private plugin data.
748 SMFS provide following method for registration:
751 int smfs_register_plugin (struct smfs_plugin *);
754 SMFS will return 0 in case of successfull registration or error code otherwise.
756 \layout Subsubsection*
758 Plugin deregistration
761 SMFS provide following method for deregistration:
764 void * smfs_deregister_plugin (int type);
767 SMFS will return private plugin data from struct smfs_plugin in case of
768 successfull registration or NULL pointer otherwise.
775 Plugin's hook function must have the same type:
778 typedef int (*smfs_plg_hook) (int opcode, void * parameter, void * plg_private);
781 Parameter can contains:
790 new_inode - used in rename
793 new_dentry - used in rename
799 handle - fsfilt transaction if exists
806 is private plugin data.
809 There are opcodes for now:
812 #define HOOK_CREATE 1
815 #define HOOK_LOOKUP 2
821 #define HOOK_UNLINK 4
824 #define HOOK_SYMLINK 5
836 #define HOOK_RENAME 9
839 #define HOOK_SETATTR 10
842 #define HOOK_WRITE 11
845 #define HOOK_READDIR 12
851 Plugin may use opcode as index for array of some type.
852 It can be table of functions for each hook type for example:
855 static cache_hook_op cache_space_hook_ops[HOOK_MAX + 1] = {
858 [HOOK_CREATE] cache_space_hook_create,
861 [HOOK_LOOKUP] cache_space_hook_lookup,
864 [HOOK_LINK] cache_space_hook_link,
867 [HOOK_UNLINK] cache_space_hook_unlink,
870 [HOOK_SYMLINK] cache_space_hook_create,
873 [HOOK_MKDIR] cache_space_hook_mkdir,
876 [HOOK_RMDIR] cache_space_hook_rmdir,
879 [HOOK_MKNOD] cache_space_hook_create,
882 [HOOK_RENAME] cache_space_hook_rename,
895 \layout Subsubsection*
900 Plugins can register helper function.
901 There are several places where helpers are needed:
904 smfs uninitialization - after deleting plugin from list it can be usefull
905 to invoke plugin deactivation
908 init_inode_info - several plugins can need to initialize own info on per-inode
912 calculate size for transaction - if plugin will participate into transaction
913 it should return extra size value.
919 smfs_helper(int code, void * parameter, void * plg_private);
929 #define SMFS_HLP_EXIT 0x01
932 #define SMFS_HLP_INODE 0x02
935 #define SMFS_HLP_TSIZE 0x03
943 SMFS returns zero value in case of successfull registration/deregistration
944 or error code otherwise.
947 Plugins will return zero value in case of successfull execution hook function
948 or error code otherwise.
951 New plugin design example
957 #define SMFS_PLG_NEW 0x03
962 //prepare hook functions
965 int new_plugin_pre_function(int opcode, void * parameter)
971 //let it be just statistic collector
982 int new_plugin_init ()
988 struct smfs_plugin data;
994 data.type = SMFS_PLG_NEW;
997 data.pre_op = &new_plugin_pre_function;
1000 data.post_op = NULL;
1006 err = smfs_register_plugin(&data);
1017 SMFS provides the ability to place a call to other modules.
1018 This can be needed when some notification needed from SMFS for example.
1021 There are quite a few things to keep in mind:
1024 The module that causes the mount of smfs, like the low level OSS and MDS
1025 server will want to register methods for upcalls.
1028 Other modules that are not mounting need to be able to find smfs and register
1032 When these registration of upcall methods have not happened, we should get
1033 reasonable error handling.
1034 Oopses and hangs are not permitted.
1039 Functional Specification
1042 We can use mechanism similar to plugin API but it is more generic.
1043 Upcall is invoked in particular place or when some conditions are occure.
1044 So each upcall has own type.
1045 There can be several upcalls with one type simultaneously.
1046 Therefore we can introduce set of types for various upcalls and make generic
1049 smfs_upcall(int type, void * arg)
1052 Module should register upcall handler in SMFS with help of registering
1054 SMFS has list of upcalls and will call functions for each upcall type walking
1064 There are defined types of upcalls:
1067 #define SMFS_UP_SMTH 0x01
1079 SMFS provide following method for registration/deregistration:
1082 int smfs_register_upcall (struct smfs_upcall *);
1085 int smfs_deregister_upcall (struct smfs_upcall *);
1088 SMFS will return 0 in case of successfull registration or error code otherwise.
1091 SMFS will invoke function for each upcall:
1094 typedef int smfs_upcall_func (int upcall_type, void * arg);
1100 upcall_type - upcall type.
1101 It is define where upcall was invoked and under what conditions;
1104 arg - any desired arguments.
1112 SMFS returns zero value in case of successfull registration/deregistration
1113 or error code otherwise.
1116 Modules should be aware about conditions under what upcall is called - locking,
git://git.whamcloud.com / fs / lustre-release.git / blob