1 #LyX 1.3 created this file. For more info see http://www.lyx.org/
14 %\usepackage{lncsexample}
18 %\usepackage{graphicx}
19 \newcommand{\lst}[3] {
20 \noindent\rule[-0.3mm]{\textwidth}{0.3mm}\vspace{-0.3mm}
21 \lstinputlisting[caption={#2},
23 showstringspaces=false,
31 keywordstyle=\color{red},
32 commentstyle=\color{cyan},
33 stringstyle=\color{green},
34 directivestyle=\color{magenta},
35 emph={1, 2, 3, 4, 5, 6, 7, 8, 9, 0, NULL, lustre, CFS},
36 emphstyle=\color{blue},
46 \paperfontsize default
53 \use_numerical_citations 0
54 \paperorientation portrait
61 \paragraph_separation skip
63 \quotes_language english
67 \paperpagestyle default
73 Storage Management File System
79 Peter Braam, Mike Pershin
83 \begin_inset LatexCommand \tableofcontents{}
93 In this document SMFS (Storage Management File System) is described.
94 It is required to do the following:
97 works as transparent pseudo-filesystem on the top of real filesystem
100 have access to Lustre's API like llog and btree
103 be portable to other file systems than ext3
106 be able to call side-functions from other modules and plugins using own
110 All features can exist together and should not affect bottom filesystem
114 Architectural overview
117 In order to achieve requirements SMFS is expected to have the following
118 major functionality divisions:
121 Backstore\SpecialChar ~
122 FS - main module that is placed in Lustre stack on top of backing
124 It is pseudo filesystem that just pass all operations to real one and can
125 invoke side-functions for each filesystem operation.
126 There is plugin API that can be used by special modules (plugins) to register
127 list of functions for various filesystem operations.
130 Plugins - major reason for SMFS existence is possibility to call side-functions
131 for any filesystem operation.
132 Let's call module with such functionality as SMFS plugin.
133 Each plugin is initialized via options when SMFS is mounted and corresponds
134 to certain method tables.
135 Plugins should register in SMFS using plugins API after that SMFS will
136 call their functions for specified filesystem operations.
140 Upcalls - very similar with plugins but more generic.
141 Modules can use this API to place own hooks in SMFS.
142 It can be needed for notification or taking control from SMFS by upper-level.
151 Functional Specification
152 \layout Subsubsection
154 Backstore FS operations
157 SMFS creates copy of backstore FS objects when needed but setup own super/inode/
158 file operations and save pointer to original one.
159 SMFS should not affect the Lustre IO path.
160 Backstore FS should remains operational in case of SMFS error as long as
162 \layout Subsubsection
164 SMFS plugins/upcalls managment
167 SMFS can registers/deregisters plugin.
168 Plugins are activated after SMFS initialization and can register own functions
169 in SMFS using special API.
173 All plugins are linked in the list and their functions are invoked for several
174 filesystem operations if defined.
178 Deactivated process starts when SMFS uninitialization occur or ioctl is
180 Ioctl interface or procfs (sysfs later) can be used also for plugin management.
183 Upcall are similar to plugins but more generic.
184 There is upcall API in SMFS.
185 Upcalls are also organized in linked list and called one by one where needed.
187 \layout Subsubsection
192 Plugins should be able to register and deregister with SMFS.
193 For these purposes SMFS has two methods
203 int smfs_register_plugin(struct smfs_plugin);
206 int smfs_deregister_plugin(int type);
209 Before calling this function plugin should fill struct smfs_plugin with
211 Registration process will check it and add to the list if possible.
212 Deregistration takes type of plugin, search it in list and delete it.
215 Plugins can insert own function hooks into SMFS.
216 Two major functions that can be setted by plugins are pre- and post-hook.
217 They will be called before filesystem operation and after it.
225 /* this is prototype for plugin hook */
228 int smfs_hook_func (int hook_opcode, void * parameters);
231 Here we pass to plugin hook_opcode and several parameters.
232 For each opcode paramaters can be different.
239 is a number from enumerated list of all possible operations where hooks
241 \layout Subsubsection
246 Plugin API is defined for using with filesystem operations specially.
247 For cases where more generic functionality is needed upcall API can be
257 int smfs_upcall (int type, void * arg);
264 is number from enumerated list of upcalls,
268 is parameter or structure with parameters related to this upcall.
269 Upcalls can be used for notification purposes.
270 Upcalls are also widely used with plugins to get some data from they or
274 Due to upcall origin there can be no some modules while SMFS already is
276 Therefore it is possible that not all functionality presents at the moment.
277 Upcall API should notify caller about this.
285 There are several plugins that use plugin API.
286 They should fit in proposed API well.
287 \layout Subsubsection
292 KML module designed to support writeback caching.
293 KML is quite simple plugin that receive notification about FS operations
294 and write they to the log using llog API.
297 KML use post_op function for logging, that are called after FS operations.
298 It should have also pre_ops for transaction handling.
299 The SMFS functionality is sufficient for it.
301 \layout Subsubsection
306 LRU is designed for cache management in local MDS on client, but can be
307 used also for HSM in future.
308 LRU module defines both pre_op and post_op.
309 Pre_op is quit simple and is used mostly to check do we need purge cache
311 Post_op does cache operations.
312 There is special thread for purging cache, it can be awaken from pre_op
314 SMFS plugin API functionality is sufficient for this module.
315 \layout Subsubsection
320 This module supports snapfs working over SMFS.
321 This module use all functionality of SMFS plugin API - pre_op and post_op.
322 Meanwhile it doesn't need extra functionality and fits in existent API.
328 \layout Subsubsection
330 Backstore FS operations
333 For succefull functioning SMFS should maintain copies of many filesystem
334 objects such as superblock, inodes, files, etc.
335 and replace their operations with own.
336 Therefore each filesystem object in SMFS has its origin object in backstore
337 FS and there is pointer on it.
338 While operating SMFS should maintain own object in consistency with real
342 Example of superblock operation handling:
345 smfs_dirty/write_inode(struct inode * inode)
351 /* getting saved backstore fs inode */
354 backfs_inode = I2CI(inode);
357 /* here can be some pre-work */
363 /* call real operation */
366 backfs_sb->s_op->dirty/write_inode(backfs_inode);
375 duplicate_inode(inode, backfs_inode);
379 \layout Subsubsection
381 Fsfilt operations handlilng
384 Fsfilt operations are handled like ordinary FS operations.
385 SMFS store fsfilt operations that corresponds to backstore FS, fsfilt module
386 just use operations declared for SMFS.
387 SMFS calls real operation with some framework.
388 There are several common steps to pass fsfilt operations to real FS:
391 get backfs sb and inode from SMFS struct inode
394 get backfs fsfilt_operations from superblock
397 call needed operation using backfs objects
400 Transactions in SMFS are handled using fsfilter operation for backstore
402 SMFS make fs_start and fs_commit in all cases where it is needed.
403 Therefore these all extra changes will be included into current transaction:
421 Transaction handler are passed to plugin in hook and can be used by plugin
422 to do proper transaction handling when it is needed.
423 \layout Subsubsection
425 SMFS plugin activation/deactivation
428 Now all plugins are compiled in SMFS module and are initialized via options
429 when SMFS is mounting.
433 Options should be passed as mountfsoptions like this:
439 SMFS parses module options and call initialization method for selected plugins.
440 It is possible to activate/deactivate plugins using ioctl (procfs or sysfs)
445 While initialization plugins should registers with SMFS:
448 smfs_register_plugin(parameters);
451 Parameters are struct smfs_plugin filled with valid data:
454 type of plugin - to distinguish it from anothers,
457 pre_op function - function that will be called before fs operation,
460 post_op function - function that will be called after fs operation,
463 helper function - helper function.
467 any private data - data which will be return to plugin in each call.
468 \layout Subsubsection
473 SMFS place special wrapper in own filesystem operations to call plugins
477 /* wrapper that calls all hooks walking through the list */
480 #define SMFS_HOOK(opcode,...)
490 smfs_hooks_op * hops;
501 list_for_each_entry(plugin_list, hops, smh_list) {
512 rc = hops->smh_hook_op(opcode, ...);
531 Hooks are placed in SMFS methods before and after calling backstore FS operation
535 /* this is how to SMFS uses hooks */
544 struct inode * backfs_inode = I2CI(inode);
547 struct smfs_file_info *sfi;
553 SMFS_HOOK(..., hook_opcode , ..., PRE_HOOK, ...);
556 backfs_inode->i_fop->some_op(sfi->c_file, ...);
559 SMFS_HOOK(..., hook_opcode, ..., POST_HOOK, ...);
563 \layout Subsubsection
568 Upcalls can be placed in any place in SMFS, where it is needed.
571 /* example of upcalls handling */
574 int smfs_upcall(int type, void * arg)
580 list_for_each(upcall_list)
583 entry->upcall(type, arg);
591 /* example of using upcall */
594 int check_uninitialization()
603 smfs_upcall(SMFS_UP_SMTH, ¶meter);
617 Functional Specification
620 Each plugin has own unique name and corresponding type.
621 There is enumerated list of plugins types.
622 Any new plugin should use next sequence number as type and name which not
623 used yet by other plugins.
624 SMFS exports two functions for plugins:
626 smfs_register_plugin()
630 smfs_deregister_plugin().
636 SMFS defines format of function that can be used by plugins to invoke own
638 Plugins should pass pointers on desired function while registration.
641 After successfull registration plugin's function will be called when selected
642 FS operations are occur.
643 SMFS passes to plugin inode structre, dentry, hook opcode and fsfilt transactio
648 Plugin should return zero value in case of successfull completion and error
650 Plugin should decide how to handle error by itself.
651 In case of critical error it can even stops operating or deregisters itself
653 If plugin have no function for selected
657 it has to return also
667 Plugin identification
670 There are defined types of plugins:
673 #define SMFS_PLG_KML 0x01
676 #define SMFS_PLG_LRU 0x02
687 Main structure for communication between plugin and SMFS.
694 struct list_head plist;
700 smfs_hook_func * pre_op;
703 smfs_hook_func * post_op;
706 smfs_helper * helper;
729 - function to handle pre-operations,
736 - function to handle post-hook operation,
739 helper - call to various helper functions needed by SMFS,
742 plugin_data - private plugin data.
748 SMFS provide following method for registration:
751 int smfs_register_plugin (struct smfs_plugin *);
754 SMFS will return 0 in case of successfull registration or error code otherwise.
761 Plugin's hook function must have the same type:
764 typedef int smfs_hook_func (int opcode, void * parameter);
767 Parameter can contains:
776 new_inode - used in rename
779 new_dentry - used in rename
785 handle - fsfilt transaction if exists
788 There are opcodes for now:
791 #define HOOK_CREATE 1
794 #define HOOK_LOOKUP 2
800 #define HOOK_UNLINK 4
803 #define HOOK_SYMLINK 5
815 #define HOOK_RENAME 9
818 #define HOOK_SETATTR 10
821 #define HOOK_WRITE 11
824 #define HOOK_READDIR 12
830 Plugin may use opcode as index for array of some type.
831 It can be table of functions for each hook type for example:
834 static cache_hook_op cache_space_hook_ops[HOOK_MAX + 1] = {
837 [HOOK_CREATE] cache_space_hook_create,
840 [HOOK_LOOKUP] cache_space_hook_lookup,
843 [HOOK_LINK] cache_space_hook_link,
846 [HOOK_UNLINK] cache_space_hook_unlink,
849 [HOOK_SYMLINK] cache_space_hook_create,
852 [HOOK_MKDIR] cache_space_hook_mkdir,
855 [HOOK_RMDIR] cache_space_hook_rmdir,
858 [HOOK_MKNOD] cache_space_hook_create,
861 [HOOK_RENAME] cache_space_hook_rename,
874 \layout Subsubsection*
879 Plugins can register helper function.
880 There are several places where helpers are needed:
883 smfs uninitialization - after deleting plugin from list it can be usefull
884 to invoke plugin deactivation
887 init_inode_info - several plugins can need to initialize own info on per-inode
891 calculate size for transaction - if plugin will participate into transaction
892 it should return extra size value.
898 smfs_helper(int code, void * parameter);
908 #define SMFS_HLP_EXIT 0x01
911 #define SMFS_HLP_INODE 0x02
914 #define SMFS_HLP_TSIZE 0x03
922 SMFS returns zero value in case of successfull registration/deregistration
923 or error code otherwise.
926 Plugins will return zero value in case of successfull execution hook function
927 or error code otherwise.
930 New plugin design example
936 #define SMFS_PLG_NEW 0x03
941 //prepare hook functions
944 int new_plugin_pre_function(int opcode, void * parameter)
950 //let it be just statistic collector
961 int new_plugin_init ()
967 struct smfs_plugin data;
973 data.type = SMFS_PLG_NEW;
976 data.pre_op = &new_plugin_pre_function;
985 err = smfs_register_plugin(&data);
996 SMFS provides the ability to place a call to other modules.
997 This can be needed when some notification needed from SMFS for example.
1000 There are quite a few things to keep in mind:
1003 The module that causes the mount of smfs, like the low level OSS and MDS
1004 server will want to register methods for upcalls.
1007 Other modules that are not mounting need to be able to find smfs and register
1011 When these registration of upcall methods have not happened, we should get
1012 reasonable error handling.
1013 Oopses and hangs are not permitted.
1018 Functional Specification
1021 We can use mechanism similar to plugin API but it is more generic.
1022 Upcall is invoked in particular place or when some conditions are occure.
1023 So each upcall has own type.
1024 There can be several upcalls with one type simultaneously.
1025 Therefore we can introduce set of types for various upcalls and make generic
1028 smfs_upcall(int type, void * arg)
1031 Module should register upcall handler in SMFS with help of registering
1033 SMFS has list of upcalls and will call functions for each upcall type walking
1043 There are defined types of upcalls:
1046 #define SMFS_UP_SMTH 0x01
1058 SMFS provide following method for registration/deregistration:
1061 int smfs_register_upcall (struct smfs_upcall *);
1064 int smfs_deregister_upcall (struct smfs_upcall *);
1067 SMFS will return 0 in case of successfull registration or error code otherwise.
1070 SMFS will invoke function for each upcall:
1073 typedef int smfs_upcall_func (int upcall_type, void * arg);
1079 upcall_type - upcall type.
1080 It is define where upcall was invoked and under what conditions;
1083 arg - any desired arguments.
1091 SMFS returns zero value in case of successfull registration/deregistration
1092 or error code otherwise.
1095 Modules should be aware about conditions under what upcall is called - locking,