--- /dev/null
+.TH LIPE_FIND3 "1" "March 2022" "lipe_find3 (lipe) 2.22_121_g2a0cb43-1" "User Commands"
+.SH NAME
+lipe_find3 \- manual page for lipe_find3 (lipe) 2.22_121_g2a0cb43-1
+.SH SYNOPSIS
+.B lipe_find3
+[\fIOPTION\fR]... \fIDEVICE \fR[\fIEXPRESSION\fR]
+.SH DESCRIPTION
+Find files on a Lustre target device
+.PP
+.B lipe_find3
+searches the inode tables of a Lustre target block device by
+evaluating the given expression from left to right, according to the
+rules of precedence (see section OPERATORS), and applies any actions
+specified in expression.
+
+If \fIEXPRESSION\fR contains no actions then the -print-relative-path
+is performed on all files for which \fIEXPRESSION\fR is true.
+
+.SH OPTIONS
+Mandatory arguments to long options are mandatory for short options too.
+.TP
+\fB\-\-debug\fR
+enable debugging
+.TP
+\fB\-\-debug\-scan\fR
+enable scan debugging
+.TP
+\fB\-\-list\-json\-attrs\fR
+list available JSON attribute names and exit
+.TP
+\fB\-\-thread\-count\fR=\fICOUNT\fR
+use COUNT scaninng threads (default 4)
+.TP
+\fB\-h\fR, \fB\-\-help\fR
+display this help text and exit
+.TP
+\fB\-v\fR, \fB\-\-version\fR
+output version information and exit
+.PP
+
+.SH EXPRESSIONS
+The \fIEXPRESSION\fR is taken to be all command line arguments
+following the DEVICE and consists of tests, actions, and operators.
+
+.SH TESTS
+Numeric arguments may be specified as \fI+n\fP for greater than \fIn\fP,
+or \fI\-n\fP for less than \fIn\fP.
+
+.IP \-true
+Return true.
+
+.IP \-false
+Return false.
+
+.IP "\-inum \fIn\fR"
+File has inode number
+.IR n .
+Note this is inode number in the backing filesystem and is not equal
+to the FID-derived inode number seen by clients.
+
+.IP "\-type \fIc\fR"
+File has type
+.RS
+.IP b
+block (buffered) special
+.IP c
+character (unbuffered) special
+.IP d
+directory
+.IP p
+named pipe (FIFO)
+.IP f
+regular file
+.IP l
+symbolic link
+.IP s
+socket
+.RE
+.IP
+To search for more than one type at once, you can supply the combined
+list of type letters separated by a comma ','.
+
+.IP "\-perm \fImode\fR"
+File permission bits are exactly \fImode\fP (octal or symbolic).
+
+.IP "\-perm \-\fImode\fR"
+File has all of the permission bits \fImode\fP (octal or symbolic) set.
+
+.IP "\-perm /\fImode\fR"
+File has any of the permission bits \fImode\fP (octal or symbolic) set.
+
+.IP "\-uid \fIn\fR"
+File has numeric user ID
+.IR n .
+
+.IP "\-user \fIuname\fR"
+File is owned by user \fIuname\fP (numeric user ID allowed).
+
+.IP "\-gid \fIn\fR"
+File has numeric group ID
+.IR n .
+
+.IP "\-group \fIgname\fR"
+File belongs to group \fIgname\fP (numeric group ID allowed).
+
+.IP "\-projid \fIid\fR"
+File has numeric project ID
+.IR n .
+See
+.BR lfs-project (1) .
+
+.IP "\-atime \fIn[smhd]\fR"
+File was last accessed
+.IR n
+seconds, minutes, hours, or days ago according to suffix. If no suffix
+is given then days is implied.
+
+.IP "\-mtime \fIn[smhd]\fR"
+File was last modified
+.IR n
+seconds, minutes, hours, or days ago according to suffix. If no suffix
+is given then days is implied.
+
+.IP "\-ctime \fIn[smhd]\fR"
+File was last changed
+.IR n
+seconds, minutes, hours, or days ago according to suffix. If no suffix
+is given then days is implied.
+
+.IP "\-crtime \fIn[smhd]\fR"
+File was created
+.IR n
+seconds, minutes, hours, or days ago according to suffix. If no suffix
+is given then days is implied.
+
+.IP "\-links \fIn\fR"
+File has \fIn\fP hard links.
+
+.IP "\-size \fIn\fR[cwbkMG]"
+File size is \fIn\fP units after rounding up to the next multiple of
+unit. The following suffixes can be used:
+.RS
+.IP `b'
+for 512-byte blocks (this is the default if no suffix is used)
+.IP `c'
+for bytes
+.IP `w'
+for two-byte words
+.IP `k'
+for kibibytes (KiB, units of 1024 bytes)
+.IP `M'
+for mebibytes (MiB, units of 1024 * 1024 = 1\|048\|576 bytes)
+.IP `G'
+for gibibytes (GiB, units of 1024 * 1024 * 1024 = 1\|073\|741\|824 bytes)
+.RE
+.IP
+For a regular file on an MDT the size is generally read from the Size
+on MDT (SoM) extended attribute. We consider the size from SoM valid
+as long as the SoM state is \fBstrict\fP or \fBlazy\fP. For other file
+types or when scanning an OST we use the size from the inode. This
+may give incorrect results for striped directories.
+
+.IP "\-blocks \fIn\fR[cwbkMG]"
+File consumes \fIn\fP units of space after rounding up to the next
+multiple of unit. The suffixes accepted are the same as for \fB\-size\fP.
+As with \fB\-size\fP, for a regular file on an MDT, we use the block
+count from the SoM xattr as long as the SoM state is \fBstrict\fR or
+\fBlazy\fR. Otherwise we use the block count from the inode. Note that
+the block count from SoM may not be strictly correct, and that even a
+correct block count may not equal what would be expected from the file
+size due to factors such as larger OST block sizes, preallocation, and
+sparse files
+
+\" ---- name and path ------------------------------------------
+
+.IP "\-name \fIpattern\fR"
+The name of file (the path with the leading directories removed)
+matches shell pattern \fIpattern\fP. For a file with multiple hard
+links, the \fB\-name\fP test returns true if \fBany\fP link matches
+\fIpattern\fP. Shell patterns are described in \fBglob\fP(7). Note
+that \fIpattern\fR should be enclosed in single quotes to avoid
+possible expansion by the shell.
+
+.IP "\-iname \fIpattern\fR"
+Like
+.BR \-name ,
+but the match is case insensitive.
+
+.IP "\-path \fIpattern\fR"
+File path relative to root of FS matches shell pattern
+\fIpattern\fP. See \fBglob\fP(7).
+
+.IP "\-ipath \fIpattern\fR"
+Like \fR\-path\fP but the match is case insensitive.
+
+\" ---- layout -------------------------------------------------
+
+.IP "\-pool \fIname\fR"
+File has a layout component specifying pool \fIname\fR.
+
+.IP "\-mirror-count \fIn\fR"
+File has \fIn\fR mirrors.
+
+.IP "\-stripe-count \fIn\fR"
+The last instantiated component of file has \fIn\fR stripes.
+
+\" ---- xattr --------------------------------------------------
+
+.IP "\-xattr \fIname\fR"
+File has an xattr with name \fBequal\fR to \fIname\fR.
+
+.IP "\-xattr-match \fIname\fR \fIvalue\fR"
+File has an xattr whose name and value matches the shell patterns \fIname\fR and \fIvalue\fR.
+
+.SH ACTIONS
+.IP "\-exec \fIcommand\fR ;"
+Execute \fIcommand\fR; true if 0 status is returned. All following
+arguments to find are taken to be arguments to the command until an
+argument consisting of ';' is encountered. The string '{}' is replaced
+by the absolute path of the first hardlink of the current file being
+processed everywhere it occurs in the arguments to the
+command. Regardless of the command, this requires a locally mounted
+client for the FS to which the device belongs. See \fBfind\fP(1) for more
+information.
+
+.IP "\-exec \fIcommand\fR +"
+Like \fB\-exec\fP but the command line is built by appending each
+selected file name at the end. See \fBfind\fP(1) for more information.
+
+.IP "\-print\-file\-fid"
+Print the current file FID when scanning an MDT, and the FID of the
+file to which the current object belongs when scanning an OST.
+
+.IP "\-print\-self\-fid"
+Print the current file FID when scanning an MDT, and the filter FID
+when scanning an OST. You probably want to use \fB\-print\-file\-fid\fP
+instead.
+
+.IP "\-print\-absolute\-path"
+Resolve and print the absolute path of the current file (or the file
+owning the current OST object) followed by a newline. If the file has
+multiple hard links then only the path of the first link will be
+printed. Path resolution requires a locally mounted client for the FS
+to which the device belongs. The local client mount will be
+automatically detected.
+
+.IP "\-print\-relative\-path"
+Like \fB\-print\-absolute\-path\fP but the path printed will be
+relative to the root of the filesystem. Also requires a locally
+mounted client.
+
+.IP "\-print"
+Alias for "\-print\-relative\-path".
+
+.IP "\-print0"
+Like \fB\-print\fP but use a null character instead of a newline.
+
+.IP "\-fprint \fIfile\fR"
+Like \fB\-print\fP but print to \fIfile\fR instead of stdout. If
+\fIfile\fP does not exist when \fBlipe_find3\fR is run then it is
+created; if it does exist, it is truncated. The file
+names '/dev/stdout' and '/dev/stderr' are handled specially; they
+refer to the standard output and standard error output,
+respectively. The output file is always created, even if the predicate
+is never matched.
+
+.IP "\-fprint0 \fIfile\fR"
+Like \fB\-print0\fP but print to \fIfile\fR instead of standard
+output.
+
+.IP "\-printf \fIformat\fR"
+Print \fIformat\fP to the standard output, interpreting '\e' escapes
+and '%' directives. The supported directives include most supported by
+\fBfind\fP(1) and the following extensions:
+.RS
+.IP %{fid}
+file FID
+.IP %{projid}
+numerical project ID
+.IP %{mirror-count}
+FLR mirror count
+.IP %{stripe-count}
+stripe count of last instantiated component
+.IP %{xattr:NAME}
+contents of NAME xattr as a string
+.RE
+
+.IP "\-fprintf \fIfile\fR \fIformat\fR"
+Like \fB\-printf\fP but print to \fIfile\fR instead of standard
+output.
+
+.IP "\-print-json \fIattrs\fR"
+Print a single line JSON description of the current file. Attributes
+to be printed should be specified as a comma separated
+list. Attributes enclosed in brackets ('[]') are
+optional. Non-optional attributes are required. Use "lipe_find3
+\--list\-json\-attrs" to see all supported JSON attributes. The
+\fBjq\fP utility may be used to format or transform the output.
+
+.IP "\-delete"
+Delete the current file. This is deletion by FID and requires a local
+client mount. This removes \fBall\fP hard links to a file. Note that a
+non empty directory cannot be removed by FID and that even when all
+the files in a directory would be removed in a scan they may not be
+removed by the time the directory is encountered.
+
+.IP "\-quit"
+Exit immediately. No child processes will be left running, but no more
+paths specified on the command line will be processed.
+
+.SH OPERATORS
+.P
+Listed in order of decreasing precedence:
+
+.IP "( \fIexpr\fR )"
+Force precedence. Note that the '(' and ')' characters must be quoted
+or slash escaped to prevent interpretation by the shell.
+
+.IP "! \fIexpr\fR"
+Logical not. Note that the '!' character must be quoted or slash
+escaped to prevent interpretation by the shell.
+
+.IP "\-not \fIexpr\fR"
+True if \fIexpr\fR is false.
+
+.IP "\fIexpr1 expr2\fR"
+Logical and. Two expressions in a row are taken to be joined with an
+implied
+.BR \-and ;
+
+.IP "\fIexpr1\fR \-a \fIexpr2\fR"
+Same as
+.IR "expr1 expr2"
+
+.IP "\fIexpr1\fR \-and \fIexpr2\fR"
+Same as
+.IR "expr1 expr2"
+
+.IP "\fIexpr1\fR \-o \fIexpr2\fR"
+Logical or.
+
+.IP "\fIexpr1\fR \-or \fIexpr2\fR"
+Same as \fIexpr1\fR
+.B \-o
+.IR expr2
+
+.IP "\fIexpr1\fR , \fIexpr2\fR"
+Evaluate \fIexpr1\fR and \fIexpr2\fR, discarding the value of
+ \fIexpr1\fR and returning value of
+.IR expr2 .
+
+.SH EXAMPLES
+
+.nf
+.B lipe_find3 /dev/mapper/vg_mdt0000_fs7-mdt0000 \-type f -user bob \-name '*.txt'
+
+.fi
+
+Print all regular files on MDT0000 owned by \fBbob\fP whose name
+matches '*.txt'. Note that moving the \fB\-name\fP test earlier in the
+expression will likely make it slower.
+
+.nf
+.B lipe_find3 /dev/mapper/vg_mdt0042_fs7-mdt0042 -type f \! \-name '*.gz' -exec gzip '{}' \\\;
+
+.fi
+LZ77 compress every regular file whose name does not match '*.gz'.
+
+.nf
+.B lipe_find3 /dev/mapper/vg_mdt0042_fs7-mdt0042 -print-json 'uid,size,blocks,[som]'
+
+.fi
+Print a JSON object describing each file on MDT0042 including the uid,
+size, block count and if avaliable the SoM attributes. For example:
+.nf
+# lipe_find3 /dev/mapper/mds1_flakey -print-json 'uid,size,blocks,[som]' | jq .
+{
+ "uid": 1001,
+ "size": 4194304,
+ "blocks": 0,
+ "som": {
+ "flags": [
+ "lazy"
+ ],
+ "_flags": 4,
+ "size": 4194304,
+ "blocks": 0
+ }
+}
+...
+
+.fi
+
+
+.nf
+.B lipe_find3 /dev/mapper/vg_mdt0042_fs0a12-mdt0042 -print-json '[all]'
+
+.fi
+Print a JSON object describing each file on MDT0042 including all
+available attributes.
+
+.SH KNOWN ISSUES
+The \fB\-size\fP, \fB\-blocks\fP, and timestamp tests give incorrect
+results for striped directories.
+
+.SH SEE ALSO
+\fBfind\fP(1), \fBlfs-find\fP(1)
--- /dev/null
+lipe_scan3 documentation
+
+XXX: lipe_scan3 is an internal utility which is not documented or
+supported for use by users. Instead it provides the scaninng core to
+the (in-progress) user facing lipe_find3 utility and to the
+(unstarted) Unified Policy Engine.
+
+lipe_scan3 is a Lustre device scanner with an embedded guile compiler,
+runtime, and interpreter (see https://www.gnu.org/software/guile/). It
+has everything guile that has plus scanning related procedures. It can
+be run in the following ways:
+
+ lipe_scan3 -i # start an interactive REPL
+ lipe_scan3 -s SCRIPT # compile and execute the scheme language SCRIPT
+ lipe_scan3 -s /dev/stdin # compile and execure scheme from stdin
+
+It also offers some predefined operations (policies) which accept one
+or more devices:
+
+ lipe_scan3 --print-file-fid DEVICE...
+ lipe_scan3 --print-absolute-path DEVICE...
+ lipe_scan3 --print-relative-path DEVICE...
+ lipe_scan3 --print-json[=ATTRS] DEVICE...
+
+In each case the device may be an MDT or an OST. The --print-file-fid
+prints the inode FID when scanning an MDT and the FID of the file
+containing the object when scanning an OST. Similarly for
+--print-*-path. In this way, we can easily find the FIDs or paths of
+all files that have objects on an given OST. When multiple devices are
+given they will be scanned sequentially.
+
+Scanning is done by the lipe-scan procedure:
+
+(lipe-scan device-path ;; string
+ client-mount-path ;; string
+ policy ;; thunk
+ required-attrs ;; unsigned int
+ thread-count)
+
+Note: A thunk is a zero argument procedure (including a closure).
+
+This will open device-path for inode scanning. If client-mount-path is
+supplied then it will be opened to for use in the policy function (for
+fid2path etc). Policy will be called for each inode on the device
+which has the required-attrs. We use thread-count concurrent scanning
+threads. So note that policy will be called concurrently. All of the
+predefined --print-* policies are implemented by calling lipe-scan
+with a corresponding scheme procedure as the policy.
+
+The usual way to call lipe-scan is to use the lipe-getopt-*
+procedures for the client-mount-path, reauired-attrs, and
+thread-count:
+
+(lipe-scan device-path ;; string
+ (lipe-getopt-client-mount-path)
+ policy ;; thunk
+ (lipe-getopt-required-attrs) ;; unsigned int
+ (lipe-getopt-thread-count))
+
+If the --client-mount-path, --required-attrs, or --thread-count
+options to lipe_scan3 were used then the corresponding lipe-getopt-*
+function returns the argument passed to the option. Otherwise they
+return sentinel values which tell lipe-scan to
+
+ Automatically determine the client mount based on the fsname of the
+ device.
+
+ Choose appropriate required-attrs based on whether the device is an
+ MDT or and OST.
+
+ Set the thread-count to half of the online CPU count.
+
+Scanning and policy
+
+lipe-scan spawns multiple threads to scan the inode of the
+device. When a scanning thread finds an inode with all of the required
+attributes, it stores the scanning context (a pointer to the current
+inode, its attributes, and some information about the device) into
+thread local storage. Then the thread calls the policy
+thunk. Generally the policy thunk examines the attributes (required
+and optional) and performs some action like printing the JSON
+representation of the inode or deleting the file.
+
+Policy attribute procedures
+
+The following procedures are defined to return attributes of the
+current inode during a call to policy. They are globally defined and
+may be referenced from anywhere (stored in global variables, or lists,
+or closed over). But will throw a *lipe-no-context* exception when
+called outside of a policy context. They depend on the thread local
+context described in the previous paragraph to access the current
+inode. The use of thunks to access an anonymous current inode is
+analagous to the implicit "this" pointer in C++ member functions.
+
+ F: FID (represented a scheme struct, encoded as string in JSON)
+ I: integer
+ S: string
+ C: consistent with value returned by stat.
+ L: Always <= value returned by stat.
+
+ (ino) inode number (I)
+ (uid) inode UID (I,C)
+ (gid) inode GID (I,C)
+ (projid) inode project ID (I)
+ (atime) inode atime in epoch seconds (I,L)
+ (mtime) inode mtime in epoch seconds (I,L)
+ (ctime) inode mtime in epoch seconds (I,L)
+ (flags) inode flags (I)
+ (nlink) inode link count (I)
+ (mode) inode type and access mode flags (I,C)
+ (size) ... (I)
+ (blocks) ... (I)
+ (self-fid) self_fid from lma xattr of inode (F)
+ (file-fid) self_fid for MDT inode, FID of corresponding file for OST object (ff_parent)
+ (links) list of pairs of parent-fid, name (list (cons F, S) ...)
+ (absolute-paths) list of all paths to file using client mount (list S ...)
+ (relative-paths) list of all paths to file relative to client mount (list S ...)
+ (xattr-ref name) xattr value as bytevector
+ (xattrs) list of name, xattr value pairs
+ (pools) list of pool names (list S ...)
+
+As much as possible, attribute values are evaluated on demand and
+cached in the context once gotten. Note that all ext4 attribute
+splitting (i_uid + i_uid_hi, ...) is handled automatically.
+
+Policy actions
+ These may only be called from a scanning context:
+
+ (print-file-fid) print (file-fid) to stdout, with brackets, no quotes.
+ (print-self-fid) print (self-fid) to stdout, ...
+ (print-json [attrs]) print JSON representation of inode to stdout
+ (print-absolute-path) ...
+ (print-realtive-path) ...
+
+Break and continue
+ These may only be called from a scanning context:
+ (lipe-scan-break STATUS) gracefully stop scanning the current device and
+ return STATUS from (lipe-scan ...)
+ (lipe-scan-continue) return from current call to policy, continue scanning
+
+Scan attribute procedures
+ These may only be called from a scanning context:
+
+ (lipe-scan-fsname) return fsname of current device (S)
+ (lipe-scan-client-mount-path) return canonicalized client mount path or "" (S)
+ (lipe-scan-client-mount-fd) return open FD on client mount or -1 (I)
+ (lipe-scan-device-name) return device name in "myfs-MDT0042" format (S)
+ (lipe-scan-device-path) return device path passed to lipe-scan (S)
+ (lipe-scan-thread-count) return number of scanning threads
+ (lipe-scan-thread-index) return scanning thread index [0, thread-count)
+
+Other procedures
+ The following are pure:
+ (make-fid seq oid ver) fid constructor
+ (fid-seq fid) return seq
+ (fid-oid fid) return oid
+ (fid-ver fid) return ver
+ (fnmatch? pattern filename [flags]) fnmatch(3) wrapper
+
+ The following can be called from any context:
+ (lipe-debug-enable [enable]) enable or disable debugging. Returns previous value.
+ (lipe-gettid) return calling thread ID (Linux tid for gdb -p, not pthread ID)
+ (lipe-getopt-client-mount-path) see above
+ (lipe-getopt-required-attrs) see above
+ (lipe-getopt-required-thread-count) see above
+
+ Every procedure included with guile.
+
+Example policies:
+ Print the FID if the UID is not equal to the GID:
+
+ (lambda ()
+ (if (not (= (uid)
+ (gid)))
+ (print-file-fid)))
+
+ Print a line containing the device name and ino:
+
+ (lambda ()
+ (format #t "~a ~a\n" (lipe-scan-device-name) (ino))
+
+ Print FIDs of files owned by ProjID 500 which are world readable:
+
+ (lambda ()
+ (if (and (= (projid) 500)
+ (positive? (logand (mode))))
+ (print-file-fid)))
+
+ See also the example scripts in lipe/src/lipe_scan3/scripts/.
+
+Warnings:
+ guile (format ...) is not multi-thread atomic in the way that stdio functions are.
+
+Notes:
+ lipe_scan3 actually uses 1 + thread-count threads. 1 for the main
+ thread and thread-count scanning threads. The main thread just waits
+ during scanning.
+
+TODOs:
+ Distinguish between missing xattr and xattr read error in loa.
+
+ Decide how lazy (size) is.
+
+ Add unambiguous size procedures (lazy-size), (real-size)
+
+ LU-15536 somsync.
+
+ Readlink support.
+
+ Properly cache llapi_layout in loa.
+
+ Add more lov based attributes.
+
+ Add lma JSON attr.
+
+ Add lov JSON attr.
+
+ Add lmv based attributes.
+
+ Add lmv JSON attr.
+
+ Use lma_incompat to ensure that we avoid agent/orphan/... inodes.
+
+ Tests for corrupted attrs.
+
+ Native xattr-ref-string.
+
+ SCM_DEFINE (scm_utf8_to_string, "utf8->string",
+
+ (use-modules (ice-9 iconv)) ; bytevector->string
+ (define (xattr-ref-string name)
+ (utf8->string (or (xattr-ref name) #vu8())))
+
+ Native xattr-ref-base64.
+
+ Makefile.am s/-coverage/-O/
+
+ Run tests on MDT0002.
+
+ Run tests on OST.
+
+ Support empty and entries attributes for striped directories.
+
+ Fixup wandering ext2fs error codes more consistently.
+
+ Enusre that we return LS3_EXIT_* values from (lipe-scan ...)
+
+ Test HA start/stop/failover during long scan.
+
+ Investigate prw-unwind throw handler.
+ (with-throw-handler #t (lambda () (links)) (lambda args (backtrace)))
+
+ Ensure that top level guile execeptions (script does not exist) have
+ the correct format.
+
+ Clarify ext2fs extra times mess. See comment in ldiskfs_decode_extra_time().
git://git.whamcloud.com / fs / lustre-release.git / commitdiff