LU-6142 llite: Fix style issues for lproc_llite.c This patch fixes issues reported by checkpatch for file lustre/llite/lproc_llite.c Test-Parameters: trivial Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Change-Id: Id4c96fa903323b73b4e1416835d8a8bb25043781 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/54006 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Timothy Day <timday@amazon.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-13802 llite: add hybrid IO SBI flag Add an SBI flag so hybrid IO can be fully disabled. Test-Parameters: trivial Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com> Signed-off-by: Qian Yingjin <qian@ddn.com> Change-Id: I2825b4cf261f98d71a18cd66d6fe3632dfabc37a Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52592 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
LU-16771 llite: add statfs_project tunable Add a client tunable and mount option to turn off project-enabled statfs() if needed, for example to speed up statfs() execution by avoiding project quota check. This new llite tunable statfs_project is set to 1 by default (feature enabled). To turn statfs_project off: lctl set_param llite.*.statfs_project=0 Additionally, statfs_project can be disabled at mount time with: mount -t lustre -o nostatfs_project ... Signed-off-by: Stephane Thiell <sthiell@stanford.edu> Change-Id: I1c3eb27e66b1d05a1c713732dfe0a4d8f7af769f Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52872 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com> Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-14361 statahead: regularized fname statahead pattern Some applications do stat() calls under a directory within which all children files have regularized file name: - mdtest benchmark tool: mdtest.$rank.$i - ML/AI with ingested data that have typically a format rule of the filename in the directory. The most common format for regularized file name is that the suffix part of the file name is number-indexing. However, in the current statahead mechanism, the statahead is populated by the order of the hash of the file name via readdir() calls, not a kind of sorting order. In this patch, we improve the statahead to prefetch attributes for the files with regularized indexing file name via asynchronous batching RPC. This patch adds the support to do statahead for these kinds of applications, which can be optimized, but without opendir()/ close() to start/stop statahead thread explicitly. Instead, the statahead thread will stop and quit when found that there was no acitivy for more than a certain time period (i.e. 30 seconds). Test-Parameters: mdtcount=4 mdscount=2 testlist=sanity env=ONLY=27p,ONLY_REPEAT=5 Test-Parameters: mdtcount=4 mdscount=2 testlist=sanity env=ONLY=27p,ONLY_REPEAT=5 Test-Parameters: mdtcount=4 mdscount=2 testlist=sanity env=ONLY=123f,ONLY_REPEAT=10 Test-Parameters: mdtcount=4 mdscount=2 testlist=sanity env=ONLY=123f,ONLY_REPEAT=10 Signed-off-by: Qian Yingjin <qian@ddn.com> Change-Id: Ide11ec5a651ae74884ddbe1cecede4f5c961e38d Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/41308 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-14361 statahead: add tunable for fname pattern statahead This patch adds a tunable parameter for fname pattern statahead. Currenty fname pattern statahead is disabled. It will be enabled by default once the patch series of fname pattern statahead work is finished and stable. Signed-off-by: Qian Yingjin <qian@ddn.com> Change-Id: I7a207e66df10b45bff1b3993e8724116489365b7 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51572 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
LU-13805 llite: add flag to disable unaligned DIO As with any new IO feature, it's a good idea to have the option to turn off unaligned DIO support if needed. It would be reasonable to merge this patch with the core patch implementing the feature; I have kept it separate for ease of review. Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com> Change-Id: Ibc86d84704151a7f30afcc538d9c03e3fdf1c38a Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51125 Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Sebastien Buisson <sbuisson@ddn.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com>
LU-17030 llite: allow setting max_cached_mb to a % Lustre's max_cached_mb parameter is hard to use because it must be set to a specific numeric value, so in effect it cannot be set on the server side unless all clients are guaranteed identical. Let's add the ability to set that to a % of memory to make it more useful. Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com> Change-Id: I1f9f5a8a5d671ab00b7ab6133bb9b1d1214ca59e Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51952 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Sebastien Buisson <sbuisson@ddn.com> Reviewed-by: Qian Yingjin <qian@ddn.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-14301 client: use EOPNOTSUPP instead of ENOTSUPP Don't return NFS-specific error code ENOTSUPP back to userspace, instead use EOPNOTSUPP. ENOTSUPP does not print a useful error message from strerror() if it is hit by an application. Signed-off-by: Andreas Dilger <adilger@whamcloud.com> Change-Id: Iabd07b31069737e8ee7ca2382fd8cff6143ebbe5 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51511 Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Neil Brown <neilb@suse.de> Reviewed-by: jsimmons <jsimmons@infradead.org> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com>
LU-14139 statahead: batched statahead processing Batched metadata processing can get a big performance boost. In this patch, it implements a batched statahead mechanism which can also increase the performance for a directory traverse or listing such as the command 'ls'. For the batched statahead, one batch getattr() RPC equals to 'N' normal lookup/getattr RPCs. It can pack a number of dentry name getting from the readdir() call and prepared lock handles one client side lock namespace into one large batched RPC transfering via bulk I/O to obtain ibits DLM locks and associated attributes for a lot of files in one blow. When MDS receives a batched getattr() RPC, it executes the sub requests in it one by one serially. A tunable parameter named "statahead_batch_max" is defined, it means the maximal items can be batched and processed within one aggregate RPC. Once the number of sub requests exceeds this predefined limit, it will pack and trigger the batched RPC. The batched RPC will also be triggered explictly when the readdir() call comes to the end position of the directory or the statahead thread exits abnormally. Batched metadata processing can get a big performance boost. The mdtest performance results without/with this patch series are as follow: mdtest-easy-stat 720.562369 kIOPS : time 118.695 seconds mdtest-easy-stat 1218.290192 kIOPS : time 70.656 seconds In this patch, we set statahead_batch_max=0 and disabled batched statahead by default. It will enable accordingly once some subsequent fixes about batched RPC have been merged. Signed-off-by: Qian Yingjin <qian@ddn.com> Change-Id: I5a80c2c377093dc8b8e21341f440e3038f017ca8 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/40720 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
LU-16621 enc: file names encryption when using secure boot Secure boot activates lockdown mode in the Linux kernel. And debugfs is restricted when the kernel is locked down. This patch moves file names encryption from debugfs to sysfs. Test-Parameters: trivial testlist=sanity-sec Signed-off-by: Alex Deiter <alex.deiter@gmail.com> Change-Id: I434714941ffac2a4694cabd33f613aef70933678 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50219 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Sebastien Buisson <sbuisson@ddn.com> Reviewed-by: jsimmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-15420 sec: handle simple fscrypt changes for 5.15 kernels The patch covers the low impact changes to the Linux kernels fscrypt API. The changes are: For Linux kernel version 5.9-rc4 the following commits are: 8b10fe68985278de4926daa56ad6af701839e40a removed the inode parameter for the fscrypt function fscrypt_fname_alloc_buffer() 5b2a828b98ec1872799b1b4d82113c76a12d594f ended up exporting fscrypt_d_revalidate() and stopped stomping on the d_ops. c8c868abc91ff23f6f5c4444c419de7c277d77e1 changed fscrypt_set_test_dummy_encryption() take a 'const char * ac4acb1f4b2b6b7e8d913537cccec8789903e164 moved the fscrypt core from using fscrypt_context to using fscrypt_policy that is user forward facing. Lastly for Linux kernel version 5.10-rc4 the commit ec0caa974cd092549ab282deb8ec7ea73b36eba0 stopped exporting fscrypt_get_encryption_info(). Use fscrypt_prepare_readdir() in its place. 70fb2612aab62d47e03f82eaa7384a8d30ca175d renamed a field in struct fscrypt_name. Since Lustre can't use fscrypt_prepare_lookup we have to deal with this change. Remove sptlrpc_enc_pool_del_user() since its an empty function that waste cycles calling it. The other large change was the replacement of fscrypt_inherit_context which is described in Linux commit a992b20cd4ee360dbbe6f69339cb07146e4304d6. This change is very large since it expects the target inode to be available. Lustre uses fscrypt_inherit_context before the inode is available so this is a much more complex change that will be done in another patch. de3cdc6e75179a2324c23400b21483a1372c95e1 makes fscrypt_require_key private. You need to test the key's presence with fscrypt_has_encryption_key() instead but that key needs to be setup first by the new function fscrypt_prepare_new_inode(). Since this is the case we wait to introduce this change. Change-Id: I4bed7fef6e3302c0258c0f1563f4e180258d7a5a Signed-off-by: James Simmons <jsimmons@infradead.org> Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/49125 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Sebastien Buisson <sbuisson@ddn.com> Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16374 enc: align Base64 encoding with RFC 4648 base64url Lustre encryption uses a Base64 encoding to encode no-key filenames (the filenames that are presented to userspace when a directory is listed without its encryption key). Make this Base64 encoding compliant with RFC 4648 base64url. And use '+' leading character to distringuish digested names. This is adapted from kernel commit ba47b515f594 fscrypt: align Base64 encoding with RFC 4648 base64url To maintain compatibility with older clients, a new llite parameter named 'filename_enc_use_old_base64' is introduced, set to 0 by default. When 0, Lustre uses new-fashion base64 encoding. When set to 1, Lustre uses old-style base64 encoding. To set this parameter globally for all clients, do on the MGS: mgs# lctl set_param -P llite.*.filename_enc_use_old_base64={0,1} Signed-off-by: Eric Biggers <ebiggers@google.com> Signed-off-by: Sebastien Buisson <sbuisson@ddn.com> Change-Id: Iaa2256da7fb591d842b5bb7aa474b2ee6de9899d Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/49581 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Tested-by: jsimmons <jsimmons@infradead.org> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: jsimmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16338 readahead: add stats for read-ahead page count This patch adds the stats for read-ahead page count: lctl get_param llite.*.read_ahead_stats llite.lustre-ffff938b7849d000.read_ahead_stats= snapshot_time 4011.320890492 secs.nsecs start_time 0.000000000 secs.nsecs elapsed_time 4011.320890492 secs.nsecs hits 4 samples [pages] misses 1 samples [pages] zero_size_window 4 samples [pages] failed_to_reach_end 1 samples [pages] failed_to_fast_read 1 samples [pages] readahead_pages 1 samples [pages] 255 255 255 Signed-off-by: Qian Yingjin <qian@ddn.com> Change-Id: Iada06eb7d78ab28cfcc7167e49d25da252da4009 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/49224 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Patrick Farrell <farr0186@gmail.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16120 build: Add support for kobj_type default_groups Linux commit v5.1-rc3-29-gaa30f47cf666 kobject: Add support for default attribute groups to kobj_type Linux commit v5.18-rc1-2-gcdb4f26a63c3 kobject: kobj_type: remove default_attrs Switch to using kobj_type default_groups when it is available. Provide support for default_attrs for older kernels. HPE-bug-id: LUS-11196 Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com> Change-Id: I43b03c67c22307293a2abc444aa1a73889ca09ee Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/48365 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Jian Yu <yujian@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16231 misc: rename lprocfs_stats functions Rename lprocfs_{alloc,register,clear,free}_stats() to be lprocfs_stats_*() so these functions can be found more easily in relation to struct lprocfs_stats. Test-Parameters: trivial Signed-off-by: Andreas Dilger <adilger@whamcloud.com> Change-Id: I671284a86ee2a1fd3c58da75923f9467e72540e5 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/48847 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Ellis Wilson <elliswilson@microsoft.com> Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16231 misc: fix stats snapshot_time to use wallclock The timestamps reported during stats collection inadvertently changed from being POSIX epoch timestamps to elapsed-from-boot timestamps. While some collection tools ignore these timestamps, or only use the delta between successive reads, having uniform timestaps in stats files simplifies stats correlation between different servers. Revert the snapshot_time back to showing wallclock time. Some "init" times were not initialized when stats were allocated or cleared, do this for all stats shown by lprocfs_stats_header(). Rename struct osc_device fields from od_ to osc_ to avoid confusion with struct osd_device. Having two od_stats was especially confusing. Add a test case to verify snapshot_time, start_time, elapsed_time. Test-Parameters: testlist=sanity env=ONLY=127a,ONLY_REPEAT=100 Fixes: ea2cd3af7b ("LU-11407 obdclass: add start time to stats files") Signed-off-by: Andreas Dilger <adilger@whamcloud.com> Change-Id: I421c3b0301c2566b48c2fc6fe7bb8b54ec48ca5d Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/48821 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Li Xi <lixi@ddn.com> Reviewed-by: Ellis Wilson <elliswilson@microsoft.com> Reviewed-by: Feng Lei <flei@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16110 lprocfs: make job_stats and rename_stats valid YAML Adjust the format of job_stats and rename_stats to make them valid YAML. This fixes the output to correctly indent the items to follow YAML formatting rules. Add a test case to verify the format of these params is valid YAML to avoid other errors being introduced in the future. Signed-off-by: Lei Feng <flei@whamcloud.com> Change-Id: Idca36621241e97ff87f8ab0448f3c5604057a460 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/48417 Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Li Xi <lixi@ddn.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com>
LU-6142 obdclass: make ccc_users in cl_client_cache a refcount_t As this is used as a refcount, it should be declared as one. Change-Id: I5af513ccb2b706a398e647ce0427affa4516a9b5 Signed-off-by: Mr. NeilBrown <neilb@suse.de> Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/48881 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Reviewed-by: Emoly Liu <emoly@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-15911 enc: null encrypted names is embedded llcrypt only enable_filename_encryption tunable only makes sense when Lustre client is built against embedded llcrypt. When built against in-kernel fscrypt, this tunable is silently ignored, as fscrypt always carries out file name encryption. So have the enable_filename_encryption tunable only when Lustre client is built against embedded llcrypt. Also fix sanity-sec test_54 so that it works for in-kernel fscrypt. Fixes: e68d496ada ("LU-15858 sec: reinstate null encryption for file names") Signed-off-by: Sebastien Buisson <sbuisson@ddn.com> Change-Id: Ibe52feb670a00c9f421907ecd438bcccc62856f0 Reviewed-on: https://review.whamcloud.com/47520 Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Tested-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-15642 obdclass: use consistent stats units Use consistent stats units, since some were "usec" and others "usecs". Most stats already use LPROCFS_TYPE_* to encode type stats type, so use this to provide units for those stats, and only explicitly provide strings for the few stats that don't match the commonly-used units. This also reduces the number of repeat static strings in the modules. Signed-off-by: Andreas Dilger <adilger@whamcloud.com> Change-Id: I25f31478f238072ddbf9a3918cd43bb08c3ebbe5 Reviewed-on: https://review.whamcloud.com/46833 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Jian Yu <yujian@whamcloud.com> Reviewed-by: Ben Evans <beevans@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>