LU-6142 uapi: Fix style issues for lustre_disk.h This patch fixes issues reported by checkpatch for file lustre/include/lustre_disk.h Test-Parameters: trivial Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Change-Id: I20cc1784602b7e95a5c1541851684d3c2199be1b Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53919 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Timothy Day <timday@amazon.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-12998 mds: add no_create parameter to stop creates Add an target tunable parameter and mount option "no_create" to disable new *directory* creation on an MDT. This sends the flag OS_STATFS_NOCREATE to the clients, and the DNE MDT space balance will avoid selecting that MDT when creating a new subdirectory, without disabling access to existing files/dirs. This allows "soft disabling" an MDT in advance of storage upgrades to minimize new directories and files created on that MDT, reduce future migration, and/or backup/restore workload. As yet it does not totally disable *file* creation on the MDT, but it may be extended to do so in the future. This is analogous to the "no_precreate" option that was added on the OSTs, and "no_create" has been added to the OSTs for consistency ("no_precreate" is kept for compatibility for now). Test-Parameters: testlist=conf-sanity env=ONLY=112b,ONLY_REPEAT=50 Signed-off-by: Andreas Dilger <adilger@whamcloud.com> Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com> Change-Id: I53cfb48ade2f844b18bfc630e7fcea6de9ce7057 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/47124 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16796 ptlrpc: Change struct lsi_mounts to use kref This patch changes struct lsi_mounts to use kref(refcount_t) instead of atomic_t Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Change-Id: Ia185b19123f535f8c54a6ea6b7a0212fbe85ffea Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51864 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Neil Brown <neilb@suse.de> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-13306 mgs: use large NIDS in the nid table on the MGS On the MGS the NIDs detected are handled using the struct mgs_target_info which currently only handles lnet_nid_t. This structure also limits the number of NIDs to 32 entries. Some sites have reported that 32 NIDs wasn't enough when they configured virtual LNet networks for isolation. Update the mgs_target_info to use NID strings instead. This has the advantage of working even if struct lnet_nid expands in the future. We place this data at the end of the mgs_target_info as a flexible array. This requires updating the ptlrpc packet handling to increase the size to some new value to contain all the NIDs registered. Also this gives us the option to use hostnames in the future. This information is then feed into a struct mgs_nidtbl_entry which is sent to the mgc on all the remote nodes. With this patch only large NIDs for small address space is translated to the original lnet_nid_t format and sent to the various clients. All the server targets, which are clients of the MGS, use the large NID format. With this patch we don't have to patch old clients when the servers are using the larger NID format. Expand LNetGetId() to return large NID addresses as well. In the future we will use the ocd_connect_flags to determine if the MSG supports large NID addresses. Change-Id: I7083d6ecfc46cf0419a0d4a582e4bf5240f193cd Signed-off-by: James Simmons <jsimmons@infradead.org> Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50896 Tested-by: Maloo <maloo@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Neil Brown <neilb@suse.de> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-6142 obd: change lmd flags to bitmap Change lmd flags to an enum that is accessible with the Linux bitmap API. This lays the foundation for creating a match table for the server options for mounting. Change-Id: If7906a9a3ba177b67d0cfbaa276a00a6ba9b7b6d Signed-off-by: James Simmons <jsimmons@infradead.org> Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/49912 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Neil Brown <neilb@suse.de> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-15420 sec: handle simple fscrypt changes for 5.15 kernels The patch covers the low impact changes to the Linux kernels fscrypt API. The changes are: For Linux kernel version 5.9-rc4 the following commits are: 8b10fe68985278de4926daa56ad6af701839e40a removed the inode parameter for the fscrypt function fscrypt_fname_alloc_buffer() 5b2a828b98ec1872799b1b4d82113c76a12d594f ended up exporting fscrypt_d_revalidate() and stopped stomping on the d_ops. c8c868abc91ff23f6f5c4444c419de7c277d77e1 changed fscrypt_set_test_dummy_encryption() take a 'const char * ac4acb1f4b2b6b7e8d913537cccec8789903e164 moved the fscrypt core from using fscrypt_context to using fscrypt_policy that is user forward facing. Lastly for Linux kernel version 5.10-rc4 the commit ec0caa974cd092549ab282deb8ec7ea73b36eba0 stopped exporting fscrypt_get_encryption_info(). Use fscrypt_prepare_readdir() in its place. 70fb2612aab62d47e03f82eaa7384a8d30ca175d renamed a field in struct fscrypt_name. Since Lustre can't use fscrypt_prepare_lookup we have to deal with this change. Remove sptlrpc_enc_pool_del_user() since its an empty function that waste cycles calling it. The other large change was the replacement of fscrypt_inherit_context which is described in Linux commit a992b20cd4ee360dbbe6f69339cb07146e4304d6. This change is very large since it expects the target inode to be available. Lustre uses fscrypt_inherit_context before the inode is available so this is a much more complex change that will be done in another patch. de3cdc6e75179a2324c23400b21483a1372c95e1 makes fscrypt_require_key private. You need to test the key's presence with fscrypt_has_encryption_key() instead but that key needs to be setup first by the new function fscrypt_prepare_new_inode(). Since this is the case we wait to introduce this change. Change-Id: I4bed7fef6e3302c0258c0f1563f4e180258d7a5a Signed-off-by: James Simmons <jsimmons@infradead.org> Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/49125 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Sebastien Buisson <sbuisson@ddn.com> Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16374 enc: align Base64 encoding with RFC 4648 base64url Lustre encryption uses a Base64 encoding to encode no-key filenames (the filenames that are presented to userspace when a directory is listed without its encryption key). Make this Base64 encoding compliant with RFC 4648 base64url. And use '+' leading character to distringuish digested names. This is adapted from kernel commit ba47b515f594 fscrypt: align Base64 encoding with RFC 4648 base64url To maintain compatibility with older clients, a new llite parameter named 'filename_enc_use_old_base64' is introduced, set to 0 by default. When 0, Lustre uses new-fashion base64 encoding. When set to 1, Lustre uses old-style base64 encoding. To set this parameter globally for all clients, do on the MGS: mgs# lctl set_param -P llite.*.filename_enc_use_old_base64={0,1} Signed-off-by: Eric Biggers <ebiggers@google.com> Signed-off-by: Sebastien Buisson <sbuisson@ddn.com> Change-Id: Iaa2256da7fb591d842b5bb7aa474b2ee6de9899d Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/49581 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Tested-by: jsimmons <jsimmons@infradead.org> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: jsimmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16385 obdlcass: stop MGC before MGS drops a reference to MGC when MGS is being umounted so that MGC doesn't try to disconnected from a missing MGS which can take long and hurt HA. Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com> Change-Id: Ib15f1ca56c47201bf6e29c12b3f81a11e55944ca Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/49378 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Mikhail Pershin <mpershin@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org>
LU-8837 lustre: make uapi...lustre_disk.h unnecessary on client uapi/linux/lustre/lustre_disk.h doesn't contain anything that is needed for client-only code, but that code doesn't compile with the file excluded, largely due to dependency on IS_SERVER() and related macros. So for client-only code provide stubs for IS_SERVER() and related macros, and don't include the uapi...lustre_disk.h file. This will cause some code to now be compiled-out on client-only, and allows some #ifdefs to be removed. A few function need to be protected with #ifdef HAVE_SERVER_SUPPORT, and llog_server.o needs to be disabled for client-only compiles. Signed-off-by: Mr NeilBrown <neilb@suse.de> Change-Id: I19c5b1612108e448f8b6a1fe3d3a448aa4abdd2a Reviewed-on: https://review.whamcloud.com/41994 Reviewed-by: James Simmons <jsimmons@infradead.org> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-13783 sec: support of native Ubuntu 20.04 HWE 5.8 kernel For Linux 5.5 kernel a patch to improve nokey names was landed that removed several variables that Lustre's llite and mdt layer were using. Rework the code to use other defines that exist. Second change for Ubuntu is several backports to handle a few variables changing across different kernel versions. One is the name change of DCACHE_ENCRYPTED_NAME and the other being is_chipertext_name. So the simpler approach Ubuntu took was to use fscrypt_prepare_lookup() and fscrypt_is_nokey_name() to work around these changes. Lastly the good news is for 5.12 the stomping of ll_d_ops no longer happens and the special revalidate_dentry fscrypto function is exported. Change-Id: I7f70fe9abddf34798e2e01b35099c9a032d92b91 Signed-off-by: James Simmons <jsimmons@infradead.org> Signed-off-by: Sebastien Buisson <sbuisson@ddn.com> Reviewed-on: https://review.whamcloud.com/46238 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Jian Yu <yujian@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-15858 sec: reinstate null encryption for file names Reinstate null encryption for file names by adding a new llite parameter named 'enable_filename_encryption', set to 0 by default. When this parameter is 0, new empty directories configured as encrypted ignore the filenames_encryption_mode and use LLCRYPT_MODE_NULL instead, which is a no-op. This LLCRYPT_MODE_NULL mode is inherited for all subdirectories and files. When this parameter is 1, new empty directories configured as encrypted use the normal encryption mode. To set this parameter globally for all clients, do on the MGS: mgs# lctl set_param -P llite.*.enable_filename_encryption=0 Also update sanity-sec test_54 to exercise the new parameter 'enable_filename_encryption'. Signed-off-by: Sebastien Buisson <sbuisson@ddn.com> Change-Id: I9d726ba26cc91a51690d59a81efe3eb98ee2995c Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-on: https://review.whamcloud.com/47355 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-14651 uapi: rename CONFIG_T_* to MGS_CFG_T_* The Linux kernel uses CONFIG_* as a way to determine if a feature is available. Using CONFIG_* in an UAPI is considered an error and in the most recent kernels will break a build. While we don't have any CONFIG_* in our UAPI headers we do have CONFIG_T_* which is used for config logs. This naming confuses the Linux kernel build system so just rename these variables to MGS_CFG_T_* instead. Test-Parameters: trivial Change-Id: I574510c2da90e9ae608c9e2374d75220b7abb19d Signed-off-by: James Simmons <jsimmons@infradead.org> Reviewed-on: https://review.whamcloud.com/43494 Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Aurelien Degremont <degremoa@amazon.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-14487 lustre: remove references to Sun Trademark. "lustre" is no longer a Trademark of Sun Microsystems. There is no need to acknowledge the trademark is every file, so just remove all these claims. Test-Parameters: trivial Signed-off-by: Mr NeilBrown <neilb@suse.de> Change-Id: I214670b39c5718f2b691193f268a64856e0cd743 Reviewed-on: https://review.whamcloud.com/41880 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org>
LU-14090 mgs: no local logs flag There is a feature that starts a target with a local copy of config log in order to avoid a delay in communicating with an MGS and to load mgs log updates later on. However, that feature is not always useful. When replace_nids adds records with new nids it does not append remote config logs but overwrite corresponding records in place. If a target starts using local config log - it gets confused by outdated nids. This patch adds tunefs.lustre --nolocallogs key that sets nolocallogs flag, which says ignore local configs copy. The flag is reset once new logs are uploaded from MGS. tunefs.lustre --nolocallogs is suggested to be executed on targets together with replace_nids on MGS. HPE-bug-id: LUS-2510 Change-Id: I949c19ac701d287e1c1199bc12445989476a707b Signed-off-by: Artem Blagodarenko <artem.blagodarenko@hpe.com> Reviewed-on: https://es-gerrit.dev.cray.com/157574 Reviewed-by: Vladimir Saveliev <c17830@cray.com> Reviewed-by: Nikitas Angelinas <nangelinas@cray.com> Tested-by: Alexander Lezhoev <c17454@cray.com> Reviewed-on: https://review.whamcloud.com/40448 Tested-by: jenkins <devops@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-12514 llite: move client mounting from obdclass to llite Mounting a lustre client is currently handled in obdclass, using services from llite. This requires obdclass to load the llite module and set up inter-module linkage. The purpose of this was for common code to support both client and server mounts. This isn't really a good idea and need to go. For lustre servers we already use a separate filesystem type. So move the mounting code from obdclass/obd_mount to llite/super25 and remove the inter-module linkages. Add some EXPORT_SYMBOL() so that llite can access some helpers that remain in obdclass. Linux-commit: a989830c88149511ee840356d9c1b34304bac576 Change-Id: Ia33bd55a042f90b178156c745a8072b516f00568 Signed-off-by: Mr NeilBrown <neilb@suse.de> Reviewed-on: https://review.whamcloud.com/37693 Tested-by: Maloo <maloo@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-12546 mdt: abort recovery between MDTs Add an option to abort recovery between MDTs in case there is a problem during recovery (e.g. MDT is missing or has broken logs), but don't abort recovery between MDT and clients. Change-Id: Id88f2b2ebae5cfa722dcac67c087b9b9a448721e Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com> Reviewed-on: https://review.whamcloud.com/36027 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
LU-12275 sec: add llcrypt as file encryption library Include fscrypt sources from Linux 5.4 into libcfs kernel module. All fscrypt_ prefixes have been replaced with llcrypt_ to avoid collision. For the exact transformations carried out, please see script contrib/scripts/fscrypt_inclusion.sh, and patches under contrib/scripts/crypto_patches/. This llcrypt library will be built if kernel provides minimum encryption support, ie IS_ENCRYPTED and S_ENCRYPTED. If kernel provides support for encryption policies v2 (included in Linux 5.4), then llcrypt library will not be built, and in-kernel fscrypt will be used instead. To be independent as much as possible from the kernel, llcrypt relies on a new 'lsi_cop' field added to struct lustre_sb_info, to point to struct llcrypt_operations, and makes use of the new 'lsi_master_keys' field to store file system keys. llcrypt also uses i_private field on struct inode to store file-specific enryption info. Test-Parameters: trivial Test-Parameters: clientdistro=ubuntu1804 testgroup=review-ldiskfs Test-Parameters: clientdistro=el8.1 testgroup=review-ldiskfs Signed-off-by: Sebastien Buisson <sbuisson@ddn.com> Change-Id: Ide2431b0c3381214aed2e54dd3084132e9545ca8 Reviewed-on: https://review.whamcloud.com/38127 Tested-by: jenkins <devops@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Yang Sheng <ys@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-12722 target: disable recovery for local clients when client is running on a server node, then the local services can't rely on that client in the contex of recovery - such a client dies with the node, can't replay requests and states and then the restarting server has to wait till recovery expires which doesn't make any sense. so the servers should recogize local clients and exclude them from recovery (i.e. don't make them part of last_rcvd). for the purpose of local testing a special mount option "local_recov" has been added to {MDS|OST}_MOUNT_OPTS in tests/cfg/local.sh to save local testing when everyting is running within a single node. Signed-off-by: Alexey Zhuravlev <bzzz@whamcloud.com> Change-Id: I4cb906c44c1192933f7d77dc782160e426e9efde Reviewed-on: https://review.whamcloud.com/36025 Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Li Xi <lixi@ddn.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-9679 llite: fix possible race with module unload. lustre_fill_super() calls client_fill_super() without holding a reference to the module containing client_fill_super. If that module is unloaded at a bad time, this can crash. To be able to get a reference to the module using try_get_module(), we need a pointer to the module. So replace lustre_register_client_fill_super() and lustre_register_kill_super_cb() with a single lustre_register_super_ops() which also passed a module pointer. Then use a spinlock to ensure the module pointer isn't removed while try_module_get() is running, and use try_module_get() to ensure we have a reference before calling client_fill_super(). Now that we take the reference to the module before calling luster_fill_super(), we don't need to take one inside lustre_fill_super(). Linux-commit: d487fe31f49e78f3cdd826923bf0c340a839ffd8 Signed-off-by: Mr NeilBrown <neilb@suse.de> Change-Id: I9474622f2a253d9882eae3f0578c50782dd11ad4 Reviewed-on: https://review.whamcloud.com/37020 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Jian Yu <yujian@whamcloud.com> Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-12036 ofd: add "no_precreate" mount option Add a mount option to disallow object creation on the OST. That allows an OST to be mounted by the administrator without it being immediately available for use by clients/applications. This may be useful if the OST needs to be added to a specific pool first, or if it is being debugged or similar. Mount option can be disabled with the obdfilter.*.no_precreate tunable parameter. Signed-off-by: Andreas Dilger <adilger@whamcloud.com> Change-Id: Icdb64a4bdd5a66b0e9e6d483e3113b97d53ebbe5 Reviewed-on: https://review.whamcloud.com/36716 Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>