LU-17308 mgs: move pool_cmd check to the kernel Several checks for pool_cmd need to be done before touching the MGS configuration. e.g: the following case should be denied before adding a destroy record in the MGS configurations: - The pool does not exist - The pool is not empty (OSTs still in the pool) This work is done in userspace (check_pool_cmd) by checking the client lov parameters for pools. But nothing guarantees those parameters to be in sync. So, only the MGS configuration should be trusted for that. This patch move those checks in the kernel. There are several reasons for this: - It guarantees the pool configurations consistency even if an external tool is used. - For standalone MGS, it limits the overhead of reading the configuration several times. This patch add a "-n|--nowait" option for pool_cmd to skip waiting for pool updates on the clients. This is useful when doing a lot of pool_cmd in a raw. And this avoids cancelling clients CONFIG lock each times (because of mgc_requeue_timeout_min). e.g: lctl pool_destroy -n lustre.old lctl pool_new -n lustre.test lctl pool_add -n lustre.test OST0001 ... lctl pool_add lustre.test OST0010 check_pool_cmd_result() is modified to compute the client wait delay with mgc_requeue_timeout_min. Add a regression test "ost-pools 2f". Test-Parameters: testlist=ost-pools Test-Parameters: testlist=ost-pools Test-Parameters: testlist=ost-pools env=ONLY=2f,ONLY_REPEAT=50 Test-Parameters: testlist=ost-pools env=ONLY=2f,ONLY_REPEAT=50 Signed-off-by: Etienne AUJAMES <eaujames@ddn.com> Change-Id: Ifbc49b5667bf17253716052a7480114936c65149 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53202 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Guillaume Courrier <guillaume.courrier@cea.fr> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-13307 nodemap: have nodemap_add_member support large NIDs Currently when mounting lustre using IPv6 address fails with Lustre: 27361:0:(nodemap_handler.c:395:nodemap_add_member()) lustre-MDT0000: error adding to nodemap, no valid NIDs found LustreError: 11-0: lustre-MDT0000-osp-MDT0003: operation mds_connect to node 0@lo failed: rc = -22 This was due to no nodemap being set so the ptlrpc layer was not seeing any new peers. Adding minimal support to nodemap allows mounting. Change-Id: If9cfe88ec92afc3f14788f3f3ded8387a1b5d8c7 Signed-off-by: James Simmons <jsimmons@infradead.org> Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53135 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Sebastien Buisson <sbuisson@ddn.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-17000 coverity: Fix Resource Leak(3) This patch fixes error reported by coverity run. CoverityID: 404744 ("Resource leak") Test-Parameters: trivial testlist=sanity,conf-sanity Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Change-Id: Ib5a22dd09870fe43a36047e407d1dd57944c9413 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52902 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Timothy Day <timday@amazon.com> Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-17000 coverity: Fix Resource Leak(2) This patch fixes error reported by coverity run. CoverityID: 397140 ("Resource leak"): lfs.c CoverityID: 397370 ("Resource leak"): dir.c CoverityID: 397378 ("Resource leak"): obd.c CoverityID: 397406 ("Resource leak"): obd.c Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Change-Id: I92bbf6749d667631b92868bcaf78af558c250441 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52686 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Andreas Dilger <adilger@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-17182 utils: pool_add send OSTs in one batch 'lctl pool_add' command sends all requests in one batch then checks results. In this way, the command won't take too long time if the OSTs are specified in command line one by one. Signed-off-by: Lei Feng <flei@whamcloud.com> Test-Parameters: trivial Change-Id: Ibd6e7ed5104e100d44c5f4288a25e7378cd9cfe8 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52654 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Jian Yu <yujian@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-17000 coverity: Fix Resource Leak(0) This patch fixes Resource leak error reported by coverity run. CoverityID: 339696 ("Resource Leak"): liblustreapi_layout.c CoverityID: 397918 ("Resource Leak"): lsnapshot.c CoverityID: 397894 ("Resource Leak"): obd.c CoverityID: 397851 ("Resource Leak"): lfs.c CoverityID: 397832 ("Resource Leak"): liblusteapi.c CoverityID: 397772 ("Resource Leak"): liblusteapi_utils.c CoverityID: 397721 ("Resource Leak"): obd.c Test-Parameters: trivial fstype=zfs testlist=sanity,conf-sanity,sanity-lsnapshot Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Change-Id: I5c0152014f987264df17fac78390a2afc12c9255 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52272 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Timothy Day <timday@amazon.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-17000 misc: remove Coverity annotations These Coverity function annotations were added around 10 years ago. Since then, Coverity seems to produce less false positives. Out of about 20 annotations, only 3 warnings get surpressed. Thus, the applicability of these annotations should be re-evaluated. Coverity has more advanced tools now for reducing false positives. Various Lustre functions and macros could be modeled rather than using function annotations. But first, we need to get a good idea of what kinds of false postives are being generated. https://scan.coverity.com/tune Test-Parameters: trivial Signed-off-by: Timothy Day <timday@amazon.com> Change-Id: Ibcb9cf55574675e20b13a4f7a1b9142a3b75e262 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51793 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16980 build: fix gcc-12 [-Werror=use-after-free] error This patch fixes the following [-Werror=use-after-free] and [-Werror=stringop-overflow=] errors detected by gcc 12: libcfs/include/libcfs/util/list.h:481:42: error: pointer 'tmp' used after 'free' [-Werror=use-after-free] 481 | pos = list_entry(pos->member.next, typeof(*pos), member), \ | ^ libcfs/include/libcfs/util/list.h:239:28: note: in definition of macro 'list_entry' 239 | ((type *)((char *)(ptr)-(char *)(&((type *)0)->member))) | ^~~ obd.c:5118:9: note: in expansion of macro 'list_for_each_entry' 5118 | list_for_each_entry(tmp, head, lpn_list) { | ^~~~~~~~~~~~~~~~~~~ obd.c:5124:17: note: call to 'free' here 5124 | free(tmp); | ^~~~~~~~~ test_brw.c: In function 'main': test_brw.c:227:22: error: 'write' specified size between 9223372036854775808 and 18446744073709551615 exceeds maximum object size 9223372036854775807 [-Werror=stringop-overflow=] 227 | rc = write(fd, buf, len); | ^~~~~~~~~~~~~~~~~~~ Change-Id: Ibe783ab0d13e2ecde1736946323932ab5db53740 Signed-off-by: Jian Yu <yujian@whamcloud.com> Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51819 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Timothy Day <timday@amazon.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16937 utils: avoid lctl shmget() if not needed lctl is dynamically allocating an IPC shared memory segment during every startup, even though it is only needed for a small number of uncommon debug commands: shmget(IPC_PRIVATE, 65680, 0600) = 196641 shmat(196641, NULL, 0) = 0x7f752b1c5000 shmctl(196641, IPC_RMID, NULL) = 0 This setup can be moved to sub-commands that actually need it. Signed-off-by: Andreas Dilger <adilger@whamcloud.com> Change-Id: I41c790ce7cba2d9c48c1ec06eb23eb94aa548242 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51526 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Reviewed-by: Timothy Day <timday@amazon.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16723 libcfs: refactor parser to be simpler The parser code has a lot of unused complexity that can be streamlined. Refactor the parser makes the interface simpler from a development perspective and can provide a consistent user experience for all of the lustre utilities. All functions which are not used outside of the parser have been made static. Functions which don't appear to be used at all have been removed. The file headers have been standardized and the SDX text added. The function names have been changed to be more standard. Test-Parameters: testlist=sanity env=ONLY=60a,ONLY_REPEAT=10 Signed-off-by: Timothy Day <timday@amazon.com> Change-Id: I3354f213de424f51aef94c840083a4cb781d7aae Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50583 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16634 misc: replace obsolete ioctl numbers Renumber the OBD_IOC_BARRIER and IOC_OSC_SET_ACTIVE ioctl numbers to use IOC type 'f' instead of 'g' and 'h', respectively. Add compatibility code to handle their deprecation in the future. Move already-deprecated ioctls OBD_IOC_GETNAME_OLD, OBD_GET_VERSION, IOC_LIBCFS_GET_NI, and IOC_LIBCFS_PING to lustre_ioctl_old.h also. Remove other already obsolete ioctl number definitions. Add new llapi_ioctl() and llapi_ioctl_dev() helper functions to handle fallback of ioctl commands to older versions when needed. Signed-off-by: Andreas Dilger <adilger@whamcloud.com> Change-Id: I519f23d34ce6197f3430a3825c5190cedb00cf14 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50390 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Reviewed-by: Vitaliy Kuznetsov <vkuznetsov@ddn.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16524 nodemap: add rbac property to nodemap Add new rbac property to nodemap. Internally this is a mask of allowed roles. Externally it defaults to all, which means all roles are allowed, and it can take the following values (multiple can be specified, comma separated), with the semantic: - byfid_ops, to allow operations by FID (e.g. 'lfs rmfid'). - chlg_ops, to allow access to Lustre Changelogs. - dne_ops, to allow operations related to DNE (e.g. 'lfs mkdir'). - file_perms, to allow modifications of file permissions and owners. - quota_ops, to allow quota modifications. Apart from all, any role not explicitly specified is forbidden. And to forbid all roles, use 'none' value. Update lctl-nodemap-modify man page to mention this new property. Signed-off-by: Sebastien Buisson <sbuisson@ddn.com> Change-Id: I4cedf03c75948f4b1e9b55292414ab9110701874 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/49873 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com> Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16518 utils: fix clang build errors This patch fixes a number of small clang build errors in Lustre utils. Many errors are related to nuances in typing or statements which appear to be tautologies. These are resolved. Some unneeded paranthesis are removed. A variable is initialized which could potentially be left uninitialized. And a comparison was added that seemed to be left out. Signed-off-by: Timothy Day <timday@amazon.com> Change-Id: Id3f40b033e640f8d2ae6386f66a88de06fc89666 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50161 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Sebastien Buisson <sbuisson@ddn.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-11912 fid: clean up OBIF_MAX_OID and IDIF_MAX_OID Define the OBIF|IDIF_MAX_OID macros to 1ULL << OBIF|IDIF_MAX_BITS - 1 Clean up the callers and remove OBIF|IDIF_OID_MASK which are not used. Test-Parameters: trivial Change-Id: I9a679b930c73da5904b2eb4c74f785fc1d27a8a0 Signed-off-by: Li Dongyang <dongyangli@ddn.com> Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/45659 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Sergey Cheremencev <scherementsev@ddn.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-15707 lod: force creation of a component without a pool This patch add the pool option "lfs setstripe -p ignore" to force the creation of component without a pool set by inheritance (from parent or root). e.g: $ lfs setstripe -p pool tdir $ lfs setstripe -E1M -p ignore -E-1 -p '' -c2 -S2M tdir/tfile $ lfs getstripe -I1 -p tdir/tfile (no pool set) $ lfs getstripe -I2 -p tdir/tfile pool (inherited from tdir) This patch add the test "ost-pools test_32" to verify this behavior. The poorly-named "-p none" keyword, which indicates the pool name should be inherited from the root or parent dir layout, will be eventually replaced by the new "-p inherit" keyword. Test-Parameters: serverdistro=el7.9 serverversion=2.12.8 testlist=ost-pools env=ONLY=32,ONLY_REPEAT=50 Test-Parameters: testlist=ost-pools env=ONLY=32,ONLY_REPEAT=50 Signed-off-by: Etienne AUJAMES <eaujames@ddn.com> Change-Id: I782cbafe209cff6857162303a4650f5e3b438be5 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/46955 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Jian Yu <yujian@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16167 obdclass: fix lctl llog_print with skipped records The 2a5b50d ignores the skipped records in configuration llog. But if ioctl OBD_IOC_LLOG_PRINT return 0 record to display, jt_llog_print_iter() will stop the processing and ignore the non-skipped records at the end of the llog. This patch returns to user space if the last index processed (by llog_print_cb) is the last of llog file. If true, jt_llog_print_iter() stops the processing. Add regression test "conf-sanity test_123ai" for this issue. Fixes: 2a5b50d ("LU-15142 lctl: fixes for set_param -P and llog_print") Test-Parameters: testlist=conf-sanity env=ONLY=123ai,SLOW=yes,ONLY_REPEAT=10 Signed-off-by: Etienne AUJAMES <etienne.aujames@cea.fr> Change-Id: I78395268c57555e4fd2a4048ccf5b6132ca2877f Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/48586 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Alexander <alexander.boyko@hpe.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-9680 obdclass: user netlink to collect devices information Our utilities can report to users a device list with various bits of data using the debugfs file 'devices'. This debugfs file is only by default available to root which prevents regular users from collecting information. Enable non-root users to collect the same information for lctl dl using netlink. The advantage of using netlink is that it also removes the 8K ioctl limit. Add the ability to present this data in YAML format as well. Change-Id: I5e6378765bd2f4c415cf29b2bc54adf0e54f308b Signed-off-by: James Simmons <jsimmons@infradead.org> Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/31618 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Jian Yu <yujian@whamcloud.com> Reviewed-by: Neil Brown <neilb@suse.de> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-15451 sec: read-only nodemap flag Add a new 'readonly_mount' property to nodemaps. When set, we return -EROFS from server side if the client is not mounting read-only. So the client will have to specify the read-only mount option to be allowed to mount. Fixes: 928714dddabb ("LU-5092 nodemap: save id maps to targets in new index file") Signed-off-by: Sebastien Buisson <sbuisson@ddn.com> Change-Id: I9931844ae46dfd5d724f592f8dfacc4a8011c7e3 Reviewed-on: https://review.whamcloud.com/46149 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-7668 utils: add lctl del_ost Add helper command: lctl del_ost [--dryrun] --target fsname-OSTxxxx Permanently disable an OST by altering the client and MDT llog catalogs on MGS. The command finds all catalog records related to the specified OST and cancel them. A --dryrun option is provided so that the system administrator can see which records would have been cancelled, but without actually cancelling them. Signed-off-by: Stephane Thiell <sthiell@stanford.edu> Change-Id: I58c4f10fa0f7164a40231e807698eb224cccf062 Reviewed-on: https://review.whamcloud.com/41449 Tested-by: jenkins <devops@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Jian Yu <yujian@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-15706 llog: deal with "SKIP" pool llog records correctly If the requested "start" llog records by one ioctl are just the useful ones "protected" by one marker, they will don't know their cfg_flags(e.g. "SKIP" or not), and then will be mis-labeled in class_config_yaml_output(). To fix this issue, this patch does the following changes: - In kernel space, remember the marker cfg_flags for 10 records earlier than the requested "start" in function llog_print_cb(), so that the "start" can be output with its correct cfg_flags. - In user space, since the pool_new/add record will be marked as "SKIP" if its corresponding pool_destroy/remove record is logged later in mgs_pool_cmd(), these "SKIP" records won't be printed by function jt_llog_print_iter(), so lpd_ost_num doesn't need to be decreased in callback function llog_search_pool_cb() as well, otherwise, it will cause "lctl pool_destroy" error. Test-Parameters: standalonemgs=true testlist=ost-pools Test-Parameters: standalonemgs=true testlist=conf-sanity env=ONLY=123,HONOR_EXCEPT=y Fixes: 2a5b50d20717 ("LU-15142 lctl: fixes for set_param -P and llog_print") Signed-off-by: Emoly Liu <emoly@whamcloud.com> Change-Id: Ie45127ac8b80a75eaeb7158559c690da52eef103 Reviewed-on: https://review.whamcloud.com/46951 Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Reviewed-by: Mike Pershin <mpershin@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: John L. Hammond <jhammond@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>