LU-16786 utils: Replace open call to WANT_FD Replace open call to WANT_FD with newly added API llapi_root_path_open() which was added under LU-16427 Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Change-Id: I97d55321cf32e40eaf7d6284c47f313199a6c406 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50764 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
LU-16310 sec: Lustre/HSM on enc file with enc key Support for Lustre/HSM on encrypted files when the encryption key is available requires similar attention as with file migration. The volatile file used for HSM restore must have the same encryption context as the Lustre file being restored, so that file content remains accessible after the layout swap at the end of the restore procedure. Please note that using Lustre/HSM with the encryption key creates clear text copies of encrypted files on the HSM backend storage. Signed-off-by: Sebastien Buisson <sbuisson@ddn.com> Change-Id: I99cba202cd2c7c747bbe5c4ec7d9208c7f6baf4b Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/49153 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: jsimmons <jsimmons@infradead.org> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Etienne AUJAMES <eaujames@ddn.com>
LU-8585 llapi: use open_by_handle_at in llapi_open_by_fid Reimplement llapi_open_by_fid() to use llapi_fid_to_handle() and open_by_handle_at(2) rather than using ioctl(). This works for opens on subdirectory mountpoints, unlike ".lustre/fid/<fid>". This patch also adds llapi_open_by_fid_at() which is similar to llapi_open_by_fid() except that it takes an open directory file descriptor or AT_CWD rather than a path as its first argument. [AD: - Move get_root_*() functions over to a new liblustreapi_root.c file in expectation of further enhancements to that code. - Cache an open file handle on the root directory so repeated calls to llapi_open_by_fid() and llapi_fid2path() do not need to search for and open the same root directory path many times. - Add man pages for newly-added functions. This reduces the system calls for llapi_fid_test significantly: original patched 14511 4315 total opens 64807 34067 total syscalls ] There may still be a need to have a fallback from open_by_handle_at() to using ".lustre/fid/<FID>" to open the fid (if available), but that can be added if this initial patch does not test well. The open_by_handle_at() method avoids reopening the "fid/" directory each time (though this fd could also be cached), but it has the drawback that it reconnects dentries to the root directory each time. Signed-off-by: Quentin Bouget <quentin.bouget@cea.fr> Change-Id: I8a4904c996389da2b0894cd9fac639a398607535 Signed-off-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/36603 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Jian Yu <yujian@whamcloud.com> Reviewed-by: Emoly Liu <emoly@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16076 utils: enhance 'lfs check' command Add optional argument to 'lfs check' command so that only the servers related to the specified lustre file system is checked. Signed-off-by: Lei Feng <flei@whamcloud.com> Test-Parameters: trivial testlist=sanityn env=ONLY=113 Change-Id: I826a8e822af0a290f06ffaadadf1bb7f86899d99 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/48155 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Emoly Liu <emoly@whamcloud.com> Reviewed-by: Jian Yu <yujian@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-15833 llapi: don't use realpath in llapi_search_fsname() This patch use st_dev value to dertermine the fsname in llapi_search_fsname(). The main purpose of this is to limit the number of lstat() (realpath()) in this function. get_root_path() is modified to search a mountpoint by dev. And the last results of get_root_path() is cached to avoid reading /proc/mount for each call. A new api function llapi_search_rootpath_by_dev() is added to get the path of Lustre mountpoint using the specified device value. **Testing:** *Environement:* VMs: 1 client, 1 MDS (2MDT), 1 OSS (2 OST) Lustre tree: test{001..100}/test{001..100}/test{01..10}/file{01..05} (500000 files + 110100 folders) OS: Centos 7 (no statx) Lustre: 2.15.50_15_g1116739 *Tests* cd <rootfs> strace lfs getstripe -r . echo 3 > /proc/sys/vm/drop_caches /usr/bin/time lfs getstripe -r . (2 iterations) *Results* times (s): ______________________________ | user | system | real | real% | _______________|______|________|______|_______| |without patch: | 6.18 | 57.3 | 427 | 0% | |_______________|______|________|______|_______| |with patch: | 2.88 | 47.3 | 404 |-5.45% | |_______________|______|________|______|_______| strace (only significant changes are displayed): (*stat = lstat + stat + fstat) _____________________________________________ | *stat | mmap | open | read | all | _______________|________|________|________|________|_________| |without patch: | 760545 | 110142 | 330379 | 330325 | 4742658 | |_______________|________|________|________|________|_________| |with patch: | 440484 | 0 | 220277 | 19 | 3541739 | |_______________|________|________|________|________|_________| -25.32% syscalls after patching. Signed-off-by: Etienne AUJAMES <etienne.aujames@cea.fr> Change-Id: I3812d922d5b1d194d52132cba95d11820424c5d7 Reviewed-on: https://review.whamcloud.com/47258 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-14732 utils: ensure hsm_user_request extent fields are zero Another patch changed the linking flags for liblustreapi which caused sanit-hsm 29c to fail. Debugging revealed the reason was the extent fields in hsm_user_request sent to the kernel was not zeroed out. For some reason the compiler flags hide this bug. I found this is easily reproduced by removing lhsmtool_posix_DEPENDENCIES which is not needed anyways since the build system can calculate those dependencies for us. Update the function llapi_hsm_user_request_alloc to zero out the struct hsm_user_request allocated. Change-Id: I02c1d6b5cec1ed3e89086a6a00e30ca81768409c Signed-off-by: James Simmons <jsimmons@infradead.org> Reviewed-on: https://review.whamcloud.com/43893 Tested-by: jenkins <devops@whamcloud.com> Reviewed-by: Ben Evans <beevans@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com> Reviewed-by: John L. Hammond <jhammond@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-14398 hsm: use llapi_fid2path_at() in the copytool In lhsmtool_posix.c and liblustreapi_hsm.c, convert several uses of uses of llapi_fid2path() to llapi_fid2path_at(). Signed-off-by: John L. Hammond <jhammond@whamcloud.com> Change-Id: Ice64d02010b4260287be4d4e26c6b75b178bc81b Reviewed-on: https://review.whamcloud.com/41408 Tested-by: jenkins <devops@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-9679 general: avoid bare return; at end of void function Having: return; } at the end of a void function is unnecessary noise. Where it is the *only* statement in the function, it can be useful, so that remain unchanged. The rest have been removed. Test-Parameters: trivial Signed-off-by: Mr NeilBrown <neilb@suse.de> Change-Id: If02f6f5b91d4134cf95a68ebccc83df28c360fb2 Reviewed-on: https://review.whamcloud.com/36654 Tested-by: jenkins <devops@whamcloud.com> Reviewed-by: Shaun Tancheff <stancheff@cray.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com> Reviewed-by: Ben Evans <bevans@cray.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-11367 som: integrate LSOM with lfs find The patch integrates LSOM functionality with lfs find so that it is possible to use LSOM functionality directly on the client. The MDS fills in the mbo_size and mbo_blocks fields from the LSOM xattr, if the actual size/blocks are not available, and then set new OBD_MD_FLLSIZE and OBD_MD_FLLBLOCKS flags in the reply so that the client knows these fields are valid. The lfs find command adds "-l|--lazy" option to allow the use of LSOM data from the MDS. Add a new version of ioctl(LL_IOC_MDC_GETINFO) call that also returns valid flags from the MDS RPC to userspace in struct lov_user_mds_data so that it is possible to determine whether the size and blocks are returned by the call. The old LL_IOC_MDC_GETINFO ioctl number is renamed to LL_IOC_MDC_GETINFO_OLD and is binary compatible, but newly-compiled applications will use the new struct lov_user_mds_data. New llapi interfaces llapi_get_lum_file(), llapi_get_lum_dir(), llapi_get_lum_file_fd(), llapi_get_lum_dir_fd() are added to fetch valid stat() attributes and LOV info to the user. Signed-off-by: Qian Yingjin <qian@ddn.com> Signed-off-by: Andreas Dilger <adilger@whamcloud.com> Change-Id: I21dfae7c2633dead5d83b438ec340fea4d3ebbe5 Reviewed-on: https://review.whamcloud.com/35167 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Li Xi <lixi@ddn.com>
LU-10092 pcc: Non-blocking PCC caching Current PCC uses refcount of PCC inode to determine whether a previous PCC-attached file can be detached. If a file is open (refcount > 1), the detaching will return -EBUSY. When another client accesses the PCC-cached file, it will trigger the restore process as the file is HSM released. During restore, the Agent needs to detach the PCC-cached file. Thus, if a PCC-attached file is keeping opened but not closed for a long time, the restore request will always return failure. In this patch, we implement a non-blocking PCC caching mechanism for Lustre. After attaching the file into PCC, the client acquires the layout lock for the file, and the layout generation is maintained in the PCC inode. Under the layout lock protection, the PCC caching state is valid and all I/O will direct into PCC. When the layout lock is revoked, in the blocking AST it will invalidate the PCC caching state and detach the file automatically. This patch is also helpful to handle the ENOSPC error for PCC write by fallback to normal I/O path which will restore the file data into OSTs (The file is in HSM released state) and redo the write again. Change-Id: I9130c04dc0e6eae879ea2ff3fdda65726e74d177 Test-Parameters: clientcount=3 testlist=sanity-pcc,sanity-pcc,sanity-pcc Signed-off-by: Qian Yingjin <qian@ddn.com> Reviewed-on: https://review.whamcloud.com/32966 Tested-by: Jenkins Reviewed-by: Wang Shilong <wshilong@ddn.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-10114 hsm: increase upper limit of maximum HSM backends registered with MDT Lustre only supports at most 32 HSM backends, which limits HSM to be applied to other features, such as LPCC. This patch breaks the limitation by allowing the system take any interger number as a valid archive-id. Test-Parameters: clientjob=lustre-b2_10 clientbuildno=136 testlist=sanity-hsm Test-Parameters: mdsjob=lustre-b2_10 ossjob=lustre-b2_10 serverbuildno=136 testlist=sanity-hsm Change-Id: I9523b92500b962db3e45a2bd6a67dba54eef5335 Signed-off-by: Teddy Zheng <teddy@ddn.com> Signed-off-by: Li Xi <lixi@ddn.com> Reviewed-on: https://review.whamcloud.com/32197 Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Tested-by: Jenkins Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-10822 utils: stop bogus buffer overflow errors Over-zealous Fortify checks assume that the buffer being used for snprintf() in get_lmd_info() is sizeof(*lmd) when in fact a larger buffer has been allocated. This causes runtime checks to fail and lfs to core dump: *** buffer overflow detected ***: /usr/bin/lfs terminated Instead of printing directly into "struct lov_user_mds_data", use a generic buffer to hold the filename passed into the ioctl and the return data. There are several places in the code which do the same operations, namely cb_getstripe(), get_lmd_info(), and ct_md_getattr(), so change them all to call get_lmd_info() or a new get_lmd_info_fd() helper to consolidate common code. Also check the return values from snprintf() in case there are new callers of this code in the future that do not actually pass large-enough buffers. Test-Parameters: clientdistro=ubuntu1604 serverdistro=el7 testlist=sanity Test-Parameters: trivial testlist=sanity-hsm Signed-off-by: Andreas Dilger <andreas.dilger@intel.com> Change-Id: I41b1fcba1f7937fbce3cc7180ed5d73d067cab07 Reviewed-on: https://review.whamcloud.com/31822 Tested-by: Jenkins Reviewed-by: John L. Hammond <john.hammond@intel.com> Tested-by: Maloo <hpdd-maloo@intel.com> Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-10308 misc: update Intel copyright messages for 2017 Update copyright messages for files updated in 2016, excluding trivial patches. Add trivial patches to updatecw.sh script exclude list. Revert some changes that were incorrectly attributed to the 2016 (d10200a80770f0029d1d665af954187b9ad883df) and 2015 (0754bc8f2623bea184111af216f7567608db35b6) copyright update patches themselves, since they were not in the exclude list when the subsequent script was run. Test-Parameters: trivial Signed-off-by: Andreas Dilger <andreas.dilger@intel.com> Change-Id: I82f21c30c4dac75792bb49fc139bee2ca51f5545 Reviewed-on: https://review.whamcloud.com/30341 Tested-by: Jenkins Tested-by: Maloo <hpdd-maloo@intel.com> Reviewed-by: Jian Yu <jian.yu@intel.com> Reviewed-by: James Nunez <james.a.nunez@intel.com> Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-10357 hsm: open file to be archived before starting In the archive case of llapi_hsm_action_begin(), open the file to be archived before calling the LL_IOC_HSM_COPY_START ioctl. Store the open FD in struct hsm_copyaction_private, return a dup of it llapi_hsm_action_get_fd(), and close it in llapi_hsm_action_end(). Calling open() first avoids 3 extra RPCs (MDS_GETATTR, LDLM_ENQUEUE for layout to get data version, LDLM_ENQUEUE for fstat()) when archiving. Signed-off-by: John L. Hammond <john.hammond@intel.com> Change-Id: Idea0a7f4cb63bd9d712ff2ce9fcb59a3b278d0f2 Reviewed-on: https://review.whamcloud.com/30456 Tested-by: Jenkins Tested-by: Maloo <hpdd-maloo@intel.com> Reviewed-by: Quentin Bouget <quentin.bouget@cea.fr> Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com> Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-6142 uapi: Get rid of lustre_fid typedef Replace it with struct lu_fid. Update the userland code and man pages to reflect this change. Linux-commit: d8f6bc9a53f97d1ea4b2b955672904338643308b Test-Parameters: trivial Change-Id: I0b7e0770dd9da9bdac55c02c2ec98aea7cea7100 Signed-off-by: James Simmons <uja.ornl@yahoo.com> Signed-off-by: Oleg Drokin <oleg.drokin@intel.com> Reviewed-on: https://review.whamcloud.com/29849 Tested-by: Jenkins Tested-by: Maloo <hpdd-maloo@intel.com> Reviewed-by: Andreas Dilger <andreas.dilger@intel.com> Reviewed-by: Ben Evans <bevans@cray.com> Reviewed-by: John L. Hammond <john.hammond@intel.com>
LU-6401 uapi: migrate remaining uapi headers to uapi directory Move all the remaining lustre headers shared between user land and kernel space to the uapi directory. OBD_MAX_IOCTL_BUFFER can be removed since the utilites don't even acknowledge it value. Change-Id: I6dcdf3f824dae6bc8a0786f0e61787ca68c10bd8 Signed-off-by: James Simmons <uja.ornl@yahoo.com> Reviewed-on: https://review.whamcloud.com/25246 Tested-by: Jenkins Reviewed-by: Quentin Bouget <quentin.bouget@cea.fr> Tested-by: Maloo <hpdd-maloo@intel.com> Reviewed-by: Ben Evans <bevans@cray.com> Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com> Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-6245 uapi: move libcfs/lnet UAPI header into own uapi directory Move all the libcfs/lnet headers shared between user land and kernel space to the new lnet uapi directory. Test-Parameters: trivial Change-Id: Id6142b0e890fd47d3318dc3c98e64f72b5df2594 Signed-off-by: James Simmons <uja.ornl@yahoo.com> Reviewed-on: https://review.whamcloud.com/28089 Tested-by: Jenkins Tested-by: Maloo <hpdd-maloo@intel.com> Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-8820 hsm: skip LL_IOC_HSM_COPY_START for HSM removes In llapi_hsm_action_begin(), if the action is a remove then skip calling the LL_IOC_HSM_COPY_START ioctl. Signed-off-by: John L. Hammond <john.hammond@intel.com> Change-Id: Id84f1877a4f05c91e063797b51bd2046f2283ec4 Reviewed-on: https://review.whamcloud.com/23700 Reviewed-by: Bob Glossman <bob.glossman@intel.com> Tested-by: Jenkins Tested-by: Maloo <hpdd-maloo@intel.com> Reviewed-by: Faccini Bruno <bruno.faccini@intel.com> Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-8587 utils: Fix incorrect indenting in llapi_hsm_log_ct_progress gcc6 highlights this case of incorrect indenting: if (progress_type == CT_RUNNING) rc = llapi_json_add_item(&json_items, "current_bytes", LLAPI_JSON_BIGNUM, ¤t); if (rc < 0) goto err; Just add the braces around, though logic-wise it's all fine. Change-Id: I770857fe2f9ce29817558247ce0987842b8d06b4 Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com> Signed-off-by: Oleg Drokin <oleg.drokin@intel.com> Reviewed-on: https://review.whamcloud.com/24570 Tested-by: Maloo <hpdd-maloo@intel.com> Reviewed-by: John L. Hammond <john.hammond@intel.com> Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
LU-8901 misc: update Intel copyright messages for 2016 Update copyright messages for files updated in 2016, excluding trivial patches. Add trivial patches to updatecw.sh script exclude list. Improve the output of patches causing the most changes to include the patch summary instead of just the hash. Test-Parameters: trivial Signed-off-by: Andreas Dilger <andreas.dilger@intel.com> Change-Id: Ifb098e8126f698ea975e10ba13f5f47c449ae4ad Reviewed-on: https://review.whamcloud.com/24233 Reviewed-by: Oleg Drokin <oleg.drokin@intel.com> Tested-by: Oleg Drokin <oleg.drokin@intel.com>