LU-16695 llite: remove O_APPEND check for sync A check for O_APPEND in determining 'sync' or not was accidentally introduced. This forces O_APPEND writes to all be synchronous, which is of course wrong. Fixes: dad7079dfd ("LU-16695 llite: switch to ki_flags from f_flags") Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com> Change-Id: Iafae63ebda527834bd45d6fcbfb0cebb0340f4e4 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/54128 Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com> Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com> Reviewed-by: Andrew Perepechko <andrew.perepechko@hpe.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com>
LU-17482 llite: short read could mess up next read offset When read reaches EOF, it could read data from stale pagecache, but we need to restore the iocb->ki_pos so that next read could continue from the correct offset. Fixes: 4468f6c9d9 ("LU-16025 llite: adjust read count as file got truncated") Signed-off-by: Bobi Jam <bobijam@whamcloud.com> Change-Id: Ib8b62c41bf65f8efec82dda53fcfbdb68ad08b38 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53827 Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Qian Yingjin <qian@ddn.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Patrick Farrell <patrick.farrell@oracle.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com>
LU-13048 mdd: allow release after a non-blocking migrate lfs setstripe -i0 file lfs hsm_archive file lfs migrate -n -i1 file lfs hsm_release file These actions lead to "Cannot send HSM request ...: Operation not permitted". This happens because of data version mismatch. This error is returned by mdt_hsm_release() when the data versions are not the same. This patch only corrects the non-blocking migrations. mdd_swap_layouts is updated to check and update the HSM archive version when possible. The new and old data versions are added as arguments to this function. If the old data version does not match the data version in the HSM attribute, we don't update the HSM attribute because we don't know what caused the inconsistency. During a swap between a volatile and a regular file, if both objects have an HSM xattr, mdd_swap_layouts was called from the MDT HSM layer (release and restore). In this case, we want to swap the HSM xattr (previously done using SWAP_LAYOUTS_MDS_HSM as a last argument to mdd_swap_layouts). If only the regular file has an HSM attribute, mdd_swap_layouts was called after a migration (blocking or not). In this case, we want to update the HSM archive version only if the file is not dirty and if the new data version is provided. Also, this patch removes the CL_LAYOUT event that was emitted for a release. Since a CL_HSM event with HE_RELEASE flag is also emitted, the CL_LAYOUT is unecessary. For "lfs swap_layouts", the operation is denied on 2 files with HSM xattr (HSM xattr swap will cause inconsistencies). With non-HSM file and archived file, the operation is allowed but the dirty flag is set on the HSM file. Add lustre_swab_close_data_special() to swab close_data fields inside the union (specific to some types of close). Add regression test sanity-hsm 607a, 607b and 607c. Test-Parameters: clientversion=2.15.4 testlist=sanity-hsm Test-Parameters: serverversion=2.15.4 testlist=sanity-hsm env=EXCEPT="114 409a" Test-Parameters: testlist=sanity-hsm env=ONLY=607,ONLY_REPEAT=15 Signed-off-by: Courrier Guillaume <guillaume.courrier@cea.fr> Signed-off-by: Etienne AUJAMES <eaujames@ddn.com> Change-Id: I6e90131235f96255b636eea366ad0cef5f4f0b19 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/49236 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-13257 llite: Disallow users to set/clear group lock flag Group locks are created/freed via dedicated ioctls. Disallow manually setting or clearing the flag. HPE-bug-id: LUS-12078 Signed-off-by: Vitaly Fertman <vitaly.fertman@hpe.com> Signed-off-by: Matt Ezell <ezellma@ornl.gov> Change-Id: Id5022cc02a7bdce2f0150592470e8336b4537a61 Reviewed-on: https://es-gerrit.hpc.amslabs.hpecorp.net/162708 Reviewed-by: Andriy Skulysh <andriy.skulysh@hpe.com> Reviewed-by: Vitaly Fertman <vitaly.fertman@hpe.com> Tested-by: Alexander Lezhoev <alexander.lezhoev@hpe.com> Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53782 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
LU-17453 llite: use dget_parent to access dentry.d_parent Use dget_parent() to aquire the d_parent member of a dentry to ensure dentry is valid while it is accessed. HPE-bug-id: LUS-11889 Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com> Change-Id: Icb0a25ece5a3a3d50da076708fcd631176652a1b Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53757 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Neil Brown <neilb@suse.de> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-17081 build: compatibility for 6.5 kernels Linux commit v6.4-rc2-29-gc6585011bc1d splice: Remove generic_file_splice_read() Prefer filemap_splice_read and provide alternates for older kernels. Linux commit v6.4-rc2-30-g3fc40265ae2b iov_iter: Kill ITER_PIPE ITER_PIPE and iov_iter_is_pipe() are removed, provide a replacement for iov_iter_is_pipe Linux commit v6.4-rc4-53-g54d020692b34 mm/gup: remove unused vmas parameter from get_user_pages() Use vma_lookup() to acquire the vma following get_user_pages() Linux commit v6.4-rc7-1884-gdc97391e6610 sock: Remove ->sendpage*() in favour of sendmsg(MSG_SPLICE_PAGES) Use sendmsg when MSG_SPLICE_PAGES is defined. Provide a wrapper using sendpage() for older kernels. HPE-bug-id: LUS-11811 Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com> Change-Id: I95a0954a602c8db08d30b38a50dcd50107c8f268 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52258 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Jian Yu <yujian@whamcloud.com> Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com> Reviewed-by: xinliang <xinliang.liu@linaro.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-15367 llite: add iotrace to open/release Add iotrace to open and release operations. Test-Parameters: trivial Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com> Change-Id: Idc20a05417398af20dee313531a3573a8aa4e4c0 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52006 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
LU-17469 llite: hold object reference in IO There could be a race between page write and inode free, hold a cl_object reference during the IO lest accessing freed object. Signed-off-by: Bobi Jam <bobijam@whamcloud.com> Change-Id: Ic70cc27430e68265aba0662fc68e9bfe2f86cfe1 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53819 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-13802 llite: add hybrid IO SBI flag Add an SBI flag so hybrid IO can be fully disabled. Test-Parameters: trivial Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com> Signed-off-by: Qian Yingjin <qian@ddn.com> Change-Id: I2825b4cf261f98d71a18cd66d6fe3632dfabc37a Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52592 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
LU-13802 llite: tag switched hybrid IOs If we switched IO type with hybrid IO, tag the IO in the cl_io. This will be used to make various choices later. Also add a more verbose debug message for DIO, printing various aspects of the IO. Test-Parameters: trivial Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com> Change-Id: I347ef059eadcd9fd3767d7defc2e3da0eeb5573b Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52703 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-13802 llite: trivial bio_dio switch check This adds a trivial version of the DIO BIO switch checking function which doesn't ever switch. This creates the basic check function which we'll add to in future patches. Test-Parameters: trivial Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com> Signed-off-by: Qian Yingjin <qian@ddn.com> Change-Id: Ia01df8d0f33246d3833c5327bcb1a07ac305492b Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52586 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
LU-13802 llite: refactor ll_file_io_generic decs The variable declarations in ll_file_io_generic are in no order at all. Put them in the standard order and convert a few 'unsigned int' to bool. Test-Parameters: trivial Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com> Change-Id: I0b808ab82bdc129853dd4f27b93b3c91b201ca8a Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52587 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Sebastien Buisson <sbuisson@ddn.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
LU-9839 clio: lov active ios accounting fix ASSERT(atomic_read(&lov->lo_active_ios)==0) is triggered due to a bug in active_ios accounting. For some cl_io_init(,CIT_MISC,,) calls increment the lov_active_ios counter is not protected by the layout lock. So the checks for active_ios != 0 are racy and not preventing another thread from starting new cl_io and incrementing the active_ios counter after any check but before the assertion. The lov_active_ios counter increment should be done under the same condition as taking the layout type lock. The ci_type=CIT_MISC and ci_ignore_layout=1 should not be used in ll_dom_finish_open() as the I/O doesn't come "from the osc layer" and may race with a layout change. HPE-bug-id: LUS-11628 Signed-off-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com> Change-Id: I35fda85b968b847a87e73dd36bbb1648c744d62c Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51638 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Mikhail Pershin <mpershin@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com> Reviewed-by: Vitaly Fertman <vitaly.fertman@hpe.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-17413 llite: protect check in ll_merge_md_attr() striping can apply in a concurrent process, so the check for striping should be serialized against any concurrent process. Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com> Change-Id: Iffac2f1f9b53abc26705d70a30c2201b48156ac8 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53639 Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com>
LU-16314 llite: Migrate LASSERTF %p to %px This change covers lustre/ec through lustre/mgs and converts LASSERTF statements to explicitly use %px. Use %px to explicitly report the non-hashed pointer value messages printed when a kernel panic is imminent. When analyzing a crash dump the associated kernel address can be used to determine the system state that lead to the system crash. As crash dumps can and are provided by customers from production systems the use of the kernel command line parameter: no_hash_pointers is not always possible. Ref: Documentation/core-api/printk-formats.rst Test-Parameters: trivial HPE-bug-id: LUS-10945 Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com> Change-Id: I708d9ef60c63f5b4006c7986599a2f39fc9e5fdf Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51213 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-14361 statahead: regularized fname statahead pattern Some applications do stat() calls under a directory within which all children files have regularized file name: - mdtest benchmark tool: mdtest.$rank.$i - ML/AI with ingested data that have typically a format rule of the filename in the directory. The most common format for regularized file name is that the suffix part of the file name is number-indexing. However, in the current statahead mechanism, the statahead is populated by the order of the hash of the file name via readdir() calls, not a kind of sorting order. In this patch, we improve the statahead to prefetch attributes for the files with regularized indexing file name via asynchronous batching RPC. This patch adds the support to do statahead for these kinds of applications, which can be optimized, but without opendir()/ close() to start/stop statahead thread explicitly. Instead, the statahead thread will stop and quit when found that there was no acitivy for more than a certain time period (i.e. 30 seconds). Test-Parameters: mdtcount=4 mdscount=2 testlist=sanity env=ONLY=27p,ONLY_REPEAT=5 Test-Parameters: mdtcount=4 mdscount=2 testlist=sanity env=ONLY=27p,ONLY_REPEAT=5 Test-Parameters: mdtcount=4 mdscount=2 testlist=sanity env=ONLY=123f,ONLY_REPEAT=10 Test-Parameters: mdtcount=4 mdscount=2 testlist=sanity env=ONLY=123f,ONLY_REPEAT=10 Signed-off-by: Qian Yingjin <qian@ddn.com> Change-Id: Ide11ec5a651ae74884ddbe1cecede4f5c961e38d Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/41308 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16695 llite: switch to ki_flags from f_flags There are possible races between IO checking f_flags and fcntl changing f_flags. The kernel fixed most of these by copying most of the file flags in to the iocb. Let's follow on and use those copied flags. This also lets us change them if we want, since they're now local to the specific IO. Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com> Signed-off-by: Guillaume Courrier <guillaume.courrier@cea.fr> Change-Id: Ib98cccec0e7888865ec10dc5f76f1d9917a1aef7 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50493 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com> Reviewed-by: Etienne AUJAMES <eaujames@ddn.com>
LU-16837 lustre: avoid the same member name There are several structures using the same member name, such as cl_ladvise_io::li_flags, layout_intent::li_flags and lfsck_instance::li_flags, and this makes it hard to find where it is used. This patch renames some structures member prefix to avoid the homonyms. Test-Parameters: trivial Signed-off-by: Bobi Jam <bobijam@whamcloud.com> Change-Id: Ie592afa06dd0abf0c1110843e5d8007a91c68145 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51766 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-13805 llite: Implement unaligned DIO connect flag Unupgraded ZFS servers may crash if they received unaligned DIO, so we need a compat flag and a test to recognize those servers. This patch implements that logic. Fixes: 7194eb6431 ("LU-13805 clio: bounce buffer for unaligned DIO") Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com> Change-Id: I5d6ee3fa5dca989c671417f35a981767ee55d6e2 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51126 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Sebastien Buisson <sbuisson@ddn.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
LU-16468 llite: protect layout before read IO going It's possible that the before the read IO, file_read_confine_iter() ->lov_attr_get() to get proper kms (known minimum size of the file), and lov_attr_get() presumes that it's called under ongoing IO, which protected the layout from changing, while it's not in this case. Signed-off-by: Bobi Jam <bobijam@whamcloud.com> Change-Id: I1b36ec6e158331e63e8026ee2b986d5a7e3cb6dc Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/49622 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Mikhail Pershin <mpershin@whamcloud.com> Reviewed-by: Yang Sheng <ys@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>