LU-17572 lov: remove noise from lov_init_sub() lov_init_sub() generates too many messages in applications like racer. let's make it a bit less noisy. Test-Parameters: trivial Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com> Change-Id: I00ae75597b550c29122d8fb9d34d4e0d24c38dd5 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/54125 Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com>
LU-10499 pcc: add readonly mode for PCC Readonly Persistent Client Cache (RO-PCC) shares the same framework with Readwrite Persistent Client Cache, expect that no HSM mechanism is used in readonly mode of PCC. Instead, RO-PCC adds a new flag field in the file object's layout named LCM_FL_PCC_RDONLY to indicate that the file is in PCC read-only state. It is protected under the layout lock. After introducing the readonly feature for the layout, the IO path has some changes. For read, if the file has been valid RO-PCC cached, the file data can be read from PCC directly; Otherwise, it will read data using normal I/O path from OSTs. For data modifying operations (write or truncate), it must clear the readonly flag of the layout on MDT (which will invaliate the RO-PCC cached state on clients via layout lock blocking callback), and then it can perform I/O. For RO-PCC, as the PCC cached file is actual a replication of Lustre file, when data read on PCC failed, it can tolerate this error by falling back to normal read path: read data from OSTs. Refer to paper (LPCC: hierarchical persistent client caching for Lustre) for more design details. Test-Parameters: clientcount=3 testlist=sanity-pcc,sanity-pcc,sanity-pcc Signed-off-by: Qian Yingjin <qian@ddn.com> Change-Id: I6badd72e00a106a0f68950621ce6f82471731a95 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/38305 Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com>
LU-16958 llite: migrate deadlock on not responding lock cancel lfs migrate race makes MDS hang with following backtrace [ 3683.248584] [<0>] ldlm_completion_ast+0x78d/0x8e0 [ptlrpc] [ 3683.250122] [<0>] ldlm_cli_enqueue_local+0x2fd/0x840 [ptlrpc] [ 3683.251363] [<0>] mdt_object_local_lock+0x50e/0xb10 [mdt] [ 3683.252615] [<0>] mdt_object_lock_internal+0x187/0x430 [mdt] [ 3683.253793] [<0>] mdt_object_lock_try+0x22/0xa0 [mdt] [ 3683.254857] [<0>] mdt_getattr_name_lock+0x1317/0x1dc0 [mdt] [ 3683.256016] [<0>] mdt_intent_getattr+0x264/0x440 [mdt] [ 3683.257105] [<0>] mdt_intent_opc+0x452/0xa80 [mdt] [ 3683.258126] [<0>] mdt_intent_policy+0x1fd/0x390 [mdt] [ 3683.259191] [<0>] ldlm_lock_enqueue+0x469/0xa90 [ptlrpc] [ 3683.260350] [<0>] ldlm_handle_enqueue0+0x61a/0x16c0 [ptlrpc] [ 3683.261596] [<0>] tgt_enqueue+0xa4/0x200 [ptlrpc] [ 3683.262662] [<0>] tgt_request_handle+0xc9c/0x1a40 [ptlrpc] [ 3683.263860] [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc] [ 3683.265220] [<0>] ptlrpc_main+0xbf3/0x1540 [ptlrpc] [ 3683.266303] [<0>] kthread+0x134/0x150 [ 3683.267111] [<0>] ret_from_fork+0x35/0x40 The deadlock happens as follows: T1: vvp_io_init() ->ll_layout_refresh() <= take lli_layout_mutex ->ll_layout_intent() ->ll_take_md_lock() <= take the CR layout lock ref ->ll_layout_conf() ->vvp_prune() ->vvp_inode_ops() <= release lli_layout_mtex ->vvp_inode_ops() <= try to acquire lli_layout_mutex -> racer wait here for T2 T2: ->ll_file_write_iter() ->vvp_io_init() ->ll_layout_refresh() <= take lli_layout_mutex ->ll_layout_intent() <= Request layout from MDT -> racer wait from server... And server want to cancel the CR layout lock T1 hold, and it won't happen. Also T1 could has take extent ldlm lock while waiting lli_layout_mutex hold by T2, and ofd_destroy_hdl does not get the lock cancellation response from T1. lli_layout_mutex is only needed for enqueuing layout lock from server, so that ll_layout_conf() does not involve with lli_layout_mutex. Fixes: 8f2c1592c3 ("LU-16958 llite: migrate vs regular ops deadlock") Signed-off-by: Bobi Jam <bobijam@whamcloud.com> Change-Id: Ib94de2c63544c3a962199aa0537418255980ae8c Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52388 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Qian Yingjin <qian@ddn.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-4974 lod: Change pool_desc to "[lod|lov]_pool_desc" This patch changes 'struct pool_desc' under lov and lod to 'lov_pool_struct' and 'lod_pool_struct' respectively. This is the first step to check if there is anything common and can be unify. Although both layer uses 'struct pool_desc' to define the pool_desc struct respectively. 'struct pool_desc' under lod has changed and grown. Therefore to remove ambiguity, prefix lod/lov is added to pool_desc struct for respective layer. This patch also adds function description wherever applicable This patch also changes space/tabs reported by checkpatch Signed-off-by: Andreas Dilger <adilger@whamcloud.com> Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Change-Id: I3fee3f2e9c321145779d9177a8e4582d123f1e8d Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/11114 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Timothy Day <timday@amazon.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16837 lov: NULL dereference in lov_delete_composite commit 14ed4a6f8f retroduced the issue fixed by commit 5da049d9ef ("LU-14389 lov: avoid NULL dereference in cleanup), this patch makes the fix cover the new case added by 14ed4a6f8f. Fixes: 14ed4a6f8f ("LU-16837 llite: handle unknown layout component") Signed-off-by: Bobi Jam <bobijam@whamcloud.com> Change-Id: I4a2b72e21139b60519ed523b4851723c91f523c1 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52826 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Vitaliy Kuznetsov <vkuznetsov@ddn.com>
LU-17110 llite: fix slab corruption with fm_extent_count=0 If userspace uses fiemap with .fm_extent_count=0, .fm_extents[0] is not allocated. Writing on the first entry without checking the extent count could lead to memory corruption (slab). This patch fix also the case when osc is disable: FIEMAP_EXTENT_LAST should be set on the extent (fe_flags) and not on the fiemap struct. Add a regression test sanityn 71d to test fiemap with fm_extent_count=0. Add a regression test sanity-hsm 408 to test fiemap on release files. Fixes: 4097196 ("LU-11848 lov: FIEMAP support for PFL and FLR file") Test-Parameters:testlist=sanityn Test-Parameters:testlist=sanityn env=ONLY=71d,ONLY_REPEAT=20 Test-Parameters:testlist=sanity-hsm env=ONLY=408,ONLY_REPEAT=20 Signed-off-by: Etienne AUJAMES <etienne.aujames@cea.fr> Change-Id: Id63c6973540187e678020977f2d555dfcbf3c634 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52352 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Feng Lei <flei@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-17088 dom: don't create different size DOM component Multiple DOM components are allowed in diffrent mirror but they must be of the same size, mirror extend should check this restraint. Fix another glitch in lov_init_composite() where dom_size is used as a __u64 value but declared as boolean. Fixes: 44a721b8c1 ("LU-11421 dom: manual OST-to-DOM migration via mirroring") Signed-off-by: Bobi Jam <bobijam@whamcloud.com> Change-Id: Ia0d08c697dbeeb3aa8d20d9849226afa06360012 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52269 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Mikhail Pershin <mpershin@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16837 llite: handle unknown layout component If lustre client encounters unknown layout component pattern in a mirror file, this patch makes client mark this mirror as invalid and skip it. Signed-off-by: Bobi Jam <bobijam@whamcloud.com> Change-Id: Ie5f44212ab96bdc706cc5a9e11f330234fc01069 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51060 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Vitaliy Kuznetsov <vkuznetsov@ddn.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-17013 lov: fill FIEMAP_EXTENT_LAST flag If file has N extents and get the fiemap with exactly N extent slots, the last extent will miss FIEMAP_EXTENT_LAST flag. Fix it. Signed-off-by: Lei Feng <flei@whamcloud.com> Test-Parameters: testlist=sanityn env=ONLY=71a+71b+71c Change-Id: I4556b31f0d04bdf8e83f323e83b871b093beaa5e Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51863 Tested-by: Maloo <maloo@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Zhenyu Xu <bobijam@hotmail.com>
LU-17000 misc: remove Coverity annotations These Coverity function annotations were added around 10 years ago. Since then, Coverity seems to produce less false positives. Out of about 20 annotations, only 3 warnings get surpressed. Thus, the applicability of these annotations should be re-evaluated. Coverity has more advanced tools now for reducing false positives. Various Lustre functions and macros could be modeled rather than using function annotations. But first, we need to get a good idea of what kinds of false postives are being generated. https://scan.coverity.com/tune Test-Parameters: trivial Signed-off-by: Timothy Day <timday@amazon.com> Change-Id: Ibcb9cf55574675e20b13a4f7a1b9142a3b75e262 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51793 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16770 llite: prune object without layout lock first lov_layout_change() calls cl_object_prune() before changing layout. It may lead to eviction from MDT in case slow responce from OST. To reduce risk of possible eviction call cl_object_prune() without layout lock held before calling lov_layout_change() vvp_prune() attempts to sync and truncate page cache pages. osc_page_delete() may encounter page cache pages in non-clean state during truncate because there's a race window between sync and truncate. Writes may stick into this window and generate dirty or writeback pages. This window is usually protected with a special truncate semaphore e.g. when truncate is requested from the truncate syscall. Let's use this semaphore to avoid write vs truncate race in vvp_prune(). Change-Id: Ie2ee29ea1e792e1b34b6de068ff2b84fd8f52f2a HPE-bug-id: LUS-9927, LUS-11612 Signed-off-by: Andriy Skulysh <andriy.skulysh@hpe.com> Reviewed-by: Vitaly Fertman <c17818@cray.com> Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com> Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50742 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andrew Perepechko <andrew.perepechko@hpe.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16958 llite: migrate vs regular ops deadlock When it need to lock inode in lov_conf_set(), it could have hold inode's lli_layout_mutex, we need unlock the layout mutex before taking its inode lock to keep the lock order. Fixes: 51d62f2122f ("LU-16637 llite: call truncate_inode_pages() in inode lock") Signed-off-by: Bobi Jam <bobijam@whamcloud.com> Change-Id: I7ee58039a6d31daefc625ac571a52baf112f8151 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51641 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-14301 client: use EOPNOTSUPP instead of ENOTSUPP Don't return NFS-specific error code ENOTSUPP back to userspace, instead use EOPNOTSUPP. ENOTSUPP does not print a useful error message from strerror() if it is hit by an application. Signed-off-by: Andreas Dilger <adilger@whamcloud.com> Change-Id: Iabd07b31069737e8ee7ca2382fd8cff6143ebbe5 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51511 Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Neil Brown <neilb@suse.de> Reviewed-by: jsimmons <jsimmons@infradead.org> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com>
LU-16637 llite: call truncate_inode_pages() in inode lock In some cases vvp_prune()->truncate_inode_pages() is get called without IO context, we need protect it with inode lock as well. So we add ll_inode_info::lli_inode_lock_owner and set it according to vfs lock rules (Documentation/filesystems/Locking or Documentation/filesystems/locking.rst), so before calling truncate_inode_pages(), we'd lock the inode if it's not locked in vfs. Fixes: ef9be34478 ("LU-16637 llite: call truncate_inode_pages() under inode lock") Signed-off-by: Bobi Jam <bobijam@whamcloud.com> Change-Id: I84d7d999a49325810062a9a7337e184d35467820 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50857 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Neil Brown <neilb@suse.de> Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-12610 llite: remove OBD_ -> CFS_ macros Remove OBD macros that are simply redefinitions of CFS macros. Signed-off-by: Timothy Day <timday@amazon.com> Signed-off-by: Ben Evans <beevans@whamcloud.com> Change-Id: I7bbcc3e1fda6418c258eb4d1c52b929a7cf72ed1 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50804 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Reviewed-by: Neil Brown <neilb@suse.de> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-15300 mdt: refresh LOVEA with LL granted this change tries to fix two problems: 1) mdt_reint_open() fetches LOVEA before layout lock is taken. this may race with another process changing the layout and may result in a stale layout returned with a granted layout lock - re-fetch LOVEA once layout lock is granted 2) lov_layout_change() should not apply old layouts which can get through when MDS doesn't take layout lock 3) LFSCK shouldn't ignore layout version stored on MDS to avoid a situation when version degrades compared to client's copy. This patch misses an optimization and can result in a number of useless calls to OSD to fetch LOVEA. To be fixed in a followup patch. Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com> Change-Id: Idee1101d152ab09947faf6d75574a8761a7690a5 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/46413 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Zhenyu Xu <bobijam@hotmail.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16518 misc: use fixed hash code There is a configure check to avoid using broken hashing code. All calls to 'hash_*' are replace by the 'cfs_hash_*' equivalents, to make use of this check. Two functions which hash then apply a mask are removed. The calls are replaced with 'cfs_hash_*' and manually applying the mask. Signed-off-by: Timothy Day <timday@amazon.com> Change-Id: Ia4b27cb6fb1329b9df45c00f748a8d22178b0654 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/49916 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: jsimmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
LU-16480 lov: fiemap improperly handles fm_extent_count=0 FIEMAP calls with fm_extent_count=0 are supposed only to return the number of extents. lov_object_fiemap() attempts to initialize stripe_last based on fiemap->fm_extents[0] which is not initialized in userspace and not even allocated in kernelspace. Eventually, the call exits with -EINVAL and "FIEMAP does not init start entry" kernel log message. Fixes: 409719608c ("LU-11848 lov: FIEMAP support for PFL and FLR file") Signed-off-by: Andrew Perepechko <andrew.perepechko@hpe.com> Change-Id: I65e706b5dd5c8a6db90a539c2602af839b4da823 HPE-bug-id: LUS-11443 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/49645 Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com>
LU-16025 llite: adjust read count as file got truncated File read will not notice the file size truncate by another node, and continue to read 0 filled pages beyond the new file size. This patch add a confinement in the read to prevent the issue and add a test case verifying the fix. Signed-off-by: Bobi Jam <bobijam@whamcloud.com> Change-Id: Ie51ba09201a1ca1464c3a3892d367590e978ee34 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/47896 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Patrick Farrell <farr0186@gmail.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
LU-15829 llite: don't use a kms if it invalid. Lockless DIO don't update a KMS as other IO type does, it caused a situation when next read don't known a real file size to be read. Lets avoid using an invalid KMS. Fixes: 6bce5367 (LU-4198 clio: turn on lockless for some kind of IO) Signed-off-by: Alexey Lyashkov <alexey.lyashkov@hpe.com> Change-Id: Ie71d3f3cc24fc16c03ed07f9f5a3a17c7fdfa684 Reviewed-on: https://review.whamcloud.com/47395 Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Bobi Jam <bobijam@hotmail.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>