LU-10499 pcc: add readonly mode for PCC Readonly Persistent Client Cache (RO-PCC) shares the same framework with Readwrite Persistent Client Cache, expect that no HSM mechanism is used in readonly mode of PCC. Instead, RO-PCC adds a new flag field in the file object's layout named LCM_FL_PCC_RDONLY to indicate that the file is in PCC read-only state. It is protected under the layout lock. After introducing the readonly feature for the layout, the IO path has some changes. For read, if the file has been valid RO-PCC cached, the file data can be read from PCC directly; Otherwise, it will read data using normal I/O path from OSTs. For data modifying operations (write or truncate), it must clear the readonly flag of the layout on MDT (which will invaliate the RO-PCC cached state on clients via layout lock blocking callback), and then it can perform I/O. For RO-PCC, as the PCC cached file is actual a replication of Lustre file, when data read on PCC failed, it can tolerate this error by falling back to normal read path: read data from OSTs. Refer to paper (LPCC: hierarchical persistent client caching for Lustre) for more design details. Test-Parameters: clientcount=3 testlist=sanity-pcc,sanity-pcc,sanity-pcc Signed-off-by: Qian Yingjin <qian@ddn.com> Change-Id: I6badd72e00a106a0f68950621ce6f82471731a95 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/38305 Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com>
LU-17464 lod: set llc_ostlist to NULL after free Default LOV striping could free component entry llc_ostlist if needed e.g. expand component entries, without set it to NULL it could be double allocated/freed later. Signed-off-by: Bobi Jam <bobijam@whamcloud.com> Change-Id: I25824cb61dd47ba284403039259593b88d25fa9d Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53797 Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Qian Yingjin <qian@ddn.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com>
LU-17261 lov: ignore broken components if some component of a mirrored file is broken, it makes sense to try another (possible valid) replica rather than give up immediately. Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com> Change-Id: I32ea0efa90109f5159bf8b6c4e0efe1d543580c3 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52996 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Zhenyu Xu <bobijam@hotmail.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-8191 lustre: convert lmv,lod,lov functions to static Static analysis shows that a number of functions could be made static. This patch declares several functions in lmv, lod, and lov static. Also, remove one unused function: lov_lsm_entry() Another function is intentionally unused for debugging purposes. It was detected by static analysis, but it has been left untouched. Test-Parameters: trivial Signed-off-by: Timothy Day <timday@amazon.com> Change-Id: If409226ea201587c7f95d4a65ffaef72671b5ac2 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51479 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16518 lod: fix new clang error in lod_lov.c The variable hsmsize was uninitialized in some situations. By moving the initialization earlier, we can avoid this. Fixes: aebb405e32e ("LU-10499 pcc: use foreign layout for PCCRO") Signed-off-by: Timothy Day <timday@amazon.com> Change-Id: I3385e3349ad00d037b8b94337cb3352623d0a40a Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52566 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Qian Yingjin <qian@ddn.com> Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
LU-17204 lod: don't panic on short LOVEA when we request LOVEA and find the existing buffer is not enough, we ask for LOVEA's size and reallocate the buffer. but LOVEA can shrink in parallel (e.g. new default striping), so our expectation that the size must be greater than size of the existing buffer is not correct. replace the corresponding assertion with a simple repeat + extra check for a livelock. Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com> Change-Id: I26ad5091228bf78858f8538478dbcbdb235cddf4 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52727 Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Mikhail Pershin <mpershin@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com>
LU-4974 lod: Change pool_desc to "[lod|lov]_pool_desc" This patch changes 'struct pool_desc' under lov and lod to 'lov_pool_struct' and 'lod_pool_struct' respectively. This is the first step to check if there is anything common and can be unify. Although both layer uses 'struct pool_desc' to define the pool_desc struct respectively. 'struct pool_desc' under lod has changed and grown. Therefore to remove ambiguity, prefix lod/lov is added to pool_desc struct for respective layer. This patch also adds function description wherever applicable This patch also changes space/tabs reported by checkpatch Signed-off-by: Andreas Dilger <adilger@whamcloud.com> Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Change-Id: I3fee3f2e9c321145779d9177a8e4582d123f1e8d Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/11114 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Timothy Day <timday@amazon.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-17084 lod: fix comparision in lod_striping_load() in if (rc > sizeof(struct lmv_foreign_md)) the latter is unsigned and gcc treats rc (which is defined as int and can be negative to encode an error) as unsigned. this way -EIO becomes greater than the size of the structure. make sizeof() signed to avoid confusion. Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com> Change-Id: Ie6735578649e397ed05b6951fab941f97051305b Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52265 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: xinliang <xinliang.liu@linaro.org> Reviewed-by: Mikhail Pershin <mpershin@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Neil Brown <neilb@suse.de>
LU-16194 lod: define negative extent offset as invalid If lu_extent.e_start/e_end is negative after converting to s64, regard it as invalid data except -1 (EOF). Signed-off-by: Lei Feng <flei@whamcloud.com> Change-Id: I79276a5185f339e9de48fe87c4da39052c7974e1 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/48684 Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com>
LU-10499 pcc: use foreign layout for PCCRO on server side This patch includes the codes about using foreign layout for PCCRO on the server side (LOD|MDD|MDT layers). Signed-off-by: Qian Yingjin <qian@ddn.com> Change-Id: I48467be9fef54bd05432528b685241aa53978d24 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51375 Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com>
LU-17088 dom: don't create different size DOM component Multiple DOM components are allowed in diffrent mirror but they must be of the same size, mirror extend should check this restraint. Fix another glitch in lov_init_composite() where dom_size is used as a __u64 value but declared as boolean. Fixes: 44a721b8c1 ("LU-11421 dom: manual OST-to-DOM migration via mirroring") Signed-off-by: Bobi Jam <bobijam@whamcloud.com> Change-Id: Ia0d08c697dbeeb3aa8d20d9849226afa06360012 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52269 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Mikhail Pershin <mpershin@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-10026 csdc: DoM pattern could be a combined value DoM pattern is LOV_PATTERN_MDT for now, and in the future it could be combined with LOV_PATTERN_COMPRESS to represent a compressed DoM component. Fix a minor glitch for lov_getstripe_old code path (in ll_lov_getstripe_ea_info), which intends to return the last component stripe info but the commit abf04e7ea3 omits to correctly set the last component stripe info before using it. Fixes: abf04e7ea3 ("LU-14337 lov: return valid stripe_count/size for PFL files") Signed-off-by: Bobi Jam <bobijam@whamcloud.com> Change-Id: Id0779c30c004b6979f88bf96b7b7b74a8b8c26e4 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51978 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
LU-17000 misc: remove Coverity annotations These Coverity function annotations were added around 10 years ago. Since then, Coverity seems to produce less false positives. Out of about 20 annotations, only 3 warnings get surpressed. Thus, the applicability of these annotations should be re-evaluated. Coverity has more advanced tools now for reducing false positives. Various Lustre functions and macros could be modeled rather than using function annotations. But first, we need to get a good idea of what kinds of false postives are being generated. https://scan.coverity.com/tune Test-Parameters: trivial Signed-off-by: Timothy Day <timday@amazon.com> Change-Id: Ibcb9cf55574675e20b13a4f7a1b9142a3b75e262 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51793 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-15671 mds: do not send OST_CREATE transno interop Send OST_CREATE RPCs from the MDS with no_resend and no_delay when communicating with an old OST that does not support the OBD_CONNECT2_REPLAY_RESEND. Likewise, the OST should not reply to the MDS RPC with rq_transno set, or this will trigger: osp_precreate_send() ASSERTION(req->rq_transno == 0) failed This can be avoided if the MDS is upgraded before the OSS, but will always be hit if OSS is upgraded first. After 2.20.53 the MDS/OSS assume that this is always true, since rolling upgrades are unsupported for larger version differences. Test-Parameters: testgroup=rolling-upgrade-oss Fixes: 63e17799a3 ("LU-8367 osp: enable replay for precreation request") Signed-off-by: Andreas Dilger <adilger@whamcloud.com> Signed-off-by: Sergey Cheremencev <scherementsev@ddn.com> Change-Id: I1ab601a2f55540dd75cf24838f7cdb7f823ed42c Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51056 Tested-by: Maloo <maloo@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-12610 llite: remove OBD_ -> CFS_ macros Remove OBD macros that are simply redefinitions of CFS macros. Signed-off-by: Timothy Day <timday@amazon.com> Signed-off-by: Ben Evans <beevans@whamcloud.com> Change-Id: I7bbcc3e1fda6418c258eb4d1c52b929a7cf72ed1 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50804 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Reviewed-by: Neil Brown <neilb@suse.de> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16135 lod: prohibit DoM pattern in plain layout DoM pattern can be set as default directory plain layout by older LFS version. It misses DoM component sanity checks if plain layout is used. Such layout is not allowed and causes later crashed when file is created under that directory. While LFS can prevent this but not in all Lustre versions, so LOD should do the check as well Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com> Change-Id: Ic58fdda2ab3e63083128cb6cf949fcb43ccd2c02 Reviewed-on: https://review.whamcloud.com/48433 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-15486 lod: mirroring a plain file in mirrored-layout dir If a file does not have a mirror in a directory with a default FLR mirror, then "lfs mirror extend" on the file fails with "cannot create volatile file: Invalid argument". This comes from the the non-striped file layout generated from LOD inheriting its FLR state from the default FLR while it contains no mirror in it, and lov_init_composite() will complain about it. if (equi(flr_state == LCM_FL_NONE, comp->lo_mirror_count > 1)) RETURN(-EINVAL); Signed-off-by: Bobi Jam <bobijam@whamcloud.com> Change-Id: I5e849acb2327ce735d0008271bfd48fa7293161c Reviewed-on: https://review.whamcloud.com/46517 Tested-by: jenkins <devops@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-15513 lod: iterate initialized stripe Fix an array out of bounds access. Do not iterate over the component objects in lod_fill_mirrors() to check non-rotational OSTs if the component is unintialized or it is a MDT or RELEASED component. Fixes: 591a990c61 ("LU-15513 lod: skip uninit component in lod_fill_mirrors") Signed-off-by: Bobi Jam <bobijam@whamcloud.com> Change-Id: I8530cd30fab750c47f15ee9278ec6d68871367cc Reviewed-on: https://review.whamcloud.com/47028 Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-15308 lod: set ldo_mirrors to NULL on error In lod_alloc_comp_entries() if OBD_ALLOC_LARGE fails and we free ldo_mirrors then we set it to NULL to avoid a double free later. Test-Parameters: trivial Signed-off-by: John L. Hammond <jhammond@whamcloud.com> Change-Id: I27552d1eaecc06720190db9cfe89ded71b9fef53 Reviewed-on: https://review.whamcloud.com/47008 Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
LU-6864 osp: manage number of modify RPCs in flight Currently we use a rpc_lock to ensure concurrent in-flight request are handled serially to prevent the execution status from being over written. This patch changes the osp component to send multiple modify RPCs in parallel to the MDT. This will improve metadata performance of cross-MDT operations. For testing replace mkdirmany with createmany -d which does the same thing. Signed-off-by: Gregoire Pichon <gregoire.pichon@bull.net> Signed-off-by: James Simmons <jsimmons@infradead.org> Change-Id: Icb601afabd6767463634a4c7943ec4206bc758ec Reviewed-on: https://review.whamcloud.com/14375 Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>