LU-10499 pcc: add readonly mode for PCC Readonly Persistent Client Cache (RO-PCC) shares the same framework with Readwrite Persistent Client Cache, expect that no HSM mechanism is used in readonly mode of PCC. Instead, RO-PCC adds a new flag field in the file object's layout named LCM_FL_PCC_RDONLY to indicate that the file is in PCC read-only state. It is protected under the layout lock. After introducing the readonly feature for the layout, the IO path has some changes. For read, if the file has been valid RO-PCC cached, the file data can be read from PCC directly; Otherwise, it will read data using normal I/O path from OSTs. For data modifying operations (write or truncate), it must clear the readonly flag of the layout on MDT (which will invaliate the RO-PCC cached state on clients via layout lock blocking callback), and then it can perform I/O. For RO-PCC, as the PCC cached file is actual a replication of Lustre file, when data read on PCC failed, it can tolerate this error by falling back to normal read path: read data from OSTs. Refer to paper (LPCC: hierarchical persistent client caching for Lustre) for more design details. Test-Parameters: clientcount=3 testlist=sanity-pcc,sanity-pcc,sanity-pcc Signed-off-by: Qian Yingjin <qian@ddn.com> Change-Id: I6badd72e00a106a0f68950621ce6f82471731a95 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/38305 Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com>
LU-16796 lov: Change struct pool_desc to use kref This patch changes struct pool_desc to use kref(refcount_t) instead of atomic_t Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Change-Id: I3829de190ec148c2e087f6a0262bf3bb76c196af Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52122 Reviewed-by: Neil Brown <neilb@suse.de> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com>
LU-4974 lod: Change pool_desc to "[lod|lov]_pool_desc" This patch changes 'struct pool_desc' under lov and lod to 'lov_pool_struct' and 'lod_pool_struct' respectively. This is the first step to check if there is anything common and can be unify. Although both layer uses 'struct pool_desc' to define the pool_desc struct respectively. 'struct pool_desc' under lod has changed and grown. Therefore to remove ambiguity, prefix lod/lov is added to pool_desc struct for respective layer. This patch also adds function description wherever applicable This patch also changes space/tabs reported by checkpatch Signed-off-by: Andreas Dilger <adilger@whamcloud.com> Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Change-Id: I3fee3f2e9c321145779d9177a8e4582d123f1e8d Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/11114 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Timothy Day <timday@amazon.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-6174 lov: use standard Linux 64 divison macros For the case of 64 bit platforms lov_do_div64() is basically the same as div_u64_rem(). For the 32 bit platform case lustre tries to work around the issue of the divisor being larger than 32 bits. We can also use div64_u64_rem() as well for 32 bit platforms since Linux supports that as well. With the move to Linux div64 operations we need to migrate fields related to 64 bit size fields. To pick the correct div64 function we need to select the proper sign value. The lov layer often uses loff_t, which is signed, for stripe sizes which doesn't make sense since sizes should always be positive. Creating a negative size by mistake will give an bizarre offset into a file. Avoid any potential issues with signed vs unsigned issues by replacing the loff_t use in this case with u64. Test-Parameters: testlist=large-lun env=ONLY=5,REFORMAT=yes Change-Id: Ieadc266d43d6be1d6d47ee14ba9ac0dab01e7d86 Signed-off-by: James Simmons <jsimmons@infradead.org> Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/39343 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Timothy Day <timday@amazon.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16837 llite: handle unknown layout component If lustre client encounters unknown layout component pattern in a mirror file, this patch makes client mark this mirror as invalid and skip it. Signed-off-by: Bobi Jam <bobijam@whamcloud.com> Change-Id: Ie5f44212ab96bdc706cc5a9e11f330234fc01069 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51060 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Vitaliy Kuznetsov <vkuznetsov@ddn.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-10026 csdc: DoM pattern could be a combined value DoM pattern is LOV_PATTERN_MDT for now, and in the future it could be combined with LOV_PATTERN_COMPRESS to represent a compressed DoM component. Fix a minor glitch for lov_getstripe_old code path (in ll_lov_getstripe_ea_info), which intends to return the last component stripe info but the commit abf04e7ea3 omits to correctly set the last component stripe info before using it. Fixes: abf04e7ea3 ("LU-14337 lov: return valid stripe_count/size for PFL files") Signed-off-by: Bobi Jam <bobijam@whamcloud.com> Change-Id: Id0779c30c004b6979f88bf96b7b7b74a8b8c26e4 Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51978 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
LU-15829 llite: don't use a kms if it invalid. Lockless DIO don't update a KMS as other IO type does, it caused a situation when next read don't known a real file size to be read. Lets avoid using an invalid KMS. Fixes: 6bce5367 (LU-4198 clio: turn on lockless for some kind of IO) Signed-off-by: Alexey Lyashkov <alexey.lyashkov@hpe.com> Change-Id: Ie71d3f3cc24fc16c03ed07f9f5a3a17c7fdfa684 Reviewed-on: https://review.whamcloud.com/47395 Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Bobi Jam <bobijam@hotmail.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-14487 modules: remove references to Sun Trademark. "lustre" is no longer a Trademark of Sun Microsystems. There is no need to acknowledge the trademark in every file, so just remove all these claims. Test-Parameters: trivial Signed-off-by: Mr NeilBrown <neilb@suse.de> Change-Id: I66941494eabc54bedf85079c5b85701187f2a8f1 Reviewed-on: https://review.whamcloud.com/42139 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Aurelien Degremont <degremoa@amazon.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
LU-6142 lustre: remove some unnecessary code. The field 'targets_proc_entry' is never set, so remove it and all reference to it. lov_dump_pool() is never used, so remove it. min_u64() is never used - remove it. max_u64() is used, but 'min3()' from linux/kernel.h makes the code clearer, so use that instead. Test-Parameters: trivial Signed-off-by: Mr NeilBrown <neilb@suse.de> Change-Id: I3ed6abc1b23e611bd4a573fa78981955a494c344 Reviewed-on: https://review.whamcloud.com/39739 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-14291 build: use tgt_pool for lov layer New general code was created for target pool handling. We can use this new code with the lov layer. Place this tgt_pool.c in the obdclass instead of having a special target directory just to build this code for the client. Change-Id: I05542c1d654d79647f5e0853bb1d587ff265fdf9 Signed-off-by: James Simmons <uja.ornl@yahoo.com> Reviewed-on: https://review.whamcloud.com/39683 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Sergey Cheremencev <sergey.cheremencev@hpe.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-6142 lov: chnage lsm_op_find() to a non-inline function. Making lsm_op_find() non-inline improves encapsulation and means that various _ops structures can now be static. Test-Parameters: trivial Signed-off-by: Mr NeilBrown <neilb@suse.de> Change-Id: Ifee2ab4f023e56bfa1e663a0106590cf7858856b Reviewed-on: https://review.whamcloud.com/39377 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com> Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-13602 flr: skip unknown FLR component types Currently, in lov_init_composite() it will quit with error when reading an unknown LOV pattern. Since FLR will be used for upcoming new features, like PCC-RO, an old client should be able to read the old format of the component types, ignore and skip the new types of FLR component. Signed-off-by: Qian Yingjin <qian@ddn.com> Change-Id: Ica3fe98203d44b52cf25b085c34c83b1a4702464 Reviewed-on: https://review.whamcloud.com/39513 Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-14128 lov: correctly set OST obj size When extends a PFL file to a size locating at a boundary of a stripe in a component, the truncate won't set the size of the OST object in the prior stripe. This patch record the prior stripe in lov_layout_raid0::lo_trunc_stripeno and add the stripe in the truncate IO and enqueue the lock covering it. Signed-off-by: Bobi Jam <bobijam@whamcloud.com> Change-Id: Ic5d8e3c16f950003736cd6dbd5af404613f818c7 Reviewed-on: https://review.whamcloud.com/40581 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Mike Pershin <mpershin@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-6142 lov: discard unused lov_dump_lmm* functions lov_dump_lmm() and lov_dump_lmm_v3() are never used, so remove them. So mark lov_lsm_pack_v1v3() and lov_lsm_pack_foreign() as static. Test-Parameters: trivial Signed-off-by: Mr NeilBrown <neilb@suse.de> Change-Id: I013c265819e7e139a9878e9e0aa18022519bd2fe Reviewed-on: https://review.whamcloud.com/39378 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com> Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-13359 quota: call rhashtable_lookup near params decl rhashtable_lookup() is an inline function which depends - for performancs - on the 'rhashtable_params' being visible and consnt. So it should only be called in the same file that declared the params. A recent patch make pools_hash_params an external variable and calls rhashtable_lookup from a separate file, which will break the optimisation. So add lov_pool_find() and use it to maintainer optimization. Fixes: 6b9f849fd5f4 ("LU-13359 quota: make used for pool correct") Signed-off-by: Mr NeilBrown <neilb@suse.de> Change-Id: Ieeb3b080491f5b2c9c825885fe7a42f4a8599a2a Reviewed-on: https://review.whamcloud.com/39676 Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Sergey Cheremencev <sergey.cheremencev@hpe.com> Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-13359 quota: make used for pool correct Before this patch used space for quota pool was a sum of a space used by user at all OSTs in a system. Now it is fixed and lfs quota --pool takes into account only OSTs form the pool. With option -v it also shows only OSTs from the pool. Change-Id: Idf1c8ed66fca7caec70246ea4182df883bcef23c Signed-off-by: Sergey Cheremencev <sergey.cheremencev@hpe.com> Reviewed-on: https://review.whamcloud.com/39298 Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com> Tested-by: jenkins <devops@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-6142 lov: convert container_of0() calls. Most calls to container_of0() in lustre/lov/ are preceded by an LINVRNT() which assures us that the pointer is valid, so container_of() can be used instead. Only in lov2obd() is there not context, so that call is changed to use container_of_safe() Signed-off-by: Mr NeilBrown <neilb@suse.de> Change-Id: I8b3ccc8f0ac1a32122f043e8feec078fcfe2452b Reviewed-on: https://review.whamcloud.com/38382 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Arshad Hussain <arshad.super@gmail.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-12580 lov: fix typo in lov_comp_md_size If the component of a PFL file is not initialized, we should use 0 as the stripe size when calculating the LOVEA size. Change-Id: I4ff5f4a78bc1d432cc1ac6fa3733461bd6b762e6 Fixes: 62f64a1077 ("LU-9489 lod: keep minimum LOVEA size") Signed-off-by: Li Dongyang <dongyangli@ddn.com> Reviewed-on: https://review.whamcloud.com/37493 Reviewed-by: Jian Yu <yujian@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Reviewed-by: Bobi Jam <bobijam@hotmail.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-8130 lov: convert lo[v|d]_pool to use rhashtable The pools hashtable can be implemented using the rhashtable implementation in lib. This has the benefit that lookups are lock-free. We need to use kfree_rcu() to free a pool so that a lookup racing with a deletion will not access freed memory. rhashtable has no combined lookup-and-delete interface, but as the lookup is lockless and the chains are short, this brings little cost. Even if a lookup finds a pool, we must be prepared for the delete to fail to find it, as we might race with another thread doing a delete. We use atomic_inc_not_zero() after finding a pool in the hash table and if that fails, we must have raced with a deletion, so we treat the lookup as a failure. Use hashlen_string() rather than a hand-crafted hash function. Note that the pool_name, and the search key, are guaranteed to be nul terminated. Based on Linux-commit: 055ed193b190edac539f37a66699b02eae3a19a9 with the port of server side pool handling to rhashtables. Change-Id: Ia5b4cbbd17515ea43a473e91719b3665f46b0d0a Signed-off-by: NeilBrown <neilb@suse.com> Reviewed-on: https://review.whamcloud.com/32662 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Neil Brown <neilb@suse.de> Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-12624 obdclass: lu_tgt_descs cleanup This patch cleans up code about lu_tgt_descs, so that it's cleaner to add MDT object QoS allocation support: * rename struct ost_pool to lu_tgt_pool. * put struct lu_qos, lmv_desc/lov_desc and lu_tgt_pool into struct lu_tgt_descs because it's more natural to manage these data there and fewer arguments are needed to pass around in related functions. * remove lu_tgt_descs.ltd_tgtnr, use lu_tgt_descs.ltd_lov_desc.ld_tgt_count instead, because they are duplicate. * other cleanups. Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com> Change-Id: I46f2e0ff06a8e580bac1dfda9a09a549b38d487d Reviewed-on: https://review.whamcloud.com/35824 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>