LU-4974 lod: Change pool_desc to "[lod|lov]_pool_desc" This patch changes 'struct pool_desc' under lov and lod to 'lov_pool_struct' and 'lod_pool_struct' respectively. This is the first step to check if there is anything common and can be unify. Although both layer uses 'struct pool_desc' to define the pool_desc struct respectively. 'struct pool_desc' under lod has changed and grown. Therefore to remove ambiguity, prefix lod/lov is added to pool_desc struct for respective layer. This patch also adds function description wherever applicable This patch also changes space/tabs reported by checkpatch Signed-off-by: Andreas Dilger <adilger@whamcloud.com> Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com> Change-Id: I3fee3f2e9c321145779d9177a8e4582d123f1e8d Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/11114 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Timothy Day <timday@amazon.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-13199 lustre: remove cl_{offset,index,page_size} helpers These helpers could be replaced with PAGE_SIZE and PAGE_SHIFT calculation directly which avoid CPU overhead. Change-Id: I624136d4399a03e599f09f00a77b86de045f19e9 Signed-off-by: Wang Shilong <wshilong@ddn.com> Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/37426 Reviewed-by: Neil Brown <neilb@suse.de> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com>
LU-16515 clio: Remove cl_page_size() cl_page_size() is just a function which does: 1 << PAGE_SHIFT and the kernel provides a macro for that - PAGE_SIZE. Maybe it didn't when this function was added, but it sure does now. So, remove cl_page_size(). Test-parameters: trivial Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com> Change-Id: I6c27f6db7cfec5d9054aab95beccffe3c2da02bb Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/49918 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Sebastien Buisson <sbuisson@ddn.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-16281 clio: append to non-existent component should return an error, but it fails now with a BUG below because @rc of lov_io_layout_at() is not checked for < 0 BUG: unable to handle kernel paging request at ffff99d3c2f74030 Call Trace: lov_stripe_number+0x19/0x40 [lov] lov_page_init_composite+0x103/0x5f0 [lov] ? kmem_cache_alloc+0x12e/0x270 cl_page_alloc+0x19f/0x660 [obdclass] cl_page_find+0x1a0/0x250 [obdclass] ll_write_begin+0x1f7/0xfb0 [lustre] HPE-bug-id: LUS-11075 Signed-off-by: Vitaly Fertman <vitaly.fertman@hpe.com> Change-Id: I4371f56cd9cdb3429d52a283831fb0a768e5c9c3 Reviewed-on: https://es-gerrit.dev.cray.com/161123 Tested-by: Jenkins Build User <nssreleng@cray.com> Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com> Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com> Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/48994 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Alexander <alexander.boyko@hpe.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-10994 clio: remove cl_page_export() and cl_page_is_vmlocked() Remove cl_page_export() and cl_page_is_vmlocked(), replacing them with direct calls to PageSetUptodate() and PageLoecked(). Signed-off-by: John L. Hammond <jhammond@whamcloud.com> Change-Id: I883d1664f4afc7a1d4006f9f4833db8125c0e8f5 Reviewed-on: https://review.whamcloud.com/47241 Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Bobi Jam <bobijam@hotmail.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-10994 lov: remove lov_page Remove the lov page layer since it does nothing but costs 24 bytes per page plus pointer chases. Signed-off-by: John L. Hammond <jhammond@whamcloud.com> Change-Id: Icd7b4b0041e0fe414a3a4143179f45845177960e Reviewed-on: https://review.whamcloud.com/47221 Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Bobi Jam <bobijam@hotmail.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-13799 lov: Cache stripe offset calculation Calculating the page offset relative to the stripe (etc) in a file is surprisingly expensive. Because i/o has already been split up to stripes by the cl_io code, calculating the stripe each time is unnecessary. We cache most of the values requiring calculation. This improves AIO/DIO page submission significantly, improving performance by a bit over 10%. Also remove lpg_generation, which isn't doing anything useful. This suggests the possibility of removing lov_page, but that's for another patch. This patch reduces i/o time in ms/GiB by: Write: 17 ms/GiB Read: 22 ms/GiB Totals: Write: 119 ms/GiB Read: 121 ms/GiB mpirun -np 1 $IOR -w -r -t 64M -b 64G -o ./iorfile --posix.odirect With previous patches in series: write 7531 MiB/s read 7179 MiB/s Plus this patch: write 8637 MiB/s read 8488 MiB/s Signed-off-by: Patrick Farrell <farr0186@gmail.com> Change-Id: I89e994592853d0fe93a034bfe8bdfb459bdaf584 Reviewed-on: https://review.whamcloud.com/39445 Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Yingjin Qian <qian@ddn.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-14487 modules: remove references to Sun Trademark. "lustre" is no longer a Trademark of Sun Microsystems. There is no need to acknowledge the trademark in every file, so just remove all these claims. Test-Parameters: trivial Signed-off-by: Mr NeilBrown <neilb@suse.de> Change-Id: I66941494eabc54bedf85079c5b85701187f2a8f1 Reviewed-on: https://review.whamcloud.com/42139 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Aurelien Degremont <degremoa@amazon.com> Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
LU-6142 obdclass: use cl_object_for_each more broadly. There are several loops which follow the pattern of the macros cl_object_for_each() or cl_object_for_each_reverse(), but don't use those macros. Change to use the macros. In some case we change the name of an 'obj' arg to a function to 'top', to avoid overloading the name 'obj'. Also include some minor formatting improvements. Signed-off-by: Mr NeilBrown <neilb@suse.de> Change-Id: I544d4148719a9284f33fd61730bd733c0d40e97c Reviewed-on: https://review.whamcloud.com/39391 Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Wang Shilong <wshilong@whamcloud.com> Reviewed-by: James Simmons <jsimmons@infradead.org> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-13134 clio: cl_page index compaction The cpl_index field is not necessary for the vvp layer, since that's the same as the vm page index, and it has already been duplicated for the lov layer. So, moving the osc usage of this to the top, we can save 24 bytes per page (in the slices) by removing this from the cl_page_slice, at the cost of 8 bytes in the cl_page. So a net savings of 16 bytes. After this patch, cl_page could be reduced from 408 to 392 bytes. Signed-off-by: Patrick Farrell <farr0186@gmail.com> Change-Id: Ib4fd75f112afb816443799314cd2ef75ff4f4c84 Reviewed-on: https://review.whamcloud.com/37417 Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Reviewed-by: Wang Shilong <wshilong@ddn.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-12535 lov: Move page index to top level When doing readahead, we see an amazing amount of time (~5-8%) just looking up the page index from the lov layer. In particular, this is more than half the time spent submitting pages: - 14.14% cl_io_submit_rw - 13.40% lov_io_submit - 8.24% lov_page_index This requires several indirections, all of which can be avoided by moving this up to the cl_page struct. Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com> Change-Id: I99bd7eb4d6556ac89c1aa9aeb4b7afc99774b212 Reviewed-on: https://review.whamcloud.com/35470 Reviewed-by: Wang Shilong <wshilong@ddn.com> Tested-by: jenkins <devops@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Li Xi <lixi@ddn.com> Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-11376 lov: new foreign LOV format This patch introduces a new layout/LOV format in order to allow to specify an arbitrary external reference for a file in Lustre namespace. The new LOV format is made of {newmagic, length, type, flags, string[length]} to be as flexible as possible. Foreign file can be created by using the open(O_LOV_DELAY_CREATE) + ioctl(LL_IOC_LOV_SETSTRIPE) operations and it can only be and remain an empty file until removed. A new API method llapi_file_create_foreign() has been introduced and "lfs [[get,set]stripe,find" modified to understand new layout. The idea behind this is to provide Lustre namespace support and layout prefetch/caching under layout protection, for user/external usage. Code has been added for lfsck to handle foreign files, and a new sub-test has been added in sanity-lfsck in order to verify if does not break foreign file and that reverse is also true. Signed-off-by: Bruno Faccini <bruno.faccini@intel.com> Change-Id: I5d9c0642fe8e7009c30918bfa946cac7c00c9af8 Reviewed-on: https://review.whamcloud.com/33755 Tested-by: Jenkins Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Tested-by: Maloo <maloo@whamcloud.com> Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
LU-6142 lov: Fix style issues for lov_page.c This patch fixes issues reported by checkpatch for file lustre/lov/lov_page.c Change-Id: I718e8080c8555be229823610fccc963f58a3aa93 Test-Parameters: trivial Signed-off-by: Arshad Hussain <arshad.super@gmail.com> Reviewed-on: https://review.whamcloud.com/33541 Tested-by: Jenkins Reviewed-by: Andreas Dilger <adilger@whamcloud.com> Tested-by: Maloo <hpdd-maloo@intel.com> Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
LU-10308 misc: update Intel copyright messages for 2017 Update copyright messages for files updated in 2016, excluding trivial patches. Add trivial patches to updatecw.sh script exclude list. Revert some changes that were incorrectly attributed to the 2016 (d10200a80770f0029d1d665af954187b9ad883df) and 2015 (0754bc8f2623bea184111af216f7567608db35b6) copyright update patches themselves, since they were not in the exclude list when the subsequent script was run. Test-Parameters: trivial Signed-off-by: Andreas Dilger <andreas.dilger@intel.com> Change-Id: I82f21c30c4dac75792bb49fc139bee2ca51f5545 Reviewed-on: https://review.whamcloud.com/30341 Tested-by: Jenkins Tested-by: Maloo <hpdd-maloo@intel.com> Reviewed-by: Jian Yu <jian.yu@intel.com> Reviewed-by: James Nunez <james.a.nunez@intel.com> Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-9771 lov: skip empty pages in lov_io_submit() empty page doesn't have corresponding bottem half so it should be completed in the LOV layer. Test-Parameters: testlist=sanity-flr Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com> Change-Id: Idb4576a8689bdf01638d66fddd123f05ffdd02cf Reviewed-on: https://review.whamcloud.com/29094 Tested-by: Jenkins Tested-by: Maloo <hpdd-maloo@intel.com> Reviewed-by: Bobi Jam <bobijam@hotmail.com> Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
LU-9771 flr: Send write intent RPC to mdt When a mirrored file is going to be written, the client needs to send a write intent RPC to the MDT. The MDT will pick a mirror as primary and mark the others as stale. The new md operation moo_layout_change() is introduced for this purpose. The MDT also transfers the latest layout version to the OST objects via do_attr_set(). Once OSTs receive the setattr RPC for layout version change, it will set the update layout version into extended attribute XATTR_NAME_FID. Test-Parameters: testlist=sanity-flr Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com> Change-Id: Ib0049a78b95895141b0032e8eff526a73a160dcb Reviewed-on: https://review.whamcloud.com/29091 Tested-by: Jenkins Reviewed-by: Lai Siyao <lai.siyao@intel.com> Tested-by: Maloo <hpdd-maloo@intel.com> Reviewed-by: Bobi Jam <bobijam@hotmail.com>
LU-9771 flr: read support for flr Avoid stale mirrors to read; If preferred mirror is inaccessible, try next one, ndelay RPC is implemented to make the error-out quick. ndelay RPC has rq_no_delay bit set that can be applied to brw and lock RPC. Test-Parameters: testlist=sanity-flr Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com> Change-Id: I52143079edd1566ecb3a734ed88dab19f882c2fc Reviewed-on: https://review.whamcloud.com/29085 Tested-by: Jenkins Reviewed-by: Bobi Jam <bobijam@hotmail.com> Reviewed-by: Fan Yong <fan.yong@intel.com> Tested-by: Maloo <hpdd-maloo@intel.com>
LU-9340 lov: readahead shouldn't exceed component boundary Otherwise, it will extend the readahead RPC to the next component while the actual lock of that component is not checked. Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com> Change-Id: Ice743d45f9df5e6fdc83b07aa6af1b182b660c9a Reviewed-on: https://review.whamcloud.com/26677 Tested-by: Jenkins Reviewed-by: Bobi Jam <bobijam@hotmail.com> (cherry picked from commit e31e234c06ac798501cdb7ec92269af83157cb21) Reviewed-on: https://review.whamcloud.com/26861 Tested-by: Maloo <hpdd-maloo@intel.com> Reviewed-by: Andreas Dilger <andreas.dilger@intel.com> Reviewed-by: Niu Yawei <yawei.niu@intel.com> Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-9008 pfl: dynamic layout modification with write/truncate * in lov_init_composite(), skip init sub object without LCME_FL_INIT layout component. * issue layout intent RPC during write/trunc ops when try to write to an un-init-ed component (even if at the lock stage). * After layout intent RPC issued, restart the IO. * get rid of unused lov_layout_operations::llo_install() interface. * add an empty mdt_layout_change() interface to handle intent layout write RPC. Reviewed-on: https://review.whamcloud.com/25317 Signed-off-by: Bobi Jam <bobijam.xu@intel.com> Change-Id: I2f79482187d2af2660dd86e55da3f5dc0138e94a Reviewed-by: Niu Yawei <yawei.niu@intel.com> Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
LU-8998 clio: Client side implementation for PFL Make client layer support composite layout. Plain layout will be stored in LOV layer as a composite layout containing a single component. Reviewed-on: https://review.whamcloud.com/24850 Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com> Signed-off-by: Bobi Jam <bobijam.xu@intel.com> Signed-off-by: Niu Yawei <yawei.niu@intel.com> Change-Id: Ic3b85a4b10c66745e5c72ff02ea313baa0b12bb5 Reviewed-by: Lai Siyao <lai.siyao@intel.com>