Whamcloud - gitweb
fs/lustre-release.git
16 months agoLU-6142 libcfs: use BIT() macro where appropriate 74/38374/2
Mr NeilBrown [Mon, 27 Apr 2020 03:36:17 +0000 (13:36 +1000)]
LU-6142 libcfs: use BIT() macro where appropriate

When accessing a bit in a bitmap/mask/flags-word it can be more
readable to use BIT(num) rather than "1 << num".

This patch makes that change to various places in libcfs.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I5e945358d2385599f8fe6f1ddff310efe17b589f
Reviewed-on: https://review.whamcloud.com/38374
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-13054 ldiskfs: split htree_lock as separate patch 72/38372/5
Yang Sheng [Sun, 26 Apr 2020 11:59:16 +0000 (19:59 +0800)]
LU-13054 ldiskfs: split htree_lock as separate patch

The htree_lock part is identical in the different
distro version of pdirop patch. So move it out as
separate patch to reduce maintenance effort.

Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: I423cc957de37ccdb097c9893f69481ce947ac78c
Reviewed-on: https://review.whamcloud.com/38372
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-13054 ldiskfs: htree_node wrongly granted 71/38371/4
Yang Sheng [Sun, 26 Apr 2020 11:56:40 +0000 (19:56 +0800)]
LU-13054 ldiskfs: htree_node wrongly granted

The thread was waken up accidently. So need check
whether the lock granted or not after wake up.
Also fix issue that major always set to 0 since
hbit initialize incorrect. The performace should be
impacted especial operate in big directory.

kernel BUG at lustre/ldiskfs/htree_lock.c:429!
 Call Trace:
 htree_node_release_all+0x5a/0x80 [ldiskfs]
 htree_unlock+0x22/0x70 [ldiskfs]
 osd_index_ea_delete+0x30e/0xb10 [osd_ldiskfs]
 lod_sub_delete+0x1c8/0x460 [lod]
 lod_delete+0x24/0x30 [lod]
 __mdd_index_delete_only+0x194/0x250 [mdd]
 __mdd_index_delete+0x46/0x290 [mdd]
 mdd_unlink+0x5f8/0xaa0 [mdd]
 mdo_unlink+0x46/0x48 [mdt]
 mdt_reint_unlink+0xbed/0x14b0 [mdt]

Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: I5972961bc78b349214c6756642717d126f0c4b26
Reviewed-on: https://review.whamcloud.com/38371
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-12511 build: don't use OpenSFS UAPI headers with --disable-modules 70/38370/2
James Simmons [Sun, 26 Apr 2020 17:33:31 +0000 (13:33 -0400)]
LU-12511 build: don't use OpenSFS UAPI headers with --disable-modules

If we configure Lustre with --disable-modules that means we want
to use the native Linux Lustre / LNet UAPI headers instead. To do
this only include OpenSFS UAPI header path when Lustre modules are
enabled.

Change-Id: Id0f4d960d51bf02a6e24d2e3ad2544df10384533
Test-Parameters: trivial
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/38370
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-9679 various: use OBD_ALLOC_PTR_ARRAY() and others 50/38250/8
Mr NeilBrown [Thu, 14 Nov 2019 03:20:01 +0000 (14:20 +1100)]
LU-9679 various: use OBD_ALLOC_PTR_ARRAY() and others

Use:
  OBD_ALLOC_PTR_ARRAY
  OBD_FREE_PTR_ARRAY
  OBD_ALLOC_PTR_ARRAY_LARGE
  OBD_FREE_PTR_ARRAY_LARGE

for allocating and freeing arrays in various modules.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I0c65759e99fde15168f190ea74cd88b3cb01f667
Reviewed-on: https://review.whamcloud.com/38250
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
16 months agoLU-9679 obdclass: use OBD_ALLOC_PTR_ARRAY() and others 21/38121/4
Mr NeilBrown [Thu, 14 Nov 2019 03:20:01 +0000 (14:20 +1100)]
LU-9679 obdclass: use OBD_ALLOC_PTR_ARRAY() and others

Use:
  OBD_ALLOC_PTR_ARRAY
  OBD_ALLOC_PTR_ARRAY_WAIT
  OBD_FREE_PTR_ARRAY
  OBD_ALLOC_PTR_ARRAY_LARGE
  OBD_FREE_PTR_ARRAY_LARGE

for allocating and freeing arrays in obdclass

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I2ebf853769a9711f1f99431aa7f69a18ba2ea552
Reviewed-on: https://review.whamcloud.com/38121
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
16 months agoLU-11023 quota: quota pools for OSTs 15/35615/52
Sergey Cheremencev [Fri, 12 Jul 2019 13:38:35 +0000 (16:38 +0300)]
LU-11023 quota: quota pools for OSTs

Patch allows to apply quota settings
not only for the whole system, but also
for different OST pools. Since this patch
each "LOD" pool is duplicated by QMT.
Thus quota pools(QP) could be tuned by
standard lctl pool_new/add/remove/erase
commands. All QPs are subset of a global pool
that includes all data devices in a system,
including DOM. However DOM is not supported.
I don't see a lot of work to add DOM support
in future - just need to decide how MDTs
could be combined in a pool.

The main idea of QP is to find all pools
for requested ID(usr/grp/prj) and apply
minimum limit. The patch doesn't affect
qsd side, so slaves know nothing about
pools and different limits. Qunit and edquot
are calculated for each slave on master.

To apply quota on QP, the patch adds key "-o"
to lfs setquota. To get quotas for QP, it
provides long option "--pool" in lfs quota.
See examples of using in sanity-quota_1b/c/d.

Now QPs work properly only on a clean system.
Support of recalculation granted space in case
of adding/removing OSTs in a pool will be added
in the next patch together with accounting
already granted space by each ID in a POOl.

Test-Parameters: testgroup=review-dne-part-4

Change-Id: I3396aded2156729b4fd15166eb59db59ee4c967e
Signed-off-by: Sergey Cheremencev <c17829@cray.com>
Reviewed-on: https://review.whamcloud.com/35615
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-13137 llite: do not flush COW pages from mapping 78/37278/17
Bobi Jam [Tue, 21 Jan 2020 07:35:17 +0000 (15:35 +0800)]
LU-13137 llite: do not flush COW pages from mapping

Do not flush COW pages from an executable mapping, as this will cause
running binaries to segfault when mapped pages disappear under them.

This was modified to allow non-blocking PCC caching, but triggers
immediate segfaults for binaries when the DLM locks are revoked, even
when PCC is not in use.

The added test case failed 100% without this patch.

This patch is only a partial fix to avoid the segfaults.  There still
needs to be something done to re-introduce the PCC functionality, but
should be conditional on PCC enabled, and not for running binaries.

There is a problem for PCC when multiple clients read/write on a
shared mmapped file. We exclude mmap_sanity tst6 in test_4 from
PCC testing temporarily.

Test-Parameters: testlist=sanity envdefinitions=ONLY=30d
Test-Parameters: testlist=sanity envdefinitions=ONLY=30d
Test-Parameters: testlist=sanity envdefinitions=ONLY=30d
Fixes: 58d744e3eaab ("LU-10092 pcc: Non-blocking PCC caching")
Change-Id: I7084e1db1f0f29cb0237b619abd33ca556aa7ec7
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/37278
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Stephan Thiell <sthiell@stanford.edu>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-9859 libcfs: add support for Xarray 91/37391/9
James Simmons [Wed, 29 Apr 2020 12:37:32 +0000 (08:37 -0400)]
LU-9859 libcfs: add support for Xarray

In the late 4.X kernel cycle Xarrays were introduced with the
goal of replacing the radix tree for the page cache. It is
highly optimized for densely packed data which is the case for
several items in Lustre such as the static array for the obd
devices and quota ids. This patch provides Xarray support for
kernels that lack. The current verison of Xarray back ported
is from the 5.4-rc2 kernel.

Change-Id: I54f9046f50a353e1cd4271c0b97207062bbf3898
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/37391
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-10973 lnet: dump cyaml to buffer 82/38082/5
Amir Shehata [Sat, 14 Dec 2019 23:20:03 +0000 (15:20 -0800)]
LU-10973 lnet: dump cyaml to buffer

Add a new API to dump cyaml to a buffer which gets returned to the
user. The user is then responsible for freeing the buffer.

This is a new API required by the LUTF in order to interface
with python more readily.

Test-Parameters: trivial
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: I6068670f26b617d7a8366bd248dd80ca8cf15e5e
Reviewed-on: https://review.whamcloud.com/38082
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-13138 tests: measure 'dd' time more accurately 27/38427/3
Andreas Dilger [Thu, 30 Apr 2020 06:36:21 +0000 (00:36 -0600)]
LU-13138 tests: measure 'dd' time more accurately

The sanity test_101d can fail with 'readahead 5s > no-readahead 5s'
because the "do_and_time" function only measures runtime in whole
seconds.  Also, in some rare cases, the VM might be slow and cause
the runtime of the do_and_time() function to be slow, even if the
"dd" command finished quickly.

Instead of timing the duration of "dd" in seconds, parse the output
from "dd" directly, which provides fractional second precision.

Test-Parameters: trivial testlist=sanityn
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I5f1ec0374c6a9b9c175db77b2b972e3b433ebbe5
Reviewed-on: https://review.whamcloud.com/38427
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-9859 libcfs: hide struct cfs_cpt_table internals 89/38389/2
James Simmons [Mon, 27 Apr 2020 15:49:33 +0000 (11:49 -0400)]
LU-9859 libcfs: hide struct cfs_cpt_table internals

Exposing the internals of struct cfs_cpt_table in the libcfs
header used by lustre makes it too tempting to use but those
fields are not available for UP systems. Make cfs_cpt_table
abstract so the lustre and LNet code will never use the
internals and instead use the functions provided in
libcfs_cpu.h.

Linux-commit: commit a6fe6f584b6a1032c2e63e94d7a7a6f1436c3774

Change-Id: Ia2b3363bcee87e042a133cc7af9a912cbafb0d92
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/38389
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-13449 tgt: fix recovery timer comparisons 66/38366/2
Andreas Dilger [Sat, 25 Apr 2020 10:19:42 +0000 (04:19 -0600)]
LU-13449 tgt: fix recovery timer comparisons

The obd_recovery_start time was changed to use a monotonic kernel
clock via ktime_get_seconds(), which is a relative kernel time in
newer kernels.  However, it was still being compared with wallclock
time in some places, which lead to incorrect calculations.

Always use the monotonic clock within the kernel, but convert the
times to wallclock time when they are printed to userspace.

Fix the added recovery-small test_140b to use the actual recovery time
rather than the entire failover time, since that may increased too
much by operations unrelated to the MDS recovery.

Fixes: 8bd04b4e5766 ("LU-12722 target: disable recovery for local clients")
Fixes: 06408a4ef381 ("LU-12769 recovery: use monotonic timer")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ibe866463cbad81010e91f630a1088990f8a48664
Reviewed-on: https://review.whamcloud.com/38366
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-13466 mgc: protect from NULL exp in mgc_enqueue() 80/38280/2
Sebastien Buisson [Mon, 20 Apr 2020 08:50:25 +0000 (10:50 +0200)]
LU-13466 mgc: protect from NULL exp in mgc_enqueue()

Return an error in mgc_enqueue() is exp parameter is NULL.
Otherwise, it might crash in class_exp2cliimp().

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I9be0bf57c6a21c83abb08b261f548128114c33f2
Reviewed-on: https://review.whamcloud.com/38280
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
16 months agoLU-13405 kernel: kernel update SLES15 SP1 [4.12.14-197.37.1] 45/38245/3
Jian Yu [Wed, 15 Apr 2020 23:35:59 +0000 (16:35 -0700)]
LU-13405 kernel: kernel update SLES15 SP1 [4.12.14-197.37.1]

Update SLES15 SP1 kernel to 4.12.14-197.37.1 for Lustre client.

Test-Parameters: trivial \
envdefinitions=SANITY_EXCEPT="56oc 100 130 136 817" \
clientdistro=sles15sp1 serverdistro=el7.8 \
testlist=sanity

Change-Id: I914a3cb756a1559142d334ec78076bc0be5e03cd
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/38245
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-13421 kernel: kernel update RHEL8.1 [4.18.0-147.8.1.el8_1] 26/38226/3
Jian Yu [Wed, 15 Apr 2020 00:24:49 +0000 (17:24 -0700)]
LU-13421 kernel: kernel update RHEL8.1 [4.18.0-147.8.1.el8_1]

Update RHEL8.1 kernel to 4.18.0-147.8.1.el8_1.

Test-Parameters: trivial \
clientdistro=el8.1 serverdistro=el8.1 \
testlist=sanity

Change-Id: I7484da780f731b89baf92b5eebebb601d9a27b86
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/38226
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-9859 libcfs: remove conditional compilation from libcfs_cpu.c 81/37881/10
Mr NeilBrown [Wed, 22 Apr 2020 12:58:39 +0000 (08:58 -0400)]
LU-9859 libcfs: remove conditional compilation from libcfs_cpu.c

libcfs_cpu.c manages CPU partitions. In the !CONFIG_SMP case, most
of this disappears and 'static inline's from libcfs_cpu.h are used.
However we still allocate a 'struct cfs_cpt_table' which is used
to keep the NUMA node mask. Event with SMP disabled Lustre wants
to handle any NUMA nodes available.

This patch removes all the !CONFIG_SMP code from libcfs_cpu.c and
conditionally compiles the whole file only when CONFIG_SMP.
We no longer allocate a 'struct cfs_cpt_table' on !CONFIG_SMP,
and don't even declare a structure.  The name "cfs_cpt_tab"
becomes always "NULL", which allows some code to be optimized away.

This means that cfs_cpt_tab can sometimes be NULL, so we need to
discard the assertion that it isn't.

Linux-commit: 7b8e2026fb185e5178f9137c463ff07fc895be5d

Test-Parameters: trivial
Change-Id: Icb84f23249d086c662de76df6ce9686de0b692aa
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-on: https://review.whamcloud.com/37881
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-13238 ofd: add OFD access logs 52/37552/16
John L. Hammond [Tue, 28 Jan 2020 15:29:52 +0000 (09:29 -0600)]
LU-13238 ofd: add OFD access logs

Add access logs to OFD layer. BRW RPC handlers will record accesss to
an in memory circular buffer which may be read in userspace through
character devices (/dev/lustre-access-log/$FSNAME-OSTxxxx). A control
device (/dev/lustre-access-log/control) is added to facilitate device
discovery. A utility (ofd_access_log_reader) to discover and consume
access logs is included.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I76b78cc5075ee01f9b234e96e7a22a1bdcf2f755
Reviewed-on: https://review.whamcloud.com/37552
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Gu Zheng <gzheng@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-13465 vfs: make ll_vfs_setxattr return -EOPNOTSUPP 01/38301/5
Sebastien Buisson [Tue, 21 Apr 2020 07:28:20 +0000 (09:28 +0200)]
LU-13465 vfs: make ll_vfs_setxattr return -EOPNOTSUPP

New compat macros ll_vfs_setxattr/ll_vfs_removexattr need to return
-EOPNOTSUPP instead of -ENOTSUPP. Indeed, the llapi_layout_get_by_fd()
code checks for "-EOPNOTSUPP" and handles this specially.

Also, the case of missing i_op->setxattr() and i_op->getxattr() can be
considered unlikely.

Test-Parameters: clientversion=2.12 envdefinitions=SANITY_EXCEPT="27M 56ra 151 156 802"
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I2474c184a1597979971881de84638fb8d281f704
Reviewed-on: https://review.whamcloud.com/38301
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-13465 mgs: re-introduce mgs_set_info for interop purpose 11/38311/4
Sebastien Buisson [Wed, 22 Apr 2020 13:54:53 +0000 (22:54 +0900)]
LU-13465 mgs: re-introduce mgs_set_info for interop purpose

Older clients might send MGS_SET_INFO requests to the MGS. Newer MGS
needs to handle that gracefully by returning:
- -EINVAL for anything other than lov.stripe*
- 0 for those parameters without doing anything, rather than
  returning -ENOTSUPP.

Test-Parameters: clientversion=2.12 envdefinitions=SANITY_EXCEPT="27M 56ra 151 156 802"
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Iba75c788eaefabd98bbb65a657f9a0150e20fafb
Reviewed-on: https://review.whamcloud.com/38311
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-12780 target: don't use ptlrpc_thread for txn_commit_thread 60/36260/10
Mr NeilBrown [Wed, 23 Oct 2019 00:30:49 +0000 (11:30 +1100)]
LU-12780 target: don't use ptlrpc_thread for txn_commit_thread

rather than ptlrpc_thread, use native kthreads functionality.

- there is no need to synchronize on startup, the initialization
  can be done before the thread is started.  This requires adding
  an lu_env to struct target_distribute_txn_data so
  distribute_txn_init() can set it up before starting the thread.

- correspondingly, the cleanup is best done outside of the thread too,
  as it is possible for kthread_stop() to stop a thread before
  the function is called even once.  So the lu_env_fini is moved to
  distrbute_txn_fini().  and ->tdtd_list is cleaned up there too
  just incase the thread didn't have a chance to run.

- kthread_stop/kthread_should_stop is used to synchornize
  shutdown

- signaling the thread is done with wake_up_process().
  The thread sets TASK_IDLE at the top of the loop, then
  sets TASK_RUNNING if it finds anything to do, and finally
  calls schedule() at the end.  This makes
  tdtd_ready_for_cancel_log() unnecesary as it just duplicates
  checks that are already present in distribute_txn_commit_thread().

Change-Id: I06c3686b90faa6c6b638b8d6c69cd4e05c2783f4
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Reviewed-on: https://review.whamcloud.com/36260
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-9859 libcfs: handle missing refcount.h 90/38390/3
James Simmons [Wed, 29 Apr 2020 12:32:39 +0000 (08:32 -0400)]
LU-9859 libcfs: handle missing refcount.h

Recently the OpenSFS tree was modified to make backported headers to
appear like native Linux kernel headers to avoid ifdef handling. One
missed is refcount.h. Rename linux-refcount.h to refcount.h to make
it look like the native kernel version. Change EXTRA_KCFLAGS to not
include the Lustre include directories since it contains the fake
kernel headers in the autoconf test. Also remove stray
LC_REPLACE_EXCEPTIONAL_ENTRIES.

Change-Id: Ia5de6dab0837db419a6b02b3b2a6ddeb6bb62f03
Test-Parameters: trivial
Fixes: 742897a967cf ("LU-13274 uapi: make lnet UAPI headers C99 compliant")
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/38390
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-6142 lustre: Remove inappropriate uses of BIT() macro. 73/38373/2
Mr NeilBrown [Mon, 27 Apr 2020 03:32:27 +0000 (13:32 +1000)]
LU-6142 lustre: Remove inappropriate uses of BIT() macro.

The BIT() macro exists for identifying a specific bit in a word when
it is being used as a bitmap (or mask for set of flags etc).
While it uses "1 << ...." it is not a general replacement for that
construct and should not be used to simply to raise '2' to some power.

Varous places in lustre and libcfs use BIT() when a size, rather than
a BIT, are required.  Convert these to explicitly use "1 << exponent".

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I41e1e188da4d2e5dff5b2c05cec607c1d104bcfa
Reviewed-on: https://review.whamcloud.com/38373
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-12511 build: avoid building rpm with --disable-modules 69/38369/3
James Simmons [Sun, 26 Apr 2020 15:12:14 +0000 (11:12 -0400)]
LU-12511 build: avoid building rpm with --disable-modules

If you configure lustre with --disable-modules this will prevent
OpenSFS lustre modules from being built but make rpms will fail
since it still expects built modules. Update the lustre.spec
file to not package any modules when --disable-modules is set.

Test-Parameters: trivial
Change-Id: Ide8d80f5b36f819b486a143e56daf494e6a1fe23
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/38369
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-12222 lnet: Primary NID of lolnd NID is the lolnd NID 13/38313/3
Chris Horn [Wed, 22 Apr 2020 16:42:27 +0000 (11:42 -0500)]
LU-12222 lnet: Primary NID of lolnd NID is the lolnd NID

We want Lustre traffic that is intended for the local peer to be sent
and received over the lolnd. The function ptlrpc_uuid_to_peer() will
currently resolve a NID to the lolnd NID, but ptlrpc_connection_get()
will overwrite this selection with the result from LNetPrimaryNID().

Have LNetPrimaryNID return the lolnd NID when it is passed the lolnd
NID.

HPE-bug-id: LUS-8457
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I02708bb45f8440091782ca7886bac7656efb0223
Reviewed-on: https://review.whamcloud.com/38313
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-12222 lnet: Introduce constant for the lolnd NID 12/38312/3
Chris Horn [Wed, 22 Apr 2020 16:39:46 +0000 (11:39 -0500)]
LU-12222 lnet: Introduce constant for the lolnd NID

This patch adds a new constant, LNET_NID_LO_0, to represent the lolnd
NID 0@lo.

HPE-bug-id: LUS-8457
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I3e57637f297b8de306905a447af8f025e31d1fcf
Reviewed-on: https://review.whamcloud.com/38312
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-9679 osd: use OBD_ALLOC_PTR_ARRAY() and others 52/38252/3
Mr NeilBrown [Thu, 14 Nov 2019 03:20:01 +0000 (14:20 +1100)]
LU-9679 osd: use OBD_ALLOC_PTR_ARRAY() and others

Use:
  OBD_ALLOC_PTR_ARRAY
  OBD_FREE_PTR_ARRAY
  OBD_ALLOC_PTR_ARRAY_LARGE
  OBD_FREE_PTR_ARRAY_LARGE

for allocating and freeing arrays in osd-ldiskfs and osd-zfs.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Iec2ec54e0b87aeefb42d5b4ab802c564850f6edd
Reviewed-on: https://review.whamcloud.com/38252
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-13389 tgt: not rollback obsolete rq 21/38221/3
Alexander Zarochentsev [Sat, 4 Apr 2020 06:44:26 +0000 (09:44 +0300)]
LU-13389 tgt: not rollback obsolete rq

Restoring old pre-LU-11444 behavior in
last_rcvd/reply_data update, do not generate
a error for obsolete requests. Othewise tx_end
tries to rollback the operation and does it partially
leaving inodes unattached and changelog records not cleared;
Fixing reply_data slot leak on error.

Fixes: 23773b3 ("LU-11444 ptlrpc: resend may corrupt the data")
HPE-bug-id: LUS-8590
Signed-off-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Change-Id: I76c2de7b820da0239ce2f0a9d17b7dedadc0519b
Reviewed-on: https://review.whamcloud.com/38221
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-13429 lov: lov_io_sub_init()) ASSERTION 69/38169/4
Bobi Jam [Wed, 8 Apr 2020 07:36:42 +0000 (15:36 +0800)]
LU-13429 lov: lov_io_sub_init()) ASSERTION

The assertion is_index_within_mirror() in lov_io_sub_init() should
only be applied for a FLR file, since a plain file does not initialize
the relevant FLR fields of the layout structure.

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I0d6ff505d789c76e7912aa08491146706b5316bc
Reviewed-on: https://review.whamcloud.com/38169
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-11025 uapi: add OBD_CONNECT2_FIDMAP 32/38232/3
Lai Siyao [Wed, 15 Apr 2020 02:46:00 +0000 (10:46 +0800)]
LU-11025 uapi: add OBD_CONNECT2_FIDMAP

Add OBD_CONNECT2_FIDMAP connect flag, which indicates whether client
supports FID mapping.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I165598e7d19878d1d3dcfa8c6f8fcc3cc233e4c9
Reviewed-on: https://review.whamcloud.com/38232
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
16 months agoLU-12785 dom: adjust DOM stripe size by free space 04/37904/6
Mikhail Pershin [Thu, 12 Mar 2020 09:53:37 +0000 (12:53 +0300)]
LU-12785 dom: adjust DOM stripe size by free space

Make DoM default stripe size dependent on filesystem free
space. If free space become less than some defined threshold
then DoM stripe size of new files is decreased along with
free space remaining up to zero. It will grow back up to
defined maximum when more free space occur.

- introduce new tunable lod.*.dom_threshold_free_mb as free
  space threshold below which default DOM stripe size will
  decrease down to zero. Units are MiB but it can be set also
  in percentage of free space:

  # lctl set_param lod.*.dom_threshold_free_mb=100GiB
  or
  # lctl set_param lod.*.dom_threshold_free_mb=10%

- rename lod.*.dom_stripesize to lod.*.dom_stripesize_max_kb to
  reflect that this is maximum limit for DOM stripe size on MDT,
  used as KiB value. Old lod.*.dom_stripesize is kept for
  compatibility.
- new read-only parameter lod.*.dom_stripesize_cur_kb to read
  current default DoM stripesize. It is autotuned depending on
  dom_threshold_free_mb parameter and amount of free space.
- OSD statfs data used in calculations are cached in LOD, so
  DoM default stripesize is not recalculated upon each create
  but happens not often that once per 10 sec and when tunables
  are being changed
- add corresponding test 270g in sanity.sh

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: Id27815ec38ae11762c74c0097c5862eaa3dbda24
Reviewed-on: https://review.whamcloud.com/37904
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
16 months agoLU-13064 sec: check permissions for changelogs access 90/36990/32
Sebastien Buisson [Mon, 9 Dec 2019 14:37:50 +0000 (23:37 +0900)]
LU-13064 sec: check permissions for changelogs access

root permissions should be checked when reading or clearing changelogs
from clients. In particular, if root is squashed via a nodemap entry,
it should not be allowed to access changelogs.
To achieve this send mdt body along with RQF_LLOG_ORIGIN_HANDLE_CREATE
and RQF_MDT_SET_INFO requests. And on server side, retrieve user
credentials and make sure they have root permission.

Test-Parameters: clientversion=2.12 envdefinitions=SANITY_EXCEPT="27M 56ra 151 156 802"
Test-Parameters: serverversion=2.12
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I0c6cc99f8a7c5a13c2b31009d73f38976931ec37
Reviewed-on: https://review.whamcloud.com/36990
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-11025 mdt: don't save remote lock if op not from client 81/37281/8
Lai Siyao [Sat, 4 Jan 2020 20:51:32 +0000 (04:51 +0800)]
LU-11025 mdt: don't save remote lock if op not from client

Some operation may not be originated from client, e.g., directory
auto split, in this case ptlrpc_request is NULL, therefore we can't
get the transaction number from request and save remote lock. This
means upon remote lock cancel this distributed transaction may not
be committed yet, that is to say, commit-on-sharing won't be triggered
if subsequent operations depend on this operation. However this won't
cause any inconsistency normally, because upon one MDT failure, this
distributed transaction can be replayed from update logs in other
MDT, and as a side effect, it avoids continuous commit in directory
auto split.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: If7608cab96f36480654d5d7a0f5a0f1c4dfd12b2
Reviewed-on: https://review.whamcloud.com/37281
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
16 months agoLU-11025 lod: refactor lod_mdt_alloc_qos/rr() 65/36865/12
Lai Siyao [Fri, 22 Nov 2019 11:41:36 +0000 (19:41 +0800)]
LU-11025 lod: refactor lod_mdt_alloc_qos/rr()

Add arguments 'stripe_idx' and 'stripe_count' to
lod_mdt_alloc_qos/rr(), so later directory split/merge can allocate
stripes from specified stripe index.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Id054ac397fd632228fa6c2786c90e5aad9aa2eb1
Reviewed-on: https://review.whamcloud.com/36865
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-13470 ptlrpc: return proper write count from ping_store 04/38304/2
Oleg Drokin [Tue, 21 Apr 2020 22:28:33 +0000 (18:28 -0400)]
LU-13470 ptlrpc: return proper write count from ping_store

If we return 0 as a measure of success, that is actually taken
by userspace as "we wrote 0 bytes, but there's no error, so
they want us to retry". So tell them we ate all their data
instead to make them happy.

Interesting that when you do shell-redirect it enters the
retry loop, but badarea_io does not.

Change-Id: Ieb0f8bcd4f010ab4375ea6f65f6ea00248a236cb
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/38304
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
16 months agoLU-10401 tests: fix error from 'tr -d=' 44/38444/2
Andreas Dilger [Thu, 30 Apr 2020 10:47:33 +0000 (04:47 -0600)]
LU-10401 tests: fix error from 'tr -d='

Fix an error from using "tr -d=" instead of "tr -d =" in
sanity test_133g.

Test-Parameters: trivial
Fixes: 83b6c6608e9 ("LU-11644 ptlrpc: show target name in req_history")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I1bac18c7a41bc45c250d8a95ef7489d73f3ebbe5
Reviewed-on: https://review.whamcloud.com/38444
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-13479 kernel: only enable integrity metadata for reads and writes 24/38324/2
Li Dongyang [Thu, 23 Apr 2020 01:16:34 +0000 (11:16 +1000)]
LU-13479 kernel: only enable integrity metadata for reads and writes

We should not enable integrity for discard bio, it will fail the
integrity allocation and the discard bio will end with EIO.

Linux-commit: e69f18f06b97ed29645d020500222bfcec2b42b2

Test-Parameters: trivial clientdistro=el7.8 serverdistro=el7.8

Change-Id: I1fc9c6c7a6c53a6287466927ecfbb9af1f2a8f27
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-on: https://review.whamcloud.com/38324
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-13097 tests: set fail_loc on all MDS sanityn 43j 18/38318/2
James Nunez [Wed, 22 Apr 2020 21:14:09 +0000 (15:14 -0600)]
LU-13097 tests: set fail_loc on all MDS sanityn 43j

Set fail_loc on all MDS nodes for sanityn test 43j.

Test-Parameters: trivial testlist=sanityn env=ONLY=43j,ONLY_REPEAT=5
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I278875ad69a8427ef2de6a7bdb94551890399abd
Reviewed-on: https://review.whamcloud.com/38318
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
16 months agoLU-13460 lfs: use correct obd index in print_failed_tgt() 64/38264/3
Emoly Liu [Fri, 17 Apr 2020 10:52:07 +0000 (18:52 +0800)]
LU-13460 lfs: use correct obd index in print_failed_tgt()

In function print_failed_tgt(), when calling llapi_obd_statfs(),
the correct obd index should be used according to obd type.

sanity.sh test_56rb is added to verify this patch.

Signed-off-by: Emoly Liu <emoly@whamcloud.com>
Change-Id: I443a5b388ee1c3efc8fc589c09aeec6f9b71600b
Reviewed-on: https://review.whamcloud.com/38264
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-12477 ldiskfs: last cleanups 39/38139/6
James Simmons [Tue, 14 Apr 2020 15:13:57 +0000 (11:13 -0400)]
LU-12477 ldiskfs: last cleanups

The patch to cleanup ldiskfs collided with the landing of the
ext4-mballoc-prefetch patch. Remove the last unsupported rhel7
bits. With the new Ubuntu 20 coming out we can drop Ubuntu16
support. Drop 3.12 kernel versions of SUSE.

Test-Parameters: trivial
Fixes: fc87b01f96e8 ("LU-12477 ldiskfs: remove obsolete ext4 patches")
Change-Id: I15f9f59ffb1275e2eaabf7ca543fd4c4829aaf9e
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/38139
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-13388 lod: unprotected access to component entries 69/38069/3
Vitaly Fertman [Wed, 25 Mar 2020 12:55:53 +0000 (15:55 +0300)]
LU-13388 lod: unprotected access to component entries

a race is possible between setattr, which may want to traverse
the component set, and the layout_change, which may change the
component set if SEL is enabled. Take an UPDATE lock on layout
write operation.

HPE-bug-id: LUS-8146
Signed-off-by: Vitaly Fertman <c17818@cray.com>
Change-Id: Idf8a5c3d423707043153892f607cdd54099d9579
Reviewed-by: Alexey Lyashkov <c17817@cray.com>
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Tested-by: Elena Gryaznova <c17455@cray.com>
Reviewed-on: https://es-gerrit.dev.cray.com/156681
Reviewed-on: https://review.whamcloud.com/38069
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-13352 tests: fix subtest duration for ONLY_REPEAT 72/37872/3
Andreas Dilger [Tue, 10 Mar 2020 22:00:25 +0000 (16:00 -0600)]
LU-13352 tests: fix subtest duration for ONLY_REPEAT

The subtest duration printed when the test passes was the total
elapsed time rather than the per-iteration time.  Print the
per-subtest time at the end of each iteration.

Fixes: e16e3d46ee8c ("LU-13169 tests: add ONLY_REPEAT parameter to repeat subtests")
Test-Parameters: trivial testlist=sanity env=ONLY=76,ONLY_REPEAT=10
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I0758df03ceb12c3c7c8d9cd62561d4ee7b3ebbe5
Reviewed-on: https://review.whamcloud.com/37872
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-13004 lnet: remove the 'struct kvec' arg from lnd_send 45/37845/7
Mr NeilBrown [Wed, 4 Dec 2019 04:33:45 +0000 (15:33 +1100)]
LU-13004 lnet: remove the 'struct kvec' arg from lnd_send

lnd_send() is now always sent a lnet_kiov_t pointer, never a
'struct kvec' pointer.
So remove the option of 'struct kvec *'.

This means that kiblnd_setup_rd_iov() is no longer used, so it can go
too.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I121f5c8c43bd7247748d6e4913c76371e9202219
Reviewed-on: https://review.whamcloud.com/37845
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-13004 lnet: remove msg_iov from lnet_msg. 44/37844/7
Mr NeilBrown [Wed, 4 Dec 2019 04:19:52 +0000 (15:19 +1100)]
LU-13004 lnet: remove msg_iov from lnet_msg.

This field is never set, so remove it and remove all uses of it.

Conversely, msg_kiov is always non-NULL, so remove code that depends
on it being NULL.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I4c5e98bd54a463ed62ea8d06b9781aa8b4e96f50
Reviewed-on: https://review.whamcloud.com/37844
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-13004 lnet: discard kvec option from lnet_libmd. 43/37843/7
Mr NeilBrown [Wed, 4 Dec 2019 04:12:08 +0000 (15:12 +1100)]
LU-13004 lnet: discard kvec option from lnet_libmd.

The 'struct kvec' field in 'struct lnet_libmd' is no longer set - the
memory is always presented as pages.  So remove that option and the
union that contained iov and kiov.

Rename md_iov.kiov to md_kiov.
Discard all code that made use of md_iov.iov.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ic74a496c4c3df786d7fc194a1ad758c4daccd819
Reviewed-on: https://review.whamcloud.com/37843
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-13004 lnet: always put a page list into struct lnet_libmd 42/37842/5
Mr NeilBrown [Tue, 17 Mar 2020 23:00:29 +0000 (10:00 +1100)]
LU-13004 lnet: always put a page list into struct lnet_libmd

'struct lnet_libmd' is only created in lnet_md_build().  It can be
given a list of pages or a virtual address.  In the latter case, the
memory will eventually be split into a list of pages.  It is cleaner
to split it into a list of pages early so that all lower levels only
need to handle one type: a page list.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Iaeb4f18f7ac63986e6fed259945e8594ea1b9f25
Reviewed-on: https://review.whamcloud.com/37842
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-13301 lfs: return error when setstripe -o on inactive OSTs 51/37751/16
Emoly Liu [Fri, 10 Apr 2020 02:20:04 +0000 (10:20 +0800)]
LU-13301 lfs: return error when setstripe -o on inactive OSTs

When running "lfs setstripe" on inactive OSTs with -o option,
error code ENOTCONN returned by osp_statfs() for OST inactive
state is mistreated as MDT disconnection by the client, then the
client keeps trying to restore the connection to MDT and resend
the request again and again.
To fix this issue, the error code "ENOTCONN" is changed to other
code (EREMOTEIO) for distinction.

sanity.sh test_27cf is added to verify this patch.

Signed-off-by: Emoly Liu <emoly@whamcloud.com>
Change-Id: I9ea10b4985c769e52ee54ff22ac69948073b7709
Reviewed-on: https://review.whamcloud.com/37751
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-12730 tests: sync file before checking LSOM 46/37146/13
Sebastien Buisson [Mon, 6 Jan 2020 16:15:44 +0000 (01:15 +0900)]
LU-12730 tests: sync file before checking LSOM

In sanity test_807, make sure files are synced before checking
LSOM info.

Test-Parameters: trivial envdefinitions=ONLY=807 testlist=sanity,sanity,sanity,sanity
Test-Parameters: envdefinitions=ONLY=807 testlist=sanity,sanity,sanity,sanity,sanity
Test-Parameters: envdefinitions=ONLY=807 testlist=sanity,sanity,sanity,sanity,sanity
Test-Parameters: envdefinitions=ONLY=807 testlist=sanity,sanity,sanity,sanity,sanity
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Id6a452cc6b734d706581e70fed3770973ccce503
Reviewed-on: https://review.whamcloud.com/37146
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-12275 sec: add llcrypt as file encryption library 27/38127/6
Sebastien Buisson [Wed, 25 Mar 2020 16:43:33 +0000 (16:43 +0000)]
LU-12275 sec: add llcrypt as file encryption library

Include fscrypt sources from Linux 5.4 into libcfs kernel module.
All fscrypt_ prefixes have been replaced with llcrypt_ to avoid
collision. For the exact transformations carried out, please see
script contrib/scripts/fscrypt_inclusion.sh, and patches under
contrib/scripts/crypto_patches/.

This llcrypt library will be built if kernel provides minimum
encryption support, ie IS_ENCRYPTED and S_ENCRYPTED.
If kernel provides support for encryption policies v2 (included in
Linux 5.4), then llcrypt library will not be built, and in-kernel
fscrypt will be used instead.

To be independent as much as possible from the kernel, llcrypt relies
on a new 'lsi_cop' field added to struct lustre_sb_info, to point to
struct llcrypt_operations, and makes use of the new 'lsi_master_keys'
field to store file system keys. llcrypt also uses i_private field on
struct inode to store file-specific enryption info.

Test-Parameters: trivial
Test-Parameters: clientdistro=ubuntu1804 testgroup=review-ldiskfs
Test-Parameters: clientdistro=el8.1 testgroup=review-ldiskfs
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ide2431b0c3381214aed2e54dd3084132e9545ca8
Reviewed-on: https://review.whamcloud.com/38127
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-10391 socklnd: properly order headers in socklnd.h 87/38287/2
James Simmons [Mon, 20 Apr 2020 17:36:27 +0000 (13:36 -0400)]
LU-10391 socklnd: properly order headers in socklnd.h

We need the proper definitions in inetdevice.h for socklnd.h.
This breaks on newer platforms like Ubuntu18 with

error: implicit declaration of function ‘endfor_ifa’ [-Werror=implicit-function-declaration]

Reversing the order of the headers resolves this build issue.

Fixes: d720fbaada ("LU-10391 socklnd: use interface index to track local addr")
Change-Id: Id8fd27ca118c592b501b3d9241f87b05db3aba70
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/38287
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-6142 fid: Fix style issues for fid_store.c 76/38276/2
Arshad Hussain [Sun, 12 Apr 2020 10:45:53 +0000 (16:15 +0530)]
LU-6142 fid: Fix style issues for fid_store.c

This patch fixes issues reported by checkpatch
for file lustre/fid/fid_store.c

Test-Parameters: trivial
Signed-off-by: Arshad Hussain <arshad.super@gmail.com>
Change-Id: I66c039e18963d2596f681a89c7ea1f30967d8879
Reviewed-on: https://review.whamcloud.com/38276
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
16 months agoLU-6142 fid: Fix style issues for fid_request.c 70/38270/3
Arshad Hussain [Sun, 12 Apr 2020 08:53:51 +0000 (14:23 +0530)]
LU-6142 fid: Fix style issues for fid_request.c

This patch fixes issues reported by checkpatch
for file lustre/fid/fid_request.c

Test-Parameters: trivial
Signed-off-by: Arshad Hussain <arshad.super@gmail.com>
Change-Id: I4530180cefc3f8677ca66a0ada7150a5dbfb1520
Reviewed-on: https://review.whamcloud.com/38270
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
16 months agoLU-6142 build: match lnet comment style to linux net layer 23/38223/4
James Simmons [Tue, 14 Apr 2020 18:31:14 +0000 (14:31 -0400)]
LU-6142 build: match lnet comment style to linux net layer

The network layer of the linux kernel has a comment style of

/* and in the future LNet will be placed in the Linux network
 * subsystem which means this comment format will be required.
 */

Test-Parameters: trivial
Change-Id: Ia3584662954c529b4fb72ab4515f3888d8d64072
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/38223
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-13417 utils: lfs setdirstripe -D -i -1 should work 60/38160/4
Andreas Dilger [Tue, 7 Apr 2020 20:25:33 +0000 (14:25 -0600)]
LU-13417 utils: lfs setdirstripe -D -i -1 should work

Allow "lfs setdirstripe -D -i -1" to work as expected to create remote
directories balanced across MDTs.  Otherwise, this sets a default
stripe count of 0, which results in only local directories created.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I8fb92e352d278b0c2985976df69751309fce7057
Reviewed-on: https://review.whamcloud.com/38160
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
16 months agoLU-13402 target: never exclude MDT/OST from last_rcvd 38/38138/6
Alex Zhuravlev [Mon, 6 Apr 2020 09:32:18 +0000 (12:32 +0300)]
LU-13402 target: never exclude MDT/OST from last_rcvd

as it's supposed to be a mandatory part of recovery process

Fixes: 8bd04b4e5766 ("LU-12722 target: disable recovery for local clients")
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I5a41c500df04dc9eeecd3b94f69e4d2722e63410
Reviewed-on: https://review.whamcloud.com/38138
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
17 months agoLU-12312 lnet: handle no discovery flag 29/38229/3
Amir Shehata [Wed, 15 Apr 2020 00:44:46 +0000 (17:44 -0700)]
LU-12312 lnet: handle no discovery flag

When a peer is being discovered, it could indicate that its
discovery is disabled. In this case we shouldn't flag the
peer for deletion. This operation is reserved for cases when
the peer initiates a push to tell us that it has changed
its status from discovery to no discovery.

Fixes: 4577410165 ("LU-13028 lnet: advertise discovery when toggled")
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: I1d357555e234d828ba19b72fab1d041f93af548e
Reviewed-on: https://review.whamcloud.com/38229
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-13274 uapi: make lustre UAPI headers pass CONFIG_UAPI_HEADER_TEST 15/38215/3
James Simmons [Mon, 13 Apr 2020 21:37:16 +0000 (17:37 -0400)]
LU-13274 uapi: make lustre UAPI headers pass CONFIG_UAPI_HEADER_TEST

The linux kernel offers a CONFIG_UAPI_HEADER_TEST that test
if the UAPI headers can be built stand alone. Running this
kernel option against the latest lustre UAPI headers expose
some few issues still left. This patch makes everything
compliant with this kernel option which should cover potential
build issues with userland applications. A few kernel-only
bits of code have been moved to the kernel headers where they
belong.

Change-Id: I1b287c8226dd1085080218fb92e7b31b32f36b07
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/38215
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Gian-Carlo DeFazio <defazio1@llnl.gov>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-13373 llite: fix to verify write in ll_file_get_iov_count() 88/37988/3
Wang Shilong [Thu, 19 Mar 2020 14:56:07 +0000 (22:56 +0800)]
LU-13373 llite: fix to verify write in ll_file_get_iov_count()

should pass VERIFY_WRITE to access_ok() if this is writing
rather than VERIFY_READ.

Change-Id: I8c6fd6696437e1d2a964728b456ea1df05daa183
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/37988
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Gu Zheng <gzheng@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
17 months agoLU-12214 build: fix make rpms 54/37954/3
Alexey Lyashkov [Tue, 17 Mar 2020 14:05:11 +0000 (17:05 +0300)]
LU-12214 build: fix make rpms

Disable an mpi tests if host don't have any MPI implementation
installed.

libselinux should a soft dependence and rpm able to handle it
internally,

Change-Id: I09eaa825a086dd349699de287222e8dc14d1fd1a
Signed-off-by: Alexey Lyashkov <c17817@cray.com>
Reviewed-on: https://review.whamcloud.com/37954
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shuichi Ihara <sihara@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-13004 lnet: merge lnet_md_alloc into lnet_md_build. 41/37841/8
Mr NeilBrown [Wed, 4 Dec 2019 03:23:58 +0000 (14:23 +1100)]
LU-13004 lnet: merge lnet_md_alloc into lnet_md_build.

lnet_md_alloc is only called twice, each time immediately before a
call to lnet_md_build(), and these are the only calls to
lnet_md_build().

So simplify the code by merging lnet_md_alloc into lnet_md_build.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I13de6d2b225052832d86d4ebe5bf9c2eae945720
Reviewed-on: https://review.whamcloud.com/37841
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-13431 llite: rework preferred IO size 76/38176/2
Sebastien Buisson [Tue, 7 Apr 2020 13:34:50 +0000 (13:34 +0000)]
LU-13431 llite: rework preferred IO size

Instead of using inode->i_blkbits to infer preferred IO size,
just set stat->blksize to the right value in ll_getattr_dentry().

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: If705c7d52bdfabdd3e669ca2d34f8cc0ca1ae08a
Reviewed-on: https://review.whamcloud.com/38176
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
17 months agoLU-12043 llite: fix to submit complete read block with ra disabled 17/35217/11
Wang Shilong [Thu, 13 Jun 2019 00:25:47 +0000 (08:25 +0800)]
LU-12043 llite: fix to submit complete read block with ra disabled

Even if readahead disabled, we need to make first cache-miss read
cover current read. Otherwise, without readahead, we are
always sending 1 page RPC which makes performances really bad.

Benchmarked with following command:

iozone -w -c -i 5 -t1 -j 2 -s 100m -r 1m data

Without patch: 39917.20 kB/sec
read write
pages per rpc         rpcs   % cum % |       rpcs   % cum %
1:      25088 100 100   |          0   0   0

With patch: 754811.62 kB/sec
read write
pages per rpc         rpcs   % cum % |       rpcs   % cum %
1:          0   0   0   |          0   0   0
2:          0   0   0   |          0   0   0
4:          0   0   0   |          0   0   0
8:          0   0   0   |          0   0   0
16:          0   0   0   |          0   0   0
32:          0   0   0   |          0   0   0
64:          0   0   0   |          0   0   0
128:          0   0   0   |          0   0   0
256:         98 100 100   |          0   0   0

We got huge peformances up x17 performances up, this is really
expected behavior without RA since we don't readahead any extra
pages, we just send RPC with one transfer size once rather than
send page by page, this help several cases that i could think of:

1) make more sense to compare RA without RA performances, for example
overhead of RA.

2)we could get better performances with multiple thread read same
file without RA.

Same test with RA enabled: 737746.56 kB/sec
read write
pages per rpc         rpcs   % cum % |       rpcs   % cum %
1:          0   0   0   |          0   0   0
2:          0   0   0   |          0   0   0
4:          0   0   0   |          0   0   0
8:          0   0   0   |          0   0   0
16:          0   0   0   |          0   0   0
32:          0   0   0   |          0   0   0
64:          0   0   0   |          0   0   0
128:          0   0   0   |          0   0   0
256:          0   0   0   |          0   0   0
512:          6  10  10   |          0   0   0
1024:         52  89 100   |          0   0   0

Change-Id: I95511fb371912a47d0b566f64a524f3a7500421c
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/35217
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-6142 fid: Fix style issues for fid_handler.c 71/38271/2
Arshad Hussain [Sun, 12 Apr 2020 08:13:52 +0000 (13:43 +0530)]
LU-6142 fid: Fix style issues for fid_handler.c

This patch fixes issues reported by checkpatch
for file lustre/fid/fid_handler.c

Test-Parameters: trivial
Signed-off-by: Arshad Hussain <arshad.super@gmail.com>
Change-Id: I78291185d592a04e457ad6fd5fb2964c27e08733
Reviewed-on: https://review.whamcloud.com/38271
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
17 months agoLU-13454 lnet: Handle health stats for reply to optimized GET 37/38237/2
Chris Horn [Tue, 14 Apr 2020 19:00:27 +0000 (14:00 -0500)]
LU-13454 lnet: Handle health stats for reply to optimized GET

The LNet messages used for replies to optimized GETs, created via
lnet_create_reply_msg(), are only ever committed for rx. As such,
their msg_txni and msg_txpeer fields are NULL. lnet_incr_hstats()
does not account for this situation, so when passed one of these
messages attempts to deref a NULL pointer.

HPE-bug-id: LUS-8730
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I36b7c46a2d439c5e62f17df64a4f9da8f01090aa
Reviewed-on: https://review.whamcloud.com/38237
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-12506 mdc: clean up code style for mdc_locks.c 17/37917/11
Andreas Dilger [Sat, 14 Mar 2020 02:22:01 +0000 (20:22 -0600)]
LU-12506 mdc: clean up code style for mdc_locks.c

Clean up code style in mdc_locks.c (whitespace, strings)

Run a useful test while we are at it.

Test-Parameters: trivial testlist=runtests mdscount=10 mdtcount=80
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I3e3e0a6a4712e8bc85c93c0a182afa00433ebbe5
Reviewed-on: https://review.whamcloud.com/37917
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-11269 ptlrpc: request's counter in import 22/37722/21
Alex Zhuravlev [Tue, 25 Feb 2020 16:44:18 +0000 (19:44 +0300)]
LU-11269 ptlrpc: request's counter in import

which is separate from imp_refcount as the latter can be
used for other purposes and it's hard to use to track
requests.

to verify the theory that imp_refcount should be checked.

Change-Id: I7c273a73e2b1bb43059c7ed003ee2b7d09273bfe
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/37722
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-10391 lnet: change lnet_sock_connect to take sockaddr 03/37703/6
Mr NeilBrown [Fri, 7 Feb 2020 02:05:57 +0000 (13:05 +1100)]
LU-10391 lnet: change lnet_sock_connect to take sockaddr

Rather than passing a u32 IPv4 address plus port to
lnet_sock_connect() pass a 'struct sockaddr'.  This will allow IPv6
addresses to be requested.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I83ba37bcaade4a84cbce47d93f02043d038c5b29
Reviewed-on: https://review.whamcloud.com/37703
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Aurelien Degremont <degremoa@amazon.com>
17 months agoLU-11025 dne: refactor dir migration 12/37712/7
Lai Siyao [Sat, 15 Feb 2020 13:26:36 +0000 (21:26 +0800)]
LU-11025 dne: refactor dir migration

Tidy up directory migration to make it easier to support
directory restripe.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Ibad46d8cf8f984e5f0062792c651b9753c2fb9a6
Reviewed-on: https://review.whamcloud.com/37712
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
17 months agoLU-9019 obd: express timeouts in timeout_t 76/31576/24
James Simmons [Wed, 22 Apr 2020 14:13:45 +0000 (10:13 -0400)]
LU-9019 obd: express timeouts in timeout_t

In the latest kernels time_t has been removed since time_t has
been a 64 bit value just like time64_t so no need for it anymore.
To avoid confusion between timestamps and timeout values Lustre
has a timeout_t typedef which is in seconds and is a s32 since
timeouts are generally short. This helps to avoid errors that
has happens in the past with certain math operation between
timeouts and timestamps that lead to overflow and underflow
cases.

Change-Id: I4524456d514561e145201079a420ff89fa829602
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/31576
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
17 months agoLU-13424 mdt: fix DOM entry check 57/38257/3
Mikhail Pershin [Thu, 16 Apr 2020 06:22:17 +0000 (09:22 +0300)]
LU-13424 mdt: fix DOM entry check

The mdt_lmm_dom_entry() tries to determine if layout has
DOM entry or not but does that correctly only if DOM entry
is the first one in layout and file is not mirrored.
Otherwise it scans through all entries and wronly reports
that DOM entry exists.

Patch does the following:
- correct initialization of 'has_dom=false' by default
- correct check for DOM+OST case, only if both entries were
  found among initiated entries
- add test case for directory migration with mirrored file

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I43c33528213132f03ebec2debec48b8d53ee229f
Reviewed-on: https://review.whamcloud.com/38257
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
17 months agoLU-10465 lod: adjust component stripe size 61/37661/4
Mikhail Pershin [Fri, 21 Feb 2020 07:07:16 +0000 (10:07 +0300)]
LU-10465 lod: adjust component stripe size

With increased default stripe size to 4MB the problem can
occur when assigning default stripe size to a composite layout
component if stripe size was not specified. There is no strict
requirement to have component boundaries be aligned by default
stripe size, so choosen component end can be not aligned with
system default stripe size, e.g. 3MB end vs 4MB default size.

Patch adds checks for default stripe size assigned to composite
layout components, stripe size should be a multiplier of component
end being the greatest power of two, so check that and reduce it
to the closest valid value when needed.

Fixes: 89693927f0b0 ("LU-8998 lod: accomodate to composite layout")
Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I823307e097fda30f91ff64cc2bc4130d83c1b211
Reviewed-on: https://review.whamcloud.com/37661
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-13349 llite: statahead should skip entry while fid changed 68/37868/5
Yang Sheng [Tue, 10 Mar 2020 17:14:45 +0000 (01:14 +0800)]
LU-13349 llite: statahead should skip entry while fid changed

The fid could change in the Statahead, so the entry should
be skipped in such case.

(lcommon_cl.c:147:cl_file_inode_init()) ASSERTION(
(((inode->i_mode) & 00170000) == 0100000) ) failed:
 kernel: LustreError:(lcommon_cl.c:147:cl_file_inode_init()) LBUG
 kernel: Pid: 106684, comm: ll_sa_94894 3.10.0-957.el7.x86_64 #1
 kernel: Call Trace:
 kernel: [<ffffffffc08587cc>] libcfs_call_trace+0x8c/0xc0 [libcfs]
 kernel: [<ffffffffc085887c>] lbug_with_loc+0x4c/0xa0 [libcfs]
 kernel: [<ffffffffc0e22ba6>] cl_file_inode_init+0x2f6/0x380 [lustre]
 kernel: [<ffffffffc0dfdce5>] ll_update_inode+0x2d5/0x5e0 [lustre]
 kernel: [<ffffffffc0e0afa3>] ll_iget+0x253/0x350 [lustre]
 kernel: [<ffffffffc0dffcf2>] ll_prep_inode+0x262/0x930 [lustre]
 kernel: [<ffffffffc0e1cc2c>] sa_handle_callback+0x37c/0x9d0 [lustre]
 kernel: [<ffffffffc0e1e156>] ll_statahead_thread+0x4d6/0x1c60 [lustre]
 kernel: [<ffffffffa64c1c31>] kthread+0xd1/0xe0
 kernel: [<ffffffffa6b74c1d>] ret_from_fork_nospec_begin+0x7/0x21

Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: Ieee99cc935a111cf05f65b64512b546d2e9b3c74
Reviewed-on: https://review.whamcloud.com/37868
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-13445 tests: enhance ha.sh to support different users 01/38201/2
Elena Gryaznova [Fri, 10 Apr 2020 13:38:53 +0000 (16:38 +0300)]
LU-13445 tests: enhance ha.sh to support different users

Patch adds the ability to run mpi workloads from different users.
The list of users is configured by parameter ha_mpi_users, default
is "mpiuser". Useful for Quota Pools testing.

Test-Parameters: trivial
Cray-bug-id: LUS-8603
Signed-off-by: Elena Gryaznova <elena.gryaznova@hpe.com>
Reviewed-by: Sergey Cheremencev <c17829@cray.com>
Reviewed-by: Vladimir Saveliev <c17830@cray.com>
Change-Id: I848e5eda4e8221ad0edde62f05d5c367d2156966
Reviewed-on: https://review.whamcloud.com/38201
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-13441 osd-zfs: remove OSD thread info cache size assertion 87/38187/2
Sam Atkinson [Thu, 19 Mar 2020 14:14:57 +0000 (10:14 -0400)]
LU-13441 osd-zfs: remove OSD thread info cache size assertion

Remove assertion which kept OSD thread info cache size under 1000.
This assertion is hit when creating a file with more than 511 stripes.

Signed-off-by: Samuel Atkinson <samatk@amazon.com>
Change-Id: If926ec6653cdd85fb654b2960bc0391442bc0674
Reviewed-on: https://review.whamcloud.com/38187
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-13353 quota: add man page for lfs quota 83/38183/5
Gian-Carlo DeFazio [Tue, 7 Apr 2020 01:47:40 +0000 (18:47 -0700)]
LU-13353 quota: add man page for lfs quota

Add a man page for lfs-quota and remove quota
command from lfs man page. References to quotaon,
quotaoff, and quotacheck are removed because
these subcommands no longer exist.

Test-Parameters: trivial
Signed-off-by: Gian-Carlo DeFazio <defazio1@llnl.gov>
Change-Id: I9321b953e072e7fbb4a5f97f5f9033dd4a3a81dc
Reviewed-on: https://review.whamcloud.com/38183
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Tested-by: Maloo <maloo@whamcloud.com>
17 months agoLU-9859 libcfs: make signal-blocking functions inline 81/38081/4
NeilBrown [Wed, 8 Apr 2020 15:15:42 +0000 (11:15 -0400)]
LU-9859 libcfs: make signal-blocking functions inline

cfs_block_sigsinv() and cfs_restore_sigs() are now
simple enough to inline them.

Linux-commit: 6b7936ceefa7ed1d7c0576ed9660cc6cb90d61df

Change-Id: I36c62a091a9c066573930cfbae9bf65836203516
Signed-off-by: NeilBrown <neilb@suse.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-on: https://review.whamcloud.com/38081
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-13375 flr: inherit the correct layout by "mirror extend -N" 79/38079/4
Emoly Liu [Thu, 9 Apr 2020 10:22:28 +0000 (18:22 +0800)]
LU-13375 flr: inherit the correct layout by "mirror extend -N"

For PFL files, if stripe information is not specified to the cmd
"lfs mirror extend -N", the layout of the last initialized comp of
this file should be inherited by the mirror.
For example,
  lcm_mirror_count:  2
  lcm_entry_count:   4
    lcme_id:             65537
    lcme_mirror_id:      1
    lcme_flags:          init
    lcme_extent.e_start: 0
    lcme_extent.e_end:   10485760
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_objects:
      - 0: { l_ost_idx: 2, l_fid: [0x100020000:0x102:0x0] }

    lcme_id:             65538
    lcme_mirror_id:      1
    lcme_flags:          init
    lcme_extent.e_start: 10485760
    lcme_extent.e_end:   20971520
      lmm_stripe_count:  2
      lmm_stripe_size:   2097152
      lmm_objects:
      - 0: { l_ost_idx: 3, l_fid: [0x100030000:0x91:0x0] }
      - 1: { l_ost_idx: 0, l_fid: [0x100000000:0x13d:0x0] }

    lcme_id:             65539
    lcme_mirror_id:      1
    lcme_flags:          0
    lcme_extent.e_start: 20971520
    lcme_extent.e_end:   EOF
      lmm_stripe_count:  -1
      lmm_stripe_size:   4194304

    lcme_id:             131073
    lcme_mirror_id:      2
    lcme_flags:          init
    lcme_extent.e_start: 0
    lcme_extent.e_end:   EOF
      lmm_stripe_count:  2
      lmm_stripe_size:   2097152
      lmm_pool:          archive
      lmm_objects:
      - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x17c:0x0] }
      - 1: { l_ost_idx: 2, l_fid: [0x100020000:0x103:0x0] }

sanity-flr.sh test_0f is modified to verify this patch.

Signed-off-by: Emoly Liu <emoly@whamcloud.com>
Change-Id: If7db3ec7ce9414a04a32e9995ba24f322fb93ab5
Reviewed-on: https://review.whamcloud.com/38079
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-12846 mdd: return error while delete failed 02/36602/28
Yang Sheng [Tue, 29 Oct 2019 06:35:59 +0000 (14:35 +0800)]
LU-12846 mdd: return error while delete failed

Since we use a global buffer, So avoid to replace the
index name while iterate the orphan directory. Also
return error code in mdd_orphan_destroy while dt_delete
failed. Else will cause a dead loop.

Fixes: e1ace3751f ("LU-8514 mdd: transaction failure should be checked")
Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: I6fc3e992333ffa61900074309223555264cfe66b
Reviewed-on: https://review.whamcloud.com/36602
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Artem Blagodarenko <c17828@cray.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-13430 vfs: add ll_vfs_getxattr/ll_vfs_setxattr compat macro 75/38175/2
Sebastien Buisson [Mon, 6 Apr 2020 09:29:10 +0000 (09:29 +0000)]
LU-13430 vfs: add ll_vfs_getxattr/ll_vfs_setxattr compat macro

Instead of checking everywhere in the code whether kernel defines
__vfs_getxattr/__vfs_setxattr/__vfs_removexattr, define compat macros
to handle this in lustre_compat.h.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Iabb54c41e6ccc4b9524f614308a4cd98c5a7ec55
Reviewed-on: https://review.whamcloud.com/38175
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-13425 lfs: support numeric hash type by "lfs migrate -H" 82/38182/8
Emoly Liu [Thu, 9 Apr 2020 11:30:12 +0000 (19:30 +0800)]
LU-13425 lfs: support numeric hash type by "lfs migrate -H"

This patch allows "lfs migrate -H" to use numertic hash type in case
of any error message returned by the server in such a form, e.g.
"run 'lfs migrate -m 1 -c 1 -H 3 dir1' to finish migration".

sanity.sh test_230d is modifed a little to verify this patch and
lfs-migrate.1 and lfs migrate usage is updated to reflect this change.

Signed-off-by: Emoly Liu <emoly@whamcloud.com>
Change-Id: I19cda608b2d28e8e392f03db366767b829ed6dc6
Reviewed-on: https://review.whamcloud.com/38182
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
17 months agoRevert "LU-4801 ldlm: discard l_lock from struct ldlm_lock." 38/38238/2
Oleg Drokin [Wed, 15 Apr 2020 21:40:12 +0000 (21:40 +0000)]
Revert "LU-4801 ldlm: discard l_lock from struct ldlm_lock."

This makes reocurence of LU-11568 back in force. I just did not realize it due to some mail filters.

This reverts commit 0584eb73dbb5b4c710a8c7eb1553ed5dad0c18d8.

Change-Id: I9efe670ab9b9b9f2ea81582fe67feaac668e54d5
Reviewed-on: https://review.whamcloud.com/38238
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-4801 ldlm: discard l_lock from struct ldlm_lock. 83/35483/19
NeilBrown [Tue, 27 Aug 2019 16:13:50 +0000 (12:13 -0400)]
LU-4801 ldlm: discard l_lock from struct ldlm_lock.

This spinlock (l_lock) is only used to stablise the l_resource
pointer while taking a spinlock on the resource.

This is not necessary - it is sufficient to take the resource
spinlock, and then check if l_resource has changed or not.  If it
hasn't then it cannot change until the resource spinlock is dropped.

We must ensure this is safe even if the resource is freed before
lock_res_and_lock() managed to get the lock.  To do this we mark the
slab as SLAB_TYPESAFE_BY_RCU and initialise the lock in an
init_once() function, but not on every allocate (and specifically
don't zero the whole struct on each allocation).
This means that if we find a resource after taking the RCU read lock,
then it is always safe to take and then drop the spinlock.
After taking the spinlock, we can check if it is more generally safe
to use.

Discarding l_lock shrinks 'struct ldlm_lock' which helps save memory.

Change-Id: I2646f198ca60bdbd2e94922bf7679fab31f45c41
Signed-off-by: NeilBrown <neilb@suse.com>
Reviewed-on: https://review.whamcloud.com/35483
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-13423 tests: cleanup_netns correctly set result 57/38157/2
Shaun Tancheff [Tue, 7 Apr 2020 23:05:06 +0000 (18:05 -0500)]
LU-13423 tests: cleanup_netns correctly set result

The existence test for 'test1pl' should not result in
cleanup_netns returning failure to the caller.

A slightly more terse if/else can be used to ensure the
caller is notified of failure only in the case of
test1pl not being deleted.

Test-Parameters: trivial
HPE-bug-id: LUS-8713
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I85dee20ec0f0ccd0be17597431fcedda9469d9da
Reviewed-on: https://review.whamcloud.com/38157
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Aurelien Degremont <degremoa@amazon.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-13418 gnilnd: CFS_ALLOC_PTR_ARRAY typos 48/38148/2
Shaun Tancheff [Tue, 7 Apr 2020 02:01:47 +0000 (21:01 -0500)]
LU-13418 gnilnd: CFS_ALLOC_PTR_ARRAY typos

CFS_FREE_PTRE_ARRAT should be CFS_FREE_PTR_ARRAY
CFS_FREE_PTRE_ARRAY should be CFS_FREE_PTR_ARRAY

Fixes: b1f6f3becedc9 ("LU-9679 libcfs: Add CFS_ALLOC_PTR_ARRAY and free")
Test-Parameters: trivial
HPE-bug-id: LUS-8704
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I0e8c487ebdd436c5a402343ffd0a1733b23af4cb
Reviewed-on: https://review.whamcloud.com/38148
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-7653 lod: fix stripe allocation during recovery 37/38137/3
Alexander Boyko [Mon, 6 Apr 2020 07:04:26 +0000 (03:04 -0400)]
LU-7653 lod: fix stripe allocation during recovery

replay-single 110f fails sometimes because it gets
less directory stripes during replay. lod_prep_md_striped_create()
skips MDT from allocation if MDT is not connected(dt_statfs returns
error).

lod_prep_md_striped_create()) try idx 0, mdt cnt 2, allocated 1
osp_statfs())Process entered
osp_statfs())Process leaving (rc=18446744073709551509 :
 -107 : ffffffffffffff95)
lod_prep_md_striped_create())lustre-MDT0001-mdtlov:
require stripes 2 only get 1

The patch allows striping if lod target is active(fid allocation
is working).

Signed-off-by: Alexander Boyko <c17825@cray.com>
Cray-bug-id: LUS-7550
Change-Id: I23ac3c2561d90a47ec04599818e88054395b7787
Reviewed-on: https://review.whamcloud.com/38137
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-13404 utils: fix lfs mirror duplicate file check 08/38108/2
Andreas Dilger [Tue, 31 Mar 2020 22:45:40 +0000 (16:45 -0600)]
LU-13404 utils: fix lfs mirror duplicate file check

Fix the check_same_file() check to work if the output file does not
exist.  Since the file is opened with O_CREAT|O_EXCL then definitely
it cannot be the same as the input file.

Add test_0j to verify "lfs mirror read -o" and "lfs mirror write -i".

Test-Parameters: trivial testlist=sanity-flr env=ONLY=0j
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ide929b53a528bd0cc77e1f997d3c561c67729179
Reviewed-on: https://review.whamcloud.com/38108
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-13399 lfsck: tune lfsck repair message 00/38100/3
Alexander Zarochentsev [Tue, 3 Mar 2020 20:50:39 +0000 (23:50 +0300)]
LU-13399 lfsck: tune lfsck repair message

namespace LFSCK repair the shard N message printed
even there was nothing to repair which is confusing.

HPE-bug-id: LUS-8548
Signed-off-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Change-Id: I34cc79f59eb76c1f6a86baeed81f8372c133b746
Reviewed-on: https://review.whamcloud.com/38100
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-13258 ptlrpc: bind pinger workqueue to CPT set 91/38091/3
James Simmons [Fri, 27 Mar 2020 16:26:30 +0000 (12:26 -0400)]
LU-13258 ptlrpc: bind pinger workqueue to CPT set

Lustre uses a pinger to determine if nodes are available which is
done using a workqueue. The workqueue can currently run on any
core. This has performance impact on applications but this can be
avoided by binding the workqueues to the cores Lustre as set aside
for its self. Additionally create a proper kernel config option
for enabling the pinger.

Change-Id: I6fefb0f1fd266ef4ac7016f092fe9e7908f010e9
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/38091
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-13394 llite: avoid sending OST RPC for project setting 89/38089/2
Wang Shilong [Fri, 27 Mar 2020 02:05:17 +0000 (10:05 +0800)]
LU-13394 llite: avoid sending OST RPC for project setting

LL_IOC_FSSETXATTR ioctl could be used to handle inode flags
setting, currently it is mainly used to hanlde project ID changing.

The only flags for project is inherit flag, there is no need
to send this to OST, so we could avoid this kind of RPC to OSS.

Change-Id: Ief97959216be807d5efe936eba368d5262a521b0
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/38089
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-12477 kernel: remove dev_read_only handling 70/38070/3
James Simmons [Wed, 25 Mar 2020 15:13:18 +0000 (11:13 -0400)]
LU-12477 kernel: remove dev_read_only handling

To test disk failures Lustre developed a method for ldiskfs to
make the backend block device appears to have failed. Modern
kernels offer the same thing with dm-flakey. Lustre has moved
to dm-flakey for some time so the old dev_read_only can now be
removed.

Change-Id: I3d5887ab8c0daf1c94c9069bcdd58786b2819ac0
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/38070
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
17 months agoLU-13005 lnet: Support checking for MD leaks. 59/38059/4
Mr NeilBrown [Wed, 25 Mar 2020 05:26:28 +0000 (16:26 +1100)]
LU-13005 lnet: Support checking for MD leaks.

Since we dropped the refcounting on LNetEQ we no longer get
confirmation that all MDs for a given handler a gone by the
time they should be.

So add lnet_assert_handler_unused() which searches the per-cpt
containers and ensures there are no MDs for a given handler, and call
that are the same place that we used to call LNetEQFree().

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ic04e5b4f67a79055c49994bd1d92e275fa71dd4b
Reviewed-on: https://review.whamcloud.com/38059
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-13156 tests: wait for nodemap update in sanity-selinux 34/38034/4
Sebastien Buisson [Mon, 23 Mar 2020 15:23:57 +0000 (16:23 +0100)]
LU-13156 tests: wait for nodemap update in sanity-selinux

In sanity-selinux test_21a and test_21b, nodemaps are used to test
SELinux status checking (sepol).
We must wait for nodemap update on all MDS nodes before carrying out
tests.

Test-Parameters: clientselinux mdscount=2 mdtcount=4 testlist=recovery-small,sanity-selinux env=ONLY="21 23",ONLY_REPEAT=100
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I363e2bec757efc199f7039f8af4bcb77e2a2a184
Reviewed-on: https://review.whamcloud.com/38034
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-13383 ofd: lazy atime update 24/38024/21
Alex Zhuravlev [Tue, 31 Mar 2020 08:06:03 +0000 (11:06 +0300)]
LU-13383 ofd: lazy atime update

OST_BRW_WRITE updates atime both in memory and on-disk
OST_BRW_READ updates atime in memory and once difference
exceeds delay (obdfilter.*.atime_delay seconds) - it's
updated on-disk.

Test-Parameters: testlist=sanity env=ONLY=39r,ONLY_REPEAT=100
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Ibe49882ec6f1984f9cf6a32f6ee9fef579ed2a03
Reviewed-on: https://review.whamcloud.com/38024
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
17 months agoLU-13347 kernel: RHEL 7.8 server support 48/37948/5
Jian Yu [Fri, 3 Apr 2020 17:55:07 +0000 (10:55 -0700)]
LU-13347 kernel: RHEL 7.8 server support

This patch makes changes to support new RHEL 7.8 release
for Lustre server (kernel 3.10.0-1127.el7).

Test-Parameters: trivial clientdistro=el7.8 serverdistro=el7.8

Change-Id: I4817fd2f9512111aa7d26109454104945fd2778f
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/37948
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-13347 kernel: new kernel [RHEL 7.8 3.10.0-1127.el7] 12/37912/5
Jian Yu [Thu, 2 Apr 2020 00:32:19 +0000 (17:32 -0700)]
LU-13347 kernel: new kernel [RHEL 7.8 3.10.0-1127.el7]

This patch makes changes to support new RHEL 7.8 release
for Lustre client.

Test-Parameters: trivial clientdistro=el7.8

Change-Id: I82f89495d5ab1d46a539a016a899307d7c8f37b7
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/37912
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-13004 lnet: don't embed whole lnet_md in lnet_event 40/37840/7
Mr NeilBrown [Mon, 6 Apr 2020 16:59:35 +0000 (12:59 -0400)]
LU-13004 lnet: don't embed whole lnet_md in lnet_event

Of the several fields that are copied into the lnet_md which is
embedded in an lnet_event, only three are ever used:
  start, user_ptr, md_options

Rather than allocating space for the whole structure, copying only
some in, and then using even fewer; just store the fields that are
needed.

Note that a comment suggests that the "threshold" value is
interesting, but this value is never used.  The comment has been
removed.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I7a69b52a65dda815b533c605f581e22fc4445be0
Reviewed-on: https://review.whamcloud.com/37840
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-13309 osd-ldiskfs: speedup osd_bufs_get/put 86/37786/4
Andrew Perepechko [Fri, 28 Feb 2020 15:53:16 +0000 (18:53 +0300)]
LU-13309 osd-ldiskfs: speedup osd_bufs_get/put

Let's remove useless page processing
in osd_get_put/osd_put_page that affects regular I/O
in non-cached mode.

Change-Id: I1972cfdec940f104c1d66e1b1ffb4faa6007929b
Signed-off-by: Andrew Perepechko <c17827@cray.com>
Cray-bug-id: LUS-8004
Reviewed-on: https://review.whamcloud.com/37786
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-13005 lnet: remove 'eq' from names, use 'handler' 78/37378/6
Mr NeilBrown [Fri, 31 Jan 2020 00:16:20 +0000 (11:16 +1100)]
LU-13005 lnet: remove 'eq' from names, use 'handler'

Now that we don't have an event-queue subsystem, using
'eq' in various names is confusing.
We have events and event handlers, so change
  lnet_eq_handler_t to lnet_handler_t
and change variable names mentioned 'eq' to just
mention 'handler'.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: If6f34ed41c565cb37ac798b5ba0411b6a471d6d0
Reviewed-on: https://review.whamcloud.com/37378
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-13169 tests: quiet debug messages during tests 38/37338/11
Andreas Dilger [Tue, 28 Jan 2020 00:29:09 +0000 (17:29 -0700)]
LU-13169 tests: quiet debug messages during tests

Quiet the output of check_network() and check_node_health() that are
printed for *every* subtest that is run, since this is distracting
when looking at the test output, and can often be more verbose than
the actual test output.

Don't get the client and server versions repeatedly, since they will
not change for the duration of a test run and can be cached instead.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ib42b01c9ca36c6c72d2c7ffffbd3a5f2643ebbe5
Reviewed-on: https://review.whamcloud.com/37338
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-13005 lnet: remove lib-eq. 47/36847/8
Mr NeilBrown [Wed, 20 Nov 2019 02:56:44 +0000 (13:56 +1100)]
LU-13005 lnet: remove lib-eq.

There is nothing of value in left in lib-eq.  An 'lnet_eq' just holds
a function pointer.
So change all 'struct lnet_eq' pointers to function pointers, and
bypass the "event queues".

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I16786e44ab712a76860fe8d83543a292555cf557
Reviewed-on: https://review.whamcloud.com/36847
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>