LU-9114 ldlm: don't compute sumsq for pool stats

Remove the calculation of sumsq from the LDLM pool stats, since
these stats are almost never used, while conversely the pools
are updated frequently.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I90612b613fb76f4500ca6cd5613f5c121b3ebbe5
Reviewed-on: https://review.whamcloud.com/39435
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14100 gnilnd: fix kgnilnd_setup_immediate_buffer

struct kvec * removed from kgnilnd_setup_immediate_buffer

Fixes: c87f41dd20 ("LU-13004 gnilnd: discard struct kvec arg.")
Test-Parameters: trivial
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: Ie3c511326af6ddb09b6bfa13bb779fc094ba30ec
Reviewed-on: https://review.whamcloud.com/40504
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9215 build: Re-add the lustre-devel package

Traditonally on rpm systems a special devel package exist that
contains userland headers and symlinks for the libraries. This
patch creates such a package.

The introduction of this new package changes the dependency
of lustre-test packages since the user land headers are tested
to make sure they compile for user land applications. Update
the dpk package as well.

Lastly we add pkg-config support to Lustre as well. This means
that if do configure --prefix=special_place it will not break
external applications if they use pkg-config.

Change-Id: I3e856d85239d1712e21f7ecc633d9fae569238e6
Test-Parameters: trivial
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/40728
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14205 osd-ldiskfs: return correct error after end io

@bi_status could be positive in newer kernel, we need use
blk_status_to_errno() helper to convert error properly.

Otherwise, a positive value could be returned to client
which might trigger unexpected error and make client
fail to resend.

Test-Parameters: clientdistro=el8.3 serverdistro=el8.3
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I2e8f257814eff311b72416788b7f6d9e37bad84a
Reviewed-on: https://review.whamcloud.com/40944
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>

LU-14149 llite: fiemap set flags for encrypted files

FIEMAP ioctl needs to set
FIEMAP_EXTENT_DATA_ENCRYPTED|FIEMAP_EXTENT_ENCODED flags for all
extents of files encrypted by fscrypt.

Also add sanity-sec test_56 to exercise the flags, with just a
plain file layout for now.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I26f788b5c4b37e96de57334f48847b40886e14c2
Reviewed-on: https://review.whamcloud.com/40852
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>

LU-14208 utils: remove copy_file_range() usage from copytool

Revert the lhsmtool_posix part of "LU-11621 utils: optimize
lhsmtool_posix with copy_file_range()". This is failing sanity-hsm
test_55() on RHEL 8.3.

Test-Parameters: clientdistro=el8.3 serverdistro=el8.3 testlist=sanity-hsm
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I0b410c2bb5f56a32c10f0ec39190735908cbb556
Reviewed-on: https://review.whamcloud.com/40966
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-10664 dom: non-blocking enqueue for DOM locks

DOM lock enqueue waits for blocking locks on MDT due to
ATOMIC flag, so MDT thread is blocked until lock is granted.
When many clients attempt to write to shared file that may
cause server thread starvation and lock contention. Switch
to non-atomic lock enqueue for DOM locks.

- switch IO lock to non-intent enqueue, so it doesn't consume
  server thread for a long time being blocked
- on client take LVB from l_lvb_data updated by completion AST and
  update l_ost_lvb used by DoM
- make glimpse performing similarly on MDT and OST, it uses one
  format with no intent buffer and return data in LVB buffer
- introduce new connect flag 'dom_lvb' for compatibility reasons
- on server handle glimpse for both old and new clients by filling
  either LVB reply buffer or mdt_body buffer
- don't take RPC slot for a DOM enqueue like it is done for EXTENT
  locks, update ldlm_cli_enqueue_fini() to accept ldlm_enqueue_info
  as parameter
- check that there is no atomic local lock issued with mandatory DOM
  bit, trybits should be used

Test-Parameters: mdssizegb=20 testlist=dom-performance
Test-Parameters: testlist=sanity-dom clientversion=2.12
Test-Parameters: testlist=sanity-dom serverversion=2.12
Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I06573018126f0663a8f76dce07e126a1879cfba5
Reviewed-on: https://review.whamcloud.com/36903
Reviewed-by: Vitaly Fertman <vitaly.fertman@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14199 sec: find policy version in use for sepol

SELinux exports the maximum kernel policy version that can be used.
When building SELinux status checking representation 'sepol', we need
to look for all possible versions of the policy, not only the max one.

Test-Parameters: clientdistro=el8.3 serverdistro=el8.2 testgroup=review-dne-selinux
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Iae4b66403ce953e5a7c0df585900713c597ff033
Reviewed-on: https://review.whamcloud.com/40918
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14095 gss: use RCU protection for sunrpc cache

Leverage sunrpc cache RCU protection for cache entry removal,
if the kernel supports it.

Test-Parameters: clientdistro=el8.2 serverdistro=el8.2 testgroup=review-dne-ssk
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I50f9dc9e9dc434b502df1e1013215b5506289ff9
Reviewed-on: https://review.whamcloud.com/40914
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14186 lov: instantiate components layout for fallocate

fallocate() need send intent lock to MDS to instantiate layout
like PFL.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: Ie14574ca0981ea00aa1867489d2a40b75eb511cc
Reviewed-on: https://review.whamcloud.com/40885
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-13783 libcfs: don't lookup symbol when not needed

Linux 5.8 and later do not export kallsyms_lookup_name() so we need to
remove dependence on it.
Looking up radix_tree_node_cachep is not needed in these kernels,
or any kernel were HAVE_XARRAY_SUPPORT is defined.
So avoid the lookup in those cases.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ia61fe9a025653985b7dcf67b1234d32667719956
Reviewed-on: https://review.whamcloud.com/40825
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Aurelien Degremont <degremoa@amazon.com>

LU-13855 tests: remove the need for libiberty

The statx.c test program uses a bunch of GNU-specific xstr*
functions that have very little, if any, benefit to the code.
They allocate copies of static format strings, but those
strings are not modified, and are never freed.  Since the
use of xstr* drags in libiberty and associated build issues
on a variety of platforms, instead avoid those functions.

Remove the use of this library completely from the statx.c
test tool, with minimal replacements to keep the build happy.
Remove the corresponding configure checks.

Fixes: 3f7853b31ef6 ("LU-10934 llite: integrate statx() API with Lustre")
Test-Parameters: trivial testlist=sanityn env=ONLY=106 clientdistro=ubuntu2004
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I2c11bac8e2a4b4754ee78caabfa9cbdba98bffff
Reviewed-on: https://review.whamcloud.com/40802
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14151 gss: use skcipher when sync_skcipher not available

Linux 4.2 introduced support for skcipher in commit
7a7ffe65c8c5 crypto: skcipher - Add top-level skcipher interface

Based on the existence of crypto_alloc_skcipher, use the skcipher API
instead of the blkcipher API, in case the sync_skcipher API is not
available.

Test-Parameters: clientdistro=el8.2 env=SHARED_KEY=true,SK_FLAVOR=skpi mdscount=2 mdtcount=4 osscount=1 ostcount=8 clientcount=2 testlist=sanity
Test-Parameters: clientdistro=el7.8 env=SHARED_KEY=true,SK_FLAVOR=skpi mdscount=2 mdtcount=4 osscount=1 ostcount=8 clientcount=2 testlist=sanity
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I4e62cd5c8219a215c5528f81d525c054215073e3
Reviewed-on: https://review.whamcloud.com/40768
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14143 lov: fix SEEK_HOLE calcs at component end

If data ends exactly at component end then LOV assumed that
is not yet hole in file and the next component will take care.
Meanwhile there can be no next component initialized yet if file
ends exactly at component boundary, so no hole offset is returned
but error

Patch fixes that issue. If component reports hole offset at
component end then it is saved to be used as result when no
other components report valid hole offset.

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I8bffd8b76546b560a3d34034701eea4963f41593
Reviewed-on: https://review.whamcloud.com/40713
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-14181 tests: except sanity test_64e 64f with SHARED_KEY

Add sanity test_64e and test_64f to ALWAYS_EXCEPT when
SHARED_KEY is used.

Test-Parameters: trivial
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Iaa9f5038a59f9ddc50dd9ac81ca81effd8bb9b1b
Reviewed-on: https://review.whamcloud.com/40865
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14121 nodemap: do not force fsuid/fsgid squashing

In the current implementation, if the real uid is squashed, then the
fsuid is similarly squashed, no matter what is the value of the
effective uid.
This squashing is a little bit too strict, and we should instead trust
mapped fsuid and fsgid values.

Also add euid_access test program and sanity-sec test_55 to verify
the issue is fixed.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Iecaecac5054b105cd42206b0a9a3868cde0269b4
Reviewed-on: https://review.whamcloud.com/40645
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14095 ssk: default rounds of Miller-Rabin for DH_check

OpenSSL 1.1.1c increased the number of rounds used for Miller-Rabin
testing of the prime provided as input parameter to DH_check(). This
makes the check roughly x10 longer, and can lead to request timeouts
when an SSK flavor is being used.

Instead, use a dynamic number of rounds based on the speed of the
check, evaluated when the lsvcgssd daemon starts. If DH_check()
runtime is fine, just use it instead of our own check.

Test-Parameters: clientdistro=el8.2 serverdistro=el8.2 testgroup=review-dne-ssk
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Id392cdd76ede196094b146c68d230bc52852aa34
Reviewed-on: https://review.whamcloud.com/40686
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12901 o2iblnd: retry qp creation with reduced queue depth

If negotiated number of frags * queue depth is too large for
successful qp creation, reduce the queue depth in a loop
until qp creation succeeds or the queue depth dips below 2.
Remember the reduced queue depth value to use for later
connections to the same peer.

Test-Parameters: trivial
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: Iaa91510d6f80d813218a06a9bc52f5f9251e8b87
Reviewed-on: https://review.whamcloud.com/40748
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

New tag 2.13.57

Change-Id: I931c35e61fe13760586d3402afa729bce029384e
Signed-off-by: Oleg Drokin <green@whamcloud.com>

LU-14042 llite: don't check layout info for page discard

The CIT_MISC+ignore_layout is indicating locks/pages manipulation
from the OSC layer, it does not care/access lov layout related info.

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Ibdf22f0d47712d370385af4c6c984052b91fc7c6
Reviewed-on: https://review.whamcloud.com/40267
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>

LU-13839 kernel: RHEL 8.3 server support

This patch makes changes to support RHEL 8.3 release with
kernel 4.18.0-240.1.1.el8_3 for Lustre server.

Test-Parameters: trivial fstype=ldiskfs \
clientdistro=el8.3 serverdistro=el8.3 testlist=sanity

Test-Parameters: trivial fstype=zfs \
clientdistro=el8.3 serverdistro=el8.3 testlist=sanity

Change-Id: I2aa2efaf86b5f244fcb1dd95be1d0bfe04e8d245
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/40729
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14170 build: Update ZFS version to 2.0.0

Major New Features
* Sequential resilver
* Persistent L2ARC
* ZStandard compression
* Redacted zfs send/receive

Notable Changes
* Added fallocate(mode-0/2) compatibility to preallocate space.
* Reorganized the zfs and zpool man pages by splitting out each
  subcommand in to its own page.
* Enabled the systemd zfs-mount-generator by default on Linux.
* More relevant and useful ZED syslog entries.
* Provided pam module for automatically loading zfs encryption
  keys for home datasets.
* Support for inheriting and setting user properties in channel
  programs.
* Improved bootloader support.
* Optionally colorized zpool status output.

Change-Id: Ia00d46a4cb54be94d4fdc0145a2d474ea987cf74
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/40822
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13826 utils: fix compatibility for LL_IOC_MDC_GETINFO

The landed patch "LU-11367 som: integrate LSOM with lfs find"
uses "LL_IOC_MDC_GETINFO_OLD", so while the IOCTL number/structs
are ABI compatible, it is not API compatible and applications
using for the header including the definition LL_IOC_MDC_GETINFO
is broken.

This patch defines versioned IOCTL number: LL_IOC_MDC_GETINFO_V1,
LL_IOC_MDC_GETINFO_V2. Then we can use the explicitly verioned
constrants everywhere for the in-tree code, and declare
LL_IOC_MDC_GETINFO in a compatible way, but external applications
can select the version that they want explicitly.

And this patch does the same fix for IOC_MDC_GETFILEINFO.

Fixes: 11aa7f8704c4 ("LU-11367 som: integrate LSOM with lfs find")
Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I39e0273c4a7331ec782de3e8686b8a756e56beec
Reviewed-on: https://review.whamcloud.com/40858
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13994 ldlm: osc_object_ast_clear() is called for mdc object on eviction

Replace osc_object_prune() with cl_object_prune()

PID: 3477   TASK: ffff9360d82fa0e0  CPU: 0   COMMAND: "ll_imp_inval"
 #0 [ffff9360d5c5b990] machine_kexec at ffffffff86865704
 #1 [ffff9360d5c5b9f0] __crash_kexec at ffffffff869209a2
 #2 [ffff9360d5c5bac0] panic at ffffffff86f7294c
 #3 [ffff9360d5c5bb40] lbug_with_loc at ffffffffc04b78cb [libcfs]
 #4 [ffff9360d5c5bb60] osc_object_ast_clear at ffffffffc0956471 [osc]
 #5 [ffff9360d5c5bbc8] ldlm_resource_foreach at ffffffffc07e2fd6 [ptlrpc]
 #6 [ffff9360d5c5bc08] ldlm_resource_iterate at ffffffffc07e3266 [ptlrpc]
 #7 [ffff9360d5c5bc38] osc_object_prune at ffffffffc0956140 [osc]
 #8 [ffff9360d5c5bc58] osc_object_invalidate at ffffffffc0956e12 [osc]
 #9 [ffff9360d5c5bcd0] osc_ldlm_resource_invalidate at ffffffffc09477bf [osc]

Change-Id: I83fd02feb3f5559dfe7b93647429f8eb57c7ad0e
HPE-bug-id: LUS-8399
Signed-off-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Alexander Boyko <c17825@cray.com>
Reviewed-by: Vitaly Fertman <c17818@cray.com>
Tested-by: Alexander Lezhoev <c17454@cray.com>
Reviewed-on: https://review.whamcloud.com/40052
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Vitaly Fertman <vitaly.fertman@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10262 mdt: mdt_reint_open: check EEXIST without lock

Many applications blindly open files with O_CREAT, and the mds gets a
write lock to the parent directory for these even if the file already
exists.
Checking for file existence first lets us take a PR lock if file
already existed even if O_CREAT was specified.

This opens up multiple races between the first lookup and the actual
locking, in each of them drop the resources we aquired and retry from
scratch to keep things as far from complicated as possible, with mixed
success.

Update (eaujames):
 - rebase the patch
 - update tests

Performance tests results:

The array below presents the average "open" syscall latency for 20
files in a single directory accessed by 400 different clients.
 _______________________________________________________________
| Test cases        | without patch | with patch | %improvement |
|___________________|_______________|____________|______________|
| readonly          | 0.960s        | 0.973s     | -1.40%       |
|___________________|_______________|____________|______________|
| readonly cached   | 0.372s        | 0.372s     | +0.01%       |
|___________________|_______________|____________|______________|
| O_CREAT+precreate | 1.645s        | 0.968s     | +41.13%      |
|___________________|_______________|____________|______________|
| O_CREAT cached    | 0.632s        | 0.623s     | +1.34%       |
|___________________|_______________|____________|______________|
| O_CREAT           | 1.261s        | 1.093s     | +13.32%      |
|___________________|_______________|____________|______________|
(for more detail, see the ticket comments section)

This patch optimizes concurent opens with O_CREAT flag when dentry are
not cached by clients.

Change-Id: I247b579d14d20036f89033c99ece457d70ba19e7
Signed-off-by: Dominique Martinet <dominique.martinet@cea.fr>
Reviewed-on: https://review.whamcloud.com/33098
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Etienne AUJAMES <eaujames@ddn.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14172 lmv: optimize dir shard revalidate

mdt_is_remote_object() will check whether child is directory shard
if parent and child are on different MDTs, which needs to read LMV
from disk, and hurt striped directory stat performance.

This can be optimized, client can just set CROSS_REF flag to do a
cross reference getattr, which avoids lots of checks.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Ib2d5a510b27c90a26f979f9cccfd40948e32d91a
Reviewed-on: https://review.whamcloud.com/40863
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>

LU-14155 mdc: avoid easize set to 0

The cl_default_mds_easize could be set to 0 in some case. Then
check it before package.

Fixes: 3ec712bd183a ("LU-11868 osd: Set max ea size to XATTR_SIZE_MAX")
Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: Icc045bfba345a61dd7904bfc7772930fd737334e
Reviewed-on: https://review.whamcloud.com/40785
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-5855 tests: enable skipped sanity-lfsck DNE ZFS tests

LFSCK has been enabled for DNE striped directories on ZFS for
some time, but the skipped tests have never been enabled for
this configuration.  Re-enable them for ZFS backends.

Fixes: 26995a3588e7 ("LU-5855 lfsck: misc fixes for zfs-based backend")
Test-Parameters: trivial fstype=zfs testlist=sanity-lfsck
Test-Parameters: fstype=zfs mdscount=2 mdtcount=4 testlist=sanity-lfsck
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I33be173c98faa61311ab2b435555cf49e1e2bacc
Reviewed-on: https://review.whamcloud.com/40761
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14073 ptlrpc: remove debugging assert using segment_eq()

In Linux 5.9-rc1 commit 428e2976a5, segment_eq() is removed as it was
only used in uaccess_kernel(). But uaccess_kernel() was introduced in Linux
v4.12-rc1, which does not cover all supported kernel versions.

As segment_eq() is only used in 1 actual debugging call, avoid adding a
new configuration test for uaccess_kernel() and just remove the 2
related LASSERTs.

Test-Parameters: trivial
Signed-off-by: Aurelien Degremont <degremoa@amazon.com>
Change-Id: I1595e9b085176d2106a2b958e1fd6f4031cc2f69
Reviewed-on: https://review.whamcloud.com/40797
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Neil Brown <neilb@suse.de>

LU-14159 build: fix gcc8 warnings on kthread_run calls

Fixes warnings from gcc8 about kthread_run calls with no format
parameter.

Signed-off-by: Quentin Bouget <quentin.bouget@cea.fr>
Signed-off-by: Sebastien Piechurski <sebastien.piechurski@atos.net>
Change-Id: I30a68a1d0e318c98f2da547ece8018564ced69c0
Reviewed-on: https://review.whamcloud.com/40790
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14154 ldiskfs: gcc10 support

gcc10 complains about uninitialized fields:
ldiskfs/extents.c: In function ‘ldiskfs_ext_map_blocks’:
ldiskfs/ldiskfs_extents.h:214:12: error: 'zero_ex2.ee_start_hi' may
be used uninitialized in this function [-Werror=maybe-uninitialized]

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I17ef56352194f6ad76bf6f08dab0be3e28d0323c
Reviewed-on: https://review.whamcloud.com/40776
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14043 lfs: use buffered IO in mirror extend

In lfs_mirror_extend() use buffered IO to read from the source
file. Mirror resync will continue to use direct IO on the source.
Call posix_fadvise() at the end of mirroring to discard cached data
from the old and new mirrors.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I7703ae9e30cf30a7e6fbf274114d5986a80d4d3a
Reviewed-on: https://review.whamcloud.com/40275
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Patrick Farrell <farr0186@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13569 lnet: Add lnet_recovery_limit to lnetctl

Allow setting/reading lnet_recovery_limit via lnetctl.

Test-Parameters: trivial
HPE-bug-id: LUS-9109
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I5aac297bad15e43a52d8b8531da08a1d3f559bea
Reviewed-on: https://review.whamcloud.com/39717
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13569 lnet: Introduce lnet_recovery_limit parameter

This parameter controls how long LNet will attempt to recover an
unhealthy interface.

Defaults to 0 to indicate indefinite recovery. This maintains the
current behavior.

Test-Parameters: trivial
HPE-bug-id: LUS-9109
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I2f7897d9a293f0979f7402de2b91e160c77790d1
Reviewed-on: https://review.whamcloud.com/39716
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13669 llite: try to improve mmap performance

We have observed slow mmap read performances for some
applications. The problem is if access pattern is neither
sequential nor stride, but could be still adjacent in a
small range and then seek a random position.

So the pattern could be something like this:

[1M data] [hole] [0.5M data] [hole] [0.7M data] [1M data]

Every time an application reads mmap data, it may not only
read a single 4KB page, but aslo a cluster of nearby pages in
a range(e.g. 1MB) of the first page after a cache miss.

The readahead engine is modified to track the range size of
a cluster of mmap reads, so that after a seek and/or cache miss,
the range size is used to efficiently prefetch multiple pages
in a single RPC rather than many small RPCs.

Benchmark:
fio --name=randread --directory=/ai400/fio --rw=randread
--ioengine=mmap --bs=128K --numjobs=32 --filesize=200G
--filename=randread --time_based --status-interval=10s
--runtime=30s --allow_file_create=1 --group_reporting
--disable_lat=1 --disable_clat=1 --disable_slat=1
--disk_util=0 --aux-path=/tmp --randrepeat=0
--unique_filename=0 --fallocate=0

               |   master  |   patched  |  speedup  |
---------------+-----------+------------+-----------+
page_fault_avg |   512usec |    52usec  |  9.75x
page_fault_max |  37698usec|    6543usec|  5.76x

Change-Id: I86436cbd027ec6df5094599e54b4acfd0e018930
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/38916
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13368 lnet: discard the callback

Lustre need a completion callback for event that request
has been sent. And then need other callback when reply
arrived. Sometime the request completion callback maybe
lost by some reason even reply has been received.
system will wait forever even timeout. We needn't to wait
request completion in such case. So provide a way to
discard the callback.

Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: If9cd8420ee76947ee5053180e0f5219f76bb94c2
Reviewed-on: https://review.whamcloud.com/38845
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13170 osd: positive lookup shouldn't return an error

because of stale OI, otherwise LFSCK has no chance to
fix direntry. lookup should initiate OI scrubber and
give it a hint about FID->dnode mapping.

This should fix sanity-lfsck/23b failing with ZFS often.

Test-Parameters: fstype=zfs testlist=sanity-lfsck env=ONLY=23b,ONLY_REPEAT=200
Change-Id: Iea826de9995e34939eff2d67a831deb6b192bd4d
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/37322
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>

LU-9555 quota: df should return projid-specific values

With local ext4 and XFS filesystems, it is possible to use
"df /path/to/directory" (statfs()) to return the current
project quota usage for that directory as "used", and
min(projid quota limit, free space) as "total".

statfs() is a natural interface for users/applications, since
it represents the used/maximum space for that subdirectory.
Otherwise, the user will get EDQUOT back when the project
quota runs out for that directory and applications will not
be able to figure out how much data they could write into
that directory.

Change-Id: I7f357e6e4de6a1465a48c2c08c4aa86c4c848dbb
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/36685
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>

LU-13543 lustre: update *pos in seq_file .next functions

Patch fixes mdt_hsm_actions_debugfs seq operations to work with
new seq file logic in newer kernels

Meanwhile it finishes remaining work originating in LU-13985 which
updates ->next functions to always increment *pos as it is expected
by kernel

Test-Parameters: clientdistro=el8.2 serverdistro=el8.2 testlist=sanity-hsm
Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I4f077b633fbf7d420bfda10cce6b0141cef07761
Reviewed-on: https://review.whamcloud.com/40757
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Neil Brown <neilb@suse.de>

LU-13980 osd-ldiskfs: print label instead of device

Print out the volume label (e.g. fsname-OST0000) in error messages
instead of the device name (e.g. dm-2) so that it is easier to
correlate this with other messages in the log.

Fix error strings to be on one line to match code style.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Iff31bc962ca32b1f3a13f579700d03835a3ebbe5
Reviewed-on: https://review.whamcloud.com/40738
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14140 osd: don't panic meeting OI dups

instead dump all info (FID->ino, LMAs) and return an error

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I4c0c7ad12cc808617c748b0a720323a968668fc9
Reviewed-on: https://review.whamcloud.com/40666
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14134 osd-ldiskfs: reduce credits for new writing

For new writing, Lustre reserve credits in worst cases,
eg for 1M write, we need reserve more than 500 credits.

This means we need much more journal space than data
blocks for each new writing thus a large journal needed
to reach good performance.

This patch try to improve the case by allowing transaction
to be restarted, only reserve few extent credits during
declare phase. Before every block allocation, one extent
insertion credits must be guaranteed, otherwise transaction
will be restarted.

A simple dd test of following in 10G device shows credits
removed from 260+ to 7 credits:

 $ dd if=/dev/zero of=data bs=1M count=1 oflag=direct

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: Ide541cbd72d8d4aa286eb8b84ca5ce09c3b5f8b6
Reviewed-on: https://review.whamcloud.com/40626
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14130 osc: skip 0 row for rpc_stats

Fix the rpc_stats statistic it should not print
0 row as it makes nosense.

Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: Id3534bb43b5453daeed4db078c18912c4cc2f51e
Reviewed-on: https://review.whamcloud.com/40613
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>

LU-14105 lfsck: don't LBUG() on disk data

lfsck_namespace_assitant_handler_p1() will LBUG() if file FID is
zero, and file name is not "..", but generally lfsck shouldn't LBUG()
on disk data. The correct way is to treat such file as bad dirent and
remove it, which is done below.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Ia4092ada5d55583508f276e5c9d59920ec31a33b
Reviewed-on: https://review.whamcloud.com/40524
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14043 llite: allow DIO with unaligned IO count

DIO only require user buffer page aligned and
IO offset page aligned, it is ok that io count is
not page aligned, remove this unnecessary limit
so that we could use DIO with file not aligned
with PAGE SIZE.

Change-Id: Iaf1e224b3f180f91d1b54016c6e6b7adaf674bf8
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/40392
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13474 gss: do not return -ERESTART when gss rpc times out

When a client receives an RPC time out while it tries to negotiate
a GSS context with the servers, it must not return -ERESTART.
Otherwise, the client would keep on sending GSS negotiation requests
to the same peer, instead of letting the HA failover mechanism try
different service nodes.

Test-Parameters: trivial
Test-Parameters: env=SHARED_KEY=true mdscount=2 mdtcount=4 osscount=1 ostcount=8 clientcount=2 testlist=sanity,recovery-small,sanity-sec
Test-Parameters: env=SHARED_KEY=true mdscount=2 mdtcount=4 osscount=1 ostcount=8 clientcount=2 testlist=sanity,recovery-small,sanity-sec
Test-Parameters: env=SHARED_KEY=true mdscount=2 mdtcount=4 osscount=1 ostcount=8 clientcount=2 testlist=sanity,recovery-small,sanity-sec
Test-Parameters: env=SHARED_KEY=true mdscount=2 mdtcount=4 osscount=1 ostcount=8 clientcount=2 testlist=sanity,recovery-small,sanity-sec
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ifc9cc02ea3ddefe6c5095cbb464a054c9027c920
Reviewed-on: https://review.whamcloud.com/38310
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12353 ldiskfs: speedup quota journalling

ldiskfs_mark_dquot_dirty() need not call
ldiskfs_write_dquot() if the dquot is already
dirty since we are guaranteed that another
thread is already writing the dquot and
will update the buffer with the latest
dquot data.

Change-Id: Ia65c5987bbb0d6d2ac94c5499cb8b51edc3c49c3
Signed-off-by: Andrew Perepechko <c17827@cray.com>
Cray-bug-id: LUS-5342
Reviewed-on: https://review.whamcloud.com/34992
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14095 gss: use hlist_unhashed() instead of ->next

In cache_detail list-mutation primitives, verifying the status of an
entry must be done using hlist_unhashed(), in case 'struct cache_head'
has a 'cache_list' field.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I1410eca9a647b74127cf40b8f3d6b68d055f773a
Reviewed-on: https://review.whamcloud.com/40514
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-12960 lod: don't set index for 2nd stripe if specific

When MDTs are specific, don't set index for the second stripe
allocation, otherwise it's not created on specific MDT.

Add sanity 31q.

Fixes: c1d0a355a6 ("LU-12624 lod: alloc dir stripes by QoS")
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I415be0fa7c6e702ec72dcaac88bba55290463d44
Reviewed-on: https://review.whamcloud.com/36735
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13571 lnd: Use NETWORK_TIMEOUT for some conn failures

For -EHOSTUNREACH and -ETIMEDOUT we cannot tell whether the
connnection failure was due to a problem with the remote or local NI,
so we should return the LNET_MSG_STATUS_NETWORK_TIMEOUT to LNet in
these cases.

HPE-bug-id: LUS-9342
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I9036f5698061ce74fe29683b8249b8f9a05f3433
Reviewed-on: https://review.whamcloud.com/39900
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13571 lnd: Use NETWORK_TIMEOUT for txs on ibp_tx_queue

TXs on the ibp_tx_queue are waiting for a connection to be
established. Failure to establish a connection could be due to a
problem with either the local NI or the remote NI, and o2iblnd cannot
currently distinguish between these failures. As such, it should
return LNET_MSG_STATUS_NETWORK_TIMEOUT to LNet so that LNet will
decrement the health value of both the local NI and the remote NI and
future sends can take these health values into account.

Test-Parameters: trivial
HPE-bug-id: LUS-9342
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: Idbbbe95483d25ec48b83e33a00685f72fa5292e6
Reviewed-on: https://review.whamcloud.com/39899
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12564 libcfs: Use vfree_atomic instead of vfree

Since vfree is unsafe to use in atomic context, implement our own
libcfs_vfree_atomic heavily based on code from linux 4.10 commit
bf22e37a641327e34681b7b6959d9646e3886770

We can't use the one in the kernel because it's not exported.

unconditionally use it in *_FREE_LARGE() macros since in_atomic()
is not recommended to be used outside of core kernel code.

Not everything is present on 3.10 (rhel7) so we also add
llist primitive and a replacement for raw_cpu_ptr there.

Change-Id: I50892f231e54a284f4d8a14d910ea9ab2fbe6a16
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/40136
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Aurelien Degremont <degremoa@amazon.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14024 ofd: Avoid use after free in ofd_inconsistency_verification_main

The ofd_inconsistency_lock should not be unlocked after we woken up
a different thread that is going to free the structure containing
said lock.

Change-Id: I913e7470664e1128a250597b0a803f791d99099e
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/40222
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-14072 llite: fix client evicition with DIO

We set lockless in file open if O_DIRECT flag is passed,
however O_DIRECT flag could be cleared by
fcntl(..., F_SETFL, ...).

Finally we comes to a case where buffer IO without lock
held properly, and hit hang:

[<ffffffffc0d421ed>] osc_extent_wait+0x21d/0x7c0 [osc]
[<ffffffffc0d44897>] osc_cache_wait_range+0x2e7/0x940 [osc]
[<ffffffffc0d4585e>] osc_cache_writeback_range+0x96e/0xff0 [osc]
[<ffffffffc0d31c45>] osc_lock_flush+0x195/0x290 [osc]
[<ffffffffc0d31d7c>] osc_lock_lockless_cancel+0x3c/0xe0 [osc]
[<ffffffffc081f488>] cl_lock_cancel+0x78/0x160 [obdclass]
[<ffffffffc0cd8079>] lov_lock_cancel+0x99/0x190 [lov]
[<ffffffffc081f488>] cl_lock_cancel+0x78/0x160 [obdclass]
[<ffffffffc081f9a2>] cl_lock_release+0x52/0x140 [obdclass]
[<ffffffffc08238a9>] cl_io_unlock+0x139/0x290 [obdclass]
[<ffffffffc08242e8>] cl_io_loop+0xb8/0x200 [obdclass]
[<ffffffffc0e1d36b>] ll_file_io_generic+0x91b/0xdf0 [lustre]
[<ffffffffc0e1dd0c>] ll_file_aio_write+0x29c/0x6e0 [lustre]
[<ffffffffc0e1e250>] ll_file_write+0x100/0x1c0 [lustre]
[<ffffffffa984aa90>] vfs_write+0xc0/0x1f0
[<ffffffffa984b8af>] SyS_write+0x7f/0xf0
[<ffffffffa9d8eede>] system_call_fastpath+0x25/0x2a
[<ffffffffffffffff>] 0xffffffffffffffff

Lock cancel time out in the server side and client
eviction happen.

Fix this problem by testing O_DIRECT flag to decide if
we could issue lockless IO.

Fixes: 6bce536725 ("LU-4198 clio: turn on lockless for some kind of IO")
Change-Id: Idbf1c748684a6540aee5f6e35c017929fbcc60b9
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/40389
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Gu Zheng <gzheng@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14071 doc: lfs setquota/quota doc for PQ

Add description of key --pool for lfs setquota
and lfs quota.

Change-Id: Ie6bfb3240f7eb8b4239d95aac29989103f009578
Test-Parameters: trivial
HPE-bug-id: LUS-9341
Signed-off-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-on: https://review.whamcloud.com/40384
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14036 build: fix lbuild for MOFED 5.1

Starting MOFED 5.1, rdma-core is required for libib*mad

Test-Parameters: trivial

Change-Id: Id26f3cdb0552933577e1b27384ac82f9f48e2b3a
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/40254
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14054 utils: add --exit-on-close to ofd_access_log_reader

Add an --exit-on-close (-e), option to ofd_access_log_reader.

Test-Parameters: trivial
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Ifded63318026b0ad3f9f077dc74008469df877d9
Reviewed-on: https://review.whamcloud.com/40312
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-13153 lib: fix llapi_get_version_string

llapi_get_version_string() has always been broken as it passes args to
get_lustre_param_value in the wrong order.

Test-Parameters: trivial
Fixes: 0c5fbd80f1ba ("LU-5969 lustreapi: replace llapi_get_version()")
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ib4bdd715295f97fe5b7080f42bfa06883f191234
Reviewed-on: https://review.whamcloud.com/40680
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14125 osc: prevent overflow of o_dropped

In osc_announce_cached(), prevent o_dropped from overflowing.
Necessary because o_dropped AKA o_misc is 32 bits, but cl_lost_grant
is 64 bits.

Add a CDEBUG call so we can tell whether this happened.

Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Change-Id: Ia459934c789ae9609f851ae0a2581de583c6fc1c
Reviewed-on: https://review.whamcloud.com/40659
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>

LU-14136 utils: correct error message in lfs mirror extend

In migrate_open_files(), use llapi_layout_file_open() instead of
lfs_component_create() to avoid printing a confusing error message and
to ensure that the type of error is correctly propagated.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I31ebf29771b1a3e0b106a0a246a260a82eed92a4
Reviewed-on: https://review.whamcloud.com/40631
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Gu Zheng <gzheng@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14131 osd-ldiskfs: reduce credits for overwritting

If all blocks are mapped which means this is overwritting
case or space has been allocated by fallocate.

There is no need to modify exten tree, and we only
need 1 credits for inode.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I907861d4862256a8a23a81812953e2330e1d9925
Reviewed-on: https://review.whamcloud.com/40604
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13390 tests: add debugging to sanity test_65n

Print out the layouts used in comparisons in case of error, so that
it is possible to debug intermittent test failures.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ib81b10ff704998518275f737f028fb16391654ed
Reviewed-on: https://review.whamcloud.com/40600
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14129 kernel: kernel update RHEL7.9 [3.10.0-1160.6.1.el7]

Update RHEL7.9 kernel to 3.10.0-1160.6.1.el7.

Test-Parameters: clientdistro=el7.9 serverdistro=el7.9

Change-Id: If2d2120082965de67a3b29ade3e4d24a4221b2c2
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/40594
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-930 utils: document 'lfs getstripe -N' option

Add the '--mirror-count|-N' option to the usage message for
"lfs getstripe" and its man page.

Fixes: 818340364d51 ("LU-11124 utils: add 'lfs getstripe -N' option")
Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I89ffb2661a48336c37b8bdd784e5f9e942cbd798
Reviewed-on: https://review.whamcloud.com/40545
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13783 build: Fix LB_LINUX_VERSION rule for v5.8

Since Commit 20b1be595282 ("kbuild: fix single target builds for
external modules") in v5.8-rc7, the LB_LINUX_VERSION autoconf
rule doesn't work.

I don't know exactly why, but it can be fixed by setting "makerule" to
an empty string.

Passing the path to the directory in $makerule is unnecessary as
LB_LINUX_COMPILE_IFELSE, which LB_LINUX_TRY_MAKE eventually calls,
passes "$MODULE_TARGET=$PWD/build" which has the required effect.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I2b381d3546aaa0f365328a1319b2d4f145f33eeb
Reviewed-on: https://review.whamcloud.com/40371
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Aurelien Degremont <degremoa@amazon.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13990 ldlm: ldlm_flock_deadlock() ASSERTION(req != lock) failed

A client gets evicted and reconnects, so there can be a window
with 2 exports with flocks from the same client.
In this case during deadlock search from the new export we can go
to the old export and back to the new one.

Failed exports should be excluded from
deadlock search.

Change-Id: I9dec50d4c6694bbbcf13b976b5ebdc29377261ce
HPE-bug-id: LUS-8635
Signed-off-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Alexander Boyko <c17825@cray.com>
Reviewed-by: Vitaly Fertman <c17818@cray.com>
Reviewed-on: https://review.whamcloud.com/40047
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Vitaly Fertman <vitaly.fertman@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13987 ldlm: Don't re-enqueue glimpse lock on read

cl_glimpse_lock() doesn't match a lock with LDLM_FL_BL_AST
even if this lock is acquired by the same thread earlier.

It needs only size to check for spare file,
so let't add LDLM_FL_CBPENDING to match flags.

 #1 [ffff9ba7326036f0] schedule at ffffffff87b67c49
 #2 [ffff9ba732603700] obd_get_request_slot at ffffffffc0dbe0a4 [obdclass]
 #3 [ffff9ba7326037b8] ldlm_cli_enqueue at ffffffffc0faedce [ptlrpc]
 #4 [ffff9ba732603878] mdc_enqueue_send at ffffffffc11b38a8 [mdc]
 #5 [ffff9ba732603938] mdc_lock_enqueue at ffffffffc11b3eb2 [mdc]
 #6 [ffff9ba7326039a8] cl_lock_enqueue at ffffffffc0dfee95 [obdclass]
 #7 [ffff9ba7326039e0] lov_lock_enqueue at ffffffffc10ef265 [lov]
 #8 [ffff9ba732603a20] cl_lock_enqueue at ffffffffc0dfee95 [obdclass]
 #9 [ffff9ba732603a58] cl_lock_request at ffffffffc0dff54b [obdclass]

HPE-bug-id: LUS-8690
Change-Id: I4c3820f754ceb502079bdd7d8e1a5389f6696eba
Signed-off-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Vitaly Fertman <c17818@cray.com>
Reviewed-by: Alexander Boyko <c17825@cray.com>
Reviewed-by: Andrew Perepechko <c17827@cray.com>
Tested-by: Elena Gryaznova <c17455@cray.com>
Reviewed-on: https://review.whamcloud.com/40044
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Vitaly Fertman <vitaly.fertman@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13891 utils: fix memory leak in llapi_ladvise()

A buffer allocated in llapi_ladvise() for ioctl() is never
released. This patch ensures the buffer is properly freed.

Fixes: e14246641c04 ("LU-4931 ladvise: Add feature of giving file access advices")
Signed-off-by: Jean-Yves VET <jyvet@ddn.com>
Change-Id: I0761e161074ae3029218473ec951670fdbbd33bd
Reviewed-on: https://review.whamcloud.com/39610
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-13837 lnet: Introduce constant for net ID of LNET_NID_ANY

This patch adds a new constant, LNET_NET_ANY, to represent the net
ID of the LNET_NID_ANY wildcard NID.

Test-Parameters: trivial
HPE-bug-id: LUS-9122
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I4b4d9e70ba2826843c6585ad5a9e365799face65
Reviewed-on: https://review.whamcloud.com/39544
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>

LU-13817 quota: print error when pool is absent

Print error for "lfs quota --pool" and
"lfs setquota -o" if pool is absent.

HPE-bug-id: LUS-9112
Test-Parameters: trivial testlist=sanity-quota
Change-Id: I4c2aa41d8c6f27742e68dfdc78c9d9365c760237
Signed-off-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-on: https://review.whamcloud.com/39505
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13746 utils: check argument logname presence in llog_print

Correction of segfault in llog_print when no logname specified.
example: lctl --device MGS llog_print

Signed-off-by: Etienne AUJAMES <eaujames@ddn.com>
Change-Id: I03a6c08dfc73ff8cb5861d162e21ce5aa581e197
Reviewed-on: https://review.whamcloud.com/39263
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>

LU-13159 osd: osd-zfs to release index back structures

otherwise those can be leaked in case of failed mount

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I7dc848a8397c11b1f56d36ef0a7155314ca9afc2
Reviewed-on: https://review.whamcloud.com/37293
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14125 obdclass: add grant fields to export procfile

Add ted_{grant,reserved,dirty} to the export
procfile for OSTs, to allow comparison between the
OST's idea of grants allocated to the client with
the client's idea.

Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Change-Id: Ib34582e2be55fe2007363b52cea4dee211b7f540
Reviewed-on: https://review.whamcloud.com/40563
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14031 tests: check client reconnect

The patch adds test 147 for recovery-small. It checks how often
client reconnects to a server with obd_timeout=200.

HPE-bug-id: LUS-8520
Test-Parameters: trivial testlist=recovery-small env=ONLY=147
Signed-off-by: Alexander Boyko <alexander.boyko@hpe.com>
Change-Id: Id5bd732fe4d949cfa45ea0272f197809cca3290d
Reviewed-on: https://review.whamcloud.com/40297
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Reviewed-by: Vitaly Fertman <vitaly.fertman@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13571 lnet: Correct handling of NETWORK_TIMEOUT status

The original intent of the LNET_MSG_STATUS_NETWORK_TIMEOUT health
status was to handle cases where the LND was unsure whether the
failure was due to the local or remote NI. In this case, we'll want
to decrement both the local and remote NI health and allow recovery
to ascertain which interface is actually healthy.

Test-Parameters: trivial
HPE-bug-id: LUS-9342
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: Ib00ac260640100123e4e97e9c566289e92fb0b6e
Reviewed-on: https://review.whamcloud.com/39898
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14027 ldlm: Do not hang if recovery restarted during lock replay

LU-13600 introduced lock ratelimiting logic, but it did not take
into account that if there's a disconnection in the REPLAY_LOCKS
phase then yet unsent locks get stuck in the sending queue so
the replay locks thread hangs with imp_replay_inflight elevated
above zero.

The direct consequence from that is recovery state machine never
advances from REPLAY to REPLAY_LOCKS status when imp_replay_inflight
is non zero.

Adjust __ldlm_replay_locks() to check if the import state changed
before attempting to send any more requests.

Add a testcase.

Change-Id: Idbaf5461f33d1884088269d67d01071c7e1bf8a5
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Fixes: 3b613a442b ("LU-13600 ptlrpc: limit rate of lock replays")
Reviewed-on: https://review.whamcloud.com/40238
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13195 obdclass: show FID for corrupted llog

to be able to remove that easily.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Iefead860b5e60e74e0eb445f715508a3b01fac87
Reviewed-on: https://review.whamcloud.com/38977
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14053 utils: detect hangup in ofd_access_log_reader

In ofd_access_log_reader, when the batch file is a pipe or socket,
detect hangups and exit.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: If7509174b992011305d8e4a7aa2766a3a1980831
Reviewed-on: https://review.whamcloud.com/40311
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14052 ofd: support for multiple access readers

ofd_access_log_reader can be passed -I, --mdt-index-filter=INDEX to
print only FIDs located on INDEX.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I9a4f09c6b7ca15623d459df17939895301a57a8b
Reviewed-on: https://review.whamcloud.com/39906
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14009 osd: missing ldiskfs_htree_unlock()

in osd_ldiskfs_it_fill()

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I5d2242e0864cbaa72af096b263d8758966a6be22
Reviewed-on: https://review.whamcloud.com/40137
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14027 ldlm: Do not wait for lock replay sending if import dsconnected

If import disconnected while we were preparing to send some lock replays
the sending RPC would get stuck on the sending list and would keep
the reconnected import state from progressing from REPLAY
to REPLAY_LOCKS state waiting for the queued replay RPCs to finish.

Set them as no_delay to ensure they don't wait.

LU-13600 exacerbated this issue some but it certainly exist
before it as well.

Change-Id: Id276a0be7657d9ad6cf40ad8e7a165d5cd341cb8
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/40272
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>

LU-14061 utils: prefer mounting with specified fstype

If the server filesystem is mounted with "-t lustre" use that type
for mounting instead of automatically selecting "lustre_tgt" for
server mounts, so that it is appearing in /proc/mounts correctly,
and "mount -t lustre" will list all of these filesystems.

Only if "mount -t lustre_tgt" is used will mount.lustre_tgt be called,
and that will internally use "lustre_tgt" as the filesystem type, and
only fall back to type "lustre" if that does not work.  This will give
userspace plenty of time to transition to using "lustre_tgt" for
server mountpoints.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ifd7560c800acdfe14c9564bbf955ecad1224f2e3
Reviewed-on: https://review.whamcloud.com/40474
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12632 hsm: wait longer in sanity-hsm test_90()

Due to the huge (51) number of HSM requests to be completed we must
wait longer for this test to pass when run using an advanced (zfs)
backend filesystem.

Test-Parameters: trivial fstype=zfs env=ONLY=90 testlist=sanity-hsm,sanity-hsm,sanity-hsm,sanity-hsm,sanity-hsm,sanity-hsm,sanity-hsm,sanity-hsm,sanity-hsm
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Ia20086bf5c072c5d120eed5b0937d37d7b4342db
Reviewed-on: https://review.whamcloud.com/40387
Reviewed-by: Ben Evans <beevans@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Nikitas Angelinas <nikitas.angelinas@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14109 doc: lnetctl manpage missing import and export options

Describe missing options in the lnetctl man page:
lnetctl import "--exec"
lnetctl export "--backup"

Test-Parameters: trivial
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Change-Id: I47e6c5e264dc8d5673f1229291be873996c02f55
Reviewed-on: https://review.whamcloud.com/40523
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Gian-Carlo DeFazio <defazio1@llnl.gov>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14102 tests: add a pause in open-close in sanity/160l

so that close has to update mtime and generate CL_MTIME
record in the changelog.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I118172229c86ed5c201299de7476678689bf4cab
Reviewed-on: https://review.whamcloud.com/40508
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13683 lfs: return -ENOENT when invoked on non-existing file

Since merge of LU-11510, lfs migrate on a non-existing file will give
the following error "lfs migrate: can't create composite layout from
file /some/path/to/file" and will exit with code 0, potentially
leaving a calling script unaware of the error.

This patch fixes it by using errno which is set in the call to
llapi_layout_get_by_path()

Signed-off-by: Sebastien Piechurski <sebastien.piechurski@atos.net>
Change-Id: I910eae78445f6071ff4e741afd43d140f554ab22
Fixes: 8bedfa377fbd ("LU-11510 lfs: migrate a composite layout file correctly")
Reviewed-on: https://review.whamcloud.com/38953
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14115 lod: fix to set inherit flag for stripe directroy

lod_xattr_set_lmv() is called to set attributes from parent
to all stripes of the directroy, however, flags are ignored
which will cause project inherit bits missed for stripes.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I10a2de303b4b8430e560752cf0d66466c93616a4
Reviewed-on: https://review.whamcloud.com/40543
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12956 ldlm: fix hrtimer using

A race could happen between hrtimer_start() and
hrtimer_expires_remaning(), cause the second one doesn't hold a lock
on timer->base. And a first one could change it between different CPU.
The following failure happened:
BUG: unable to handle kernel NULL pointer dereference at 000000000028
IP: [<ffffffffc0fc773f>] target_handle_connect+0x12ff/0x2b50 [ptlrpc]
at remaining = hrtimer_expires_remaining(timer), timer->base was NULL

The fix changes hrtimer_expires_remaining() to hrtimer_get_remaining()
which helds a lock and prevents race.

Fixes: 9334f1d51249 ("LU-11771 ldlm: use hrtimer for recovery to fix timeout messages")
HPE-bug-id: LUS-9514
Signed-off-by: Alexander Boyko <alexander.boyko@hpe.com>
Change-Id: I2cea1e5e2d523f131f1acb3346cf0324adae624e
Reviewed-on: https://review.whamcloud.com/40513
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andrew Perepechko <andrew.perepechko@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14077 kernel: kernel update SLES15 SP1 [4.12.14-197.64.1]

Update SLES15 SP1 kernel to 4.12.14-197.64.1 for Lustre client.

Test-Parameters: trivial \
env=SANITY_EXCEPT="56oc 100 130 136 817" \
clientdistro=sles15sp1 serverdistro=el7.8 \
testlist=sanity

Change-Id: If2d8ecd89b307c0e49cf39a7ef33c298a2690f27
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/40477
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14083 build: Don't overwrite KBUILD_EXTRA_SYMBOLS

The gnilnd requires that KBUILD_EXTRA_SYMBOLS be populated with the
symbols for some of its dependencies. Don't overwrite
KBUILD_EXTRA_SYMBOLS for the kernel tests.

Test-Parameters: trivial
HPE-bug-id: LUS-9448
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: Ie10d3be0078e163f3e36f1c880d87ebc185c49c7
Reviewed-on: https://review.whamcloud.com/40435
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13988 mdt: ASSERTION(!lustre_handle_is_used(&lh->mlh_reg_lh)) failed

mdt_brw_enqueue() should clear lock handle on error.

HPE-bug-id: LUS-8740
Change-Id: Ic2201189af6bcb2b1a114a599e138f9c22f3d99d
Signed-off-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Alexandr Boyko <c17825@cray.com>
Reviewed-by: Alexander Zarochentsev <c17826@cray.com>
Reviewed-on: https://review.whamcloud.com/40045
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13822 ptlrpc: fixes for RCU-related stalls

ptlrpc_expired_set() may need to process a lot
of requests, so the processing loop needs to
schedule from time to time to avoid RCU-related
stalls.

Change-Id: I14b0aaf14ab127805699729adbb26459a2f4f07c
Signed-off-by: Andrew Perepechko <andrew.perepechko@hpe.com>
HPE-bug-id: LUS-8939
Reviewed-on: https://review.whamcloud.com/39514
Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13709 tests: test lfs mkdir -c without -i

Almost every test with lfs mkdir -c in the test suite also
uses option -i, so lfs mkdir -c (same as -i -1, where lustre
chooses the MDTs) is poorly tested.  Add a test for that
case, sanity test_300s.

Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Change-Id: Iede537d52cf445c9c9a6353338670e55a11364da
Reviewed-on: https://review.whamcloud.com/39457
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Gian-Carlo DeFazio <defazio1@llnl.gov>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-11290 ldlm: page discard speedup

Improving check_and_discard_cb, allowing to cache
negative result of dlm lock lookup and avoid
excessive osc_dlm_lock_at_pgoff() calls.

HPE-bug-id: LUS-6432
Change-Id: I253f944bf430b06d0e1a300d22e5d9b2e97412bf
Signed-off-by: Alexander Zarochentsev <c17826@cray.com>
Reviewed-on: https://review.whamcloud.com/39327
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Vitaly Fertman <vitaly.fertman@hpe.com>
Reviewed-by: Andrew Perepechko <andrew.perepechko@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13839 kernel: new kernel [RHEL 8.3 4.18.0-240.1.1.el8_3]

This patch makes changes to support new RHEL 8.3 release
for Lustre client.

Test-Parameters: trivial clientdistro=el8.3

Change-Id: I06a46735b42ac258e576b1dd5c0beb17f4fd3e47
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/40536
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13344 gss: Update crypto to use sync_skcipher

As of linux v4.19-rc2-66-gb350bee5ea0f the change
   crypto: skcipher - Introduce crypto_sync_skcipher

Enabled the deprecation of blkcipher which was dropped
as of linux v5.4-rc1-159-gc65058b7587f
    crypto: skcipher - remove the "blkcipher" algorithm type

Based on the existence of SYNC_SKCIPHER_REQUEST_ON_STACK
use the sync_skcipher API or provide wrappers for the
blkcipher API

Test-Parameters: testlist=sanity,recovery-small,sanity-sec mdscount=2 mdtcount=4 ostcount=8 clientcount=2 env=SHARED_KEY=true,SK_FLAVOR=skn
Test-Parameters: testlist=sanity,recovery-small,sanity-sec mdscount=2 mdtcount=4 ostcount=8 clientcount=2 env=SHARED_KEY=true,SK_FLAVOR=ska
Test-Parameters: testlist=sanity,recovery-small,sanity-sec mdscount=2 mdtcount=4 ostcount=8 clientcount=2 env=SHARED_KEY=true,SK_FLAVOR=ski
Test-Parameters: testlist=sanity,recovery-small,sanity-sec mdscount=2 mdtcount=4 ostcount=8 clientcount=2 env=SHARED_KEY=true,SK_FLAVOR=skpi
HPE-bug-id: LUS-8589
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I7683c20957213fd687ef5cf6dea64c842928db5b
Reviewed-on: https://review.whamcloud.com/38586
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10059 tests: sanityn 32a error messages

There are three checks for file size in sanityn test 32a
that, if they fail, have the same error message
"wrong file size".  Let's add additional information to
the error messages to help distinguish the different errors.

Test-Parameters: trivial testlist=sanityn
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I71aae8de67125d5be93c4fb4728b2c20d26df49c
Reviewed-on: https://review.whamcloud.com/40496
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Vikentsi Lapa <vlapa@whamcloud.com>

LU-14093 ptlrpc: redefine uop_params_off to support gcc10

otherwise gcc10 complains about out-of-boundry access:

llog_swab.c: In function lustre_swab_update_ops:
llog_swab.c:137:46: error: array subscript 65534 is outside the bounds
of an interior zero-length array__u16[0] {aka short unsigned int[]}
[-Werror=zero-length-bounds]
  137 |    __swab16s(&uops->uops_op[i].uop_params_off[j]);

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I27981cbc79991cbd7a79cb90aec97bd1dc8b2f1b
Reviewed-on: https://review.whamcloud.com/40486
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14045 sec: fix O_DIRECT and encrypted files

Sometimes, we can end up in a situation where
osc_release_bounce_pages() mistakenly consider pages as fscrypt
bounce pages, and tries to free them.
Fix the way we consider bounce pages by always setting the PageChecked
flag on them.
Also remove sanity test_426 from the ALWAYS_EXCEPT list.

Test-Parameters: testlist=sanity-sec envdefinitions=ONLY="36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 52 53 54" clientdistro=ubuntu2004 fstype=ldiskfs mdscount=2 mdtcount=4
Test-Parameters: testlist=sanity-sec envdefinitions=ONLY="36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 52 53 54" clientdistro=ubuntu2004 fstype=zfs mdscount=2 mdtcount=4
Fixes: 728036f256 ("LU-12275 sec: O_DIRECT for encrypted file")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ic54ae031c3b0baa17ffed8a6b6b90ff44f87ff58
Reviewed-on: https://review.whamcloud.com/40295
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14039 obdclass: set LA_TYPE when update_log init

Fix for LBUG:
osp_md_create()) ASSERTION( attr->la_valid & LA_TYPE ) failed:
osp_md_create()) LBUG
Pid: 6024, comm: lod0002_rec0001 3.10.0-1062.18.1.el7_lustre
Call Trace:
 libcfs_call_trace+0x8c/0xc0 [libcfs]
 lbug_with_loc+0x4c/0xa0 [libcfs]
 osp_md_create+0x42a/0x470 [osp]
 llog_osd_get_cat_list+0x8d4/0xbd0 [obdclass]
 lod_sub_prep_llog+0xb9/0x783 [lod]
 lod_sub_recovery_thread+0x383/0xcf0 [lod]
 kthread+0xd1/0xe0
 ret_from_fork_nospec_begin+0x7/0x21

Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: I035f3f2da5dba54b86431cec65c48e9c5010224c
Reviewed-on: https://review.whamcloud.com/40264
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>