LU-15317 llite: Add FID to async ra iotrace

IOtrace log entries need to include the FID of the file
concerned.  Add this to async readahead.

test-parameters: trivial

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I8d788969f29412ce88f1cafa229977f6efa20962
Reviewed-on: https://review.whamcloud.com/45912
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15317 llite: Add strided readahead to iotrace

We need to capture some additional parameters to correctly
understand the behavior of strided readahead.  Add these
parameters to the existing iotrace message.

test-parameters: trivial

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I7caddf9dfaf9ba5f2645d045d5a4a50562cc1b54
Reviewed-on: https://review.whamcloud.com/45888
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15317 llite: Make iotrace logging quieter

Most of the time, we don't read any pages with readahead,
since we're moving through the window and aren't ready to
read more yet.  That's important for readahead debug, but
there's no need to log it for iotrace.  (This matters
because without this change, this messsage is the large
majority of iotrace messages.)

test-parameters: trivial

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I58197acd1ef97c903320a2433ec1d5dcb0e46bd0
Reviewed-on: https://review.whamcloud.com/45887
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-11787 test: Fix checkfilemap tests for 64K page

File mapping is page size aligned. Modify the tests to handle 64K
page.

Test-Parameters: trivial clientdistro=el8.5 clientarch=aarch64 testlist=sanityn env=ONLY="71a 71b"
Test-Parameters: trivial clientdistro=el8.5 clientarch=ppc64le testlist=sanityn env=ONLY="71a 71b"
Change-Id: I316a197db8cdd0f9064431f8c572b43adf6110b8
Signed-off-by: James Simmons <jsimmons@infradead.org>
Signed-off-by: Xinliang Liu <xinliang.liu@linaro.org>
Reviewed-on: https://review.whamcloud.com/45629
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15583 build: Update ZFS version to 2.1.2

Update ZFS version to 2.1.2. The changes are listed in:
https://github.com/openzfs/zfs/releases/tag/zfs-2.1.2

Change-Id: If7c81a4b1fe13e29eea1c277b896223f5c06b31a
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45591
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15207 libcfs: reset hs_rehash_bits

if rehash work is cancelled, then nobody resets
hs_rehash_bits and the first iterator asserts
at LASSERT(!cfs_hash_is_rehashing(hs)) in
cfs_hash_for_each_relax().

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I1a567f6be77ca6c45e5d4f256722206b12588554
Reviewed-on: https://review.whamcloud.com/45533
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15193 quota: expand QUOTA_MAX_TRANSIDS to 12

In some rare cases 12 quota ids are needed.
Usually (user, group) * (block, inode) * (inode, parent) = 8 qids
are needed. But with project id,
(user, group, project) * (block, inode) * (inode, parent) = 12 qids
are needed.

Change-Id: I4b3ee197f6e274abda06edf60b246f089fe28d10
Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial testlist=sanity-quota
Reviewed-on: https://review.whamcloud.com/45456
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Stephane Thiell <sthiell@stanford.edu>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15082 osp: invalidate statfs data from the timer callback

osp_statfs_timer_cb() can be called just before statfs data gets
stale. this in turn may cause early wakeup to the precreate thread
which would find statfs data still up-to-data and go back to slepp.
if no precreate happens to this OSP (e.g. due to current space
usage), then the precreate thread will stay asleep for a long time,
statfs data won't get refreshed and this may block new objects
on the corresponding OST.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I86e16eed6f1068702db696a9ddec7a22994180b7
Reviewed-on: https://review.whamcloud.com/45199
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14979 lnet: set max recovery interval duration

Add a tunable parameter to limit the recovery ping interval which was
previously statically set to 900.
This can be done by using:

  lnetctl set max_recovery_ping_interval <value>

Modify sanity-lnet test 210/211 to validate this new functionality.

Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Cyril Bordage <cbordage@whamcloud.com>
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I766ceb6e03ffdab125005e16472b6f9eeadfb9d5
Reviewed-on: https://review.whamcloud.com/44927
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14264 tests: make PARALLEL available to all suites

The PARALLEL environment variable is checked and a default
value is set in the sanity test suite, but recovery-small
also uses PARALLEL and does not check/initialize it.

Move check and set the PARALLEL environment variable to a
default value in the init_test_env() routine in
test-framework.sh.

Fixes: 688d5da6a89 (“LU-12846 mdd: return error while delete failed”)
Fixes: 26e8f1137b8 (“LU-13116 mgc: do not lose sptlrpc config lock”)

Test-Parameters: trivial testlist=sanity,recovery-small
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: If773e2ea7300056a0ef00a2cb24e13e20a971bd6
Reviewed-on: https://review.whamcloud.com/44763
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15483 tests: Improve test 398b

Test 398b is currently only writing 3 MiB (12 MiB/4) of
data from each thread in the racing set (4 DIO, 4 BIO).
This is such a small amount that it's probably finishing
too quickly to generate the intended overlap & races most
of the time.

It also currently writes only at PAGE_SIZE, which excludes
a lot of possible races.

This is currently a pretty fast test - ~ 2 seconds on my
local machine, similar on Maloo - so let's have it write 4
times more data and use 4 different block sizes, so it will
hit more races.

This is 16x more I/O, but the test still only takes around
16 seconds on my local machine.

This is motivated by difficulties dealing with rare
failures while developing the LU-13799 patch series and
hoping to make them easier to catch.

Test-Parameters: trivial testlist=sanity env=ONLY=398b,ONLY_REPEAT=50
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I1092023a526c085dfacd1bd112c2ebb714da63e5
Reviewed-on: https://review.whamcloud.com/44321
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Wang Shilong <wangshilong1991@gmail.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14832 llite: Correct cl_env comments

The comments related to cl_env caching behavior are
dangerously out of date and misleading, describing an
old caching mechanism which was linked to threads.

This has not been present for some time, and we cannot use
cl_env_get to get the environment for a thread as it
describes.

Correct the various comments and remove a now extraneous
include.

Test-Parameters: trivial
Fixes: 4533271278 ("LU-4257 obdclass: Get rid of cl_env hash table")
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I0a855094c66281bfb35e234ecad1af3b923e8538
Reviewed-on: https://review.whamcloud.com/44191
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Wang Shilong <wangshilong1991@gmail.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-14771 ptlrpc: Rearrange version mismatch message

Minor change to reposition the client version string on
console warning message, and merge split string to one line

Test-Parameters: trivial
Signed-off-by: Andrew Elwell <Andrew.Elwell@gmail.com>
Change-Id: I89242cff873028f4f76bafdd5a14f169a98f7875
Reviewed-on: https://review.whamcloud.com/44029
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-8137 utils: fix llverdev for use on regular files

Allow llverdev to work with regular files to verify data contents.

Add a parameter to allow specifying the maximum file size to use.
Allow the use of unit suffix when specifying sizes as arguments.
If the file size is zero, write until the filesystem is full.
Use pread() and pwrite() to avoid having to deal with file offsets.

Add sanityn test_56a/56b to verify proper operation on a large file.

Update llverdev.8 man page to describe options properly.

Test-Parameters: trivial testlist=sanityn env=ONLY=56a,56b ostsizegb=12
Test-Parameters: fstype=zfs testlist=sanityn env=ONLY=56a,56b ostsizegb=12
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ib06dc989fe692fd6aae388a79d9aa28f702540e5
Reviewed-on: https://review.whamcloud.com/43230
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Anjus George <georgea@ornl.gov>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15460 test: wait quota pool to be prepared

When one OST pool was created, the corresponding quota pool
could need more time to prepare after it was created, then
fail to check the info of the quota pool

Test-Parameters: trivial
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Change-Id: Ibea33403639087f27e438d71c0e87fea5367bc3e
Reviewed-on: https://review.whamcloud.com/46853
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14503 o2iblnd: clean up zombie connections on shutdown

Clean up zombie connections on net shutdown in o2iblnd
Wake up connd threads and wait for them to do the clean-up
before proceeding.

Test-Parameters: trivial
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: Iff8424d9be7401987046fe9aef6e7a787f5efe83
Reviewed-on: https://review.whamcloud.com/42068
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14422 utils: check empty stripe-count/offset setting

Prohibit "" stripe count and stripe offset setting in lfs setstripe.

Test-Parameters: trivial
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I4d0cc2354c4249a31222e0f90de39e16ec4694a5
Reviewed-on: https://review.whamcloud.com/41793
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13970 llite: add option to disable Lustre inode cache

A tunable option is added to disable Lustre inode cache:
"llite.*.inode_cache=0" (default =1)

When it's turned off, ll_drop_inode() always returns 1, then
the last iput() will release inode.

Add sanity test_433.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I0642bdc694dc365a05395c3fae98131e1e7723c6
Reviewed-on: https://review.whamcloud.com/39973
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>

LU-13906 build: consistent use of %{name}

Make the of the %{name} macro consistent across lustre packages

Fixes: cfaf0eb92005 ("LU-12214 build: fixes if the name is not just 'lustre'")
Test-Parameters: trivial
HPE-bug-id: LUS-5914, LUS-5915
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I71c0a0e77a0fc7319a166311e33d3ca9de60e499
Reviewed-on: https://review.whamcloud.com/39662
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10973 lutf: Fix library path

Correct library path pointing the lnetconfig library.

Test-Parameters: trivial
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: I60089ba0e86d2c3406e283c44530793d84065674
Reviewed-on: https://review.whamcloud.com/46502
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13547 tests: remove ea_inode from mkfs MDT options

The large_dir and ea_inode is enabled by default when MDTs
are formatted.  Thus, remove 'ea_inode' and 'large_dir'
from the mkfs options in the test-framework.

Add a test to make sure that these MDT features are enabled
by default.

Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I1b3f1bee7e0c091f8fffc9da0e769eed921dac8f
Reviewed-on: https://review.whamcloud.com/38582
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12186 ec: add necessary structure member for EC file

Added basic structure members for erasure-coding layout.

Test-Parameters: trivial
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I697dc144877d6c5fbe9335dc721200e43749f5d9
Reviewed-on: https://review.whamcloud.com/38319
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14195 libcfs: test for nla_strscpy

During the development of the Linux 5.10 kernel the function
nla_strlcpy() was replaced by nla_strscpy(). Handle this
change for Lustre.

Test-parameters: trivial
Change-Id: I47f12add619cfd88a3692f0760b8bcc35b7877d9
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/46876
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15788 lmv: try another MDT if statfs failed

With lazystatfs option statfs could fail if MDT0 is offline.
This leads to MPICH->IOR fail during FOFB tests. A client
could get statfs data from different MDT at DNE setup.

HPE-bug-id: LUS-10581
Signed-off-by: Alexander Boyko <alexander.boyko@hpe.com>
Change-Id: Icec83aba0c3ddbc749b782787b1b52faadf34a3e
Reviewed-on: https://review.whamcloud.com/47152
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12756 lnet: Avoid redundant peer NI lookups

Each caller of lnet_peer_ni_traffic_add() performs a subsequent call
to lnet_peer_ni_find_locked(). We can avoid the extra lookup by having
lnet_peer_ni_traffic_add() return a peer NI pointer (or ERR_PTR as
appropriate).

lnet_peer_ni_traffic_add() now takes a ref on the peer NI to mimic
the behavior of lnet_peer_ni_find_locked().

lnet_nid2peerni_ex() only has a single caller that always passes
LNET_LOCK_EX for the cpt argument, so this function argument is
removed.

Some duplicate code dealing with ln_state handling is removed from
lnet_peerni_by_nid_locked()

Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I8e9e2449ef2b958b53abd59cd2c122e5492fbb34
Reviewed-on: https://review.whamcloud.com/36623
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-6864 osp: manage number of modify RPCs in flight

Currently we use a rpc_lock to ensure concurrent in-flight
request are handled serially to prevent the execution status
from being over written. This patch changes the osp component
to send multiple modify RPCs in parallel to the MDT. This will
improve metadata performance of cross-MDT operations.

For testing replace mkdirmany with createmany -d which does the
same thing.

Signed-off-by: Gregoire Pichon <gregoire.pichon@bull.net>
Signed-off-by: James Simmons <jsimmons@infradead.org>
Change-Id: Icb601afabd6767463634a4c7943ec4206bc758ec
Reviewed-on: https://review.whamcloud.com/14375
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15841 lod: iterate component to collect avoid array

In newly created file, the mirror information hasn't been established
as LOD is trying allocate OST for its components, so we need to
iterate component instead of mirror to collect the avoid guidance
information.

Test-Parameters: testlist=sanity-flr env=ONLY=47,ONLY_REPEAT=40
Fixes: fabf3fe7 ("LU-9007 lod: improve obj alloc for FLR file")
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I6bbe5f6b6dfea06c5213b77b7ebb6a5d28aa0d17
Reviewed-on: https://review.whamcloud.com/47293
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15786 tests: get maxage param on mds1 properly

The correct maxage parameters on mds1:
  osp.${FSNAME}-MDT000[1-N]-osp-MDT0000.maxage

To reproduce the failure just run the following on
failover setup where mds1_HOST != mds1failover_HOST:
  sh llmount.sh
  ONLY="100b 100c" sh replay-single.sh

  error: get_param: param_path
   'osp/*MDT0000*MDT0001/maxage': No such file or directory
  sleep: missing operand
  Try 'sleep --help' for more information.

Fixes: 436cd4fd21 ("LU-14938 tests: fail_abort() in t-f to take care of MDTs")
Test-Parameters: trivial testlist=replay-single,recovery-small
Signed-off-by: Elena Gryaznova <elena.gryaznova@hpe.com>
HPE-bug-id: LUS-10804
Reviewed-by: Vladimir Saveliev <vlaidimir.saveliev@hpe.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Change-Id: Icbedb044c4a008868bd3a99d44aa1c350e7c9eaa
Reviewed-on: https://review.whamcloud.com/47145
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Vladimir Saveliev <vladimir.saveliev@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15754 lfsck: skip an inode if iget() returns -ENOMEM

After the change
commit c2b6d621c4ffe9936adf7a55c8b1c769672c306f
Author: Al Viro <viro@zeniv.linux.org.uk>
Date: Thu Jun 28 15:53:17 2018 -0400 new primitive: discard_new_inode()

find_inode_fast() returns -ESTALE, but iget_locked() replaces
it to the NULL and finally ldiskfs_inode_attach_jinode()
returns -ENOMEM.

So this check in osd_iit_iget() doesn't work.

if (rc == -ENOENT || rc == -ESTALE)
    RETURN(SCRUB_NEXT_CONTINUE);

As a solution we can skip an inode if -ENOMEM returned

Hpe-bug-id: LUS-10833
Change-Id: Icb30610e46e2ab899a512761b63aea248c4f2ada
Signed-off-by: Artem Blagodarenko <artem.blagodarenko@hpe.com>
Reviewed-on: https://review.whamcloud.com/47079
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andrew Perepechko <andrew.perepechko@hpe.com>
Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15724 tests: MDT failover hang reproducer

The patch adds recovery-small 144a test to reproduce
MDT failover hang when precreate threads are blocked on objects.

LustreError: 0-0: Forced cleanup waiting for mdt-kjcf05-MDT0001_UUID
namespace with 46 resources in use, (rc=-110)

Test-Parameters: trivial testlist=recovery-small env=ONLY=144a
HPE-bug-id: LUS-10750
Signed-off-by: Alexander Boyko <alexander.boyko@hpe.com>
Change-Id: I2743a1b5c8911d6982b527f7e7b7bbbaf310cd04
Reviewed-on: https://review.whamcloud.com/47006
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Reviewed-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15724 osp: wakeup all precreate threads

Number of threads could sleep at osp_precreate_reserve() and
wait objects from OST. When MDT stops Lustre should wakeup
all threads. When opd_pre_recovering is set any wakeup of
opd_pre_user_waitq is useless. Failover of MDT does not produce
disconnect event, only inactive, so osp_precreate_cleanup_orphans()
can not be awakened.

LustreError: 0-0: Forced cleanup waiting for mdt-kjcf05-MDT0001_UUID
namespace with 46 resources in use, (rc=-110)

 schedule_timeout at ffffffff8e551cd3
 osp_precreate_reserve at ffffffffc17d2d83 [osp]
 osp_declare_create at ffffffffc17c7eb9 [osp]
 lod_sub_declare_create at ffffffffc156415b [lod]
 lod_qos_declare_object_on at ffffffffc155bf42 [lod]
 lod_ost_alloc_rr.constprop.23 at ffffffffc155db2f [lod]
 lod_qos_prep_create at ffffffffc15630a6 [lod]
 lod_declare_instantiate_components at ffffffffc154b237 [lod]

HPE-bug-id: LUS-10750
Signed-off-by: Alexander Boyko <alexander.boyko@hpe.com>
Change-Id: If0164cfbecb1e358d9857421cb234559dc8cecbc
Reviewed-on: https://review.whamcloud.com/47005
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Reviewed-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15132 mdc: Use early cancels for hsm requests

HSM RELEASE and RESTORE requests take EX layout lock on the MDT side.
So the client can use early cancel for its local lock on the resource
to limit the contention (mdt side).

This patch does not pack ldlm request inside the hsm request because
the field (RMF_DLM_REQ) does not exist in the request. Adding this
field inside the request would break compatibility with _old_ servers.

Signed-off-by: Etienne AUJAMES <eaujames@ddn.com>
Change-Id: I30a57b4855c28eef9c55a9645d3b6c491f962b13
Reviewed-on: https://review.whamcloud.com/47181
Reviewed-by: Nikitas Angelinas <nikitas.angelinas@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13363 lod: do object allocation in OST pool

Currently, the ltd->ltd_qos.lq_same_space boolean that decides
whether the LOD QOS allocator is active for an allocation or not
is tracked for the entire LOV. But when a pool is specified, this
judgement should be tracked on a per-pool basis.

sanity.sh test_116c is added to verify this patch.

Test-Parameters: ostcount=6 testlist=sanity env=ONLY=116c
Signed-off-by: Emoly Liu <emoly@whamcloud.com>
Change-Id: I463d5927c7a9c9171483615d2cec629ec10dc666
Reviewed-on: https://review.whamcloud.com/38136
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15800 ofd: take a read lock for fallocate

there is no need to take an write (exclusive) object's
lock for fallocate - we just need to serialize fallocate
vs destroy, all internal structures should be protected
by OSD and disk filesystem like the write path does.

Fixes: cdaaa87f6b ("LU-14214 ofd: fix locking in ofd_object_fallocate()")
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I65986745865ee329c5257a7efca5e79403830608
Reviewed-on: https://review.whamcloud.com/47268
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15718 lnet: improve lnet_selftest speed

lets replace a global spinlock with atomic variables,
to avoid cpu power limit in testing.

Test-Parameters: trivial testlist=lnet-selftest
HPE-bug-id: LUS-10812
Signed-off-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Change-Id: Ib12a3b3fe2fe300354e5a7c502bc6f5165e7d05c
Reviewed-on: https://review.whamcloud.com/47002
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15832 lod: clear .do_index_ops in striping free

LDLM lock can guarantee LOD object directory striping is safe to
access, but lod_striping_free_nolock() should clear .do_index_ops,
otherwise upon some failure the directory striping is freed, while a
subsequent dt_try_as_dir() skips striping initialization, and call
.do_index_ops directly, which will cause crash.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Ib94a4ef2f8bf5f0d34521abff77d8be46ecbf428
Reviewed-on: https://review.whamcloud.com/47256
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15776 tgt: fix transaction handling in tgt_brw_write()

Hotfix to prevent possible data loss during WRITE replay.
Since commit f0f92773ee18 from LU-14187 the obd_commitrw()
may restart write transaction in OFD and MDT. That causes
transaction number to be assigned multiple times if such
restart happens. Without flag tti_mult_trans the first
transaction number is stored only so later one could remain
not applied causing data loss after recovery.

Patch sets tti_mult_trans for tgt_brw_write() so the latest
transaction number will be used as request transno.

Fixes: f0f92773ee ("LU-14187 osd-ldiskfs: fix locking in write commit")
Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I364b478591942be5562c3e98ee6e6aa487f3e0c5
Reviewed-on: https://review.whamcloud.com/47371
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wang Shilong <wangshilong1991@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15887 test: add always_except()

In test-framework.sh, add a new function (always_except()) to replace
manual manipulation of $ALWAYS_EXECPT. Convert sanity.sh to use
always_except() and add a line to contrib/scripts/spelling.txt to
suggest its use.

Test-Parameters: trivial
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I1b39fe9555bab59e70db00cef73d13102668500a
Reviewed-on: https://review.whamcloud.com/47452
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15722 osd-ldiskfs: fix IO write gets stuck for 64K PAGE_SIZE

This fixes below IO write stuck issue:
-----
[606895.151765] LustreError:
334886:0:(ofd_io.c:1389:ofd_commitrw_write()) lustre-OST0000: restart IO
write too many times: 10000
[606895.207345] LustreError:
334886:0:(ofd_io.c:1389:ofd_commitrw_write()) Skipped 8 previous similar
messages
-------

Which goes into an infinite loop:
ofd_commitrw_write()->osd_write_commit()->osd_ldiskfs_map_inode_pages()
    ->ldiskfs_map_blocks()->ofd_commitrw_write()

The cause is that:
For 64K PAGE_SIZE blocks allocation/mapping. m_lblk should be the
first un-allocated block if m_lblk points at an already allocated
block when create = 1, ldiskfs_map_blocks() will just return with
already allocated blocks and without allocating any new requested
blocks for the extent.

This stuck issue won't happen on 4K PAGE_SIZE. Because for
PAGE_SIZE = blocksize case, if m_lblk points at an already
allocated block it will point at an un-allocated block in next
restart transaction, because the already mapped block/page will
be filtered out in next restart transaction via flag
OBD_BRW_DONE in osd_declare_write_commit().

Change-Id: Iadba0be8875a15a2e2f158ec9571f5ece5637ae0
Signed-off-by: Xinliang Liu <xinliang.liu@linaro.org>
Reviewed-on: https://review.whamcloud.com/47004
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15658 lod: ost list and pool name conflict

If the OST list is given on setstripe with the -o option, the pool is
unconditionally dropped even if all the OSTs are in. Let the pool stay
in this case.

Also, if the start index given on setstripe with the -i option is out
of the pool, make it similar to the -o option - drop the pool.

HPE-bug-id: LUS-10868
Fixes: b384ea39e5 ("LU-14480 pool: wrong usage with ost list")
Signed-off-by: Vitaly Fertman <vitaly.fertman@hpe.com>
Change-Id: I718c237e273689048eb74044eea73de6c212395e
Reviewed-on: https://review.whamcloud.com/46967
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Etienne AUJAMES <eaujames@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15683 ofd: proper initialize filter_fid in ofd fallocate

Intialize filter_fid buffer and call xattr set XATTR_NAME_FID properly
in ofd_object_fallocate().

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Ied573c39dde77f935622e9fbedb2d71eb3bd8f5d
Reviewed-on: https://review.whamcloud.com/46919
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15519 quota: fallocate does not increase projectid usage

fallocate() was not accounting for projectid quota usage.
This was happening due to two reasons. 1) the projectid
was not properly passed to md_op_data in ll_set_project()
and 2) the OBD_MD_FLPROJID flag was not set receive the
projctid.

This patch addresses the above reasons.

Test-case: sanity-quota/78a added

Fixes: 48457868a02a ("LU-3606 fallocate: Implement fallocate preallocate operation")
Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Change-Id: I3ed44e7ef7ca8fe49a08133449c33b62b1eff500
Reviewed-on: https://review.whamcloud.com/46676
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15487 mdd: print FID in mdd_dir_page_build() error

Print the MDT name and FID in mdd_dir_page_build() when an error
is hit.  Because this changes the callback function signature,
also update dt_index_page_build() to print a more useful message.

Add OBD_FAIL_MDS_DIR_PAGE_WALK to allow triggering this codepath
to see if this is the source of problems in error handling.

Minor code style and whitespace cleanups in related functions.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ic475f4a2c775871ff5af59a47e0966ba3eed7013
Reviewed-on: https://review.whamcloud.com/46368
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15284 llite: access lli_lsm_md with lock in all places

lli_lsm_md should be accessed with lock in all places. Among all the
changes, ll_rease_page() is inside lock already, except statahead
code.

Test-Parameters: mdscount=2 mdtcount=4 testlist=racer,racer,racer
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I1e09402812ce51ce7ab80d9529d488cb5b2879ee
Reviewed-on: https://review.whamcloud.com/46355
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15278 lod: distinguish DIR/REGULAR lod_object members

In lod_striping_free_nolock(), we need to distinguish lod_object
type, since DIR/REGULAR lod_object structure share the same memory
region, it could accidently free some unintended memory if it treat
DIR lod_object as REGULAR one, or vice versa.

Fixes: 6a20bdcc608b ("LU-11376 lov: new foreign LOV format")
Fixes: fdad38781ccc ("LU-11376 lmv: new foreign LMV format")
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I2d4c563725b35f7a75f0f1fbf9c1d35b1799eff4
Reviewed-on: https://review.whamcloud.com/45710
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10391 socklnd: switch ksocknal_del_peer to lnet_processid

ksocknal_del_peer now takes a pointer to a lnet_processid,
with room for a large address.
A NULL means "ANY NID, AND PID".
The "ip" argument was completely unused, so has been removed.

This was the last use of 'struct lnet_process_id' in ksocklnd.

Test-Parameters: trivial testlist=sanity-lnet
Test-Parameters: serverversion=2.12 serverdistro=el7.9 testlist=runtests
Test-Parameters: clientversion=2.12 testlist=runtests
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I2f3879fdafc9b1effd540af3644febf0d06eb5e2
Reviewed-on: https://review.whamcloud.com/44623
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10391 socklnd: large processid for ksocknal_get_peer_info

Have ksocknal_launch_packet() report a 'struct lnet_processid'
with a large address.

Test-Parameters: trivial testlist=sanity-lnet
Test-Parameters: serverversion=2.12 serverdistro=el7.9 testlist=runtests
Test-Parameters: clientversion=2.12 testlist=runtests
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I6dd4b11e0361d893dea519287448028ca0a1ab97
Reviewed-on: https://review.whamcloud.com/44622
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10391 socklnd: pass large processid to ksocknal_add_peer

Teach ksocknal_add_peer() to handle large-address processid, and now
ksocknal_launch_packet() can support IPv6 addresses as well as IPv4.

Test-Parameters: trivial testlist=sanity-lnet
Test-Parameters: serverversion=2.12 serverdistro=el7.9 testlist=runtests
Test-Parameters: clientversion=2.12 testlist=runtests
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I90655b8d6e1a2e9fc38a7bf9d492542f76086c70
Reviewed-on: https://review.whamcloud.com/44621
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10391 lnet: switch LNetIsPeerLocal() to take 16-byte addr

LNetIsPeerLocal() now takes a 'struct lnet_nid'

Test-Parameters: trivial
Test-Parameters: serverversion=2.12 serverdistro=el7.9 testlist=runtests
Test-Parameters: clientversion=2.12 testlist=runtests
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Id72a92cda510de6864c82d88912c582745cc9727
Reviewed-on: https://review.whamcloud.com/43621
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10391 lnet: change LNetGet to take 16byte nid and pid.

"self" is now passed to LNetGet as a pointer to a 16-byte-addr nid, or
NULL for "ANY".  "target" is passed as a 16-bytes-addr process_id.

Test-Parameters: trivial
Test-Parameters: serverversion=2.12 serverdistro=el7.9 testlist=runtests
Test-Parameters: clientversion=2.12 testlist=runtests
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I8e0fcd442d5195991b799a8db3ec8030c81f9400
Reviewed-on: https://review.whamcloud.com/43620
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10391 lnet: convert LNetPut to take 16byte nid and pid.

LNetPut() now takes a 16byte nid for self and similar process_id for
target.

Test-Parameters: trivial
Test-Parameters: serverversion=2.12 serverdistro=el7.9 testlist=runtests
Test-Parameters: clientversion=2.12 testlist=runtests
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I240caf6fb8b02b1814b9d4883aceda33894786a4
Reviewed-on: https://review.whamcloud.com/43619
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10391 lnet: Change LNetDist to work with struct lnet_nid

LNetDist now takes and returns 'struct lnet_nid'
lustre_uuid_to_peer() is also updated.

The 'dst' and 'src' parameters to LNetDist are now both pointers, and
that can point to the same 'struct lnet_nid'.  Code needs to be
careful not to set *src until after the last use of *dst.

Test-Parameters: trivial
Test-Parameters: serverversion=2.12 serverdistro=el7.9 testlist=runtests
Test-Parameters: clientversion=2.12 testlist=runtests
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I267f3333e3164778a7df39d6b90f0a9a913fcdcf
Reviewed-on: https://review.whamcloud.com/43618
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10391 lnet: alter lnet_drop_rule_match() to take lnet_nid

The local nid passed to lnet_drop_rule_match() is now a 16-byte nid.
Various support functions are also changed to embrace 'struct
lnet_nid'.

Test-Parameters: trivial
Test-Parameters: serverversion=2.12 serverdistro=el7.9 testlist=runtests
Test-Parameters: clientversion=2.12 testlist=runtests
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I738bff9cfc8c5a70c736639fdd64a66d2aded186
Reviewed-on: https://review.whamcloud.com/43617
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10391 lnet: change LNetPrimaryNID to use struct lnet_nid

Rather than taking and returning a 4-byte-addr nid, LNetPrimaryNID now
takes a pointer to a struct lnet_nid, and updates it in-place.

Test-Parameters: trivial
Test-Parameters: env=SHARED_KEY=true testlist=sanity,sanity-sec
Test-Parameters: serverversion=2.12 serverdistro=el7.9 testlist=runtests
Test-Parameters: clientversion=2.12 testlist=runtests
Change-Id: I74f193e5c533125c282f230d272a506129baa365
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Reviewed-on: https://review.whamcloud.com/43616
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14627 utils: quiet spurious lustre_rmmod message

Quiet unnecessary "LNET ready to unload" message from lustre_rmmod.

Lustre-change: https://review.whamcloud.com/47116
Lustre-commit: TBD (from ee7aba2beabbf983ccffe8e4881e792943a15b09)

Fixes: 32304d863ae98 ("LU-14627 tests: Create unload_modules_local")
Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I02efa36bbc1a3f4ab63767176ef53956dcafa589
Reviewed-on: https://review.whamcloud.com/47116
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15000 llog: read canceled records in llog_backup

llog_backup() do not reproduce index "holes" in the generated copy.
This could result to a llog copy indexes different from the source.
Then it might confuse the configuration update mechanism that rely on
indexes between the MGS source and the target copy.

This index gaps can be caused by "lctl --device MGS llog_cancel".

This patch add "raw" read mode to llog_process* to read canceled
records. So now llog_backup is able to reproduce an exact copy of
the original.

Signed-off-by: Etienne AUJAMES <etienne.aujames@cea.fr>
Change-Id: I811e23de8f4545bed36a44fedc2638d7418830dd
Reviewed-on: https://review.whamcloud.com/46552
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Dominique Martinet <qhufhnrynczannqp.f@noclue.notk.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: DELBARY Gael <gael.delbary@cea.fr>
Reviewed-by: Stephane Thiell <sthiell@stanford.edu>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15542 osd-ldiskfs: exclude EA inode from processing

EA inode is ldiskfs internal object and this should be
excluded at any osd access.
Panic fixed by this change because iterator will return
an error when tries to access to the ea inode.

osd_ea_fid_get()) Process leaving (rc=18446744073709551614 : -2
osd_it_ea_rec()) Process leaving (rc=18446744073709551614 : -2)
osd_it_ea_next()) Process entered

HPe-bug-id: LUS-10683
Signed-off-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Change-Id: I73e5325721ae28ea380e9d216c6c6cf7fa0ac4ea
Reviewed-on: https://review.whamcloud.com/46486
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andrew Perepechko <andrew.perepechko@hpe.com>
Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15502 mdt: cleanup whitespace in mdt_open.c

Change spaces to tabs for indent and do not align local variables.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I1b89ffd54d41ce42738b061738f990d0326bf9af
Reviewed-on: https://review.whamcloud.com/46385
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15189 lnet: fix memory mapping.

Nvidia GDS have a bug which caused incorrect page type detect.
It may return an GPU flag for the kmalloc buffer (ptlrpc_message
in my case).
To workaround this - Whamcloud have both mapping calls, but it's
costly and caused an extra RDMA operations as ko2iblnd trust
an msg_rdma_force flag.
Lets drop extra Nvidia calls and check just an real "user" pages
or GPU flag.

HPe-bug-id: LUS-10520
Signed-off-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Change-Id: I5d70c5e0630b0f16e130a7db0385de2443c11a63
Reviewed-on: https://review.whamcloud.com/45482
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15189 build: add GDS configure options

add ability to specify GDS patch, pickup an GDS API directly from the
GDS sources.

Test-Parameters: trivial
Signed-off-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Change-Id: Iccc52765540cc90c6a69af34d230ed24b2eb996a
Reviewed-on: https://review.whamcloud.com/45480
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Reviewed-by: Andrew Perepechko <andrew.perepechko@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14642 test: add fsx mirror file test mode

- add fsx mirror file test mode with "-M" option so that fsx can exert
  its IO to FLR file as well as extend/split/resync the FLR file.

- add sanity-flr test_70b() to test fsx with flrmode.

- fix a bug in "lfs mirror verify" to accomodate max mirror count
  instead of (max - 1) mirrors.

- improve "lfs mirror verify -v" print proper data range of its crc-32
  checksum values.

Test-Parameters: trivial testlist=sanity-flr env=ONLY=100,FSXNUM=10000
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Ib55c7b25dcd82fa0b197ad21268b16c82aab5da9
Reviewed-on: https://review.whamcloud.com/43473
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10073 tests: re-enable lnet selftest smoke test for PPC + ARM

LNet selftest has failed on platforms with 64K pages. Recent work
by Alexy Lyashkov should have resolved these issues. Lets enable
the lnet selftest to valid these changes.

Test-Parameters: trivial clientdistro=el8.5 clientarch=aarch64 testlist=lnet-selftest
Test-Parameters: trivial clientdistro=el8.5 clientarch=ppc64le testlist=lnet-selftest

Change-Id: I97684101ea2f90bb41f452955d5f864ca35d61ec
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/38857
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: xinliang <xinliang.liu@linaro.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15404 ldiskfs: port truncate fix to Ubuntu 20 HWE

A fixed landed for the LU-15404 bug which a truncate vs setxattr
race caused a kernel crash. This fix wasn't added in for new
Ubuntu 20 HWE kernels and the 5.10 kernel.

Change-Id: I43b33cd28335d1bc5796aa6651ecf847b4ea24e6
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/46480
Reviewed-by: Andrew Perepechko <andrew.perepechko@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15858 sec: reinstate null encryption for file names

Reinstate null encryption for file names by adding a new llite
parameter named 'enable_filename_encryption', set to 0 by default.
When this parameter is 0, new empty directories configured as
encrypted ignore the filenames_encryption_mode and use
LLCRYPT_MODE_NULL instead, which is a no-op. This LLCRYPT_MODE_NULL
mode is inherited for all subdirectories and files.
When this parameter is 1, new empty directories configured as
encrypted use the normal encryption mode.

To set this parameter globally for all clients, do on the MGS:
mgs# lctl set_param -P llite.*.enable_filename_encryption=0

Also update sanity-sec test_54 to exercise the new parameter
'enable_filename_encryption'.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I9d726ba26cc91a51690d59a81efe3eb98ee2995c
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47355
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15848 ldiskfs: escape encrypted file names

When a Lustre MDT is mounted as ldiskfs, the names of the encrypted
files have to be escaped to avoid breaking the shell.
On CentOS 7, the LDISKFS_ENCRYPT_FL flag does not exist. So we add it,
and when the target is mounted as ldiskfs (LDISKFS_MOUNT_DIRDATA flag
not present) we critical-encode encrypted file names before
presentation. And conversely, we critical-decode names upon lookup.
On CentOS 8, the LDISKFS_ENCRYPT_FL flag exists. The fscrypt functions
from kernel 4.18 are also wired up, but they all refer to -EOPNOTSUPP
or equivalent, so they cannot be used to present usable names. So when
the target is mounted as ldiskfs (LDISKFS_MOUNT_DIRDATA flag not
present) we proceed to critical-encoding of encrypted file names
before presentation, and to critical-decoding upon lookup.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Iaca467eaa233be8142356efa822962953754c2ce
Reviewed-on: https://review.whamcloud.com/47309
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14541 llite: Check vmpage in releasepage

We cannot release a page if the vmpage reference count is
>1, otherwise we will detach a vmpage from Lustre when the
page is still referenced in the VM.

This creates a situation where page discard for lock
cancellation will not find the page, so we can get stale
data reads.

This re-introduces the LU-12587 issue where direct I/O on
a client falls back to buffered I/O if there are pages in
cache, since it cannot flush them.  This is annoying but
not a huge problem.

Fixes: e59f0c9a245f ("LU-12587 llite: don't check vmpage refcount in ll_releasepage()")
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I3aa1cd7330f5e7d1ba2ddb0c12779aa22f3d70b7
Reviewed-on: https://review.whamcloud.com/47262
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>

LU-14541 llite: add rw_seq_cst_vs_drop_caches

Add a reproducer (rw_seq_cst_vs_drop_caches) for the read/write vs
drop_caches sequnetial consistency violation described in
LU-14541. Add an always excepted test (sanityn test_16f) to run
rw_seq_cst_vs_drop_caches.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I557ae7386b38214110a4d85ba0515e95fed7a11e
Reviewed-on: https://review.whamcloud.com/47243
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14541 llite: reenable fast_read by default

Reenable fast_read by default.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I9985beb88d13d21ee538c632fc288470a58b0257
Reviewed-on: https://review.whamcloud.com/47298
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15876 test: restore missing spaces in ALWAYS_EXCEPT

In sanity-flr, restore missing spaces in ALWAYS_EXCEPT.

Fixes: a45fe93cd8 ("LU-14765 test: disable sanity-flr/44c")
Test-Parameters: trivial testlist=sanity-flr fstype=zfs
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Id2fa88b72d021cf114a80ae7dbd1f80bfb31a8b2
Reviewed-on: https://review.whamcloud.com/47406
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14765 test: disable sanity-flr/44c

file blocks number accounting does not include blocks occupied by
multiple mirrors, disable the test for now.

Test-Parameters: trivial
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I1a5d7e0c203d5c7dc0da53ce6c020795bf17b9fc
Reviewed-on: https://review.whamcloud.com/44074
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13578 test: sleep longer in sanity test_39

In sanity test_39r(), sleep for 2 * atime_diff rather than atime_diff + 1.

Test-Parameters: trivial testlist=sanity env=ONLY=39r,ONLY_REPEAT=50
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Ied508e12c848f6935d2317fb86bddc5341a6156e
Reviewed-on: https://review.whamcloud.com/47346
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-11170 test: increase time limit in sanity test_415

In sanity test_415() double the time limit on renaming 500
files. Really we should not be doing arbitrary performance tests using
VMs on oversubscribed test nodes but whatever.

Test-Parameters: trivial testlist=sanity env=ONLY=415,ONLY_REPEAT=10
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Ie2677df00f13ffc85d21ceba5ba7115fff6f0980
Reviewed-on: https://review.whamcloud.com/47341
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15757 test: disable sanityn test_102() for zfs

sanityn test_102() on zfs causes clients to crash on umount about half
the time. Disable this test until this can be sorted out.

Test-Parameters: trivial testlist=sanityn fstype=zfs mdtcount=4
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: If9f7c77f8c68eee88b3f050fc26c42c21828e2c9
Reviewed-on: https://review.whamcloud.com/47311
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15830 mdt: mkdir to lookup target name

distributed mkdir should lookup the target name to
avoid rollback as much as possible as the latter is
very expensive due to llog re-initialization.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: If28760e0afb804dca11e1e7501e0a53ff9067ca1
Reviewed-on: https://review.whamcloud.com/47226
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15757 llite: check s_root ll_md_blocking_ast()

ll_md_blocking_ast() can be called in the context of import
invalidation which in turn caused by umount. this way
ll_md_blocking_ast() and umount can race and ll_md_blocking_ast()
can found sb->s_root NULL which should be checked before
calling into is_root_inode().

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I61c1d29a7de3084ad1dfd0c216cee628418b7038
Reviewed-on: https://review.whamcloud.com/47086
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15827 osd: respect filldir buffer limits

In osd_ldiskfs_filldir() ensure that the encoded name also fits in the
buffer. In osd_ldiskfs_filldir() and obj_name2lu_name() remove a
superfluous and potentially incorrect check on names that appear to be
FIDs.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I1540f058801b002474ceac1206317344cebb1084
Reviewed-on: https://review.whamcloud.com/47224
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Revert "LU-12019 build: Recognize Debian Kernel and set KMP dir"

This reverts commit 230d4500d5a9dfada392199d77fc413382f24750.

Revert this commit because lustre failed to load on MOFED 5.5
See LU-15831 for details

Change-Id: I845431ad2126743c1ca9a59d1b56e1a35dbc9e38
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47238
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: Oleg Drokin <green@whamcloud.com>

LU-15810 sec: fix enc dir migration

Now that the encryption context is stored in an xattr named
"encryption.c" instead of "security.c", we need to fetch this xattr
explicitly in case of encrypted directory migration. Indeed, there is
no xattr handler in ldiskfs for this "encryption." xattr type, so it
is not returned when listing all xattrs to migrate.

Fixes: 4231fab66e ("LU-13717 sec: make client encryption compatible with ext4")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I628f9b253e86343db0b71f6a5b1ad2c5728ca38d
Reviewed-on: https://review.whamcloud.com/47201
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Post-branch-off of 2.15 master is now 2.16

Change-Id: I93ff129b038dff4f541ca9497ba21b77eaf82de8
Signed-off-by: Oleg Drokin <green@whamcloud.com>

LU-15787 sec: document enc-unaware clients on enc files

Document the behavior of encryption-unaware clients when they access
encrypted files.

Test-Parameters: trivial
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I0354e3051e10aa0542baeb8e34c6201d47e65710
Reviewed-on: https://review.whamcloud.com/47182
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

New RC 2.15.0-RC4

Change-Id: Iabd55b89a5551e5f42f7f6b112c7a7187da8ae7b
Signed-off-by: Oleg Drokin <green@whamcloud.com>

LU-15815 llite: disable fast_read and workaround

Revert the fast_read stale data workaround from LU-14541 and disable
fast_read by default. The workaround causes applications to receive
spurious SIGBUGs when reclaim is concurrent with mmap page fault
handlers. We disable fast read to avoid the stale data issue entirely.

This reverts commit f2a16793fa4316fc9ccdc46bcfe54f6b8d1e442b and
re-exposes us to the consistency issues described in LU-14541.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I716b1fd6ab22242b9267b8883f0371a360aaecef
Reviewed-on: https://review.whamcloud.com/47204
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15761 obdclass: fix locking in llog_cat_refresh()

the patch fixes two problems:
1) pairing up_write() should be used with cathandle
2) llog_read_header() manipulates loghandle's internal
   structures (header, last_idx, etc) which are supposed
   to stay consistent from another user's point of view
   (like llog_add_rec())

Fixes: 71f409c9b31b ("LU-11418 llog: refresh remote llog upon -ESTALE")
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Ib86e10a925b541d02c22d74e6ddbc4368345ac11
Reviewed-on: https://review.whamcloud.com/47185
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15803 sec: correctly handle page lock in ll_io_zero_page

In ll_io_zero_page(), we need to make sure we have locked the page,
and it is up-to-date, before zeroing. So modify ll_io_read_page()
behavior to not disown the clpage for our use case. It avoids being
exposed to concurrent modifications.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I58e4cf80374a798c9c4302364cf2fb39da9033bb
Reviewed-on: https://review.whamcloud.com/47170
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15787 sec: block enc unaware clients on enc files

Prevent encryption unaware clients from manipulating encrypted files
and directories. Those can be old clients, or clients built without
encryption support (intentionally or because they run on an old
kernel).
In the mdt layer, check that clients have the OBD_CONNECT2_ENCRYPT
connection flag, and if not, block access if they try to manipulate
a file or directory that has the LUSTRE_ENCRYPT_FL flag.
The forbidden operations from encryption unaware clients are:
- open
- create
- link
- rename
- migrate
Improve sanity-sec test_54 to test this use case.

Test-Parameters: testlist=sanity-sec mdscount=2 mdtcount=4 osscount=1 ostcount=8 clientcount=2 serverdistro=el7.9
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ief0639e49c0a8e1a1a0cb19cb13c006edfdff6c4
Reviewed-on: https://review.whamcloud.com/47156
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15479 tests: clean up sanity test_316/test_319

These tests use the deprecated "lfs mv" insted of "lfs migrate".

Fix test_316 to specify which MDT to create 'd' on, so MDT space
balancing does not result in 'd' being created on MDT0001, which
defeats the purpose of the test trying to migrate it to MDT0001.

Fix test_319 to call "lfs migrate" on the parent directory and
not on the test file, since this returns an "ENOTDIR" error.

Also fix some minor script style issues.

Test-Parameters: trivial testlist=sanity env=ONLY="316 319",ONLY_REPEAT=100 mdscount=2 mdtcount=4
Test-Parameters: fstype=zfs testlist=sanity env=ONLY="316 319",ONLY_REPEAT=100 mdscount=2 mdtcount=4
Fixes: ae7ca90713b4 ("LU-11926 ldlm: Lost lease lock on migrate error")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I08d55ce70cbd900c2f4d31b27baa1a33423ebbe5
Reviewed-on: https://review.whamcloud.com/46959
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15740 tests: add more stats to runtests

Print out the space usage at the start, middle, and end of
runtests, so that it is easier to see where the space is
going, and how much is used at peak consumption.  The goal
of the test is to avoid space leakage in object create and
destroy, but it is OK if there is some usage for internal
files like llogs, quotas, etc.

Move the "mkdirmany" call to the first phase, add a statmany
to the middle phase, and "rmdirmany" to the end so that it
is also checking the directory validity after a remount.

Update createmany and statmany to be a bit easier to use.

Test-Parameters: trivial testlist=runtests env=SLOW=yes mdscount=2 mdtcount=4
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Iec8cb56501c7e75b620951a2d669b0dd6bb0a36f
Reviewed-on: https://review.whamcloud.com/47065
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: cliff white <cwhite@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15575 tests: interop skip sanity/103e before 2.14.52

Older MDSes can *set* default ACLs with many entries, but it
causes an error when trying to *create* files that inherit them.
Skip test_103e for older MDS versions.

Test-Parameters: trivial testlist=sanity env=ONLY=103e serverversion=2.14.0
Fixes: aa92caa21fa2 ("LU-14430 mdt: fix maximum ACL handling")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I4fbed3f788020c6d158ba83496ebd5cd68ba57cb
Reviewed-on: https://review.whamcloud.com/47125
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15358 tests: fix check in sanityn test_80b

Shellcheck found checks in sanityn test_80b using bad logic:

        [ $rc -ne 0 -o $rc -ne 16 ] || {
                echo "touch file failed with $rc1"
                break;
        }

This can never be false, so the subtest will never detect errors.
Fix these checks, along with some related style issues.

Test-Parameters: trivial testlist=sanityn mdscount=2 mdtcount=4 env=ONLY=80,ONLY_REPEAT=50
Fixes: 220e6cbfa65c5 ("LU-6475 mdt: race between open and migrate")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I541fe6f3ae253ea1c4d7fa2bcfad9052e374e60c
Reviewed-on: https://review.whamcloud.com/46706
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14826 tests: fix sanityn.sh test_102 version check

The MDS version check in test_102 had a syntax error, found
by 'shellcheck'.

Also quiet spurios error messages from check_fhandle_syscalls
if the "lctl" binary cannot be found.

Fixes: cbc62b0b829 ("LU-14826 mdt: getattr_name("..") under striped directory")
Test-Parameters: trivial testlist=sanityn mdscount=2 mdtcount=4 env=ONLY=102,ONLY_REPEAT=100
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Iee432bc147f63809fea095c4ce9e1694f7ce7057
Reviewed-on: https://review.whamcloud.com/46693
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15645 obdclass: llog to handle gaps

due to old errors an update llog can contaain gaps in index.
this shouldn't block llog processing and recovery. actual
gaps in transaction sequence should be catched by VBR.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I11ec817e356f9658118c34706ef3a533e7faba83
Reviewed-on: https://review.whamcloud.com/46837
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15748 osc: fallocate interop for 2.14 clients

fallocate() start and end are passed in o_size and o_blocks
on the wire.  Clients 2.15.0 and newer should always set
the OBD_MD_FLSIZE and OBD_MD_FLBLOCKS valid flags, but some
older client versions did not.  We permit older clients to
not set these flags, checking their version by proxy using
the missing OBD_CONNECT_TRUNCLOCK to imply 2.14.0 or older.

Test-Parameters: serverversion=2.14.0 testlist=sanity env=SANITY_EXCEPT="64h 103e"
Fixes: 2f496148c31d ("LU-15551 Return EOPNOTSUPP instead of EPROTO")
Fixes: 163870abfb7c ("LU-14382 mdt: implement fallocate in MDC/MDT")
Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Change-Id: I1ea47854f40d54297bceb03ad32b24737efa4ae7
Reviewed-on: https://review.whamcloud.com/47098
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>

LU-15719 doc: Update changelog to properly reflect supported kernels

According to https://wiki.lustre.org/Release_2.15.0

Also update e2fsprogs version from the same document

Change-Id: Ife26db67b543bb83651b56df41beaf81fea97721
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46993
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Peter Jones <pjones@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

New RC 2.15.0-RC3

Change-Id: If8b76076bedd13b39dac94ab582e1855feb6add4
Signed-off-by: Oleg Drokin <green@whamcloud.com>

LU-15670 clio: Disable lockless for DIO with O_APPEND

Lockless O_DIRECT with O_APPEND can allow interleaved / racy
appends from concurrent I/O.

Disable lockless I/O when O_APPEND is set

HPE-bug-id: LUS-9776
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I5c56f92c90e631c295f56e5958985f516e1990f8
Reviewed-on: https://review.whamcloud.com/46890
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15616 lnet: ln_api_mutex deadlocks

LNetNIFini() acquires the ln_api_mutex and holds onto it throughout
various shutdown routines. Meanwhile, LND threads (via
lnet_nid2peerni_locked()) or the discovery thread (via
lnet_peer_data_present()) may need to acquire this mutex in order to
progress.

Address these potential deadlocks by setting the_lnet.ln_state to
LNET_STATE_STOPPING earlier in LNetNIFini(), and release the mutex
prior to any call into LND module or before any wait.

LNetNIInit() is modified to return -ESHUTDOWN if it finds that there
is a concurrent shutdown in progress.

HPE-bug-id: LUS-10681
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: Ia8b28cc95ff71e66a0f99aed4f2c22ec9d44ce1e
Reviewed-on: https://review.whamcloud.com/46727
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13714 lnet: only update gateway NI status on discovery

Move the NI status from DOWN to UP only when receiving
a discovery PING. The discovery PING should be the only
message which should update the NI status since it's used
as the gateway NI keep alive mechanism.

This is done to avoid the following scenario:

The gateway itself can push its updates to the peers which
have removed it from its routing table. The peers would
respond to the PUSH with an ACK, the ACK will bring the
gateway's NI status to up. Therefore other peers which have
avoid_asym_router_failure=1 will have their route status
remain up even though the symmetrical route is gone.

Note: there is no way for the gateway to differentiate between
a keep alive discovery and a manually triggered discovery or ping.
However, this a narrow case which will not be handled.

net_last_alive converted to use ktime_get_seconds() instead of
ktime_get_real_seconds() since the NTP adjustment is not needed.

Test-Parameters: trivial
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: Ifd5b06d4cf783b68b36413ada63f0a1d0095fb5b
Reviewed-on: https://review.whamcloud.com/39176
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15363 tests: don't use lustre module to test lnet

Currently sanity-lnet.sh loads the lustre modules to properly
initialize the lnet modules. This doesn't work with the native
Linux client since it only starts up LNet after mounting the
file system. We shouldn't be using lustre to test lnet so
load lnet modules with config_on_load option to properly setup
the default LNet configuration.

Also fix ksocklnd-config to use bash so sanity-lnet.sh can
pass on Ubuntu.

Test-Parameters: trivial testlist=sanity-lnet
Change-Id: Ifffc51625f5c2ffbb3ab811b75739c0e6407a821
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/45834
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15545 utils: fix double free in lgss_sk

Fix double free issue in lgss_sk if write_config_file fails.

Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial
Change-Id: Icd1673b27d699fba78b01fe53f587291e6c36ed6
Reviewed-on: https://review.whamcloud.com/46594
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>

LU-15692 lmv: change default hash back to fnv_1a_64

Until performance issue is resolved, change the default directory
hash type from 'crush' back to 'fnv_1a_64'.

Fixes: bb60caa1c6e7 ("LU-14459 lmv: change default hash type to crush")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: If2cef70298773f42dc1c62809eea98519b3ebbe5
Reviewed-on: https://review.whamcloud.com/46950
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Shuichi Ihara <sihara@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>