LU-13296 obd: make statfs cache working again

Once statfs raced on mutex, lets read a cached data instead
of trash.

Test-Parameters: testlist=sanity envdefinitions=ONLY=423,ONLY_REPEAT=500
Fixes: 1c41a6ac390b ("LU-12368 obdclass: don't send multiple statfs RPCs")
Signed-off-by: Alexey Lyashkov <c17817@cray.com>
Change-Id: I268782875c30c078f239c194f69cdf7506d66169
Reviewed-on: https://review.whamcloud.com/37753
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>

LU-13209 build: Fix vvp_account_page_dirtied

Fix compile error vvp_account_page_dirtied undefined

Fixes: 788e464a72 ("LU-13288 llite: Find account_page_dirtied on module init")
Cray-bug-id: LUS-8472
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: Iae0a3af1b091165423a2bb24a9159a7af8ab3cbe
Reviewed-on: https://review.whamcloud.com/37778
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13241 utils: use libext2fs for ldiskfs operations

Instead of using debugfs to stat and read the CONFIGS/mountdata,
we can switch to libext2fs and control the flags used when
opening the ldiskfs, to reduce the mount time on huge targets.

Change-Id: I9da8fc1c77d149fe5cf3bd19b0ff76d892620101
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-on: https://review.whamcloud.com/37656
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13314 test: add 56ob to ALWAYS_EXCEPT for now

There is a problem that 'lfs find' try to calculate
365 days for one year, this will be problem for leap year.

We need fix test or codes, before that let's disable this test
to make other patches landed possibly.

Change-Id: I79d34ce29657b4d149a0fbe82220cfefa0c60378
Test-Parameters: trivial testlist=sanity envdefinitions=ONLY=56ob
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/37767
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-12988 ldiskfs: mballoc to prefetch groups

ahead of scanning. prefething is done in 8 * flex_bg groups, so
it should be 8 read-ahead reads for a single allocating thread.
at the end of allocation the allocating thread waits for read-ahead
completion and initializes buddy information so that read-aheads
are not lost in case of memory pressure.
at cr=0 the number of prefetching IOs is limited per allocation
context to prevent a situation when mballoc loads thousands of
bitmaps looking for a perfect group and ignoring groups with
good chunks.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Ibce9a060900544abe994f4d7a20ac9c2979ccc56
Reviewed-on: https://review.whamcloud.com/37633
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>

LU-12043 llite: move tunable params to sysfs_memparse()

Move the max_read_ahead_* tunables from debugfs to sysfs, since
they follow the one-value-per-file rule and should be visible to
regular users.

Rename the functions and constants from *readahead* to *read_ahead*
or *READ_AHEAD* to match the tunable names from procfs.

Deprecate usage of llprocfs_str_with_units_to_s64(), lu_str_to_s64(),
llprocfs_str_with_units_to_u64(), and lu_str_to_u64(), and instead
use sysfs_memparse() to parse sizes in the few remaining places
where they are used.  A separate patch will remove those functions.

Minor fix to the "lctl set_param" man page.

Fixes: adb5aca3d673 ("LU-8066 llite: Move all remaining procfs entries to debugfs")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I2cdf5f8f0aeca458ed1989366102c33ae83ebbe5
Reviewed-on: https://review.whamcloud.com/34849
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13096 llite: fix potential overflow in ll_max_cached_mb_seq_write()

atomic_long_cmpxchg() return long, however @rc is int, if we have
a larger memory etc 24T, we will get overflow, and @diff will never
change thus we get a hang loop there.

Test-Parameters: trivial
Fixes: adb5aca3d673 ("LU-8066 llite: Move all remaining procfs entries to debugfs")
Change-Id: I20d6feff9797ba10a089587bee0d8691bee460df
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/37707
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13283 tests: add racer to nonmpi load

Patch adds the ability to run racer as one of
nonmpi loads. All racer parameters can be set over
RACERP:
Example:
  RACERP="MDSCOUNT=3 DURATION=600 RACER_ENABLE_STRIPED_DIRS=false"
etc.

ha_start_mpi_loads() is improved to not create
machinefiles if no mpi load set.

Test-Parameters: trivial
Cray-bug-id: LUS-8297
Signed-off-by: Elena Gryaznova <elena.gryaznova@hpe.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-by: Vladimir Saveliev <vladimir.saveliev@hpe.com>
Change-Id: I974838ce199897284396ef53d1a487d1a1ae1774
Reviewed-on: https://review.whamcloud.com/37671
Reviewed-by: Vladimir Saveliev <c17830@cray.com>
Reviewed-by: Alexandr Boyko <c17825@cray.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13281 tests: ha.sh improvements

Path adds the possbility to set the different layouts
for clients directories. This allows to run the clients loads
on the directories with DoM, PFL, SEL, etc. including the simple
layouts set on old clients, which is useful for inter-operation
testing.

Patch adds the possibility to split the full client list to
separate subsets by specifying NCLIENTSSET equal to 1 by
default. For NCLIENTSSET=2 two subsets will be created, each
one are passed to corresponding machinefile. This allow
to split the loads which is also useful for inter-operation:
old clients can operate with directories with old layouts while
the new clients operate with directories with new DoM, PFL, SEL,
etc. layouts.
For NCLIENTSSET=2, ${#ha_clients[@]}=4 and
CLIENTSSTRIPE='"-E $((64*1024)) -L mdt -E EOF" "-c -1 "'
layout "-E $((64*1024)) -L mdt -E EOF" will be applied on
even clients ${ha_clients[0]} and ${ha_clients[2]},
layout "-c -1 " will be applied on odd clients ${ha_clients[1]}
and ${ha_clients[3]}.

Test-Parameters: trivial
Cray-bug-id: LUS-6906
Signed-off-by: Elena Gryaznova <elena.gryaznova@hpe.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-by: Vladimir Saveliev <vladimir.saveliev@hpe.com>
Change-Id: Ia6f8c07e3936ea773706c3e79217672c68d77bd0
Reviewed-on: https://review.whamcloud.com/37666
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexandr Boyko <c17825@cray.com>
Reviewed-by: Vladimir Saveliev <c17830@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13219 tests: add nfs-server service in setup-nfs.sh

For RHEL 8.1, the NFS server service is nfs-server
instead of nfs.

Test-Parameters: trivial \
clientdistro=el8.1 serverdistro=el8.1 \
testlist=parallel-scale-nfsv3,parallel-scale-nfsv4

Change-Id: I5a97c8fe419187412dfc02047ed66141f567d7fb
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/37663
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13270 tests: dom-performance fixes

Patch makes the fs cleanup at the start of dom-performance
optional and depending on t-f global DO_CLEANUP option
value. For DO_CLEANUP=false script removes only the
directories created by previous dom-performance session.
Without this fix test run 2nd time fail with:
  lfs setdirstripe: dirstripe error on '/mnt/testfs/dp_dne':
      stripe already set
  lfs setdirstripe: cannot create stripe dir '/mnt/testfs/dp_dne':
      File exists

Patch removes the check of the files createmany, statmany
and smalliomany which are part of lustre/tests. With
the existing check test_smallio() is always skipped
if run on PWD != lustre/tests.

Patch removes the comparison with MDSSIZE and adds the
comparison with real mds size. Without this fix the test
works incorrectly if run on existing fs (formatall was not
done during this session: Lustre was created with mds size
differs from MDSSIZE).

Add skip_env() to report about missing mdtest/dbench/IOR/etc.
Without this fix  the tests are skipped silently.

Test-Parameters: trivial mdssizegb=20 testlist=dom-performance
Cray-bug-id: LUS-7349
Signed-off-by: Elena Gryaznova <elena.gryaznova@hpe.com>
Reviewed-by: Vladimir Saveliev <vladimir.saveliev@hpe.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Change-Id: I5fdf1ad480edb17598cbe427bc550396ebe97808
Reviewed-on: https://review.whamcloud.com/37644
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13269 tests: make lnet-selftest.sh more flexible

lnet-selftest.sh adds both LST tests: from client to server and
from server to client, but sometimes we are interesting to execute
the tests only from server or only from client.

Patch allows to set required "from" parameter via lst_FROM variable.
The following values are used:
lst_FROM=c : add the tests from clients to server only
lst_FROM=s : add the tests from server to clients only

Test-Parameters: testlist=lnet-selftest envdefinitions=lst_FROM=c
Cray-bug-id: LUS-8178
Signed-off-by: Elena Gryaznova <elena.gryaznova@hpe.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Vladimir Saveliev <vladimir.saveliev@hpe.com>
Change-Id: I0111a4386b3d8acb022db11f11a0be970864696d
Reviewed-on: https://review.whamcloud.com/37640
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Vladimir Saveliev <c17830@cray.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13268 tests: customize lnet-selftest for performance

Sometimes we need to run read/write/ping tests separately
with different lst check flags.

Patch allows to:
  set the required list of tests via new lst_TESTS variable;
  set lst check to LST_BRW_CHECK_NONE, LST_BRW_CHECK_FULL
      or LST_BRW_CHECK_SIMPLE.

Test-Parameters: trivial testlist=lnet-selftest
Cray-bug-id: LUS-8005
Signed-off-by: Elena Gryaznova <elena.gryaznova@hpe.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Reviewed-by: Vladimir Saveliev <vladimir.saveliev@hpe.com>
Change-Id: I502028f645f39f391c19b2028fc6ba5b7bdb1a96
Reviewed-on: https://review.whamcloud.com/37639
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Vladimir Saveliev <c17830@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13267 tests: improve racer cleanup

Add RACER_MAX_CLEANUP_WAIT parameter to specify
timeout for racer cleanup to avoid long waiting in case which
racer processes went non-killable.

Loop in racer_cleanup() contains inaccuracy which made the loop
to sleep less that it was supposed to. Fix it.

Test-Parameters: trivial testlist=racer
Cray-bug-id: LUS-8498
Signed-off-by: Elena Gryaznova <elena.gryaznova@hpe.com>
Reviewed-by: Vladimir Saveliev <vladimir.saveliev@hpe.com>
Reviewed-by: Andriy Skulysh <andriy.skulysh@hpe.com>
Change-Id: I32dac8bc11ef2041a1a580054c2782780bb5980e
Reviewed-on: https://review.whamcloud.com/37638
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Vladimir Saveliev <c17830@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13227 LDLM: update LVB if it is server lock

ldlm_glimpse_ast() is registered for server lock which means
when client send a glimpse request, it just return a special error
for this lock, it is possible that local object has size expanding
with this PW lock, so we should try update LVB upon error.

Originally, ldlm_cb_interpret() has codes to handle this error,
but it only try to handle case with some clients race, it doesn't
cover server lock cases especially after we turn on lockless for DIO.

Fixes: 6bce536725 ("LU-4198 clio: turn on lockless for some kind of IO")
Change-Id: Ic84fd19d9eaf7f8245b8f7a2165ee5913849ac01
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/37611
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13160 tests: fix sanity-hsm monitor setup

On RHEL8, even we are using pdsh -R ssh,
the ssh still waits for the remote cat process
to finish.
Use the subshell to avoid the time out.

Change-Id: Id5b8d492b5ce9a235da73448ade475ade145bbed
Test-Parameters: trivial clientdistro=el8.1 testlist=sanity-hsm
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-on: https://review.whamcloud.com/37595
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13211 ldiskfs: rework data-in-dirent for linux 5.4.7+

Linux commit v5.4-rc3-92-g109ba779d6cc
ext4: check for directory entries too close to block end

This impacts the ext4-data-in-dirent.patch due to the usage
of EXT4_DIR_REC_LEN

Invert the original patch from:
  EXT4_DIR_REC_LEN(<int>)        => __EXT4_DIR_REC_LEN(<int>)
  EXT4_DIR_REC_LEN(de->name_len) => EXT4_DIR_REC_LEN(de)
to:
  EXT4_DIR_REC_LEN(<int>)        => EXT4_DIR_REC_LEN(<int>)
  EXT4_DIR_REC_LEN(de->name_len) => EXT4_DIR_ENTRY_LEN(de)

Doing this allows the patch to apply cleanly over a wider
range of kernel releases.

Test-Parameters: trivial
Cray-bug-id: LUS-8478
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I95349743f323bb150854ff4541bd2c88f01662a6
Reviewed-on: https://review.whamcloud.com/37467
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12945 lnet: Disable zero copy when running on VM

When running on a hypervisor platform zero copy buffers
may still be referenced when write queue size is zero

So when running on a hypervisor push the zero copy size limit
above max payload size of 16M.

Use the hypervisor test added to linux v4.14-119-g79cc74155218
and provide a replacement for earlier kernels.

kernel-commit: 79cc74155218316b9a5d28577c7077b2adba8e58

Cray-bug-id: LUS-8072
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I5582a6aa8da6f48deafaf13d60cf67a09d7a7231
Reviewed-on: https://review.whamcloud.com/37300
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13110 kernel: kernel update SLES12 SP4 [4.12.14-95.45.1]

Update SLES12 SP4 kernel to 4.12.14-95.45.1 for Lustre client.

Test-Parameters: trivial clientdistro=sles12sp4 \
envdefinitions=LNET_SELFTEST_EXCEPT=smoke,SANITY_EXCEPT="103a 817"

Change-Id: I1f7024465b4b6334488b7314f1073fafa10331d6
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/37123
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13082 tests: enable lgss_keyring debug traces

Enable lgss_keyring debug traces in test framework, and collect
systemd journal in case of test failure.
Also, if needed, dump SSK keys, keyring and nodemap definitions.

Test-Parameters: trivial
Test-Parameters: envdefinitions=SHARED_KEY=true testlist=sanity
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I201d97b640045e3cbc8dcd4cd4b25e0e4e644037
Reviewed-on: https://review.whamcloud.com/37042
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13097 tests: set fail_loc on all MDS nodes for pdir tests

Set fail_loc on all MDS nodes for pdir tests, not only $SINGLEMDS.

Test-Parameters: trivial testlist=sanityn,sanityn,sanityn,sanityn
Test-Parameters: testlist=sanityn,sanityn,sanityn,sanityn,sanityn
Test-Parameters: testlist=sanityn,sanityn,sanityn,sanityn,sanityn
Test-Parameters: testlist=sanityn,sanityn,sanityn,sanityn,sanityn
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I80d935eae0b06e39712abfe48a56b8ce08537926
Reviewed-on: https://review.whamcloud.com/37145
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>

LU-13004 ptlrpc: remove *GET*KVEC macros and fields.

GET_KVEC, BD_GET_KVEC, GET_ENC_KVEC, BD_GET_ENC_KVEC
are no longer used.
There are the only users of the bd_kvec field of bd_u,
so that field can be removed, and bd_u can be discarded
and its other field (bd_kiov) promoted.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I31d19f6867c907029aa8d9ceee27c5ac9c9225a1
Reviewed-on: https://review.whamcloud.com/36972
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13293 llite: don't abort readahead too aggressively

We will stop readahead if we have covered @ria_end_idx_min
but lock contention case happen, this could cause a problem
with small read with SSF mode.

Because even lock contention happen, but every client could
potentially have a large consecutive pages to read, if this
is a 4K read, it will end up with a lot small read.

To fix this problem, at least allow readahead continue with
one stripe size, this is exact behavior before the commit

Without Patch:
Max Write: 13082.37 MiB/sec (13717.85 MB/sec)
Max Read:  854.17 MiB/sec (895.67 MB/sec)

With Patch:
Max Write: 12448.90 MiB/sec (13053.61 MB/sec)
Max Read: 23921.73 MiB/sec (25083.75 MB/sec)

Fixes: cfbeae9 ("LU-12043 llite: extend readahead locks for striped file")
Change-Id: I59963592f6dbe6babd746cd01441f4a99a8cafcb
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/37697
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13288 llite: Find account_page_dirtied on module init

Kernel v5.2-5678-gac1c3e4 no longer exports account_page_dirtied
Use kallsyms_lookup_name to find and use it as
vvp_account_page_dirtied on module init to avoid any
performance regressions due to symbol_get.

Test-Parameters: clientdistro=el8.1
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: Ie27abb07ffbf9e5be67fe64601ebc62409f829fd
Reviewed-on: https://review.whamcloud.com/37686
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Shuichi Ihara <sihara@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13291 ldiskfs: mballoc don't skip uninit-on-disk groups

as those need no IO to initialize buddy structures and the best
candidates for new blocks.

Fixes: 6a7a700a1490 ("LU-12988 ldiskfs: skip non-loaded groups at cr=0/1")
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Ic3c5a238d8825024d7a0fec6a25e842b7ba1f100
Reviewed-on: https://review.whamcloud.com/37687
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13004 lustre: remove support for KVEC bulk descriptors

KVEC descriptors are no long used nor needed.
KIOV are sufficient for all needs.

This allows us to remove
  PTLRPC_BULK_BUF_KVEC
and
  PTLRPC_BULK_BUF_KIOV
flags - the distinction no longer exists.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ic3a6ec942b60a05c7ce6c5b05659700e1399d0b9
Reviewed-on: https://review.whamcloud.com/36971
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13026 lnet: MR selection of gateway ni

Multi-Rail gateways can only have 1 route pointing to them. Use
the MR selection algorithm to get the best lpni on the MR
gateway to use.

Using the selection algorithm to find the gateway NI, allows us
to use the standard MR criteria: health, preference and credits.

Test-parameters: trivial

Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: I65526c6b64a5702734949c9a583a7558614ceae2
Reviewed-on: https://review.whamcloud.com/36913
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13025 lnet: pick healthiest peer net

When iterating over the peer nets, select the healthiest one.
Node might be able to reach a peer over multiple nets, and therefore
the health of these peer nets must be considered.

Test-parameters: trivial

Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: I155888dca358627fcb63c2ed0e51114bc49a9ff1
Reviewed-on: https://review.whamcloud.com/36912
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12678 lnet: remove lnet_me_alloc/lnet_me_free

These functions are simple wrapper that do not benefit
readability, so remove them.

Move the DEBUJG messages to the places where allocation happens.  This
introduces only a tiny amount of code duplication.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ieb40f8a7547cba30b05c1e5e526c762e354f3c47
Reviewed-on: https://review.whamcloud.com/36910
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13040 lmv: Pool name string handling

KASAN picked up a case where pool_name may not be null
terminated.

There are a few cases where strncpy is used, however strncpy does
not guarantee the target string is null terminated. The preference
is to use strlcpy.

Cray-bug-id: LUS-8229
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I414e955b5964c24b061edd76ed9e64a8985c537d
Reviewed-on: https://review.whamcloud.com/36908
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Ben Evans <beevans@whamcloud.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-12775 test: reorder 'tar' command options

'tar' in RHEL8 is stricter in command option order.

Test-Parameters: trivial \
envdefinitions=ONLY="32c" \
clientdistro=el8.1 serverdistro=el8.1 \
mdscount=2 mdtcount=4 testlist=conf-sanity

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I814203808efae4a746166abd3ba08f2bc5fce8f7
Reviewed-on: https://review.whamcloud.com/36907
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12981 lnet: Check MTU accurately

The existing check for MTU lacks checking for the KIOV/IOV cases,
and false positive triggered for very large incomming
buffer.

Cray-bug-id: LUS-7948
Signed-off-by: Alexey Lyashkov <c17817@cray.com>
Change-Id: Id3497e5f63470c24b2e51703fc564b02c9516aa6
Reviewed-on: https://review.whamcloud.com/36796
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12761: tests: make version_code() accept two number versions too

There's now a user in sanity test 103a that calls version_code with
2.6.  Andreas rightfully points instead of fixing the caller we can
just update the code to accept this usage.

Change-Id: I5915cd08a36946c6d26f2e231aa7a820a3eef46a
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/36275
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>

LU-12780 osp: use native kthreads for opd_pre_thread

rather than ptlrpc_thread, use native kthreads functionality.

- provide an opt_args structure which is allocated
  and initialized before the thread is started so errors
  cannot happen in the thread.
- include a completion to synchronize startup so we can be sure
  the thread function actually runs, and so will clean up.
- use kthread_stop and kthread_should_stop to
  synchronize shutdown.

The ptlrpc_thread was not used for signaling the thread about
work-to-do, so no change is needed there.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ib0e2041da3fa4d613b17f743b18700c84a79fac2
Reviewed-on: https://review.whamcloud.com/36263
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9859 libcfs: simplify linux-prim.c

cfs_block_sigs() is never used.
cfs_clear_sigpending() is only used in lustre_lib.h so move it
to that header. Based on

Linux-commit: 99c1ffc99a570c68cef906d9763edb47b316ea1a

Change-Id: Ia0d5ecb736c4107c5a7b666bda85714d6819fbca
Signed-off-by: NeilBrown <neilb@suse.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-on: https://review.whamcloud.com/35410
Reviewed-by: Neil Brown <neilb@suse.de>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10467 ptlrpc: convert waiting in ptlrpc_hr_main()

This is a basic conditional wait. Instead of using
l_wait_condition() use the linux standard wait_event_idle().

Change-Id: I5c81914de003468ac20b6c65f1b6bee2d4cf6891
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/37696
Reviewed-by: Neil Brown <neilb@suse.de>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9679 osc: convert a while loop to for

This loop uses 'continue' in several places,
and each one is proceeded by
   ext = next_extent(ext)
which also appears at the end.
This is exactly the pattern that a 'for' loop
simplifies.  So change to a for loop.

Linux-Commit 9083b739197b ("lustre: osc: convert a while loop to for")

Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: Ie622690134f9b3ee829255bcf997d06289abd6e6
Reviewed-on: https://review.whamcloud.com/37606
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9679 osc: centralize handling of PTLRPCD_SET

Various places test if a given rqset is PTLRPCD_SET
and call either ptlrpcd_add_req() or ptlrpc_set_add_req()
depending on the result.

This can be unified by putting the test of PTLRPCD_SET in
ptlrpc_set_add_req(), and always calling that function.

This results in there being only one place that tests PTLRPCD_SET.

Linux-Commit: 6a587cd4c705 ("lustre: centralize handling of
PTLRPCD_SET")

Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I879aa9ebb7e841dc2d1240a32d1c5d07e582e0b2
Reviewed-on: https://review.whamcloud.com/37603
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>

LU-9679 osc: use overlapped() consistently.

osc_extent_is_overlapped() open-codes exactly the test that
overlapped() performs.
So use overlapped() instead, to make the code more obviously
consistent.

Linux-Commit: 270995b08634 ("lustre: osc: use overlapped()
consistently.")

Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I3a3ed2ee04343a294ae94f205f5d12be98f99bf3
Reviewed-on: https://review.whamcloud.com/37602
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>

LU-9679 osc: remove cl_io_cancel()

cl_io_cancel() is never used, so remove it and various
other things that it is the only user of.

Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I6cf9b53aa7fc3379e57fa0ac0ea236ccda4ff6b7
Reviewed-on: https://review.whamcloud.com/37597
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9679 osc: use assert_spin_locked()

assert_spin_locked() is preferred to spin_is_locked() for affirming
that a spinlock is locked.

__osc_extent_sanity_check() is only ever called with obj already
locked, so change the check into an assertion.

Linux-Commit: a12d8284b574 ("lustre: osc_cache: use
assert_spin_locked()")

Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: Iaae6deb5af4dec4d31893749924f211ba0489c47
Reviewed-on: https://review.whamcloud.com/37596
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9679 general: add missing spaces near punctuation

Many places in lustre fold a long string onto multiple lines, usually
at word breaks.  Sometimes the word-break has punctuation, such as
comma, colon, or period, but needs a space as well to be properly
readable.  Because the string is folded, the missing space isn't
immediately obvious in the code.

This patch adds those spaces after punctuation where is seems to be
needed, and joins the affected strings onto a single line, in accord
with current policy.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I9e76778b1e9687bc26e85500006b4b9d9ae6f93a
Reviewed-on: https://review.whamcloud.com/37402
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>

LU-12914 mdt: mdt_prep_ma_buf_from_rep() is called twice

In some rare cases (replay of file open with O_LOV_DELAY_CREATE
when object found dead on mdt) mdt_prep_ma_buf_from_rep() can
be called twice (in either mdt_reint_open() and mdt_open_by_fid())
during the same request handling.
So remove assert checking if LMV or LOV has already been found and
set in ma.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: I78e0456ea59c37cab4276383c75c4fa5cc9f4829
Reviewed-on: https://review.whamcloud.com/36611
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <farr0186@gmail.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-11668 mdd: use mdd_object_fid() instead of mdo2fid()

Both mdd_object_fid() and mdo2fid() helper functions are the same.
Replace mdo2fid() with the better-named mdd_object_fid(mdd_obj)
function everywhere.  Use mdd_obj_dev_name(mdd_obj) for console
error messages instead of mdd2obd_dev(mdd)->obd_name for consistency.

It would be nice to consistently use "mdd_obj" for objects (instead of
"o" or "mo" or "obj", ...) and "mdd" for devices (instead of "m"), but
that is too big to include in this patch.  Just replace them in the
few wrapper functions already affected by this patch.

Fix up whitespace and string formatting style in affected code.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I6de748bada06f0f66123e4567115deb2633ebbe5
Reviewed-on: https://review.whamcloud.com/35047
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12931 gnilnd: use time_after() to compare jiffies

Fix a potential bug in gnilnd it is directly comparing a timeout
against jiffies instead of using time_after() to handle jiffies wrap.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ie4d190e9c04e807f2152b71dc28ef0b0463ebbe5
Reviewed-on: https://review.whamcloud.com/36702
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13264 osc: ensure lu_ref work atomic from osc_lock_upcall()

Since osc_lock_upcall() uses per-cpu env via
cl_env_percpu_[get,put](), all undelying work must execute on the
same CPU, meaning that no sleep()/scheduling must occur.
This implies all lu_ref related work to no longer use lu_ref_add(),
which calls might_sleep() (likely to cause a
scheduling/cpu-switch...), but lu_ref_add_atomoc() instead.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: Ide33d4c415e9e382f0bc344e2114182a1f122de6
Reviewed-on: https://review.whamcloud.com/37629
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexandr Boyko <c17825@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13263 osc: use LDLM_LOCK_RELEASE() if no lu_ref added

In osc_ldlm_glimpse_ast(), LDLM_LOCK_PUT() is used after
LDLM_LOCK_GET() when no lu_ref has been added.
This causes a LBUG when USE_LU_REF is configured, so
change LDLM_LOCK_PUT() to LDLM_LOCK_RELEASE().

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: Id522a02878f01ae565e6c2418fe8cd85c945dde9
Reviewed-on: https://review.whamcloud.com/37625
Reviewed-by: Patrick Farrell <farr0186@gmail.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13258 libcfs: make apply_workqueue_attrs() available for Lustre

Currently Lustre work queues can run on any core which introduces
noise on the system. The linux kernel has a function called
apply_workqueue_attrs() which allows you to control which cores
a work queue will execute on. Manually export this function so
Lustre can use it.

Test-Parameters: trivial

Change-Id: I467539cb8def7029fa9dfff2386234de5e0fe133
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/37613
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13004 target: take offset into account in tgt_send_buffer

While calculating amount of pages needed, take buffer offset into
account because it can be non-aligned after allocations in
out_read().

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: Ib7c9b35d328d366a27cc553ffe2f2c5930949cf4
Reviewed-on: https://review.whamcloud.com/37571
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12251 tests: stop running sanity-flr for PPC

Stop running all sanity-flr tests for PPC client
testing until we understand and the sanity-pfl test
suite to passes all testing for PPC clients.

Test-Parameters: trivial clientarch=ppc64 testlist=sanity-flr
Test-Parameters: testlist=sanity-flr

Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: Iee044e6995ed4f6cca5f6b7f92eee6b59cb7018b
Reviewed-on: https://review.whamcloud.com/37563
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-12198 libcfs: always copy ioctl header back to user

lnetctl_get_peer_list fills back the required size in header if the
given buffer was too small. Userspace needs the info back to grow
the buffer and try again.

Note we only replace err on failure if err was previously not set

Fixes: fba98579efc4 ("LU-6202 libcfs: replace libcfs_register_ioctl with a blocking notifier_chain")
Signed-off-by: Dominique Martinet <dominique.martinet@cea.fr>
Change-Id: I2b6e319aceeb00d488572053d27023891afe1928
Reviewed-on: https://review.whamcloud.com/37559
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Neil Brown <neilb@suse.de>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13225 utils: bash completion for lfs and lctl

Add a bash completion for "lfs" and improve completion for "lctl".
Rename the "lctl" completion script to "lustre" since the two
commands share helper routines for fsnames, pools, etc. and install
"lfs" and "lctl" symlinks to the common command file.

The completion prints available sub-commands and their options,
and for some sub-commands it completes available arguments
(e.g. mount points, pool names, and MDT/OST names).

A couple of minor changes to "lfs" and "lctl" usage messages to make
the sub-command options easier to parse.  More needs to be done to
make all sub-commands have proper long options.

There is definitely a lot more that could be added to the completions,
but this is a good start and provides a framework for the future.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ie989b2ef4c0d6d8565e5c6753205bb6ed83ebbe5
Reviewed-on: https://review.whamcloud.com/37483
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Dominique Martinet <dominique.martinet@cea.fr>
Reviewed-by: Quentin Bouget <quentin.bouget@cea.fr>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13133 tests: sanity-selinux test_21{a,b} sepol update

We need to make sure MDS receives updated sepol info from MGS.
In case of combined MGT/MDT, directly setting fileset on the node
will mask llog-based info retrieval mechanism. So always use
'lctl set_param -P' to set sepol value.

Test-Parameters: trivial
Test-Parameters: clientselinux testlist=sanity-selinux
Test-Parameters: clientselinux testlist=sanity-selinux
Test-Parameters: clientselinux testlist=sanity-selinux
Test-Parameters: clientselinux testlist=sanity-selinux
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Iaf8ff13364b9ba5f5d8b733be0247d79e05a6b3d
Reviewed-on: https://review.whamcloud.com/37224
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13071 lnet: reduce log severity for health events

No need to print an error when the health of an interface is
reduced. Changed it to debug level.

Test-Parameters: trivial
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: Ia60ade12efab732ea4b0388a3803976bf65938ab
Reviewed-on: https://review.whamcloud.com/37002
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13004 osp: use correct page count in osp_prep_update_req

A fix that went into patchset 3 of
 https://review.whamcloud.com/#/c/36828/3
disappeared in patchset 5.
We should restore it.

Specifically, 'page_count' should be a count of pages,
but it is currently a count of the bytes in all the pages.

Fixes: f32fbf189fab ("LU-13004 osp: use KIOV in osp_prep_update_req")
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ic8dcdac414d16b4f2c1c6e0367d496de7e0a8cff
Reviewed-on: https://review.whamcloud.com/37587
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12861 libcfs: Cleanup libcfs_debug_msg use of snprintf

scnprintf returns the number of bytes written to the buffer.
snprintf returns the size of the buffer needed to satisfy
the request.

Prefer scnprintf when result is being used as the number
of bytes in a buffer.

Use snprintf when the result is used for sizing or resizing
a buffer.

Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I8c4b8f7dcc081f8b9dfffc35059011172be2e091
Reviewed-on: https://review.whamcloud.com/36901
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Ben Evans <beevans@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12191 utils: Make "lctl list_param" read exact path under sysfs tree

"lctl list_param -R" currently checks for the param_name
in the path and reads the sysfs tree under that. But it can
give erroneous results in the following example -

For path like /sys/fs/lnet/net/o2ib1/ib0, command
"lctl list_param -R" doesn't go down the "net" tree
because it matches "net" with "lnet" and just stop
there.

This patch changes how param_name is checked for
in the path. Like in the above example, instead
of checking for "net", it should check for
"/net". So, this patch adds this change in
param_display() in lustre/utils/lustre_cfg.c

Change-Id: Ieb3ad0d1248eee2192246ff5c4d77a71d87dc446
Signed-off-by: Sonia Sharma <sharmaso@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/36852
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13005 lnet: eq: discard struct lnet_handle_eq

The Portals API uses a cookie 'handle' to identify an EQ.  This is
appropriate for a user-space API for objects maintained by the kernel,
but it brings no value when the API client and implementation are both
in the kernel, as is the case with Lustre and LNet.

Instead of using a 'handle', a pointer to the 'struct lnet_eq` can be
used.  This object is not reference counted and is always freed
correctly, so there can be no case where the cookie becomes invalid
while it is still held.

So use 'struct lnet_eq *' directly instead of having indirection
through a 'struct lnet_handle_eq'.
Also:
 - have LNetEQAttach() return the pointer, using ERR_PTR() to return
   errors.
 - discard ln_eq_containers and don't store the me there-in.
   This means we don't free any eq that have not already been freed,
   but all eq that are allocated are properly freed, so that is not
   a problem.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I0d6e5b654e39e749b39d46f68d0fb3e47a3256e9
Reviewed-on: https://review.whamcloud.com/36841
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12780 llite: avoid ptlrpc_thread for ll_statahead_thread

Instead of ptlrpc_thread use more direct interfaces.

- Instead of waiting for thread startup to complete, perform
  the startup before starting the thread.
- as nothing waits for the thread to finish we cannot use
  kthread_should_stop().  Instead, set the task pointer
  sai_task to NULL when the thread is finishing up.
- As we don't use kthread_should_stop(), we can safely do cleanup
  in the thread, because it is sure to run.
- use wake_up_process to signal the thread that there
  is work to do.
- the wake_up that is currently at the end of sa_put() becomes
  a little more complicated and is move to after the one place
  where sa_put() is called.

Change-Id: If694dafc6864348fe5203a4935f4c128ce5ff255
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Reviewed-on: https://review.whamcloud.com/36259
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12780 llite: don't use ptlrpc_thread for sai_agl_thread

Instead of ptlrpc_thread use native kthread functionality.

- instead of waiting for the thread to start-up, perform
  all early initialization before starting the thread,
  and cleanup happens after thread is stopped.
- use kthread_stop()/ kthread_should_stop() to negotiate
  shutdown.
- wake_up_process to tell the thread if there is more work
  to do.  The thread sets current->state to TASK_IDLE before
  checking, so that if it gets the wakeup, the 'schedule()'
  call won't block.
  We clear ->sai_agl_task under a spinlock, from which it is
  also woken, to avoid races.

Linux-commit c044fb0f835c

Change-Id: I73294dd2f28087f56c3463ecfad1a8b32a44b2d7
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Reviewed-on: https://review.whamcloud.com/36258
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10467 lfsck: use wait_event_idle()

This l_wait_event() call is equivalent to the more standard
wait_event_idle().
So switch over to wait_event_idle().

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I8e13360a40dd1eec740f597d649c0f230533eb3d
Reviewed-on: https://review.whamcloud.com/37610
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10467 ldlm: use wait_event_idle() instead of l_wait_event

This l_wait_event() is equivalent to wait_event_idle() which is now
supported in lustre.  So switch over to it.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: If1ee81a0d562516534665d049fb24c1f39b59b95
Reviewed-on: https://review.whamcloud.com/37609
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13254 mdt: clear mti_mdt in mdt_thread_info_fini()

Clear mti_mdt when finalizing mdt_thread_info to prevent
its reuse my other handler later. Usually that may happen
at mdt_lvbo_fill/update() which takes thread info as is,
without initialization because at this point it is not
clear was it already initialized or not. So mti_mdt may be
used there being initialized by some other handler from
different MDT or even with garbage at old pointer.
Meanwhile there is no need to use any mdt_thread_info values
like mti_mdt in mdt_lvbo_fill() because there is MDT device
taken from namespace and the only fields are used from
mdt_thread_info are temporary storage for FID and lu_buffer.

Patch zeros mti_mdt upon thread finalizing and removes also
usages of info->mti_mdt from mdt_lvbo_fill/update() replacing
that with MDT taken from lock namespace.

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: Ib350093f0b70c777932c056b34cb56a9702b650d
Reviewed-on: https://review.whamcloud.com/37592
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10467 mdc: change ssleep to msleep_interruptible

During review of the mdc wait_idle* changes for mdc_getpage()
it was pointed out that the use of ssleep() prevents the code
from being interruptible. Change ssleep to msleep_interruptible()
to allow breaking out of the sleep if an application sends
and INTR signal.

Change-Id: I2fcb90ecdd6f2c4f2ee6fbc54d253622e8beee29
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/37488
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12811 ptlrpc: pass buffer size to the swabbing functions

By adding a separate rmf_swab_len() function pointer to
req_msg_field, the buffer size can be passed to the swabbing
functions, e.g. lustre_swab_fiemap() in this patch, to avoid
invalid access, especially for small buffer.

Signed-off-by: Emoly Liu <emoly@whamcloud.com>
Change-Id: I997e6a828f2f1cdfdb8a5fa241fa43ca0ae3677e
Reviewed-on: https://review.whamcloud.com/36435
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>

LU-11915 tests: add debugging to conf-sanity test_115

After updating the e2fsprogs build version to 1.45.2.wc2, this
test is not longer being skipped, and is failing to mount the
newly-formatted OST0000 due to errors registering with the MGS
(target index already in use).  Since the MDS+MGS was just
reformatted, that doesn't make sense.

Continue to skip this test until we understand why it is failing,
but use ALWAYS_EXCEPT instead of blaming the e2fsprogs version.

Test-Parameters: trivial testlist=conf-sanity
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I59c689763481c4fc3677ca1807101de09599bb77
Reviewed-on: https://review.whamcloud.com/37548
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10073 tests: skip test smoke for PPC

The lnet-selftest test smoke fails consistently for
PPC client testing.  Thus, stop running this test until
we understand the failure; add smoke to the ALWAYS_EXCEPT
list.

Test-Parameters: trivial
Test-Parameters: clientarch=ppc64 testlist=lnet-selftest
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I090ec05d7ad934bb4c68e976572adb29eb29a676
Reviewed-on: https://review.whamcloud.com/37450
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13186 tests: stop running tests for PPC clients

Stop running tests, put on the ALWAYS_EXCEPT list, that fail
consistently when testing PPC clients including:

sanity-hsm tests
(LU-12251) 1a 1b 1d 1e 12c 12f 12g 12h 12m 12n 12o 12p 12q
21 22 23 24a 24b 24d 24e 24f 25b 30c 37 57 58 90 110b 111b
113 222b 222d 228 260a 260b 260c
(LU-12252) 220A 220a 221 222a 222c 223a 223b 224A 224a 226
227 600 601 602 603 604 605

sanity-pfl tests
(LU-13186) 14
(LU-13205) 16a
(LU-13207) 16b
(LU-13215) 17

Test-Parameters: trivial
Test-Parameters: clientarch=ppc64 testlist=sanity-pfl,sanity-hsm
Test-Parameters: clientarch=ppc64 testlist=sanity-pfl,sanity-hsm
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I847a8121d2675b9671bc9a39c4f6ccff67b208fa
Reviewed-on: https://review.whamcloud.com/37397
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13226 ldiskfs: Add support for Ubuntu eoan 5.3

Ubuntu eoan kernel is close enough to 5.4.7+ mainline to
use the patch series directly.
Update the configure script to select it.

Test-Parameters: trivial
Cray-bug-id: LUS-8485
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: Iadb9b87a153a88846399d91699c972c72a5e1e7a
Reviewed-on: https://review.whamcloud.com/37554
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13232 tests: add stack_trap to clean up sanity 160j

When sanity test 160j fails at any point in the test before
clean up, a client can be left with no file system mounted
or the second file system mount could be left mounted.  We
need to call stack_trap after each of these commands to
clean up the mount points in case of the test failing.

Test-Parameters: trivial
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I631cc2bb2d664a0cdcfe5942d16cd1d011a822ef
Reviewed-on: https://review.whamcloud.com/37550
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9859 libcfs: rename cfs_cpt_table to cfs_cpt_tab

The variable "cfs_cpt_table" has the same name as
the structure "struct cfs_cpt_table".
This makes it hard to use #define to make one disappear
on a uni-processor build, but keep the other.
So rename the variable to cfs_cpt_tab.

Linux-commit: 457d63ea5c1aa81fe0b9a66a77a2282856b88983

Test-Parameters: trivial

Change-Id: I77cc6694183df2485974c8a962a5766a905fb5f9
Signed-off-by: NeilBrown <neilb@suse.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-on: https://review.whamcloud.com/37519
Reviewed-by: Neil Brown <neilb@suse.de>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10756 ptlrpc: fix IMP_CLOSED state is being never set

Commit cf78502e48d checks the new state for IMP_CLOSED value
instead of import current state so instead of keeping import
closed it prevents import state from being set to IMP_CLOSE

Patch restores original check to keep import closed by
checking its current state

Fixes: cf78502e48d ("LU-10756 ptlrpc: change IMPORT_SET_* macros into real functions")
Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I7df2798f09ce7023381c03957adf530da4149c2d
Reviewed-on: https://review.whamcloud.com/37405
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13190 mds: send mbo_max_mdsize in open intent reply

 - client sends open|create intent before a connection to OST
   cl_default_mds_easize is 0 since initialization
 - MDS replies back without UPDATE bit in LDLM lock, but wit EAh
    (MDS doesn't send OBD_MD_FLMODEASIZE and mbo_max_mdsize back
 - client's cl_default_mds_easize is still 0
 - client sends getattr intent with 0-size buffer for EA
 - MDS replies LAYOUT lock, but empty EA due to 0-size buffer
 - client sets local layout to EMPTY
 - all subsequent I/O fails with -EBADF

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Iadd5595d956f0469e3916cdc1cca2ac8f802a149
Reviewed-on: https://review.whamcloud.com/37400
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12722 target: disable recovery for local clients

when client is running on a server node, then the local
services can't rely on that client in the contex of
recovery - such a client dies with the node, can't replay
requests and states and then the restarting server has to
wait till recovery expires which doesn't make any sense.

so the servers should recogize local clients and exclude
them from recovery (i.e. don't make them part of last_rcvd).

for the purpose of local testing a special mount option
"local_recov" has been added to {MDS|OST}_MOUNT_OPTS in
tests/cfg/local.sh to save local testing when everyting
is running within a single node.

Signed-off-by: Alexey Zhuravlev <bzzz@whamcloud.com>
Change-Id: I4cb906c44c1192933f7d77dc782160e426e9efde
Reviewed-on: https://review.whamcloud.com/36025
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12280 quota: add notify grace

Add an option to get notify when the quota is over soft limit but
prevents it from becoming the hard limit.

Change-Id: I01ae1266c3683198b82af7bad119db280c1e3a07
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/36017
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9859 libcfs: remove unnecessary cfs_block_allsigs() calls

Threads started by kthread_run() ignore all signals,
as kthreadd() calls ignore_signals(), and this is
inherited by all children.
So there is no need to call cfs_block_allsigs() in functions
that are only run from kthread_run().

For the case of lnet_ping_md_unlink() it is not from a kernel
thread but nothing in that function should be affected by
signals so it is safe to remove.

For lnet_ping() we need to manually block signals since
LNetEQPool() can unconditionally abort when a signal is
recieved.

Linux-commit: 1b2dad1459e480028a2714439048d8a634132857

Signed-off-by: NeilBrown <neilb@suse.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Change-Id: I124dccf78a3187d5f4a31c7b76db5369aaafc369
Reviewed-on: https://review.whamcloud.com/35350
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12477 lustre: remove obsolete config checks

Remove from the lustre kernel code all the support for kernels
earlier than the RHEL7 3.10+. This greatly simplifies the code
and makes build times much better.

Change-Id: If52091ac5249b2719b992032040ccf30cc5bf0e4
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/37085
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10235 mdt: mdt_create: check EEXIST without lock

mkdir() currently gets a write lock on the parent even if the new
directory already exists.

This patch adds an initial lookup of the new directory without a DLM
lock so that other clients do not need to cancel their DLM lock if the
"new" directory already exists, but will continue as usual if directory
did not exist.

There is a small race window that child was created by others after our
check and before locking parent, but this can be detected later during
index insert.

Performance change on two haswell 16-core VMs with ib, mean values of
mpirun -n 8 ./mdtest -D -i 8 -I 1000

test environment | directory creation | tree creation
local, no patch  | 1725/s             | 769/s
local, patch     | 1821/s             | 788/s
remote, no patch | 1729/s             | 772/s
remote, patch    | 1687/s             | 787/s

The differences are of the order of the noise here, with all mkdirs
being effective.

If directories exist, some simple stress on four nodes shows intended
improvements:
clush -w vm[0-3] 'seq 0 10000 |
    xargs -P 7 -I{} sh -c "(({}%3==0)) &&
        mkdir /mnt/lustre/testdir/foo 2>/dev/null ||
        stat /mnt/lustre/testdir > /dev/null"'

with patch: 10s
without patch: 19s
(the difference grows exponentially with number of clients and hangs
with over 60 clients without the patch; exact time was not re-measured
with patch)

Updated sanityn.sh 43a 45a to avoid race conditions.

Add sanityn.sh test_43j to verify above scenario.

Test-Parameters: envdefinitions=SLOW=yes testlist=replay-vbr,replay-vbr
Change-Id: I37fc9c8ffc7ab334c0645042beda5bef01284564
Signed-off-by: Dominique Martinet <dominique.martinet@cea.fr>
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/30880
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Quentin Bouget <quentin.bouget@cea.fr>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-11597 tests: skip sanityn tests for PPC

Several sanityn test suite tests fail consistenly when
testing PPC clients.  These tests should be skipped,
added to the ALWAYS_EXCEPT list, until the failures are
understood and fixed.

Tests to skip in sanityn are
16a (LU-11597)
71a (LU-11787)

Test-Parameters: trivial clientarch=ppc64 testlist=sanityn
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I39cc9d22e8a47eb8ef59ce8d30e1b6e9aa616a9a
Reviewed-on: https://review.whamcloud.com/37561
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-11269 ptlrpc: do not expose transient IDLE state

to avoid cases when anyone sending an RPC observes the connection
in this state while it's going to reconnect right away.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I9ca89051c4176fe321262f8b2f52969c382e401e
Reviewed-on: https://review.whamcloud.com/37523
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13228 clio: mmap write when overquota

Flagging client by overquota flag should not
cause mmap write access to sigbus the app.

Cray-bug-id: LUS-8221
Signed-off-by: Alexander Zarochentsev <c17826@cray.com>
Change-Id: I29d5901fa5078b5cfca40391a02531cf27efce93
Reviewed-on: https://review.whamcloud.com/37495
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andrew Perepechko <c17827@cray.com>
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13131 osc: remove redundant osc_list() helper

The osc_list() helper function is the same as list_empty_marker(),
and we don't need both.  Remove osc_list() from the code.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I07d2a519906f52fca8c95613a14ad7389a3ebbe5
Reviewed-on: https://review.whamcloud.com/37479
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9679 various: use list_splice and list_splice_init

The construct
   list_add(to, from);
   list_del(from);
is equivalent to
   list_splice(from, to);
providing 'to' has been initialized.
Similarly with list_del_init and list_splice_init.
There is no need to check if list_empty(from) first.

Also looping over a list moving individiual entries to
another list can more easily be done with list_splice.

These changes improve code clarity.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I710eb3bbd83c75e6c8f00b8d0a4c256ad28f9082
Reviewed-on: https://review.whamcloud.com/37457
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10391 lnet: discard lnet_sock_accept()

There is no longer any important difference between
lnet_sock_accept(), and kernel_accept(..., O_NONBLOCK).
So remove lnet_sock_accept().

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Iad7c91abe2359758982e3740a21c91232c919aa0
Reviewed-on: https://review.whamcloud.com/37303
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10391 lnet: use data_ready callback to trigger accept()

Rather than blocking in lnet_sock_accept(), set up a data_ready
callback, and use that to find out when to call lnet_sock_accept()
again.

This simplifies lnet_sock_accept() (which will be removed in
next patch), and means that we could listen on multiple
sockets, which will be useful for IPv6 support.

The code design in based on that in net/sunrpc/svcsock.c.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I3015f2f6b6d420af5c8454b6c1a99611b48e7702
Reviewed-on: https://review.whamcloud.com/37302
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9679 llite: Discard LUSTRE_FPRIVATE()

The LUSTRE_FPRIVATE() macro adds no value.
Instead of
  LUSTRE_FPRIVATE(file)
use
  file->private_data

which is shorter and more familiar, and widely used
elsewhere in lustre.

Also re-indent several functions where this was changed, to
use TABs.
Also join together some strings that were split across 2
lines.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I811aea8069b22beed15fd96d8c6bef8eca42defd
Reviewed-on: https://review.whamcloud.com/36652
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10467 ptlrpc: convert final users of LWI_TIMEOUT_INTERVAL

LWI_TIMEOUT_INTERVAL causes l_wait_event to perform a slow
poll loop.  This is only needed if the event can happen without
triggering a wakeup on the wait-queue.

On this case, the event is a counter reaching zero, and we can
easily ensure a wakeup is sent whenever that counter becomes
zero.
So let's add those wake_ups, and change this to a simple
wait_event_idle_timeout().

At the same time, change all wake_up_all() calls on this wait queue to
simple wake_up().  wake_up_all() is only needed where there are
exclusive waiters, and this queue has no exclusive waiters.

Change-Id: I2bea069150f21b725025bacc7a4fa0cf4d95ab20
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Reviewed-on: https://review.whamcloud.com/35979
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10467 lustre: use l_wait_event_abortable where appropriate.

If the lwi passed to l_wait_event() was created with

    lwi = LWI_INTR(LWI_ON_SIGNAL_NOOP, NULL);

the effect is to wait with no timeout and blocking any
non-fatal signals.
For this, we now have l_wait_event_abortable(), or for one
case l_wait_event_abortable_exclusive();
So use those.

l_wait_event_abortable() will return -ERESTARTSYS if a signal was
received, while l_wait_event() returens -EINTR.  We need to be
careful to handle this difference.

Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: Iadf0fab92fcfd46802766198dcbe6b6b349214fa
Reviewed-on: https://review.whamcloud.com/35975
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12988 ldiskfs: revert prefetch patch

as a problem leading to IO errors was found.
also, the patch for 4.18 kernel needs fixes.

Revert "LU-12988 ldiskfs: mballoc to prefetch groups"

This reverts commit 05f31782be20fc4c46082dba02c10bcea59539e3.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I478a011e561633516524697f3a4aa03734791790
Reviewed-on: https://review.whamcloud.com/37619
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13166 osd-ldiskfs: fix to allow to get system inode

Lustre need load ldiskfs system inode for quota accounting purpose,
so pass LDISKFS_IGET_SPECIAL flag to ldiskfs_iget(), otherwise,
support of centos8 quota will be broken.

Fixes: 8ab3aa50a14 ("LU-12355 ldiskfs: Added ext4_iget_flags to ext4_iget")
Change-Id: I3a30ec540444b149bc3398a62951d2826eb7b9ce
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/37421
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9859 libcfs: move tracefile locking from linux-tracefile.c
 to tracefile.c

There is no value in keeping it separate.

Linux-commit: 49209c598d93289ca077575615e98f242b1d8156

Test-Parameters: trivial

Change-Id: I24ee7545a40fd6d2ac15018f089d51142736fa27
Signed-off-by: NeilBrown <neilb@suse.com>
Reviewed-on: https://review.whamcloud.com/37408
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12747 tests: wait properly for orhpan thread stop

Use wait_update_facet() to check if the MDD orphan cleanup thread has
exited, rather than a fixed 5s timeout.  We can hope that most cases
will finish faster than 5s, but don't gratuitously fail if it takes
somewhat longer.  We clearly aren't having a fatal problem here, or
there would be serious failures at cleanup time.

Fixes: fffef5c29e3b ("LU-11418 mdd: delete name if orphan doesn't exist")
Test-Parameters: trivial testlist=sanity envdefinitions=ONLY=811,ONLY_REPEAT=100
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I16b0281a519d47b5b98d495bf17040153c3ebbe5
Reviewed-on: https://review.whamcloud.com/37395
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13169 tests: add ONLY_REPEAT parameter to repeat subtests

Add the ONLY_REPEAT environment variable, to allow tests specified
by ONLY to be run multiple times, to ensure that the test is passing
consistently (or fixing an intermittent bug).  This is faster than
restarting the test session multiple times for only a few subtests.

Have the iteration around the subshell started for run_one() so that
any registered stack_trap EXIT calls are triggered between iterations,
the fail_loc is reset, grant/health/error checks are done, and so on.

Remove $tdir and $tfile files after each iteration to avoid failures
with the subsequent subtest runs.  For tests that do not follow the
standard naming convention for test directories and files, they need
to be updated to use $tdir and $tfile, which is good in any case.

YAML output splits each iteration into a separate subtest for Maloo.
The output from run_one() is appended to a single output file for all
iterations so all output is captured instead of just the last one.

The iterations will continue until $ONLY_REPEAT loops pass, or until
the subtest hits an error.  Trying to continue for all iterations in
the face of errors would likely end up with all of later iterations
failing also due to leftover state from the previous failure, and the
goal is for the subtests to pass consistently.  If we are trying to
determine rates of intermittent failures, this can be computed using
1/num_passes about the same as num_failures/ONLY_REPEAT iterations.

Rename variables in subtests to avoid clash with testnum, testname,
and TESTNAME, and use them consistently in functions and subtests.

Test-Parameters: testlist=sanity envdefinitions=ONLY=27l,ONLY_REPEAT=100
Signed-off-by: Andreas Dilger <adilger@dilger.ca>
Change-Id: I5449590dc3e25c113b059974fb7b96c892434380
Reviewed-on: https://review.whamcloud.com/37321
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Charlie Olmstead <charlie@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12637 kernel: RHEL 8.1 server support

This patch makes changes to support RHEL 8.1 release with
kernel 4.18.0-147.3.1.el8_1 for Lustre server.

Test-Parameters: trivial \
envdefinitions=SANITY_EXCEPT="411 812b" \
clientdistro=el8.1 serverdistro=el8.1 \
testlist=sanity

Change-Id: Iee6ae3dc20c62caaac1d740b14c5877ff7bfb4d5
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/36968
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13178 build: Update ZFS version to 0.8.3

New Features 0.8.0

* Native encryption #5769 - The encryption property enables the
creation of encrypted filesystems and volumes. The aes-256-ccm
algorithm is used by default. Per-dataset keys are managed with zfs
load-key and associated subcommands.

* Raw encrypted 'zfs send/receive' #5769 - The zfs send -w option
allows an encrypted dataset to be sent and received to another pool
without decryption. The received dataset is protected by the original
user key from the sending side. This allows datasets to be efficiently
backed up to an untrusted system without fear of the data being
compromised.

* Device removal #6900 - This feature allows single and mirrored
top-level devices to be removed from the storage pool with zpool
remove. All data is copied in the background to the remaining
top-level devices and the pool capacity is reduced accordingly.

* Pool checkpoints #7570 - The zpool checkpoint subcommand allows
you to preserve the entire state of a pool and optionally revert back
to that exact state. It can be thought of as a pool wide snapshot.
This is useful when performing complex administrative actions which
are otherwise irreversible (e.g. enabling a new feature flag,
destroying a dataset, etc).

* Pool TRIM #8419 - The zpool trim subcommand provides a way to
notify the underlying devices which sectors are no longer allocated.
This allows an SSD to more efficiently manage itself and helps prevent
performance from degrading. Continuous background trimming can be
enabled via the new autotrim pool property.

* Pool initialization #8230 - The zpool initialize subcommand writes
a pattern to all the unallocated space. This eliminates the first
access performance penalty, which may exist on some virtualized
storage (e.g. VMware VMDKs).

* Project accounting and quota #6290 - This features adds project
based usage accounting and quota enforcement to the existing space
accounting and quota functionality. Project quotas add an additional
dimension to traditional user/group quotas. The zfs project and zfs
projectspace subcommands have been added to manage projects, set quota
limits and report on usage.

* Channel programs #6558 - The zpool program subcommand can be used
to perform compound ZFS administrative actions via Lua scripts in a
sandboxed environment (with time and memory limits).

* Pyzfs #7230 - The new pyzfs library is intended to provide a
stable interface for the programmatic administration of ZFS. This
wrapper provides a one-to-one mapping for the libzfs_core API
functions, but the signatures and types are more natural to Python.

* Python 3 compatibility #8096 - The arcstat, arcsummary, and
dbufstat utilities have been updated to be compatible with Python 3.

* Direct IO #7823 - Adds support for Linux's direct IO interface.

Performance

* Sequential scrub and resilver #6256 - When scrubbing or
resilvering a pool the process has been split into two phases. The
first phase scans the pool metadata in order to determine where the
data blocks are stored on disk. This allows the second phase to issue
scrub I/O as sequentially as possible, greatly improving performance.

* Allocation classes #5182 - Allows a pool to include a small number
of high-performance SSD devices that are dedicated to storing specific
types of frequently accessed blocks (e.g. metadata, DDT data, or small
file blocks). A pool can opt-in to this feature by adding a special or
dedup top-level device.

* Administrative commands #7668 - Improved performance due to
targeted caching of the metadata required for administrative commands
like zfs list and zfs get.

* Parallel allocation #7682 - The allocation process has been
parallelized by creating multiple "allocators" per-metaslab group.
This results in improved allocation performance on high-end systems.

* Deferred resilvers #7732 - This feature allows new resilvers to be
postponed if an existing one is already in progress. By waiting for
the running resilver to complete redundancy is restored as quickly as
possible.

* ZFS Intent Log (ZIL) #6566 - New log blocks are created and issued
while there are still outstanding blocks being serviced by the
storage, effectively reducing the overall latency observed by the
application.

* Volumes #8615 - When a pool contains a large number of volumes
they are more promptly registered with the system and made available
for use after a zpool import.

* QAT #7295 #7282 #6767 - Support for accelerated SHA256 checksums,
AES-GCM encryption, and the new QAT Intel(R) C62x Chipset / Atom(R)
C3000 Processor Product Family SoC.

Changes in Behavior

* Relaxed (ref)reservation constraints on volumes, they may now be
set larger than the volume size.

* The arcstat.py, arc_summary.py, and dbufstat.py commands have been
renamed arcstat, arc_summary, and dbufstat respectively.

* The SPL source is now included in the ZFS repository removing the
need for separate packages.

* The dedupditto pool property and zfs send -D option have been
deprecated and will be removed in a future release.

Changes for 0.8.1

* Fix comparison signedness in arc_is_overflowing() #8873
* Fix incorrect error message for raw receive #8863
* arc_summary: prefer python3 version and install when there is no
python #8851
* Fix %post and %postun generation in kmodtool #8866
* Reinstate raw receive check when truncating #8852 #8857
* If $ZFS_BOOTFS contains guid, replace the guid portion with $pool
* Fix integer overflow of ZTOI(zp)->i_generation #8858
* hkdf_test binary should only have one icp instance #8850
* Fixed a small typo in man/man1/raidz_test.1 #8855
* Allow TRIM_UNUSED_KSYM when build as a builtin-module #8820
* Make Python detection optional and more portable #8809 #8731
* Wait in 'S' state when send/recv pipe is blocking #8733 #8752
* Make zfs_async_block_max_blocks handle zero correctly #8829 #8289
* Revert "Report holes when there are only metadata changes" #8816
* Exclude log device ashift from normal class #8735
* Fix integer overflow in get_next_chunk() #8778 #8797
* Double-free of encryption wrapping key due to invalid pool
properties #8791
* Endless loop in zpool_do_remove() on platforms with unsigned char
* Fix embedded bp accounting in count_block() #8800 #8766
* Disable parallel processing for 'zfs mount -l' #8762 #8811
* Linux 5.2 compat: Directly call wait_on_page_bit() #8794
* Linux 5.2 compat: Fix config/kernel-shrink.m4 test failure #8776
* Linux 5.2 compat: Remove config/kernel-set-fs-pwd.m4 #8777
* zpool: status -t is not documented in help message #8782
* VERIFY3P() message is missing a space character #8786
columns #8785
* zfs: don't pretty-print objsetid property #8784
* zfs: missing newline character in zfs_do_channel_program() error
message #8783
* Fix ksh-path for random_readwrite_fixed.ksh #8779
* Linux 2.6.39 compat: Test if kstrtoul() exists #8760 #8761
* Device removal panics on 32-bit systems #8790
* zpool: trim -p is not a valid option #8781
* Fix coverity defects: CID 186143 #8788
* Fix kstat state update during pool transition #8746
* Linux 5.2 compat: rw_tryupgrade() #8730

Changes for 0.8.2

* Disabled resilver_defer feature leads to looping resilvers #9299
  #9338
* Fix dsl_scan_ds_clone_swapped logic #9140 #9163
* Scrubbing root pools may deadlock on kernels without
  elevator_change() (#9321) #9321
* QAT related bug fixes #9276 #9303
* kmodtool: depmod path #8724 #9310
* Fix /etc/hostid on root pool deadlock #9256 #9285
* BuildRequires libtirpc-devel needed for RHEL 8 #9289
* Fix zpool subcommands error message with some unsupported options
  #9270
* Fix zfs-dkms .deb package warning in prerm script #9271
* zvol_wait script should ignore partially received zvols #9260
* New service that waits on zvol links to be created #8975
* Always refuse receving non-resume stream when resume state exists
  #9252
* Fix Intel QAT / ZFS compatibility on v4.7.1+ kernels #9268 #9269
* etc/init.d/zfs-functions.in: remove arch warning
* zfs_handle used after being closed/freed in change_one callback
  #9165
* Fix zil replay panic when TX_REMOVE followed by TX_CREATE #7151
  #8910 #9123 #9145
* zfs_ioc_snapshot: check user-prop permissions on snapshotted
  datasets #9179 #9180
* Fix Plymouth passphrase prompt in initramfs script #9202
* Fix deadlock in 'zfs rollback' #9203
* Make slog test setup more robust #9194
* zfs-mount-genrator: dependencies should be space-separated #9174
* Linux 5.3: Fix switch() fall though compiler errors #9170
* Linux 5.3 compat: Makefile subdir-m no longer supported #9169
* Fix out-of-order ZIL txtype lost on hardlinked files #8769 #9061
* Increase default zcmd allocation to 256K #9084
* Improve performance by using dmu_tx_hold_*_by_dnode() #9081
* Fix channel programs on s390x #8992 #9080
* Race between zfs-share and zfs-mount services #9083
* Implement secpolicy_vnode_setid_retain() #9035 #9043
* zed crashes when devid not present #9054 #9060
* Don't directly cast unsigned long to void* #9065
* Fix module_param() type for zfs_read_chunk_size #9051
* Move some tests to cli_user/zpool_status #9057
* Race condition between spa async threads and export #9015 #9044
* hdr_recl calls zthr_wakeup() on destroyed zthr #9047
* Fix wrong comment on zcr_blksz_{min,max} #9052
* Retire unused spl_{mutex,rwlock}_{init_fini} #9029
* Linux 5.3 compat: retire rw_tryupgrade() #9029
* Linux 5.3 compat: rw_semaphore owner #9029
* Fix lockdep recursive locking false positive in dbuf_destroy #8984
* Add missing __GFP_HIGHMEM flag to vmalloc #9031
* Use zfsctl_snapshot_hold() wrapper #9039
* Minor style cleanup #9030
* Fix get_special_prop() build failure #9020
* systemd encryption key support #8750 #8848
* Drop redundant POSIX ACL check in zpl_init_acl() #9009
* Export dnode symbols #9027
* Ensure dsl_destroy_head() decrypts objsets #9021
* Disable unused pathname::pn_path* (unneeded in Linux) #9025
* Fixes: #8934 Large kmem_alloc #8934 #9011
* Fix ZTS killed processes detection #9003
* pkg-utils python sitelib for SLES15 #8969
* Fix race in parallel mount's thread dispatching algorithm #8450
  #8833 #8878
* Fix dracut Debian/Ubuntu packaging #8990 #8991
* Remove VERIFY from dsl_dataset_crypt_stats() #8976
* Improve "Unable to automount" error message. #8959
* Check b_freeze_cksum under ZFS_DEBUG_MODIFY conditional #8979
* Fix error text for EINVAL in zfs_receive_one() #8977
* Don't use d_path() for automount mount point for chroot'd process
  #8903 #8966
* nopwrites on dmu_sync-ed blocks can result in a panic #8957
* Avoid extra taskq_dispatch() calls by DMU #8909
* -Y option for zdb is valid #8926
* Fix error message on promoting encrypted dataset #8905 #8935
* Fix out-of-tree build failures #8921 #8943
* dn_struct_rwlock can not be held in dmu_tx_try_assign() #8929
* Remove arch and relax version dependency #8914
* Add libnvpair to libzfs pkg-config #8919
* Let zfs mount all tolerate in-progress mounts #8881
* zstreamdump: add per-record-type counters and an overhead counter
  #8432
* Fix comments on zfs_bookmark_phys #8945
* Add SCSI_PASSTHROUGH to zvols to enable UNMAP support #8933
* Prevent pointer to an out-of-scope local variable #8924 #8940
* dedup=verify doesn't clear the blkptr's dedup flag #8936
* Update vdev_ops_t from illumos #8925
* Allow unencrypted children of encrypted datasets #8737 #8870
* Replace whereis with type in zfs-lib.sh #8920 #8938
* Use ZFS_DEV macro instead of literals #8912
* Fix memory leak in check_disk() #8897 #8911
* kmod-zfs-devel rpm should provide kmod-spl-devel #8930
* ZTS: Fix mmp_interval failure #8906
* Minimize aggsum_compare(&arc_size, arc_c) calls. #8901
* Python config cleanup #8895
* lz4_decompress_abd declared but not defined #8894
* panic in removal_remap test on 4K devices #8893
* compress metadata in later sync passes #8892
* Move write aggregation memory copy out of vq_lock #8890
* Restrict filesystem creation if name referred either '.' or '..'
  #8842 #8564
* ztest: dmu_tx_assign() gets ENOSPC in spa_vdev_remove_thread()
  #8889
* Fix lockdep warning on insmod #8868 #8884
* fat zap should prefetch when iterating #8862
* Target ARC size can get reduced to arc_c_min #8864
* Fix typo in vdev_raidz_math.c #8875 #8880
* Improve ZTS block_device_wait debugging #8839
* Block_device_wait does not return an error code #8839
* Remove redundant redundant remove #8839
* Fix logic error in setpartition function #8839
* Allow metaslab to be unloaded even when not freed from #8837
* Avoid updating zfs_gitrev.h when rev is unchanged #8860
* l2arc_apply_transforms: Fix typo in comment #8822
* Reduced IOPS when all vdevs are in the
  zfs_mg_fragmentation_threshold #8859
* Drop objid argument in zfs_znode_alloc() (sync with OpenZFS) #8841
* Remove vn_set_fs_pwd()/vn_set_pwd() (no need to be at / during
  insmod) #8826
* grammar: it is / plural agreement #8818
* Refactor parent dataset handling in libzfs zfs_rename() #8815
* Update comments to match code #8759
* Update descriptions for vnops #8767
* Drop local definition of MOUNT_BUSY #8765
* kernel timer API rework #8647

Changes for 0.8.3

* Fix zfs-0.8.3 "qat.h"
* Prevent unnecessary resilver restarts #9155 #9378 #9551 #9588
* Fix QAT allocation failure return value #9784 #9788
* Fix zfs-0.8.3 zfs_receive_raw test case
* zdb: print block checksums with 6 d's of verbosity
* zfs-load-key.sh: ${ZFS} is not the zfs binary #9780
* Avoid some crashes when importing a pool with corrupt metadata #9022
* In initramfs, do not prompt if keylocation is "file://" #9764
* libspl: declare aok extern in header #9752
* Cancel initialize and TRIM before vdev_metaslab_fini() #8602 #9751
* Update maximum kernel version to 5.4 #9754 #9759
* Fix for ARC sysctls ignored at runtime
* cppcheck: (warning) Possible null pointer dereference: nvh #9732
* cppcheck: (error) Address of local auto-variable assigned #9732
* cppcheck: (error) Null pointer dereference: who_perm #9732
* cppcheck: (warning) Possible null pointer dereference: dnp #9732
* cppcheck: (error) Memory leak: vtoc #9732
* cppcheck: (error) Shifting signed 64-bit value by 63 bits #9732
* cppcheck: (error) Uninitialized variable #9732
* Exchanged two "${ZFS} get -H -o value" commands #9736
* Create symbolic links in /dev/disk/by-vdev for nvme disk devices
  #9730
* Force systems with kernel option "quiet" to display prompt for
  password #9731
* initramfs: setup keymapping and video for prompts #9723
* Don't fail to apply umask for O_TMPFILE files #8997 #8998
* Allow empty ds_props_obj to be destroyed #9704
* Fix use-after-free of vd_path in spa_vdev_remove() #9706
* zio_decompress_data always ASSERTs successful decompression #9612
  #9630
* Exclude data from cores unconditionally and metadata conditionally
  #9691
* Set send_realloc_files.ksh to use properties.shlib #9679
* Fix reporting of L2ARC hits/misses in arc_summary3 #9669
* Fix zdb_read_block using zio after it is destroyed #9644 #9657
* Fix use-after-free in case of L2ARC prefetch failure #9648
* Increase allowed 'special_small_blocks' maximum value #9131 #9355
* Adapt gitignore for modules #9656
* Fix encryption logic in systemd mount generator #9611
* Fix non-absolute path in systemd mount generator #9611
* Fix small typo in systemd mount generator #9611
* Implement -A (ignore ASSERTs) for zdb #9610
* Remove zfs_vdev_elevator module option #9417 #9609
* Add display of checksums to zdb -R #9607
* Check for unlinked znodes after igrab() #9602
* Remove requirement for -d 1 for zfs list and zfs get with bookmarks
  #9589
* Break out of zfs_zget early if unlinked znode #9583
* Remove inappropiate error message suggesting to use '-r' #9574
* Change zed.service to zfs-zed.service in man page #9581
* Prevent NULL pointer dereference in blkg_tryget() on EL8 kernels
  #9546 #9577
* Add missing documentation for some KMC flags #9034
* Fix zpool create -o <property> error message #9550 #9568
* Improve logging of 128KB writes #9409
* Skip loading already loaded key #9495 #9529
* Add a notice in /etc/defaults/zfs for systemd users #9544
* Include prototypes for vdev_initialize #9535
* dracut/zfs-load-key.sh: properly remove prefixes #9520
* Fix contrib/zcp/Makefile.am #9527
* Fix 'zfs change-key' with unencrypted child #9524
* Fix zpool history unbounded memory usage #9516
* Fix incremental recursive encrypted receive #9494
* Use correct format string when printing int8 #9486
* Name anonymous enum of KMC_BIT constants #9478
* Update skc_obj_alloc for spl kmem caches that are backed by Linux
  #9474
* Modify sharenfs=on default behavior #9397 #9425
* Implement ZPOOL_IMPORT_UDEV_TIMEOUT_MS #9436
* Clarify loop variable name in zfs copies test #9445
* Fix pool creation with feature@allocation_classes disabled #9427
  #9429
* Update zfs program command usage #9056 #9428
* Fix automount for root filesystems #9381 #9384
* Rename rangelock_ functions to zfs_rangelock_ #9402
* Workaround to avoid a race when /var/lib is a persistent dataset
  #9360
* Fix for zfs-dracut regression #8913 #9379
* Perform KABI checks in parallel #8547 #9132 #9341
* SIMD: Use alloc_pages_node to force alignment #9608 #9674
* Linux 5.0 compat: SIMD compatibility
* Add warning for zfs_vdev_elevator option removal #9317
* diff_cb() does not handle large dnodes #7678 #8931 #9343
* Use signed types to prevent subtraction overflow #9355
* Refactor libzfs_error_init newlines #9330
* Device removal of indirect vdev panics the kernel #9327
* Fix clone handling with encryption roots #9267 #9294
* Canonicalize Python shebangs #9314
* Fix stalled txg with repeated noop scans #9300
* Clean up do_vol_test in zfs_copies tests #9286
* Fix noop receive of raw send stream #9221 #9173
* Clean up zfs_clone_010_pos #9284
* Refactor checksum operations in tests #9280
* Use the right booleans #9264
* Fix panic on DilOS with kstat per dataset statistics #9254 #9151
* maxinflight can overflow in spa_load_verify_cb() #9272
* Fix typos #9251
* Fix typos in module/zfs/ #9240
* Fix typos in lib/ #9237
* Fix typos in module/ #9241
* Fix typos in modules/icp/ #9239
* Fix typos in include/ #9238
* Fix typos in etc/ #9236
* Fix typos in contrib/ #9235
* Fix typos in cmd/ #9234
* Fix typos in man/ #9233
* Fix typos in config/ #9232
* Fix refquota_007_neg.ksh #9257
* Prevent metaslab_sync panic due to spa_final_dirty_txg #9185 #9186
  #9231 #9253
* Simplify deleting partitions in libtest #9224
* Use compatible arg order in tests #9228
* Use smaller default slack/delta value for schedule_hrtimeout_range()
  #9217
* Prefer for(;;) to while (TRUE) #9219
* Add regression test for "zpool list -p" #9134
* Split argument list, satisfy shellcheck SC2086 #9212
* Fix install error introduced by #9089
* Document ZFS_DKMS_ENABLE_DEBUGINFO in userland configuration #9191
* Dedup IOC enum values in libzfs_input_check #9188
* Enhance ioctl number checks #9187
* Minor cleanup in Makefile.am #9189
* zfs-functions.in: in_mtab() always returns 1 #9168
* Fix lockdep circular locking false positive involving sa_lock #9110
* Set "none" scheduler if available (initramfs) #9042
* Add more refquota tests #9139
* initramfs: fixes for (debian) initramfs #7904 #9089
* dmu_tx_wait() hang likely due to cv_signal() in
  dsl_pool_dirty_delta() #9137
* Improve write performance by using dmu_read_by_dnode() #9156
* Assert that a dnode's bonuslen never exceeds its recorded size #8348
* Make txg_wait_synced conditional in zfsvfs_teardown #9115
* Prevent race in blkptr_verify against device removal #9112
* Fix device expansion when VM is powered off #9111
* spa_load_verify() may consume too much memory #9146
* Change boolean-like uint8_t fields in znode_t to boolean_t #9092
* Drop KMC_NOEMERGENCY #9119
* Don't wakeup unnecessarily in 'zpool events -f' #9091
* Test cancelling a removal in ZTS #9101
* lockdep false positive - move txg_kick() outside of ->dp_lock #9094
* Add channel program for property based snapshots #8443 #9050
* install path fixes #9087
* Don't activate metaslabs with weight 0 #8968
* OpenZFS 9318 - vol_volsize_to_reservation does not account for raidz
  skip blocks #8973
* Concurrent small allocation defeats large allocation #8843
* Fix bp_embedded_type enum definition #8951
* OpenZFS 9425 - channel programs can be interrupted #8904
* looping in metaslab_block_picker impacts performance on fragmented
  pools #8877
* single-chunk scatter ABDs can be treated as linear #8580
* make zil max block size tunable #8865

Test-Parameters: testlist=sanity,sanity-hsm,sanityn,sanity-quota fstype=zfs ostcount=2 mdscount=2
Signed-off-by: Nathaniel Clark <nclark@whamcloud.com>
Change-Id: I28e9cd70e56a2d73fc9b8347a9ddfe28e0a85090
Reviewed-on: https://review.whamcloud.com/37373
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9855 lustre: replace LPROCFS_CLIMP_CHECK()

The usage pattern for LPROCFS_CLIMP_CHECK() is clumsy.
It must be paired with LPROCFS_CLIMP_EXIT(), but
not doing this does not produce a compile-time error.
The 'import' should not be dereferenced before the CHECK, or
used after the EXIT, but sometimes it is.

Replace it with a structure macro/statement:

 with_obd_imp_lock(obd, imp, rc) {
     statements;
 }

statements are protected by the semaphore and only run if imp can be
set to a non-NULL pointer.
rc can be changed by the statements, and should be returned
afterwards as it may have been set to -ENODEV.

Errors fixed with this patch:
- some code tested u.cli.cl_import no-NULL even after
  LPROCFS_CLIMP_CHECK()
- some code dereferences cl_import before calling
  LPROCFS_CLIMP_CHECK()
- short_io_bytes_store() and max_procs_in_flight_store() don't access
  the import, so don't need LPROCFS_CLIMP_CHECK
- lprocfs_import_seq_write() set count to an error before 'goto out'
  which would free memory of length "count+1".
- lprocfs_import_seq_write() also called ptlrpc_recover_import()
  on the imp *after* dropping the semaphore.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: If9d5eb452157d7f76796f690569ef13fec111d76
Reviewed-on: https://review.whamcloud.com/36956
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12678 lnet: socklnd: mark all ksock_proto struct 'const'.

These structs are always read-only, so tell the compiler.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Icc7c3209135a2ab0d04a822b7053231fd2d9ff0c
Reviewed-on: https://review.whamcloud.com/36894
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13004 osp: use KIOV in osp_prep_update_req

Convert osp_prep_update_req to use a BULK_BUF_KIOV
rather than a BULK_BUF_KVEC descriptor.

This is a step towards remove KIOV support.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I2fdf84d73ba2d34c678b6eb6a8bbd323a761dfe4
Reviewed-on: https://review.whamcloud.com/36828
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13004 target: use KIOV for out_handle

Convert out_handle() use use a BULK_BUF_KIOV rather than
a BULK_BUF_KVEC.

This is a step towards removed KVEC support and standardizing
on KIOV.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I3f5b1b06183a716ba57d6f7f2a28bf5aa0f76dfe
Reviewed-on: https://review.whamcloud.com/36826
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13004 osp: break read request into pages.

Rather than breaking up a read request into arbitrarily
sized (4K) pieces of memory in virtual address space,
break it up into pages (which might be 64K) and
use a kiov rather than kvec to manage them.

This is a step towards removing kvec suport and
standardizing on kiov.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: If688764c53066a9c4db212682085fa899d4dde1b
Reviewed-on: https://review.whamcloud.com/36825
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>