LU-9859 libcfs: remove Lustre specific bitmap handling

Only the NRS TBF handling uses the Lustre specific bitmap
handling. Convert to the Linux bitmap API and remove the
Lustre specific bitmap handling.

Test-Parameters: trivial testlist=sanityn env=ONLY=77
Change-Id: I58dcf869778d6cf6349c16e73d75e53735ffb97d
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/48222
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-16085 llite: fix stat attributes_mask

Fix stat attributes_mask to return STATX_ATTR_ENCRYPTED whenever it is
possible. Also fix sanityn test_106c to expect at least the 0x30 flag
for attributes_mask.

Test-Parameters: trivial
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Icd16beff058c42d77e9b04ad1a287ec2ac04dfed
Reviewed-on: https://review.whamcloud.com/48208
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-16093 kernel: kernel update SLES12 SP5 [4.12.14-122.130.1]

Update SLES12 SP5 kernel to 4.12.14-122.130.1 for Lustre client.

Test-Parameters: trivial clientdistro=sles12sp5 \
env=SANITY_EXCEPT="56oc 430c 817" testlist=sanity

Change-Id: Ib2180a056889d481a7b55c41cbcd98c8e0e272d8
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/48204
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Colin Faber <cfaber@ddn.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-16084 tests: fix lustre-patched filefrag check

Fix sanity test_130b thru test_130g to check for "filefrag -l"
instead of "filefrag -e", since the "-e" option has been in
upstream e2fsprogs since commit v1.42.6-50-g2508eaa7.  The "-l"
option (logical extent ordering) is really what is needed to
handle Lustre-striped files anyway.

While there, fix the code style in these subtests:
- use "local" and lower-case names for local variables
- use $(...) for subshells
- use (( ... )) for numeric comparisons
- use preferred "check || action" style checks
- use "skip_env" for environment configuration checks (e2fsprogs)
- use "skip" for test-related checks that can't be "fixed"
- use pre-defined $ost1_FSTYPE for checking OST filesystem type

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I8eb7f17a9532796ab0274247194dd52cbc8a141c
Reviewed-on: https://review.whamcloud.com/48188
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15994 tests: add testing for io_uring via fio

This patch adds test case for io_uring I/O engine via fio.

Test-Parameters: trivial testlist=sanity
Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I0f2e371f91c02dc76644f42e5d1055ec200597c6
Reviewed-on: https://review.whamcloud.com/48167
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15548 tests: skip conf-sanity/131 for older servers

Skip conf-sanity.sh test_131 when running against older servers that
do not support the trusted.projid xattr.

Test-Parameters: trivial testlist=conf-sanity env=ONLY=131
Test-Parameters: testlist=conf-sanity env=ONLY=131 serverversion=2.14.0
Fixes: e4d07f2c30 ("LU-12056 ldiskfs: add trusted.projid virtual xattr")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: If1858502ab50ffd10e494eab793e3bc0f883fe9e
Reviewed-on: https://review.whamcloud.com/48151
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15873 obd: skip checking read-only fs health

Health check upon read-only file system would fail and STONITH
ensues.

Add obd_device::obd_read_only to record read-only flag of the
obd_device. And skip checking the health of read-only devices.

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Ica83b9c871f7bee62cef6504deb0dcc32dd20afb
Reviewed-by: Wang Shilong <wshilong@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-on: https://review.whamcloud.com/48095
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-1904 idl: add checks for OBD_CONNECT flags

Make it harder to accidentally declare OBD_CONNECT flags without
properly defining their names.  Otherwise, this can cause serious
compatibility problems if two features are using the same flag.

Add the definition lines into spelling.txt so there is *always*
a warning generated, since this always needs proper attention.

Make it clear whom to contact when reserving a new feature flag.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I9a5e2c97c40c39ea57d20979d4b130854edc785a
Reviewed-on: https://review.whamcloud.com/48053
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-16012 sec: fix detection of SELinux enforcement

On newer distros (e.g. RHEL 9.0), on which selinux_is_enabled() does
not exist anymore, the only way to find out if SELinux is enforced
when initializing the security context is to fetch the length of the
security attribute name. If it is 0, we conclude SELinux is disabled.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ifcdcb8ffbb7f9ad50d16d7d3317e94d0d212fa42
Reviewed-on: https://review.whamcloud.com/48049
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-16048 build: Update ZFS version to 2.1.5

Update ZFS version to 2.1.5. The changes are listed in:
https://github.com/openzfs/zfs/releases/tag/zfs-2.1.5

Change-Id: I9f25aafe889f87fb80677e59dbe4679932d8b920
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/48040
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-16045 enc: force use of new enc xattr on new servers

When an older client uses encryption with a newer server, the client
wants to see the encryption context in security.c xattr. But
internally on server side, we force use of newer encryption.c xattr
for consistency purpose. When required, the encryption context is put
in the request to the client as usual, which interprets it as desired.

Fixes: 4231fab66e ("LU-13717 sec: make client encryption compatible with ext4")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I667e123bdff912acc270666e8c74ebda6f0534e7
Reviewed-on: https://review.whamcloud.com/48035
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-16035 kfilnd: Initial kfilnd implementation

Initial implementation of the kfabric Lustre Network Driver.

Test-Parameters: trivial
HPE-bug-id: LUS-6565
Signed-off-by: Doug Oucharek <dougso@me.com>
Signed-off-by: Ian Ziemba <ian.ziemba@hpe.com>
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I48a070ca0ba37e4923cd6dcb3327676ae6ddaae1
Reviewed-on: https://review.whamcloud.com/48009
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Ron Gredvig <ron.gredvig@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-16027 tests: sanity:test_66: specify blocksize explicitly

Fix test_66() to be independent from BLOCKSIZE environment
variable.

To reproduce the failure, just run:
  llmount.sh
  export BLOCKSIZE=4096; ONLY=66 sh sanity.sh
  == sanity test 66: update inode blocks count on client ==
  8+0 records in
  8+0 records out
  8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00589935 s, 1.4 MB/s
  sanity test_66: @@@@@@ FAIL: /mnt/lustre/f66 blocks 2 < 8

Test-Parameters: trivial testlist=sanity env=ONLY=66
Signed-off-by: Elena Gryaznova <elena.gryaznova@hpe.com>
HPE-bug-id: LUS-11014
Reviewed-by: Vladimir Saveliev <vlaidimir.saveliev@hpe.com>
Change-Id: I0adca724518cb955e3664d33a36628ae19a1712d
Reviewed-on: https://review.whamcloud.com/47983
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Vladimir Saveliev <vladimir.saveliev@hpe.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15999 tests: format journal with correct block size

Without "-b block-size" mke2fs calculates block size itself based upon
device size. In result, filesystem and journal may be formatted with
different block sizes. For example, 32M device gets formatted to journal
with block size 1K and llmount.sh fails to create Lustre with external
journal:
   mke2fs: Filesystem has unexpected block size while trying
           to open journal device /dev/vdb
because the target device itself is created with default
"Block size: 4096".
Let's make sure that journal gets formatted with correct block size.

Patch also adds the ability to create fs with different block
sizes if BLCKSIZE or <facet>_BLOCKSIZE are set.

Fixes: d01d4c697a ("LU-957 scrub: Proc interfaces and tests for OI scrub")
Test-Parameters: trivial
Signed-off-by: Elena Gryaznova <elena.gryaznova@hpe.com>
HPE-bug-id: LUS-11008
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-by: Vladimir Saveliev <vlaidimir.saveliev@hpe.com>
Change-Id: I0a82e34efc23d318bbd52946916ae8f2b7cd94eb
Reviewed-on: https://review.whamcloud.com/47930
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Vladimir Saveliev <vladimir.saveliev@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15393 tests: check QoS hang with OST failover

Patch adds recovery small test 152, to reproduce situation
where MDT object allocation sleeps on OST failover at
lod_ost_alloc_rr under lq_rw_sem read. And all other creation threads
hang at lod_ost_alloc_qos at down_write(lq_rw_sem).

HPE-bug-id: LUS-10388
Test-Parameters: trivial testlist=recovery-small env=ONLY=152
Signed-off-by: Alexander Boyko <alexander.boyko@hpe.com>
Change-Id: I7b9c5a5c9870a559e673a5fd253dcaea40d9fe63
Reviewed-on: https://review.whamcloud.com/47715
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andriy Skulysh <andriy.skulysh@hpe.com>
Reviewed-by: Vitaly Fertman <vitaly.fertman@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-16081 lnet: Memory leak on adding existing interface

In the function lnet_dyn_add_ni an lnet_ni structure is allocated.
In case of an error the function returns without freeing the memory of
the structure.
Added handling of possible lnet_net structure memory leaks.

Test-parameters: trivial testlist=sanity-lnet
Signed-off-by: Frank Sehr <fsehr@whamcloud.com>
Change-Id: I7544a9379093b99f77aaddb8d021b4a5bf221082
Reviewed-on: https://review.whamcloud.com/48173
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15694 quota: keep grace time while setting default limits

The quota grace time should only be changed by "lfs setquota -t",
and it should be kept while setting default quota limits.

This patch also fixes an issue of not saving the grace time while
writing glboal quota record.

Signed-off-by: Hongchao Zhag <hongchao@whamcloud.com>
Change-Id: I89ca49d09dc41deffe4bc77e53721b5bb4f4be37
Reviewed-on: https://review.whamcloud.com/46935
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15642 obdclass: use consistent stats units

Use consistent stats units, since some were "usec" and others "usecs".
Most stats already use LPROCFS_TYPE_* to encode type stats type, so
use this to provide units for those stats, and only explicitly provide
strings for the few stats that don't match the commonly-used units.
This also reduces the number of repeat static strings in the modules.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I25f31478f238072ddbf9a3918cd43bb08c3ebbe5
Reviewed-on: https://review.whamcloud.com/46833
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Ben Evans <beevans@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15930 lnet: Remove duplicate checks for peer sensitivity

Callers of lnet_inc_lpni_healthv_locked() and
lnet_dec_healthv_locked() currently check whether the parent peer
has a peer specific sensitivity defined. To remove this code
duplication, this logic is rolled into
lnet_inc_lpni_healthv_locked() and lnet_dec_lpni_healthv_locked().
The latter is a new wrapper around lnet_dec_healthv_locked().

lnet_dec_healthv_locked() is changed to return a bool indicating
whether the health value was actually modified so that the peer
net health is only updated when the peer NI health actually changes.

Test-Parameters: trivial testlist=sanity-lnet
HPE-bug-id: LUS-11018
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I624561167392ad625ea7478689e9c5975cec3f2e
Reviewed-on: https://review.whamcloud.com/46626
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15929 lnet: Correct net selection for router ping

lnet_find_best_ni_on_local_net() contains logic for restricting
the NI selection to a net specified by lnet_peer::lp_disc_net_id. The
purpose of this is to ensure that LNet peers ping every interface on
a router at a regular interval as part of the LNet router health
feature. However, this logic is flawed because lnet_msg_discovery()
is used to determine whether the message being sent is a discovery
message, but that function actually determines whether a given message
can _trigger_ discovery.

Introduce a new function, lnet_msg_is_ping(), which determines whether
a given lnet_msg is a GET on the LNET_RESERVED_PORTAL.
Modify lnet_find_best_ni_on_local_net() to restrict NI selection to
lp_disc_net_id iff:
1. lp_disc_net_id is non-zero
2. The peer has the LNET_PEER_RTR_DISCOVERY flag set.
3. lnet_msg_is_ping() returns true

Test-Parameters: trivial testlist=sanity-lnet
HPE-bug-id: LUS-11017
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I3dbdfd5c44b6167d24b7b6e0b1097db0b3c5cb76
Reviewed-on: https://review.whamcloud.com/47527
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15595 lnet: LNet peer aliveness broken

The peer health feature used on LNet routers is intended to detect if
a peer is dead or alive by keeping track of the last time it received
a message from the peer. If the last alive value is outside of a
configurable interval then the peer is considered dead and the router
will drop messages to that peer rather than attempt to send to it.

This feature no longer works as intended because even if the
last alive value is outside the interval the router will still
consider the peer NI to be alive if the health value of the NI and
the cached status both indicate the peer NI is alive.

So even if a router has not received any messages from the client in
days, as long as the router thinks the peer's interfaces are healthy
then it will consider the peer alive. This doesn't make any sense as
peers are supposed to regularly ping the router, and if they don't do
so then they should not be considered alive.

Fix the issue by relying solely on the last alive value to determine
peer aliveness. Do not consider the health value or cached status
when determining whether to drop the message.

lnet_peer_alive_locked() has single caller that only checks whether
zero was returned. We can convert lnet_peer_alive_locked() to return
bool rather than int.

Rename lnet_peer_alive_locked() to lnet_check_message_drop() to
better reflect the purpose of the function. The return value is
inverted to reflect the name change.

Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: Iaabdf5109676ffd18bdba9627afea7e041ddc1e1
Reviewed-on: https://review.whamcloud.com/46623
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15595 tests: Add various router tests

Add test cases to exercise LNet routing.

Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I4a077937b3e3b8b07707afeb0c5c23ec1c9074f4
Reviewed-on: https://review.whamcloud.com/46622
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14955 lnet: Use fatal NI if none other available

Allow NI in fatal state to be selected for sending if there are no
NIs in non-fatal state.

Test-Parameters: trivial testlist=sanity-lnet
HPE-bug-id: LUS-11019
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: Iab8ef6ee5c5f45896196dbd88a2f61e004278297
Reviewed-on: https://review.whamcloud.com/44746
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-16058 build: proc_ops check fails with SUBARCH undefined

During configure with config.cache enabled SUBARCH may not
be defined.

Move the definition to a location that must be traversed.

Test-Parameters: trivial
Fixes: a5084c2f2e ("LU-14937 build: re-use config cache in 'make rpms/debs'")
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I0a7b4de3ecccd41b1c55e8b2df29039517e0c416
Reviewed-on: https://review.whamcloud.com/48101
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12514 target: move server mount code to target layer

Currently the server mount code for lustre_tgt is all in obdclass. If
we change the stack to initialize the LNet / ptlrpc layer after mounting
then we will end up with modular circular dependencies. To avoid this
move all the lustre_tgt mounting code to the target layer. This way the
mounting code can use both ptlrpc and obdclass module routiens. Also include
MODULE_ALAIS("lustre_tgt") so mount -t lustre_tgt will load ptlrpc which
contains the target layer.

Change-Id: I392602e8fd18d001cb97b05b909c366ba5b8fa82
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/47160
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15811 llite: Refactor DIO/AIO free code

Refactor the DIO/AIO free code and add some asserts.

This removes a potential use-after-free in the freeing
code.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I335b18fc7a28fc426a25675e2449d3d192cba596
Reviewed-on: https://review.whamcloud.com/48115
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15811 llite: Unify range unlock

Correct parallel_dio condition and unify range unlock code
block.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Ib66e8def571054df5117c279e238894bc3b58bce
Reviewed-on: https://review.whamcloud.com/48000
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15003 sec: use enc pool for bounce pages

Take pages from the enc pool so that they can be used for
encryption, instead of letting llcrypt allocate a bounce page
for every call to the encryption primitives.
Pages are taken from the enc pool a whole array at a time.

This requires modifying the llcrypt API, so that new functions
llcrypt_encrypt_page() and llcrypt_decrypt_page() are exported.
These functions take a destination page parameter.
Until this change is pushed in upstream fscrypt, this performance
optimization is not available when Lustre is built and run against
the in-kernel fscrypt lib.

Using enc pool for bounce pages is a worthwhile performance win. Here
are performance penalties incurred by encryption, without this patch,
and with this patch:

                     ||=====================|=====================||
                     || Performance penalty | Performance penalty ||
                     ||    without patch    |     with patch      ||
||==========================================|=====================||
|| Bandwidth – write |        30%-35%       |   5%-10% large IOs  ||
||                   |                      |    15% small IOs    ||
||------------------------------------------|---------------------||
|| Bandwidth – read  |         20%          |    less than 10%    ||
||------------------------------------------|---------------------||
||      Metadata     |         N/A          |         5%          ||
|| creat,stat,remove |                      |                     ||
||==========================================|=====================||

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Signed-off-by: James Simmons <jsimmons@infradead.org>
Change-Id: I3078d0a3349b3d24acc5e61ab53ac434b5f9d0e3
Reviewed-on: https://review.whamcloud.com/47149
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14719 utils: dir migration stop on error

Once directory migration fails, it should stop immediately since
current migration won't succceed, and subsequent migration may
fail on the same error.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I96c1693d1b1da0856c925b9b22c1ab7f3181f0d8
Reviewed-on: https://review.whamcloud.com/47040
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15357 iokit: fix the obsolete usage of cfg_device

The LCTL command "cfg_device" is obsolete and some operations
(such as "cleanup", "detach") don't support it anymore.
In mds_survey and lfsck-performance it causes the echo client
device not to be destroyed and causes LBUG when umounting the
related Lustre device.

Change-Id: If7f6eff080906e395023289652fcd2a78dfb6fb7
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45872
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Colin Faber <cfaber@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15811 llite: Rework upper/lower DIO/AIO

One of the patches for LU-13799,
"Implement lower/upper aio"
(https://review.whamcloud.com/44209/) created a
complicated setup where the cl_dio_aio struct was used
both for the top level DIO or AIO and for the lower level
sub I/Os (corresponding to stripes).

This is quite complicated and hard to follow, so this
rewrites these two uses to be separate structs.  This
incidentally fixes at least one possible memory leak, but
is mostly a cleanup.

Fixes: 46ff761371 "LU-13799 Implement lower/upper aio"
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Ide4a2b84f48624ee97dfb57fe80d201fbb7fe8d0
Reviewed-on: https://review.whamcloud.com/47187
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10391 lnet: change ni_status in lnet_ni to u32*

struct lnet_ni.ni_status points to a 'struct lnet_ni_status', but only
the ns_status field of that structure is ever accessed.

Change ni_status to point directly to just the ns_status field.
This will provide flexibility for introducing a variant for 'struct
lnet_ni_status' which holds a large-address nid.

Test-Parameters: trivial testlist=sanity-lnet
Test-Parameters: serverversion=2.12 serverdistro=el7.9 testlist=runtests
Test-Parameters: clientversion=2.12 testlist=runtests
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I5570608e98bc2aa1156b8d885df2a56f8ae7b6f7
Reviewed-on: https://review.whamcloud.com/44626
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-16056 libcfs: restore umask handling in kernel threads

This reverts commit 9013eb2bb5 which incorrectly assumes that Lustre
service threads do not modify umask.  A quick grep shows that umask
is modified in osd-ldiskfs __osd_create().

If some other thread sharing the same fs context is modifying umask
in an incompatible way (which includes all Lustre threads after
this patch) then it will occasionally break created file access
permissions for Lustre.

Change-Id: I589b72e4286dc84f4e3f1a0c54fe31aa988e6c18
Fixes: 9013eb2bb5 (LU-9859 libcfs: don't call unshare_fs_struct()")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/48233
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Shuichi Ihara <sihara@ddn.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-16082 ldiskfs: old-style EA inode handling fix

The upstream version of EA inodes support coming
with RHEL8 (linux kernel 4.18+) have a slightly different
implementation of EA inodes support and also have a
compatibility code to recognize old-style Lustre-only EAs.
Unfortunately the compatibility code is broken and makes
old xattr data unaccessible due to a wrong hash value check.

HPE-bug-id: LUS-11133
Signed-off-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Change-Id: Icd6f93d4ebb33dcd03b58f9eb364905c18ae81dc
Reviewed-on: https://review.whamcloud.com/48174
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-16011 lnet: use preallocate bulk for server

Server side want to have a preallocate bulk to avoid large lock
contention on the page cache.
Without it LST limited with 35Gb/s speed with 3 rail host (HDR each)
due large CPU usage.
Preallocate bulks increase a memory consumption for small bulk,
but performance improved dramatically up to 74Gb/s with very low
cpu usage.

Test-Parameters: trivial testlist=sanity-lnet,lnet-selftest
Signed-off-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Change-Id: If1eaf5addf6c9d9f695a892dc66023b3bc293208
Reviewed-on: https://review.whamcloud.com/47952
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Andrew Perepechko <andrew.perepechko@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15959 kernel: new kernel [SLES15 SP4 5.14.21-150400.24.18.1]

This patch makes changes to support new SLES15 SP4 release
with kernel 5.14.21-150400.24.18.1 for Lustre client.

Test-Parameters: trivial clientdistro=sles15sp4 \
env=SANITY_EXCEPT="27J 244a" testlist=sanity
Test-Parameters: trivial clientdistro=sles15sp3

Change-Id: I0bf548835578163767d2f6a2a5e5bd2b33154871
Co-Authored-By: Minh Diep <mdiep@whamcloud.com>
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47696
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14745 tests: ensure sanity/51d has enough objects

Ensure that sanity test_51d has precreated enough objects on the
OSTs for the tests to finish without running out.  Otherwise,
some OSTs may be skipped and skew the results.

Test-Parameters: trivial testlist=sanity env=ONLY=51d,ONLY_REPEAT=5 mdscount=2 mdtcount=4
Test-Parameters: testlist=sanity env=ONLY=51d,ONLY_REPEAT=5 mdscount=2 mdtcount=4
Test-Parameters: testlist=sanity env=ONLY=51d,ONLY_REPEAT=5 mdscount=2 mdtcount=4
Test-Parameters: testlist=sanity env=ONLY=51d,ONLY_REPEAT=5 mdscount=2 mdtcount=4
Test-Parameters: testlist=sanity env=ONLY=51d,ONLY_REPEAT=5 mdscount=2 mdtcount=4
Test-Parameters: testlist=sanity env=ONLY=51d,ONLY_REPEAT=5 mdscount=2 mdtcount=4
Test-Parameters: testlist=sanity env=ONLY=51d,ONLY_REPEAT=5 mdscount=2 mdtcount=4
Fixes: fd5c915eff ("LU-15282 tests: improve sanity test_51d coverage")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ifaf79173aa34fc7f3b3b1ad3d2876b65c16d1474
Reviewed-on: https://review.whamcloud.com/48086
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>

LU-15874 kernel: new kernel [RHEL 9.0 5.14.0-70.22.1.el9_0]

This patch makes changes to support new RHEL 9.0 release
for Lustre client.

fix lbuild to include modified find-requires.ksyms

Test-Parameters: trivial clientdistro=el9.0 \
env=SANITY_EXCEPT="130 244a" testlist=sanity

Change-Id: Ib7fdf9d3946df626759d395b5000b375391da344
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47847
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-16078 o2iblnd: Salt comp_vector

If conns_per_peer is greater than 1, all the connections targeting
the same peer are assigned the same comp_vector. This results in
multiple IB CQs targeting the same peer to be serialized on a single
comp_vector.

Help spread out the IB CQ work to multiple cores by salting
comp_vector based on number of connections.

1 client to 1 server LST 1M write results with 4 conns_per_peer and
RXE configured to spread out work based on comp_vector.

Before: 1377.92 MB/s
After: 3828.48 MB/s

Test-Parameters: trivial
HPE-bug-id: LUS-11043
Change-Id: I4e3e2056947ee54d6d65f17e238163c9dc38cd61
Signed-off-by: Ian Ziemba <ian.ziemba@hpe.com>
Reviewed-on: https://review.whamcloud.com/48148
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13813 tests: fix stack_trap in conf-sanity test 110/111

This patch fixes stack_trap in conf-sanity test 110 and 111
to restore test environment.

Test-Parameters: trivial env=SLOW=yes,ENABLE_QUOTA=yes,CONF_SANITY_EXCEPT=107 \
clientdistro=el8.5 serverdistro=el8.5 testlist=conf-sanity

Test-Parameters: trivial env=SLOW=yes,ENABLE_QUOTA=yes \
fstype=zfs \
clientdistro=el8.5 serverdistro=el8.5 testlist=conf-sanity

Test-Parameters: trivial env=SLOW=yes,ENABLE_QUOTA=yes,CONF_SANITY_EXCEPT=107 \
mdscount=2 mdtcount=4 \
clientdistro=el8.5 serverdistro=el8.5 testlist=conf-sanity

Change-Id: I540d96e8ad2c4990e7da18fe22256b44e9a19c72
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/48022
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14139 statahead: add total hit/miss count stats

In this patch, it adds total hit/miss count stats for statahead.
These statistics are updated when the statahead thread terminated.

This patch also adds support to clear all statahead stats:
$LCTL set_param llite.*.statahead_stats=0

Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I8b11d26385234305631c232a15711224dcfb0668
Reviewed-on: https://review.whamcloud.com/46309
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14139 llite: simplify callback handling for async getattr

In this patch, it prepares the inode and set lock data directly in
the callback interpret of the intent async getattr RPC request (in
ptlrpcd context), simplifies the old impementation that defer this
work in the statahead thread.

If the statahead entry is a striped directory, it may generate
new RPCs in the ptlrpcd interpret context to obtain the
attributes for slaves of the striped directory:
@ll_prep_inode()->@lmv_revaildate_slaves()
This is dangerous and may result in deadlock in ptlrpcd interpret
context, thus we use work queue to handle these extra RPCs.
Add sanity 123d to verify that it works correctly.

According to the benchmark result, the workload "ls -l" to a large
directory on a client without any caching (server and client),
containing 1M files (47001 bytes) shows the results with measured
elapsed time:
- w/o patch: 180 seconds;
- w patch: 181 seconds;

There is no any obvious performance regession.

Test-Parameters: testlist=racer,racer,racer
Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I43aba0f609243f34f7e7b674c7fff5fa417b1c02
Reviewed-on: https://review.whamcloud.com/45648
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-16061 osd-ldiskfs: clear EXTENT_FL for symlink agent inode

The flag should be cleared for "fast" symlinks otherwise
e2fsck complains about inode correctness.
New agent inodes of symlink type may have EXT4_EXTENT_FL flag
set if the fs has "extent" feature and it is not cleared as in
other places where "fast" symlinks are created.

HPE-bug-id: LUS-10237

Signed-off-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Change-Id: Ib7b807bb1298cc3a9fd4fdba35747b4bda6fe034
Reviewed-on: https://review.whamcloud.com/48093
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-16060 osd-ldiskfs: copy nul byte terminator in writelink

memcpy() call in osd_ldiskfs_writelink() doesn't copy the nul
terminator byte from the source buffer, leaving the space
after target link name uninialized which is ok for the kernel
code and debugfs but not e2fsck.

HPE-bug-id: LUS-11103

Signed-off-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Change-Id: I914f2c78e1a6571bf360a23b0ede8c70502bf0df
Reviewed-on: https://review.whamcloud.com/48092
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Reviewed-by: Andrew Perepechko <andrew.perepechko@hpe.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-16037 build: remove quotes from %{mkconf_options}

This patch fixes lustre-dkms.spec.in to remove quotes
from %{mkconf_options} passed to dkms.mkconf, so as to
resolve the following build issue:

dkms.conf: Error! Directive 'DEST_MODULE_LOCATION'
does not begin with '/kernel', '/updates', or '/extra'
in record #0.

Test-Parameters: trivial

Change-Id: I0b365d7a96cb632680bc2321e87b28a3bf076e47
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/48044
Reviewed-by: Colin Faber <cfaber@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

New tag 2.15.51

Change-Id: I2fc9e9fae7a975f047528c4670276b239d77ac26
Signed-off-by: Oleg Drokin <green@whamcloud.com>

LU-15994 llite: use fatal_signal_pending in range_lock

FIO io_uring failed with one file shared by two FIO processes
under Unubtu 2204 kernel.
After analyzed, we found that range_lock() function return
-ERESTARTSYS when there pending signal on current process in
Lustre I/O. This causes -EINTR returned to the application.

we solve this bug by replacing @signal_pending(current) with
@fatal_signal_pending(current) in range_lock(). The range_lock()
function only returns -ERESTARTSYS when the current process has
fatal pending signal such as SIGKILL.

Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I0a0be8fa3b4ba5c89f7866286b2bdc6595f18026
Reviewed-on: https://review.whamcloud.com/48106
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-16042 tests: can not get cache size on Arm64

This fix the test fail on Arm64, the cache size can not be
display on /proc/cpuinfo. And even in the VM and somee
older Arm64 CPU, we can not get the cachesize. So it's
better to fallback to a pre-set value here if we don't get
the cache size.

Signed-off-by: Kevin Zhao <kevin.zhao@linaro.org>
Change-Id: I17ce1d8accc69d1489db2071a2741b3927fff302
Reviewed-on: https://review.whamcloud.com/48030
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-16019 llite: fully disable readahead in kernel I/O path

In the new kernel (rhel9 or ubuntu 2204), the readahead path may
be out of the control of Lustre CLIO engine:

generic_file_read_iter()
  ->filemap_read()
    ->filemap_get_pages()
      ->page_cache_sync_readahead()
        ->page_cache_sync_ra()

void page_cache_sync_ra()
{
if (!ractl->ra->ra_pages || blk_cgroup_congested()) {
if (!ractl->file)
return;
req_count = 1;
do_forced_ra = true;
}

/* be dumb */
if (do_forced_ra) {
force_page_cache_ra(ractl, req_count);
return;
}
...
}

From the kernel readahead code, even if read-ahead is disabled
(via @ra_pages == 0), it still issues this request as read-ahead
as we will need it to satisfy the requested range. The forced
read-ahead will do the right thing and limit the read to just
the requested range, which we will set to 1 page for this case.

Thus it can not totally avoid the read-ahead in the kernel I/O
path only by setting @ra_pages with 0.
To fully disable the read-ahead in the Linux kernel I/O path, we
still need to set @io_pages to 0, it will set I/O range to 0 in
@force_page_cache_ra():
void force_page_cache_ra()
{
...
max_pages = = max_t(unsigned long, bdi->io_pages,
    ra->ra_pages);
nr_to_read = min_t(unsigned long, nr_to_read, max_pages);
while (nr_to_read) {
...
}
...
}

After set bdi->io_pages with 0, it can pass the sanity/101j.

Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I859a6404abb9116d9acfa03de91e61d3536d3554
Reviewed-on: https://review.whamcloud.com/47993
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15938 llog: llog_reader to detect more corruptions

Improve llog_reader to determine more corruptions and report
errors
 - notify if llog bitmap has bits set with no records in llog
 - compare header records count with amount of records really
   found
 - fix amount of records to output, preventing wrong output of
   NOT SET record
 - list missing records in gap if found
 - count all errors found, add prefix 'error:' in output for
   better output processing by third-party scripts
 - don't exit immediately in case of error but continue if
   possible and output all read valid data

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: Ic47dc6bb6cbdd9db6f888a0b892254403a628912
Reviewed-on: https://review.whamcloud.com/47934
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15996 quota: change 'none' to 'expired' for grace

If the grace time is expired, the output of grace in 'lfs quota'
is better to use 'expired' than 'none'.

Test-Parameters: trivial testlist=sanity-quota
Signed-off-by: Honghao Zhang <hongchao@whamcloud.com>
Change-Id: I7a3fac77ca6e16ad406bef0bd7642d6d50feb4b2
Reviewed-on: https://review.whamcloud.com/47912
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15896 gss: support OpenSSLv3

Lustre GSS code makes use of some OpenSSL API that has been
deprecated in v3, namely all the functions in the DH_* family.
So replace them with their EVP_PKEY_* counterparts if Lustre is
built on a system with OpenSSLv3.

Fixes: ee60c14360 ("LU-15896 gss: ignore OpenSSLv3 deprecated API")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I78a4ca18b25aca3c34fe84e41413a33caddc01b6
Reviewed-on: https://review.whamcloud.com/47717
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9859 lfsck: use Linux bitmap API

Replace the use of the libcfs specific bitmap API used by lfsck
with the standard Linux bitmap API.

Change-Id: Icc0d9d2ceb9ca7b4b94dd728d9b9c499cf4d2414
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/47579
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14975 utils: non-recursive dir migration fix

If sem_init() doesn't return 0, llapi_semantic_traverse() won't
call sem_fini() in directory traverse, therefore
cb_migrate_mdt_init() shouldn't increase param->fp_depth if it
reaches max depth in non-recursive mode.

Update sanity 230w.

Fixes: 5604a6d270b ("LU-14975 dne: dir migration in non-recursive mode")
Signed-off-by: Lai Syao <lai.siyao@whamcloud.com>
Change-Id: I8814aaae7c267cec51654175f9fa0708f7685a5a
Reviewed-on: https://review.whamcloud.com/47012
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15486 lod: mirroring a plain file in mirrored-layout dir

If a file does not have a mirror in a directory with a default FLR
mirror, then "lfs mirror extend" on the file fails with
"cannot create volatile file: Invalid argument".

This comes from the the non-striped file layout generated from
LOD inheriting its FLR state from the default FLR while it contains
no mirror in it, and lov_init_composite() will complain about it.

 if (equi(flr_state == LCM_FL_NONE, comp->lo_mirror_count > 1))
         RETURN(-EINVAL);

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I5e849acb2327ce735d0008271bfd48fa7293161c
Reviewed-on: https://review.whamcloud.com/46517
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-8837 lustre: make uapi...lustre_disk.h unnecessary on client

uapi/linux/lustre/lustre_disk.h doesn't contain anything that is
needed for client-only code, but that code doesn't compile with the
file excluded, largely due to dependency on IS_SERVER() and related
macros.

So for client-only code provide stubs for IS_SERVER() and related
macros, and don't include the uapi...lustre_disk.h file.

This will cause some code to now be compiled-out on client-only, and
allows some #ifdefs to be removed.

A few function need to be protected with #ifdef HAVE_SERVER_SUPPORT,
and llog_server.o needs to be disabled for client-only compiles.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I19c5b1612108e448f8b6a1fe3d3a448aa4abdd2a
Reviewed-on: https://review.whamcloud.com/41994
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-8238 ldlm: rid of obsolete param of ldlm_resource_get()

The second parameter @parent of ldlm_resource_get() is obsolete, just
remove it.

Test-Parameters: trivial
Change-Id: I88af99c6984eda50a21da4d516ce7dea8cba60f5
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/20631
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12511 utils: fix regression for UAPI headers for native client

A patch landed to add wiretest for the GSS wire protocol which is
lacking for the native client. Add disabling the new test code
for the native Linux client build.

Test-Parameters: trivial
Fixes: 7dfbc71350 ("LU-9243 gss: fix GSS struct definition badness")
Change-Id: I31c387b757a77f4503b923c784911afc16c878a0
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/47803
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9680 utils: fix Netlink / YAML library handling

Testing the implementation of Netlink with lustre with early
code revealed some userland lnetconfig library bugs. Several bugs
fixed are:

1) First issue was that the YAML parser and emitter can share an
   netlink socket. This means the netlink callbacks will expect
   that same void argument passed in. We were in the case of error
   handling expecting an struct yaml_netlink_output but all other
   callbacks were using struct yaml_netlink_input. This mismatch
   can cause the application to segfault. So move all netlink
   callback handling to use just the yaml_parser. The yaml
   emittter now is used to just send Netlink packets to the
   kernel.

   Also fix the Netlink ext_ack error message handling.

2) In my board testing I found various bugs related to the
   paring of the YAML to create Netlink packet to send to the
   kernel. This patch resolves all the known issues. Most
   related to the complex layering of sequences, mappings and
   flows in a YAML block.

3) Fix up nla_strlcpy autoconf test which always fails with
   Oleg's special setup.

4) Add a Netlink protocol version YAML function.

Test-Parameters: trivial
Change-Id: I7e7c755ceaa969dffff8c6f771c2ac048dc55720
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/47802
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Joe Atzinger <joseph.atzinger@microsoft.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-8367 osp: enable replay for precreation request

Lustre has some kind of deadlock between osp_precreate_thread()
and stripe allocation at osp_precreate_reserve(). Stripe allocation
thread allocated objects and sleeps for more objects at
osp_precreate_reserve() in case of OST failover. After reconnection,
osp_precreate_thread() calls osp_precreate_cleanup_orphans() to
synchronize last id and clean-up unused objects, but it waits zero
object reservation(d->opd_pre_reserved). So, no more objects could
be created at OST and no reserved objects could be freed.
This produce slow creates messages and MDT creation threads hang
osp_precreate_reserve()) kjcf05-OST0003-osc-MDT0001: slow creates,
 last=[0x340000400:0x23a4f483:0x0], next=[0x340000400:0x23a4f378:0x0],
 reserved=267, sync_changes=0, sync_rpcs_in_progress=0, status=0
The issue reproduced more often with over stripe feature.

No need to do orphan clean-up phase when MDT supports
resend/replay for precreation request. This behaviour resolves the
osp_precreate_cleanup_orphans() hang and unblocks objects creation.

Force creation logic is added to support reformatted OST with a same
index. It was done during orphan clean-up phase before this.

Sanity tests 27S and 822 become invalid. 27S is based on orphan
clean-up after reconnection, 822 is based on not resendable
OST_CREATE request. These tests are removed.

HPE-bug-id: LUS-10793
Signed-off-by: Alexander Boyko <alexander.boyko@hpe.com>
Change-Id: I21287b51252e573e796fac69ee3df6ac90e28c10
Reviewed-on: https://review.whamcloud.com/46889
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Vitaly Fertman <vitaly.fertman@hpe.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15850 lmv: always space-balance r-r directories

If the MDT free space is imbalanced, use QOS space balancing for
round-robin subdirectory creation, regardless of the depth
of the directory tree.  Otherwise, new subdirectories created
in parents with round-robin default layout may suddenly become
"sticky" on the parent MDT and upset the space balancing and
load distribution.

Add sanity/test_413h to check that round-robin dirs always balance.

Test-Parameters: testlist=sanity env=ONLY=413h,ONLY_REPEAT=100
Fixes: 38c4c538f5 ("LU-15216 lmv: improve MDT QOS space balance")
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Ia1d0b5b1a027cf14236f93ae34b5cf4929e76d23
Reviewed-on: https://review.whamcloud.com/47578
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15846 tests: don't use comma-separated debug flags

To avoid test interop issues between 2.15 clients and 2.12/2.14
servers, don't use comma-separated debug flags in sanity-quota.sh
quota_init() and quota_fini().

Test-Parameters: trivial testlist=sanity-quota.sh env=ONLY=0 serverversion=2.14.0
Fixes: 6b6fde1026 ("LU-13055 libcfs: allow comma-separated masks")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ifca39054d14292bca8bcff9b8e03ae58fd5cc3a8
Reviewed-on: https://review.whamcloud.com/47308
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14651 ldiskfs: add 5.11 kernel support

The Ubuntu 20.04.3 LTS moved to the 5.11 kernel. Support for this
kernel is a small step from the 5.10 kernel support. This patch
adds these small changes to support ldiskfs.

Test-Parameters: trivial
Change-Id: I3055736658b628fe79a6a9fc20ac01e7e1597630
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/47900
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15821 ldlm: Prioritize blocking callbacks

The current code places bl_ast lock callbacks at the end of
the global BL callback queue.  This is bad because it
causes urgent requests from the server to wait behind
non-urgent cleanup tasks to keep lru_size at the right
level.

This can lead to evictions if there is a large queue of
items in the global queue so the callback is not serviced
in a timely manner.

Put bl_ast callbacks on the priority queue so they do not
wait behind the background traffic.

Add some additional debug in this area.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Ic6eb65819a4a93e9d30e807d386ca18380b30c7d
Reviewed-on: https://review.whamcloud.com/47215
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15850 llite: pass dmv inherit depth instead of dir depth

In directory creation, once it's ancestor has default LMV, pass
the inherit depth, otherwise pass the directory depth to ROOT.

This depth will be used in QoS allocation.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Id480f32c1718e9f62314c2dfe8905be5db94d1f2
Reviewed-on: https://review.whamcloud.com/47577
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-16010 kernel: kernel update RHEL8.6 [4.18.0-372.16.1.el8_6]

Update RHEL8.6 kernel to 4.18.0-372.16.1.el8_6.

Test-Parameters: trivial fstype=ldiskfs \
clientdistro=el8.6 serverdistro=el8.6 testlist=sanity

Test-Parameters: trivial fstype=zfs \
clientdistro=el8.6 serverdistro=el8.6 testlist=sanity

Change-Id: I08db577f31a1d686b88804384a05d5b418e634d5
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47948
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15910 llite: use max default EA size to get default LMV

Subdir mount will fetch ROOT default LMV and set it, but the default
EA size cl_default_mds_easize may not be set for MDT0 yet, because
it's updated upon getattr/enqueue, and if subdir mount is not on MDT0,
it may not be initialized yet. Use max EA size to fetch default
layout in ll_dir_get_default_layout().

Fixes: a162e24d2d ("LU-15910 llite: enforce ROOT default on subdir mount")
Fixes: 716ac65ef6 ("LU-15910 tests: skip sanity/413g for SSK")
Test-Parameters: env=SHARED_KEY=true,ONLY="413g" testlist=sanity mdscount=1 mdtcount=2
Test-Parameters: env=SHARED_KEY=true,ONLY="413g" testlist=sanity mdscount=1 mdtcount=2
Test-Parameters: env=SHARED_KEY=true,ONLY="413g" testlist=sanity mdscount=1 mdtcount=2
Test-Parameters: env=SHARED_KEY=true,ONLY="413g" testlist=sanity mdscount=1 mdtcount=2
Test-Parameters: env=SHARED_KEY=true,ONLY="413g" testlist=sanity mdscount=1 mdtcount=2
Test-Parameters: env=SHARED_KEY=true,ONLY="413b 413g" testlist=sanity mdscount=1 mdtcount=2
Test-Parameters: env=SHARED_KEY=true,ONLY="413b 413g" testlist=sanity mdscount=1 mdtcount=2
Test-Parameters: env=SHARED_KEY=true,ONLY="413b 413g" testlist=sanity mdscount=1 mdtcount=2
Test-Parameters: env=SHARED_KEY=true,ONLY="413b 413g" testlist=sanity mdscount=1 mdtcount=2
Test-Parameters: env=SHARED_KEY=true,ONLY="413b 413g" testlist=sanity mdscount=1 mdtcount=2
Test-Parameters: env=SHARED_KEY=true,ONLY="413b 413c 413g" testlist=sanity mdscount=1 mdtcount=2
Test-Parameters: env=SHARED_KEY=true,ONLY="413b 413c 413g" testlist=sanity mdscount=1 mdtcount=2
Test-Parameters: env=SHARED_KEY=true,ONLY="413b 413c 413g" testlist=sanity mdscount=1 mdtcount=2
Test-Parameters: env=SHARED_KEY=true,ONLY="413b 413c 413g" testlist=sanity mdscount=1 mdtcount=2
Test-Parameters: env=SHARED_KEY=true,ONLY="413b 413c 413g" testlist=sanity mdscount=1 mdtcount=2
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I3c762cd371a80c2bea12d7fdbc16c6b14b3214e6
Reviewed-on: https://review.whamcloud.com/47937
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12189 ec: code to add support for M to N parity

This code adds basic functionality for calculating N parities
for M data units. This allows much more than just working with
raid6 calculations. The code is derived from the Intel isa-l
userland library. Keep the code in an separate module for easy
merger upstream at a latter time.

Test-Parameters: trivial
Change-Id: Ie0bb5af2514c213db40de33139e03e16f9605ce8
Signed-off-by: James Simmons <jsimmons@infradead.org>
Signed-off-by: Adam Disney <disneyaw@ornl.gov>
Reviewed-on: https://review.whamcloud.com/47628
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15851 lnet: Adjust niov checks for large MD

An LNet user can allocate a large contiguous MD. That MD can have >
LNET_MAX_IOV pages which causes some LNDs to assert on either niov
argument passed to lnd_recv() or the value stored in
lnet_msg::msg_niov. This is true even in cases where the actual
transfer size is <= LNET_MTU and will not exceed limits in the LNDs.

Adjust ksocklnd_send()/ksocklnd_recv() to assert on the return value
of lnet_extract_kiov().

Remove the assert on msg_niov (payload_niov) from kiblnd_send().
kiblnd_setup_rd_kiov() will already fail if we exceed ko2iblnd's
available scatter gather entries.

HPE-bug-id: LUS-10878
Test-Parameters: trivial
Fixes: 857f11169f ("LU-13004 lnet: always put a page list into struct lnet_libmd")
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: Iaa851d90f735d04e5167bb9c07235625759245b2
Reviewed-on: https://review.whamcloud.com/47319
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-16036 test: make sanity-lfsck 15d more robust

Now migrating directory LFSCK is not fully supported, thus accessing
such directory may fail. To make sanity-lfsck 15d more robust,
reformat servers after test.

Test-Parameters: trivial mdscount=2 mdtcount=4 testlist=sanity-lfsck env=ONLY=15d,ONLY_REPEAT=100
Fixes: 54a2d4662b58 ("LU-15868 lfsck: don't crash upon dir migration failure")
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Ie613737ab3e8d294e1f9e5709ceb35baa75790ad
Reviewed-on: https://review.whamcloud.com/48048
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-16023 tests: sanity-quota/8 should return success

sanity-quota/8 should return success explicitly

Test-Parameters: trivial testlist=sanity-quota
Fixes: bc69a8d058 ("LU-8621 utils: cmd help to stdout or short cmd error")
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Id30bfd3e0bafedb6516471accbc0519cc640d2bd
Reviewed-on: https://review.whamcloud.com/47966
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15282 tests: relax test_51d thresholds somewhat

Added combinations for sanity.sh test_51d are failing some fraction
of the time.  Relax thresholds somewhat to avoid spurious failures,
while keeping added configs to detect major regressions.

Test-Parameters: trivial
Fixes: fd5c915eff ("LU-15282 tests: improve sanity test_51d coverage")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I2d28f77fa377a1eb96a00cf35827b9ebc5af806b
Reviewed-on: https://review.whamcloud.com/47978
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15983 lnet: Define KFILND network type

Define the KFILND network type. This reserves the network type number
for future implementation and allows creation of kfi peers and
adding routes to kfi peers.

Test-Parameters: trivial testlist=sanity-lnet
HPE-bug-id: LUS-11060
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I9111645f1290c8af4937d1b2689a068df81922a4
Reviewed-on: https://review.whamcloud.com/47830
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15850 mdt: pack default LMV in open reply

Add flag MDS_OPEN_DEFAULT_LMV to indicate that default LMV should be
packed in open reply, otherwise if open fetches LOOKUP lock, client
won't know directory has default LMV, and in subdir creation default
LMV won't take effect.

Test-Parameters: clientversion=2.14 testlist=sanity mdtcount=4 mdscount=2 env=SANITY_EXCEPT="39l 134b 150b 160g 205a 208 230e 230p 270a 300g 807"
Test-Parameters: serverversion=2.14 testlist=sanity mdtcount=4 mdscount=2 env=SANITY_EXCEPT="65n 247f"
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: If2300ca39f406169eff9eab8f973ca1c2bfc8202
Reviewed-on: https://review.whamcloud.com/47576
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14472 quota: skip non-exist or inact tgt for lfs_quota

The nonexistent or inactive targets (MDC or OSC) should be skipped
for "lfs quota".

Change-Id: I25eece413715e4e05dd94ccbfd101220da7477f9
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/41771
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Feng, Lei <flei@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-11407 tgt: cleanup job_stats output printing

Escape non-printable and other special characters in the JobID
name, which may be passed from the client environment, to avoid
breaking YAML format parsing.  We can't use the kernel "%*pE"
escape format, because that doesn't have any option to escape
printable characters like quotes or regular spaces.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: If6a0bc4276fae03305f94e8c85d8f109913ebbe5
Reviewed-on: https://review.whamcloud.com/37764
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Ben Evans <beevans@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-6612 utils: strengthen llog_reader vs wrong format/header

The following snippet shows that llog_reader can be puzzled due to
an invalid 0 for the number of records when parsing an expected
LLOG file header :
root# dd if=/dev/zero bs=4096 count=1 of=/tmp/zeroes
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.000263962 s, 15.5 MB/s
root# llog_reader /tmp/zeroes
Memory Alloc for recs_buf error.
Could not pack buffer; rc=-12

Test-Parameters: trivial testlist=sanity,sanity-hsm
Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: I12be79e6c6a5da384a5fd81878a76a7ea8aa5834
Reviewed-on: https://review.whamcloud.com/15654
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15984 o2iblnd: debug message is missing a newline

Add missing newline to one of the debug messages in
kiblnd_pool_alloc_node.

Test-Parameters: trivial
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: I541622322ea6166892270dbfd1567cc64f8c314c
Reviewed-on: https://review.whamcloud.com/47933
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15938 lod: prevent endless retry in recovery thread

- abort lod_sub_recovery_thread() by obd_abort_recov_mdt in
  addition to obd_abort_recovery
- handle 'short llog' situation gracefully, when remote llog
  is shorter than local copy header expects, trust remote llog
  data and consider llog processing as finished
- on other errors during remote llog read, set obd_abort_recov_mdt
  but not obd_abort_recovery in attempt to skip MDT-MDT recovery
  only and continue with client recovery while possible
- fix parsing problem with 'abort_recov' and 'abort_recov_mdt' in
  lmd_parse() causing no MDT recovery abort but client recovery
  abort always. Allow also 'abort_recovery_mdt' mount option name

The original case with endless retry is caused by such de-sync
between local llog structures and remote llog. The local llog
header says there is record with some ID, so recovery thread
is trying to get that record from remote llog. Meanwhile there
is no such record on remote server, so it reads whole llog and
return it back properly but llog processing consider that as
incomplete llog due to network issues and retry endlessly.

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: Ib127fd0d1abd5289d90c7b4b3ca74ab6fc78bc71
Reviewed-on: https://review.whamcloud.com/47698
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10994 clio: Remove cl_2queue_add wrapper

Remove the wrapper function cl_2queue_add() and replace all its calls in
different files with the function it wrapped. Also, comments are added
wherever necessary to make the working of function clear. Prototype of
the function is also removed from the header file as it is no longer
needed.

Linux-commit: 53f1a12768a55e53b2c40e00a8804b1edfa739b3

Change-Id: Ic746c45e3dda9fdf3f1d2f8c6204d80fec5c058f
Signed-off-by: Shivani Bhardwaj <shivanib134@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-on: https://review.whamcloud.com/47651
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15925 lnet: add debug messages for IB

If net debug is enabled, information about connection, when
tx status is ECONNABORTED, is collected (only for IB).

Test-Parameters: trivial
Signed-off-by: Cyril Bordage <cbordage@whamcloud.com>
Change-Id: I44a33703931630b85cc0e847e2a038217b7967c6
Reviewed-on: https://review.whamcloud.com/47583
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15902 obdclass: dt_try_as_dir() check dir exists

If an object is not directory, but dt_lookup() is called on it, it
may crash because .do_lookup is NULL for non-directory file.

Add argument to check object existence and type in dt_try_as_dir(),
and for object to create, skip this check.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I51df0cbb5a4e7abca370ee27dac678f995b76159
Reviewed-on: https://review.whamcloud.com/47483
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15880 quota: fix issues in reserving quota

Calling "chgrp" with unprivileged user will reserve quota space
before changing the GID of the file, and the reserved quota space
will be freed after its transaction is committed. there are some
issues in the current implementation,
1, the reserved quota isn't freed in case of error in "mdd_attr_set"
   and "tgt_cb_last_committed".
2, during freeing the reserved quota, the quota space to free is
   set as the same parameter as reserving the quota, which could
   be wrong, for instance, the reserving quota space will be 0 if
   the corresponding quota ID isn't enforces, but the call will
   return without error.

Like the "qsd_op_begin/qsd_op_end",The patch also adds reference to
the lquota_entry gotten during reserving quota and release it during
freeing the reserved quota to prevent potential issue.

Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Change-Id: I098cde7d5e89fe8b9eaab0ae4bc285a4ac6c2281
Reviewed-on: https://review.whamcloud.com/47425
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15991 kernel: kernel update RHEL7.9 [3.10.0-1160.71.1.el7]

Update RHEL7.9 kernel to 3.10.0-1160.71.1.el7.

Test-Parameters: trivial clientdistro=el7.9 serverdistro=el7.9

Change-Id: I89215145ea8da2925e5c8c01cdf963ba8a087877
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47865
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-16000 utils: align updatelog parameters in llog_reader

Parameters in update log records are aligned on 64bits. llog_reader
do not aligned these parameters: if a parameters size is not mutiple
of 8, the next parameter size will be read incorrectly.

Test-Parameters: trivial
Fixes: 9962d6f ("LU-14617 utils: llog_reader updatelog support")
Signed-off-by: Etienne AUJAMES <eaujames@ddn.com>
Change-Id: I6871614ab4ea79d59c3c3b4644b377de395bad56
Reviewed-on: https://review.whamcloud.com/47913
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15993 ofd: don't leak pages if nodemap fails

ofd_commitrw() shouldn't exit w/o calling ofd_commitrw_write(),
otherwise the pages taken in ofd_preprw() are leaked.

same in mdt_obd_commitrw()

Fixes: bbfdc7c167 ("LU-14739 quota: fix quota with root squash enabled")

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Icd60c7ab80c5a7b65603d7da0d2e83872dc6b97f
Reviewed-on: https://review.whamcloud.com/47873
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-8582 target: send error reply on wrong opcode

Unknown opcode does not necessarily means insane client. A new client
might send RPCs with new opcodes to an old server. The client might
desperately stuck there waiting for a reply. So, send an error back
when RPC has a wrong opcode.

This patch returns the EOPNOTSUPP to client instead of block. ENOTSUPP
is not used here since strerror() does not understand ENOTSUPP.

OBD_FAIL_OST_OPCODE=0x253 is added for fault injection test of opcode.
To test whether an invalid opcode is handled properly on OST, use the
following command:

lctl set_param fail_val=${opcode} fail_loc=0x253

Change-Id: I46ca62bc532b92368e06a4f883b102c7186c453c
Signed-off-by: Li Xi <lixi@ddn.com>
Reviewed-on: https://review.whamcloud.com/47761
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15888 build: Debian dkms-debs requires ed and libkeyutils

dkms install/build needs dependencies on libmount-dev,
libkeyutils1, and libkeyutils-dev

Debian does not install the 'ed' package by default.
Without the 'ed' package the version is not correctly added
to the changelog and parsed to the package names.

Debian does not have linux-image or linux-headers psuedo
packages so require the arch specific ones, ex:
   linux-image | linux-image-amd64 | linux-image-arm64
and:
   linux-headers | linux-headers-amd64 | linux-headers-arm64
respectively.

o2ib fails to find Debian in-kernel Module.symvers and
should check $LINUX_OBJ/Module.symvers before failing.

HPE-bug-id: LUS-10984
Test-Parameters: trivial
Fixes: 85a6eebeca1 ("LU-15652 build: On Debian detect -common kernel headers")
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I82e2689f3af4b9ce106ee3ab6b4109d2709c8872
Reviewed-on: https://review.whamcloud.com/47455
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15886 lfsck: remove unreasonable assertions

Remove unreasonable assertions in LFSCK code:
* lfsck->li_obj_dir and lfsck->li_lmv may be NULL if object wan't
  initialized successfully.
* orphan objects under ldiskfs /lost+found may not exist.
* object may not be directory in lfsck_verify_lpf()->
  lfsck_verify_linkea().
* for corner case
  (leh->leh_reccount == 0 && leh->leh_overflow_time != 0),
  LASSERT(ldata->ld_leh->leh_reccount > 0) will be triggerred in
  lfsck_namespace_linkea_clear_overflow(), remove this assertion,
  and this corner case can be handled correctly in current lfsck
  code.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: If114c7882a2c083e83fcfac5981eddfa526d1426
Reviewed-on: https://review.whamcloud.com/47447
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15868 lfsck: don't crash upon dir migration failure

LFSCK against directories that were migrated, but failed may crash,
it's because lost+found directory may not be initialized correctly,
and this error is skipped on purpose, add check in code that
dereference it.

lfsck_verify_lpf() may dereference NULL "child2".

lmv_name_to_stripe_index() should support stripe LMV, which is used
by LFSCK to verify name hash.

Add OBD_FAIL_OUT_EIO to simulate sub transaction failure.

Add sanity-lfsck 15d to verify LFSCK won't crash upon directory
migration failure.

Update sanity-lfsck 4 and 5 to start mds1 with OI scrub enabled, and
wait for mds1 OI scrub finish, otherwise LFSCK may fail to verify
lost+found later.

Test-Parameters: mdscount=2 mdtcount=4 testlist=sanity-lfsck \
env=ONLY=15d,ONLY_REPEAT=100
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I1b1872da2b4ef8f7403effc4d1d3e298c6a0b7e6
Reviewed-on: https://review.whamcloud.com/47381
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15626 tests: Fix "error" reported by shellcheck for recovery-random-scale

This patch fixes "error" issues reported by shellcheck
for file lustre/tests/recovery-random-scale.sh. This
patch also moves spaces to tabs.

Test-Parameters: trivial clientcount=6 mdtcount=2 mdscount=2 osscount=2 austeroptions=-R failover=true iscsi=1 env=FAILOVER_PERIOD=180,SLOW=no testlist=recovery-random-scale
Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Change-Id: I9e7c6fcd726d5fc2141a7ced73ea78ee6a43ec22
Reviewed-on: https://review.whamcloud.com/46866
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15626 tests: Fix "error" reported by shellcheck for recovery-mds-scale

This patch fixes "error" issues reported by shellcheck
for file lustre/tests/recovery-mds-scale.sh. This patch
also moves spaces to tabs.

Test-Parameters: trivial clientcount=6 mdtcount=2 mdscount=2 osscount=2 austeroptions=-R failover=true iscsi=1 env=FAILOVER_PERIOD=180,DURATION=600,SLOW=no testlist=recovery-mds-scale
Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Change-Id: I6c098809835950e1f781e04a6898895592407948
Reviewed-on: https://review.whamcloud.com/46865
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15132 hsm: Protect against parallel HSM restore requests

Multiple parallel accesses (read/write) to the same released file
could cause multiple HSM restore requests to be sent.
On the MDT side, each restore request waits the first one to complete
before grabbing the MDS_INODELOCK_LAYOUT LCK_EX and registering the
llog record.

This could cause several MDT threads to hang for the same restore
request sent in parallel. In the worst case, all MDT threads can
hang and the MDS is not longer able to handle requests.

This patch checks if an HSM restore handle exists before taking the
lock.

Test-Parameters: testlist=sanity-hsm,sanity-hsm
Test-Parameters: testlist=sanity-hsm env=ONLY=12s,ONLY_REPEAT=50
Signed-off-by: Etienne AUJAMES <eaujames@ddn.com>
Change-Id: I9584edc2c7411aa41b2e318e55f57c117d1c3dfb
Reviewed-on: https://review.whamcloud.com/45367
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Nikitas Angelinas <nikitas.angelinas@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15913 mdt: disable parallel rename for striped dirs

Parallel rename should not be done within striped directories to
avoid remote updates.  These are like cross-directory renames.

Add tunables for parallel directory rename in case of problems.
These can be configured separately for files and directories.

    mdt.*.enable_parallel_rename_dir
    mdt.*.enable_parallel_rename_file

Fixes: 90979ab390 ("LU-12125 mds: allow parallel directory rename")
Fixes: d76cc65d5d ("LU-12125 mds: allow parallel regular file rename")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I384976cd1c9f401169336ee7a479ba0e3dd9f4ee
Reviewed-on: https://review.whamcloud.com/47593
Reviewed-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15521 spec: fix more bare words error with rpm 4.16

This patch fixes more bare words errors and extra tokens
warnings with rpm 4.16.

Test-Parameters: trivial

Change-Id: Ic40b5763d1cb362d5aa77b06e9a5768b2abbc708
Fixes: 597a6bf9e085 ("LU-15521 spec: fix bare words error with rpm 4.16")
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47833
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15653 client: able to cleanup devices manually

Using 'lctl cleanup/detach' could be needed in situations
with unclean umount. Meanwhile that doesn't work now for
LMV and also could cause panic after all

Patch restores ability to cleanup/detach client devices
manually.
- debugfs and lprocfs cleanup in lmv_precleanup() are moved
  lmv_cleanup() to be not cleared too early. This prevents
  hang on 'lctl cleanup' for LMV device
- test 172 is added in sanity. It skips device cleanup during
  normal umount, keeping device alive without client mount
  then manually cleanups/detaches them
- prevent negative lov_connections in lov_disconnect() and
  handle it gracefully
- remove obd_cleanup_client_import() in mdc_precleanup(),
  it is called already inside osc_precleanup_common()

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I8a3868fabd1d805e827d04852d1614a3fe57ce35
Reviewed-on: https://review.whamcloud.com/46859
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15894 ofd: revert range locking in ofd

After commit 301d76a711 (LU-14876), range locking is no longer needed
in ofd, because 301d76a711 itself prevents the original data corruption
fixed by range locking. At the same time, range locking in ofd adds
unnecessary overhead, we can even see serialization under specific load.

This patch reverts range locking but keeps recovery-small test 148
to test the original corruption scenario case.

Change-Id: Ic795bcfb1e249c4927f66b6bad456f5511819861
Signed-off-by: Andrew Perepechko <andrew.perepechko@hpe.com>
HPE-bug-id: LUS-9890
Fixes: 35679a730 ("LU-10958 ofd: data corruption due to RPC reordering")
Reviewed-on: https://review.whamcloud.com/47466
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15759 libcfs: debugfs file_operation should have an owner

If debugfs a file is open when unloading the libcfs/lnet module, it
produces a kernel Oops (debugfs file_operations callbacks no longer
exist).

Crash generated with routerstat (/sys/kernel/debug/lnet/stats):
[ 1449.750396] IP: [<ffffffffab24e093>] SyS_lseek+0x83/0x100
[ 1449.750412] PGD 9fa14067 PUD 9fa16067 PMD d4e5d067 PTE 0
[ 1449.750428] Oops: 0000 [#1] SMP
[ 1449.750883]  [<ffffffffab7aaf92>] system_call_fastpath+0x25/0x2a
[ 1449.750897]  [<ffffffffab7aaed5>] ?
system_call_after_swapgs+0xa2/0x13a

This patch adds an owner to debugfs file_operation for libcfs and
lnet_router entries (/sys/kernel/debug/lnet/*).

The following behavior is expected:
$ modprobe lustre
$ routerstat 10 > /dev/null &
$ lustre_rmmod
rmmod: ERROR: Module lnet is in use
Can't read statfile (ENODEV)
[1]+  Exit 1                  routerstat 10 > /dev/null
$ lustre_rmmod

Note that the allocated 'struct file_operations' cannot be freed until
the module_exit() function is called, as files could still be open
until then.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ia0920313e0c2a4b6cdc875fed08221e174a12a73
Reviewed-on: https://review.whamcloud.com/47335
Reviewed-by: Etienne AUJAMES <eaujames@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15779 ofd: don't hold read lock over bulk

as this can block all operations on OST:

1) ofd_preprw_read() takes a shared object lock and initiates BULK
2) OUT needs an exclusive object lock on the same object
3) ofd_commitrw_write() starts transaction and now has to wait
   for OUT to get and release that exclusive object lock (step 2)
4) number of threads can get stuck waiting for ofd_commit_write()
   to stop it's transaction

this patch drops a shared object lock before BULK transfer.
at the moment it's not clear how such read would race with
object removal on ZFS - this should be investigated.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I131493abd90283e9ca897f904e00c25d26e3d8d3
Reviewed-on: https://review.whamcloud.com/47126
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15727 lod: honor append_pool with default composite layouts

In lod_get_default_lov_striping(), correct the handling of composite
default layouts in the case where append_stripe_count is nonzero.
Align the names of the append members of struct dt_allocation_hint
with the mdd params. Remove the unused dah_mode member of struct
dt_allocation_hint.

Add sanity test_27U() to verify.

Fixes: e2ac6e1eaa ("LU-9341 lod: Add special O_APPEND striping")
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I66b426d24d6476fb483397f290229983f3da4be5
Reviewed-on: https://review.whamcloud.com/47014
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>