LU-9679 osc: discard oe_intree

An rbnode knows if it is in the tree or not, using RB_EMPTY_NODE().
There is no need for an extra flag.

Linux-Commit: a25a82301800 ("lustre: osc_cache: discard oe_intree")

Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: Iddd4ef623d96c1cf2e0789343c02ecb095f7b6a1
Reviewed-on: https://review.whamcloud.com/37600
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12955 lnet: fix routed final dest selection

The final hop gateway should rely on the message initiator to
select the final destination NID. IE if gateway is handing off
the message to the final destination, always use the NID provided
in the message.

If the initiator has discovery off, then it'll always pick the
same destination NID.

Otherwise if the initiator has discovery on && there exists a route to
each of the interfaces on the destination, the initiator will select
these interfaces in round robin

This will work for both single-hop and multi-hop cases. The logic
on the routers will be:

if destination NID in the message is local then
   send the message to the destination NID
else if destination NID is remote && there exists a route
   if router is MR then
      select NI according to selection criteria

Test-parameters: trivial

Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: Ifd8dd65fe7695d69609ac4c1af5102429d8aa379
Reviewed-on: https://review.whamcloud.com/36916
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-6174 osd-ldiskfs: perform proper division

The used value calculated for osd_inode_iteration() and
sf_items_updated_prior are both 64 bit values. The code
wrongly using do_div() which could truncate sf_items_updated_prior
to a 32 bit value. Use div64_u64() instead.

Change-Id: I3d76e399e6cf50460d98b1174de83263d9c2d3cc
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/36750
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>

LU-13156 tests: skip sanity-selinux test_21a+21b

These tests are failing most of the time, but are the last failures
for both review-dne-selinux and review-dne-selinux-ssk sessions of
this script.  Add them to ALWAYS_EXCEPT so that these sessions can be
enforced to prevent further regressions.  They can be fixed afterward.

Test-Parameters: trivial testlist=sanity-selinux
Test-Parameters: testlist=sanity-selinux env=SHARED_KEY=true
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ib51e3749b1e262336ae1db53278909dd78d2be63
Reviewed-on: https://review.whamcloud.com/37899
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13116 tests: properly clean keyring in sanity-sec test_30

sanity-sec test_30 makes use of a bogus key to test Lustre behaviour
in case of invalid shared key. This bogus key must be cleaned and
removed properly so that it does not interfere with subsequent tests.

Test-Parameters: trivial envdefinitions=SHARED_KEY=true testlist=sanity-sec
Test-Parameters: envdefinitions=SHARED_KEY=true testlist=sanity-sec
Test-Parameters: envdefinitions=SHARED_KEY=true testlist=sanity-sec
Test-Parameters: envdefinitions=SHARED_KEY=true testlist=sanity-sec
Test-Parameters: envdefinitions=SHARED_KEY=true testlist=sanity-sec
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I32d6b70948b2fedfb60f2750f56ef772a404d08c
Reviewed-on: https://review.whamcloud.com/37151
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13116 ptlrpc: return error for conn with NULL export

In import_select_connection(), instead of asserting when export on
chosen connection is NULL, return an error.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I60dcb26693f2ba177d612ded7efff9721febf601
Reviewed-on: https://review.whamcloud.com/37251
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13216 ptlrpc: sptlrpc_req_refresh_ctx's timeout semantic

sptlrpc_req_refresh_ctx() is only called with two possible timeouts
in the current code: either we do not wait at all, or we wait
indefinitely.
So change semantic of sptlrpc_req_refresh_ctx()'s timeout parameter,
to be more consistent with wait* primitives. timeout == 0 means
'do not wait at all', and timeout == MAX_SCHEDULE_TIMEOUT means
'wait indefinitely'. Other timeout values are unsupported.

Consequently, remove code that handles the case of a non zero and
non infinite timeout. And remove the unsed ctx_refresh_timeout()
callback.
It solves a problem in the current code, because in case of indefinite
wait, sptlrpc_req_refresh_ctx() is not supposed to try to refresh the
request, so it must not call the ctx_refresh_timeout() callback.

Fixes: c1fad6a9a5 ("LU-10467 ptlrpc: convert waiting in sptlrpc_req_refresh_ctx()")
Test-Parameters: trivial envdefinitions=SHARED_KEY=true testlist=sanity-sec
Test-Parameters: envdefinitions=SHARED_KEY=true testlist=sanity-sec
Test-Parameters: envdefinitions=SHARED_KEY=true testlist=sanity-sec
Test-Parameters: envdefinitions=SHARED_KEY=true testlist=sanity-sec
Test-Parameters: envdefinitions=SHARED_KEY=true testlist=sanity-sec
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I1b20457c86863d57f97f4d998f7c5fa8ec928b31
Reviewed-on: https://review.whamcloud.com/37473
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12992 gss: retry in case of short computed shared key

Lustre uses OpenSSL's DH_compute_key() to compute shared secret key.
There is around 1 chance out of 256 that the returned shared key
is shorter than expected.
https://www.qacafe.com/articles/router-vpn-implementation-pitfalls/
If the returned key is short by 1 or 2 bytes, we prepend it with 0s.
Otherwise, we retry twice as it should finally be the expected length.

Test-Parameters: envdefinitions=SHARED_KEY=true testlist=sanity,recovery-small
Test-Parameters: envdefinitions=SHARED_KEY=true testlist=sanity,recovery-small
Test-Parameters: envdefinitions=SHARED_KEY=true testlist=sanity,recovery-small
Test-Parameters: envdefinitions=SHARED_KEY=true testlist=sanity,recovery-small
Test-Parameters: envdefinitions=SHARED_KEY=true testlist=sanity,recovery-small
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I519194ef52539ca9a6305120fbf00b60ca985b1f
Reviewed-on: https://review.whamcloud.com/37064
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jeremy Filizetti <jeremy.filizetti@gmail.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13116 mgc: do not lose sptlrpc config lock

Multiple targets on the same node can make use of the same lock on
sptlrpc config log. So if a target is being stopped, sptlrpc
config log should not be directly marked as stopping.

Fixes: 0ad54d5977 ("LU-11185 mgc: config lock leak")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I09c2b0c276daadb66721df88c33f734ebba86114
Reviewed-on: https://review.whamcloud.com/37344
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexander Boyko <c17825@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-4198 clio: Remove pl_owner

The pl_owner field is only used for invariant testing, but standard
tests to enable that testing, so it doesn't really serve any purpose.
If you do enable it, you will finx the invariant in cl_page_list_del()
fails.

As async-io allows completion to happen in a different thread to
submission, this might not be a surprise.

So let's remove the field.

Sample stack trace:
LustreError: 2813:0:(cl_io.c:823:cl_page_list_del()) ASSERTION(
    plist->pl_owner == get_current() ) failed:
LustreError: 2813:0:(cl_io.c:823:cl_page_list_del()) LBUG
Pid: 2813, comm: ptlrpcd_01_01 3.10.0-7.7-debug #1 SMP Fri Oct 25
    00:19:37 EDT 2019
Call Trace:
 [<ffffffffa017de4c>] libcfs_call_trace+0x8c/0xc0 [libcfs]
 [<ffffffffa017defc>] lbug_with_loc+0x4c/0xa0 [libcfs]
 [<ffffffffa0318e21>] cl_page_list_del+0x1c1/0x200 [obdclass]
 [<ffffffffa031aeee>] cl_aio_end+0x5e/0x180 [obdclass]
 [<ffffffffa031a531>] cl_sync_io_note+0xa1/0x280 [obdclass]
 [<ffffffffa0315bf0>] cl_page_completion+0xd0/0x410 [obdclass]
 [<ffffffffa0822d79>] osc_ap_completion.isra.29+0x169/0x4c0 [osc]
 [<ffffffffa082a1c2>] osc_extent_finish+0x3a2/0xb10 [osc]
 [<ffffffffa080eb53>] brw_interpret+0x1f3/0xd60 [osc]
 [<ffffffffa05debc0>] ptlrpc_check_set+0x520/0x1ec0 [ptlrpc]
 [<ffffffffa060d0fb>] ptlrpcd_check+0x48b/0x580 [ptlrpc]
 [<ffffffffa060d4f3>] ptlrpcd+0x303/0x6c0 [ptlrpc]
 [<ffffffff810b8254>] kthread+0xe4/0xf0
 [<ffffffff817e5ddd>] ret_from_fork_nospec_begin+0x7/0x21
 [<ffffffffffffffff>] 0xffffffffffffffff

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Icc3af742140ddc5dd31426154d578da0ba4300fe
Reviewed-on: https://review.whamcloud.com/37824
Tested-by: Maloo <maloo@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Patrick Farrell <farr0186@gmail.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9859 libcfs: prepare for switch to container_of_safe()

Upstream Linux uses container_of_safe() rather than
container_of0().

So provide container_of_safe() for all kernels, and use spelling.txt
to suggest new code uses it.  We can then start switching code
over gradually.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ie938b76d569de6bb10b51fb0b12d79fcd0f43e7e
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Reviewed-on: https://review.whamcloud.com/37736
Tested-by: Maloo <maloo@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13096 llite: serialize max_cached_mb write operation

It is possible that two threads try to change max_cached_mb concurrently,
if they both try to reclaim LRU pages, it will be possible
that one thread finish, but another thread loop forever to reclaim
enough LRU pages.

Morever, this operations try to change two value, they should be
protected together to guarantee atomic, the whole operations could
be slow, this patch introduce another mutex lock to serialize operation.

Patch also fix to avoid reclaim too many LRU pages with large memory.

Change-Id: I1e88e37607f9e8db2ba67f1f7e5670f25c136465
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/37710
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-13273 tests: run_ior() parameters order fix

IOR requires particular options to be specified
before script file option. Otherwise those options
make no effect.

After script file processing any options which
change initialTestParams values have no effect
because test set is already created based on
initialTestParams initialized and modified with
preceding options.

To have initialTestParams->testFileName equal to
<testFile> the parameters order should be:
   -o <testFile> -f <scriptFile>
With the order
   -f <scriptFile> -o <testFile>
initialTestParams->testFileName is equal to default
value "testFile" set in init_IOR_Param_t.

Fixes: 2d7419478e79 (LU-11078 tests: customise run_ior())
Test-Parameters: trivial
Test-Parameters: testlist=parallel-scale envdefinitions=ONLY=iorssf
Cray-bug-id: LUS-7734
Signed-off-by: Elena Gryaznova <elena.gryaznova@hpe.com>
Reviewed-by: Vladimir Saveliev <vladimir.saveliev@hpe.com>
Reviewed-by: Vitaly Fertman <vitaly.fertman@hpe.com>
Change-Id: I16a797466355c69e655259aaf8d9a3ae69d4a31d
Reviewed-on: https://review.whamcloud.com/37651
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Vladimir Saveliev <c17830@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13271 tests: skip 87, 115 tests if FILESET set

When FILESET is set the tests require Lustre root mounted
on $MOUNT are to be skipped.

Test-Parameters: trivial testlist=conf-sanity envdefinitions="FILESET=/subdir" envdefinitions=ONLY=87
Test-Parameters: testlist=conf-sanity envdefinitions="FILESET=/subdir" envdefinitions=ONLY=115
Signed-off-by: Elena Gryaznova <c17455@cray.com>
Reviewed-by: Vladimir Saveliev <vladimir.saveliev@hpe.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Change-Id: I8931d0419d3613caf53497c99ad8185dc2b56fad
Reviewed-on: https://review.whamcloud.com/37647
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Vladimir Saveliev <c17830@cray.com>

LU-13255 lnet: introduce wait_var_event_warning.

LNet has a recurring pattern of waiting for some variable to reach a
particular value, and generating a warning every second that it
hasn't.  In many cases the warning has a higher priority if the wait
has been for a power-of-2 seconds.

This patch embodies that pattern in a new macro
  wait_var_event_warning()

and uses wake_up_var() to cause the wait to complete as soon as
possible.

This patch does not include any change to gnilnd - that is left for
a separate patch.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I557a64713cbe379a566a775944f58ddf93dbd800
Reviewed-on: https://review.whamcloud.com/37593
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12214 build: add right kernel detect

add kabi package dependence and define to use it.

add ability to use an 'epoch' in kernel version
(confuse a kmodtool)

don't replace a requested kernel version with
kernel source provided
(remove an epoch)

Cray-bug-id: LUS-6033, LUS-7204
Test-parameters: trivial
Change-Id: Ia3dbca4520119ac7b6e83800b862b4f29f763796
Signed-off-by: Alexey Lyashkov <c17817@cray.com>
Reviewed-on: https://review.whamcloud.com/36428
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexander Boyko <c17825@cray.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12780 lod: don't use ptlrpc_thread for recovery thread

rather than ptlrpc_thread, use native kthreads functionality.

- This saves a memory allocation - and associated error handling
  and memory freeing - for a 'struct ptlrpc_thread'

- startup-up code is moved out of the thread to before the
  thread is started.  This make error handling clearer,
  though it does require putting a 'struct lu_env' in
  struct lod_recovery_data.

- As freeing lod_recovery_data and the env is most easily done
  in the thread, add a completion to ensure the thread actually
  starts.  Without this it is possible for a kthread_stop() to
  stop the thread before it calls the thread function at all.

- As the thread exits independently but is also waited for, we
  need a handshake around kthread_stop().  Both thread_exit and
  code that stops the thread use xchg() to store a NULL in the
  task pointer ltd_recovery_task or lod_child_recovery_thread.
  If the thread reads out a non-NULL pointer, it can exit because
  nothing will wait for it.
  If it gets NULL, it must wait for kthread_should_stop().  The waiter
  will call kthread_shop() immediately after the xchg, so the wait
  will be short.

  We use wait_var_event() for this wait even though there is no
  wake_up_var().  wait_var_event() is just a convenient macro that
  handles thread state change for us - the wait queue is irrelevant.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I95403fc38bac99f21a4353ec7814816b813184d0
Reviewed-on: https://review.whamcloud.com/36261
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12988 ldiskfs: port ext4-mballoc-prefetch.patch to RHEL 8.1

ext4: mballoc to prefetch groups ahead of scanning

Prefetching is done in 8 * flex_bg groups, so it should be 8 read-ahead
reads for a single allocating thread. At the end of allocation the
thread waits for read-ahead completion and initializes buddy information
so that read-aheads are not lost in case of memory pressure.

At cr=0 the number of prefetching IOs is limited per allocation context
to prevent a situation when mballoc loads thousands of bitmaps looking
for a perfect group and ignoring groups with good chunks.

Change-Id: I34e3e3dd604379fb65d711b620e8b30ae5b7df6e
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Lustre-bug-id: https://jira.whamcloud.com/browse/LU_12988
Reviewed-on: https://review.whamcloud.com/37811
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-12642 ofd: change last_id output in proc

Don't show IDIF 32:48 bits in sequence like below:
lctl get_param obdfilter.lustre-OST0002.last_id
0x100020001:4295098401
The sequence should be the same with seq on MDS:
lctl get_param osp.lustre-OST0002-osc-MDT0000.prealloc_last_seq
0x100020000
lctl get_param osp.lustre-OST0002-osc-MDT0000.prealloc_last_id
4295098401

Change-Id: Ie3b7db3d69305a26c06325869628e5b38196e5d7
Signed-off-by: Sergey Cheremencev <c17829@cray.com>
Cray-bug-id: LUS-6454
Reviewed-on: https://es-gerrit.dev.cray.com/155308
Reviewed-by: Alexander Boyko <c17825@cray.com>
Tested-by: Elena Gryaznova <c17455@cray.com>
Reviewed-by: Alexander Zarochentsev <c17826@cray.com>
Reviewed-on: https://review.whamcloud.com/35730
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-8221 osc: fix for lu_ref feature in osc_extent_put()

Trying to enable and use the USE_LU_REF/lu_ref feature, a previous
clio related changes have been found to silently (as feature
has not been tested since!) break it.

In osc_extent_put(), there is no need to remove an
implicit "handle" reference, by using LDLM_LOCK_PUT() macro, since
it has never been acquired before. This was introduced by
3f3a24d LU-3259 clio: cl_lock simplification

And the "osc_extent" reference must be deleted instead added.

This patch makes the necessary changes to fix all of these.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: I86599ff97310b16c9599b6a52a88693872ae6d30
Reviewed-on: https://review.whamcloud.com/20519
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9679 osc: convert while to for in get_write_extents()

This 'while' loop is really a 'for' loop in disguise.
So change it to 'for'.

Linux-Commit: d3ba9dc5cc66 ("lustre: osc_cache: convert while to for
in get_write_extents()")

Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: Ifd4295cbdad400d2c49d83e0c54ddd0e4c5ae548
Reviewed-on: https://review.whamcloud.com/37608
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <farr0186@gmail.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>

LU-9679 osc: simplify osc_extent_wait()

Taking a spinlock to check the current value of the state is
unnecessary.  The wake_up() and wait_event() calls have sufficient
barriers to ensure that the value will be seen and the wait will abort
properly.

In most cases, osc_extent_wait() is followed by osc_object_lock()
before any shared data is touched - in those cases there is no need
for osc_extent_wait() to wait for the spinlock to be released.

The one case where osc_object_lock() does not immediately follow is in
osc_cache_truncate_start().  The extra locking was introduced in a
patch which fixed a problem with truncation, so it is likely that this
is the call that was thought to be relevant.

In that case, following osc_extent_wait(), an extent that had been
detached from the per-object list (oe_link linkage) is worked on it
without any locking.

In this case the code is waiting for OES_TRUNC, so any changes that
happen after the osc_extent_state_set(ext, OES_TRUNC) and when the
lock is dropped, might not be seen by the woken code.  The only thing
changed is ->oe_trunc_pending, and the woken code doesn't look at
that.

The only remaining possible need for extra synchronization is if some
other value was changed before the wakeup and is needed after the
wait.  According to memory-barriers.txt, a barrier might be needed to
ensure that is visible.  Such a barrier is most clearly presented by
used smp_store_release() to set the state before wakeup, and
smp_load_acquire() to view it after waiting.

Also use a simple wake_up() instead of wake_up_all() - the latter is
only needed when exclusive waiting is being used.

Linux-Commit: 2a7e8a40cb11 ("lustre: osc: simplify osc_extent_wait()")

Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I0945902b55d598af263779787b9641a86e9476c3
Reviewed-on: https://review.whamcloud.com/37598
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <farr0186@gmail.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-6174 nrs: perform proper division

For NRS TBF both tr_nsecs and tr_rpc_rate are 64 bit values. The
code is wrongly using do_div() which could truncate tr_rpc_rate
to a 32 bit value. With current hardware setups the rpc rate is
less than the 4 billion limit of u32, currently I see 64K of rpcs
with a single IB port. Since this is the case we can change
tr_rpc_rate to a 32 bit value.

Change-Id: I62eabf3a8a30f5c9062aaf4c32fa878710a788c8
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/36749
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>

LU-13265 lnet: Modify push MD to have single threshold

Drop LNET_MD_MANAGE_REMOTE from the push target buffer and adjust
lnet_peer_push_event() handler accordingly.

Set the lazy flag on the LNET_RESERVED_PORTAL.

Set md.threshold to 1 on the MD for push target buffer. This forces
the buffer to unlink when there is any operation on it, thus
preventing any chance of concurrent access.

Refactor lnet_push_target_resize() into two functions:
 - lnet_push_target_post(): Posts the specified buffer
 - lnet_push_target_resize() allocates new ping buffer and calls
     lnet_push_target_post() to post it.

Add flag to lnet_ping_buffer struct to indicate if the buffer needs
to be posted. Flag is set by the push target event handler when it
has finished processing the buffer. The discovery thread checks this
flag and takes appropriate action.

Cray-bug-id: LUS-8461
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: If7f2fe26eb31bc2ad5e7023a809ef107ada9e99c
Reviewed-on: https://review.whamcloud.com/37677
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Olaf Weber <olaf.weber@hpe.com>
Reviewed-by: Stephen Champion <stephen.champion@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12811 ptlrpc: do some cleanups in swabber_dumper_helper()

This patch combines the process of dump/swab/dump calls for both
array and non-array cases in function swabber_dumper_helper(),
also does some code cleanups.

Signed-off-by: Emoly Liu <emoly@whamcloud.com>
Change-Id: Iccd487999ad998068b4fae6a614bad3b17737d07
Reviewed-on: https://review.whamcloud.com/36848
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>

LU-13300 ldiskfs: port patches to improve extent status shrink

We see serious extent status shrink problem on some customer sites,
the backtrace could be:

NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [kswapd0:106]
[<ffffffffae1958c2>] merge+0x62/0xc0
[<ffffffffc123fc50>] ? ldiskfs_init_inode_table+0x410/0x410 [ldiskfs]
[<ffffffffae1959bb>] list_sort+0x9b/0x250
[<ffffffffc124014e>] __ldiskfs_es_shrink+0x1ce/0x2a0 [ldiskfs]
[<ffffffffc12402d4>] ldiskfs_es_shrink+0xb4/0x130 [ldiskfs]
[<ffffffffadfcf805>] shrink_slab+0x175/0x340
[<ffffffffae03e1d7>] ? vmpressure+0x87/0x90
[<ffffffffadfd3538>] balance_pgdat+0x3a8/0x5e0

Backport following Linux upstrem commits since v3.18 to RHEL7:

Linux-commit: eb68d0e2fc5a4e5c06324ea5f485fccbae626d05
ext4: track extent status tree shrinker delay statictics

Linux-commit: 50460fe8c6d1d95b16427936e351f277a1c72d43
ext4: remove extent status procfs files if journal load

Linux-commit: edaa53cac8fd4b96ed4b8f96c4933158ff2dd337
ext4: change LRU to round-robin in extent status tree shrinker

Linux-commit: b0dea4c1651f3cdb6d17604fa473e72cb74cdc6b
ext4: move handling of list of shrinkable inodes into extent status code

Linux-commit: dd4759255188771e60cf3455982959a1ba04f4eb
ext4: limit number of scanned extents in status tree shrinker

Linux-commit: 624d0f1dd7c80d2bac4fc3066b2ff3947f890883
ext4: cleanup flag definitions for extent status tree

Linux-commit: 2be12de98a1cc21c4de4e2d6fb2bf5aa0a279947
ext4: introduce aging to extent status tree

Test-Parameters: fstype=ldiskfs serverdistro=el7.7
Change-Id: Idd97872b1663bc001a63274a430eaade66efd37d
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/37749
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13355 crypto: adler32 wrapper in libcfs

libcfs has wrappers in order to be able to use adler32 as any other
crypto module. But libcfs should not be considered as the owner of
this implementation.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I66da322b7deae7aeb7f2fb95ede9ee1eff43c70c
Reviewed-on: https://review.whamcloud.com/37878
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>

LU-9859 ptlrpc: change imp_refcount to refcount_t

The lustre portals handle was removed from the OBD import so
it missed the move to refcount_t with the LU-12452 work. Moving
to refcount_t also gives us security protection due to overflow
issues with atomic_t. Lastly refcount_t gives use the same
equivalent as LASSERT_ATOMIC_* with CONFIG_REFCOUNT_FULL thus
allowing us to remove the LASSERT_ATOMIC_* wrappers in time.

Change-Id: Ifdbd51c39bd3921e3f5d18d60efed6d1ff58c5c6
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/37733
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13252 mgs: Handle possible failure from name_create_mdt

name_create_mdt may not succeed if memory allocation fails.

Cray-bug-id: LUS-8503
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I692cfd962e39b7f41d21bcfff1d7577a722ab80d
Reviewed-on: https://review.whamcloud.com/37584
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>

LU-13227 ofd: update lvb before dropping server lock

If this is a local lock on a server namespace and this was
the last reference, lock will be canceled directly instead
of staying at LRU, this might be problem that we will lose
O_DIRECT write size expanding without LVB updating.

This patch try to fix the problem by updating LVB just after
writing but before lock is dropped.

Fixes: 6bce536725 ("LU-4198 clio: turn on lockless for some kind of IO")
Change-Id: I9d1fbac30c8599d677358e8720fe9b3ca81fe818
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/37790
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9859 lustre: remove current_pid() and current_comm()

Just use current->pid and current->comm directly, instead
of having wrappers.

Linux-commit: 63fd7d04580b6345ff1e0aab906c034f973d493e

Test-Parameters: trivial
Change-Id: I278f32d6dd8c370a7ab211c5147ee8d246ea1893
Signed-off-by: NeilBrown <neilb@suse.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-on: https://review.whamcloud.com/37776
Reviewed-by: Neil Brown <neilb@suse.de>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>

LU-930 utils: fix lfs setstripe option description

Document the "-o mode" for the file create mode in the usage
message instead of the deprecated "-m mode" for setdirstripe.

Fixes: fdad38781ccc ("LU-11376 lmv: new foreign LMV format")
Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I49259c06be57e234083c3d0f981ab674931b25d2
Reviewed-on: https://review.whamcloud.com/37730
Tested-by: Maloo <maloo@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Joseph Gmitter <jgmitter@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13274 uapi: make lustre UAPI headers C99 compliant

Attempting to compile strict C99 user land applications or
libraries with the Lustre UAPI headers will fail. These same
errors can be seen by enabling CONFIG_UAPI_HEADER_TEST as well.
Update the Lustre UAPI headers to be compilable with -std=c99.
Enhance our current test covering UAPI header handling.

For OpenSFS branch we can't include <linux/stat.h> since we support
kernels before struct statx existed and they will collide with the
special definitions in lustre_user.h.

Change-Id: Ifb0da33180dc3c7e116d6bf2b7f603ad0528277a
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/37678
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Gian-Carlo DeFazio <defazio1@llnl.gov>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13209 build: SUSE 15 SP2 fix for KBUILD_SRC removed

Linux v5.2-rc4-45-g051f278e9d81 changed

KBUILD_SRC := $(abs_srctree)
  to
building_out_of_srctree := 1

So adjust the build to handler either case

Test-Parameters: trivial
Cray-bug-id: LUS-8472
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I385d84cfe0b557f18d17d5732ab943b16b1e1d9c
Reviewed-on: https://review.whamcloud.com/37465
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Ben Evans <beevans@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9855 lustre: rename obd_device variables to obd

Most variables of type 'struct obd_device *' are named 'obd' -
655 of them.

81 are named 'dev'
29 are named 'obddev'

The remainder (< 100) have various names specific to the used (e.g
disk_obd, src, tgt etc), none used more than 11 times (mgc).

As 'obd', 'dev', and 'obddev' are all simple generic names, it would
be nice to be consistent and only use one of these.

This patch changes all 'dev' and 'obddev' variables to 'obd'.

Also change some "sprintf" to "scnprintf .. PAGE_SIZE" to avoid
warnings.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I640c4f121d7db6aec1fef848a84463d46f68f262
Reviewed-on: https://review.whamcloud.com/36969
Tested-by: Maloo <maloo@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-12756 lnet: Use info cached in lnet_msg on resend

The src_nid and rtr_nid supplied to lnet_send() are cached in the
lnet_msg object. Use this information when resending messages via
lnet_resend_pending_msgs_locked()

Test-Parameters: trivial
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I7a9f54d6d28f26492508f86e447fd01814874f35
Reviewed-on: https://review.whamcloud.com/36541
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-11310 kernel: new kernel [SLES15 SP1 4.12.14-197.29]

This patch makes changes to support new SLES15 SP1 release
with kernel 4.12.14-197.29 for Lustre client.

Test-Parameters: trivial \
envdefinitions=SANITY_EXCEPT="100 130 136 817" \
clientdistro=sles15sp1 serverdistro=el7.7 \
testlist=sanity

Cray-bug-id: LUS-6001
Signed-off-by: Chris Horn <hornc@cray.com>
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Change-Id: I07d4dce16f7a01744c7776c826ba738ace83e58f
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/32693
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10467 libcfs: remove l_wait_event() and related macros.

Now that all usage of l_wait_event() has been converted to standard
wait_event* macros, we can remove l_wait_event and related macros.

Also clean up a few comments that mentioned these removed macros.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ie5fdc21ac94520e52021aa870d351230d39b0bc7
Reviewed-on: https://review.whamcloud.com/37729
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10391 lnet: remove 'accept' variable.

lnet/acceptor.c has two variables, "accept" and "accept_type",
which have identical content and purpose.
Remove "accept" and just use "accept_type".

As the module parameter "accept" will now represent the value stored
in the variable "accept_type", we need to use module_param_named() to
allow the module parameter to have a different name to the kernel
variable.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I938418c971dfdd9f6e5c8c22c03e992432082334
Reviewed-on: https://review.whamcloud.com/37694
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12811 ptlrpc: pass buflen to lustre_swab_object_update_*()

Pass the buffer size to the swabbing functions
lustre_swab_object_update_reply() and
lustre_swab_object_update_request() to avoid invalid access,
especially for small buffer.

Signed-off-by: Emoly Liu <emoly@whamcloud.com>
Change-Id: I229c1946ec3c088c7804762655f143dc419b2ea0
Reviewed-on: https://review.whamcloud.com/36867
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12207 tests: allow some margin for sanity/76

With newer slab allocators it is possible that the kernel may keep
some inodes in the per-cpu cache and not release all of them.  This
was resulting sanity test_76 failures with a few (often 12) inodes
not being freed of the inodes that are created by the test, like:

    inode slab grew from 3224 to 3236
    inode slab grew from 70368 to 70380
    inode slab grew from 68878 to 68890

Allow some small number of inodes (8 per core) cached on the client
without considering it a test failure failure.

Clean up test_76 code style to current standards.

Test-Parameters: trivial testlist=sanity env=ONLY=76,ONLY_REPEAT=100
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ia3f60de2fb471bb32da27d36665a3a0fd43ebbe5
Reviewed-on: https://review.whamcloud.com/37827
Tested-by: Maloo <maloo@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Ben Evans <beevans@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10447 tests: remove use of SETSTRIPE from sanity

The recently landed test_136() was using $SETSTRIPE instead of
$LFS setstripe, but that was removed from test-framework.sh in a
later patch (written earlier and didn't take it into account).
This test doesn't fail during normal testing because it is skipped
due to SLOW testing.  Change it to use $LFS setstripe instead.

Fixes: cc1092291932 ("LU-13069 obdclass: don't skip records for wrapped catalog")
Fixes: be3b7e772d3a ("LU-10447 tests: deprecate use of $SETSTRIPE/$GETSTRIPE")
Test-Parameters: trivial testlist=sanity env=ONLY=136,ONLY_REPEAT=10
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ibb1e8220e2de711e7034d91baa329bdc04687c72
Reviewed-on: https://review.whamcloud.com/37831
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexander Boyko <c17825@cray.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13343 tests: skip recovery-small test_140 with SSK

recovery-small test_140a and test_140b are using a 'local client',
ie a client mounted on a server node.
This is not compatible with SSK keys installed by the test framework,
so just skip these tests when running with SSK.

Test-Parameters: trivial
Test-Parameters: env=SHARED_KEY=true,ONLY=140 testlist=recovery-small
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I420deed9dbc50da622766648a51f885ed01203c6
Reviewed-on: https://review.whamcloud.com/37832
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>

LU-13309 osd-ldiskfs: remove per-page object_get/put in brw

According to profiling data, object_get/put calls consume a lot
of CPU ticks:

99.99%  [kernel.vmlinux]
           |
           |--28.82%--_atomic_dec_and_lock
           |          |
           |           --28.82%--lu_object_put
           |                     |
           |                      --28.73%--osd_bufs_put
           |                                ofd_commitrw
           |                                tgt_brw_read
           |                                tgt_request_handle
           |                                ptlrpc_server_handle_request
           |                                ptlrpc_main
           |                                kthread
           |                                ret_from_fork
           |
           |--26.51%--osd_bufs_get
           |          |
           |           --26.51%--ofd_preprw
           |                     tgt_brw_read
           |                     tgt_request_handle
           |                     ptlrpc_server_handle_request
           |                     ptlrpc_main
           |                     kthread
           |                     ret_from_fork
           |
           |--18.09%--lu_object_put
           |          |
           |           --18.01%--osd_bufs_put
           |                     ofd_commitrw
           |                     tgt_brw_read
           |                     tgt_request_handle
           |                     ptlrpc_server_handle_request
           |                     ptlrpc_main
           |                     kthread
           |                     ret_from_fork

ofd_preprw_read() and ofd_preprw_write() pin the corresponding
ofd object, later ofd_commitrw_read() and ofd_commitrw_write()
unpin it. When the ofd object is pinned, its underlying
osd object cannot go away, so object_get/object_put
in osd_bufs_get()/osd_bufs_put() are basically no-op.

Change-Id: Ic48f793de5ef3e62505f44879c91050922160000
Signed-off-by: Andrew Perepechko <c17827@cray.com>
Cray-bug-id: LUS-4388
Reviewed-on: https://review.whamcloud.com/37758
Reviewed-by: Alexander Boyko <c17825@cray.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13258 libcfs: fixes for cfs_arch_init()

The introduction of cfs_arch_init() brought two problems.

1/ wait_bit_init() wasn't known due to a missing include file.
2/ cfs_arch_init() was not marked __init, but it called
   a function (wait_bit_init) that was.

This patch fixes both of these.

Fixes: 3453c95f513c ("LU-13258 libcfs: make apply_workqueue_attrs() available for Lustre")
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I6f19e11e68f52ca8071332364d369ed3a717d5c9
Reviewed-on: https://review.whamcloud.com/37727
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12477 lustre: restore time_after32()

time_after32 is needed when compiling with kernels earlier than v4.14,
which haven't had it backported.  E.g. SLE12-SP3-LTSS needs it.

Fixes: 8e88bbfef579 ("LU-12477 lustre: remove obsolete config checks")
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ifb51b9ed0945ad28f8e8aa34dada3860107b95df
Reviewed-on: https://review.whamcloud.com/37725
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-13261 mdt: PFL layout changed while accessing

The PFL layout EA could be enlarged when the corresponding layout
of some IO range is started to be written, which can cause other
thread to get incorrect layout size at "mdt_intent_layout" and cause
"mdt_lvbo_fill" to fail checking the real layout size.

In Lustre, "ldlm_handle_enqueue0" has processed the error "-ERANGE"
and it will retry after expanding the layout buffer size, then it
only needs to decrease the debug level of the log in "mdt_lvbo_fill"

Change-Id: Iad722d1dac187f57ae77606a4d4587525412cd68
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/37684
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>

LU-13276 lnet: Update nnis to avoid infinite loop

The goto loop in lnet_push_target_resize() is infinite because
the loop variable 'nnis' is not updated with the new value from
ln_push_target_nnis.

Cray-bug-id: LUS-8526
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I65b0bc0b56393f2296bafa3a964c59840baa0643
Reviewed-on: https://review.whamcloud.com/37676
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13277 lnet: Discovery thread can deadlock on shutdown

Drop the net_lock/EX before breaking out of the loop to avoid
deadlock.

Cray-bug-id: LUS-8525
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: Ie62b55fa45b5795937eb1480a1fcabe295fed0ee
Reviewed-on: https://review.whamcloud.com/37675
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13282 tests: wait $LFSCK_BG_PID properly

ha.sh fails with:
  wait: pid $LFSCK_BG_PID is not a child of this shell
because wait $LFSCK_BG_PID duplicates wait in ha_stop_loads().

Patch fixes this issue.

Test-Parameters: trivial
Cray-bug-id: LUS-7930
Signed-off-by: Elena Gryaznova <elena.gryaznova@hpe.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-by: Vladimir Saveliev <vladimir.saveliev@hpe.com>
Change-Id: Ifdeab0dc570e9da889ccb34c0b47473e1077bfdc
Reviewed-on: https://review.whamcloud.com/37668
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Vladimir Saveliev <c17830@cray.com>
Reviewed-by: Alexander Boyko <c17825@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13253 libcfs: protect libcfs_debug_dumplog()

as it uses global state to wait for dumping thread completion and
it doesn't make sense to dump concurrently.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Ie6916a0bb53a0a3aa8992413811fe7a908f86276
Reviewed-on: https://review.whamcloud.com/37588
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13234: osd-ldiskfs: hold inode mutex for ldiskfs_orphan_add()

See following warning:

/home/green/git/lustre-release/ldiskfs/namei.c:3310 ldiskfs_orphan_add+0x11e/0x2a0 [ldiskfs]
...
Call Trace:
[<ffffffff817d1711>] dump_stack+0x19/0x1b
[<ffffffff8108ba58>] __warn+0xd8/0x100
[<ffffffff8108bb9d>] warn_slowpath_null+0x1d/0x20
[<ffffffffa0adbfce>] ldiskfs_orphan_add+0x11e/0x2a0 [ldiskfs]
[<ffffffffa0bb3b73>] osd_punch+0x2a3/0x6c0 [osd_ldiskfs]
[<ffffffffa0692541>] ? tgt_fmd_put+0x41/0x120 [ptlrpc]
[<ffffffffa0e8b0a8>] ofd_object_punch+0x7e8/0xce0 [ofd]
[<ffffffffa0e7a5d3>] ofd_punch_hdl+0x4f3/0xa70 [ofd]
[<ffffffffa0673f55>] tgt_request_handle+0x965/0x1620 [ptlrpc]
[<ffffffffa0229dde>] ? libcfs_nid2str_r+0xfe/0x130 [lnet]
[<ffffffffa0616f60>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<ffffffff810c6941>] ? __wake_up_common_lock+0x91/0xc0
[<ffffffff810c6250>] ? sched_feat_set+0xf0/0xf0
[<ffffffffa061b1c0>] ptlrpc_main+0xcb0/0x1cb0 [ptlrpc]
[<ffffffff810c665d>] ? finish_task_switch+0x5d/0x1b0
[<ffffffffa061a510>] ? ptlrpc_register_service+0xff0/0xff0 [ptlrpc]
[<ffffffff810b8254>] kthread+0xe4/0xf0
[<ffffffff810b8170>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817e5ddd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810b8170>] ? kthread_create_on_node+0x140/0x140

Hold mutex lock for ldiskfs_orphan_add() to eliminate warning.

Fixes: f64e9f19f68e ("LU-12977 ldiskfs: properly take inode_lock() for truncates")
Change-Id: I810c3c170649b3c96d98f227a480bb1092f2386c
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/37528
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13004 ptlrpc: simplify bd_vec access.

Now that there are no kvecs in ptlrpc_bulk_desc, only bdvecs, we can
simplify the access, discarding the containing struct and the macros,
and just accessing the fields directly.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I068a7a280f130bf0b53b9c572ed47ef0cc999102
Reviewed-on: https://review.whamcloud.com/36973
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13005 lnet: remove the 'queue' from LNetEQ

All calls to LNetEQAlloc pass a size of 0, so no queue
is ever allocated.
So remove the 'size' arg, and all code that depends on
it being non-zero.
Similarly remove eq_size, eq_deq_seq eq_enq_seq and eq_events
as they are always 0/NULL.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Icb7bb352fa61cd6ea46847676e583e738eeeda8c
Reviewed-on: https://review.whamcloud.com/36844
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13005 lnet: remove LNetEQPoll

There are no longer any users for LNetEQPoll, so remove it
and any mention of it.

Also remove the ln_eq_waitq which is no longer used.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: If0c1a2634e879c62925314a55b4c2ae0512d1837
Reviewed-on: https://review.whamcloud.com/36843
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13005 lnet: don't use LNetEQPoll for ping replies.

lnet_ping() is the only user of LNetEQPoll.  All other event queues
register a callback and don't make use of queuing.
The queuing provided by lib-eq isn't really needed in the kernel where
callbacks are so easy.

So as a step towards simplifying lib-eq, change lnet_ping() to
register a callback which uses a completion to notify when the ping is
finally complete.

Note that the handling of 'rc' in the current code is a little
confused.
If LNetGet() succeeds, but no LNET_EVENT_REPLY is received, then
rc will be set to -EIO.  There is code to try to set it to ETIMEOUT
or the return value from LNetEQPoll, but this code is ineffectual.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ie937887524bfa3ca9c2828d673d431411720c7ae
Reviewed-on: https://review.whamcloud.com/36842
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12214 build: fix build with MPI

 lack of MPI library dependence will lost a mpi tests
 from build

Cray-bug-id: LUS-6033, LUS-7204
Test-parameters: trivial
Change-Id: I88b8ad67a9a2863fdcd02e2df5289fdd480c2a74
Signed-off-by: Alexey Lyashkov <c17817@cray.com>
Reviewed-on: https://review.whamcloud.com/36427
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Alexander Boyko <c17825@cray.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12214 build: fix build with mofed

add a MOFED / OPA modules dependency to produce a right
package.

Test-parameters: trivial
Cray-bug-id: LUS-6033, LUS-7204
Change-Id: I843762945c5dbb59d53e1913b7382813bfba86ad
Signed-off-by: Alexey Lyashkov <c17817@cray.com>
Reviewed-on: https://review.whamcloud.com/36426
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Alexander Boyko <c17825@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12214 build: fixes if the name is not just 'lustre'

fix using a %{name} macro in spec file.
this allow to have right names for server packages.

Cray-bug-id: LUS-5915
Test-parameters: trivial
Change-Id: I2ae271b5344fb899bb053f82d2534355ce60aa3a
Signed-off-by: Alexey Lyashkov <c17817@cray.com>
Reviewed-on: https://review.whamcloud.com/36424
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexander Boyko <c17825@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12780 osp: don't use ptlrpc_thread for opd_update_thread

rather than ptlrpc_thread, use native kthreads functionality.

- There is no need to synchornized on startup - do all the startup
  that can fail before starting the thread.  This involves puting the
  lu_env in the per-thread struct osp_updates.

- Synchronization on shutdown is down with kthread_stop() and
  kthread_should_stop().  lu_env_put() call is moved to after
  kthread_stop() call, as it is theoretically possible that the
  thread function never gets called, so it isn't safe for it to
  be responsible for cleanup.

- opd_update_thread is replace with ou_update_task in struct
  osp_updates.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I78dafbadf419ee9b80a9bd0046152fb6f293f191
Reviewed-on: https://review.whamcloud.com/36264
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-11510 lfs: migrate a composite layout file correctly

The patch fixes the following issues:
- in function migrate_open_files(), "layout" pointer should be used
  instead of "param" pointer to tell whether a comp file should be
  created or not, because "param" pointer is always not null and
  the composite layout file will never be created;
- make --copy and --yaml options work correctly in lfs_migrate tool;
- when a composite layout file is migrated, "--copy" option will be
  added to preserve its layout in both lfs_migrate and "lfs migrate",
  and in such situation, pool name will be saved as well;
- when a file is restriped with -R option by lfs_migrate, the file
  will be set with its parent's stripe by default, by adding
  "--copy $parent_dir" option;
- do some code cleanup in lfs_migrate and sanity.sh test_56wb/c

sanity.sh test_56xd/xe are added to verify this patch.

Signed-off-by: Emoly Liu <emoly@whamcloud.com>
Change-Id: I85779c69e74444eb869f28add4363ad3a6835b97
Reviewed-on: https://review.whamcloud.com/36082
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12542 ldlm: don't access l_resource when not locked.

lock->l_resource can (sometimes) change when the resource
isn't locked.
So dereferencing lock->l_resource and then locking the
resource looks wrong.
As lock_res_and_lock() returns the locked resource, this
code can easily be more obviously correct by using
that return value.

Change-Id: Iced0bf1af4fa8ddedffa817e00f1c6a02b035d76
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Reviewed-on: https://review.whamcloud.com/35484
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-11505 tests: customise run_*() functions

For PFL testing we need the ability to run
parallel-scale tests with composite files.
Patch adds the subtests' specific variables *_STRIPEPARAMS
which allow to set any specified striping/composite layout
for test directory.

Test-Parameters: trivial testlist=parallel-scale,parallel-scale-nfsv3
Signed-off-by: Elena Gryaznova <elena.gryaznova@hpe.com>
Cray-bug-id: LUS-5936
Reviewed-by: Alexander Boyko <c17825@cray.com>
Reviewed-by: Vladimir Saveliev <c17830@cray.com>
Change-Id: I767fa17523f5c40d50260892901dc947a1da8a24
Reviewed-on: https://review.whamcloud.com/33352
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9868 llite: remove lld_it field of ll_dentry_data

This field is never set nor used.  Let's remove it.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: If01ba4deec34a6528b3c36c45a378b9f3824ca2f
Reviewed-on: https://review.whamcloud.com/37741
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9679 osc: fix to return right weight in osc_lock_weight()

cl_io_init() might return a negative value, it could be indicated
lock should be not eliminated, osc_lock_weight() expect a positive
value, fix to return 1 in this case.

This is not a problem for now, as osc_io_init() only return 0,
To avoid any potential issues in the future, better fix it.

Test-Parameters: trivial
Change-Id: I657e5dcc4bb7691bf3b4ca06df7cb87945008a93
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/37735
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9679 osc: use rb_entry_safe

Use rb_entry_safe() instead of container_of() to simplify the code.

Linux-Commit: e3e0293ca9b9 ("staging: lustre: osc: use rb_entry_safe")

Signed-off-by: Geliang Tang <geliangtang@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I9f8e19d45d859a6c7b7aa01093a1c2211063874c
Reviewed-on: https://review.whamcloud.com/37604
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9679 osc: simplify osc_page_gang_lookup()

osc_page_gang_lookup() has 4 values that it can receive from a
callback, and that it can return to the caller:
CLP_GANG_OKAY,
CLP_GANG_RESCHED,
CLP_GANG_AGAIN,
CLP_GANG_ABORT

"AGAIN" is never used.
"RESCHED" is not needed as a cond_resched() can safely be called at
the point this is returned, rather than returning it.
That leaves "OKAY" and "ABORT" which can simply by "true" and "false"
boolean values.

Internalizing the RESCHED case means the callers don't need to loop
themselves.  This simplifies calling patterns.

Linux-Commit: 1e8fd6f9806c ("lustre: osc_cache: simplify
osc_page_gang_lookup()")

Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I603963b72e4299bebcdaf4064428d6281fd12def
Reviewed-on: https://review.whamcloud.com/37599
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13102 llog: fix processing of a wrapped catalog

The logic for rereading a llog buffer had an exception
for a full catalog, when lgh_last_idx = llh_cat_idx and a first
processing index is a llh_cat_idx+1. This check is based on
a value lh_last_idx, which stays unchanged between buffer read.
But llh_cat_idx could go forward, and this lead to a wrong check
where reread doesn't happen. As a result Lustre got ENOENT for
a record and stoped osp processing.

llog_cat_set_first_idx())
catlog [0x6:0x1:0x0] first idx 34730, last_idx 34731
osp_sync_process_queues()) 1 changes, 0 in progress, 0 in flight
llog_process_thread())
stop processing plain 0x76941:1:0 index 64767 count 1
llog_process_thread())
index: 34731, lh_last_idx: 34730 synced_idx: 34730 lgh_last_idx: 34731
llog_cat_process_common()) processing log [0x2780f:0x1:0x0]:0
at index 34731 of catalog [0x6:0x1:0x0]
llog_cat_id2handle()) snx11281-OST0001-osc-MDT0001:
error opening log id [0x2780f:0x1:0x0]:0:rc = -2

The patch fixes logic and also adds/changes debugging for
llog and osp.

Cray-bug-id: LUS-8193
Signed-off-by: Alexander Boyko <c17825@cray.com>
Change-Id: I9463223a1ea904b96643b19e1580f5894142c12b
Reviewed-on: https://review.whamcloud.com/37102
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>

LU-9679 libcfs: Add CFS_ALLOC_PTR_ARRAY and free

Following the pattern of CFS_ALLOC_PTR() and the kernel
pattern of kmalloc_array(), add
  CFS_ALLOC_PTR_ARRAY()
and
  CFS_FREE_PTR_ARRAY()

which allocate and free arrays given a pointer to
the array, and a number of elements

This makes code easier to read and could be a step
toward using the kernel's hardened alloc_array interfaces,
which insure the multiplication doesn't overflow.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I54c919bd9ce22fbc72715c3da16f8c29e7135ccc
Reviewed-on: https://review.whamcloud.com/36975
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-12950 osd-ldiskfs: increase supported size to 1024tb

Currently attempts of creating ldisk file system with size >512TB
finished with message:

    LDISKFS-fs does not support file systems greater than 512TB
    and can cause data corruption. Use "force_over_512tb" mount
    option to override.

Change "force_over_512tb" mount option to "force_over_1024tb" as
testing for these large filesystems have not shown any serious
functional problems (though there are performance issues to address).

Test-Parameters: trivial fstype=ldiskfs testlist=conf-sanity
Signed-off-by: Artem Blagodarenko <c17828@cray.com>
Cray-bug-id: lus-6815
Change-Id: I0f84fe85aaab05ab0bafa5f0e6074e1690d79899
Reviewed-on: https://review.whamcloud.com/36705
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>

LU-13229 ldlm: unlock request memory leak

resending an unlock request can caused an memory leak as new request
will allocated on resend. Lets finish a request before resend.

Fixes: 85a12c6c8d7 ("LU-12828 ldlm: FLOCK request can be processed twice")
Cray-bug-id: LUS-8447
Signed-off-by: Alexey Lyashkov <c17817@cray.com>
Change-Id: I15dfe0388d1305c8eb5be18b19b0ffc687454ef1
Reviewed-on: https://review.whamcloud.com/37670
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13296 obd: make statfs cache working again

Once statfs raced on mutex, lets read a cached data instead
of trash.

Test-Parameters: testlist=sanity envdefinitions=ONLY=423,ONLY_REPEAT=500
Fixes: 1c41a6ac390b ("LU-12368 obdclass: don't send multiple statfs RPCs")
Signed-off-by: Alexey Lyashkov <c17817@cray.com>
Change-Id: I268782875c30c078f239c194f69cdf7506d66169
Reviewed-on: https://review.whamcloud.com/37753
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>

LU-13209 build: Fix vvp_account_page_dirtied

Fix compile error vvp_account_page_dirtied undefined

Fixes: 788e464a72 ("LU-13288 llite: Find account_page_dirtied on module init")
Cray-bug-id: LUS-8472
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: Iae0a3af1b091165423a2bb24a9159a7af8ab3cbe
Reviewed-on: https://review.whamcloud.com/37778
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13241 utils: use libext2fs for ldiskfs operations

Instead of using debugfs to stat and read the CONFIGS/mountdata,
we can switch to libext2fs and control the flags used when
opening the ldiskfs, to reduce the mount time on huge targets.

Change-Id: I9da8fc1c77d149fe5cf3bd19b0ff76d892620101
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-on: https://review.whamcloud.com/37656
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13314 test: add 56ob to ALWAYS_EXCEPT for now

There is a problem that 'lfs find' try to calculate
365 days for one year, this will be problem for leap year.

We need fix test or codes, before that let's disable this test
to make other patches landed possibly.

Change-Id: I79d34ce29657b4d149a0fbe82220cfefa0c60378
Test-Parameters: trivial testlist=sanity envdefinitions=ONLY=56ob
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/37767
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-12988 ldiskfs: mballoc to prefetch groups

ahead of scanning. prefething is done in 8 * flex_bg groups, so
it should be 8 read-ahead reads for a single allocating thread.
at the end of allocation the allocating thread waits for read-ahead
completion and initializes buddy information so that read-aheads
are not lost in case of memory pressure.
at cr=0 the number of prefetching IOs is limited per allocation
context to prevent a situation when mballoc loads thousands of
bitmaps looking for a perfect group and ignoring groups with
good chunks.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Ibce9a060900544abe994f4d7a20ac9c2979ccc56
Reviewed-on: https://review.whamcloud.com/37633
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>

LU-12043 llite: move tunable params to sysfs_memparse()

Move the max_read_ahead_* tunables from debugfs to sysfs, since
they follow the one-value-per-file rule and should be visible to
regular users.

Rename the functions and constants from *readahead* to *read_ahead*
or *READ_AHEAD* to match the tunable names from procfs.

Deprecate usage of llprocfs_str_with_units_to_s64(), lu_str_to_s64(),
llprocfs_str_with_units_to_u64(), and lu_str_to_u64(), and instead
use sysfs_memparse() to parse sizes in the few remaining places
where they are used.  A separate patch will remove those functions.

Minor fix to the "lctl set_param" man page.

Fixes: adb5aca3d673 ("LU-8066 llite: Move all remaining procfs entries to debugfs")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I2cdf5f8f0aeca458ed1989366102c33ae83ebbe5
Reviewed-on: https://review.whamcloud.com/34849
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13096 llite: fix potential overflow in ll_max_cached_mb_seq_write()

atomic_long_cmpxchg() return long, however @rc is int, if we have
a larger memory etc 24T, we will get overflow, and @diff will never
change thus we get a hang loop there.

Test-Parameters: trivial
Fixes: adb5aca3d673 ("LU-8066 llite: Move all remaining procfs entries to debugfs")
Change-Id: I20d6feff9797ba10a089587bee0d8691bee460df
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/37707
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13283 tests: add racer to nonmpi load

Patch adds the ability to run racer as one of
nonmpi loads. All racer parameters can be set over
RACERP:
Example:
  RACERP="MDSCOUNT=3 DURATION=600 RACER_ENABLE_STRIPED_DIRS=false"
etc.

ha_start_mpi_loads() is improved to not create
machinefiles if no mpi load set.

Test-Parameters: trivial
Cray-bug-id: LUS-8297
Signed-off-by: Elena Gryaznova <elena.gryaznova@hpe.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-by: Vladimir Saveliev <vladimir.saveliev@hpe.com>
Change-Id: I974838ce199897284396ef53d1a487d1a1ae1774
Reviewed-on: https://review.whamcloud.com/37671
Reviewed-by: Vladimir Saveliev <c17830@cray.com>
Reviewed-by: Alexandr Boyko <c17825@cray.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13281 tests: ha.sh improvements

Path adds the possbility to set the different layouts
for clients directories. This allows to run the clients loads
on the directories with DoM, PFL, SEL, etc. including the simple
layouts set on old clients, which is useful for inter-operation
testing.

Patch adds the possibility to split the full client list to
separate subsets by specifying NCLIENTSSET equal to 1 by
default. For NCLIENTSSET=2 two subsets will be created, each
one are passed to corresponding machinefile. This allow
to split the loads which is also useful for inter-operation:
old clients can operate with directories with old layouts while
the new clients operate with directories with new DoM, PFL, SEL,
etc. layouts.
For NCLIENTSSET=2, ${#ha_clients[@]}=4 and
CLIENTSSTRIPE='"-E $((64*1024)) -L mdt -E EOF" "-c -1 "'
layout "-E $((64*1024)) -L mdt -E EOF" will be applied on
even clients ${ha_clients[0]} and ${ha_clients[2]},
layout "-c -1 " will be applied on odd clients ${ha_clients[1]}
and ${ha_clients[3]}.

Test-Parameters: trivial
Cray-bug-id: LUS-6906
Signed-off-by: Elena Gryaznova <elena.gryaznova@hpe.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-by: Vladimir Saveliev <vladimir.saveliev@hpe.com>
Change-Id: Ia6f8c07e3936ea773706c3e79217672c68d77bd0
Reviewed-on: https://review.whamcloud.com/37666
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexandr Boyko <c17825@cray.com>
Reviewed-by: Vladimir Saveliev <c17830@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13219 tests: add nfs-server service in setup-nfs.sh

For RHEL 8.1, the NFS server service is nfs-server
instead of nfs.

Test-Parameters: trivial \
clientdistro=el8.1 serverdistro=el8.1 \
testlist=parallel-scale-nfsv3,parallel-scale-nfsv4

Change-Id: I5a97c8fe419187412dfc02047ed66141f567d7fb
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/37663
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13270 tests: dom-performance fixes

Patch makes the fs cleanup at the start of dom-performance
optional and depending on t-f global DO_CLEANUP option
value. For DO_CLEANUP=false script removes only the
directories created by previous dom-performance session.
Without this fix test run 2nd time fail with:
  lfs setdirstripe: dirstripe error on '/mnt/testfs/dp_dne':
      stripe already set
  lfs setdirstripe: cannot create stripe dir '/mnt/testfs/dp_dne':
      File exists

Patch removes the check of the files createmany, statmany
and smalliomany which are part of lustre/tests. With
the existing check test_smallio() is always skipped
if run on PWD != lustre/tests.

Patch removes the comparison with MDSSIZE and adds the
comparison with real mds size. Without this fix the test
works incorrectly if run on existing fs (formatall was not
done during this session: Lustre was created with mds size
differs from MDSSIZE).

Add skip_env() to report about missing mdtest/dbench/IOR/etc.
Without this fix  the tests are skipped silently.

Test-Parameters: trivial mdssizegb=20 testlist=dom-performance
Cray-bug-id: LUS-7349
Signed-off-by: Elena Gryaznova <elena.gryaznova@hpe.com>
Reviewed-by: Vladimir Saveliev <vladimir.saveliev@hpe.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Change-Id: I5fdf1ad480edb17598cbe427bc550396ebe97808
Reviewed-on: https://review.whamcloud.com/37644
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13269 tests: make lnet-selftest.sh more flexible

lnet-selftest.sh adds both LST tests: from client to server and
from server to client, but sometimes we are interesting to execute
the tests only from server or only from client.

Patch allows to set required "from" parameter via lst_FROM variable.
The following values are used:
lst_FROM=c : add the tests from clients to server only
lst_FROM=s : add the tests from server to clients only

Test-Parameters: testlist=lnet-selftest envdefinitions=lst_FROM=c
Cray-bug-id: LUS-8178
Signed-off-by: Elena Gryaznova <elena.gryaznova@hpe.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Vladimir Saveliev <vladimir.saveliev@hpe.com>
Change-Id: I0111a4386b3d8acb022db11f11a0be970864696d
Reviewed-on: https://review.whamcloud.com/37640
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Vladimir Saveliev <c17830@cray.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13268 tests: customize lnet-selftest for performance

Sometimes we need to run read/write/ping tests separately
with different lst check flags.

Patch allows to:
  set the required list of tests via new lst_TESTS variable;
  set lst check to LST_BRW_CHECK_NONE, LST_BRW_CHECK_FULL
      or LST_BRW_CHECK_SIMPLE.

Test-Parameters: trivial testlist=lnet-selftest
Cray-bug-id: LUS-8005
Signed-off-by: Elena Gryaznova <elena.gryaznova@hpe.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Reviewed-by: Vladimir Saveliev <vladimir.saveliev@hpe.com>
Change-Id: I502028f645f39f391c19b2028fc6ba5b7bdb1a96
Reviewed-on: https://review.whamcloud.com/37639
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Vladimir Saveliev <c17830@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13267 tests: improve racer cleanup

Add RACER_MAX_CLEANUP_WAIT parameter to specify
timeout for racer cleanup to avoid long waiting in case which
racer processes went non-killable.

Loop in racer_cleanup() contains inaccuracy which made the loop
to sleep less that it was supposed to. Fix it.

Test-Parameters: trivial testlist=racer
Cray-bug-id: LUS-8498
Signed-off-by: Elena Gryaznova <elena.gryaznova@hpe.com>
Reviewed-by: Vladimir Saveliev <vladimir.saveliev@hpe.com>
Reviewed-by: Andriy Skulysh <andriy.skulysh@hpe.com>
Change-Id: I32dac8bc11ef2041a1a580054c2782780bb5980e
Reviewed-on: https://review.whamcloud.com/37638
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Vladimir Saveliev <c17830@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13227 LDLM: update LVB if it is server lock

ldlm_glimpse_ast() is registered for server lock which means
when client send a glimpse request, it just return a special error
for this lock, it is possible that local object has size expanding
with this PW lock, so we should try update LVB upon error.

Originally, ldlm_cb_interpret() has codes to handle this error,
but it only try to handle case with some clients race, it doesn't
cover server lock cases especially after we turn on lockless for DIO.

Fixes: 6bce536725 ("LU-4198 clio: turn on lockless for some kind of IO")
Change-Id: Ic84fd19d9eaf7f8245b8f7a2165ee5913849ac01
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/37611
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13160 tests: fix sanity-hsm monitor setup

On RHEL8, even we are using pdsh -R ssh,
the ssh still waits for the remote cat process
to finish.
Use the subshell to avoid the time out.

Change-Id: Id5b8d492b5ce9a235da73448ade475ade145bbed
Test-Parameters: trivial clientdistro=el8.1 testlist=sanity-hsm
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-on: https://review.whamcloud.com/37595
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13211 ldiskfs: rework data-in-dirent for linux 5.4.7+

Linux commit v5.4-rc3-92-g109ba779d6cc
ext4: check for directory entries too close to block end

This impacts the ext4-data-in-dirent.patch due to the usage
of EXT4_DIR_REC_LEN

Invert the original patch from:
  EXT4_DIR_REC_LEN(<int>)        => __EXT4_DIR_REC_LEN(<int>)
  EXT4_DIR_REC_LEN(de->name_len) => EXT4_DIR_REC_LEN(de)
to:
  EXT4_DIR_REC_LEN(<int>)        => EXT4_DIR_REC_LEN(<int>)
  EXT4_DIR_REC_LEN(de->name_len) => EXT4_DIR_ENTRY_LEN(de)

Doing this allows the patch to apply cleanly over a wider
range of kernel releases.

Test-Parameters: trivial
Cray-bug-id: LUS-8478
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I95349743f323bb150854ff4541bd2c88f01662a6
Reviewed-on: https://review.whamcloud.com/37467
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12945 lnet: Disable zero copy when running on VM

When running on a hypervisor platform zero copy buffers
may still be referenced when write queue size is zero

So when running on a hypervisor push the zero copy size limit
above max payload size of 16M.

Use the hypervisor test added to linux v4.14-119-g79cc74155218
and provide a replacement for earlier kernels.

kernel-commit: 79cc74155218316b9a5d28577c7077b2adba8e58

Cray-bug-id: LUS-8072
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I5582a6aa8da6f48deafaf13d60cf67a09d7a7231
Reviewed-on: https://review.whamcloud.com/37300
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13110 kernel: kernel update SLES12 SP4 [4.12.14-95.45.1]

Update SLES12 SP4 kernel to 4.12.14-95.45.1 for Lustre client.

Test-Parameters: trivial clientdistro=sles12sp4 \
envdefinitions=LNET_SELFTEST_EXCEPT=smoke,SANITY_EXCEPT="103a 817"

Change-Id: I1f7024465b4b6334488b7314f1073fafa10331d6
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/37123
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13082 tests: enable lgss_keyring debug traces

Enable lgss_keyring debug traces in test framework, and collect
systemd journal in case of test failure.
Also, if needed, dump SSK keys, keyring and nodemap definitions.

Test-Parameters: trivial
Test-Parameters: envdefinitions=SHARED_KEY=true testlist=sanity
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I201d97b640045e3cbc8dcd4cd4b25e0e4e644037
Reviewed-on: https://review.whamcloud.com/37042
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13097 tests: set fail_loc on all MDS nodes for pdir tests

Set fail_loc on all MDS nodes for pdir tests, not only $SINGLEMDS.

Test-Parameters: trivial testlist=sanityn,sanityn,sanityn,sanityn
Test-Parameters: testlist=sanityn,sanityn,sanityn,sanityn,sanityn
Test-Parameters: testlist=sanityn,sanityn,sanityn,sanityn,sanityn
Test-Parameters: testlist=sanityn,sanityn,sanityn,sanityn,sanityn
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I80d935eae0b06e39712abfe48a56b8ce08537926
Reviewed-on: https://review.whamcloud.com/37145
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>

LU-13004 ptlrpc: remove *GET*KVEC macros and fields.

GET_KVEC, BD_GET_KVEC, GET_ENC_KVEC, BD_GET_ENC_KVEC
are no longer used.
There are the only users of the bd_kvec field of bd_u,
so that field can be removed, and bd_u can be discarded
and its other field (bd_kiov) promoted.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I31d19f6867c907029aa8d9ceee27c5ac9c9225a1
Reviewed-on: https://review.whamcloud.com/36972
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13293 llite: don't abort readahead too aggressively

We will stop readahead if we have covered @ria_end_idx_min
but lock contention case happen, this could cause a problem
with small read with SSF mode.

Because even lock contention happen, but every client could
potentially have a large consecutive pages to read, if this
is a 4K read, it will end up with a lot small read.

To fix this problem, at least allow readahead continue with
one stripe size, this is exact behavior before the commit

Without Patch:
Max Write: 13082.37 MiB/sec (13717.85 MB/sec)
Max Read:  854.17 MiB/sec (895.67 MB/sec)

With Patch:
Max Write: 12448.90 MiB/sec (13053.61 MB/sec)
Max Read: 23921.73 MiB/sec (25083.75 MB/sec)

Fixes: cfbeae9 ("LU-12043 llite: extend readahead locks for striped file")
Change-Id: I59963592f6dbe6babd746cd01441f4a99a8cafcb
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/37697
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13288 llite: Find account_page_dirtied on module init

Kernel v5.2-5678-gac1c3e4 no longer exports account_page_dirtied
Use kallsyms_lookup_name to find and use it as
vvp_account_page_dirtied on module init to avoid any
performance regressions due to symbol_get.

Test-Parameters: clientdistro=el8.1
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: Ie27abb07ffbf9e5be67fe64601ebc62409f829fd
Reviewed-on: https://review.whamcloud.com/37686
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Shuichi Ihara <sihara@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13291 ldiskfs: mballoc don't skip uninit-on-disk groups

as those need no IO to initialize buddy structures and the best
candidates for new blocks.

Fixes: 6a7a700a1490 ("LU-12988 ldiskfs: skip non-loaded groups at cr=0/1")
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Ic3c5a238d8825024d7a0fec6a25e842b7ba1f100
Reviewed-on: https://review.whamcloud.com/37687
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13004 lustre: remove support for KVEC bulk descriptors

KVEC descriptors are no long used nor needed.
KIOV are sufficient for all needs.

This allows us to remove
  PTLRPC_BULK_BUF_KVEC
and
  PTLRPC_BULK_BUF_KIOV
flags - the distinction no longer exists.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ic3a6ec942b60a05c7ce6c5b05659700e1399d0b9
Reviewed-on: https://review.whamcloud.com/36971
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13026 lnet: MR selection of gateway ni

Multi-Rail gateways can only have 1 route pointing to them. Use
the MR selection algorithm to get the best lpni on the MR
gateway to use.

Using the selection algorithm to find the gateway NI, allows us
to use the standard MR criteria: health, preference and credits.

Test-parameters: trivial

Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: I65526c6b64a5702734949c9a583a7558614ceae2
Reviewed-on: https://review.whamcloud.com/36913
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13025 lnet: pick healthiest peer net

When iterating over the peer nets, select the healthiest one.
Node might be able to reach a peer over multiple nets, and therefore
the health of these peer nets must be considered.

Test-parameters: trivial

Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: I155888dca358627fcb63c2ed0e51114bc49a9ff1
Reviewed-on: https://review.whamcloud.com/36912
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12678 lnet: remove lnet_me_alloc/lnet_me_free

These functions are simple wrapper that do not benefit
readability, so remove them.

Move the DEBUJG messages to the places where allocation happens.  This
introduces only a tiny amount of code duplication.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ieb40f8a7547cba30b05c1e5e526c762e354f3c47
Reviewed-on: https://review.whamcloud.com/36910
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>