Whamcloud - gitweb
fs/lustre-release.git
11 months agoLU-13169 tests: quiet debug messages during tests 38/37338/11
Andreas Dilger [Tue, 28 Jan 2020 00:29:09 +0000 (17:29 -0700)]
LU-13169 tests: quiet debug messages during tests

Quiet the output of check_network() and check_node_health() that are
printed for *every* subtest that is run, since this is distracting
when looking at the test output, and can often be more verbose than
the actual test output.

Don't get the client and server versions repeatedly, since they will
not change for the duration of a test run and can be cached instead.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ib42b01c9ca36c6c72d2c7ffffbd3a5f2643ebbe5
Reviewed-on: https://review.whamcloud.com/37338
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
11 months agoLU-13005 lnet: remove lib-eq. 47/36847/8
Mr NeilBrown [Wed, 20 Nov 2019 02:56:44 +0000 (13:56 +1100)]
LU-13005 lnet: remove lib-eq.

There is nothing of value in left in lib-eq.  An 'lnet_eq' just holds
a function pointer.
So change all 'struct lnet_eq' pointers to function pointers, and
bypass the "event queues".

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I16786e44ab712a76860fe8d83543a292555cf557
Reviewed-on: https://review.whamcloud.com/36847
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
11 months agoLU-13005 lnet: drop refcounting from event queues. 46/36846/10
Mr NeilBrown [Wed, 20 Nov 2019 02:31:09 +0000 (13:31 +1100)]
LU-13005 lnet: drop refcounting from event queues.

The refcount is no longer incremented or decremented
as there is no interesting state and no long term
references.  Nothing checks if LNetEQFree returns -EBUSY
any more - though a few calls LASSERT() that the return
value is 0.

So discard the refcounting, change LNetEQFree to return
'void', and remove all the LASSERTs.

Change-Id: I976834e7c6193765948e3b1a4f02069e33cdbe1e
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Reviewed-on: https://review.whamcloud.com/36846
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
11 months agoLU-12780 osp: don't use ptlrpc_thread for osp_sync_thread() 65/36265/8
Mr NeilBrown [Wed, 23 Oct 2019 00:30:50 +0000 (11:30 +1100)]
LU-12780 osp: don't use ptlrpc_thread for osp_sync_thread()

Instead of ptlrpc_thread, just use native kthread functionality.

- Move allocations to before the thread is created, so error
  reporting is easier.
- add a completion so that we can be sure that the thread is stared,
  and hence know that cleanup will eventually happen.
- use kthread_stop() to tell the thread to stop, and
  kthread_should_stop() to see if it has bene told.
- clear ->opd_sync_task before calling kthread_stop(), and
  use a test for NULL to see if the thread is stopping

- As some error conditions can cause the thread to exit before being
  asked, and as kthread_stop() doesn't handle this possibility, add
  an extra handshake.
  Both thread-exit and osp_sync_fini() atomically exchange NULL for
  the thread pointer.
  If thread-exit is first, the exit proceeds and osp_sync_fini() never
  waits.  For osp_sync_fini() is first, it calls kthread_exit() to
  wait, and thread-exit doesn't happen until it is sure that
  kthread_stop() is waiting.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I952cbdae90252476853854d89b8f1a78d95ddbde
Reviewed-on: https://review.whamcloud.com/36265
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
11 months agoLU-11025 tests: only set crush for newer builds 07/38107/4
Andreas Dilger [Tue, 31 Mar 2020 23:25:15 +0000 (17:25 -0600)]
LU-11025 tests: only set crush for newer builds

Don't set default DNE hash type to crush if MDS doesn't support it.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I1c91f1149d9472bbe1f30ce3f01565835ed7ab01
Reviewed-on: https://review.whamcloud.com/38107
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
11 months agoLU-11814 obd: Crashed while mount in parallel 74/37974/10
Yang Sheng [Tue, 10 Mar 2020 13:53:26 +0000 (21:53 +0800)]
LU-11814 obd: Crashed while mount in parallel

Ensure obd_type init finished before obd_setup was started.
Also included a test case to reproduce the failure.

 (osd_handler.c:7132:osd_device_init0()) ASSERTION( info ) failed:
 (osd_handler.c:7132:osd_device_init0()) LBUG
 Pid: 2457, comm: mount.lustre 3.10.0-862.9.1.el7_lustre.ddn1.x86_64
Call Trace:
 [<ffffffffc042f7cc>] libcfs_call_trace+0x8c/0xc0 [libcfs]
 [<ffffffffc042f87c>] lbug_with_loc+0x4c/0xa0 [libcfs]
 [<ffffffffc10cc6d5>] osd_device_alloc+0x615/0x770 [osd_ldiskfs]
 [<ffffffffc099ae2a>] obd_setup+0x11a/0x2b0 [obdclass]
 [<ffffffffc099cc58>] class_setup+0x2a8/0x840 [obdclass]
 [<ffffffffc09a08ed>] class_process_config+0x196d/0x2420 [obdclass]
 [<ffffffffc09a45f8>] do_lcfg+0x258/0x500 [obdclass]
 [<ffffffffc09a8e68>] lustre_start_simple+0x88/0x210 [obdclass]
 [<ffffffffc09d59b4>] server_fill_super+0xf34/0x185a [obdclass]
 [<ffffffffc09abfe8>] lustre_fill_super+0x328/0x950 [obdclass]
 [<ffffffff9781f3bf>] mount_nodev+0x4f/0xb0
 [<ffffffffc09a4008>] lustre_mount+0x38/0x60 [obdclass]
 [<ffffffff9781ff3e>] mount_fs+0x3e/0x1b0
 [<ffffffff9783d4c7>] vfs_kern_mount+0x67/0x110
 [<ffffffff9783faef>] do_mount+0x1ef/0xce0
 [<ffffffff97840923>] SyS_mount+0x83/0xd0
 [<ffffffff97d20795>] system_call_fastpath+0x1c/0x21

Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: If3129df5fd11226636fb84a9275f481cefb749f7
Reviewed-on: https://review.whamcloud.com/37974
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
11 months agoLU-13356 client: don't use OBD_CONNECT_MNE_SWAB 80/37880/11
Alexander Boyko [Wed, 11 Mar 2020 10:40:52 +0000 (06:40 -0400)]
LU-13356 client: don't use OBD_CONNECT_MNE_SWAB

OBD_CONNECT_MNE_SWAB is equal to OBD_CONNECT_MDS_MDS, and
it was used at MGC client in past for mne swabbing during interop.
Right now it is interpreted at MGS like OBD_CONNECT_MDS_MDS and skip
these clients from eviction and lock canceling after timeout.

Fixes: 1bdc4fd0594e ("LU-6307 obdclass: distinguish MGC/MDT connection properly")
Test-Parameters: testlist=runtests clientversion=2.12 envdefinitions=MDS_MOUNT_OPTS="-orw",OST_MOUNT_OPTS="-orw"
Test-Parameters: testlist=runtests serverversion=2.12 envdefinitions=MDS_MOUNT_OPTS="-orw",OST_MOUNT_OPTS="-orw"
Test-Parameters: testlist=runtests clientversion=2.10 clientdistro=el7.6 envdefinitions=MDS_MOUNT_OPTS="-orw",OST_MOUNT_OPTS="-orw"
Test-Parameters: testlist=runtests serverversion=2.10 serverdistro=el7.6 envdefinitions=MDS_MOUNT_OPTS="-orw",OST_MOUNT_OPTS="-orw"
Signed-off-by: Alexander Boyko <c17825@cray.com>
Cray-bug-id: LUS-8484
Change-Id: I4f8ddeb1808cfaee7507e0efcdefa24040cfcbb6
Reviewed-on: https://review.whamcloud.com/37880
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Artem Blagodarenko <c17828@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
11 months agoLU-6142 lustre: no need to check debugfs return values 48/37748/4
Mr NeilBrown [Thu, 27 Feb 2020 05:17:41 +0000 (16:17 +1100)]
LU-6142 lustre: no need to check debugfs return values

When calling debugfs functions, there is no need to ever check the
return value.  The function can work or not, but the code logic should
never do something different based on this.

Clean up the lustre core code by not caring about the value of debugfs
calls.  This ends up removing a number of lines of code that are not
needed.

Test-Parameters: trivial
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I78bddd60a8ae26418e1b4f99938b21cd652aea08
Reviewed-on: https://review.whamcloud.com/37748
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
11 months agoLU-6142 lustre: remove ldebugfs_add_simple() wrapper. 47/37747/6
Mr NeilBrown [Thu, 27 Feb 2020 05:13:59 +0000 (16:13 +1100)]
LU-6142 lustre: remove ldebugfs_add_simple() wrapper.

This is never used, so it not needed.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Id711102ba9b71fe326e7fe792cb2ed5e511328fe
Reviewed-on: https://review.whamcloud.com/37747
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
11 months agoLU-6142 lustre: remove ldebugfs_register_stats() wrapper 46/37746/6
Mr NeilBrown [Thu, 27 Feb 2020 05:11:58 +0000 (16:11 +1100)]
LU-6142 lustre: remove ldebugfs_register_stats() wrapper

It was just calling debugfs_create_file() so unwind things and just
call the real function instead.  This ends up saving a number of lines
as there was never any error handling happening anyway, so that all
can be removed as well.

Test-Parameters: trivial
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I7e30139cc8e810ab4bc49ae9c298e6199c3f2942
Reviewed-on: https://review.whamcloud.com/37746
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
11 months agoLU-6142 lustre: remove ldebugfs_seq_create wrapper function 45/37745/5
Mr NeilBrown [Thu, 27 Feb 2020 05:00:32 +0000 (16:00 +1100)]
LU-6142 lustre: remove ldebugfs_seq_create wrapper function

It was just calling debugfs_create_file() so unwind things and just
call the real function instead.  This ends up saving a number of lines
as there was never any error handling happening anyway, so that all
can be removed as well.

Test-Parameters: trivial
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I7103ef2a44348f28b958730413e93e0878eba4d2
Reviewed-on: https://review.whamcloud.com/37745
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
11 months agoLU-6142 lustre: make ldebugfs_add_vars a void function 44/37744/6
Mr NeilBrown [Thu, 27 Feb 2020 04:43:57 +0000 (15:43 +1100)]
LU-6142 lustre: make ldebugfs_add_vars a void function

The call to ldebugfs_add_vars() can not really fail, so have it just
return nothing, which allows us to clean up a lot of unused error
handling code.

Linux-Commit d5bf5cf9e150 ("staging: lustre: make ldebugfs_add_vars a
void function")

Test-Parameters: trivial
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I338c573fe5cb7dbd3595247f5c3647f5c1bf949b
Reviewed-on: https://review.whamcloud.com/37744
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
11 months agoLU-10391 socklnd: use interface index to track local addr 02/37702/11
Mr NeilBrown [Fri, 7 Feb 2020 02:05:57 +0000 (13:05 +1100)]
LU-10391 socklnd: use interface index to track local addr

socklnd currently uses local IP addresses to track the local end of a
route or connection.  This will not work so well for IPv6 where
addresses are generally more dynamic - an interface can have several
addresses and they can come and go.

Even with IPv4, an interface can have multple addresses, though this
is partially hidden as there we normally use aliases which make it
looks like different interfaces: eth0:1 eth0:2 etc.  These are really
all the same interface, just with different addresses.

It is really the local interface, rather than the local address,
which is important.  Choosing the right interface ensures the traffic
goes over the desired network hardware.  Using an address, as the code
currently does, is just a convenient indirection.

With IPv6, using the interface directly will be easier, and it is
quite possible with IPv4.

So: this patch changes ksock_route to store an interface index for the
source end of the route, and adds the index nubmer to ksock_interface.

lnet_sock_listen() loses the local IP address which is never used, and
lnet_sock_connect() now is passed a source interface rather than a
source IP address.
lock_sock_create() also gets a local source interface.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ib3e0f92d10ad6ba4d66782bae22638e9935a1e4e
Reviewed-on: https://review.whamcloud.com/37702
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Aurelien Degremont <degremoa@amazon.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
11 months agoLU-10391 lnet: simplify use of lnet_count_acceptor_nets() 95/37695/8
Mr NeilBrown [Fri, 7 Feb 2020 02:05:57 +0000 (13:05 +1100)]
LU-10391 lnet: simplify use of lnet_count_acceptor_nets()

Instead of calling this in various places before either
lnet_acceptor_start() or lnet_acceptor_stop(), only call
it in those functions.
It is already called in lnet_acceptor_start().

At shutdown, lnet_acceptor_stop() must succeed unconditionally, so
include a test on the_lnet.ln_refcount.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I11ef6c1a226e7b752b540f37651c766e83c5cb52
Reviewed-on: https://review.whamcloud.com/37695
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
11 months agoLU-9679 osc: convert oe_refc to kref 01/37601/9
NeilBrown [Wed, 12 Dec 2018 06:49:53 +0000 (17:49 +1100)]
LU-9679 osc: convert oe_refc to kref

oe_refc is used like a kref, so make it one.

As cl_object_put() needs an 'env', we cannot quite use
kref_put() as intended.  Maybe that can be fixed one day.

This requires kref_read() to be provided for older kernels.

The original Linux-commit also converted oe_users to a refcount_t
but that was later reverted.

Linux-Commit: 35ea2a7c694b ("lustre: osc: convert oe_refc and oe_users
to kref and refcount_")

Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I42fd17c4261b85b273387936b97fce463273e2b8
Reviewed-on: https://review.whamcloud.com/37601
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
11 months agoLU-9679 lustre: add OBD_ALLOC_PTR_ARRAY() and others 76/36976/12
Mr NeilBrown [Thu, 14 Nov 2019 03:20:01 +0000 (14:20 +1100)]
LU-9679 lustre: add OBD_ALLOC_PTR_ARRAY() and others

Similar to CFS_ALLOC_PTR_ARRAY(), add
  OBD_ALLOC_PTR_ARRAY
  OBD_ALLOC_PTR_ARRAY_WAIT
  OBD_FREE_PTR_ARRAY
  OBD_ALLOC_PTR_ARRAY_LARGE
  OBD_FREE_PTR_ARRAY_LARGE

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I253d4497ed7cbbe1afc5c66c2aa753bb272a4ad5
Reviewed-on: https://review.whamcloud.com/36976
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
11 months agoLU-9868 llite: Get rid of ll_dcompare 75/24175/5
Oleg Drokin [Fri, 3 Apr 2020 03:24:04 +0000 (23:24 -0400)]
LU-9868 llite: Get rid of ll_dcompare

The problem with real_lookup, and real_lookup itself are long gone,
so let's kick out invalid dentries in revalidate now.

Change-Id: Ieb274645245a19ac4917e0ec4643a0c44aa32cd3
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: https://review.whamcloud.com/24175
Reviewed-by: Neil Brown <neilb@suse.de>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
11 months agoLU-13127 libcfs: make noise to console if CRC32 is missing 14/37214/9
Alex Zhuravlev [Mon, 13 Jan 2020 09:37:44 +0000 (12:37 +0300)]
LU-13127 libcfs: make noise to console if CRC32 is missing

if CRC32 support is disabled in the kernel by mistake, then
early replies don't work and the actual cause isn't obvious.

the patch also adds a check at configure to ensure that
target kernel support CRC32 crypto function.

Change-Id: I17999ae89eaf6c4fb7c7db12200e2092cc6afea8
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/37214
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
12 months agoLU-6202 obd: replace OBD_IOC_NO_TRANSNO with sysfs file 93/35993/22
James Simmons [Wed, 25 Mar 2020 11:57:40 +0000 (07:57 -0400)]
LU-6202 obd: replace OBD_IOC_NO_TRANSNO with sysfs file

We can replace the ioctl OBD_IOC_NO_TRANSNO, which is server side
only, with a sysfs file. The original ioctl set an internal obd
device flags to simulate transactions not reaching the disk. The
new write only sysfs file behaves the exact same way.

Change-Id: Ie78feec61fab46a1982bd5caca3e89613356d2e8
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/35993
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13274 uapi: make lnet UAPI headers C99 compliant 71/37971/11
James Simmons [Tue, 24 Mar 2020 18:49:48 +0000 (14:49 -0400)]
LU-13274 uapi: make lnet UAPI headers C99 compliant

Attempting to compile strict C99 user land applications or
libraries with the Lustre UAPI headers will fail. These same
errors can be seen by enabling CONFIG_UAPI_HEADER_TEST as well.
Update the LNet UAPI headers to be compilable with -std=c99.
Add LNet test covering UAPI header handling.

Test-Parameters: trivial testlist=sanity-lnet

Change-Id: Ic1c30ed2d4d9d1d7f11b657d3817a05161367e13
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/37971
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13386 llite: allow current readahead to exceed reservation 47/38047/4
Wang Shilong [Wed, 25 Mar 2020 01:14:07 +0000 (09:14 +0800)]
LU-13386 llite: allow current readahead to exceed reservation

ll_ra_count_get() is designed to allow over reservation for performance
reasons, this could happen that multiple threads consuming a small
readahead window limit, it will fallback to small 4K read which hurts
performances.

Fixes: 8fbef5e("LU-12043 llite: make sure readahead cover current read")
Change-Id: Ied4ee0ffcd55c568c70f7d3074e810cd0034ef04
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/38047
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13385 lnet: Calculate IB service-id 45/38045/6
Amir Shehata [Tue, 24 Mar 2020 22:18:18 +0000 (15:18 -0700)]
LU-13385 lnet: Calculate IB service-id

The service-id is determined by the:
1. port space (currently hard coded to TCP)
2. service port number (defaults to 987)

options ko2iblnd service=<num>

Can be used to change the service port number. However, when
configuring QoS, we need to know the service-id, which can then
be mapped to the service level in the opensm's qos-policy.conf file.

This patch adds a new lnetctl command which returns the service-id.

lnetctl service-id

Test-Parameters: trivial
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: Ifab46473088239a391314040bac91d0f791ea3e4
Reviewed-on: https://review.whamcloud.com/38045
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Olaf Weber <olaf.weber@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13377 llite: fix dead loop for short write 18/38018/6
Wang Shilong [Sat, 21 Mar 2020 01:58:09 +0000 (09:58 +0800)]
LU-13377 llite: fix dead loop for short write

|->vvp_io_write_start()
 |->__generic_file_write_iter()
    |->iov_iter_advance() if write succeed()
  |->vvp_io_write_commit()
     |->update ci_nob

The problem is we will move forward iov iter inside
__generic_file_write_iter(), but @ci_nob will be updated
after vvp_io_write_commit().

If out of quota or some other problems happen, this could
cause a mismatch with @ci_nob and @vui_iter.

And @vui_iter->count will be reset using @ci_nob in
iov_iter_reexpand(), this will make @vui_iter->count
more than what it really left, and we could dead loop
in vvp_mmap_locks() if IO need be retried or restarted:

vvp_io_write_lock+0x45/0x80 [lustre]
cl_io_lock+0x5f/0x3d0 [obdclass]
cl_io_loop+0x92/0x190 [obdclass]
ll_file_io_generic+0x7b3/0xc90 [lustre]
ll_file_aio_write+0x12d/0x1f0 [lustre]
ll_file_write+0xce/0x1e0 [lustre]
vfs_write+0xc0/0x1f0
SyS_write+0x7f/0xf0
system_call_fastpath+0x22/0x27

Change-Id: I5fb4c18cf02fb17bf50122b63decacef678caa01
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/38018
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
12 months agoLU-13275 tests: enhance racer with failover 52/37652/5
Elena Gryaznova [Thu, 20 Feb 2020 18:50:44 +0000 (21:50 +0300)]
LU-13275 tests: enhance racer with failover

Patch adds the possibilty to run facet_failovier <random_facet>
if RACER_FAILOVER is set to true.
The type of facets are set over FAIL_TARGETS variable,
default value is "MDS OST".

Test-Parameters: trivial \
envdefinitions=RACER_FAILOVER=true testlist=racer
Cray-bug-id: LUS-8280
Signed-off-by: Elena Gryaznova <c17455@cray.com>
Reviewed-by: Vladimir Saveliev <vladimir.saveliev@hpe.com>
Reviewed-by: Andriy Skulysh <andriy.skulysh@hpe.com>
Change-Id: Iacb6ac855bf77d7c6930fdbb7ec192ecc46264b3
Reviewed-on: https://review.whamcloud.com/37652
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Vladimir Saveliev <c17830@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13272 tests: customise dom-performance 50/37650/7
Elena Gryaznova [Thu, 20 Feb 2020 18:01:40 +0000 (21:01 +0300)]
LU-13272 tests: customise dom-performance

dom-performance compares the performance benefits
between the normal file and DoM file.
However it is also interesting to demostrate that if
the file grows beyond the DoM size the performance
for it is not worse compared to the normal file.

Patch allows to set file size beyond the DoM size.

Test-Parameters: trivial mdssizegb=20 \
testlist=dom-performance envdefinitions=ONLY=IOR
Cray-bug-id: LUS-7359
Signed-off-by: Elena Gryaznova <elena.gryaznova@hpe.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Change-Id: I2412b4f1e1d0a591c47a9ceadc37f730c4deedbe
Reviewed-on: https://review.whamcloud.com/37650
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-10395 tests: add test_280 sanity 35/37635/6
Alexander Boyko [Thu, 20 Feb 2020 10:12:03 +0000 (05:12 -0500)]
LU-10395 tests: add test_280 sanity

The test reproduce a race between client llog processing
and MGS umount.

osd_fid2oi()) ASSERTION( osd->od_oi_table != NULL &&
osd->od_oi_count >= 1 ) failed: [0xa:0xb:0x0]

 #3 lbug_with_loc at ffffffffc0c098cb [libcfs]
 #4 __osd_oi_lookup at ffffffffc13e5610 [osd_ldiskfs]
 #5 osd_oi_lookup at ffffffffc13e76fd [osd_ldiskfs]
 #6 osd_fid_lookup at ffffffffc13e2cc5 [osd_ldiskfs]
 #7 osd_object_init at ffffffffc13e44e1 [osd_ldiskfs]
 #8 lu_object_start at ffffffffc0e262ab [obdclass]
 #9 lu_object_find_at at ffffffffc0e2a121 [obdclass]
 #10 dt_locate_at at ffffffffc0e2b6dd [obdclass]
 #11 llog_osd_open at ffffffffc0deae9e [obdclass]
 #12 llog_open at ffffffffc0dd8f3a [obdclass]
 #13 llog_origin_handle_next_block at  ffffffffc1200a06[ptlrpc]
 #14 tgt_llog_next_block at ffffffffc124b6c3 [ptlrpc]

Cray-bug-id: LUS-8505
Test-Parameters: standalonemgs=false trivial testlist=sanity envdefinitions=ONLY=280,ONLY_REPEAT=5
Signed-off-by: Alexander Boyko <c17825@cray.com>
Change-Id: I449b18779bfe391784ad532451213d5d5456c1b1
Reviewed-on: https://review.whamcloud.com/37635
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-10395 osd: stop OI at device shutdown 15/37615/5
Alex Zhuravlev [Tue, 18 Feb 2020 15:04:44 +0000 (18:04 +0300)]
LU-10395 osd: stop OI at device shutdown

and not at obd_cleanup(). otherwise a race is possible:
 umount <MDT> stopping OI vs MGS accessing same OSD which
results in the assertion:
ASSERTION( osd->od_oi_table != NULL && osd->od_oi_count >= 1 )

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I24fccea718f2e2663166cfb0ff26571039357535
Reviewed-on: https://review.whamcloud.com/37615
Reviewed-by: Alexander Boyko <c17825@cray.com>
Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-930 doc: update MAINTAINERS and mailmap 49/37549/7
Sergey Cheremencev [Wed, 12 Feb 2020 12:22:40 +0000 (15:22 +0300)]
LU-930 doc: update MAINTAINERS and mailmap

Change cray emails with new hpe.
Add new reviewers from HPE.

Change-Id: I7b5b3270fa16beec84c50ffdf4dc0fb7f85db761
Test-Parameters: trivial
Signed-off-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-on: https://review.whamcloud.com/37549
Reviewed-by: Andrew Perepechko <andrew.perepechko@hpe.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13224 utils: expose llapi_param* functions 45/37545/15
Gian-Carlo DeFazio [Tue, 11 Feb 2020 21:08:18 +0000 (13:08 -0800)]
LU-13224 utils: expose llapi_param* functions

Added an interface to find files in sysfs,procfs,etc.,
and read them into a file or buffer.
The interface is meant to return the same results as
the "lctl list_param" and "lctl get_param" commands,
but doesn't require starting an lctl process.

The interface is being used by lctl for parts of
its "list_param" and "get_param" commands.

The output of lctl get_param is altered slightly.
Previously, for some multi-line files lctl would
print the param name and first line of the file on
the same line such as:

param=line 1
line 2
line 3

Now multi-line files consistently put a newline
after the param name and before the content such as:

param=
line 1
line 2
line 3

Added debug output to failing test sanity 300d.

Signed-off-by: Gian-Carlo DeFazio <defazio1@llnl.gov>
Change-Id: I2726b27b0042d58c97284f8348970deb6efc43d1
Reviewed-on: https://review.whamcloud.com/37545
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13134 clio: cl_page index compaction 17/37417/5
Patrick Farrell [Tue, 4 Feb 2020 01:05:27 +0000 (20:05 -0500)]
LU-13134 clio: cl_page index compaction

The cpl_index field is not necessary for the vvp layer,
since that's the same as the vm page index, and it has
already been duplicated for the lov layer.

So, moving the osc usage of this to the top, we can save
24 bytes per page (in the slices) by removing this from
the cl_page_slice, at the cost of 8 bytes in the cl_page.
So a net savings of 16 bytes.

After this patch, cl_page could be reduced from 408 to
392 bytes.

Signed-off-by: Patrick Farrell <farr0186@gmail.com>
Change-Id: Ib4fd75f112afb816443799314cd2ef75ff4f4c84
Reviewed-on: https://review.whamcloud.com/37417
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13134 obdclass: use slab allocation for cl_page 25/37225/17
Wang Shilong [Thu, 16 Jan 2020 07:05:08 +0000 (15:05 +0800)]
LU-13134 obdclass: use slab allocation for cl_page

Currently we use kmalloc() for cl_page allocation, this is because
cl_page size could be different with different objects.

For most cases (except obdecho object) we have 408 bytes for cl_page.
kmalloc() uses 512 bytes slab for it. So for 4KB PAGE_SIZE, we could
allocate 8 cl_page in a single page.

If we use private slab cache, it can fit 10 cl_page for 4KB PAGE_SIZE,
and even more struct cl_page can fit if we reduce the struct size.
For 4GB of pages the cl_page memory usage will shrink from 512M to 408M.

With clients already having 100GB of memory, it will save us 2.5GB
memory usage for cl_page, and some systems have as much as 24TB today!

This patch uses a static array with N (currently 16) entries storing
the { size, slab pointer }, with only 2-3 entries in current usage.
If the static array is not large enough, then fall back to kmalloc().

Benchmark numbers:

    4KB Random Read
    fio -iodepth=128 -direct=1 -size=2g -runtime=60 -numjobs=256
        -group_reporting  -directory=/ai400/out -create_serialize=0
        -filename_format='f.$jobnum.$filenum'

                        master+     master+
                        LU-4198     LU-4198+
                                    LU-13134   delta
      QD=1, numjobs=1     4518       4538      +0.44%
    QD=128, numjobs=256   2177K      2207k     +1.38%

Change-Id: I565b09616b22706f93d4c0fdc0df396d06cd51cc
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/37225
Reviewed-by: Neil Brown <neilb@suse.de>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
12 months agoLU-13258 llite: bind readahead workqueue to CPT set 17/37717/9
James Simmons [Wed, 25 Mar 2020 02:47:00 +0000 (22:47 -0400)]
LU-13258 llite: bind readahead workqueue to CPT set

A workqueue is used by Lustre to optimize readahead. This work
queue can run on any core and can easily be over surscribed. This
will have a negative impact on HPC applications running on a
Lustre client. Limit the number of threads a workqueue can run
to the size of the CPU allocated for Lustre and only allow those
threads to run on the cores belonging to the CPT set.

Change-Id: Ifcc662d52843f5028c34d55695c1d6297e5c00b0
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/37717
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Stephen Champion <stephen.champion@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-12846 tests: verify orphan upgrade compatibilty 36/37636/7
Artem Blagodarenko [Wed, 19 Feb 2020 16:38:31 +0000 (11:38 -0500)]
LU-12846 tests: verify orphan upgrade compatibilty

There was a discussion in LU-13070 that name compatibility
is useless, because orphan's name is not taken into account
in orphans deletion loop.

This test shows that orphan with a name format before 2.11
can not be deleted in mdd_orphan_destroy() that can lead
to an endless loop:

    mdd_orphan_destroy() t32fs-MDD0000: could not delete
        orphan [0x200000bd0:0x1:0x0]: rc = -2

Print a banner in each test_32 subtest to separate images.

Test-Parameters: trivial testlist=conf-sanity envdefinitions=ONLY=32,ONLY_REPEAT=4
Signed-off-by: Artem Blagodarenko <artem.blagodarenko@hpe.com>
Change-Id: I351ff3e5f3d10dabfd76cb1c9f07d3fc597004a6
Reviewed-on: https://review.whamcloud.com/37636
Tested-by: Maloo <maloo@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
12 months agoLU-12780 quota: don't use ptlrpc_thread for qmt_reba_thread 56/36556/6
Mr NeilBrown [Wed, 23 Oct 2019 00:30:52 +0000 (11:30 +1100)]
LU-12780 quota: don't use ptlrpc_thread for qmt_reba_thread

Instead of ptlrpc_thread, use native kthread functionality.

- for startup, perform allocations before creating the thread so that
  once created it cannot fail.
  We still use a completion to ensure the thread function runs before
  kthread_stop is called, so that cleanup will happen.

- As lu_env_add() can fail, and needs to know which task owns
  the env, we add a new function lu_env_add_task()

- for shutdown, use kthread_stop() and kthread_should_stop()

- to alert thread of a new event, use wake_up_process()
  Do this under qmt_reba_lock so the thread cannot disappear while
  being woken.
  The thread sets TASK_IDLE, then if any tests show there is work
  to do it sets TASK_RUNNING, so the following schedule() won't
  block and the loop will check again.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: If3f0444d5aa38ea84990d95f85ad18202f99d5df
Reviewed-on: https://review.whamcloud.com/36556
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13131 osc: Always send all HP RPCs requests 57/38057/2
Oleg Drokin [Fri, 7 Feb 2020 01:21:43 +0000 (20:21 -0500)]
LU-13131 osc: Always send all HP RPCs requests

that are resulting from lock cancel activity.

Change-Id: I4007167f0f39b0699e977c14bd160f475d8288ad
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/38057
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
12 months agoLU-6174 lod: remove incorrect ll_do_div64 01/38101/2
James Simmons [Mon, 30 Mar 2020 17:06:19 +0000 (13:06 -0400)]
LU-6174 lod: remove incorrect ll_do_div64

ll_do_div64() is just a wrapper around do_div() which is handled
incorrectly. do_div() expects a 32 bit value but stripe_count is
cast to 64 bits which is incorrect. Instead lets just avoid
casting and use do_div() directly.

Change-Id: I08601fdf8e9d5fdf26981f073dd602775c12281b
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/38101
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
12 months agoNew tag 2.13.53 2.13.53 v2_13_53
Oleg Drokin [Mon, 6 Apr 2020 20:55:59 +0000 (16:55 -0400)]
New tag 2.13.53

Change-Id: I0ac2858038f2710a19189d4b621064355ccbbb7d
Signed-off-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13379 tests: don't use localrecov for older servers 22/38022/6
Andreas Dilger [Sun, 22 Mar 2020 07:53:22 +0000 (01:53 -0600)]
LU-13379 tests: don't use localrecov for older servers

Don't add the "-o localrecov" mount option to older MDT and OST
mounts, since that option is not supported there.

Fixes: 8bd04b4e5766 ("LU-12722 target: disable recovery for local clients")
Test-Parameters: trivial testlist=runtests serverversion=2.12
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Idc0ba22c9b648d49515f5e0f2d8271c8573ebbe5
Reviewed-on: https://review.whamcloud.com/38022
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-6179 llite: remove LOCKAHEAD_OLD compatibility 09/38109/2
Andreas Dilger [Wed, 1 Apr 2020 01:14:00 +0000 (19:14 -0600)]
LU-6179 llite: remove LOCKAHEAD_OLD compatibility

The OBD_CONNECT_LOCKAHEAD_OLD feature was added for a short time for
compatibility with an implementation of LDLM lockahead that was later
replaced by OBD_CONNECT_LOCKAHEAD2.

Remove the compatibility code for this old implementation that has
been disabled since the 2.13 release.

Also remove other obsolete compatibility code dating back to 2.8.53.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I4444eff180b2c6e2b27d260413f2debbb2ce7057
Reviewed-on: https://review.whamcloud.com/38109
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
12 months agoLU-8454 Remove struct mgs_send_param and its users 06/38106/4
Oleg Drokin [Wed, 1 Apr 2020 00:31:24 +0000 (20:31 -0400)]
LU-8454 Remove struct mgs_send_param and its users

This code has been somewhat dormant since 2.9 and now it's
time to remove it.

Change-Id: I392c29d30e11b4feb362ac2f396a2f687c81c52a
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/38106
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
12 months agoLU-13321 tests: force even DNE file distribution 29/38129/5
Andreas Dilger [Fri, 3 Apr 2020 00:54:11 +0000 (18:54 -0600)]
LU-13321 tests: force even DNE file distribution

Force even file creation across DNE-striped directories in sanity
test_160f to avoid random test failures (about 15%) when no files
are created on a specific MDT.

Fixes: 0a1cf8da8069 ("LU-11025 dne: introduce new directory hash type: 'crush'")
Test-Parameters: trivial testlist=sanity env=ONLY=160f,ONLY_REPEAT=50
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I8bead0c20efeda90b9b89f933a022cc18e3ebbe5
Reviewed-on: https://review.whamcloud.com/38129
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-12312 tests: stop running sanity-sec test 31 28/38128/2
James Nunez [Thu, 2 Apr 2020 17:59:32 +0000 (11:59 -0600)]
LU-12312 tests: stop running sanity-sec test 31

sanity-sec test 31 is failing a high percent of time
and the failure is not understood.  Let's stop running
this test, add the test to the ALWAYS_EXCEPT list, until
we have a fix for this issue.

Test-Parameters: trivial testlist=sanity-sec
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I07c3cb1b5d12c920f78efb942460608c664000b3
Reviewed-on: https://review.whamcloud.com/38128
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13226 ldiskfs: Add support for SUSE 15 SP2 90/37490/7
Shaun Tancheff [Wed, 12 Feb 2020 20:18:07 +0000 (14:18 -0600)]
LU-13226 ldiskfs: Add support for SUSE 15 SP2

SUSE 15 SP2 kernel is close enough to 5.4.7+ mainline to
use the patch series directly.
Update the configure script to select it.

Test-Parameters: trivial
Cray-bug-id: LUS-8485
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I10e5b9eef6783ac153429be602bbda8d870c46d5
Reviewed-on: https://review.whamcloud.com/37490
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13344 servers: change request timeouts to s32 50/37950/2
James Simmons [Tue, 17 Mar 2020 11:31:08 +0000 (07:31 -0400)]
LU-13344 servers: change request timeouts to s32

In the latest kernels time_t has been removed since time_t has
been a 64 bit value just like time64_t so no need for it anymore.
The field sf_run_time is stored on local media and its expected
to be 32 bit in size so change it to s32 explictly. The fields
the MDT coordinator that are currently time_t are a simple change
to s32 since they represent a time period and not a time stamp.
Lastly just update a comment in llog_swab.c since time_t will no
longer exist but we want to point out time in the llog is 32 bits
in size.

Change-Id: I0849fc9798c191d1fed77f7bba1357a209924418
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/37950
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13344 sec: remove time_t usage 42/37942/4
James Simmons [Fri, 20 Mar 2020 23:49:06 +0000 (19:49 -0400)]
LU-13344 sec: remove time_t usage

In the latest kernels time_t has been removed since time_t has
been a 64 bit value just like time64_t so no need for it anymore.
Change ps_sepol_mtime to ktime_t which is the preferred time
format in the linux kernel. For the case of the Lustre GGS code
the fields marked as time_t are timestamps so just change it to
time64_t.

Change-Id: I9f40592d440cc8d68c7b73c0c31d88bcecb3723b
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/37942
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Tested-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13344 ptlrpc: change request timeouts to s32 33/37933/6
James Simmons [Fri, 20 Mar 2020 21:40:25 +0000 (17:40 -0400)]
LU-13344 ptlrpc: change request timeouts to s32

In the latest kernels time_t has been removed since time_t has
been a 64 bit value just like time64_t so no need for it anymore.
This means all kernel time keeping are all 64 bits but Lustre
sends 32 bit values, in seconds, for its timeout between nodes.
To keep backwards compatibility as well as distinguish timeouts
from timestamps change the Lustre fields representing timeouts
to a new typedef timeout_t which is a s32. The reason for picking
a signed number is that it makes it easier to detect if an
incorrect timeout was calculated. In the past negative timeouts
have happened which would look like an enormous timeout if its
treated as an unsigned number.

Change-Id: I7491bbac65023c544e8f4a0488b1d07315a739e5
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/37933
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13309 ofd: optimize the brw codepath 95/37795/3
Andrew Perepechko [Fri, 13 Mar 2020 09:44:24 +0000 (12:44 +0300)]
LU-13309 ofd: optimize the brw codepath

1) currently, ofd_commitrw() calls lvbo_update()
from the read path with req == NULL.

The only effect of this call is lvb update from
disk. However, the read request does not update
inode info such as atime/mtime/ctime/i_size, so
there's no need for lvb update.

The comment above the removed code claims that
ofd_preprw_read() updated ondisk atime. In fact,
there's no code in the read path which updates
atime. Atime update was implemented by
obdo_to_inode() before we moved to osd.

We don't seem to suffer from this loss, so
attr_get and lvb update, which currently do nothing,
can be simply dropped.

2) ofd_commitrw_*() calls ofd_object_find()
whereas the corresponding preprw call has not
only received the pointer to the object, but
also keeps a reference on it when exits so
ofd_commitrw_*() has to put it twice.

We don't really need to look up the needed
object twice, just record it in the ofd info
and then take it from there.

Change-Id: If9692906e6c8c4e1254d166620d1ea7e68a933b6
Signed-off-by: Andrew Perepechko <c17827@cray.com>
Cray-bug-id: LUS-8127
Reviewed-on: https://review.whamcloud.com/37795
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexander Boyko <c17825@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13324 llite: page delete race 93/37793/4
Bobi Jam [Tue, 10 Mar 2020 13:12:53 +0000 (21:12 +0800)]
LU-13324 llite: page delete race

Page could be raced by truncate during its deletion by
another thread, check vmpage mapping before accessing it.

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I3172f5ffd3928926b16ab6fd7362b05da0c7cfd5
Reviewed-on: https://review.whamcloud.com/37793
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
12 months agoLU-12542 lustre: remove INIT_LIST_HEAD_RCU support. 89/37789/2
Mr NeilBrown [Wed, 4 Mar 2020 03:31:16 +0000 (14:31 +1100)]
LU-12542 lustre: remove INIT_LIST_HEAD_RCU support.

Not all kernels provide INIT_LIST_HEAD_RCU() so it is provided for
older kernels.  But lustre no longer uses this macro and is unlikely
to - it is rarely used at all.

So remove the back-compat support.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I0556e9c75c575ee269f253a90600788747578593
Reviewed-on: https://review.whamcloud.com/37789
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13255 gnilnd: Use wait_var_event_warning() 94/37594/6
Mr NeilBrown [Thu, 7 Nov 2019 06:00:07 +0000 (17:00 +1100)]
LU-13255 gnilnd: Use wait_var_event_warning()

LNet has a recurring pattern of waiting for some variable to
reach a particular value, and generating a warning every second that
it hasn't.  In many cases the warning has a higher priority if the
wait has been for a power-of-2 seconds.

This pattern is now  embodied in
  wait_var_event_warning()

Use that macro in various places in gnilnd.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I7be82a175c7b1b8da4df99199378e7af5328e0e7
Reviewed-on: https://review.whamcloud.com/37594
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13157 mdd: migrate readlink from wrong place 85/37285/4
Lai Siyao [Mon, 20 Jan 2020 04:23:45 +0000 (12:23 +0800)]
LU-13157 mdd: migrate readlink from wrong place

In osd_read(), if symlink name buf length is smaller than i_size,
return -EOVERFLOW, and compare inline data length with i_size
instead of buflen.

Updated sanity.sh 230b.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Ia13b1f1079efc4ebd22ec400f1a909ff7ec2095d
Reviewed-on: https://review.whamcloud.com/37285
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
12 months agoLU-9091 procfs: remove old string parsing routines 75/37275/4
Andreas Dilger [Tue, 24 Mar 2020 13:57:39 +0000 (09:57 -0400)]
LU-9091 procfs: remove old string parsing routines

Remove the llprocfs_str_with_units_to_s64(), lu_str_to_s64(),
llprocfs_str_with_units_to_u64(), and lu_str_to_u64() functions.
All new code should use sysfs_memparse() to parse sizes from
userspace.

Mark patch forbuildonly so that it does not land before we have
a chance to remove any patches with users of these functions.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I6dc321e60eb587ef25652f4b5ae8fa478d3ebbe5
Reviewed-on: https://review.whamcloud.com/37275
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13004 net: discard LNET_MD_IOVEC 74/36974/6
Mr NeilBrown [Thu, 21 Nov 2019 04:26:04 +0000 (15:26 +1100)]
LU-13004 net: discard LNET_MD_IOVEC

This flag is now never set, so discard it.  lnet_md memory descriptors
are either a single virtual address, or a list of bvec.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I9b2280f801108e5d9fc718c90c83692042991f12
Reviewed-on: https://review.whamcloud.com/36974
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-12292 lnet: keep health even if recovery failed 21/36921/11
Amir Shehata [Tue, 3 Dec 2019 17:22:03 +0000 (09:22 -0800)]
LU-12292 lnet: keep health even if recovery failed

Don't decrement the interface's health value when recovery
message fails. If we've already determined that an interface
is unhealthy, there is no need to continue decrementing
it's health every 1 second. It'll take too long to come back
into service when it becomes healthy.

Clean up where health is decremented in order not to have
repetitive decrements. No need to decrement in lnet_notify()
because in order for the LND to call this an existing transmit
must've failed. This means a message has already failed which
will result in the health being decremented.

When a recovery send fails make sure to flag the recovery as
failed because there is no reply expected in this case.

Test-parameters: trivial

Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: Ifb3500a77a5a5be51e7079269c8ddba85ed0c2a7
Reviewed-on: https://review.whamcloud.com/36921
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-12303 lnet: recover health at same rate as dec 20/36920/9
Amir Shehata [Tue, 3 Dec 2019 01:09:07 +0000 (17:09 -0800)]
LU-12303 lnet: recover health at same rate as dec

When there is a failure to send to or over an interface the
interface's health value is decremented by lnet_health_sensitivity.
Originally, when it recovers the health value goes up by 1.
This patch makes incrementing and decrementing health value
symmetrical, by lnet_health_sensitivity.

In this way if a site wants to stop using and start using an interface
quickly, then it can set lnet_health_sensitivity to a large value
compared to the health value maximum, which is hard coded to 1000.
For example setting lnet_health_sensitivity to 500 will reduce the
health value of an interface to 0 after two failed sends and
reset it to maximum health value after two successful sends.

Test-parameters: trivial

Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: Ib5cedb063d9ccf79c574edac291551f8c94bcce4
Reviewed-on: https://review.whamcloud.com/36920
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13028 lnet: advertise discovery when toggled 19/36919/9
Amir Shehata [Mon, 2 Dec 2019 01:05:24 +0000 (17:05 -0800)]
LU-13028 lnet: advertise discovery when toggled

When discovery is toggled send a push message to all peers.
When a node receives a push notification that discovery is
turned off while previously it was on, then delete the peer
information. If the peer is a router, recreate the routes.

Test-parameters: trivial

Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: I58f9f42542e4c05763128d7c9d23108c3e7f13a3
Reviewed-on: https://review.whamcloud.com/36919
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-8130 lu_object: use RCU to free lu_object_header 65/37965/2
Mr NeilBrown [Wed, 18 Mar 2020 02:55:19 +0000 (13:55 +1100)]
LU-8130 lu_object: use RCU to free lu_object_header

Any struct containing an lu_object_header is now freed using
call_rcu() or rcu_kfree() when freed by loo_object_free.

This will allow RCU protocols to protect lookups once we
switch the lu_object cache to rhashtable in a future patch.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ib0e1134ae2ac73532d6f953ff4510076f1d8a78a
Reviewed-on: https://review.whamcloud.com/37965
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-12780 quota: don't use ptlrpc_thread for qsd_upd_thread() 67/36267/11
Mr NeilBrown [Wed, 23 Oct 2019 00:30:51 +0000 (11:30 +1100)]
LU-12780 quota: don't use ptlrpc_thread for qsd_upd_thread()

Instead of ptlrpc_thread, use native kthread functionality.

- for startup, perform allocations before starting the thread,
  and use a completion to ensure thread function runs before
  kthread_stop() is ever called, so cleanup is guaranteed to
  happen.
- for shutdown, use kthread_stop/kthread_should_stop
- for signalling the thread, use wake_up_process.  The thread
  sets TASK_IDLE while checking if there is anything to do,
  and TASK_RUNNING when it finds something to work on, so
  the schedule_timeout() only blocks if there was nothing to do.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I2ec2e28320e14251f342b8749a1a738f136f3575
Reviewed-on: https://review.whamcloud.com/36267
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
12 months agoLU-12390 tests: customise mdsrate tests 68/35068/4
Elena Gryaznova [Wed, 5 Jun 2019 11:46:18 +0000 (14:46 +0300)]
LU-12390 tests: customise mdsrate tests

Patch adds the following customizations:
  set custom layout for filesystem root
  directory by specifying fs_STRIPEPARAMS
  and
  set the custom layout for mdsrate test's
  working directory BASEDIR by specifying
  mdsrate_STRIPEPARAMS.

Test-Parameters: trivial testlist=performance-sanity
Signed-off-by: Elena Gryaznova <c17455@cray.com>
Cray-bug-id: LUS-7249
Reviewed-by: Alexander Boyko <c17825@cray.com>
Reviewed-by: Vladimir Saveliev <c17830@cray.com>
Change-Id: I28524c203f86e7e94a4db47a8d8c4b3518883600
Reviewed-on: https://review.whamcloud.com/35068
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-11596 tests: wait for grant to stop changing 76/37876/3
Andreas Dilger [Wed, 11 Mar 2020 04:52:35 +0000 (22:52 -0600)]
LU-11596 tests: wait for grant to stop changing

Wait for the client and server grant to stop changing in sanity
test-42d, test_64[acd] to avoid intermittent failures.

Test-Parameters: trivial testlist=sanity env=ONLY=64a,ONLY_REPEAT=100
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I3ef3a93d04e73ffb0098f712373c1462c53ebbe5
Reviewed-on: https://review.whamcloud.com/37876
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-11025 lmv: simplify name to stripe mapping 11/37711/4
Lai Siyao [Sat, 15 Feb 2020 02:48:59 +0000 (10:48 +0800)]
LU-11025 lmv: simplify name to stripe mapping

Handle layout change internally when mapping name to stripe:
* Move layout changing related code into lmv name to stripe mapping
  so callers doesn't need to take care of the internals.
* lmv_name_to_stripe_index() maps name in new layout, and
  lmv_name_to_stripe_index_old() in old layout.
* rename lmv_migrate_existence_check() to lmv_old_layout_lookup()
  to support directory restripe in the future.
* support layout changing directory in LFSCK.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Icf8bda5db884784f761a2d373a6f81d7e13f525f
Reviewed-on: https://review.whamcloud.com/37711
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-11025 dne: change dir layout via dt_layout_change 34/37634/5
Lai Siyao [Fri, 14 Feb 2020 09:37:58 +0000 (17:37 +0800)]
LU-11025 dne: change dir layout via dt_layout_change

This patch includes following changes:

* Use existing mo_layout_change()/dt_layout_change() interface to
  change directory layout, which is cleaner and easier to add new
  features. The supported layout change operations are:
  1. MD_LAYOUT_DETACH: detach stripes from directory master object.
  2. MD_LAYOUT_ATTACH: attach stripes after target's.
  3. MD_LAYOUT_SHRINK: destroy stripes from specific offset.

* Previously layout change is done in MDD layer if striped directory
  is remote, however LOD can handle remote striped directory the same
  as local one, NB, remember to initialize index operations before
  altering directory layout. In this way layer violation can be
  avoided.

* Use setxattr(XATTR_NAME_LMV, LU_XATTR_CREATE) to create stripes
  for directories, and '0' or 'LU_XATTR_REPLACE' to set or replace
  directory LMV instead of setxattr(XATTR_NAME_LMV".set").

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Iac226f9e0e504f19837e14e93cee2542a2cb5e3d
Reviewed-on: https://review.whamcloud.com/37634
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-11025 dne: introduce new directory hash type: "crush" 75/36775/20
Lai Siyao [Wed, 31 Jul 2019 02:30:15 +0000 (10:30 +0800)]
LU-11025 dne: introduce new directory hash type: "crush"

Current directory striping strategy is fixed striping, i.e., it
calls (hash(filename) % stripe_count) to decide stripe for file.
The problem with this approach is that if stripe_count changes,
most of the files will need to be relocated between MDTs. This
makes directory split/merge quite expensive.

This patch introduces consistent hash striping strategy:
it calls (hash(filename) % LMV_CRUSH_PG_COUNT) to locate PG_ID
(placement group index), and then calls
crush_hash(PG_ID, stripe_index) to get a straw for each stripe,
and the stripe with the highest staw will be used to place this
file.

As we can see, it uses the CRUSH algorithm, but it only uses it
to map placement group pseudo-randomly among all stripes, while
doesn't use it to choose MDTs if MDT is not specified. The latter
is done by MDT object QoS allocation in LMV and LOD (LMV decides
the starting stripe MDT, while LOD decides the rest stripes).

This implementation contains below changes:
* new hash type "crush", and lmv_name_to_stripe_index() will take
  care of it in mapping name to stripe index.
* add "mdt_hash" sysfs tunable for LOD, which will be the default LMV
  hash type used in mkdir.
* if 'lfs setdirstripe' doesn't set hash type, server will choose
  the default LMV hash type.
* place temp file on MDT where target is: map temporary file created
  by rsync or mpifileutils dstripe on the same MDT where target is
  located, so later rename won't make target a remote entry. And the
  same for backup file ends with .bak, .sav, .orig or ~.

Compatibility:
* if server doesn't support OBD_CONNECT2_CRUSH, client will switch
  to "fnv_1a_64" hash type if "crush" is specified.
* if client doesn't know "crush" hash type, it will try all stripes
  in lookup.
* client will set hash type to LMV_HASH_TYPE_UNKNOWN if setdirstripe
  doesn't set it explicitly, and server will use the default hash
  type to create striped directory, NB, for backward compatibility,
  client will set it to LMV_HASH_TYPE_DEFAULT if server < 2.14.

test_mkdir() in sanity test will create striped directory with
random hash type if not specified.

sanity test 160g, 160h and 160i are fragile, which require a certain
amount of files created under each stripe, use 'fnv_1a_64' hash type
to fullfill this requirement.

sanity-lfsck test 31b has the same limit, create more subdirs to
make it more robust.

Add sanity 33h for temp file mapping.

Test-Parameters: trivial
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I669e561c667f926c35cf1338f4c6604249e1ee51
Reviewed-on: https://review.whamcloud.com/36775
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
12 months agoLU-13131 osc: Make sure we don't accidentally deprioritize extents 67/37967/3
Oleg Drokin [Sat, 14 Mar 2020 07:11:29 +0000 (03:11 -0400)]
LU-13131 osc: Make sure we don't accidentally deprioritize extents

Looks like in some cases high priority extents could migrate off
oo_hp_exts list back onto urgent list.

Change-Id: I96bf0a3a005b166f34dba215463c0806dfe2526a
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/37967
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
12 months agoLU-12580 lov: fix out of bound usercopy 69/37469/4
Li Dongyang [Fri, 7 Feb 2020 12:16:26 +0000 (23:16 +1100)]
LU-12580 lov: fix out of bound usercopy

When handling ioctl LL_IOC_LOV_GETSTRIPE, the user
could pass a limited buffer which is bigger than
lov_comp_md_size(), it will crash the client because
we are doing the usercopy with the user provided buffer
size.

Make sure the copy works, also for the PFL file,
we should only copy the chosen component.

Change-Id: I92bcf6d7b7f7a4387a9936a0b58332e50a88e542
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-on: https://review.whamcloud.com/37469
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-12580 lov: fix typo in lov_comp_md_size 93/37493/4
Li Dongyang [Mon, 10 Feb 2020 04:32:58 +0000 (15:32 +1100)]
LU-12580 lov: fix typo in lov_comp_md_size

If the component of a PFL file is not initialized,
we should use 0 as the stripe size when calculating
the LOVEA size.

Change-Id: I4ff5f4a78bc1d432cc1ac6fa3733461bd6b762e6
Fixes: 62f64a1077 ("LU-9489 lod: keep minimum LOVEA size")
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-on: https://review.whamcloud.com/37493
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-11591 llog: remove partial_chunk variable 30/37830/3
Mr NeilBrown [Mon, 9 Mar 2020 02:20:41 +0000 (13:20 +1100)]
LU-11591 llog: remove partial_chunk variable

This variable is set but never used, so remove it.

Test-Parameters: trivial
Fixes: ec4194e4e78c ("LU-11591 llog: add synchronization for the last record")
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I59ec75ba66a3ca37990d3ffec1d1b8f80350ae52
Reviewed-on: https://review.whamcloud.com/37830
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alexander Boyko <c17825@cray.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
12 months agoLU-12748 readahead: limit async ra requests 27/37927/5
Wang Shilong [Sun, 15 Mar 2020 14:06:40 +0000 (22:06 +0800)]
LU-12748 readahead: limit async ra requests

Currently async readahead is limited by following factors:

1) @ra_max_pages_per_file
2) @ra_max_read_ahead_whole_pages;
3) @ra_async_pages_per_file_threshold

If admin change a large value 4G to @ra_max_read_ahead_whole_pages,
with 16M RPC we could have 256 async readahead requests
flighting at the same time, this could consume all CPU
resources for readahead without limiting.

Even though we could set @max_active for workqueue,
RA requests still kept in the workqueue pool which help
prevent from CPU busying, the problem is RA still try to
use CPU later, we might still submit too many requests
to workqueue, so instead of limiting it in the workqueue,
we could limit it earlier, if there has been too many
async RA requests in the system(let's say default is 1/2
of CPU cores), we just fallback to sync RA, which limit
read threads using all CPU resources.

Change-Id: I370c04e014f24c795c1a28effca9c51b1db2a417
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/37927
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13354 capa: remove remained lustre capa code 95/37895/2
Mikhail Pershin [Wed, 11 Mar 2020 13:28:15 +0000 (16:28 +0300)]
LU-13354 capa: remove remained lustre capa code

Remove remaining unused lustre capability structures and
defines.

Test-Parameters: trivial
Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: Iebd5bf87537c19fb30804adaf623c185983c351b
Reviewed-on: https://review.whamcloud.com/37895
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13354 ptlrpc: update wirecheck with missed checks 77/37877/3
Mikhail Pershin [Wed, 11 Mar 2020 08:59:10 +0000 (11:59 +0300)]
LU-13354 ptlrpc: update wirecheck with missed checks

- add missed values and defines in wirecheck.c
- prefer compile-time checks with CVALUE/CDEFINE for new
  blocks of checks, one-line updates are added with the
  same checks style as checks around it.
- update wirehdr.c to generate proper wiretest.c. Changes
  were made to wiretest.c directly, change wirehdr.c to
  reflect these changes.
- remove obsoleted and not used capa enum and defines found
  while updating checks

Test-Parameters: trivial
Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: Ia1ea4ce30d61c58b20e3f20bed083812391f5998
Reviewed-on: https://review.whamcloud.com/37877
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
12 months agoLU-13344 libcfs: adler32: don't set CRYPTO_TFM_RES_BAD_KEY_LEN 70/37870/3
Mr NeilBrown [Tue, 10 Mar 2020 20:34:41 +0000 (16:34 -0400)]
LU-13344 libcfs: adler32: don't set CRYPTO_TFM_RES_BAD_KEY_LEN

This flag was not being checked by most callers and has been removed since
Commit 674f368a952c ("crypto: remove CRYPTO_TFM_RES_BAD_KEY_LEN")
so remove from lustre too.

Change-Id: Ia616430a98d05c7bb3b22a0be543e83855272d2f
Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Reviewed-on: https://review.whamcloud.com/37870
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-9859 libcfs: open code cfs_trace_max_debug_mb() into cfs_trace_set_debug_mb() 29/37829/5
Mr NeilBrown [Thu, 12 Mar 2020 19:52:56 +0000 (15:52 -0400)]
LU-9859 libcfs: open code cfs_trace_max_debug_mb() into cfs_trace_set_debug_mb()

This function in used twice.
In libcfs_debug_init() the usage is pointless as
the only place that set libcfs_debug_mb ensures
that it does not exceed the maximum.  So checking
again can never help.

So open-code the small function into the only
other place that it is used - in cfs_trace_set_debug_mb(),
which is used to set libcfs_debug_mb.

HPC machines are being deployed with 24 TiB or more memory.
Change the variables uses to calculate the max_debug_mb
to unsigned long to avoid a potential overflow and change
80 / 100 scaling factors to 4 / 5.

Linux-commit: 5b5df84f3582b0e3a2ff70d49867bc3e03f67502

Change-Id: I497502bb7418e0b3d24b24c507029652bc9a9e55
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Reviewed-on: https://review.whamcloud.com/37829
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-9859 libcfs: always range-check libcfs_debug_mb setting. 28/37828/4
Mr NeilBrown [Mon, 9 Mar 2020 01:35:57 +0000 (21:35 -0400)]
LU-9859 libcfs: always range-check libcfs_debug_mb setting.

When the libcfs_debug_mb module parameter is set
at module-load time it isn't range-checked.  When
it is set via sysfs it is.  This inconsistency
makes the code harder to follow.

It is quite safe to call cfs_trace_set_debug_mb()
and cfs_trace_get_debug_mb() before the module
is initialized as cfs_tcd_for_each() does nothing
before initializtion.

So change cfs_trace_set_debug_mb() - which does
range checking - to returned the ranged checked number (it currently
always returns zero) and use that as the new value, unless
cfs_trace_get_debug_mb() now returns a non-zero value.

Linux-commit: 205b154f3bd6fa3e95ceca8bba8d00122bb8fb04

Change-Id: I495097bbae138625e54a012bf06c96e62d206228
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Reviewed-on: https://review.whamcloud.com/37828
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13120 build: Fix ZFS dependancies for osd-zfs-mount 09/37809/5
Shaun Tancheff [Fri, 6 Mar 2020 21:37:47 +0000 (15:37 -0600)]
LU-13120 build: Fix ZFS dependancies for osd-zfs-mount

lustre-osd-zfs-mount depends on zfs
lustre-osd-zfs-mount depends on kmod-lustre-osd-zfs

SuSE packaging style prefers kmp package naming so prepare
for adopting a kmp named zfs package

Test-Parameters: fstype=zfs
Cray-bug-id: LUS-7077
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: Ifcee1fa90bf95925cbcff0975d95f12ae04f670d
Reviewed-on: https://review.whamcloud.com/37809
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13312 ldlm: fix to stop iterating tree early in ldlm_kms_shift_cb() 62/37762/5
Wang Shilong [Sat, 29 Feb 2020 14:02:55 +0000 (22:02 +0800)]
LU-13312 ldlm: fix to stop iterating tree early in ldlm_kms_shift_cb()

It is very possible that old_kms is equal (last extent end + 1),
and if we fail last one check, we will have to iterate all PR extent locks
in tree which is O(N).

This is very likely to happen in IO500 hard mode, because firstly
we try to write which generates a lot of PW locks, after that read
is triggered which need cancel all PW locks to PR locks.

Be careful to avoid overflow, as @l_extent.end could be OBD_OBJECT_EOF,
if that is case, it just means we could stop iterating.

Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: Ie142fb738482a8abcc7d0c9d67d6e6eb520459db
Reviewed-on: https://review.whamcloud.com/37762
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Gu Zheng <gzheng@ddn.com>
12 months agoLU-13294 libcfs: incorrect rotor behaviour 09/37709/5
Andrew Perepechko [Fri, 14 Feb 2020 02:20:09 +0000 (05:20 +0300)]
LU-13294 libcfs: incorrect rotor behaviour

Signed int cpt rotor is set to -1 on initialization.
cfs_cpt_spread_node() improperly handles this value
via "if (!rotor--)" check. The condition is never true
with negative rotor values, so for_each_node_mask()
only exits with node = MAX_NUMNODES.

kmalloc_node() attempts to determine the zonelist based
on the passed node id and maps MAX_NUMNODES to some
random pointer. Crash.

BUG: unable to handle kernel paging request at 0000000100002007
IP: [<ffffffff847c0da7>] __alloc_pages_nodemask+0x97/0x420

Change-Id: I4df74e394bdfc2a918d66aa12e6852ff0f6738ab
Signed-off-by: Andrew Perepechko <c17827@cray.com>
Cray-bug-id: LUS-8492
Reviewed-on: https://review.whamcloud.com/37709
Reviewed-by: Alexander Boyko <c17825@cray.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13278 lnet: Reconcile discovery push and reply handling 74/37674/3
Chris Horn [Mon, 10 Feb 2020 20:11:49 +0000 (14:11 -0600)]
LU-13278 lnet: Reconcile discovery push and reply handling

Reconcile the logic for updating the multi-rail flag of a peer when
processing a discovery PUSH with the logic used when processing a
discovery REPLY.

Cray-bug-id: LUS-8516
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: Idfb4c3729822d03b71f9440ac66176ae6b886022
Reviewed-on: https://review.whamcloud.com/37674
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Stephen Champion <stephen.champion@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13280 tests: customize run_mdtest for several mount points 65/37665/6
Elena Gryaznova [Wed, 26 Feb 2020 16:05:11 +0000 (19:05 +0300)]
LU-13280 tests: customize run_mdtest for several mount points

Patch adds the possibility to run mdtest on $mdtest_Nmntp
number of Lustre mount points.

Test-Parameters: trivial testlist=parallel-scale \
envdefinitions="mdtest_Nmntp=3" \
envdefinitions=ONLY=mdtestfpp envdefinitions=ONLY=mdtestssf
Cray-bug-id: LUS-7686
Signed-off-by: Elena Gryaznova <elena.gryaznova@hpe.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Change-Id: I9bc68d4f296970290cf460bf25cc26537bcc3281
Reviewed-on: https://review.whamcloud.com/37665
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexander Boyko <c17825@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13260 lov: fix size check when stripe is zero 23/37623/3
Yang Sheng [Wed, 19 Feb 2020 11:08:33 +0000 (19:08 +0800)]
LU-13260 lov: fix size check when stripe is zero

Set correct max size while stripe is zero.

Fixes: f3f6515562 (LU-8998 lov: add composite layout unpacking)
Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: I9b76283fcc65f58e3be6adf49f035236687ac85c
Reviewed-on: https://review.whamcloud.com/37623
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-9091 obdclass: allow bare KMGTPE param suffix 20/37620/9
Andreas Dilger [Fri, 10 May 2019 21:09:41 +0000 (15:09 -0600)]
LU-9091 obdclass: allow bare KMGTPE param suffix

Allow sysfs_memparse() to parse bare "KMGTPE" suffixes as base-2.
This simplifies the code to not append the unnecessary "iB" if
the suffix is one of KMGTPE already, since these are assumed to
be binary units already.

Don't append a bare defunit of "B", since this unnecessarily copies
the input string and takes the slow path for parsing the string, but
doesn't actually change the resulting value.

The test_string_to_size_one() may only have printed an error and not
caused module load to actually fail.  Print a message and return an
error if the tests fail.  Some checks expect to fail, so add
test_string_to_size_err() to verify that an error was returned.

Mark the obd_init_checks() function with __init so it is dropped
after the module is loaded.

Fixes: d9e0c9f346d0 ("LU-9091 sysfs: use string helper like functions for sysfs")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I3cdf5f8f0aeca458ed1989366102c33ae83ebbe5
Reviewed-on: https://review.whamcloud.com/37620
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-12477 ldiskfs: remove obsolete ext4 patches 52/37352/14
James Simmons [Fri, 6 Mar 2020 01:40:47 +0000 (20:40 -0500)]
LU-12477 ldiskfs: remove obsolete ext4 patches

Drop support for ldiskfs kernels earlier than RHEL7.6.

Test-Parameters: trivial

Change-Id: I30450904c508ec8aa5388cbfd9bd967028f88b28
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/37352
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13004 modules: replace lnet_kiov_t with struct bio_vec 52/37852/7
Mr NeilBrown [Thu, 21 Nov 2019 05:53:57 +0000 (16:53 +1100)]
LU-13004 modules: replace lnet_kiov_t with struct bio_vec

lnet_kiov_t has the same structure and general purpose
as struct bio_vec - only the names a different.
The difference brings no value - so let's remove it.
sed -e 's/lnet_kiov_t/struct biovec/g'
    -e 's/kiov_page/bv_page/g'
    -e 's/kiov_len/bv_len/g'
    -e 's/kiov_offset/bv_offset/g'

plus some cleaning up.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ia718e5880d0fbbb9aa9aed0edf48a2b2c42b54d3
Reviewed-on: https://review.whamcloud.com/37852
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13041 lnet: fix router primary_nid deletion 18/36918/8
Amir Shehata [Sun, 1 Dec 2019 22:49:31 +0000 (14:49 -0800)]
LU-13041 lnet: fix router primary_nid deletion

For MR routers it is possible to update the primary_nid. This will
result in a push to update the peers of the changes. The peer should
be able to update the primary_nid of the router.

When the peer receives the push notifications and deletes the primary
NID of the router, it should update the router's primary NID to be
the next peer NI in the list.

Test-parameters: trivial

Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: I5928882dde513cb885301f632535282695429bb0
Reviewed-on: https://review.whamcloud.com/36918
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13027 lnet: fix route consolidation 17/36917/8
Amir Shehata [Sun, 1 Dec 2019 20:18:38 +0000 (12:18 -0800)]
LU-13027 lnet: fix route consolidation

After discovery one or more peers can be merged together. Some
of those peers might be gateways. The routes from the gateways
should be transferred to the MR peer. However, if the MR peer
is already a gateway, then we shouldn't add routes to a remote
net for which a route already exists, unless that route is
higher priority or has less number of hops, in which case we
should replace the route with the better one.

Test-parameters: trivial

Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: I8790f39a604f4e3d222c1f98c3d28d3dfb03a237
Reviewed-on: https://review.whamcloud.com/36917
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13035 lnet: fix remote peer ni selection 15/36915/7
Amir Shehata [Fri, 29 Nov 2019 05:22:30 +0000 (21:22 -0800)]
LU-13035 lnet: fix remote peer ni selection

When selecting the NI of a remote peer we can only round robin
over the NIs. Remote peer NIs have no credits and no health.
However, we'd like to still spread the traffic over the remote
peer NIs as much as we can.

Remote peer NIs sequence number was not being incremented and
therefore the round robin algorithm wasn't working.

Test-parameters: trivial

Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: I83dba3597987844b7bddc980c53e8c6d90bcd3b2
Reviewed-on: https://review.whamcloud.com/36915
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13029 lnet: fix asym routing with multi-hop 14/36914/7
Amir Shehata [Thu, 28 Nov 2019 23:44:27 +0000 (15:44 -0800)]
LU-13029 lnet: fix asym routing with multi-hop

When a system is setup with multi-hop routing the next-hop
gateway, one off the nodes, will not have an interface
on the remote network. The route should still be considered
alive even if lnet_avoid_asym_routing is set

The code sets a route to be multi-hop if the gateway it uses
does not have an NI on the remote net of the route. The underlying
assumption here is that routing configuration is correct

To aid in fault detection a warning message is printed out if
the route is set to be multi-hop but the route hop count is less
than 1.

Added a YAML field in the:
lnetctl route show -v
output to print the type of the route: single-hop or multi-hop.

Test-parameters: trivial

Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: I0a96263e2903974ec50d46bd56e550cde7d86e8c
Reviewed-on: https://review.whamcloud.com/36914
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Doug Oucharek <dougso@me.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13005 ptlrpc: use percpu refcount to track requests. 45/36845/10
Mr NeilBrown [Wed, 20 Nov 2019 02:16:04 +0000 (13:16 +1100)]
LU-13005 ptlrpc: use percpu refcount to track requests.

ptlrpc needs to wait for outstanding RPC requests to complete before
ptlrpc_ni_fini() finishes.
It currently does this using a refcount in the event-queue, but that
refcount is otherwise unused, and it will shortly be removed.

So add a percpu refcount that explicitly tracks these requests, and
wait for it with a completion.

Also move the declaration of ptlrpc_eq to ptlrpc_internal.h which is a
more appropriate location.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I479df29d1ddfb4e3d82a26b5fb567f3432dad8af
Reviewed-on: https://review.whamcloud.com/36845
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-12546 lfsck: print logs for LLIT_UNMATCHED_PAIR 44/36744/9
Hongchao Zhang [Thu, 26 Dec 2019 20:22:42 +0000 (15:22 -0500)]
LU-12546 lfsck: print logs for LLIT_UNMATCHED_PAIR

Print the information of MDT/OST objects for LLIT_UNMATCHED_PAIR
to help track the affected files.

Change-Id: Ifa94ef9365fc6fa9875bbcdb9ad377150a52700d
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/36744
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-12780 quota: don't use ptlrpc_thread for qsd_reint_main() 68/36268/8
Mr NeilBrown [Wed, 23 Oct 2019 00:30:51 +0000 (11:30 +1100)]
LU-12780 quota: don't use ptlrpc_thread for qsd_reint_main()

Instead of ptlrpc_thread, use native kthread operations.

- perform allocation before starting the thread, and don't
  bother with synchronization at thread start
- use kthread_stop() and kthread_should_stop() to signal
  shutdown
- as the thread can stop independently, use xchg() on
  qqi->qqi_reint_task to determine if there is a waiter, and
  if so, the thread should synchronize with the waiter.
- use wake_up_var and wait_var_event for event signalling.

Also use kthread's thread-name formatting rather than
allocating a 'name' for formating the name separately.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ia1334a3e865b67ba5ae78c7b8342704a340229f7
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Reviewed-on: https://review.whamcloud.com/36268
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-12780 osp: don't use ptlrpc_thread for lwp_notify_main() 66/36266/7
Mr NeilBrown [Wed, 23 Oct 2019 00:30:50 +0000 (11:30 +1100)]
LU-12780 osp: don't use ptlrpc_thread for lwp_notify_main()

Instead of ptlrpc_thread, use native kthread functionality.

- There is no need to synchronise on start-up.  But we do
  need to ensure class_export_get/put are balanced, and we
  cannot be certain that the thread function will run.
  So call class_export_get() before starting the thread,
  and class_export_put() after it has finished.

- As the thread exits independantly, waiting for it to finish
  is a little more complex than just using kthread_stop().

  If the thread stops before lwp_device_fini() runs, it sets
  ->lpd_notify_task to NULL, and lwp_device_fini() doesn't wait.
  If lwp_device_fini() runs first, it set ->lpd_notify_task to
  NULL and calls kthread_stop().  The thread notices this, and
  deliberately waits for kthread_should_stop() to avoid any
  races in kthread code.

  xchg() is used to ensure exactly one of the thread and
  lwp_device_fini() sets lpd_notify_task to NULL.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I45621f325d14194af5a44f86f7e605bdafbbb9be
Reviewed-on: https://review.whamcloud.com/36266
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-11310 ldiskfs: Support for SUSE 15 GA and SP1 94/36094/9
Shaun Tancheff [Sat, 7 Mar 2020 16:26:35 +0000 (10:26 -0600)]
LU-11310 ldiskfs: Support for SUSE 15 GA and SP1

Add support for ldiskfs SUSE 15 SP1 and late GA:
GA 150 series kernels and SP1 195, and 197 series

Test-Parameters: trivial
Cray-bug-id: LUS-7572
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I2525acc475503166fb71bed59663edeee603d38c
Reviewed-on: https://review.whamcloud.com/36094
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
12 months agoLU-12670 osd-ldiskfs: Shrink rcu critical section 05/35805/14
Shaun Tancheff [Mon, 2 Dec 2019 18:14:30 +0000 (12:14 -0600)]
LU-12670 osd-ldiskfs: Shrink rcu critical section

Drop ENTRY/EXIT macros to exclude them rcu critical section
scope without introducing additional work around logic.

Cray-bug-id: LUS-7771
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Change-Id: I70486ba137592d1a57bc712837b8a300b8017a36
Reviewed-on: https://review.whamcloud.com/35805
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-9859 libcfs: improve API and implementation of blocking signals. 11/35411/11
NeilBrown [Wed, 11 Mar 2020 13:01:09 +0000 (09:01 -0400)]
LU-9859 libcfs: improve API and implementation of blocking signals.

According to comment for set_current_blocked() in
kernel/signal.c, changing ->blocked directly is wrong.
sigprocmask() should be called instead.

So change cfs_block_sigsinv() and cfs_restore_sigs()
to use sigprocmask().
For consistency, change them to pass the sigset_t by reference
rather than by value.

Also fix cfs_block_sigsinv() so that it correctly blocks
signals above 32 on a 32bit host.

Linux-commit: 84e07b9d0ac8728b1865b23498d746861a8ab4c2

Change-Id: Iffb8b39ee2b988e9909ceaaba50446eecdf7f249
Signed-off-by: NeilBrown <neilb@suse.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-on: https://review.whamcloud.com/35411
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-13362 lnet: Disc reply race with finalize and routed recv 37/37937/2
Chris Horn [Fri, 13 Mar 2020 19:34:23 +0000 (14:34 -0500)]
LU-13362 lnet: Disc reply race with finalize and routed recv

A race exists between a thread handling a discovery reply, and
another thread in the lnet_finalize() call path, or in any of
the code paths that result in lnet_post_routed_recv_locked().

The discovery reply handler takes the lp_lock, and while holding
that lock, tries to acquire the lpni_lock for each lpni associated
with the lnet_peer object.

In lnet_return_rx_credits_locked() (lnet_finalize() code path) and
lnet_post_routed_recv_locked() (called via a couple different code
paths) the lpni_lock is taken, and then the lp_lock is taken for the
associated lnet_peer object.

Thread A: spin_lock(lp_lock)
Thread B: spin_lock(lpni_lock)
Thread B: spin_lock(lp_lock)
Thread A: spin_lock(lpni_lock)

This results in deadlock. The lp_lock and lpni_lock do not need to be
held at the same time in lnet_return_rx_credits_locked() nor in
lnet_post_routed_recv_locked().

Cray-bug-id: LUS-8607
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: Ie4e9a172b4705d9f5723a6da1ff251b380ad47ac
Reviewed-on: https://review.whamcloud.com/37937
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-6142 llite: Fix style issues for llite/statahead.c 65/37765/7
Andreas Dilger [Sun, 1 Mar 2020 11:40:22 +0000 (04:40 -0700)]
LU-6142 llite: Fix style issues for llite/statahead.c

Fix style issues for llite/statahead.c.

Test-Parameters: trivial testlist=sanity
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ie6d8507f35cbd586ca51ad6f1c5dc1ef7abc3305
Reviewed-on: https://review.whamcloud.com/37765
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-12506 changelog: support large number of MDT 59/37759/7
Hongchao Zhang [Wed, 4 Mar 2020 14:07:19 +0000 (09:07 -0500)]
LU-12506 changelog: support large number of MDT

At client, the changelog of each MDT is associated with
one miscdevice, but the number of miscdevice is limited
to 64 in Linux kernel, then it will fail if there are
more than 64 MDTs.

This patch replaces miscdevice with dynamic devices to
support more MDTs.

Change-Id: Ie3ce76cbe1c613bf17d6350ea95546524b6d66b8
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/37759
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-6142 lustre: remove remaining users of ldebugfs_register 43/37743/3
Mr NeilBrown [Thu, 27 Feb 2020 04:28:50 +0000 (15:28 +1100)]
LU-6142 lustre: remove remaining users of ldebugfs_register

ldebugfs_register() is just a call to debugfs_create_dir() and
ldebugfs_add_vars() if the list option is set.  Fix up the remaining
users of this function to just call these two functions instead, and
delete the now unused ldebugfs_register() call.

This ends up cleaning up more code and making things smaller, always a
good thing.

Linux-Commit 00905f006608 ("staging: lustre: remove last two
users of ldebugfs_register()")

Test-Parameters: trivial
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I08f4cdcdd3367f50c37d899a5dc4d753e72c0fda
Reviewed-on: https://review.whamcloud.com/37743
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
12 months agoLU-6142 lustre: unwrap some ldebugfs_register() calls 42/37742/3
Mr NeilBrown [Thu, 27 Feb 2020 03:58:57 +0000 (14:58 +1100)]
LU-6142 lustre: unwrap some ldebugfs_register() calls

When the third option (list) to ldebugfs_register() is NULL, it's the
same as just calling debugfs_create_dir().  So unwind this and call
debugfs_create_dir() directly.

This ends up saving lots of code as we do not need to do any error
checking of the return value (because it does not matter).

Note: this depends on Commit a59d6293e537 ("debugfs: change parameter
check in debugfs_remove() functions") which landed in v3.6-rc1.

Linux-Commit b145f49f233d ("staging: lustre: unwrap some
ldebugfs_register() calls")

Test-Parameters: trivial
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I6bff3b1d416de1f15bb95b22fa6075b1ae30a534
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Reviewed-on: https://review.whamcloud.com/37742
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-6142 fld: remove function typedefs 40/37740/3
Mr NeilBrown [Thu, 27 Feb 2020 02:59:27 +0000 (13:59 +1100)]
LU-6142 fld: remove function typedefs

typedefs a generally discouraged in upstream linux.
fld_hash_func_t and fld_scan_func_t are only used
once immediately after the definition, so having them
doesn't help readability or maintainability at all.

So remove them.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I37288238f375d5673b7cb8d68d30102f43e44107
Reviewed-on: https://review.whamcloud.com/37740
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-12511 fld: use list_last_entry in fld_cache_shrink 39/37739/2
Mr NeilBrown [Thu, 27 Feb 2020 02:47:38 +0000 (13:47 +1100)]
LU-12511 fld: use list_last_entry in fld_cache_shrink

fld_cache_shrink can be substantially simplified by using
list_last_entry().

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I0dcbd8ffefbd113786f422b7662298605020687d
Reviewed-on: https://review.whamcloud.com/37739
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>