Whamcloud - gitweb
fs/lustre-release.git
2 months agoLU-13776 tests: make sure pjdfstest.sh writes to tmp 38/39338/4
Oleg Drokin [Sat, 11 Jul 2020 04:51:27 +0000 (00:51 -0400)]
LU-13776 tests: make sure pjdfstest.sh writes to tmp

no writes to random Lustre source locations as they could be readonly

Test-Parameters: trivial
Test-Parameters: fstype=ldiskfs testlist=pjdfstest
Test-Parameters: fstype=zfs testlist=pjdfstest
Change-Id: Icd262a698390eadf4b53cd5d311bc6c2a561a79e
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/39338
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
2 months agoLU-12275 sec: introduce null algo for filename encryption 81/38881/8
Sebastien Buisson [Tue, 9 Jun 2020 15:11:42 +0000 (15:11 +0000)]
LU-12275 sec: introduce null algo for filename encryption

Introduce a "null" algorithm for client side filename encryption,
which is basically a no-op.
This is needed because first implementation step only supports
content encryption, and not names. So give the ability to support
encryption policies that have a 'filenames_encryption_mode' property
internally set to LLCRYPT_MODE_NULL.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I3470f89f227b3a03e56766fe1ba5e36ae92ec27b
Reviewed-on: https://review.whamcloud.com/38881
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13485 build: Make parallel tests names unique 61/39161/4
Shaun Tancheff [Wed, 24 Jun 2020 18:36:38 +0000 (13:36 -0500)]
LU-13485 build: Make parallel tests names unique

This iteration re-used the serial tests internal name for the
parallel test variant. This name clashing causes builds to fail.

Remedy this by adding a postfix tag the internal unique identifier
of the converted parallel tests.

Test-Parameters: trivial
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: Ia39bf2e0004abd10a0c7b146eb2ef1cf62e6d891
Reviewed-on: https://review.whamcloud.com/39161
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <farr0186@gmail.com>
2 months agoLU-13366 tests: add SEL support to racer 59/37959/6
Vitaly Fertman [Fri, 13 Mar 2020 12:47:00 +0000 (15:47 +0300)]
LU-13366 tests: add SEL support to racer

some files are created with a sel layout if RACER_ENABLE_SEL is set

Test-Parameters: testlist=racer envdefinitions="RACER_ENABLE_SEL=true"
Signed-off-by: Vitaly Fertman <c17818@cray.com>
Change-Id: I1f699988fdd0a0dbb19b71d6ea353d68326d0d5b
Reviewed-by: Alexander Zarochentsev <c17826@cray.com>
Reviewed-by: Elena Gryaznova <c17455@cray.com>
Tested-by: Elena Gryaznova <c17455@cray.com>
Reviewed-on: https://es-gerrit.dev.cray.com/156106
HPE-bug-id: LUS-7591
Reviewed-on: https://review.whamcloud.com/37959
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13366 lod: check for extension size at instantiation time 61/37961/8
Vitaly Fertman [Fri, 13 Mar 2020 13:42:35 +0000 (16:42 +0300)]
LU-13366 lod: check for extension size at instantiation time

lod_statfs_and_check() may consider an OST as OK, but SEL needs its
extension size to fit the free space and the OST may turn out to be
low-on-space, thus inappropriate OST may be chosen, SEL cannot use
it and ENOSPC occurs.

Take the extension size into account at the object allocation time.
If none of OSTs is able to allocate an object, try to allocate in old
manner instead of returning ENOSPC from lod_qos_prep_create().

Adjust tests to work on smaller extension sizes to fit into OSTs
used for sanity tests to enable the new logic above.

Signed-off-by: Vitaly Fertman <c17818@cray.com>
Change-Id: If5159200a72a68a7261aa7031e58a1ac6a8e3f24
Reviewed-by: Alexey Lyashkov <c17817@cray.com>
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Tested-by: Elena Gryaznova <c17455@cray.com>
Reviewed-on: https://es-gerrit.dev.cray.com/156372
HPE-bug-id: LUS-8157
Reviewed-on: https://review.whamcloud.com/37961
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13344 all: Separate debugfs and procfs handling 34/37834/14
Shaun Tancheff [Thu, 4 Jun 2020 21:27:44 +0000 (16:27 -0500)]
LU-13344 all: Separate debugfs and procfs handling

Linux 5.6 introduces proc_ops with v5.5-8862-gd56c0d45f0e2
proc: decouple proc from VFS with "struct proc_ops"

Separate debugfs usage and procfs usage to prepare for the divergence
of debugfs using file_operations and procfs using proc_ops

HPE-bug-id: LUS-8589
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I1746e563b55a9e89f90ac01843c304fe6b690d8b
Reviewed-on: https://review.whamcloud.com/37834
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13196 llite: Remove mutex on dio read 19/37419/6
Patrick Farrell [Tue, 4 Feb 2020 03:10:23 +0000 (22:10 -0500)]
LU-13196 llite: Remove mutex on dio read

DIO reads in Lustre are protected by Lustre range locking
and do not need the inode mutex.  This code was removed
in LU-1669, the range lock was added for DIO reads in
LU-6227, and then the mutex was accidentally re-introduced
in LU-6260.

Remove it again.

Test-Parameters: envdefinitions=ONLY=16 testlist=sanityn,sanityn,sanityn,sanityn,sanityn,sanityn,sanityn,sanityn
Fixes: 98883bd3e2cc ("LU-6260 llite: add support for direct IO api changes")
Signed-off-by: Patrick Farrell <farr0186@gmail.com>
Change-Id: I5b3570e83a4b4ff36d9a22bc6bd3be5d5f991924
Reviewed-on: https://review.whamcloud.com/37419
Reviewed-by: Wang Shilong <wshilong@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Tested-by: Shuichi Ihara <sihara@ddn.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-9679 mdc: create mdc_acl.c 92/39292/4
Mr NeilBrown [Mon, 6 Jul 2020 12:34:47 +0000 (08:34 -0400)]
LU-9679 mdc: create mdc_acl.c

This new C file contains acl related code so it can be
conditionally compiled.

Also remove conditional code around the call to mdc_unpack_acl(), as
those tests are already performed inside mdc_unpack_acl().

In the Makefile, use CONFIG_FS_POSIX_ACL to control conditional
compilation as CONFIG_LUSTRE_FS_POSIX_ACL is not available, but the
two are semantically equivalent in this context.

This removes all #ifdef of CONFIG_LUSTRE_FS_POSIX_ACL from C files in
'mdc'.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ic9060bee4a2ad55580d8879fe32fee01b1cb8884
Reviewed-on: https://review.whamcloud.com/39292
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-9679 lustre: make ptlrpc_connection_put() static inline 91/39291/3
Mr NeilBrown [Mon, 6 Jul 2020 12:34:48 +0000 (08:34 -0400)]
LU-9679 lustre: make ptlrpc_connection_put() static inline

This function needs to be called from the obdclass modules,
but is currently defined in a module that depends on that module.
The get around this interdependence, a global variable
  ptlrpc_put_connection_superhack
is used to make a pointer to the function available.

Rather than this hack, we can make ptlrpc_connection_put()
static-inline.  This does expose some details of ptlrpc to obdclass,
but there is already a fairly tight connection.

Also change the return value to 'void' as it is never used,
and don't bother checking for NULL before calling, as the
function has its own test for NULL.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I37333b78e410e8c46bddc468c31bed61dd9e7b33
Reviewed-on: https://review.whamcloud.com/39291
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-9679 lov: annotate nested locking of obd_dev_mutex 48/39248/2
Mr NeilBrown [Thu, 2 Jul 2020 21:43:45 +0000 (07:43 +1000)]
LU-9679 lov: annotate nested locking of obd_dev_mutex

obd_statfs() can call lmv_statfs() with ->obd_dev_mutex helds.
lmv_statfs will then call obd_statfs() on a different device
and ->obd_dev_mutex will be taken again.  This is a *different*
mutex, but lockdep cannot see the difference, so it complains.

We can tell lockdep not to worry in this case using
mutex_lock_interruptible_nested().

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I0776407b722dd29ab1321289953b63f76fce7ceb
Reviewed-on: https://review.whamcloud.com/39248
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13754 gss: open sptlrpc init channel in R+W mode 97/39297/3
Sebastien Buisson [Tue, 7 Jul 2020 14:59:08 +0000 (23:59 +0900)]
LU-13754 gss: open sptlrpc init channel in R+W mode

Linux 5.3 changed struct cache_detail readers to writers.
As this mechanism is used by GSS authentication in Lustre via SunRPC,
we need to make sure lsvcgssd daemon does open
/proc/net/rpc/auth.sptlrpc.init/channel in R+W mode.

It also affects CentOS/RHEL 7.8, as the kernel commit was ported to
these distros.

Test-Parameters: trivial
Test-Parameters: env=SHARED_KEY=true,SANITY_EXCEPT="56w 405" mdscount=2 mdtcount=4 osscount=2 ostcount=8 testlist=sanity,recovery-small,sanity-sec
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: If88802d4f2bc3168dda4f79fe57f2f44ac7ef39e
Reviewed-on: https://review.whamcloud.com/39297
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12586 lov: one more fix to write_intent end for trunc 12/38412/8
Bobi Jam [Wed, 29 Apr 2020 09:14:04 +0000 (17:14 +0800)]
LU-12586 lov: one more fix to write_intent end for trunc

This patch fixes another case where the truncate write intent
extent is set incorrectly.  This may cause errors when truncating
PFL files to exactly the boundary between two extents.

Fixes: c32c7401426d ("LU-12586 lov: Correct write_intent end for trunc")
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I40f14a48c53bfd1e6442e4414ee30f9eb159fc02
Reviewed-on: https://review.whamcloud.com/38412
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
2 months agoLU-13729 osd-ldiskfs: race access to iam_formats during setup 13/39213/2
Wang Shilong [Tue, 30 Jun 2020 01:12:48 +0000 (09:12 +0800)]
LU-13729 osd-ldiskfs: race access to iam_formats during setup

It might be possible during OST mounting, two targets reach
iam_format_guess() at the same time, if @initialized is 0,
they both access iam_lxx_format_init(), however list operation
inside is not protected by any locking which cause list corruptions
finally.

We could fix this by doing formats registration in module init,
since there are only two formats, just remove pointless list.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I6dd5a4d1297792b47fb4b94052465a7e0f9123aa
Reviewed-on: https://review.whamcloud.com/39213
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13713 lnet: check rtr_nid is a gateway 75/39175/2
Amir Shehata [Wed, 24 Jun 2020 23:46:58 +0000 (16:46 -0700)]
LU-13713 lnet: check rtr_nid is a gateway

The rtr_nid is specified for all REPLY/ACK. However it is possible
for the route through the gateway specified by rtr_nid to be removed.
In this case we don't want to use it. We should lookup alternative
paths.

This patch checks if the peer looked up is indeed a gateway. If it's
not a gateway then we attempt to find another path. There is no need
to fail right away. It's not a hard requirement to fail if the default
rtr_nid is not valid.

Test-Parameters: trivial
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: Ic1c93b7c6c2c8060e2cfeb8fb1cb875dbc3010f7
Reviewed-on: https://review.whamcloud.com/39175
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12678 socklnd: change ksnd_nthreads to atomic_t 21/39121/3
Mr NeilBrown [Sun, 7 Jun 2020 23:24:36 +0000 (19:24 -0400)]
LU-12678 socklnd: change ksnd_nthreads to atomic_t

This variable is treated like an atomic_t, but a global spinlock is
used to protect updates - and also unnecessarily to protect reads.

Change to atomic_t and avoid using the spinlock.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Id94d280875a9e115dc077253c49e97a725dc91e1
Reviewed-on: https://review.whamcloud.com/39121
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13697 llite: fix short io for AIO 04/39104/8
Wang Shilong [Fri, 19 Jun 2020 14:19:18 +0000 (22:19 +0800)]
LU-13697 llite: fix short io for AIO

The problem is currently AIO could not handle i/o size > stripe size:

We need cl io loop to handle io across stripes, since -EIOCBQUEUED is
returned for AIO, io loop will be stopped thus short io happen.

The patch try to fix the problem by making IO engine aware of
special error, and it will be proceed to finish all IO requests.

Fixes: d1dde ("LU-4198 clio: AIO support for direct IO")
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I1885e0fc510571417d888249b381f9c2f130ca5a
Reviewed-on: https://review.whamcloud.com/39104
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13460 lfs: make function print_failed_tgt() work correctly 59/38959/6
Emoly Liu [Wed, 17 Jun 2020 07:43:00 +0000 (15:43 +0800)]
LU-13460 lfs: make function print_failed_tgt() work correctly

param->fp_xxx_index is the number of the specified targets, not
the specific index, so it can't be passed into llapi_obd_statfs()
directly to get the statfs information. Instead, all the targets
listed in param->fp_xxx_indexes should be passed one by one.

Also, $mdt_idx in sanity.sh test_56rb should be initialized with
the correct mdt index.

Signed-off-by: Emoly Liu <emoly@whamcloud.com>
Change-Id: Idffdddac1b5b249aa903b97912c767826f3b146c
Reviewed-on: https://review.whamcloud.com/38959
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12549 utils: Check range of quota ID for "lfs" arguments 38/38938/6
Etienne AUJAMES [Tue, 9 Jun 2020 17:07:37 +0000 (19:07 +0200)]
LU-12549 utils: Check range of quota ID for "lfs" arguments

strtoul function return a 64bits value on a 64bits system, so an
overflow occurs when we store user value into a quota/project
structure.

This commit apply the same 32 bits verification for "lfs" project,
quota,setquota and find commands on uid, gid and project id arguments.

Signed-off-by: Etienne AUJAMES <eaujames@ddn.com>
Change-Id: I809e9ac55d4bc676c20b18c6c198a69eaba9cff6
Reviewed-on: https://review.whamcloud.com/38938
Reviewed-by: Wang Shilong <wshilong@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
2 months agoLU-13667 ptlrpc: fix endless loop issue 15/38915/4
Hongchao Zhang [Fri, 19 Jun 2020 02:53:12 +0000 (10:53 +0800)]
LU-13667 ptlrpc: fix endless loop issue

In ptlrpc_pinger_main, if the process to ping the recoverable
clients or obd_update_maxusage takes too long time, it could
be stuck in endless loop because of the negative value returned
by pinger_check_timeout.

Change-Id: Ib7fc22b3cc31255223bc2be60224ced1a3585f87
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/38915
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13648 lnet: Set remote NI status in lnet_notify 62/38862/2
Chris Horn [Fri, 22 May 2020 01:49:53 +0000 (20:49 -0500)]
LU-13648 lnet: Set remote NI status in lnet_notify

The gnilnd receives node health information asynchronous from any tx
failure, so aliveness of lpni as reported by lnet_is_peer_ni_alive()
may not match what LND is telling us. Use existing reset flag to
set cached NI status down so we can be sure that remote NIs are
correctly set down.

Test-Parameters: trivial
HPE-bug-id: LUS-8897
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I1ab36b63d83fb35803eb13a330d698cfa49f17e9
Reviewed-on: https://review.whamcloud.com/38862
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12678 lnet: remove LNetMEUnlink and clean up related code 46/38646/13
Mr NeilBrown [Mon, 18 May 2020 00:45:05 +0000 (10:45 +1000)]
LU-12678 lnet: remove LNetMEUnlink and clean up related code

LNetMEUnlink is not particularly useful, and exposing it as an LNet
interface only provides the opportunity for it to be misused.

Every successful call to LNetMEAttach() is followed by a call to
LNetMDAttach().  If that call succeeds, the ME is owned by
the MD and the caller mustn't touch it again.
If the call fails, the caller is currently required to call
LNetMEUnlink(), which all callers do, and these are the only places
that LNetMEUnlink() are called.

As LNetMDAttach() knows when it will fail, it can unlink the ME itself
and save the caller the effort.
This allows LNetMEUnlink() to be removed which simplifies
the LNet interface.

LNetMEUnlink() is also used in in ptl_send_rpc() in a situation where
ptl_send_buf() fails.  In this case both the ME and the MD need to be
unlinked, as as they are interconnected, LNetMEUnlink() or
LNetMDUnlink() can equally do the job.  So change it to use
LNetMDUnlink().

LNetMEUnlink() is primarily a call the lnet_me_unlink(). It also
 - has some handling if ->me_md is not NULL, but that is never the
   case
 - takes the lnet_res_lock().  However LNetMDAttach() already
   takes that lock.
So none of this functionality is useful to LNetMDAttach().
On failure, it can call lnet_me_unlink() directly while ensuring
it still has the lock.

This patch:
 - moves the calls to lnet_md_validate() into lnet_md_build()
 - changes LNetMDAttach() to always take the lnet_res_lock(),
   and to call lnet_me_unlink() on failure.
 - removes all calls to LNetMEUnlink() and sometimes simplifies
   surrounding code.
 - changes lnet_md_link() to 'void' as it only ever returns
   '0', and thus simplify error handling in LNetMDAttach()
   and LNetMDBind()

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ied4e8fb544dbe1b32df7dc70439161dc74366a1d
Reviewed-on: https://review.whamcloud.com/38646
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12514 utils: try lustre_tgt filesystem for mount 99/38799/7
James Simmons [Sun, 21 Jun 2020 14:36:17 +0000 (10:36 -0400)]
LU-12514 utils: try lustre_tgt filesystem for mount

Now that Lustre supports a separate file system for the server
targets we can update mount.lustre to try "lustre_tgt" for the
server backend. For backwards compatibility if using "lustre_tgt"
fails try the original "lustre" file system type.

Change-Id: Ica08ff2abc3ad06c78d6d435db7e2fa3897e037e
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/38799
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
2 months agoLU-12687 osc: consume grants for direct I/O 96/35896/26
Vladimir Saveliev [Mon, 29 Jun 2020 11:26:57 +0000 (14:26 +0300)]
LU-12687 osc: consume grants for direct I/O

New IO engine implementation lost consuming grants by direct I/O
writes. That led to early emergence of out of space condition during
direct I/O. The below illustrates the problem:
  # OSTSIZE=100000 sh llmount.sh
  # dd if=/dev/zero of=/mnt/lustre/file bs=4k count=100 oflag=direct
  dd: error writing ‘/mnt/lustre/file’: No space left on device

Consume grants for direct I/O.

Try to consume grants in osc_queue_sync_pages() when it is called for
pages which are being writted in direct i/o.

Tests are added to verify grant consumption in buffered and direct i/o
and to verify direct i/o overwrite when ost is full.
The overwrite test is for ldiskfs only as zfs is unable to overwrite
when it is full.

Fixes: 9fe4b52ad2 ("LU-1030 osc: new IO engine implementation")
Signed-off-by: Vladimir Saveliev <c17830@cray.com>
Change-Id: I9a199452c564e8e8ad02f79231e8481166f3666e
Cray-bug-id: LUS-7036
Reviewed-on: https://review.whamcloud.com/35896
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
2 months agoLU-5338 tests: sanity-lfsck 11b allow larger last_id 49/34349/8
Andreas Dilger [Fri, 1 Mar 2019 00:54:41 +0000 (17:54 -0700)]
LU-5338 tests: sanity-lfsck 11b allow larger last_id

In sanity-lfsck test_11b allow the OST to report the last_id
larger than the previously-used OID on the MDS.  This may
happen if the OST preallocates additional objects or skips
some objects during recovery.

Add wait_update_cond() and wait_update_facet_cond() to
allow checking an arbitrary condition against the expected
result rather than only exactly equal to the expected value.

Remove the wait_result() helper function, which is essentially
the same thing as wait_update_facet(), and only has a few users.

Test-Parameters: trivial mdscount=2 mdtcount=4 testlist=sanity-lfsck
Test-Parameters: fstype=zfs mdscount=2 mdtcount=4 testlist=sanity-lfsck
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I1443d16699d8115fce664a331134ca6076ecab07
Reviewed-on: https://review.whamcloud.com/34349
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-9679 llite: annotate non-owner locking 34/39234/3
Mr NeilBrown [Thu, 2 Jul 2020 01:02:02 +0000 (11:02 +1000)]
LU-9679 llite: annotate non-owner locking

The lli_lsm_sem locks taken by ll_prep_md_op_data() are sometimes
released by a different thread.  This confuses lockdep unless we
explain the situation.

So use down_read_non_owner() and up_read_non_owner().

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ie6543706c658fc427461ef03448f3fcf90abaab7
Reviewed-on: https://review.whamcloud.com/39234
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-6142 fld: Fix style issues for fld_cache.c 05/39205/2
Arshad Hussain [Tue, 23 Jun 2020 19:20:27 +0000 (00:50 +0530)]
LU-6142 fld: Fix style issues for fld_cache.c

This patch fixes issues reported by checkpatch
for file lustre/fld/fld_cache.c

Test-Parameters: trivial
Signed-off-by: Arshad Hussain <arshad.super@gmail.com>
Change-Id: I91377fca9437fa31091ca039afba8f0c2ad8da7b
Reviewed-on: https://review.whamcloud.com/39205
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Ben Evans <beevans@whamcloud.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-6142 libcfs: resolve debug.c checkpatch issues 18/39118/4
James Simmons [Sun, 28 Jun 2020 12:14:01 +0000 (08:14 -0400)]
LU-6142 libcfs: resolve debug.c checkpatch issues

Cleanup up libcfs debug.c to resolve various checkpatch issues.
This also brings us into alignment with the Linux Lustre client.

Change-Id: I011a11ce7f0d5189186f5f9d8e32954ec0ce1ff7
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/39118
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13514 tests: remove upgrade images for conf-sanity 09/39109/4
James Nunez [Fri, 19 Jun 2020 18:01:42 +0000 (12:01 -0600)]
LU-13514 tests: remove upgrade images for conf-sanity

conf-sanity test 32a is hanging at a high rate.  We need to
explore if the issue involves old images are having problems
upgrading to the latest version of Lustre.

Test-Parameters: trivial
Test-Parameters: env=ONLY=32a,ONLY_REPEAT=20 fstype=ldiskfs testlist=conf-sanity
Test-Parameters: env=ONLY=32 fstype=zfs testlist=conf-sanity
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I0ff1e9e1304192b1008551b82133d95a0010c86a
Reviewed-on: https://review.whamcloud.com/39109
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
2 months agoLU-6142 osd-ldiskfs: Fix style issues for osd_io.c 88/38788/3
Arshad Hussain [Fri, 29 May 2020 12:13:37 +0000 (17:43 +0530)]
LU-6142 osd-ldiskfs: Fix style issues for osd_io.c

This patch fixes issues reported by checkpatch
for file lustre/osd-ldiskfs/osd_io.c

Test-Parameters: trivial
Signed-off-by: Arshad Hussain <arshad.super@gmail.com>
Change-Id: Iab3fdfa20271f53a794a88d71abfa4f0e72afb07
Reviewed-on: https://review.whamcloud.com/38788
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Ben Evans <beevans@whamcloud.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13677 quota: qunit sorting doesn't work 42/38942/2
Sergey Cheremencev [Mon, 15 Jun 2020 15:11:09 +0000 (18:11 +0300)]
LU-13677 quota: qunit sorting doesn't work

As sorting doesn't work correctly, sometimes
new decreased qunit may be not sent to OST.
If qunit reaches it's minimum and is not sent
to OST, this may cause a client to hung on
write as OST gets EINPROGRESS instead of
EDQUOT. Patch solves this issue.

Change-Id: I3ae22cc4d080968132ca762b9c0915a994ac126e
Signed-off-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-on: https://review.whamcloud.com/38942
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13606 lnet: Allow router to forward to healthier NID 98/38798/6
Chris Horn [Tue, 26 May 2020 18:47:50 +0000 (13:47 -0500)]
LU-13606 lnet: Allow router to forward to healthier NID

When a final-hop router (aka edge router) is forwarding a message,
if both the originator and destination of the message are mutli-rail
capable, then allow the router to choose a new destination lpni if
the one selected by the message originator is unhealthy or down.

Test-Parameters: trivial
HPE-bug-id: LUS-8905
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I4676edc0395584c9a8c396930f2db3d6ffd99eba
Reviewed-on: https://review.whamcloud.com/38798
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13581 build: xarray and lockdep_is_held const clash 50/39150/4
Shaun Tancheff [Sun, 28 Jun 2020 03:06:28 +0000 (22:06 -0500)]
LU-13581 build: xarray and lockdep_is_held const clash

xarray support added to lustre breaks building with RHEL
debug kernels. The root cause is due to an change in
the signature of lock_is_held when CONFIG_LOCKDEP is enabled.

Provide a workaround when the const mismatch conditions
exist to enable building RHEL debug kernel packages.

Also narrow the test for xarray support to explicitly require
xa_is_value be defined to protect against relying on
incomplete xarray support.

The same xarray issue is present in MOFED 5 so the same
scheme is used to protect lock_is_held change to require
const parameter from breaking.

Signed-off-by: James Simmons <jsimmons@infradead.org>
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: Icd51cfb111be6b30adf6f720fb680459ca8cf5b4
Reviewed-on: https://review.whamcloud.com/39150
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13723 lustre: Convert ERR_PTR(PTR_ERR()) to ERR_CAST() 04/39204/3
Arshad Hussain [Sat, 27 Jun 2020 12:25:18 +0000 (17:55 +0530)]
LU-13723 lustre: Convert ERR_PTR(PTR_ERR()) to ERR_CAST()

This patch converts ERR_PTR(PTR_ERR()) to ERR_CAST()
This fixes warning thrown by coccinelle

It also updates contrib/scripts/spelling.txt to catch
any future misuse of ERR_PTR(PTR_ERR())

Test-Parameters: trivial
Signed-off-by: Arshad Hussain <arshad.super@gmail.com>
Change-Id: I2214f46c08e5295dd139e7b4b245881cd9f6495a
Reviewed-on: https://review.whamcloud.com/39204
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Ben Evans <beevans@whamcloud.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13693 lfs: check early for MDS_OPEN_DIRECTORY 59/39159/4
John L. Hammond [Tue, 23 Jun 2020 18:00:55 +0000 (13:00 -0500)]
LU-13693 lfs: check early for MDS_OPEN_DIRECTORY

In mdt_reint_open() check earlier for MDS_OPEN_DIRECTORY/O_DIRECTORY
to avoid breaking leases used by lfs mirror when calling lfs
getstripe. Add a multi client version of sanity test_210 to sanityn.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I1860fc76c8014da3e637d83b487cb28b037ba71b
Reviewed-on: https://review.whamcloud.com/39159
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-9859 libcfs: don't call unshare_fs_struct() 32/39132/3
Mr NeilBrown [Sun, 7 Jun 2020 23:24:25 +0000 (19:24 -0400)]
LU-9859 libcfs: don't call unshare_fs_struct()

A kthread runs with the same fs_struct as init.
It is only helpful to unshare this if the thread
will change one of the fields in the fs_struct:
 root directory
 current working directory
 umask.

No lustre kthread changes any of these, so there is
no need to call unshare_fs_struct().

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I7309b6ed184b14a272bad7dc5149ad36281f948e
Reviewed-on: https://review.whamcloud.com/39132
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12678 socklnd: convert various refcounts to refcount_t 30/39130/2
Mr NeilBrown [Sun, 7 Jun 2020 23:24:27 +0000 (19:24 -0400)]
LU-12678 socklnd: convert various refcounts to refcount_t

Each of these refcounts exactly follows the expectations of
refcount_t, so change the atomic_t to refcoun_t.

We can remove the LASSERTs on incref/decref as they can now be enabled
at build time with CONFIG_REFCOUNT_FULL

Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I50f13465b588c30a70568a5800619cc7ec26293d
Reviewed-on: https://review.whamcloud.com/39130
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12678 socklnd: use list_for_each_entry_safe() 29/39129/2
Mr NeilBrown [Sun, 7 Jun 2020 23:24:28 +0000 (19:24 -0400)]
LU-12678 socklnd: use list_for_each_entry_safe()

Several loops use list_for_each_safe(), then call list_entry() as
first step.  These can be merged using list_for_each_entry_safe().

In one case, the 'safe' version is clearly not needed, so just use
list_for_each_entry().

Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I52eb3598cf4308dfac0aad2a493c4b91c4102e7d
Reviewed-on: https://review.whamcloud.com/39129
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12678 socklnd: use need_resched() 28/39128/2
Mr NeilBrown [Sun, 7 Jun 2020 23:24:29 +0000 (19:24 -0400)]
LU-12678 socklnd: use need_resched()

Rather than using a counter to decide when to drop the lock and see if
we need to reshedule we can use need_resched(), which is a precise
test instead of a guess.

Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: If13871a4a4c57ca87cbb1e22af85cb7fd24ab006
Reviewed-on: https://review.whamcloud.com/39128
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12678 o2iblnd: Use list_for_each_entry_safe 26/39126/2
Mr NeilBrown [Sun, 7 Jun 2020 23:24:31 +0000 (19:24 -0400)]
LU-12678 o2iblnd: Use list_for_each_entry_safe

Several loops use list_for_each_safe(), then call
list_entry() as first step.
These can be merged using list_for_each_entry_safe().

Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I01ba77b98bd4863fa37cbbe6b4072ac2513e5821
Reviewed-on: https://review.whamcloud.com/39126
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12678 o2iblnd: use need_resched() 25/39125/2
Mr NeilBrown [Sun, 7 Jun 2020 23:24:32 +0000 (19:24 -0400)]
LU-12678 o2iblnd: use need_resched()

Rather than using a counter to decide when to drop
the lock and see if we need to reshedule we can
use need_resched(), which is a precise test instead of a guess.

Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I43a1d9d0963622953761f25e13bc4781c2b02be2
Reviewed-on: https://review.whamcloud.com/39125
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12768 o2iblnd: wait properly for fps->increasing. 24/39124/2
Mr NeilBrown [Sun, 7 Jun 2020 23:24:33 +0000 (19:24 -0400)]
LU-12768 o2iblnd: wait properly for fps->increasing.

If we need to allocate a new fmr_pool and another thread is currently
allocating one, we call schedule() and then try again.  This can spin,
consuming a CPU and wasting power.

Instead, use wait_var_event() and wake_up_var() to
wait for fps_increasing to be cleared.

Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I16210fc6904d7605f4671f5edfa2f490526c3a16
Reviewed-on: https://review.whamcloud.com/39124
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12678 o2iblnd: Use ib_mtu_int_to_enum() 23/39123/2
Mr NeilBrown [Sun, 7 Jun 2020 23:24:34 +0000 (19:24 -0400)]
LU-12678 o2iblnd: Use ib_mtu_int_to_enum()

Rather than bespoke code for converting an MTU into the enum,
use ib_mtu_int_to_enum().
This has slightly different behaviour for invalid values,
but those are caught when the parameter is set.

Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I9ba4230fd1aad2c6c59233bac9558870cb1ffeda
Reviewed-on: https://review.whamcloud.com/39123
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-9859 libcfs: cleanup tracefile.h 16/39116/2
Mr NeilBrown [Sat, 20 Jun 2020 02:34:48 +0000 (22:34 -0400)]
LU-9859 libcfs: cleanup tracefile.h

With many things moved into tracefile.c we can cleanup a lot of
things in tracefile.h. Move some items that are only used in
tracefile.c from tracefile.h into tracefile.c. In tracefile.h
we have the ifdef LUSTRE_TRACEFILE_PRIVATE which has several
duplicate defines. We can remove those duplicates.

Test-Parameters: trivial

Change-Id: Ic45c4fa6ebfb21e26c536b6ff5cda780a7ca85ac
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Reviewed-on: https://review.whamcloud.com/39116
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13134 llite: fix to free cl_dio_aio properly 03/39103/6
Wang Shilong [Fri, 19 Jun 2020 13:59:42 +0000 (21:59 +0800)]
LU-13134 llite: fix to free cl_dio_aio properly

@cl_dio_aio is allocated by slab, we should use slab
free helper to free its memory.

Fixes: 3db1d ("LU-13134 obdclass: use slab allocation for cl_dio_aio")
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: Ic82f4dec5e9bb6de7a531c224fa23248d4f7d63d
Reviewed-on: https://review.whamcloud.com/39103
Reviewed-by: Patrick Farrell <farr0186@gmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
2 months agoLU-13680 osd-ldiskfs: handle large allocations 43/38943/4
Andreas Dilger [Mon, 15 Jun 2020 19:46:07 +0000 (13:46 -0600)]
LU-13680 osd-ldiskfs: handle large allocations

Use OBD_ALLOC_PAGE_ARRAY_LARGE() for oti_dio_pages, as this allocation
can be as large as 512KB due to large PTLRPC_MAX_BRW_PAGES.

Test-Parameters: trivial
Fixes: 72372486a5e9 ("LU-11347 osd: do not use pagecache for I/O")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I0a0557e42bb5db5612c78e6d9b87f366a23ebbe5
Reviewed-on: https://review.whamcloud.com/38943
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13673 misc: quiet compiler warning on armv7l 27/38927/6
Andreas Dilger [Fri, 12 Jun 2020 21:16:09 +0000 (15:16 -0600)]
LU-13673 misc: quiet compiler warning on armv7l

Add AM_CFLAGS when building lfs.c and lfs_project.c.

Avoid overflow in jobid_get_from_environ() and lu_prandom_u64_max().

Quiet printk() warning for mismatched type of size_t variables
by using %z modifier for those variables.

Fixes: 742897a967cf ("LU-13274 uapi: make lnet UAPI headers C99 compliant")
Fixes: df21a3b9eb01 ("LU-12790 obdclass: print jobid error message properly")
Fixes: 645b72c5c058 ("LU-12495 obdclass: generate random u64 max correctly")
Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ib424497d1543fd7840d630facf3210b6fa1344c7
Reviewed-on: https://review.whamcloud.com/38927
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13608 tgt: abort recovery while reading update llog 46/38746/5
Hongchao Zhang [Tue, 30 Jun 2020 11:22:10 +0000 (19:22 +0800)]
LU-13608 tgt: abort recovery while reading update llog

Abort the reading update LLOG fromt other MDTs when the recovery
is aborted, then the recovery process can be aborted in time.

This patch also adds watchdog for the process of the replay request
to detect possible stale process.

Change-Id: Ie2de041360c9eba95ef9bfd14b00ac2709e6eace
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/38746
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13519 osd-ldiskfs: expand inode project quota for upgrading 05/38505/8
Wang Shilong [Wed, 6 May 2020 04:45:25 +0000 (12:45 +0800)]
LU-13519 osd-ldiskfs: expand inode project quota for upgrading

When upgrading filesystem, it is possible that inode
it not big enough to hold project id field, and in that case
set project ID will return EOVERFLOW error.

Since ldiskfs have the logic to expand inode size automatically,
we could add similar logic for project quota.

Considering this as an rare case, we just call
ldiskfs_mark_inode_dirty() which will try to expand instead
of exporting more functions.

Change-Id: I941f33ce8f45d2015acc0a33c5b54cf3a771a452
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/38505
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13476 llite: Fix lock ordering in pagevec_dirty 17/38317/5
Shaun Tancheff [Wed, 6 May 2020 08:19:48 +0000 (03:19 -0500)]
LU-13476 llite: Fix lock ordering in pagevec_dirty

In vvp_set_pagevec_dirty lock order between i_pages and
lock_page_memcg was inverted with the expectation that
no other users would conflict.

However in vvp_page_completion_write the call to
test_clear_page_writeback does expect to be able
to lock_page_memcg then lock i_pages which appears
to conflict with the original analysis.

The reported case shows as RCU stalls with
vvp_set_pagevec_dirty blocked attempting to lock i_pages.

Fixes: a7299cb012f ("LU-9920 vvp: dirty pages with pagevec")
HPE-bug-id: LUS-8798
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I47c2107ddbef4a76325928e982abfc0ea666f39b
Reviewed-on: https://review.whamcloud.com/38317
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Patrick Farrell <farr0186@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13134 osc: re-declare ops_from/to to shrink osc_page 87/37487/8
Wang Shilong [Sat, 8 Feb 2020 11:41:46 +0000 (19:41 +0800)]
LU-13134 osc: re-declare ops_from/to to shrink osc_page

@ops_from and @ops_to is within PAGE_SIZE, use PAGE_SHIFT
bits to limit it is fine, on x86_64 platform, this patch
will reduce another 8 bytes.

Notice, previous @ops_to is exclusive which could be PAGE_SIZE,
this patch change it to inclusive which means max value will be
PAGE_SIZE - 1, and be careful to calculate its length.

After this patch, cl_page size could reduce from 320 to 312 bytes,
and we are able to allocate 13 objects using slab pool for 4K page.

Change-Id: Ic260c0a6580292301b5397276042e399c0f07e11
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/37487
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13134 obdclass: re-declare cl_page variables to reduce its size 80/37480/11
Wang Shilong [Sat, 8 Feb 2020 02:19:07 +0000 (10:19 +0800)]
LU-13134 obdclass: re-declare cl_page variables to reduce its size

With following changes:
1) make CPS_CACHED declare start from 1 consistent with CPT_CACHED
2) add CPT_NR to indicate max allowed CPT state value.
3) Reserve 4 bits for @cp_state which allow 15 kind of states
4) Reserve 2 bits for @cp_type which allow 3 kinds of cl_page types
5) use short int for @cp_kmem_index and We still have another 16 bits
reserved for future extension.
6)move @cp_lov_index after @cp_ref to fill 4 bytes hole.

After this patch, cl_page size could reduce from 336 bytes to 320 bytes

Change-Id: I92d5652a42850890ac6ce61e54884450dda25cc7
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/37480
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13134 obdclass: use offset instead of cp_linkage 28/37428/12
Wang Shilong [Tue, 4 Feb 2020 13:44:23 +0000 (21:44 +0800)]
LU-13134 obdclass: use offset instead of cp_linkage

Since we have fixed-size cl_page allocations, we could use
offset array to store every slices pointer for cl_page.

With this patch, we will reduce cl_page size from 392 bytes
to 336 bytes which means we could allocate from 10 to 12 objects.

Change-Id: I323bd589941125bfddf104f53a335d0cfee5c548
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/37428
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-9679 lustre: remove some "#ifdef CONFIG*" from .c files. 31/39131/4
Mr NeilBrown [Sun, 7 Jun 2020 23:24:26 +0000 (19:24 -0400)]
LU-9679 lustre: remove some "#ifdef CONFIG*" from .c files.

It is Linux policy to avoid #ifdef in C files where
convenient - .h files are OK.

This patch defines a few inline functions which differ
depending on CONFIG_LUSTRE_FS_POSIX_ACL, and removes
some #ifdefs from .c files.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I680bcf568d3a09d3768cc992a53671352bd125fd
Reviewed-on: https://review.whamcloud.com/39131
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-9679 nrs: remove ts_opcodes bitmap. 75/38975/2
Mr NeilBrown [Wed, 29 Apr 2020 02:46:43 +0000 (12:46 +1000)]
LU-9679 nrs: remove ts_opcodes bitmap.

This bitmap is never used.  There is one place that tests if it has
been allocated or not, but that place can easily be satisfied by a
strcmp().

So discard the field.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I97ca6753236f106d1865d514034b72a679f9b28a
Reviewed-on: https://review.whamcloud.com/38975
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13609 llog: list all the log files correctly on MGS/MDT 17/38917/6
Emoly Liu [Fri, 12 Jun 2020 08:12:00 +0000 (16:12 +0800)]
LU-13609 llog: list all the log files correctly on MGS/MDT

"lctl --device xxx llog_catlist" should list all the config log on
MGS and catalog on MDT correctly without any buffer size limit.
If data can't be fetched in one time, data->ioc_count is used to
save the number of all the fetched logs and then continue.

conf-sanity.sh test_123af is added to verify this patch.

Signed-off-by: Emoly Liu <emoly@whamcloud.com>
Change-Id: I364d563446833751b1f017fa2bef0351dab56235
Reviewed-on: https://review.whamcloud.com/38917
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Ben Evans <beevans@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-11814 obdcalss: ensure LCT_QUIESCENT take sync 16/38416/5
Yang Sheng [Wed, 29 Apr 2020 15:16:43 +0000 (23:16 +0800)]
LU-11814 obdcalss: ensure LCT_QUIESCENT take sync

Add locking in lu_device_init ensure LCT_QUIESCENT
operating can be seen on other thread in parallel
mounting. Also add extra checking before unset the
flag to make sure we don't do it after device has
been started.

(osd_handler.c:7730:osd_device_init0()) ASSERTION( info ) failed:
(osd_handler.c:7730:osd_device_init0()) LBUG
Pid: 28098, comm: mount.lustre 3.10.0-1062.9.1.el7_lustre.x86_64
Call Trace:
 libcfs_call_trace+0x8c/0xc0 [libcfs]
 lbug_with_loc+0x4c/0xa0 [libcfs]
 osd_device_alloc+0x778/0x8f0 [osd_ldiskfs]
 obd_setup+0x129/0x2f0 [obdclass]
 class_setup+0x48f/0x7f0 [obdclass]
 class_process_config+0x190f/0x2830 [obdclass]
 do_lcfg+0x258/0x500 [obdclass]
 lustre_start_simple+0x88/0x210 [obdclass]
 server_fill_super+0xf55/0x1890 [obdclass]
 lustre_fill_super+0x498/0x990 [obdclass]
 mount_nodev+0x4f/0xb0
 lustre_mount+0x18/0x20 [obdclass]
 mount_fs+0x3e/0x1b0
 vfs_kern_mount+0x67/0x110
 do_mount+0x1ef/0xce0
 SyS_mount+0x83/0xd0
 system_call_fastpath+0x25/0x2a
 0xffffffffffffffff
 Kernel panic - not syncing: LBUG

Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: Iccf3d545a5fc7c4a3b2320f1c7c7edcfbc1d17bb
Reviewed-on: https://review.whamcloud.com/38416
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-6142 lov: convert container_of0() calls. 82/38382/4
Mr NeilBrown [Mon, 27 Apr 2020 05:31:55 +0000 (15:31 +1000)]
LU-6142 lov: convert container_of0() calls.

Most calls to container_of0() in lustre/lov/ are preceded by an
LINVRNT() which assures us that the pointer is valid, so
container_of() can be used instead.

Only in lov2obd() is there not context, so that call is changed to use
container_of_safe()

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I8b3ccc8f0ac1a32122f043e8feec078fcfe2452b
Reviewed-on: https://review.whamcloud.com/38382
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-6142 lustre: convert use of container_of0 in include/ 80/38380/5
Mr NeilBrown [Mon, 27 Apr 2020 04:54:13 +0000 (14:54 +1000)]
LU-6142 lustre: convert use of container_of0 in include/

Most uses of container_of0() are changed to the upstream-standard
interface container_of_safe().  There is no clear context suggesting
that the pointer is known to be value, so it is consistent with the
current code to use the _safe() version.

In a few cases it is clear that the pointer must be valid.
This may be because:
 - it is a '.next' for a struct list_head
 - it from lo_object_next() whic his a special case of above
 - the returned value is confirmed to be valid by an LINVRNT()

So change all container_of0() to either container_of_safe() or
container_of().

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I9fdb8b7216667e58d3837ea555889f9346e4b10a
Reviewed-on: https://review.whamcloud.com/38380
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13653 mdt: ignore quota when creating slave stripe 75/38875/7
Hongchao Zhang [Wed, 24 Jun 2020 09:53:55 +0000 (17:53 +0800)]
LU-13653 mdt: ignore quota when creating slave stripe

When creating striped directory, the quota limit has been checked
on master MDT, the quota should be ignored when creating the slave
stripe object.

Change-Id: Ia53b1975a8d66c78725feb313659f7a9b889e735
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/38875
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13721 utils: fix 'lfs find --pool' for PFL files 96/39196/3
Andreas Dilger [Sat, 27 Jun 2020 06:04:42 +0000 (00:04 -0600)]
LU-13721 utils: fix 'lfs find --pool' for PFL files

Fix "lfs find --pool" to check the lov_user_md_v3 for the specified
pool name in find_check_pool() for composite files (PFL or FLR).
The v3 pointer was initialized from the main layout xattr, but was
not being refreshed for each of the components in the file.

Add a test case for "lfs find --pool" usage.

Fixes: 5a76aee24476 ("LU-8998 lfs: user space tools for PFL")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I6b6f21fd2fdf58f46972704cb6fb425a943ebbe5
Reviewed-on: https://review.whamcloud.com/39196
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13600 ptlrpc: re-enterable signal_completed_replay() 40/39140/4
Mikhail Pershin [Mon, 22 Jun 2020 18:04:34 +0000 (21:04 +0300)]
LU-13600 ptlrpc: re-enterable signal_completed_replay()

The signal_completed_replay() can meet race conditions while
checking imp_replay_inflight counter, so remove assertion and
check race conditions instead.

Fixes: 3b613a442b ("LU-13600 ptlrpc: limit rate of lock replays")
Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: Ib7c372b1757556b7285f380b40167742f9b71ec6
Reviewed-on: https://review.whamcloud.com/39140
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13731 llite: include linux/mm_types.h for VM_FAULT_RETRY 22/39222/2
Jian Yu [Wed, 1 Jul 2020 04:52:11 +0000 (21:52 -0700)]
LU-13731 llite: include linux/mm_types.h for VM_FAULT_RETRY

In RHEL 8.2 kernel 4.18.0-193.6.3.el8_2, VM_FAULT_RETRY is
defined in linux/mm_types.h instead of linux/mm.h. This
patch adds the #include in llite_internal.h to define
VM_FAULT_RETRY.

Test-Parameters: clientdistro=el8.2 serverdistro=el8.2 \
testlist=sanity

Test-Parameters: clientdistro=el8.1 serverdistro=el8.1 \
testlist=sanity

Change-Id: I0e48b32f661dab25392abf7924b5ac44f334d639
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/39222
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12678 socklnd: don't fall-back to tcp_sendpage. 34/39134/2
Mr NeilBrown [Sun, 7 Jun 2020 23:24:23 +0000 (19:24 -0400)]
LU-12678 socklnd: don't fall-back to tcp_sendpage.

sk_prot->sendpage is never NULL, so there is no
need for a fallback to tcp_sendpage.

Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Iaabf47790f2809fe98a0f09da31aa441021b26ab
Reviewed-on: https://review.whamcloud.com/39134
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12678 lnet: Fix some out-of-date comments. 27/39127/2
Mr NeilBrown [Sun, 7 Jun 2020 23:24:30 +0000 (19:24 -0400)]
LU-12678 lnet: Fix some out-of-date comments.

The structures these comments describe have changed or been removed,
but the comments weren't updated at the time.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I342edb56790290a0158d4907a5e775e88361ce08
Reviewed-on: https://review.whamcloud.com/39127
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12678 o2iblnd: allocate init_qp_attr on stack. 22/39122/2
Mr NeilBrown [Sun, 7 Jun 2020 23:24:35 +0000 (19:24 -0400)]
LU-12678 o2iblnd: allocate init_qp_attr on stack.

'struct ib_qp_init_attr' is not so large that it cannot be allocated
on the stack.  It is about 100 bytes, various other function in Linux
allocate it on the stack, and the stack isn't as constrained as it
once was.

So allocate on stack instead of using kmalloc and handling errors.

Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Id1f2f695f298d1883a5d6817092a6f89f1e072ef
Reviewed-on: https://review.whamcloud.com/39122
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-9859 libcfs: fold cfs_tracefile_*_arch into their only callers. 17/39117/2
Mr NeilBrown [Sat, 20 Jun 2020 02:47:19 +0000 (22:47 -0400)]
LU-9859 libcfs: fold cfs_tracefile_*_arch into their only callers.

There is no need to separate "arch" init/fini from
the rest, so fold it all in.
This requires some slightly subtle changes to clean-up
to make sure we don't walk lists before they are
initialized.

Test-Parameters: trivial

Change-Id: Id78385cfc29ab3d5162a5fb84283e7a6ce6bf91a
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Reviewed-on: https://review.whamcloud.com/39117
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-930 misc: update URLs in README 85/38985/3
Andreas Dilger [Thu, 18 Jun 2020 22:39:15 +0000 (16:39 -0600)]
LU-930 misc: update URLs in README

Update the URLs in the README file to reference the pages at
https://wiki.lustre.org/ instead of the WC pages.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I209956b74c19e8b1fd1c0f6d93b8b217073ebbe5
Reviewed-on: https://review.whamcloud.com/38985
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Peter Jones <pjones@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-9859 libcfs: move tgt_descs to standard Linux bitmaps. 81/38981/3
James Simmons [Fri, 19 Jun 2020 13:39:34 +0000 (09:39 -0400)]
LU-9859 libcfs: move tgt_descs to standard Linux bitmaps.

Originally the Linux kernel was lacking a uniform bitmap API so
Lustre created its own. Todays modern kernels support a standard
bitmap API so migrate tgt_descs to the standard API.

Change-Id: If43b520c29d16355189b1eb7f9fdab7309446545
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/38981
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13595 scripts: Add a debug option to lustre_rmmod 41/38941/5
Etienne AUJAMES [Thu, 11 Jun 2020 14:38:18 +0000 (16:38 +0200)]
LU-13595 scripts: Add a debug option to lustre_rmmod

The option is "-d" or "--debug-kernel", it uses "lctl debug_kernel" to
get debug message before unload libcfs.
For each module the script add a debug mark before unloading.

Example:
$ lustre_rmmod -d > /tmp/lustre_rmmod.log
$ lustre_rmmod -d lustre lnet > /tmp/lustre_rmmod2.log

Test-Parameters: trivial
Signed-off-by: Etienne AUJAMES <eaujames@ddn.com>
Change-Id: I6e44a24f2e786c08faf1db27de94e0f88ca65dc7
Reviewed-on: https://review.whamcloud.com/38941
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
2 months agoLU-13566 socklnd: fix local interface binding 43/38743/6
Amir Shehata [Wed, 17 Jun 2020 22:25:36 +0000 (15:25 -0700)]
LU-13566 socklnd: fix local interface binding

When a node is configured with multiple interfaces in
Multi-Rail config, socklnd was not utilizing the local interface
requested by LNet. In essence LNet was using all the NIDs in round
robin, however the socklnd module was not binding to the correct
interface. Traffic was thus sent on a subset of the interfaces.

The reason is that the route interface number was not being set.
In most cases lnet_connect() is called to create a socket. The
socket is bound to the interface provided and then
ksocknal_create_conn() is called to create the socklnd connection.
ksocknal_create_conn() calls ksocknal_associate_route_conn_locked()
at which point the route's local interface is assigned. However,
this is already too late as the socket has already been created
and bound to a local interface.

Therefore, it's important to assign the route's interface before
calling lnet_connect() to ensure socket is bound to correct local
interface.

To address this issue, the route's interface index is initialized
to the NI's interface index when it's added to the peer_ni.

Another bug fixed:
The interface index was not being initialized in the startup
routine.

Note: We're strictly assuming that there is one interface for each
NI. This is because tcp bonding will be removed from the socklnd as
it has been deprecated by LNet mutli-rail.

Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: Ibfa202ba009e07dbd69b19f1180790f1ea978ab1
Reviewed-on: https://review.whamcloud.com/38743
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13437 mdt: rename misses remote LOOKUP lock revoke 81/38181/16
Lai Siyao [Wed, 8 Apr 2020 14:55:22 +0000 (22:55 +0800)]
LU-13437 mdt: rename misses remote LOOKUP lock revoke

In rename, all objects but target may be remote, so to check whether
source is remote object on source parent, we need to compare which
MDTs they are located if both are remote. Add a helper function
mdt_rename_source_lock() to handle all possible combinations. If target
parent is remote, take remote LOOKUP for target on where target parent
is.

Add sanityn.sh 81c.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I2c134970d6abc8761528d01950b23495292cdf93
Reviewed-on: https://review.whamcloud.com/38181
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-930 doc: update James Simmons contact info 56/37756/3
James Simmons [Sun, 21 Jun 2020 14:50:17 +0000 (10:50 -0400)]
LU-930 doc: update James Simmons contact info

Update my email address to the one I have had for over 20 years.

Test-Parameters: trivial
Change-Id: Id29e9ef0ffb49fc1b568913d47bb24686b3175fc
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/37756
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
2 months agoLU-13180 osc: disable ext merging for rdma only pages and non-rdma 67/37567/2
Wang Shilong [Fri, 14 Feb 2020 06:50:11 +0000 (14:50 +0800)]
LU-13180 osc: disable ext merging for rdma only pages and non-rdma

This patch try to add logic to prevent CPU memory pages and RDMA
memory pages from merging into one RPC, codes which set OBD_BRW_RDMA_ONLY
will be added whenever RDMA only codes added later.

Change-Id: I11e8beda52cc533f17b2a40c34713f441e93d5b6
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/37567
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Gu Zheng <gzheng@ddn.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
2 months agoLU-8130 lu_object: convert lu_object cache to rhashtable 07/36707/32
Mr NeilBrown [Thu, 14 May 2020 12:03:33 +0000 (08:03 -0400)]
LU-8130 lu_object: convert lu_object cache to rhashtable

The lu_object cache is a little more complex than the other lustre
hash tables for two reasons.
1/ there is a debugfs file which displays the contents of the cache,
  so we need to use rhashtable_walk in a way that works for seq_file.

2/ There is a (sharded) lru list for objects which are no longer
   referenced, so finding an object needs to consider races with the
   lru as well as with the hash table.

The debugfs file already manages walking the libcfs hash table keeping
a current-position in the private data.  We can fairly easily convert
that to a struct rhashtable_iter.  The debugfs file actually reports
pages, and there are multiple pages per hashtable object.  So as well
as rhashtable_iter, we need the current page index.

For the double-locking, the current code uses direct-access to the
bucket locks that libcfs_hash provides.  rhashtable doesn't provide
that access - callers must provide their own locking or use rcu
techniques.

The lsb_waitq.lock is still used to manage the lru list, but with
this patch it is no longer nested *inside* the hashtable locks, but
instead is outside.  It is used to protect an object with a refcount
of zero.

When purging old objects from an lru, we first set
LU_OBJECT_HEARD_BANSHEE while holding the lsb_waitq.lock,
then remove all the entries from the hashtable separately.

When removing the last reference from an object, we first take the
lsb_waitq.lock, then decrement the reference and add to the lru list
or discard it setting LU_OBJECT_UNHASHED.

When we find an object in the hashtable with a refcount of zero, we
take the corresponding lsb_waitq.lock and check that neither
LU_OBJECT_HEARD_BANSHEE or LU_OBJECT_UNHASHED is set.  If neither is,
we can safely increment the refcount.  If either are, the object is
gone.

This way, we only ever manipulate an object with a refcount of zero
while holding the lsb_waitq.lock.

As there is nothing to stop us using the resizing capabilities of
rhashtable, the code to try to guess the perfect hash size has been
removed.

Also: the "is_dying" variable in lu_object_put() is racey - the value
could change the moment it is sampled.  It is also not needed as it is
only used to avoid a wakeup, which is not particularly expensive.
In the same code as comment says that 'top' could not be accessed, but
the code then immediately accesses 'top' to calculate 'bkt'.
So move the initialization of 'bkt' to before 'top' becomes unsafe.

Also: Change "wake_up_all()" to "wake_up()".  wake_up_all() is only
relevant when an exclusive wait is used.

Moving from the libcfs hashtable to rhashtable also gives the
benefit of a very large performance boost.

Before patch:

SUMMARY rate: (of 5 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   Directory creation:      12036.610      11091.880      11452.978        318.829
   Directory stat:          25871.734      24232.310      24935.661        574.996
   Directory removal:       12698.769      12239.685      12491.008        149.149
   File creation:           11722.036      11673.961      11692.157         15.966
   File stat:               62304.540      58237.124      60282.003       1479.103
   File read:               24204.811      23889.091      24048.577        110.245
   File removal:             9412.930       9111.828       9217.546        120.894
   Tree creation:            3515.536       3195.627       3442.609        123.792
   Tree removal:              433.917        418.935        428.038          5.545

After patch:

SUMMARY rate: (of 5 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   Directory creation:      11873.308        303.626       9371.860       4539.539
   Directory stat:          31116.512      30190.574      30568.091        335.545
   Directory removal:       13082.121      12645.228      12943.239        157.695
   File creation:           12607.135      12293.319      12466.647        138.307
   File stat:              124419.347     105240.996     116919.977       7847.165
   File read:               39707.270      36295.477      38266.011       1328.857
   File removal:             9614.333       9273.931       9477.299        140.201
   Tree creation:            3572.602       3017.580       3339.547        207.061
   Tree removal:              487.687          0.004        282.188        230.659

Change-Id: I618dc2e2da003c240a887126f600e7eac5df951c
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Reviewed-on: https://review.whamcloud.com/36707
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-8130 ptlrpc: convert conn_hash to rhashtable 16/33616/20
Mr NeilBrown [Tue, 7 Apr 2020 16:20:47 +0000 (12:20 -0400)]
LU-8130 ptlrpc: convert conn_hash to rhashtable

Linux has a resizeable hashtable implementation in lib,
so we should use that instead of having one in libcfs.

This patch converts the ptlrpc conn_hash to use rhashtable.
In the process we gain lockless lookup.

As connections are never deleted until the hash table is destroyed,
there is no need to count the reference in the hash table.  There
is also no need to enable automatic_shrinking.

Linux-commit: ac2370ac2bc5215daf78546cd8d925510065bb7f

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I0537564ba544ed06be42ba243606a884a1290f20
Reviewed-on: https://review.whamcloud.com/33616
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-9679 obdclass: remove init to 0 from lustre_init_lsi() 35/39135/2
Mr NeilBrown [Sun, 7 Jun 2020 23:24:21 +0000 (19:24 -0400)]
LU-9679 obdclass: remove init to 0 from lustre_init_lsi()

After allocating a struct with OBD_ALLOC, there is no value in setting
a few of the fields to zero.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ib1291661ba9124219e69c7a4d3c6ee4dcf14e021
Reviewed-on: https://review.whamcloud.com/39135
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13525 sec: better struct sepol_downcall_data 80/38580/7
Sebastien Buisson [Tue, 12 May 2020 15:58:15 +0000 (00:58 +0900)]
LU-13525 sec: better struct sepol_downcall_data

struct sepol_downcall_data is badly formed for several reasons:
- it uses a __kernel_time_t field, which can be variably sized,
  depending on the size of __kernel_long_t. Replace it with a
  fixed-size __s64 type;
- it has __u32 sdd_magic that is immediately before a potentially
  64-bit field, whereas the 64-bit fields in a structure should
  always be naturally aligned on 64-bit boundaries to avoid potential
  incompatibility in the structure definition;
- it has __u16 sdd_sepol_len which may be followed by padding.

So create a better struct sepol_downcall_data, while maintaining
compatibility with 2.12 by keeping a struct sepol_downcall_data_old.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I07c573c2eef64fb0c796d8af4acdc3428e0761a8
Reviewed-on: https://review.whamcloud.com/38580
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13365 ldlm: check slv and limit before updating 69/37969/3
Wang Shilong [Wed, 18 Mar 2020 07:51:17 +0000 (15:51 +0800)]
LU-13365 ldlm: check slv and limit before updating

slv and limit do not change for most of time,
ldlm_cli_update_pool() could be called for each RPC reply,
try hold read lock to check firstly could avoid heavy write
lock in hot path.

Change-Id: I4ef33132a463b3bac23863c09a2652f2d2782aae
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/37969
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
2 months agoLU-9679 osc: simplify osc_extent_find() 07/37607/7
NeilBrown [Thu, 13 Dec 2018 00:32:56 +0000 (11:32 +1100)]
LU-9679 osc: simplify osc_extent_find()

osc_extent_find() contains some code with the same functionality as
osc_extent_merge().  So replace that code with a call to
osc_extent_merge().

This requires that we set cur->oe_grants earlier, as
osc_extent_merge() needs that.

Also:

 - fix a pre-existing bug - osc_extent_merge() should never try to
   merge two extends with different ->oe_mppr as later alignment
   checks can get confused.
 - Remove a redundant list_del_init() which is already included in
   __osc_extent_remove().

Linux-Commit: 85ebb57ddc5b ("lustre: osc: simplify osc_extent_find()")

Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I5fa56e04ed707ee91f99179030dae4bd45456061
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Reviewed-on: https://review.whamcloud.com/37607
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-13711 build: fix typo on SSL dependency for Ubuntu 67/39167/3
Sebastien Buisson [Wed, 24 Jun 2020 14:57:54 +0000 (16:57 +0200)]
LU-13711 build: fix typo on SSL dependency for Ubuntu

On Ubuntu, SSL dependency is "libssl1.1".

Fixes: e1bf37870d ("LU-12214 build: fix build with gss enabled")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I186631ac2f08a3f5be0fb54fadec3d1455960e8c
Reviewed-on: https://review.whamcloud.com/39167
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
2 months agoLU-13701 tests: stop running sanity-lnet tests 55/39155/3
James Nunez [Tue, 23 Jun 2020 14:31:29 +0000 (08:31 -0600)]
LU-13701 tests: stop running sanity-lnet tests

sanity-lnet tests 204, 205, 206, 207 and 208 are failing at
a very high rate for ARM client testing.  We need to stop
running these tests, add them to the ALWAYS_EXCEPT list,
until we understand why these tests are failing.

Test-Parameters: trivial testgroup=review-ldiskfs-arm
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: Ia3e970748d719f25b156666a0b37ac9d89535532
Reviewed-on: https://review.whamcloud.com/39155
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
3 months agoLU-13693 lfs: avoid opening regular files for getstripe 79/38979/3
John L. Hammond [Thu, 18 Jun 2020 15:16:52 +0000 (10:16 -0500)]
LU-13693 lfs: avoid opening regular files for getstripe

In get_mds_md_size() just return a large enough size for all striping
attributes. This saves lfs getstripe from opening the file which was
interfering with leases used for mirroring.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Ia45eb8f6aa942507a55965afccbc28375788dff2
Reviewed-on: https://review.whamcloud.com/38979
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
3 months agoLU-13690 mdd: remove warnings in obd_lookup() 71/38971/2
John L. Hammond [Wed, 17 Jun 2020 21:07:58 +0000 (16:07 -0500)]
LU-13690 mdd: remove warnings in obd_lookup()

In obf_lookup() remove CWARN()s about invalid FID formats. Return
-ENOENT instead of -EINVAL for invalid FIDs. This removes noisy
console messages around volatile file handling in .lustre/fid/.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Ia76f0da38baa86b8d1173cfe0ede52e275f68a28
Reviewed-on: https://review.whamcloud.com/38971
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 months agoLU-12780 scrub: all update to bitfields must be protected. 74/38974/2
Mr NeilBrown [Thu, 18 Jun 2020 01:38:25 +0000 (11:38 +1000)]
LU-12780 scrub: all update to bitfields must be protected.

When a structure contains bitfields, these are updated by
a read-modify-write of the whole word.
If two of these updates can race, corruption can occurs - updates can
be lost.

To avoid this, it is best to protect updates with a spinlock.
Many updates to the os_* bit fields are already protected by
->os_lock.  This patch addes the lock to the remaining updates.
In many cases, this only requires moving a 'lock' earlier, or an
'unlock' later.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I2335d08cd53dcda98d8046d730829347456a6c5d
Reviewed-on: https://review.whamcloud.com/38974
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 months agoLU-12905 tests: wrappers for createmany and unlinkmany 85/36585/15
Alex Zhuravlev [Fri, 29 May 2020 09:38:43 +0000 (12:38 +0300)]
LU-12905 tests: wrappers for createmany and unlinkmany

which set debug=0 if number of operations is high enough.
this is to speedup testing.

according to Maloo sanity in review-ldiskfs takes ~9min less.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I0e0a0ef6cf217ecddd1b780103d01e2109fc33d9
Reviewed-on: https://review.whamcloud.com/36585
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 months agoLU-10934 tests: fix compilation without SELINUX 73/38973/2
Mr NeilBrown [Wed, 17 Jun 2020 23:46:39 +0000 (09:46 +1000)]
LU-10934 tests: fix compilation without SELINUX

lustre/tests/statx.c does not compile is SELINUX support
is not installed.
So add some #ifdefs to fix it.

Test-Parameters: trivial
Fixes: 3f7853b31ef6 ("LU-10934 llite: integrate statx() API with Lustre")
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ia0f285e1bb04270aff753bea71ffbe15a911db5f
Reviewed-on: https://review.whamcloud.com/38973
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
3 months agoLU-13657 kernel: kernel update RHEL8.2 [4.18.0-193.6.3.el8_2] 02/38902/2
Jian Yu [Thu, 11 Jun 2020 20:01:43 +0000 (13:01 -0700)]
LU-13657 kernel: kernel update RHEL8.2 [4.18.0-193.6.3.el8_2]

Update RHEL8.2 kernel to 4.18.0-193.6.3.el8_2.

Test-Parameters: trivial \
clientdistro=el8.2 serverdistro=el8.2 \
testlist=sanity

Change-Id: I7092768f227f93260f5a12c131154f1ec6fda1fd
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/38902
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 months agoLU-13662 lnet: handle undefined parameters 94/38894/2
Amir Shehata [Wed, 10 Jun 2020 22:27:23 +0000 (15:27 -0700)]
LU-13662 lnet: handle undefined parameters

If peer_tx_credits or peer_credits are 0, they should be
defaulted to the system defaults 8 and 256 respectively

Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: I351ff37cba0a9adaa1a6c25ff9c7da701724db2b
Reviewed-on: https://review.whamcloud.com/38894
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 months agoLU-13597 ofd: add more information to job_stats 16/38816/7
Emoly Liu [Wed, 3 Jun 2020 10:48:00 +0000 (18:48 +0800)]
LU-13597 ofd: add more information to job_stats

Request processing times/latency and basic IO size information
is added to the job_stats output. This allows monitoring per-job
request processing performance.
Except read_bytes and write_bytes in bytes units, all the others
use "usecs" units and show min/max/sum values. What's more, two
new counters for read and write time are added to calculate
bandwidth. The output format is like:
write_bytes: { samples: 1, unit: bytes, min: x, max: x, sum: x,
sumsq: x}

sanity.sh test_205b is modified to verify this patch.

Signed-off-by: Emoly Liu <emoly@whamcloud.com>
Change-Id: I7a5b77ca0ba464f6330a4bc56735c7762e167019
Reviewed-on: https://review.whamcloud.com/38816
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 months agoLU-13562 build: SUSE build support for azure, cray_ari_s 20/38620/2
Shaun Tancheff [Fri, 15 May 2020 14:21:22 +0000 (09:21 -0500)]
LU-13562 build: SUSE build support for azure, cray_ari_s

The lustre build for SUSE is hard coded for the default flavor
(aka kernel-default-devel) however it is useful to be able to
build for other flavors so the resulting packages can resolve
package dependencies as well as follow the expected naming
convention.

Test-Parameters: trivial
HPE-bug-id: LUS-8554
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: Id83e89ff4a2b9bf86b3f40c7a217440aa2b4fe94
Reviewed-on: https://review.whamcloud.com/38620
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 months agoLU-13345 kernel: kernel update SLES12 SP4 [4.12.14-95.48.1] 47/38247/6
Jian Yu [Mon, 8 Jun 2020 22:33:10 +0000 (15:33 -0700)]
LU-13345 kernel: kernel update SLES12 SP4 [4.12.14-95.48.1]

Update SLES12 SP4 kernel to 4.12.14-95.48.1 for Lustre client.

Test-Parameters: trivial clientdistro=sles12sp4 \
env=LNET_SELFTEST_EXCEPT=smoke,SANITY_EXCEPT="56oc 817"

Change-Id: I1c6971001e813807d37cc177c9969bad78048cf0
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/38247
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 months agoLU-13437 mdt: don't fetch LOOKUP lock for remote object 61/38561/9
Lai Siyao [Sun, 10 May 2020 07:22:36 +0000 (15:22 +0800)]
LU-13437 mdt: don't fetch LOOKUP lock for remote object

Pack parent FID in getattr by FID, which will be used to check whether
child is remote object on parent. The helper function is called
mdt_is_remote_object(). NB, directory shard is not treated as remote
object, because if so, client needs to revalidate shards when dir is
accessed, which will hurt performance much.

For getattr by FID, if object is remote file on parent, don't fetch
LOOKUP lock, otherwise client may see stale dir entries.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Id181ecc053579ee394080381a82706334503ced0
Reviewed-on: https://review.whamcloud.com/38561
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 months agoLU-12214 build: fix build with gss enabled 30/36430/18
Alexey Lyashkov [Fri, 4 Oct 2019 13:04:50 +0000 (16:04 +0300)]
LU-12214 build: fix build with gss enabled

provide a right dependences for the gss enabled lustre

Cray-bug-id: LUS-6033, LUS-7204
Test-parameters: trivial
Change-Id: Ib530a112f7f1629f7aea35cd4bad7c3f89e781ff
Signed-off-by: Alexey Lyashkov <c17817@cray.com>
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-on: https://review.whamcloud.com/36430
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 months agoLU-13675 o2iblnd: revert 'Timed out tx' patch 58/38958/4
Andreas Dilger [Wed, 17 Jun 2020 02:19:52 +0000 (02:19 +0000)]
LU-13675 o2iblnd: revert 'Timed out tx' patch

Revert "LU-1742 o2iblnd: 'Timed out tx' error message" patch
as this is causing crashes in o2iblnd consistently.

This reverts commit 7308662efc02fde077216f54728ecf278f31311b.

Test-Parameters: trivial
Change-Id: I470023f41eb1123de92aa3d86b32c7893363bc4e
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/38958
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: Shuichi Ihara <sihara@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 months agoLU-6142 mdt: Fix style issues for mdt_identity.c 28/38928/2
Arshad Hussain [Tue, 2 Jun 2020 19:27:02 +0000 (00:57 +0530)]
LU-6142 mdt: Fix style issues for mdt_identity.c

This patch fixes issues reported by checkpatch
for file lustre/mdt/mdt_identity.c

Test-Parameters: trivial
Signed-off-by: Arshad Hussain <arshad.super@gmail.com>
Change-Id: Ifb7f51ae4bdc8ab7dd816411a8017c379b298763
Reviewed-on: https://review.whamcloud.com/38928
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Ben Evans <beevans@whamcloud.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 months agoLU-13617 llite: don't hold inode_lock for security notify 92/38792/3
Alexander Boyko [Mon, 1 Jun 2020 12:32:11 +0000 (08:32 -0400)]
LU-13617 llite: don't hold inode_lock for security notify

With selinux enabled client has a dead lock which leads to
client eviction from MDS.
1 thread                    2 thread
do file open                do stat
inode_lock(parend dir)
                            got LDLM_PR(parent dir)
enqueue LDLM_CW(parent dir) waits on inode_lock to notify security
waits
timeout on enqueue
and client eviction because client didn't cancel a LDLM_PR lock

security_inode_notifysecctx()->selinux_inode_notifysecctx()->
selinux_inode_setsecurity()
The call of selinux_inode_setsecurity doesn't need to hold
inode_lock.

Fixes: 1d44980bcb ("LU-8956 llite: set sec ctx on client's inode at create time")
Signed-off-by: Alexander Boyko <alexander.boyko@hpe.com>
Cray-bug-id: LUS-8924
Change-Id: I4727da45590734bde57bee9d378b61c30b5d515a
Reviewed-on: https://review.whamcloud.com/38792
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andrew Perepechko <andrew.perepechko@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 months agoLU-6142 utils: Fix style issues for lfs.c 07/38707/4
Arshad Hussain [Fri, 22 May 2020 18:58:14 +0000 (00:28 +0530)]
LU-6142 utils: Fix style issues for lfs.c

This patch fixes issues reported by checkpatch
for file lustre/utils/lfs.c

Test-Parameters: trivial
Signed-off-by: Arshad Hussain <arshad.super@gmail.com>
Change-Id: Icc9ca0967c937e1fcd7b64f36f1e36f1a1f04f01
Reviewed-on: https://review.whamcloud.com/38707
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: Ben Evans <beevans@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 months agoLU-9812 spec: Fail rpmbuild if with servers but unconfigured 82/28282/9
Nathaniel Clark [Mon, 11 Mar 2019 18:10:30 +0000 (14:10 -0400)]
LU-9812 spec: Fail rpmbuild if with servers but unconfigured

Fail rpm build if "--with servers" used, but servers are not being
built after ./configure.  This would happen if --without ldiskfs but
zfs isn't found and thus configure turns off server support.

Test-Parameters: trivial
Change-Id: I57b07c7b7c5ad5bd73165238969ad3b9d2f3a5ab
Signed-off-by: Nathaniel Clark <nclark@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/28282
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 months agoLU-12275 sec: ioctls to handle encryption policies 73/37673/22
Sebastien Buisson [Thu, 20 Feb 2020 14:53:22 +0000 (14:53 +0000)]
LU-12275 sec: ioctls to handle encryption policies

Introduce support for fscrypt IOCTLs that handle encryption
policies v2. It enables setting/getting encryption policies on
individual directories, letting users decide how they want to
encrypt specific directories.

fscrypt encryption policies v2 are supported from Linux 5.4.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I0dc8c9ca1291ddd9c44617feb5df845b57d7dcc9
Reviewed-on: https://review.whamcloud.com/37673
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 months agoLU-12275 tests: exercise file content encryption/decryption 91/36191/36
Sebastien Buisson [Mon, 16 Sep 2019 15:00:24 +0000 (15:00 +0000)]
LU-12275 tests: exercise file content encryption/decryption

Add new tests to sanity-sec in order to exercise file content
encryption/decryption. Also test encrypted file length, especially
when multiple threads are writting to the same file in parallel.

Test-Parameters: trivial
Test-Parameters: clientdistro=el8.1 testgroup=review-ldiskfs
Test-Parameters: clientdistro=el8.1 testgroup=review-ldiskfs-dne
Test-Parameters: clientdistro=el8.1 testgroup=review-ldiskfs-arm
Test-Parameters: clientdistro=el8.1 testgroup=review-dne-selinux
Test-Parameters: clientdistro=el8.1 testgroup=review-dne-part-1 env=SANITYN_EXCEPT=106
Test-Parameters: clientdistro=el8.1 testgroup=review-dne-part-2 env=SANITY_PCC_EXCEPT="4"
Test-Parameters: clientdistro=el8.1 testgroup=review-dne-part-3
Test-Parameters: clientdistro=el8.1 testgroup=review-dne-part-4
Test-Parameters: clientdistro=el8.1 testgroup=review-zfs
Test-Parameters: clientdistro=el8.1 testgroup=review-dne-zfs-part-1 env=SANITYN_EXCEPT=106
Test-Parameters: clientdistro=el8.1 testgroup=review-dne-zfs-part-2 env=SANITY_PCC_EXCEPT="4"
Test-Parameters: clientdistro=el8.1 testgroup=review-dne-zfs-part-3
Test-Parameters: clientdistro=el8.1 testgroup=review-dne-zfs-part-4
Test-Parameters: testlist=sanity-sec envdefinitions=ONLY="36 37 38 39 40 41 42 43 44 45 46 47 48" clientdistro=el8.1 fstype=ldiskfs
Test-Parameters: testlist=sanity-sec envdefinitions=ONLY="36 37 38 39 40 41 42 43 44 45 46 47 48" clientdistro=el8.1 fstype=zfs
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I339e748e0213da980fe7779bb06ae9b3bd91bf5c
Reviewed-on: https://review.whamcloud.com/36191
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>