Whamcloud - gitweb
fs/lustre-release.git
9 months agoRevert "LU-13120 build: Fix ZFS dependancies for osd-zfs-mount" 20/37320/3
Andreas Dilger [Thu, 23 Jan 2020 23:02:35 +0000 (23:02 +0000)]
Revert "LU-13120 build: Fix ZFS dependancies for osd-zfs-mount"

This reverts commit fb687e35402fa6755589657a67dbe30be09ba9c5.

All review-dne-zfs-part-[1234] sessions fail with the most recent
master landings, and this seems like the likely culprit.

Change-Id: Id0295d65a642e7c2ef6367dac72d89acfca8a6b4
Reviewed-on: https://review.whamcloud.com/37320
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
9 months agoLU-11276 ldlm: fix lock convert races 66/36466/11
Vitaly Fertman [Wed, 16 Oct 2019 16:07:56 +0000 (19:07 +0300)]
LU-11276 ldlm: fix lock convert races

The blocking cb may be triggered in parallel and the convert logic
of the DOM lock must be ready that the cancel_bits could be already
zeroed by the first executor.

As there may be several blocking cb parallel executors and several
conversion callers, each requesting for different inode bits, setup
the following logic:
- the lock keeps the aggregated set of bits requested for cancelling
  by different parties, where 0 means the whole lock is to be
  cancelled, and where the CBPENDING flag means there is a canceling
  job pending;
- once completed, the cancel_bits are zeroed and the CBPENDING flag
  is dropped, meaning the next request will be a part of the next job;
- once a local lock is converted, its state is changed appropriately
  and no cleanup is left for the interpret time as the lock is ready
  for the next usage;
- as the lock is unlocked in a process of conversion and more bits
  may appear, check it and repeat appropriately;
- let just 1 conversion executor to work at a time, others are waiting
  similar to ldlm_cli_cancel();
- there are others who may want to cancel unused locks (cancel_lru,
  cancel_resource_local), consider CANCELING as a request to cancel
  the full lock independently of the cancel_bits;

Some cleanups are done:
- move the cache drop logic to the CANCELING part of the blocking cb
  from the BLOCKING one;
- remove the convert RPC interpret, as the lock cleanups are already
  done in advance; the convert RPC is re-sendable and an error means
  there is a serioes net problem;

Test-Parameters: testlist=racer,racer,racer
Signed-off-by: Vitaly Fertman <c17818@cray.com>
Change-Id: I901de34241704ed801152f071cb7f610fe6f4bfe
Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/36466
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-13119 osd-ldiskfs: set f_cred for app armour 84/37184/4
James Simmons [Fri, 10 Jan 2020 14:30:47 +0000 (09:30 -0500)]
LU-13119 osd-ldiskfs: set f_cred for app armour

The function interate_dir() interfaces with the security layer.
For some kernel versions on platforms that use app armour it
expects f_cred to be set. Currently osd-ldiskfs open codes the
creation of struct file so it is missing a cred. Fix this by
setting f_cred to the default current_cred().

Test-Parameters: testlist=sanity-lfsck serverdistro=sles12sp3

Change-Id: I38487e8ae99a0f70d6e430935b7d19523d414b4b
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/37184
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
9 months agoLU-13121 llite: fix deadlock in ll_update_lsm_md() 82/37182/2
Lai Siyao [Tue, 7 Jan 2020 13:30:38 +0000 (21:30 +0800)]
LU-13121 llite: fix deadlock in ll_update_lsm_md()

Deadlock may happen in in following senario: a lookup process called
ll_update_lsm_md(), it found lli->lli_lsm_md is NULL, then
down_write(&lli->lli_lsm_sem). but another lookup process initialized
lli->lli_lsm_md after this check and before write lock, so the first
lookup process called up_read(&lli->lli_lsm_sem) and return, so the
write lock is never released, which cause subsequent lookups deadlock.

Rearrange the code to simplify the locking:
1. take read lock.
2. if lsm was initialized and unchanged, release read lock and return.
3. otherwise release read lock and take write lock.
4. free current lsm and initialize with new lsm.
5. release write lock.
6. initialize stripes with read lock.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Ifcc25a957983512db6f29105b5ca5b6ec914cb4b
Reviewed-on: https://review.whamcloud.com/37182
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-13120 build: Fix ZFS dependancies for osd-zfs-mount 69/37169/3
Shaun Tancheff [Thu, 9 Jan 2020 11:57:29 +0000 (05:57 -0600)]
LU-13120 build: Fix ZFS dependancies for osd-zfs-mount

lustre-osd-zfs-mount depends on zfs
lustre-osd-zfs-mount depends on kmod-lustre-osd-zfs

SuSE packaging style prefers kmp package naming so prepare
for adopting a kmp named zfs package

Test-Parameters: trivial
Cray-bug-id: LUS-7077
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Change-Id: I510a46dd3d0e6d58a1e0db36226d412ee06016ec
Reviewed-on: https://review.whamcloud.com/37169
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-13117 libcfs: fix to match right key in cfs_get_environ() 56/37156/4
Wang Shilong [Wed, 8 Jan 2020 01:45:27 +0000 (09:45 +0800)]
LU-13117 libcfs: fix to match right key in cfs_get_environ()

It does the memcmp() to match the environment variable
with the desired key, then accounts for the "=" when
calculating length. But it fails to check that the next
character is actually an equals sign. In the case of
any key which is also the prefix to some other variable

Also add debug information for debugging similar issue
in the future.

Test-Parameters: trivial
Change-Id: Ia2b4ccd1f10c89059cecc224d4e2ba8d1d75b825
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/37156
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-13098 ptlrpc: supress connection restored message 86/37086/4
Alex Zhuravlev [Sat, 21 Dec 2019 15:40:20 +0000 (18:40 +0300)]
LU-13098 ptlrpc: supress connection restored message

if that happens on idling connection.

Fixes: 5a6ceb664f07 ("LU-7236 ptlrpc: idle connections can disconnect")
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I506665d427f3e77477f53e2d3059bcb1daaf0318
Reviewed-on: https://review.whamcloud.com/37086
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-13092 lbuild: include lbuild-{fc,rhel,sles} to SIGNATURE 76/37076/4
Wang Shilong [Thu, 9 Jan 2020 01:34:28 +0000 (09:34 +0800)]
LU-13092 lbuild: include lbuild-{fc,rhel,sles} to SIGNATURE

We should include these files to calculate SIGNATURE, for example
bump kernel extra tags could happen there.

Test-Parameters: trivial
Change-Id: I2c62ad765d3c6a1b9e99affe3be95a404d6140c5
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/37076
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Gu Zheng <gzheng@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-13053 tests: fix conf-sanity call to umount_ldiskfs 49/36949/3
James Nunez [Fri, 6 Dec 2019 16:38:13 +0000 (09:38 -0700)]
LU-13053 tests: fix conf-sanity call to  umount_ldiskfs

conf-sanity test 87 calls umount_ldiskfs(), but the function
in test-framework.sh is unmount_ldiskfs().  We need to
change the function call in test 87 to unmount_ldiskfs().

Test-Parameters: trivial testlist=conf-sanity
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I3e0818a229341c4fab8aee923cad2253b7dd634d
Reviewed-on: https://review.whamcloud.com/36949
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12678 lnet: remove locking protection ln_testprotocompat 56/36856/4
Mr NeilBrown [Fri, 17 Jan 2020 19:36:51 +0000 (14:36 -0500)]
LU-12678 lnet: remove locking protection ln_testprotocompat

lnet_net_lock(LNET_LOCK_EX) is a heavy-weight lock that is not
necessary here.  The bits in this field are only set rarely - via an
ioctl - and the pattern for reading and clearing them exactly
matches test_and_clear_bit().  So change the field to "unsigned
long" (so test_and_clear_bit() can be used), and use
test_and_clear_bit(), discarding all other locking.

Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ie420fcb3d547d9ec04025b921d5b24bd8f2fcce3
Reviewed-on: https://review.whamcloud.com/36856
Reviewed-by: Alexey Lyashkov <c17817@cray.com>
Reviewed-by: Chris Horn <hornc@cray.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12678 lnet: make "struct lnet_lnd" always "const". 32/36832/3
Mr NeilBrown [Sun, 24 Nov 2019 23:00:50 +0000 (10:00 +1100)]
LU-12678 lnet: make "struct lnet_lnd" always "const".

Every place where "struct lnet_lnd" appears, "const" is
added in front. Now all those structs can be in read-only
memory which is generally more secure.

Linux-commit 07499855083e ("lnet: make "struct lnet_lnd"
                            always "const".")

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I54a73d5b12de8c6b9a98182577c3c30d05c00222
Reviewed-on: https://review.whamcloud.com/36832
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Chris Horn <hornc@cray.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12678 socklnd: initialize the_ksocklnd at compile-time. 31/36831/5
Mr NeilBrown [Wed, 15 Jan 2020 15:31:45 +0000 (10:31 -0500)]
LU-12678 socklnd: initialize the_ksocklnd at compile-time.

All other lnds initialize this struct at compile-time.
It is best for socklnd to do so too.

Test-Parameters: trivial testlist=sanity-lnet

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I3acd636f6f5ba783a2c60bf18ffc46c98e091c13
Reviewed-on: https://review.whamcloud.com/36831
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-11385 odbclass: Handle gracefully if nsproxy is NULL 02/36802/5
Serguei Smirnov [Tue, 19 Nov 2019 22:18:17 +0000 (14:18 -0800)]
LU-11385 odbclass: Handle gracefully if nsproxy is NULL

Gracefully handle the case if current->nsproxy is NULL:
check for the condition and return an error, avoiding attempts
to dereference the pointer.

Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: Ia102d2bacdb0e54b0339985396447e6d25465c56
Reviewed-on: https://review.whamcloud.com/36802
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12968 mgs: Prevent reading past end of buffer 53/36753/5
Shaun Tancheff [Thu, 14 Nov 2019 01:28:46 +0000 (19:28 -0600)]
LU-12968 mgs: Prevent reading past end of buffer

KASAN reported
  BUG: KASAN: slab-out-of-bounds in mgs_wlp_lcfg+0xb3/0x4a0 [mgs]
  Read of size 64 at addr ffff8880b8f9fe40 by task ll_mgs_0002/17603

On memory allocated here.
  mgs_write_log_target+0x2ae/0x910 [mgs]

In mgs_wlp_lcfg( ..., char *ptr) ptr is a string so use strlcpy
instead of memcpy to avoid reading past the end of the buffer

Cray-bug-id: LUS-8137
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Change-Id: I539c0b4d878d26c44f64a4cd5746a8fba1bef2fa
Reviewed-on: https://review.whamcloud.com/36753
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
9 months agoLU-12904 ldiskfs: Add ldiskfs support for linux 5.4 83/36583/6
Shaun Tancheff [Wed, 15 Jan 2020 13:47:00 +0000 (07:47 -0600)]
LU-12904 ldiskfs: Add ldiskfs support for linux 5.4

Linux 5.4 ext4 has some changes from 5.0 this
fixes up the ldiskfs patches to apply against 5.4

Test-Parameters: trivial
Cray-bug-id: LUS-8042
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Change-Id: I116226ec9297eead4dfd3403be748f732e67f54f
Reviewed-on: https://review.whamcloud.com/36583
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-13141 ldiskfs: block alloc performance patch 50/37250/2
Shaun Tancheff [Wed, 15 Jan 2020 12:22:13 +0000 (06:22 -0600)]
LU-13141 ldiskfs: block alloc performance patch

Add block alloc performance patch to CentOS 7.7, 8.0 and
Ubuntu 19.04 5.0 kernel.

Cray-bug-id: LUS-8402
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Change-Id: Ifeb78839e5dbe8731bbb5532906708b97d4d9d33
Reviewed-on: https://review.whamcloud.com/37250
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Artem Blagodarenko <c17828@cray.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12791 kernel: kernel update RHEL 8.0 [4.18.0-80.11.2.el8_0] 27/36527/5
Jian Yu [Fri, 10 Jan 2020 18:01:04 +0000 (10:01 -0800)]
LU-12791 kernel: kernel update RHEL 8.0 [4.18.0-80.11.2.el8_0]

Update RHEL 8.0 kernel to 4.18.0-80.11.2.el8_0.

Test-Parameters: trivial clientdistro=el8 \
testlist=sanity

Change-Id: I4081719fa9a8c83ea0e8bff46dc9d54774cabb56
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/36527
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12214 selinux: Remove concatenating of selinux context 23/36423/5
Alexey Lyashkov [Fri, 8 Nov 2019 07:58:28 +0000 (10:58 +0300)]
LU-12214 selinux: Remove concatenating of selinux context

Remove concatenating of context for the temporary mount point
if selinux is enabled.
mount.zfs don't have that option, so revert it for consistency.
It can be added with -o option if needed.

Cray-bug-id: LUS-5992
Test-Parameters: clientselinux mdtcount=4 testlist=sanity,recovery-small,sanity-sec,sanity-selinux

Change-Id: If471de13e201c5cdcb28631b90b2efa13d8f2b4f
Signed-off-by: Alexey Lyashkov <c17817@cray.com>
Reviewed-on: https://review.whamcloud.com/36423
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12521 llapi: add separate fsname and instance API 51/35451/10
Andreas Dilger [Tue, 9 Jul 2019 18:47:19 +0000 (12:47 -0600)]
LU-12521 llapi: add separate fsname and instance API

The llapi_getname() function returns the combined fsname and client
instance as one string, which is fine when using the entire string,
but the output cannot be safely parsed into separate fsname and
instance strings in all cases.

Introduce new llapi_get_fsname() and llapi_get_instance() functions
that return only the fsname and instance strings, since the source
string returned from the kernel can be unambiguously separated before
it is returned in a combined string via llapi_getname().

Fix the lfs_getname() '-n' and '-i' options to use the new routines
rather than parsing the output from llapi_getname().

Add man pages for these functions.

Fixes: 2a4821b836c8 ("LU-12159 utils: improve lfs getname functionality")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Iaf5846a0ae147a428f66ec8a1d0251e7e12540e5
Reviewed-on: https://review.whamcloud.com/35451
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12806 llapi: use name_to_handle_at in llapi_fd2fid 92/36292/19
Quentin Bouget [Thu, 12 Dec 2019 08:47:23 +0000 (09:47 +0100)]
LU-12806 llapi: use name_to_handle_at in llapi_fd2fid

Reimplement llapi_fd2fid so as to use name_to_handle_at() rather than
using an ioctl() call.

This patch also updates llapi_fid2path as using file descriptor
obtained from a call to open() + O_PATH is valid with
name_to_handle_at(), is more efficient and also works for symlinks
out of the box.

Signed-off-by: Quentin Bouget <quentin.bouget@cea.fr>
Change-Id: Ic7e83c7fdf924363ed59a0681267d960e660db6d
Reviewed-on: https://review.whamcloud.com/36292
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12919 lnet: Fix source specified route selection 22/36622/2
Chris Horn [Thu, 31 Oct 2019 02:57:45 +0000 (21:57 -0500)]
LU-12919 lnet: Fix source specified route selection

If lnet_send() is called with a specific src_nid, but
rtr_nid == LNET_NID_ANY and the message needs to be routed, then we
need to ensure that the lnet_peer_ni of our next hop is on the same
network as the lnet_ni associated with the src_nid. Otherwise we
may end up choosing an lnet_peer_ni that cannot be reached from
the specified source.

Test-Parameters: trivial
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: Idc5b808f90170a3480d523ba4726cc48c3387ddb
Reviewed-on: https://review.whamcloud.com/36622
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexandr Boyko <c17825@cray.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-10467 ptlrpc: refactor waiting in ptlrpc_set_wait() 81/35981/13
Mr NeilBrown [Fri, 10 Jan 2020 14:01:32 +0000 (09:01 -0500)]
LU-10467 ptlrpc: refactor waiting in ptlrpc_set_wait()

ptlrpc_set_wait can wait either with or without signals blocked.
After it has waited, it possibly checks if a signal is pending and if
so, marks the set as interrupted.

The code for the check examines lwi.lwi_allow_intr which was set
before the wait.  Converting this to use upstream wait primitives
will remove lwi, so we need another way to handle this.

The current test looks wrong.  It is
        if (rc == -ETIMEDOUT &&
            (!lwi.lwi_allow_intr || set->set_allow_intr) &&
            signal_pending(current)) {

but if lwi.lwi_allow_intr is true, then the wait will have allowed
signals and so the set will already have been interrupted if needed.
So the case where lwi.lwi_allow_intr is true and set->set_allow_intr
is also true, should be irrelevant.

i.e. the condition should just be
        if (rc == -ETIMEDOUT &&
            !lwi.lwi_allow_intr &&
            signal_pending(current)) {

which it was before
 Commit afcf3026c6ad ("LU-6684 lfsck: stop lfsck even
                       if some servers offline")

Given this, if we move the l_wait_event() into each branch of the
'if', we can then move the extra condition and
ptlrpc_interrupted_set() call into the 'else' branch - the only place
the condition would fire, and simplify the condition to

        if (rc == -ETIMEDOUT &&
            signal_pending(current)) {

This will make the two waits separate, so they can be easily
converted.

Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I1e1819c697cea47607d5fc4a018c898236b33f4b
Reviewed-on: https://review.whamcloud.com/35981
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12470 tests: increase pdirops timeout 04/37304/2
Andreas Dilger [Wed, 22 Jan 2020 07:32:38 +0000 (00:32 -0700)]
LU-12470 tests: increase pdirops timeout

There are pretty regular failures of the sanityn pdirops test_40-47.
Increase the timeout slightly to reduces the frequency of failures.

Fixes: 743b85a32e24 ("LU-2233 tests: improve tests sanityn/40-47")
Test-Parameters: trivial testlist=sanityn,sanityn,sanityn,sanityn
Test-Parameters: testlist=sanityn,sanityn,sanityn,sanityn,sanityn
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ie1ea5704edab97b61f563135b4cc2491dc3ebbe5
Reviewed-on: https://review.whamcloud.com/37304
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12460 llite: replace lli_trunc_sem 71/35271/11
NeilBrown [Fri, 19 Jul 2019 18:57:16 +0000 (14:57 -0400)]
LU-12460 llite: replace lli_trunc_sem

lli_trunc_sem can lead to a deadlock.

vvp_io_read_start takes lli_trunc_sem, and can take
mmap sem in the direct i/o case, via
generic_file_read_iter->ll_direct_IO->get_user_pages_unlocked

vvp_io_fault_start is called with mmap_sem held (taken in
the kernel page fault code), and takes lli_trunc_sem.

These aren't necessarily the same mmap_sem, but can be if
you mmap a lustre file, then read into that mapped memory
from the file.

These are both 'down_read' calls on lli_trunc_sem so they
don't directly conflict, but if vvp_io_setattr_start() is
called to truncate the file between these, it does
'down_write' on lli_trunc_sem.  As semaphores are queued,
this down_write blocks subsequent reads.

This means if the page fault has taken the mmap_sem,
but not yet the lli_trunc_sem in vvp_io_fault_start,
it will wait behind the lli_trunc_sem down_write from
vvp_io_setattr_start.

At the same time, vvp_io_read_start is holding the
lli_trunc_sem and waiting for the mmap_sem, which will not
be released because vvp_io_fault_start cannot get the
lli_trunc_sem because the setattr 'down_write' operation is
queued in front of it.

Solve this by replacing with a hand-coded semaphore, using
atomic counters and wait_var_event().  This allows a
special down_read_nowait which ignores waiting down_write
operations.  This combined with waking up all waiters at
once guarantees that down_read_nowait can always 'join'
another down_read, guaranteeing our ability to take the
semaphore twice for read and avoiding the deadlock.

I'd like there to be a better way to fix this, but I
haven't found it yet.

Signed-off-by: NeilBrown <neilb@suse.com>
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Ibd3abf4df1f1f6f45e440733a364999bd608b191
Reviewed-on: https://review.whamcloud.com/35271
Reviewed-by: Neil Brown <neilb@suse.de>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-8066 lfsck: use underscores in lfsck status files 15/36715/5
Andreas Dilger [Wed, 8 Jan 2020 15:51:52 +0000 (10:51 -0500)]
LU-8066 lfsck: use underscores in lfsck status files

Use underscores to separate words in the lfsck_layout
and oi_scrub files instead of hyphens, to match the use
in lfsck_namespace and consistency with other files.

Test-Parameters: trivial testlist=sanity-lfsck,sanity-scrub
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I5306d82a4e6056a5098d1753f53cb4ee4e2540e5
Reviewed-on: https://review.whamcloud.com/36715
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12928 tests: start running recovery-small 136 09/37009/2
James Nunez [Fri, 13 Dec 2019 16:19:53 +0000 (09:19 -0700)]
LU-12928 tests: start running recovery-small 136

recovery-small test 136 was not run due to repeated crashes
when run with SELinux and shared secrect key enabled.  The
crash was fixed and we need to start running this test
again meaning recovery-small test 136 needs to be removed
from the ALWAYS_EXCEPT list.

Test-Parameters: trivial
Test-Parameters: envdefinitions=SHARED_KEY=true clientselinux testlist=recovery-small
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: If7e2e7932c3916f0588f943404b59aa5653bbf17
Reviewed-on: https://review.whamcloud.com/37009
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
9 months agoLU-10664 tests: fix MPI tests in dom-performance.sh 44/37044/4
Mikhail Pershin [Mon, 16 Dec 2019 18:16:24 +0000 (21:16 +0300)]
LU-10664 tests: fix MPI tests in dom-performance.sh

Make MPI tests in dom-performance.sh to be ran under
mpiuser instead of root

Test-Parameters: trivial mdssizegb=20 testlist=dom-performance
Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: Ief3036434191a0bc153d5c8e380183d5e5067dc4
Reviewed-on: https://review.whamcloud.com/37044
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-13039 quota: Ensure local buffer is null terminated 99/36899/3
Shaun Tancheff [Sat, 30 Nov 2019 18:14:56 +0000 (12:14 -0600)]
LU-13039 quota: Ensure local buffer is null terminated

Found via KASAN, copy_from_user may not set null
terminator.

Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Change-Id: I20de911c2b2d50a1715a27e3edfe65442eaa2be6
Reviewed-on: https://review.whamcloud.com/36899
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Sergey Cheremencev <c17829@cray.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-13005 lnet: discard LNetEQGet and LNetEQWait 40/36840/4
Mr NeilBrown [Tue, 7 Jan 2020 18:14:51 +0000 (13:14 -0500)]
LU-13005 lnet: discard LNetEQGet and LNetEQWait

These interfaces are never used and are not particularly useful,
so discard them.

Test-Parameters: trivial testlist=sanity-lnet

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Iaf2bc9ec2638820c3e4334e40cf2cf6993237f7d
Reviewed-on: https://review.whamcloud.com/36840
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-13004 ptlrpc: Allow BULK_BUF_KIOV to accept a kvec 24/36824/7
Mr NeilBrown [Sun, 5 Jan 2020 16:11:30 +0000 (11:11 -0500)]
LU-13004 ptlrpc: Allow BULK_BUF_KIOV to accept a kvec

Bulk descriptor of type PTLRPC_BULK_BUF_KIOV are comprised
of a list of page+offset+len.
If the calling code actually has a virtual-address+len, it
cannot current use BULK_BUF_KIOV and must use BULK_BUF_KVEC.

However it is quite easy to convert virtual-address+len
to a list of page+offset+len.

So we can add a ->add_iov_frag interface for KIOV descriptors, and
then we will be able to use KIOV descriptors for everything.  The
caller must ensure to allocate a large enough descriptor, taking
into account the size of each exptected kvec.

Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: If8bc5dc9f6e89a196bd72d3ac9b88c4ea5da83d1
Reviewed-on: https://review.whamcloud.com/36824
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12865 tests: fix sanity 160f to be more robust 68/36468/7
Andreas Dilger [Thu, 17 Oct 2019 07:19:26 +0000 (16:19 +0900)]
LU-12865 tests: fix sanity 160f to be more robust

The sanity test_160f test was failing intermittently because the first
Changelog user ("cl6") was being unregistered in some cases when it
set changelog_max_idle_time=10, but the test slept for 9s and then did
some operations that could be slow.  In rare cases the test runs too
long and the MDS evicts the "good" user along with the bad user:

   MDD0000: Force deregister of ChangeLog user cl7 idle more than 35s
   MDD0000: Force deregister of ChangeLog user cl6 idle more than 11s

Change the test sleep interval to be half of the max_idle limit so
that there is no risk of the "good" Changelog user being evicted.

Add some logging to the test so that it is easier to correlate test
script actions with events in the MDS debug log.

Fixes: 31fef6845e8b ("LU-10680 mdd: create gc thread when no current transaction")
Test-Parameters: trivial envdefinitions=ONLY=160 testlist=sanity,sanity
Test-Parameters: envdefinitions=ONLY=160 mdscount=2 testlist=sanity,sanity
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I0e4c9c271d98a2716f848e75676780b0383ebbe5
Reviewed-on: https://review.whamcloud.com/36468
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-8130 lu_object: factor out extra per-bucket data 16/36216/8
NeilBrown [Thu, 12 Dec 2019 23:51:01 +0000 (18:51 -0500)]
LU-8130 lu_object: factor out extra per-bucket data

The hash tables managed by lu_object store some extra
information in each bucket in the hash table.  This prevents the use
of resizeable hash tables, so lu_site_init() goes to some trouble
to try to guess a good hash size.

There is no real need for the extra data to be closely associated with
hash buckets.  There is a small advantage as both the hash bucket and
the extra information can then be protected by the same lock, but as
these locks have low contention, that should rarely be noticed.

The extra data is updated frequently and accessed rarely, such an lru
list and a wait_queue head.  There could just be a single copy of this
data for the whole array, but on a many-cpu machine, that could become
a contention bottle neck.  So it makes sense keep multiple shards and
combine them only when needed.  It does not make sense to have many
more copies than there are CPUs.

This patch takes the extra data out of the hash table buckets and
creates a separate array, which never has more entries than twice the
number of possible cpus.  As this extra data contains a
wait_queue_head, which contains a spinlock, that lock is used to
protect the other data (counter and lru list).

The code currently uses a very simple hash to choose a
hash-table bucket:

 (fid_seq(fid) + fid_oid(fid)) & (CFS_HASH_NBKT(hs) - 1)

There is no documented reason for this and I cannot see any value in
not using a general hash function. We can use hash_32() and hash_64()
on the fid value with a random seed created for each lu_site. The
hash_*() functions where picked over the jhash() functions since
it performances way better.

The lock ordering requires that a hash-table lock cannot be taken
while an extra-data lock is held.  This means that in
lu_site_purge_objects() we much first remove objects from the lru
(with the extra information locked) and then remove each one from the
hash table.  To ensure the object is not found between these two
steps, the LU_OBJECT_HEARD_BANSHEE flag is set.

As the extra info is now separate from the hash buckets, we cannot
report statistic from both at the same time.  I think the lru
statistics are probably more useful than the hash-table statistics, so
I have preserved the former and discarded the latter.  When the
hashtable becomes resizeable, those statistics will be irrelevant.

As the lru and the hash table are now managed by different locks
we need to be careful to prevent htable_lookup() finding an
object that lu_site_purge_objects() is purging.
To help with this we introduce a new lu_object flag to say
that and object is being purged.  Once set, the object will
be quickly removed from the hash table, and is already
removed from the lru.

Change-Id: I2a7402a348377d3b17f76e8617216e5b7ff9b99a
Signed-off-by: NeilBrown <neilb@suse.com>
Reviewed-on: https://review.whamcloud.com/36216
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12756 lnet: Refactor lnet_compare_routes 21/36621/2
Chris Horn [Thu, 31 Oct 2019 02:26:14 +0000 (21:26 -0500)]
LU-12756 lnet: Refactor lnet_compare_routes

Restrict lnet_compare_routes() to only comparing the lnet_route
objects passed as arguments. This saves us from doing unecessary
calls to lnet_find_best_lpni_on_net().

Rename lnet_compare_peers to lnet_compare_gw_lpnis to better
reflect what is done by this routine.

Test-Parameters: trivial
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I2d7b5dcc2aacb371b21908ceebf2dd6a349fa74c
Reviewed-on: https://review.whamcloud.com/36621
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexandr Boyko <c17825@cray.com>
Reviewed-by: Alexey Lyashkov <c17817@cray.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12756 lnet: Remove unused vars in lnet_find_route_locked 20/36620/2
Chris Horn [Sun, 27 Oct 2019 19:35:05 +0000 (14:35 -0500)]
LU-12756 lnet: Remove unused vars in lnet_find_route_locked

The lp and lp_best variables are not needed in
lnet_find_route_locked().

Test-Parameters: trivial
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I61a7097ab66703a1af1346c7301b9efc7e4392c9
Reviewed-on: https://review.whamcloud.com/36620
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexandr Boyko <c17825@cray.com>
Reviewed-by: Alexey Lyashkov <c17817@cray.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12756 lnet: Avoid extra lnet_remotenet lookup 36/36536/4
Chris Horn [Tue, 22 Oct 2019 00:46:14 +0000 (19:46 -0500)]
LU-12756 lnet: Avoid extra lnet_remotenet lookup

We can keep track of the lnet_remotenet object associated with the
"best" lnet_peer_net, and pass that lnet_remotenet directly to
lnet_find_route_locked().

Test-Parameters: trivial
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: Ib9808ca885c698ba6c73c5243fbce8b3f499b790
Reviewed-on: https://review.whamcloud.com/36536
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexandr Boyko <c17825@cray.com>
Reviewed-by: Alexey Lyashkov <c17817@cray.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-13115 mdt: handle mdt_pack_sectx_in_reply() errors 48/37148/4
Mikhail Pershin [Tue, 7 Jan 2020 09:49:20 +0000 (12:49 +0300)]
LU-13115 mdt: handle mdt_pack_sectx_in_reply() errors

The mdt_pack_secctx_in_reply() contains mo_xattr_get() call
which -ENOENT error should be checked and exit by error path
if needed.

In DNE environment lu_object may lost its LOHA_EXISTS flag
during osp_xattr_get() and that should be handled to don't
proceed with code paths for existent objects.

Test-Parameters: clientselinux mdtcount=4 envdefinitions=ONLY=185a testlist=sanity,sanity,sanity,sanity
Test-Parameters: clientselinux mdtcount=4 testlist=sanity,recovery-small,sanity-sec
Test-Parameters: clientselinux mdtcount=4 testgroup=review-dne-selinux
Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I55ad666f58dd3fae3ed097018aa23ed94818d246
Reviewed-on: https://review.whamcloud.com/37148
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-9679 llite: fix possible race with module unload. 20/37020/4
Mr NeilBrown [Fri, 3 Jan 2020 00:50:55 +0000 (19:50 -0500)]
LU-9679 llite: fix possible race with module unload.

lustre_fill_super() calls client_fill_super() without holding a
reference to the module containing client_fill_super.  If that
module is unloaded at a bad time, this can crash.

To be able to get a reference to the module using
try_get_module(), we need a pointer to the module.

So replace
  lustre_register_client_fill_super() and
  lustre_register_kill_super_cb()
with a single
  lustre_register_super_ops()
which also passed a module pointer.

Then use a spinlock to ensure the module pointer isn't removed
while try_module_get() is running, and use try_module_get() to
ensure we have a reference before calling client_fill_super().

Now that we take the reference to the module before calling
luster_fill_super(), we don't need to take one inside
lustre_fill_super().

Linux-commit: d487fe31f49e78f3cdd826923bf0c340a839ffd8

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I9474622f2a253d9882eae3f0578c50782dd11ad4
Reviewed-on: https://review.whamcloud.com/37020
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoNew tag 2.13.51 2.13.51 v2_13_51
Oleg Drokin [Sat, 11 Jan 2020 03:21:29 +0000 (22:21 -0500)]
New tag 2.13.51

Change-Id: I2ce973d9b599ed426b56d7892176205ba6822910

9 months agoLU-12923 lnet: Replace CLASSERT() with BUILD_BUG_ON() 13/37113/3
Arshad Hussain [Sun, 29 Dec 2019 12:46:19 +0000 (18:16 +0530)]
LU-12923 lnet: Replace CLASSERT() with BUILD_BUG_ON()

This patch replaces CLASSERT() with kernel defined
BUILD_BUG_ON()

Test-Parameters: trivial
Signed-off-by: Arshad Hussain <arshad.super@gmail.com>
Change-Id: I94292ca4729c19e0651fad285943ae02584afc03
Reviewed-on: https://review.whamcloud.com/37113
Reviewed-by: Aurelien Degremont <degremoa@amazon.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12923 lustre: Replace CLASSERT() with BUILD_BUG_ON() 11/37111/5
Arshad Hussain [Sun, 29 Dec 2019 08:34:31 +0000 (14:04 +0530)]
LU-12923 lustre: Replace CLASSERT() with BUILD_BUG_ON()

This patch replaces remaining CLASSERT() with kernel defined
BUILD_BUG_ON()

Test-Parameters: trivial
Signed-off-by: Arshad Hussain <arshad.super@gmail.com>
Change-Id: Ie23846f8d67cac1872bda9c7e20fe9bc888bf365
Reviewed-on: https://review.whamcloud.com/37111
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
9 months agoLU-13090 utils: fix lfs_migrate -p for file with pool 67/37067/6
Andreas Dilger [Thu, 19 Dec 2019 11:51:41 +0000 (04:51 -0700)]
LU-13090 utils: fix lfs_migrate -p for file with pool

If "lfs_migrate -p <pool>" is run to migrate a file with an existing
pool, the given pool is overridden by the existing pool from the file
during migration.  Fix this to use the OST pool requested by the user.

Don't print a warning about deprecated -n option if --dry-run is used.

If a pool is specified, use it with "lfs df" to find OST free space.

Change temp filename to work better with new DNE "crush" hash.

Don't return an error if falling back to rsync and no links are found.

Add test for "lfs_migrate -p" and update man page and usage to match.
Clean up debug-level helpers in test-framework.sh.

Test-Parameters: trivial testlist=ost-pools
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ief69a620fc969aeff24ec0633a3314c3b83ebbe5
Reviewed-on: https://review.whamcloud.com/37067
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-13088 ldlm: Fix sleeping function called in atomic 63/37063/2
Mr NeilBrown [Thu, 19 Dec 2019 05:55:35 +0000 (16:55 +1100)]
LU-13088 ldlm: Fix sleeping function called in atomic

target_recovery_overseer() can sleep while holding a spinlock, which
triggers a BUG warning.

It is easily fixed by dropping the spinlock before waiting.  In the
case where the task waits, no useful information that could be
protected by the spinlock is held, so nothing can be lost by dropping
it.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I8bb3d02523b5dcfadac19f01ccb736d7b7f28239
Reviewed-on: https://review.whamcloud.com/37063
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-13059 kernel: kernel update RHEL7.7 [3.10.0-1062.9.1.el7] 60/36960/3
Jian Yu [Mon, 9 Dec 2019 09:38:44 +0000 (01:38 -0800)]
LU-13059 kernel: kernel update RHEL7.7 [3.10.0-1062.9.1.el7]

Update RHEL7.7 kernel to 3.10.0-1062.9.1.el7.

Test-Parameters: trivial clientdistro=el7.7 serverdistro=el7.7

Change-Id: I11fc7a2c382a5c234698bfb30a38a08ed29fef03
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/36960
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-930 doc: fix formatting errors in lfs_migrate.1 59/36959/2
Andreas Dilger [Mon, 9 Dec 2019 09:04:49 +0000 (02:04 -0700)]
LU-930 doc: fix formatting errors in lfs_migrate.1

Add missing .TP sections for the command-line options.
Remove duplicate EXAMPLES section and '--yes' from bad merge.

Test-Parameters: trivial
Fixes: 99d7a8ed43b ("LU-8207 scripts: add auto-stripe option to lfs_migrate")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I5de290e00c5fd718e53ac0fc801d44e1cf3ebbe5
Reviewed-on: https://review.whamcloud.com/36959
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12637 kernel: new kernel [RHEL 8.1 4.18.0-147.3.1.el8_1] 46/36946/7
Jian Yu [Fri, 3 Jan 2020 07:28:20 +0000 (23:28 -0800)]
LU-12637 kernel: new kernel [RHEL 8.1 4.18.0-147.3.1.el8_1]

This patch makes changes to support new RHEL 8.1 release
for Lustre client.

Test-Parameters: trivial clientdistro=el8.1 \
envdefinitions=SANITY_EXCEPT="411" \
testlist=sanity

Change-Id: Ifcc0a15c3ad9afa99b670641f91b23c1a5c0668e
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/36946
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12678 lnet: use lnet_accept_magic, not le32_to_cpu. 57/36857/3
Mr NeilBrown [Wed, 6 Nov 2019 05:59:45 +0000 (16:59 +1100)]
LU-12678 lnet: use lnet_accept_magic, not le32_to_cpu.

This le32_to_cpu() looks wrong, as the argument is a CPU value, not
le32, and the value is being compared to something that might be
le32.  Previous code used lnet_accept_magic() for tests on 'magic',
so it seems to make sense to use lnet_accept_magic() here too.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I3f04bb087d4ae3d6785e77072b51132f9440bd32
Reviewed-on: https://review.whamcloud.com/36857
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12678 lnet: lnet_startup_lndnet: avoid use-after-free 55/36855/2
Mr NeilBrown [Wed, 6 Nov 2019 05:42:34 +0000 (16:42 +1100)]
LU-12678 lnet: lnet_startup_lndnet: avoid use-after-free

If lnet_startup_lndni() fails it will free 'ni' (via lnet_ni_free()).
So we mustn't de-reference it in the LASSERT() in that case

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I01e35013e028a8f95f169e25aeb0c344b2310380
Reviewed-on: https://review.whamcloud.com/36855
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12678 lnet: change list_for_each in ksocknal_debug_peerhash 36/36836/3
Mr NeilBrown [Mon, 18 Nov 2019 01:24:09 +0000 (12:24 +1100)]
LU-12678 lnet: change list_for_each in ksocknal_debug_peerhash

This list_for_each() loop searches for a particular entry,
then acts of in.  It currently acts after the loop by testing
if the variable is NULL.  When we convert to list_for_each_entry()
it won't be NULL.

Change the code so the acting happens inside the loop.
 list_for_each_entry() {
    if (this isn't it)
        continue;
    act on entry;
    goto done; // break out of 2 loops
}

Note that identing is deliberately left unchanged,
as the next patch will change the 2 loops to a single loop,
after which the current indents will be correct.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Idea32bf2ab4037650d6698d4f82f6b6764b4d1b2
Reviewed-on: https://review.whamcloud.com/36836
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12678 lnet: discard struct ksock_peer 35/36835/4
Mr NeilBrown [Tue, 31 Dec 2019 18:10:47 +0000 (13:10 -0500)]
LU-12678 lnet: discard struct ksock_peer

struct ksock_peer is declared in a forward-ref, but
never defined or used.  Let's remove it, and change
some spaces to TABs while we are there.

Test-Parameters: trivial testlist=sanity-lnet

Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I8a86a77a5cad606a374e60a5b8920be28308587d
Reviewed-on: https://review.whamcloud.com/36835
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12678 lnet: prepare to make lnet_lnd const. 30/36830/5
Mr NeilBrown [Mon, 30 Dec 2019 15:54:09 +0000 (10:54 -0500)]
LU-12678 lnet: prepare to make lnet_lnd const.

Preferred practice is for structs containing function
pointers to be 'const'.  Such structs are generally tempting
attack vectors, and making them const allows linux to place
them in read-only memory, thus reducing the attack surface.

'struct lnet_lnd' is mostly function pointers, but contains
one writable field - a list_head.

Rather than keeping registered lnds in a linked-list, we can place
them in an array indexed by type - type numbers are at most 15 so
this is not a burden.

With these changes, no part of an lnet_lnd is ever modified.

Test-Parameters: trivial
Test-Parameters: testlist=sanity-lnet

Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I08c7df551109e05ca4a3cef866e8df737d1a1ad4
Reviewed-on: https://review.whamcloud.com/36830
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-8130 ldlm: simplify ldlm_ns_hash_defs[] 20/36220/3
NeilBrown [Fri, 20 Dec 2019 14:55:07 +0000 (09:55 -0500)]
LU-8130 ldlm: simplify ldlm_ns_hash_defs[]

As the ldlm_ns_types are dense, we can use the type as
the index to the array, rather than searching through
the array for a match.
We can also discard nsd_hops as all hash tables now
use the same hops.
This makes the table smaller and the code simpler.

Change-Id: I2aebb9d533d676bed51a7422801545be4fbb7e1e
Signed-off-by: NeilBrown <neilb@suse.com>
Reviewed-on: https://review.whamcloud.com/36220
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
9 months agoLU-12542 handle: rename ops to owner 98/35798/10
NeilBrown [Fri, 20 Dec 2019 14:37:48 +0000 (09:37 -0500)]
LU-12542 handle: rename ops to owner

Now that portals_handle_ops contains only a char*,
it is functioning primarily to identify the owner of each handle.
So change the name to h_owner, and the type to const char*.

Note: this h_owner is now quite different from the similar h_owner
in the server code.  When server code it merged the
"med" pointer should be stored in the "mfd" and validated separately.

Change-Id: Ie2e9134ea22c4929683c84bf45c41b96b348d0a2
Signed-off-by: NeilBrown <neilb@suse.com>
Reviewed-on: https://review.whamcloud.com/35798
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-9091 sysfs: use string helper like functions for sysfs 58/35658/22
James Simmons [Fri, 3 Jan 2020 02:47:07 +0000 (21:47 -0500)]
LU-9091 sysfs: use string helper like functions for sysfs

For a very long time the Linux kernel has supported the function
memparse() that allowed the passing in of memory sizes with the
suffix set of K, M, G, T, P, E. Lustre adopted this approach
with its proc / sysfs implmentation. The difference being that
lustre expanded this functionality to allow sizes with a
fractional component, such as 1.5G for example. The code used to
parse for the numerical value is heavily tied into the debugfs
seq_file handling and stomps on the passed in buffer which you
can't do with sysfs files.

Similar functionality to what Lustre does today exist in newer
linux kernels in the form of string helpers. Currently the
string helpers only convert a numerical value to human readable
format. A new function, string_to_size(), was created that takes
a string and turns it into a numerical value. This enables the
use of string helper suffixes i.e MiB, kB etc with the lustre
tunables and we can now support 10 base numbers i.e MB, kB as
well. Already string helper suffixes are used for debugfs files
so I expect this to be adopted over time so it should be
encouraged to use string_to_size() for newer lustre sysfs files.

At the same time we want to perserve the original behavior of
using the suffix set of K, M, G, T, P, E. To do this we create
the function sysfs_memparse() that supports the new string helper
suffixes as well as the older set of suffixes. This new code is
also way simpler than what is currently done with the current code.

Change-Id: Ia437db44f2a987aa11ab4ff3e9df23e9aeba04d7
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/35658
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12477 libcfs: Remove obsolete config checks 42/35342/11
Patrick Farrell [Mon, 30 Dec 2019 13:24:49 +0000 (08:24 -0500)]
LU-12477 libcfs: Remove obsolete config checks

Remove a few config checks for kernel versions we no longer
support. Only 3.10+ kernels are now supported.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I4f4177c512a37fb7a78bab69aa89aa7199ab30b4
Reviewed-on: https://review.whamcloud.com/35342
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12756 lnet: Avoid comparing route to itself 35/36535/4
Chris Horn [Tue, 22 Oct 2019 00:35:42 +0000 (19:35 -0500)]
LU-12756 lnet: Avoid comparing route to itself

The first iteration of the route selection loop compares the first
route in the list with itself.

Test-Parameters: trivial
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I1a51b04b248dbaa9a47a7a69e2995c21e515fb2b
Reviewed-on: https://review.whamcloud.com/36535
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexandr Boyko <c17825@cray.com>
Reviewed-by: Alexey Lyashkov <c17817@cray.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12756 lnet: Refactor lnet_find_best_lpni_on_net 34/36534/4
Chris Horn [Mon, 21 Oct 2019 22:15:27 +0000 (17:15 -0500)]
LU-12756 lnet: Refactor lnet_find_best_lpni_on_net

Replace lnet_send_data argument.

Test-Parameters: trivial
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: Ic346eaf6870f2a7c68c7f4c45d424f4f924370d9
Reviewed-on: https://review.whamcloud.com/36534
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexandr Boyko <c17825@cray.com>
Reviewed-by: Alexey Lyashkov <c17817@cray.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-13069 obdclass: don't skip records for wrapped catalog 96/36996/2
Alexander Boyko [Thu, 12 Dec 2019 07:59:41 +0000 (02:59 -0500)]
LU-13069 obdclass: don't skip records for wrapped catalog

osp_sync_thread() uses opd_sync_last_catalog_idx as a start point of
catalog processing. It is used at llog_cat_process_cb also, to skip
records from processing. When catalog is wrapped, processing starts
from second part of catalog and then a first part. So, a first part
would be skipped at llog_cat_process_cb() base on lpd_startcat.

osp_sync_thread() restarts a processing loop with a
opd_sync_last_catalog_idx. For a wrapped it increases last
index and one more increase do a llog_process_thread. This leads
to a skipped records at catalog, they would not be processed.
The patch fixes these issues.
It also adds sanity test 135 and 136 as regression tests.

Signed-off-by: Alexander Boyko <c17825@cray.com>
Cray-bug-id: LUS-8053,LUS-8236
Change-Id: Ic75af1bf4468b9ef2de32cbf6d834b6a81376e88
Reviewed-on: https://review.whamcloud.com/36996
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Alexander Zarochentsev <c17826@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-9679 lnet: always check return of try_module_get() 54/36854/6
Mr NeilBrown [Wed, 6 Nov 2019 04:45:27 +0000 (15:45 +1100)]
LU-9679 lnet: always check return of try_module_get()

try_module_get() can fail, so the return value should be checked.
If we *know* that we already hold a reference, __module_get()
should be used instead.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Id526f9ae3829a50fe7df7069230804322cd4558e
Reviewed-on: https://review.whamcloud.com/36854
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Chris Horn <hornc@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-9679 obdclass: don't manage module refs in open/close. 21/37021/3
Mr NeilBrown [Sun, 15 Dec 2019 21:29:30 +0000 (08:29 +1100)]
LU-9679 obdclass: don't manage module refs in open/close.

Core Linux code for managing char-devs ensures that the relevant
module is held active while a char-dev is open - see cdev_get()
and cdev_put().
So there is no need for lustre/obd_class to manage the module
ref count as well.

As this is all that obd_class_open and obd_class_close do, those
functions can be removed.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I84b0dc81c830cefc2383f184d12beeb2cfa22404
Reviewed-on: https://review.whamcloud.com/37021
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-10467 osp: use wait_event_idle_timeout() 88/35988/10
Mr NeilBrown [Wed, 28 Aug 2019 23:28:20 +0000 (09:28 +1000)]
LU-10467 osp: use wait_event_idle_timeout()

osp has 4 LWI_TIMEOUT() calls that pass an on_timeout
function.
In each case, the on_timeout function returns 1, so this
is equivalent to using wait_event_idle_timeout(), and
calling the function if the timeout happened.

One of the two functions passed does nothing except return 1, so it
can be ignored.
The other function, used only once, contains a CDEBUG message,
so we now call that when wait_event_idle_timeout() returns 0.

Change-Id: Ic153266e412d684c4aa6c7204ff5755d991d83c6
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Reviewed-on: https://review.whamcloud.com/35988
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-10467 ptlrpc: convert waiting in sptlrpc_req_refresh_ctx() 87/35987/10
Mr NeilBrown [Mon, 30 Dec 2019 15:01:24 +0000 (10:01 -0500)]
LU-10467 ptlrpc: convert waiting in sptlrpc_req_refresh_ctx()

The l_wait_event call in sptlrpc_req_refresh_ctx() is somewhat complex
as it is passed both an on_timeout and on_signal handler, and
on_timeout doesn't return a constant value.

The net effect is to wait for the timeout with signals blocked.  Then,
if the condition still isn't true, run the on_timeout handler and if
that returns zero, wait again - indefinitely this time - and allow
some signals.  If a signal was received, call the on_signal handler.

This is fairly straight forward to write out in C, as shown in the
patch.

Change-Id: I7f9cfb8a8ff234bed4045ab21b53d018337cd615
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Reviewed-on: https://review.whamcloud.com/35987
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-10467 lustre: use wait_event_idle_timeout() as appropriate. 77/35977/18
Mr NeilBrown [Fri, 3 Jan 2020 00:30:32 +0000 (19:30 -0500)]
LU-10467 lustre: use wait_event_idle_timeout() as appropriate.

If l_wait_event() is passed an lwi initialised with
one of
   LWI_TIMEOUT_INTR( time, NULL, NULL, NULL)
   LWI_TIMEOUT_INTR( time, NULL, LWI_ON_SIGNAL_NOOP, NULL)
   LWI_TIMEOUT( time, NULL, NULL)
where time != 0, then it behaves much like
wait_event_idle_timeout().
All signals are blocked, and it waits either for the
condition to be true, or for the timeout (in jiffies).

Note that LWI_ON_SIGNAL_NOOP has no effect here.

l_wait_event() returns 0 when the condition is true, or -ETIMEDOUT
when the timeout occurs.  wait_event_idle_timeout() instead returns a
positive number when the condition is true, and 0 when the timeout
occurs.  So in the cases where return value is used, handling needs to
be adjusted accordingly.

Note that in some cases where cfs_fail_val gives the time to wait for,
the current code re-tests the wait time against zero as cfs_fail_val
can change asynchronously.  This is because l_wait_event() behaves
quite differently if the timeout is zero.

The new code doesn't need to do that as wait_event_idle_timeout()
treat 0 just as a very short wait, which is exactly the correct
behavior here.

This patch also removes a comment which is no longer meaningful
(CAN_MATCH) and corrects a debug message which reported the wait time
as "seconds" rather than the correct "jiffies".

This patch doesn't change the timed wait in cl_sync_io_wait().
That is a bit more complicated, so it left to a separate patch.

Change-Id: I632afc290935e321926f45b144d5367799a01381
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Reviewed-on: https://review.whamcloud.com/35977
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-13087 target: init lcd last transno from reply data 60/37060/2
Mikhail Pershin [Thu, 5 Dec 2019 21:23:01 +0000 (00:23 +0300)]
LU-13087 target: init lcd last transno from reply data

Init lcd_last_transno value from reply data to keep it
valid so tgt_release_reply_data() will keep a slot with
the highest transno and on-disk data is not lost.

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: Id31b3b250616fb6afd3d145c31b12af30ac86be8
Reviewed-on: https://review.whamcloud.com/37060
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-13061 osp: check catlog FID after reading in 98/36998/4
Hongchao Zhang [Thu, 19 Dec 2019 02:52:29 +0000 (21:52 -0500)]
LU-13061 osp: check catlog FID after reading in

In osp_sync_llog_init, the catlog FID read from "CATALOGS"
should be checked whether it is sane or not.

Change-Id: I4342b21b7d5c6d408a9ab52a1e30815ae1d5f563
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/36998
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
9 months agoLU-13077 pfl: cleanup xattr checking 10/37010/7
Sebastien Buisson [Fri, 13 Dec 2019 16:39:08 +0000 (01:39 +0900)]
LU-13077 pfl: cleanup xattr checking

Cleanup xattr checking in mdd and lod layers for PFL.

Reported-by: Clement Barthelemy <clement.barthelemy@nextino.eu>
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I2841b615ee304785fbf316b829d8280eefc3878a
Reviewed-on: https://review.whamcloud.com/37010
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-13043 quota: remove annoying message in osd_declare_inode_qid() 06/36906/5
Wang Shilong [Tue, 3 Dec 2019 06:32:22 +0000 (14:32 +0800)]
LU-13043 quota: remove annoying message in osd_declare_inode_qid()

The admin shouldn't be getting console error messages when a user goes
over quota(this would be happening continuously at some sites).

In some call paths, the "*flags" parameter may be NULL, don't try to
access it in that case.

As a general cleanup, move the QUOTA_FL_* flags over to a named enum
"enum osd_quota_local_flags" so that it is easier to see what this field
actually holds, rather than a totally generic "int *flags" argument that
has to be hunted through the code.

Fixes: d30f9e6b6c5d ("LU-11425 quota: support quota for DoM")
Change-Id: Id5686ecdb8a943e48a2888067e321f83b8569188
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/36906
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Stephan Thiell <sthiell@stanford.edu>
9 months agoLU-12781 ptlrpc: fix inline reply buffer grow 32/36732/4
Mikhail Pershin [Mon, 11 Nov 2019 21:24:39 +0000 (00:24 +0300)]
LU-12781 ptlrpc: fix inline reply buffer grow

In req_capsule_server_grow() reply buffer can be increased
without re-allocation if has enough size already, don't do
that though if rs->rs_repbuf is a wrapper, e.g. with security
enabled. In that case re-allocation is still needed.

Re-enable test 272a in sanity.sh with SHARED_KEY

Test-Parameters: mdscount=2 mdtcount=4 envdefinitions=SHARED_KEY=true testlist=sanity,sanity-pfl
Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I0632b9513f877bea989b7a61a729e2db488dcfcc
Reviewed-on: https://review.whamcloud.com/36732
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12036 ofd: add "no_precreate" mount option 16/36716/9
Andreas Dilger [Fri, 8 Nov 2019 09:04:52 +0000 (02:04 -0700)]
LU-12036 ofd: add "no_precreate" mount option

Add a mount option to disallow object creation on the OST.  That
allows an OST to be mounted by the administrator without it being
immediately available for use by clients/applications.  This may
be useful if the OST needs to be added to a specific pool first,
or if it is being debugged or similar.

Mount option can be disabled with the obdfilter.*.no_precreate
tunable parameter.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Icdb64a4bdd5a66b0e9e6d483e3113b97d53ebbe5
Reviewed-on: https://review.whamcloud.com/36716
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12941 lnet: Add peer level aliveness information 78/36678/5
Chris Horn [Wed, 11 Sep 2019 20:42:55 +0000 (15:42 -0500)]
LU-12941 lnet: Add peer level aliveness information

Keep track of the aliveness of a peer so that we can optimize for
situations where an LNet router hasn't responded to a ping. In
this situation we consider all routes down, and we needn't spend time
inspecting each route, or inspecting all of the router's local and
remote interfaces in order to determine the router's aliveness.

Cray-bug-id: LUS-7860
Test-Parameters: trivial
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: Ie63c1ef40de3ad818639bae6b040923898fd5b46
Reviewed-on: https://review.whamcloud.com/36678
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexey Lyashkov <c17817@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12898 utils: %llu mismatch with type __u64 on ppcle64 58/36558/7
Olaf Faaland [Tue, 22 Oct 2019 16:44:51 +0000 (09:44 -0700)]
LU-12898 utils: %llu mismatch with type __u64 on ppcle64

Fix build errors like this one on ppcle64:

BUILDSTDERR: libmount_utils_zfs.c: In function 'zfs_mkfs_opts':
BUILDSTDERR: libmount_utils_zfs.c:573:5: error: format '%llu' expects
argument of type 'long long unsigned int', but argument 4 has type
'__u64' [-Werror=format=]
BUILDSTDERR:      mop->mo_device_kb * 1024);

__u64 was treated as an unsigned long long which breaks the build on
ppc64le, where they are not the same size.

In printf cases, cast to unsigned long long to match the printf format
so the format is compatible with the type and it is guaranteed
not to lose any data.

In the case of sscanf(), replace the call with strtoull() to eliminate
the issue.

Test-Parameters: trivial
Change-Id: I02fd82e0be4d756881c15aa9faedb9b40961661a
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Reviewed-on: https://review.whamcloud.com/36558
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
9 months agoLU-8130 ldlm: add a counter to the per-namespace data 19/36219/5
NeilBrown [Tue, 17 Sep 2019 19:34:39 +0000 (15:34 -0400)]
LU-8130 ldlm: add a counter to the per-namespace data

When we change the resource hash to rhashtable we won't have
a per-bucket counter.  We could use the nelems global counter,
but ldlm_resource goes to some trouble to avoid having any
table-wide atomics, and hopefully rhashtable will grow the
ability to disable the global counter in the near future.
Having a counter we control makes it easier to manage the
back-reference to the namespace when there is anything in the
hash table.

So add a counter to the ldlm_ns_bucket.

Change-Id: Ic79e96f95d5cacfb5e7bb02350f5f4fafb207b44
Signed-off-by: NeilBrown <neilb@suse.com>
Reviewed-on: https://review.whamcloud.com/36219
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12904 gss: struct cache_detail readers changed to writers 80/36580/4
Shaun Tancheff [Tue, 17 Dec 2019 04:29:15 +0000 (22:29 -0600)]
LU-12904 gss: struct cache_detail readers changed to writers

Linux 5.3 changed struct cache_detail readers to writers
SUNRPC: Track writers of the 'channel' file to improve ...

kernel-commit: 64a38e840ce5940253208eaba40265c73decc4ee

Test-Parameters: trivial
Cray-bug-id: LUS-8042
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Change-Id: I7750303937cd6fc560e458efa79f25e521fefec7
Reviewed-on: https://review.whamcloud.com/36580
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12410 lnet: Delete unused nid parsing code 10/35310/11
Chris Horn [Fri, 20 Sep 2019 15:08:03 +0000 (10:08 -0500)]
LU-12410 lnet: Delete unused nid parsing code

Delete the nid parsing code from liblnetconfig that is no longer used.

Test-Parameters: trivial
Test-Parameters: testlist=sanity-lnet
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I6fbb3450756c7976836c3b6731d3ecd9f93cbf8d
Reviewed-on: https://review.whamcloud.com/35310
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12410 tests: Add gni tests to sanity-lnet 06/35506/11
Chris Horn [Sun, 24 Nov 2019 18:02:16 +0000 (12:02 -0600)]
LU-12410 tests: Add gni tests to sanity-lnet

Add test-cases to validate handling of gni nids to sanity-lnet.sh

Also add some additional tests to validate error handling.

Test-Parameters: trivial
Test-Parameters: testlist=sanity-lnet
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I7947e237e0d3e12e2e30752bca384cef2b66072c
Reviewed-on: https://review.whamcloud.com/35506
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12410 lnet: Convert lnetctl route add and del 08/35308/13
Chris Horn [Sun, 24 Nov 2019 18:01:33 +0000 (12:01 -0600)]
LU-12410 lnet: Convert lnetctl route add and del

Convert the lnetctl route add and delete commands to utilize the new
capabilities provided by the nidstrings library.

Test-Parameters: trivial
Test-Parameters: testlist=sanity-lnet
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: Ifcaf67575ed1de40c9a3c92f40ec6dca7fd08d9e
Reviewed-on: https://review.whamcloud.com/35308
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12410 lnet: Convert yaml peer configuration 07/35307/13
Chris Horn [Mon, 24 Jun 2019 03:56:47 +0000 (22:56 -0500)]
LU-12410 lnet: Convert yaml peer configuration

Convert the yaml peer config handlers to utilize the new capabilities
provided by the nidstrings library.

Test-Parameters: trivial
Test-Parameters: testlist=sanity-lnet
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I89a53ded636877661a3600822ca49030c8841540
Reviewed-on: https://review.whamcloud.com/35307
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12410 lnet: Convert lnetctl peer add and del 05/35305/13
Chris Horn [Mon, 24 Jun 2019 00:27:42 +0000 (19:27 -0500)]
LU-12410 lnet: Convert lnetctl peer add and del

Convert the lnetctl peer add and del commands to utilize the new
capabilities provided by the nidstrings library.

Test-Parameters: trivial
Test-Parameters: testlist=sanity-lnet
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I50693a2af6fef2e1ef3b34fd02c7423625cb7665
Reviewed-on: https://review.whamcloud.com/35305
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-12410 lnet: Implement method to tokenize nidstrings 05/35505/11
Chris Horn [Sun, 14 Jul 2019 18:57:56 +0000 (13:57 -0500)]
LU-12410 lnet: Implement method to tokenize nidstrings

The CLI for various lnetctl operations allows the user to specify
multiple, comma separated nidstrings. Implement a common method
for tokenizing nidstrings that can be leveraged by the operations
that require it.

Test-Parameters: trivial
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I2f8ab6d5d9e7c3d5bde3a11b85bdf38fbf6fdf29
Reviewed-on: https://review.whamcloud.com/35505
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
9 months agoLU-13070 mdd: try old format for orphan names during recovery 49/37049/3
Artem Blagodarenko [Tue, 17 Dec 2019 09:12:36 +0000 (12:12 +0300)]
LU-13070 mdd: try old format for orphan names during recovery

mdd_orphan_destroy() loop caused by compatibility issue on upgrade to
2.11 or later. The format for names of orphans in the PENDING directory
was changed in Lustre 2.11. The old format names are not recognized by
mdd_orphan_destroy() in Lustre 2.11, but compatibility code added to
handle this was incomplete, leading to an endless loop. There's a check
for the old format name, used in mdd_orphan_delete(), but that check
was not included in mdd_orphan_destroy().

This patch adds compatibility check for mdd_orphan_destroy().

Fixes: a02fd4573fe ("LU-7787 mdd: clean up orphan object handling")
Signed-off-by: Artem Blagodarenko <c17828@cray.com>
Cray-bug-id: LUS-8270
Change-Id: I9f42188dcb00f9d536996c14771de7df02502b40
Reviewed-on: https://review.whamcloud.com/37049
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
9 months agoLU-13042 tests: give more time in sanity-selinux test_21b 05/36905/3
Sebastien Buisson [Tue, 3 Dec 2019 02:06:48 +0000 (11:06 +0900)]
LU-13042 tests: give more time in sanity-selinux test_21b

In sanity-selinux test_21b, set sepol refresh time to 1000 seconds
instead of 10. This gives plenty of time for file/dir access tests,
and also cache drop, to complete. Then reset send_sepol to a smaller,
already expired value, to force sepol refresh.

Test-Parameters: trivial
Test-Parameters: clientselinux mdtcount=4 testlist=sanity-selinux envdefinitions=ONLY="21b"
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I57f72faad4bd55736a3240cdefdac2e5814eba79
Reviewed-on: https://review.whamcloud.com/36905
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Alexey Lyashkov <c17817@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
10 months agoLU-12787 tests: skip project quota if it is disabled 97/36997/2
Alexander Boyko [Thu, 19 Sep 2019 13:22:57 +0000 (09:22 -0400)]
LU-12787 tests: skip project quota if it is disabled

quota_scan touchs project quota in case of errors or logs.
When project quota is not supported, this leads to error:
    Unexpected quotactl error: Operation not supported
    ...
    Some errors happened when getting quota info. Some devices
    may be not working or deactivated. The data in "[]" is inaccurate.

The fix adds a check before touching project quota.

Cray-bug-id: LUS-7811
Test-Parameters: testlist=sanity-quota
Signed-off-by: Alexander Boyko <c17825@cray.com>
Change-Id: Ia733b666d6937ea9e8e99ef856d2ae1246dc44d1
Reviewed-on: https://review.whamcloud.com/36997
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Alexander Zarochentsev <c17826@cray.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
10 months agoLU-11607 tests: remove duplicate code lnet-selftest 65/36965/3
James Nunez [Mon, 9 Dec 2019 18:37:14 +0000 (11:37 -0700)]
LU-11607 tests: remove duplicate code lnet-selftest

lnet-selftest.sh and test-framework.sh both have a function
called is_mounted() that check if the file system is
mounted.  Since both functions do and return the same
thing, let's remove the is_mounted() function from
lnet-selftest.

Test-Parameters: trivial
Test-Parameters: fstype=zfs testlist=lnet-selftest,lnet-selftest
Test-Parameters: fstype=ldiskfs testlist=lnet-selftest,lnet-selftest
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I05ce84002cfa8ac96ac4f1e8169fb2233b66f378
Reviewed-on: https://review.whamcloud.com/36965
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
10 months agoLU-12923 libcfs: Use BUILD_BUG_ON() for hash.c 02/36902/6
Arshad Hussain [Sun, 1 Dec 2019 01:43:57 +0000 (07:13 +0530)]
LU-12923 libcfs: Use BUILD_BUG_ON() for hash.c

This patch replaces all CLASSERT() with kernel defined
BUILD_BUG_ON() for file libcfs/libcfs/hash.c

Test-Parameters: trivial
Signed-off-by: Arshad Hussain <arshad.super@gmail.com>
Change-Id: Ie5dc744fc10b6e5f303fca93d342629e99a2403d
Reviewed-on: https://review.whamcloud.com/36902
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
10 months agoLU-12991 lnet: fix rspt counter 95/36895/3
Alexey Lyashkov [Fri, 29 Nov 2019 10:42:59 +0000 (13:42 +0300)]
LU-12991 lnet: fix rspt counter

rsp entries must freed via lnet_rspt_free function to avoid counter
leak. handle NULL allocation properly.

Test-parameters: trivial

Cray-bug-id: LUS-8189
Change-Id: I7630d375387593e28bfbe2c4a3ea3712a239f64f
Signed-off-by: Alexey Lyashkov <c17817@cray.com>
Reviewed-on: https://review.whamcloud.com/36895
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexandr Boyko <c17825@cray.com>
Reviewed-by: Chris Horn <hornc@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
10 months agoLU-12923 utils: Use BUILD_BUG_ON() for wirehdr.c & wirecheck.c 11/36711/4
Arshad Hussain [Tue, 29 Oct 2019 12:55:04 +0000 (18:25 +0530)]
LU-12923 utils: Use BUILD_BUG_ON() for wirehdr.c & wirecheck.c

This patch replaces all CLASSERT() with user defined
BUILD_BUG_ON() for file lustre/utils/wirehdr.c and
lustre/utils/wirecheck.c

Test-Parameters: trivial
Signed-off-by: Arshad Hussain <arshad.super@gmail.com>
Change-Id: If46603158a9ad311762fe51839000c39a0b15307
Reviewed-on: https://review.whamcloud.com/36711
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
10 months agoLU-11656 llite: fetch default layout for a directory 09/36609/11
Jian Yu [Tue, 19 Nov 2019 22:19:24 +0000 (14:19 -0800)]
LU-11656 llite: fetch default layout for a directory

For a directory that does not have trusted.lov xattr, the current
"lfs getstripe" will only print the stripe_count, stripe_size,
and stripe_index that are fetched from the /sys/fs/lustre/lov values.
It doesn't show the actual default layout that will be used when
new files will be created in that directory.

This patch fixes the above issue in ll_dir_getstripe_default() by
fetching the layout from root FID after ll_dir_get_default_layout()
returns -ENODATA from a directory that does not have trusted.lov xattr.

Change-Id: Icbf1f8f4fa5e5b8788217fcb0cfd24a3b80a27d9
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/36609
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
10 months agoLU-930 doc: add man pages to make file 98/36598/7
James Nunez [Mon, 28 Oct 2019 18:54:21 +0000 (12:54 -0600)]
LU-930 doc: add man pages to make file

There are several existing man pages that are not installed on the
Lustre client or server because they are not included in the make
file.  Add the following pages to the Makefile.am file:

lctl-pool_add
lctl-pool_new
l_getsepol
lfs-getname
lfs-getsom
lfs-mirror-copy
lfs-mirror-read
lfs-mirror-write
lfs-rmfid
llapi_get_lum_dir
llapi_get_lum_dir_fd
llapi_get_lum_file
llapi_get_lum_file_fd
llapi_layout_extension_size_get
llapi_layout_extension_size_set
llapi_rmfid
llapi_search_mdt
llapi_search_ost
llapi_search_tgt

Fixes: ca34df3815f7 (LU-930 doc: man pages for lctl pool_new, pool_add)
Fixes: 2a4821b836c8 (LU-12159 utils: improve lfs getname functionality)
Fixes: 697e8fe6f325 (LU-11473 doc: add lfs-getsom man page)
Fixes: c6e7c0788d7c (LU-10258 lfs: lfs mirror copy command)
Fixes: 1fd63fcb045c (LU-12090 utils: lfs rmfid)
Fixes: e82adfcbd00f (LU-930 doc: man page for l_getsepol)
Fixes: 11aa7f8704c4 (LU-11367 som: integrate LSOM with lfs find)
Fixes: fed241911f61 (LU-10070 lod: SEL: Add flag & setstripe support)
Fixes: 096db80e0810 (LU-11264 llapi: clean up llapi_search_tgt() code)

Test-Parameters: trivial

Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I72c34693a9cf03d2e241d60903020a72339c75b1
Reviewed-on: https://review.whamcloud.com/36598
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Joseph Gmitter <jgmitter@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
10 months agoLU-12904 utils: zfs properly detect spa_multihost 82/36582/6
Shaun Tancheff [Tue, 10 Dec 2019 09:20:26 +0000 (03:20 -0600)]
LU-12904 utils: zfs properly detect spa_multihost

spa_multihost is used in a user space tool and the
compile test for spa_multihost reflect that.

Test-Parameters: trivial
Cray-bug-id: LUS-8042
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Change-Id: Ie85fffb80e84a2b65547e3d48dc0cff31c3325b4
Reviewed-on: https://review.whamcloud.com/36582
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
10 months agoLU-12904 build: account_page_dirtied is not exported 75/36575/4
Shaun Tancheff [Fri, 25 Oct 2019 20:11:37 +0000 (15:11 -0500)]
LU-12904 build: account_page_dirtied is not exported

Linux 5.2 does not export account_page_dirtied
mm: remove the account_page_dirtied export

Use symbol_get() to access account_page_dirtied for Lustre

kernel-commit: ac1c3e49a9a734150b33297eeca5b43d92fd5be8

Test-Parameters: trivial
Cray-bug-id: LUS-8042
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Change-Id: I9cd432556e183d06784537b000a4bda657116d88
Reviewed-on: https://review.whamcloud.com/36575
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
10 months agoLU-12861 libcfs: provide an scnprintf and start using it 53/36453/8
Shaun Tancheff [Tue, 29 Oct 2019 22:14:18 +0000 (17:14 -0500)]
LU-12861 libcfs: provide an scnprintf and start using it

snprintf() returns the number of chars that would be needed to hold
the complete result, which may be larger that the buffer size.

scnprintf differs in it's return value is number of chars actually
written (not including the terminating null).

Correct the few patterns where the return from snprintf() is used and
expected not to exceed the passed buffer size.

Test-Parameters: trivial
Cray-bug-id: LUS-7999
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Change-Id: Ie42458be16e8c0ba1cb6d688fd418683f18de21e
Reviewed-on: https://review.whamcloud.com/36453
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
10 months agoLU-12634 llite: Use __xa_set_mark if it is available 73/36373/3
Shaun Tancheff [Fri, 4 Oct 2019 17:57:16 +0000 (12:57 -0500)]
LU-12634 llite: Use __xa_set_mark if it is available

Linux v4.19-rc5-248-g9b89a0355144
xarray: Add XArray marks

Test for and use __xa_set_mark() for marking page cache pages.

Move kernel compat wrappers in to inline functions.
Co-locate the configure test macros for 4.20 in kernel version order.

Test-Parameters: trivial
Cray-bug-id: LUS-7689
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Change-Id: I659a0e3af0a648d50205f44f2649ba8b982bfa42
Reviewed-on: https://review.whamcloud.com/36373
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Ben Evans <bevans@cray.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
10 months agoLU-12678 lnet: use list_move where appropriate. 39/36339/2
NeilBrown [Tue, 1 Oct 2019 15:59:59 +0000 (11:59 -0400)]
LU-12678 lnet: use list_move where appropriate.

There are several places in lustre where "list_del" (or occasionally
"list_del_init") is followed by "list_add" or "list_add_tail" which
moves the object to a different list.
These can be combined into "list_move" or "list_move_tail".

Test-Parameters: trivial testlist=sanity-lnet

Change-Id: I481de128ea40928186f78a0a0cc26e89b43f1645
Signed-off-by: NeilBrown <neilb@suse.com>
Reviewed-on: https://review.whamcloud.com/36339
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
10 months agoLU-12542 handle: discard OBD_FREE_RCU 97/35797/9
NeilBrown [Tue, 1 Oct 2019 00:58:55 +0000 (20:58 -0400)]
LU-12542 handle: discard OBD_FREE_RCU

OBD_FREE_RCU and the hop_free call-back together form an overly
complex mechanism equivalent to kfree_rcu() or call_rcu(...).
Discard them and use the simpler approach.

This removes the only use for the field h_size, so discard
that too.

Change-Id: I3b4135565dab6a9aa5034f42ae3f9b66851cae31
Signed-off-by: NeilBrown <neilb@suse.com>
Reviewed-on: https://review.whamcloud.com/35797
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
10 months agoLU-8066 obdclass: don't copy ops structures in to new type. 87/35687/8
NeilBrown [Fri, 11 Oct 2019 01:26:06 +0000 (21:26 -0400)]
LU-8066 obdclass: don't copy ops structures in to new type.

The obd_ops and md_ops structures passed to class_register_type() are
read-only, and have a lifetime that is exceeds the lifetime of the
obd_type - they are static in a module which unregisters the type before
being unloaded.

So there is no need to copy the ops, just store a pointer.

Also mark all the structures as read-only to confirm they don't get
written. This is best-practice for structures of function pointers.

Linux-commit: 2233f57f1b95b9a85a3129ddcc2860ddbc4c2a94

Signed-off-by: NeilBrown <neilb@suse.com>
Change-Id: Id0be1477925e0c878e3edb6a9d892f3c89a8b19b
Reviewed-on: https://review.whamcloud.com/35687
Reviewed-by: Neil Brown <neilb@suse.de>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
10 months agoLU-12514 obdclass: remove vfsmount option from client_fill_super 27/35427/4
NeilBrown [Thu, 12 Dec 2019 14:58:32 +0000 (09:58 -0500)]
LU-12514 obdclass: remove vfsmount option from client_fill_super

This arg is always NULL and is never used.
So discard it from this and related functions.

Linux-commit: 7dc2155195586ec75f53d6dcd381f935ccc35d02

Change-Id: I00b16115edbff0de7605768121981b928585552c
Signed-off-by: NeilBrown <neilb@suse.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-on: https://review.whamcloud.com/35427
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
10 months agoLU-12514 obdclass: remove pointless struct lustre_mount_data2 26/35426/7
NeilBrown [Wed, 2 Oct 2019 15:26:10 +0000 (11:26 -0400)]
LU-12514 obdclass: remove pointless struct lustre_mount_data2

This is used to pass a void* and NULL to lustre_fill_super().
It is easier just to pass the void*. The "NULL" passed is
sometimes a "struct vfsmount". This pointer is passed to
ll_fill_super() which then passes it to client_common_fill_super()
that just ignores it.

Linux-commit: 998831a00192a38a9f1425b2fb2d6faf3e34e665

Change-Id: If5e229d80c08b7c16e89d11a03fc766584c24f7c
Signed-off-by: NeilBrown <neilb@suse.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-on: https://review.whamcloud.com/35426
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
10 months agoLU-8130 obd: convert obd uuid hash to rhashtable 29/34429/8
NeilBrown [Thu, 12 Dec 2019 23:46:38 +0000 (18:46 -0500)]
LU-8130 obd: convert obd uuid hash to rhashtable

The rhashtable data type is a perfect fit for the
export uuid hash table, so use that instead of
cfs_hash (which will eventually be removed).

As rhashtable supports lookups and insertions in atomic
context, there is no need to drop a spinlock while
inserting a new entry, which simplifies code quite a bit.

Linux-commit: 4206c444e4a89dae9f67f08d9c29d58c37c960cd

Change-Id: Icadf64d572982409008a1ef4d23eb0fe1e3c8cd0
Signed-off-by: NeilBrown <neilb@suse.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-on: https://review.whamcloud.com/34429
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
10 months agoLU-9679 all: prefer sizeof(*var) for ALLOC/FREE 61/36661/3
Mr NeilBrown [Mon, 4 Nov 2019 05:38:24 +0000 (16:38 +1100)]
LU-9679 all: prefer sizeof(*var) for ALLOC/FREE

The construct
   LIBCFS_ALLOC(var, sizeof(*var));
is more obviously correct than
   LIBCFS_ALLOC(var, sizeof(struct something));
and is preferred upstream (where it is actually kzalloc
or similar of course).

When it is that simple, and there is no multiplier for
the size,
   CFS_ALLOC_PTR(var);
is even better.

The same logic applies to OBD_ALLOC(), LIBCFS_FREE(),
and OBD_FREE().

So convert allocations and frees that use sizeof(struct..)
to use one of the simpler constructs.

In mgs_write_log_mdt0, uuid is better declared as a
"struct obd_uuid *" which is a struct that contain a 'char'
array.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I8cd97c75241bbb87d15cc6b7c9ac2a7d6184d700
Reviewed-on: https://review.whamcloud.com/36661
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
10 months agoLU-12410 lnet: Define enum for lnetctl commands 04/35504/11
Chris Horn [Thu, 27 Jun 2019 03:51:57 +0000 (22:51 -0500)]
LU-12410 lnet: Define enum for lnetctl commands

The enum values can be used to faciliate code sharing amongst the
lnetctl routines.

Test-Parameters: trivial
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I1085a70a17aefa300f3bf949cf867b2712131a0f
Reviewed-on: https://review.whamcloud.com/35504
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
10 months agoLU-12410 lnet: Implement DLC wrapper for cfs_parse_nidlist 04/35304/13
Chris Horn [Sat, 22 Jun 2019 17:03:09 +0000 (12:03 -0500)]
LU-12410 lnet: Implement DLC wrapper for cfs_parse_nidlist

Implement a simple wrapper around cfs_parse_nidlist to be used by DLC
commands. The wrapper serves to sanitize the nidstr, so that is
suitable to be input to cfs_parse_nidlist. We do not want to allow an
asterisk character in this string because the resultant nidlist, when
expanded, would define too many nids for the various operations that
will utilize this functionality.

Test-Parameters: trivial
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I59280317af4af05eca1c8c598eadf8871e28bcf1
Reviewed-on: https://review.whamcloud.com/35304
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>