Whamcloud - gitweb
fs/lustre-release.git
2 years agoLU-14423 osd: recognize holes in osd_is_mapped() 04/45504/2
Alex Zhuravlev [Thu, 11 Feb 2021 14:33:01 +0000 (17:33 +0300)]
LU-14423 osd: recognize holes in osd_is_mapped()

ldiskfs_fiemap() can return {0,0,0} for last non-allocated
region.  osd_is_mapped() should be able to recognize and
cache this state.

Lustre-change: https://review.whamcloud.com/41481
Lustre-commit: 2eaa49ef0f16798d564883b16cea9e96fad52495

Fixes: 144b5a65c1 ("LU-7132 osd-ldiskfs: speedup rewrites")
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I03883038c2c0ec84754377a442c4947c7e3021a9
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andrew Perepechko <andrew.perepechko@hpe.com>
Reviewed-on: https://review.whamcloud.com/45504
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
2 years agoLU-14733 o2iblnd: Avoid double posting invalidate 96/44296/5
Original Author Mike Marciniszyn [Wed, 7 Jul 2021 19:16:01 +0000 (15:16 -0400)]
LU-14733 o2iblnd: Avoid double posting invalidate

When the kib_tx is provisioned during kiblnd_fmr_pool_map(), spare
WRs in the kib_fast_reg_descriptor are setup and the mapping of
pages is given to the mr.

kiblnd_post_tx_locked() then posts the spare WRs from the
kib_fast_reg_descriptor.

if (rc == 0)
return 0;

The code returns and the kib_fast_reg_descriptor is still contains
the spare WRs.   The next time the kib_tx is used, the
now obsolete WRs will be inadvertently posted.   For rdmavt, the
obsolete invalidate will cause an -EINVAL to be returned from
the post send.

Fix by adding a state variable frd_posted to the kib_fast_reg_descriptor.
The variable is set to false in kiblnd_fmr_pool_unmap().
kiblnd_post_tx_locked() is adjusted to avoid prepending the
kib_fast_reg_descriptor WRs when frd_posted is true.   After
the post succeeds, the frd_posted is set to true.

Lustre-change: https://review.whamcloud.com/44190
Lustre-commit: 5930576791e864529e6ef9b46f3e09cc4b635fc2
Test-Parameters: trivial

Signed-off-by: Mike Marciniszyn <mike.marciniszyn@cornelisnetworks.com>
Change-Id: I426dd05e635392e75d1aa48808782a229e83ce5f
Signed-off-by: Gian-Carlo DeFazio <defazio1@llnl.gov>
Reviewed-on: https://review.whamcloud.com/44296
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
2 years agoLU-14733 o2iblnd: Move racy NULL assignment 95/44295/4
Original Author Mike Marciniszyn [Wed, 7 Jul 2021 19:16:00 +0000 (15:16 -0400)]
LU-14733 o2iblnd: Move racy NULL assignment

kiblnd_fmr_pool_unmap() can race map and subsequent processing
because of this flaw in unmap:

if (frd) {
frd->frd_valid = false;
spin_lock(&fps->fps_lock);
list_add_tail(&frd->frd_list, &fpo->fast_reg.fpo_pool_list);
spin_unlock(&fps->fps_lock);
fmr->fmr_frd = NULL;
}

The fmr can be pulled off the list in kiblnd_fmr_pool_unmap() on
another CPU an fmr_frd could be in a state of flux and
potentially be seen incorrectly later on as the kib_tx is processed.

Fix my moving the fmr_frd assignment to before the fmr is added to the
list.

Lustre-change: https://review.whamcloud.com/44189
Lustre-commit: 023113fb8946f3565529e7327fdcd90ab9db3ba3
Test-Parameters: fortestonly testgroup=review-dne-zfs-part-1

Signed-off-by: Mike Marciniszyn <mike.marciniszyn@cornelisnetworks.com>
Change-Id: Ibddf132a363ecfe9db3cc06287cec873c021d2fb
Signed-off-by: Gian-Carlo DeFazio <defazio1@llnl.gov>
Reviewed-on: https://review.whamcloud.com/44295
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
2 years agoLU-14690 kernel: RHEL 8.4 server support 74/44174/6
Original Author ian Yu [Fri, 4 Jun 2021 07:47:14 +0000 (00:47 -0700)]
LU-14690 kernel: RHEL 8.4 server support

This patch makes changes to support RHEL 8.4 release with
kernel 4.18.0-305.3.1.el8_4 for Lustre server.

Test-Parameters: trivial fstype=ldiskfs fortestonly \
clientdistro=el8.4 serverdistro=el8.4 testlist=sanity

Test-Parameters: trivial fstype=zfs fortestonly \
clientdistro=el8.4 serverdistro=el8.4 testlist=sanity

Lustre-change: https://review.whamcloud.com/43791
Lustre-commit: 644a14196810f0c6b663957720414e042d2ae965

Change-Id: I484af80c4764367b40b28ce459a6ff9d87edf3a8
Signed-off-by: Jian Yu <yujian@whamcloud.com>
lustre/kernel_patches/which_patch conflect resolved
by using most up to date version.
Signed-off-by: Gian-Carlo DeFazio <defazio1@llnl.gov>
Reviewed-on: https://review.whamcloud.com/44174
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: Oleg Drokin <green@whamcloud.com>
2 years agoLU-14690 kernel: new kernel [RHEL 8.4 4.18.0-305.3.1.el8_4] 73/44173/4
Original Author [Fri, 4 Jun 2021 07:37:18 +0000 (00:37 -0700)]
LU-14690 kernel: new kernel [RHEL 8.4 4.18.0-305.3.1.el8_4]

This patch makes changes to support new RHEL 8.4 release
for Lustre client.

Test-Parameters: trivial clientdistro=el8.4

Lustre-change: https://review.whamcloud.com/43725
Lustre-commit: f269497ac7a730880e590eb9e8405f082522c5e0

Test-Parameters: fortestonly
Change-Id: I47d4706f9175d489ef0e6226492af20f44f0677e
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Removed libcfs/include/libcfs/linux/linux-net.h
Signed-off-by: Gian-Carlo DeFazio <defazio1@llnl.gov>
Reviewed-on: https://review.whamcloud.com/44173
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: Oleg Drokin <green@whamcloud.com>
2 years agoLU-14553 changelog: eliminate mdd_changelog_clear warning 72/44172/3
Original Author Olaf Faaland [Thu, 25 Mar 2021 01:35:10 +0000 (18:35 -0700)]
LU-14553 changelog: eliminate mdd_changelog_clear warning

When handling a changelog_clear request, the user may specify a
range of indices which do not exist.  Similarly, the user may
specify a changelog user which does not exist.  Neither indicates
a problem within Lustre that justifies a a console warning.

Change those cases to CDEBUG.

Lustre-change: https://review.whamcloud.com/43125
Lustre-commit: 6b183927e19715d093c80a35ebc42a1cda5e70e2

Test-Parameters: trivial fortestonly
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Change-Id: I64bab12ef4978c4bf7139f5f36a39f9b109616fb
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Signed-off-by: Gian-Carlo DeFazio <defazio1@llnl.gov>
Reviewed-on: https://review.whamcloud.com/44172
Tested-by: jenkins <devops@whamcloud.com>
2 years agoLU-12836 osd-zfs: Catch all ZFS pool change events 71/44171/3
Original Author Tony Hutter [Fri, 12 Mar 2021 01:23:16 +0000 (17:23 -0800)]
LU-12836 osd-zfs: Catch all ZFS pool change events

This change adds the following symlinks:

  vdev_attach-lustre -> statechange-lustre.sh
  vdev_remove-lustre -> statechange-lustre.sh
  vdev_clear-lustre -> statechange-lustre.sh

This makes it so the statechange-lustre.sh script is also called on
all ZFS events that could change the pool state.

Lustre-change: https://review.whamcloud.com/43552
Lustre-commit: e11a47da71a2e2482e4c4cf582d663cd76a2ecab

Test-Parameters: fortestonly
Signed-off-by: Tony Hutter <hutter2@llnl.gov>
Change-Id: I18edc86749e8ab91bb45f21aafd3fd47e78cbaef
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Signed-off-by: Gian-Carlo DeFazio <defazio1@llnl.gov>
Reviewed-on: https://review.whamcloud.com/44171
Tested-by: jenkins <devops@whamcloud.com>
2 years agoLU-14752 obdclass: handle EBUSY returned for lu_object hashtable 70/44170/3
Original Author James Simmons [Thu, 10 Jun 2021 16:53:57 +0000 (12:53 -0400)]
LU-14752 obdclass: handle EBUSY returned for lu_object hashtable

When the rhashtable grows to a certain size it will be rescaled.
When rescaling you can be returned a ENOMEM or EBUSY error. This
we reported as:

LustreError: 3594004:0:(lu_object.c:2472:lu_object_assign_fid()) ASSERTION( rc == 0 ) failed: failed hashtable insertion: rc = -16
LustreError: 3594004:0:(lu_object.c:2472:lu_object_assign_fid()) LBUG
Pid: 3594004, comm: mdt01_020 4.18.0-240.22.1.1toss.t4.x86_64 #1 SMP Tue Apr 13 17:18:40 PDT 2021
Call Trace TBD:
Kernel panic - not syncing: LBUG
...
Call Trace:
dump_stack+0x5c/0x80
panic+0xe7/0x2a9
lbug_with_loc.cold.10+0x18/0x18 [libcfs]
lu_object_assign_fid+0x3b8/0x3c0 [obdclass]

Add handling the EBUSY case for our lu_object hash.

Lustre-change: https://review.whamcloud.com/43968
Lustre-commit: 285a29d3b5e47f63a94c0682040ddbf09614f130

Fixes: aff14dbc522 ("LU-8130 lu_object: convert lu_object cache to rhashtable")
Change-Id: Id85f32633117e02850b799e8d95e3e35d982cbd4
Test-Parameters: fortestonly
Signed-off-by: James Simmons <jsimmons@infradead.org>
Signed-off-by: Gian-Carlo DeFazio <defazio1@llnl.gov>
Reviewed-on: https://review.whamcloud.com/44170
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 years agoLU-14460 lnet: fix mismatched printf format 62/44162/4
Original Author Lei Feng [Thu, 25 Feb 2021 00:31:56 +0000 (08:31 +0800)]
LU-14460 lnet: fix mismatched printf format

Original "%llx" does not work on all platforms. Fix it.

Lustre-change: https://review.whamcloud.com/41755
Lustre-commit: 58e05ff5af3d1fcd7b059dc56955a5f8d94db4ab

Signed-off-by: Lei Feng <flei@whamcloud.com>
Change-Id: I2edecbf66ccb2141c72294d324ade79574f5c084
Test-Parameters: trivial fortestonly
Signed-off-by: Gian-Carlo DeFazio <defazio1@llnl.gov>
Reviewed-on: https://review.whamcloud.com/44162
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoNew release 2.14.0 2.14.0 v2_14_0
Oleg Drokin [Fri, 19 Feb 2021 19:28:17 +0000 (14:28 -0500)]
New release 2.14.0

Change-Id: I2eb99af8fbeaab80b6614e427b77949b1225b406
Signed-off-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14345 misc: update e2fsprogs to 1.45.6.wc5 33/41433/3
Andreas Dilger [Mon, 8 Feb 2021 11:56:25 +0000 (04:56 -0700)]
LU-14345 misc: update e2fsprogs to 1.45.6.wc5

Update Changelog to reference new e2fsprogs release.

4aea203f LU-5949 e2fsck: call delete_inode() properly
8725134d LU-5949 e2fsck: simplify inode badness handling
71b74579 LU-14345 e2fsck: fix check of directories over 4GB

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ie0845eeed410f6f9f8ef985342fc19d160aa8cb0
Reviewed-on: https://review.whamcloud.com/41433
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Artem Blagodarenko <artem.blagodarenko@hpe.com>
Reviewed-by: Peter Jones <pjones@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoNew RC 2.14.0-RC3 2.14.0-RC3 v2_14_0-RC3
Oleg Drokin [Sat, 13 Feb 2021 00:52:28 +0000 (19:52 -0500)]
New RC 2.14.0-RC3

Change-Id: I594b5c6d0da7f067bef69fa7a7027374d4434dd8
Signed-off-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14424 Revert "LU-9679 osc: simplify osc_extent_find()" 98/41498/2
Oleg Drokin [Fri, 12 Feb 2021 15:55:42 +0000 (10:55 -0500)]
LU-14424 Revert "LU-9679 osc: simplify osc_extent_find()"

It looks like there are performance regressions atttributed to this patch.

This reverts commit 80e21cce3dd6748fd760786cafe9c26d502fd74f.

Change-Id: I55e0abd50573dd82a9d216f9c3b01483f99c3223
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/41498
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
3 years agoNew release candidate 2.14.0 RC2 2.14.0-RC2 v2_14_0-RC2
Oleg Drokin [Mon, 8 Feb 2021 22:13:56 +0000 (17:13 -0500)]
New release candidate 2.14.0 RC2

Change-Id: Iad3d71e7dcf96173d192717ef4fef3f0dc12b051
Signed-off-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-13751 tests: remove read of changelog sanity 160j 17/41317/7
James Nunez [Tue, 26 Jan 2021 01:15:49 +0000 (18:15 -0700)]
LU-13751 tests: remove read of changelog sanity 160j

sanity test 160j tries to read the changelog after one of two
client mounts is unmounted.  In this case, we can fail to read
the changelog and get a "Cannot send after transport endpoint
shutdown" error.

The intention of sanity test 160j is to check that
there is no LBUG due to missed obd device.  So, do not try to
read from the changelog after file system unmount.

Test-Parameters: trivial testlist=sanity env=ONLY=160j,ONLY_REPEAT=200
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I1746a422b25d546b9aae38ae8438d9c08bce8827
Reviewed-on: https://review.whamcloud.com/41317
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
3 years agoLU-14355 ptlrpc: do not output error when imp_sec is freed 10/41310/2
Sebastien Buisson [Mon, 25 Jan 2021 08:24:19 +0000 (17:24 +0900)]
LU-14355 ptlrpc: do not output error when imp_sec is freed

There is a race condition on client reconnect when the import is being
destroyed.  Some outstanding client bound requests are being processed
when the imp_sec has already been freed.
Ensure to output the error message in import_sec_validate_get() only
if import is not already in the zombie work queue.

Fixes: 135fea8fa9 ("LU-4423 obdclass: use workqueue for zombie management")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I4b431128e04f11b1e3ee7de47090af87538c3558
Reviewed-on: https://review.whamcloud.com/41310
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14299 test: sleep to enable quota acquire again 89/41389/3
Hongchao Zhang [Fri, 29 Jan 2021 20:51:43 +0000 (04:51 +0800)]
LU-14299 test: sleep to enable quota acquire again

sanity-quota test_61 fails with incorrect quota exceeded
errors because quota acquire will be disabled for 5 seconds
after edquot flag is set.  The test should introduce some
delay between the test of over quota and normal one.

Test-Parameters: trivial fstype=zfs testlist=sanity-quota env=ONLY=61,ONLY_REPEAT=20
Fixes: 530881fe4ee20 ("LU-7816 quota: add default quota setting support")
Change-Id: I8040ba960f32cf01cb7cee3a77c06ad4bd732f0e
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/41389
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-13449 tests: fix recovery-small test_140b check 09/39909/2
Andreas Dilger [Mon, 14 Sep 2020 23:07:17 +0000 (17:07 -0600)]
LU-13449 tests: fix recovery-small test_140b check

The recovery timer is printed in MM:SS format, but the current test
is unhappy if the SS part is printed as "08" or "09" since that is
interpreted by bash as an invalid octal number.  Also, the current
check does not handle the case if recovery is longer than a minute.

Change the code to convert MM:SS back to seconds for the comparison.

Test-Parameters: trivial testlist=recovery-small env=ONLY=140
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ie1dc77a88bb0e8fd5025f2b5ca57d4a61d3ebbe5
Reviewed-on: https://review.whamcloud.com/39909
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14316 llite: quiet spurious ioctl warning 27/41427/2
Andreas Dilger [Fri, 5 Feb 2021 20:13:10 +0000 (13:13 -0700)]
LU-14316 llite: quiet spurious ioctl warning

Calling "lfs setstripe" prints a suprious warning about using the old
ioctl(LL_IOC_LOV_GETSTRIPE) when that is not actually the case.

Remove the ioctl warning for now and deal with related issues later.

Fixes: 364ec95f3688 ("LU-9367 llite: restore ll_file_getstripe in ll_lov_setstripe")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I20f5a7adb60a30fce27e49827bd46229e2ce7057
Reviewed-on: https://review.whamcloud.com/41427
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-11607 tests: replace lustre_version/fstype in sanity-scrub 29/35929/10
James Nunez [Mon, 26 Aug 2019 22:45:40 +0000 (16:45 -0600)]
LU-11607 tests: replace lustre_version/fstype in sanity-scrub

The routine get_lustre_env() is available to all Lustre test
suites and sets an environment variable for the Lustre
version and file system types of servers.

In sanity-scrub, sanity-hsm and sanity-lfsck, replace calls
to lustre_version_code() and facet_fstype() for all server
types with definitions from get_lustre_env().

Clean up around any modifications by removing calls to
return after skip() or skip_env().

Fixes: c8790ae52393 (LU-1538 tests: standardize test script init - dne-part-2)
Fixes: c54b6ca2bdb5 (LU-13718 tests: add LU numbers to skipped tests)
Test-Parameters: trivial fstype=ldiskfs testlist=sanity-scrub,sanity-hsm,sanity-lfsck
Test-Parameters: fstype=zfs testlist=sanity-scrub,sanity-hsm,sanity-lfsck
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I5aa2898f6efef127cf2d7f4a2f08838f503c51ab
Reviewed-on: https://review.whamcloud.com/35929
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Vikentsi Lapa <vlapa@whamcloud.com>
3 years agoLU-14389 lov: avoid NULL dereference in cleanup 98/41398/2
Andreas Dilger [Sun, 31 Jan 2021 07:20:47 +0000 (00:20 -0700)]
LU-14389 lov: avoid NULL dereference in cleanup

Running racer concurrently with file migration crashes easily
when the layout changes for a file in an unexpected way:

  lov_init_composite() lustre-clilov: DOM entries with different sizes
  lov_layout_change() lustre-clilov: cannot apply new layout on
      [0x200000402:0x3e6a:0x0] : rc = -22
  BUG: unable to handle kernel NULL pointer dereference at 0x00000014
  IP: [<ffffffffa08baef4>] lov_delete_composite+0x104/0x540 [lov]
  Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
  CPU: 1 PID: 20227 Comm: ln

Avoid the NULL dereference if the entry is not fully initialized
during cleanup.

Test-Parameters: testlist=racer env=DURATION=3600
Fixes: 61a002cd863 ("LU-13602 flr: skip unknown FLR component types")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I8fe17f1b49ca2bccc7a285febe47032d023ebbe5
Reviewed-on: https://review.whamcloud.com/41398
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-13669 llite: make readahead aware of hints 28/41228/3
Wang Shilong [Sat, 9 Jan 2021 10:28:43 +0000 (18:28 +0800)]
LU-13669 llite: make readahead aware of hints

Calling madvise(MADV_SEQUENTIAL) and madvise(MADV_RANDOM) sets the
VM_SEQ_READ and VM_RAND_READ hints in vma->vm_flags.  These should
be used to guide the Lustre readahead for better performance.

Disable the kernel readahead for mmap() pages and use the llite
readahead instead.  There was also a bug in ll_fault0() that would
set both VM_SEQ_READ and VM_RAND_READ at the same time, which was
confusing the detection of the VM_SEQ_READ case, since VM_RAND_READ
was being checked first.

This changes the readahead for mmap from submitting mostly 4KB RPCs
to a large number of 1MB RPCs for the application profiled:

  llite.*.read_ahead_stats     before        patched
  ------------------------     ------        -------
  hits                           2408         135924 samples [pages]
  misses                        34160           2384 samples [pages]

  osc.*.rpc_stats           read before    read patched
  ---------------          -------------  --------------
  pages per rpc            rpcs   % cum%   rpcs   % cum%
     1:                    6542  95  95     351  55  55
     2:                     224   3  99      76  12  67
     4:                      32   0  99      28   4  72
     8:                       2   0  99       9   1  73
    16:                      25   0  99      32   5  78
    32:                       0   0  99       8   1  80
    64:                       0   0  99       5   0  80
   128:                       0   0  99      15   2  83
   256:                       2   0  99     102  16  99
   512:                       0   0  99       0   0  99
  1024:                       1   0 100       3   0 100

Readahead hit rate improved from 6% to 98%, and 4KB RPCs dropped from
95% to 55% and 1MB+ RPCs increased from 0% to 16% (79% of all pages).

Add debug to ll_file_mmap(), ll_fault() and ll_fault_io_init() to
allow tracing VMA state functions for future IO optimizations.

Fixes: 62ef9c949753 ("add 2.6.27 kernel support")
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I4bbb028db05b21ae01dafe6a7bea398e9b74d8a4
Reviewed-on: https://review.whamcloud.com/41228
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-10958 ofd: data corruption due to RPC reordering 81/32281/18
Andrew Perepechko [Mon, 9 Dec 2019 17:13:50 +0000 (20:13 +0300)]
LU-10958 ofd: data corruption due to RPC reordering

Without read-only cache, it is possible that a client
resends a BRW RPC, receives a reply from the original
BRW RPC, modifies the same data and sends a new BRW
RPC, however, because of RPC reordering stale data
gets to disk.

Let's use range locking to protect against this race.

Change-Id: I35cbf95594601eacfc5f108b14e4c447962b0bbf
Signed-off-by: Andrew Perepechko <c17827@cray.com>
Cray-bug-id: LUS-5578,LUS-8943
Reviewed-on: https://review.whamcloud.com/32281
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoNew release candidate 2.14.0-RC1 2.14.0-RC1 v2_14_0-RC1
Oleg Drokin [Mon, 1 Feb 2021 19:11:10 +0000 (14:11 -0500)]
New release candidate 2.14.0-RC1

Change-Id: I54bced5067f605bae67faffce46d89383dc69a39
Signed-off-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-13184 tests: wait for OST startup in test_112 93/37393/8
Andreas Dilger [Fri, 31 Jan 2020 21:09:14 +0000 (14:09 -0700)]
LU-13184 tests: wait for OST startup in test_112

Wait for OST0000 to finish mounting and connecting to the MDS before
forcing to create any files there.  Otherwise, intermittent failures
are seen because the OST has no objects and returns -ERANGE:

    lfs setstripe: 'f112.conf-sanity.0': Numerical result out of range
    problem creating f112.conf-sanity.0 on OST0000

Test-Parameters: trivial testlist=conf-sanity envdefinitions=ONLY=112,ONLY_REPEAT=100
Fixes: 416e67222b76 ("LU-12036 ofd: add 'no_precreate' mount option")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: If2d547a42c51a7028803ec25680931a7593ebbe5
Reviewed-on: https://review.whamcloud.com/37393
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14307 quota: fix NULL pointer dereference 64/41264/7
Sergey Cheremencev [Mon, 18 Jan 2021 22:11:45 +0000 (01:11 +0300)]
LU-14307 quota: fix NULL pointer dereference

Fix NULL pointer dereference at 0x20 in
qmt_trans_start_with_slv->lquota_lqe_debug.

HPE-bug-id: LUS-9662
Change-Id: Iead0df053ae0dcb7453c1910a4b4b7a3728da829
Signed-off-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-on: https://review.whamcloud.com/41264
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14356 utils: change position of the -lext2fs option 05/41305/2
Jian Yu [Sat, 23 Jan 2021 20:47:29 +0000 (12:47 -0800)]
LU-14356 utils: change position of the -lext2fs option

This patch changes the position of the -lext2fs option
in the gcc command line so as to resolve the following
issue:

mount_osd_ldiskfs.so: undefined symbol: unix_io_manager

Change-Id: I9ceaca867697c132b8d4a7800169101a024d17b8
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/41305
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-13578 test: use a single read() in sanity test_39r 73/40973/3
John L. Hammond [Tue, 15 Dec 2020 15:34:34 +0000 (09:34 -0600)]
LU-13578 test: use a single read() in sanity test_39r

In sanity test_39r() ensure that the we only call read() once on the
file ot update it's atime. Otherwise the file atime may be greater
than the OST object atime due to a final read() done by dd which
returns no bytes and does not generate a BRW RPC to the OST. Even
though it returns 0 bytes, it requested a non-zero number of bytes and
is therefore required to update the file access time.

Fixes: 7c9ce8aac ("LU-13383 ofd: lazy atime update")
Test-Parameters: trivial testlist=sanity env=ONLY=39r,ONLY_REPEAT=400
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I1dd4e93ac150b34f4fc943fe25b92bf9119b0461
Reviewed-on: https://review.whamcloud.com/40973
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
3 years agoLU-14198 build: Use pax tar format for build of dist-targets 15/40915/2
Daniel Ahlin [Tue, 8 Dec 2020 23:30:34 +0000 (00:30 +0100)]
LU-14198 build: Use pax tar format for build of dist-targets

   The tar ustar format used during build of dist-* targets (which is
   used when building e.g. rpms and debs) has several limitations and
   will prevent e.g. users with uid > 2097151 (2^21) from building
   these targets (or from passing the autoconf test for tar).

   This commit changes format from ustar to POSIX.1-2001/pax which
   will remove this and several other limits (with path-length being
   one relevant example, see LU-12078).

Signed-off-by: Daniel Ahlin <ahlin@google.com>
Change-Id: Ic66ca696ede2e359a04c179c6d630baacaa9bcb1
Reviewed-on: https://review.whamcloud.com/40915
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
3 years agoLU-14286 osd-ldiskfs: enable fallocate by default 15/41315/3
Andreas Dilger [Mon, 25 Jan 2021 23:18:09 +0000 (16:18 -0700)]
LU-14286 osd-ldiskfs: enable fallocate by default

Enable fallocate on ldiskfs OSTs by default now that the known
problems have been resolved.  The default mode=0 is the standard
"allocate unwritten extents" behavior used by ext4.  This is by
far the fastest for space allocation, but requires the unwritten
extents to be split and/or zeroed when they are overwritten.

The OST fallocate mode=1 can also be set to use "zeroed extents",
which may be handled by "WRITE SAME", "TRIM zeroes data", or
other low-level functionality in the underlying block device.
This is somewhat slower at fallocate() time (especially for very
large allocations), but still avoids sending any data over the
network, avoids runtime overhead from managing the extents.  There
is not yet an FALLOC_FL_* flag to request this behavior from the
client on a per-file basis.

If problems are hit in the field, fallocate can also be disabled
with mode=-1 at runtime or persistently.

   lctl set_param [-P] osd-ldiskfs.*.fallocate_zero_blocks=<mode>

Ensure that all of the tests which currently use fallocate() are
enabling it for test runs, even if the default changes again.

Fixes: 4f18e08099e5 ("LU-14286 osd-ldiskfs: fallocate with unwritten extents")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Iefa71c525597d54fc82a3d6de27a50d4d2ce7057
Reviewed-on: https://review.whamcloud.com/41315
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14337 lov: return stripe_count=1 instead of 0 for DoM files 65/41265/2
Emoly Liu [Tue, 19 Jan 2021 04:06:06 +0000 (12:06 +0800)]
LU-14337 lov: return stripe_count=1 instead of 0 for DoM files

Return stripe_count=1 instead of 0 for DoM files to avoid
divide-by-zero for older userspace that calls this ioctl,
e.g. lustre ADIO driver.

Signed-off-by: Emoly Liu <emoly@whamcloud.com>
Change-Id: I43c5e01bdee834f9a05a669a3e6f3d5cd926cb87
Reviewed-on: https://review.whamcloud.com/41265
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14283 osc: avoid crash if ocd reset 25/41225/4
Andreas Dilger [Thu, 14 Jan 2021 16:01:21 +0000 (09:01 -0700)]
LU-14283 osc: avoid crash if ocd reset

Avoid divide-by-zero if OSC obd_connect_data is not fully initialized.
cl_ocd_grant_param is only set after cl_max_extent_pages is OK to use.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ibcee30b46e24ca3d4c2b571b27f3c0bb43f4bf71
Reviewed-on: https://review.whamcloud.com/41225
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14339 obdclass: add option %H for jobid 62/41262/3
Yang Sheng [Mon, 18 Jan 2021 17:46:05 +0000 (01:46 +0800)]
LU-14339 obdclass: add option %H for jobid

Add a option %H to avoid jobid too long in some cases.

Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: Iaf70da5de25fd321a21e6e6cd7f7d211dca1adf3
Reviewed-on: https://review.whamcloud.com/41262
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
3 years agoLU-14306 sec: get rid of bad rss-counter state messages 99/41199/12
Sebastien Buisson [Mon, 11 Jan 2021 09:36:06 +0000 (09:36 +0000)]
LU-14306 sec: get rid of bad rss-counter state messages

When doing O_DIRECT IOs on encrypted files, messages about bad
rss-counter state can be seen in the console. The mm get confused
because we twist the Lustre pages used for RPCs so that they are
suitable for llcrypt API.
In order to do this properly, the original mapping on these pages
must be preserved outside of the encryption/decryption needs.

Fixes: 728036f256 ("LU-12275 sec: O_DIRECT for encrypted file")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I80ebcd3f96c51a3d158d7ef66f23b8da13904c52
Reviewed-on: https://review.whamcloud.com/41199
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14286 osd-ldiskfs: fallocate with unwritten extents 04/41204/10
Andreas Dilger [Tue, 12 Jan 2021 04:38:31 +0000 (21:38 -0700)]
LU-14286 osd-ldiskfs: fallocate with unwritten extents

The osd_fallocate() code should typically be allocating unwritten
extents with LDISKFS_GET_BLOCKS_CREATE_UNWRIT_EXT instead of actually
zeroing the blocks on disk with LDISKFS_GET_BLOCKS_CREATE_ZERO.

Writing zeroes during fallocate() is typically slower initially, and
is causing timeouts in sanity test_150e, which is trying to fill up
all OSTs to 90%.  In some cases, zeroing the underlying blocks can
use the underlying storage support for efficient zeroing (WRITE_SAME),
so it may be faster for later use than uninitialized extents that have
to be converted to initialized extents by (possibly) splitting them
into smaller extents and/or zero filling them when they are paritally
being overwritten.

Add a tunable parameter osd-ldiskfs.*.fallocate_zero_blocks to allow
selecting this behavior at runtime.  The default is -1, to disable
fallocate completely (return -EOPNOTSUPP) due to current bugs.

Test-Parameters: testlist=sanityn env=ONLY=16,ONLY_REPEAT=10
Fixes: 72617588ac8c ("LU-14286 osd-ldiskfs: fallocate() should zero new blocks")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ida3692c487fdc8918863fc5c99459caaba17d92e
Reviewed-on: https://review.whamcloud.com/41204
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14312 ldlm: don't change GROUP lock GID on client 68/41268/5
Mikhail Pershin [Tue, 19 Jan 2021 14:21:57 +0000 (17:21 +0300)]
LU-14312 ldlm: don't change GROUP lock GID on client

GROUP lock GID is part of inodebits policy and is passed
to the server from client in policy li_gid field.
Meanwhile the ldlm_ibits_policy_wire_to_local() is used on
client also when server reply or completion AST is processed,
so client original GID can be overwritten by server value.
This is not problem if both server and client have the same
Lustre version but if server is older then it can have garbage
in li_gid field and client lock policy is updated with it.

Considering that GROUP lock GID is never changed and server should
not do that, the solution is to ignore returned li_gid from server
and never update original GID of GROUP lock on client from server
response.

Test-Parameters: testlist=sanity serverversion=2.12.6 env=ONLY=272b
Test-Parameters: testlist=sanity serverversion=2.13.0 serverdistro=el7.7 env=ONLY=272b

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I9a82f4b3513fd93d63b92a9527cb7b89c635e61b
Reviewed-on: https://review.whamcloud.com/41268
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
3 years agoLU-14034 hsm: add PID file handling to lhsmtool_posix 53/40253/9
John L. Hammond [Tue, 29 Dec 2020 02:10:31 +0000 (21:10 -0500)]
LU-14034 hsm: add PID file handling to lhsmtool_posix

Add pid-file handling to lhsmtool_posix to prevent accidentally
running concurrent instances of the copytool. (Multiple instances are
still allowed if you do not use PID files or use separate files.)

Use the PID file to avoid needing libtool when stopping, continuing,
or killing the copytool from test scripts.

Remove unnecessary libtool usage from test scripts.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I4f012a88c58f3a86a731df3b7d35ff32db047c2d
Reviewed-on: https://review.whamcloud.com/40253
Tested-by: Maloo <maloo@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Ben Evans <beevans@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14326 osc: correctly update size/kms for fallocate 72/41272/2
Bobi Jam [Fri, 15 Jan 2021 05:33:17 +0000 (13:33 +0800)]
LU-14326 osc: correctly update size/kms for fallocate

* fallocate chose oa->o_size for falloc_offset and o->o_blocks for
  falloc_end, but forgot to change attr->cat_size and attr->cat_kms
  to use sa_attr.lvb_size to update osc's lvb and kms if it expands
  the file's size.

  Other setattr IO uses @size (sa_falloc_offset in fallocate case) to
  update the lvb and kms.

* lock request extent for fallocate should be
  [sa_falloc_offset, sa_falloc_end)

* calculate sa_attr.lvb_size correctly for osc objects
  (lov_io_sub_inherit())

Test-Parameters: testlist=sanityn env=ONLY=16,COUNT=50000,ONLY_REPEAT=10
Test-Parameters: testlist=sanity-benchmark env=ONLY=fsx,COUNT=50000,ONLY_REPEAT=10
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Fixes: 48457868a02a ("LU-3606 fallocate: Implement fallocate preallocate operation")
Change-Id: I7dbed3bc6899a3db53284c8aac3cb9476e7958f5
Reviewed-on: https://review.whamcloud.com/41272
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14303 tests: parallel-scale test rr_alloc fails 92/41192/2
Yang Sheng [Sun, 10 Jan 2021 15:50:44 +0000 (23:50 +0800)]
LU-14303 tests: parallel-scale test rr_alloc fails

Correct the parameter for DNE environment. Else the
test case will fail on 'No such file or directory'.

Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: Ib94c5b17a3b49153ac229bfc4dfcee39bd9f60d4
Reviewed-on: https://review.whamcloud.com/41192
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14334 lnet: update changelog 63/41263/4
Serguei Smirnov [Mon, 18 Jan 2021 19:20:47 +0000 (11:20 -0800)]
LU-14334 lnet: update changelog

Updated changelog to indicate changes in OFED/MOFED support
as well as new features added in this version of LNet.

Test-Parameters: trivial
Change-Id: I264f6566324da42fd51a8e159d172cbf0ae1a28b
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/41263
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14283 obdclass: connect vs disconnect race 56/41256/3
Wang Shilong [Sat, 16 Jan 2021 12:46:13 +0000 (20:46 +0800)]
LU-14283 obdclass: connect vs disconnect race

There might be a possible race if setup (connect)
and cleanup (disconnect) are tangled together(similar
comments in osc_disconnect()):

  Thread1: Thread2:
   connecting  class_cleanup
     ptlrpc_connect_interpret
   obd->obd_setup = 0
obd_import_event
  if (obd->obd_set_up)
osc_init_grant() /*skipped*/
        ptlrpc_activate_import..

And If RPC was waked up and send out before
class_disconnect_exports(), It might hit divide zero crash
in osc_announce_cached() because @cl_max_extent_pages is zero.

The problem is we clear @obd_setup too early, It should be cleared
when OBD is really shutdown.

Fixes: 45900a ("LU-4134 obdclass: obd_device improvement")
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I898b6f53602c05221a3154a61615a0e270167ac6
Reviewed-on: https://review.whamcloud.com/41256
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-13602 flr: skip unknown FLR component types 13/39513/8
Qian Yingjin [Mon, 27 Jul 2020 03:56:22 +0000 (11:56 +0800)]
LU-13602 flr: skip unknown FLR component types

Currently, in lov_init_composite() it will quit with error when
reading an unknown LOV pattern.
Since FLR will be used for upcoming new features, like PCC-RO,
an old client should be able to read the old format of the
component types, ignore and skip the new types of FLR component.

Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: Ica3fe98203d44b52cf25b085c34c83b1a4702464
Reviewed-on: https://review.whamcloud.com/39513
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-12125 llite: send file mode with rename RPC 84/41184/5
Andreas Dilger [Sat, 9 Jan 2021 07:23:39 +0000 (00:23 -0700)]
LU-12125 llite: send file mode with rename RPC

In preparation for parallel rename operations, send renamed file mode
to the MDS in order to allow the rename locking to be more efficient.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I4691f30d151a8ff81e443d24109234341b3ebbe5
Reviewed-on: https://review.whamcloud.com/41184
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14286 osd-ldiskfs: don't read unwritten blocks 16/41216/4
Alex Zhuravlev [Wed, 13 Jan 2021 15:48:54 +0000 (18:48 +0300)]
LU-14286 osd-ldiskfs: don't read unwritten blocks

which were allocated using fallocate(2), instead fill
the pages with zeroes.

Add a test to verify that fallocated blocks read as zeroes.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I9c0c90f93fd33f26f834144e225b2643cf9fffb7
Reviewed-on: https://review.whamcloud.com/41216
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
3 years agoLU-14194 cksum: add lprocfs checksum support in MDC/MDT 71/40971/5
Mikhail Pershin [Tue, 15 Dec 2020 13:56:21 +0000 (16:56 +0300)]
LU-14194 cksum: add lprocfs checksum support in MDC/MDT

Add missed support for checksum parameters in MDC and MDT
Handle T10-PI parameters in MDT similar to OFD, move all
functionality to target code and unify its usage in both
targets

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I7d397067304e028bf597d5c3ab16250731ccba9d
Reviewed-on: https://review.whamcloud.com/40971
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
3 years agoLU-13676 tools: find unique stacks from sysrq-t 56/40956/5
Andreas Dilger [Sun, 13 Dec 2020 11:57:23 +0000 (04:57 -0700)]
LU-13676 tools: find unique stacks from sysrq-t

Enhance the crash-find-unique-traces.awk script to be more useful:
- allow parsing output from sysrq-t
- skip leading fields from input (e.g. timestamps, syslog header)
- sort messages to avoid duplicates
- handle some variance in kernel function names
- print thread names as well as PIDs for all matching threads
- print number of threads matching a particular stack trace
- warn if parts of the stack traces were lost due to buffer overflow

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Icfd70fbf255f413ca1aab0b94c62754e4c00628d
Reviewed-on: https://review.whamcloud.com/40956
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14317 ldiskfs: ‘%llu’ mismatch with type ‘long int’ on arm64 91/41191/2
Jian Yu [Mon, 11 Jan 2021 03:46:42 +0000 (19:46 -0800)]
LU-14317 ldiskfs: â€˜%llu’ mismatch with type â€˜long int’ on arm64

This patch fixes build issues like this one on arm64:

lustre-release/ldiskfs/mballoc.c: In function â€˜ldiskfs_mb_release’:
lustre-release/ldiskfs/mballoc.c:3180:4: warning:
format â€˜%llu’ expects argument of type â€˜long long unsigned int’,
but argument 4 has type â€˜long int’ [-Wformat=]
    "mballoc: (%llu, %llu, %llu) useless c(0,1,2) loops",
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Change-Id: I07018302fae8da3b71c0cc649753bdcccdc0d34f
Test-Parameters: trivial
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/41191
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
3 years agoLU-14263 gss: unlink revoked key 47/41047/4
Sebastien Buisson [Fri, 18 Dec 2020 14:49:11 +0000 (23:49 +0900)]
LU-14263 gss: unlink revoked key

When a GSS context is destroyed, it is unbound from its key, marking
the key as revoked.
The key also needs to be unlinked from the session keyring. This way,
a subsequent context initialization will manage to create a new valid
key and link it to the keyring.

Similarly, add a new '-r' flag to 'lfs flushctx', in order to reap the
revoked keys from the keyring when flushing the GSS context.

Test-Parameters: trivial
Test-Parameters: clientdistro=el7.9 testgroup=review-dne-ssk
Test-Parameters: clientdistro=el8.3 testgroup=review-dne-ssk
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ida4b4ea53202c1f40ad93816fb4ec96fec2bf8bc
Reviewed-on: https://review.whamcloud.com/41047
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14286 tests: reduce fallocate space sanity 150e 11/41211/3
James Nunez [Tue, 12 Jan 2021 17:56:32 +0000 (10:56 -0700)]
LU-14286 tests: reduce fallocate space sanity 150e

Changes to fallocate have increase the time to allocate
space.  The increased run time is making sanity test
150e run long enough to time out.  We need to reduce the
amount of space santiy test 150e fallocates from 90%
to 100 MB per OST or 60% of available space if running
in SLOW mode to allow the test to complete.

Test-Parameters: trivial env=ONLY=150e fstype=ldiskfs testlist=sanity
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: Ic6ab41daf8f451c31d092849ccf6906747baa72d
Reviewed-on: https://review.whamcloud.com/41211
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
3 years agoLU-14324 tests: Fix fsx fallocate detection 02/41202/5
Oleg Drokin [Mon, 11 Jan 2021 23:57:40 +0000 (18:57 -0500)]
LU-14324 tests: Fix fsx fallocate detection

Avoid kernel playing fake tricks on us by optimizing away small
increases in file size by fallocate, so to ensure the truth make
large fallocate call instead.
Additionally always truncate to zero after all call, as even a
failed call can result in increased file size.

Test-Parameters: trivial testlist=sanity env=ONLY=150
Test-Parameters: testlist=sanityn env=ONLY=16,COUNT=10000
Test-Parameters: testlist=sanity-dom env=ONLY=fsx,sanityn,SANITYN_ONLY=16
Test-Parameters: fstype=zfs testlist=sanity env=ONLY=150
Test-Parameters: fstype=zfs testlist=sanityn env=ONLY=16,COUNT=10000
Test-Parameters: fstype=zfs testlist=sanity-dom env=ONLY=fsx,sanityn,SANITYN_ONLY=16
Change-Id: I8cddb7d0d30bdf89bf13d18ae36a5f950adc392d
Fixes: 03247bda765c ("LU-14286 osd-ldiskfs: fallocate() should zero new blocks")
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/41202
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Tested-by: Maloo <maloo@whamcloud.com>
3 years agoLU-14128 lov: correctly set OST obj size 81/40581/6
Bobi Jam [Mon, 9 Nov 2020 15:20:10 +0000 (23:20 +0800)]
LU-14128 lov: correctly set OST obj size

When extends a PFL file to a size locating at a boundary of a stripe
in a component, the truncate won't set the size of the OST object
in the prior stripe.

This patch record the prior stripe in
lov_layout_raid0::lo_trunc_stripeno and add the stripe in the
truncate IO and enqueue the lock covering it.

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Ic5d8e3c16f950003736cd6dbd5af404613f818c7
Reviewed-on: https://review.whamcloud.com/40581
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14296 lnet: use an unbound cred in kiblnd_resolve_addr() 37/41137/3
John L. Hammond [Tue, 5 Jan 2021 22:08:34 +0000 (16:08 -0600)]
LU-14296 lnet: use an unbound cred in kiblnd_resolve_addr()

In kiblnd_resolve_addr() call prepare_kernel_cred(NULL) rather than
prepare_creds() to get a cred with unbound capabilities.

Fixes: 1e4bd16acfa ("LU-14006 o2ib: raise bind cap before resolving address")
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Ib2500d82aa4eab5590fa95ae73fd75b2ab542bcd
Reviewed-on: https://review.whamcloud.com/41137
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14301 llite: return EOPNOTSUPP if fallocate is not supported 48/41148/3
John L. Hammond [Wed, 6 Jan 2021 19:48:35 +0000 (13:48 -0600)]
LU-14301 llite: return EOPNOTSUPP if fallocate is not supported

In ll_fallocate() if the server returns the NFSv3 specific error code
ENOTSUPP then replace it with EOPNOTSUPP to avoid confusing
applications.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I85952be27808704112a6902684900cd938d576ac
Reviewed-on: https://review.whamcloud.com/41148
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Wang Shilong <wshilong@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
3 years agoLU-14281 tests: fix sanity-sec test_56 for ZFS 30/41130/4
Sebastien Buisson [Tue, 5 Jan 2021 09:23:32 +0000 (10:23 +0100)]
LU-14281 tests: fix sanity-sec test_56 for ZFS

Skip sanity-sec test_56 in case of ZFS backend, as it does not
support FIEMAP.
Also simplify sanity-sec test_56, as all we care about is whether the
"encrypted" flag is set at all.

Fixes: 33322f3a2488 ("LU-14149 llite: fiemap set flags for encrypted files")
Test-Parameters: trivial
Test-Parameters: testlist=sanity-sec env=ONLY=56 clientdistro=el8.3 serverdistro=el8.3 fstype=ldiskfs
Test-Parameters: testlist=sanity-sec env=ONLY=56 clientdistro=el8.3 serverdistro=el8.3 fstype=zfs
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I69125d3e06acb89dbda421e573d18a2c22beac37
Reviewed-on: https://review.whamcloud.com/41130
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
3 years agoLU-13783 o2iblnd: make FMR-pool support optional. 87/40287/6
Mr NeilBrown [Fri, 16 Oct 2020 04:53:34 +0000 (15:53 +1100)]
LU-13783 o2iblnd: make FMR-pool support optional.

Linux 5.8 removes the FMR-pool API.  This patch makes
all use for this API optional, selected only if the
support exists in the kernel.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I4c40f3a766f5b46ae4f26d7d3ecf8434a6e5a0cb
Reviewed-on: https://review.whamcloud.com/40287
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sergey Gorenko <sergeygo@nvidia.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14286 osd-ldiskfs: fallocate() should zero new blocks 19/41119/10
Arshad Hussain [Thu, 31 Dec 2020 19:40:34 +0000 (01:10 +0530)]
LU-14286 osd-ldiskfs: fallocate() should zero new blocks

The ldiskfs osd_fallocate() does not correctly zero the
allocated blocks. This is not noticed during autotest and
local developer testing because the underlying storage is
zero-filled due to using a sparse backing file and will
usually read back as zero. Also, the fsx data correctness
test was not properly detecting fallocate() support, and
as a result this behavior was not detected properly.

'fsx' uses test_fallocate() to determine if fallocate()
along with its various FLAGS is enabled and could be
called. It was found that test_fallocate() always returned
false as a bug. Resulting is fallocate being skipped always
for all fsx runs.

This patches fixes test_fallocate() to properly determine
if fallocate calls could be supported or not and return
appropriately and correctly (1 or 0) true/false.

After patch:
~~~~~~~~~~~
$ MOUNT_2="yes" bash ./lustre/tests/llmount.sh

$ lfs setstripe -c -1 /mnt/lustre/f110

$ fsx -c 50 -p 100 -N 2500 -S 0 -l 10485760 /mnt/lustre*/f110
Chance of close/open is 1 in 50
Seed set to 4481
fd 0: /mnt/lustre/f110
fd 1: /mnt/lustre2/f110
fsx: test_fallocate: does not support fallocate mode 0x3, disabling!
fsx: test_fallocate: does not support fallocate mode 0x10, disabling!
skipping zero size read
:
fallocating to largest ever: 0x30ce
:
1900[0] 1609444486.316868 fallocate  0x7f6c thru 0xab3a (0x2bcf bytes)
:
2300[1] 1609444487.241910 fallocate  0x0697 thru 0xed8e (0xe6f8 bytes)
:
All operations completed A-OK!
$

Test-Parameters: trivial testlist=sanity,sanityn,sanity-dom env=COUNT=10000
Fixes: 48457868a02a ("LU-3606 fallocate: Implement fallocate preallocate operation")
Fixes: 853d180121a6 ("LU-3606 fsx: Add fallocate operation to fsx")
Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Change-Id: Idbb092db6bd09cfda129f6077dcbfae4f4c9d0d0
Reviewed-on: https://review.whamcloud.com/41119
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
3 years agoLU-14298 tests: fix sanity-sec test_44 for aarch64 client 46/41146/6
Sebastien Buisson [Wed, 6 Jan 2021 10:13:56 +0000 (11:13 +0100)]
LU-14298 tests: fix sanity-sec test_44 for aarch64 client

PAGE_SIZE on aarch64 client is 64KB instead of 4KB, so sizes of IOs
performed in sanity-sec test_44 must be adapted.

Test-Parameters: testgroup=review-ldiskfs-arm env=ONLY=44 clientdistro=el8.3 serverdistro=el8.3
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I3764a75f6df871d71d751764627ed618bedf8e0b
Reviewed-on: https://review.whamcloud.com/41146
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
3 years agoLU-14187 osd-ldiskfs: fix locking in write commit 91/40991/12
Wang Shilong [Wed, 16 Dec 2020 07:10:56 +0000 (15:10 +0800)]
LU-14187 osd-ldiskfs: fix locking in write commit

Restart transaction in osd layer break rules that
locks are taken afer transaction start.

This patch try to fix by moving transaction restart
to OFD layer.

We will record how many extents we declare with
@oh_declared_ext. And during IO we check if it
run out before we restart whole transaction.

We track an average extent bytes per filesystem and
use it as a hint during declare write commit. This
will avoid us restart transaction ofen if filesystem
is heavily fragmented.

Fixes: 0271b17b ("LU-14134 osd-ldiskfs: reduce credits for new writing")
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I289a3a6775befe159b7fef29004eaaaff873e2c6
Reviewed-on: https://review.whamcloud.com/40991
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
3 years agoLU-13641 socklnd: announce deprecation of 'use_tcp_bonding' 88/41088/5
Serguei Smirnov [Thu, 24 Dec 2020 01:43:21 +0000 (17:43 -0800)]
LU-13641 socklnd: announce deprecation of 'use_tcp_bonding'

Add warning to be printed if 'use_tcp_bonding' option is used
notifying the user that the feature is being deprecated.
It is suggested to use MR configuration with dynamic discovery
instead.

Multi-Rail feature doesn't need to be explicitly enabled.
To use MR instead of tcp bonding, group the interfaces
on the same network using the lnetctl utility:

lnetctl net add --net tcp --if eth2,eth3

or via the modprobe configuration file (/etc/modprobe.d/lnet.conf
or /etc/modprobe.d/lustre.conf):

        options lnet networks="tcp(eth2,eth3)"

and make sure dynamic discovery is enabled:

        lnetctl set discovery 1

MR will aggregate the throughput of all configured and available
networks/interfaces shared between peer nodes.

Test-Parameters: trivial
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: I34288ae1c8a1c4092f88b45a571312691f145218
Reviewed-on: https://review.whamcloud.com/41088
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
3 years agoLU-14267 osd: do not update inode each write 69/41069/3
Alex Zhuravlev [Tue, 22 Dec 2020 07:08:16 +0000 (10:08 +0300)]
LU-14267 osd: do not update inode each write

there is no need to update in-bh inode each write,
overwrites can skip that. otherwise:
1) transaction can exceed reserved credits as
   osd_declare_write() doesn't count inode for overwrites
2) copy is not free and this can be called few times for
   each operation: last_rcvd data, changelog.

Fixes: a23aac2219 ("LU-13416 ldiskfs: don't corrupt data on journal replay")
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I47f463d9421cea7fb32e03fa5ba0ab6524511a29
Reviewed-on: https://review.whamcloud.com/41069
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
3 years agoLU-14265 utils: allow 'lfs mkdir' without arguments 54/41054/2
Andreas Dilger [Fri, 18 Dec 2020 22:47:09 +0000 (15:47 -0700)]
LU-14265 utils: allow 'lfs mkdir' without arguments

If no arguments are given to "lfs mkdir <newdir>" assume that
"-i -1 -c 1" were given, to create the specified <newdir> on
an MDT with above-average free space/inodes.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I319f68240832a905a86833a70a4631cde5ce7057
Reviewed-on: https://review.whamcloud.com/41054
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14218 kernel: kernel update SLES15 SP1 [4.12.14-197.75.1] 30/41030/3
Jian Yu [Thu, 17 Dec 2020 18:15:50 +0000 (10:15 -0800)]
LU-14218 kernel: kernel update SLES15 SP1 [4.12.14-197.75.1]

Update SLES15 SP1 kernel to 4.12.14-197.75.1 for Lustre client.

Test-Parameters: trivial \
env=SANITY_EXCEPT="56oc 100 130 136 817" \
clientdistro=sles15sp1 serverdistro=el7.8 \
testlist=sanity

Change-Id: I79d217d3c87da6a108b58d8ce8c02da60d9401a2
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/41030
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-12702 quota: wait pending write before acquiring remotely 23/41023/9
Wang Shilong [Mon, 21 Dec 2020 02:18:45 +0000 (10:18 +0800)]
LU-12702 quota: wait pending write before acquiring remotely

There is a window that ZFS has updated usage, but
pending_write has not been reduced. This could cause
us to grant more space.

With soft least qunit introduced, after timer reach,
local slave might still have some space to consume,
it is possible that we might exceed this limit.

Test-Parameters: fstype=zfs testlist=sanity-quota,sanity-quota,sanity-quota
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I0e1a94fd0b637e154642de5922a0670780c56ef2
Reviewed-on: https://review.whamcloud.com/41023
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14082 ldlm: Use req_mode while lock cleanup 33/40433/4
Yang Sheng [Wed, 28 Oct 2020 13:03:47 +0000 (21:03 +0800)]
LU-14082 ldlm: Use req_mode while lock cleanup

For local lock, the decref cannot count exactly by
granted_mode if the lock has not been granted.
LustreError: (ldlm_lock.c:354:ldlm_lock_destroy_internal())
 ### lock still has references ns: ??
 lock: ffff88342aa07200/0x9b92ad3407bea22a
 lrc: 4/0,1 mode: --/PW res: ?? rrc=?? type: ???
 flags: 0x10106400000000 nid: local
 remote: 0x5248822d3123ac19 expref: -99
 pid: 14515 timeout: 0 lvb_type: 0
LustreError: (ldlm_lock.c:355:ldlm_lock_destroy_internal()) LBUG
Pid: 14562, comm: ll_imp_inval 3.10.0-693.21.1.el7.x86_64 #1 SMP
Call Trace:
[] save_stack_trace_tsk+0x22/0x40
[] libcfs_call_trace+0x8c/0xc0 [libcfs]
[] lbug_with_loc+0x4c/0xa0 [libcfs]
[] ldlm_lock_destroy_internal+0x269/0x2a0 [ptlrpc]
[] ldlm_lock_destroy_nolock+0x2b/0x110 [ptlrpc]
[] ldlm_flock_completion_ast+0x4f5/0x1080 [ptlrpc]
[] cleanup_resource+0x18e/0x370 [ptlrpc]
[] ldlm_resource_clean+0x53/0x60 [ptlrpc]
[] cfs_hash_for_each_relax+0x250/0x450 [libcfs]
[] cfs_hash_for_each_nolock+0x75/0x1c0 [libcfs]
[] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc]
[] mdc_import_event+0x1b6/0xa20 [mdc]
[] ptlrpc_invalidate_import+0x220/0x8f0 [ptlrpc]
[] ptlrpc_invalidate_import_thread+0x48/0x2b0 [ptlrpc]
[] kthread+0xd1/0xe0

Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: Ic029b3fb82a3dedc507e2bada265322efae5b40c
Reviewed-on: https://review.whamcloud.com/40433
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14097 tests: conf-sanity/91 should check specific MDT 01/40501/4
Alex Zhuravlev [Sat, 31 Oct 2020 05:55:10 +0000 (08:55 +0300)]
LU-14097 tests: conf-sanity/91 should check specific MDT

for exports afer eviction. otherwise 91 fails if few MDTs
share same node as in local setup.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I047473311c01f94e210e8f87aced4cbd7de76b7f
Reviewed-on: https://review.whamcloud.com/40501
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
3 years agoLU-11365 tests: test-framework::add() to load ldiskfs 40/33140/16
Alex Zhuravlev [Tue, 11 Sep 2018 17:54:43 +0000 (20:54 +0300)]
LU-11365 tests: test-framework::add() to load ldiskfs

when the tests run from in-tree build, then mkfs.lustre
can not rely on modprobe and ldiskfs needs to be loaded
explicitly. otherwise some tests like conf-sanity/55 fail.

Change-Id: Ic15f0eab8e483d5de9f63606433fbb26f754fd55
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/33140
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14214 ofd: fix locking in ofd_object_fallocate() 69/40969/2
Alex Zhuravlev [Tue, 15 Dec 2020 05:44:59 +0000 (08:44 +0300)]
LU-14214 ofd: fix locking in ofd_object_fallocate()

the rule is that locks are taken after transaction start.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I5252be0033611f03b355c1f98e21422cdde55c9e
Reviewed-on: https://review.whamcloud.com/40969
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Wang Shilong <wshilong@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-8066 osd-ldiskfs: quiet debug mount message 68/40968/3
Andreas Dilger [Mon, 14 Dec 2020 23:08:46 +0000 (16:08 -0700)]
LU-8066 osd-ldiskfs: quiet debug mount message

We don't need a message printed to the console for every
mount reporting that tunable parameters were configured.

Test-Parameters: trivial
Fixes: 493cd8088388 ("LU-8066 osd: migrate from proc to sysfs")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I12fb89f8f15a86657fe5c1f46359f184ce3ebbe5
Reviewed-on: https://review.whamcloud.com/40968
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14192 mdt: check for overstriping correctly 96/40896/3
Vitaly Fertman [Mon, 19 Oct 2020 20:46:15 +0000 (23:46 +0300)]
LU-14192 mdt: check for overstriping correctly

check for overstriping in the first component as well

HPE-bug-id: LUS-9430
Signed-off-by: Vitaly Fertman <c17818@cray.com>
Change-Id: Ice125b885681b0cf71dc55c5238c4c4d7d7bb970
Reviewed-on: https://es-gerrit.dev.cray.com/157999
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Tested-by: Elena Gryaznova <c17455@cray.com>
Reviewed-by: Alexander Zarochentsev <c17826@cray.com>
Reviewed-on: https://review.whamcloud.com/40896
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14191 lod: comp stripe count limit check 95/40895/2
Vitaly Fertman [Thu, 26 Nov 2020 18:37:43 +0000 (21:37 +0300)]
LU-14191 lod: comp stripe count limit check

when creating a PFL file of 2K stripes, they are shrinked to 700+,
because lod_get_stripe_count() checks the space left after taking
given 2K stripes into account, and the total amount of stripes that
fits into EA is 2700+.

when creating a SEL file of 2K stipes, it fails because the extension
component is also taking into account despite the fact it is never
initialised and due to the above logic 2x2k stripes leaves no space
in EA.

HPE-bug-id: LUS-9589
Signed-off-by: Vitaly Fertman <c17818@cray.com>
Change-Id: Ibeb063472d477e393646860010823fed5acb8382
Reviewed-by: Alexander Boyko <c17825@cray.com>
Reviewed-by: Artem Blagodarenko <c17828@cray.com>
Tested-by: Elena Gryaznova <c17455@cray.com>
Reviewed-on: https://review.whamcloud.com/40895
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-by: Artem Blagodarenko <artem.blagodarenko@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14158 mdc: process changelogs_catalog from the oldest rec 86/40786/2
Etienne AUJAMES [Fri, 27 Nov 2020 20:26:15 +0000 (21:26 +0100)]
LU-14158 mdc: process changelogs_catalog from the oldest rec

The chlg_load use the LLOG_CAT_FIRST to process changelogs. This
values will process record in the catalog always starting with index
0 to the newest record. So when catalog reach the end of indexes and
when records are saved at the begining of catalog, the
llog_cat_process will ignore records at the end.

This patch change the "startcat" value LLOG_CAT_FIRST to 0 to scan
the catalog from the oldest record to the newest.

Fixes: e2150028 (LU-12553  mdc: polling mode for changelog reader)
Signed-off-by: Etienne AUJAMES <eaujames@ddn.com>
Change-Id: Ib00a4373d87fd63bfb5d05731f1fdbd286c0b6f5
Reviewed-on: https://review.whamcloud.com/40786
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-4165 tests: restore skipped sanity-lfsck tests 60/40760/5
Andreas Dilger [Wed, 25 Nov 2020 22:50:04 +0000 (15:50 -0700)]
LU-4165 tests: restore skipped sanity-lfsck tests

Many subtests in sanity-lfsck were incorrectly being skipped due
to a cleanup patch that was intended to skip those tests for very
old Lustre versions being run on the MDS, or for ZFS MDTs.

Instead, a large number of tests were *always* being skipped.

Restore those accidentally-skipped tests.  Hopefully they still pass.

Clean up code style for the script:
- remove the needless "return" after "skip"
- use mds1_FSTYPE and MDS1_VERSION instead of getting values from MDS

Fixes: c54b6ca2bdb ("LU-13718 tests: add LU numbers to skipped tests")
Test-Parameters: trivial fstype=ldiskfs testlist=sanity-lfsck
Test-Parameters: fstype=ldiskfs mdscount=2 mdtcount=4 testlist=sanity-lfsck
Test-Parameters: fstype=zfs testlist=sanity-lfsck
Test-Parameters: fstype=zfs mdscount=2 mdtcount=4 testlist=sanity-lfsck
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I4e49d79826369a546576672f77f7012f4a3ebbe5
Reviewed-on: https://review.whamcloud.com/40760
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14075 kernel: kernel update RHEL8.2 [4.18.0-193.28.1.el8_2] 17/40417/5
Jian Yu [Tue, 15 Dec 2020 08:13:20 +0000 (00:13 -0800)]
LU-14075 kernel: kernel update RHEL8.2 [4.18.0-193.28.1.el8_2]

Update RHEL8.2 kernel to 4.18.0-193.28.1.el8_2.

Test-Parameters: trivial \
clientdistro=el8.2 serverdistro=el8.2 \
testlist=sanity

Change-Id: I34e1e51241c3090d1041dedef8379c2e212f58a5
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/40417
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-13745 pcc: fall back normal splice read for detached file 96/40396/4
Qian Yingjin [Mon, 26 Oct 2020 02:37:26 +0000 (10:37 +0800)]
LU-13745 pcc: fall back normal splice read for detached file

For the file that was once cached into PCC but detached from PCC
later, it should fall back into the normal I/O path to direct I/Os
to Lustre OSTs for splice read, not just return zero.

Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: Iff3449069a0a12ac56b58618e474749361c31bb7
Reviewed-on: https://review.whamcloud.com/40396
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14205 osd-ldiskfs: fix return of osd_extend_restart_trans() 89/40989/2
Wang Shilong [Wed, 16 Dec 2020 02:50:24 +0000 (10:50 +0800)]
LU-14205 osd-ldiskfs: fix return of osd_extend_restart_trans()

ldiskfs_journal_ensure_credits() could possibly return 1 if
transaction restarted, Lustre osd expect it return 0 or negative.

Fixes: cf48a2d23b ("LU-13839 kernel: RHEL 8.3 server support")
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I51ff0bd50575902b66529cbd71ff14710c202d03
Reviewed-on: https://review.whamcloud.com/40989
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-11848 misc: update e2fsprogs to 1.45.6.wc3 67/40967/2
Andreas Dilger [Mon, 14 Dec 2020 21:34:55 +0000 (14:34 -0700)]
LU-11848 misc: update e2fsprogs to 1.45.6.wc3

Update Changelog to reference new e2fsprogs release.

23547cbe57df LU-14126 e2fsck: update mmp block race
0b6718f5e22d LU-11848 filefrag: support PFL and FLR file
502f2e17bd5b LU-13797 e2fsck: quiet 'extent tree could be narrower'
94318cef3508 misc: replace remaining loff_t with ext2_loff_t

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I4b3859dfe3950f3b23e47f69ecf8b3ab053ebbe5
Reviewed-on: https://review.whamcloud.com/40967
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@whamcloud.com>
Reviewed-by: Peter Jones <pjones@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-11848 tests: handle 4-digit version numbers properly 97/40897/4
Andreas Dilger [Mon, 7 Dec 2020 20:08:45 +0000 (13:08 -0700)]
LU-11848 tests: handle 4-digit version numbers properly

Allow version_code() to handle and compare 4-digit version numbers
properly.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I8f57537f87bb1c087def277d0d6192def43ebbe5
Reviewed-on: https://review.whamcloud.com/40897
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14227 tests: Add missing space in sanity-lnet test 21 03/41003/2
Oleg Drokin [Wed, 16 Dec 2020 14:51:40 +0000 (09:51 -0500)]
LU-14227 tests: Add missing space in sanity-lnet test 21

otherwise echo"Add peer with nidrange (gni)" is not a valid command

Change-Id: Ibdff3f38338bb880dd4ae23e31ca2517f4626010
Test-Parameters: trivial testlist=sanity-lnet
Fixes: 6a8335be5c2b ("LU-12410 tests: Add gni tests to sanity-lnet")
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/41003
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
3 years agoLU-11848 lov: FIEMAP support for PFL and FLR file 66/40766/23
Bobi Jam [Thu, 26 Nov 2020 06:13:28 +0000 (14:13 +0800)]
LU-11848 lov: FIEMAP support for PFL and FLR file

* use the high 16 bits of fe_device to record the absolute stripe
  number from the beginning we are processing, so that continuous call
  can resume from the stripe specified by it.

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Ie519a9361b2d3ae4977ed2ccf925c92065aa9787
Reviewed-on: https://review.whamcloud.com/40766
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
3 years agoLU-13839 ldiskfs: fix detecting ext4_journal_ensure_credits 39/40939/4
Mr NeilBrown [Thu, 10 Dec 2020 21:41:34 +0000 (08:41 +1100)]
LU-13839 ldiskfs: fix detecting ext4_journal_ensure_credits

As __ext4_journal_ensure_credits() is not exported, the current
detection doesn't work.
So add a local definition so that the build succeeds.

Fixes: cf48a2d23bca ("LU-13839 kernel: RHEL 8.3 server support")
Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I5f4b0883e8425d26760ea77e89379103dce30855
Reviewed-on: https://review.whamcloud.com/40939
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
3 years agoLU-14185 osp: alloc fails for OSP obj xattr cache 71/40871/2
Vitaly Fertman [Fri, 4 Dec 2020 19:18:48 +0000 (22:18 +0300)]
LU-14185 osp: alloc fails for OSP obj xattr cache

osp_oac_xattr_find_or_add() may get ENOMEM as it tries to allocate
over len=buf->lb_len=XATTR_SIZE_MAX, thus order:5. Use large alloc
instead.

HPE-bug-id: LUS-9551
Signed-off-by: Vitaly Fertman <c17818@cray.com>
Change-Id: I98c08d16d0343cb60f6efcb63d20739103f2af1c
Reviewed-by: Sergey Cheremencev <c17829@cray.com>
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Tested-by: Alexander Lezhoev <c17454@cray.com>
Reviewed-on: https://review.whamcloud.com/40871
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
3 years agoLU-14080 gnilnd: updates for SUSE 15 SP2 26/40426/3
Shaun Tancheff [Tue, 27 Oct 2020 21:03:13 +0000 (16:03 -0500)]
LU-14080 gnilnd: updates for SUSE 15 SP2

gnilnd updates for SUSE 15 SP2

Use -Werror when checking for -Wno-stringop-truncation to
ensure the compile test is valid

Adjust for kernel dropping time_t, timeval, timespec
Adjust for kernel switching to timer_setup

Adjust for kernel dropping global_page_state(), use
nr_free_pages instead

Cleanup string format for stricter checking

Test-Parameters: trivial
HPE-bug-id: LUS-9453
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I5dcda1497fa5b1f2cf4a215517700f07374fcf7f
Reviewed-on: https://review.whamcloud.com/40426
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14161 obdclass: fix some problems with obd_nid_hash 46/40846/5
Mr NeilBrown [Thu, 3 Dec 2020 00:48:41 +0000 (11:48 +1100)]
LU-14161 obdclass: fix some problems with obd_nid_hash

There are a few of problems with the handling of the obd_nid_hash
rhl-table.

- obd_export_evict_by_nid() drops out of rcu_readlock() while holding
  a reference into the table.  This is theoretically unsafe as changed
  to the table might cause entries to be missed.
- nid_keycmp() ignores entries with exp_failed set.  This is a problem
  because rhltable_lookup() only compares the key for the *first*
  object in a list of objects with the same key - on a match it
  returns the whole list.
  So if exp_failed was set on the first in the list, all would become
  invisible.

These can be fixed by moving the test on ->exp_failed to after the
call to rhltabke_lookup(), and by using obd_nid_export_for_each() in
obd_export_evict_by_nid(), and repeatedly dealing with the first exp
returned.

Fixes: 580ef453d1d6 ("LU-8130 obd: convert obd_nid_hash to rhashtable")
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I65861c179e93c26cf39c5db8a58e3e4c9b962fe3
Reviewed-on: https://review.whamcloud.com/40846
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-9114 ldlm: don't compute sumsq for pool stats 35/39435/2
Andreas Dilger [Fri, 17 Jul 2020 10:08:08 +0000 (04:08 -0600)]
LU-9114 ldlm: don't compute sumsq for pool stats

Remove the calculation of sumsq from the LDLM pool stats, since
these stats are almost never used, while conversely the pools
are updated frequently.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I90612b613fb76f4500ca6cd5613f5c121b3ebbe5
Reviewed-on: https://review.whamcloud.com/39435
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14100 gnilnd: fix kgnilnd_setup_immediate_buffer 04/40504/4
Shaun Tancheff [Thu, 10 Dec 2020 10:15:54 +0000 (04:15 -0600)]
LU-14100 gnilnd: fix kgnilnd_setup_immediate_buffer

struct kvec * removed from kgnilnd_setup_immediate_buffer

Fixes: c87f41dd20 ("LU-13004 gnilnd: discard struct kvec arg.")
Test-Parameters: trivial
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: Ie3c511326af6ddb09b6bfa13bb779fc094ba30ec
Reviewed-on: https://review.whamcloud.com/40504
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-9215 build: Re-add the lustre-devel package 28/40728/5
James Simmons [Sun, 6 Dec 2020 02:03:22 +0000 (21:03 -0500)]
LU-9215 build: Re-add the lustre-devel package

Traditonally on rpm systems a special devel package exist that
contains userland headers and symlinks for the libraries. This
patch creates such a package.

The introduction of this new package changes the dependency
of lustre-test packages since the user land headers are tested
to make sure they compile for user land applications. Update
the dpk package as well.

Lastly we add pkg-config support to Lustre as well. This means
that if do configure --prefix=special_place it will not break
external applications if they use pkg-config.

Change-Id: I3e856d85239d1712e21f7ecc633d9fae569238e6
Test-Parameters: trivial
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/40728
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14205 osd-ldiskfs: return correct error after end io 44/40944/2
Wang Shilong [Fri, 11 Dec 2020 07:17:50 +0000 (15:17 +0800)]
LU-14205 osd-ldiskfs: return correct error after end io

@bi_status could be positive in newer kernel, we need use
blk_status_to_errno() helper to convert error properly.

Otherwise, a positive value could be returned to client
which might trigger unexpected error and make client
fail to resend.

Test-Parameters: clientdistro=el8.3 serverdistro=el8.3
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I2e8f257814eff311b72416788b7f6d9e37bad84a
Reviewed-on: https://review.whamcloud.com/40944
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
3 years agoLU-14149 llite: fiemap set flags for encrypted files 52/40852/5
Sebastien Buisson [Thu, 3 Dec 2020 14:43:29 +0000 (15:43 +0100)]
LU-14149 llite: fiemap set flags for encrypted files

FIEMAP ioctl needs to set
FIEMAP_EXTENT_DATA_ENCRYPTED|FIEMAP_EXTENT_ENCODED flags for all
extents of files encrypted by fscrypt.

Also add sanity-sec test_56 to exercise the flags, with just a
plain file layout for now.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I26f788b5c4b37e96de57334f48847b40886e14c2
Reviewed-on: https://review.whamcloud.com/40852
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
3 years agoLU-14208 utils: remove copy_file_range() usage from copytool 66/40966/2
John L. Hammond [Mon, 14 Dec 2020 18:36:32 +0000 (12:36 -0600)]
LU-14208 utils: remove copy_file_range() usage from copytool

Revert the lhsmtool_posix part of "LU-11621 utils: optimize
lhsmtool_posix with copy_file_range()". This is failing sanity-hsm
test_55() on RHEL 8.3.

Test-Parameters: clientdistro=el8.3 serverdistro=el8.3 testlist=sanity-hsm
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I0b410c2bb5f56a32c10f0ec39190735908cbb556
Reviewed-on: https://review.whamcloud.com/40966
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
3 years agoLU-10664 dom: non-blocking enqueue for DOM locks 03/36903/25
Mikhail Pershin [Sun, 1 Dec 2019 06:41:37 +0000 (09:41 +0300)]
LU-10664 dom: non-blocking enqueue for DOM locks

DOM lock enqueue waits for blocking locks on MDT due to
ATOMIC flag, so MDT thread is blocked until lock is granted.
When many clients attempt to write to shared file that may
cause server thread starvation and lock contention. Switch
to non-atomic lock enqueue for DOM locks.

- switch IO lock to non-intent enqueue, so it doesn't consume
  server thread for a long time being blocked
- on client take LVB from l_lvb_data updated by completion AST and
  update l_ost_lvb used by DoM
- make glimpse performing similarly on MDT and OST, it uses one
  format with no intent buffer and return data in LVB buffer
- introduce new connect flag 'dom_lvb' for compatibility reasons
- on server handle glimpse for both old and new clients by filling
  either LVB reply buffer or mdt_body buffer
- don't take RPC slot for a DOM enqueue like it is done for EXTENT
  locks, update ldlm_cli_enqueue_fini() to accept ldlm_enqueue_info
  as parameter
- check that there is no atomic local lock issued with mandatory DOM
  bit, trybits should be used

Test-Parameters: mdssizegb=20 testlist=dom-performance
Test-Parameters: testlist=sanity-dom clientversion=2.12
Test-Parameters: testlist=sanity-dom serverversion=2.12
Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I06573018126f0663a8f76dce07e126a1879cfba5
Reviewed-on: https://review.whamcloud.com/36903
Reviewed-by: Vitaly Fertman <vitaly.fertman@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14199 sec: find policy version in use for sepol 18/40918/2
Sebastien Buisson [Wed, 9 Dec 2020 12:34:05 +0000 (13:34 +0100)]
LU-14199 sec: find policy version in use for sepol

SELinux exports the maximum kernel policy version that can be used.
When building SELinux status checking representation 'sepol', we need
to look for all possible versions of the policy, not only the max one.

Test-Parameters: clientdistro=el8.3 serverdistro=el8.2 testgroup=review-dne-selinux
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Iae4b66403ce953e5a7c0df585900713c597ff033
Reviewed-on: https://review.whamcloud.com/40918
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
3 years agoLU-14095 gss: use RCU protection for sunrpc cache 14/40914/2
Sebastien Buisson [Tue, 8 Dec 2020 14:17:40 +0000 (15:17 +0100)]
LU-14095 gss: use RCU protection for sunrpc cache

Leverage sunrpc cache RCU protection for cache entry removal,
if the kernel supports it.

Test-Parameters: clientdistro=el8.2 serverdistro=el8.2 testgroup=review-dne-ssk
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I50f9dc9e9dc434b502df1e1013215b5506289ff9
Reviewed-on: https://review.whamcloud.com/40914
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14186 lov: instantiate components layout for fallocate 85/40885/3
Wang Shilong [Mon, 7 Dec 2020 03:06:08 +0000 (11:06 +0800)]
LU-14186 lov: instantiate components layout for fallocate

fallocate() need send intent lock to MDS to instantiate layout
like PFL.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: Ie14574ca0981ea00aa1867489d2a40b75eb511cc
Reviewed-on: https://review.whamcloud.com/40885
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Tested-by: Maloo <maloo@whamcloud.com>
3 years agoLU-13783 libcfs: don't lookup symbol when not needed 25/40825/2
Mr NeilBrown [Mon, 19 Oct 2020 02:11:36 +0000 (13:11 +1100)]
LU-13783 libcfs: don't lookup symbol when not needed

Linux 5.8 and later do not export kallsyms_lookup_name() so we need to
remove dependence on it.
Looking up radix_tree_node_cachep is not needed in these kernels,
or any kernel were HAVE_XARRAY_SUPPORT is defined.
So avoid the lookup in those cases.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ia61fe9a025653985b7dcf67b1234d32667719956
Reviewed-on: https://review.whamcloud.com/40825
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Aurelien Degremont <degremoa@amazon.com>
3 years agoLU-13855 tests: remove the need for libiberty 02/40802/11
Andreas Dilger [Mon, 30 Nov 2020 20:35:46 +0000 (13:35 -0700)]
LU-13855 tests: remove the need for libiberty

The statx.c test program uses a bunch of GNU-specific xstr*
functions that have very little, if any, benefit to the code.
They allocate copies of static format strings, but those
strings are not modified, and are never freed.  Since the
use of xstr* drags in libiberty and associated build issues
on a variety of platforms, instead avoid those functions.

Remove the use of this library completely from the statx.c
test tool, with minimal replacements to keep the build happy.
Remove the corresponding configure checks.

Fixes: 3f7853b31ef6 ("LU-10934 llite: integrate statx() API with Lustre")
Test-Parameters: trivial testlist=sanityn env=ONLY=106 clientdistro=ubuntu2004
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I2c11bac8e2a4b4754ee78caabfa9cbdba98bffff
Reviewed-on: https://review.whamcloud.com/40802
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14151 gss: use skcipher when sync_skcipher not available 68/40768/9
Sebastien Buisson [Thu, 26 Nov 2020 16:22:49 +0000 (11:22 -0500)]
LU-14151 gss: use skcipher when sync_skcipher not available

Linux 4.2 introduced support for skcipher in commit
7a7ffe65c8c5 crypto: skcipher - Add top-level skcipher interface

Based on the existence of crypto_alloc_skcipher, use the skcipher API
instead of the blkcipher API, in case the sync_skcipher API is not
available.

Test-Parameters: clientdistro=el8.2 env=SHARED_KEY=true,SK_FLAVOR=skpi mdscount=2 mdtcount=4 osscount=1 ostcount=8 clientcount=2 testlist=sanity
Test-Parameters: clientdistro=el7.8 env=SHARED_KEY=true,SK_FLAVOR=skpi mdscount=2 mdtcount=4 osscount=1 ostcount=8 clientcount=2 testlist=sanity
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I4e62cd5c8219a215c5528f81d525c054215073e3
Reviewed-on: https://review.whamcloud.com/40768
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14143 lov: fix SEEK_HOLE calcs at component end 13/40713/5
Mikhail Pershin [Thu, 19 Nov 2020 13:24:18 +0000 (16:24 +0300)]
LU-14143 lov: fix SEEK_HOLE calcs at component end

If data ends exactly at component end then LOV assumed that
is not yet hole in file and the next component will take care.
Meanwhile there can be no next component initialized yet if file
ends exactly at component boundary, so no hole offset is returned
but error

Patch fixes that issue. If component reports hole offset at
component end then it is saved to be used as result when no
other components report valid hole offset.

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I8bffd8b76546b560a3d34034701eea4963f41593
Reviewed-on: https://review.whamcloud.com/40713
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
3 years agoLU-14181 tests: except sanity test_64e 64f with SHARED_KEY 65/40865/2
Sebastien Buisson [Fri, 4 Dec 2020 08:22:04 +0000 (09:22 +0100)]
LU-14181 tests: except sanity test_64e 64f with SHARED_KEY

Add sanity test_64e and test_64f to ALWAYS_EXCEPT when
SHARED_KEY is used.

Test-Parameters: trivial
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Iaa9f5038a59f9ddc50dd9ac81ca81effd8bb9b1b
Reviewed-on: https://review.whamcloud.com/40865
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14121 nodemap: do not force fsuid/fsgid squashing 45/40645/6
Sebastien Buisson [Fri, 13 Nov 2020 10:36:14 +0000 (19:36 +0900)]
LU-14121 nodemap: do not force fsuid/fsgid squashing

In the current implementation, if the real uid is squashed, then the
fsuid is similarly squashed, no matter what is the value of the
effective uid.
This squashing is a little bit too strict, and we should instead trust
mapped fsuid and fsgid values.

Also add euid_access test program and sanity-sec test_55 to verify
the issue is fixed.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Iecaecac5054b105cd42206b0a9a3868cde0269b4
Reviewed-on: https://review.whamcloud.com/40645
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14095 ssk: default rounds of Miller-Rabin for DH_check 86/40686/6
Sebastien Buisson [Tue, 17 Nov 2020 16:13:08 +0000 (17:13 +0100)]
LU-14095 ssk: default rounds of Miller-Rabin for DH_check

OpenSSL 1.1.1c increased the number of rounds used for Miller-Rabin
testing of the prime provided as input parameter to DH_check(). This
makes the check roughly x10 longer, and can lead to request timeouts
when an SSK flavor is being used.

Instead, use a dynamic number of rounds based on the speed of the
check, evaluated when the lsvcgssd daemon starts. If DH_check()
runtime is fine, just use it instead of our own check.

Test-Parameters: clientdistro=el8.2 serverdistro=el8.2 testgroup=review-dne-ssk
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Id392cdd76ede196094b146c68d230bc52852aa34
Reviewed-on: https://review.whamcloud.com/40686
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>