Whamcloud - gitweb
fs/lustre-release.git
3 years agoLU-11365 tests: test-framework::add() to load ldiskfs 40/33140/16
Alex Zhuravlev [Tue, 11 Sep 2018 17:54:43 +0000 (20:54 +0300)]
LU-11365 tests: test-framework::add() to load ldiskfs

when the tests run from in-tree build, then mkfs.lustre
can not rely on modprobe and ldiskfs needs to be loaded
explicitly. otherwise some tests like conf-sanity/55 fail.

Change-Id: Ic15f0eab8e483d5de9f63606433fbb26f754fd55
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/33140
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14214 ofd: fix locking in ofd_object_fallocate() 69/40969/2
Alex Zhuravlev [Tue, 15 Dec 2020 05:44:59 +0000 (08:44 +0300)]
LU-14214 ofd: fix locking in ofd_object_fallocate()

the rule is that locks are taken after transaction start.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I5252be0033611f03b355c1f98e21422cdde55c9e
Reviewed-on: https://review.whamcloud.com/40969
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Wang Shilong <wshilong@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-8066 osd-ldiskfs: quiet debug mount message 68/40968/3
Andreas Dilger [Mon, 14 Dec 2020 23:08:46 +0000 (16:08 -0700)]
LU-8066 osd-ldiskfs: quiet debug mount message

We don't need a message printed to the console for every
mount reporting that tunable parameters were configured.

Test-Parameters: trivial
Fixes: 493cd8088388 ("LU-8066 osd: migrate from proc to sysfs")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I12fb89f8f15a86657fe5c1f46359f184ce3ebbe5
Reviewed-on: https://review.whamcloud.com/40968
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14192 mdt: check for overstriping correctly 96/40896/3
Vitaly Fertman [Mon, 19 Oct 2020 20:46:15 +0000 (23:46 +0300)]
LU-14192 mdt: check for overstriping correctly

check for overstriping in the first component as well

HPE-bug-id: LUS-9430
Signed-off-by: Vitaly Fertman <c17818@cray.com>
Change-Id: Ice125b885681b0cf71dc55c5238c4c4d7d7bb970
Reviewed-on: https://es-gerrit.dev.cray.com/157999
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Tested-by: Elena Gryaznova <c17455@cray.com>
Reviewed-by: Alexander Zarochentsev <c17826@cray.com>
Reviewed-on: https://review.whamcloud.com/40896
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14191 lod: comp stripe count limit check 95/40895/2
Vitaly Fertman [Thu, 26 Nov 2020 18:37:43 +0000 (21:37 +0300)]
LU-14191 lod: comp stripe count limit check

when creating a PFL file of 2K stripes, they are shrinked to 700+,
because lod_get_stripe_count() checks the space left after taking
given 2K stripes into account, and the total amount of stripes that
fits into EA is 2700+.

when creating a SEL file of 2K stipes, it fails because the extension
component is also taking into account despite the fact it is never
initialised and due to the above logic 2x2k stripes leaves no space
in EA.

HPE-bug-id: LUS-9589
Signed-off-by: Vitaly Fertman <c17818@cray.com>
Change-Id: Ibeb063472d477e393646860010823fed5acb8382
Reviewed-by: Alexander Boyko <c17825@cray.com>
Reviewed-by: Artem Blagodarenko <c17828@cray.com>
Tested-by: Elena Gryaznova <c17455@cray.com>
Reviewed-on: https://review.whamcloud.com/40895
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-by: Artem Blagodarenko <artem.blagodarenko@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14158 mdc: process changelogs_catalog from the oldest rec 86/40786/2
Etienne AUJAMES [Fri, 27 Nov 2020 20:26:15 +0000 (21:26 +0100)]
LU-14158 mdc: process changelogs_catalog from the oldest rec

The chlg_load use the LLOG_CAT_FIRST to process changelogs. This
values will process record in the catalog always starting with index
0 to the newest record. So when catalog reach the end of indexes and
when records are saved at the begining of catalog, the
llog_cat_process will ignore records at the end.

This patch change the "startcat" value LLOG_CAT_FIRST to 0 to scan
the catalog from the oldest record to the newest.

Fixes: e2150028 (LU-12553  mdc: polling mode for changelog reader)
Signed-off-by: Etienne AUJAMES <eaujames@ddn.com>
Change-Id: Ib00a4373d87fd63bfb5d05731f1fdbd286c0b6f5
Reviewed-on: https://review.whamcloud.com/40786
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-4165 tests: restore skipped sanity-lfsck tests 60/40760/5
Andreas Dilger [Wed, 25 Nov 2020 22:50:04 +0000 (15:50 -0700)]
LU-4165 tests: restore skipped sanity-lfsck tests

Many subtests in sanity-lfsck were incorrectly being skipped due
to a cleanup patch that was intended to skip those tests for very
old Lustre versions being run on the MDS, or for ZFS MDTs.

Instead, a large number of tests were *always* being skipped.

Restore those accidentally-skipped tests.  Hopefully they still pass.

Clean up code style for the script:
- remove the needless "return" after "skip"
- use mds1_FSTYPE and MDS1_VERSION instead of getting values from MDS

Fixes: c54b6ca2bdb ("LU-13718 tests: add LU numbers to skipped tests")
Test-Parameters: trivial fstype=ldiskfs testlist=sanity-lfsck
Test-Parameters: fstype=ldiskfs mdscount=2 mdtcount=4 testlist=sanity-lfsck
Test-Parameters: fstype=zfs testlist=sanity-lfsck
Test-Parameters: fstype=zfs mdscount=2 mdtcount=4 testlist=sanity-lfsck
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I4e49d79826369a546576672f77f7012f4a3ebbe5
Reviewed-on: https://review.whamcloud.com/40760
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14075 kernel: kernel update RHEL8.2 [4.18.0-193.28.1.el8_2] 17/40417/5
Jian Yu [Tue, 15 Dec 2020 08:13:20 +0000 (00:13 -0800)]
LU-14075 kernel: kernel update RHEL8.2 [4.18.0-193.28.1.el8_2]

Update RHEL8.2 kernel to 4.18.0-193.28.1.el8_2.

Test-Parameters: trivial \
clientdistro=el8.2 serverdistro=el8.2 \
testlist=sanity

Change-Id: I34e1e51241c3090d1041dedef8379c2e212f58a5
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/40417
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-13745 pcc: fall back normal splice read for detached file 96/40396/4
Qian Yingjin [Mon, 26 Oct 2020 02:37:26 +0000 (10:37 +0800)]
LU-13745 pcc: fall back normal splice read for detached file

For the file that was once cached into PCC but detached from PCC
later, it should fall back into the normal I/O path to direct I/Os
to Lustre OSTs for splice read, not just return zero.

Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: Iff3449069a0a12ac56b58618e474749361c31bb7
Reviewed-on: https://review.whamcloud.com/40396
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14205 osd-ldiskfs: fix return of osd_extend_restart_trans() 89/40989/2
Wang Shilong [Wed, 16 Dec 2020 02:50:24 +0000 (10:50 +0800)]
LU-14205 osd-ldiskfs: fix return of osd_extend_restart_trans()

ldiskfs_journal_ensure_credits() could possibly return 1 if
transaction restarted, Lustre osd expect it return 0 or negative.

Fixes: cf48a2d23b ("LU-13839 kernel: RHEL 8.3 server support")
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I51ff0bd50575902b66529cbd71ff14710c202d03
Reviewed-on: https://review.whamcloud.com/40989
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-11848 misc: update e2fsprogs to 1.45.6.wc3 67/40967/2
Andreas Dilger [Mon, 14 Dec 2020 21:34:55 +0000 (14:34 -0700)]
LU-11848 misc: update e2fsprogs to 1.45.6.wc3

Update Changelog to reference new e2fsprogs release.

23547cbe57df LU-14126 e2fsck: update mmp block race
0b6718f5e22d LU-11848 filefrag: support PFL and FLR file
502f2e17bd5b LU-13797 e2fsck: quiet 'extent tree could be narrower'
94318cef3508 misc: replace remaining loff_t with ext2_loff_t

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I4b3859dfe3950f3b23e47f69ecf8b3ab053ebbe5
Reviewed-on: https://review.whamcloud.com/40967
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@whamcloud.com>
Reviewed-by: Peter Jones <pjones@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-11848 tests: handle 4-digit version numbers properly 97/40897/4
Andreas Dilger [Mon, 7 Dec 2020 20:08:45 +0000 (13:08 -0700)]
LU-11848 tests: handle 4-digit version numbers properly

Allow version_code() to handle and compare 4-digit version numbers
properly.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I8f57537f87bb1c087def277d0d6192def43ebbe5
Reviewed-on: https://review.whamcloud.com/40897
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14227 tests: Add missing space in sanity-lnet test 21 03/41003/2
Oleg Drokin [Wed, 16 Dec 2020 14:51:40 +0000 (09:51 -0500)]
LU-14227 tests: Add missing space in sanity-lnet test 21

otherwise echo"Add peer with nidrange (gni)" is not a valid command

Change-Id: Ibdff3f38338bb880dd4ae23e31ca2517f4626010
Test-Parameters: trivial testlist=sanity-lnet
Fixes: 6a8335be5c2b ("LU-12410 tests: Add gni tests to sanity-lnet")
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/41003
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
3 years agoLU-11848 lov: FIEMAP support for PFL and FLR file 66/40766/23
Bobi Jam [Thu, 26 Nov 2020 06:13:28 +0000 (14:13 +0800)]
LU-11848 lov: FIEMAP support for PFL and FLR file

* use the high 16 bits of fe_device to record the absolute stripe
  number from the beginning we are processing, so that continuous call
  can resume from the stripe specified by it.

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Ie519a9361b2d3ae4977ed2ccf925c92065aa9787
Reviewed-on: https://review.whamcloud.com/40766
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
3 years agoLU-13839 ldiskfs: fix detecting ext4_journal_ensure_credits 39/40939/4
Mr NeilBrown [Thu, 10 Dec 2020 21:41:34 +0000 (08:41 +1100)]
LU-13839 ldiskfs: fix detecting ext4_journal_ensure_credits

As __ext4_journal_ensure_credits() is not exported, the current
detection doesn't work.
So add a local definition so that the build succeeds.

Fixes: cf48a2d23bca ("LU-13839 kernel: RHEL 8.3 server support")
Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I5f4b0883e8425d26760ea77e89379103dce30855
Reviewed-on: https://review.whamcloud.com/40939
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
3 years agoLU-14185 osp: alloc fails for OSP obj xattr cache 71/40871/2
Vitaly Fertman [Fri, 4 Dec 2020 19:18:48 +0000 (22:18 +0300)]
LU-14185 osp: alloc fails for OSP obj xattr cache

osp_oac_xattr_find_or_add() may get ENOMEM as it tries to allocate
over len=buf->lb_len=XATTR_SIZE_MAX, thus order:5. Use large alloc
instead.

HPE-bug-id: LUS-9551
Signed-off-by: Vitaly Fertman <c17818@cray.com>
Change-Id: I98c08d16d0343cb60f6efcb63d20739103f2af1c
Reviewed-by: Sergey Cheremencev <c17829@cray.com>
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Tested-by: Alexander Lezhoev <c17454@cray.com>
Reviewed-on: https://review.whamcloud.com/40871
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
3 years agoLU-14080 gnilnd: updates for SUSE 15 SP2 26/40426/3
Shaun Tancheff [Tue, 27 Oct 2020 21:03:13 +0000 (16:03 -0500)]
LU-14080 gnilnd: updates for SUSE 15 SP2

gnilnd updates for SUSE 15 SP2

Use -Werror when checking for -Wno-stringop-truncation to
ensure the compile test is valid

Adjust for kernel dropping time_t, timeval, timespec
Adjust for kernel switching to timer_setup

Adjust for kernel dropping global_page_state(), use
nr_free_pages instead

Cleanup string format for stricter checking

Test-Parameters: trivial
HPE-bug-id: LUS-9453
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I5dcda1497fa5b1f2cf4a215517700f07374fcf7f
Reviewed-on: https://review.whamcloud.com/40426
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14161 obdclass: fix some problems with obd_nid_hash 46/40846/5
Mr NeilBrown [Thu, 3 Dec 2020 00:48:41 +0000 (11:48 +1100)]
LU-14161 obdclass: fix some problems with obd_nid_hash

There are a few of problems with the handling of the obd_nid_hash
rhl-table.

- obd_export_evict_by_nid() drops out of rcu_readlock() while holding
  a reference into the table.  This is theoretically unsafe as changed
  to the table might cause entries to be missed.
- nid_keycmp() ignores entries with exp_failed set.  This is a problem
  because rhltable_lookup() only compares the key for the *first*
  object in a list of objects with the same key - on a match it
  returns the whole list.
  So if exp_failed was set on the first in the list, all would become
  invisible.

These can be fixed by moving the test on ->exp_failed to after the
call to rhltabke_lookup(), and by using obd_nid_export_for_each() in
obd_export_evict_by_nid(), and repeatedly dealing with the first exp
returned.

Fixes: 580ef453d1d6 ("LU-8130 obd: convert obd_nid_hash to rhashtable")
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I65861c179e93c26cf39c5db8a58e3e4c9b962fe3
Reviewed-on: https://review.whamcloud.com/40846
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-9114 ldlm: don't compute sumsq for pool stats 35/39435/2
Andreas Dilger [Fri, 17 Jul 2020 10:08:08 +0000 (04:08 -0600)]
LU-9114 ldlm: don't compute sumsq for pool stats

Remove the calculation of sumsq from the LDLM pool stats, since
these stats are almost never used, while conversely the pools
are updated frequently.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I90612b613fb76f4500ca6cd5613f5c121b3ebbe5
Reviewed-on: https://review.whamcloud.com/39435
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14100 gnilnd: fix kgnilnd_setup_immediate_buffer 04/40504/4
Shaun Tancheff [Thu, 10 Dec 2020 10:15:54 +0000 (04:15 -0600)]
LU-14100 gnilnd: fix kgnilnd_setup_immediate_buffer

struct kvec * removed from kgnilnd_setup_immediate_buffer

Fixes: c87f41dd20 ("LU-13004 gnilnd: discard struct kvec arg.")
Test-Parameters: trivial
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: Ie3c511326af6ddb09b6bfa13bb779fc094ba30ec
Reviewed-on: https://review.whamcloud.com/40504
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-9215 build: Re-add the lustre-devel package 28/40728/5
James Simmons [Sun, 6 Dec 2020 02:03:22 +0000 (21:03 -0500)]
LU-9215 build: Re-add the lustre-devel package

Traditonally on rpm systems a special devel package exist that
contains userland headers and symlinks for the libraries. This
patch creates such a package.

The introduction of this new package changes the dependency
of lustre-test packages since the user land headers are tested
to make sure they compile for user land applications. Update
the dpk package as well.

Lastly we add pkg-config support to Lustre as well. This means
that if do configure --prefix=special_place it will not break
external applications if they use pkg-config.

Change-Id: I3e856d85239d1712e21f7ecc633d9fae569238e6
Test-Parameters: trivial
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/40728
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14205 osd-ldiskfs: return correct error after end io 44/40944/2
Wang Shilong [Fri, 11 Dec 2020 07:17:50 +0000 (15:17 +0800)]
LU-14205 osd-ldiskfs: return correct error after end io

@bi_status could be positive in newer kernel, we need use
blk_status_to_errno() helper to convert error properly.

Otherwise, a positive value could be returned to client
which might trigger unexpected error and make client
fail to resend.

Test-Parameters: clientdistro=el8.3 serverdistro=el8.3
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I2e8f257814eff311b72416788b7f6d9e37bad84a
Reviewed-on: https://review.whamcloud.com/40944
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
3 years agoLU-14149 llite: fiemap set flags for encrypted files 52/40852/5
Sebastien Buisson [Thu, 3 Dec 2020 14:43:29 +0000 (15:43 +0100)]
LU-14149 llite: fiemap set flags for encrypted files

FIEMAP ioctl needs to set
FIEMAP_EXTENT_DATA_ENCRYPTED|FIEMAP_EXTENT_ENCODED flags for all
extents of files encrypted by fscrypt.

Also add sanity-sec test_56 to exercise the flags, with just a
plain file layout for now.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I26f788b5c4b37e96de57334f48847b40886e14c2
Reviewed-on: https://review.whamcloud.com/40852
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
3 years agoLU-14208 utils: remove copy_file_range() usage from copytool 66/40966/2
John L. Hammond [Mon, 14 Dec 2020 18:36:32 +0000 (12:36 -0600)]
LU-14208 utils: remove copy_file_range() usage from copytool

Revert the lhsmtool_posix part of "LU-11621 utils: optimize
lhsmtool_posix with copy_file_range()". This is failing sanity-hsm
test_55() on RHEL 8.3.

Test-Parameters: clientdistro=el8.3 serverdistro=el8.3 testlist=sanity-hsm
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I0b410c2bb5f56a32c10f0ec39190735908cbb556
Reviewed-on: https://review.whamcloud.com/40966
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
3 years agoLU-10664 dom: non-blocking enqueue for DOM locks 03/36903/25
Mikhail Pershin [Sun, 1 Dec 2019 06:41:37 +0000 (09:41 +0300)]
LU-10664 dom: non-blocking enqueue for DOM locks

DOM lock enqueue waits for blocking locks on MDT due to
ATOMIC flag, so MDT thread is blocked until lock is granted.
When many clients attempt to write to shared file that may
cause server thread starvation and lock contention. Switch
to non-atomic lock enqueue for DOM locks.

- switch IO lock to non-intent enqueue, so it doesn't consume
  server thread for a long time being blocked
- on client take LVB from l_lvb_data updated by completion AST and
  update l_ost_lvb used by DoM
- make glimpse performing similarly on MDT and OST, it uses one
  format with no intent buffer and return data in LVB buffer
- introduce new connect flag 'dom_lvb' for compatibility reasons
- on server handle glimpse for both old and new clients by filling
  either LVB reply buffer or mdt_body buffer
- don't take RPC slot for a DOM enqueue like it is done for EXTENT
  locks, update ldlm_cli_enqueue_fini() to accept ldlm_enqueue_info
  as parameter
- check that there is no atomic local lock issued with mandatory DOM
  bit, trybits should be used

Test-Parameters: mdssizegb=20 testlist=dom-performance
Test-Parameters: testlist=sanity-dom clientversion=2.12
Test-Parameters: testlist=sanity-dom serverversion=2.12
Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I06573018126f0663a8f76dce07e126a1879cfba5
Reviewed-on: https://review.whamcloud.com/36903
Reviewed-by: Vitaly Fertman <vitaly.fertman@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14199 sec: find policy version in use for sepol 18/40918/2
Sebastien Buisson [Wed, 9 Dec 2020 12:34:05 +0000 (13:34 +0100)]
LU-14199 sec: find policy version in use for sepol

SELinux exports the maximum kernel policy version that can be used.
When building SELinux status checking representation 'sepol', we need
to look for all possible versions of the policy, not only the max one.

Test-Parameters: clientdistro=el8.3 serverdistro=el8.2 testgroup=review-dne-selinux
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Iae4b66403ce953e5a7c0df585900713c597ff033
Reviewed-on: https://review.whamcloud.com/40918
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
3 years agoLU-14095 gss: use RCU protection for sunrpc cache 14/40914/2
Sebastien Buisson [Tue, 8 Dec 2020 14:17:40 +0000 (15:17 +0100)]
LU-14095 gss: use RCU protection for sunrpc cache

Leverage sunrpc cache RCU protection for cache entry removal,
if the kernel supports it.

Test-Parameters: clientdistro=el8.2 serverdistro=el8.2 testgroup=review-dne-ssk
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I50f9dc9e9dc434b502df1e1013215b5506289ff9
Reviewed-on: https://review.whamcloud.com/40914
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14186 lov: instantiate components layout for fallocate 85/40885/3
Wang Shilong [Mon, 7 Dec 2020 03:06:08 +0000 (11:06 +0800)]
LU-14186 lov: instantiate components layout for fallocate

fallocate() need send intent lock to MDS to instantiate layout
like PFL.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: Ie14574ca0981ea00aa1867489d2a40b75eb511cc
Reviewed-on: https://review.whamcloud.com/40885
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Tested-by: Maloo <maloo@whamcloud.com>
3 years agoLU-13783 libcfs: don't lookup symbol when not needed 25/40825/2
Mr NeilBrown [Mon, 19 Oct 2020 02:11:36 +0000 (13:11 +1100)]
LU-13783 libcfs: don't lookup symbol when not needed

Linux 5.8 and later do not export kallsyms_lookup_name() so we need to
remove dependence on it.
Looking up radix_tree_node_cachep is not needed in these kernels,
or any kernel were HAVE_XARRAY_SUPPORT is defined.
So avoid the lookup in those cases.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ia61fe9a025653985b7dcf67b1234d32667719956
Reviewed-on: https://review.whamcloud.com/40825
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Aurelien Degremont <degremoa@amazon.com>
3 years agoLU-13855 tests: remove the need for libiberty 02/40802/11
Andreas Dilger [Mon, 30 Nov 2020 20:35:46 +0000 (13:35 -0700)]
LU-13855 tests: remove the need for libiberty

The statx.c test program uses a bunch of GNU-specific xstr*
functions that have very little, if any, benefit to the code.
They allocate copies of static format strings, but those
strings are not modified, and are never freed.  Since the
use of xstr* drags in libiberty and associated build issues
on a variety of platforms, instead avoid those functions.

Remove the use of this library completely from the statx.c
test tool, with minimal replacements to keep the build happy.
Remove the corresponding configure checks.

Fixes: 3f7853b31ef6 ("LU-10934 llite: integrate statx() API with Lustre")
Test-Parameters: trivial testlist=sanityn env=ONLY=106 clientdistro=ubuntu2004
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I2c11bac8e2a4b4754ee78caabfa9cbdba98bffff
Reviewed-on: https://review.whamcloud.com/40802
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14151 gss: use skcipher when sync_skcipher not available 68/40768/9
Sebastien Buisson [Thu, 26 Nov 2020 16:22:49 +0000 (11:22 -0500)]
LU-14151 gss: use skcipher when sync_skcipher not available

Linux 4.2 introduced support for skcipher in commit
7a7ffe65c8c5 crypto: skcipher - Add top-level skcipher interface

Based on the existence of crypto_alloc_skcipher, use the skcipher API
instead of the blkcipher API, in case the sync_skcipher API is not
available.

Test-Parameters: clientdistro=el8.2 env=SHARED_KEY=true,SK_FLAVOR=skpi mdscount=2 mdtcount=4 osscount=1 ostcount=8 clientcount=2 testlist=sanity
Test-Parameters: clientdistro=el7.8 env=SHARED_KEY=true,SK_FLAVOR=skpi mdscount=2 mdtcount=4 osscount=1 ostcount=8 clientcount=2 testlist=sanity
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I4e62cd5c8219a215c5528f81d525c054215073e3
Reviewed-on: https://review.whamcloud.com/40768
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14143 lov: fix SEEK_HOLE calcs at component end 13/40713/5
Mikhail Pershin [Thu, 19 Nov 2020 13:24:18 +0000 (16:24 +0300)]
LU-14143 lov: fix SEEK_HOLE calcs at component end

If data ends exactly at component end then LOV assumed that
is not yet hole in file and the next component will take care.
Meanwhile there can be no next component initialized yet if file
ends exactly at component boundary, so no hole offset is returned
but error

Patch fixes that issue. If component reports hole offset at
component end then it is saved to be used as result when no
other components report valid hole offset.

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I8bffd8b76546b560a3d34034701eea4963f41593
Reviewed-on: https://review.whamcloud.com/40713
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
3 years agoLU-14181 tests: except sanity test_64e 64f with SHARED_KEY 65/40865/2
Sebastien Buisson [Fri, 4 Dec 2020 08:22:04 +0000 (09:22 +0100)]
LU-14181 tests: except sanity test_64e 64f with SHARED_KEY

Add sanity test_64e and test_64f to ALWAYS_EXCEPT when
SHARED_KEY is used.

Test-Parameters: trivial
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Iaa9f5038a59f9ddc50dd9ac81ca81effd8bb9b1b
Reviewed-on: https://review.whamcloud.com/40865
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14121 nodemap: do not force fsuid/fsgid squashing 45/40645/6
Sebastien Buisson [Fri, 13 Nov 2020 10:36:14 +0000 (19:36 +0900)]
LU-14121 nodemap: do not force fsuid/fsgid squashing

In the current implementation, if the real uid is squashed, then the
fsuid is similarly squashed, no matter what is the value of the
effective uid.
This squashing is a little bit too strict, and we should instead trust
mapped fsuid and fsgid values.

Also add euid_access test program and sanity-sec test_55 to verify
the issue is fixed.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Iecaecac5054b105cd42206b0a9a3868cde0269b4
Reviewed-on: https://review.whamcloud.com/40645
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14095 ssk: default rounds of Miller-Rabin for DH_check 86/40686/6
Sebastien Buisson [Tue, 17 Nov 2020 16:13:08 +0000 (17:13 +0100)]
LU-14095 ssk: default rounds of Miller-Rabin for DH_check

OpenSSL 1.1.1c increased the number of rounds used for Miller-Rabin
testing of the prime provided as input parameter to DH_check(). This
makes the check roughly x10 longer, and can lead to request timeouts
when an SSK flavor is being used.

Instead, use a dynamic number of rounds based on the speed of the
check, evaluated when the lsvcgssd daemon starts. If DH_check()
runtime is fine, just use it instead of our own check.

Test-Parameters: clientdistro=el8.2 serverdistro=el8.2 testgroup=review-dne-ssk
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Id392cdd76ede196094b146c68d230bc52852aa34
Reviewed-on: https://review.whamcloud.com/40686
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-12901 o2iblnd: retry qp creation with reduced queue depth 48/40748/4
Serguei Smirnov [Tue, 24 Nov 2020 19:59:46 +0000 (14:59 -0500)]
LU-12901 o2iblnd: retry qp creation with reduced queue depth

If negotiated number of frags * queue depth is too large for
successful qp creation, reduce the queue depth in a loop
until qp creation succeeds or the queue depth dips below 2.
Remember the reduced queue depth value to use for later
connections to the same peer.

Test-Parameters: trivial
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: Iaa91510d6f80d813218a06a9bc52f5f9251e8b87
Reviewed-on: https://review.whamcloud.com/40748
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoNew tag 2.13.57 2.13.57 v2_13_57
Oleg Drokin [Wed, 9 Dec 2020 20:29:02 +0000 (15:29 -0500)]
New tag 2.13.57

Change-Id: I931c35e61fe13760586d3402afa729bce029384e
Signed-off-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14042 llite: don't check layout info for page discard 67/40267/3
Bobi Jam [Fri, 16 Oct 2020 11:44:25 +0000 (19:44 +0800)]
LU-14042 llite: don't check layout info for page discard

The CIT_MISC+ignore_layout is indicating locks/pages manipulation
from the OSC layer, it does not care/access lov layout related info.

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Ibdf22f0d47712d370385af4c6c984052b91fc7c6
Reviewed-on: https://review.whamcloud.com/40267
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
3 years agoLU-13839 kernel: RHEL 8.3 server support 29/40729/7
Jian Yu [Mon, 7 Dec 2020 22:31:56 +0000 (14:31 -0800)]
LU-13839 kernel: RHEL 8.3 server support

This patch makes changes to support RHEL 8.3 release with
kernel 4.18.0-240.1.1.el8_3 for Lustre server.

Test-Parameters: trivial fstype=ldiskfs \
clientdistro=el8.3 serverdistro=el8.3 testlist=sanity

Test-Parameters: trivial fstype=zfs \
clientdistro=el8.3 serverdistro=el8.3 testlist=sanity

Change-Id: I2aa2efaf86b5f244fcb1dd95be1d0bfe04e8d245
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/40729
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14170 build: Update ZFS version to 2.0.0 22/40822/4
Jian Yu [Wed, 2 Dec 2020 00:07:09 +0000 (16:07 -0800)]
LU-14170 build: Update ZFS version to 2.0.0

Major New Features
* Sequential resilver
* Persistent L2ARC
* ZStandard compression
* Redacted zfs send/receive

Notable Changes
* Added fallocate(mode-0/2) compatibility to preallocate space.
* Reorganized the zfs and zpool man pages by splitting out each
  subcommand in to its own page.
* Enabled the systemd zfs-mount-generator by default on Linux.
* More relevant and useful ZED syslog entries.
* Provided pam module for automatically loading zfs encryption
  keys for home datasets.
* Support for inheriting and setting user properties in channel
  programs.
* Improved bootloader support.
* Optionally colorized zpool status output.

Change-Id: Ia00d46a4cb54be94d4fdc0145a2d474ea987cf74
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/40822
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-13826 utils: fix compatibility for LL_IOC_MDC_GETINFO 58/40858/4
Qian Yingjin [Fri, 4 Dec 2020 02:44:25 +0000 (10:44 +0800)]
LU-13826 utils: fix compatibility for LL_IOC_MDC_GETINFO

The landed patch "LU-11367 som: integrate LSOM with lfs find"
uses "LL_IOC_MDC_GETINFO_OLD", so while the IOCTL number/structs
are ABI compatible, it is not API compatible and applications
using for the header including the definition LL_IOC_MDC_GETINFO
is broken.

This patch defines versioned IOCTL number: LL_IOC_MDC_GETINFO_V1,
LL_IOC_MDC_GETINFO_V2. Then we can use the explicitly verioned
constrants everywhere for the in-tree code, and declare
LL_IOC_MDC_GETINFO in a compatible way, but external applications
can select the version that they want explicitly.

And this patch does the same fix for IOC_MDC_GETFILEINFO.

Fixes: 11aa7f8704c4 ("LU-11367 som: integrate LSOM with lfs find")
Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I39e0273c4a7331ec782de3e8686b8a756e56beec
Reviewed-on: https://review.whamcloud.com/40858
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
3 years agoLU-13994 ldlm: osc_object_ast_clear() is called for mdc object on eviction 52/40052/4
Andriy Skulysh [Sun, 29 Mar 2020 13:15:29 +0000 (16:15 +0300)]
LU-13994 ldlm: osc_object_ast_clear() is called for mdc object on eviction

Replace osc_object_prune() with cl_object_prune()

PID: 3477   TASK: ffff9360d82fa0e0  CPU: 0   COMMAND: "ll_imp_inval"
 #0 [ffff9360d5c5b990] machine_kexec at ffffffff86865704
 #1 [ffff9360d5c5b9f0] __crash_kexec at ffffffff869209a2
 #2 [ffff9360d5c5bac0] panic at ffffffff86f7294c
 #3 [ffff9360d5c5bb40] lbug_with_loc at ffffffffc04b78cb [libcfs]
 #4 [ffff9360d5c5bb60] osc_object_ast_clear at ffffffffc0956471 [osc]
 #5 [ffff9360d5c5bbc8] ldlm_resource_foreach at ffffffffc07e2fd6 [ptlrpc]
 #6 [ffff9360d5c5bc08] ldlm_resource_iterate at ffffffffc07e3266 [ptlrpc]
 #7 [ffff9360d5c5bc38] osc_object_prune at ffffffffc0956140 [osc]
 #8 [ffff9360d5c5bc58] osc_object_invalidate at ffffffffc0956e12 [osc]
 #9 [ffff9360d5c5bcd0] osc_ldlm_resource_invalidate at ffffffffc09477bf [osc]

Change-Id: I83fd02feb3f5559dfe7b93647429f8eb57c7ad0e
HPE-bug-id: LUS-8399
Signed-off-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Alexander Boyko <c17825@cray.com>
Reviewed-by: Vitaly Fertman <c17818@cray.com>
Tested-by: Alexander Lezhoev <c17454@cray.com>
Reviewed-on: https://review.whamcloud.com/40052
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Vitaly Fertman <vitaly.fertman@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-10262 mdt: mdt_reint_open: check EEXIST without lock 98/33098/17
Dominique Martinet [Fri, 31 Aug 2018 09:03:36 +0000 (18:03 +0900)]
LU-10262 mdt: mdt_reint_open: check EEXIST without lock

Many applications blindly open files with O_CREAT, and the mds gets a
write lock to the parent directory for these even if the file already
exists.
Checking for file existence first lets us take a PR lock if file
already existed even if O_CREAT was specified.

This opens up multiple races between the first lookup and the actual
locking, in each of them drop the resources we aquired and retry from
scratch to keep things as far from complicated as possible, with mixed
success.

Update (eaujames):
 - rebase the patch
 - update tests

Performance tests results:

The array below presents the average "open" syscall latency for 20
files in a single directory accessed by 400 different clients.
 _______________________________________________________________
| Test cases        | without patch | with patch | %improvement |
|___________________|_______________|____________|______________|
| readonly          | 0.960s        | 0.973s     | -1.40%       |
|___________________|_______________|____________|______________|
| readonly cached   | 0.372s        | 0.372s     | +0.01%       |
|___________________|_______________|____________|______________|
| O_CREAT+precreate | 1.645s        | 0.968s     | +41.13%      |
|___________________|_______________|____________|______________|
| O_CREAT cached    | 0.632s        | 0.623s     | +1.34%       |
|___________________|_______________|____________|______________|
| O_CREAT           | 1.261s        | 1.093s     | +13.32%      |
|___________________|_______________|____________|______________|
(for more detail, see the ticket comments section)

This patch optimizes concurent opens with O_CREAT flag when dentry are
not cached by clients.

Change-Id: I247b579d14d20036f89033c99ece457d70ba19e7
Signed-off-by: Dominique Martinet <dominique.martinet@cea.fr>
Reviewed-on: https://review.whamcloud.com/33098
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Etienne AUJAMES <eaujames@ddn.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14172 lmv: optimize dir shard revalidate 63/40863/3
Lai Siyao [Thu, 3 Dec 2020 21:07:01 +0000 (05:07 +0800)]
LU-14172 lmv: optimize dir shard revalidate

mdt_is_remote_object() will check whether child is directory shard
if parent and child are on different MDTs, which needs to read LMV
from disk, and hurt striped directory stat performance.

This can be optimized, client can just set CROSS_REF flag to do a
cross reference getattr, which avoids lots of checks.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Ib2d5a510b27c90a26f979f9cccfd40948e32d91a
Reviewed-on: https://review.whamcloud.com/40863
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
3 years agoLU-14155 mdc: avoid easize set to 0 85/40785/2
Yang Sheng [Fri, 27 Nov 2020 14:57:51 +0000 (22:57 +0800)]
LU-14155 mdc: avoid easize set to 0

The cl_default_mds_easize could be set to 0 in some case. Then
check it before package.

Fixes: 3ec712bd183a ("LU-11868 osd: Set max ea size to XATTR_SIZE_MAX")
Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: Icc045bfba345a61dd7904bfc7772930fd737334e
Reviewed-on: https://review.whamcloud.com/40785
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
3 years agoLU-5855 tests: enable skipped sanity-lfsck DNE ZFS tests 61/40761/4
Andreas Dilger [Wed, 25 Nov 2020 23:20:19 +0000 (16:20 -0700)]
LU-5855 tests: enable skipped sanity-lfsck DNE ZFS tests

LFSCK has been enabled for DNE striped directories on ZFS for
some time, but the skipped tests have never been enabled for
this configuration.  Re-enable them for ZFS backends.

Fixes: 26995a3588e7 ("LU-5855 lfsck: misc fixes for zfs-based backend")
Test-Parameters: trivial fstype=zfs testlist=sanity-lfsck
Test-Parameters: fstype=zfs mdscount=2 mdtcount=4 testlist=sanity-lfsck
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I33be173c98faa61311ab2b435555cf49e1e2bacc
Reviewed-on: https://review.whamcloud.com/40761
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14073 ptlrpc: remove debugging assert using segment_eq() 97/40797/4
Aurelien Degremont [Fri, 13 Nov 2020 16:17:37 +0000 (16:17 +0000)]
LU-14073 ptlrpc: remove debugging assert using segment_eq()

In Linux 5.9-rc1 commit 428e2976a5, segment_eq() is removed as it was
only used in uaccess_kernel(). But uaccess_kernel() was introduced in Linux
v4.12-rc1, which does not cover all supported kernel versions.

As segment_eq() is only used in 1 actual debugging call, avoid adding a
new configuration test for uaccess_kernel() and just remove the 2
related LASSERTs.

Test-Parameters: trivial
Signed-off-by: Aurelien Degremont <degremoa@amazon.com>
Change-Id: I1595e9b085176d2106a2b958e1fd6f4031cc2f69
Reviewed-on: https://review.whamcloud.com/40797
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Neil Brown <neilb@suse.de>
3 years agoLU-14159 build: fix gcc8 warnings on kthread_run calls 90/40790/2
Sebastien Piechurski [Fri, 15 May 2020 18:22:44 +0000 (20:22 +0200)]
LU-14159 build: fix gcc8 warnings on kthread_run calls

Fixes warnings from gcc8 about kthread_run calls with no format
parameter.

Signed-off-by: Quentin Bouget <quentin.bouget@cea.fr>
Signed-off-by: Sebastien Piechurski <sebastien.piechurski@atos.net>
Change-Id: I30a68a1d0e318c98f2da547ece8018564ced69c0
Reviewed-on: https://review.whamcloud.com/40790
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
3 years agoLU-14154 ldiskfs: gcc10 support 76/40776/2
Alex Zhuravlev [Fri, 27 Nov 2020 07:11:27 +0000 (10:11 +0300)]
LU-14154 ldiskfs: gcc10 support

gcc10 complains about uninitialized fields:
ldiskfs/extents.c: In function â€˜ldiskfs_ext_map_blocks’:
ldiskfs/ldiskfs_extents.h:214:12: error: 'zero_ex2.ee_start_hi' may
be used uninitialized in this function [-Werror=maybe-uninitialized]

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I17ef56352194f6ad76bf6f08dab0be3e28d0323c
Reviewed-on: https://review.whamcloud.com/40776
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
3 years agoLU-14043 lfs: use buffered IO in mirror extend 75/40275/4
John L. Hammond [Fri, 16 Oct 2020 18:37:36 +0000 (13:37 -0500)]
LU-14043 lfs: use buffered IO in mirror extend

In lfs_mirror_extend() use buffered IO to read from the source
file. Mirror resync will continue to use direct IO on the source.
Call posix_fadvise() at the end of mirroring to discard cached data
from the old and new mirrors.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I7703ae9e30cf30a7e6fbf274114d5986a80d4d3a
Reviewed-on: https://review.whamcloud.com/40275
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Patrick Farrell <farr0186@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-13569 lnet: Add lnet_recovery_limit to lnetctl 17/39717/14
Chris Horn [Fri, 21 Aug 2020 19:27:07 +0000 (14:27 -0500)]
LU-13569 lnet: Add lnet_recovery_limit to lnetctl

Allow setting/reading lnet_recovery_limit via lnetctl.

Test-Parameters: trivial
HPE-bug-id: LUS-9109
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I5aac297bad15e43a52d8b8531da08a1d3f559bea
Reviewed-on: https://review.whamcloud.com/39717
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-13569 lnet: Introduce lnet_recovery_limit parameter 16/39716/12
Chris Horn [Fri, 21 Aug 2020 18:33:12 +0000 (13:33 -0500)]
LU-13569 lnet: Introduce lnet_recovery_limit parameter

This parameter controls how long LNet will attempt to recover an
unhealthy interface.

Defaults to 0 to indicate indefinite recovery. This maintains the
current behavior.

Test-Parameters: trivial
HPE-bug-id: LUS-9109
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I2f7897d9a293f0979f7402de2b91e160c77790d1
Reviewed-on: https://review.whamcloud.com/39716
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-13669 llite: try to improve mmap performance 16/38916/10
Wang Shilong [Sun, 28 Jun 2020 08:35:07 +0000 (16:35 +0800)]
LU-13669 llite: try to improve mmap performance

We have observed slow mmap read performances for some
applications. The problem is if access pattern is neither
sequential nor stride, but could be still adjacent in a
small range and then seek a random position.

So the pattern could be something like this:

[1M data] [hole] [0.5M data] [hole] [0.7M data] [1M data]

Every time an application reads mmap data, it may not only
read a single 4KB page, but aslo a cluster of nearby pages in
a range(e.g. 1MB) of the first page after a cache miss.

The readahead engine is modified to track the range size of
a cluster of mmap reads, so that after a seek and/or cache miss,
the range size is used to efficiently prefetch multiple pages
in a single RPC rather than many small RPCs.

Benchmark:
fio --name=randread --directory=/ai400/fio --rw=randread
--ioengine=mmap --bs=128K --numjobs=32 --filesize=200G
--filename=randread --time_based --status-interval=10s
--runtime=30s --allow_file_create=1 --group_reporting
--disable_lat=1 --disable_clat=1 --disable_slat=1
--disk_util=0 --aux-path=/tmp --randrepeat=0
--unique_filename=0 --fallocate=0

               |   master  |   patched  |  speedup  |
---------------+-----------+------------+-----------+
page_fault_avg |   512usec |    52usec  |  9.75x
page_fault_max |  37698usec|    6543usec|  5.76x

Change-Id: I86436cbd027ec6df5094599e54b4acfd0e018930
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/38916
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-13368 lnet: discard the callback 45/38845/8
Yang Sheng [Mon, 4 May 2020 10:57:33 +0000 (18:57 +0800)]
LU-13368 lnet: discard the callback

Lustre need a completion callback for event that request
has been sent. And then need other callback when reply
arrived. Sometime the request completion callback maybe
lost by some reason even reply has been received.
system will wait forever even timeout. We needn't to wait
request completion in such case. So provide a way to
discard the callback.

Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: If9cd8420ee76947ee5053180e0f5219f76bb94c2
Reviewed-on: https://review.whamcloud.com/38845
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-13170 osd: positive lookup shouldn't return an error 22/37322/8
Alex Zhuravlev [Fri, 24 Jan 2020 12:12:54 +0000 (15:12 +0300)]
LU-13170 osd: positive lookup shouldn't return an error

because of stale OI, otherwise LFSCK has no chance to
fix direntry. lookup should initiate OI scrubber and
give it a hint about FID->dnode mapping.

This should fix sanity-lfsck/23b failing with ZFS often.

Test-Parameters: fstype=zfs testlist=sanity-lfsck env=ONLY=23b,ONLY_REPEAT=200
Change-Id: Iea826de9995e34939eff2d67a831deb6b192bd4d
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/37322
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
3 years agoLU-9555 quota: df should return projid-specific values 85/36685/17
Wang Shilong [Wed, 6 Nov 2019 13:01:18 +0000 (21:01 +0800)]
LU-9555 quota: df should return projid-specific values

With local ext4 and XFS filesystems, it is possible to use
"df /path/to/directory" (statfs()) to return the current
project quota usage for that directory as "used", and
min(projid quota limit, free space) as "total".

statfs() is a natural interface for users/applications, since
it represents the used/maximum space for that subdirectory.
Otherwise, the user will get EDQUOT back when the project
quota runs out for that directory and applications will not
be able to figure out how much data they could write into
that directory.

Change-Id: I7f357e6e4de6a1465a48c2c08c4aa86c4c848dbb
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/36685
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
3 years agoLU-13543 lustre: update *pos in seq_file .next functions 57/40757/4
Mikhail Pershin [Wed, 25 Nov 2020 19:45:38 +0000 (22:45 +0300)]
LU-13543 lustre: update *pos in seq_file .next functions

Patch fixes mdt_hsm_actions_debugfs seq operations to work with
new seq file logic in newer kernels

Meanwhile it finishes remaining work originating in LU-13985 which
updates ->next functions to always increment *pos as it is expected
by kernel

Test-Parameters: clientdistro=el8.2 serverdistro=el8.2 testlist=sanity-hsm
Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I4f077b633fbf7d420bfda10cce6b0141cef07761
Reviewed-on: https://review.whamcloud.com/40757
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Neil Brown <neilb@suse.de>
3 years agoLU-13980 osd-ldiskfs: print label instead of device 38/40738/4
Andreas Dilger [Tue, 24 Nov 2020 07:35:17 +0000 (00:35 -0700)]
LU-13980 osd-ldiskfs: print label instead of device

Print out the volume label (e.g. fsname-OST0000) in error messages
instead of the device name (e.g. dm-2) so that it is easier to
correlate this with other messages in the log.

Fix error strings to be on one line to match code style.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Iff31bc962ca32b1f3a13f579700d03835a3ebbe5
Reviewed-on: https://review.whamcloud.com/40738
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14140 osd: don't panic meeting OI dups 66/40666/5
Alex Zhuravlev [Tue, 24 Nov 2020 09:34:00 +0000 (12:34 +0300)]
LU-14140 osd: don't panic meeting OI dups

instead dump all info (FID->ino, LMAs) and return an error

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I4c0c7ad12cc808617c748b0a720323a968668fc9
Reviewed-on: https://review.whamcloud.com/40666
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14134 osd-ldiskfs: reduce credits for new writing 26/40626/12
Wang Shilong [Thu, 12 Nov 2020 09:38:15 +0000 (17:38 +0800)]
LU-14134 osd-ldiskfs: reduce credits for new writing

For new writing, Lustre reserve credits in worst cases,
eg for 1M write, we need reserve more than 500 credits.

This means we need much more journal space than data
blocks for each new writing thus a large journal needed
to reach good performance.

This patch try to improve the case by allowing transaction
to be restarted, only reserve few extent credits during
declare phase. Before every block allocation, one extent
insertion credits must be guaranteed, otherwise transaction
will be restarted.

A simple dd test of following in 10G device shows credits
removed from 260+ to 7 credits:

 $ dd if=/dev/zero of=data bs=1M count=1 oflag=direct

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: Ide541cbd72d8d4aa286eb8b84ca5ce09c3b5f8b6
Reviewed-on: https://review.whamcloud.com/40626
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14130 osc: skip 0 row for rpc_stats 13/40613/2
Yang Sheng [Wed, 11 Nov 2020 20:02:55 +0000 (04:02 +0800)]
LU-14130 osc: skip 0 row for rpc_stats

Fix the rpc_stats statistic it should not print
0 row as it makes nosense.

Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: Id3534bb43b5453daeed4db078c18912c4cc2f51e
Reviewed-on: https://review.whamcloud.com/40613
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
3 years agoLU-14105 lfsck: don't LBUG() on disk data 24/40524/2
Lai Siyao [Tue, 3 Nov 2020 02:17:36 +0000 (10:17 +0800)]
LU-14105 lfsck: don't LBUG() on disk data

lfsck_namespace_assitant_handler_p1() will LBUG() if file FID is
zero, and file name is not "..", but generally lfsck shouldn't LBUG()
on disk data. The correct way is to treat such file as bad dirent and
remove it, which is done below.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Ia4092ada5d55583508f276e5c9d59920ec31a33b
Reviewed-on: https://review.whamcloud.com/40524
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14043 llite: allow DIO with unaligned IO count 92/40392/3
Wang Shilong [Mon, 2 Nov 2020 06:46:42 +0000 (14:46 +0800)]
LU-14043 llite: allow DIO with unaligned IO count

DIO only require user buffer page aligned and
IO offset page aligned, it is ok that io count is
not page aligned, remove this unnecessary limit
so that we could use DIO with file not aligned
with PAGE SIZE.

Change-Id: Iaf1e224b3f180f91d1b54016c6e6b7adaf674bf8
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/40392
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-13474 gss: do not return -ERESTART when gss rpc times out 10/38310/17
Sebastien Buisson [Mon, 27 Apr 2020 11:44:53 +0000 (13:44 +0200)]
LU-13474 gss: do not return -ERESTART when gss rpc times out

When a client receives an RPC time out while it tries to negotiate
a GSS context with the servers, it must not return -ERESTART.
Otherwise, the client would keep on sending GSS negotiation requests
to the same peer, instead of letting the HA failover mechanism try
different service nodes.

Test-Parameters: trivial
Test-Parameters: env=SHARED_KEY=true mdscount=2 mdtcount=4 osscount=1 ostcount=8 clientcount=2 testlist=sanity,recovery-small,sanity-sec
Test-Parameters: env=SHARED_KEY=true mdscount=2 mdtcount=4 osscount=1 ostcount=8 clientcount=2 testlist=sanity,recovery-small,sanity-sec
Test-Parameters: env=SHARED_KEY=true mdscount=2 mdtcount=4 osscount=1 ostcount=8 clientcount=2 testlist=sanity,recovery-small,sanity-sec
Test-Parameters: env=SHARED_KEY=true mdscount=2 mdtcount=4 osscount=1 ostcount=8 clientcount=2 testlist=sanity,recovery-small,sanity-sec
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ifc9cc02ea3ddefe6c5095cbb464a054c9027c920
Reviewed-on: https://review.whamcloud.com/38310
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-12353 ldiskfs: speedup quota journalling 92/34992/3
Andrew Perepechko [Wed, 11 Jul 2018 13:32:33 +0000 (16:32 +0300)]
LU-12353 ldiskfs: speedup quota journalling

ldiskfs_mark_dquot_dirty() need not call
ldiskfs_write_dquot() if the dquot is already
dirty since we are guaranteed that another
thread is already writing the dquot and
will update the buffer with the latest
dquot data.

Change-Id: Ia65c5987bbb0d6d2ac94c5499cb8b51edc3c49c3
Signed-off-by: Andrew Perepechko <c17827@cray.com>
Cray-bug-id: LUS-5342
Reviewed-on: https://review.whamcloud.com/34992
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
3 years agoLU-14095 gss: use hlist_unhashed() instead of ->next 14/40514/4
Sebastien Buisson [Mon, 2 Nov 2020 14:27:00 +0000 (23:27 +0900)]
LU-14095 gss: use hlist_unhashed() instead of ->next

In cache_detail list-mutation primitives, verifying the status of an
entry must be done using hlist_unhashed(), in case 'struct cache_head'
has a 'cache_list' field.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I1410eca9a647b74127cf40b8f3d6b68d055f773a
Reviewed-on: https://review.whamcloud.com/40514
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
3 years agoLU-12960 lod: don't set index for 2nd stripe if specific 35/36735/4
Lai Siyao [Sat, 7 Sep 2019 09:38:46 +0000 (17:38 +0800)]
LU-12960 lod: don't set index for 2nd stripe if specific

When MDTs are specific, don't set index for the second stripe
allocation, otherwise it's not created on specific MDT.

Add sanity 31q.

Fixes: c1d0a355a6 ("LU-12624 lod: alloc dir stripes by QoS")
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I415be0fa7c6e702ec72dcaac88bba55290463d44
Reviewed-on: https://review.whamcloud.com/36735
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-13571 lnd: Use NETWORK_TIMEOUT for some conn failures 00/39900/15
Chris Horn [Fri, 11 Sep 2020 19:13:32 +0000 (14:13 -0500)]
LU-13571 lnd: Use NETWORK_TIMEOUT for some conn failures

For -EHOSTUNREACH and -ETIMEDOUT we cannot tell whether the
connnection failure was due to a problem with the remote or local NI,
so we should return the LNET_MSG_STATUS_NETWORK_TIMEOUT to LNet in
these cases.

HPE-bug-id: LUS-9342
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I9036f5698061ce74fe29683b8249b8f9a05f3433
Reviewed-on: https://review.whamcloud.com/39900
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-13571 lnd: Use NETWORK_TIMEOUT for txs on ibp_tx_queue 99/39899/15
Chris Horn [Fri, 11 Sep 2020 18:42:42 +0000 (13:42 -0500)]
LU-13571 lnd: Use NETWORK_TIMEOUT for txs on ibp_tx_queue

TXs on the ibp_tx_queue are waiting for a connection to be
established. Failure to establish a connection could be due to a
problem with either the local NI or the remote NI, and o2iblnd cannot
currently distinguish between these failures. As such, it should
return LNET_MSG_STATUS_NETWORK_TIMEOUT to LNet so that LNet will
decrement the health value of both the local NI and the remote NI and
future sends can take these health values into account.

Test-Parameters: trivial
HPE-bug-id: LUS-9342
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: Idbbbe95483d25ec48b83e33a00685f72fa5292e6
Reviewed-on: https://review.whamcloud.com/39899
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-12564 libcfs: Use vfree_atomic instead of vfree 36/40136/3
Oleg Drokin [Thu, 1 Oct 2020 05:51:55 +0000 (01:51 -0400)]
LU-12564 libcfs: Use vfree_atomic instead of vfree

Since vfree is unsafe to use in atomic context, implement our own
libcfs_vfree_atomic heavily based on code from linux 4.10 commit
bf22e37a641327e34681b7b6959d9646e3886770

We can't use the one in the kernel because it's not exported.

unconditionally use it in *_FREE_LARGE() macros since in_atomic()
is not recommended to be used outside of core kernel code.

Not everything is present on 3.10 (rhel7) so we also add
llist primitive and a replacement for raw_cpu_ptr there.

Change-Id: I50892f231e54a284f4d8a14d910ea9ab2fbe6a16
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/40136
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Aurelien Degremont <degremoa@amazon.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Tested-by: Maloo <maloo@whamcloud.com>
3 years agoLU-14024 ofd: Avoid use after free in ofd_inconsistency_verification_main 22/40222/2
Oleg Drokin [Mon, 12 Oct 2020 20:12:55 +0000 (16:12 -0400)]
LU-14024 ofd: Avoid use after free in ofd_inconsistency_verification_main

The ofd_inconsistency_lock should not be unlocked after we woken up
a different thread that is going to free the structure containing
said lock.

Change-Id: I913e7470664e1128a250597b0a803f791d99099e
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/40222
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
3 years agoLU-14072 llite: fix client evicition with DIO 89/40389/3
Wang Shilong [Sat, 24 Oct 2020 01:47:23 +0000 (09:47 +0800)]
LU-14072 llite: fix client evicition with DIO

We set lockless in file open if O_DIRECT flag is passed,
however O_DIRECT flag could be cleared by
fcntl(..., F_SETFL, ...).

Finally we comes to a case where buffer IO without lock
held properly, and hit hang:

[<ffffffffc0d421ed>] osc_extent_wait+0x21d/0x7c0 [osc]
[<ffffffffc0d44897>] osc_cache_wait_range+0x2e7/0x940 [osc]
[<ffffffffc0d4585e>] osc_cache_writeback_range+0x96e/0xff0 [osc]
[<ffffffffc0d31c45>] osc_lock_flush+0x195/0x290 [osc]
[<ffffffffc0d31d7c>] osc_lock_lockless_cancel+0x3c/0xe0 [osc]
[<ffffffffc081f488>] cl_lock_cancel+0x78/0x160 [obdclass]
[<ffffffffc0cd8079>] lov_lock_cancel+0x99/0x190 [lov]
[<ffffffffc081f488>] cl_lock_cancel+0x78/0x160 [obdclass]
[<ffffffffc081f9a2>] cl_lock_release+0x52/0x140 [obdclass]
[<ffffffffc08238a9>] cl_io_unlock+0x139/0x290 [obdclass]
[<ffffffffc08242e8>] cl_io_loop+0xb8/0x200 [obdclass]
[<ffffffffc0e1d36b>] ll_file_io_generic+0x91b/0xdf0 [lustre]
[<ffffffffc0e1dd0c>] ll_file_aio_write+0x29c/0x6e0 [lustre]
[<ffffffffc0e1e250>] ll_file_write+0x100/0x1c0 [lustre]
[<ffffffffa984aa90>] vfs_write+0xc0/0x1f0
[<ffffffffa984b8af>] SyS_write+0x7f/0xf0
[<ffffffffa9d8eede>] system_call_fastpath+0x25/0x2a
[<ffffffffffffffff>] 0xffffffffffffffff

Lock cancel time out in the server side and client
eviction happen.

Fix this problem by testing O_DIRECT flag to decide if
we could issue lockless IO.

Fixes: 6bce536725 ("LU-4198 clio: turn on lockless for some kind of IO")
Change-Id: Idbf1c748684a6540aee5f6e35c017929fbcc60b9
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/40389
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Gu Zheng <gzheng@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14071 doc: lfs setquota/quota doc for PQ 84/40384/3
Sergey Cheremencev [Fri, 23 Oct 2020 13:43:07 +0000 (16:43 +0300)]
LU-14071 doc: lfs setquota/quota doc for PQ

Add description of key --pool for lfs setquota
and lfs quota.

Change-Id: Ie6bfb3240f7eb8b4239d95aac29989103f009578
Test-Parameters: trivial
HPE-bug-id: LUS-9341
Signed-off-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-on: https://review.whamcloud.com/40384
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
3 years agoLU-14036 build: fix lbuild for MOFED 5.1 54/40254/7
Minh Diep [Wed, 14 Oct 2020 23:57:51 +0000 (16:57 -0700)]
LU-14036 build: fix lbuild for MOFED 5.1

Starting MOFED 5.1, rdma-core is required for libib*mad

Test-Parameters: trivial

Change-Id: Id26f3cdb0552933577e1b27384ac82f9f48e2b3a
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/40254
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
3 years agoLU-14054 utils: add --exit-on-close to ofd_access_log_reader 12/40312/6
John L. Hammond [Wed, 30 Sep 2020 19:37:26 +0000 (14:37 -0500)]
LU-14054 utils: add --exit-on-close to ofd_access_log_reader

Add an --exit-on-close (-e), option to ofd_access_log_reader.

Test-Parameters: trivial
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Ifded63318026b0ad3f9f077dc74008469df877d9
Reviewed-on: https://review.whamcloud.com/40312
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
3 years agoLU-13153 lib: fix llapi_get_version_string 80/40680/3
Mr NeilBrown [Wed, 18 Nov 2020 00:52:15 +0000 (11:52 +1100)]
LU-13153 lib: fix llapi_get_version_string

llapi_get_version_string() has always been broken as it passes args to
get_lustre_param_value in the wrong order.

Test-Parameters: trivial
Fixes: 0c5fbd80f1ba ("LU-5969 lustreapi: replace llapi_get_version()")
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ib4bdd715295f97fe5b7080f42bfa06883f191234
Reviewed-on: https://review.whamcloud.com/40680
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
3 years agoLU-14125 osc: prevent overflow of o_dropped 59/40659/3
Olaf Faaland [Wed, 11 Nov 2020 22:38:25 +0000 (14:38 -0800)]
LU-14125 osc: prevent overflow of o_dropped

In osc_announce_cached(), prevent o_dropped from overflowing.
Necessary because o_dropped AKA o_misc is 32 bits, but cl_lost_grant
is 64 bits.

Add a CDEBUG call so we can tell whether this happened.

Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Change-Id: Ia459934c789ae9609f851ae0a2581de583c6fc1c
Reviewed-on: https://review.whamcloud.com/40659
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
3 years agoLU-14136 utils: correct error message in lfs mirror extend 31/40631/2
John L. Hammond [Thu, 12 Nov 2020 16:13:33 +0000 (10:13 -0600)]
LU-14136 utils: correct error message in lfs mirror extend

In migrate_open_files(), use llapi_layout_file_open() instead of
lfs_component_create() to avoid printing a confusing error message and
to ensure that the type of error is correctly propagated.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I31ebf29771b1a3e0b106a0a246a260a82eed92a4
Reviewed-on: https://review.whamcloud.com/40631
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Gu Zheng <gzheng@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
3 years agoLU-14131 osd-ldiskfs: reduce credits for overwritting 04/40604/2
Wang Shilong [Wed, 11 Nov 2020 06:51:09 +0000 (14:51 +0800)]
LU-14131 osd-ldiskfs: reduce credits for overwritting

If all blocks are mapped which means this is overwritting
case or space has been allocated by fallocate.

There is no need to modify exten tree, and we only
need 1 credits for inode.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I907861d4862256a8a23a81812953e2330e1d9925
Reviewed-on: https://review.whamcloud.com/40604
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-13390 tests: add debugging to sanity test_65n 00/40600/3
Andreas Dilger [Wed, 11 Nov 2020 00:56:38 +0000 (17:56 -0700)]
LU-13390 tests: add debugging to sanity test_65n

Print out the layouts used in comparisons in case of error, so that
it is possible to debug intermittent test failures.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ib81b10ff704998518275f737f028fb16391654ed
Reviewed-on: https://review.whamcloud.com/40600
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14129 kernel: kernel update RHEL7.9 [3.10.0-1160.6.1.el7] 94/40594/2
Jian Yu [Tue, 10 Nov 2020 18:18:44 +0000 (10:18 -0800)]
LU-14129 kernel: kernel update RHEL7.9 [3.10.0-1160.6.1.el7]

Update RHEL7.9 kernel to 3.10.0-1160.6.1.el7.

Test-Parameters: clientdistro=el7.9 serverdistro=el7.9

Change-Id: If2d2120082965de67a3b29ade3e4d24a4221b2c2
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/40594
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-930 utils: document 'lfs getstripe -N' option 45/40545/3
Andreas Dilger [Thu, 5 Nov 2020 07:40:47 +0000 (00:40 -0700)]
LU-930 utils: document 'lfs getstripe -N' option

Add the '--mirror-count|-N' option to the usage message for
"lfs getstripe" and its man page.

Fixes: 818340364d51 ("LU-11124 utils: add 'lfs getstripe -N' option")
Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I89ffb2661a48336c37b8bdd784e5f9e942cbd798
Reviewed-on: https://review.whamcloud.com/40545
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-13783 build: Fix LB_LINUX_VERSION rule for v5.8 71/40371/3
Mr NeilBrown [Fri, 23 Oct 2020 04:07:36 +0000 (15:07 +1100)]
LU-13783 build: Fix LB_LINUX_VERSION rule for v5.8

Since Commit 20b1be595282 ("kbuild: fix single target builds for
external modules") in v5.8-rc7, the LB_LINUX_VERSION autoconf
rule doesn't work.

I don't know exactly why, but it can be fixed by setting "makerule" to
an empty string.

Passing the path to the directory in $makerule is unnecessary as
LB_LINUX_COMPILE_IFELSE, which LB_LINUX_TRY_MAKE eventually calls,
passes "$MODULE_TARGET=$PWD/build" which has the required effect.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I2b381d3546aaa0f365328a1319b2d4f145f33eeb
Reviewed-on: https://review.whamcloud.com/40371
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Aurelien Degremont <degremoa@amazon.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-13990 ldlm: ldlm_flock_deadlock() ASSERTION(req != lock) failed 47/40047/4
Andriy Skulysh [Tue, 19 May 2020 12:10:52 +0000 (15:10 +0300)]
LU-13990 ldlm: ldlm_flock_deadlock() ASSERTION(req != lock) failed

A client gets evicted and reconnects, so there can be a window
with 2 exports with flocks from the same client.
In this case during deadlock search from the new export we can go
to the old export and back to the new one.

Failed exports should be excluded from
deadlock search.

Change-Id: I9dec50d4c6694bbbcf13b976b5ebdc29377261ce
HPE-bug-id: LUS-8635
Signed-off-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Alexander Boyko <c17825@cray.com>
Reviewed-by: Vitaly Fertman <c17818@cray.com>
Reviewed-on: https://review.whamcloud.com/40047
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Vitaly Fertman <vitaly.fertman@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-13987 ldlm: Don't re-enqueue glimpse lock on read 44/40044/3
Andriy Skulysh [Wed, 1 Apr 2020 18:20:58 +0000 (21:20 +0300)]
LU-13987 ldlm: Don't re-enqueue glimpse lock on read

cl_glimpse_lock() doesn't match a lock with LDLM_FL_BL_AST
even if this lock is acquired by the same thread earlier.

It needs only size to check for spare file,
so let't add LDLM_FL_CBPENDING to match flags.

 #1 [ffff9ba7326036f0] schedule at ffffffff87b67c49
 #2 [ffff9ba732603700] obd_get_request_slot at ffffffffc0dbe0a4 [obdclass]
 #3 [ffff9ba7326037b8] ldlm_cli_enqueue at ffffffffc0faedce [ptlrpc]
 #4 [ffff9ba732603878] mdc_enqueue_send at ffffffffc11b38a8 [mdc]
 #5 [ffff9ba732603938] mdc_lock_enqueue at ffffffffc11b3eb2 [mdc]
 #6 [ffff9ba7326039a8] cl_lock_enqueue at ffffffffc0dfee95 [obdclass]
 #7 [ffff9ba7326039e0] lov_lock_enqueue at ffffffffc10ef265 [lov]
 #8 [ffff9ba732603a20] cl_lock_enqueue at ffffffffc0dfee95 [obdclass]
 #9 [ffff9ba732603a58] cl_lock_request at ffffffffc0dff54b [obdclass]

HPE-bug-id: LUS-8690
Change-Id: I4c3820f754ceb502079bdd7d8e1a5389f6696eba
Signed-off-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Vitaly Fertman <c17818@cray.com>
Reviewed-by: Alexander Boyko <c17825@cray.com>
Reviewed-by: Andrew Perepechko <c17827@cray.com>
Tested-by: Elena Gryaznova <c17455@cray.com>
Reviewed-on: https://review.whamcloud.com/40044
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Vitaly Fertman <vitaly.fertman@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-13891 utils: fix memory leak in llapi_ladvise() 10/39610/6
Jean-Yves VET [Fri, 7 Aug 2020 13:49:36 +0000 (15:49 +0200)]
LU-13891 utils: fix memory leak in llapi_ladvise()

A buffer allocated in llapi_ladvise() for ioctl() is never
released. This patch ensures the buffer is properly freed.

Fixes: e14246641c04 ("LU-4931 ladvise: Add feature of giving file access advices")
Signed-off-by: Jean-Yves VET <jyvet@ddn.com>
Change-Id: I0761e161074ae3029218473ec951670fdbbd33bd
Reviewed-on: https://review.whamcloud.com/39610
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
3 years agoLU-13837 lnet: Introduce constant for net ID of LNET_NID_ANY 44/39544/4
Chris Horn [Thu, 30 Jul 2020 16:29:30 +0000 (11:29 -0500)]
LU-13837 lnet: Introduce constant for net ID of LNET_NID_ANY

This patch adds a new constant, LNET_NET_ANY, to represent the net
ID of the LNET_NID_ANY wildcard NID.

Test-Parameters: trivial
HPE-bug-id: LUS-9122
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I4b4d9e70ba2826843c6585ad5a9e365799face65
Reviewed-on: https://review.whamcloud.com/39544
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
3 years agoLU-13817 quota: print error when pool is absent 05/39505/8
Sergey Cheremencev [Fri, 24 Jul 2020 10:20:17 +0000 (13:20 +0300)]
LU-13817 quota: print error when pool is absent

Print error for "lfs quota --pool" and
"lfs setquota -o" if pool is absent.

HPE-bug-id: LUS-9112
Test-Parameters: trivial testlist=sanity-quota
Change-Id: I4c2aa41d8c6f27742e68dfdc78c9d9365c760237
Signed-off-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-on: https://review.whamcloud.com/39505
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-13746 utils: check argument logname presence in llog_print 63/39263/3
Etienne AUJAMES [Fri, 3 Jul 2020 08:17:29 +0000 (10:17 +0200)]
LU-13746 utils: check argument logname presence in llog_print

Correction of segfault in llog_print when no logname specified.
example: lctl --device MGS llog_print

Signed-off-by: Etienne AUJAMES <eaujames@ddn.com>
Change-Id: I03a6c08dfc73ff8cb5861d162e21ce5aa581e197
Reviewed-on: https://review.whamcloud.com/39263
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
3 years agoLU-13159 osd: osd-zfs to release index back structures 93/37293/12
Alex Zhuravlev [Mon, 20 Jan 2020 17:12:58 +0000 (20:12 +0300)]
LU-13159 osd: osd-zfs to release index back structures

otherwise those can be leaked in case of failed mount

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I7dc848a8397c11b1f56d36ef0a7155314ca9afc2
Reviewed-on: https://review.whamcloud.com/37293
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
3 years agoLU-14125 obdclass: add grant fields to export procfile 63/40563/4
Olaf Faaland [Thu, 2 Jul 2020 21:25:32 +0000 (14:25 -0700)]
LU-14125 obdclass: add grant fields to export procfile

Add ted_{grant,reserved,dirty} to the export
procfile for OSTs, to allow comparison between the
OST's idea of grants allocated to the client with
the client's idea.

Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Change-Id: Ib34582e2be55fe2007363b52cea4dee211b7f540
Reviewed-on: https://review.whamcloud.com/40563
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14031 tests: check client reconnect 97/40297/7
Alexander Boyko [Mon, 19 Oct 2020 13:07:10 +0000 (09:07 -0400)]
LU-14031 tests: check client reconnect

The patch adds test 147 for recovery-small. It checks how often
client reconnects to a server with obd_timeout=200.

HPE-bug-id: LUS-8520
Test-Parameters: trivial testlist=recovery-small env=ONLY=147
Signed-off-by: Alexander Boyko <alexander.boyko@hpe.com>
Change-Id: Id5bd732fe4d949cfa45ea0272f197809cca3290d
Reviewed-on: https://review.whamcloud.com/40297
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Reviewed-by: Vitaly Fertman <vitaly.fertman@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-13571 lnet: Correct handling of NETWORK_TIMEOUT status 98/39898/14
Chris Horn [Fri, 11 Sep 2020 18:41:39 +0000 (13:41 -0500)]
LU-13571 lnet: Correct handling of NETWORK_TIMEOUT status

The original intent of the LNET_MSG_STATUS_NETWORK_TIMEOUT health
status was to handle cases where the LND was unsure whether the
failure was due to the local or remote NI. In this case, we'll want
to decrement both the local and remote NI health and allow recovery
to ascertain which interface is actually healthy.

Test-Parameters: trivial
HPE-bug-id: LUS-9342
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: Ib00ac260640100123e4e97e9c566289e92fb0b6e
Reviewed-on: https://review.whamcloud.com/39898
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14027 ldlm: Do not hang if recovery restarted during lock replay 38/40238/4
Oleg Drokin [Wed, 14 Oct 2020 03:55:02 +0000 (23:55 -0400)]
LU-14027 ldlm: Do not hang if recovery restarted during lock replay

LU-13600 introduced lock ratelimiting logic, but it did not take
into account that if there's a disconnection in the REPLAY_LOCKS
phase then yet unsent locks get stuck in the sending queue so
the replay locks thread hangs with imp_replay_inflight elevated
above zero.

The direct consequence from that is recovery state machine never
advances from REPLAY to REPLAY_LOCKS status when imp_replay_inflight
is non zero.

Adjust __ldlm_replay_locks() to check if the import state changed
before attempting to send any more requests.

Add a testcase.

Change-Id: Idbaf5461f33d1884088269d67d01071c7e1bf8a5
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Fixes: 3b613a442b ("LU-13600 ptlrpc: limit rate of lock replays")
Reviewed-on: https://review.whamcloud.com/40238
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
3 years agoLU-13195 obdclass: show FID for corrupted llog 77/38977/2
Alex Zhuravlev [Thu, 18 Jun 2020 07:00:26 +0000 (10:00 +0300)]
LU-13195 obdclass: show FID for corrupted llog

to be able to remove that easily.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Iefead860b5e60e74e0eb445f715508a3b01fac87
Reviewed-on: https://review.whamcloud.com/38977
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14053 utils: detect hangup in ofd_access_log_reader 11/40311/7
John L. Hammond [Thu, 24 Sep 2020 21:05:39 +0000 (16:05 -0500)]
LU-14053 utils: detect hangup in ofd_access_log_reader

In ofd_access_log_reader, when the batch file is a pipe or socket,
detect hangups and exit.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: If7509174b992011305d8e4a7aa2766a3a1980831
Reviewed-on: https://review.whamcloud.com/40311
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14052 ofd: support for multiple access readers 06/39906/15
Alex Zhuravlev [Tue, 15 Sep 2020 12:42:17 +0000 (15:42 +0300)]
LU-14052 ofd: support for multiple access readers

ofd_access_log_reader can be passed -I, --mdt-index-filter=INDEX to
print only FIDs located on INDEX.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I9a4f09c6b7ca15623d459df17939895301a57a8b
Reviewed-on: https://review.whamcloud.com/39906
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14009 osd: missing ldiskfs_htree_unlock() 37/40137/4
Alex Zhuravlev [Sun, 4 Oct 2020 10:41:26 +0000 (13:41 +0300)]
LU-14009 osd: missing ldiskfs_htree_unlock()

in osd_ldiskfs_it_fill()

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I5d2242e0864cbaa72af096b263d8758966a6be22
Reviewed-on: https://review.whamcloud.com/40137
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 years agoLU-14027 ldlm: Do not wait for lock replay sending if import dsconnected 72/40272/4
Oleg Drokin [Fri, 16 Oct 2020 14:25:58 +0000 (10:25 -0400)]
LU-14027 ldlm: Do not wait for lock replay sending if import dsconnected

If import disconnected while we were preparing to send some lock replays
the sending RPC would get stuck on the sending list and would keep
the reconnected import state from progressing from REPLAY
to REPLAY_LOCKS state waiting for the queued replay RPCs to finish.

Set them as no_delay to ensure they don't wait.

LU-13600 exacerbated this issue some but it certainly exist
before it as well.

Change-Id: Id276a0be7657d9ad6cf40ad8e7a165d5cd341cb8
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/40272
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
3 years agoLU-14061 utils: prefer mounting with specified fstype 74/40474/3
Andreas Dilger [Thu, 29 Oct 2020 20:47:40 +0000 (14:47 -0600)]
LU-14061 utils: prefer mounting with specified fstype

If the server filesystem is mounted with "-t lustre" use that type
for mounting instead of automatically selecting "lustre_tgt" for
server mounts, so that it is appearing in /proc/mounts correctly,
and "mount -t lustre" will list all of these filesystems.

Only if "mount -t lustre_tgt" is used will mount.lustre_tgt be called,
and that will internally use "lustre_tgt" as the filesystem type, and
only fall back to type "lustre" if that does not work.  This will give
userspace plenty of time to transition to using "lustre_tgt" for
server mountpoints.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ifd7560c800acdfe14c9564bbf955ecad1224f2e3
Reviewed-on: https://review.whamcloud.com/40474
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>