RM-620 build: New tag 2.14.0-ddn20

New tag 2.14.0-ddn20

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I211d330581f8998fc190add692d3d73cc5800d41

LU-14300 quota: avoid nested lqe lookup

lqe_locate called from qmt_pool_lqes_lookup for lqe
that hasn't an entry on a disk calls qmt_lqe_set_default.
This may call qmt_set_id_notify->qmt_pool_lqes_spec
and rewrite already added lqes in a qti. Rewritten
lqes may trigger an assertion:

LustreError: 5072:0:(qmt_pool.c:838:qmt_pool_lqes_lookup())
ASSERTION( (((qmt_info(env)->qti_lqes_num) > 16 ?
qmt_info(env)->qti_lqes : qmt_info(env)->qti_lqes_small)[(
qmt_info(env)->qti_glbl_lqe_idx)])->lqe_is_global ) failed:
LustreError: 5072:0:(qmt_pool.c:838:qmt_pool_lqes_lookup()) LBUG
Pid: 5072, comm: mdt_rdpg00_003 3.10.0-957.1.3957.1.3.x4.1.15.x86_64
Call Trace:
 [<ffffffffc046f62c>] libcfs_call_trace+0x8c/0xc0 [libcfs]
 [<ffffffffc046f94c>] lbug_with_loc+0x4c/0xa0 [libcfs]
 [<ffffffffc0e4ae38>] qmt_pool_lqes_lookup+0x798/0x8f0 [lquota]
 [<ffffffffc0e3b0ce>] qmt_intent_policy+0x86e/0xe00 [lquota]
 [<ffffffffc109d53d>] mdt_intent_opc+0x3bd/0xb40 [mdt]
 [<ffffffffc10a5134>] mdt_intent_policy+0x1a4/0x360 [mdt]
 [<ffffffffc0a7bedb>] ldlm_lock_enqueue+0x3cb/0xad0 [ptlrpc]
 [<ffffffffc0aa4a46>] ldlm_handle_enqueue0+0xa56/0x1610 [ptlrpc]
 [<ffffffffc0b304b2>] tgt_enqueue+0x62/0x210 [ptlrpc]
 [<ffffffffc0b3753a>] tgt_request_handle+0x7ea/0x1750 [ptlrpc]

or a deadlock(2 same lqes qti_lqes array):

 call_rwsem_down_write_failed+0x17/0x30
 qti_lqes_write_lock+0xb1/0x1b0 [lquota]
 qmt_dqacq0+0x2ee/0x1ac0 [lquota]
 qmt_intent_policy+0xbfe/0xe00 [lquota]
 mdt_intent_opc+0x3ba/0xb50 [mdt]
 mdt_intent_policy+0x1a1/0x360 [mdt]
 ldlm_lock_enqueue+0x3d6/0xaa0 [ptlrpc]
 ldlm_handle_enqueue0+0xa76/0x1620 [ptlrpc]
 tgt_enqueue+0x62/0x210 [ptlrpc]
 tgt_request_handle+0x96a/0x1680 [ptlrpc]
 kthread+0xd1/0xe0

Patch adds a sanity-quota_73b to check that the isssue
doesn't exist anymore.

Lustre-change: https://review.whamcloud.com/43326
Lustre-commit: 188112fc806c8c61d536ba3230b8d50f65e4f8fc

Change-Id: Ib1ebe82c3b6e819b2538f30af08930060bd659ae
HPE-bug-id: LUS-9902
Signed-off-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Alexander Zarochentsev <c17826@cray.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Wang Shilong <wshilong@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45183
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13952 quota: default OST Pool Quotas

Patch makes ability to set default quota
limits per OST pool.
Patch also adds sanity-quota_73.

Lustre-change: https://review.whamcloud.com/39873
Lustre-commit: 25a70a88c9eb35b7e43347c0d8220e334591d25e

Test-Parameters: testlist=sanity-quota
HPE-bug-id: LUS-9133
Change-Id: I9e49def231aeeed4588e5e3fbcd29fdd62a35855
Signed-off-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-on: https://review.whamcloud.com/45182
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>

EX-3880 pcc: add pcc_async_affinity for async PCC attach

This patch adds a tunable parameter "llite.*.pcc_async_affinity"
that enables or disables the CPT selection in PCC-RO asynchronous
attach for testing.

Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I1473a7547555a2d6c615d37182b6cc359194aae0
Reviewed-on: https://review.whamcloud.com/45011
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-6142 lod: return pools_hash_params to being static.

A recent patch changes pools_hash_params in lod_pool.c to no longer
be 'static'.  This is not ideal.

rhashtable interfaces are mostly 'static inlines' which contain a lot
of code which is mostly optimised away providing that the 'params'
structure is const and locally visible.  When these interfaces are
called with a params structure in another file, the code produces is
quite inefficient and wasteful.

It is generally cleaner to provide accessor functions which can be
exported to other compilation units.  It is even beneficial to do that
within the one file.

This patch introduces
   lod_pool_exists()
and
   lod_pool_find()

The first is 'extern' and thus 'pools_hash_params' can not be static.
The second is used in several places in lod_pool.c, improving code
quality and maintainability.

Lustre-change: https://review.whamcloud.com/45070
Lustre-commit: 9ec5e2329bf3d7e38fa899a259221aa58fb48cd4

Fixes: 0a998f4723f5 ("LU-14825 lod: pool spilling")

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ieafe2f23fe5cc71d9bdce73cbe7360f5cb540edf
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45185
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15011 lod: count all spilling events

when target pool is used to as the original has no enough space.
lctl lod.*.pool.<poolname>.spill_hit can be used to get the counter.

Lustre-change: https://review.whamcloud.com/44947
Lustre-commit: TBD (from 6348594defc0b1a414b45abe309a8a18b1da303e)

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I6d54a2b910705da182b5f4118e535d196cdab004
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44948
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-4041 tests: look for lamigo log on MDS

hot-pools tests 7, 8 and 10 grep the lamigo log from a
client node looking for job completion or other lamigo
information.  The lamigo log is written to a space on the
MDS.  Change the tests to cat the log file on the MDS
and pipe that to grep.

Test-Parameters: trivial testlist=hot-pools

Change-Id: I78813d6188216b2529e1f6e41252bc6fe1fc9514
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45218
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-15106 ofd: quiet deprecated param warning

There are a number of obdfilter parameter files that report a
warning even when they are read, which is confusing for users
if there is a tool that is scraping all available parameters:

    # lctl get_param obdfilter.*.*
    ofd: 'obdfilter.*.read_cache_enabled' is deprecated,
         use 'osd-*.read_cache_enabled' instead
    ofd: 'obdfilter.*.readcache_max_filesize' is deprecated,
         use 'osd-*.readcache_max_filesize' instead
    ofd: 'obdfilter.*.sync_on_lock_cancel' is deprecated,
         use 'obdfilter.*.sync_lock_cancel' instead
    ofd: 'obdfilter.*.writethrough_cache_enabled' is deprecated,
         use 'osd-*.writethrough_cache_enabled' instead

It should only print a message if the parameters are actually written.
Also fix the messages to reference the correct parameter names.

Most of these parameter links were added in 2.4 with the addition of
osd-ldiskfs.  However, the deprecation warnings were only added in
2.12.53 and slated for removal in 2.15, but were not backported to
2.12 LTS, and there hasn't been an LTS release since then, so it is
better bump removal so the upcoming 2.15 LTS release includes them.

Fix the test scripts to only use the new parameter names, to avoid
spurious warning messages.  We don't test interop with 2.3 anymore.

Lustre-change: https://review.whamcloud.com/45246
Lustre-commit: TBD (from cf05bfb35f00c72f27ec36619259cc77eb56186d)

Test-Parameters: trivial
Fixes: 7df7347b7b18 ("LU-12967 ofd: restore sync_on_lock_cancel tunable")
Fixes: 493cd8088388 ("LU-8066 osd: migrate from proc to sysfs")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ie548e5b6af5463959fb4774e31996097373ebbe5
Reviewed-on: https://review.whamcloud.com/45247
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>

LU-15099 kernel: kernel update RHEL7.9 [3.10.0-1160.45.1.el7]

Update RHEL7.9 kernel to 3.10.0-1160.45.1.el7.

Test-Parameters: trivial \
env=SANITY_EXCEPT="415 418" \
clientdistro=el7.9 serverdistro=el7.9 \
testlist=sanity

Change-Id: I11c307bfd6a6b353bc7b6fe40bb5d604bc9b3fdc
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45228
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

NVDA-86 pcc: print kern_path() error on failure

When looking up a PCC cache pathname in kern_path() returns an error,
print the pathname and returned error before it is ignored.

Test-Parameters: trivial testlist=sanity-pcc
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I9ef59ffedc2c6d11cc7ab1abb4098c56f23ebbe5
Reviewed-on: https://review.whamcloud.com/44962
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>

LU-13997 tests: fix sanity test_418 lock cancellation

Use "do_nodes" directly to cancel DLM locks, rather than
"do_rpc_nodes", since that is very heavy to use in a loop
(each call takes 3s, but the loop delay is only 0.2s).

Due to DoM reserving grant space for the DoM files, the "avail"
space shown by "df" may be smaller in the aggregate returned by
the MDT compared to the individual values from "lfs df".

Skip this part of the check until MDC grant cancel is fixed.

Lustre-change: https://review.whamcloud.com/45231
Lustre-commit: TBD (from e4bb910bba91199af21199080c5f3eb1070c59c9)

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I83d989688ce671f0ff9c62ebdf3144746a3ebbe5
Reviewed-on: https://review.whamcloud.com/45232
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn19

New tag 2.14.0-ddn19

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ia08fb0df6f54adb5b1713e2e85e547a4a6ebec66

LU-14195 obdclass: change list_sort to use const pointers

Kernel 5.10.70 commit 4f0f586bf0c898233d8f316f471a21db2abd522d
defines the list_cmp_func_t type and changes the comparison
function types of all list_sort() callers to use const pointers
to avoid type mismatches.

Lustre-change: https://review.whamcloud.com/45219
Lustre-commit: TBD (from 01ef4ea77d6798bfb7d9c94b3a4dc809838c07b0)

Change-Id: I40d37ec0f0d485d0deebaa9dc3493f2865f76ec9
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45220
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-4017 build: remove extra line cause scriptlet error

Remove unnecessary line

Test-Parameters: trivial

Change-Id: Ic4e5d42161a5db15155fbf3f6e52444d50a5bb9a
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45217
Reviewed-by: Gaurang Tapase <gtapase@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-14321 tests: create PFL file in sanityn 51b

sanityn test 51b was modified to integrate statx API in
Lustre version 2.13.54.  When we run version interop testing
with servers less than 2.13.54 and later clients, the test
will fail.

We should modify the test to create a PFL file without the
'extension-size' lfs setstripe option which will allow this
test to run with servers less than 2.13.54.

Lustre-change: https://review.whamcloud.com/44027
Lustre-commit: 82d1c06b0c12a724817fa82ff4b3ef088679cdff

Fixes: 3f7853b31ef6 ("LU-10934 llite: integrate statx() API with Lustre")

Test-Parameters: trivial
Test-Parameters: serverversion=2.12.6-ddn42 serverdistro=el7.9 env=ONLY=51b testlist=sanityn

Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: Ic3feb72771aa2db050b792159175624260e71f5b
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-on: https://review.whamcloud.com/45184
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14448 lod: verify LOV before set/inherit

DoM layout can only be set as entry in composite layout, and its
stripe count should always be 0.

Verify LOV before setting and inheriting.

Add sanity 270i.

Lustre-change: https://review.whamcloud.com/45039
Lustre-commit: TBD (from f6cb64a29f6445356e0557597ddf83e761de8ca1)

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I016d1a202960bfebc72dd808de5f80e09051a01e
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45204
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-3890 lipe: raise pool spilling threshold

Raise the default fast pool spill threshold to 90%.

Lustre-change: https://review.whamcloud.com/45147
Lustre-commit: TBD (from b37fd65b3d0110100d964370f598edebe2d322af)

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I3c8ce85a1eb3aae6c2b49e8b195a8986558335af
Reviewed-on: https://review.whamcloud.com/45205
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn18

New tag 2.14.0-ddn18

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ie927d5fc981b968f871622a8c9dcd1e5b0461b9d

LU-14337 lov: return valid stripe_count/size for PFL files

Dump struct lov_comp_md_v1 in function ll_lov_getstripe_ea_info()
correctly to avoid stripe_count=0 or stripe_size=0 returned by
old interface llapi_file_get_stripe(), which will cause
divide-by-zero for older userspace that calls this ioctl,
e.g. lustre ADIO driver.
The rule is:
- if stripe_count=0, return stripe_count=1;
- if stripe_size=0,
  -- for DoM files, return the stripe size of the second component,
     since the first component of DoM file data is placed on the
     MDT for faster access;
  -- else, return the stripe size of the last component.

Also, lov_getstripe_old.c and santy-pfl.sh test_25 is added to
verify this patch.

Lustre-change: https://review.whamcloud.com/41803
Lustre-commit: abf04e7ea356e8b1523e4183d5c46c0462f1f605

Test-Parameters: testlist=sanity-pfl env=ONLY=25
Signed-off-by: Emoly Liu <emoly@whamcloud.com>
Change-Id: I4023ca4baff1b1ad2a439aa497baaabc56e891d2
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-on: https://review.whamcloud.com/45028
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-4006 pcc: improve pcc-detach and man pages

Improve "lfs pcc detach_fid" to have a "--mnt|-m" option to match
the corresponding option for "lfs pcc attach_fid".  There isn't
any reason that they should be different.

Separate lfs-pcc-attach.1 and lfs-pcc-detach.1 from the
lfs-pcc.1 man page and improve descriptions of these commands.
Add links for lfs-pcc-attach_fid.1 and lfs-pcc-detach_fid.1.

Test-Parameters: trivial testlist=sanity-pcc
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ieff40b7b133aef64200a9a3f6c1a948d7c3ebbe5
Reviewed-on: https://review.whamcloud.com/45178
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Feng, Lei <flei@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-4006 pcc: make "pccro=1" default

To avoid a risk that users will accidentally configure PCC-RW and
potentially lose data if those client nodes go offline, this patch
makes "pccro=1" default for PCC backends.

This patch adds a new option "--w|--write" for PCC-RW cache
mode when attach a file.
Also It makes "--r|--readonly" as a default option for PCC attach
command.

Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I56735b0ebe8f0d9ef22b3f7e39e8cccfa3aad443
Reviewed-on: https://review.whamcloud.com/45162
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-3825 pcc: Check if PCC copy is unlinked for state output

In this patch, it adds support for the command "lfs pcc state" to
check whether the PCC copy is in the local client cache or
unlinked improperly.

Test-Parameters: testlist=sanity-pcc env=ONLY=48
Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: Ic50c901df78adfaf5b56990120f832e5d74a117c
Reviewed-on: https://review.whamcloud.com/45082
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15064 tests: sanity-sec test_58 must read its own dir

sanity-sec test_58 should restrict file listing to its own
directory, and not try to list content of the entire Lustre tree.
This is useless for the purpose of the test, and exposes to unclean
remnants from previous tests.

Lustre-change: https://review.whamcloud.com/45150
Lustre-commit: TBD (6d7eda03355dc641bfb0c40e849df3d29c5088f8)

Test-Parameters: trivial
Test-Parameters: testlist=sanity-sec env=ONLY=58
Fixes: bf291f2212 ("LU-14989 sec: access to enc file's xattrs")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ic9ab0860da0ab86355a207ad9d50feb3975adf68
Reviewed-on: https://review.whamcloud.com/45151
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-14989 sec: keep encryption context in xattr cache

When an inode is being cleared, its xattr cache must be completely
wiped. But in case of lock cancel, we want to keep the encryption
context, as further processing might need to check it.

Lustre-change: https://review.whamcloud.com/45148
Lustre-commit: TBD (bc79d1bc90f9d094598985127f8386894d0b7355)

Fixes: bf291f2212 ("LU-14989 sec: access to enc file's xattrs")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I8a2f4497129353a7fbf86cdaaa13fae6e0988790
Reviewed-on: https://review.whamcloud.com/45149
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-3825 pcc: no error to detach missing file

Do not print an error message if "lfs pcc detach" tries to detach
a file that is already removed from the cache.  This might happen
for a wide variety of reasons (external cache cleanup process, etc).

Test-Parameters: trivial testlist=sanity-pcc
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Iad07b4bbab809bd356ed06c9f5cfcc35fa3ebbe5
Reviewed-on: https://review.whamcloud.com/45132
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Feng, Lei <flei@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>

LU-11407 tgt: cleanup job_stats output printing

Escape non-printable and other special characters in the JobID
name, which may be passed from the client environment, to avoid
breaking YAML format parsing.

Lustre-change: https://review.whamcloud.com/37764
Lustre-commit: TBD (from 953c674fa9b9ae6f20e2cbdd8ed17cadd17922c9)

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: If6a0bc4276fae03305f94e8c85d8f109913ebbe5
Reviewed-on: https://review.whamcloud.com/45112
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-3875 lustre: add allow_intr param to mdc and osc

Add an obd_import parameter ("allow_intr") to make all requests from a
given import interruptible. Remove hot-pools test_16() from
ALWAYS_EXCEPT.

Remove the unconditional setting of allow_intr added by EX-3749 mdc:
non blocking close and changelog release.

Lustre-change: https://review.whamcloud.com/45005
Lustre-commit: c5803ee670b57d05568f4c0ef527be6977772540

Test-Parameters: trivial testlist=hot-pools env=ONLY=16,ONLY_REPEAT=20
Change-Id: Ibc258a0e89dab204f2a10ba29e3eea19030ad444
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45129
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn17

New tag 2.14.0-ddn17

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ib9987cfd4898bec88b1ffebe418b426335eaa534

EX-2067 lpurge: check layout before opening

In lpurge_mirror_delete(), fetch the layout by getxattr() and check if
we would be deleting the last stale mirror before we open the
file. This is to avoid breaking the lease held by a concurrent mirror
resync.

Lustre-change: https://review.whamcloud.com/45033
Lustre-commit: 912766d05efd95886597a46dcbaaf10237e56ebf

Test-Parameters: trivial testlist=hot-pools
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Change-Id: I8b1addd14b290faa537fcc15514cae8b18c802f2
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45111
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-3937 lnet: include linux/ethtool.h

Commit a8929b9f0b3e86ae91fd0037edc37a356020a947 added
a check to see if linux/ethtool.h is included in
linux/netdevice.h. However, a more simple way is to
include linux/ethtool.h directly in the files that use
"struct ethtool_ops".

Test-Parameters: trivial

Fixes: a8929b9f0b3 ("EX-3937 lnet: check if ethtool.h is included in netdevice.h")
Change-Id: I5d1b45a15194a0ba2dbd52ce47fdb9bf9fda028c
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45110
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14629 sec: do not block rename of topmost encrypted dir

We intentionally forbid file and directory rename from encrypted to
unencrypted directory. But we must not block rename of the topmost
encrypted directory.

Lustre-change: https://review.whamcloud.com/45054
Lustre-commit: 3db8e31797535570e5f2e99f6c8471b49d395efe

Fixes: 1158386ac9 ("LU-14629 sec: forbid file rename from enc to unencrypted dir")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I480a24b2b0327e1d9104f216da54720e4f351636
Reviewed-on: https://review.whamcloud.com/45085
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14989 sec: access to enc file's xattrs

Encryption context is stored in 'security.c' xattr. This is put in the
xattr cache via ll_xattr_cache_insert() to avoid sending a getxattr
request to the server. But this operation declares the xattr cache for
the inode as 'valid', with two consequences. It prevents any further
filling with other xattrs, and trying to read an xattr value will
directly return -ENODATA, without any attempt to fetch the xattr from
the server.
This is solved by adding a new ll_file_flags 'LLIF_XATTR_CACHE_FILLED'
that tells if the xattr cache for the inode has been filled. This bit
is set only by ll_xattr_cache_refill(), and 'valid' now just means the
xattr cache for the inode has been initialized.

Lustre-change: https://review.whamcloud.com/44855
Lustre-commit: 1faf54e8bf19c28a4de7b069309d607cb3f10f91

Fixes: 40d91eafe2 ("LU-12275 sec: atomicity of encryption context getting/setting")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I6c2b6870df29f26f048dedeb7212d1c801ca69e1
Reviewed-on: https://review.whamcloud.com/45084
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

DDN-2073 tests: fix sanity-sec test_15

sanity-sec test_15 should not inherit trusted and admin properties for
newly created nodemaps from the default nodemap. Instead, it must
explicitly set both to 0, as the whole point is to exercise id
mappings.

Test-Parameters: trivial testlist=sanity-sec
Test-Parameters: clientdistro=ubuntu2004 testlist=sanity-sec
Change-Id: Ie80a10528268024d59cd079949620ebd1a836395
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-on: https://review.whamcloud.com/45100
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15011 tests: pool spill test modifications

Make the following modifications to the ost-pools
test suite:
test 29 - change check for ‘when striping is specified
explicitly ‘ file from ‘file-2’ to ‘file-3’

test 30 - Add bad parameter check for setting the threshold
below zero

test 31 - ‘do_nodes $mdts $LCTL get_param lod.*.pool.*’
doesn’t print anything. Change to
‘do_nodes $mdts $LCTL get_param lod.*.pool.*.spill*’

Lustre-change: https://review.whamcloud.com/45056
Lustre-commit: TBD (from a097f6ea7e5a9514c08995d6ed721cba05bb1fb0)

Fixes: 0a998f4723 (“LU-14825 lod: pool spilling”)

Test-Parameters: trivial testlist=ost-pools
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: Icbdc3d42b7f7609bc57cc37830975d831125d659
Reviewed-on: https://review.whamcloud.com/45103
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-14781 osp: osp object header could be NULL

Don't call lu_object_header_fini upon NULL header in
osp_object_free().

Call trace:
lu_object_free.isra.30+0xf2/0x170 [obdclass]
lu_object_find_at+0x496/0x930 [obdclass]
lod_initialize_objects+0x3e4/0xba0 [lod]
lod_parse_striping+0x693/0xc20 [lod]
lod_striping_load+0x2b2/0x660 [lod]
lod_declare_destroy+0x12b/0x600 [lod]
mdd_declare_finish_unlink+0x91/0x210 [mdd]
mdd_unlink+0x48f/0xab0 [mdd]
mdt_reint_unlink+0xc32/0x1550 [mdt]
mdt_reint_rec+0x83/0x210 [mdt]
mdt_reint_internal+0x6e1/0xb00 [mdt]
mdt_reint+0x67/0x140 [mdt]
tgt_request_handle+0xaee/0x15f0 [ptlrpc]
ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
ptlrpc_main+0xb34/0x1470 [ptlrpc]
kthread+0xd1/0xe0

Lustre-change: https://review.whamcloud.com/44055/
Lustre-commit: 90dab62d635666001b8b4e2b7aa8cc1f23bc6f5b

Test-Parameters: trivial
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Iec23cf06dffaa64c6f5853c28382ba930ee1076b
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45101
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-3749 mdc: non blocking close and changelog release

Use interruptible RPCs in mdc_close() and the llog client
functions. In chlg_release() send a SIGKILL to the changelog producer
thread to wake from waiting on any RPC responses. Add sanity.sh test
160r to verify.

Lustre-change: https://review.whamcloud.com/44842
Lustre-commit: 83e0eb2be4fa61344137b2f50724a1ca1dc532ae

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I5f8b7cd839f5d6cf704ac07c7d583c45013921c4
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44960
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

EX-3937 lnet: check if ethtool.h is included in netdevice.h

Kernel 5.11.0-36 removes including linux/ethtool.h from
linux/netdevice.h, which caused the following build error:

dereferencing pointer to incomplete type ‘const struct ethtool_ops’

This patch fixes the above issue by checking if ethtool.h
is included in netdevice.h.

Change-Id: Ifc25de5acaebf2b5fd5bb6f1c303366ab9ea6ef6
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45097
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-15036 kernel: kernel update SLES15 SP3 [5.3.18-59.24.1]

Update SLES15 SP3 kernel to 5.3.18-59.24.1 for Lustre client.

Test-Parameters: trivial

Change-Id: I77987472f91afd9f695f988997e88233a1a5de99
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45061
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-10848 test: wait to process inodes in phase2

In test_8 of sanity-lfsck, the "LF_INCONSISTENT" flag was set
when processing the inodes with corrupted LinkEA in phase2,
LFSCK could have no chance to process it yet because of the delay
OBD_FAIL_LFSCK_DELAY3.

Lustre-change: https://review.whamcloud.com/44658
Lustre-commit: 4efcd6f9b8f9bb768c6167a880777413950619c4

Test-Parameters: trivial testlist=sanity-lfsck env=ONLY=8,ONLY_REPEAT=20
Change-Id: Id414728c998d527fbc27f877c6d31dcedcc12457
Signed-off-by: Hongchao Zhagn <hongchao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45013
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-3803 build: build lipe-lpcc on ubuntu 20.04 client

* only support client. need to use ./configure --disable-server

Test-Parameters: trivial

Change-Id: I83c7a712cc0c58dcadd5df59bb49e3fc3d85f9b1
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45095
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Feng, Lei <flei@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14729 osd-ldiskfs: declare should consider concurrency

Write in Lustre OSD is different than Ext4 since write
is serialized in local filesystem, however in OSD side,

many concurrent threads may grow tree before transaction starts.

Also fix to use @dirty_groups rather than @extents, remove
unnecessary @depth assignment.

Lustre-change: https://review.whamcloud.com/44316
Lustre-commit: 934758366dbde211743d53b8c048c57819603cb3

Fixes: 9810341a8 ("LU-14729 osd-ldiskfs: fix to declare write commits")
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I1e0fc9069a579736a74b0ba2607056fe980574c3
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45077
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

EX-3808 pcc: reset file mapping for pcc copy

The inode mapping of PCC copy will be replaced since mmap().
Then we must check & reset it while io starting. Else system
will crash on:
CPU: 3 PID: 31556 Comm: dd Kdump: loaded Tainted: P
Call Trace:
[<c1424463>] ll_readpage+0x4f3/0x870 [lustre]
[<a4bbf400>] generic_file_aio_read+0x3f0/0x790
[<c0418dc3>] ext4_file_read+0x43/0x130 [ext4]
[<c14564a3>] pcc_file_read_iter+0x323/0x390 [lustre]
[<c13f9ee2>] ll_file_aio_read+0x1f2/0x510 [lustre]
[<a4bfa59c>] ? mmap_region+0x20c/0x670
[<c13fa300>] ll_file_read+0x100/0x1c0 [lustre]
[<a4c4a6df>] vfs_read+0x9f/0x170
[<a4c4b59f>] SyS_read+0x7f/0xf0
[<a518de21>] ? system_call_after_swapgs+0xae/0x146
[<a518dede>] system_call_fastpath+0x25/0x2a
[<a518de21>] ? system_call_after_swapgs+0xae/0x146

Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: I719e9fac7362fe76f0ece399dda693b9e965c5bc
Reviewed-on: https://review.whamcloud.com/45052
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

EX-3846 lipe: backport lpcc script to python2

Since python2 is used by all other scripts in lipe,
backport lpcc to python2 too.

Change-Id: I5cffef7f4a38fe32ad071070076643814ca53736
Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial testlist=sanity-pcc env=ONLY=200-202,ONLY_REPEAT=50 \
                 clientextra_install_params="--packages lipe-lpcc"
Reviewed-on: https://review.whamcloud.com/45015
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Tested-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>

LU-14740 quota: reject invalid project id on server side

do sanity check before transfer project ID, reject invalid
project id if it comes from some older clients.

Lustre-change: https://review.whamcloud.com/44339
Lustre-commit: d6a3e06cb0f0db57a2637d029b1ff3bfd1de3d7d

Test-Parameters: trivial testlist=sanity-quota
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: If89e320c7808d188e615f5f0923c2322774b2ceb
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44946
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>

LU-14740 llite: avoid project quota overflow

Currently, project ID is stored as u32, max possible
value for it is 4294967295.

However, VFS reserve max value for special usage, see
following function:

  static inline bool
  qid_has_mapping(struct user_namespace *ns, struct kqid qid)
  {
          return from_kqid(ns, qid) != (qid_t) -1;
  }

So qid_has_mapping() could return 0 for id 4294967295.
A further try on chown test:

  $ chown 4294967295:4294967295 c.sh
  chown: invalid user: "4294967295:4294967295"
  $ chown 4294967294:4294967294 c.sh

Fix to check max possible value for project ID in the
client kernel side, and add a test case for this.

Lustre-change: https://review.whamcloud.com/43939
Lustre-commit: 3ffa5d680f0092ae51ffa84bd94a9983f9a8c99e

Test-Parameters: trivial testlist=sanity-quota
Fixes: 7b5c1f1404c3 ("LU-13845 utils: Quota id 0xFFFFFFFF is invalid")
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: Ide8b9cc79d9b7f2a8b9860a0c0f683ec903b8f91
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44938
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-11872 quota: add get/set project support for non-dir/file

Add ablity to get/set non-dir/file's project ID and state.

Lustre-change: https://review.whamcloud.com/44006
Lustre-commit: b31792b0e72425c8c7850d69837f08c9f3e95a9c

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: Ib8eee09254f9751797b5deb7f753c34eb2c0d5a5
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Reviewed-on: https://review.whamcloud.com/44922
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>

EX-3409 tests: handle pcc async attach better

Reset llite.*.pcc_async_threshold to the original value when the
subtest exits, so that it doesn't affect later tests.

Wait for async pcc attach to finish in test_47.

Test-Parameters: trivial testlist=sanity-pcc env=ONLY=46-47,ONLY_REPEAT=100
Test-Parameters: testlist=sanity-pcc
Test-Parameters: testlist=sanity-pcc
Test-Parameters: testlist=sanity-pcc
Test-Parameters: testlist=sanity-pcc
Test-Parameters: testlist=sanity-pcc
Test-Parameters: testlist=sanity-pcc
Test-Parameters: testlist=sanity-pcc
Test-Parameters: testlist=sanity-pcc
Test-Parameters: testlist=sanity-pcc
Test-Parameters: testlist=sanity-pcc
Fixes: 6c693d126197 ("EX-3764 pcc: avoid panic in asynchronous attach thread")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I9447fa1fc5ac20fd86340c2eda9bc08ba23ebbe5
Reviewed-on: https://review.whamcloud.com/44982
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>

EX-3720 tests: skip lpcc_purge tests if not installed

Skip lpcc_purge tests if package is not installed on client.

Test-Parameters: trivial testlist=sanity-pcc env=ONLY=200-202                  clientextra_install_params="--packages lipe-lpcc"
Test-Parameters: testlist=sanity-pcc env=ONLY=200-202
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ic977c1cad602cc1ae33b273493bfb9d0f9dbe37b
Reviewed-on: https://review.whamcloud.com/44994
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>

EX-3720 lipe: add test cases for lpcc_purge

add sanity test cases for lpcc_purge

Change-Id: I420f461f116c41767340329f02920118bf183673
Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial testlist=sanity-pcc env=ONLY=200-202,ONLY_REPEAT=50 \
                 clientextra_install_params="--packages lipe-lpcc"
Test-Parameters: trivial testlist=sanity-pcc env=ONLY=200-202,ONLY_REPEAT=50 \
                 clientextra_install_params="--packages lipe-lpcc"
Reviewed-on: https://review.whamcloud.com/44792
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14924 osd-ldiskfs: fix T10PI verify/generate_fn

We are making wrong assumptions in the verify/generate_fn
of T10PI.

Consider this case: we have 4 pages lnb[0-3] in osd_iobuf.
lnb[2] is mapped to a hole, so it won't be added to bio.
If lnb[3] happens to be contiguous after lnb[1], lnb[3] will
be added to bio, with a bi_idx of 2.
In the verify/generate_fn, we work out which niobuf_local
to feed the guard tags to using bi_idx and obp_start_page_idx
and we will end up with wrong niobuf and set the guard tags
for lnb[2].

Contiguous blocks in bio doesn't necessarily mean we are looking
at contiguous niobuf_local/lnb in osd_iobuf->dr_lnbs

Lustre-change: https://review.whamcloud.com/44548
Lustre-commit: 92fc8708a2d5b20e317a2cec97056c752a8b7bd4

Test-Parameters: env=ONLY=77n testlist=sanity
Change-Id: I1ea1b6498692044e680c8754cd31e2c2b7bc9539
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44941
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15008 build: fix RHEL8.4 detection for lipe

The following error was generated by detect-distro.sh:

   I don't know what distro name RedHatEnterprise and version 8.4 is.

while detect-distro.sh is looking for RedhatEnterpriseServer.

Test-Parameters: trivial
Fixes: 37de8547a945 ("EX-3723 build: build lipe-lpcc on client")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ida0c16e6067ca3d736e35c889ac18940c43ebbe5
Reviewed-on: https://review.whamcloud.com/44983
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn16

LU-14323 tests: skip sanity-flr/pfl tests for older servers

sanity-flr test 46 sub tests 7, 8, 9 and 10 and sanity-pfl
test 16c were added to lustre-master version 2.13.53.205.
When we run version interop testing, these sanity-flr and
sanity-pfl tests will fail.  Thus skip sanity-flr test 46
subtests 7, 8, 9, and 10 and sanity-pfl test 16c when run
with servers with version less than 2.13.53.205 and clients
with later version.

Lustre-change: https://review.whamcloud.com/44494
Lustre-commit: 0442eb6205344c5c907c63df77ad09bc0a6f89a6

Fixes: ee916af10de2 (“LU-13366 utils: SEL yaml and copy file support “)

Test-Parameters: trivial
Test-Parameters: env=ONLY=46 testlist=sanity-flr
Test-Parameters: env=ONLY=16 testlist=sanity-pfl

Test-Parameters: serverversion=2.12.6-ddn42 serverdistro=el7.9 env=ONLY=46 testlist=sanity-flr
Test-Parameters: serverversion=2.12.6-ddn42 serverdistro=el7.9 env=ONLY=16 testlist=sanity-pfl

Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I09b88351a10891f63dceb9a2a74c92e4fffc13c5
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44977
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14880 libcfs: Use crypto/sha2.h if available

As of Linux commit a24d22b225ce158651378869a6b88105c4bdb887
   crypto: sha - split sha.h into sha1.h and sha2.h

sha.h is removed and sha2.h or sha3.h is preferred.

Lustre-change: https://review.whamcloud.com/44373
Lustre-commit: e9cffb256d10a3bb28c957f0bf06c61793a9610b

Test-Parameters: trivial
Fixes: a813e81870 ("LU-12275 sec: add llcrypt as file encryption library")
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: Iead5e1cb23e79a400da3cbfeb4c35c834e821d62
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-on: https://review.whamcloud.com/44996
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-14881 libcfs: Complete testing for tcp_sock_set_*

Linux commits:
  v5.7-rc6-2504-gddd061b8daed
  tcp: add tcp_sock_set_quickack

  v5.7-rc6-2508-gd41ecaac903c
  tcp: add tcp_sock_set_keepintvl

  v5.7-rc6-2509-g480aeb9639d6
  tcp: add tcp_sock_set_keepcnt

Introduced a series of helper functions that may be
back ported individually.

Lustre-change: https://review.whamcloud.com/44374
Lustre-commit: bbd9646f91fd13ea84f730296f3ce6983030aea2

Test-Parameters: trivial
Fixes: 99d9638d6c ("LU-13783 libcfs: support removal of kernel_setsockopt()")
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I4fce67b801979ec7857265b6bd0370c05737e268
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Aurelien Degremont <degremoa@amazon.com>
Reviewed-on: https://review.whamcloud.com/44995
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

EX-3658 pumount: refine exit statuses

   0  Unmount succeeded and no processes using FS remained.
   1  Unmount succeeded but one or more processes using FS remained.
   2  Unmount failed or some other major error occurred.
   3  An invalid option or argument was supplied.

Add sanity-pumount 57a-c to verify.

Test-Parameters: trivial testlist=sanity-pumount clientextra_install_params="--packages pumount"
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I7d0d67129af08d5c56ab5e676d4d92b3a14f2d9d
Reviewed-on: https://review.whamcloud.com/44758
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44958
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn15

New tag 2.14.0-ddn15

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I040766fd6ce457bfb02edbeaae9178fbbe0ae913

EX-3658 pumount: correct path escaping

Correct the test for whether a character needs escaping in
pu_str_escape2(). Add sanity-pumount 55f-g to verify.

Test-Parameters: trivial testlist=sanity-pumount clientextra_install_params="--packages pumount"
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I248f56752b30836636734f43d16741511717709c
Reviewed-on: https://review.whamcloud.com/44759
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44959
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-14917 llite: Switch mmap readahead logic

The mmap readahead logic has shown to be badly suited for
certain workloads (kdb).  Experimentation showed that for
these workloads, the standard readahead algorithm works
better.

This patch allows switching the readahead algorithm for
mmap to the 'standard' readahead algorithm.

The added tunable is:
mmap_read_ahead_logic in llite
(lctl get_param llite.*.mmap_read_ahead_logic)

It defaults to '1', which is the special mmap readahead
logic.

Setting it to 0 switches mmap readahead to use the standard
readahead logic.

This patch also fixes the existing mmap readahead test,
which was not running.

Note the results for the existing test are better with the
standard readahead logic, ie, with mmap readahead logic
disabled.  This suggests we should default to non-mmap
readahead logic.

However, the mmap readahead logic was carefully tuned for
certain workloads, and this would be a large change.  So,
this question is deferred until we can do a larger look at
readahead behavior.

NOTE:
Fixing the mmap readahead tests revealed that mmap
readahead is not working well currently.  Fixing that is
complicated and would delay landing this patch (needed for
a customer), so this patch disables the failing portion of
the tests.

Lustre-change: https://review.whamcloud.com/44526
Lustre-commit: TBD (7f6e16938f8d4e5dadd0594b7e1b8f4c70d79c68)

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I36c5a3ccfc34379d75666bb41605344a4ae70881
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44943
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-14949 llite: Always do lookup on ENOENT in open

When there is no valid dentry found for a file we want to
open, we perform a full lookup, which goes to the server
and looks up the file by name. When we find an existing
dentry in cache *but the file is not open on the node*, we
do not do a full lookup.  We move directly to opening the
file.

When we open files, we use the FID of the file.  The
problem occurs when a new file is renamed *over* the file
we were trying to open.  This removes the FID we are
trying to open, but the file *name* userspace called open()
on is still present.  In this case, we will return ENOENT,
even though there is a file matching the name used in the
open() call.

The solution is when we get an ENOENT on open (indicating
our open raced with an unlink), we always send ESTALE back
to the VFS, which restarts the open and forces a lookup to
the server (by forcing Lustre to consider the dentry
invalid, see comments in ll_intent_file_open and code in
ll_revalidate_dentry).

This causes a lookup by name, which will correctly handle
the rename, allowing the open to proceed normally.

This should only generate extra retries in the case where a
positive dentry exists on the client but the file has been
removed on the server, ie, open racing with unlink.

This should hopefully be rare.

Lustre-change: https://review.whamcloud.com/44675
Lustre-commit: 72c1f7095203cc1badadf581c66f9546476438ab

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Ice19f4bdbea9a2cbeb337336f7e7098afa6b4be3
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-on: https://review.whamcloud.com/44949
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-14782 kernel: new kernel [SLES15 SP3 5.3.18-59.19.1]

This patch makes changes to support new SLES15 SP3 release
with kernel 5.3.18-59.19.1 for Lustre client.

Test-Parameters: trivial

Lustre-change: https://review.whamcloud.com/44062
Lustre-commit: 0a900c5b4a28c4c63055dde00c7b91e39c4912a0

Change-Id: Idf6fad9773dd242c02859a5c7b14401675c4ecf4
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44976
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-3723 build: build lipe-lpcc on client

Introduce config param --disable-server to build
rpms that needed for lustre client.

Test-Parameters: trivial

Change-Id: If1812b17f993ebf4031cae30303dda22c179bdc9
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44835
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15008 kernel: kernel update RHEL8.4 [4.18.0-305.19.1.el8_4]

Update RHEL8.4 kernel to 4.18.0-305.19.1.el8_4 for Lustre client.

Test-Parameters: trivial clientdistro=el8.4

Change-Id: Icedc6cf2a5678cfbce76c47507137c0ea41d0b06
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44953
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14489 utils: fix 'lfs find --mdt-count'

Running "lfs find --mdt-count" causes the find to exit if there
is no directory striping, rather than continuing to the next item.

If cb_get_dirstripe() receives ENODATA then it should consider
that directory as not having any striping and move on, rather
than returning this error to the caller.

Don't crash in cb_getdirstripe() if it is called with a NULL
directory pointer or no directory is opened.

Lustre-change: https://review.whamcloud.com/43866
Lustre-commit: baba1fd07a977a62295482919e9218f877c0535a

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: If8dd135a86a6a8911bf804542132b2e7a3ce7057
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Reviewed-on: https://review.whamcloud.com/44945
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-11776 utils: add support lfs find with mdt hash flag

The lfs find can use mdt hash flag as a condition. Also
change it can find with one more mdt hash type.

Lustre-change: https://review.whamcloud.com/39340
Lustre-commit: 00141b1a746d4733c2f52c7a7edec36da4cedcac

Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: I599bb1a3cc2c9ea2a523f50f119bd93a5520d213
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44944
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-14780 llite: failed ASSERTION(ldlm_has_layout(lock))

When setting layout in layout lock, the lock could lost its layout
bits, and we'd try fetch the layout lock again.

Lustre-change: https://review.whamcloud.com/44054
Lustre-commit: 1b166d6dd6a2f39dfe35b60be169b288665d0283

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I10f96e4cb03cfe228d3c1ea1500b1a8d8e4e5e54
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44934
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-3342 tests: correct Lustre version in test skip checks

Many patches land to the EXAScaler branches as ports from
other branches.  Sometimes the tests that are included with
the ported patches check the version of Lustre to ensure
that the feature it tests exists in this version of Lustre.
These version values are not always changed when patches
are ported from one branch to another.

Change Lustre test suite version checks to be relative to
this branch.

Fixes: 5cfcd52d (“LU-13417 mdd: set default LMV on ROOT”)
Fixes: c75e68e5 (“LU-14804 nodemap: do not return error for improper ACL”)
Fixes: f2d1c4ee (“LU-14647 flr: mmap write/punch does not stale other mirrors”)
Fixes: 86847243 (“LU-13730 lod: don't confuse stale with primary flag”)

Test-Parameters: trivial env=ONLY="0 432" testlist=sanity
Test-Parameters: env=ONLY="50 207" testlist=sanity-flr

Change-Id: I9f1f0c1b89d5df7082e4fcaee385b724a453f331
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44906
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14967 obdclass: EAGAIN after rhashtable_walk_next()

rhashtable_walk_next() can return -EAGAIN when concurrent resizing
has happened. so the callers should check for this error and just
repeat rhashtable_walk_next().

Lustre-change: https://review.whamcloud.com/44766
Lustre-commit: 96aa615f91cd25b04c393f16f122e33f6744fdc9

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I15ba2cdf16c2678e18836b4f16b56a3b8bfdacd0
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Aurelien Degremont <degremoa@amazon.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Neil Brown <neilb@suse.de>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44937
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14934 kernel: kernel update SLES12 SP5 [4.12.14-122.83.1]

Update SLES12 SP5 kernel to 4.12.14-122.83.1 for Lustre client.

Test-Parameters: trivial clientdistro=sles12sp5 \
env=SANITY_EXCEPT="56oc 160h 430c 817" testlist=sanity

Change-Id: I2b35d129550b895324bb3e2e61910ad10e846f03
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44865
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-3814 pcc: print help msg more clearly for detach

In this patch, it prints the help message for detach_fid and
detach commands more clearly when not given the required
parameters such as mount point or FIDs.

It also ignores the -EINPROGRESS error if the file is being
attached, i.e. copy data from Lustre OSTs into PCC.

Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I547e80e5c9c213b159039b9b79da176cdb91c4bc
Reviewed-on: https://review.whamcloud.com/44954
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14895 client: allow case-insensitive checksum types

The current t10ip4K and t10crc4K checksum types use an upper-case 'K'
in the name, unlike the other checksum types which are all lower-case.
This is distinction is difficult to see in some fonts, and can cause
usage errors.  Accept upper-case variants of the checksum type names.

Lustre-change: https://review.whamcloud.com/44530
Lustre-commit: TBD (from 48a8218fdd0d0ed876fb39d29542fd1751c2e341)

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I97673ffa98cf8e5fc601ac7df5aaafb24b3ebbe5
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44940
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14806 o2iblnd: clear fatal error on successful failover

In IB bonding configuration link down event causes fatal error
flag to be set on the bonded interface so it is not selected by
LNet for tx, e.g. when just one of the two cables is pulled.
This change allows for the interface status to be restored on
successful failover.

Lustre-change: https://review.whamcloud.com/44139
Lustre-commit: 4668283cd13079dd6d86482704aef593f5c01dff

Test-Parameters: testlist=sanity-lnet
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: Ifd55b141e73d01a187c02ede3f021f0eab18e0bb
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44933
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14733 o2iblnd: Avoid double posting invalidate

When the kib_tx is provisioned during kiblnd_fmr_pool_map(), spare
WRs in the kib_fast_reg_descriptor are setup and the mapping of
pages is given to the mr.

kiblnd_post_tx_locked() then posts the spare WRs from the
kib_fast_reg_descriptor.

if (rc == 0)
return 0;

The code returns and the kib_fast_reg_descriptor is still contains
the spare WRs.   The next time the kib_tx is used, the
now obsolete WRs will be inadvertently posted.   For rdmavt, the
obsolete invalidate will cause an -EINVAL to be returned from
the post send.

Fix by adding a state variable frd_posted to kib_fast_reg_descriptor.
The variable is set to false in kiblnd_fmr_pool_unmap().
kiblnd_post_tx_locked() is adjusted to avoid prepending the
kib_fast_reg_descriptor WRs when frd_posted is true.   After
the post succeeds, the frd_posted is set to true.

Lustre-change: https://review.whamcloud.com/44190
Lustre-commit: 5930576791e864529e6ef9b46f3e09cc4b635fc2

Signed-off-by: Mike Marciniszyn <mike.marciniszyn@cornelisnetworks.com>
Change-Id: I426dd05e635392e75d1aa48808782a229e83ce5f
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44932
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14733 o2iblnd: Move racy NULL assignment

kiblnd_fmr_pool_unmap() can race map and subsequent processing
because of this flaw in unmap:

if (frd) {
frd->frd_valid = false;
spin_lock(&fps->fps_lock);
list_add_tail(&frd->frd_list, &fpo->fast_reg.fpo_pool_list);
spin_unlock(&fps->fps_lock);
fmr->fmr_frd = NULL;
}

The fmr can be pulled off the list in kiblnd_fmr_pool_unmap() on
another CPU an fmr_frd could be in a state of flux and
potentially be seen incorrectly later on as the kib_tx is processed.

Fix my moving the fmr_frd assignment to before the fmr is added to the
list.

Lustre-change: https://review.whamcloud.com/44189
Lustre-commit: 023113fb8946f3565529e7327fdcd90ab9db3ba3

Signed-off-by: Mike Marciniszyn <mike.marciniszyn@cornelisnetworks.com>
Change-Id: Ibddf132a363ecfe9db3cc06287cec873c021d2fb
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44931
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-12577 llog: protect partial updates from readers

llog_osd_write_rec() adds a record in few steps: the header is
updated first, then the record itself is appended. per-loghandle
semaphore is used, but remote readers allocate a new separate
loghandle for every access (header reading, blocks), the the
readers can't use loghandle's semaphore to avoid accessing partial
updates. use object-based locking [censored] to serialize the writer
vs the readers.

Lustre-change: https://review.whamcloud.com/43589
Lustre-commit: 03dd1bb036d426a692584d73f66bcdb221658d79

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Ie4e4d4a1e9a6fcdea9fcca7d80b0da920e786424
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44935
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14877 llite: Remove inode locking in ll_fsync

It does not appear to be necessary

Lustre-change: https://review.whamcloud.com/44368
Lustre-commit: e8d76d1090e912ee5d916284ca5c8ba9195ddd9b

Change-Id: I0142a9dca4ecc6893521275b69a0a46012eab0b0
Fixes: 8f3ef1e961 ("LU-812 llite: 3.0+ kernel fsync should call write")
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44921
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>

LU-14752 obdclass: handle EBUSY returned for lu_object hashtable

When the rhashtable grows to a certain size it will be rescaled.
When rescaling you can be returned a ENOMEM or EBUSY error. This
we reported as:

LustreError: 3594004:0:(lu_object.c:2472:lu_object_assign_fid()) ASSERTION( rc == 0 ) failed: failed hashtable insertion: rc = -16
LustreError: 3594004:0:(lu_object.c:2472:lu_object_assign_fid()) LBUG
Pid: 3594004, comm: mdt01_020 4.18.0-240.22.1.1toss.t4.x86_64 #1 SMP Tue Apr 13 17:18:40 PDT 2021
Call Trace TBD:
Kernel panic - not syncing: LBUG
...
Call Trace:
dump_stack+0x5c/0x80
panic+0xe7/0x2a9
lbug_with_loc.cold.10+0x18/0x18 [libcfs]
lu_object_assign_fid+0x3b8/0x3c0 [obdclass]

Add handling the EBUSY case for our lu_object hash.

Lustre-change: https://review.whamcloud.com/43968
Lustre-commit: 285a29d3b5e47f63a94c0682040ddbf09614f130

Fixes: aff14dbc522 ("LU-8130 lu_object: convert lu_object cache to rhashtable")
Change-Id: Id85f32633117e02850b799e8d95e3e35d982cbd4
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Gian-Carlo DeFazio <defazio1@llnl.gov>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44926
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-13997 tests: sanity/418 to cancel all client locks

verify idea about dirty client's data

Lustre-change: https://review.whamcloud.com/44803
Lustre-commit: TBD (from 42a090928b36bc14d8cb9a73a7f5b719eff38a2e)

Test-Parameters: trivial
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Ifef58a98b26c7790274d2a57aa52e4475e923dd0
Reviewed-on: https://review.whamcloud.com/44936
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-2859 pcc: keep mtime unchange when attach file into PCC

Modifying the timestamps of the files for the attach will cause
problems for the cache manager, since all files will appear new
at the time they are imported into the cache.
And that may also confuse applications if the file mtime has
changed just because of attach.

In this patch, it keeps the file mtime unchanged when attach it
into PCC.

Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I26f0c19c7b6cc1af0d62c192931d0042c9614993
Reviewed-on: https://review.whamcloud.com/44909
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-3804 lod: use kstrtoint_from_user()

which checks the address to be presented in memory

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I573ac593a8886caf6ee49c285674aed870eb6b2f
Reviewed-on: https://review.whamcloud.com/44928
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-2240 lipe: remove fake lustre headers

The fake lustre headers should be removed and we should include
normal lustre headers instead. The two headers are fake_lustre_idl.h
and fake_lustre_disk.h. Including the fake headers makes it harder
to include other lustre headers due to conflicts.

Test-Parameters: testlist=sanity-lipe

Change-Id: I95d6b50c9f9fd9f675eceba00464053124b7279c
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44897
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-13055 changelog: use default mask if server has no mask

When registering a new maskless user and server has no specific
mask set then effective mask to be set to DEFAULT value

Lustre-commit: 1c91131941b3c02c60c2dc852b23490dce3f2485
Lustre-change: https://review.whamcloud.com/44404

Fixes: a15eb4f132 ("LU-13055 mdd: per-user changelog names and mask")
Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: If799cb5cc29c60cce6ef6c987f2e493145e00e31
Reviewed-on: https://review.whamcloud.com/44411
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-3764 pcc: avoid panic in asynchronous attach thread

When PCC attach a file asynchronously, it wrongly uses
@pccx->pccx_file after put the file in @pcc_attach_context_free().
This may casue panic in pcc_readonly_attach_thread().
This patch fixes it by using @pccx->pccx_file before put it.

Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: Iaa93403e4db7497923033d327e689627790fa6a0
Reviewed-on: https://review.whamcloud.com/44881
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14767 utils: mkfs.lustre allow lazy_itable_init=1

When "lazy_itable_init=0" was added to the mke2fs options the call
to append_unique() to see whether "lazy_itable_init" was already
listed in the mke2fs options was incorrect. It checks to see if
"lazy_itable_init=0" is already present in the options, and doesn't
match "lazy_itable_init=1" if it was specified on the command-line.

Separate the key and value passed to append_unique() so that it can
check if any form of the key is present in the existing options.

Lustre-change: https://review.whamcloud.com/44019
Lustre-commit: a81c093a935c62b9e4586ae930aab7439948d538

Test-Parameters: trivial testlist=conf-sanity
Fixes: 701cc249594e ("LU-13533 utils: ext4lazyinit should be disabled")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ic7a6dbb81f004dd35f0f1c5f5ddec0fb363ebbe5
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Artem Blagodarenko <artem.blagodarenko@hpe.com>
Reviewed-on: https://review.whamcloud.com/44920
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-3587 build: Use explicit python2

Use explicit python2 since lipe is not
ready to move to python3 yet

Change-Id: I289775f522d0f0e284fc8cfba3ca1737f3e27c79
Test-Parameters: trivial
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44503
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-8837 utils: move lustre_disk_data back to lustre_disk.h

This patch moves struct lustre_disk_data from mount_utils.h
back to lustre_disk.h so that it can be used in other codes
without including mount_utils.h.

Lustre-change: https://review.whamcloud.com/44829
Lustre-commit: TBD

Fixes: d62efba975d2 ("LU-8837 utils: make tools lightweight for lustre clients")
Change-Id: I589da2710e3cbe7d93a59928143f2b5cac955e6e
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44892
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

EX-3050 lipe: lpcc_purge supports version and revision

Print unified lipe version and revision in lpcc_purge --version command
and stats dumpfile.

Change-Id: I78e500d4f765b638662a90f21f4e5d7ebd2209e2
Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial
Reviewed-on: https://review.whamcloud.com/44699
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-3541 lipe: aggregate LPCC information in lpcc status command

Aggregate LPCC backend information in lpcc status command. Including
the configuration of LPCC, lpcc_purge and stats of lpcc_purge.

Change-Id: I6fd394038b0b9b6279a592bd324b76f90585808e
Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial
Reviewed-on: https://review.whamcloud.com/44696
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-3554 lipe: improve log messages for lpcc_purge

lpcc service handle log_level option for lpcc_purge.
Add more log messages for lpcc_purge.

Change-Id: I1fd41d32cc6add00acea60c38985774cd5b7071a
Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial
Reviewed-on: https://review.whamcloud.com/44519
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn14

New tag 2.14.0-ddn14

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I96f78051cb1425adc5b6188a7c458e91678134f0

EX-3780 lipe: clean up specfile

Clean up specfile error due to previous merge

Change-Id: I1dc33310027286860c446b9eb0cc01c16b2c5407
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44891
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn13

New tag 2.14.0-ddn13

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Iaa7a9b6e6dab2fbd4704a4bb8b737921270a5fe7

LU-14993 kernel: kernel update RHEL8.4 [4.18.0-305.17.1.el8_4]

Update RHEL8.4 kernel to 4.18.0-305.17.1.el8_4 for Lustre client.

Test-Parameters: trivial clientdistro=el8.4

Change-Id: I95e97e1b39e8c49a80f12c9c1b2076553c3dcd49
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44874
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14994 kernel: kernel update RHEL7.9 [3.10.0-1160.42.2.el7]

Update RHEL7.9 kernel to 3.10.0-1160.42.2.el7.

Test-Parameters: trivial clientdistro=el7.9 serverdistro=el7.9

Change-Id: I377ea5d1e28c50b1087dfca7cb32f44afb9bf5f5
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44879
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-2921 lipe: merge lipe changes from b_es5_2

Merge commit 'd4e8316dafcd4c3cdb5092dc6f2a857dc28065fa' into b_es6_0

$ git checkout b_es5_2
$ git subtree split --prefix=lipe
6085b19ae7daa054857bf14d05740ff1224aef01
$ git checkout b_es6_0
$ git subtree merge --prefix=lipe --squash 6085b19ae7daa054857bf14d05740ff1224aef01

Change-Id: I4b8bfa69d312bfe93ad37da1737df9025c9ed0b5

Squashed 'lipe/' changes from b7b776f968..6085b19ae7

6085b19ae7 EX-3738 hotpools: Strict ordering of client and lamigo
a33bdd5d61 EX-3441 lipe: add pool spilling to stratagem-hp-{config,convert}.sh
4c16ba7e00 EX-3092 lipe: remove the lipe-hsm RPM
ef76926047 EX-3725 lipe: fix json.h include

git-subtree-dir: lipe
git-subtree-split: 6085b19ae7daa054857bf14d05740ff1224aef01

EX-3738 hotpools: Strict ordering of client and lamigo

Ensure strict ordering for start and stop of client mount
and lamigo/lpurge.
Ensure enough of a timeout for start and stop.

Test-Parameters: trivial
Signed-off-by: Nathaniel Clark <nclark@whamcloud.com>
Change-Id: I7c50abc5ff82f0cc3fd117fd961f7421ad2df9be
Reviewed-on: https://review.whamcloud.com/44807
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Gaurang Tapase <gtapase@ddn.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

EX-3430: Improvement in HP config for client mount

Use custom lustre-client resource agent for client
mount/unmount which in turn uses the pumount utility
to unmount. This makes sure the client is unmounted
gracefully while stopping the resource.

Remove stratagem-hp-convert.sh as it is no longer used.

Test-Parameters: trivial

Change-Id: Idc5fc2a59e4783a4011fc65a0ae3d69281b1a20f
Signed-off-by: Gaurang Tapase <gtapase@ddn.com>
Reviewed-on: https://review.whamcloud.com/44547
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-14677 sec: do not expose security.c to listxattr/getxattr

security.c xattr, which contains encryption context, should not be
exposed by the xattr-related system calls such as listxattr() and
getxattr() because of its special semantics.
Update sanity-sec test_57 to test this.

Lustre-change: https://review.whamcloud.com/44101
Lustre-commit: TBD (db49ec2ae2c96a09fae054e5fcb3d1959e26f83d)

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I919f5cbafc53f5745fbfb5b9d2d7316e892d8c9f
Reviewed-on: https://review.whamcloud.com/44183
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14677 llite: move env contexts to ll_inode_info level

Contrary to file, inode is always available, so move the list of
env contexts from the file data to the ll_inode_info level.
This is needed because we will have to handle env properties in
ll_get_context() and ll_xattr_list()/ll_listxattr().
This also requires changing lli_lock from a spinlock to an rwlock.

Lustre-change: https://review.whamcloud.com/44198
Lustre-commit: TBD (932007c91333117b7b0905ce5601aafc9b3bdd4e)

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I478d2a8eabfcb09074ba52601f05840d047a6da2
Reviewed-on: https://review.whamcloud.com/44199
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>