LU-14629 sec: do not block rename of topmost encrypted dir

We intentionally forbid file and directory rename from encrypted to
unencrypted directory. But we must not block rename of the topmost
encrypted directory.

Lustre-change: https://review.whamcloud.com/45054
Lustre-commit: 3db8e31797535570e5f2e99f6c8471b49d395efe

Fixes: 1158386ac9 ("LU-14629 sec: forbid file rename from enc to unencrypted dir")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I480a24b2b0327e1d9104f216da54720e4f351636
Reviewed-on: https://review.whamcloud.com/45085
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14989 sec: access to enc file's xattrs

Encryption context is stored in 'security.c' xattr. This is put in the
xattr cache via ll_xattr_cache_insert() to avoid sending a getxattr
request to the server. But this operation declares the xattr cache for
the inode as 'valid', with two consequences. It prevents any further
filling with other xattrs, and trying to read an xattr value will
directly return -ENODATA, without any attempt to fetch the xattr from
the server.
This is solved by adding a new ll_file_flags 'LLIF_XATTR_CACHE_FILLED'
that tells if the xattr cache for the inode has been filled. This bit
is set only by ll_xattr_cache_refill(), and 'valid' now just means the
xattr cache for the inode has been initialized.

Lustre-change: https://review.whamcloud.com/44855
Lustre-commit: 1faf54e8bf19c28a4de7b069309d607cb3f10f91

Fixes: 40d91eafe2 ("LU-12275 sec: atomicity of encryption context getting/setting")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I6c2b6870df29f26f048dedeb7212d1c801ca69e1
Reviewed-on: https://review.whamcloud.com/45084
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

DDN-2073 tests: fix sanity-sec test_15

sanity-sec test_15 should not inherit trusted and admin properties for
newly created nodemaps from the default nodemap. Instead, it must
explicitly set both to 0, as the whole point is to exercise id
mappings.

Test-Parameters: trivial testlist=sanity-sec
Test-Parameters: clientdistro=ubuntu2004 testlist=sanity-sec
Change-Id: Ie80a10528268024d59cd079949620ebd1a836395
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-on: https://review.whamcloud.com/45100
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15011 tests: pool spill test modifications

Make the following modifications to the ost-pools
test suite:
test 29 - change check for ‘when striping is specified
explicitly ‘ file from ‘file-2’ to ‘file-3’

test 30 - Add bad parameter check for setting the threshold
below zero

test 31 - ‘do_nodes $mdts $LCTL get_param lod.*.pool.*’
doesn’t print anything. Change to
‘do_nodes $mdts $LCTL get_param lod.*.pool.*.spill*’

Lustre-change: https://review.whamcloud.com/45056
Lustre-commit: TBD (from a097f6ea7e5a9514c08995d6ed721cba05bb1fb0)

Fixes: 0a998f4723 (“LU-14825 lod: pool spilling”)

Test-Parameters: trivial testlist=ost-pools
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: Icbdc3d42b7f7609bc57cc37830975d831125d659
Reviewed-on: https://review.whamcloud.com/45103
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-14781 osp: osp object header could be NULL

Don't call lu_object_header_fini upon NULL header in
osp_object_free().

Call trace:
lu_object_free.isra.30+0xf2/0x170 [obdclass]
lu_object_find_at+0x496/0x930 [obdclass]
lod_initialize_objects+0x3e4/0xba0 [lod]
lod_parse_striping+0x693/0xc20 [lod]
lod_striping_load+0x2b2/0x660 [lod]
lod_declare_destroy+0x12b/0x600 [lod]
mdd_declare_finish_unlink+0x91/0x210 [mdd]
mdd_unlink+0x48f/0xab0 [mdd]
mdt_reint_unlink+0xc32/0x1550 [mdt]
mdt_reint_rec+0x83/0x210 [mdt]
mdt_reint_internal+0x6e1/0xb00 [mdt]
mdt_reint+0x67/0x140 [mdt]
tgt_request_handle+0xaee/0x15f0 [ptlrpc]
ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
ptlrpc_main+0xb34/0x1470 [ptlrpc]
kthread+0xd1/0xe0

Lustre-change: https://review.whamcloud.com/44055/
Lustre-commit: 90dab62d635666001b8b4e2b7aa8cc1f23bc6f5b

Test-Parameters: trivial
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Iec23cf06dffaa64c6f5853c28382ba930ee1076b
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45101
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-3749 mdc: non blocking close and changelog release

Use interruptible RPCs in mdc_close() and the llog client
functions. In chlg_release() send a SIGKILL to the changelog producer
thread to wake from waiting on any RPC responses. Add sanity.sh test
160r to verify.

Lustre-change: https://review.whamcloud.com/44842
Lustre-commit: 83e0eb2be4fa61344137b2f50724a1ca1dc532ae

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I5f8b7cd839f5d6cf704ac07c7d583c45013921c4
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44960
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

EX-3937 lnet: check if ethtool.h is included in netdevice.h

Kernel 5.11.0-36 removes including linux/ethtool.h from
linux/netdevice.h, which caused the following build error:

dereferencing pointer to incomplete type ‘const struct ethtool_ops’

This patch fixes the above issue by checking if ethtool.h
is included in netdevice.h.

Change-Id: Ifc25de5acaebf2b5fd5bb6f1c303366ab9ea6ef6
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45097
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-15036 kernel: kernel update SLES15 SP3 [5.3.18-59.24.1]

Update SLES15 SP3 kernel to 5.3.18-59.24.1 for Lustre client.

Test-Parameters: trivial

Change-Id: I77987472f91afd9f695f988997e88233a1a5de99
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45061
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-10848 test: wait to process inodes in phase2

In test_8 of sanity-lfsck, the "LF_INCONSISTENT" flag was set
when processing the inodes with corrupted LinkEA in phase2,
LFSCK could have no chance to process it yet because of the delay
OBD_FAIL_LFSCK_DELAY3.

Lustre-change: https://review.whamcloud.com/44658
Lustre-commit: 4efcd6f9b8f9bb768c6167a880777413950619c4

Test-Parameters: trivial testlist=sanity-lfsck env=ONLY=8,ONLY_REPEAT=20
Change-Id: Id414728c998d527fbc27f877c6d31dcedcc12457
Signed-off-by: Hongchao Zhagn <hongchao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45013
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-3803 build: build lipe-lpcc on ubuntu 20.04 client

* only support client. need to use ./configure --disable-server

Test-Parameters: trivial

Change-Id: I83c7a712cc0c58dcadd5df59bb49e3fc3d85f9b1
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45095
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Feng, Lei <flei@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14729 osd-ldiskfs: declare should consider concurrency

Write in Lustre OSD is different than Ext4 since write
is serialized in local filesystem, however in OSD side,

many concurrent threads may grow tree before transaction starts.

Also fix to use @dirty_groups rather than @extents, remove
unnecessary @depth assignment.

Lustre-change: https://review.whamcloud.com/44316
Lustre-commit: 934758366dbde211743d53b8c048c57819603cb3

Fixes: 9810341a8 ("LU-14729 osd-ldiskfs: fix to declare write commits")
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I1e0fc9069a579736a74b0ba2607056fe980574c3
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45077
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

EX-3808 pcc: reset file mapping for pcc copy

The inode mapping of PCC copy will be replaced since mmap().
Then we must check & reset it while io starting. Else system
will crash on:
CPU: 3 PID: 31556 Comm: dd Kdump: loaded Tainted: P
Call Trace:
[<c1424463>] ll_readpage+0x4f3/0x870 [lustre]
[<a4bbf400>] generic_file_aio_read+0x3f0/0x790
[<c0418dc3>] ext4_file_read+0x43/0x130 [ext4]
[<c14564a3>] pcc_file_read_iter+0x323/0x390 [lustre]
[<c13f9ee2>] ll_file_aio_read+0x1f2/0x510 [lustre]
[<a4bfa59c>] ? mmap_region+0x20c/0x670
[<c13fa300>] ll_file_read+0x100/0x1c0 [lustre]
[<a4c4a6df>] vfs_read+0x9f/0x170
[<a4c4b59f>] SyS_read+0x7f/0xf0
[<a518de21>] ? system_call_after_swapgs+0xae/0x146
[<a518dede>] system_call_fastpath+0x25/0x2a
[<a518de21>] ? system_call_after_swapgs+0xae/0x146

Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: I719e9fac7362fe76f0ece399dda693b9e965c5bc
Reviewed-on: https://review.whamcloud.com/45052
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

EX-3846 lipe: backport lpcc script to python2

Since python2 is used by all other scripts in lipe,
backport lpcc to python2 too.

Change-Id: I5cffef7f4a38fe32ad071070076643814ca53736
Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial testlist=sanity-pcc env=ONLY=200-202,ONLY_REPEAT=50 \
                 clientextra_install_params="--packages lipe-lpcc"
Reviewed-on: https://review.whamcloud.com/45015
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Tested-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>

LU-14740 quota: reject invalid project id on server side

do sanity check before transfer project ID, reject invalid
project id if it comes from some older clients.

Lustre-change: https://review.whamcloud.com/44339
Lustre-commit: d6a3e06cb0f0db57a2637d029b1ff3bfd1de3d7d

Test-Parameters: trivial testlist=sanity-quota
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: If89e320c7808d188e615f5f0923c2322774b2ceb
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44946
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>

LU-14740 llite: avoid project quota overflow

Currently, project ID is stored as u32, max possible
value for it is 4294967295.

However, VFS reserve max value for special usage, see
following function:

  static inline bool
  qid_has_mapping(struct user_namespace *ns, struct kqid qid)
  {
          return from_kqid(ns, qid) != (qid_t) -1;
  }

So qid_has_mapping() could return 0 for id 4294967295.
A further try on chown test:

  $ chown 4294967295:4294967295 c.sh
  chown: invalid user: "4294967295:4294967295"
  $ chown 4294967294:4294967294 c.sh

Fix to check max possible value for project ID in the
client kernel side, and add a test case for this.

Lustre-change: https://review.whamcloud.com/43939
Lustre-commit: 3ffa5d680f0092ae51ffa84bd94a9983f9a8c99e

Test-Parameters: trivial testlist=sanity-quota
Fixes: 7b5c1f1404c3 ("LU-13845 utils: Quota id 0xFFFFFFFF is invalid")
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: Ide8b9cc79d9b7f2a8b9860a0c0f683ec903b8f91
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44938
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-11872 quota: add get/set project support for non-dir/file

Add ablity to get/set non-dir/file's project ID and state.

Lustre-change: https://review.whamcloud.com/44006
Lustre-commit: b31792b0e72425c8c7850d69837f08c9f3e95a9c

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: Ib8eee09254f9751797b5deb7f753c34eb2c0d5a5
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Reviewed-on: https://review.whamcloud.com/44922
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>

EX-3409 tests: handle pcc async attach better

Reset llite.*.pcc_async_threshold to the original value when the
subtest exits, so that it doesn't affect later tests.

Wait for async pcc attach to finish in test_47.

Test-Parameters: trivial testlist=sanity-pcc env=ONLY=46-47,ONLY_REPEAT=100
Test-Parameters: testlist=sanity-pcc
Test-Parameters: testlist=sanity-pcc
Test-Parameters: testlist=sanity-pcc
Test-Parameters: testlist=sanity-pcc
Test-Parameters: testlist=sanity-pcc
Test-Parameters: testlist=sanity-pcc
Test-Parameters: testlist=sanity-pcc
Test-Parameters: testlist=sanity-pcc
Test-Parameters: testlist=sanity-pcc
Test-Parameters: testlist=sanity-pcc
Fixes: 6c693d126197 ("EX-3764 pcc: avoid panic in asynchronous attach thread")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I9447fa1fc5ac20fd86340c2eda9bc08ba23ebbe5
Reviewed-on: https://review.whamcloud.com/44982
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>

EX-3720 tests: skip lpcc_purge tests if not installed

Skip lpcc_purge tests if package is not installed on client.

Test-Parameters: trivial testlist=sanity-pcc env=ONLY=200-202                  clientextra_install_params="--packages lipe-lpcc"
Test-Parameters: testlist=sanity-pcc env=ONLY=200-202
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ic977c1cad602cc1ae33b273493bfb9d0f9dbe37b
Reviewed-on: https://review.whamcloud.com/44994
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>

EX-3720 lipe: add test cases for lpcc_purge

add sanity test cases for lpcc_purge

Change-Id: I420f461f116c41767340329f02920118bf183673
Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial testlist=sanity-pcc env=ONLY=200-202,ONLY_REPEAT=50 \
                 clientextra_install_params="--packages lipe-lpcc"
Test-Parameters: trivial testlist=sanity-pcc env=ONLY=200-202,ONLY_REPEAT=50 \
                 clientextra_install_params="--packages lipe-lpcc"
Reviewed-on: https://review.whamcloud.com/44792
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14924 osd-ldiskfs: fix T10PI verify/generate_fn

We are making wrong assumptions in the verify/generate_fn
of T10PI.

Consider this case: we have 4 pages lnb[0-3] in osd_iobuf.
lnb[2] is mapped to a hole, so it won't be added to bio.
If lnb[3] happens to be contiguous after lnb[1], lnb[3] will
be added to bio, with a bi_idx of 2.
In the verify/generate_fn, we work out which niobuf_local
to feed the guard tags to using bi_idx and obp_start_page_idx
and we will end up with wrong niobuf and set the guard tags
for lnb[2].

Contiguous blocks in bio doesn't necessarily mean we are looking
at contiguous niobuf_local/lnb in osd_iobuf->dr_lnbs

Lustre-change: https://review.whamcloud.com/44548
Lustre-commit: 92fc8708a2d5b20e317a2cec97056c752a8b7bd4

Test-Parameters: env=ONLY=77n testlist=sanity
Change-Id: I1ea1b6498692044e680c8754cd31e2c2b7bc9539
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44941
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15008 build: fix RHEL8.4 detection for lipe

The following error was generated by detect-distro.sh:

   I don't know what distro name RedHatEnterprise and version 8.4 is.

while detect-distro.sh is looking for RedhatEnterpriseServer.

Test-Parameters: trivial
Fixes: 37de8547a945 ("EX-3723 build: build lipe-lpcc on client")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ida0c16e6067ca3d736e35c889ac18940c43ebbe5
Reviewed-on: https://review.whamcloud.com/44983
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn16

LU-14323 tests: skip sanity-flr/pfl tests for older servers

sanity-flr test 46 sub tests 7, 8, 9 and 10 and sanity-pfl
test 16c were added to lustre-master version 2.13.53.205.
When we run version interop testing, these sanity-flr and
sanity-pfl tests will fail.  Thus skip sanity-flr test 46
subtests 7, 8, 9, and 10 and sanity-pfl test 16c when run
with servers with version less than 2.13.53.205 and clients
with later version.

Lustre-change: https://review.whamcloud.com/44494
Lustre-commit: 0442eb6205344c5c907c63df77ad09bc0a6f89a6

Fixes: ee916af10de2 (“LU-13366 utils: SEL yaml and copy file support “)

Test-Parameters: trivial
Test-Parameters: env=ONLY=46 testlist=sanity-flr
Test-Parameters: env=ONLY=16 testlist=sanity-pfl

Test-Parameters: serverversion=2.12.6-ddn42 serverdistro=el7.9 env=ONLY=46 testlist=sanity-flr
Test-Parameters: serverversion=2.12.6-ddn42 serverdistro=el7.9 env=ONLY=16 testlist=sanity-pfl

Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I09b88351a10891f63dceb9a2a74c92e4fffc13c5
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44977
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14880 libcfs: Use crypto/sha2.h if available

As of Linux commit a24d22b225ce158651378869a6b88105c4bdb887
   crypto: sha - split sha.h into sha1.h and sha2.h

sha.h is removed and sha2.h or sha3.h is preferred.

Lustre-change: https://review.whamcloud.com/44373
Lustre-commit: e9cffb256d10a3bb28c957f0bf06c61793a9610b

Test-Parameters: trivial
Fixes: a813e81870 ("LU-12275 sec: add llcrypt as file encryption library")
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: Iead5e1cb23e79a400da3cbfeb4c35c834e821d62
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-on: https://review.whamcloud.com/44996
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-14881 libcfs: Complete testing for tcp_sock_set_*

Linux commits:
  v5.7-rc6-2504-gddd061b8daed
  tcp: add tcp_sock_set_quickack

  v5.7-rc6-2508-gd41ecaac903c
  tcp: add tcp_sock_set_keepintvl

  v5.7-rc6-2509-g480aeb9639d6
  tcp: add tcp_sock_set_keepcnt

Introduced a series of helper functions that may be
back ported individually.

Lustre-change: https://review.whamcloud.com/44374
Lustre-commit: bbd9646f91fd13ea84f730296f3ce6983030aea2

Test-Parameters: trivial
Fixes: 99d9638d6c ("LU-13783 libcfs: support removal of kernel_setsockopt()")
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I4fce67b801979ec7857265b6bd0370c05737e268
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Aurelien Degremont <degremoa@amazon.com>
Reviewed-on: https://review.whamcloud.com/44995
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

EX-3658 pumount: refine exit statuses

   0  Unmount succeeded and no processes using FS remained.
   1  Unmount succeeded but one or more processes using FS remained.
   2  Unmount failed or some other major error occurred.
   3  An invalid option or argument was supplied.

Add sanity-pumount 57a-c to verify.

Test-Parameters: trivial testlist=sanity-pumount clientextra_install_params="--packages pumount"
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I7d0d67129af08d5c56ab5e676d4d92b3a14f2d9d
Reviewed-on: https://review.whamcloud.com/44758
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44958
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn15

New tag 2.14.0-ddn15

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I040766fd6ce457bfb02edbeaae9178fbbe0ae913

EX-3658 pumount: correct path escaping

Correct the test for whether a character needs escaping in
pu_str_escape2(). Add sanity-pumount 55f-g to verify.

Test-Parameters: trivial testlist=sanity-pumount clientextra_install_params="--packages pumount"
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I248f56752b30836636734f43d16741511717709c
Reviewed-on: https://review.whamcloud.com/44759
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44959
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-14917 llite: Switch mmap readahead logic

The mmap readahead logic has shown to be badly suited for
certain workloads (kdb).  Experimentation showed that for
these workloads, the standard readahead algorithm works
better.

This patch allows switching the readahead algorithm for
mmap to the 'standard' readahead algorithm.

The added tunable is:
mmap_read_ahead_logic in llite
(lctl get_param llite.*.mmap_read_ahead_logic)

It defaults to '1', which is the special mmap readahead
logic.

Setting it to 0 switches mmap readahead to use the standard
readahead logic.

This patch also fixes the existing mmap readahead test,
which was not running.

Note the results for the existing test are better with the
standard readahead logic, ie, with mmap readahead logic
disabled.  This suggests we should default to non-mmap
readahead logic.

However, the mmap readahead logic was carefully tuned for
certain workloads, and this would be a large change.  So,
this question is deferred until we can do a larger look at
readahead behavior.

NOTE:
Fixing the mmap readahead tests revealed that mmap
readahead is not working well currently.  Fixing that is
complicated and would delay landing this patch (needed for
a customer), so this patch disables the failing portion of
the tests.

Lustre-change: https://review.whamcloud.com/44526
Lustre-commit: TBD (7f6e16938f8d4e5dadd0594b7e1b8f4c70d79c68)

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I36c5a3ccfc34379d75666bb41605344a4ae70881
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44943
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-14949 llite: Always do lookup on ENOENT in open

When there is no valid dentry found for a file we want to
open, we perform a full lookup, which goes to the server
and looks up the file by name. When we find an existing
dentry in cache *but the file is not open on the node*, we
do not do a full lookup.  We move directly to opening the
file.

When we open files, we use the FID of the file.  The
problem occurs when a new file is renamed *over* the file
we were trying to open.  This removes the FID we are
trying to open, but the file *name* userspace called open()
on is still present.  In this case, we will return ENOENT,
even though there is a file matching the name used in the
open() call.

The solution is when we get an ENOENT on open (indicating
our open raced with an unlink), we always send ESTALE back
to the VFS, which restarts the open and forces a lookup to
the server (by forcing Lustre to consider the dentry
invalid, see comments in ll_intent_file_open and code in
ll_revalidate_dentry).

This causes a lookup by name, which will correctly handle
the rename, allowing the open to proceed normally.

This should only generate extra retries in the case where a
positive dentry exists on the client but the file has been
removed on the server, ie, open racing with unlink.

This should hopefully be rare.

Lustre-change: https://review.whamcloud.com/44675
Lustre-commit: 72c1f7095203cc1badadf581c66f9546476438ab

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Ice19f4bdbea9a2cbeb337336f7e7098afa6b4be3
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-on: https://review.whamcloud.com/44949
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-14782 kernel: new kernel [SLES15 SP3 5.3.18-59.19.1]

This patch makes changes to support new SLES15 SP3 release
with kernel 5.3.18-59.19.1 for Lustre client.

Test-Parameters: trivial

Lustre-change: https://review.whamcloud.com/44062
Lustre-commit: 0a900c5b4a28c4c63055dde00c7b91e39c4912a0

Change-Id: Idf6fad9773dd242c02859a5c7b14401675c4ecf4
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44976
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-3723 build: build lipe-lpcc on client

Introduce config param --disable-server to build
rpms that needed for lustre client.

Test-Parameters: trivial

Change-Id: If1812b17f993ebf4031cae30303dda22c179bdc9
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44835
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15008 kernel: kernel update RHEL8.4 [4.18.0-305.19.1.el8_4]

Update RHEL8.4 kernel to 4.18.0-305.19.1.el8_4 for Lustre client.

Test-Parameters: trivial clientdistro=el8.4

Change-Id: Icedc6cf2a5678cfbce76c47507137c0ea41d0b06
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44953
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14489 utils: fix 'lfs find --mdt-count'

Running "lfs find --mdt-count" causes the find to exit if there
is no directory striping, rather than continuing to the next item.

If cb_get_dirstripe() receives ENODATA then it should consider
that directory as not having any striping and move on, rather
than returning this error to the caller.

Don't crash in cb_getdirstripe() if it is called with a NULL
directory pointer or no directory is opened.

Lustre-change: https://review.whamcloud.com/43866
Lustre-commit: baba1fd07a977a62295482919e9218f877c0535a

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: If8dd135a86a6a8911bf804542132b2e7a3ce7057
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Reviewed-on: https://review.whamcloud.com/44945
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-11776 utils: add support lfs find with mdt hash flag

The lfs find can use mdt hash flag as a condition. Also
change it can find with one more mdt hash type.

Lustre-change: https://review.whamcloud.com/39340
Lustre-commit: 00141b1a746d4733c2f52c7a7edec36da4cedcac

Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: I599bb1a3cc2c9ea2a523f50f119bd93a5520d213
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44944
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-14780 llite: failed ASSERTION(ldlm_has_layout(lock))

When setting layout in layout lock, the lock could lost its layout
bits, and we'd try fetch the layout lock again.

Lustre-change: https://review.whamcloud.com/44054
Lustre-commit: 1b166d6dd6a2f39dfe35b60be169b288665d0283

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I10f96e4cb03cfe228d3c1ea1500b1a8d8e4e5e54
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44934
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-3342 tests: correct Lustre version in test skip checks

Many patches land to the EXAScaler branches as ports from
other branches.  Sometimes the tests that are included with
the ported patches check the version of Lustre to ensure
that the feature it tests exists in this version of Lustre.
These version values are not always changed when patches
are ported from one branch to another.

Change Lustre test suite version checks to be relative to
this branch.

Fixes: 5cfcd52d (“LU-13417 mdd: set default LMV on ROOT”)
Fixes: c75e68e5 (“LU-14804 nodemap: do not return error for improper ACL”)
Fixes: f2d1c4ee (“LU-14647 flr: mmap write/punch does not stale other mirrors”)
Fixes: 86847243 (“LU-13730 lod: don't confuse stale with primary flag”)

Test-Parameters: trivial env=ONLY="0 432" testlist=sanity
Test-Parameters: env=ONLY="50 207" testlist=sanity-flr

Change-Id: I9f1f0c1b89d5df7082e4fcaee385b724a453f331
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44906
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14967 obdclass: EAGAIN after rhashtable_walk_next()

rhashtable_walk_next() can return -EAGAIN when concurrent resizing
has happened. so the callers should check for this error and just
repeat rhashtable_walk_next().

Lustre-change: https://review.whamcloud.com/44766
Lustre-commit: 96aa615f91cd25b04c393f16f122e33f6744fdc9

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I15ba2cdf16c2678e18836b4f16b56a3b8bfdacd0
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Aurelien Degremont <degremoa@amazon.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Neil Brown <neilb@suse.de>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44937
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14934 kernel: kernel update SLES12 SP5 [4.12.14-122.83.1]

Update SLES12 SP5 kernel to 4.12.14-122.83.1 for Lustre client.

Test-Parameters: trivial clientdistro=sles12sp5 \
env=SANITY_EXCEPT="56oc 160h 430c 817" testlist=sanity

Change-Id: I2b35d129550b895324bb3e2e61910ad10e846f03
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44865
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-3814 pcc: print help msg more clearly for detach

In this patch, it prints the help message for detach_fid and
detach commands more clearly when not given the required
parameters such as mount point or FIDs.

It also ignores the -EINPROGRESS error if the file is being
attached, i.e. copy data from Lustre OSTs into PCC.

Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I547e80e5c9c213b159039b9b79da176cdb91c4bc
Reviewed-on: https://review.whamcloud.com/44954
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14895 client: allow case-insensitive checksum types

The current t10ip4K and t10crc4K checksum types use an upper-case 'K'
in the name, unlike the other checksum types which are all lower-case.
This is distinction is difficult to see in some fonts, and can cause
usage errors.  Accept upper-case variants of the checksum type names.

Lustre-change: https://review.whamcloud.com/44530
Lustre-commit: TBD (from 48a8218fdd0d0ed876fb39d29542fd1751c2e341)

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I97673ffa98cf8e5fc601ac7df5aaafb24b3ebbe5
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44940
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14806 o2iblnd: clear fatal error on successful failover

In IB bonding configuration link down event causes fatal error
flag to be set on the bonded interface so it is not selected by
LNet for tx, e.g. when just one of the two cables is pulled.
This change allows for the interface status to be restored on
successful failover.

Lustre-change: https://review.whamcloud.com/44139
Lustre-commit: 4668283cd13079dd6d86482704aef593f5c01dff

Test-Parameters: testlist=sanity-lnet
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: Ifd55b141e73d01a187c02ede3f021f0eab18e0bb
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44933
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14733 o2iblnd: Avoid double posting invalidate

When the kib_tx is provisioned during kiblnd_fmr_pool_map(), spare
WRs in the kib_fast_reg_descriptor are setup and the mapping of
pages is given to the mr.

kiblnd_post_tx_locked() then posts the spare WRs from the
kib_fast_reg_descriptor.

if (rc == 0)
return 0;

The code returns and the kib_fast_reg_descriptor is still contains
the spare WRs.   The next time the kib_tx is used, the
now obsolete WRs will be inadvertently posted.   For rdmavt, the
obsolete invalidate will cause an -EINVAL to be returned from
the post send.

Fix by adding a state variable frd_posted to kib_fast_reg_descriptor.
The variable is set to false in kiblnd_fmr_pool_unmap().
kiblnd_post_tx_locked() is adjusted to avoid prepending the
kib_fast_reg_descriptor WRs when frd_posted is true.   After
the post succeeds, the frd_posted is set to true.

Lustre-change: https://review.whamcloud.com/44190
Lustre-commit: 5930576791e864529e6ef9b46f3e09cc4b635fc2

Signed-off-by: Mike Marciniszyn <mike.marciniszyn@cornelisnetworks.com>
Change-Id: I426dd05e635392e75d1aa48808782a229e83ce5f
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44932
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14733 o2iblnd: Move racy NULL assignment

kiblnd_fmr_pool_unmap() can race map and subsequent processing
because of this flaw in unmap:

if (frd) {
frd->frd_valid = false;
spin_lock(&fps->fps_lock);
list_add_tail(&frd->frd_list, &fpo->fast_reg.fpo_pool_list);
spin_unlock(&fps->fps_lock);
fmr->fmr_frd = NULL;
}

The fmr can be pulled off the list in kiblnd_fmr_pool_unmap() on
another CPU an fmr_frd could be in a state of flux and
potentially be seen incorrectly later on as the kib_tx is processed.

Fix my moving the fmr_frd assignment to before the fmr is added to the
list.

Lustre-change: https://review.whamcloud.com/44189
Lustre-commit: 023113fb8946f3565529e7327fdcd90ab9db3ba3

Signed-off-by: Mike Marciniszyn <mike.marciniszyn@cornelisnetworks.com>
Change-Id: Ibddf132a363ecfe9db3cc06287cec873c021d2fb
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44931
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-12577 llog: protect partial updates from readers

llog_osd_write_rec() adds a record in few steps: the header is
updated first, then the record itself is appended. per-loghandle
semaphore is used, but remote readers allocate a new separate
loghandle for every access (header reading, blocks), the the
readers can't use loghandle's semaphore to avoid accessing partial
updates. use object-based locking [censored] to serialize the writer
vs the readers.

Lustre-change: https://review.whamcloud.com/43589
Lustre-commit: 03dd1bb036d426a692584d73f66bcdb221658d79

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Ie4e4d4a1e9a6fcdea9fcca7d80b0da920e786424
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44935
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14877 llite: Remove inode locking in ll_fsync

It does not appear to be necessary

Lustre-change: https://review.whamcloud.com/44368
Lustre-commit: e8d76d1090e912ee5d916284ca5c8ba9195ddd9b

Change-Id: I0142a9dca4ecc6893521275b69a0a46012eab0b0
Fixes: 8f3ef1e961 ("LU-812 llite: 3.0+ kernel fsync should call write")
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44921
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>

LU-14752 obdclass: handle EBUSY returned for lu_object hashtable

When the rhashtable grows to a certain size it will be rescaled.
When rescaling you can be returned a ENOMEM or EBUSY error. This
we reported as:

LustreError: 3594004:0:(lu_object.c:2472:lu_object_assign_fid()) ASSERTION( rc == 0 ) failed: failed hashtable insertion: rc = -16
LustreError: 3594004:0:(lu_object.c:2472:lu_object_assign_fid()) LBUG
Pid: 3594004, comm: mdt01_020 4.18.0-240.22.1.1toss.t4.x86_64 #1 SMP Tue Apr 13 17:18:40 PDT 2021
Call Trace TBD:
Kernel panic - not syncing: LBUG
...
Call Trace:
dump_stack+0x5c/0x80
panic+0xe7/0x2a9
lbug_with_loc.cold.10+0x18/0x18 [libcfs]
lu_object_assign_fid+0x3b8/0x3c0 [obdclass]

Add handling the EBUSY case for our lu_object hash.

Lustre-change: https://review.whamcloud.com/43968
Lustre-commit: 285a29d3b5e47f63a94c0682040ddbf09614f130

Fixes: aff14dbc522 ("LU-8130 lu_object: convert lu_object cache to rhashtable")
Change-Id: Id85f32633117e02850b799e8d95e3e35d982cbd4
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Gian-Carlo DeFazio <defazio1@llnl.gov>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44926
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-13997 tests: sanity/418 to cancel all client locks

verify idea about dirty client's data

Lustre-change: https://review.whamcloud.com/44803
Lustre-commit: TBD (from 42a090928b36bc14d8cb9a73a7f5b719eff38a2e)

Test-Parameters: trivial
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Ifef58a98b26c7790274d2a57aa52e4475e923dd0
Reviewed-on: https://review.whamcloud.com/44936
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-2859 pcc: keep mtime unchange when attach file into PCC

Modifying the timestamps of the files for the attach will cause
problems for the cache manager, since all files will appear new
at the time they are imported into the cache.
And that may also confuse applications if the file mtime has
changed just because of attach.

In this patch, it keeps the file mtime unchanged when attach it
into PCC.

Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I26f0c19c7b6cc1af0d62c192931d0042c9614993
Reviewed-on: https://review.whamcloud.com/44909
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-3804 lod: use kstrtoint_from_user()

which checks the address to be presented in memory

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I573ac593a8886caf6ee49c285674aed870eb6b2f
Reviewed-on: https://review.whamcloud.com/44928
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-2240 lipe: remove fake lustre headers

The fake lustre headers should be removed and we should include
normal lustre headers instead. The two headers are fake_lustre_idl.h
and fake_lustre_disk.h. Including the fake headers makes it harder
to include other lustre headers due to conflicts.

Test-Parameters: testlist=sanity-lipe

Change-Id: I95d6b50c9f9fd9f675eceba00464053124b7279c
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44897
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-13055 changelog: use default mask if server has no mask

When registering a new maskless user and server has no specific
mask set then effective mask to be set to DEFAULT value

Lustre-commit: 1c91131941b3c02c60c2dc852b23490dce3f2485
Lustre-change: https://review.whamcloud.com/44404

Fixes: a15eb4f132 ("LU-13055 mdd: per-user changelog names and mask")
Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: If799cb5cc29c60cce6ef6c987f2e493145e00e31
Reviewed-on: https://review.whamcloud.com/44411
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-3764 pcc: avoid panic in asynchronous attach thread

When PCC attach a file asynchronously, it wrongly uses
@pccx->pccx_file after put the file in @pcc_attach_context_free().
This may casue panic in pcc_readonly_attach_thread().
This patch fixes it by using @pccx->pccx_file before put it.

Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: Iaa93403e4db7497923033d327e689627790fa6a0
Reviewed-on: https://review.whamcloud.com/44881
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14767 utils: mkfs.lustre allow lazy_itable_init=1

When "lazy_itable_init=0" was added to the mke2fs options the call
to append_unique() to see whether "lazy_itable_init" was already
listed in the mke2fs options was incorrect. It checks to see if
"lazy_itable_init=0" is already present in the options, and doesn't
match "lazy_itable_init=1" if it was specified on the command-line.

Separate the key and value passed to append_unique() so that it can
check if any form of the key is present in the existing options.

Lustre-change: https://review.whamcloud.com/44019
Lustre-commit: a81c093a935c62b9e4586ae930aab7439948d538

Test-Parameters: trivial testlist=conf-sanity
Fixes: 701cc249594e ("LU-13533 utils: ext4lazyinit should be disabled")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ic7a6dbb81f004dd35f0f1c5f5ddec0fb363ebbe5
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Artem Blagodarenko <artem.blagodarenko@hpe.com>
Reviewed-on: https://review.whamcloud.com/44920
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-3587 build: Use explicit python2

Use explicit python2 since lipe is not
ready to move to python3 yet

Change-Id: I289775f522d0f0e284fc8cfba3ca1737f3e27c79
Test-Parameters: trivial
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44503
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-8837 utils: move lustre_disk_data back to lustre_disk.h

This patch moves struct lustre_disk_data from mount_utils.h
back to lustre_disk.h so that it can be used in other codes
without including mount_utils.h.

Lustre-change: https://review.whamcloud.com/44829
Lustre-commit: TBD

Fixes: d62efba975d2 ("LU-8837 utils: make tools lightweight for lustre clients")
Change-Id: I589da2710e3cbe7d93a59928143f2b5cac955e6e
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44892
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

EX-3050 lipe: lpcc_purge supports version and revision

Print unified lipe version and revision in lpcc_purge --version command
and stats dumpfile.

Change-Id: I78e500d4f765b638662a90f21f4e5d7ebd2209e2
Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial
Reviewed-on: https://review.whamcloud.com/44699
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-3541 lipe: aggregate LPCC information in lpcc status command

Aggregate LPCC backend information in lpcc status command. Including
the configuration of LPCC, lpcc_purge and stats of lpcc_purge.

Change-Id: I6fd394038b0b9b6279a592bd324b76f90585808e
Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial
Reviewed-on: https://review.whamcloud.com/44696
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-3554 lipe: improve log messages for lpcc_purge

lpcc service handle log_level option for lpcc_purge.
Add more log messages for lpcc_purge.

Change-Id: I1fd41d32cc6add00acea60c38985774cd5b7071a
Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial
Reviewed-on: https://review.whamcloud.com/44519
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn14

New tag 2.14.0-ddn14

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I96f78051cb1425adc5b6188a7c458e91678134f0

EX-3780 lipe: clean up specfile

Clean up specfile error due to previous merge

Change-Id: I1dc33310027286860c446b9eb0cc01c16b2c5407
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44891
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn13

New tag 2.14.0-ddn13

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Iaa7a9b6e6dab2fbd4704a4bb8b737921270a5fe7

LU-14993 kernel: kernel update RHEL8.4 [4.18.0-305.17.1.el8_4]

Update RHEL8.4 kernel to 4.18.0-305.17.1.el8_4 for Lustre client.

Test-Parameters: trivial clientdistro=el8.4

Change-Id: I95e97e1b39e8c49a80f12c9c1b2076553c3dcd49
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44874
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14994 kernel: kernel update RHEL7.9 [3.10.0-1160.42.2.el7]

Update RHEL7.9 kernel to 3.10.0-1160.42.2.el7.

Test-Parameters: trivial clientdistro=el7.9 serverdistro=el7.9

Change-Id: I377ea5d1e28c50b1087dfca7cb32f44afb9bf5f5
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44879
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-2921 lipe: merge lipe changes from b_es5_2

Merge commit 'd4e8316dafcd4c3cdb5092dc6f2a857dc28065fa' into b_es6_0

$ git checkout b_es5_2
$ git subtree split --prefix=lipe
6085b19ae7daa054857bf14d05740ff1224aef01
$ git checkout b_es6_0
$ git subtree merge --prefix=lipe --squash 6085b19ae7daa054857bf14d05740ff1224aef01

Change-Id: I4b8bfa69d312bfe93ad37da1737df9025c9ed0b5

Squashed 'lipe/' changes from b7b776f968..6085b19ae7

6085b19ae7 EX-3738 hotpools: Strict ordering of client and lamigo
a33bdd5d61 EX-3441 lipe: add pool spilling to stratagem-hp-{config,convert}.sh
4c16ba7e00 EX-3092 lipe: remove the lipe-hsm RPM
ef76926047 EX-3725 lipe: fix json.h include

git-subtree-dir: lipe
git-subtree-split: 6085b19ae7daa054857bf14d05740ff1224aef01

EX-3738 hotpools: Strict ordering of client and lamigo

Ensure strict ordering for start and stop of client mount
and lamigo/lpurge.
Ensure enough of a timeout for start and stop.

Test-Parameters: trivial
Signed-off-by: Nathaniel Clark <nclark@whamcloud.com>
Change-Id: I7c50abc5ff82f0cc3fd117fd961f7421ad2df9be
Reviewed-on: https://review.whamcloud.com/44807
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Gaurang Tapase <gtapase@ddn.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

EX-3430: Improvement in HP config for client mount

Use custom lustre-client resource agent for client
mount/unmount which in turn uses the pumount utility
to unmount. This makes sure the client is unmounted
gracefully while stopping the resource.

Remove stratagem-hp-convert.sh as it is no longer used.

Test-Parameters: trivial

Change-Id: Idc5fc2a59e4783a4011fc65a0ae3d69281b1a20f
Signed-off-by: Gaurang Tapase <gtapase@ddn.com>
Reviewed-on: https://review.whamcloud.com/44547
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-14677 sec: do not expose security.c to listxattr/getxattr

security.c xattr, which contains encryption context, should not be
exposed by the xattr-related system calls such as listxattr() and
getxattr() because of its special semantics.
Update sanity-sec test_57 to test this.

Lustre-change: https://review.whamcloud.com/44101
Lustre-commit: TBD (db49ec2ae2c96a09fae054e5fcb3d1959e26f83d)

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I919f5cbafc53f5745fbfb5b9d2d7316e892d8c9f
Reviewed-on: https://review.whamcloud.com/44183
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14677 llite: move env contexts to ll_inode_info level

Contrary to file, inode is always available, so move the list of
env contexts from the file data to the ll_inode_info level.
This is needed because we will have to handle env properties in
ll_get_context() and ll_xattr_list()/ll_listxattr().
This also requires changing lli_lock from a spinlock to an rwlock.

Lustre-change: https://review.whamcloud.com/44198
Lustre-commit: TBD (932007c91333117b7b0905ce5601aafc9b3bdd4e)

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I478d2a8eabfcb09074ba52601f05840d047a6da2
Reviewed-on: https://review.whamcloud.com/44199
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14677 sec: migrate/extend/split on encrypted file

lfs migrate/extend/split makes use of volatile files to swap layouts.
When operation is carried out on an encrypted file, the volatile file
must be assigned the same encryption context as the original file, so
that data moved/copied to different OSTs is identical to the original
file's.
Also update sanity-sec test_52 to exercise these commands.

Lustre-change: https://review.whamcloud.com/43878
Lustre-commit: 09c558d16f0a80f436522edde89367c088fe2055

Change-Id: I3878b5e9e6d3738dfee0ce0f89a3646e6a7b976f
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-on: https://review.whamcloud.com/43879
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-3741 pcc: add pcc_mode parameter for permission check

This patch introduced a "llite.*.pcc_mode" parameter for PCC.
By this parameter, administrator can determine what file access
permissions should be allowed to bring files into PCC device for
caching.
This paramter is set with 0 by default.
Add sanity-pcc test_46 to verify it.

In this patch, it also ignores the EEXIST error when found that the
file had already attached into PCC during the manual attach.

Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I1e006e4f723c1c177ae84c64ad32c6049a57110f
Reviewed-on: https://review.whamcloud.com/44804
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13798 llite: fix LL_SBI_FLAGS array declaration

Fix the LL_SBI_FLAGS to string mapping array.  On master this
has "foreign_symlink" and "foreign_symlink_upcall" before the
"parallel_dio" option, but those do not exist on b_es6_0.
Instead, there is "snapshot" in one of those slots, and the
second is unused.

Since these are in-memory flags only, the actual values are
not critical, and there is a patch in-flight to clean up this
code to be more robust.

In the meantime, what is important is that LL_SBI_PARALLEL_DIO
has the proper "parallel_dio" string in the right spot.

Test-Parameters: trivial
Fixes: 00152903a180 ("LU-13798 llite: parallelize direct i/o issuance")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ie7134c051e85a5a2a90dbeb3145e8d8c09f6d24e
Reviewed-on: https://review.whamcloud.com/44776
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>

LU-14977 kernel: kernel update RHEL7.9 [3.10.0-1160.41.1.el7]

Update RHEL7.9 kernel to 3.10.0-1160.41.1.el7.

Test-Parameters: trivial clientdistro=el7.9 serverdistro=el7.9

Change-Id: Ib57c3ad3b750f4af93cecb372fa5547ccee68fee
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44812
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-3752 pcc: show attaching state for PCC state output

When set llite.*.pcc_async_threshold=0, the client will do PCC
attach in asynchronous way.
When the file is large, attaching the file into PCC may take some
time.
In this patch, we improve that output of the PCC command
"lfs pcc state" to show that the file is in PCC attaching state
when the file is still in the phase of copying from Lustre OSTs
to PCC.

Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I101d87638f5afac41fb4f55b4aaf95d938bc8ccd
Reviewed-on: https://review.whamcloud.com/44852
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14844 tests: make sure mgc_requeue_timeout_min exist.

The module parameter mgc_requeue_timeout_min was introduced to reduce
testing times. Currently the test framework always tries to set this
value but it doesn't exist in earlier Lustre versions which breaks
interop testing. Set the module parameter only if it exist.

Lustre-change: https://review.whamcloud.com/44215
Lustre-commit: dfeb63f2ee3701ef731ffcea3f79fb70d513a9dc

Test-Parameters: trivial
Change-Id: I64f62e3d6e2faeba99ced98363d241083f95d92e
Fixes: 04b2da6180d ("LU-14516 mgc: configurable wait-to-reprocess time")
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44789
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14516 mgc: configurable wait-to-reprocess time

so we can set it shorter, for testing purposes at least. to change
minimal wait time MGC module option 'mgc_requeue_timeout_min'
should be used (in seconds). additionally a random value up to
mgc_requeue_timeout_min is added to avoid a flood of config re-read
requests from clients. if mgc_requeue_timeout_min is set to 0,
then random part will be up to 1 second.

  ost-pools: before: 5840s, after:a 3474s
  sanity-flr: before: 1575s, after: 1381s
  sanity-quota: before: 10679s, after: 9703s

Lustre-change: https://review.whamcloud.com/42020
Lustre-commit: 04b2da6180d3c8eda21f7ab36c676462be041b74

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Iff7dad4ba14d687b7e891a1c346397e4c370800d
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Aurelien Degremont <degremoa@amazon.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/44788
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14825 lod: pool spilling

To avoid the problem of the fast pool becoming full this patch
introduces so-called pool spilling: for every OST pool a target
pool can be assigned which will be used instead of original one
if the original one's use is over specified threshold:

  lctl set_param lod.*.pool.pool1.spill_target=pool2
  lctl set_param lod.*.pool.pool1.spill_threshold_pct=80

i.e. once pool1 is 80+% used, then new files will be created on
pool2.

A chain (up to 10 at the moment) can be configured using the
settings like above when different OST pools are considered
one by one.

Lustre-change: https://review.whamcloud.com/43989
Lustre-commit: TBD (from be958a7bde7351856db6632d06e72e23ce916b13)

Test-Parameters: testlist=ost-pools
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I7f6dd4931ba64f3db8a7ae6a3b185f942a629ed7
Reviewed-on: https://review.whamcloud.com/44303
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-14965 ldiskfs: hold inode mutex for ldiskfs_orphan_add()

See following warning:

ldiskfs/namei.c:3331 ldiskfs_orphan_add+0x11e/0x290 [ldiskfs]
Call Trace:
dump_stack+0x19/0x1b
__warn+0xd8/0x100
warn_slowpath_null+0x1d/0x20
ldiskfs_orphan_add+0x11e/0x290 [ldiskfs]
ldiskfs_xattr_inode_orphan_add+0xbb/0x110 [ldiskfs]
ldiskfs_xattr_delete_inode+0x5c/0x350 [ldiskfs]
ldiskfs_evict_inode+0x1a8/0x630 [ldiskfs]
evict+0xb4/0x180
iput+0xfc/0x190
osd_object_delete+0x1f8/0x370 [osd_ldiskfs]
lu_object_free.isra.27+0xb8/0x1c0 [obdclass]
lu_object_put+0xa5/0x460 [obdclass]
mdt_object_put+0x30/0x110 [mdt]
mdt_reint_unlink+0x8e0/0x1890 [mdt]
mdt_reint_rec+0x83/0x210 [mdt]
mdt_reint_internal+0x720/0xaf0 [mdt]
mdt_reint+0x67/0x140 [mdt]
tgt_request_handle+0x7ea/0x1750 [ptlrpc]
ptlrpc_server_handle_request+0x256/0xb10 [ptlrpc]
ptlrpc_main+0xb3c/0x14e0 [ptlrpc]
kthread+0xd1/0xe0
ret_from_fork_nospec_begin+0x21/0x21

Need to hold inode mutex on the external EA for ldiskfs_orphan_add()
to soothe the warning.

Lustre-change: https://review.whamcloud.com/44754
Lustre-commit: TBD (from 047a859723d3df090af5b1db44adf1f191a6c77c)

Fixes: f64e9f19f68e ("LU-12977 ldiskfs: properly take inode_lock() for truncates")
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I3a1abfde3289c0bbd46e0d5a5b9d2ff7d7cf9273
Reviewed-on: https://review.whamcloud.com/44771
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14986 kernel: kernel update SLES15 SP2 [5.3.18-24.78.1]

Update SLES15 SP2 kernel to 5.3.18-24.78.1 for Lustre client.

Test-Parameters: trivial \
env=SANITY_EXCEPT="100 130 136 817" \
clientdistro=sles15sp2 serverdistro=el7.9 \
testlist=sanity

Change-Id: I2778f5bdacf243da95b7c8c74881ab4d20ad3d91
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44862
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-3750 tests: fix hot-pools and sanity-lipe merges

Restore missing changes in lustre/tests/hot-pools and .../sanity-lipe.
These were not included in the lipe/ subtree merge since they were
outside that dir.

Fix hot-pools test_4 to scan the lamigo log on the MDS.
Add tests for lipe_scan2

Test-Parameters: trivial testlist=hot-pools,sanity-lipe
Fixes: 2f05a3e06928 ("EX-2453 lipe: fixup striped directory paths")
Fixes: 3eee8554c0fb ("EX-2453 lipe: add SoM handling to lipe_scan2")
Fixes: fa19bceb68e0 ("EX-1325 lamigo: improve debug/error messages")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I4c34a3a8f48edf81b5231d208f1a0ff3d9a811f8
Reviewed-on: https://review.whamcloud.com/44859
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-3676 tests: skip conf-sanity test_5a

Skip conf-sanity test_5a since it is causing constant failures since
/sbin/umount.lustre was added.

Test-Parameters: trivial testlist=conf-sanity env=ONLY=1-5 \
austeroptions=-H
Fixes: 6d62073950ac ("EX-3209 lipe: add lpcc util and service")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I77531407d02c5accc78fc239f65c4d05d995502f
Reviewed-on: https://review.whamcloud.com/44839
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-1135 lipe: build lipe_convert_expr against RHEL 8

This patch changes the python platform to python2 to
resolve the following build failure against RHEL 8:

*** ERROR: ambiguous python shebang in
/usr/bin/lipe_convert_expr: #!/usr/bin/python -u.
Change it to python3 (or python2) explicitly.

Test-Parameters: trivial

Change-Id: Id889846573b0c6e4998ef4502616e827cd69f1fa
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44825
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn12

New tag 2.14.0-ddn12

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I7aaf690a1452923202f5441d7de8aa2b92e8eefd

EX-3409 revert: "pcc: add owner capacity check for open attach"

This reverts commit 7cce3772e267afee328d63da9367875c63e6ad43,
since this prevented users with read permission on a file to
auto-attach or manually attach the file into the local cache.

Test-Parameters: trivial testlist=sanity-pcc
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ib562f2c62acf3ed564309fa8ae56ba21bd31577c
Reviewed-on: https://review.whamcloud.com/44816
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>

RM-620 build: New tag 2.14.0-ddn11

New tag 2.14.0-ddn11

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I45b0e969517a5875cba100c0ab2920911fc0d9c1

LU-14954 socklnd: fix link state detection

Due to matching only the device index, link detection implemented
in LU-14742 has issues with confusing the link events for the
virtual interfaces with the link events for the interface that
LNet was actually configured to use. Fix this by improving
the identification of the event source: use both device name and
device index.

Also, to make sure the link fatal state is cleared only when
the device is bound to the IP address used at NI creation,
subscribe to inetaddr events in addition to the netdev events.

Lustre-change: https://review.whamcloud.com/44732
Lustre-commit: TBD (from d4dbbf3cfd692ed548c82e2dda9fdcadae052a62)

Test-Parameters: trivial
Fixes: b842fb6fd5 ("LU-14742: detect link state to set fatal error")
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: Ib1996c66a8ae2596970d66e3d920702190851e3f
Reviewed-on: https://review.whamcloud.com/44787
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-3686 revert: "LU-13799 llite: Adjust dio refcounting"

This reverts commit 8a31964534358dd1a5db6cf86b9c6014d3c98d48
("LU-13799 llite: Adjust dio refcounting")

This patch is causing several tests to crash with messages
similar to the following:

  BUG: Bad page state in process ptlrpcd_01_01
  BUG: Bad page map in process iozone
  page:ffffdafec7f35640 count:0 mapcount:-1 mapping: (null) index:0x7f5
  page flags: 0x6fffff00080018(uptodate|dirty|swapbacked)
  page dumped because: bad pte
    addr:7f5524800000 vm_flags:8100073
    anon_vma:ffff9e8465843fa0 mapping: (null) index:7f5524800
  WARNING: CPU: 1 PID: 9325 at lib/list_debug.c:62 list_del corruption

Test-Parameters: trivial testlist=sanity
Test-Parameters: testlist=sanity-lfsck
Test-Parameters: testlist=sanity-dom
Test-Parameters: testlist=sanity-flr
Test-Parameters: testlist=replay-single

Change-Id: I97b77b671ff0dea4cd13428f700b7643d9e94f09
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44806
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-3636 pcc: set invalid cache state for fallback I/O

When fallback I/O to Lustre (not PCC backend), it should set the
cache state correctly (with *cached = false).

Fixes: c3cf63c830 ("EX-3636 pcc: reset file mmaping for the file once mmaped")
Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I4e7dad93c589ac8062fe6c08423bf209f08432b6
Reviewed-on: https://review.whamcloud.com/44793
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-3730 pcc: add test for concurrent read from 2 clients

This patch add a test case with concurrent read access from 2
clients.
The purpose is to verify that the client will not re-attach file
into PCC backend once attached when the file is read access
concurrently from 2 mount points on a client according to the
PCC attach stats.

This patch also fixes the help message for PCC-RO.

Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: Ibb038bd3a74f43031b6fab4e65565620c416909e
Reviewed-on: https://review.whamcloud.com/44791
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-3715 pcc: add stats for attach|detach|auto_attach

In this patch, we add stats for PCC attach, detach and
auto_attach.
With this feature, we verify that PCC can auto-attach the file
into PCC cache without having to re-fetch the data of the whole
file.
Add sanity-pcc test_44.

Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: Ia0c1cd6b414998e72859aaf34c125b5a4e4e743c
Reviewed-on: https://review.whamcloud.com/44764
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14693 mdt: skip DLM when opening volatile files

In mdt_reint_open(), when opening a volatile file skip taking a
MDS_INODELOCK_UPDATE lock on the parent directory.

Lustre-change: https://review.whamcloud.com/43742
Lustre-commit: 8c04afb5236f8d130419aa0bf5aaf0f52a2ad297

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I8ee89710f52e8097e1412897de91159702560e4a
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44551
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-3725 lipe: fix json.h include

Fix json.h include in lipe_scan2.c.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I2547f14de3fce9ccd91cb32ea1eeb8116566692d

RM-620 build: New tag 2.14.0-ddn10

New tag 2.14.0-ddn10

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ie920dc93c5f27a0fc43a76b9643067e263f12020

LU-14930 mdt: abort_recov_mdt shouldn't abort client recovery

When abort_recov_mdt is set to abort MDT-MDT recovery then
abort_recovery flag is set too inside target_stop_recovery_thread()
call, that causes not just MDT-MDT recovery abort but aborts
also clients/MDT recovery.

Lustre-commit: 6fd75f264c5f5c186bbfe559e1a98fb3769d8128
Lustre-change: https://review.whamcloud.com/44610

Fixes: dd9e79b64d ("LU-12546 mdt: abort recovery between MDTs")
Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: Ibda05e91a2da90156e2b6c9fdcb2169cdbd50fe4
Reviewed-on: https://review.whamcloud.com/44669
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-3663 pcc: auto attach should skip if already attached

When try to auto attach a file into PCC, if found that the file
had already attached into PCC, it should skip the auto attach
processing. Otherwise, it will result in wrong PCC inode refcount
when multiple threads try to auto attach a file at the same time.

For a file once mmapped into PCC and detached due to layout lock
shrinking or manual detach command, If found that file is still
valid cached (attach into PCC again by another thread), in the
@pcc_mmap_io_init(), it should set the mapping of PCC copy with
the one of Lustre file again.

Test-Parameters: testlist=sanity-pcc,sanity-pcc,sanity-pcc
Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I5f049ca7d6db8708712e79e9ad459fc60b80f2be
Reviewed-on: https://review.whamcloud.com/44697
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-3636 pcc: reset file mmaping for the file once mmaped

For a file once mmaped and cached on PCC, a new open will set the
mapping for the file handle of PCC copy (@file->f_mapping) with
the one of the Lustre file handle. When the file is detached from
PCC due to manual detach or layout lock shrinking, the normal I/O
(read/write) will auto-attach the file into PCC again during I/O
as the layout version is unchanged. However, it still needs to
reset the file mapping (@pcc_file->f_mapping) with the mapping of
the PCC copy. Otherwise it will cause panic as follows:
[  935.516823] RIP: 0010:_raw_read_lock+0xa/0x20
[  935.517077]  ll_cl_find+0x19/0x60 [lustre]
[  935.517098]  ll_readpage+0x51/0x820 [lustre]
[  935.517110]  read_pages+0x122/0x190
[  935.517119]  __do_page_cache_readahead+0x1c1/0x1e0
[  935.517131]  ondemand_readahead+0x1f9/0x2c0
[  935.517142]  pagecache_get_page+0x30/0x2c0
[  935.517165]  generic_file_buffered_read+0x556/0xa00
[  935.517189]  pcc_try_auto_attach+0x3ac/0x400 [lustre]
[  935.517552]  pcc_io_init+0x146/0x560 [lustre]
[  935.517906]  pcc_file_read_iter+0x24d/0x2b0 [lustre]
[  935.518259]  ll_file_read_iter+0x74/0x2e0 [lustre]
[  935.518604]  new_sync_read+0x121/0x170
[  935.518937]  vfs_read+0x8a/0x140

This patch adds sanity-pcc test_98 to verify it.

I/O for a file previously opened before attach into PCC or once
opened while in ATTACHING state will fallback to Lustre OSTs.
For the later mmap() on the file, the mmap() I/O also needs to
fallback to Lustre OSTs and cannot read directly from local valid
cached PCC copy until all fallback file handles are closed as the
mapping of the PCC copy is replaced with the one of Lustre file
when mmapped a file.
Add sanity-pcc test_97 to verify it.

And we also forbid to auto attach the file which is still in
mmapped I/O.

This patch disables "mmap_conv" by default.

Test-Parameters: testlist=sanity-pcc,sanity-pcc,sanity-pcc
Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I11195b0bdb6fb1d0d68d0b0cd02a0af8ee1fc297
Reviewed-on: https://review.whamcloud.com/44592
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14925 kernel: kernel update RHEL8.4 [4.18.0-305.12.1.el8_4]

Update RHEL8.4 kernel to 4.18.0-305.12.1.el8_4 for Lustre client.

Test-Parameters: trivial clientdistro=el8.4

Change-Id: Ic7a782270dd350a211094c18411faa60a35013e9
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/44603
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>

Squashed 'lipe/' changes from 2a015e67c4..b7b776f968

b7b776f968 Fix json.h include.
2ba146383f Update lipe version to 1.19.
f007d85105 EX-2453 lipe: fixup striped directory paths
64847ef214 EX-2453 lipe: add SoM handling to lipe_scan2
75045a94c5 EX-3701 lipe: lamigo ssh logging improvements
984381d796 DDN-2223 lipe: add ngc_exp_remv and tests
9e5f54e3e7 EX-3588 lamigo: use POSIX redirection syntax
a6240ef3bf EX-3701 lipe: lamigo lamigo log message improvements

git-subtree-dir: lipe
git-subtree-split: b7b776f9687f084300eabe4b5cebdc20f316d8e0

EX-2921 lipe: merge lipe changes from b_es5_2

Merge commit '72399232bb9bbc85fb509fa48e12d88cc7471724' into b_es6_0

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I1624187cffbaba64d914dbe517d0b7e16878006a