LU-16101 tests: skip sanity/27J for more kernels

This is a bug in the kernel that is not present in older kernels
before commit v5.11-10234-gcbd59c48ae2b (5.12), and is fixed with
commit v6.2-rc4-61-g5956592ce337 (6.2).

Move this from ALWAYS_EXCEPT (bug that needs to be fixed) to skip
(test that is known to fail in some configs but has been fixed).

Lustre-change: https://review.whamcloud.com/51567
Lustre-commit: b711af7d243f3773cec3a37f64c0e0aa8bbc363f

Fixes: af6f49698a18 ("LU-16101 tests: add sanity/27J to always_except")
Test-Parameters: trivial testlist=sanity clientdistro=el9.2 env=ONLY=27J
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I8ec0a6d25a90e05672b039cd6c2b2fbf8a3ebbe5
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51583
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15519 quota: fallocate does not increase projectid usage

fallocate() was not accounting for projectid quota usage.
This was happening due to two reasons. 1) the projectid
was not properly passed to md_op_data in ll_set_project()
and 2) the OBD_MD_FLPROJID flag was not set receive the
projctid.

This patch addresses the above reasons.

Test-case: sanity-quota/78a added

Lustre-change: https://review.whamcloud.com/46676
Lustre-commit: 5fc934ebbbe665f24e2f11fe224065dd8e9a08ba

Fixes: 48457868a02a ("LU-3606 fallocate: Implement fallocate preallocate operation")
Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Change-Id: I3ed44e7ef7ca8fe49a08133449c33b62b1eff500
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51639
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn93

New tag 2.14.0-ddn93

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ic6530f389228232cf1f763810d2abcbebc9b51d7

LU-16937 utils: avoid lctl shmget() if not needed

lctl is dynamically allocating an IPC shared memory segment
during every startup, even though it is only needed for a
small number of uncommon debug commands:

    shmget(IPC_PRIVATE, 65680, 0600)        = 196641
    shmat(196641, NULL, 0)                  = 0x7f752b1c5000
    shmctl(196641, IPC_RMID, NULL)          = 0

This setup can be moved to sub-commands that actually need it.

Lustre-change: https://review.whamcloud.com/51526
Lustre-commit: TBD (from 309713169fde9e162c26e909bc83cb43cccd67ba)

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I41c790ce7cba2d9c48c1ec06eb23eb94aa548242
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51516
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16934 kernel: update RHEL 8.8 [4.18.0-477.15.1.el8_8]

Update RHEL 8.8 kernel to 4.18.0-477.15.1.el8_8.

Lustre-change: https://review.whamcloud.com/51517
Lustre-commit: TBD (from 7174f706328cb4e6a52c898c1cd7719b81e26c0d)

Test-Parameters: trivial fstype=ldiskfs \
clientdistro=el8.8 serverdistro=el8.8 testlist=sanity

Test-Parameters: trivial fstype=zfs \
clientdistro=el8.8 serverdistro=el8.8 testlist=sanity

Change-Id: I66365dce63065a0a07958a182a3c705e9948d424
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51519
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7683 utils: always try to use our own lz4/lz4hc

lz4/lz4hc provided by the kernel do not grok a compression level.
The built-in lz4/lz4hc do, so always build them as dedicated kernel
modules llz4.ko and llz4hc.ko, with the same .cra_name but with a
slightly higher .cra_priority = 110, so that they are preferred over
the in-kernel modules if any.

And try to manually load the llz4/llz4hc kernel modules when a file
requires compression with the corresponding alg. This is a "one-shot"
try that allows us to prefer our modules that has level support, but
continues to at least compress/decompress files even if our own
modules are not available.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I0bdf267f998e21df81e460250a653aed34e3215d
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51474
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-8582 tests: skip sanity/905 for old OSTs

The fail_loc used in sanity test_905 does not exist in older OSTs.
Skip this subtest for older OSTs.

Lustre-change: https://review.whamcloud.com/51568
Lustre-commit: TBD (from 2ced1e0898aacd741c95c25d44350dfefa953853)

Fixes: 566edb8c43 ("LU-8582 target: send error reply on wrong opcode")
Test-Parameters: trivial testlist=sanity serverversion=2.12.9 env=ONLY=905
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I8fa50ec0f66afd9f24d562e0be57a416c04d8ba8
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51569
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15235 tests: skip sanity/56od in interop

Sanity test_56oc and test_56od were using the btime_supported()
function to check it "lfs find" supported file birth time, but
this did not properly check whether the MDS supported this option.

Remove the btime_supported() check and just use the version, since
this has been around a few releases already.

Lustre-change: https://review.whamcloud.com/51580
Lustre-commit: TBD (from 8add332bda0c58d9908478b9263e8aea46edc135)

Fixes: 186b97e68abb ("LU-11971 utils: Send file creation time to clients")
Test-Parameters: trivial testlist=sanity serverversion=2.12.9 env=ONLY=56
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I0c85103c843d3b993e3e112bf5d0da976d3ebbe5
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51581
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-7814 tests: fixed shell syntax error

Fixed a regression introduced in LU-16399.
Auster assumes that all scripts are written in bash,
so this commit removes hardcoded sh calls.

Test-Parameters: trivial testlist=parallel-scale-nfsv4
Fixes: ea13f42719 ("LU-16399 tests: add subtests setup/cleanup records")
Signed-off-by: Alex Deiter <adeiter@tintri.com>
Change-Id: I20d136db3b763143df038d25a167eb84f646444f
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51556
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7758 test: Add sanity-compr.sh to run sanity with PFL layout

Add sanity-compr.sh to run sanity and sanityn with PFL layout
Also fix sanity subtests problem of 56ba,57b,65e,65g,65n,204e

Test-Parameters: trivial testlist=sanity-compr

Signed-off-by: Wei Liu <sarah@whamcloud.com>
Change-Id: Iefdc7757697629eb5c57d7694456249d62a2049e
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51577
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15934 lod: clear up the message

Print out the precise info while llog context error.

Lustre-change: https://review.whamcloud.com/49528
Lustre-commit: 9882d4e933fd8cdbc4a9bc8bf6b29655009f7e03

Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: I492201cd3ae5eb39ad34f3a873d7bb346b52430f
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51555
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-6856 utils: add -Z option for 'lfs getstripe/find'

Add support for "lfs getstripe -Z" to get the last instantiated
component compression information.

Add support for "lfs find -Z <type>[:[+-]<level>]" to keep consistent
options with "lfs setstripe -Z".

Fixes: 093bd2f343 ("EX-6856 utils: add 'lfs find' support for compressed file")
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Id3e788761656d05604bc9a72fb1e51c5f2a0ad3b
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51497
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15934 tests: add a test case for update llog

A test case to simulate the update llog corruption
situation. It replaces the catalog file with a
random data. The recovery of mdt will be blocked
if without the fixing patch.

Lustre-change: https://review.whamcloud.com/51208
Lustre-commit: 54301fe4f598eef5aebdbdb0c7f3dddea9541c4e

Fixes: 814691bcff ("LU-15934 lod: renew the update llog")
Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: I0ade8d0ff33ddc06b622e5e67cf4b4775dfff129
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51570
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15934 lod: renew the update llog

Skip and renew the update llog file while it was
corrupted.

Lustre-change: https://review.whamcloud.com/49569
Lustre-commit: 814691bcffab0a19240121740fb85a1912886a3c

Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: I3491858dce42b4a8ed11db55ebbf8a12ef5f521d
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Mikhail Pershin <mpershin@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51552
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-6127 tests: test compression without bzip2/HDF5

Run as much of sanity test_460a compression tests as possible,
even if bunzip2 or HDF5 file are unavailable.

Print a clear message in test_84 if bunzip2 unavailable.

Fixes: f43b9ce9af ("EX-6127 osc: osc brw request compression")
Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I36251834f636600eb9b0194ccd14c8b203da32e5
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51532
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16340 quota: notify only global lqe

Don't notify slaves with new limits when set new
limits to the pools. Do this only for lqes that
belong to the global pool.

The fix helps to avoid a case when slaves do not
apply new limit because slaves` data version is
greater than the one comes from the MDT. It was
possible, if set a lot of times different limits to PQ.
After that new limits from the global pool could not
be be applied:

qsd_upd_schedule()) lustre-OST0000: discarding glb
update from glimpse ver:7 local ver:203

For details about the problem see "check indexes versions"
test in sanity-quota.sh.

Add test 25 "check indexes versions" into sanity-quota.
Without the fix it reproduces above problem.

Fix checkpatch to don't print "Invalid vsprintf pointer
extension" for %px.

HPE-bug-id: LUS-10705
Signed-off-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Lustre-change: https://review.whamcloud.com/c/fs/lustre-release/+/49239/
Lustre-commit: 513b1cdbca58913249eb524a37374c418fdec44f

Change-Id: Idb091a10894e9db9f67d215baef2926723d6c65d
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51551
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>

EX-7818 osc: don't do decomp. with compression disabled

decompress_request likely has a significant performance
cost for checking if data is compressed, so we should not
call it when compression is disabled.

This is a stop-gap solution for the preview until we can
have the server tell the client if data is compressed, as
described in EX-7818.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Iba8feba3ab0fe620d0594f59c2c6ddea25faeb4f
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51563
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7807 osc: don't discard decompress_request error

The error handling for decompress_request is unusual - non-zero
returns are just discarded. And rc2 is just discarded.
The read() doesn't fail or get a short read.

Fix this so if decompression fails with an error.

Reported-by: Patrick Farrell <pfarrell@whamcloud.com>
Signed-off-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Fixes: f43b9ce9af ("EX-6127 osc: osc brw request compression")
Change-Id: Idd01947c7375c9586a64f064dd6ee0ac2308ea86
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51531
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7812 llite: use CIT_MISC in check_compression

ll_mmap_check_compression uses a CIT_FAULT io type and
initializes it with cl_io_rw_init.  This doesn't
initialize the IO correctly and this sometimes results
in the following crash during io_fini, because the ft_page
pointer is initialized to something else by cl_io_rw_init:

[ 1512.276302] BUG: unable to handle kernel NULL pointer dereference at 000000000000004a
[ 1512.280778] RIP: 0010:cl_pagevec_put+0x9f/0x3a0 [obdclass]
[ 1512.288693]  vvp_io_fault_fini+0x21/0x40 [lustre]
[ 1512.289294]  cl_io_fini+0x7a/0x230 [obdclass]
[ 1512.289876]  ll_mmap_check_compression+0x403/0x540 [lustre]

Fixes: 95df962a92 ("EX-6265 llite: disable mmap on compressed files"

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Ib1f270f071370d6c045bb9d799ab5b7b41a6c6be
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51553
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-6127 llite: fix chunk_bits usage in readahead

For the minimum compression chunk size, chunk bits is zero,
so we cannot use if (chunk_bits) to determine if we're
doing compression.

This also fixes two other things:
1. A rounding error when rounding to chunk
2. Move rounding of end_idx to before first usage of
end_idx, so calculation of number of pages is correct
Without this, when the user reads 1 page or less, readahead
will calculate the readahead page count as 0 and will exit
without reading the chunk.

Fixes: c05d5990f4 ("EX-6127 llite: getting stripe info optimization")
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I273506fd4f6ed5f0b8b5020357fd7caf0531e61c
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51504
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-6856 utils: support 'lfs find --printf %LZ'

Add support for "lfs find --printf %LZ" to print the compression
type:level of the last instantiated component of a file.

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Iaf1b6c031b06c70e7b5be51354697aa6bdcc9850
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51498
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn92

New tag 2.14.0-ddn92

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Icc54de2ab22bff722ec19c3293767706458f684b

EX-6127 osc: add compression to can_merge_pages

Some BRW flags are OK to have on only some pages in a BRW,
others are not.  can_merge_pages has a whitelist of the
flags which are safe to have on only some pages in a BRW,
and prints a warning if other flags are seen.

Add compression to the white list, because while all pages
in an niobuf must be compressed, it is normal to have only
some pages in a BRW compressed.

Prior to this patch, this warning was printing during
normal usage of compression.

Test-Parameters: trivial
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Ia262d4fc878e5328bd956865047e997aa77946f0
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51528
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-6206 lustre: add lgzip kernel module

lgzip kernel module implements compression according to the
deflate/zlib algorithm, through the kernel Crypto API.
It provides 2 cipher drivers under the generic name 'deflate':
* deflate-lustre-generic of type compression
* deflate-lustre-scomp of type scomp

Note the 'deflate' name is identical to the in-kernel module, but
lgzip registers it with a slightly higher .cra_priority = 110, so that
it is preferred over the in-kernel module.
Our 'deflate' is also different in that it accepts a compression level
as explained below.

lgzip kernel module sources are copied from linux v6.2-rc5 and renamed
to gzip.c to avoid name collisions. It implements the Crypto API
interface, and rely on the deflate/zlib kernel library for compression
implementation. It has been modified to grok a compression level, as
read from the top 4 bits of the crypto_tfm flags, and pass it to the
underlying library.
The deflate/zlib library sources are also copied from linux v6.2-rc5
and built statically. Headers have also been copied from linux
v6.2-rc5 for consistency, and source files modified to include the
copied headers instead of the system headers.
All aforementioned sources are located in the lustre/gzip directory.
The lgzip module is always built with Lustre.

This patch enhances the test kernel module kcompr.ko to exercise the
compression level of the provided 'deflate' module.
It also tries to manually load the lgzip kernel module when a file
requires compression with the 'delfate' alg. This is a "one-shot" try
that allows us to prefer our module that has level support, but
continues to at least compress/decompress files even if our own module
is not available.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I2b0386457bc781d91816172dea6b52ce3dd273f4
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51422
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7784 tests: skip sanity 460 on 64k PAGE_SIZE

Skip this test on the system with 64k PAGE_SIZE.

Test-Parameters: trivial testlist=sanity clientdistro=el8.7 clientarch=aarch64
Signed-off-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Change-Id: I9324ae975bfc3c4f08d5048d7c977447ef62cc78
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51501
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14301 revert: client: use EOPNOTSUPP instead of ENOTSUPP

This reverts commit 3deafa8a39a964c67f533173a5b2f90d0d2d7730.
Accidentally landed and is likely to cause problems.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>

EX-7386 build: Update lipe tag to 2.24

New version tag.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I711231b7ec063ed9bf8a0976e18360bebdc84d3b

LU-14301 client: use EOPNOTSUPP instead of ENOTSUPP

Don't return NFS-specific error code ENOTSUPP back to userspace,
instead use EOPNOTSUPP.

Lustre-change: https://review.whamcloud.com/51511
Lustre-commit: TBD (from 1a0d553000d5a869f9039bab74dbdbb20d4259b0)

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Iabd07b31069737e8ee7ca2382fd8cff6143ebbe5

EX-6265 llite: allow mmap reads of compressed files

mmap reads of compressed files work, so we should only
block writes.

We cannot block the actual fault operations because that
will cause the application to get a SIGBUS, so we check
the file open mode when we go to create the memory mapping.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I636f398fd247ddcd153f94bf8116440540e8469c
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51506
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Qian Yingjin <qian@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-6265 llite: disable mmap on compressed files

disable mmap(2) on compressed files until well tested.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I3464a03b16708edcd0692bc9db337eb8473ea047
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51413
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16399 tests: add subtests setup/cleanup records

* Added setup/cleanup records for subtests

Lustre-change: https://review.whamcloud.com/49582
Lustre-commit: d0ae0079d747d05f74f733fb594d8edb512f8b16

Change-Id: Icb203a864fa8785e423a073b4ee0f02ea3d3ac77
Signed-off-by: Alex Deiter <adeiter@tintri.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51406
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16836 lnet: ensure dev notification on lnd startup

Look up device and link state on lnd startup so that
the initial NI state may be set properly.

Reduce code duplication by adding lnet_set_link_fatal_state() and
lnet_get_link_status() functions which are shared across LNDs.
LND-specific versions of these are removed.

This fixes the issue with adding LNet NI using an interface with
cable unplugged which results in the NI state initialized as "up".

Lustre-change: https://review.whamcloud.com/51057
Lustre-commit: 09c6e2b872287c847d15620788f6cf50b3a9f30b

Fixes: c4df48116d ("LU-16563 lnet: use discovered ni status")
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: I16084092cc21a4e42dfef4624adfbf57eb4fdecb
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51310
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7509 tests: enable neterror logging in sanity-benchmark/iozone

This patch enables LNet error logging in sanity-benchmark test_iozone()
to gather network errors while connection issue occurs.

Test-Parameters: trivial env=SLOW=yes,ENABLE_QUOTA=yes \
 clientdistro=el8.6 serverdistro=el7.9 testlist=sanity-benchmark

Change-Id: I398779abc95525fe5579fc7505e6e6221c32bf90
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51483
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-8582 target: send error reply on wrong opcode

Unknown opcode does not necessarily means insane client. A new client
might send RPCs with new opcodes to an old server. The client might
desperately stuck there waiting for a reply. So, send an error back
when RPC has a wrong opcode.

This patch returns the EOPNOTSUPP to client instead of block. ENOTSUPP
is not used here since strerror() does not understand ENOTSUPP.

OBD_FAIL_OST_OPCODE=0x253 is added for fault injection test of opcode.
To test whether an invalid opcode is handled properly on OST, use the
following command:

    lctl set_param fail_val=${opcode} fail_loc=0x253

Lustre-change: https://review.whamcloud.com/47761
Lustre-commit: 03c1ddf19c83891683e1726f240a2449941e8b22

Change-Id: I46ca62bc532b92368e06a4f883b102c7186c453c
Signed-off-by: Li Xi <lixi@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51513
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14301 lustre: add ENOTSUPP to spelling.txt

Add a spelling check for ENOTSUPP to suggest use of EOPNOTSUPP
instead. Note:

ENOTSUPP (524) and defined only in the kernel errno.h and is a NFSv3
specific errno. If ENOTSUPP is returned to userspace then strerror()
will print "Unknown error 524".

EOPNOTSUPP (95) is defined in kernel and userspace errno.h.

ENOTSUP is defined in userspace errno.h as an alias for EOPNOTSUPP.

Lustre-change: https://review.whamcloud.com/41280
Lustre-commit: e00733f0f87659c936039a58ea738cfb070638bc

Test-Parameters: trivial
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I13b0389c9ec0853f43d8ab4a8f6538eb24c8a2ad
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51512
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>

EX-7683 man: document supported compression types

Update lfs-setstripe to mention the currently supported compression
types: gzip, lz4, lz4fast and lzo.

Test-Parameters: trivial
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I9b587516d27b882bec5855da4948f489e5a0041f
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51485
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7775 utils: fix cp_comp_type size

cp_comp_type should be 8 bits, as llch_compr_type and all associated
variables are declared as u8.
So remove useless cp_comp_enabled and fix code to test for compressed
component with cp_comp_type against LL_COMPR_TYPE_NONE.
And update LL_COMPR_TYPE_MAX value to 255 to avoid conflicts with
future compression types.

Fixes: f43b9ce9af ("EX-6127 osc: osc brw request compression")
Test-Parameters: trivial
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ia15868ac0ac003b62942540a57f782226ae8c141
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51481
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7775 utils: fix LL_COMPR_TYPE wire checks

Fix LL_COMPR_TYPE* wire checks to avoid duplication.
The value of LL_COMPR_TYPE_UNCHANGED is also declared as 15 so that
is does not conflict with other potential real compression types in
the future.

Fixes: 67d4601737 ("EX-6249 csdc: set compress component for file")
Test-Parameters: trivial
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Iab9830f09f0778e1e1f3b1ea4c9878ce1017de8d
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51473
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7683 utils: update handling of compr level for lz4

The way lz4 compression level or acceleration factor is handled needs
to be adapted in order to match what is provided by the lz4 userspace
tool:
- any level between 0 and 2 is interpreted as the default lz4
  acceleration factor of 1;
- any level from 3 and up to 16 is interpreted as a compression level
  for internal lz4hc. Increasing the compression level trades CPU
  time for improved compression ratio;
- acceleration factor can be specified for the lz4fast compression
  type, from 1 to 26. This acceleration factor trades compression
  ratio for faster speed.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I4711217c1a6601f29f78d262567da5998f657fc9
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51380
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7715 utils: error for invalid compression level

Not all compression types accept a compression level (e.g. lzo in its
kernel implementation). For those, return an error if a compression
level is provided on 'lfs setstripe' command-line.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I6c367c4bfd76cc81c890a89ba9f994f4fd9f4f80
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51394
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-6931 tests: add version check in sanity test_904

The patch that added hide virtual projid xattr functionality
to b_es6_0 branch was submitted under a tag of 2.14.0-ddn55.
In case of interop testing with older servers test 904 fails.
At the same time the rest of the test should still run on
older servers, so I moved parts of the code related to hide
virtual projid xattr functionality under version check
condition.

Sanity test 904 was modified.

Fixes: 8acf413647 ("LU-15548 osd-ldiskfs: hide virtual projid xattr")
Test-Parameters: trivial testlist=sanity env=ONLY="904"
Test-Parameters: testlist=sanity env=ONLY="904" serverversion=2.14.0-ddn54
Signed-off-by: Alena Nikitenko <anikitenko@ddn.com>
Change-Id: Ie3bc67cb8a7a83954f3b2048f009f84ab77bf53d
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51484
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7752 test: hot-pools test_59: not expected flag 'stale'

Make the 1st mirror preferable for write for the test purpose.

Test-Parameters: trivial env=ONLY=59 testlist=hot-pools
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I06c33badfc10ff9743df83a6e421e27afc9b6dbd
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51502
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Colin Faber <cfaber@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn91

New tag 2.14.0-ddn91

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I8634b3650bf43aadaf303e48b9ebe2931e6af6ab

LU-16923 kernel: update RHEL 8.8 [4.18.0-477.13.1.el8_8]

Update RHEL 8.8 kernel to 4.18.0-477.13.1.el8_8.

Lustre-change: https://review.whamcloud.com/51411
Lustre-commit: TBD (from a04c2ff950a2a6b65e55184113b1513fc9bf0058)

Test-Parameters: trivial fstype=ldiskfs \
clientdistro=el8.8 serverdistro=el8.8 testlist=sanity

Test-Parameters: trivial fstype=zfs \
clientdistro=el8.8 serverdistro=el8.8 testlist=sanity

Change-Id: I23c6439aedd6f8e9473ddb629ff7e01c50d9c8fc
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51420
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7680 llite: skip hole/data lseek() on compressed file

Doesn't execute real lseek() on compressed file with
SEEK_HOLE/SEEK_DATA origin but consider file always
non-sparsed and do generic lseek() only

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I2cd587cc0205e85758e06bbaafafe0e2959e0ade
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51429
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15671 mds: do not send OST_CREATE transno interop

Send OST_CREATE RPCs from the MDS with no_resend and no_delay
when communicating with an old OST that does not support the
OBD_CONNECT2_REPLAY_RESEND.  Likewise, the OST should not reply
to the MDS RPC with rq_transno set, or this will trigger:

   osp_precreate_send() ASSERTION(req->rq_transno == 0) failed

This can be avoided if the MDS is upgraded before the OSS, but
will always be hit if OSS is upgraded first.

After 2.20.53 the MDS/OSS assume that this is always true, since
rolling upgrades are unsupported for larger version differences.

Lustre-change: https://review.whamcloud.com/51056
Lustre-commit: 9ee1281060d0a00a9c5d715a9a6d9b99c27123ff

Test-Parameters: testgroup=rolling-upgrade-oss
Fixes: 63e17799a3 ("LU-8367 osp: enable replay for precreation request")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Sergey Cheremencev <scherementsev@ddn.com>
Change-Id: I1ab601a2f55540dd75cf24838f7cdb7f823ed42c
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51425
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-6127 llite: getting stripe info optimization

ll_lov_getstripe_ea_info() is expensive call and should be avoided
if possible. Let's use cached chunk size rather than get it from
stripe info every time.

Signed-off-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Change-Id: Id08487ec782f797e242e3f673c4a4dd8d526c9cc
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51321
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7601 llite: restrict overwrites during preview

EX-7601 is an issue where when modifying a compressed file
we do not correctly read-up and re-write existing
compressed data.

To avoid this, we can only allow writes which are not
aligned to compression chunk size when they are not
overwriting existing data, ie, when they are extending the
file.

This returns EINVAL for all writes to compressed files
which are not either chunk aligned or extending the file.
This should prevent users from hitting the data corruption
issue but still allows some basic usage.  This is intended
just for the preview period.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I0eea01e2249866a074afd0d0642fe6dce9a49664
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51259
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-6127 llite: DIO fallback on compressed files

Fully supporting direct I/O on compressed files is tricky
because we cannot pull the full chunk in to the page cache
(because there is no page cache for DIO).

So instead we fall back to buffered I/O for DIO on
compressed files.

This patch adds the check and a test for this.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I8224ef9b8ad1d912d8a11eccad37d3dff8dd8498
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51200
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7528 lov: fallocate is not allowed for compressed files

Client Side Data Compression allocates blocks after a compression.
It is impossible to preallocate blocks for the whole file, so
fallocate should be disabled in case of compression.

Signed-off-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Change-Id: Ie834ace183fdcec0d7d6f747237e0964c3c4120b
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51059
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Vitaliy Kuznetsov <vkuznetsov@ddn.com>

EX-6127 osc: osc brw request compression

This patch adds client-side compression/decompression.

The client-side data compression project (CSDC) reduces storage and network
utilization by leveraging the more plentiful memory and CPU resources on the
local client. Data is sent compressed over the network, saved directly to
storage on the server side, and decompressed back on the client side.
Uncompressed data is kept in client page cache, all while being functionally
transparent to the end user and application.

As an example, a test file is compressed and decompressed.
The resulting file is compared with the original one.

The test case shows 2.5x compression ratio:
356K /mnt/lustre/d460.sanity/sanity.sh
884K /tmp/cmp-46ofie/decompressed_sanity.sh

Compression should read whole chunk even if offset
and size differ.

Let's modify readahead to force reading data from the
offset and size multiple to the chunk size.

Test-Parameters: testlist=sanity env=ONLY=460
Signed-off-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Change-Id: I9b41ab815db3df9ad7bdea5fca4c093cbda8814b
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/49511
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16837 csdc: reserve connect bits for compressed layout

Add connect data bit for compressed layout (OBD_CONNECT2_COMPRESS)
and another connect data bit to be used (OBD_CONNECT2_LARGE_NID).

Also reserve obd_connect_data::ocd_compr_type which is a bitmask of
supported compression type to be negotiated between client and MDS.

Lustre-change: https://review.whamcloud.com/51108
Lustre-commit: 83189aef3b23f18cb8c1deae34994a00f8582039

Test-Parameters: trivial
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I21029c6c3e8a7e690ecc8d489bbb95aec3ab1fa8
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51409
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-1904 idl: add checks for OBD_CONNECT flags

Make it harder to accidentally declare OBD_CONNECT flags without
properly defining their names.  Otherwise, this can cause serious
compatibility problems if two features are using the same flag.

Add the definition lines into spelling.txt so there is *always*
a warning generated, since this always needs proper attention.

Make it clear whom to contact when reserving a new feature flag.

Lustre-change: https://review.whamcloud.com/48053
Lustre-commit: d851381ea6947244842ae6f138cd0bfd399b7ef4

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I9a5e2c97c40c39ea57d20979d4b130854edc785a
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51408
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn90

New tag 2.14.0-ddn90

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I8a73e4233db5fc835eb33849e0508204417e7429

EX-7593 csdc: don't set compression layout when disabled

When llite_enable_compression is disabled
(lfs set_param llite.*.enable_compression=0), we should check
it before sending it to MDS lest we get a file with compressed
component which we cannot handle.

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Ib1e2123ffdb239c3e1401d682ae9c2c49e3f4a6f
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51250
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7714 utils: logical AND for 'lfs find' compression exprs

All search expressions provided to 'lfs find' must be combined as a
logical AND. Fix newly added options for compression support, so that
they comply with this logical AND.

Fixes: 093bd2f343 ("EX-6856 utils: add 'lfs find' support for compressed file")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I3b28cd87c1d304df6d04753b413d46f5abcfe16e
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51393
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16778 tests: sanity-quota_75 fix

Change conf=fsync to oflag=direct to avoid
cache write.

Lustre-change: https://review.whamcloud.com/51158
Lustre-commit: TBD (from b1c5e39335820602abeecbb91a3afb86879f84f2)

Test-Parameters: trivial testlist=sanity-quota env=ONLY=75,ONLY_REPEAT=100
Signed-off-by: Sergey Cheremencev <scherementsev@ddn.com>
Change-Id: Iff04bac63f772dc2d0d0ad765d210b2539fbe33e
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51407
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16922 kernel: update RHEL 9.2 [5.14.0-284.18.1.el9_2]

Update RHEL 9.2 kernel to 5.14.0-284.18.1.el9_2 for Lustre client.

Lustre-change: https://review.whamcloud.com/51410
Lustre-commit: TBD (from 6111eb7a418fb4396e0451d0613af43870951f72)

Test-Parameters: trivial env=SANITY_EXCEPT=27J clientdistro=el9.2 testlist=sanity
Test-Parameters: trivial env=SANITY_EXCEPT=27J clientdistro=el9.2 serverdistro=el8.8 testlist=sanity

Signed-off-by: Jian Yu <yujian@whamcloud.com>
Change-Id: Ifa8f13200550e5f473b7d7d641155e349c453c03
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51416
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7596 tests: don't fail if metadata_csum_seed unset

Fix logic in the sanity-pcc cache filesystem setup.  With mke2fs
1.47.0-wc1 it enabled metadata_csum_seed unconditionally, but it
caused problems on el7.9 kernels. In 1.47.0-wc2 it disabled that
feature, caused the check for removing the feature to fail.

Since metadata_csum_seed has been available since 1.44.2 it
shouldn't be a problem to force it off duing mke2fs.

Test-Parameters: trivial testlist=sanity-pcc env=ONLY=1
Fixes: 7d35dd13273e7 ("EX-7596 tests: disable metadata_csum_seed for pcc cache device")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ic04ab4043981dc9b5c32e01c4aa85be343e3f3f8
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51417
Reviewed-by: Feng Lei <flei@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>

LU-16888 gss: fix ptlrpc_gss automatic loading

ptlrpc_gss kernel module is automatically loaded when a GSS security
flavor is enforced. Loading success is recorded in a static variable
in the ptlrpc module, which prevents further reloading in case
ptlrpc_gss is unloaded while keeping ptlrpc loaded.
Get rid of this static variable as it is not required in order to
avoid calling request_module("ptlrpc_gss") when not needed. Indeed,
once loaded, the static array policies[] has an entry at the
SPTLRPC_POLICY_GSS index, indicating that the ptlrpc_gss module is
loaded.

Lustre-change: https://review.whamcloud.com/51264
Lustre-commit: b80d6defb7b018250ef4fafccff7c980aed6a444

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I9bb100a202fe9c3fc455a2ffba6ee6398e19b9bf
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51374
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16772 quota: access lqe_glbl_data only under mutex

Hold mutex to protect lqe_glbl_data against freeing in
qmt_lvbo_update, qmt_allock_lock_array and qmt_setup_id_desc.
This patch should help against below(LU-14434) and similar panics.

  BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
  RIP: 0010:qmt_id_lock_cb+0x69/0x100 [lquota]
  qmt_id_lock_cb+0x69/0x100 [lquota]
  qmt_glimpse_lock.isra.19+0x27e/0xfb0 [lquota]
  qmt_reba_thread+0x5da/0x9b0 [lquota]
  kthread+0x112/0x130
  ret_from_fork+0x35/0x40

It is the 2nd part of 50ff4d1da63e8. The latest patchset of
https://review.whamcloud.com/c/fs/lustre-release/+/50748/ has extra
changes, but occasionally has been landed without inspection.
So it's not simple porting from upstream.

Fixes: 50ff4d1da63 ("LU-16772 quota: protect lqe_glbl_data in qmt_site_recalc_cb")
Signed-off-by: Sergey Cheremencev <scherementsev@ddn.com>
Change-Id: I5753ace94a739c6df70dd8ea3bde828e2b5ed812
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51368
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16804 tests: rename 'complete' to 'complete_test'

The test-framework.sh "complete" function conflicts with "complete"
exported from bash_completion, and this causes lustre-initialization
to fail in some configurations now that the lustre test config
is loaded earlier during test-framework.sh init_test_env() setup.

Rename "complete" to "complete_test" to avoid this conflict.

Lustre-change: https://review.whamcloud.com/51383
Lustre-commit: TBD (from 9bf8d1944995e7bf627a602aa1d0523d810c84b6)

Test-Parameters: trivial
Fixes: fdbb2bc849 ("LU-16804 tests: load CONFIG at beginning of init_test_env")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ic72d8d5cc4a65feec6bfb2a76ac5f9b9d78e3f75
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51389
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Charlie Olmstead <charlie@whamcloud.com>

LU-16587 utils: give lfs migrate a larger buffer

lfs migrate is slow because it mostly uses a small 1MB buffer. Bigify.

[root@kjlmo4n00 16G]# time lfs migrate -S 1M -p flash 16G.1
real 0m25.341s
[root@kjlmo4n00 16G]# time /root/tools/lfs_nzr migrate -S 1M -p flash 16G.1
real 0m6.526s

Lustre-change: https://review.whamcloud.com/50118
Lustre-commit: 23224e03dc30c89dd449de5a7fe99b0bd3aca495

Signed-off-by: Nathan Rutman <nathan.rutman@hpe.com>
Change-Id: I850ca475fcd0efe2d71d26e4d1544f462c60252a
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51352
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-6510 csdc: prefer uncompressed mirror for write

When writing to mirrored files with both compressed and uncompressed
mirrors, prefer the uncompressed components to write, and that is
better for performance, more compatible with older clients, and better
fits the model of compressing files after initial write.

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I62a117d5cc3d34e2c0c96d1a9ade8eef0a2d1291
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50974
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-6261 ptlrpc: extend sec bulk functionality

Client Side Data Compression(CSDC) needs buffers pool for efficient
work. Encryption used ptlrpc sec bulk, but it works with pages.

This patch extends sec bulk functionality to allocate different
size buffers. Memory shrinking and other usefull features
should still work as expected.

Signed-off-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Change-Id: I929b4dfdcb0e8197f3804629b000af0d4bd6f2a0
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50616
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn89

New tag 2.14.0-ddn89

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Id5516daccbef1da2f342afd79e912bc91bc9da13

LU-15046 osp: precreate thread vs connect race

lcs_exp (required for fid client) was initialized in osp_obd_connect()
which races with osp_precreate_thread(). the latter can get stuck if
lcs_exp is not initialized and then the whole precreation logic is
blocked until remount. instead the precreation thread can just wait
preliminary until lcs_exp is initialized properly.

Lustre-change: https://review.whamcloud.com/45099
Lustre-commit: 7e0a2b073701e23f6c941d249e034abe1043ccd6
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I7a42bf4b17ce5d46bc25bd548d81eb55f168804b
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51308
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16850 socklnd: remove ksnr_myiface from ksock_conn_cb

Drop ksnr_myiface: it is no longer needed since socklnd
TCP bonding got removed. There's one interface per
connection cb per peer_ni, and it can be accessed as
net->ksnn_interface.ksni_index.

Fix setting of ksni_nroutes accordingly. Duplication of
interface index in conn_cb and ksnn_interface was causing
the assertion
ASSERTION( net->ksnn_interface.ksni_nroutes == 0 )
in ksocknal_shutdown() to fail if the corresponding
device is deregistered before lnd shutdown.

Modify test_214 of sanity-lnet to create connections so that
the scenario of socklnd shutdown with NI on a deregistered
interface is covered.

Lustre-change: https://review.whamcloud.com/51148
Lustre-commit: f6be07c457385cfacd9b802e4cade9f6f6ab7d6f

Fixes: a7ee03d7ca4185 ("LU-16378 lnet: handles unregister/register events)
Test-Parameters: trivial
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: I4de164c9e64aa770164a8320b9460fadce49aa06
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51326
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13641 socklnd: remove remnants of tcp bonding

->ksnp_n_passive_ips is now always zero, so remove it and all uses of
it.  ->ksnp_passive_ips is gone too, as is ksocknal_ip2iface().

Lustre-change: https://review.whamcloud.com/48568
Lustre-commit: 3630e1eaf9db562a1de707762cd649db815459c8

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I5de6d027c545087c961673d8704f68c4f3dd5076
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51325
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-4130 lipe: Remove old hotpool scripts

Hotpool scripts now live in EMF repo.
The scrips in lipe/ are unmaintained and non-functional.

Change-Id: I2f288f7f68886015ddb9a85da4e437c9f03c3928
Signed-off-by: Nathaniel Clark <nclark@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51215
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Colin Faber <cfaber@ddn.com>
Reviewed-by: Shuichi Ihara <sihara@ddn.com>
Reviewed-by: Artur Novik <anovik@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15047 gss: gss integrity check with multi-rail

With multi-rail, a primary NID is used as node identifier, but LNet
decides which NID is actually used for sending/receiving data, on a
per request basis.
For the integrity check mechanism implemented as part of GSS, the
primary NID must be used in order to compute HMAC with the correct
key, independently of the actual NID for the current request.

Lustre-change: https://review.whamcloud.com/45277
Lustre-commit: c8301a65c5672a1d081669343466746df983eabc

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I2bf3974d3aa0e8365a9413dca56c69ee3734c12b
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51274
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-7495 utils: add --links option for lfs find

This adds a "--links" option for lfs find to filter files and
directories by the number of hard links. It also adds a printf format
'%n' to print the number of links for a file.

This commit also fixes '-l' as a short option for '--lazy' which was
added in 11aa7f8704c490b011f60f234c3ac9929ce76948 but the short option
did not work.

Lustre-change: https://review.whamcloud.com/50886
Lustre-commit: f759d6386d5d0edb95d683d97ca8d84c80080c1c

Signed-off-by: Thomas Bertschinger <bertschinger@lanl.gov>
Change-Id: I5d15bc290df8e8a08402f8d5cfa0a7139791b0a4
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Anjus George <georgea@ornl.gov>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51327
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16799 tests: fix sanity-krb5

sanity-krb5.sh needs to be fixed in several ways.
It cannot assume that the Kerberos credentials cache is FILE, and
expect ccache files to be under /tmp/krb5cc_*.
The lsvcgssd daemon must be launched with -vvv flags for easier
debugging.
Keyring needs to be cleared appropriately after using 'lfs flushctx'.

Lustre-change: https://review.whamcloud.com/50864
Lustre-commit: f8f8b3c574e95cb7272310bba19f97fe68cd9b11

Test-Parameters: trivial testlist=sanity-krb5 kerberos=true
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I31ca8d2d97e137c7ba9fa478d5432aeedb5135a8
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51265
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-12514 llite: move client mounting from obdclass to llite

Mounting a lustre client is currently handled
in obdclass, using services from llite.
This requires obdclass to load the llite module
and set up inter-module linkage.

The purpose of this was for common code to support both
client and server mounts.  This isn't really a good idea
and need to go. For lustre servers we already use a
separate filesystem type.

So move the mounting code from obdclass/obd_mount to llite/super25
and remove the inter-module linkages.
Add some EXPORT_SYMBOL() so that llite can access some helpers
that remain in obdclass.

Linux-commit: a989830c88149511ee840356d9c1b34304bac576

Lustre-change: https://review.whamcloud.com/37693
Lustre-commit: 53fa81765750e38f7879ed5092fd729c1bdc8a0f

Change-Id: Ia33bd55a042f90b178156c745a8072b516f00568
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51315
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16425 tests: skip interop recovery-small/144a/144b

Skip recovery-small test_144a and test_144b for old MDS
missing the fix and for its corresponding test.

Lustre-change: https://review.whamcloud.com/49679
Lustre-commit: 64faf832a6128cc55c0f3ffa0595d9715d3bdd25

Fixes: 240938f7b1 ("LU-8367 tests: cleanup_orphans hang reproducer")
Fixes: aa6250b741 ("LU-15724 tests: MDT failover hang reproducer")
Test-Parameters: trivial testlist=recovery-small env=ONLY=144 serverversion=2.14.0
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I77bfdf55d0218aa9e252f742cc90f1c61216d506
Reviewed-by: Sarah Liu <sarah@whamcloud.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51328
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16167 obdclass: fix lctl llog_print with skipped records

The 2a5b50d ignores the skipped records in configuration llog.
But if ioctl OBD_IOC_LLOG_PRINT return 0 record to display,
jt_llog_print_iter() will stop the processing and ignore the
non-skipped records at the end of the llog.

This patch returns to user space if the last index processed
(by llog_print_cb) is the last of llog file. If true,
jt_llog_print_iter() stops the processing.

Add regression test "conf-sanity test_123ai" for this issue.

Lustre-change: https://review.whamcloud.com/48586
Lustre-commit: c6da54aa7546440339265c644538d3d109e46bde

Fixes: 2a5b50d ("LU-15142 lctl: fixes for set_param -P and llog_print")
Test-Parameters: testlist=conf-sanity env=ONLY=123ai,SLOW=yes,ONLY_REPEAT=10
Signed-off-by: Etienne AUJAMES <etienne.aujames@cea.fr>
Change-Id: I78395268c57555e4fd2a4048ccf5b6132ca2877f
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alexander <alexander.boyko@hpe.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51316
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16619 build: Ubuntu jammy 5.19 client support

Ubuntu 5.19 kernel removed lsmcontext_init() and changed
security_dentry_init_security to require struct context *

Linux kernel linux-hwe-5.19
LSM: Removed scaffolding function lsmcontext_init

Linux kernel linux-hwe-5.19
LSM: security_dentry_init_security with struct lsmcontext

Lustre-change: https://review.whamcloud.com/50210
Lustre-commit: TBD (from d7001a6c68c334d15d3daeb932b8456b101153d2)

Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: Ib6479a2cd20df5e565ae6203e05df2afa3f3de31
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51002
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16327 llite: read_folio, release_folio, filler_t

Linux commit v5.18-rc5-221-gb7446e7cf15f
  fs: Remove aop flags parameter from grab_cache_page_write_begin()
flags have been dropped from write_begin() and
grab_cache_page_write_begin()

Linux commit v5.18-rc5-241-g5efe7448a142
  fs: Introduce aops->read_folio
Provide a ll_read_folio handler around ll_readpage

Linux commit v5.18-rc5-280-ge9b5b23e957e
  fs: Change the type of filler_t
Affects read_cache_page, provides a wrapper for read_cache_page
and wrappers for filler functions

Linux commit v5.18-rc5-282-gfa29000b6b26
  fs: Add aops->release_folio
Provide an ll_release_folio function based on ll_releasepage

Lustre-change: https://review.whamcloud.com/49199
Lustre-commit: 133ed0cf6f0c84d2b5b84e1de3ff2c54b1fb902d

Test-Parameters: trivial
HPE-bug-id: LUS-11357
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: Ibd4ec1133c80cd0eb8400c4cd07b50e421dd35c5
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50977
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13485 kernel: Parallel core configure tests

Transform the compile tests in lustre-core to run in parallel

Lustre-change: https://review.whamcloud.com/38361
Lustre-commit: a346cf6cf22c285e05177a5a5e70a7d8e5bd9fa8

LU-16586 build: Remove old check for linux_selinux_enabled

LC_SRC_HAS_LINUX_SELINUX_ENABLED is used and not defined, it
should be removed.

LC_PROG_LINUX_SRC and LC_PROG_LINUX_RESULTS are defined twice,
remove the unused ones.

Lustre-change: https://review.whamcloud.com/50111
Lustre-commit: 2bdff20f97707cf05118e822adf202ccb209e3a5

Fixes: a346cf6cf22 ("LU-13485 kernel: Parallel core configure tests")

Test-Parameters: trivial
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I46fa852659eb4db86a12ec4ad3efddd0fdd3a655
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50976
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Tested-by: Shuichi Ihara <sihara@ddn.com>
Reviewed-by: Shuichi Ihara <sihara@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16745 mdc: md_open_data should keep ref on close_req

md_open_data should keep a ref on mod_close_req,
otherwise the mod_close_req could be freed before
we try to access mod_close_req via md_open_data.

Lustre-change: https://review.whamcloud.com/50656
Lustre-commit: ca716f763f89af192ab86678ee9d14f49c80cae6

Change-Id: I621f7db389854326db298d99957a0bce43024b6e
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51198
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15123 tests: check quota reintegration after recovery

4th step of quota reintegration (reconciliation) waits for recovery
completion. So the tests (like sanity-quota/7a) should wait for
recovery completion before checking reintegration results.

Lustre-change: https://review.whamcloud.com/50688
Lustre-commit: 4432b6e2824775e292f96e202d6fc0db231bc749

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Id0aa5db01658621103d94ad6dafe91b2960b3a33
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sergey Cheremencev <scherementsev@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51232
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-14277 lod: statfs should not block a create

lod_qos_statfs_update() need a guarantee that targets
isn't changed, so it doesn't need to take a QoS mutex.

Lustre-change: https://review.whamcloud.com/41497
Lustre-commit: 6cdb08a0c598b20a4b0ce27f7fe4763b0ada3118

HPE-bug-id: LUS-2106
Signed-off-by: Alexander Boyko <alexander.boyko@hpe.com>
Signed-off-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Change-Id: Id9f0ea2fa006bee601d05e14b2e515fcf8248249
Reviewed-by: Andrew Perepechko <andrew.perepechko@hpe.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51246
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16755 kernel: update RHEL 8.8 [4.18.0-477.10.1.el8_8]

Update RHEL 8.8 kernel to 4.18.0-477.10.1.el8_8.

Lustre-change: https://review.whamcloud.com/51051
Lustre-commit: TBD (from 266266a8874cab2df8e9b5cc909d578dc239a1b1)

Test-Parameters: trivial fstype=ldiskfs \
clientdistro=el8.8 serverdistro=el8.8 testlist=sanity

Test-Parameters: trivial fstype=zfs \
clientdistro=el8.8 serverdistro=el8.8 testlist=sanity

Change-Id: I6d7703512f9c5a8b686f06e94f32f0e51c9b2001
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51054
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16756 kernel: new kernel [RHEL 9.2 5.14.0-284.11.1.el9_2]

This patch makes changes to support new RHEL 9.2 release
for Lustre client.

Lustre-change: https://review.whamcloud.com/50745
Lustre-commit: dd390cd315f505456e8e8a29e16a0f24b6878e27

Test-Parameters: trivial env=SANITY_EXCEPT=27J clientdistro=el9.2 testlist=sanity
Test-Parameters: trivial env=SANITY_EXCEPT=27J clientdistro=el9.2 serverdistro=el8.7 testlist=sanity

Change-Id: I4886bbf30d6d6a93c4adbfb68871e9d91f5b64de
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50964
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16755 kernel: RHEL 8.8 client and server support

This patch makes changes to support RHEL 8.8 release
with kernel 4.18.0-477.el8 for Lustre client and server.

Lustre-change: https://review.whamcloud.com/50708
Lustre-commit: fccc55866bc7da0b4908130d3f16bd355ffb7dc6

Test-Parameters: trivial fstype=ldiskfs \
clientdistro=el8.8 serverdistro=el8.8 testlist=sanity

Test-Parameters: trivial fstype=zfs \
clientdistro=el8.8 serverdistro=el8.8 testlist=sanity

Change-Id: Ie47f131e0340a601c8a5d748ecf9b1b73d4baa1f
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50963
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16517 build: pass extra configure options to "make debs"

While running "make debs", the configure command in debian/rules
ignores some user defined configure options. This patch fixes
the issue by adding the detection of the extra options into
debian/rules.

Lustre-change: https://review.whamcloud.com/50464
Lustre-commit: 3989529f22f5c54a98e445674b4b3cc443a3af5f

Test-Parameters: trivial clientdistro=ubuntu2204

Change-Id: Ia9db4e05abf33834cb3c853f4f0829dadc8d7400
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51174
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-6142 obdclass: resolve lu_ref checkpatch issues

Fix up all the checkpatch issues reported for the code handling
lu_ref. Also change USE_LU_REF to CONFIG_LUSTRE_DEBUG_LU_REF
which will match what will be upstream.

Lustre-change: https://review.whamcloud.com/44088
Lustre-commit: 5c98de856618f301035c7579e61fffa10cc726cc

Change-Id: I100e2679fc04c97eb67e4d44c4f6a6b530da6fa8
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51173
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-6142 lustre: change all 'struct seq_operations' to const

All 'struct seq_operations' are now 'static const'.

Lustre-change: https://review.whamcloud.com/39396
Lustre-commit: c5b90540731f13c37e6ad62c194ebef6725608a7

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ia0f6a2863835ec02fb7d283a099eea489f65ab3a
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51172
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15668 osd-ldiskfs: fix osd_bio_private double free

In osd_do_bio(), if the IO is fragmented and bio_alloc()
fails to allocate new bio, the bio_private still holds
osd_bio_private for last bio and will be double freed
in osd_do_bio() and dio_integrity_complete_routine().

Lustre-change: https://review.whamcloud.com/46879
Lustre-commit: 82d44b71f16bea216e04f7797ed032a783301d66

Test-Parameters: trivial
Change-Id: I42eaf95a85ec99a60359122054efb5beb0fb6104
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51187
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16829 gss: quiet noisy warnings

GSS code has plenty of debugging messages as warnings.
Update the code to make them just debugging messages and
nothing more.

Lustre-change: https://review.whamcloud.com/50613
Lustre-commit: 2247cf23ce55398ea4bebc1f8093275534736a0a

Test-Parameters: trivial

Change-Id: I53c471d758b0309ae10bab000211fa0381cabf87
Signed-off-by: Aurelien Degremont <adegremont@nvidia.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51190
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-6511 csdc: prefer uncompressed mirror for read

When accessing a mirrored file with both compressed and uncompressed
components, choose uncompressed components for read.

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I043b27bd891c039901075a08c76630f8f0f9f182
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50791
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7327 utils: add warning for 'lfs setstripe' compression

Add warning message to indicate that file compression is a preview
feature, not for production use.

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I1d5f9bdb11e33e8dbd65c6048d62e612020af3a3
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50658
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Reviewed-by: Colin Faber <cfaber@ddn.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>

EX-6856 utils: add 'lfs find' support for compressed file

* Add "--comp-flags=[^]compress" to locate file with/without
  compressed components.
* Add "--comp-flags=[^]nocompr" to locate file with/without setting
  component compress preference.
* Add "[!] --layout=compress" to locate file with/without compressed
  components.
* Add "[!] --compress-type=<compress-type>" to locate compressed file
  with/without specified compress algorithm.
* Add "[!] --compress-level=[+-]<compress-level>" to locate compressed
  file with/without specified compress level.
* Add "[!] --compress-chunk=[+-]<compress-chunk>" to locate compressed
  file with/without specified compress chunk in KiB.

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Ic36946738c3463fd862aeca4ee2e2c2ed85eff84
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/49930
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>

EX-6249 csdc: set compress component for file

* 'lfs setstripe' to support compress component

  --compress|-Z <type>[:<level>]
  Set component compression algorithm <type> and compress <level>;

  --compress-chunk=<size>
  Set compress data chunk size in KiB used by the compression
  algorithm, the value will be adjusted to power-of-two multiples of
  the base 64KiB.

  Example:
$ lfs setstripe -Eeof -Z lz4:5 --compress-chunk=512 <file>

* 'lfs getstripe' to show compress component parameters
  Display component's compression parameters if possible.

  --compress-type|--compr-type
  Print only the compress type if possible.

  --compress-level|--compr-level
  Print only the compress level if possible.

  --compress-chunk|--compr-chunk
  Print only the compress chunk size in KiB if possible.

  Example:
$ lfs getstripe <file>
    lcme_compr_type:     lz4
    lcme_compr_lvl:      5
    lcme_compr_chunk_kb: 512
      lmm_pattern:       raid0,compress

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Ife0382469cbc5099e0c6dc96534bb169ddeff61e
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/49509
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>

EX-6245 csdc: define compress component layout

Define a layout containing compress component with a compress
pattern so that old client cannot process it.

Add a client parameter to enable file compression support.

lctl get_param llite.*.enable_compression
lctl set_param llite.*.enable_compression={0,1}

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: If88d446fc7425644ea9484004ffa4e6d2a05f7e6
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/49508
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>

LU-16717 mdt: resume dir migration with bad_type

LFSCK may set hash type to "none,bad_type" upon migration failure,
set it back to "fnv_1a_64,migrating,bad_type,fixed" to allow
migration resumption. fnv_1a_64 is set because it's the default hash
type, and now that we don't know the hash type in the original
migration command, just try with it.

LFSCK just add "bad_type" flag on such directory, so that such
migration can always be resumed in the future.

Add sanity 230z.

Lustre-change: https://review.whamcloud.com/50797
Lustre-commit: 151650e468ab423e831c30d635ea380e0434a122

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I19606aefcb9115e6724843785aea89a1c380e23f
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51230
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-16717 mdt: treat unknown hash type as sane type

Directory migration failure may leave directory hash type as
LMV_HASH_TYPE_UNKNOWN|LMV_HASH_FLAG_BAD_TYPE, which should be treated
as sane hash type on existing directories, otherwise such directories
can't be unlinked.

Add sanity 230y.

Lustre-change: https://review.whamcloud.com/50796
Lustre-commit: 05cdb71ba6813570123613993f3cfcf74fc83561

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Ieffc0808d1db989d0bf9723f05cddb06f349e208
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51202
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-7596 tests: disable metadata_csum_seed for pcc cache device

latest e2fsprogs enable metadata_csum_seed feature when mkfs.ext4
a device, which is used in pcc test as a cache device. Such a ext4
fs cannot be mounted on an old kernel of el7. So disable this
feature for pcc cache device if it is detected in sanity-pcc test.

Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial testlist=sanity-pcc clientdistro=el7.9
Change-Id: I2ef3ff27b49fb479ec348b742cf614a43321813b
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51234
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn88

New tag 2.14.0-ddn88

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ib22ed479bd8d4f50b327b4287dd4c68cb1530db5

EX-7506 test: change ALWAYS_EXCEPT with function always_except

Change ALWAYS_EXCEPT with function always_except in replay-single.sh

Test-Parameters: trivial testlist=replay-single

Change-Id: I6fb4c350b1aa6f8c38920bf52e8ae42b64bdcd64
Signed-off-by: Wei Liu <sarah@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51086
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>