LU-11407 obdclass: add start time to stats files

When the stats files are initialized or reset, store the current
timestamp with the stats.  That allows computing average IO and
RPC rates over the accumulated stats lifetime, in addition to the
normal incremental operation rates found by comparing successive
values read from the stats file with the read interval.

Any stats that currently print the "snapshot_time:" header will
now also print "start_time:" and "elapsed_time:" fields as well.
Consolodate this printing into a helper function instead of
duplicating very similar code in many different functions.  Output
can't be exactly the same for all callers, because these fields are
embedded into different types of output files, but it is very close.

Change struct rename_stats and brw_stats to use a common name prefix.

Change the obd_job_stats timestamps to ktime_t so that we can use the
common helper function for printing the header.  It is easier to store
ojs_cleanup_interval internally as 1/2 of the maximum stats age, since
since division is more easily done when the value is initially set as
seconds compared to when it is ktime_t.  This may also be a tiny bit
more efficient since we don't do a divide/shift on each access.

Lustre-change: https://review.whamcloud.com/33201
Lustre-commit: ea2cd3af7bfabfa6876727ee44495f4c331bea8e

LU-16231 misc: fix stats snapshot_time to use wallclock

merged into this patch here to avoid landing broken stats.

Some "init" times were not initialized when stats were allocated or
cleared, do this for all stats shown by lprocfs_stats_header().

Rename struct osc_device fields from od_ to osc_ to avoid confusion
with struct osd_device. Having two od_stats was especially confusing.

Add a test case to verify snapshot_time, start_time, elapsed_time.

Lustre-change: https://review.whamcloud.com/48821
Lustre-commit: e42efe35eec7b9725f7f4fff86aaee04093366b0

LU-16110 lprocfs: make job_stats and rename_stats valid YAML

related to lprocfs_stats_header() are include here that were not
present when that patch was backported.  This fixes the output to
correctly indent the items to follow YAML formatting rules.

Lustre-change: https://review.whamcloud.com/48417
Lustre-commit: e96cb6ff1fea7a2bc62a6c0786fb0e07cbfda81a

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Iacefa17def455ef53a28fd14b6d4c670463ebbe5
Reviewed-by: Ben Evans <beevans@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Feng Lei <flei@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/49745
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-4141 test: extend expected number of keepalive msgs

Make test_165g tolerate wide range of number of
keepalive messages received by lamigo

Test-Parameters: trivial env=ONLY=165g testlist=sanity
Test-Parameters: trivial env=ONLY=165g testlist=sanity
Test-Parameters: trivial env=ONLY=165g testlist=sanity
Test-Parameters: trivial env=ONLY=165g testlist=sanity
Test-Parameters: trivial env=ONLY=165g testlist=sanity
Signed-off-by: Alexandre Ioffe <aioffe@ddn.com>
Change-Id: Ia3132d96420d571e9ed67d2baacdb25da2d52c4d
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50778
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7389 pcc: open file for detach O_RDONLY

llapi_pcc_detach_file is rdwr, but should just be rdonly.
This means files can be attached but not detached if the
client is mounted rdonly.

The fix is just to open the file in detach with O_RDONLY.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I3e289ab52ff760a8ab84a209b968109517953b52
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50749
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16768 lfs: copy optarg string other than using it directly

Copy optarg string for fp_format_printf_str lest it be messed
later.

Lustre-change: https://review.whamcloud.com/50733
Lustre-commit: 75db98cef3df8f9a0e1b6e7a5150f3c332e6167b

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Ib32883d3261ae921adf0fdd7b05bcbf728de7557
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50690
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16662 autoconf: fix configure test compile for CONFIG_KEYS

This fixes below configure error on Linux v5.19+:
$ ./configure --disable-server
...
checking whether to enable gss keyring backend... yes
checking if Linux kernel was built with CONFIG_KEYS in or as module...
no
configure: WARNING: GSS keyring backend requires that CONFIG_KEYS be
enabled in your kernel.
checking for keyctl_search in -lkeyutils... yes
configure: error: Cannot enable gss_keyring. See above for details.
$ grep CONFIG_KEYS -rn /boot/config-*
6884:CONFIG_KEYS=y

For in-tree IB support and without passing Linux src path when run
./configure, the LINUX_OBJ maybe just a soft link to O2IBPATH, they
are pointing to the same dir. E.g.:
O2IBPATH='/usr/src/kernels/6.1.8-3.0.0.7.oe1.aarch64'
LINUX_OBJ='/lib/modules/6.1.8-3.0.0.7.oe1.aarch64/build'
$ ls -l /lib/modules/6.1.8-3.0.0.7.oe1.aarch64/build
lrwxrwxrwx 1 root root 42 Feb  7 00:00
/lib/modules/6.1.8-3.0.0.7.oe1.aarch64/build ->
/usr/src/kernels/6.1.8-3.0.0.7.oe1.aarch64
In this case, current configure will put kernel's Module.symvers to
variable KBUILD_EXTRA_SYMBOLS. This should be avoided after kernel
v5.19 which contains commit "b8422711080f modpost: make multiple export
error". This making multiple export symbol as an error from a warning
which can be seen in the config.log:
...
ERROR: modpost: vmlinux: 'init_uts_ns' exported twice. Previous export
was in vmlinux
...

Lustre-change: https://review.whamcloud.com/50399
Lustre-commit: 321a533b868908f37d01a4b787f5a463a02e427c

Test-Parameters: trivial
Change-Id: I35295b3acc7fffb93716362f5d8c659eb922afcb
Signed-off-by: Xinliang Liu <xinliang.liu@linaro.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50569
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15300 mdt: refresh LOVEA with LL granted

this change tries to fix two problems:
1) mdt_reint_open() fetches LOVEA before layout lock is taken.
   this may race with another process changing the layout and
   may result in a stale layout returned with a granted layout
   lock - re-fetch LOVEA once layout lock is granted

2) lov_layout_change() should not apply old layouts which
   can get through when MDS doesn't take layout lock

3) LFSCK shouldn't ignore layout version stored on MDS to avoid
   a situation when version degrades compared to client's copy.

This patch misses an optimization and can result in a number of
useless calls to OSD to fetch LOVEA. To be fixed in a followup
patch.

Lustre-change: https://review.whamcloud.com/46413
Lustre-commit: efbe0f63eff8a9a7b192607382f6859e3b0088b8

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Idee1101d152ab09947faf6d75574a8761a7690a5
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50705
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16670 enc: make sure DoM files are correctly decrypted

Make sure DoM files are decrypted upon read by loading their
associated encryption context, via llcrypt_prepare_readdir()/
llcrypt_get_encryption_info().

Fix sanity-sec test_50 accordingly.

Lustre-change: https://review.whamcloud.com/50429
Lustre-commit: 1c424252d37c64e3c223c19dced3cad2649c1f61

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ie9ef3cbb08d2295a2fd10b9e9ab0862119c7723e
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Mikhail Pershin <mpershin@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50431
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-10026 csdc: reserve layout bits for compress component

Add layout bits for compress component layout.

* lcme_compr_type: compression type (gzip, lz4, lz4hc, lzo, etc.)
* lcme_compr_lvl: compression level (0=default, 1-15)
* lcme_compr_chunk_log_bits: chunk size = 2^(16+chunk_log_bits)

Component pattern:
* LOV_PATTERN_COMPRESS - file contains compressed data chunks and
       cannot be read by a client without decompression support.

Compress component flags:
* LCME_FL_COMPRESS - the component should be compressed with the
       compression algorithm stored in lcme_comp_type, at level
       lcme_comp_level, with chunk size 2^(16+lcme_chunk_log_bits)
* LCME_FL_PARTIAL - the component holds some uncompressed chunks due
       to poor write size/alignment, and may benefit from being
       recompressed as the full file data is available
* LCME_FL_NOCOMPR - the component should not be compressed because
       the data was found to be incompressible, or by user request

Lustre-change: https://review.whamcloud.com/49170
Lustre-commit: TBD (from 147d4eb27b85b4994a47539be6aceff212365ee5)

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Idca22cca87b01bba8a5b3c85ca62044abe1d30eb
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/49321
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16677 utils: add bitfields and ifdefs to wiretest

Add CHECK_BITFIELD() for "checking" bitfields in data structures
(currently only adds a comment to wiretest.c, maybe improve later).

Add CHECK_COND_START/FINISH() for adding #ifdef/#endif conditions
into wiretest, mainly for structs not used by (upstream) client.

Lustre-change: https://review.whamcloud.com/50479
Lustre-commit: 5a730827147714136b7d5035ca6115545a6b5ef0

LU-16677 utils: synchronize wirecheck.c and wiretest.c

wirecheck.c is not compilable and out of sync with wiretest.c.
The patch adds forgotten changes to wirecheck.c and replaces
wiretest.c by a version generated by "make newwiretest".

Lustre-change: https://review.whamcloud.com/50456
Lustre-commit: d43eb211995e0afb35690946c78ef6c82b9f86ad

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I8ecc4bdd5b5d651faa42f65ce8ea46da003ebbe5
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50751
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16524 sec: add fscrypt_admin rbac role

The purpose of the new fscrypt_admin rbac role is to control admin
tasks related to fscrypt. When not set, it is forbidden to all users
including root to modify existing protectors or policies, or create
new ones. But it remains possible to lock and unlock encrypted
directories.

Internally, this is achieved by marking fscrypt metadata files and
directories, i.e. everything under ROOT/.fscrypt, with a special mdt
object flag LOHA_FSCRYPT_MD.
Upon request processing, the mdt layer returns -EPERM if the flag
LOHA_FSCRYPT_MD is found on an object that is the target of a modify
request.
The LUSTRE_IMMUTABLE_FL flag is also returned to clients for such
objects.

sanity-sec test_64f is added to exercise the new fscrypt_admin flag.

Lustre-change: https://review.whamcloud.com/50184
Lustre-commit: 22bef9b6c64ef394a2efb41ce1388be71300af0d

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I99956499133994444ccd88e33340067790a182ce
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50339
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16524 sec: enforce rbac roles

There are 5 different rbac roles defined via nodemap:
- byfid_ops, to allow operations by FID (e.g. 'lfs rmfid').
- chlg_ops, to allow access to Lustre Changelogs.
- dne_ops, to allow operations related to DNE (e.g. 'lfs mkdir').
- file_perms, to allow modifications of file permissions and owners.
- quota_ops, to allow quota modifications.
Enforce these roles by checking the value of the 'rbac' nodemap
property on server side and returning -EPERM if operation is
forbidden.

Add sanity-sec test_64* to exercise these capabilities.

Lustre-change: https://review.whamcloud.com/49907
Lustre-commit: 971e025f5fb77f4eaaa1e9070598dfa6292a9678

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I37057f0ab50c02fa99db03cb04149a437e35ee0a
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50312
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-7391 lipe: lpcc_purge calculates disk usage as df

lpcc_purge calculate disk usage in the same way as df command.

Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial testlist=sanity-pcc
Change-Id: I43fe60188b1363d0ba58ea659b560b97807dc019
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50753
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Qian Yingjin <qian@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16057 obdclass: set OBD_MD_FLGROUP for ladvise RPC

ladvise RPC doesn't have OBD_MD_FLGROUP set, when RPC
reaches server, tgt_validate_obdo() will corrupt the FID
if it's seq is in FID_SEQ_NORMAL range.

Do not mess with seq in obdo_to_ioobj() and tgt_validate_obdo(),
since 2.0 all RPCs should have OBD_MD_FLGROUP set.

Add OBD_MD_FLGROUP for ladvise RPC to fix new client talking
to old servers.

Lustre-change: https://review.whamcloud.com/48080
Lustre-commit: bee803c6e440ba6b55e0ff356e5324f44cfa63eb

Change-Id: I373b7f32458b18e29d9bb716a912fe4a54eccac5
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-on: https://review.whamcloud.com/48080
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50755
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-12347 llite: do not take mod rpc slot for getxattr

The following scenario may lead to client eviction:
clientA                clientB                  MDS
threadA1: write to file F1, get
and hold DoM MDC LDLM lock L1:
   ->cl_io_loop()
    ->cl_io_lock()
     :
     ->mdc_lock_granted()
      ->lock->l_writers++
      [hold ref until write done]

threadA2-A8: create files F2-F8:
   ->ll_file_open()
    ->mdc_enqueue_base()
     ->ldlm_cli_enqueue()
      ->ptlrpc_get_mod_rpc_slot()
      ->ptlrpc_queue_wait()
      [hold RPC slot until create done]

                                              OST(s) in recovery.
                                              MDS waiting on OST(s) to
                                              precreate new objects.

threadA1:
   -> cl_io_start()
    -> __generic_file_aio_write()
     -> file_remove_suid()
      -> ll_xattr_cache_refill()
       -> mdc_xattr_common()
        -> ptlrpc_get_mod_rpc_slot()
        [blocked waiting for RPC slot]

                       threadB1: write file F1,
       enqueue DoM MDC lock L1

                                              MDS sends blocking AST
                                              to clientA for lock L1

ldlm_threadA3: cannot cancel busy lock L1:
   -> ldlm_handle_bl_callback()
   ["Lock L1 referenced, will be cancelled later"]

                                              MDS evicts clientA for
                                              not cancelling lock L1

threadA1: never completes write:
  ->cl_io_end()
   ->cl_io_unlock()
    ->osc_lock_cancel()
     ->lock->l_writers--;

The fix is to add IT_GETXATTR to list of operations which do not
need mod rpc slot.

Tests to illustrate the issue is added.

wait_for_function(): total sleep time (wait) is to be equal to max
when 1 is returned.

Lustre-change: https://review.whamcloud.com/44151
Lustre-commit: eb64594e4473af859e74a0e831316cead0f5c49b

Signed-off-by: Vladimir Saveliev <vlaidimir.saveliev@hpe.com>
HPE-bug-id: LUS-7271
Change-Id: I1b80677df084bda141b9ac58a78b765bd0b14a41
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andrew Perepechko <andrew.perepechko@hpe.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50754
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16341 quota: fix panic in qmt_site_recalc_cb

The panic occurred due to empty qit_lqes array after
qmt_pool_lqes_lookup_spec. Sometimes it is possible if
global lqe is not enforced. Return -ENOENT from
qmt_pool_lqes_lookup_spec if no lqes have been added.
It fixes following panic:
BUG: unable to handle kernel NULL pointer dereference at
00000000000000f8
...
RIP: 0010:qmt_site_recalc_cb+0x2ec/0x780 [lquota]
...
[ffffa5564118fda0] cfs_hash_for_each_tight at ffffffffc0c72c81
[libcfs]
[ffffa5564118fe08] qmt_pool_recalc at ffffffffc142dec7 [lquota]
[ffffa5564118ff10] kthread at ffffffffb45043a6
[ffffa5564118ff50] ret_from_fork at ffffffffb4e00255

Add test sanity-quota_14 that reproduces above panic
without the fix.

HPE-bug-id: LUS-11007
Change-Id: Ie51396269fae7ed84379bef5fc964cce789eba7c
Signed-off-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50793
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sergey Cheremencev <scherementsev@ddn.com>

LU-14541 llite: Check for page deletion after fault

Before completing a page fault and returning to the kernel,
we lock the page and verify it has not been truncated.  But
we must also verify the page has not been deleted from
Lustre, or we can return a disconnected (ie, not tracked by
Lustre) page to the kernel.

We mark deleted pages !uptodate, but this doesn't matter
for faulted pages, because the kernel assumes they are
returned uptodate, and maps them in to the process address
space.  Once mapped, the page state is not checked until
the page is unmapped.

But because the page is referenced by the mapping, it stays
in the page cache even though it's been disconnected from
Lustre.

Because the page is disconnected from Lustre, it will not
be found and cancelled on lock cancellation.  This can
result in stale data reads.

This is particularly an issue with releasepage (called from
drop_caches or under memory pressure), which can delete
pages separately from cancelling covering locks.

If releasepage is disabled, which is effectively what
"LU-14541 llite: Check vmpage in releasepage"
does, this is not an issue.  But disabling releasepage
causes other problems and is incorrect anyway.

Lustre-change: https://review.whamcloud.com/49653
Lustre-commit: b3d2114e538cf95a7e036f8313e9095fe821da79

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: If1164db8f8e92a1cf811431d56d15f30d8eb3faa
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50303
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13081 tests: skip sanity test_151/test_156

Skip both sanity test_151 and test_156 during interop testing,
since this is really testing server-side functionality only
(OSS caching behavior). And it makes sense to just exclude
test_151 and test_156 during interop testing, otherwise it
seems that the client version of the test can become
inconsistent with the caching behavior/tunables on the OSS
and the failures don't mean anything. There is enough
non-interop testing to catch any regressions in the OSS
cache behavior.

Lustre-change: https://review.whamcloud.com/50777
Lustre-commit: TBD (from 62cd8d19ff103e6e8a2b4bb7cdc00815ddb0edac)

Test-Parameters: trivial
Signed-off-by: Alex Deiter <adeiter@tintri.com>
Change-Id: I39a8b54894d5b0c7573e6c56d1f8e1ba02b3e3fe
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50887
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16775 tests: cleanup target after sanity-sec test_31

sanity-sec test_31 adds an LNet network tcp999, and associated
servicenode param on MDS target. This param must be cleared when
exiting the test, otherwise it can lead to incorrect client HA
behavior, trying to reach out to the fake service nodes.

Lustre-change: https://review.whamcloud.com/50766
Lustre-commit: TBD (e8b6c5a1128210a3a2bf525b5b489224932c8f88)

Test-Parameters: trivial
Test-Parameters: mdscount=2 mdtcount=4 osscount=1 ostcount=8 clientcount=2 \
    clientselinux testlist=sanity-sec env=SHARED_KEY=true,ONLY="31 61"
Fixes: c508c94268 ("LU-16557 client: -o network needs add_conn processing")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: If3a1926855bd23e9154c9a32b7a555e934e94565
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50767
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16515 tests: disable sanity test_118c/118d

Temporarily disable sanity test_118c and test_118d until there is
a fix available, since this is failing a large fraction of tests.

Lustre-change: https://review.whamcloud.com/50470
Lustre-commit: 7c52cbf65218d77c0594f92981173aa7d78f6758

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I16ebbc470a126bb99b5c3ecdf93407d6b73ebbe5
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50794
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn86

New tag 2.14.0-ddn86

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Iad3d4ffca7cdbb06657a679b5b580af788b6a5ec

LU-16603 protocol: add OBD_BRW_COMPRESSED

so the client can hint OST the data is compressed

Test-Parameters: trivial
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I4b721db3ad349d5745ee6698de368d0cb0138954
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50154
Reviewed-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Lustre-change: with short Gerrit URL https://review.whamcloud.com/50154
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50372
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15374 tests: check FULL and IDLE for client import state

The client-to-OST import state can be FULL or IDLE.

Lustre-change: https://review.whamcloud.com/49298
Lustre-commit: 25fb82eb413389b6023e0e61f7efb71e91d15c01

Test-Parameters: trivial testgroup=review-dne-part-3

Fixes: 25606a2ce1 ("LU-15342 tests: escape "|"")
Fixes: 3da8f014fd ("LU-12857 tests: allow clients to be IDLE after recovery")
Fixes: 5a6ceb664f ("LU-7236 ptlrpc: idle connections can disconnect")

Change-Id: I3582ceb273d241ee71fe907f6d1423746e453faa
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50632
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15342 tests: escape "|"

escape "|" on want="FULL|IDLE" to protect interpretation
by shell:
  sh: IDLE: command not found

Lustre-change: https://review.whamcloud.com/45788
Lustre-commit: 25606a2ce19e94c13694d46c3f15e9a10df40a91

Fixes: af666bef05 ("LU-12857 tests: allow clients to be IDLE after recovery")
Test-Parameters: trivial
Signed-off-by: Elena Gryaznova <elena.gryaznova@hpe.com>
Change-Id: I2f885ea225ba43537f37b8dad1c2e0cd8f652a79
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50631
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-12857 tests: allow clients to be IDLE after recovery

If clients are not connected to an OST when it fails (connection
is IDLE), they do not need to be involved in recovery, so this
should not be considered an error when checking the client state.

Lustre-change: https://review.whamcloud.com/45318
Lustre-commit: af666bef058c5b7997527fc851a84a89375912fb

Test-Parameters: trivial testlist=recovery-mds-scale env=SLOW=no
Test-Parameters: testlist=conf-sanity
Test-Parameters: testlist=replay-dual,replay-single
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I6cfeb718acd233378ed1608f22061bc15c3ebbe5
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50630
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16706 kernel: update RHEL 9.1 [5.14.0-162.22.2.el9_1]

Update RHEL 9.1 kernel to 5.14.0-162.22.2.el9_1 for Lustre client.

Lustre-change: https://review.whamcloud.com/50512
Lustre-commit: 9c316c69b9788ac219540e560a7831b88b81b690

Test-Parameters: trivial clientdistro=el9.1 testlist=sanity
Test-Parameters: trivial serverdistro=el8.7 clientdistro=el9.1 testlist=sanity
Change-Id: Ib5186e6f0dcd89660b7000db7f37c0c5a29f944f
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50528
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16710 kernel: update RHEL 8.7 [4.18.0-425.19.2.el8_7]

Update RHEL 8.7 kernel to 4.18.0-425.19.2.el8_7.

Lustre-change: https://review.whamcloud.com/50548
Lustre-commit: TBD (from 8cd92d6bdb3726fb25286ead1276622b884805d6)

Test-Parameters: trivial fstype=ldiskfs \
clientdistro=el8.7 serverdistro=el8.7 testlist=sanity

Test-Parameters: trivial fstype=zfs \
clientdistro=el8.7 serverdistro=el8.7 testlist=sanity

Change-Id: I17e43d98d1a3c7217e61771e8ed78a7123f9313f
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50551
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16563 lnet: use discovered ni status to set initial health

If not routing, track local NI status in the ping buffer
such that locally recognized "down" state, for example,
due to a downed network interface/link, is available
to any discovering peer.

On the active side of discovery, check peer NI status so if NI
is down, decrement its health score and queue for recovery.

Lustre-change: https://review.whamcloud.com/50027/
Lustre-commit: da230373bd14306cb97fb48748ebce205f09d468

Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: I513c7942099c0da9088fa6d4460f76386ea91d3b
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50040
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16637 llite: call truncate_inode_pages() under inode lock

truncate_inode_pages() is required to be called under (and serialised
by) inode lock.

Lustre-change: https://review.whamcloud.com/50284
Lustre-commit: ef9be34478036db0544753e33030fff7e32bfe44

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I0f1a09c8756522f87a2e5d8030d12f80e2f630b4
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50365
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16621 enc: file names encryption when using secure boot

Secure boot activates lockdown mode in the Linux kernel.
And debugfs is restricted when the kernel is locked down.
This patch moves file names encryption from debugfs to sysfs.

Lustre-change: https://review.whamcloud.com/50219
Lustre-commit: 716675fff642655c4d4715654b0b4880b96139b6

Test-Parameters: trivial testlist=sanity-sec
Signed-off-by: Alex Deiter <alex.deiter@gmail.com>
Change-Id: I434714941ffac2a4694cabd33f613aef70933678
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50578
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>

EX-4141 lipe: keepalive message from ofd_access_log_reader

Lamigo checks ofd_access_log_reader for keepalive message
feature availability. Start ofd_access_log_reader with keepalive
message option if it is available.
Added tests for lamigo and ofd_access_log_reader to
test keepalive message feature
Fixed a bug when trace file is opened.
Moved FATAL, ERROR, DEBUG macros to include and
Used FATAL macro wherever possible

Signed-off-by: Alexandre Ioffe <aioffe@ddn.com>
Test-Parameters: trivial testlist=hot-pools,sanity
Test-Parameters: trivial testlist=hot-pools,sanity
Test-Parameters: trivial testlist=hot-pools,sanity
Change-Id: I0f218e2394c9d0ab6cd425860ba79956a10cbd58
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50389
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16615 utils: add messages in l_getidentity

Add time related messages in l_getidentity to help debug upon
timeout, which may cause -EACCES error in user applications.

Lustre-change: https://review.whamcloud.com/50213
Lustre-commit: d5b26443a3d33d68e5747fecc591baa887bc5b89

Test-Parameters: trivial
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I87ebfb85d05e19886d8becc6b14ed0233eaed42d
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/49717
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16524 nodemap: add rbac property to nodemap

Add new rbac property to nodemap. Internally this is a mask of allowed
roles. Externally it defaults to all, which means all roles are
allowed, and it can take the following values (multiple can be
specified, comma separated), with the semantic:
- byfid_ops, to allow operations by FID (e.g. 'lfs rmfid').
- chlg_ops, to allow access to Lustre Changelogs.
- dne_ops, to allow operations related to DNE (e.g. 'lfs mkdir').
- file_perms, to allow modifications of file permissions and owners.
- quota_ops, to allow quota modifications.
Apart from all, any role not explicitly specified is forbidden. And to
forbid all roles, use 'none' value.

Update lctl-nodemap-modify man page to mention this new property.

Lustre-change: https://review.whamcloud.com/49873
Lustre-commit: 5e48ffca322c3c72d3b83b0719f245fc6f13c8e4

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I4cedf03c75948f4b1e9b55292414ab9110701874
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50293
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16338 readahead: add stats for read-ahead page count

This patch adds the stats for read-ahead page count:

lctl get_param llite.*.read_ahead_stats
llite.lustre-ffff938b7849d000.read_ahead_stats=
snapshot_time             4011.320890492 secs.nsecs
start_time                0.000000000 secs.nsecs
elapsed_time              4011.320890492 secs.nsecs
hits                      4 samples [pages]
misses                    1 samples [pages]
zero_size_window          4 samples [pages]
failed_to_reach_end       1 samples [pages]
failed_to_fast_read       1 samples [pages]
readahead_pages           1 samples [pages] 255 255 255

Lustre-change: https://review.whamcloud.com/49224
Lustre-commit: cdcf97e17e73dfdd65c4e46bb30c4a07f5e710cf

Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: Iada06eb7d78ab28cfcc7167e49d25da252da4009
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <farr0186@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50070
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16579 llite: fix the wrong beyond read end calculation

During the test, we found a dead loop in the read path which
retruns AOP_TRUNCATED_PAGE(0x8001) endless.
The reason is that the calculation of the ending beyond offset is
wrong: (iter->count + iocb->ki_pos).
The ending beyond offset was supposed to be not changed during
the read I/O loop for each page in buffered I/O mode.
However, @iter->count is decreased with read bytes when finished
the read of each page: @iter->count -= read_bytes.

In this patch, we store the ending beyond page index in
@lcc->lcc_end_index before call @generic_file_read_iter into a
loop for each read page and solve this bug.

Lustre-change: https://review.whamcloud.com/50065
Lustre-commit: ae356dc325877bd130ad94acc5f3610898de8a8a

Fixes: 2f8f38effa ("LU-16412 llite: check read page past requested")
Signed-off-by: Qian Yingjin <qian@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I5bb7ab82e5e2de8b9bd911798fb8ae65fc7c91af
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50068
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16494 fileset: check fileset for operations by fid

Some operations by FID, such as lfs rmfid, must be aware of
subdirectory mount (fileset) so that they do not operate on files
that are outside of the namespace currently mounted by the client.

For lfs rmfid, we first proceed to a fid2path resolution. As fid2path
is already fileset aware, it fails if a file or a link to a file is
outside of the subdirectory mount. So we carry on with rmfid only
for FIDs for which the file and all links do appear under the
current fileset.

This new behavior is enabled as soon as we detect a subdirectory mount
is done (either directly or imposed by a nodemap fileset). This means
the new behavior does not impact normal, whole-namespace client mount.

sanity test_421h is added to exercise this new capability.

Lustre-change: https://review.whamcloud.com/49696
Lustre-commit: 9a72c073d33b0454229402c0cc930dc4e796107b

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I47136ac0a3324b9afdd01b0f902abc37938bd361
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50072
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16310 sec: Lustre/HSM on enc file with enc key

Support for Lustre/HSM on encrypted files when the encryption key is
available requires similar attention as with file migration.
The volatile file used for HSM restore must have the same encryption
context as the Lustre file being restored, so that file content
remains accessible after the layout swap at the end of the restore
procedure.

Please note that using Lustre/HSM with the encryption key creates
clear text copies of encrypted files on the HSM backend storage.

Lustre-change: https://review.whamcloud.com/49153
Lustre-commit: df7a8d92d2378e236ee83b559e7a1158f84e63f4

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I99cba202cd2c7c747bbe5c4ec7d9208c7f6baf4b
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Etienne AUJAMES <eaujames@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/49899
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16205 sec: fid2path for encrypted files

Add support of fid2path for encrypted files. Server side returns raw
encrypted path name to client, which needs to process the returned
string. This is done from top to bottom, by iteratively decrypting
parent name and then doing a lookup on it, so that child can in turn
be decrypted.

For encrypted files that do not have their names encrypted, lookups
can be skipped. Indeed, name decryption is a no-op in this case, which
means it is not necessary to fetch the encryption key associated with
the parent inode.

Without the encryption key, lookups are skipped for the same reason.
But names have to be encoded and/or digested. So server needs to
insert FIDs of individual path components in the returned string.
These FIDs are interpreted by the client to build encoded/digested
names.

Add sanity-sec test_63 to exercise this new capability.

Lustre-change: https://review.whamcloud.com/48930
Lustre-commit: fa9da556ad22b1485c53cf0337dc6872d89aedfa

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I165bf2e5657037ae2e25c9378e4713537ea94bec
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/49898
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16205 sec: reserve flag for fid2path for encrypted files

Reserve OBD_CONNECT2_ENCRYPT_FID2PATH connection flag for fid2path
support for encrypted files.
This connection flag is required so that newer servers continue to
return -ENODATA to older clients.

Lustre-change: https://review.whamcloud.com/49028
Lustre-commit: 6f74bb60ff6c58f4a2647556124c501100330f4c

Test-Parameters: trivial
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I505b90a061687a7ef481adacca98908c96e487be
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/49897
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16369 ldiskfs: do not check enc context at lookup

On rhel8, ldiskfs should not check for encryption context of inodes
upon lookup. On these kernels, ext4 is not encryption aware, so just
assume context is fine when target is mounted as ldiskfs.

Lustre-change: https://review.whamcloud.com/49324
Lustre-commit: 540c293a4d0fc80253670b3db8d6722da43284ad

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I9f9813d290ea24b34f710e2c8219e856ca8fbc58
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/49894
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16091 enc: S_ENCRYPTED flag on OST objects for enc files

Add a dumb encryption context on OST objects being created, when the
LUSTRE_ENCRYPT_FL flag gets set in the LMA, for ldiskfs backend
targets. This leads ldiskfs to internally set the LDISKFS_ENCRYPT_FL
flag on the on-disk inode. Also, it makes e2fsprogs happy to see an
enc ctx for an inode that has the LDISKFS_ENCRYPT_FL flag.

Add a dumb encryption context on OST objects being opened, if there is
not already one, for ldiskfs backend targets. This is done by adding
the LUSTRE_ENCRYPT_FL flag if necessary, at the same time as atime
gets updated. It is some sort of live self-check that fixes OST
objects created with an older Lustre version.

Enhance lfsck to detect and fix OST objects belonging to encrypted
files that are missing the encryption flag. This is implemented in the
MDT-OST consistency routine, as part of the layout checking.

Also add sanity-sec test_62 and sanity-lfsck test_42 to exercise this.

Note this patch does not add any dumb encryption context on OST
objects when the backend is ZFS.

Lustre-change: https://review.whamcloud.com/48198
Lustre-commit: 348446d6370b3f63f0da8a96997b3295f896c6fb

Test-Parameters: testlist=sanity-sec mdscount=2 mdtcount=4 osscount=1 ostcount=8 clientcount=2 fstype=zfs
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I6bee3c82ee4d1a52275facf9e2b0d60061e0beef
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/49896
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16454 mdt: Add a per-MDT "max_mod_rpcs_in_flight"

Value max_mod_rpcs_per_client doesn't define a static number of
slots for the per-client replies or anything, and the only
thing it is used for is to pass the limit to the client. For the
same reason, there also doesn't appear to be a particularly hard
limitation why the client cannot change and exceed the
server-provided parameter, except to avoid overloading the server
with too many RPCs at once, but that may also be true of the
current limit with a larger number of clients, no different than
"max_rpcs_in_flight".

This fix adds a tunable parameter "max_mod_rpcs_in_flight" per MDT
to lustre/mdt/mdt_lproc.c so that it can be set
with "lctl set_param" at runtime. The max_mod_rpcs_per_client global
setting is marked "deprecated" but is still used as the default
value when creating an MDT.

Merge
8449bd91ba LU-16558 mdt: Fix max limit for "max_mod_rpcs_in_flight"
into this patch.

Lustre-change: https://review.whamcloud.com/49749
Lustre-commit: f16c31ccd91d66caba69d3ceea6a61c1682df59e
Lustre-change: https://review.whamcloud.com/50010
Lustre-commit: 8449bd91ba45c47614231a9bfe141e700dec8bb9

Signed-off-by: Vitaliy Kuznetsov <vkuznetsov@ddn.com>
Change-Id: I27cfcb68e1a534e80e6a2dbf2e1affc430803b49
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50450
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-8367 osp: wait for precreate on reformatted OST

We should wait for precreate rpc to finish when we see a just
reformatted/replaced OST, otherwise the client could try
to access the object on OST before it's created.

Do not use sync_trans when recreating the objects on the
reformatted/replaced OST.

Fix detecting reformatted OST for FID_SEQ_NORMAL, for such
seqs the oid will be initialized as LUSTRE_FID_INIT_OID,
which is 1.

Change-Id: I4aebb9d573aa352dd7897e5f1129dc2117a084bb
Fixes: 63e17799a3 ("LU-8367 osp: enable replay for precreation request")
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/49151
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50410

LU-8367 osp: enable replay for precreation request

Lustre has some kind of deadlock between osp_precreate_thread()
and stripe allocation at osp_precreate_reserve(). Stripe allocation
thread allocated objects and sleeps for more objects at
osp_precreate_reserve() in case of OST failover. After reconnection,
osp_precreate_thread() calls osp_precreate_cleanup_orphans() to
synchronize last id and clean-up unused objects, but it waits zero
object reservation(d->opd_pre_reserved). So, no more objects could
be created at OST and no reserved objects could be freed.
This produce slow creates messages and MDT creation threads hang
osp_precreate_reserve()) kjcf05-OST0003-osc-MDT0001: slow creates,
 last=[0x340000400:0x23a4f483:0x0], next=[0x340000400:0x23a4f378:0x0],
 reserved=267, sync_changes=0, sync_rpcs_in_progress=0, status=0
The issue reproduced more often with over stripe feature.

No need to do orphan clean-up phase when MDT supports
resend/replay for precreation request. This behaviour resolves the
osp_precreate_cleanup_orphans() hang and unblocks objects creation.

Force creation logic is added to support reformatted OST with a same
index. It was done during orphan clean-up phase before this.

Sanity tests 27S and 822 become invalid. 27S is based on orphan
clean-up after reconnection, 822 is based on not resendable
OST_CREATE request. These tests are removed.

Lustre-change: https://review.whamcloud.com/46889
Lustre-commit: 63e17799a369e2ff0b140fd41dc5d7d8656d2bf0

HPE-bug-id: LUS-10793
Signed-off-by: Alexander Boyko <alexander.boyko@hpe.com>
Change-Id: I21287b51252e573e796fac69ee3df6ac90e28c10
Reviewed-by: Vitaly Fertman <vitaly.fertman@hpe.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/49821
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15660 statahead: statahead thread doesn't stop

Add a barrier to ensure sai_task changing can be seen
when access it without locking. Else the statahead
thread could sleep forever since wake_up was lost.

Lustre-change: https://review.whamcloud.com/47673
Lustre-commit: b977caa2dc7dddcec9e20d393ee79dfa9fe31c0d

Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: I211e99f1bdddaaaf028a205658f603fda034d389
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50559
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16478 target: disconnected export

eviction can race with a reconnect and this in turn can lead
to a leaked export reference prevent further umount -
mdt_obd_reconnect() grabs a reference via nodemap_add_member().
call obd_disconnect() if such a case observed to balance
obd_reconnect().

Lustre-change: https://review.whamcloud.com/50041
Lustre-commit: 654d5f3fa4df2a0f7275a6da0f050a18881f4f75

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I3fd49429ef40ef391d58e042e091258dcb9add72
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50427
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16263 lov: continue fsync on other OST objs even on -ENOENT

When fsync races with truncate, we'd continue to other OST object's
fsync even some stripe fsync returns -ENOENT, so that on client it
could potentially discard caching pages by calling
osc_io_fsync_start()->osc_cache_writebase_range().

Lustre-change: https://review.whamcloud.com/50005
Lustre-commit: 927b5cd49c3369d533d7f8dc5c8df497aaf33b6e

Test-Parameters: testlist=sanity ostcount=4 env=ONLY="273b 273c",ONLY_REPEAT=120
Test-Parameters: testlist=sanity ostcount=4 env=ONLY="273b 273c",ONLY_REPEAT=120
Test-Parameters: testlist=sanity ostcount=4 env=ONLY="273b 273c",ONLY_REPEAT=120
Test-Parameters: testlist=sanity ostcount=4 env=ONLY="273b 273c",ONLY_REPEAT=120
Test-Parameters: testlist=sanity ostcount=4 env=ONLY="273b 273c",ONLY_REPEAT=120
Test-Parameters: testlist=sanity ostcount=4 env=ONLY="273b 273c",ONLY_REPEAT=120
Test-Parameters: testlist=sanity ostcount=4 env=ONLY="273b 273c",ONLY_REPEAT=120
Test-Parameters: testlist=sanity ostcount=4 env=ONLY="273b 273c",ONLY_REPEAT=120
Test-Parameters: testlist=sanity ostcount=4 env=ONLY="273b 273c",ONLY_REPEAT=120
Test-Parameters: testlist=sanity ostcount=4 env=ONLY="273b 273c",ONLY_REPEAT=120
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I457ba80063086e310df55aaa22778b51a6ea211e
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50195
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16465 llite: fix LSOM blocks for ftruncate and close

LSOM is updated on close and setattr request.
For the setattr, clients do not know the numbers blocks yet (OSTs
setattr requests have to finish). So the blocks number is set to 1 by
the server.

The close request send after a ftruncate() will wrongly update LSOM
back to its old blocks number. This is because clients do not update
the inode.i_blocks after an OST setattr.

Then the MDS will denied a client close request to update LSOM to its
correct blocks number. Only truncates are allowed to decrease the
blocks number (server side).

This patch force the client inode update at the end of an OST setattr.
And it tries (if no contention on the inode_size) to update the inode
at the end of an OST fsync or a sync IO.

Update sanity test 806/807 for this use case.

Lustre-change: https://review.whamcloud.com/49675
Lustre-commit: dfb08bbf77a1362f79c3738cc3952f8db2e46511

Test-Parameters: testlist=sanity env=ONLY=806,807,ONLY_REPEAT=20
Test-Parameters: fstype=zfs testlist=sanity-flr env=ONLY=70,ONLY_REPEAT=10
Signed-off-by: Etienne AUJAMES <eaujames@ddn.com>
Signed-off-by: Xing Huang <hxing@ddn.com>
Change-Id: Ib1afde97071ebae56f0b413ec444403c3cdebd02
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Qian Yingjin <qian@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50611
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15404 ldiskfs: use per-filesystem workqueues to avoid deadlocks

Calling flush_scheduled_work() under s_umount is dangerous and may
cause deadlocks. This patch backports the fix from
https://lore.kernel.org/all/20220402084023.1841375-1-anserper@ya.ru/

Lustre-change: https://review.whamcloud.com/50354
Lustre-commit: 616fa9b581798e1b66e4d36113c29531ad7e41a0

Fixes: e239a14001 ("LU-15404 ldiskfs: truncate during setxattr leads to kernel panic")
Signed-off-by: Andrew Perepechko <andrew.perepechko@hpe.com>
Change-Id: Ia191b70166f94f34e96a282ec18bd8650871e108
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50585
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16371 ldlm: clear lock converting flag on resource cleanup

During resource cleanup clear lock's converting flag so that
ldlm_cli_cancel() won't erroneously trip the assertion, the assertion
is used for normal lock revoke callbacks.

Lustre-change: https://review.whamcloud.com/49339
Lustre-commit: 4990f4ef5eb81d8017c9992c1f6924527dc8ce60

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I1be4d7f16dbc7e026b460fd5358a0fe509b97a59
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/49337
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16612 llite: protect cp_state with vmpage lock

cl_page_make_ready() calls cl_page_io_start() without vmpage lock
protection, and that could mess up cl_page's cp_state/cp_owner.

Lustre-change: https://review.whamcloud.com/50180
Lustre-commit: d03b038d0dd8360dc896ceb7f3cee99245551cb8

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Id0df7e14246aa561494a9b6e581cebc55241c4b9
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50182
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16646 krb: improve lookup of user's credentials

Rather than only looking up for user's credentials in hard-coded
FILE:/tmp/krb5cc_<uid>, try first the default ccache on the system,
and then fallback to files matching /tmp/*krb5cc* and
/run/user/<uid>/*krb5cc*.

Lustre-change: https://review.whamcloud.com/50377
Lustre-commit: 4b12f2b9ae14556285529b40985a7c3aacdbb7f4

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ic2bedb4cc12e9adad0ce63bd0617b2e0ec13907e
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50653
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16646 krb: use system ccache for Lustre services

Instead of hard-coding a FILE credentials cache for Lustre services,
comply with the system configuration in place and use the default
ccache.

Lustre-change: https://review.whamcloud.com/50342
Lustre-commit: 9784178eff0ad61acedb50ed4bfbb423cafe32b4

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ib89fe117e9f1d937925a02c7ed786a81cd8954cb
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50652
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16646 krb: get rid of MEMORY private cache for krb creds

On client side, Kerberos credentials for root are cached in MEMORY,
but this is just in addition to the original credentials cache.
As there is no need to cache credentials twice, get rid of this
MEMORY private cache.

Lustre-change: https://review.whamcloud.com/50341
Lustre-commit: 5731acd4997f422e3e5561d0ed73359537a7b65f

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ifd02ee6cfafc27347b3c31e0dbbaab15190cf883
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50651
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16630 sec: improve Kerberos cross-realm trust remapping

Improve Kerberos cross-realm trust remapping by leveraging existing
Kerberos mechanisms. gss_localname() can be used to resolve usernames:
it goes through the auth_to_local translation rules in krb5.conf and
thus can easily be configured by security administrators.
This new mechanism does not replace the existing and rudimentary
mapping based on /etc/lustre/idmap.conf. If /etc/lustre/idmap.conf
exists, it is used for user mapping. If not, the new mechanism based
on gss_localname() gets involved.
But we now print a warning that idmap.conf is deprecated if we detect
it is in use.

Lustre-change: https://review.whamcloud.com/50259
Lustre-commit: 3214d4d860e36b6aa07addad9e600fd754fc9149

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Iaaf15a757dc246673e2f412181219cc978079fab
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50292
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15690 tests: skip replay-ost-single/12a on old server

Skip 12a of replay-ost-single for older server version.

Lustre-change: https://review.whamcloud.com/50701
Lustre-commit: 3201bd4ac497540f74c7295b9ec541aa9775537c

Test-Parameters: trivial testlist=replay-ost-single env=ONLY=12a
Fixes: 28769c65987c ("LU-15195 ofd: missing OST object")
Signed-off-by: Xing Huang <hxing@ddn.com>
Change-Id: I473452a5326691f4394c9e3ab2ab5dfecbc6ec58
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50706
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16734 gss: fix lookup_user_key() bug

With more recent kernels, like on Ubuntu 22.04, trying to
delete some keyring resources trigger a kernel warning message
and cleaning is not successful, leading to stuck resources
and warning messages being regularly printed.

This is because Linux 5.8, in commit 8c0637e, introduced an API
change for lookup_user_key() that was not taken in account.

Update the lookup_user_key() call from _user_key() to fix it.

Lustre-change: https://review.whamcloud.com/50623
Lustre-commit: 013a6711503045b9e7154b8ff786ee85cdc3ecdd

Change-Id: I34ef4dac3f56cbb4aac6bc5a3bad36feb66b8675
Signed-off-by: Aurelien Degremont <adegremont@nvidia.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Jonathan Calmels <jcalmels@nvidia.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50721
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn85

New tag 2.14.0-ddn85

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I5bfbaad7178fca1a2af35531579e98f9f03215cf

LU-15495 tests: fixed dbench test

* Using awk to get list shared libraries
* Fixed shellcheck warnings

Lustre-change: https://review.whamcloud.com/49088
Lustre-commit: 01fb7bda971ee9d5dcd3950f40668131f306b8c5

Test-Parameters: trivial testlist=sanity \
    clientdistro=el7.9 clientarch=x86_64 env=SLOW=yes,ONLY=71
Test-Parameters: trivial testlist=sanity \
    clientdistro=el8.6 clientarch=x86_64 env=SLOW=yes,ONLY=71
Test-Parameters: trivial testlist=sanity \
    clientdistro=el8.7 clientarch=aarch64 env=SLOW=yes,ONLY=71
Test-Parameters: trivial testlist=sanity \
    clientdistro=el9.0 clientarch=x86_64 env=SLOW=yes,ONLY=71
Test-Parameters: trivial testlist=sanity \
    clientdistro=sles15sp4 clientarch=x86_64 env=SLOW=yes,ONLY=71
Test-Parameters: trivial testlist=sanity \
    clientdistro=ubuntu2004 clientarch=x86_64 env=SLOW=yes,ONLY=71
Test-Parameters: trivial testlist=sanity \
    clientdistro=ubuntu2204 clientarch=x86_64 env=SLOW=yes,ONLY=71
Test-Parameters: trivial env=SLOW=yes,ONLY=26 testlist=replay-dual
Test-Parameters: trivial env=SLOW=yes,ONLY=70b testlist=replay-single
Test-Parameters: trivial env=SLOW=yes,ONLY=8 testlist=sanity-pfl
Test-Parameters: trivial env=SLOW=yes,ENABLE_QUOTA=yes,ONLY=8 \
    testlist=sanity-quota

Change-Id: Ic28bd67dcfb5ff24e65e33278ac867409a2c1cc6
Signed-off-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50572
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16672 tests: auster node.yml labels Alma and Rocky as CentOS

release() assumes a node with /etc/centos-release is CentOS. This patch
removes that assumption and uses the name in the centos-release file.
Corrected the os-release code to strip off the last word if present.

Lustre-change: https://review.whamcloud.com/50442
Lustre-commit: TBD (from 3d6c37836b4bb7b1e6ea90bb7aacf4715e44c667)

Test-Parameters: trivial
Other-Id: Ia5acbce3351ca23f4d9265d1aaf8d952a2c8b502
Change-Id: Id16ec38d3530c4ece4fbdbb56c23af24d4c55b99
Signed-off-by: Charlie Olmstead <charlie@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50617
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15053 tests: reset quota if ENABLE_QUOTA=1

Quota limits set in setup_quota() with ENABLE_QUOTA=1
should be cleaned up in the end to avoid failures in
the next sessions

Lustre-change: https://review.whamcloud.com/49823
Lustre-commit: 2d40d96b4ec86327ec510be293f2ce4711f00826

Test-Parameters: testlist=sanity-quota env=ENABLE_QUOTA=yes
Test-Parameters: testgroup=review-dne-part-4 env=ENABLE_QUOTA=yes
Signed-off-by: Sergey Cheremencev <scherementsev@ddn.com>
Change-Id: Ia6b034739cfe800c6661f199420d0a4dbe7110fc
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50634
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16510 build: include unsafe_memcpy definition

The original LU-16510 missed a key part of the
unsafe_memcpy code from the upstream kernel, and so we
weren't actually defining unsafe_memcpy() as intended.

Thanks to Aurelien Degremont <adegremont@nvidia.com> for
pointing this out.

Fixes: 5099a3b7 ("LU-16510 build: fortified memcpy from linux 6.1")
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Ib9e2d56ed0b3691f1ab9fcd25403fa86ac784b6d
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50574
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7251 lnet: update locking multiple NIDs of the MR peer

Port updates to
LU-16709 lnet: fix locking multiple NIDs of the MR peer

This allows for the first of the two locked NIDs
to stay primary as intended for the purpose of communicating
with Lustre even if peer discovery succeeded
using a different NID of MR peer.

Lustre-change: https://review.whamcloud.com/50530
Lustre-commit: TBD (ddc9652a238e146e215157572b2e7e119de0e63b)

Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: Ic66c3b6d4dec98540e4fa2d7fa51c0e5e2f442ed
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50603
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-14661 obdclass: Add peer NI when processing llog (again)

Construct peers when processing the config log so that LNet has
complete information about peer info stored in the config log.

These are "temporary" peers which can be overwritten by discovery.

In client_import_add_nids_to_conn(), we do not need to hold the
import lock when adding NIDs to the obd_uuid, and LNet needs to
take the LNet API mutex when adding/modifying peers. We don't want
to take the mutex while a spin lock is already being held, so drop
the spin lock prior to calling class_add_nids_to_uuid().

Lustre-change: https://review.whamcloud.com/43510
Lustre-commit: 16321de596f6395153be6cbb6192250516963077

[This was problematic when the patch first landed, but was fixed
 by commit aacb16191a ("LU-14668 lnet: Lock primary NID logic")]

Fixes: 759d488fa0 ("EX-6349 revert: Add peer NI when processing llog")
HPE-bug-id: LUS-9293
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: Ib587ef478251e9722b21210e896838e0344d0e47
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50589
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn84

New tag 2.14.0-ddn84

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I04f278c63bbc7c646dcf2fa32bf4d11d128bb7db

EX-7251 lnet: fix locking multiple NIDs of the MR peer

If LNetPrimaryNID is called on multiple NIDs of the same node,
as a result of peer discovery it is possible that
the discovered peer is found to contain a NID which is locked
as primary by a different existing peer record.
In this case it is safe to delete one of the peer records,
but the NID which got locked the earliest should be
kept as primary.

This allows for the first NID listed in the mount command's
comma-separated list to stay primary as intended
for the purpose of communicating with Lustre even if peer
discovery succeeded using a different NID of MR peer.

Lustre-change: https://review.whamcloud.com/50530
Lustre-commit: TBD (47df7c726987d49e92b7145c128414daa413835f)

Fixes: aacb16191a ("LU-14668 lnet: Lock primary NID logic")
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: Iec9f8b70053fe24cddee552358500dfad0234b7f
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50533
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Shuichi Ihara <sihara@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn83

New tag 2.14.0-ddn83

Signed-off-by: Andreas Dilger <adilger@dilger.ca>
Change-Id: Ib5162a7d3c15b84a298ce5d30374a9b08fc7c5db

LU-16642 tests: improve sanity-sec test_61

Improve sanity-sec test_61 by using a client-specific nodemap rather
than the default nodemap.

Lustre-change: https://review.whamcloud.com/50317
Lustre-commit: a7222127c7a6437e3f3561fc55f3dc4ba69a97e5

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ie0c9e381e42a93d89558947dee9a60537cf01e65
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sergey Cheremencev <scherementsev@ddn.com>
LU-16683 tests: fix sanity-sec test_61 for SSK

When SHARED_KEY is in use, nodemap specific shared keys must be loaded
explicitly because sanity-sec test_61 defines a nodemap dedicated to
the client.

Lustre-change: https://review.whamcloud.com/50476
Lustre-commit: 05e5cb0b0c07e15f51ce4e8fa26e12c178ab404a

Fixes: a7222127c7 ("LU-16642 tests: improve sanity-sec test_61")
Test-Parameters: trivial
Test-Parameters: testlist=sanity-sec env=ONLY=61
Test-Parameters: testlist=sanity-sec env=SHARED_KEY=true,ONLY=61
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I206205496352b6f36341c8b962bb7de4b71541d5
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50502
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16708 tests: fix zconf_mount_clients for SSK

When SHARED_KEY is in use, zconf_mount_clients can add 'skpath' mount
option to load nodemap-specific keys. But this must not overwrite
already specified mount options.

Lustre-change: https://review.whamcloud.com/50523
Lustre-commit: TBD (bd0d4644467192f5272bcc483663060314aec05a)

Fixes: eee562b0a9 ("LU-16683 tests: fix sanity-sec test_61 for SSK")
Test-Parameters: trivial
Test-Parameters: testlist=sanity-sec env=ONLY=61
Test-Parameters: testlist=sanity-sec env=SHARED_KEY=true,ONLY=61
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I3dbc6ffe72722659ead77fb1c2c2675873c7aff2
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50524
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-11785 tests: fix conf-sanity/98 mount check on 64K page

This patch fix the mount option length check expectation
fail on 64K page. Since the maxopt_len is the minmium
value of page_size or 64K page_size, but the test cases
only hard code the length of option to the 4K one. This
patch add the mount options according to the page size.

Lustre-change: https://review.whamcloud.com/48177
Lustre-commit: TBD (from c7682adb9d021794a4aada49f15e3531d0ecea39)

Test-Parameters: trivial testlist=conf-sanity env=ONLY=98
Signed-off-by: Kevin Zhao <kevin.zhao@linaro.org>
Change-Id: Icdeb8b73308056e216c3f4ce71907b0c928d2c30
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50506
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16163 tests: skip racer_on_nfs for NFSv3

This test is continually failing and nobody is available to
fix it (it may be an NFS bug or a Lustre bug, unsure).

This same test passes on NFSv4 regularly.

Lustre-change: https://review.whamcloud.com/50385
Lustre-commit: 513eb670b01f15104cbeb2909a141d2174dcc874

Test-Parameters: trivial testlist=parallel-scale-nfsv3
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I7d9ac390c26aa8478dd35457ba20061747c2b92e
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50461
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16676 build: always include llcrypt sources

llcrypt sources should always be included in source packages.
Binary build will decide whether to include llcrypt in the build
objects or not.

Lustre-change: https://review.whamcloud.com/50446
Lustre-commit: TBD (ec8866ef1b022f719bddd318016df59aceab37ee)

Test-Parameters: trivial
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I726d7deb27687bffebce55f6c09d578e6290aac7
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50453
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16658 tests: disable performance-sanity test_6

This test is likely failing due to a bug in mdsrate, which is no
longer actively developed.  It should be replaced by mdtest.

Lustre-change: https://review.whamcloud.com/50386
Lustre-commit: TBD (from c4b23f1975e750e8c8cbe64741b8427d708cb7a6)

Test-Parameters: trivial testlist=performance-sanity
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I05378fb75ed30e56983f4668c03725824ad5a8ab
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50426
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16529 test: wait quota synced on quota slaves

Check and wait the quota setting to be synchronized on
quota slaves before running actual sanity-quota test_84.

Lustre-change: https://review.whamcloud.com/50197
Lustre-commit: TBD (from 0a94395230e7482029ff03381bf77b4783a93e2d)

Test-Parameters: trivial testlist=sanity-quota env=ONLY=84,ONLY_REPEAT=100
Fixes: a2fd4d3aee ("LU-15880 quota: fix insane grant quota")
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Change-Id: I7752bff33f24d1d38dc340b2addbfc98d6f7c857
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50445
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn82

New tag 2.14.0-ddn82

Signed-off-by: Andreas Dilger <adilger@dilger.ca>
Change-Id: Ic8e3dd420b2d120e12c13f64943468cfaea70843

EX-7013 lipe: printf formats added to lipe_find3

Printf formats added to lipe_find3: %LS, %Lc, %LN,
%LF, %{stripe-size}. Sanity-lipe-find3 updated to test
these new formats.

Test-Parameters: trivial testlist=sanity-lipe-find3,sanity-lipe-scan3
Signed-off-by: Alexandre Ioffe <aioffe@ddn.com>
Change-Id: I6377e131961634572a246909587f81da9190dd28
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50366
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Colin Faber <cfaber@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16659 build: Detect the mofed path based on running kernel

Test-Parameters: trivial

Signed-off-by: Gaurang Tapase <gtapase@ddn.com>
Change-Id: I519e93e8c26807da6143e2cf4d825ccf4a4180e4
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50352
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16629 osd: refill the existing env

During the LDLM lock callback, the "lu_env" is created in
ldlm_bl_thread_main, which is initiated by "ldlm_setup",
and it could have no key of "osd_thread_info" yet, then it
need to call "lu_env_refill" to refill the keys.

Lustre-change: https://review.whamcloud.com/50270
Lustre-commit: TBD (from c9673c45a9c279b12ccb8804d54508905a1e79ee)

Change-Id: Ibae978a5a10826c2e3186012911870ce7bf0b147
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50273
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-11596 osc: Fix and re-enable sanity grant test for ARM

If both OST and OSC support OBD_CONNECT_GRANT_PARAM, OST side will not
change client side claimed grant (a.k.a. o_grant_used) regardless of
the client page size. So no grant loss in this case.

Lustre-change: https://review.whamcloud.com/40758
Lustre-commit: 7d3edce0650f0b66bdd280373a54a16cb28b8469

Test-Parameters: trivial clientarch=aarch64 clientdistro=el8.6 testlist=sanity
Fixes: bd1e41672c97 ("LU-2049 grant: add support for OBD_CONNECT_GRANT_PARAM")
Change-Id: Ia0d3da587cb551400fec0c054dc65b116e6bd95b
Signed-off-by: James Simmons <jsimmons@infradead.org>
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Signed-off-by: Xinliang Liu <xinliang.liu@linaro.org>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50279
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15129 tests: sanity-quota_75_dom fix

oflag=sync causes dd to write page by page instead
of sending several pages in RPC. Furthermore, when
granted space becomes closer to soft_limit(i.e. over
9MB if soft_limit is 10MB), OST can not preacquire
space anymore. Also OST could acquire only requested
amount of space - see qmt_alloc_expand. Thus OST has
to send quota acquire request at MDT for each BRW request
from the client. Sometimes 20 seconds is not enough to write
10MB. Change oflag=sync with conv=fsync to reduce the number
of RPCs between the client and OST and between QSDs and QMT.
One fsync at close should help to avoid timeout failure.

Lustre-change: https://review.whamcloud.com/50164
Lustre-commit: 7d05a687ee5d4f4b95585244a7f60394475fe0ba

Test-Parameters: trivial testlist=sanity-quota
Test-Parameters: testlist=sanity-quota env=ONLY=75,ONLY_REPEAT=50
Signed-off-by: Sergey Cheremencev <scherementsev@ddn.com>
Change-Id: Iad363fdc8a0984861055c295ea9cc3f23110fd9f
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50336
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-7013 lipe: printf formats added to lipe_find3

Printf formats added to lipe_find3: fid, mirror-count,
stripe-count. Sanity-lipe-find3 updated with printf
format tests

Test-Parameters: trivial testlist=sanity-lipe-find3,sanity-lipe-scan3
Signed-off-by: Alexandre Ioffe <aioffe@ddn.com>
Change-Id: Ic5b0d6786c705e51d809eba19e123287d2794740
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50313
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Colin Faber <cfaber@ddn.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7065 build: add Rocky distro to lipe

Test-Parameters: trivial

Change-Id: I83298fee9da925a698246d3f52e2c9164cb5cf4b
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50294
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16589 tests: fix hard-link failure in sanityn/55d

Since coreutils version 8.31, the stat() and lstat()
operations were removed from ln by commit 571f63f5010b,
which caused the following dir hard-link failure in
sanityn/55d:

ln: failed to create hard link '/mnt/lustre2/d55d.sanityn/d55d.sanityn/'
=> '/mnt/lustre2/d55d.sanityn/f1': No such file or directory

This actually reveals a kernel issue which is fixed by commit
v5.18-rc2-188-gb3d4650d82c7.

To avoid the kernel issue and keep the test effective,
this patch appends the target filename to the $tdir/
so as to fix the hard-link failure.

Lustre-change: https://review.whamcloud.com/50127
Lustre-commit: TBD (from 6dea4fabe02ebd90bc6e4dbb8641003a9a66c967)

Test-Parameters: trivial env=ONLY=55d testlist=sanityn
Test-Parameters: trivial clientdistro=el9.1 env=ONLY=55d testlist=sanityn
Test-Parameters: trivial clientdistro=el9.0 env=ONLY=55d testlist=sanityn
Test-Parameters: trivial clientdistro=sles15sp4 env=ONLY=55d testlist=sanityn
Test-Parameters: trivial clientdistro=sles15sp3 env=ONLY=55d testlist=sanityn

Change-Id: I42313e43eaea3d94007d534bf38efdeacf2ede43
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50350
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn81

New tag 2.14.0-ddn81

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I189f73281dfe12d4835c3d54b10e41099fe47af8

LU-16285 ldlm: BL_AST lock cancel still can be batched

The previous patch makes BLAST locks to be cancelled separately.
However the main problem is flushing the data under the other batched
locks, thus still possible to batch it with those with no data.
Could be optimized for not yet CANCELLING locks only, otherwise it is
already in the l_bl_ast list.

Lustre-change: https://review.whamcloud.com/50158
Lustre-commit: 9d79f92076b6a9ca735dbe4420c122f47d240263

Fixes: b65374d9 ("LU-16285 ldlm: send the cancel RPC asap")
Signed-off-by: Vitaly Fertman <vitaly.fertman@hpe.com>
Signed-off-by: Xing Huang <hxing@ddn.com>
Change-Id: Ie4a7c7f3e0f5462290f72af7c3b2ff410a31f5e7
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50274
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16221 kernel: update RHEL 9.1 [5.14.0-162.18.1.el9_1]

Update RHEL 9.1 kernel to 5.14.0-162.18.1.el9_1 for Lustre client.

Lustre-change: https://review.whamcloud.com/50177
Lustre-commit: TBD (from ae128e3a29f10fc9bdb5711c1c27e2edaa35a82b)

Test-Parameters: trivial clientdistro=el9.1 testlist=sanity

Change-Id: I032f69f1ecba60248729bb856a3aad78e5f05680
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50204
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16509 lnet: memcpy false positive in brw_test

The flexible array at the end of srcp_bulk is triggering a
false positive in fortified memcpy().  Quash it with
unsafe_memcpy().

Lustre-change: https://review.whamcloud.com/50185/
Lustre-commit: 299bd671a9f7c0955b2be59e36bb8239797c1b82 (tbd)

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I13386c0a8e73b04af8d398aa49361bfdf6a05ad8
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50218
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn80

New tag 2.14.0-ddn80

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ia899d8b47c4df16020bcfd8c20cc60d59f761daa

LU-16509 lnet: quash memcpy WARN_ONCE false positive

Linux v6.1-rc1-4-g6f7630b1b5bc
  fortify: Capture __bos() results in const temp var

In lnet_peer_push_event() the memcpy triggers a WARN_ONCE
due to the flexible array at the end of
struct lnet_ping_info contained in struct lnet_ping_buffer

Use unsafe_memcpy() to avoid this false positive warning.

Lustre-change: https://review.whamcloud.com/49801
Lustre-commit: 18e24419599666bd536d072705052a8c64b681ab (tbd)

Test-Parameters: trivial
HPE-bug-id: LUS-11455
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I4aa8f38678cd1522004d98b58a3f440d8a38589c
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50217
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16363 build: fiemap flexible array

Linux commit v5.19-rc2-1-g94dfc73e7cf4
 treewide: uapi: Replace zero-length arrays with flexible-array
 members
Adjust wiretest to handle flexible array when
sizeof(fiemap->fm_extents) is undefined.

Lustre-change: https://review.whamcloud.com/49305
Lustre-commit: fedf1e8bd70ccb2aaa64cb90111a7298d9bb2bf7

Test-Parameters: trivial
HPE-bug-id: LUS-11388
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: Ia2692d126a871b43e9144e5d151215166604702d
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50216
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16557 client: -o network needs add_conn processing

Mount option -o network restricts client import to use
only selected network. It processes connection UUID/NIDs
during 'setup' config command handling but skips any
'add_conn' command if its UUID has no mention about that
network. Meahwhile connection UUID is just a name and may
have many NIDs configured including those on restricted
network which are skipped as well. Therefore client import
configuration misses failover NIDs on restricted network.

Patch makes import to save restricted network information
after 'setup' command processing, so it is applied to any
client_import_add_conn() call. The 'add_conn' command is
always processed now and its NIDs will be filtered in the
same way as for 'setup'.
Test 31 in sanity-sec.sh is extended to check imports
failover_nids has all and only NIDs on restricted network

Lustre-change: https://review.whamcloud.com/49986
Lustre-commit: c508c9426838f16256223ab0bbd648bfbec25e46

Test-Parameters: env=ONLY=31 testlist=sanity-sec
Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: Id70ebd836f061f154e3779b07b52f1baea9a1776
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50172
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-9162 lod: option to set max stripe count per filesystem

Add an option to set max default stripe count when the stripe count
is set to -1.

Lustre-change: https://review.whamcloud.com/45532
Lustre-commit: 3c9580931d1a7753e6c03900df01fd05ab3ca73d

Change-Id: I02634a02a6f6579750fe964662b7e644af1689d6
Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50231
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-6169 lipe: debug sanity-lipe-find3 problem

Added test print for files/FIDs left behind in filesystem.

Test-Parameters: trivial testlist=sanity-lipe-find3 env=ONLY="90 350"
Test-Parameters: testlist=sanity-lipe-scan3,sanity-lipe-find3
Test-Parameters: testlist=sanity-quota,sanity-lipe-find3
Test-Parameters: testlist=sanity-pcc,sanity-lipe-find3
Test-Parameters: testlist=sanity-quota,sanity-lipe-find3
Test-Parameters: testlist=sanity-pcc,sanity-lipe-find3
Test-Parameters: testlist=sanity-quota,sanity-lipe-find3
Test-Parameters: testlist=sanity-lipe-scan3,sanity-lipe-find3
Test-Parameters: testlist=sanity-quota,sanity-lipe-find3
Test-Parameters: testlist=sanity-pcc,sanity-lipe-find3
Test-Parameters: testlist=sanity-quota,sanity-lipe-find3
Test-Parameters: testlist=sanity-quota,sanity-lipe-find3
Test-Parameters: testlist=sanity-quota,sanity-lipe-find3
Test-Parameters: testlist=sanity-quota,sanity-lipe-find3
Test-Parameters: testlist=sanity-quota,sanity-lipe-find3
Test-Parameters: testlist=sanity-quota,sanity-lipe-find3
Test-Parameters: testlist=sanity-quota,sanity-lipe-find3
Signed-off-by: Alexandre Ioffe <aioffe@ddn.com>
Change-Id: I2ad8dc24f6825075f4089c1759c148bee26bfbfe
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50142
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-11170 tests: add debugging to sanity/415

Add a loop of renames without the concurrent 'touch' operation to
measure the test time, and then a second loop that has the 'touch'
so that we can see whether slow renames are because of COS (which
would make the test failure a kernel bug to be fixed), or because
the test is running in a VM and the server/disk is slow (which is
something to be fixed in the test (e.g. by making "slow" relative
to the non-touch baseline time).

Lustre-change: https://review.whamcloud.com/49724
Lustre-commit: 6594babc73851fab335c514cd1fee018425e7bb3

Test-Parameters: trivial testlist=sanity env=ONLY=415,ONLY_REPEAT=120 mdscount=2 mdtcount=4
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ic1a952be0b861005b46da3e673216e455f3ebbe5
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50203
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-13417 tests: fix interop version check for tests

Fix version check in tests so that they actually run.

Add a spelling.txt rule for wrong version checks used in tests.

Test-Parameters: trivial testlist=sanity-hsm env=ONLY=12s
Fixes: ca1f7f6899 ("LU-15132 hsm: Protect against parallel HSM restore requests")
Fixes: b1b3c1413a ("LU-15727 lod: honor append_pool with default composite layouts")
Fixes: f1a5ef423e ("LU-15938 llog: more checks in llog_reader")
Fixes: 4e43fe8d0c ("LU-14719 lod: distributed transaction check space")
Fixes: 56ea02d192 ("LU-15850 lmv: always space-balance r-r directories")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I5241d4b419876b8f9cc6799757d6f205ee7b9e82
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50256
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16349 o2iblnd: Fix key mismatch issue

If a pool memory region (mr) is mapped then unmapped without being
used, its key becomes out of sync with the RDMA subsystem.

At pool mr map time, the present code will create a local
invalidate work request (wr) using the mr's present key and then
change the mr's key.  When the mr is first used after being mapped,
the local invalidate wr will invalidate the original mr key, and
then a fast register wr is used with the modified key.  The fast
register will update the RDMA subsystem's key for the mr.

The error occurs when the mr is never used.  The next time the mr
is mapped, a local invalidate wr will again be created, but this
time it will use the mr's modified key.  The RDMA subsystem never
saw the original local invalidate, so now the RDMA subsystem's
key for the mr and o2iblnd's key for the mr are out of sync.

Fix the issue by tracking if the invalidate has been used.
Repurpose the boolean frd->frd_valid.  Presently, frd_valid is
always false.  Remove the code that used frd_valid to conditionally
split the invalidate from the fast register.  Instead, use frd_valid
to indicate when a new invalidate needs to be generated.  After a
post, evaluate if the invalidate was successfully used in the post.

These changes are only meaningful to the FRWR code path.  The failure
has only been observed when using Omni-Path Architecture.

Lustre-change: https://review.whamcloud.com/49714
Lustre-commit: 0c93919f1375ce16d42ea13755ca6ffcc66b9969

Signed-off-by: Cyril Bordage <cbordage@whamcloud.com>
Signed-off-by: Xing Huang <hxing@ddn.com>
Change-Id: I532a11f10ae6a5917a4c054f37747d08eb4d6331
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50211
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16555 obdclass: quote jobid with ':' char

quote jobid with ':' char even though ':' is printed literally.

Lustre-change: https://review.whamcloud.com/50150
Lustre-commit: TBD (from e3f53a5fa5c086b685eb1f0e8d3c96780faabd49)

Fixes: 12604ac4de (LU-16555 obdclass: print more special chars for jobid)
Test-Parameters: trivial
Signed-off-by: Lei Feng <flei@whamcloud.com>
Change-Id: I8bf642180b00f068921598c3a0ea30b4f5b99c1e
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50151
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16591 build: use vendor tag for LUSTRE_FIX

Improve the regexp for the LUSTRE_FIX (fourth) component of the
LUSTRE_VERSION parsing so that if there is an additional component
in the version like "2.15.2-abc123" the "123" will be reported as
the fourth component of the version in obd_connect_data.ocd_version.
If there is already a fix version specified (e.g. 2.15.2.4-abc123)
then it will continue to be used for LUSTRE_FIX instead of the extra
"-abcNN" value.

The build version is shown in "{mdt,obdfilter}.*.exports.*.export"
param on servers for connected clients, and in "{mdc,osc}.*.import"
param on clients for connected servers.  Displaying the full version
improves debuggability of remote peers to know their specific build
instead of showing the first three digits and always ".0" at the end.

Since ocd_version is a numeric field it is not possible to include
the "abc" part of the version string on the peer nodes.

Lustre-change: https://review.whamcloud.com/50136
Lustre-commit: ee67a78cf2b102230dc3a7beea2274866e643224

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I96b38f99b2522c5ea3f3b3e2ddd7cd64f1ce7057
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50137
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shuichi Ihara <sihara@ddn.com>

LU-16601 kernel: update SLES15 SP4 [5.14.21-150400.24.46.1]

Update SLES15 SP4 kernel to 5.14.21-150400.24.46.1 for Lustre client.

Lustre-change: https://review.whamcloud.com/50179
Lustre-commit: TBD (from c1e032746e318daba275eec8995136a187de4a3c)

Test-Parameters: trivial clientdistro=sles15sp4 testlist=sanity

Change-Id: I5b9e39359e61e929adaeddece60f4d247996a00a
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50206
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16600 kernel: update RHEL 8.7 [4.18.0-425.13.1.el8_7]

Update RHEL 8.7 kernel to 4.18.0-425.13.1.el8_7.

Lustre-change: https://review.whamcloud.com/50178
Lustre-commit: TBD (from 0fbefb1c42e98d0323a8febe104ec4b8afc4ad22)

Test-Parameters: trivial fstype=ldiskfs \
clientdistro=el8.7 serverdistro=el8.7 testlist=sanity

Test-Parameters: trivial fstype=zfs \
clientdistro=el8.7 serverdistro=el8.7 testlist=sanity

Change-Id: I4e4d7024e31597566d936bdec4b65a1d4ed925a0
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50205
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15524 mdd: trigger changelog GC by free space

if amount of space consumed by changelog become comparable
with system free space then start emergency GC for changelog
by purging the oldest user

Such behavior is enabled by default and can be disabled via
mdd_changelog_free_space_gc parameter

Test 160t is added to sanity.sh

Lustre-change: https://review.whamcloud.com/46467
Lustre-commit: bec1334954a73ed668fad409e8c728f9dfd6bb99

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: Ia63cc71e708b0f10cdf54f45f0809c0e86950101
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50243
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>