LU-16850 socklnd: remove ksnr_myiface from ksock_conn_cb

Drop ksnr_myiface: it is no longer needed since socklnd
TCP bonding got removed. There's one interface per
connection cb per peer_ni, and it can be accessed as
net->ksnn_interface.ksni_index.

Fix setting of ksni_nroutes accordingly. Duplication of
interface index in conn_cb and ksnn_interface was causing
the assertion
ASSERTION( net->ksnn_interface.ksni_nroutes == 0 )
in ksocknal_shutdown() to fail if the corresponding
device is deregistered before lnd shutdown.

Modify test_214 of sanity-lnet to create connections so that
the scenario of socklnd shutdown with NI on a deregistered
interface is covered.

Lustre-change: https://review.whamcloud.com/51148
Lustre-commit: f6be07c457385cfacd9b802e4cade9f6f6ab7d6f

Fixes: a7ee03d7ca4185 ("LU-16378 lnet: handles unregister/register events)
Test-Parameters: trivial
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: I4de164c9e64aa770164a8320b9460fadce49aa06
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51326
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13641 socklnd: remove remnants of tcp bonding

->ksnp_n_passive_ips is now always zero, so remove it and all uses of
it.  ->ksnp_passive_ips is gone too, as is ksocknal_ip2iface().

Lustre-change: https://review.whamcloud.com/48568
Lustre-commit: 3630e1eaf9db562a1de707762cd649db815459c8

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I5de6d027c545087c961673d8704f68c4f3dd5076
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51325
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-4130 lipe: Remove old hotpool scripts

Hotpool scripts now live in EMF repo.
The scrips in lipe/ are unmaintained and non-functional.

Change-Id: I2f288f7f68886015ddb9a85da4e437c9f03c3928
Signed-off-by: Nathaniel Clark <nclark@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51215
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Colin Faber <cfaber@ddn.com>
Reviewed-by: Shuichi Ihara <sihara@ddn.com>
Reviewed-by: Artur Novik <anovik@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15047 gss: gss integrity check with multi-rail

With multi-rail, a primary NID is used as node identifier, but LNet
decides which NID is actually used for sending/receiving data, on a
per request basis.
For the integrity check mechanism implemented as part of GSS, the
primary NID must be used in order to compute HMAC with the correct
key, independently of the actual NID for the current request.

Lustre-change: https://review.whamcloud.com/45277
Lustre-commit: c8301a65c5672a1d081669343466746df983eabc

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I2bf3974d3aa0e8365a9413dca56c69ee3734c12b
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51274
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-7495 utils: add --links option for lfs find

This adds a "--links" option for lfs find to filter files and
directories by the number of hard links. It also adds a printf format
'%n' to print the number of links for a file.

This commit also fixes '-l' as a short option for '--lazy' which was
added in 11aa7f8704c490b011f60f234c3ac9929ce76948 but the short option
did not work.

Lustre-change: https://review.whamcloud.com/50886
Lustre-commit: f759d6386d5d0edb95d683d97ca8d84c80080c1c

Signed-off-by: Thomas Bertschinger <bertschinger@lanl.gov>
Change-Id: I5d15bc290df8e8a08402f8d5cfa0a7139791b0a4
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Anjus George <georgea@ornl.gov>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51327
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16799 tests: fix sanity-krb5

sanity-krb5.sh needs to be fixed in several ways.
It cannot assume that the Kerberos credentials cache is FILE, and
expect ccache files to be under /tmp/krb5cc_*.
The lsvcgssd daemon must be launched with -vvv flags for easier
debugging.
Keyring needs to be cleared appropriately after using 'lfs flushctx'.

Lustre-change: https://review.whamcloud.com/50864
Lustre-commit: f8f8b3c574e95cb7272310bba19f97fe68cd9b11

Test-Parameters: trivial testlist=sanity-krb5 kerberos=true
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I31ca8d2d97e137c7ba9fa478d5432aeedb5135a8
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51265
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-12514 llite: move client mounting from obdclass to llite

Mounting a lustre client is currently handled
in obdclass, using services from llite.
This requires obdclass to load the llite module
and set up inter-module linkage.

The purpose of this was for common code to support both
client and server mounts.  This isn't really a good idea
and need to go. For lustre servers we already use a
separate filesystem type.

So move the mounting code from obdclass/obd_mount to llite/super25
and remove the inter-module linkages.
Add some EXPORT_SYMBOL() so that llite can access some helpers
that remain in obdclass.

Linux-commit: a989830c88149511ee840356d9c1b34304bac576

Lustre-change: https://review.whamcloud.com/37693
Lustre-commit: 53fa81765750e38f7879ed5092fd729c1bdc8a0f

Change-Id: Ia33bd55a042f90b178156c745a8072b516f00568
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51315
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16425 tests: skip interop recovery-small/144a/144b

Skip recovery-small test_144a and test_144b for old MDS
missing the fix and for its corresponding test.

Lustre-change: https://review.whamcloud.com/49679
Lustre-commit: 64faf832a6128cc55c0f3ffa0595d9715d3bdd25

Fixes: 240938f7b1 ("LU-8367 tests: cleanup_orphans hang reproducer")
Fixes: aa6250b741 ("LU-15724 tests: MDT failover hang reproducer")
Test-Parameters: trivial testlist=recovery-small env=ONLY=144 serverversion=2.14.0
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I77bfdf55d0218aa9e252f742cc90f1c61216d506
Reviewed-by: Sarah Liu <sarah@whamcloud.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51328
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16167 obdclass: fix lctl llog_print with skipped records

The 2a5b50d ignores the skipped records in configuration llog.
But if ioctl OBD_IOC_LLOG_PRINT return 0 record to display,
jt_llog_print_iter() will stop the processing and ignore the
non-skipped records at the end of the llog.

This patch returns to user space if the last index processed
(by llog_print_cb) is the last of llog file. If true,
jt_llog_print_iter() stops the processing.

Add regression test "conf-sanity test_123ai" for this issue.

Lustre-change: https://review.whamcloud.com/48586
Lustre-commit: c6da54aa7546440339265c644538d3d109e46bde

Fixes: 2a5b50d ("LU-15142 lctl: fixes for set_param -P and llog_print")
Test-Parameters: testlist=conf-sanity env=ONLY=123ai,SLOW=yes,ONLY_REPEAT=10
Signed-off-by: Etienne AUJAMES <etienne.aujames@cea.fr>
Change-Id: I78395268c57555e4fd2a4048ccf5b6132ca2877f
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alexander <alexander.boyko@hpe.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51316
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16619 build: Ubuntu jammy 5.19 client support

Ubuntu 5.19 kernel removed lsmcontext_init() and changed
security_dentry_init_security to require struct context *

Linux kernel linux-hwe-5.19
LSM: Removed scaffolding function lsmcontext_init

Linux kernel linux-hwe-5.19
LSM: security_dentry_init_security with struct lsmcontext

Lustre-change: https://review.whamcloud.com/50210
Lustre-commit: TBD (from d7001a6c68c334d15d3daeb932b8456b101153d2)

Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: Ib6479a2cd20df5e565ae6203e05df2afa3f3de31
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51002
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16327 llite: read_folio, release_folio, filler_t

Linux commit v5.18-rc5-221-gb7446e7cf15f
  fs: Remove aop flags parameter from grab_cache_page_write_begin()
flags have been dropped from write_begin() and
grab_cache_page_write_begin()

Linux commit v5.18-rc5-241-g5efe7448a142
  fs: Introduce aops->read_folio
Provide a ll_read_folio handler around ll_readpage

Linux commit v5.18-rc5-280-ge9b5b23e957e
  fs: Change the type of filler_t
Affects read_cache_page, provides a wrapper for read_cache_page
and wrappers for filler functions

Linux commit v5.18-rc5-282-gfa29000b6b26
  fs: Add aops->release_folio
Provide an ll_release_folio function based on ll_releasepage

Lustre-change: https://review.whamcloud.com/49199
Lustre-commit: 133ed0cf6f0c84d2b5b84e1de3ff2c54b1fb902d

Test-Parameters: trivial
HPE-bug-id: LUS-11357
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: Ibd4ec1133c80cd0eb8400c4cd07b50e421dd35c5
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50977
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13485 kernel: Parallel core configure tests

Transform the compile tests in lustre-core to run in parallel

Lustre-change: https://review.whamcloud.com/38361
Lustre-commit: a346cf6cf22c285e05177a5a5e70a7d8e5bd9fa8

LU-16586 build: Remove old check for linux_selinux_enabled

LC_SRC_HAS_LINUX_SELINUX_ENABLED is used and not defined, it
should be removed.

LC_PROG_LINUX_SRC and LC_PROG_LINUX_RESULTS are defined twice,
remove the unused ones.

Lustre-change: https://review.whamcloud.com/50111
Lustre-commit: 2bdff20f97707cf05118e822adf202ccb209e3a5

Fixes: a346cf6cf22 ("LU-13485 kernel: Parallel core configure tests")

Test-Parameters: trivial
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I46fa852659eb4db86a12ec4ad3efddd0fdd3a655
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50976
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Tested-by: Shuichi Ihara <sihara@ddn.com>
Reviewed-by: Shuichi Ihara <sihara@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16745 mdc: md_open_data should keep ref on close_req

md_open_data should keep a ref on mod_close_req,
otherwise the mod_close_req could be freed before
we try to access mod_close_req via md_open_data.

Lustre-change: https://review.whamcloud.com/50656
Lustre-commit: ca716f763f89af192ab86678ee9d14f49c80cae6

Change-Id: I621f7db389854326db298d99957a0bce43024b6e
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51198
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15123 tests: check quota reintegration after recovery

4th step of quota reintegration (reconciliation) waits for recovery
completion. So the tests (like sanity-quota/7a) should wait for
recovery completion before checking reintegration results.

Lustre-change: https://review.whamcloud.com/50688
Lustre-commit: 4432b6e2824775e292f96e202d6fc0db231bc749

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Id0aa5db01658621103d94ad6dafe91b2960b3a33
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sergey Cheremencev <scherementsev@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51232
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-14277 lod: statfs should not block a create

lod_qos_statfs_update() need a guarantee that targets
isn't changed, so it doesn't need to take a QoS mutex.

Lustre-change: https://review.whamcloud.com/41497
Lustre-commit: 6cdb08a0c598b20a4b0ce27f7fe4763b0ada3118

HPE-bug-id: LUS-2106
Signed-off-by: Alexander Boyko <alexander.boyko@hpe.com>
Signed-off-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Change-Id: Id9f0ea2fa006bee601d05e14b2e515fcf8248249
Reviewed-by: Andrew Perepechko <andrew.perepechko@hpe.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51246
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16755 kernel: update RHEL 8.8 [4.18.0-477.10.1.el8_8]

Update RHEL 8.8 kernel to 4.18.0-477.10.1.el8_8.

Lustre-change: https://review.whamcloud.com/51051
Lustre-commit: TBD (from 266266a8874cab2df8e9b5cc909d578dc239a1b1)

Test-Parameters: trivial fstype=ldiskfs \
clientdistro=el8.8 serverdistro=el8.8 testlist=sanity

Test-Parameters: trivial fstype=zfs \
clientdistro=el8.8 serverdistro=el8.8 testlist=sanity

Change-Id: I6d7703512f9c5a8b686f06e94f32f0e51c9b2001
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51054
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16756 kernel: new kernel [RHEL 9.2 5.14.0-284.11.1.el9_2]

This patch makes changes to support new RHEL 9.2 release
for Lustre client.

Lustre-change: https://review.whamcloud.com/50745
Lustre-commit: dd390cd315f505456e8e8a29e16a0f24b6878e27

Test-Parameters: trivial env=SANITY_EXCEPT=27J clientdistro=el9.2 testlist=sanity
Test-Parameters: trivial env=SANITY_EXCEPT=27J clientdistro=el9.2 serverdistro=el8.7 testlist=sanity

Change-Id: I4886bbf30d6d6a93c4adbfb68871e9d91f5b64de
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50964
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16755 kernel: RHEL 8.8 client and server support

This patch makes changes to support RHEL 8.8 release
with kernel 4.18.0-477.el8 for Lustre client and server.

Lustre-change: https://review.whamcloud.com/50708
Lustre-commit: fccc55866bc7da0b4908130d3f16bd355ffb7dc6

Test-Parameters: trivial fstype=ldiskfs \
clientdistro=el8.8 serverdistro=el8.8 testlist=sanity

Test-Parameters: trivial fstype=zfs \
clientdistro=el8.8 serverdistro=el8.8 testlist=sanity

Change-Id: Ie47f131e0340a601c8a5d748ecf9b1b73d4baa1f
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50963
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16517 build: pass extra configure options to "make debs"

While running "make debs", the configure command in debian/rules
ignores some user defined configure options. This patch fixes
the issue by adding the detection of the extra options into
debian/rules.

Lustre-change: https://review.whamcloud.com/50464
Lustre-commit: 3989529f22f5c54a98e445674b4b3cc443a3af5f

Test-Parameters: trivial clientdistro=ubuntu2204

Change-Id: Ia9db4e05abf33834cb3c853f4f0829dadc8d7400
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51174
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-6142 obdclass: resolve lu_ref checkpatch issues

Fix up all the checkpatch issues reported for the code handling
lu_ref. Also change USE_LU_REF to CONFIG_LUSTRE_DEBUG_LU_REF
which will match what will be upstream.

Lustre-change: https://review.whamcloud.com/44088
Lustre-commit: 5c98de856618f301035c7579e61fffa10cc726cc

Change-Id: I100e2679fc04c97eb67e4d44c4f6a6b530da6fa8
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51173
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-6142 lustre: change all 'struct seq_operations' to const

All 'struct seq_operations' are now 'static const'.

Lustre-change: https://review.whamcloud.com/39396
Lustre-commit: c5b90540731f13c37e6ad62c194ebef6725608a7

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ia0f6a2863835ec02fb7d283a099eea489f65ab3a
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51172
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15668 osd-ldiskfs: fix osd_bio_private double free

In osd_do_bio(), if the IO is fragmented and bio_alloc()
fails to allocate new bio, the bio_private still holds
osd_bio_private for last bio and will be double freed
in osd_do_bio() and dio_integrity_complete_routine().

Lustre-change: https://review.whamcloud.com/46879
Lustre-commit: 82d44b71f16bea216e04f7797ed032a783301d66

Test-Parameters: trivial
Change-Id: I42eaf95a85ec99a60359122054efb5beb0fb6104
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51187
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16829 gss: quiet noisy warnings

GSS code has plenty of debugging messages as warnings.
Update the code to make them just debugging messages and
nothing more.

Lustre-change: https://review.whamcloud.com/50613
Lustre-commit: 2247cf23ce55398ea4bebc1f8093275534736a0a

Test-Parameters: trivial

Change-Id: I53c471d758b0309ae10bab000211fa0381cabf87
Signed-off-by: Aurelien Degremont <adegremont@nvidia.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51190
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-6511 csdc: prefer uncompressed mirror for read

When accessing a mirrored file with both compressed and uncompressed
components, choose uncompressed components for read.

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I043b27bd891c039901075a08c76630f8f0f9f182
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50791
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7327 utils: add warning for 'lfs setstripe' compression

Add warning message to indicate that file compression is a preview
feature, not for production use.

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I1d5f9bdb11e33e8dbd65c6048d62e612020af3a3
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50658
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Reviewed-by: Colin Faber <cfaber@ddn.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>

EX-6856 utils: add 'lfs find' support for compressed file

* Add "--comp-flags=[^]compress" to locate file with/without
  compressed components.
* Add "--comp-flags=[^]nocompr" to locate file with/without setting
  component compress preference.
* Add "[!] --layout=compress" to locate file with/without compressed
  components.
* Add "[!] --compress-type=<compress-type>" to locate compressed file
  with/without specified compress algorithm.
* Add "[!] --compress-level=[+-]<compress-level>" to locate compressed
  file with/without specified compress level.
* Add "[!] --compress-chunk=[+-]<compress-chunk>" to locate compressed
  file with/without specified compress chunk in KiB.

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Ic36946738c3463fd862aeca4ee2e2c2ed85eff84
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/49930
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>

EX-6249 csdc: set compress component for file

* 'lfs setstripe' to support compress component

  --compress|-Z <type>[:<level>]
  Set component compression algorithm <type> and compress <level>;

  --compress-chunk=<size>
  Set compress data chunk size in KiB used by the compression
  algorithm, the value will be adjusted to power-of-two multiples of
  the base 64KiB.

  Example:
$ lfs setstripe -Eeof -Z lz4:5 --compress-chunk=512 <file>

* 'lfs getstripe' to show compress component parameters
  Display component's compression parameters if possible.

  --compress-type|--compr-type
  Print only the compress type if possible.

  --compress-level|--compr-level
  Print only the compress level if possible.

  --compress-chunk|--compr-chunk
  Print only the compress chunk size in KiB if possible.

  Example:
$ lfs getstripe <file>
    lcme_compr_type:     lz4
    lcme_compr_lvl:      5
    lcme_compr_chunk_kb: 512
      lmm_pattern:       raid0,compress

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Ife0382469cbc5099e0c6dc96534bb169ddeff61e
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/49509
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>

EX-6245 csdc: define compress component layout

Define a layout containing compress component with a compress
pattern so that old client cannot process it.

Add a client parameter to enable file compression support.

lctl get_param llite.*.enable_compression
lctl set_param llite.*.enable_compression={0,1}

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: If88d446fc7425644ea9484004ffa4e6d2a05f7e6
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/49508
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>

LU-16717 mdt: resume dir migration with bad_type

LFSCK may set hash type to "none,bad_type" upon migration failure,
set it back to "fnv_1a_64,migrating,bad_type,fixed" to allow
migration resumption. fnv_1a_64 is set because it's the default hash
type, and now that we don't know the hash type in the original
migration command, just try with it.

LFSCK just add "bad_type" flag on such directory, so that such
migration can always be resumed in the future.

Add sanity 230z.

Lustre-change: https://review.whamcloud.com/50797
Lustre-commit: 151650e468ab423e831c30d635ea380e0434a122

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I19606aefcb9115e6724843785aea89a1c380e23f
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51230
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-16717 mdt: treat unknown hash type as sane type

Directory migration failure may leave directory hash type as
LMV_HASH_TYPE_UNKNOWN|LMV_HASH_FLAG_BAD_TYPE, which should be treated
as sane hash type on existing directories, otherwise such directories
can't be unlinked.

Add sanity 230y.

Lustre-change: https://review.whamcloud.com/50796
Lustre-commit: 05cdb71ba6813570123613993f3cfcf74fc83561

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Ieffc0808d1db989d0bf9723f05cddb06f349e208
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51202
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-7596 tests: disable metadata_csum_seed for pcc cache device

latest e2fsprogs enable metadata_csum_seed feature when mkfs.ext4
a device, which is used in pcc test as a cache device. Such a ext4
fs cannot be mounted on an old kernel of el7. So disable this
feature for pcc cache device if it is detected in sanity-pcc test.

Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial testlist=sanity-pcc clientdistro=el7.9
Change-Id: I2ef3ff27b49fb479ec348b742cf614a43321813b
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51234
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn88

New tag 2.14.0-ddn88

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ib22ed479bd8d4f50b327b4287dd4c68cb1530db5

EX-7506 test: change ALWAYS_EXCEPT with function always_except

Change ALWAYS_EXCEPT with function always_except in replay-single.sh

Test-Parameters: trivial testlist=replay-single

Change-Id: I6fb4c350b1aa6f8c38920bf52e8ae42b64bdcd64
Signed-off-by: Wei Liu <sarah@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51086
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16842 fsx: tolerate delete last non-stale mirror error

fsx mirror split test could try to delete the last non-stale mirror
of a file and that's a tolerable error scenario. The fsx FLR test
randomly choose a mirror operation and this situation could happen.

Lustre-change: https://review.whamcloud.com/51090
Lustre-commit: TBD (from 9bc22dcefbe3a7ac66a3e8f2ad042f1e018d45a9)

Fixes: 04ab0cc869c (LU-14156 utils: mirror split to check for last in-sync early)
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I80c294da80740b21e00ae72a092fd8883ec7d60e
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51092
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14668 tests: verify state of peer added with '--lock_prim'

Add peer state verification to sanity-lnet test_26:
check that peer state has corresponding bit set for a peer
created with '--lock_prim' option.

Lustre-change: https://review.whamcloud.com/50249
Lustre-commit: 9b6fcfa334b153e52caec16d4cfd180306826a3a

Test-Parameters: trivial testlist=sanity-lnet
Fixes: 54a28697c ("LU-14668 lnet: add 'force' option to lnetctl peer del")
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: Id5fde036907f9dd19a21e8e6611a070321310f0e
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50972
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14668 lnet: add 'lock_prim_nid" lnet module parameter

Add 'lock_prim_nid' lnet module parameter to allow control
of how Lustre peer primary NID is selected.
If set to 1 (default), the NID specified by Lustre when
calling LNet API is designated as primary for the peer,
allowing for non-blocking discovery in the background.
If set to 0, peer discovery is blocking until complete
and the NID listed first in discovery response is designated
as primary.

Lustre-change: https://review.whamcloud.com/50159
Lustre-commit: fc7a0d6013b46ebc17cdfdccc04a5d1d92c6af24

Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: I6ed1cb0c637f4aa7a7340a6f01819ba9a85858f4
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50971
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14668 lnet: add 'force' option to lnetctl peer del

Add --force option to 'lnetctl peer del' command.
If the peer has primary NID locked, this option allows
for the peer to be deleted manually:
lnetctl peer del --prim_nid <nid> --force

Add --prim_lock option to 'lnetctl peer add' command.
If specified, the primary NID of the peer is locked
such that it is going to be the NID used to identify
the peer in communications with Lustre layer.

Lustre-change: https://review.whamcloud.com/50149
Lustre-commit: f1b2d8d60c593a670b36006bcf9b040549d8c13a

Test-Parameters: trivial
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: Ia6001856cfbce7b0c3288cff9b244b569d259647
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50970
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-11623 mdt: return UPDATE|PERM on open

This patch includes the following changes:
* try lock UPDATE|PERM ibits on open to speed up subsequent stat.
* open returns PR lock to client by default, because CR mode
  UPDATE|PERM ibits don't make sense since all modifications take
  PW lock.
* don't lock UPDATE|PERM ibits for PCC attach, otherwise these ibits
  revoke will cause file detach.
* update sanity-pcc 13a to make it fail on single client test if
  anything went wrong.
* update sanity-lfsck 31d because previously CR UPDATE lock is
  fetched, thus the test pass by mistake.

This should help common workloads with open followed by a stat
or other such operation.

Benchmark results:

This patch can significantly improve open-create + stat on the same
client.

This patch in combination with two others:

https://review.whamcloud.com/32157
https://review.whamcloud.com/33584

Improves the 'stat' side of open-create + stat by >10x.

Without patches (master branch commit 26a7abe):

mpirun -np 24 --allow-run-as-root /work/tools/bin/mdtest -n 50000 -d
/cache1/out/ -F -C -T -v -w 32k

   Operation                      Max            Min           Mean
   ---------                      ---            ---           ----
   File creation     :       3838.205       3838.204       3838.204
   File stat         :      33459.289      33459.249      33459.271
   File read         :          0.000          0.000          0.000
   File removal      :          0.000          0.000          0.000
   Tree creation     :       3146.841       3146.841       3146.841
   Tree removal      :          0.000          0.000          0.000

With the three patches:

mpirun -np 24 --allow-run-as-root /work/tools/bin/mdtest -n 50000 -d
/cache1/out/ -F -C -T -v -w 32k
SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean
   ---------                      ---            ---           ----
   File creation     :       3822.440       3822.439       3822.440
   File stat         :     350620.140     350615.980     350617.193
   File read         :          0.000          0.000          0.000
   File removal      :          0.000          0.000          0.000
   Tree creation     :       2076.727       2076.727       2076.727
   Tree removal      :          0.000          0.000          0.000

Note 33K stats/second vs 350K stats/second.

ls -l time of the mdtest directory is also reduced from 23.5 seconds
to 5.8 seconds.

Lustre-change: https://review.whamcloud.com/33585
Lustre-commit: 4b44204930cf4d312425dd95e51720fe5d3e801b

Change-Id: Ib3410629c190de6f74246a4a92f8216537fa2b95
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Signed-off-by: Qian Yingjiin <qian@ddn.com>
Tested-by: Shuichi Ihara <sihara@ddn.com>
Reviewed-by: Mikhail Pershin <mpershin@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51071
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-7668 utils: add lctl del_ost

Add helper command:

   lctl del_ost [--dryrun] --target fsname-OSTxxxx

Permanently disable an OST by altering the client and MDT llog
catalogs on MGS. The command finds all catalog records related to the
specified OST and cancel them. A --dryrun option is provided so that
the system administrator can see which records would have been
cancelled, but without actually cancelling them.

Lustre-change: https://review.whamcloud.com/41449
Lustre-commit: 1121816c4a4e1bb2ef097c4a9802362181c43800

LU-7668 tests: skip conf-sanity test_33a for old MGS

Skip del_ost test for MGS versions that do not have this command.

Lustre-change: https://review.whamcloud.com/50221
Lustre-commit: ba5346b050fb395844252a706d4dba2ef0e0d8dc

Signed-off-by: Stephane Thiell <sthiell@stanford.edu>
Change-Id: I58c4f10fa0f7164a40231e807698eb224cccf062
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50923
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-6275 lustre: add lz4 and lz4hc kernel modules

lz4 and lz4hc kernel modules implement compression according to the
lz4 and lz4hc algorithms respectively, through the kernel Crypto API.
lz4 module provides 2 cipher drivers under the generic name 'lz4':
* lz4-lustre-generic of type compression
* lz4-lustre-scomp of type scomp
lz4hc module provides 2 cipher drivers under the generic name 'lz4hc':
* lz4hc-lustre-generic of type compression
* lz4hc-lustre-scomp of type scomp

lz4 and lz4hc kernel module sources are copied from linux v6.1-rc5,
and renamed to llz4.c and llz4hc.c respectively to avoid name
collisions. Use of vmalloc has been changed to kvmalloc since it is
faster in most cases. They implement the Crypto API interface, and
rely on the lz4/lz4hc kernel library for compression implementation.
They have been modified to grok a compression acceleration/level, as
read from the top 4 bits of the crypto_tfm flags, and pass it to the
underlying library.
The lz4/lz4hc library sources are also copied from linux v6.1-rc5 and
built statically, so lz4_compress, lz4_decompress and lz4hc_compress
sources have been "de-modulified", and EXPORT_SYMBOLs removed. Headers
have also been copied from linux v6.1-rc5 for consistency, and source
files modified to include the copied headers instead of the system
headers.
All aforementioned sources are located in the lustre/lz4 directory.

The lz4/lz4hc modules are built only if the kernel does not provide
them. This is verified by checking if the CONFIG_CRYPTO_LZ4 and
CONFIG_CRYPTO_LZ4HC kernel config options are defined.

This patch provides unit testing of several compression modules, such
as lz4, lz4hc, lzo and gzip. This is done via a new test kernel module
kcompr.ko, and new sanity test_84. File lustre/tests/kernel/kcompr.c
contains examples of how to call the compression/decompression
routines.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I74ce95ff18194c6052d291588d7e8c79596a7f23
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/49208
Reviewed-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14294 tests: fixed NFS configuration issue

* Used the systemctl command to manage system services
* Used the same order of parameters to setup and cleanup NFS
* Used tab for indentation

Lustre-change: https://review.whamcloud.com/49062
Lustre-commit: 1a8fe55b17ac2bc2195aaba446467ccdac67b564

Test-Parameters: trivial clientdistro=el7.9 \
testlist=parallel-scale-nfsv3,parallel-scale-nfsv4
Test-Parameters: clientdistro=el8.7 \
testlist=parallel-scale-nfsv3,parallel-scale-nfsv4
Test-Parameters: clientdistro=el9.0 \
testlist=parallel-scale-nfsv3,parallel-scale-nfsv4
Test-Parameters: clientdistro=sles15sp4 \
testlist=parallel-scale-nfsv3,parallel-scale-nfsv4
Test-Parameters: clientdistro=ubuntu2004 \
testlist=parallel-scale-nfsv3,parallel-scale-nfsv4
Test-Parameters: clientdistro=ubuntu2204 \
testlist=parallel-scale-nfsv3,parallel-scale-nfsv4

Change-Id: I6b087035ac7524aa99c0facad48f8c3fb7444cbc
Signed-off-by: Alex Deiter <adeiter@tintri.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50889
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-11912 fid: clean up OBIF_MAX_OID and IDIF_MAX_OID

Define the OBIF|IDIF_MAX_OID macros to 1ULL << OBIF|IDIF_MAX_BITS - 1
Clean up the callers and remove OBIF|IDIF_OID_MASK which are not used.

Lustre-change: https://review.whamcloud.com/45659
Lustre-commit: bb2f0dac868cf1321277bc3d7d6fc71f016d921b

Test-Parameters: trivial
Change-Id: I9a679b930c73da5904b2eb4c74f785fc1d27a8a0
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sergey Cheremencev <scherementsev@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50759
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15670 clio: Disable lockless for DIO with O_APPEND

Lockless O_DIRECT with O_APPEND can allow interleaved / racy
appends from concurrent I/O.

Disable lockless I/O when O_APPEND is set

Lustre-change: https://review.whamcloud.com/46890
Lustre-commit: 649d638467c0375797cd59ab7c9ac4113e6c682e

HPE-bug-id: LUS-9776
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I5c56f92c90e631c295f56e5958985f516e1990f8
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/49666
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-7544 lipe: set lpcc interval to 30 by default

set the default value of interval lpcc.conf and lpcc_purge
to 30 seconds everywhere.

Test-Parameters: trivial testlist=sanity-pcc
Signed-off-by: Lei Feng <flei@whamcloud.com>
Change-Id: Iaee812bdc0d3c4d04549092e5c8aa78b0a2f1dbd
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51058
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16331 utils: fix 'lfs find -O <uuid>' with gaps

Fix the UUID parsing in llapi_get_target_uuids() so that the OST
UUIDs are put into the right slots when there is a gap in numbering.
Otherwise, "lfs find -O <uuid>" will not be able to find the given
UUID if it is the first OST after the gap.

Add test case for 'lfs find' and 'lfs getstripe' with large/sparse
OST indices in conf-sanity test_81.

Lustre-change: https://review.whamcloud.com/49207
Lustre-commit: 05334b90a5d3ddd6c8eabc3683fd487f47df6e35

Test-Parameters: trivial testlist=conf-sanity env=ONLY="81-82"
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ia0581f85f016c202514148114924509118a0f792
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Feng Lei <flei@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51076
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-7553 - reduce b_es6_0 failover-part-3  duration

Reduced recovery-mds-scale duration to 12 hours

Test-Parameters: trivial
Test-Parameters: austeroptions="-R" clientcount=4 clientdistro=el7.9 \
                 env=SLOW=yes failover=true iscsi=1 mdscount=2 \
                 mdssizegb=10 osscount=2 ostsizegb=2 serverdistro=el7.9 \
                 testgroup=failover-part-3

Change-Id: I791044863b5c52f44a0db36e816581b9d4655287
Signed-off-by: Charlie Olmstead <charlie@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51099
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14692 tests: allow FID_SEQ_NORMAL for MDT0000

Fix the tests asssuming objects created for MDT0000
always have a seq number of 0, to prepare for
deprecating IDIF sequence.

Fix sanity test_312 on ZFS to properly identify which
OST the object was created on, but do not re-enable it.

Lustre-change: https://review.whamcloud.com/46293
Lustre-commit: eaae4655567b16260237764dadb7ab57df8b0edd
Lustre-change: https://review.whamcloud.com/49720
Lustre-commit: 8767d2e44110fc19e624e963d5ebc788409339d3

Test-Parameters: trivial
Test-Parameters: testlist=sanity-scrub env=ONLY=19
Test-Parameters: testlist=sanity-sec env=ONLY=37
Change-Id: I4bffabe25a6f84cdba760aabea1da3429715a283
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50756
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15252 mdt: reduce contention at mdt_lsom_update

mot_som_mutex serialize all close requests with lsom updates for
a same mdt_object. For a massive open/read/close single shared
file load, it leads to high load avarage cause many threads sleep
on mutex.
This patch introduces a cached lsom size, and uses a mutex at update
part only. Close requests with lsom size less or equal to cached size
would not take a mutex at all.

Test results MPI open/flock/funlock/close SSF
10 iterations 10 node 100 thread each, 1000 file ops per thread
close time secs master patch MDT load avarage master patch
avg             0.142  0.086                  47.05  38.89
max             0.164  0.129                  49.39  44.77
min             0.097  0.041                  44.44  34.7

Lustre-change: https://review.whamcloud.com/45709
Lustre-commit: c8b7afe4970415f8dae84f5e20661f8a3b3681a0

Signed-off-by: Alexander Boyko <alexander.boyko@hpe.com>
Change-Id: I807b468b128295df9391b0467e74d4f10240662e
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andrew Perepechko <andrew.perepechko@hpe.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51030
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-6841 tests: do not truncate over s_maxbytes

sanity-lipe-scan3/104 truncates file to MAX_LFS_FILESIZE
but it should not go over s_maxbytes.

Fixes: c05dbbbbca ("EX-4015 lipe: add lipe_scan3")
Test-Parameters: trivial testlist=sanity-lipe-scan3 env=ONLY=104
Change-Id: I4a14befb0dfc34b2611052850b77a0841ff853aa
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50899
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexandre Ioffe <aioffe@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16529 test: verify the grant > 0 before test

In the test 84 in sanity-quota, the grant should be larger than 0
after the dd complete, this patch adds check for it and forces the
quota to be re-integrated during the check.

Lustre-change: https://review.whamcloud.com/50799
Lustre-commit: fdffbd3d7842a2f99e5fcdcb5b6e6766949f6333

Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Change-Id: I9cddcf0c4afc12a11f3535792262ebb35a1e480e
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sergey Cheremencev <scherementsev@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51094
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16674 obdclass: optimize job_stats reads

This patch has 2 objectives:

1/ limit the lock time on ojs_list (list of job stats)

"lctl get_param mdt.*.job_stats" can not dump job_stats in a single
read (seq_file buffer is limited to 4k). So, several reads are needed
to dump the full job list.
For each read, we have to find the job entry corresponding to the file
offset. For now, we walk ojs_list from the beginning to get this
entry.

This patch saved the last known entry and the corresponding offset to
start the next read from here.

2/ avoid the lock contention when reading job_stats

This patch replaces the read lock on ojs_lock by RCU locking, this
enables userspace processes reading the job_stats not to interfere
with the kernel target threads.

Add the stress test sanity 205g to check for possible races.
Add stack_trap in sanity test 205a and 205e to restore jobid_name and
jobid_var.

* Performance *

The following command is used to capture records:
$ time grep -c job_id /proc/fs/lustre/mdt/lustrefs-MDT0000/job_stats

- job_stats dump with no fs activity

Here are results after ending sanity test 205g with slow mode and
job_cleanup_interval=300s.
               ___________________________________
              | nbr of job | time | rate          |
 _____________|____________|______|_______________|
|without patch| 14749      | 1.3s | 11345 jobid/s |
|_____________|____________|______|_______________|
|with patch   | 22209      | 0.6s | 37015 jobid/s |
|_____________|____________|______|_______________|
|diff %       | +43%       | -54% | +226%         |
|_____________|____________|______|_______________|

- job_stats dump with fs activity

Here are results before ending sanity test 205g with slow mode and
job_cleanup_interval=300s.
               ___________________________________
              | nbr of job | time | rate          |
 _____________|____________|______|_______________|
|without patch| 14849      | 2.3s | 6428  jobid/s |
|_____________|____________|______|_______________|
|with patch   | 22776      | 1.2s | 18823 jobid/s |
|_____________|____________|______|_______________|
|diff %       | +53%       | -47% | +192%         |
|_____________|____________|______|_______________|

Lustre-change: https://review.whamcloud.com/50459
Lustre-commit: c6890a955f89508db46fd8ffbf22b05b145976cd

Test-Parameters: testlist=sanity env=SLOW=yes,ONLY=205g,ONLY_REPEAT=10
Test-Parameters: testlist=sanity env=ONLY=205g serverversion=2.14.0
Test-Parameters: testlist=sanity env=SLOW=yes,ONLY=205
Signed-off-by: Etienne AUJAMES <eaujames@ddn.com>
Signed-off-by: Xing Huang <hxing@ddn.com>
Change-Id: Ic4cd90965720af76eff0ed4e00ca897518bfbc66
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Feng Lei <flei@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51110
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16712 cksum: fix generating T10PI guard tags for partialbrw

To get better performance, we allocate a page as the buffer for
T10PI guard tags, we fill the buffer going over every page for brw,
when the buffer is considered full, we use
cfs_crypto_hash_update_page() to update the hash and reuse the buffer.

It doesn't work when there's a page in the brw gets clipped, and the
checksum sector size is 512. For a page with PAGE_SIZE of 4096,
and offset at 1024, we will end up with 6 guard tags, and won't have
enough space in the very end of the buffer for a full brw page, which
needs 8.

Work out the number of guard tags for each page, update the
checksum hash and reuse the buffer when needed.

Lustre-change: https://review.whamcloud.com/50540
Lustre-commit: 3999627447c01eebd96c14cc5cf8bba93f89a66b

Change-Id: Ic591e63b24534f2a42b670669520895cb35a9546
Fixes: b1e7be00cb ("LU-10472 osc: add T10PI support for RPC checksum")
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51079
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-12353 ldiskfs: add ext4-dquot-commit-speedup patch to more series

Add ext4-dquot-commit-speedup.patch to RHEL 8.x ldiskfs patch series.

Lustre-change: https://review.whamcloud.com/50853
Lustre-commit: TBD (from 06da805983c298f0957decfdb1d08cf7c39fd99b)

Test-Parameters: trivial clientdistro=el8.7 serverdistro=el8.7 testlist=sanity
Change-Id: Ib0ac325bde442b4eafedde9ba44984b02d5ea061
Fixes: dad25f258e50 ("LU-12353 ldiskfs: speedup quota journalling")
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50984
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn87

New tag 2.14.0-ddn87

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I6450c71a5e3ada3e0f63670867420a57c88bde5d

LU-16649 llite: EIO is possible on a race with page reclaim

We must clear the 'uptodate' page flag when we delete a
page from Lustre, or stale reads can occur.  However,
generic_file_buffered_read requires any pages returned from
readpage() be uptodate.

So, we must retry reading if page truncation happens in
parallel with the read.

This implements the same fix as:
https://review.whamcloud.com/49647
b4da788a819f82d35b685d6ee7f02809c05ca005

did for the mmap path.

Lustre-change: https://review.whamcloud.com/50344
Lustre-commit: 1d98e5c32b41e19bb1247958e666bb66e69dbc4c

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Iae0d1eb343f25a0176135347e54c309056c2613a
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50346
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

Revert "LU-14541 llite: Check vmpage in releasepage"

This reverts commit c524079f4f59a39b99467d9868ee4aafdcf033e9,
because it breaks releasepage for Lustre and does not
completely fix the data consistency issue in LU-14541.

Breaking releasepage matters because it prevents direct I/O
from working if there is page cache data present, and
because it causes similar issues with GDS, which must be
able to flush page cache pages before doing I/O.

With patches:
"LU-16160 llite: SIGBUS is possible on a race with page reclaim"/
d9c23a7934747eb19e23470b30806482a1aa60f8
and
"LU-14541 llite: Check for page deletion after fault"/
19678e30147f50f813e72e8216cfb0453fe0ca6e
LU-14541 is fully resolved, so we can revert this patch.

Lustre-change: https://review.whamcloud.com/49654
Lustre-commit: e3cfb688ed7116a57b2c7f89a3e4f28291a0b69f

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I613bdb4f27161ffc3638d1d8ea38827af5a7bd47
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50304
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7395 pcc: use llapi_open_by_fid to check pinned files

When check whether a file was pinned in PCC backend, it reported:
"cannot read or parse pin xattr of file
'/lustre/fsr/.lustre/fid/[0x780001b83:0x2138:0x0]'.: No such file
or directory (2)"

The failure reason is that open by FID is not configured for
subdirectory mounts.
In this patch, we use llapi_open_by_fid (which supports for
subdirectory mount) to open the file to solve this error.

Change-Id: If0120d745418836cfdf449a795b6f524c40f9c27
Signed-off-by: Qian Yingjin <qian@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50770
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-11170 tests: don't fail sanity/415 in VM

Don't fail sanity test_415 when running in a VM due to variable
runtimes for the tests.

A proper solution would be to examine the logs to determine if
the renames are blocked or just all slow due to VM contention.

Lustre-change: https://review.whamcloud.com/50654
Lustre-commit: 73a7b1c2a3f0114618db7781adb56974ed682f24

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I1a5d0f601705c9ec8559e760c4ec27c7f83ebbe5
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-by: Vikentsi Lapa <vlapa@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50938
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-10733 tests: increase conf-sanity/106 OST size

conf-sanity test_106 is trying to create ~64k files, but OST0000
only has about 48k objects in this case, so the file creates are
failing during the test.  This makes the test somewhat unreliable
and hitting errors not related to what was originally intended
(llog wrap handling).

Increase the OSTSIZE for this test to handle the number of objects
needed by the test so it can run more reliably.

Lustre-change: https://review.whamcloud.com/50732
Lustre-commit: 334d780617561c66c91697fb1681ce24b5379387

Test-Parameters: trivial testlist=conf-sanity env=ONLY=106
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ie33825801172ea565d9d1d5fb81595d2cad65677
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50714
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16650 kernel: update RHEL 7.9 [3.10.0-1160.88.1.el7]

Update RHEL 7.9 kernel to 3.10.0-1160.88.1.el7.

Lustre-change: https://review.whamcloud.com/50553
Lustre-commit: bd0d79456b91db58a75eeb717c7805d78d8a9a1a

Test-Parameters: trivial clientdistro=el7.9 serverdistro=el7.9

Change-Id: I4119595943940cca94d1853b59c94a02fed8cb71
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50556
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14699 tests: fix server version to check

tests 160g and 160s check Lustre version wrongly, patch
corrects these checks to use correct version

Test-Parameters: trivial
Fixes: 735136ead955 (LU-14699 mdd: proactive changelog garbage collection)
Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: Ib6a78a55672629686b59f0227e81d78f28bb81ac
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50957
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16739 uapi: make lustre_disk.h buildable in user land

The rbac work introduced a regression that makes lustre_disk.h
UAPI header no longer buildable in user land. This is causing
sanity test 400b to fail with:

lustre_disk.h:266:18: error: 'LUSTRE_NODEMAP_NAME_LENGTH' undeclared here (not in a function)
  char   ncr_name[LUSTRE_NODEMAP_NAME_LENGTH + 1];
                  ^~~~~~~~~~~~~~~~~~~~~~~~~~
lustre_disk.h:267:20: error: 'ncr_flags' is narrower than values of its type [-Werror]
  enum nm_flag_bits ncr_flags:8;
                    ^~~~~~~~~
lustre_disk.h:267:20: error: field 'ncr_flags' has incomplete type
lustre_disk.h:268:21: error: 'ncr_flags2' is narrower than values of its type [-Werror]
  enum nm_flag2_bits ncr_flags2:8;
                     ^~~~~~~~~~
lustre_disk.h:268:21: error: field 'ncr_flags2' has incomplete type
lustre_disk.h:277:2: error: unknown type name 'lnet_nid_t'
  lnet_nid_t nrr_start_nid;
  ^~~~~~~~~~
lustre/lustre_disk.h:278:2: error: unknown type name 'lnet_nid_t'
  lnet_nid_t nrr_end_nid;
  ^~~~~~~~~~

To fix this move several pieces of nodemap handling from lustre_idl.h
to lustre_disk.h.

The git commit 5e6a51787fef20b849682d8c49ec9c2beed5c373 for Linux
kernel version 6.2.0-rc5 made guid_t only available for kernel code.
The only UAPI data structure left is uuid_le. Thankfully MCE requires
this otherwise even uuid_le would be removed. We will need to keep
an eye on this.

Lustre-change: https://review.whamcloud.com/50641
Lustre-commit: 5a6725d19d4037026d7cab2442b0c639d1511e5d

Test-Parameters: trivial testlist=sanity envdefinitions=ONLY=400b
Fixes: b3b61b85cad ("LU-16524 nodemap: add rbac property to nodemap")
Change-Id: I4b962572ec2bf76159a17807c564390ded00d630
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50912
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-7449 pcc: output valid state for valid cached files

There are two cases where the command 'lfs pcc state' reports
'none' for PCC status of a file:
- File has not been cached at all into PCC. When the file is read,
  data will come from remote Lustre filesystem.
- File was cached into PCC but system cache dropped on client
  later (e.g. 'sysctl -w vm.drop_caches=3'). When file is read,
  file layout version needs to be compared against remote file
  system. And if verion is matching, data will com from PCC.

This patch adds a valid flag to distinguish between these two
states. For the latter case, the command 'lfs pcc state' will
output as follows:
$ lfs pcc state /mnt/lustre/f105.sanity-pcc
file: /mnt/lustre/f105.sanity-pcc, type: none, flags: valid

Add sanity-pcc/test_105 to verify it works as expected.

Test-Parameters: trivial testlist=sanity-pcc
Change-Id: I1d729bfe550b1bde0e78e8b3ec8217cd598fb64c
Signed-off-by: Qian Yingjin <qian@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50860
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16758 krb: use Kerberos machine principal in client

In addition to having Lustre client rely on the
lustre_root/<hostname>@REALM principal to authenticate, support the
more standard Kerberos machine principal host/<hostname>@REALM.
That avoids the need for additional keytab entries, and brings Lustre
in line with other services such as OpenSSH and NFS.

Lustre-change: https://review.whamcloud.com/50709
Lustre-commit: 74890266a39297c1c3a41263a7bfd86e0d8e426a

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Id50cef1a3a94248b958ce9ea42b5ae356f29cbf1
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50911
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7447 utils: lamigo to ignore ENODATA

lamigo shouldn't print an error message when replication fails
with ENODATA - this is a valid case as llapi_layout_get_by_path()
returns ENODATA in all case when a layout can't be fetching,
including non-existing file.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I8184d4af78514016b1afb3bca2eb34caf123d3ca
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50858
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alexandre Ioffe <aioffe@ddn.com>

LU-16782 kernel: update RHEL 9.1 [5.14.0-162.23.1.el9_1]

Update RHEL 9.1 kernel to 5.14.0-162.23.1.el9_1 for Lustre client.

Lustre-change: https://review.whamcloud.com/50785
Lustre-commit: TBD (from ecb01fccaf9deb30ae0688e353a2b379a30ce65e)

Test-Parameters: trivial clientdistro=el9.1 testlist=sanity
Test-Parameters: trivial serverdistro=el8.7 clientdistro=el9.1 testlist=sanity

Change-Id: I961bac2129b98da2950694fa03e0bf47b780d85c
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50787
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15140 tests: cleanup of recovery-*-scale tests fails

Bash trap handler is executed only after completition of
current command, so under big I/O load it can be executed
after test and cleanup phase finished.

Run I/O load in background overcome bash limitation.

Lustre-commit: f252abc6690247ee9608dbde80238add0ecaed8c
Lustre-change: https://review.whamcloud.com/45824

Test-Parameters: trivial clientcount=6 mdtcount=2 mdscount=2 osscount=2 austeroptions=-R failover=true iscsi=1 env=FAILOVER_PERIOD=180 testlist=recovery-double-scale env=SLOW=yes
Test-Parameters: clientcount=5 mdtcount=2 mdscount=2 osscount=2 austeroptions=-R failover=true iscsi=1 env=FAILOVER_PERIOD=180 env=DURATION=82800 testlist=recovery-mds-scale env=SLOW=yes
Test-Parameters: clientcount=5 mdtcount=2 mdscount=2 osscount=2 austeroptions=-R failover=true iscsi=1 env=DURATION=82800 testlist=recovery-random-scale env=SLOW=yes

Signed-off-by: Elena Gryaznova <elena.gryaznova@hpe.com>
HPE-bug-id: LUS-2649
Change-Id: I3c91cac4d3f9af9863e8f48ba8a6bae02190ccb4
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/46749
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16163 tests: skip racer_on_nfs for NFSv3

Export ALWAYS_EXCEPT env for child NFS test

Lustre-change: https://review.whamcloud.com/50579
Lustre-commit: 892d726f274c7cd4e505689ad69194ac68dc323b

Fixes: 513eb670b0 ("LU-16163 tests: skip racer_on_nfs for NFSv3")
Test-Parameters: trivial testlist=parallel-scale-nfsv3
Signed-off-by: Alex Deiter <adeiter@tintri.com>
Change-Id: Ibb4a9916166f13ab9bd2374b33d4313453972276
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50801
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16772 quota: protect lqe_glbl_data in qmt_site_recalc_cb

lqe_glbl_data should be protected with lqe_glbl_data_lock in
qmt_site_reaclc_sb like it did in other places. Otherwise it
may cause following panic:

  BUG: unable to handle kernel NULL pointer at 00000000000000f8
  qmt_site_recalc_cb+0x2f8/0x790 [lquota]
  cfs_hash_for_each_tight+0x121/0x310 [libcfs]
  qmt_pool_recalc+0x372/0x9f0 [lquota]

Also protect lqe_glbl_data access with lqe_glbl_data_lock in
qmt_lvbo_free().  Add debugging to see how often this is hit.

Lustre-change: https://review.whamcloud.com/50748
Lustre-commit: TBD (from e3511c6dbfb097308f48957e2e2df8c25f87030a)

Fixes: 1dbcbd70f8 ("LU-15021 quota: protect lqe_glbl_data in lqe")
Signed-off-by: Sergey Cheremencev <scherementsev@ddn.com>
Change-Id: I030f14b02062151f1708a03ac7414a9991f798f6
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50784
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16735 test: cancel MDC locks and wait recovery

During test_35, the MDC LDLM locks should also be cancelled to
flush the pending operations and the recovery should be waited
to complete before checking the quota.

Lustre-change: https://review.whamcloud.com/50638
Lustre-commit: 6c42ca6e445ff41f17c92f1ec875479388a59212

Test-Parameters: trivial testlist=sanity-quota mdscount=2 mdtcount=4
Change-Id: I6508644976be77ad2895107abf90144b51790cfe
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50644
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16286 ldiskfs: reimplement nodelalloc optimization

fiemap calls perform costly delayed extent search affecting
BRW performance, however, in Lustre we don't use delayed
allocation at all. Let's skip this search completely as we did
in RHEL7.

Lustre-change: https://review.whamcloud.com/49007
Lustre-commit: 3dd73b5c5d61a219c702873711055cb1cc80394a

LU-16286 ldiskfs: add ext4_find_delayed_extent patch to more series

Add rhel8.4/ext4-optimize-find_delayed_extent.patch to RHEL 8.7
ldiskfs patch series.

Test-Parameters: trivial clientdistro=el8.6 serverdistro=el8.6 testlist=sanity
Test-Parameters: trivial clientdistro=el8.7 serverdistro=el8.7 testlist=sanity

Change-Id: I2c3562cf5cbdf3c5532e4b79b28a040a995322b7
Signed-off-by: Andrew Perepechko <andrew.perepechko@hpe.com>
HPE-bug-id: LUS-11161
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50811
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>

LU-16523 lprocfs: adjust the format of rename_stats

Adjust the format of rename_stats to a more human-friendly YAML.

Lustre-change: https://review.whamcloud.com/49869
Lustre-commit: 73b5d7db7e8d3ede42524fc447fb30fa05ea7a3f

Fixes: bedb797c5d ("LU-16110 lprocfs: make job_stats and rename_stats valid YAML")
Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial
Change-Id: I20e6d07c974e907bb2e30412dd1899f845de2021
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/49871
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16087 lprocfs: add histogram to stats counter

Add histogram to stats counter.
Enable histogram for read/write_bytes in mdt/obdfilter
job stats.

Sample job_stats:
- job_id:          md5sum.0
snapshot_time   : 3143196.864165417 secs.nsecs
start_time      : 3143196.707206168 secs.nsecs
elapsed_time    : 0.156959249 secs.nsecs
  read_bytes:      { samples: 2, ..., hist: { 32K: 1, 1M: 1 } }
  write_bytes:     { samples: 1, ..., hist: { 1K: 1 } }

Lustre-change: https://review.whamcloud.com/48278
Lustre-commit: fde40ce32c91c804cb85be085f2aaf06170047b6

Signed-off-by: Lei Feng <flei@whamcloud.com>
Change-Id: I75b6909c8b63f08b74c3c411ff3dcd27881bb839
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Shuichi Ihara <sihara@ddn.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/49760
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15642 obdclass: use consistent stats units

Use consistent stats units, since some were "usec" and others "usecs".
Most stats already use LPROCFS_TYPE_* to encode type stats type, so
use this to provide units for those stats, and only explicitly provide
strings for the few stats that don't match the commonly-used units.
This also reduces the number of repeat static strings in the modules.

Lustre-change: https://review.whamcloud.com/46833
Lustre-commit: b515c6ec2ab84598c77c65eb78f1afd5e67b1ede

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I25f31478f238072ddbf9a3918cd43bb08c3ebbe5
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Ben Evans <beevans@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/49759
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-11407 obdclass: add start time to stats files

When the stats files are initialized or reset, store the current
timestamp with the stats.  That allows computing average IO and
RPC rates over the accumulated stats lifetime, in addition to the
normal incremental operation rates found by comparing successive
values read from the stats file with the read interval.

Any stats that currently print the "snapshot_time:" header will
now also print "start_time:" and "elapsed_time:" fields as well.
Consolodate this printing into a helper function instead of
duplicating very similar code in many different functions.  Output
can't be exactly the same for all callers, because these fields are
embedded into different types of output files, but it is very close.

Change struct rename_stats and brw_stats to use a common name prefix.

Change the obd_job_stats timestamps to ktime_t so that we can use the
common helper function for printing the header.  It is easier to store
ojs_cleanup_interval internally as 1/2 of the maximum stats age, since
since division is more easily done when the value is initially set as
seconds compared to when it is ktime_t.  This may also be a tiny bit
more efficient since we don't do a divide/shift on each access.

Lustre-change: https://review.whamcloud.com/33201
Lustre-commit: ea2cd3af7bfabfa6876727ee44495f4c331bea8e

LU-16231 misc: fix stats snapshot_time to use wallclock

merged into this patch here to avoid landing broken stats.

Some "init" times were not initialized when stats were allocated or
cleared, do this for all stats shown by lprocfs_stats_header().

Rename struct osc_device fields from od_ to osc_ to avoid confusion
with struct osd_device. Having two od_stats was especially confusing.

Add a test case to verify snapshot_time, start_time, elapsed_time.

Lustre-change: https://review.whamcloud.com/48821
Lustre-commit: e42efe35eec7b9725f7f4fff86aaee04093366b0

LU-16110 lprocfs: make job_stats and rename_stats valid YAML

related to lprocfs_stats_header() are include here that were not
present when that patch was backported.  This fixes the output to
correctly indent the items to follow YAML formatting rules.

Lustre-change: https://review.whamcloud.com/48417
Lustre-commit: e96cb6ff1fea7a2bc62a6c0786fb0e07cbfda81a

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Iacefa17def455ef53a28fd14b6d4c670463ebbe5
Reviewed-by: Ben Evans <beevans@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Feng Lei <flei@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/49745
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-4141 test: extend expected number of keepalive msgs

Make test_165g tolerate wide range of number of
keepalive messages received by lamigo

Test-Parameters: trivial env=ONLY=165g testlist=sanity
Test-Parameters: trivial env=ONLY=165g testlist=sanity
Test-Parameters: trivial env=ONLY=165g testlist=sanity
Test-Parameters: trivial env=ONLY=165g testlist=sanity
Test-Parameters: trivial env=ONLY=165g testlist=sanity
Signed-off-by: Alexandre Ioffe <aioffe@ddn.com>
Change-Id: Ia3132d96420d571e9ed67d2baacdb25da2d52c4d
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50778
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7389 pcc: open file for detach O_RDONLY

llapi_pcc_detach_file is rdwr, but should just be rdonly.
This means files can be attached but not detached if the
client is mounted rdonly.

The fix is just to open the file in detach with O_RDONLY.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I3e289ab52ff760a8ab84a209b968109517953b52
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50749
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16768 lfs: copy optarg string other than using it directly

Copy optarg string for fp_format_printf_str lest it be messed
later.

Lustre-change: https://review.whamcloud.com/50733
Lustre-commit: 75db98cef3df8f9a0e1b6e7a5150f3c332e6167b

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Ib32883d3261ae921adf0fdd7b05bcbf728de7557
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50690
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16662 autoconf: fix configure test compile for CONFIG_KEYS

This fixes below configure error on Linux v5.19+:
$ ./configure --disable-server
...
checking whether to enable gss keyring backend... yes
checking if Linux kernel was built with CONFIG_KEYS in or as module...
no
configure: WARNING: GSS keyring backend requires that CONFIG_KEYS be
enabled in your kernel.
checking for keyctl_search in -lkeyutils... yes
configure: error: Cannot enable gss_keyring. See above for details.
$ grep CONFIG_KEYS -rn /boot/config-*
6884:CONFIG_KEYS=y

For in-tree IB support and without passing Linux src path when run
./configure, the LINUX_OBJ maybe just a soft link to O2IBPATH, they
are pointing to the same dir. E.g.:
O2IBPATH='/usr/src/kernels/6.1.8-3.0.0.7.oe1.aarch64'
LINUX_OBJ='/lib/modules/6.1.8-3.0.0.7.oe1.aarch64/build'
$ ls -l /lib/modules/6.1.8-3.0.0.7.oe1.aarch64/build
lrwxrwxrwx 1 root root 42 Feb  7 00:00
/lib/modules/6.1.8-3.0.0.7.oe1.aarch64/build ->
/usr/src/kernels/6.1.8-3.0.0.7.oe1.aarch64
In this case, current configure will put kernel's Module.symvers to
variable KBUILD_EXTRA_SYMBOLS. This should be avoided after kernel
v5.19 which contains commit "b8422711080f modpost: make multiple export
error". This making multiple export symbol as an error from a warning
which can be seen in the config.log:
...
ERROR: modpost: vmlinux: 'init_uts_ns' exported twice. Previous export
was in vmlinux
...

Lustre-change: https://review.whamcloud.com/50399
Lustre-commit: 321a533b868908f37d01a4b787f5a463a02e427c

Test-Parameters: trivial
Change-Id: I35295b3acc7fffb93716362f5d8c659eb922afcb
Signed-off-by: Xinliang Liu <xinliang.liu@linaro.org>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50569
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15300 mdt: refresh LOVEA with LL granted

this change tries to fix two problems:
1) mdt_reint_open() fetches LOVEA before layout lock is taken.
   this may race with another process changing the layout and
   may result in a stale layout returned with a granted layout
   lock - re-fetch LOVEA once layout lock is granted

2) lov_layout_change() should not apply old layouts which
   can get through when MDS doesn't take layout lock

3) LFSCK shouldn't ignore layout version stored on MDS to avoid
   a situation when version degrades compared to client's copy.

This patch misses an optimization and can result in a number of
useless calls to OSD to fetch LOVEA. To be fixed in a followup
patch.

Lustre-change: https://review.whamcloud.com/46413
Lustre-commit: efbe0f63eff8a9a7b192607382f6859e3b0088b8

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Idee1101d152ab09947faf6d75574a8761a7690a5
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50705
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16670 enc: make sure DoM files are correctly decrypted

Make sure DoM files are decrypted upon read by loading their
associated encryption context, via llcrypt_prepare_readdir()/
llcrypt_get_encryption_info().

Fix sanity-sec test_50 accordingly.

Lustre-change: https://review.whamcloud.com/50429
Lustre-commit: 1c424252d37c64e3c223c19dced3cad2649c1f61

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ie9ef3cbb08d2295a2fd10b9e9ab0862119c7723e
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Mikhail Pershin <mpershin@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50431
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-10026 csdc: reserve layout bits for compress component

Add layout bits for compress component layout.

* lcme_compr_type: compression type (gzip, lz4, lz4hc, lzo, etc.)
* lcme_compr_lvl: compression level (0=default, 1-15)
* lcme_compr_chunk_log_bits: chunk size = 2^(16+chunk_log_bits)

Component pattern:
* LOV_PATTERN_COMPRESS - file contains compressed data chunks and
       cannot be read by a client without decompression support.

Compress component flags:
* LCME_FL_COMPRESS - the component should be compressed with the
       compression algorithm stored in lcme_comp_type, at level
       lcme_comp_level, with chunk size 2^(16+lcme_chunk_log_bits)
* LCME_FL_PARTIAL - the component holds some uncompressed chunks due
       to poor write size/alignment, and may benefit from being
       recompressed as the full file data is available
* LCME_FL_NOCOMPR - the component should not be compressed because
       the data was found to be incompressible, or by user request

Lustre-change: https://review.whamcloud.com/49170
Lustre-commit: TBD (from 147d4eb27b85b4994a47539be6aceff212365ee5)

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Idca22cca87b01bba8a5b3c85ca62044abe1d30eb
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/49321
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16677 utils: add bitfields and ifdefs to wiretest

Add CHECK_BITFIELD() for "checking" bitfields in data structures
(currently only adds a comment to wiretest.c, maybe improve later).

Add CHECK_COND_START/FINISH() for adding #ifdef/#endif conditions
into wiretest, mainly for structs not used by (upstream) client.

Lustre-change: https://review.whamcloud.com/50479
Lustre-commit: 5a730827147714136b7d5035ca6115545a6b5ef0

LU-16677 utils: synchronize wirecheck.c and wiretest.c

wirecheck.c is not compilable and out of sync with wiretest.c.
The patch adds forgotten changes to wirecheck.c and replaces
wiretest.c by a version generated by "make newwiretest".

Lustre-change: https://review.whamcloud.com/50456
Lustre-commit: d43eb211995e0afb35690946c78ef6c82b9f86ad

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I8ecc4bdd5b5d651faa42f65ce8ea46da003ebbe5
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50751
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16524 sec: add fscrypt_admin rbac role

The purpose of the new fscrypt_admin rbac role is to control admin
tasks related to fscrypt. When not set, it is forbidden to all users
including root to modify existing protectors or policies, or create
new ones. But it remains possible to lock and unlock encrypted
directories.

Internally, this is achieved by marking fscrypt metadata files and
directories, i.e. everything under ROOT/.fscrypt, with a special mdt
object flag LOHA_FSCRYPT_MD.
Upon request processing, the mdt layer returns -EPERM if the flag
LOHA_FSCRYPT_MD is found on an object that is the target of a modify
request.
The LUSTRE_IMMUTABLE_FL flag is also returned to clients for such
objects.

sanity-sec test_64f is added to exercise the new fscrypt_admin flag.

Lustre-change: https://review.whamcloud.com/50184
Lustre-commit: 22bef9b6c64ef394a2efb41ce1388be71300af0d

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I99956499133994444ccd88e33340067790a182ce
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50339
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16524 sec: enforce rbac roles

There are 5 different rbac roles defined via nodemap:
- byfid_ops, to allow operations by FID (e.g. 'lfs rmfid').
- chlg_ops, to allow access to Lustre Changelogs.
- dne_ops, to allow operations related to DNE (e.g. 'lfs mkdir').
- file_perms, to allow modifications of file permissions and owners.
- quota_ops, to allow quota modifications.
Enforce these roles by checking the value of the 'rbac' nodemap
property on server side and returning -EPERM if operation is
forbidden.

Add sanity-sec test_64* to exercise these capabilities.

Lustre-change: https://review.whamcloud.com/49907
Lustre-commit: 971e025f5fb77f4eaaa1e9070598dfa6292a9678

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I37057f0ab50c02fa99db03cb04149a437e35ee0a
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50312
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-7391 lipe: lpcc_purge calculates disk usage as df

lpcc_purge calculate disk usage in the same way as df command.

Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial testlist=sanity-pcc
Change-Id: I43fe60188b1363d0ba58ea659b560b97807dc019
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50753
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Qian Yingjin <qian@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16057 obdclass: set OBD_MD_FLGROUP for ladvise RPC

ladvise RPC doesn't have OBD_MD_FLGROUP set, when RPC
reaches server, tgt_validate_obdo() will corrupt the FID
if it's seq is in FID_SEQ_NORMAL range.

Do not mess with seq in obdo_to_ioobj() and tgt_validate_obdo(),
since 2.0 all RPCs should have OBD_MD_FLGROUP set.

Add OBD_MD_FLGROUP for ladvise RPC to fix new client talking
to old servers.

Lustre-change: https://review.whamcloud.com/48080
Lustre-commit: bee803c6e440ba6b55e0ff356e5324f44cfa63eb

Change-Id: I373b7f32458b18e29d9bb716a912fe4a54eccac5
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-on: https://review.whamcloud.com/48080
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50755
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-12347 llite: do not take mod rpc slot for getxattr

The following scenario may lead to client eviction:
clientA                clientB                  MDS
threadA1: write to file F1, get
and hold DoM MDC LDLM lock L1:
   ->cl_io_loop()
    ->cl_io_lock()
     :
     ->mdc_lock_granted()
      ->lock->l_writers++
      [hold ref until write done]

threadA2-A8: create files F2-F8:
   ->ll_file_open()
    ->mdc_enqueue_base()
     ->ldlm_cli_enqueue()
      ->ptlrpc_get_mod_rpc_slot()
      ->ptlrpc_queue_wait()
      [hold RPC slot until create done]

                                              OST(s) in recovery.
                                              MDS waiting on OST(s) to
                                              precreate new objects.

threadA1:
   -> cl_io_start()
    -> __generic_file_aio_write()
     -> file_remove_suid()
      -> ll_xattr_cache_refill()
       -> mdc_xattr_common()
        -> ptlrpc_get_mod_rpc_slot()
        [blocked waiting for RPC slot]

                       threadB1: write file F1,
       enqueue DoM MDC lock L1

                                              MDS sends blocking AST
                                              to clientA for lock L1

ldlm_threadA3: cannot cancel busy lock L1:
   -> ldlm_handle_bl_callback()
   ["Lock L1 referenced, will be cancelled later"]

                                              MDS evicts clientA for
                                              not cancelling lock L1

threadA1: never completes write:
  ->cl_io_end()
   ->cl_io_unlock()
    ->osc_lock_cancel()
     ->lock->l_writers--;

The fix is to add IT_GETXATTR to list of operations which do not
need mod rpc slot.

Tests to illustrate the issue is added.

wait_for_function(): total sleep time (wait) is to be equal to max
when 1 is returned.

Lustre-change: https://review.whamcloud.com/44151
Lustre-commit: eb64594e4473af859e74a0e831316cead0f5c49b

Signed-off-by: Vladimir Saveliev <vlaidimir.saveliev@hpe.com>
HPE-bug-id: LUS-7271
Change-Id: I1b80677df084bda141b9ac58a78b765bd0b14a41
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andrew Perepechko <andrew.perepechko@hpe.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50754
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16341 quota: fix panic in qmt_site_recalc_cb

The panic occurred due to empty qit_lqes array after
qmt_pool_lqes_lookup_spec. Sometimes it is possible if
global lqe is not enforced. Return -ENOENT from
qmt_pool_lqes_lookup_spec if no lqes have been added.
It fixes following panic:
BUG: unable to handle kernel NULL pointer dereference at
00000000000000f8
...
RIP: 0010:qmt_site_recalc_cb+0x2ec/0x780 [lquota]
...
[ffffa5564118fda0] cfs_hash_for_each_tight at ffffffffc0c72c81
[libcfs]
[ffffa5564118fe08] qmt_pool_recalc at ffffffffc142dec7 [lquota]
[ffffa5564118ff10] kthread at ffffffffb45043a6
[ffffa5564118ff50] ret_from_fork at ffffffffb4e00255

Add test sanity-quota_14 that reproduces above panic
without the fix.

HPE-bug-id: LUS-11007
Change-Id: Ie51396269fae7ed84379bef5fc964cce789eba7c
Signed-off-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50793
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sergey Cheremencev <scherementsev@ddn.com>

LU-14541 llite: Check for page deletion after fault

Before completing a page fault and returning to the kernel,
we lock the page and verify it has not been truncated.  But
we must also verify the page has not been deleted from
Lustre, or we can return a disconnected (ie, not tracked by
Lustre) page to the kernel.

We mark deleted pages !uptodate, but this doesn't matter
for faulted pages, because the kernel assumes they are
returned uptodate, and maps them in to the process address
space.  Once mapped, the page state is not checked until
the page is unmapped.

But because the page is referenced by the mapping, it stays
in the page cache even though it's been disconnected from
Lustre.

Because the page is disconnected from Lustre, it will not
be found and cancelled on lock cancellation.  This can
result in stale data reads.

This is particularly an issue with releasepage (called from
drop_caches or under memory pressure), which can delete
pages separately from cancelling covering locks.

If releasepage is disabled, which is effectively what
"LU-14541 llite: Check vmpage in releasepage"
does, this is not an issue.  But disabling releasepage
causes other problems and is incorrect anyway.

Lustre-change: https://review.whamcloud.com/49653
Lustre-commit: b3d2114e538cf95a7e036f8313e9095fe821da79

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: If1164db8f8e92a1cf811431d56d15f30d8eb3faa
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50303
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13081 tests: skip sanity test_151/test_156

Skip both sanity test_151 and test_156 during interop testing,
since this is really testing server-side functionality only
(OSS caching behavior). And it makes sense to just exclude
test_151 and test_156 during interop testing, otherwise it
seems that the client version of the test can become
inconsistent with the caching behavior/tunables on the OSS
and the failures don't mean anything. There is enough
non-interop testing to catch any regressions in the OSS
cache behavior.

Lustre-change: https://review.whamcloud.com/50777
Lustre-commit: TBD (from 62cd8d19ff103e6e8a2b4bb7cdc00815ddb0edac)

Test-Parameters: trivial
Signed-off-by: Alex Deiter <adeiter@tintri.com>
Change-Id: I39a8b54894d5b0c7573e6c56d1f8e1ba02b3e3fe
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50887
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16775 tests: cleanup target after sanity-sec test_31

sanity-sec test_31 adds an LNet network tcp999, and associated
servicenode param on MDS target. This param must be cleared when
exiting the test, otherwise it can lead to incorrect client HA
behavior, trying to reach out to the fake service nodes.

Lustre-change: https://review.whamcloud.com/50766
Lustre-commit: TBD (e8b6c5a1128210a3a2bf525b5b489224932c8f88)

Test-Parameters: trivial
Test-Parameters: mdscount=2 mdtcount=4 osscount=1 ostcount=8 clientcount=2 \
    clientselinux testlist=sanity-sec env=SHARED_KEY=true,ONLY="31 61"
Fixes: c508c94268 ("LU-16557 client: -o network needs add_conn processing")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: If3a1926855bd23e9154c9a32b7a555e934e94565
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50767
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16515 tests: disable sanity test_118c/118d

Temporarily disable sanity test_118c and test_118d until there is
a fix available, since this is failing a large fraction of tests.

Lustre-change: https://review.whamcloud.com/50470
Lustre-commit: 7c52cbf65218d77c0594f92981173aa7d78f6758

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I16ebbc470a126bb99b5c3ecdf93407d6b73ebbe5
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50794
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn86

New tag 2.14.0-ddn86

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Iad3d4ffca7cdbb06657a679b5b580af788b6a5ec

LU-16603 protocol: add OBD_BRW_COMPRESSED

so the client can hint OST the data is compressed

Test-Parameters: trivial
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I4b721db3ad349d5745ee6698de368d0cb0138954
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50154
Reviewed-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Lustre-change: with short Gerrit URL https://review.whamcloud.com/50154
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50372
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15374 tests: check FULL and IDLE for client import state

The client-to-OST import state can be FULL or IDLE.

Lustre-change: https://review.whamcloud.com/49298
Lustre-commit: 25fb82eb413389b6023e0e61f7efb71e91d15c01

Test-Parameters: trivial testgroup=review-dne-part-3

Fixes: 25606a2ce1 ("LU-15342 tests: escape "|"")
Fixes: 3da8f014fd ("LU-12857 tests: allow clients to be IDLE after recovery")
Fixes: 5a6ceb664f ("LU-7236 ptlrpc: idle connections can disconnect")

Change-Id: I3582ceb273d241ee71fe907f6d1423746e453faa
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50632
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15342 tests: escape "|"

escape "|" on want="FULL|IDLE" to protect interpretation
by shell:
  sh: IDLE: command not found

Lustre-change: https://review.whamcloud.com/45788
Lustre-commit: 25606a2ce19e94c13694d46c3f15e9a10df40a91

Fixes: af666bef05 ("LU-12857 tests: allow clients to be IDLE after recovery")
Test-Parameters: trivial
Signed-off-by: Elena Gryaznova <elena.gryaznova@hpe.com>
Change-Id: I2f885ea225ba43537f37b8dad1c2e0cd8f652a79
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50631
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-12857 tests: allow clients to be IDLE after recovery

If clients are not connected to an OST when it fails (connection
is IDLE), they do not need to be involved in recovery, so this
should not be considered an error when checking the client state.

Lustre-change: https://review.whamcloud.com/45318
Lustre-commit: af666bef058c5b7997527fc851a84a89375912fb

Test-Parameters: trivial testlist=recovery-mds-scale env=SLOW=no
Test-Parameters: testlist=conf-sanity
Test-Parameters: testlist=replay-dual,replay-single
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I6cfeb718acd233378ed1608f22061bc15c3ebbe5
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50630
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16706 kernel: update RHEL 9.1 [5.14.0-162.22.2.el9_1]

Update RHEL 9.1 kernel to 5.14.0-162.22.2.el9_1 for Lustre client.

Lustre-change: https://review.whamcloud.com/50512
Lustre-commit: 9c316c69b9788ac219540e560a7831b88b81b690

Test-Parameters: trivial clientdistro=el9.1 testlist=sanity
Test-Parameters: trivial serverdistro=el8.7 clientdistro=el9.1 testlist=sanity
Change-Id: Ib5186e6f0dcd89660b7000db7f37c0c5a29f944f
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50528
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16710 kernel: update RHEL 8.7 [4.18.0-425.19.2.el8_7]

Update RHEL 8.7 kernel to 4.18.0-425.19.2.el8_7.

Lustre-change: https://review.whamcloud.com/50548
Lustre-commit: TBD (from 8cd92d6bdb3726fb25286ead1276622b884805d6)

Test-Parameters: trivial fstype=ldiskfs \
clientdistro=el8.7 serverdistro=el8.7 testlist=sanity

Test-Parameters: trivial fstype=zfs \
clientdistro=el8.7 serverdistro=el8.7 testlist=sanity

Change-Id: I17e43d98d1a3c7217e61771e8ed78a7123f9313f
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50551
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>