LU-17034 tests: memory corruption in PQ

Add conf-sanity_33c to test that there is no
memory corruption in PQ. The test uses OST
with index 0x7c6 to cause access out of
lqeg_arr which size is 64 by default.

Test-Parameters: trivial testlist=conf-sanity env=ONLY=33c
Signed-off-by: Sergey Cheremencev <scherementsev@ddn.com>
Change-Id: I401ce80b86701ff611df5f7078b6aecad147d6db
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52198
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17034 quota: tmp fix against memory corruption

Change QMT_INIT_SLV_CNT from 64 to 2000 to avoid accessing
memory out of array lqeg_arr. It could happen when at least
one of OSTs has index larger than the whole number of OSTs.
It is a temporary solution and maximum supported OST index
is 0x7d0. Later it will be changed with the longterm
solution.

Signed-off-by: Sergey Cheremencev <scherementsev@ddn.com>
Change-Id: Ic892352b7e833c58ea14bb7cfb98b4946f4ca9bb
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52180
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17005 obdclass: only print seconds for job_stats

Only print the seconds field for the job_stats snapshot_time: field.
Otherwise the addition of the usec and "secs.usec" units field can
break the output parsing.

There is no way to print only the seconds in lprocfs_stats_header(),
since all of the other snapshot_time: fields previously printed the
microseconds field also, so use seq_printf() with the old format.

Test-Parameters: trivial
Fixes: 5efb892396e3 ("LU-11407 obdclass: add start time to stats files")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I1bda709e3bc4231d42f6a98e7487f0b11445f056
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52212
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Feng Lei <flei@whamcloud.com>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>

LU-17071 o2iblnd: IBLND_REJECT_EARLY condition causes LBUG

The message printed when kiblnd_passive_connect recognizes
IBLND_REJECT_EARLY condition introduced by LU-16393 is trying
to derefence a NULL pointer in the parameter list. Fix this.

Lustre-change: https://review.whamcloud.com/52202
Lustre-commit: TBD (from a0fa2440765ee81b173de810b85a5bdb325bd274)

Test-parameters: trivial

Fixes: 1ea489f05b3 ("LU-16393 o2iblnd: add IBLND_REJECT_EARLY reject reason")
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: I711e5855383c140b9f7c35b27f48995f3f0e25ee
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52211
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Gaurang Tapase <gtapase@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn99

New tag 2.14.0-ddn99

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I2a58d93788f4bf1d7dbf74dc1490677252de95ce

EX-7601 llite: rename compr information in io

The compression related information in the cl_io is named
poorly.  It's not that the IO is compressed, it's that the
IO is to a compressed file.  The compr_chunk_log_bits is
the maximum from the entire file, not the maximum hit by
this IO.  (This is used by readahead, so the maximum in the
whole file is what's desired.)

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I46981a98628f127e7b147280caaf7544fa288786
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51771
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7601 lov: refactor lov_io_lsme_at

lov_io_lsme_at needs some minor changes to be called from
lov_io_slice_init().

Test-Parameters: trivial
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I0611d66052e22d349932eb26257369e07b9b8167
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51495
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-8027 pcc: add wait option when remove a PCC backend

In this patch, we add a "wait" option for the PCC tool when remove
PCC backend from a client:
lctl pcc del --wait $MOUNT $pcc_path
lctl pcc clear --wait $MOUNT

With this option, the caller must wait for all in-progress
attaches finished when remove the PCC backend from a client.

Change-Id: Ic8386329087a7129b0583fa823cbb50673893d0d
Signed-off-by: Qian Yingjin <qian@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51944
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-8027 pcc: wait for in-progress attaches when remove PCC

When remove a PCC backend from a client, it should wait for all
in-progress attaches finished. Othwise, it results in the failure
of the PCC backend umount operation.

The reason is that the PCC copy is referenced in the kernel, not
used by any applications in user space and the tool "lsof" can not
check whether the target PCC backend is used or not.

Change-Id: I05b268e75841f9f17e77819ed20c85c78d7c6ad6
Signed-off-by: Qian Yingjin <qian@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51940
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Feng Lei <flei@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-7073 tests: Add file migration to racer

Make racer run both blocking and non-blocking "lfs migrate" commands.
Implement this within the file_create.sh script, since it is already
selecting among different layout types.

Update Makefile.am to avoid listing every racer filename explicitly
to make it easier to add new types of operations in the future.

Lustre-change: https://review.whamcloud.com/c/fs/lustre-release/+/13669
Lustre-commit: e83569da38138859a51c660dfb5ca5bf45c70a37

Test-Parameters: trivial testlist=racer,racer,racer
Test-Parameters: fstype=zfs testlist=racer,racer,racer
Signed-off-by: Henri Doreau <henri.doreau@cea.fr>
Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I51b3f19c78029ff47102e96a71ec4a0fc472183a
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Zhenyu Xu <bobijam@hotmail.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52069
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-17015 gss: support large kerberos token on client

If the current Kerberos setup is using large token, like
when PAC feature is enabled for Kerberos, client can crash.

Return an error instead of asserting to avoid the crash
and increase the default buffer size to 4kB instead of 1kB.
This will only increase the SEC_CTX_INIT request size, and
the buffer is shrunk before being sent over the wire.

This will allow security token up to 2kB to be properly
handled by Lustre. Above that size, a different issue will
happen on server side that will require another patch.

Lustre-change: https://review.whamcloud.com/51946
Lustre-commit: TBD (from 374417f3f7c1e74e402a01ae9737ff01334d1dd4)

Test-Parameters: trivial kerberos=true testlist=sanity-krb5
Signed-off-by: Aurelien Degremont <adegremont@nvidia.com>
Change-Id: I9ce30ee7f8c95bfe41525c49986ffac45ffac97c
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51951
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17011 utils: monotonic clock in lfs mirror

use monotonic clocks instead of realtime to avoid affecting
bandwidth or hanging the transfer if the clock is changed.

Lustre-change: https://review.whamcloud.com/51852
Lustre-commit: TBD (from 81498f782e7c31e6e950352f4dbb2aa6f8052131)

Test-Parameters: trivial
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I58cf327d235448e93fa2ed63cefdf4dd01306e71
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51896
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16949 lnet: get monitor thread to update ping buffer

Make sure that ping buffer updates requested by o2iblnd and
socklnd are performed by the LNet monitor thread.
Having the LNDs do these updates via an LNet API directly caused a
lock-up due to spinlock acquisition while in an interrupt context
in Centos 7.9 environment.
To avoid LNet trying to update the ping buffer for an LNI which is
still initializing, check that o2iblnd net is fully initialized
(IBLND_INIT_ALL) before requesting the ping buffer update.

Lustre-change: https://review.whamcloud.com/51635/
Lustre-commit: 7ac399c5aec01186ad4c9a7153aea400777c897f

Fixes: da230373bd ("LU-16563 lnet: use discovered ni status")
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: I87ff8791937f5a0ead6096ff33e8c0a8087f8ddd
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51704
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17013 lov: fill FIEMAP_EXTENT_LAST flag

If file has N extents and get the fiemap with exactly N
extent slots, the last extent will miss FIEMAP_EXTENT_LAST
flag. Fix it.

Lustre-change: https://review.whamcloud.com/51863
Lustre-commit: TBD (from 264ab0c258adbce93d582e5e97f05ff7bf87c18a)

Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial testlist=sanityn env=ONLY="71a 71b 71c"
Change-Id: I4556b31f0d04bdf8e83f323e83b871b093beaa5e
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52114
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15926 nrs: fix tbf realtime rules

tc_nsecs_resid should be reset to 0 when changing a rule otherwise
this could lead to mds crashes for realtime policies.

nrs_tbf_req_get(): ASSERTION( cli->tc_nsecs_resid < cli->tc_nsecs )

Lustre-change: https://review.whamcloud.com/47585
Lustre-commit: 530861b344e46bef51c80adac4640c4586d8463a

Fixes: d11fa2c27959 ("LU-9228 nrs: TBF realtime policies under congestion")
Signed-off-by: Etienne AUJAMES <eaujames@ddn.com>
Signed-off-by: Xing Huang <hxing@ddn.com>
Change-Id: I280acb42e104088c6b8750a0bb7bf9c50cf96e73
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52115
Reviewed-by: Qian Yingjin <qian@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16760 utils: support 'lfs find --attrs' and '-printf %La'

Add support to "lfs find" to filter on file attribute flags, with the
syntax "[!] --attrs=[^]ATTR[,...]".
Add support to "lfs find" to print file attribute flags with
"-printf %La".

Lustre-change: https://review.whamcloud.com/51562
Lustre-commit: f0ab3ac6d6e31472c20ef538b799b96a512087f7

Add sanity-sec test_65 for Encrypted and Immutable flags.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I5e5cfe5c8c8cbed8bb79f3ad6d8116347ecfe6ac
Reviewed-by: Zhenyu Xu <bobijam@hotmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52067
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-8150 tests: hot-pools doesn't need yq

just a single test using yq for the trivial check..
we don't really need yq, IMO.

Test-Parameters: fortestonly testlist=hot-pools
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Ib11bd85938646ba1387d26e0d39cc54dcfe04bf0
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52092
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17043 enc: fix osd lookup cache for long encrypted names

Fix osd lookup cache to support files with long encrypted names.
Those encrypted names can be up to 256 bytes, not NUL terminated.

Lustre-change: https://review.whamcloud.com/52016
Lustre-commit: TBD (from 51a526bfa61bb5391a7ac33108e264f590cd3f0c)

Fixes: 07a7befdc1 ("LU-16405 osd: lookup cache")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ica2329c8a0990395307a14fe9bb9d43db3b364ed
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52017
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16405 osd: lookup cache

MDT may need to re-lookup just checked names (after locking).
introduce a trivial tiny per-thread cache in OSD in order to
make such a repeating lookup cheap.

the original issue is that ext4_add_entry() doesn't really
check for possible duplicate (that would be expensive as
a whole 4K block must be scanned).

important: the cache is reset upon request processing completion as
we don't update iversion on a disk (due to conflict with VBR).

Lustre-change: https://review.whamcloud.com/50521
Lustre-commit: 29f8eb2a67ba2806d91d93de1e82e05a63f76382

Fixes: 79acb9a9e7 ("LU-10235 mdt: mdt_create: check EEXIST without lock")
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I40c3ee702f7895c3bda00b380f904cd587e0a1c4
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51809
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16393 o2iblnd: add IBLND_REJECT_EARLY reject reason

Add IBLND_REJECT_EARLY reason for rejecting connection request:
to be used when the device doesn't have any nets added yet or
when there's no active NIs on the net to handle the connection.
These conditions are supposed to occur only when LNI is being
added/initialized, so report at CNETERROR level vs. CERROR.

In lnet, set NI state to ACTIVE only after it has been added
to the list of NIs for the net, so that LND can know that
the NI can be used to accept connections.

Lustre-change: https://review.whamcloud.com/51651
Lustre-commit: 673ff86a84ad5d11cde24aa7411c45385ad1c633

Test-parameters: trivial
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: I59efb2fdf5d5ceabb6ff23f638ec85da82d57b99
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52015
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16943 tests: use primary ost1 server in replay-single/135

This patch fixes replay-single test_135() to make sure
the primary ost1 server is used at the beginning of the test.

Test-Parameters: trivial testlist=replay-single

Test-Parameters: trivial env=FAILURE_MODE=HARD \
    clientcount=4 mdtcount=1 mdscount=2 osscount=2 \
    austeroptions=-R failover=true iscsi=1 \
    testlist=replay-single,mmp

Fixes: 18a424a0db1d ("LU-16943 tests: fix replay-single/135 under hard failure mode")
Change-Id: Ia25314255c9f00ba71687e1f757517f37031caed
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51913
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16298 ldiskfs: Periodically write ldiskfs superblock

This patch introduces a mechanism to periodically check and update
the superblock within the ext4 file system. The main purpose of this
patch is to keep the disk superblock up to date. The update will be
performed if more than one hour has passed since the last update,
and if more than 16MB of data have been written to disk.

This check and update is performed within the
ext4_journal_commit_callback function, ensuring that the superblock
is written while the disk is active, rather than based on a timer
that may trigger during disk idle periods.

Lustre-change: https://review.whamcloud.com/51340
Lustre-commit: e27a7b33d6351ff8b8bae101079af88f4eedac99

Signed-off-by: Vitaliy Kuznetsov <vkuznetsov@ddn.com>
Change-Id: I06eb9624b663a6ca6b15c6af2373b82f1bb63de6
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51717
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16872 tests: exercise sanity test_27M more fully

Improve the sanity.sh test_27M to precreate a bunch of files with
specific OST striping so that it is more likely to trigger the code
path that accessed the stale OST list when using O_APPEND layout.

Also clean up code style in the rest of this subtest.

Lustre-change: https://review.whamcloud.com/51602
Lustre-commit: 7bb1685048bf999df03ceadab39faa09b8a5560d

Test-Parameters: trivial testlist=sanity env=ONLY=27M,ONLY_REPEAT=200
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ie94e3a32fc48198e4e15f44a55d1f8ccf61c74f5
Reviewed-by: Thomas Bertschinger <bertschinger@lanl.gov>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52013
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16872 lod: reset llc_ostlist when using O_APPEND stripes

Files created with O_APPEND can have special striping set with the
parameters mdd.*.append_stripe_count and mdd.*.append_pool, and
should not inherit a list of OSTs to use from a parent directory
when these parameters are set. However, if a file is created with
O_APPEND and its create is handled by a kernel thread that has
previously created a file with a default list of OSTs, then those
defaults were erroneously applied to the O_APPEND file. This can
lead to the create returning EINVAL or to a crash.

This commit ensures that llc_ostlist is cleared when a file is
created with special append stripes.

Lustre-change: https://review.whamcloud.com/51559
Lustre-commit: 766b35a9700f36aa08b652fa9d18b890d34bf4a5

Signed-off-by: Thomas Bertschinger <bertschinger@lanl.gov>
Change-Id: Ib2023e17c9ef31a2e029e09e67b257eb2c77b113
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52012
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

Revert "LU-16651 llite: hold invalidate_lock when
 invalidate  cache pages"

This reverts patch 4debbda73f because of the hang issues
documented in NVDA-182.

Lustre-change: https://review.whamcloud.com/50371
Lustre-commit: bba59b1287c9cd8c30a85fafb4fd5788452bd05c

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I5a7b67b79d9594f9f03150acca4fc542c09c0798
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52024
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16958 llite: call truncate_inode_pages() in inode lock

In some cases vvp_prune()->truncate_inode_pages() is get called
without IO context, we need protect it with inode lock as well.

So we add ll_inode_info::lli_inode_lock_owner and set it according to
vfs lock rules (Documentation/filesystems/Locking or
Documentation/filesystems/locking.rst), so before calling
truncate_inode_pages(), we'd lock the inode if it's not locked in
vfs.

And in lov_conf_set(), when it requires inode lock, we'd take heed of
the possible inode size lock, inode layout lock and lov conf lock that
have been taken by itself, and it also need to take these locks in
order lest deadlock being ensued.

Lustre-commit: 51d62f2122fee14fbb3ff8333b5a830e1181e4e5
Lustre-change: https://review.whamcloud.com/50857
Lustre-commit: 8f2c1592c3bbd0351ab3984a88a3eed7075690c8
Lustre-change: https://review.whamcloud.com/51641

Fixes: ef9be34478 ("LU-16637 llite: call truncate_inode_pages() under inode lock")
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I7ee58039a6d31daefc625ac571a52baf112f8151
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51644
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn98

New tag 2.14.0-ddn98

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I21ed1babb45f0b7f66e32fa4cfd425fab1da29b3

RM-620 build: New tag lipe-2.28

New tag lipe-2.28

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I40ca1c408e11bed1e7ddfcf700930c0ffe9df0fc

LU-15535 llite: deadlock on lli_lsm_sem

it may happen that one process is doing lookup, and after reply while
holding the LDLM lock is trying to update LSM/default LSM under the
write lli_lsm_sem for a dir.

another process has taken the read lli_lsm_sem (taken for all the MD
ops in ll_prep_md_op_data()) and is waiting for a conflicting PW LDLM
lock on server for its modification for this dir.

it may happen on restriping with LSM, on changing the default LSM, but
even more often way is racer run even without striped dirs:
- racer does LFS mkdir -i $i <subdir> per each MDS, what creates a default
  LSM on these subdirs inherited endlessly - to keep the MDS index;
- racer also does mkdir -p <path>, in which case we do:
ll_new_node - create a parent dir, no RMF_DEFAULT_MDT_MD in reply
ll_lookup parent it=open - no RMF_DEFAULT_MDT_MD in reply
ll_new_node - create a child
the default LSM is inherited on the parent creation, however as those RPCs
do not have lookup LDLM lock and no data - the default layout is not set
for the parent in inode at the time of a child creation. thus a parallel
lookup which gets the LSM deadlocks with this ll_new_node().

at the same time, similar to CLIO, we do not need to hold a sem nor an
LDLM lock over the whole operation to avoid LSM modification on server,
we just need to take an uptodate LSM (this is a subject for LU-16320)
and to guarantee this op will be working on the client on this LSM for
the whole operation.

the solution is to let MD ops to work on a copy of LSM therefore letting
others to modify LSM attached to inode in parallel if needed.

Lustre-change: https://review.whamcloud.com/50489
Lustre-commit: 3ebc8e0528e34a11ffeff1e6be347de18b248069

HPE-bug-id: LUS-10725
Signed-off-by: Vitaly Fertman <vitaly.fertman@hpe.com>
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I3137300b5bcce2e890994ce8751cdf7fce2f3f54
Reviewed-on: https://es-gerrit.hpc.amslabs.hpecorp.net/161525
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51828
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15728 llite: fix relatime support

relatime behavior is properly managed by VFS, however
Lustre also stores acmtime on OST objects and atime
updates for OST objects should honor relatime behavior.

This patch updates 'ci_noatime' feature which was introduced to
properly honor noatime option for OST objects, to also support
'relatime'.
file_is_noatime() code already comes from upstream touch_atime().
Add missing parts from touch_atime() to also support relatime.

It also forces atime to disk on MDD if ondisk atime is older than
ondisk mtime/ctime to match relatime (even if relatime is not enabled)

Add a new test for relatime feature.

Lustre-change: https://review.whamcloud.com/47017
Lustre-commit: c10c6eeb37dd553166367b96369dca25183ace3b

Signed-off-by: Aurelien Degremont <degremoa@amazon.com>
Change-Id: I7a26f39841300a60c015944f9e544115b4446ead
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
LU-15728 mdd: fix sanity-56oc failure

sanity 56oc starts failing since relatime patch was landed.
'relatime' patch introduced an atime behavior change.  It was forcing
atime uptime to disk on MDD if ondisk atime is older than ondisk
mtime/ctime to match relatime (even if relatime was not enabled).

This was an optimization, trying to have a slightly better atime value
cheaply. This is unfortunately causing regression in sanity-56oc.
Let's remove it for now until we understand that better.

Lustre-change: https://review.whamcloud.com/50009
Lustre-commit: 8beeec77c3b426a01e1f10ca51149c7ca7e01b7e

Fixes: c10c6ee ("LU-15728 llite: fix relatime support")
Change-Id: Ieed4d4c7523c26cfc5bc230986d96b2acf152dee
Signed-off-by: Aurelien Degremont <degremoa@amazon.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51858
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16973 ptlrpc: flush delayed file desc if idle

The use of alloc_file_pseudo() allocates a real file descriptor,
so fput() will use a deferred cleanup for the descriptor, either
when the thread "finishes the syscall" (which never happens for
kernel threads), or a unmount time.  This accumulates too many
file descriptors (millions) on a busy system.

Instead of waiting to cleanup these file descriptors at unmount
time, call flush_delayed_fput() to clean them up when a ptlrpcd
thread becomes idle before it goes to sleep.

For kernels 5.3 and later when flush_delayed_fput() was first added,
and before kernel 5.6 when it was EXPORT_SYMBOL'd, grab a pointer
to the function with kallsyms_lookup_name() so it can be called.

Delete LN_CONFIG_STRSCPY_EXISTS reference that generates configure
warnings, since this check was renamed and moved to libcfs.

Lustre-change: https://review.whamcloud.com/51805
Lustre-change: 2feb4a7bb01c5e98763a62fb0bd64edf933c95de

Fixes: eed43b2a427b ("LU-13783 osd-ldiskfs: use alloc_file_pseudo to create fake files")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I24a08f9568d7d636a69672c5c3132ab25b292407
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51813
Tested-by: Shuichi Ihara <sihara@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16341 tests: skip sanity-quota/test_14 for old MDS

Skip sanity-quota test_14 for old MDS missing the fix
for LU-16341 kernel NULL in qmt_site_recalc_cb.

Lustre-change: https://review.whamcloud.com/51949
Lustre-commit: TBD (from 7d359e4de4a0159ac0fc1d15a18e93ca24d4e360)

Fixes: d965d63415 ("LU-16341 quota: fix panic in qmt_site_recalc_cb")
Test-Parameters: trivial testlist=sanity-quota env=ONLY=14
Signed-off-by: Alex Deiter <adeiter@tintri.com>
Change-Id: I1a23daa06f0cd306c2b034df18617c2650945b28
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51950
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17005 obdclass: allow jobstats histogram to be disabled

Allow the jobstats IO size histogram to be disabled with the
"enable_stats_header" tunable parameter.  This maintains
compatibility with older collectd and other stats-scraping
scripts that do not understand these new fields.

Default to disabled for EXA6.2.1, since it is new for this release.

Test-Parameters: trivial
Fixes: fde40ce32c ("LU-16087 lprocfs: add histogram to stats counter")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ie8bff8ef033f0b6e28c19548b7302d0913ddf905
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51966
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>

EX-8086 tests: skip sanity-compr for old servers

Skip for old servers.

Test-Parameters: trivial testlist=sanity-compr serverversion=EXA6.2.0
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ia42f1f32fe166bae84a7c38c02d6a218303cc51e
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51967
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Colin Faber <cfaber@ddn.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>

LU-16228 utils: improve lljobstats parsing

Change the hard-coded python3 interpreter path to use "env".

Allow lljobstats to parse JobIDs from Insight that have a leading
"@" if the jobname is not specified.  Otherwise it is not valid
YAML and causes the parser to choke.

Test-Parameters: trivial
Signed-off-by: Bjoern Olausson <bolausson@ddn.com>
Change-Id: Ibd7eba46fe80d39993810c06a3ada0f06ee694c6
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51964
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Feng Lei <flei@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16510 build: check if CONFIG_FORTIFY_SOURCE is defined

The linux/fortify-string.h header file should not be
included while the kernel config option CONFIG_FORTIFY_SOURCE
is not defined.

Lustre-change: https://review.whamcloud.com/51973
Lustre-commit: TBD (from 06956ef63fd7a0489648fe85c2f1e1904e78424e)

Fixes: 919b93b951d4 ("LU-16510 build: fortified memcpy from linux 6.1")
Change-Id: I2e1905406e892b182f143d512a2d3722b141e52d
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52000
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7806 csdc: set DoM compression component

Allow to create file with compression DoM component.

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I8b5c4d5fd18033e796b1c7555c37251e4f7f9a49
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51968
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17031 build: fix refefine __compiletime_strlen error

Lustre build failed on Ubuntu 2204 kernel v5.17 with "redefine
__compiletime_strlen".
This patch fixes this build error.

Lustre-change: https://review.whamcloud.com/51953
Lustre-commit: TBD (from aa50b0579aec9039e1ceab0bfdaac45e95df747d)

Fixes: 919b93b951 ("LU-16510 build: fortified memcpy from linux 6.1")
Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: Ic26daecd6b91614e01b5b0030f40eede205a21f7
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52001
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17036 utils: make sure resize option is legit

To align the metadata on 1MB boundaries we manually
set the resize blocks to 16368G for 4K block size,
however mke2fs expects the resize blocks is bigger
than device size.

For devices between 16368G and 16384G the mke2fs
will fail with:
The resize maximum must be greater than the filesystem size.

Lustre-change: https://review.whamcloud.com/51970
Lustre-commit: TBD (from adcaa9173e1d09573e3a4cbfe1b3a2494ee13ba3)

Change-Id: I4567a79c1405e9527d7f0f9bec4c8a7aae0eba6c
Test-Parameters: trivial
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51971
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7451 lipe: Add memory and access log reader stats in lamigo

- Name lamigo threads
- Add memory allocation and deallocation stats in lamigo_alr
- Add stats on total received ALR's, max elements to heat sort,
  current number of allocated heat records
- Add stats on heat thread cycle elapsed time
- Add stats on receiving access log records
  by heat thread
- Add stats on ALR and changelog file cache usage
- Increase number of hash buckets to accomodate
  a million files in the cache by default. This speeds up
  processing input ALR's and prevents excessive memory usage
  for input ALR's
- Modified prints in lamigo_alr_dump_heat_table()
- Add timestamp when stats is dumped on USR1 signal
- Add stats on individual access log readers when
  used USR1 signal stats dump
- Add stats prints in periodical INFO message

Test-Parameters: trivial testlist=hot-pools
Signed-off-by: Alexandre Ioffe <aioffe@ddn.com>
Change-Id: I57215f988278bfe42ddd5f1fd7607c3dbb3fdcdb
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50698
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15460 test: wait quota pool to be prepared

When one OST pool was created, the corresponding quota pool
could need more time to prepare after it was created, then
fail to check the info of the quota pool

Lustre-commit: 83dd308db59a9b95c1db5efa12ea5af22e9c8346
Lustre-change: https://review.whamcloud.com/46853

Test-Parameters: trivial
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Change-Id: Ibea33403639087f27e438d71c0e87fea5367bc3e
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51947
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15791 tests: Drop local traffic during health test

Existing drop rules for health tests omit local nids for the
destination so it is possible for local NI health values to recover
while the tests execute. Add drop rules for local NIDs to prevent
their health from recovering.

Lustre-change: https://review.whamcloud.com/48661
Lustre-commit: 43344697dccdcdb0f4b8dba4899be9571d640131

Test-Parameters: trivial
Test-Parameters: testlist=sanity-lnet env=ONLY=205,ONLY_REPEAT=100
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I6a4a06b3fa76effd21e21449abf47cd0e14bbf18
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51916
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17010 lfsck: don't create trans in dryrun mode

In LFSCK, the LFSCK transaction should not be created in
dryrun mode, which is related to the following patch,

Lustre-change: https://review.whamcloud.com/51849
Lustre-commit: 441902fa3d445791a8c54026c130ab357f7469d7

Fixes: 0c1ae1cb9c19 ("LU-13124 scrub: check for multiple linked file")
Change-Id: Id543bc3c0e300c1cc14b670d724ebcacac3bf71b
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51975
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-3598 lamigo: dump lfs's stdout/stderr

in case of error

Lustre-change: https://review.whamcloud.com/44505
Lustre-commit: f8d365d8d7eaa24715fe4f68687051a6addc94bd

Test-Parameters: trivial testlist=hot-pools
Change-Id: Iea0c6fc097255f02df3698c1e3a31b39bfa09ca1
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51711
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn97

New tag 2.14.0-ddn97

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I3f491760c578bc075694134fba903238a56b1e88

RM-620 build: New tag lipe-2.27

New tag lipe-2.27

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I82a09c9359575ec68f1422b2fb2272c84414ef03

EX-8041 utils: lpurge to use 2nd-stripe objects

in some cases lpurge must be considering objects which are not
first stripe (another lpurge can be waiting for thresholds, etc).

Test-Parameters: mdscount=2 mdtcount=4 trivial testlist=hot-pools

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Iccc0b6af6f283287b41d6f16c875c56f83413e2e
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51884
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17018 build: add 'linux-image-generic' as Depends

Add 'linux-image-generic >= 3.10' as a dependency for Debian dkms
package for Ubuntu support

Lustre-change: https://review.whamcloud.com/51879
Lustre-change: TBD (from 0f43f2a808a33df44a11bae9ff9518167cf60544)

Test-Parameters: trivial
Fixes: 621e0bc2f9 ("LU-16661 build: improve lustre.spec.in Requires")
Signed-off-by: Raphael Druon <rdruon@ddn.com>
Change-Id: Ie8bacbd55c379632d5554de8d72606c818c1771e
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51882
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16831 tests: add version check to sanity-pfl/0e

This patch adds MDS version check to sanity-pfl test 0e
to avoid interop test failure.

Lustre-change: https://review.whamcloud.com/51930
Lustre-commit: TBD (from 5c10bd215ac0c0ce8351795215067071387bfc5b)

Test-Parameters: trivial serverdistro=el7.9 \
serverjob=lustre-b_es5_2 serverbuildno=583 \
env=ONLY=0e testlist=sanity-pfl

Test-Parameters: trivial env=ONLY=0e testlist=sanity-pfl

Fixes: a337688d6e89 ("LU-16831 lfs: limit stripe count for component size")
Change-Id: I79df1f36f07f6b376525364708eacc687f85a061
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51931
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-11036 test: race in sanity-lfsck test_8

In sanity-lfsck test_8, "sleep 1" is run after START_NAMESPACE,
but it still has chance that LFSCK status is complete but LFSCK
thread not quit yet, therefore the following START_NAMESPACE may fail
with -EALREADY. Just check the first lfsck started scanning.

And similarly use wait_update to check flags for DELAY3.

Lustre-change: https://review.whamcloud.com/51720
Lustre-commit: f1ddb4093ed623c1382f75e47807e4081962cc3d

Test-Parameters: trivial MDSCOUNT=2 MDTCOUNT=4 testlist=sanity-lfsck env=ONLY=8,ONLY_REPEAT=10
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Ie1f612bebb52c4755e5b4e13d58ab8bf2aeb2832
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51927
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17009 tests: fix runtests to read file name with backslash

If a file in /etc dir has a name with backslash, then runtests
will fail because the read command considers the backslash as
an escape character. This patch fixes the issue by adding "-r"
option to read.

Lustre-change: https://review.whamcloud.com/51847
Lustre-commit: TBD (from 14044bdfe5fa600bd2c2339032ae8bb54e24b5fb)

Test-Parameters: trivial testlist=runtests

Change-Id: Iab912ba9708f5b64e6bb8d8adc266ff23ed32de5
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51869
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sarah Liu <sarah@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-8059 tests: hot-pools to work in a single node

hot-pools.sh doesn't start on a single node setup because it tries
to find lpurge on $(all_osts_nodes) is empty. make single node
setup an exclusion.

Test-Parameters: trivial
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I6520ff7bef92d62fd39f4c9347462d046844ecea
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51925
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16974 utils: lfs mirror resync to show progress

lfs mirror resync should be able to:
 - show progress like lfs mirror extend --stats does
 - throttle like lfs mirror extend -W does

use 64MB buffer for mirror resync by default.

Lustre-change: https://review.whamcloud.com/51750
Lustre-commit: TBD (from d03e56590243c2d854ce087e2fcb118ccb126d93)

Change-Id: Ibe60748542ff4a3731aa6a4a9907be82427a0ae9
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51878
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17021 socklnd: fix late ksnr_max_conns set

ksnr_max_conns was set to the correct value after it was used.

Lustre-change: https://review.whamcloud.com/51890
Lustre-commit: TBD (from cc894288304fb8e6caa44543e6b44b8ec18deb9b)

Test-Parameters: trivial
Signed-off-by: Cyril Bordage <cbordage@whamcloud.com>
Change-Id: I9f2454d915ee1ab27db96f5247028db94965a11f
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51891
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn96

New tag 2.14.0-ddn96

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I5f7b1fb93402c433e1e3b85b3ef656dd12650972

RM-620 build: New tag lipe-2.26

New tag lipe-2.26

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I6c580359ab0a8051135cc25a7b9eb9179487b540

RM-620 build: New tag 2.14.0-ddn95

New tag 2.14.0-ddn95

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ia76cc084f09794ceed012ca9c512efa7a2a904b8

RM-620 build: New tag lipe-2.25

New tag lipe-2.25

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I0beb892085dff9b28495483df8b5a877ac078ea8

EX-8016 lipe: Identify lfs command issued by lamigo

Each shell command issued by lamigo on OSS is identied
by attached comment. For example,
"lfs mirror extend -N ...  > /dev/null 2>&1 # lamigo testfs-MDT0000"
This can help identify the source of the command and track down
a reason why we have too many command running concurrently.

Test-Parameters: trivial testlist=hot-pools
Signed-off-by: Alexandre Ioffe <aioffe@ddn.com>
Change-Id: I93ed62b030d9c1048c9c3fa7146077d54d2bde37
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51826
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-17002 kernel: update RHEL 7.9 [3.10.0-1160.95.1.el7]

Update RHEL 7.9 kernel to 3.10.0-1160.95.1.el7.

Lustre-change: https://review.whamcloud.com/51801
Lustre-commit: TBD (from b8ff9e0ee7a0d5a9c0fed584eb275ae5ce124a60)

Test-Parameters: trivial clientdistro=el7.9 serverdistro=el7.9

Change-Id: I759d686660a99909b5d6536ac8f3500ef2145fe5
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51804
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16985 utils: adaptive bufsize for mirroring

if bandwidth limit is requested, then change default bufsize to
make I/O rather smooth than like a saw.

Lustre-change: https://review.whamcloud.com/51773
Lustre-commit: TBD (from 390235cc52bd51dc807492db19d8a1f905dbd9d0)

Test-Parameters: trivial
Fixes: 23224e03dc ("LU-16587 utils: give lfs migrate a larger buffer")
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Ibc8e7d30ded201a4ff3d699530f5c9f8be5ce7f1
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51839
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16979 utils: enable throttling mirror extend

this can be useful in some scenarios like massive mirror
creation.

Lustre-change: https://review.whamcloud.com/51758
Lustre-commit: 6f6804aca91d6a2a9e764a5b929feb85db138d88

Test-Parameters: trivial
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Ia84054f3519cd5cef37aaabb2ae605fb6ea200e0
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51830
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16790 man: fix lctl-nodemap-modify man page

The section about the rbac property is malformed, regarding line feeds
in particular.
The forbid_encryption property is not mentioned.
Default values for admin, trusted, deny_unknown, audit_mode and
readonly_mount properties are not expressed correctly. These
properties grok 0 or 1.

Lustre-change: https://review.whamcloud.com/50918
Lustre-commit: 7e252d4c6c7049b16e28359c776d748d74c0ff3c

Fixes: 84e7220830 ("LU-16524 sec: add fscrypt_admin rbac role")
Test-Parameters: trivial
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I712e4d21b3038d87c50bb12206cbcc92ef4b615f
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51831
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-16980 build: fix gcc-12 [-Werror=address] error

This patch fixes the following [-Werror=address] error
detected by gcc 12 and other errors caused by incomplete
porting in commit 1ef38b1258f1 to b_es6_0:

lustre/utils/obd.c: In function 'llog_search_pool_cb':
lustre/utils/obd.c:3816:28: error: the comparison will always evaluate
as 'true' for the address of 'lpd_ostname' will never be NULL
 3816 |       } else if (lpd->lpd_ostname && lpd->lpd_ostname[0]) {
      |                  ^~~
obd.c:3747:14: note: 'lpd_ostname' declared here
 3747 |       char lpd_ostname[MAX_OBD_NAME + 1];
      |            ^~~~~~~~~~~

lustre/tests/kernel/kcompr.c: In function 'fill_input_with_rand':
lustre/tests/kernel/kcompr.c:144:41: error: implicit declaration of
function 'prandom_u32_max'; did you mean 'prandom_u32_state'?
  144 |       p += sizeof(randbuf2) + prandom_u32_max(1 << 13);
      |                               ^~~~~~~~~~~~~~~
      |                               prandom_u32_state

lustre/llite/file.c: In function 'll_set_acl':
lustre/llite/file.c:5900:43: error: 'inode' undeclared
 5900 |       struct ll_sb_info *sbi = ll_i2sbi(inode);
      |                                         ^~~~~

Fixes: 1ef38b1258f1 ("LU-16594 build: get_random_u32_below, get_acl with dentry")
Change-Id: I9206fa169880ae16e192d17c8f144ed3b83c65a3
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51829
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7998 lustre: Fix build with crypto_tfm_ctx on kernel 6.2

In recent kernels, commit e634ac4
"crypto: api - Add crypto_tfm_ctx_dma"
moved crypto_tfm_ctx into algapi.h. So this file must be included
from lustre_crypto.h.

Fixes: 8238c4e7fe ("EX-6275 lustre: add lz4 and lz4hc kernel modules")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: If10f721355da3e58ca541c17615e978334e8d718
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51812
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Jian Yu <yujian@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-12477 libcfs: Further reduce  complexity for shrinkers.

Commit c4c17fa4a3f5 ("LU-12477 libcfs: Remove obsolete config checks")
reduced the complexity of shinkers by removing support for older
kernels, but could have gone a lot further.  This patch adds
further simplification.

Lustre-change: https://review.whamcloud.com/40831
Lustre-commit: 782725b11021b2f1c3467863216836ee94bcd874

LU-12477 lustre: check return status of register_shrinker()

register_shrinker() can fail with -ENOMEM.  We should check for that
and abort the relevant initialization functions when it happens.

For ldlm_pools, ldlm_pools_fini() can be called when ldlm_pools_init()
fails, or even in case where it hasn't been called.  So add a static
flag to ensure we ldlm_pools_fini() do not undo things that haven't been
done.

For lu_global_init() we need to add proper cleanup if anything fails.

Lustre-change: https://review.whamcloud.com/40883
Lustre-commit: 812b2ccf0284df42e8a88a6b7c4c3874dd721c71

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Signed-off-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Change-Id: Ibcc84f61e542b503f795b16a7144e430f8b73582
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51778
Tested-by: Jian Yu <yujian@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16980 build: fix gcc-12 [-Werror=format-truncation=] error

This patch fixes the following [-Werror=format-truncation=] errors
detected by gcc 12:

liblnetconfig.c: In function 'open_sysfs_file':
liblnetconfig.c:106:49: error: '%s' directive output may be truncated
writing up to 127 bytes into a region of size between 1 and 128
[-Werror=format-truncation=]
  106 |         snprintf(filename, sizeof(filename), "%s%s",
      |                                                 ^~

lfs_project.c: In function 'lfs_project_handle_dir':
lfs_project.c:324:50: error: '%s' directive output may be truncated
writing up to 255 bytes into a region of size between 1 and 4095
[-Werror=format-truncation=]
  324 |                 snprintf(fullname, PATH_MAX, "%s/%s", pathname,
      |                                                  ^~

statx.c: In function 'do_dir_list':
statx.c:1427:58: error: '%s' directive output may be truncated
writing up to 255 bytes into a region of size between 1 and 4095
[-Werror=format-truncation=]
 1427 |                         snprintf(fullname, PATH_MAX, "%s/%s",
      |                                                          ^~

Lustre-change: https://review.whamcloud.com/51765
Lustre-commit: TBD (from c976fe2f854e621dc16810ff8d0afafe4278bd14)

Change-Id: I514a1022d879f8b7af89f6ded68e9b453cd11408
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51774
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16980 build: fix gcc-12 [-Werror=use-after-free] error

This patch fixes the following [-Werror=use-after-free] and
[-Werror=stringop-overflow=] errors detected by gcc 12:

libcfs/include/libcfs/util/list.h:481:42: error: pointer 'tmp'
used after 'free' [-Werror=use-after-free]
  481 |              pos = list_entry(pos->member.next, typeof(*pos), member),  \
      |                                          ^
libcfs/include/libcfs/util/list.h:239:28: note: in definition of macro 'list_entry'
  239 |         ((type *)((char *)(ptr)-(char *)(&((type *)0)->member)))
      |                            ^~~
obd.c:5118:9: note: in expansion of macro 'list_for_each_entry'
 5118 |         list_for_each_entry(tmp, head, lpn_list) {
      |         ^~~~~~~~~~~~~~~~~~~
obd.c:5124:17: note: call to 'free' here
 5124 |                 free(tmp);
      |                 ^~~~~~~~~

test_brw.c: In function 'main':
test_brw.c:227:22: error: 'write' specified size between 9223372036854775808
and 18446744073709551615 exceeds maximum object size 9223372036854775807
[-Werror=stringop-overflow=]
  227 |                 rc = write(fd, buf, len);
      |                      ^~~~~~~~~~~~~~~~~~~

Lustre-change: https://review.whamcloud.com/51819
Lustre-commit: TBD (from 8fa48300347ac3c7dffb361d1686c434127fe3e1)

Change-Id: Ibe783ab0d13e2ecde1736946323932ab5db53740
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51825
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7989 utils: lpurge to dump empty slots

empty time slots may still have detailed stats explaining
slot's emptiness: how many objects were scanned, how many
had no pfid, no mirror, etc.

Test-Parameters: trivial testlist=hot-pools
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I7e8f594d8615a5e964c01ec1baf2a2b5d3e9cce3
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51796
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Colin Faber <cfaber@ddn.com>
Reviewed-by: Alexandre Ioffe <aioffe@ddn.com>

LU-17005 obdclass: allow stats header to be disabled

Add a global "enable_stats_header" tunable parameter that can be
set to enable/disable the "start_time" and "elapsed_time" fields
in the standard lprocfs "stats" files.

Default to disabled for EXA6.2.1, since this is new for this release.

Lustre-change: https://review.whamcloud.com/51823
Lustre-commit: TBD (from 370a81bf07f3e3856f73900f276495fb7a2b5223)

Test-Parameters: trivial
Fixes: 5efb892396e3 ("LU-11407 obdclass: add start time to stats files")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I460b957447bfb83e6d4fd7395b79ce994f3ebbe5
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51824
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Feng Lei <flei@whamcloud.com>

LU-16667 build: struct mnt_idmap, linux/filelock.h

Linux commit v6.2-rc3-9-g5970e15dbcfe
  filelock: move file locking definitions to separate
            header file

Add configure test for linux/filelock.h and include it
where needed.

linux kernel v6.2-rc1-4-gb74d24f7a74f
  fs: port ->getattr() to pass mnt_idmap
linux kernel v6.2-rc1-3-gc1632a0f1120
  fs: port ->setattr() to pass mnt_idmap

Add a configure test for mnt_idmap and fallback to using
user_namespace for older kernels.

Lustre-change: https://review.whamcloud.com/50420
Lustre-commit: 3011aa564a8c682aafbc6071b9866e266d8a6232

Test-Parameters: trivial
HPE-bug-id: LUS-11557
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: Ib8cbb3157fb11b4f1fc55f1626c2998cb202bd8c
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51756
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16637 revert: "llite: call truncate_inode_pages() in inode lock"

This reverts commit 1a084b1378ac37fb4269ac2e6dcac5c172fa00d2 due to
lock ordering issues between migrate vs. write reported in LU-16958.

Change-Id: I7966f19a5d7aabbf578a59c55459ddee332a6601
Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51772
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-7968 utils: lamigo to accept multi-word args

so we can put something like the following into the config file:
mirror-cmd=lfs mirror extend -N --stats --stats-interval=1 -W1M

Test-Parameters: trivial testlist=hot-pools

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Ib01c4b4c76a46ee44189cc6fa763306b059886b3
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51768
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16843 ldiskfs: merge extent blocks

There are cases (e.g. file written synchronously with discontiguous
blocks that are later filled in) when a lot of extents are created
initially, then the extents get merged over time, but there is no
way to merge the index blocks.  This can cause a very deep extent
index tree (above 5 levels) and cause problems like:

  inode has invalid extent depth: 6

Merge leave/index blocks (one at each level at most) to right/left
when extents are removed from the index.

submitted to ext4@ maillist:
https://lore.kernel.org/linux-ext4/7A2B8861-96AA-4815-BB58-180F63F62436@whamcloud.com/

Lustre-change: https://review.whamcloud.com/51096
Lustre-commit: 0f7e6c02a9ea5b85d8d97f724bed318268cea60f

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I746c0917e746eb442d3c69a23f591d9cdade76fa
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51272
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16691 ldiskfs: limit length of per-inode prealloc list

In the scenario of writing sparse files, the per-inode prealloc list may
be very long, resulting in high overhead for ext4_mb_use_preallocated().
To circumvent this problem, we limit the maximum length of per-inode
prealloc list to 512 and allow users to modify it.

After patching, we observed that the sys ratio of cpu has dropped, and
the system throughput has increased significantly. We created a process
to write the sparse file, and the running time of the process on the
fixed kernel was significantly reduced, as follows:

Running time on unfixed kernel：
    # time taskset 0x01 ./sparse /data1/sparce.dat
    real    0m2.051s
    user    0m0.008s
    sys     0m2.026s

Running time on fixed kernel：
    # time taskset 0x01 ./sparse /data1/sparce.dat
    real    0m0.471s
    user    0m0.004s
    sys     0m0.395s

Link: https://lore.kernel.org/r/d7a98178-056b-6db5-6bce-4ead23f4a257@gmail.com
Lustre-change: https://review.whamcloud.com/50481
Lustre-commit: b16c9333a00802faea419dfe6fbb013c4477c9c6

Signed-off-by: Chunguang Xu <brookxu@tencent.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I5e4ea3acfc07f6e69890690211bf6a34c1230979
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sergey Cheremencev <scherementsev@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51712
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16661 build: improve lustre.spec.in Requires

Add Suggests: bash-completion for lustre-client and lustre for
lctl and lfs sub-command completion.

Move perl from Requires to Recommends, since there are only some
uncommonly used tools (llstat, llobdstat) that are using perl.
Remove a couple of ancient obsolete test scripts that used perl.

lustre-iokit incorrectly Required perl instead of python3.

Set minimum kernel version for client to be 3.10 or later.

Change "netstat" to "ss" in tests to avoid dependency issues.
Fix sanity.sh and conf-sanity.sh tests for sles12sp5 issues.

Lustre-change: https://review.whamcloud.com/50397
Lustre-commit: 621e0bc2f9d4e7d9aed0ec14e939eae4451a8f1e

Test-Parameters: trivial testlist=runtests clientdistro=el9.1
Test-Parameters: trivial testlist=runtests clientdistro=sles15sp3
Fixes: 7521473bdd ("LU-16382 spec: add more dependencies for lustre-tests")
Fixes: fd734cffb3 ("b=18443 tests: remove obsolete tests scripts")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I48c6819596c81cb044e983bc64f1edf1ee3ebbe5
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: xinliang <xinliang.liu@linaro.org>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51751
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16661 build: add Recommends and Suggests for Debian

Add Suggests: bash-completion for lustre-client-tools and
lustre-server-tools for lctl and lfs completion.

Move perl from Depends to Recommends, since there are only some
uncommonly used tools (llstat, llobdstat) that are using perl.

Add python3 to Recommends for lustre-server-tools for lljobstat.
Remove python3 from lustre-iokit since it isn't used anywhere.

Change Maintainer for Debian packages to the lustre-devel mailing
list, instead of someone who hasn't worked on Lustre for 6 years.

Increase minimum kernel version for client from 2.6.32 to 3.10.

Improve package descriptions slightly.

Lustre-change: https://review.whamcloud.com/50396
Lustre-commit: 73ed44854a740b78523ed1f050960038c8adf552

Test-Parameters: trivial testlist=runtests clientdistro=ubuntu2204
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I43248cc78ae6a47ad77817c27ba11de25b3ebbe5
Reviewed-by: Thomas Stibor <thomas@stibor.net>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-by: Timothy Day <timday@amazon.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51749
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16984 tests: replay-dual/31 checks file from DIR2

Move replay-dual/test_31 to correct place.
In replay-dual/test_31, check file existence from DIR2.
Add more messages for diagnosis.

Lustre-change: https://review.whamcloud.com/51762
Lustre-commit: TBD (from 025aa6e78ca6d3f4fc872b2b616034ec5b28952a)

Fixes: 07764c4eeb ("LU-16953 tests: wait longer in replay-dual/test_31")
Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial testlist=replay-dual env=ONLY=31,ONLY_REPEAT=100
Change-Id: Iee679ee94ac2cb51baad1651bfaddf452fafdbd1
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51764
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16290 lnet: Signal completion on ping send failure

Call complete() on the ping_data::completion if we get
LNET_EVENT_SEND with non-zero status. Otherwise the thread which
issued the ping is stuck waiting for the full ping timeout.

A pd_unlinked member is added to struct ping_data to indicate whether
the associated MD has been unlinked. This is checked by lnet_ping() to
determine whether it needs to explicitly called LNetMDUnlink().

Lastly, in cases where we do not receive a reply, we now return the
value of pd.rc, if it is non-zero, rather than -EIO. This can provide
more information about the underlying ping failure.

Lustre-change: https://review.whamcloud.com/49020
Lustre-commit: 48c34c71de65e8a251a218bc9ecb7c5ed522d786

HPE-bug-id: LUS-11317
Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I1bc573cf7397e319993fa8aabb31c5f3b59768e7
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51700
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>

LU-16551 tests: Ensure all peer credits used in MR

sanity-lnet test_254 needs to ensure that all peer credits are
consumed. Because of the raciness of the round robin code in LNet,
we cannot rely on just issuing the appropriate number of pings.
Instead we should use the --source argument to lnetctl ping to ensure
that we send the correct number of pings over each interface.

To simplify matters, only perform this test, and the other tests that
call setup_health_test(), in non-routed configurations.

Lustre-change: https://review.whamcloud.com/49979
Lustre-commit: aa66cb46761d72ec2c861ff9665ea6b6dbbe76a8

Test-Parameters: trivial testlist=sanity-lnet
Fixes: 52db11cdce ("LU-16303 lnet: Drop LNet message if deadline exceeded")
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I05a7ffec37d16c14711fe696232708f927357b1c
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51699
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>

LU-16303 lnet: Drop LNet message if deadline exceeded

The LNet message deadline is set when a message is committed for
sending. A message can be queued while waiting for send credit(s)
after it has been committed. Thus, it is possible for a message
deadline to be exceeded while on the queue. We should check for this
when posting messages to LND layer.

Lustre-change: https://review.whamcloud.com/49078
Lustre-commit: 52db11cdceef0851b972410cf8f7014d20fd194e

HPE-bug-id: LUS-11333
Test-Parameters: trivial testlist=sanity-lnet env=ONLY=253,ONLY_REPEAT=100
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I1315b2351536e63b9d4f22d9336a57415031e0c7
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51698
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15595 tests: Router test interop check and aarch fix

setup_router_test() executes load_lnet() on remote nodes, but
this function was only added in 2.15. Add a version check for it.

Enabling routing may fail on nodes with small amount of memory (like
aarch config). Define small number of router buffers to work around
this issue. Modify the functions which calculate the number of buffers
to allow small sizes to be specified via parameters.

Lustre-change: https://review.whamcloud.com/48578
Lustre-commit: 1aba6b0d9b661d3699cbd4624e9db334a13fc647

Test-Parameters: trivial testlist=sanity-lnet serverversion=2.12.9
Test-Parameters: testgroup=review-ldiskfs-arm testlist=sanity-lnet
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: If0b76747fe09e883546f18da9f3322c72263e29d
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51664
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>

LU-15595 lnet: Always use ping reply to set route lr_alive

We currently process discovery ping replies in different ways
depending on whether the gateway has discovery enabled or disabled
(or the local peer doing the processing has discovery enabled or
disabled).

When DD is disabled we process the ping reply to set the lr_alive
field of lnet_route because the peer objects for non-MR routers do
not contain all the information needed to calculate the route
aliveness when a message is being sent.

When DD is enabled then we don't do any special processing of the
ping reply. We simply let discovery update the NI status for the
GW's peer NIs and then we calculate the route aliveness on every
send.

We issue discovery pings to routers every alive_router_check_interval
seconds (default 60), but we calculate route aliveness on every send
to a remote network (1000s of times per seconds). Thus, it is better
to slightly duplicate the effort expended when we receive a discovery
reply so that we can avoid calculating route aliveness on every send.

Since both lr_alive and hop type are being set on each ping reply, for
both DD enabled and disabled cases, we can remove the code for
updating lr_alive and hop type from lnet_router_discovery_complete().

If discover encounters a fatal error, we still set the status of each
peer NI, as well as all routes, to down in
lnet_router_discovery_complete().

Lustre-change: https://review.whamcloud.com/46624
Lustre-commit: 1ea6c87d415144522a1c50c08063c9d30d585325

Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: If4838c269a89885ba3763f62847e294804edf62e
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51663
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15595 tests: Add various router tests

Add test cases to exercise LNet routing.

Lustre-change: https://review.whamcloud.com/46622
Lustre-commit: 8ee85e15412d32fbe60f70c474c0a28ff15b8351

Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I4a077937b3e3b8b07707afeb0c5c23ec1c9074f4
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51661
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15363 tests: don't use lustre module to test lnet

Currently sanity-lnet.sh loads the lustre modules to properly
initialize the lnet modules. This doesn't work with the native
Linux client since it only starts up LNet after mounting the
file system. We shouldn't be using lustre to test lnet so
load lnet modules with config_on_load option to properly setup
the default LNet configuration.

Also fix ksocklnd-config to use bash so sanity-lnet.sh can
pass on Ubuntu.

Lustre-change: https://review.whamcloud.com/45834
Lustre-commit: e41f91dc90a0977f7ea85b199b7e5809c56b810e

Test-Parameters: trivial testlist=sanity-lnet
Change-Id: Ifffc51625f5c2ffbb3ab811b75739c0e6407a821
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51666
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15440 lnet: lnet_peer_data_present() memory leak

If the ping buffer has nnis <= 1 then the ref on the ping buffer does
not get dropped. This causes a memory leak.

Lustre-change: https://review.whamcloud.com/46052
Lustre-commit: 56384a4fc39ff99c8abb3538f93d303f2be6ab45

Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I5e3c651ffecbe4f8860afb86770cecef23ebe862
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51684
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16911 sec: quiet messages from identity upcall retry mech

Do not use CERROR to print messages about failed identity acquire
upcalls. And make a difference between initial attempt before retry,
and final failure.

Lustre-change: https://review.whamcloud.com/51355
Lustre-commit: 765e59d37dc2b4c85e9bdaea2765c9023226778b

Fixes: 61c3b3a9bb ("LU-16165 sec: retry mechanism for identity cache")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Signed-off-by: Xing Huang <hxing@ddn.com>
Change-Id: I35e04ca31b623d6037bb49e4ded4ea96d653f074
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51797
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-7965 utils: resync-cmd option to lamigo

so we can change default "lfs mirror resync" to something else,
add extra options, etc.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Idf445128e811297651e09babc51fb5b0f2b9b20d
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51757
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16831 lfs: limit stripe count for component size

If stripe count is larger than component_size/stripe_size, some
allocated OST objects are created but inaccessible. This patch
reduces the number of stripes in that case to avoid this.

Lustre-change: https://review.whamcloud.com/51143
Lustre-commit: a250ecb959a98c2ec0a01bbca9d943a19b8fa078

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I117ed8a7696c6c6adcdd0c2c6531a958cc53bd51
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51719
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Zhenyu Xu <bobijam@hotmail.com>

LU-17001 pcc: fix the build failures for mmap_conv

PCC has implemented mmap page conversion for performance reason.
However, it is not stable, and disabled by default.

The Lustre build meets some failures against the newer kernel such
as Ubuntu 2204 kernel 6.2.0-26-generic.
The reason is that @pagevec_lookup and @add_to_page_cache_locked
are removed and no longer exported by the kernel.

This patch gives a simple solution. It disables the mmap_conv
option for PCC and does not perform mmap page conversion for
the newer kernel without @add_to_page_cache_locked exported.

Change-Id: Ie314830f225848b7f888709ad000fd7d36d3c0b2
Signed-off-by: Qian Yingjin <qian@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51798
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16983 mdc: check errcode prior mdc_fill_lvb() call

The mdc_enqueue_fini() can be called with negative
errcode parameter if request processing was failed.
In that case the mdc_fill_lvb() shouldn't be called.

Issue may occur with DoM files, old server (<2.14) and
new client. The problem is in new client code.

Lustre-change: https://review.whamcloud.com/51761
Lustre-commit: TBD (from c422797b3949e0b3b7fac7164f03f01dfa6aca0b)

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I884398beada4286bc07875247e15b41120f73a3e
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51769
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16667 build: kernel_cap_t contains u64

linux kernel v6.2-13111-gf122a08b197d
  capability: just use a 'u64' instead of a 'u32[2]' array

Add configure test for kernel_cap_t as u64 and provide
and accessor for the least significant 32 bits.

As of linux commit v3.6-10973-g607ca46e97a1 lustre implicitly
started to ignore some capabilities, see:
   include/uapi/linux/capability.h

The last capability flag was added by:
   linux commit v5.8-rc5-1-g124ea650d307

The capabilities the Lustre currently ignores are:
 - CAP_MAC_OVERRIDE
 - CAP_MAC_ADMIN
 - CAP_SYSLOG
 - CAP_WAKE_ALARM
 - CAP_BLOCK_SUSPEND
 - CAP_AUDIT_READ
 - CAP_PERFMON
 - CAP_BPF
 - CAP_CHECKPOINT_RESTORE

None of which appear to be important to Lustre operations
and should be fine to continue ignore.

Lustre-change: https://review.whamcloud.com/50421
Lustre-commit: ea9532fb731bbfe041010e2224219479c2c0d71b

Test-Parameters: trivial
HPE-bug-id: LUS-11557
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I48ad7b1a34fff378c260dc73ea91b22aaa0d7469
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51726
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-9859 libcfs: discard cfs_cap_t, use kernel_cap_t

lustre only sends 32bits of capabilities in on-the-wire RPC calls.
It current strips off higher bits and uses a 32bit cfs_cap_t
throughout.
Though there is a small memory cost, it is cleaner to use
kernel_cap_t throughout and only truncate when marshalling
data for RPC calls.

So this patch replaces cfs_cap_t with kernel_cap_t throughout,
and where a cfs_cap_t was previous stored in a __u32, we now
store cap.cap[0] instead.

With this, we can remove include/linux/libcfs/curproc.h

Linux-commit: 18f92a6e3d6bd00941ddfb5837835348f72d39dc

Lustre-change: https://review.whamcloud.com/43171
Lustre-commit: 0a6beb2a50f7a4f953a258b63c2ea1fbfd373092

Change-Id: If7dd7a16c218dfc0d520e189f021ed6bda3b93fd
Signed-off-by: Mr. NeilBrown <neilb@suse.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51743
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-9859 libcfs: simplify capability dropping.

Lustre has a 'squash credentials' concept similar to the "anon_uid"
for nfsd.  When accessing a file with squashed credentials, we
need to also drop capabilities.
Linux has cap_drop_fs_set() and cap_drop_nfsd_set().  Rather than
taking a completely different approach, this patch changes lustre
to use this same cap_drop_*_set() approach.

With this change we also drop CAP_MKNOD and CAP_MAC_OVERRIDE
which are probably appropriate, and don't drop
CAP_SYS_ADMIN or CAP_SYS_BOOT which should be irrelevant for
file permission checking

Calling both cap_drop_*_set() seems a bit clumsy, but gets
the job done.

Linux-commit: f497115d4cf8a430c5d9902ce35716ba5f9c21ef

Lustre-change: https://review.whamcloud.com/41957
Lustre-commit: ee06281d89871c46310ee4b0585ee4489d2c2f3f

Change-Id: I2f4f691bc4ad090f6abaa4e13eb473bf8d904b23
Signed-off-by: NeilBrown <neilb@suse.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51745
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-9859 libcfs: remove linux-curproc.c

The only real functionality remaining here is
cfs_curproc_cap_pack(),
and it can be trivially implemented as an inline
in curproc.h.
So do that and remove the file.

The rest can be moved to jobid.c

Linux-commit: 37d3b407dc14a13ec8bba3a4d7737c92f996e9c0

Lustre-change: https://review.whamcloud.com/41938
Lustre-commit: f6cd6a880cd03fd1d3fd3d8e03133f0835ad275d

Change-Id: I3546841fa44accb19d0867099c17b16ede48228e
Signed-off-by: Mr. NeilBrown <neilb@suse.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51742
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-9859 libcfs: remove cfs_capable

Use capable() directly.

Linux-commit: 2eb90a757e9d953c9e2a8fce530422189992fb1b

Lustre-change: https://review.whamcloud.com/41783
Lustre-commit: e34fdc5479037bf7b6efed3239362585a1c1ff3b

Change-Id: Iadaa3c743a350def37558b23d954f5dfd4e0844a
Signed-off-by: Peng Tao <bergwolf@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51741
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-9859 libcfs: replace all CFS_CAP_* macros with CAP_*

Lustre defines a few CFS_CAP_* macros which are exactly the
same as the corresponding CAP_* macro, with one exception.

CFS_CAP_SYS_BOOT is 23
CAP_SYS_BOOT is 22.

CFS_CAP_SYS_BOOT is only used through CFS_CAP_FS_MASK and
causes capability 23 (CAP_SYS_NICE) to be dropped in certain
circumstances.
It is probable that the intention was to drop CAP_SYS_BOOT,
and this is what is now done.

CFS_CAP_CHOWN_MASK and CFS_CAP_SYS_RESOURCE_MASK are never
used, so they have been removed.

Linux-commit: 5ebaa2d14850205e44757c4d5fdd4097712d01ef

Lustre-change: https://review.whamcloud.com/39875
Lustre-commit: a076975f9fed44e2b3a8b516aa7fe2ee6fbdb2bb

Change-Id: Ifb90c0a36e204c76b90ff23ac609345d11b878da
Signed-off-by: Mr. NeilBrown <neilb@suse.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51740
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn94

New tag 2.14.0-ddn94

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ie82ae2d2198bdfd780acf1c2050c64287bc0b7ec

LU-16637 llite: call truncate_inode_pages() in inode lock

In some cases vvp_prune()->truncate_inode_pages() is get called
without IO context, we need protect it with inode lock as well.

So we add ll_inode_info::lli_inode_lock_owner and set it according to
vfs lock rules (Documentation/filesystems/Locking or
Documentation/filesystems/locking.rst), so before calling
truncate_inode_pages(), we'd lock the inode if it's not locked in
vfs.

Lustre-commit: 51d62f2122fee14fbb3ff8333b5a830e1181e4e5
Lustre-change: https://review.whamcloud.com/50857

Fixes: ef9be34478 ("LU-16637 llite: call truncate_inode_pages() under inode lock")
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I84d7d999a49325810062a9a7337e184d35467820
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51643
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16594 build: get_random_u32_below, get_acl with dentry

Linux commit v6.1-13825-g3c202d14a9d7
  prandom: remove prandom_u32_max()

Use get_random_u32_below() and provide a replacement
when get_random_u32_below is not available.

Linux commit v6.1-rc1-2-g138060ba92b3
  fs: pass dentry to set acl method
Linux commit v6.1-rc1-4-g7420332a6ff4
  fs: add new get acl method

get_acl() and set_acl() have new signatures

Lustre-change: https://review.whamcloud.com/50193
Lustre-commit: 3ef773db80fc346455c9939ad108f3f56990ee9c

Test-Parameters: trivial
HPE-bug-id: LUS-11556
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I1de02f86fd2719fc75de4f014f51d73736d83c33
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51734
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>