LU-17136 ldiskfs: increase max extent tree depth

increase max extent tree depth to 8.
this is an workaround until LU-16843 ready

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Ie1b6bd64ff6d5179b47b6a537c6b9f85670c3f69
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52758
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17193 build: fix gcc-12 compiler warnings

A few instances of QCTL_COPY() should be QCTL_COPY_NO_PNAME()
as the zero-length array to hold the pool name is not
allocated in these cases.

Lustre-change: https://review.whamcloud.com/c/fs/lustre-release/+/52687
Lustre-commit: 1b0de05f81372eeda9a2a38142553ead7e88a431

Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I72bda8b46c51dbd42fb42bf569ba29572526acfe
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52834
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17151 tests: increase sanity/411b memory limit

This test fails most of the time when run using
arm clients. It seems like the cgroup memory limit
was increased in a past revision for a similar issue.
Increase it a bit more for aarch64. Increase it a
smaller amount for x86.

Also, add some better logging for some other issues.

There's likely a better fix for this, but hopefully
this will let the test pass and provide some value
without having to do a full revert.

Lustre-change: https://review.whamcloud.com/52610
Lustre-commit: 0e878390e1c8c5883bccd01758392eaa16a67f31

Fixes: 8aa231a99 ("LU-16713 llite: writeback/commit pages under memory pressure")
Test-Parameters: trivial
Test-Parameters: testgroup=review-ldiskfs-arm testlist=sanity env=ONLY=411b,ONLY_REPEAT=50
Test-Parameters: clientdistro=el8.7 testlist=sanity env=ONLY=411b,ONLY_REPEAT=50
Test-Parameters: clientdistro=el9.1 testlist=sanity env=ONLY=411b,ONLY_REPEAT=50
Signed-off-by: Timothy Day <timday@amazon.com>
Change-Id: If850077c0d7f6466082433776d370d24eee9736c
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52610
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Qian Yingjin <qian@ddn.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52838

LU-16713 llite: writeback/commit pages under memory pressure

Lustre buffered I/O does not work well with restrictive memcg
control. This may result in OOM when the system is under memroy
pressure.

Lustre has implemented unstable pages support similar to NFS.
But it is disabled by default due to the performance reason.

In Lustre, a client pins the cache pages for writes until the
write transcation is committed on the server (OST) even these
pinned pages have been finished writeback. The server starts
a transaction commit either because the commit interval (5
second, by default) for the backend storage (i.e. OST/ldiskfs)
has been reached or there is not enough room in the journal
for a particular handle to start. Before the write transcation
has been committed and notify the client, these pages are
pinned and not flushable in any way by the kernel.
This means that when a client hits memory pressure there can
be a large number of unfreeable (pinned and uncommitted) pages,
so the application on the client will end up OOM killed because
when asked to free up memory it can not.
This is particularly common with cgroups. Because when cgroups
are in use, the memory limit is generally much lower than the
total system memory limits and it is more likely to reach the
limits.

Linux kernel has matured memory reclaim mechanism to avoid OOM
even with cgroups.
After perform dirtied write for a page, the kernel calls
@balance_dirty_pages(). If the dirtied and uncommitted pages
are over background threshold for the global memory limits or
memory cgroup limits, the writeback threads are woken to perform
some writeout.
When allocate a new page for I/O under memory pressure, the
kernel will try direct reclaim and then allocating. For cgroup,
it will try to reclaim pages from the memory cgroup over soft
limit. The slow page allocation path with direct reclaim will
call @wakeup_flusher_threads() with WB_REASON_VMSCAN to start
writeback dirty pages.

Our solution uses the page reclaim mechanism in the kernel
directly.
In the completion of page writeback (in @brw_interpret), call
@__mark_inode_dirty() to add this dirty inode which has pinned
uncommitted pages into the @bdi_writeback where each memory
cgroup has itw own @bdi_writeback to contorl the writeback for
buffered writes within it.
Thus under memory pressure, the writeback threads will be woken
up, and it will call @ll_writepages() to write out data.
For background writeout (over background dirty threshold) or
writeback with WB_REASON_VMSCAN for direct reclaim, we first
flush dirtied pages to OSTs and then sync them to OSTs and force
to commit these pages to release them quickly.

When a cgroup is under memory pressure, the kernel asks to do
writeback and then it does a fsync to OSTs. This will commit
uncommitted/unstable pages, and then the kernel can free them
finally.

In the following, we will give out some performance results.
The client has 512G memory in total.
1. dd if=/dev/zero of=$test bs=1M count=$size
I/O size 128G 256G 512G 1024G
unpatch (GB/s) 2.2 2.2 2.1 2.0
patched (GB/s) 2.2 2.2 2.1 2.0
There is no preformance regession after enable unstable page
account with the patch.

2. One process under different memcg limits and total I/O
size varies from 2X memlimit to 0.5 memlimit:
dd if=/dev/zero of=$file bs=1M count=$((memlimit_mb * time))
memcg limits 1G 4G 16G 64G
2X memlimit (GB/s) 1.7 1.6 1.8 1.7
1X memlimit (GB/s) 1.9 1.9 2.2 2.2
.5X memlimit(GB/s) 2.3 2.3 2.2 2.3
Without this patch, dd with I/O size > memcg limit will be
OOM-killed.

3. Multiple cgroups Testing:
8 cgroups in total each with memory limit of 8G.
Run dd write on each cgrop with I/O size of 2X memory limit
(16G).
17179869184 bytes (17 GB, 16 GiB) copied, 12.7842 s, 1.3 GB/s
17179869184 bytes (17 GB, 16 GiB) copied, 12.7889 s, 1.3 GB/s
17179869184 bytes (17 GB, 16 GiB) copied, 12.9504 s, 1.3 GB/s
17179869184 bytes (17 GB, 16 GiB) copied, 12.9577 s, 1.3 GB/s
17179869184 bytes (17 GB, 16 GiB) copied, 13.4066 s, 1.3 GB/s
17179869184 bytes (17 GB, 16 GiB) copied, 13.5397 s, 1.3 GB/s
17179869184 bytes (17 GB, 16 GiB) copied, 13.5769 s, 1.3 GB/s
17179869184 bytes (17 GB, 16 GiB) copied, 13.6605 s, 1.3 GB/s

4. Two dd writers one (A) is under memcg control and another
(B) is not. The total write data is 128G. Memcg limits varies
from 1G to 128G.
cmd: ./t2p.sh $memlimit_mb
memlimit dd writer (A) dd writer (B)
1G 1.3GB/s 2.2GB/s
4G 1.3GB/s 2.2GB/s
16G 1.4GB/s 2.2GB/s
32G 1.5GB/s 2.2GB/s
64G 1.8GB/s 2.2GB/s
128G 2.1GB/s 2.1GB/s

The results demonstrates that the process with memcg limits
nearly has no impact on the performance of the process without
limits.

Lustre-change: https://review.whamcloud.com/50544
Lustre-commit: 8aa231a994683a9224d42c0e7ae48aaebe2f583c

Test-Parameters: clientdistro=el8.7 testlist=sanity env=ONLY=411b,ONLY_REPEAT=10
Test-Parameters: clientdistro=el9.1 testlist=sanity env=ONLY=411b,ONLY_REPEAT=10
Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I7b548dcc214995c9f00d57817028ec64fd917eab
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52527
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-8362 scripts: Improve estimated ratio

ll_compression_scan does not take in account the size of the
sampled files, this might lead to uncorrect estimated ratio for non
homogeneous file.

This patch apply the compression ratio estimated with the sampled data
and applies it to the entire file size, assuming the file will have
the same compression ratio across it.

Test-Parameters: trivial
Signed-off-by: Raphael Druon <rdruon@ddn.com>
Change-Id: Ic4a26460e17c666b9edf4c0d8d450a06fad5920f
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52759
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16837 lov: NULL dereference in lov_delete_composite

commit 14ed4a6f8f retroduced the issue fixed by commit
5da049d9ef ("LU-14389 lov: avoid NULL dereference in cleanup), this
patch makes the fix cover the new case added by 14ed4a6f8f.

Lustre-change: https://review.whamcloud.com/52826
Lustre-commit: TBD (from 10b4a14b389cb00e1033e2f49e3d1f5a554b259a)

Fixes: 14ed4a6f8f ("LU-16837 llite: handle unknown layout component")
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I4a2b72e21139b60519ed523b4851723c91f523c1
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52827
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16827 obdfilter: Fix obdfilter-survery/1a

local_node() under test-framework is used
to determine if the node is remote or local
local_node() returns "true" if the node is
local. Else for remote node it return "false"

This patch fixes obdfilter/1a test case which
which was making reverse logic call to
local_node() to determine remote/local node

This patch modifies local_node() to return
"true"/"false" instead of 0/1

This patch also replaces lctl with $LCTL

Lustre-change: https://review.whamcloud.com/51035
Lustre-commit: 91a3b286ba57bb491b5c17600d7cec9e516a428f

Test-Parameters: testlist=obdfilter-survey,sanity-lipe-scan3,sanity-lipe-find3
Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Signed-off-by: Vitaliy Kuznetsov <vkuznetsov@ddn.com>
Change-Id: I7bcb483975ec46d9847e0050e5a1f22f68663c80
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52800
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15461 test: add pool quota check

The test_79 in sanity-quota needs quota pool support.
The removal of the "stop file" is also improved not to
trigger the test error if it has been deleted.

Lustre-change: https://review.whamcloud.com/52737
Lustre-commit: TBD (from a4b3cd91ae157a63644350769ebb248f21dd6eac)

Test-Parameters: trivial testlist=sanity-quota
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Change-Id: I4acd36e61faf4259c2821293ffb7913d4cca76bd
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52659
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-17220 kernel: update RHEL 7.9 [3.10.0-1160.102.1.el7]

Update RHEL 7.9 kernel to 3.10.0-1160.102.1.el7.

Lustre-change: https://review.whamcloud.com/52819
Lustre-commit: TBD (from 1feea616fd7addf842afdc836e7f32686ea159ae)

Test-Parameters: trivial clientdistro=el7.9 serverdistro=el7.9

Change-Id: Ifc56766dedf055dc3762e200835beb220fd63afb
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52843
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17221 kernel: update SLES15 SP4 [5.14.21-150400.24.92.1]

Update SLES15 SP4 kernel to 5.14.21-150400.24.92.1 for Lustre client.

Lustre-change: https://review.whamcloud.com/52820
Lustre-commit: TBD (from 92cf005d01e327e53bd312b411211ed2f1d827b9)

Test-Parameters: trivial clientdistro=sles15sp4 testlist=sanity

Change-Id: Id82d0ce48179df1f12dc367cced8cf84e1b918d9
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52825
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17222 kernel: update SLES15 SP5 [5.14.21-150500.55.31.1]

Update SLES15 SP5 kernel to 5.14.21-150500.55.31.1 for Lustre client.

Lustre-change: https://review.whamcloud.com/c/fs/lustre-release/+/52821
Lustre-commit: TBD (from b2159275aaf3595776ae89b3efeda4ec8bde14ff)

Test-Parameters: trivial clientdistro=sles15sp5 testlist=sanity

Change-Id: I5719e8c79740a58223b2e0bea6f6b269f281968a
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52824
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16868 tests: skip conf-sanity/32 in interop

Do not run conf-sanity.sh test_32* in interop testing.  Otherwise,
it is possible that the version of the test script running on the
client does not perform the upgrades with the right steps needed
for remote servers that are running a different version.

Lustre-change: https://review.whamcloud.com/52835
Lustre-commit: TBD (from 6368e97e593707d2ae1423dcb41c7f001f1d2152)

Test-Parameters: trivial testlist=conf-sanity env=ONLY=32a
Test-Parameters: testlist=conf-sanity env=ONLY=32a serverversion=EXA5
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Iabe1469a87d58c49e3c38b76ab18f8997f3ebbe5
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52836
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>

LU-13453 osd-ldiskfs: do not leak inode if OI insertion fails

osd_create() should destroy just created inode if OI insertion
fails.

also fixes lustre_index_restore() to drop nlink for object to
be removed.

the patch adds two tests:
 - ENOSPC on OI insertion
 - ENOSPC on .. insertion, i.e. directory block allocation

Lustre-change: https://review.whamcloud.com/38235
Lustre-commit: e45e8a92a2ecab742b3680716a55aaa1d9827057

Test-Parameters: testlist=sanity-scrub mdscount=2 mdtcount=4
Test-Parameters: testlist=sanity-scrub mdscount=2 mdtcount=4
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I2a5db657c7dab54b8dc2c50bc29365d5ee754a2e
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52846
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn110

New tag 2.14.0-ddn110

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Iab6709369cc3d4e50fd799fcd6db3796202905e7

LU-14659 test: improve generate_uneven_mdts() in sanity.sh

Improve generate_uneven_mdts() in several places:
1. set qos maxage to 1, so the result is up to date, and avoid filling
   up MDT.
2. fill MDT with files of size 64K other than 1M, so MDT imbalance is
   quicker to achieve.
3. when checking minimum imbalance after test, lookup max value from
   the result, other than by index stored before directory creation,
   because the result is dynamic if several MDTs have almost the same
   free space and inodes.

Lustre-change: https://review.whamcloud.com/44649
Lustre-commit: d45be79a069f527657c1ce91630183031ea42b27

Test-Parameter: trivial mdscount=2 mdtcount=4 testlist=sanity
Fixes: 233344d451e ("LU-13417 test: generate uneven MDTs early for sanity 413")
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I2807101ff632404e25fdb640840d83d1991c88d9
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52751
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7507 test: fix sanity-lfsck test_8

The backport of LU-15738 is not correct, which may cause test failure.

Fixes: 779c10b7c64 ("LU-15738 test: check lfsck status before starting")
Test-Parameters: trivial testlist=sanity-lfsck env=ONLY=8,ONLY_REPEAT=10
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Ibbc5b576db7d502095257031dd4619fe5103df3b
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52774
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17129 tests: cleanup fileset info on nodemaps

In sanity-sec, fileset info added to nodemaps via 'set_param -P' must
be removed afterwards with 'set_param -P -d', otherwise those commands
will remain in the llogs.

Lustre-change: https://review.whamcloud.com/52408
Lustre-commit: 16e4383e90f630dfcce20e2675bb887471b6a9c8

Test-Parameters: trivial mdscount=2 mdtcount=4 testlist=sanity-sec env=ONLY=27
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I75bd0dc263f71c7f5d9ece028cc038eb1f2ca9a4
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52706
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15564 osd: add allocation time histogram

add block mapping/allocation histogram to brw stats to debug
mballoc related issues.

$ lctl get_param osd*.*OST*.brw_stats
                           read      |     write
block maps msec        maps  % cum % |  maps        % cum %
1:    1522360 100 100   | 49272  99  99
2:          0   0 100   |    1   0  99
4:          0   0 100   |    1   0  99
8:          0   0 100   |    0   0  99
16:          0   0 100   |    0   0  99
32:          0   0 100   |    0   0  99
64:          0   0 100   |    1   0 100

Lustre-change: https://review.whamcloud.com/46550
Lustre-commit: f97ce54357bc91f7b1285febfc50d6087dd94c13

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I1185386adc64e844de71e25a4e439e493e5e5bc5
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52767
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14927 osd: share brw_stats code between OSD back ends.

Both the ldiskfs and ZFS OSD backend handle brw_stats. With the
stricter GPL requirement ZFS can no longer carry the brw_stats
code. So move the common code to lprocfs_status_server.c as
well as move brw_stats to debugfs as well.

Lustre-change: https://review.whamcloud.com/44690
Lustre-commit: 8a84c7f9c7d65f6f880be6fe4d94fca26a405d81

Change-Id: I294e5df3557552266dd3a02d3bc9844c42c01f60
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52766
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16694 tests: replace resolveip script

The resolveip script can be replaced with a bash one-liner,
using getent and awk.

Lustre-change: https://review.whamcloud.com/50491
Lustre-commit: f8cc7db39dd22fdb6330402a60af7bb30c78449e

Test-Parameters: trivial
Signed-off-by: Timothy Day <timday@amazon.com>
Signed-off-by: Vitaliy Kuznetsov <vkuznetsov@ddn.com>
Change-Id: I207ea011e43b7b236d5082994ffb51654d8d782c
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52781
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-8349 lipe: Fix problem with fallocate in test_306

This patch modifies the fallocate command so that the
size type is specified correctly on all versions.
This should will fix the issue with invalid
length value.

Test-Parameters: trivial
Signed-off-by: Vitaliy Kuznetsov <vkuznetsov@ddn.com>
Change-Id: Ie25dc87aed8ea9c882d9b352b11a58071c9c460d
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52779
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-8421 llite: disable kernel readahead for pcc mmap

Set ra_pages to 0 for PCC files when mmaped, because
otherwise this setting carries through to Lustre and will
cause crashes and possible inconsistencies.  This happens
because the PCC file and Lustre file share a mapping, which
is a weird trick required to have mmap work on PCC.

Add a set of asserts which confirm kernel readahead is
disabled and wasn't used for mmap.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I117042d68fac25158e8141c243acba698cf1930f
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52732
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Qian Yingjin <qian@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-8369 ldiskfs: mballoc to store group in pa

mballoc should store and then use group number from pa,
not calculate it from pa_pstart which change and point to
a subsequent group in the case of full pa consumption.

Fixes: f36eda6a1e ("LU-10026 osd-ldiskfs: use preallocation for dense writes")
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I5e3c5c7e51fe9688f4d847a9ea0591486d8975d8
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52754
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn109

New tag 2.14.0-ddn109

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I6d46cc9e8edf9362e5a639644da4a9e205b2932a

LU-17182 utils: pool_add send OSTs in one batch

'lctl pool_add' command sends all requests in one batch
then checks results. In this way, the command won't take
too long time if the OSTs are specified in command line
one by one.

Lustre-change: https://review.whamcloud.com/52654
Lustre-commit: TBD (from 7a9b37c84921687ab8ac0c765aab6b5a6b339468)

Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial
Change-Id: Ibd6e7ed5104e100d44c5f4288a25e7378cd9cfe8
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52719
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn108

New tag 2.14.0-ddn108

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I6c0a1e6da1605c4c569c2bfbb7543c318fd96327

RM-620 build: New tag lipe-2.34

New tag lipe-2.34

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I5ace0da818b9ffb5bb52a158bc2e3a17ff65be8a

LU-17202 target: set SB_KERNMOUNT

set SB_KERNMOUNT on server's mountpoint so umount process block
until the server (MDS/OST/MGS) is really umounted and released
the corresponding block device.

Lustre-change: https://review.whamcloud.com/52724
Lustre-commit: TBD (from 005a85f1c17a6c3477786a47dccfb67f2b36cb09)

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I03ac2dc6bb7cd4d93f5a1729fccb976aa1ebd5d4
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52725
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-8245 osc: use correct count

Using the number of bytes in the compressed page creates
gaps in the RDMA, which IB memory registration cannot
accept.

Fix this by always setting count to PAGE_SIZE for
compressed pages and otherwise using the count from the
original source page.  Setting PAGE_SIZE for compressed
pages is valid because client only does compression for
aligned IO, except for the trailing chunk.  For the
trailing chunk, the file size is set on the server, so any
trailing bytes are ignored.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Ied89d3ac328fb6020079392f5a8812ad5637b4a4
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52550
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-11912 tests: force new seq in runtests

If seq rollover happens during runtests/1,
the new seq on OST will consume some space and
this will fail the free space check.

Force a new seq before running test case to
prevent this.

Lustre-change: https://review.whamcloud.com/52741
Lustre-commit: TBD

Change-Id: I7bb1156127eb423889626bf84bc6c87dd68e6ece
Test-Parameters: trivial testlist=runtests
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52743
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15740 tests: scale fs_log_size by OSTCOUNT

The fs_log_size "free space skew" was being scaled by MDSCOUNT,
but in fact this parameter is only ever used to compare the OST
free space usage, so the OSTCOUNT should be used when scaling it.

It is likely that the skew is actually caused by blocks allocated
by OST object directories and not llogs (no llogs used on OSTs for
many years), but it isn't worthwhile to rename the function.

Lustre-change: https://review.whamcloud.com/50419
Lustre-commit: fabec6f2cb39950a2f208567dac716e21880fa9f

Test-Parameters: trivial testlist=replay-single env=ONLY="20b 89"
Test-Parameters: testlist=runtests clientdistro=ubuntu2204
Test-Parameters: testlist=replay-ost-single env=ONLY="6 7"
Test-Parameters: testlist=sanity-sec env=ONLY="16-22
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I97f05b10fa7ec367534b5bdce09feae5e93ebbe5
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52742
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-13748 mdt: remove LASSERT in mdt_dump_lmm()

Change LASSERT() in mdt_dump_lmm() into a CERROR(), since this
function is normally used for debugging and it doesn't help to
crash the MDS just when it complaining about some file layout.

Add sanity test_27Cg that triggered this initially.

Lustre-change: https://review.whamcloud.com/50532
Lustre-commit: 97d29eb800e8d9faba04f0744376cb50b239e2e9

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I27f0b4489de48d6f197e0143589d76428d85e549
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52728
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-17084 lod: fix comparision in lod_striping_load()

in if (rc > sizeof(struct lmv_foreign_md)) the latter
is unsigned and gcc treats rc (which is defined as int
and can be negative to encode an error) as unsigned.
this way -EIO becomes greater than the size of the
structure.  make sizeof() signed to avoid confusion.

Lustre-change: https://review.whamcloud.com/52265
Lustre-commit: ce54b5281c3172401ce4526a4de65d2d584fa0e7

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Signed-off-by: Xing Huang <hxing@ddn.com>
Change-Id: Ie6735578649e397ed05b6951fab941f97051305b
Reviewed-by: xinliang <xinliang.liu@linaro.org>
Reviewed-by: Mikhail Pershin <mpershin@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52721
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-12896 gss: key can be unlinked when timeout expires

The key associated with a GSS context could appear to be already
unlinked when the upcall timeout expires. In this case, do not assert
but report this case with a warning message.

Lustre-change: https://review.whamcloud.com/52473
Lustre-commit: 4c6290087b3bf0838a00de8f8b1cfde86efbc409

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I714af3a1ce54648c4ba29ef13015f9291de52765
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52705
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-8400 lnet: revert LU-13485

This reverts commit 889a8c41b9799256f9c1f54abc221b7c3d1ed3a8.
("LU-13485 lnet: Parallel configure tests for lnet")
The commit breaks the rhel8.8 build with implicit declaration of
ib_get_dma_mr.

Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: I77f6ddb46d49b27fa49594026223df54e423779d
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52718
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn107

New tag 2.14.0-ddn107

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ida621bbb0a593a5a18123a95cf3cba4fb1340a2c

EX-8349 lipe: Fix problem with fallocate in test_306

This patch removes the check every time a file is created,
and adds a general check for the presence of created files.

Test-Parameters: trivial testlist=sanity-lipe-scan3
Signed-off-by: Vitaliy Kuznetsov <vkuznetsov@ddn.com>
Change-Id: I7d7e0c9b0e6517853bb3b77e8f938bbd877b3003
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52590
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13575 lnet: Ensure round robin selection of peer NIs

Use the peer net sequence number to set the peer NI sequence number to
ensure round robin selection of peer NIs on each peer net.

Lustre-change: https://review.whamcloud.com/45004
Lustre-commit: c51763948abfdbdc8e3f3ea7e73f2632320a095a

HPE-bug-id: LUS-10349
Test-Parameters: trivial
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I1fa14ad675ead4ae2c5b1d4edad250caa4498df2
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52471
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>

LU-13575 lnet: Ensure round robin selection of local NIs

Use the net sequence number to set the NI sequence number to ensure
round robin selection of NIs on each net.

Lustre-change: https://review.whamcloud.com/45003
Lustre-commit: a18c4a16246e6185919eda805eca52772bbc3efe

Test-Parameters: trivial
HPE-bug-id: LUS-10349
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I6ce0b088fcad6312186e6fbad4ab14283aee55eb
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52470
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>

LU-9121 lnet: select best peer and local net

Select the healthiest and highest priority peer and local net when
sending a message.

Lustre-change: https://review.whamcloud.com/34352
Lustre-commit: dff6587805ddad212ab48e5bedacbc7846542b7b

Test-Parameters: trivial testlist=lnet-selftest,sanity-lnet
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: I42717e7fdc3226c6faa7c59c713f18422e27f2e5
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52444
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>

LU-17088 dom: don't create different size DOM component

Multiple DOM components are allowed in diffrent mirror but they
must be of the same size, mirror extend should check this restraint.

Fix another glitch in lov_init_composite() where dom_size is used
as a __u64 value but declared as boolean.

Lustre-change: https://review.whamcloud.com/52269
Lustre-commit: e2539c0667525aff8d985d018c4ed077d95ba882

Fixes: 44a721b8c1 ("LU-11421 dom: manual OST-to-DOM migration via mirroring")
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Ia0d08c697dbeeb3aa8d20d9849226afa06360012
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mikhail Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52601
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14637 flr: get rid of excluding dom+flr support test

Now that DoM+FLR are supported, fix the tests that expect this
combination of features on a file to fail.

Lustre-change: https://review.whamcloud.com/44185
Lustre-commit: 4b52ea1d30b45900787271c4c035fad124abf34a

Fixes: 0bff64be320fd ("LU-9771 flr: to not support dom+flr for phase 1")
Fixes: 44a721b8c1063 ("LU-11421 dom: manual OST-to-DOM migration via mirroring)
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I9fc76e797e469744107e5d0453b78729226be0ee
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52600
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-8344 lipe: Update manual page

This small patch expands the explanations for some
commands with information from the development files.
Adds one "todo-list.md" file instead of different files
with similar information.

Test-Parameters: trivial
Signed-off-by: Vitaliy Kuznetsov <vkuznetsov@ddn.com>
Change-Id: I0a3ebda49525d62cd6ca398f12601e588dc2dd42
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52589
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Colin Faber <cfaber@ddn.com>
Reviewed-by: Alexandre Ioffe <aioffe@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17015 gss: support large kerberos token for rpc sec init

If the current Kerberos setup is using large token, like when PAC
feature is enabled for Kerberos, authentication can fail due to server
side unable to exchange token between kernel and userspace.
This limitation is inherent to the sunrpc cache mechanism, that can
only handle tokens up to PAGE_SIZE.

For RPC sec init phase, use Lustre's upcall cache mechanism
instead of deprecated kernel's sunrpc cache. The upcall calls a new
userspace command 'l_getauth', that fowards the sec init request to
the lsvcgssd daemon via Unix domain sockets.

Lustre-change: https://review.whamcloud.com/52224
Lustre-commit: TBD (from 8acd059ee2b8d1e4c48c3d9dbb380bca75e1b3be)

Test-Parameters: kerberos=true testlist=sanity-krb5
Change-Id: I709cd79894a5a13fc4cdfab2109c86f2230db3b8
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52653
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17015 build: rework upcall cache

EX-4333 introduced in upcall_cache.c a dependency on md_object.h for
struct lu_ucred. Rework files to move this dependency to a differnt
file, so that upcall_cache.c can be built in client-only mode.

Fixes: fb0082bba1 ("EX-4333 sec: support supplementary groups from client")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I4bcc7e07a4f4886c5994d17cbef72ea09eb1be1d
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52670
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17015 gss: bump token buffer size to 16KiB

A 4 KiB large buffer is not enough to hold the GSS token under some
circumstances. So bump GSS_CTX_INIT_MAX_LEN value to 16 KiB.

Lustre-change: https://review.whamcloud.com/52475
Lustre-commit: TBD (from 43a540207da0198cc9c45b3c6312c555702b56cb)

Fixes: 9758129177 ("LU-17015 gss: support large kerberos token on client")
Test-Parameters: trivial kerberos=true testlist=sanity-krb5
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I8e72f1447593d2bf2ae537fcc920ceee20e93c09
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52628
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-13485 ldiskfs: Parallel configure tests for ldiskfs

Transform the compile tests in ldiskfs to run in parallel

Lustre-change: https://review.whamcloud.com/38351
Lustre-commit: 3774b6afbe3b67e869bb61c9cb212cc37e8705fa

Test-Parameters: trivial
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I3a097ab5cd18b57e9311980d9aa708ed25f58464
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52655
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13485 libcfs: Remove unused iter_type check

The iter_type member check is not used, remove it.

Lustre-change: https://review.whamcloud.com/48091
Lustre-commit: c755373c567090c49589e5aa0d3134847d4b952e

Test-Parameters: trivial
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I48d536a27738e73314feb88317d41d8479c72528
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52683
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-13485 lnet: Parallel configure tests for lnet

Transform the compile tests in lustre-lnet to run in parallel
Also fixes the generated Makefile to work with MOFED and in-kernel
OFED.

configure build times on an 8 core 8G vm vs current serial:

             serial      parallel
            --------     --------
    real    8m27.824s    1m28.375s
    user    5m29.448s    2m11.558s
    sys     3m48.258s    0m51.763s

Lustre-change: https://review.whamcloud.com/38368
Lustre-commit: fc84caa81b7fb9d27e82229d39f046e83b5ebb7e

Test-Parameters: trivial
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I4f0cb8584e1c3149ec3f005dd55fed0c47b50472
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52678
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-8038 csdc: sending compression info to server

Client fills in layout compression info into obdo and passes it
to server.

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Ieb5d7b3609da41f35f8622ed6116f19ce7567ddb
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52669
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn106

New tag 2.14.0-ddn106

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ibc557d9c0a56d9994b7fed147e7e183a1b5528db

RM-620 build: New tag lipe-2.33

New tag lipe-2.33

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I638f4b593856e9e8c15b180328be9cc23ec3365e

LU-16837 llite: handle unknown layout component

If lustre client encounters unknown layout component pattern in
a mirror file, this patch makes client mark this mirror as invalid
and skip it.

Lustre-change: https://review.whamcloud.com/51060
Lustre-commit: 14ed4a6f8f231fe94392906f991a32f07e7d7883

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Ie5f44212ab96bdc706cc5a9e11f330234fc01069
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51061
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7948 utils: lamigo to track mirror progress

pass --stats to lfs mirror/resync commands and then read
lfs's output over ssh channel.
this way we can keep ssh channel alive and interrupt
replication if it doesn't report progress.

the very first time agent is used lamigo checks whether
agent's lfs utility supports stats.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Iee5b43cb85dae62550d74667b16e00336f1bf52f
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51744
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexandre Ioffe <aioffe@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-9121 lnet: Select NI/peer NI with highest prio

Modify the selection algorithm to select the highest priority
local and peer NI. Health always trumps all other selection
criteria

Lustre-commit: 3fc2e0e0b3c8353a8fecc6d127ee55d255d7acb7
Lustre-change: https://review.whamcloud.com/34351

Test-Parameters: trivial testlist=lnet-selftest,sanity-lnet
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: I487a706f4da30311d0bd59fe03f72dbe68a52425
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52289
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-9121 lnet: foundation patch for selection mod

Add the priority and preferred NIDs fields in the lnet_ni,
lnet_net, lnet_peer_net and lnet_peer_ni. Switched
the implementation of the preferred NIDs list to list_head
instead of array, because the code is more straight forward.
There is more memory overhead due to list_head, but these lists
are expected to be small, so I chose code simplicity over memory.

Lustre-commit: 51b2c0f75f727f0562b3145015357cbff5cbb3b5
Lustre-change: https://review.whamcloud.com/34350

Test-Parameters: trivial testlist=lnet-selftest,sanity-lnet
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: I0c75855b736345c25e1604083eee2b65d38ef28d
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52288
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16097 tests: skip quota subtests in interop

Skip subtests in sanity-quota.sh to avoid interop test failures,
backdated to check all new tests since 2.14.0 for completeness.

Test-Parameters: trivial testlist=sanity-quota serverversion=EXA6.1.0
Fixes: 513b1cdbca ("LU-16340 quota: notify only global lqe")
Fixes: d4978678b4 ("LU-15694 quota: keep grace time while setting default")
Fixes: 25a70a88c9 ("LU-13952 quota: default OST Pool Quotas")
Fixes: 188112fc80 ("LU-14300 quota: avoid nested lqe lookup")
Fixes: 8c19365416 ("LU-13971 quota: report Pool Quotas for a user")
Fixes: a4fbe7341b ("LU-14739 quota: nodemap squashed root cannot bypass quota")
Fixes: 789038c97a ("LU-15167 quota: fallocate send UID/GID for quota")
Fixes: c9901b68b4 ("LU-13587 quota: protect qpi in proc")
Fixes: 61ec1e0f2c ("LU-15031 quota: reseed glbe in qmt_lvbo_udate")
Fixes: dfe7d2dd2b ("LU-16341 quota: fix panic in qmt_site_recalc_cb")
Fixes: 862f0baa7c ("LU-15097 quota: stop pool_recalc before killing pool")
Fixes: 61481796ac ("LU-15193 quota: expand QUOTA_MAX_TRANSIDS to 12")
Fixes: a2fd4d3aee ("LU-15880 quota: fix insane grant quota")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Sergey Cheremencev <scherementsev@ddn.com>
Change-Id: Ife8bfd83d0f217c534f3b12b4c9d108d370ed6b7
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52009
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52582

EX-8281 tests: Fix sanity test_56ab when CSDC is enabled

Use /dev/urandom in sanity test_56ab so the data cannot be compressed

Lustre-change: https://review.whamcloud.com/52572
Lustre-commit: TBD  (from 1ce661dd56fb4b6ecc9e909805c6101bbd9c3161)

Test-Parameters: trivial testlist=sanity env=ONLY=56ab

Signed-off-by: Wei Liu <sarah@whamcloud.com>
Change-Id: I0ceb9afcbdc8443b5e04dff486e41621479dbd23
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52501
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Colin Faber <cfaber@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-17152 tests: unmount NFS clients with zconf_umount_clients

This patch fixes cleanup_nfs() to unmount NFS clients by running
zconf_umount_clients(), which can find and kill active processes
that are accessing the NFS mount point so as to avoid the
"device is busy" failure.

The patch also adds racer_on_nfs test into always_except list for
parallel-scale-nfsv4 due to LU-17154.

Lustre-change: https://review.whamcloud.com/52533
Lustre-commit: TBD (from 52a2147e8b0eca74f38b1b87991b53ccf25663cd)

Test-Parameters: trivial testlist=parallel-scale-nfsv4

Change-Id: I37a38502362399540c28e78d1343e768b490ce8b
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52534
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>

EX-3860 llog: extended debug for -ENOTDIR error

Debug patch to catch trace and debug log for -ENOTDIR
error in distribute_txn_cancel_records()

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: Ie1bb7c138282bfa05a2fafcceafdb436d45f28d3
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52394
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-13814 clio: add cp_inode to page allocation

cp_inode can be set correctly during page allocation,
rather than after.  This is a prelude to moving cp_inode to
the osc_transfer_page, but that's better done in a separate
patch.

Lustre-change: https://review.whamcloud.com/52208
Lustre-commit: TBD (from f2afaf4eb10d70c36ad6bdbc2def66bee4fcdc23)
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Signed-off-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Change-Id: I509f6cfbae8e5a6ec6b07c8253d68f6dd2794e59
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52557
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7601 osc: move common CSDC code to the library

CSDC repacks a chunk on the server side in case of the
partial rewrite. There are routines that can be shared
between client and server.

This patch moves common compression code to the
libcfs.

Signed-off-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Change-Id: I824211a3435b0479f7a3b8f08598a5b567b67d3c
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52262
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17087 lmv: update stale tgt statfs every 1 hour

Some tgt statfs may not be initialized upon mount due to network
issues, if the filesystem is imbalanced, these tgts won't be chosen to
create directory because their bavail and ffree are 0.

If MDT is chosen by QoS, update tgt statfs that is one hour overdue,
otherwise check update the statfs of the tgt that is chosen.

Lustre-commit: e262e0ffbe792ae2f8b47ccdafac38a36151a300
Lustre-change: https://review.whamcloud.com/52270

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I06af8b8bd342f66cb794471df3ee0f3b127ffe05
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52560
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17136 ldiskfs: increase max extent tree depth

this is an workaround until LU-16843 ready

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I5829c10888bf32649fe7a7a72c8ee697647a89cc
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52540
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-8253 tests: skip sanity-scrub/4e until fixed

Subtest 18 is failing about 1/5 of sanity-scrub runs, after test_4e
was landed.  Disable test_4e to see if that fixes the issue.

Test-Parameters: trivial
Test-Parameters: testlist=sanity-scrub,sanity-scrub,sanity-scrub
Test-Parameters: testlist=sanity-scrub,sanity-scrub,sanity-scrub
Test-Parameters: testlist=sanity-scrub,sanity-scrub,sanity-scrub
Test-Parameters: testlist=sanity-scrub,sanity-scrub,sanity-scrub
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ia8f1bd9dbf0fdbfabf79b1ead63a0421a8892c82
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52564
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn105

New tag 2.14.0-ddn105

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I28dd2c8ac1550a3c6abb630a32e88423fdbf492e

RM-620 build: New tag lipe-2.32

New tag lipe-2.32

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I4556f6234985973170f94c06c3572dbfce3204c1

EX-8191 lipe: Fix test for --collect-fsize-stats in lipe3

This patch modifies the test for collecting statistics
in lipe3 and corrects:
1. Error getting a username if it doesnt already exist.
2. Error comparing file sizes after changing table
generation rules.
3. Converts the test from reading yaml to reading json
4. Now many files of different sizes are generated
for the test.
5. Now the data for comparison is retrieved from
the ls utility.
6. The test has added a check for creating a user with a
large UID, GID, which checks the availability of reports
for this user.

Test-Parameters: trivial testlist=sanity-lipe-scan3
Signed-off-by: Vitaliy Kuznetsov <vkuznetsov@ddn.com>
Change-Id: I7d0bdcc407bc0d27441c4204511dab2e6a421a5f
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52424
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16954 llite: add SB_I_CGROUPWB on super block for cgroup

Cgroup support can be enabled per super_block by setting
SB_I_CGROUPWB in ->s_iflags.
Cgroup writeback requires support from both the bdi and
filesystem.
This patch adds SB_I_CGROUPWB flag on super block for Lustre.
This is required by the subsequent patch series to support
cgroup in Lustre.

Adding this flags for Lustre super block will cause the remount
failure on Maloo testing on Unbutu 2204 v5.15 kernel due to the
duplicate filename (sysfs) for bdi device.
To avoid remount failure, we explicitly unregister the sysfs for
the @bdi.

Lustre-change: https://review.whamcloud.com/51955
Lustre-commit: dcc1dd39a67f15de9174e7acdda599e3c54c1421

Test-Parameters: clientdistro=ubuntu2204 testlist=sanity-sec
Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I7fff4f26aa1bfdb0e5de0c4bdbff44ed74d18c2d
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52538
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17133 kernel: update SLES15 SP4 [5.14.21-150400.24.84.1]

Update SLES15 SP4 kernel to 5.14.21-150400.24.84.1 for Lustre client.

Lustre-change: https://review.whamcloud.com/52481
Lustre-commit: TBD (from 5dcdbe687d136d7e976f578faccbb3bde1b0acc9)

Test-Parameters: trivial clientdistro=sles15sp4 testlist=sanity

Change-Id: I5bce1642fc5bd212fd89dd65d9e1beb32ccd744d
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52546
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17010 lfsck: don't dump stack repeatedly

If there are transactions started with LFSCK in dry-run mode, don't
dump the stack repeatedly, as this can spam the console logs and
significantly hurt performance.

Lustre-commit: dc360cd3eff20618f243ab89097a62f8ecf2c929
Lustre-change: https://review.whamcloud.com/52356

Test-Parameters: trivial testlist=sanity-lfsck
Fixes: 0c1ae1cb9c ("LU-13124 scrub: check for multiple linked file")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I0b0d64911453dc8ab947e284656311b5d0300c1e
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52541
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>

LU-17095 build: avoid modules.order nonexistence failure

The modules.order is a temporary output file generated by
kbuild while running "make" command. Sometimes, there is
a race condition that causes the file not created and makes
make command fail as follows:

cat: ...//modules.order: No such file or directory

This patch creates an empty modules.order file to avoid
the error.

Lustre-change: https://review.whamcloud.com/52323
Lustre-commit: dbe4f860977455a9abe50165645a025bb6c46350

Test-Parameters: trivial

Change-Id: If779a727731f18e9409c35c0cd0deddd79559d3a
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52544
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-8245 ptlrpc: always do vmalloc

If we were ever to do an allocation with kmalloc, we could
get non-page aligned memory.  So just use vmalloc directly.

Sadly, this isn't the problem with infiniband.  We never
ask for < 8192, which is the libcfs kmalloc/vmalloc cutoff.

Still, this is a timebomb if we ever changed the libcfs
kmalloc/vmalloc cutoff, so, fix it.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Id20898065b516d363d9dc280e71be1b5cfb6f4a7
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52532
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7342 revert: "test: remove extra cleanup and qp check"

This reverts commit 1dbe9be20011893ca46ccbbd2676e8063af4158d.
This causes 100% sanity-quota timeouts in test_79.

Test-Parameters: trivial testlist=sanity-quota
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I77a168bde4b53b69a197a4036b31b36f792ebae3
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52561
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>

RM-620 build: New tag 2.14.0-ddn104

New tag 2.14.0-ddn104

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I118f71235f7200f1e880424d5e7ac43334186ba3

LU-16699 osc: Prefer NR_ZONE_WRITE_PENDING

Linux commit v4.7-5966-g5a1c84b404a7
 mm: remove reclaim and compaction retry approximations

Introduced NR_ZONE_WRITE_PENDING which should be used
in mod_zone_page_state.

Older kernels should fallback to NR_UNSTABLE_NFS
or NR_WRITEBACK.

Lustre-change: https://review.whamcloud.com/50499
Lustre-commit: d4094475c990d6ee8bf9e6e32a93f7c86a78f57a

Test-Parameters: trivial
HPE-bug-id: LUS-11559
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I90f22d4bd56f5986eaa5d4a042a2c8ed31fbf752
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Qian Yingjin <qian@ddn.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52526
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16671 osc: fix unstable pages for short IO

Unstable pages was written with theoretical support for
short IO (ie, no bulk, data-in-rpc, LU-1757), but since the
short IO code wasn't merged until years later, they were
probably never tested together.  And when you do, it
crashes.

In truth, short IO has no separate pages to be tracked,
which is why this is crashing.  This means that small write
RPCs won't be tracked in unstable pages, but that's a very
minor limitation and unlikely to cause trouble.  (and since
RPC allocations are not 'pages', they're just malloc'ed,
there's no good way to track them anyway)

Lustre-change: https://review.whamcloud.com/50451
Lustre-commit: 4ba4976f525e957ef4c3ca7981bea01f72109ed6

Fixes: 70f092a ("LU-1757 brw: add short io osc/ost transfer.")
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I34b09f8324424c3ff0b0c09c86f01c938b643e37
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Qian Yingjin <qian@ddn.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52524
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-17015 obdclass: new primitives for upcall cache

This patch adds 2 new primitives to the upcall cache mechanism:
- upcall_cache_get_entry_raw: get a ref on an existing entry;
- upcall_cache_update_entry: modify expiry time and state of an entry.

Lustre-change: https://review.whamcloud.com/52389
Lustre-commit: 2ddb1d33245c23c4cafe64fb917323bdf567c81f

Test-Parameters: trivial
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I4825f09ae807abb52ebe0e24719dcd915e8c8aef
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52497
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-8277 llite: set max compression size to 64 MiB

Compression size should never be larger than RPC size, so
set it to a maximum of 64 MiB.

Test-Parameters: trivial
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Ia5958db3504f4f442fbd41e48416924debc26192
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52466
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7342 test: remove extra cleanup and qp check

The test_79 in sanity-quota needs quota pool support, and
the cleanup of the "stop file" is also included in the
stack_trap, then it is no need to to do it explicitly.

Test-Parameters: trivial
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Change-Id: If86a1d0187b4b95d0c5e24f11f5f058280726e64
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52472
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-8189 osc: do not compress resends

There's some issue with doing compression on resent
requests, so this patch works around it with two things:
1. Use the uncompressed page array for resend
(this was always necessary unless we modified resend to
know it already had compressed pages as input)
2. Disable compression on resend (not clear why 1. wasn't
enough)

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I5fbbdc2771f8c2c7b5c28f0b70d89b8b6015147f
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52484
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16973 osd: adds SB_KERNMOUNT flag

During umount mntput() is called. It uses delayed_mntput()
function, and it could take much time to finish. A block
device is occupied during delayed work.

[ 8753.941980] Lustre: server umount XXX complete
[ 8800.129136] sysrq: SysRq : Trigger a crash

PID: 319306   TASK:XXXX   CPU: 2    COMMAND: "kworker/2:0"
 #0 __schedule at ffffffff9754e1d4
 #1 preempt_schedule_common at ffffffff9754e6fa
 #2 _cond_resched at ffffffff9754e72d
 #3 invalidate_mapping_pages at ffffffff96e72da5
 #4 invalidate_bdev at ffffffff96f5d13c
 #5 ldiskfs_put_super at ffffffffc1c82e34 [ldiskfs]
 #6 generic_shutdown_super at ffffffff96f1bdcc
 #7 kill_block_super at ffffffff96f1bed1
 #8 deactivate_locked_super at ffffffff96f1b784
 #9 cleanup_mnt at ffffffff96f3b86b

Let's use SB_KERNMOUNT flag during mount, it leads to
synchronous mntput().
It also calls flush_delayed_fput during umount to finish
delayed fput.

Lustre-change: https://review.whamcloud.com//51731
Lustre-commit: eff11c8ce1f89f30dcc5af88b67b3d6c15a631a6

Change-Id: Ia6729f6cbac85c3626562e946a4b96665a143714
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52495
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16218 utils: add component flags "prefrd" and "prefwr"

The initial implementation of "lfs setstripe ... --comp-flags=prefer"
only allowed specifying a single "prefer" argument for a given
mirror component, which would set both the "LCME_FL_PREF_RD" and
"LCME_FL_PREF_WR" flags at the same time.

This patch adds the separated component flags "prefrd" and "prefwr"
to allow setting the individual flags on a component.

Lustre-change: https://review.whamcloud.com/52508
Lustre-commit: TBD (from a4cd76c790c46b4bf6d85e386b1054d4b925e095)

Test-Parameters: trivial testlist=sanity-flr

Change-Id: I3e413cb37fab7ab2834946536705ce61a3feeed4
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52525
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Colin Faber <cfaber@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16896 flr: resync should not change file size

mirror resync could punch a hole reaching the end of file in a
mirror, which could change the file size when the mirror is referred.

This patch calls truncate after punch in this case to keep the file
size unchanged in the mirror.

Lustre-change: https://review.whamcloud.com/51344
Lustre-commit: b9ce342ee196af48d2d25e2811121fe4471f5fd2

Also pick up commit 4cd4bfba473fb370767e1f2014d9fe1531889f82
("LU-16813 utils: move mirror_end initialization) to move
initialization for mirror_end variable in llapi_mirror_resync_many(),
otherwise lfs mirror resync may fail since mirror_end gets reset on
each pass of the loop.

Lustre-change: https://review.whamcloud.com/50919
Lustre-commit: 4cd4bfba473fb370767e1f2014d9fe1531889f82

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Ia0fc1f220a32a60f3516c69e86867796ae5c35c7
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52061
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-11912 tests: consume precreated objects in parallel

Run the force_new_seq_all() file creations to run in parallel, since
this can take a significant amount of time when there are multiple
MDTs and OSTs (up to 1000s for 4x MDTs and 8x OSTs).

Lustre-change: https://review.whamcloud.com/51292
Lustre-commit: 656fc937cfd3fc3b65cb21a7f93a6bd4cc07fc0e

Test-Parameters: trivial testlist=replay-dual mdscount=2 mdtcount=4
Test-Parameters: testlist=replay-ost-single mdscount=2 mdtcount=4
Test-Parameters: testlist=replay-single mdscount=2 mdtcount=4
Test-Parameters: testlist=sanity-pfl env=ONLY="0 1 16 27" mdscount=2 mdtcount=4
Fixes: 2fdb1f8d01b9f ("LU-11912 tests: SEQ rollover fixes")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I849370586fe320d1f7df069f0b83980449658d97
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51496
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-11912 ofd: reduce LUSTRE_DATA_SEQ_MAX_WIDTH

Reduce LUSTRE_DATA_SEQ_MAX_WIDTH from ~4B to ~32M
to limit the number of objects under /O/[seq]/d[0..31]
dir on OSTs.
This makes the directories stay optimial for ldiskfs,
to avoid going into the largedir/3-level htree territory.

Remove the hard-coded LUSTRE_DATA_SEQ_MAX_WIDTH checks
in ofd, make them check the seq->lcs_width which is
a tunable set to LUSTRE_DATA_SEQ_MAX_WIDTH by default,
allow the value up to IDIF_MAX_OID if a larger seq width
is needed.

Use the odbo->o_size in the OST_CREATE rpc reply on ofd,
to update osp with the current seq width setting.
osp then uses this seq width to determine when to rollover
to a new seq.

The seq will rollover when the seq width is exhausted,
the default is LUSTRE_DATA_SEQ_MAX_WIDTH.
For seq >= FID_SEQ_NORMAL objects, the upper limit of
seq width is OBIF_MAX_OID,
For IDIF/MDT0 objects, the upper limit is IDIF_MAX_OID.
The seq FID_SEQ_OST_MDT0 will change to a normal seq after the
rollover.

Fix osp_precreate_reserve when the last precreated is the end
of the seq and the osp_objs_precreated can not host all
the requested objects, the mdt thread would stuck:
it wakes up osp precreate thread in a loop for progress,
but osp thread will not try to do anything until the seq
is used up. This can be seen easier when seq->lcs_width is
set to a low number and try to create an overstripe with stripe
number bigger than seq->lcs_width.

Fix the precreate thread spinning when the precreate pool
is at the end of the seq, and is nearly empty.

Change the seq->lcs_width to 16384 for all tests in
test-framework.sh, except a few slow tests to avoid timeouts,
and some overstriping tests creating LOV_MAX_STRIPE_COUNT to
avoid overstriping creating less objects than expected,
when precreate pool is at the end of the seq, and there are
not enough objects.

Fix the problem where seq could still change after
replay_barrier. To achieve this, introduce new fail_loc
OBD_FAIL_OSP_FORCE_NEW_SEQ and force_new_seq/force_new_seq_all
to drain the objects in the precreate pool then rollover to a
new seq. This applies to a bunch of test suites heavily using
replay_barrier.

Lustre-change: https://review.whamcloud.com/38424
Lustre-commit: 0ecb2a167c56ffff8e4fcb5cf576fb8c5d9e64fe

LU-14692 tests: wait for osp in conf-sanity/84

Wait for osp to change the first IDIF SEQ to a
normal SEQ, before using replay_barrier.
Otherwise the SEQ change could get lost and we
will trigger LASSERT during replay.

Lustre-change: https://review.whamcloud.com/50477
Lustre-commit: a9b7d73964b8b655c6c628820464342309f11356

Change-Id: I2749c1004b7bf3197b691cc94527f90145bcdef8
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sergey Cheremencev <scherementsev@ddn.com>
LU-11912 tests: SEQ rollover fixes

To avoid changeing SEQ after replay_barrier, we
use force_new_seq when starting the test suites heavily
using replay_barrier, e.g. replay-single.
However when there are fewer OSTs, the default 16384
SEQ width could not last the entire test suite, SEQ
rollover could still happen randomly after replay_barrier.

To overcome this, change the default OSTSEQWIDTH to
65536, and divide by number of OSTs, so the SEQ width is
larger with fewer OSTs. For 8 OSTs, the SEQ width is 16384,
and make sure we don't go under it.

Use force_new_seq_all for the test suites using replay_barrier
on MDTs other than mds1.

Add force_new_seq_all for replay-ost-single, which is using
replay_barrier on OST. If SEQ rollover happens after that,
the SEQ range update on ofd is lost due to replay_barrier,
the next time when we try to allocate a new SEQ will end up
with an old one.

Use force_new_seq_all for the test cases(namely sanity-pfl/0b
0c 1c 16b sanity/27Cd) checking for number of stripes created
with overstriping, to make sure we have enough objects
in the precreate pool.

Lustre-change: https://review.whamcloud.com/50478
Lustre-commit: 2fdb1f8d01b9f55f8270b48edc0e105e40d42f55

Test-Parameters: ostcount=4 testlist=replay-single
Test-Parameters: ostcount=2 testlist=replay-single
Test-Parameters: mdtcount=2 testlist=conf-sanity env=ONLY=122a,ONLY_REPEAT=10
Test-Parameters: testlist=sanity,sanity-pfl
Test-Parameters: testlist=sanity-scrub,replay-single,obdfilter-survey,replay-ost-single,large-scale
Fixes: 0ecb2a167c ("LU-11912 ofd: reduce LUSTRE_DATA_SEQ_MAX_WIDTH")
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Change-Id: I2749c1004b7bf3197b691cc94527f90145bcdef8
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50760
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14692 osp: deprecate IDIF sequence for MDT0000

Always return true for IDIF seq osp_fid_end_seq
so osp precreate will rollover to a new seq in
the FID_SEQ_NORMAL range for MDT0000.

Remove conf-sanity test_122b:
Check OST sequence wouldn't change when IDIF 32bit overflows

Lustre-change: https://review.whamcloud.com/45822
Lustre-commit: 6d2e7d191a7b27cde62b605dbed14488cfd4d410

Change-Id: I85a0e38266331c96d971d68ec353949ccac3fc21
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sergey Cheremencev <scherementsev@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/50758
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-10026 osd-ldiskfs: use preallocation for dense writes

use inode's preallocation chunks as per-inode group preallocation:
just grab the very first available blocks from the window.

Lustre-change: https://review.whamcloud.com//50171
Lustre-commit: TBD (from 986340bcdfa572a1f6bab34014e0474c89f47691))

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I9d36701f569f4c6305bc46f3373bfc054fcd61a9
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51468
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Zhenyu Xu <bobijam@hotmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn103

New tag 2.14.0-ddn103

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Iab8b19200c1ff9947b21c0e150ede242af5cede5

LU-17128 build: fix lnet.service missing issue on Ubuntu 22.04

The lnet.service file was in the lustre-client-utils package
built on Ubuntu 20.04, but it was missing from the package
built on Ubuntu 22.04.

This is caused by the dpkg-buildpackage change introduced by
dpkg version 1.21.1ubuntu2.1 installed by default on Ubuntu 22.04.
To fix this issue, we need to specify build profiles explicitly
to dpkg-buildpackage via -P|--build-profiles option instead of
just setting the environment variable DEB_BUILD_PROFILES.

Lustre-change: https://review.whamcloud.com/52404
Lustre-commit: TBD (from 59eef55fe08d761be91d9bce207d43f99769cf08)

Test-Parameters: trivial clientdistro=ubuntu2004
Test-Parameters: trivial clientdistro=ubuntu2204

Signed-off-by: Jian Yu <yujian@whamcloud.com>
Change-Id: I9975ef357f0aba722c56d27eaa9b2cfbccc9c524
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52405
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7795 scripts: add whole file to compression scan

Add a mode where the compression scan script compresses the
entire file, which in theory should 100% match the
compression results from using CSDC and allow a test to
calculate the exact space usage reduction expected by
using CSDC.

This is intended to be used mostly for testing.

Change help documentation slightly to make clear this can
also accept a path to a single file.

Test-Parameters: trivial
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I606a33d686d87dd631bf5b33dc85ee8c24fe9f67
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52406
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-8270 ptlrpc: rename 'pools' to 'ptr_pages'

This finalizes the removal of the overloading of 'pools'
to also mean pointers of pages to items in each page pool.

This patch is currently the last in the series so it gets
full testing.  (This may change, but it's true now.)

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I0f4aba95f573f4afdc6f5d92f22fd67391fa6dab
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52461
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-8270 ptlrpc: rename npools to nptr_pages

Continue removal of 'pool' as a name for a page of pointers
to items in a pool.

Test-Parameters: trivial
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I97b320027a0a6b5870d246e1527fa3fbe15fccb5
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52460
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-8270 ptlrpc: rename max_pools to max_ptr_pages

Continue removal of referring to page pointers as pools
with another rename.

Test-Parameters: trivial
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I14796f670a7f06fbec3b40ec23b4dd2e50f22d46
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52459
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-8270 ptlrpc: change "pool" to "page_ptrs"

The page pool code *also* likes to refer to each page of
pointers it uses to track items in it as a "POOL", which is
incredibly confusing.

This patch works on renaming that to page_ptrs, but leaves
some steps for a future patch.

Test-Parameters: trivial
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I56ee54c7f39b52d7cceffec9e3decf71bd313ddc
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52458
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-8270 ptlrpc: remove PAGES_PER_POOL macro

The page pool code *also* likes to refer to each page of
pointers it uses to track items in it as a "POOL", which is
incredibly confusing.

Start unwinding this by removing the PAGES_PER_POOL macro.

Test-Parameters: trivial
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Ie29434f53eeb945b8d35df7c1212ae3f51a2aafa
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52457
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-8270 ptlrpc: remove PAGES_POOL macro

PAGES_POOL is just the order 0 pool now, so remove the
special naming, and adjust a few associated functions.

Test-Parameters: trivial
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I09e1debeadecbce33c7be43a8859815084623358
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52456
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-8270 ptlrpc: rename INDEX macros

Rename INDEX macros to ORDER.

Test-Parameters: trivial
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Ic1123d25bc855dc7671c9cb587a0d6680662b729
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52455
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-8270 ptlrpc: rename ppp_index to ppp_order

Rename ppp_index to ppp_order.

Other renames will be in a subsequent patch, to keep these
as simple as possible.

Test-Parameters: trivial
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I96559e27a67b7cc4e56e06378e5686370438850c
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52454
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-8270 ptlrpc: begin renaming pool_index to order

Replace local variables for pool_index with pool_order.

Other renames will be in a subsequent patch, to keep these
as simple as possible.

Test-Parameters: trivial
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: If347ff39776f9a75c0f7d9d9981d01e19bc2cbc9
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52453
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>