EX-7849 quota: extra debug messages

Add extra debug messages into qmt to find the
root cause of panic:

  qmt_id_lock_glimpse()) ASSERTION( lqe->lqe_gl )

Signed-off-by: Sergey Cheremencev <scherementsev@ddn.com>
Change-Id: I05377222e1887b660f759ed11de53cd9e4023ed1
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52906
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16868 tests: skip conf-sanity/66 in interop

Do not run conf-sanity.sh test_66* in interop testing.  Otherwise,
it is possible that the version of the test script running on the
client does not perform the upgrades with the right steps needed
for remote servers that are running a different version.

Lustre-change: https://review.whamcloud.com/52899
lustre-commit: TBD (from 774e626146ddcbeb527c0939e0210f92bab4c6c3)

Test-Parameters: trivial testlist=conf-sanity env=ONLY=66
Test-Parameters: testlist=conf-sanity env=ONLY=66 serverversion=2.12.9
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I7b28b5f123a7348f87d43c54c806eaf6173ebbe5
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52900
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>

EX-5258 lipe: add hidden option -show_counters

This patch adds a hidden output policy option
to lipe_find3 that shows how many total inodes
were scanned and the number of inodes
that matched the filters.

Usage example:
lipe_find3 /dev/nvme1n1 -path dir-1/* -show-counters
Output:
scanned: 1460
matched: 200

Test-Parameters: trivial
Signed-off-by: Vitaliy Kuznetsov <vkuznetsov@ddn.com>
Change-Id: Ibcd22a94e01ea6322cd38fd414e6058314aac8ef
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52937
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-8441 lipe: lamigo fix compatibility with older lustre

- lfs mirror extend may dump help text either to stdout or stderr.
  Lamigo will adopt both cases
- Make correct exit from loop when ssh remote session fails
- Skip hot-pools tests 75a,75b,75c if lfs mirror extend
  does not support --stats-interval
- Minor code fixes

Test-Parameters: trivial testlist=hot-pools
Signed-off-by: Alexandre Ioffe <aioffe@ddn.com>
Change-Id: Iba035043bc4868e7898f3739d03607d5d3e21574
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52898
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn111

New tag 2.14.0-ddn111

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I95b2ba340e1e65cee4b661b8d986d365863fa633

EX-8421 llite: remove LBUG() from ll_readpage()

This LBUG() has been hitting sometimes in sanity-PCC, which
means EX-8421 is not completely fixed.

Until we can fully sort out EX-8421, we don't want to have
this LBUG enabled on customer systems.  The underlying bug
has been present for some time and the first attempt at an
EX-8421 fix improves the situation.

So, remove the LBUG(), with the intent of putting it back
later once EX-8421 is fixed for real.

Fixes: 3a701bf587 ("EX-8421 llite: disable kernel readahead for pcc mmap")
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I21f951d38f67b37626f33068d2a4b64377f4c46a
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52858
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-8236 pcc: abort in-progress attach by PCC detach command

A user may want to abort in-progress attach for some purposes such
as freeing space for PCC backend.
To support this operation, we add an "abort" option for PCC detach
command to abort the in-progress attach.

Change-Id: I49fb1c42838f8d7e9728a5c4c6f3d60e959b233b
Signed-off-by: Qian Yingjin <qian@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52375
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-8027 pcc: add --wait option for PCC detach command

This patch adds "--wait" option for PCC detach command.
PCC detach with this option must wait for in-progress attach on
this file finished.
Add sanity-pcc/test_107 to verify it.

Change-Id: I63d52d514884b15a7b534d0f03deee441a12d3f1
Signed-off-by: Qian Yingjin <qian@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52374
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-8027 pcc: wait for attach finished for detach command

When detach a file from a PCC backend, this file may be still
attaching state. At this time, we add a flag to wait for the
attach finished (PCC_DEATCH_FL_ATTACHING_WAIT). After that, retry
the detach.

Change-Id: If85d95be744e3f7d6a07f880e78de5b68b579ed6
Signed-off-by: Qian Yingjin <qian@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52373
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7849 tests: add "+quota" to racer

Set temporarily "quota" debug level in racer.sh. This
should be removed after the debugging.

Test-Parameters: trivial
Signed-off-by: Sergey Cheremencev <scherementsev@ddn.com>
Change-Id: Ie03b8f51bd3298d272d78447f5e6ff6969901886
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52831
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17138 enc: prefer specific crypto engines

Some ciphers provided by external accelerators might register under
the generic cipher name. To avoid using them with Lustre, prefer the
AES-NI variant implemented directly in the CPU. And fallback to the
generic cipher if AES-NI is not available.

Introduce a new libcfs kernel module parameter named
'client_encryption_engine' to give the ability to choose the cipher.
By default its value is 'aes-ni', which makes Lustre look for the
AES-NI cipher first. This parameter can be set to 'system-default'
whic makes Lustre pick the generic cipher.

Lustre-change: https://review.whamcloud.com/52477
Lustre-commit: 056eb9dcc0d5f80451c400342d54037f6de24bd9

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I8b00f1c3c8dcf11c58e9f40a410b57b2f255e642
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52828
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16623 lod: handle object allocation consistently

Consistently handle the various OS_STATFS_* flags that indicate
an OST or MDT is full or otherwise marked ineligible for use.

Fix lod_statfs_check() so it skips MDTs with OS_STATFS_ENOINO
for allocating dir stripes instead of only checking OST targets.

In the LOD code, ltd_active=0 indicates that the device is not
usable for new object allocations for a variety of reasons. That
includes out of space or inodes, read-only, max_create_count=0,
or disconnected export, not *only* that the OSP is disconnected
from the OST as with imp_deactive.  Targets marked ltd_active=0
will not be counted in ld_active_tgt_count, so these OSTs will
not count toward stripe_count for stripe_count=-1 files.

Set flags = LOD_USES_DEFAULT_STRIPE in lod_qos_prep_create() for
stripe_count = -1 layouts and pass it to lod_stripe_count_min()
to avoid use of *all* OSTs when free space is imbalanced or OSTs
are not available, and be happy with allocations on 3/4 of OSTs.
It looks like this functionality was missed when object allocations
transitioned from the LOV to LOD module.  Put the LOV_USES_* into
an enum and rename to LOD_USES_* for consistency with current code.

Apply the lod.*.max_stripe_count limits to PFL components as well
as plain file layouts in lod_comp_entry_stripe_count().

Rename ltd_connecting to ltd_discon, since there is no guarantee
that this target is actually *connecting*, only that it is currently
disconnected.  Use ltd_discon in places that checked ltd_active to
decide if the OSP was disconnected from the OST, which shouldn't be
skipped just because the OST is full or has creates disabled.

Lustre-change: https://review.whamcloud.com/50250
Lustre-commit: ced540165ef573570b8a8cba6e43f79e5fc6539f

LU-16981 lod: update llc_stripe_count after ost inactive

If an OST gets deactivated while lod_ost_alloc_qos() is trying to
allocate stripes for a file create, then normally this is caught and
EAGAIN is returned which causes the lod_comp->llc_stripe_count to
get updated to accurately reflect the stripe count. But there is a
race condition and if the OST is deactivated after the call to
ltd_qos_is_usable() but before the stripes are allocated, then
updating the stripe count never occurred.

This causes an LBUG later in lod_striped_create() because fewer
stripes are allocated than the number in llc_stripe_count so it
finds a stripe that is NULL.

The solution is to properly update lod_comp->llc_stripe_count when
the number of stripes created is less than expected.

Lustre-change: https://review.whamcloud.com/51759
Lustre-commit: 78336aa166f4a7a0128a5891c747eecf26ff9565

Test-Parameters: testlist=sanity env=ONLY=27V,ONLY_REPEAT=100
Signed-off-by: Thomas Bertschinger <bertschinger@lanl.gov>
Fixes: 7b124fef76 ("LU-4277 lod: handle os_state as a flag, check READONLY")
Fixes: 5b147e47de ("LU-11115 lod: skip max_create_count=0 OST in QoS and RR algorithms")
Fixes: c7f2e70a27 ("LU-1303 lod: QoS allocation policy")
Fixes: c1d0a355a6 ("LU-12624 lod: alloc dir stripes by QoS")
Fixes: 3c9580931d ("LU-9162 lod: option to set max stripe count per filesystem")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Sergey Cheremencev <scherementsev@ddn.com>
Change-Id: Ifb9443fe6c80b4d7f82b442060db7ac8423ebbe5
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52729
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16334 llite: update statx size/ctime for fallocate

In the VFS interface ->fallocate(), it should update i_size and
i_ctime returned by statx() accordingly when the file size grows.

Add sanity/150h.

fallocate() call does not update the attributes on MDT.
We use STATX with cached-always mode to verify it as it will not
send Glimpse lock RPCs to OSTs to obtain file size information
and use the caching attributes (size) on the client side as much
as possible.

Lustre-change: https://review.whamcloud.com/49221
Lustre-commit: 51851705e936b2dbc9cf141ecf7ab4e3be04333a

Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: Ib8128892222a01cd00250c704328bd13cfb12e2d
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52736
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-8353 csdc: remove holes from struct ll_compr_hdr

This patch reorganizes struct ll_compr_hdr to remove
alignment holes.

Test-Parameters: trivial
Change-Id: I59800b00e3a17972d621bae21ba06509a39b1036
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52753
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7331 sec: disable compression for encrypted files

In case a read-modify-write io pattern is carried out on a compressed
file, it has to be handled on server side.
But because encryption cannot be done on server side for security
reasons, we are not able to handle that kind of io pattern if the file
is encrypted + compressed.
So just disable compression for all encrypted files.

Fixes: eb70ba19e9 ("EX-7331 sec: add support for encryption plus compression")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I84881fb1235f015d022751d4cce2d43a7231c2b4
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52746
Reviewed-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15404 ldiskfs: fix truncate during setxattr for el7.9

Backport the ext4-delayed-iput.patch to rhel7.9 kernels so the
delayed osd-ldiskfs truncate can use s_misc_wq consistently.

This moves the call to the final iput in a separate thread.
This way, setxattr transactions will never be split into two.
Since the setxattr code adds xattr inodes with nlink=0 into the
orphan list, old xattr inodes will be properly cleaned up in
any case.

Lustre-change: https://review.whamcloud.com/51335
Lustre-commit: 471ce3d95651ca06209a76973cae3bbdb5b6aa2f

Test-Parameters: trivial serverdistro=el7.9
Fixes: e239a14001 ("LU-15404 ldiskfs: truncate during setxattr leads to kernel panic")
Change-Id: Idd70befa6a83818ece06daccf9bb6256813ebbe5
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andrew Perepechko <andrew.perepechko@hpe.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52809
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>

LU-11912 tests: fix racing in force_new_seq_all

We run force_new_seq in parallel to reduce time spent
on consuming precreated objects.

However this could be racy when multiple MDTs are on
the same MDS, a task could finish for one MDT early
and reset the fail_loc to 0 on MDS while other tasks
are still working on other MDTs.

Replace OBD_FAIL_OSP_FORCE_NEW_SEQ with a new param
prealloc_force_new_seq for osp, so we can control
the seq rollover individually for each osp device.

Lustre-change: https://review.whamcloud.com/52801
Lustre-commit: TBD (from af6dcd597d7f5134de553349c05091e51e0f3dd6)

Change-Id: I52dbd550564ca628a8a85c42951694d58b2b93a9
Fixes: 656fc937cf ("LU-11912 tests: consume precreated objects in parallel")
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52802
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-16966 osd: take trunc_lock for fallocate

as fallocate may need few transactions (or transaction restarted)
we have to avoid any concurrent writes/truncates on this object
until fallocate supports 'restart-from-beginning' - first stop the
transaction, then release the lock, then repeat again (like
the write path does).

Lustre-change: https://review.whamcloud.com/52264
Lustre-commit: 51529fb57f85210e292a15c882cf25a4689ea77d

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I0bf38b1886fbf24656b45fe0f87fcbad2227672a
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52709
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17040 scrub: inconsistent item

When OI does not include the fid, scrub will attempt to
fix it with zero inode number. There is
low chance that fid would be found during full inode
scan. But inode scan requires an empty inconsistent
list. With repeated EINPROGRESS replies, inconsistent list is
always not empty.

Move fid with zero inode numbers to stale list.

1 scrub fix to print real OI resurect and
skip not related
2 out_handle debug for dt_locate() fid failed
3 debug for out requests when it was interrupted

Lustre-change: https://review.whamcloud.com/51997
Lustre-commit: 461e3867ea11240c77ccd1bb71a3758506cf882e

HPE-bug-id: LUS-10780
Signed-off-by: Alexander Boyko <alexander.boyko@hpe.com>
Change-Id: Iad9e9cba90b4648eb0fe8fa6c99984ada60fde70
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52839
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-17144 mdt: set dmv by setxattr

Client side: convert setxattr("trusted.dmv") to "setdirstripe -D", as
will help restore directory default LMV from backup.

Server side: add a tunable to enable setxattr("trusted.dmv"), it can
be turned on by "lctl set_param -n mdt.*.enable_dmv_xattr=1". It's
off by default. Since empty buffer can be set by setxattr, add check
in server code to avoid crash.

Add sanity 413j.

Lustre-change: https://review.whamcloud.com/52510
Lustre-commit: 1ebe91ec0ab55f686a730d448e7a1ba2ce99639a

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I27d784998a9c4a182b4fffb8b06c84e9d9190919
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52511
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17136 ldiskfs: increase max extent tree depth

increase max extent tree depth to 8.
this is an workaround until LU-16843 ready

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Ie1b6bd64ff6d5179b47b6a537c6b9f85670c3f69
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52758
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17193 build: fix gcc-12 compiler warnings

A few instances of QCTL_COPY() should be QCTL_COPY_NO_PNAME()
as the zero-length array to hold the pool name is not
allocated in these cases.

Lustre-change: https://review.whamcloud.com/c/fs/lustre-release/+/52687
Lustre-commit: 1b0de05f81372eeda9a2a38142553ead7e88a431

Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I72bda8b46c51dbd42fb42bf569ba29572526acfe
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52834
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17151 tests: increase sanity/411b memory limit

This test fails most of the time when run using
arm clients. It seems like the cgroup memory limit
was increased in a past revision for a similar issue.
Increase it a bit more for aarch64. Increase it a
smaller amount for x86.

Also, add some better logging for some other issues.

There's likely a better fix for this, but hopefully
this will let the test pass and provide some value
without having to do a full revert.

Lustre-change: https://review.whamcloud.com/52610
Lustre-commit: 0e878390e1c8c5883bccd01758392eaa16a67f31

Fixes: 8aa231a99 ("LU-16713 llite: writeback/commit pages under memory pressure")
Test-Parameters: trivial
Test-Parameters: testgroup=review-ldiskfs-arm testlist=sanity env=ONLY=411b,ONLY_REPEAT=50
Test-Parameters: clientdistro=el8.7 testlist=sanity env=ONLY=411b,ONLY_REPEAT=50
Test-Parameters: clientdistro=el9.1 testlist=sanity env=ONLY=411b,ONLY_REPEAT=50
Signed-off-by: Timothy Day <timday@amazon.com>
Change-Id: If850077c0d7f6466082433776d370d24eee9736c
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52610
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Qian Yingjin <qian@ddn.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52838

LU-16713 llite: writeback/commit pages under memory pressure

Lustre buffered I/O does not work well with restrictive memcg
control. This may result in OOM when the system is under memroy
pressure.

Lustre has implemented unstable pages support similar to NFS.
But it is disabled by default due to the performance reason.

In Lustre, a client pins the cache pages for writes until the
write transcation is committed on the server (OST) even these
pinned pages have been finished writeback. The server starts
a transaction commit either because the commit interval (5
second, by default) for the backend storage (i.e. OST/ldiskfs)
has been reached or there is not enough room in the journal
for a particular handle to start. Before the write transcation
has been committed and notify the client, these pages are
pinned and not flushable in any way by the kernel.
This means that when a client hits memory pressure there can
be a large number of unfreeable (pinned and uncommitted) pages,
so the application on the client will end up OOM killed because
when asked to free up memory it can not.
This is particularly common with cgroups. Because when cgroups
are in use, the memory limit is generally much lower than the
total system memory limits and it is more likely to reach the
limits.

Linux kernel has matured memory reclaim mechanism to avoid OOM
even with cgroups.
After perform dirtied write for a page, the kernel calls
@balance_dirty_pages(). If the dirtied and uncommitted pages
are over background threshold for the global memory limits or
memory cgroup limits, the writeback threads are woken to perform
some writeout.
When allocate a new page for I/O under memory pressure, the
kernel will try direct reclaim and then allocating. For cgroup,
it will try to reclaim pages from the memory cgroup over soft
limit. The slow page allocation path with direct reclaim will
call @wakeup_flusher_threads() with WB_REASON_VMSCAN to start
writeback dirty pages.

Our solution uses the page reclaim mechanism in the kernel
directly.
In the completion of page writeback (in @brw_interpret), call
@__mark_inode_dirty() to add this dirty inode which has pinned
uncommitted pages into the @bdi_writeback where each memory
cgroup has itw own @bdi_writeback to contorl the writeback for
buffered writes within it.
Thus under memory pressure, the writeback threads will be woken
up, and it will call @ll_writepages() to write out data.
For background writeout (over background dirty threshold) or
writeback with WB_REASON_VMSCAN for direct reclaim, we first
flush dirtied pages to OSTs and then sync them to OSTs and force
to commit these pages to release them quickly.

When a cgroup is under memory pressure, the kernel asks to do
writeback and then it does a fsync to OSTs. This will commit
uncommitted/unstable pages, and then the kernel can free them
finally.

In the following, we will give out some performance results.
The client has 512G memory in total.
1. dd if=/dev/zero of=$test bs=1M count=$size
I/O size 128G 256G 512G 1024G
unpatch (GB/s) 2.2 2.2 2.1 2.0
patched (GB/s) 2.2 2.2 2.1 2.0
There is no preformance regession after enable unstable page
account with the patch.

2. One process under different memcg limits and total I/O
size varies from 2X memlimit to 0.5 memlimit:
dd if=/dev/zero of=$file bs=1M count=$((memlimit_mb * time))
memcg limits 1G 4G 16G 64G
2X memlimit (GB/s) 1.7 1.6 1.8 1.7
1X memlimit (GB/s) 1.9 1.9 2.2 2.2
.5X memlimit(GB/s) 2.3 2.3 2.2 2.3
Without this patch, dd with I/O size > memcg limit will be
OOM-killed.

3. Multiple cgroups Testing:
8 cgroups in total each with memory limit of 8G.
Run dd write on each cgrop with I/O size of 2X memory limit
(16G).
17179869184 bytes (17 GB, 16 GiB) copied, 12.7842 s, 1.3 GB/s
17179869184 bytes (17 GB, 16 GiB) copied, 12.7889 s, 1.3 GB/s
17179869184 bytes (17 GB, 16 GiB) copied, 12.9504 s, 1.3 GB/s
17179869184 bytes (17 GB, 16 GiB) copied, 12.9577 s, 1.3 GB/s
17179869184 bytes (17 GB, 16 GiB) copied, 13.4066 s, 1.3 GB/s
17179869184 bytes (17 GB, 16 GiB) copied, 13.5397 s, 1.3 GB/s
17179869184 bytes (17 GB, 16 GiB) copied, 13.5769 s, 1.3 GB/s
17179869184 bytes (17 GB, 16 GiB) copied, 13.6605 s, 1.3 GB/s

4. Two dd writers one (A) is under memcg control and another
(B) is not. The total write data is 128G. Memcg limits varies
from 1G to 128G.
cmd: ./t2p.sh $memlimit_mb
memlimit dd writer (A) dd writer (B)
1G 1.3GB/s 2.2GB/s
4G 1.3GB/s 2.2GB/s
16G 1.4GB/s 2.2GB/s
32G 1.5GB/s 2.2GB/s
64G 1.8GB/s 2.2GB/s
128G 2.1GB/s 2.1GB/s

The results demonstrates that the process with memcg limits
nearly has no impact on the performance of the process without
limits.

Lustre-change: https://review.whamcloud.com/50544
Lustre-commit: 8aa231a994683a9224d42c0e7ae48aaebe2f583c

Test-Parameters: clientdistro=el8.7 testlist=sanity env=ONLY=411b,ONLY_REPEAT=10
Test-Parameters: clientdistro=el9.1 testlist=sanity env=ONLY=411b,ONLY_REPEAT=10
Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I7b548dcc214995c9f00d57817028ec64fd917eab
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52527
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-8362 scripts: Improve estimated ratio

ll_compression_scan does not take in account the size of the
sampled files, this might lead to uncorrect estimated ratio for non
homogeneous file.

This patch apply the compression ratio estimated with the sampled data
and applies it to the entire file size, assuming the file will have
the same compression ratio across it.

Test-Parameters: trivial
Signed-off-by: Raphael Druon <rdruon@ddn.com>
Change-Id: Ic4a26460e17c666b9edf4c0d8d450a06fad5920f
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52759
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16837 lov: NULL dereference in lov_delete_composite

commit 14ed4a6f8f retroduced the issue fixed by commit
5da049d9ef ("LU-14389 lov: avoid NULL dereference in cleanup), this
patch makes the fix cover the new case added by 14ed4a6f8f.

Lustre-change: https://review.whamcloud.com/52826
Lustre-commit: TBD (from 10b4a14b389cb00e1033e2f49e3d1f5a554b259a)

Fixes: 14ed4a6f8f ("LU-16837 llite: handle unknown layout component")
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I4a2b72e21139b60519ed523b4851723c91f523c1
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52827
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16827 obdfilter: Fix obdfilter-survery/1a

local_node() under test-framework is used
to determine if the node is remote or local
local_node() returns "true" if the node is
local. Else for remote node it return "false"

This patch fixes obdfilter/1a test case which
which was making reverse logic call to
local_node() to determine remote/local node

This patch modifies local_node() to return
"true"/"false" instead of 0/1

This patch also replaces lctl with $LCTL

Lustre-change: https://review.whamcloud.com/51035
Lustre-commit: 91a3b286ba57bb491b5c17600d7cec9e516a428f

Test-Parameters: testlist=obdfilter-survey,sanity-lipe-scan3,sanity-lipe-find3
Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Signed-off-by: Vitaliy Kuznetsov <vkuznetsov@ddn.com>
Change-Id: I7bcb483975ec46d9847e0050e5a1f22f68663c80
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52800
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15461 test: add pool quota check

The test_79 in sanity-quota needs quota pool support.
The removal of the "stop file" is also improved not to
trigger the test error if it has been deleted.

Lustre-change: https://review.whamcloud.com/52737
Lustre-commit: TBD (from a4b3cd91ae157a63644350769ebb248f21dd6eac)

Test-Parameters: trivial testlist=sanity-quota
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Change-Id: I4acd36e61faf4259c2821293ffb7913d4cca76bd
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52659
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-17220 kernel: update RHEL 7.9 [3.10.0-1160.102.1.el7]

Update RHEL 7.9 kernel to 3.10.0-1160.102.1.el7.

Lustre-change: https://review.whamcloud.com/52819
Lustre-commit: TBD (from 1feea616fd7addf842afdc836e7f32686ea159ae)

Test-Parameters: trivial clientdistro=el7.9 serverdistro=el7.9

Change-Id: Ifc56766dedf055dc3762e200835beb220fd63afb
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52843
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17221 kernel: update SLES15 SP4 [5.14.21-150400.24.92.1]

Update SLES15 SP4 kernel to 5.14.21-150400.24.92.1 for Lustre client.

Lustre-change: https://review.whamcloud.com/52820
Lustre-commit: TBD (from 92cf005d01e327e53bd312b411211ed2f1d827b9)

Test-Parameters: trivial clientdistro=sles15sp4 testlist=sanity

Change-Id: Id82d0ce48179df1f12dc367cced8cf84e1b918d9
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52825
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17222 kernel: update SLES15 SP5 [5.14.21-150500.55.31.1]

Update SLES15 SP5 kernel to 5.14.21-150500.55.31.1 for Lustre client.

Lustre-change: https://review.whamcloud.com/c/fs/lustre-release/+/52821
Lustre-commit: TBD (from b2159275aaf3595776ae89b3efeda4ec8bde14ff)

Test-Parameters: trivial clientdistro=sles15sp5 testlist=sanity

Change-Id: I5719e8c79740a58223b2e0bea6f6b269f281968a
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52824
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16868 tests: skip conf-sanity/32 in interop

Do not run conf-sanity.sh test_32* in interop testing.  Otherwise,
it is possible that the version of the test script running on the
client does not perform the upgrades with the right steps needed
for remote servers that are running a different version.

Lustre-change: https://review.whamcloud.com/52835
Lustre-commit: TBD (from 6368e97e593707d2ae1423dcb41c7f001f1d2152)

Test-Parameters: trivial testlist=conf-sanity env=ONLY=32a
Test-Parameters: testlist=conf-sanity env=ONLY=32a serverversion=EXA5
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Iabe1469a87d58c49e3c38b76ab18f8997f3ebbe5
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52836
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>

LU-13453 osd-ldiskfs: do not leak inode if OI insertion fails

osd_create() should destroy just created inode if OI insertion
fails.

also fixes lustre_index_restore() to drop nlink for object to
be removed.

the patch adds two tests:
 - ENOSPC on OI insertion
 - ENOSPC on .. insertion, i.e. directory block allocation

Lustre-change: https://review.whamcloud.com/38235
Lustre-commit: e45e8a92a2ecab742b3680716a55aaa1d9827057

Test-Parameters: testlist=sanity-scrub mdscount=2 mdtcount=4
Test-Parameters: testlist=sanity-scrub mdscount=2 mdtcount=4
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I2a5db657c7dab54b8dc2c50bc29365d5ee754a2e
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52846
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn110

New tag 2.14.0-ddn110

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Iab6709369cc3d4e50fd799fcd6db3796202905e7

LU-14659 test: improve generate_uneven_mdts() in sanity.sh

Improve generate_uneven_mdts() in several places:
1. set qos maxage to 1, so the result is up to date, and avoid filling
   up MDT.
2. fill MDT with files of size 64K other than 1M, so MDT imbalance is
   quicker to achieve.
3. when checking minimum imbalance after test, lookup max value from
   the result, other than by index stored before directory creation,
   because the result is dynamic if several MDTs have almost the same
   free space and inodes.

Lustre-change: https://review.whamcloud.com/44649
Lustre-commit: d45be79a069f527657c1ce91630183031ea42b27

Test-Parameter: trivial mdscount=2 mdtcount=4 testlist=sanity
Fixes: 233344d451e ("LU-13417 test: generate uneven MDTs early for sanity 413")
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I2807101ff632404e25fdb640840d83d1991c88d9
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52751
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7507 test: fix sanity-lfsck test_8

The backport of LU-15738 is not correct, which may cause test failure.

Fixes: 779c10b7c64 ("LU-15738 test: check lfsck status before starting")
Test-Parameters: trivial testlist=sanity-lfsck env=ONLY=8,ONLY_REPEAT=10
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Ibbc5b576db7d502095257031dd4619fe5103df3b
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52774
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17129 tests: cleanup fileset info on nodemaps

In sanity-sec, fileset info added to nodemaps via 'set_param -P' must
be removed afterwards with 'set_param -P -d', otherwise those commands
will remain in the llogs.

Lustre-change: https://review.whamcloud.com/52408
Lustre-commit: 16e4383e90f630dfcce20e2675bb887471b6a9c8

Test-Parameters: trivial mdscount=2 mdtcount=4 testlist=sanity-sec env=ONLY=27
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I75bd0dc263f71c7f5d9ece028cc038eb1f2ca9a4
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52706
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15564 osd: add allocation time histogram

add block mapping/allocation histogram to brw stats to debug
mballoc related issues.

$ lctl get_param osd*.*OST*.brw_stats
                           read      |     write
block maps msec        maps  % cum % |  maps        % cum %
1:    1522360 100 100   | 49272  99  99
2:          0   0 100   |    1   0  99
4:          0   0 100   |    1   0  99
8:          0   0 100   |    0   0  99
16:          0   0 100   |    0   0  99
32:          0   0 100   |    0   0  99
64:          0   0 100   |    1   0 100

Lustre-change: https://review.whamcloud.com/46550
Lustre-commit: f97ce54357bc91f7b1285febfc50d6087dd94c13

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I1185386adc64e844de71e25a4e439e493e5e5bc5
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52767
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14927 osd: share brw_stats code between OSD back ends.

Both the ldiskfs and ZFS OSD backend handle brw_stats. With the
stricter GPL requirement ZFS can no longer carry the brw_stats
code. So move the common code to lprocfs_status_server.c as
well as move brw_stats to debugfs as well.

Lustre-change: https://review.whamcloud.com/44690
Lustre-commit: 8a84c7f9c7d65f6f880be6fe4d94fca26a405d81

Change-Id: I294e5df3557552266dd3a02d3bc9844c42c01f60
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52766
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16694 tests: replace resolveip script

The resolveip script can be replaced with a bash one-liner,
using getent and awk.

Lustre-change: https://review.whamcloud.com/50491
Lustre-commit: f8cc7db39dd22fdb6330402a60af7bb30c78449e

Test-Parameters: trivial
Signed-off-by: Timothy Day <timday@amazon.com>
Signed-off-by: Vitaliy Kuznetsov <vkuznetsov@ddn.com>
Change-Id: I207ea011e43b7b236d5082994ffb51654d8d782c
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52781
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-8349 lipe: Fix problem with fallocate in test_306

This patch modifies the fallocate command so that the
size type is specified correctly on all versions.
This should will fix the issue with invalid
length value.

Test-Parameters: trivial
Signed-off-by: Vitaliy Kuznetsov <vkuznetsov@ddn.com>
Change-Id: Ie25dc87aed8ea9c882d9b352b11a58071c9c460d
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52779
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-8421 llite: disable kernel readahead for pcc mmap

Set ra_pages to 0 for PCC files when mmaped, because
otherwise this setting carries through to Lustre and will
cause crashes and possible inconsistencies.  This happens
because the PCC file and Lustre file share a mapping, which
is a weird trick required to have mmap work on PCC.

Add a set of asserts which confirm kernel readahead is
disabled and wasn't used for mmap.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I117042d68fac25158e8141c243acba698cf1930f
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52732
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Qian Yingjin <qian@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-8369 ldiskfs: mballoc to store group in pa

mballoc should store and then use group number from pa,
not calculate it from pa_pstart which change and point to
a subsequent group in the case of full pa consumption.

Fixes: f36eda6a1e ("LU-10026 osd-ldiskfs: use preallocation for dense writes")
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I5e3c5c7e51fe9688f4d847a9ea0591486d8975d8
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52754
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn109

New tag 2.14.0-ddn109

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I6d46cc9e8edf9362e5a639644da4a9e205b2932a

LU-17182 utils: pool_add send OSTs in one batch

'lctl pool_add' command sends all requests in one batch
then checks results. In this way, the command won't take
too long time if the OSTs are specified in command line
one by one.

Lustre-change: https://review.whamcloud.com/52654
Lustre-commit: TBD (from 7a9b37c84921687ab8ac0c765aab6b5a6b339468)

Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial
Change-Id: Ibd6e7ed5104e100d44c5f4288a25e7378cd9cfe8
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52719
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn108

New tag 2.14.0-ddn108

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I6c0a1e6da1605c4c569c2bfbb7543c318fd96327

RM-620 build: New tag lipe-2.34

New tag lipe-2.34

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I5ace0da818b9ffb5bb52a158bc2e3a17ff65be8a

LU-17202 target: set SB_KERNMOUNT

set SB_KERNMOUNT on server's mountpoint so umount process block
until the server (MDS/OST/MGS) is really umounted and released
the corresponding block device.

Lustre-change: https://review.whamcloud.com/52724
Lustre-commit: TBD (from 005a85f1c17a6c3477786a47dccfb67f2b36cb09)

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I03ac2dc6bb7cd4d93f5a1729fccb976aa1ebd5d4
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52725
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-8245 osc: use correct count

Using the number of bytes in the compressed page creates
gaps in the RDMA, which IB memory registration cannot
accept.

Fix this by always setting count to PAGE_SIZE for
compressed pages and otherwise using the count from the
original source page.  Setting PAGE_SIZE for compressed
pages is valid because client only does compression for
aligned IO, except for the trailing chunk.  For the
trailing chunk, the file size is set on the server, so any
trailing bytes are ignored.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Ied89d3ac328fb6020079392f5a8812ad5637b4a4
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52550
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-11912 tests: force new seq in runtests

If seq rollover happens during runtests/1,
the new seq on OST will consume some space and
this will fail the free space check.

Force a new seq before running test case to
prevent this.

Lustre-change: https://review.whamcloud.com/52741
Lustre-commit: TBD

Change-Id: I7bb1156127eb423889626bf84bc6c87dd68e6ece
Test-Parameters: trivial testlist=runtests
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52743
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15740 tests: scale fs_log_size by OSTCOUNT

The fs_log_size "free space skew" was being scaled by MDSCOUNT,
but in fact this parameter is only ever used to compare the OST
free space usage, so the OSTCOUNT should be used when scaling it.

It is likely that the skew is actually caused by blocks allocated
by OST object directories and not llogs (no llogs used on OSTs for
many years), but it isn't worthwhile to rename the function.

Lustre-change: https://review.whamcloud.com/50419
Lustre-commit: fabec6f2cb39950a2f208567dac716e21880fa9f

Test-Parameters: trivial testlist=replay-single env=ONLY="20b 89"
Test-Parameters: testlist=runtests clientdistro=ubuntu2204
Test-Parameters: testlist=replay-ost-single env=ONLY="6 7"
Test-Parameters: testlist=sanity-sec env=ONLY="16-22
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I97f05b10fa7ec367534b5bdce09feae5e93ebbe5
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52742
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-13748 mdt: remove LASSERT in mdt_dump_lmm()

Change LASSERT() in mdt_dump_lmm() into a CERROR(), since this
function is normally used for debugging and it doesn't help to
crash the MDS just when it complaining about some file layout.

Add sanity test_27Cg that triggered this initially.

Lustre-change: https://review.whamcloud.com/50532
Lustre-commit: 97d29eb800e8d9faba04f0744376cb50b239e2e9

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I27f0b4489de48d6f197e0143589d76428d85e549
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52728
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-17084 lod: fix comparision in lod_striping_load()

in if (rc > sizeof(struct lmv_foreign_md)) the latter
is unsigned and gcc treats rc (which is defined as int
and can be negative to encode an error) as unsigned.
this way -EIO becomes greater than the size of the
structure.  make sizeof() signed to avoid confusion.

Lustre-change: https://review.whamcloud.com/52265
Lustre-commit: ce54b5281c3172401ce4526a4de65d2d584fa0e7

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Signed-off-by: Xing Huang <hxing@ddn.com>
Change-Id: Ie6735578649e397ed05b6951fab941f97051305b
Reviewed-by: xinliang <xinliang.liu@linaro.org>
Reviewed-by: Mikhail Pershin <mpershin@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52721
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-12896 gss: key can be unlinked when timeout expires

The key associated with a GSS context could appear to be already
unlinked when the upcall timeout expires. In this case, do not assert
but report this case with a warning message.

Lustre-change: https://review.whamcloud.com/52473
Lustre-commit: 4c6290087b3bf0838a00de8f8b1cfde86efbc409

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I714af3a1ce54648c4ba29ef13015f9291de52765
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52705
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-8400 lnet: revert LU-13485

This reverts commit 889a8c41b9799256f9c1f54abc221b7c3d1ed3a8.
("LU-13485 lnet: Parallel configure tests for lnet")
The commit breaks the rhel8.8 build with implicit declaration of
ib_get_dma_mr.

Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: I77f6ddb46d49b27fa49594026223df54e423779d
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52718
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn107

New tag 2.14.0-ddn107

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ida621bbb0a593a5a18123a95cf3cba4fb1340a2c

EX-8349 lipe: Fix problem with fallocate in test_306

This patch removes the check every time a file is created,
and adds a general check for the presence of created files.

Test-Parameters: trivial testlist=sanity-lipe-scan3
Signed-off-by: Vitaliy Kuznetsov <vkuznetsov@ddn.com>
Change-Id: I7d7e0c9b0e6517853bb3b77e8f938bbd877b3003
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52590
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13575 lnet: Ensure round robin selection of peer NIs

Use the peer net sequence number to set the peer NI sequence number to
ensure round robin selection of peer NIs on each peer net.

Lustre-change: https://review.whamcloud.com/45004
Lustre-commit: c51763948abfdbdc8e3f3ea7e73f2632320a095a

HPE-bug-id: LUS-10349
Test-Parameters: trivial
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I1fa14ad675ead4ae2c5b1d4edad250caa4498df2
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52471
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>

LU-13575 lnet: Ensure round robin selection of local NIs

Use the net sequence number to set the NI sequence number to ensure
round robin selection of NIs on each net.

Lustre-change: https://review.whamcloud.com/45003
Lustre-commit: a18c4a16246e6185919eda805eca52772bbc3efe

Test-Parameters: trivial
HPE-bug-id: LUS-10349
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I6ce0b088fcad6312186e6fbad4ab14283aee55eb
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52470
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>

LU-9121 lnet: select best peer and local net

Select the healthiest and highest priority peer and local net when
sending a message.

Lustre-change: https://review.whamcloud.com/34352
Lustre-commit: dff6587805ddad212ab48e5bedacbc7846542b7b

Test-Parameters: trivial testlist=lnet-selftest,sanity-lnet
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: I42717e7fdc3226c6faa7c59c713f18422e27f2e5
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52444
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>

LU-17088 dom: don't create different size DOM component

Multiple DOM components are allowed in diffrent mirror but they
must be of the same size, mirror extend should check this restraint.

Fix another glitch in lov_init_composite() where dom_size is used
as a __u64 value but declared as boolean.

Lustre-change: https://review.whamcloud.com/52269
Lustre-commit: e2539c0667525aff8d985d018c4ed077d95ba882

Fixes: 44a721b8c1 ("LU-11421 dom: manual OST-to-DOM migration via mirroring")
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Ia0d08c697dbeeb3aa8d20d9849226afa06360012
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mikhail Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52601
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14637 flr: get rid of excluding dom+flr support test

Now that DoM+FLR are supported, fix the tests that expect this
combination of features on a file to fail.

Lustre-change: https://review.whamcloud.com/44185
Lustre-commit: 4b52ea1d30b45900787271c4c035fad124abf34a

Fixes: 0bff64be320fd ("LU-9771 flr: to not support dom+flr for phase 1")
Fixes: 44a721b8c1063 ("LU-11421 dom: manual OST-to-DOM migration via mirroring)
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I9fc76e797e469744107e5d0453b78729226be0ee
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52600
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-8344 lipe: Update manual page

This small patch expands the explanations for some
commands with information from the development files.
Adds one "todo-list.md" file instead of different files
with similar information.

Test-Parameters: trivial
Signed-off-by: Vitaliy Kuznetsov <vkuznetsov@ddn.com>
Change-Id: I0a3ebda49525d62cd6ca398f12601e588dc2dd42
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52589
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Colin Faber <cfaber@ddn.com>
Reviewed-by: Alexandre Ioffe <aioffe@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17015 gss: support large kerberos token for rpc sec init

If the current Kerberos setup is using large token, like when PAC
feature is enabled for Kerberos, authentication can fail due to server
side unable to exchange token between kernel and userspace.
This limitation is inherent to the sunrpc cache mechanism, that can
only handle tokens up to PAGE_SIZE.

For RPC sec init phase, use Lustre's upcall cache mechanism
instead of deprecated kernel's sunrpc cache. The upcall calls a new
userspace command 'l_getauth', that fowards the sec init request to
the lsvcgssd daemon via Unix domain sockets.

Lustre-change: https://review.whamcloud.com/52224
Lustre-commit: TBD (from 8acd059ee2b8d1e4c48c3d9dbb380bca75e1b3be)

Test-Parameters: kerberos=true testlist=sanity-krb5
Change-Id: I709cd79894a5a13fc4cdfab2109c86f2230db3b8
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52653
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17015 build: rework upcall cache

EX-4333 introduced in upcall_cache.c a dependency on md_object.h for
struct lu_ucred. Rework files to move this dependency to a differnt
file, so that upcall_cache.c can be built in client-only mode.

Fixes: fb0082bba1 ("EX-4333 sec: support supplementary groups from client")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I4bcc7e07a4f4886c5994d17cbef72ea09eb1be1d
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52670
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17015 gss: bump token buffer size to 16KiB

A 4 KiB large buffer is not enough to hold the GSS token under some
circumstances. So bump GSS_CTX_INIT_MAX_LEN value to 16 KiB.

Lustre-change: https://review.whamcloud.com/52475
Lustre-commit: TBD (from 43a540207da0198cc9c45b3c6312c555702b56cb)

Fixes: 9758129177 ("LU-17015 gss: support large kerberos token on client")
Test-Parameters: trivial kerberos=true testlist=sanity-krb5
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I8e72f1447593d2bf2ae537fcc920ceee20e93c09
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52628
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-13485 ldiskfs: Parallel configure tests for ldiskfs

Transform the compile tests in ldiskfs to run in parallel

Lustre-change: https://review.whamcloud.com/38351
Lustre-commit: 3774b6afbe3b67e869bb61c9cb212cc37e8705fa

Test-Parameters: trivial
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I3a097ab5cd18b57e9311980d9aa708ed25f58464
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52655
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13485 libcfs: Remove unused iter_type check

The iter_type member check is not used, remove it.

Lustre-change: https://review.whamcloud.com/48091
Lustre-commit: c755373c567090c49589e5aa0d3134847d4b952e

Test-Parameters: trivial
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I48d536a27738e73314feb88317d41d8479c72528
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52683
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-13485 lnet: Parallel configure tests for lnet

Transform the compile tests in lustre-lnet to run in parallel
Also fixes the generated Makefile to work with MOFED and in-kernel
OFED.

configure build times on an 8 core 8G vm vs current serial:

             serial      parallel
            --------     --------
    real    8m27.824s    1m28.375s
    user    5m29.448s    2m11.558s
    sys     3m48.258s    0m51.763s

Lustre-change: https://review.whamcloud.com/38368
Lustre-commit: fc84caa81b7fb9d27e82229d39f046e83b5ebb7e

Test-Parameters: trivial
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I4f0cb8584e1c3149ec3f005dd55fed0c47b50472
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52678
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-8038 csdc: sending compression info to server

Client fills in layout compression info into obdo and passes it
to server.

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Ieb5d7b3609da41f35f8622ed6116f19ce7567ddb
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52669
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn106

New tag 2.14.0-ddn106

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ibc557d9c0a56d9994b7fed147e7e183a1b5528db

RM-620 build: New tag lipe-2.33

New tag lipe-2.33

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I638f4b593856e9e8c15b180328be9cc23ec3365e

LU-16837 llite: handle unknown layout component

If lustre client encounters unknown layout component pattern in
a mirror file, this patch makes client mark this mirror as invalid
and skip it.

Lustre-change: https://review.whamcloud.com/51060
Lustre-commit: 14ed4a6f8f231fe94392906f991a32f07e7d7883

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Ie5f44212ab96bdc706cc5a9e11f330234fc01069
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51061
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7948 utils: lamigo to track mirror progress

pass --stats to lfs mirror/resync commands and then read
lfs's output over ssh channel.
this way we can keep ssh channel alive and interrupt
replication if it doesn't report progress.

the very first time agent is used lamigo checks whether
agent's lfs utility supports stats.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Iee5b43cb85dae62550d74667b16e00336f1bf52f
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/51744
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexandre Ioffe <aioffe@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-9121 lnet: Select NI/peer NI with highest prio

Modify the selection algorithm to select the highest priority
local and peer NI. Health always trumps all other selection
criteria

Lustre-commit: 3fc2e0e0b3c8353a8fecc6d127ee55d255d7acb7
Lustre-change: https://review.whamcloud.com/34351

Test-Parameters: trivial testlist=lnet-selftest,sanity-lnet
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: I487a706f4da30311d0bd59fe03f72dbe68a52425
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52289
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-9121 lnet: foundation patch for selection mod

Add the priority and preferred NIDs fields in the lnet_ni,
lnet_net, lnet_peer_net and lnet_peer_ni. Switched
the implementation of the preferred NIDs list to list_head
instead of array, because the code is more straight forward.
There is more memory overhead due to list_head, but these lists
are expected to be small, so I chose code simplicity over memory.

Lustre-commit: 51b2c0f75f727f0562b3145015357cbff5cbb3b5
Lustre-change: https://review.whamcloud.com/34350

Test-Parameters: trivial testlist=lnet-selftest,sanity-lnet
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: I0c75855b736345c25e1604083eee2b65d38ef28d
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52288
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16097 tests: skip quota subtests in interop

Skip subtests in sanity-quota.sh to avoid interop test failures,
backdated to check all new tests since 2.14.0 for completeness.

Test-Parameters: trivial testlist=sanity-quota serverversion=EXA6.1.0
Fixes: 513b1cdbca ("LU-16340 quota: notify only global lqe")
Fixes: d4978678b4 ("LU-15694 quota: keep grace time while setting default")
Fixes: 25a70a88c9 ("LU-13952 quota: default OST Pool Quotas")
Fixes: 188112fc80 ("LU-14300 quota: avoid nested lqe lookup")
Fixes: 8c19365416 ("LU-13971 quota: report Pool Quotas for a user")
Fixes: a4fbe7341b ("LU-14739 quota: nodemap squashed root cannot bypass quota")
Fixes: 789038c97a ("LU-15167 quota: fallocate send UID/GID for quota")
Fixes: c9901b68b4 ("LU-13587 quota: protect qpi in proc")
Fixes: 61ec1e0f2c ("LU-15031 quota: reseed glbe in qmt_lvbo_udate")
Fixes: dfe7d2dd2b ("LU-16341 quota: fix panic in qmt_site_recalc_cb")
Fixes: 862f0baa7c ("LU-15097 quota: stop pool_recalc before killing pool")
Fixes: 61481796ac ("LU-15193 quota: expand QUOTA_MAX_TRANSIDS to 12")
Fixes: a2fd4d3aee ("LU-15880 quota: fix insane grant quota")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Sergey Cheremencev <scherementsev@ddn.com>
Change-Id: Ife8bfd83d0f217c534f3b12b4c9d108d370ed6b7
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52009
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52582

EX-8281 tests: Fix sanity test_56ab when CSDC is enabled

Use /dev/urandom in sanity test_56ab so the data cannot be compressed

Lustre-change: https://review.whamcloud.com/52572
Lustre-commit: TBD  (from 1ce661dd56fb4b6ecc9e909805c6101bbd9c3161)

Test-Parameters: trivial testlist=sanity env=ONLY=56ab

Signed-off-by: Wei Liu <sarah@whamcloud.com>
Change-Id: I0ceb9afcbdc8443b5e04dff486e41621479dbd23
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52501
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Colin Faber <cfaber@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-17152 tests: unmount NFS clients with zconf_umount_clients

This patch fixes cleanup_nfs() to unmount NFS clients by running
zconf_umount_clients(), which can find and kill active processes
that are accessing the NFS mount point so as to avoid the
"device is busy" failure.

The patch also adds racer_on_nfs test into always_except list for
parallel-scale-nfsv4 due to LU-17154.

Lustre-change: https://review.whamcloud.com/52533
Lustre-commit: TBD (from 52a2147e8b0eca74f38b1b87991b53ccf25663cd)

Test-Parameters: trivial testlist=parallel-scale-nfsv4

Change-Id: I37a38502362399540c28e78d1343e768b490ce8b
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52534
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>

EX-3860 llog: extended debug for -ENOTDIR error

Debug patch to catch trace and debug log for -ENOTDIR
error in distribute_txn_cancel_records()

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: Ie1bb7c138282bfa05a2fafcceafdb436d45f28d3
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52394
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-13814 clio: add cp_inode to page allocation

cp_inode can be set correctly during page allocation,
rather than after.  This is a prelude to moving cp_inode to
the osc_transfer_page, but that's better done in a separate
patch.

Lustre-change: https://review.whamcloud.com/52208
Lustre-commit: TBD (from f2afaf4eb10d70c36ad6bdbc2def66bee4fcdc23)
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Signed-off-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Change-Id: I509f6cfbae8e5a6ec6b07c8253d68f6dd2794e59
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52557
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7601 osc: move common CSDC code to the library

CSDC repacks a chunk on the server side in case of the
partial rewrite. There are routines that can be shared
between client and server.

This patch moves common compression code to the
libcfs.

Signed-off-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Change-Id: I824211a3435b0479f7a3b8f08598a5b567b67d3c
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52262
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17087 lmv: update stale tgt statfs every 1 hour

Some tgt statfs may not be initialized upon mount due to network
issues, if the filesystem is imbalanced, these tgts won't be chosen to
create directory because their bavail and ffree are 0.

If MDT is chosen by QoS, update tgt statfs that is one hour overdue,
otherwise check update the statfs of the tgt that is chosen.

Lustre-commit: e262e0ffbe792ae2f8b47ccdafac38a36151a300
Lustre-change: https://review.whamcloud.com/52270

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I06af8b8bd342f66cb794471df3ee0f3b127ffe05
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52560
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17136 ldiskfs: increase max extent tree depth

this is an workaround until LU-16843 ready

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I5829c10888bf32649fe7a7a72c8ee697647a89cc
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52540
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-8253 tests: skip sanity-scrub/4e until fixed

Subtest 18 is failing about 1/5 of sanity-scrub runs, after test_4e
was landed.  Disable test_4e to see if that fixes the issue.

Test-Parameters: trivial
Test-Parameters: testlist=sanity-scrub,sanity-scrub,sanity-scrub
Test-Parameters: testlist=sanity-scrub,sanity-scrub,sanity-scrub
Test-Parameters: testlist=sanity-scrub,sanity-scrub,sanity-scrub
Test-Parameters: testlist=sanity-scrub,sanity-scrub,sanity-scrub
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ia8f1bd9dbf0fdbfabf79b1ead63a0421a8892c82
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52564
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn105

New tag 2.14.0-ddn105

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I28dd2c8ac1550a3c6abb630a32e88423fdbf492e

RM-620 build: New tag lipe-2.32

New tag lipe-2.32

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I4556f6234985973170f94c06c3572dbfce3204c1

EX-8191 lipe: Fix test for --collect-fsize-stats in lipe3

This patch modifies the test for collecting statistics
in lipe3 and corrects:
1. Error getting a username if it doesnt already exist.
2. Error comparing file sizes after changing table
generation rules.
3. Converts the test from reading yaml to reading json
4. Now many files of different sizes are generated
for the test.
5. Now the data for comparison is retrieved from
the ls utility.
6. The test has added a check for creating a user with a
large UID, GID, which checks the availability of reports
for this user.

Test-Parameters: trivial testlist=sanity-lipe-scan3
Signed-off-by: Vitaliy Kuznetsov <vkuznetsov@ddn.com>
Change-Id: I7d0bdcc407bc0d27441c4204511dab2e6a421a5f
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52424
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16954 llite: add SB_I_CGROUPWB on super block for cgroup

Cgroup support can be enabled per super_block by setting
SB_I_CGROUPWB in ->s_iflags.
Cgroup writeback requires support from both the bdi and
filesystem.
This patch adds SB_I_CGROUPWB flag on super block for Lustre.
This is required by the subsequent patch series to support
cgroup in Lustre.

Adding this flags for Lustre super block will cause the remount
failure on Maloo testing on Unbutu 2204 v5.15 kernel due to the
duplicate filename (sysfs) for bdi device.
To avoid remount failure, we explicitly unregister the sysfs for
the @bdi.

Lustre-change: https://review.whamcloud.com/51955
Lustre-commit: dcc1dd39a67f15de9174e7acdda599e3c54c1421

Test-Parameters: clientdistro=ubuntu2204 testlist=sanity-sec
Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I7fff4f26aa1bfdb0e5de0c4bdbff44ed74d18c2d
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52538
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17133 kernel: update SLES15 SP4 [5.14.21-150400.24.84.1]

Update SLES15 SP4 kernel to 5.14.21-150400.24.84.1 for Lustre client.

Lustre-change: https://review.whamcloud.com/52481
Lustre-commit: TBD (from 5dcdbe687d136d7e976f578faccbb3bde1b0acc9)

Test-Parameters: trivial clientdistro=sles15sp4 testlist=sanity

Change-Id: I5bce1642fc5bd212fd89dd65d9e1beb32ccd744d
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52546
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-17010 lfsck: don't dump stack repeatedly

If there are transactions started with LFSCK in dry-run mode, don't
dump the stack repeatedly, as this can spam the console logs and
significantly hurt performance.

Lustre-commit: dc360cd3eff20618f243ab89097a62f8ecf2c929
Lustre-change: https://review.whamcloud.com/52356

Test-Parameters: trivial testlist=sanity-lfsck
Fixes: 0c1ae1cb9c ("LU-13124 scrub: check for multiple linked file")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I0b0d64911453dc8ab947e284656311b5d0300c1e
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52541
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>

LU-17095 build: avoid modules.order nonexistence failure

The modules.order is a temporary output file generated by
kbuild while running "make" command. Sometimes, there is
a race condition that causes the file not created and makes
make command fail as follows:

cat: ...//modules.order: No such file or directory

This patch creates an empty modules.order file to avoid
the error.

Lustre-change: https://review.whamcloud.com/52323
Lustre-commit: dbe4f860977455a9abe50165645a025bb6c46350

Test-Parameters: trivial

Change-Id: If779a727731f18e9409c35c0cd0deddd79559d3a
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52544
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-8245 ptlrpc: always do vmalloc

If we were ever to do an allocation with kmalloc, we could
get non-page aligned memory.  So just use vmalloc directly.

Sadly, this isn't the problem with infiniband.  We never
ask for < 8192, which is the libcfs kmalloc/vmalloc cutoff.

Still, this is a timebomb if we ever changed the libcfs
kmalloc/vmalloc cutoff, so, fix it.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Id20898065b516d363d9dc280e71be1b5cfb6f4a7
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52532
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7342 revert: "test: remove extra cleanup and qp check"

This reverts commit 1dbe9be20011893ca46ccbbd2676e8063af4158d.
This causes 100% sanity-quota timeouts in test_79.

Test-Parameters: trivial testlist=sanity-quota
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I77a168bde4b53b69a197a4036b31b36f792ebae3
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52561
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>

RM-620 build: New tag 2.14.0-ddn104

New tag 2.14.0-ddn104

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I118f71235f7200f1e880424d5e7ac43334186ba3

LU-16699 osc: Prefer NR_ZONE_WRITE_PENDING

Linux commit v4.7-5966-g5a1c84b404a7
 mm: remove reclaim and compaction retry approximations

Introduced NR_ZONE_WRITE_PENDING which should be used
in mod_zone_page_state.

Older kernels should fallback to NR_UNSTABLE_NFS
or NR_WRITEBACK.

Lustre-change: https://review.whamcloud.com/50499
Lustre-commit: d4094475c990d6ee8bf9e6e32a93f7c86a78f57a

Test-Parameters: trivial
HPE-bug-id: LUS-11559
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I90f22d4bd56f5986eaa5d4a042a2c8ed31fbf752
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Qian Yingjin <qian@ddn.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52526
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-16671 osc: fix unstable pages for short IO

Unstable pages was written with theoretical support for
short IO (ie, no bulk, data-in-rpc, LU-1757), but since the
short IO code wasn't merged until years later, they were
probably never tested together.  And when you do, it
crashes.

In truth, short IO has no separate pages to be tracked,
which is why this is crashing.  This means that small write
RPCs won't be tracked in unstable pages, but that's a very
minor limitation and unlikely to cause trouble.  (and since
RPC allocations are not 'pages', they're just malloc'ed,
there's no good way to track them anyway)

Lustre-change: https://review.whamcloud.com/50451
Lustre-commit: 4ba4976f525e957ef4c3ca7981bea01f72109ed6

Fixes: 70f092a ("LU-1757 brw: add short io osc/ost transfer.")
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I34b09f8324424c3ff0b0c09c86f01c938b643e37
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Qian Yingjin <qian@ddn.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52524
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-17015 obdclass: new primitives for upcall cache

This patch adds 2 new primitives to the upcall cache mechanism:
- upcall_cache_get_entry_raw: get a ref on an existing entry;
- upcall_cache_update_entry: modify expiry time and state of an entry.

Lustre-change: https://review.whamcloud.com/52389
Lustre-commit: 2ddb1d33245c23c4cafe64fb917323bdf567c81f

Test-Parameters: trivial
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I4825f09ae807abb52ebe0e24719dcd915e8c8aef
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52497
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-8277 llite: set max compression size to 64 MiB

Compression size should never be larger than RPC size, so
set it to a maximum of 64 MiB.

Test-Parameters: trivial
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Ia5958db3504f4f442fbd41e48416924debc26192
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52466
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-7342 test: remove extra cleanup and qp check

The test_79 in sanity-quota needs quota pool support, and
the cleanup of the "stop file" is also included in the
stack_trap, then it is no need to to do it explicitly.

Test-Parameters: trivial
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Change-Id: If86a1d0187b4b95d0c5e24f11f5f058280726e64
Reviewed-on: https://review.whamcloud.com/c/ex/lustre-release/+/52472
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>