Whamcloud - gitweb
Alex Deiter [Thu, 22 Jun 2023 13:28:48 +0000 (17:28 +0400)]
LU-14697 tests: change performance-sanity to use mdtest
Replace mdsrate by mdtest in performance-sanity.sh
Test-Parameters: trivial
Test-Parameters: testlist=performance-sanity clientdistro=el7.9
Test-Parameters: testlist=performance-sanity clientdistro=el8.8
Test-Parameters: testlist=performance-sanity clientdistro=el9.2
Test-Parameters: testlist=performance-sanity clientdistro=ubuntu2204
Signed-off-by: Alex Deiter <adeiter@tintri.com>
Change-Id: I1a80bab4ccbe085d3ff8d8b332c8e117e14ea9cb
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51414
Tested-by: Maloo <maloo@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Patrick Farrell [Wed, 9 Aug 2023 16:16:25 +0000 (12:16 -0400)]
LU-13805 obd: Reserve unaligned DIO connect flag
Unaligned DIO generally requires only client changes, but
an assert must be removed from ZFS servers for it to work
correctly. This means we need a connect flag to recognize
whether or not a server running ZFS can safely use
unaligned DIO.
All OSTs will present this flag - to keep things simple -
but if the flag is not present, we'll still do unaligned
DIO to ldiskfs OSTs.
Actual implementation will be in another patch, this one
just creates the flag itself.
Test-Parameters: trivial
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I8b149cc54f4fb11e64182c65f2fbb01f8a3d3868
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51075
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Qian Yingjin <qian@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Mr NeilBrown [Tue, 24 Nov 2020 23:10:14 +0000 (10:10 +1100)]
LU-9859 lnet: simplifiy cfs_ip_addr_parse()
cfs_ip_add_parse() is now always passed a string that it is safe to
modify. So change the parsing to benefit from this and use standard
tools like strsep().
Note that the 'len' argument is now ignored. It cannot be removed
without a larger change.
Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: Ice492edf109dca2e411132b891514f0caa535d8c
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50842
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
James Simmons [Thu, 3 Aug 2023 20:57:02 +0000 (16:57 -0400)]
LU-10391 obdclass: handle large NIDs for mount strings
Mount strings support using ':' as a delimiter but this is also
a part of the some NID strings like IPv6, so rework class_parse_value()
to only look at ':' when it occurs after '@'.
The mount utilities use the function convert_hostnames() to ensure
the mount string containing an NID is valid. This only works for
small size nids so migrate the function to handle large NIDs. This
should allow mounting with IPv6 or other large NID addresses.
In testing the userland libcfs_ip_str2addr_size() had bugs that
rendered incorrect NID strings. Fix those issues.
Fixes:
b6c702df5d4 ("LU-10391 libcfs: add large-nid string conversion functions.")
Change-Id: Ic9b2a368456ba75ceb5911ac7f75ae00d6123870
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50362
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Shaun Tancheff [Fri, 14 Jul 2023 09:21:45 +0000 (16:21 +0700)]
LU-16962 build: parallel header checks
Add LB2_CHECK_LINUX_HEADER_SRC and LB2_CHECK_LINUX_HEADER_RESULT
macros to use for running header checks in parallel.
Migrate (most) header checks to parallel and run a subset
early as the results of those tests are required by other
configure tests.
Test-Parameters: trivial
HPE-bug-id: LUS-11710
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: Ia765261179d25e96912e65e31c81824b4507e604
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51673
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Shaun Tancheff [Fri, 14 Jul 2023 08:34:02 +0000 (15:34 +0700)]
LU-16957 build: Improve parallel --config-cache
The parallel build should consider the configure cache before
adding tests to the parallel build pass.
Track the number of compile tests needed, skip the make when
no build tests are needed.
Also unify libcfs, core, and ldiskfs build passes to a single step.
Configure timings vs master
master master w/cache | patch patch w/cache
-------------- --------------- | --------------- ----------------
real 1m3.493s real 0m34.024s | real 1m3.903s real 0m8.404s
user 1m34.587s user 1m16.547s | user 1m37.191s user 0m4.292s
sys 0m35.119s sys 0m22.687s | sys 0m35.297s sys 0m5.514s
Test-Parameters: trivial
HPE-bug-id: LUS-11706
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I6696b350e8315190a67c1463435b18a87d45813e
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51637
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Patrick Farrell [Wed, 8 Feb 2023 04:09:17 +0000 (23:09 -0500)]
LU-13805 llite: Add copy of iovec to sub-dio
It's very useful to move some of the direct I/O processing
from the main thread to the ptlrpc threads. This is done
by associating the processing with each sub DIO, or DIO
'chunk'. This requires a local copy of the iovec in each
chunk, because we:
A) need the chunk-current state of the iovec (as we move
along the iovec as chunks are created)
B) some operations will modify the iovec, and so to do
them from multiple ptlrpc threads, each needs to work on a
separate copy of the iovec.
This will be used by copy_page_to_iter in completing
unaligned DIO reads.
This has been split out from the main unaligned DIO patch
for simplification.
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I5645e904b6f9423eafc69cc0f59349cb3dcb9920
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/49940
Reviewed-by: Zhenyu Xu <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Qian Yingjin <qian@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Patrick Farrell [Tue, 14 Feb 2023 18:33:35 +0000 (13:33 -0500)]
LU-13805 clio: Add write to sdio
Unaligned DIO will need to know if an sdio is a write or
a read, so we add this info to the sub-dio.
Test-parameters: trivial
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I8ee042ca5a0461db672ba98b7fa6c64b01a8bba2
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/49991
Reviewed-by: Zhenyu Xu <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Qian Yingjin <qian@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Patrick Farrell [Tue, 4 Apr 2023 18:53:08 +0000 (14:53 -0400)]
LU-13805 tests: Add racing tests of BIO, DIO
We're a bit short on racing tests for buffered IO and
direct IO. This patch adds a number of tests. These
were originally part of the unaligned DIO patch, but
some of them have shown issues without unaligned IO.
So this patch puts in these tests with only aligned IO so
we can see which failures are specific to the unaligned IO
changes and which are not.
This patch should be landable like this.
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I861bcaec785936cb9c3752f8148dcab4054f6078
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50529
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Qian Yingjin <qian@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Henri Doreau [Fri, 6 Feb 2015 09:01:36 +0000 (10:01 +0100)]
LU-7073 tests: Add file migration to racer
Make racer run both blocking and non-blocking "lfs migrate" commands.
Implement this within the file_create.sh script, since it is already
selecting among different layout types.
Update Makefile.am to avoid listing every racer filename explicitly
to make it easier to add new types of operations in the future.
Test-Parameters: trivial testlist=racer,racer,racer
Test-Parameters: fstype=zfs testlist=racer,racer,racer
Signed-off-by: Henri Doreau <henri.doreau@cea.fr>
Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I51b3f19c78029ff47102e96a71ec4a0fc472183a
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/13669
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Zhenyu Xu <bobijam@hotmail.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Andreas Dilger [Sun, 26 Mar 2023 23:40:19 +0000 (17:40 -0600)]
LU-930 misc: update MAINTAINERS addresses
Update email addresses for various maintainers, and remove those
people who are no longer working on Lustre.
Change "M:" to "R:", since "M:" means "Mail to" and not "Maintainer"
as I thought. Use "R:" for "Reviewer", and remove other tags that
we don't want in this file for Lustre.
Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I3e73963b08181154fa48f308cb3d1d0a533ebbe5
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52025
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Chris Horn [Thu, 27 Jul 2023 15:47:22 +0000 (09:47 -0600)]
LU-16997 kfilnd: Correct RX buffer size
The immediate receive buffers are large buffers where incoming
messages are byte packed into the buffer until all buffer space is
exhausted. The size of the buffer is kkfilnd module parameter
credits * 4096. The number of buffers is controlled by kkfilnd module
parameter immediate_rx_buf_count.
With the current defaults this results in only 1MiB of buffers per RX
context (i.e. CPT) to sync these messages. While kfilnd tries to
replenish these buffers as fast as possible, it may not be fast enough
and replenishing can be delayed based on CPU availability. Change
default credits to 512 so that we have have 8x 2MiB buffers.
Test-Parameters: trivial
HPE-bug-id: LUS-10548
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I83f813ba2e295e6087131dcdfb12fc0feebb4834
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51789
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Ron Gredvig <ron.gredvig@hpe.com>
Reviewed-by: Ian Ziemba <ian.ziemba@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Ron Gredvig [Thu, 15 Dec 2022 20:04:20 +0000 (14:04 -0600)]
LU-16996 kfilnd: Wrong traffic class assigned
The wrong traffic class was being assigned to a transmit
context when multiple networks were assigned to the same
interface.
This was discovered by noticing a currently unsupported
traffic class didn't fail when it was used. The traffic
class from the shared domain was being used instead.
This was fixed by explicitly specifying the traffic
class when creating a transmit context for an endpoint.
Test-Parameters: trivial
HPE-bug-id: LUS-11415
Signed-off-by: Ron Gredvig <ron.gredvig@hpe.com>
Change-Id: I21236c01d4bef53b62e2f303c8e24e059ce83c0a
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51788
Reviewed-by: Ian Ziemba <ian.ziemba@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Chris Horn [Thu, 13 Apr 2023 15:36:37 +0000 (09:36 -0600)]
LU-16995 kfilnd: Handle TAG_RX_OK in TN_STATE_FAIL
It is possible for the fabric to delay packets such that the retry
handler cancels the message but it is still delivered to the target.
If the timing is right then the initiator may receive a TAG_RX_OK
event after the transaction has transitioned to TN_STATE_FAIL. This
currently trips an LBUG, but instead we can allow the transaction to
complete normally.
Test-Parameters: trivial
HPE-bug-id: LUS-11572
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I381d64713a7942fed09d41b30f64be602193057f
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51787
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Ron Gredvig <ron.gredvig@hpe.com>
Reviewed-by: Ian Ziemba <ian.ziemba@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Ron Gredvig [Thu, 20 Apr 2023 19:12:08 +0000 (19:12 +0000)]
LU-16994 kfilnd: Add work queue parameters
Added kfilnd work queue parameters to allow tuning.
The wq_high_priority parameter enables the work queue to run
as high priority. Default is enabled.
The wq_cpu_intensive parameter enables the work queue to run
as cpu intensive. Default is disabled.
The wq_max_active parameter sets the max in-flight work items
of the work queue. Default is 512.
The prov_cpu_exclusive parameter enables reserving one of a
CPT's CPUs for the exlusive use of the kfabric provider.
Default is disabled.
Test-Parameters: trivial
HPE-bug-id: LUS-11605
Signed-off-by: Ron Gredvig <ron.gredvig@hpe.com>
Change-Id: Ic4db95787a864efca3ea1234953ce3ea828f3594
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51786
Reviewed-by: Ian Ziemba <ian.ziemba@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Ron Gredvig [Fri, 12 May 2023 19:44:48 +0000 (19:44 +0000)]
LU-16993 kfilnd: Add count value to debugfs stats
The debugfs stats for initialor and target include
min. max and average. It is hard to interprest the
average without knowing how many sample were included
to calculate it.
Added the count value for extra context.
Test-Parameters: trivial
HPE-bug-id: LUS-11627
Signed-off-by: Ron Gredvig <ron.gredvig@hpe.com>
Change-Id: I3d840c250653b4f29b40c169254b9c9b4c88f584
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51785
Reviewed-by: Ian Ziemba <ian.ziemba@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Ian Ziemba [Wed, 22 Feb 2023 23:56:21 +0000 (17:56 -0600)]
LU-16992 kfilnd: Expand debugfs to record min/max
This will be useful in determining how long kfilnd transactions take
when underlying NIC gets congested.
Test-Parameters: trivial
HPE-bug-id: LUS-11497
Signed-off-by: Ian Ziemba <ian.ziemba@hpe.com>
Change-Id: I5506329086e6284e04ec7b609485d582a35ca0b5
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51784
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Ron Gredvig <ron.gredvig@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Ian Ziemba [Wed, 22 Feb 2023 22:00:43 +0000 (16:00 -0600)]
LU-16991 kfilnd: Init LNet NI data in dev alloc
LNet ni_nid was being set outside of kfilnd_dev_alloc(). This was
causing the incorrect debugfs directories to be generated inside
kfilnd_dev_alloc().
Fix this by setting all LNet NI fields inside kfilnd_dev_alloc().
Test-Parameters: trivial
HPE-bug-id: LUS-11496
Signed-off-by: Ian Ziemba <ian.ziemba@hpe.com>
Change-Id: I4eecfa05966cb7793a01b92b0bc49ffca252976e
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51783
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Ron Gredvig <ron.gredvig@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Chris Horn [Thu, 9 Mar 2023 00:18:41 +0000 (18:18 -0600)]
LU-16990 kfilnd: Use NETWORK_TIMEOUT for TAG_RX_CANCEL
We can get ECANCELED for some tagged receives which results in
transaction failure with TN_EVENT_TAG_RX_CANCEL. This can occur due
to problems with either the source or the target, so we should
use NETWORK_TIMEOUT message status.
Test-Parameters: trivial
HPE-bug-id: LUS-11520
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: Ic3c1910f8a8c43447cbbc28129e23350e726830d
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51782
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Ron Gredvig <ron.gredvig@hpe.com>
Reviewed-by: Ian Ziemba <ian.ziemba@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Chris Horn [Mon, 12 Dec 2022 23:28:54 +0000 (16:28 -0700)]
LU-16989 kfilnd: Handle TX_FAIL in WAIT_SEND_COMP
It is possible for us to get a TN_EVENT_TX_FAIL while transaction is
in TN_STATE_WAIT_SEND_COMP state. We should gracefully handle this
situation rather than LBUG.
Test-Parameters: trivial
HPE-bug-id: LUS-11344
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: Ib6fc5ed41f12762843fe9f638ffd523699936556
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51781
Tested-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Serguei Smirnov [Thu, 13 Jul 2023 00:29:56 +0000 (17:29 -0700)]
LU-16393 o2iblnd: add IBLND_REJECT_EARLY reject reason
Add IBLND_REJECT_EARLY reason for rejecting connection request:
to be used when the device doesn't have any nets added yet or
when there's no active NIs on the net to handle the connection.
These conditions are supposed to occur only when LNI is being
added/initialized, so report at CNETERROR level vs. CERROR.
In lnet, set NI state to ACTIVE only after it has been added
to the list of NIs for the net, so that LND can know that
the NI can be used to accept connections.
Test-parameters: trivial
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: I59efb2fdf5d5ceabb6ff23f638ec85da82d57b99
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51651
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Cyril Bordage [Tue, 8 Aug 2023 13:06:23 +0000 (15:06 +0200)]
LU-17021 socklnd: fix late ksnr_max_conns set
ksnr_max_conns was set to the correct value after it was used.
Test-Parameters: trivial
Fixes:
a5cbe7883db6 ("LU-12815 socklnd: allow dynamic setting of conns_per_peer")
Signed-off-by: Cyril Bordage <cbordage@whamcloud.com>
Change-Id: I9f2454d915ee1ab27db96f5247028db94965a11f
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51890
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Hongchao Zhang [Sat, 22 Jul 2023 08:29:57 +0000 (16:29 +0800)]
LU-17010 lfsck: don't create trans in dryrun mode
In LFSCK, the LFSCK transaction should not be created in
dryrun mode, which is related to the following patch,
Fixes:
0c1ae1cb9c19 ("LU-13124 scrub: check for multiple linked file")
Change-Id: Id543bc3c0e300c1cc14b670d724ebcacac3bf71b
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51849
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Timothy Day [Sat, 29 Jul 2023 19:58:47 +0000 (19:58 +0000)]
LU-17000 lnet: fix use-after-free in lnet_startup_lndnet
If the lnd_startup function returns a positive
error code, the ni will get freed. But the code
incorrectly checks only for negative error codes,
leading to a potential use-after-free.
Addresses-Coverity-ID: 397786 ("Use after free")
Test-Parameters: trivial
Signed-off-by: Timothy Day <timday@amazon.com>
Change-Id: I36dd4dbfc0b409de010257e5d9ae9d983fd1817f
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51806
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Alex Zhuravlev [Wed, 2 Aug 2023 11:46:28 +0000 (14:46 +0300)]
LU-16974 utils: lfs mirror resync to show progress
lfs mirror resync should be able to:
- show progress like lfs mirror extend --stats does
- throttle like lfs mirror extend -W does
use 64MB buffer for mirror resync by default.
Change-Id: Ibe60748542ff4a3731aa6a4a9907be82427a0ae9
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51750
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Timothy Day <timday@amazon.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Timothy Day [Fri, 23 Jun 2023 20:44:17 +0000 (20:44 +0000)]
LU-8191 lnet: convert functions in utils to static
Static analysis shows that a number of functions
could be made static. This patch declares several
functions in various LNet utils and lnetconfig to
static.
In LNet selftest (lst), one unused function was
removed entirely. Some declarations were moved to
made static.
Test-Parameters: trivial
Signed-off-by: Timothy Day <timday@amazon.com>
Change-Id: Ia4528281b3c87d77e46abb95f47ab0bdc72168c0
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51427
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Shaun Tancheff [Thu, 8 Jun 2023 06:32:17 +0000 (13:32 +0700)]
LU-16883 ldiskfs: update for ext4-delayed-iput for SUSE 15
ext4-delayed-iput patch does not apply cleanly to SUSE 15
SP4 and SP5 series 5.14.21 kernel.
Adjust the minor conflict in ext4_put_super()
Test-Parameters: trivial
Fixes:
616fa9b581 ("LU-15404 ldiskfs: use per-filesystem workqueues to avoid deadlocks")
HPE-bug-id: LUS-11661
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: Iee424bd6d455853d9f82e6e5b08e4ab44deb432c
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51252
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Xinliang Liu [Thu, 1 Jun 2023 10:00:04 +0000 (10:00 +0000)]
LU-16862 rpm: set kmod-lustre-tests requires kmod-lustre explicitly
Kmod-lustre-tests rpm should be installed along with kmod-lustre rpm
if there is one.
Test-Parameters: trivial
Change-Id: Ib265298381c317a03c4244f8ea380c6d64f0aef5
Signed-off-by: Xinliang Liu <xinliang.liu@linaro.org>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51191
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
James Simmons [Tue, 16 May 2023 13:59:15 +0000 (09:59 -0400)]
LU-9859 libcfs: remove cfs_size_round()
Now that everyone is moved to round_up() we can safely remove the
macro cfs_size_round().
Test-Parameters: trivial
Change-Id: If8e1aff5e89007eeb38c5810c68282d51e37f019
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51009
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Timothy Day <timday@amazon.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Mr NeilBrown [Tue, 25 Jul 2023 18:47:27 +0000 (14:47 -0400)]
LU-9859 lnet: simplify cfs_parse_nidlist()
By duplicating the string being parsed, we can use mutating parsign
primitives and simplify the code.
Note that the kernel-space cfs_parse_nidlist() is now different from
the user-space version. As they are both declared in the same header
file, this needs an #ifdef until the headers can be separated.
Change some function that return 0 on error to match the kernel
convention of 0 on success and -ve on error.
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I1d5155a1cee82f798bec0863d80d75af92399cf1
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50841
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Mr NeilBrown [Tue, 25 Jul 2023 18:34:49 +0000 (14:34 -0400)]
LU-9859 llite: simplify pcc_fname_list_parse()
Now that pcc_fname_list_parse() is passed a mutable string, mutable
parsing with standard tools can be used.
Test-Parameters: trivial testlist=sanity-pcc
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I8da6720ae6822c410a339b5fff9fdd4256bb3685
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50840
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Mr NeilBrown [Wed, 19 Jul 2023 18:15:57 +0000 (14:15 -0400)]
LU-9859 ptlrpc: simplify nrs_tbf_opcode_list_parse()
If nrs_tbf_opcode_list_parse() duplicates the string it is passed, it
can use standard mutating functions for parsing the string.
Test-Parameters: trivial
Test-Parameters: testlist=sanityn
Test-Parameters: testlist=sanity
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I4acb57de52abde97fa9c2d133cf10a432b12f604
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50837
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Etienne AUJAMES <eaujames@ddn.com>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Xinliang Liu [Mon, 3 Apr 2023 03:47:23 +0000 (03:47 +0000)]
LU-16673 tests: add checking for user USER0
Add checking for user USER0 in tests 125, 154a, 154b.
Test-Parameters: trivial testlist=sanity env=ONLY="125 154a 154b"
Change-Id: Id42d4b6dca4c05757d02483ddedd65be55df96d6
Signed-off-by: Xinliang Liu <xinliang.liu@linaro.org>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50501
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Sebastien Buisson [Wed, 12 Jul 2023 13:32:26 +0000 (15:32 +0200)]
LU-16374 enc: rename O_FILE_ENC to O_CIPHERTEXT
Rename O_FILE_ENC to O_CIPHERTEXT as per discussion in linux-fscrypt
mailing-list.
Also change the flag combination to be:
O_NOCTTY | O_NDELAY | O_DSYNC
to avoid the risk of accidental issues with tar that already opens
files with the 'O_NOCTTY | O_NDELAY' combination.
O_DSYNC does not make much sense for O_RDONLY files, but will force
writes on encrypted restore to be synchronous. With O_DIRECT and large
enough writes (32MB?) that might be OK, but not ideal for small files.
Fixes:
fdbf2ffd41 ("LU-14677 sec: no encryption key migrate/extend/resync/split")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I36fed17a413ee690bc445c3e76674ed5fc337de5
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51640
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Vladimir Saveliev [Fri, 28 Jul 2023 14:44:33 +0000 (10:44 -0400)]
LU-16851 llite: use right predicate in ll_atomic_open
Using d_unhashed() brings a race window with d_add() and d_drop()
leading to dentry hash table corruption. If dentry which is in hash
already is added to hash table, it gets looped to itself via next
pointer:
dentry 0xffff8fd34cc08840
...
d_hash = {
next = 0xffff8fd34cc08848,
pprev = 0x0
},
See for reference:
commit
00699ad8571afd7fb8bc2c61f67c86c2428680ab
Author: Al Viro <viro@zeniv.linux.org.uk>
Date: Tue Jul 5 09:44:53 2016 -0400
Use the right predicate in ->atomic_open() instances
Keep using d_unhashed() if d_in_lookup() is not provided by kernel.
HPE-bug-id: LUS-11560
Signed-off-by: Vladimir Saveliev <vladimir.saveliev@hpe.com>
Change-Id: I6c27f031d0d7e7d571752d6172a32406ad68e913
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51147
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Shaun Tancheff [Fri, 14 Jul 2023 09:09:38 +0000 (16:09 +0700)]
LU-16962 build: parallel configure cleanup
LC_REGISTER_SHRINKER_FORMAT_NAMED macro should use
register_shrinker_format
LC_HAVE_FILEMAP_GET_FOLIOS_CONTIG macro should use
filemap_get_folios_contig
LN_SRC_CONFIG_STRSCPY_EXISTS and LN_CONFIG_STRSCPY_EXISTS
are not defined. Remove the references.
Test-Parameters: trivial
HPE-bug-id: LUS-11709
Fixes:
0006eb3644 ("LU-16328 llite: migrate_folio, vfs_setxattr")
Fixes:
ca992899d5 ("LU-16351 llite: Linux 6.1 prandom, folios_contig, vma_iterator")
Fixes:
7fe7f4ca06 ("LU-16520 build: Move strscpy to libcfs common header")
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I0cb630d035a23edfa353040f4c0d25c46eb417d8
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51670
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Petros Koutoupis <petros.koutoupis@hpe.com>
Reviewed-by: xinliang <xinliang.liu@linaro.org>
James Simmons [Tue, 25 Jul 2023 18:14:03 +0000 (14:14 -0400)]
LU-8130 nrs: convert NRS ORR/TRR to rhashtable
Move away from the cfs hash implementation to rhashtable
for NRS ORR handling. Since rhashtable is lockless it
should also increase performance. For the NRS TRR handling
we can use Xarray since its based on OST index which are
sequential which will give better performance than an hashtable.
TRR entries are added to the Xarray but not removed until the
Xarray is destroyed.
Test-Parameters: testlist=sanityn env=ONLY=77c+77d,ONLY_REPEAT=100
Change-Id: I5206a7586d4b4c8991b7163fd9253f017e6d3969
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/40113
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Etienne AUJAMES <eaujames@ddn.com>
James Simmons [Wed, 2 Aug 2023 18:36:11 +0000 (14:36 -0400)]
LU-8130 libcfs: don't use radix tree for xarray
For newer kernels the radix tree is totally based on Xarray. For Lustre
support for RHEL7 we backported Xarray but it still was using the
radix tree. Their is a mismatch between what the radix tree expects
and using a struct xa_node when allocating and freeing memory. Instead
abandon all use of the radix tree with Xarray. We use our own private
kmem cache which is based on radix tree but it uses xa_node.
Test-Parameters: trivial
Fixes:
84e12028be9a ("LU-9859 libcfs: add support for Xarray")
Change-Id: I87607aa0e55a4aca039f2fef5a76fbff0bedd9b3
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51840
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Andreas Dilger [Sat, 29 Jul 2023 08:59:28 +0000 (02:59 -0600)]
LU-16973 ptlrpc: flush delayed file desc if idle
The use of alloc_file_pseudo() allocates a real file descriptor,
so fput() will use a deferred cleanup for the descriptor, either
when the thread "finishes the syscall" (which never happens for
kernel threads), or a unmount time. This accumulates too many
file descriptors (millions) on a busy system.
Instead of waiting to cleanup these file descriptors at unmount
time, call flush_delayed_fput() to clean them up when a ptlrpcd
thread becomes idle before it goes to sleep.
For kernels 3.6 and later when flush_delayed_fput() was first added,
and before kernel 5.4 when it was EXPORT_SYMBOL'd, grab a pointer
to the function with kallsyms_lookup_name() so it can be called.
Delete LN_CONFIG_STRSCPY_EXISTS reference that generates configure
warnings, since this check was renamed and moved to libcfs.
Fixes:
eed43b2a427b ("LU-13783 osd-ldiskfs: use alloc_file_pseudo to create fake files")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I24a08f9568d7d636a69672c5c3132ab25b292407
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51805
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Andrew Perepechko <andrew.perepechko@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Alexey Lyashkov [Tue, 30 May 2023 08:30:02 +0000 (11:30 +0300)]
LU-16847 ldiskfs: refactor bio submit.
extract common code into osd_submit_bio() function to avoid
code duplication.
fix a debug output for the last bio, currently it lost from output.
HPe-bug-id: LUS-11645
Signed-off-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Change-Id: I5f957a832b73ad08ccccfba7866393d93d4ae538
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51236
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andrew Perepechko <andrew.perepechko@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
James Simmons [Wed, 19 Jul 2023 18:04:15 +0000 (14:04 -0400)]
LU-13306 mgs: use large NIDS in the nid table on the MGS
On the MGS the NIDs detected are handled using the struct
mgs_target_info which currently only handles lnet_nid_t.
This structure also limits the number of NIDs to 32 entries.
Some sites have reported that 32 NIDs wasn't enough when
they configured virtual LNet networks for isolation.
Update the mgs_target_info to use NID strings instead.
This has the advantage of working even if struct lnet_nid
expands in the future. We place this data at the end of
the mgs_target_info as a flexible array. This requires
updating the ptlrpc packet handling to increase the size
to some new value to contain all the NIDs registered.
Also this gives us the option to use hostnames in the
future. This information is then feed into a
struct mgs_nidtbl_entry which is sent to the mgc on all
the remote nodes. With this patch only large NIDs for
small address space is translated to the original
lnet_nid_t format and sent to the various clients.
All the server targets, which are clients of the MGS,
use the large NID format. With this patch we don't
have to patch old clients when the servers are using
the larger NID format.
Expand LNetGetId() to return large NID addresses as well.
In the future we will use the ocd_connect_flags to
determine if the MSG supports large NID addresses.
Change-Id: I7083d6ecfc46cf0419a0d4a582e4bf5240f193cd
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50896
Tested-by: Maloo <maloo@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Xinliang Liu [Tue, 28 Jun 2022 08:34:46 +0000 (08:34 +0000)]
LU-15978 osp: fix striped directory deletion fails for 64K PAGE_SIZE
This fixes the rmdir errors below:
rmdir: failed to remove '/mnt/lustre/d1.sanity/d2': Invalid argument
LustreError: 381691:0:(osp_object.c:1998:osp_it_next_page())
lustre-MDT0000-osp-MDT0001: invalid magic (0 !=
8a6d6b6c) for page 0/1
while read layout orphan index.
For 64K PAGE_SIZE, when created an striped directory, e.g. created
with function test_mkdir() defined in test-framework.sh when MDSCOUNT
>= 2, deleting it will fail.
For PAGE_SIZE > LU_PAGE_SIZE, if the end system page fills less than
LU_PAGE_COUNT lu_idxpages, init the header of the remain lu_idxpages.
So that the clients handle this partial filling correctly.
Also make goto labels meaningful and avoid not freeing pages for
lip_nr == 0 in osp_it_next_page().
This patch also fixes wrong page idx for page kunmap in
dt_index_walk().
This server end fix also necessary for the idxpage reading clients
nodemap_process_idx_pages() and qsd_reint_entries(). So this patch also
includes fix for LU-15992: nodemap create and check failed on 64K page
size.
Fixes:
77eea1985bb1 ("LU-3336 lfsck: orphan OST-objects iteration")
Change-Id: I75bd9603c31bed8ea15fdba693677d41affaf61c
Signed-off-by: Xinliang Liu <xinliang.liu@linaro.org>
Co-authored-by: Kevin Zhao <kevin.zhao@linaro.org>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/47812
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Xinliang Liu [Mon, 6 Jun 2022 08:59:54 +0000 (08:59 +0000)]
LU-15722 osd-ldiskfs: fix write stuck for 64K PAGE_SIZE
This reverts a previous commit for large PAGE_SIZE to fix a stuck IO
issue in another way.
One more ldiskfs_map_blocks() can't fix the write stuck for PAGE_SIZE
> BLOCK_SIZE. It still gets stuck in some tests like sanity-dom fsx.
Because each time ldiskfs_map_blocks() lookup it only return a
continuous range physical blocks. If a page has multiple continuous
range blocks, then it needs multiple ldiskfs_map_blocks() lookups to
find out all the already mapped blocks.
The fixed idea here is to record the already written blocks of the
start page and skip them at the next write retry.
This also fix and cleanup osd_mark_page_io_done() when start_blocks
is non-zero.
Fixes:
176ea3a4599e ("LU-15722 osd-ldiskfs: fix IO write gets stuck for 64K PAGE_SIZE")
Change-Id: I9c14d5d0aa23e81837dacb01d050c091e6a79148
Signed-off-by: Xinliang Liu <xinliang.liu@linaro.org>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/47563
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Andreas Dilger [Fri, 7 Jul 2023 19:57:52 +0000 (13:57 -0600)]
LU-16872 tests: exercise sanity test_27M more fully
Improve the sanity.sh test_27M to precreate a bunch of files with
specific OST striping so that it is more likely to trigger the code
path that accessed the stale OST list when using O_APPEND layout.
Also clean up code style in the rest of this subtest.
Test-Parameters: trivial testlist=sanity env=ONLY=27M,ONLY_REPEAT=200
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ie94e3a32fc48198e4e15f44a55d1f8ccf61c74f5
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51602
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Thomas Bertschinger <bertschinger@lanl.gov>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Thomas Bertschinger [Fri, 7 Jul 2023 14:57:40 +0000 (10:57 -0400)]
LU-16872 lod: reset llc_ostlist when using O_APPEND stripes
Files created with O_APPEND can have special striping set with the
parameters mdd.*.append_stripe_count and mdd.*.append_pool, and
should not inherit a list of OSTs to use from a parent directory
when these parameters are set. However, if a file is created with
O_APPEND and its create is handled by a kernel thread that has
previously created a file with a default list of OSTs, then those
defaults were erroneously applied to the O_APPEND file. This can
lead to the create returning EINVAL or to a crash.
This commit ensures that llc_ostlist is cleared when a file is
created with special append stripes.
Signed-off-by: Thomas Bertschinger <bertschinger@lanl.gov>
Change-Id: Ib2023e17c9ef31a2e029e09e67b257eb2c77b113
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51559
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Patrick Farrell [Tue, 8 Aug 2023 19:39:17 +0000 (15:39 -0400)]
LU-17019 tests: fix 'v' usage in test 185
Test 185 provides 'v' to multiop when using the
multiop_bg_pause function, which already adds 'v' to the
multiop command. Using a verbosity level > 2 breaks the
code which checks the multiop output for PAUSING, now that
there is other text in that output.
Fixes:
84376bf674 ("LU-13805 tests: add unaligned io to multiop")
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I7023e47956e32a3f681fec504a1fc9200dbbf24f
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51898
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Jian Yu [Mon, 31 Jul 2023 17:24:02 +0000 (10:24 -0700)]
LU-16980 build: fix gcc-12 [-Werror=use-after-free] error
This patch fixes the following [-Werror=use-after-free] and
[-Werror=stringop-overflow=] errors detected by gcc 12:
libcfs/include/libcfs/util/list.h:481:42: error: pointer 'tmp'
used after 'free' [-Werror=use-after-free]
481 | pos = list_entry(pos->member.next, typeof(*pos), member), \
| ^
libcfs/include/libcfs/util/list.h:239:28: note: in definition of macro 'list_entry'
239 | ((type *)((char *)(ptr)-(char *)(&((type *)0)->member)))
| ^~~
obd.c:5118:9: note: in expansion of macro 'list_for_each_entry'
5118 | list_for_each_entry(tmp, head, lpn_list) {
| ^~~~~~~~~~~~~~~~~~~
obd.c:5124:17: note: call to 'free' here
5124 | free(tmp);
| ^~~~~~~~~
test_brw.c: In function 'main':
test_brw.c:227:22: error: 'write' specified size between
9223372036854775808
and
18446744073709551615 exceeds maximum object size
9223372036854775807
[-Werror=stringop-overflow=]
227 | rc = write(fd, buf, len);
| ^~~~~~~~~~~~~~~~~~~
Change-Id: Ibe783ab0d13e2ecde1736946323932ab5db53740
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51819
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Timothy Day <timday@amazon.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Timothy Day [Fri, 28 Jul 2023 05:11:49 +0000 (05:11 +0000)]
LU-17000 lod: fix lod_gen_component_id not wrapping
The end variable is set to SEQ_ID_MAX. The code
for checking whether to try and search the
remaining ids checks LCME_ID_MAX. Thus, it
never gets called. Change the check to look for
SEQ_ID_MAX. Also, change the start and end such
that all ids are checked once.
Addresses-Coverity-ID: 397902 ("Logically dead code")
Test-Parameters: trivial
Signed-off-by: Timothy Day <timday@amazon.com>
Change-Id: I1bd06b38314686f3d5a1c9ad42b38ad197f1a4e7
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51795
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Alex Zhuravlev [Wed, 26 Jul 2023 19:00:29 +0000 (22:00 +0300)]
LU-16985 utils: adaptive bufsize for mirroring
if bandwidth limit is requested, then change default bufsize to
make I/O rather smooth than like a saw.
Fixes:
23224e03dc ("LU-16587 utils: give lfs migrate a larger buffer")
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Ibc8e7d30ded201a4ff3d699530f5c9f8be5ce7f1
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51773
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Timothy Day <timday@amazon.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Jian Yu [Mon, 31 Jul 2023 09:03:46 +0000 (02:03 -0700)]
LU-16980 build: fix gcc-12 [-Werror=format-truncation=] error
This patch fixes the following [-Werror=format-truncation=] errors
detected by gcc 12:
liblnetconfig.c: In function 'open_sysfs_file':
liblnetconfig.c:106:49: error: '%s' directive output may be truncated
writing up to 127 bytes into a region of size between 1 and 128
[-Werror=format-truncation=]
106 | snprintf(filename, sizeof(filename), "%s%s",
| ^~
lfs_project.c: In function 'lfs_project_handle_dir':
lfs_project.c:324:50: error: '%s' directive output may be truncated
writing up to 255 bytes into a region of size between 1 and 4095
[-Werror=format-truncation=]
324 | snprintf(fullname, PATH_MAX, "%s/%s", pathname,
| ^~
statx.c: In function 'do_dir_list':
statx.c:1427:58: error: '%s' directive output may be truncated
writing up to 255 bytes into a region of size between 1 and 4095
[-Werror=format-truncation=]
1427 | snprintf(fullname, PATH_MAX, "%s/%s",
| ^~
Change-Id: I514a1022d879f8b7af89f6ded68e9b453cd11408
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51765
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Timothy Day <timday@amazon.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Arshad Hussain [Tue, 18 Jul 2023 11:50:47 +0000 (17:20 +0530)]
LU-16605 lfs: Add -0 option to fid2path
Currently fid2path adds '\n' after printing
pathnames. Add '-0' option to fid2path to
add NUL('\0') at the end after printing out
pathnames instead of '\n'. This allows
pathnames that contain newlines('\n') to be
correctly interpreted by binaries like xargs.
Without -0 option:
$ lfs fid2path /mnt/lustre 0x200000401:0x1:0x0
/mnt/lustre/Test
_file
/mnt/lustre/Link_
file
With -0 option:
$ lfs fid2path -0 /mnt/lustre 0x200000401:0x1:0x0 | xargs --null
/mnt/lustre/test
_file /mnt/lustre/link_
file
Test-case sanity/226e added.
Test-Parameters: trivial testlist=sanity
Reported-by: Simon Westersund <simon.westersund@csc.fi>
Signed-off-by: Simon Westersund <simon.westersund@csc.fi>
Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Change-Id: I9e3e32cde6c6abe83df48afd191ec167c74ac7e6
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51736
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Timothy Day <timday@amazon.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Serguei Smirnov [Tue, 11 Jul 2023 22:40:37 +0000 (15:40 -0700)]
LU-16949 lnet: get monitor thread to update ping buffer
Make sure that ping buffer updates requested by o2iblnd and
socklnd are performed by the LNet monitor thread.
Having the LNDs do these updates via an LNet API directly caused a
lock-up due to spinlock acquisition while in an interrupt context
in Centos 7.9 environment.
To avoid LNet trying to update the ping buffer for an LNI which is
still initializing, check that o2iblnd net is fully initialized
(IBLND_INIT_ALL) before requesting the ping buffer update.
Fixes:
da230373bd ("LU-16563 lnet: use discovered ni status")
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: I87ff8791937f5a0ead6096ff33e8c0a8087f8ddd
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51635
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Andreas Dilger [Sat, 10 Jun 2023 13:36:38 +0000 (07:36 -0600)]
LU-16521 tests: racer should work without DNE
Don't assume there are multiple MDTs for the target directory
when running dir_migrate and dir_remote. If only one MDT
(or none, if running on a non-Lustre directory for some reason),
just exit from these workloads instead of spewing errors.
Test-parameters: trivial testlist=sanityn
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I47cbe9de2c4f9afd79228b6a505dde023f2540e5
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51267
Tested-by: Maloo <maloo@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Thomas Bertschinger [Fri, 5 May 2023 21:05:22 +0000 (17:05 -0400)]
LU-13031 jobstats: store jobid in xattr when files are created
This change stores the jobid of the process that creates a file in an
extended attribute in the file's MDT inode, at file creation time.
The name of the xattr is determined by a new sysfs parameter
"mdt.*.job_xattr" so that the admin can choose a name that does
not conflict with other uses they may have for a given xattr.
The default value is "user.job". A value of "NONE" means that
the jobid will not be stored in the inode.
If the name is in the user namespace "user.", then the name portion
can be up to 7 alphanumeric characters long. The admin can choose
the trusted namespace to prevent users from modifying the value,
but only "trusted.job" is allowed in this namespace.
Allowing users to modify the contents of the xattr is helpful so
that the jobid can be preserved even when files are moved with tools
like `cp` or `rsync`, and when copied from one filesystem to another.
Signed-off-by: Thomas Bertschinger <bertschinger@lanl.gov>
Change-Id: Iad78a5ec6fbc4b761ff481141763bdd0cdcd0128
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50982
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Mr NeilBrown [Wed, 25 Nov 2020 00:04:09 +0000 (11:04 +1100)]
LU-9859 lnet: simplify cfs_expr_list_parse()
If we dup the string first, we can use a mutating approach to parsing,
allowing us to use standard tools like strsep, strcmp, kstrtouint,
etc.
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I40a1f7bc65bd122d22cd53cf3c645f4a3730f82e
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50843
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Andriy Skulysh [Tue, 26 Jul 2022 11:10:43 +0000 (14:10 +0300)]
LU-16770 llite: prune object without layout lock first
lov_layout_change() calls cl_object_prune() before
changing layout. It may lead to eviction from MDT
in case slow responce from OST.
To reduce risk of possible eviction call cl_object_prune()
without layout lock held before calling lov_layout_change()
vvp_prune() attempts to sync and truncate page cache pages.
osc_page_delete() may encounter page cache pages in non-clean state
during truncate because there's a race window between sync and truncate.
Writes may stick into this window and generate dirty or writeback pages.
This window is usually protected with a special truncate semaphore e.g.
when truncate is requested from the truncate syscall.
Let's use this semaphore to avoid write vs truncate race in vvp_prune().
Change-Id: Ie2ee29ea1e792e1b34b6de068ff2b84fd8f52f2a
HPE-bug-id: LUS-9927, LUS-11612
Signed-off-by: Andriy Skulysh <andriy.skulysh@hpe.com>
Reviewed-by: Vitaly Fertman <c17818@cray.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50742
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andrew Perepechko <andrew.perepechko@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Timothy Day [Sun, 30 Jul 2023 16:33:06 +0000 (16:33 +0000)]
LU-6142 lod: remove custom macros in lod_qos.c
lod_qos.c redefines some common Lustre macros,
for use only in that file. This is not needed.
Replace these macros with the usual Lustre
macros instead. Remove some unused macros at
the same time.
There was some debug code under a '#if 0'.
Since enabling this requires a change to the
source code anyway, a potential debugger can
just re-add this code themselves if they
really need it.
Test-Parameters: trivial
Signed-off-by: Timothy Day <timday@amazon.com>
Change-Id: I1752eab59ce5792f1ca3e9f698bda370d9ac75b1
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51808
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Vitaliy Kuznetsov [Fri, 28 Jul 2023 15:40:14 +0000 (19:40 +0400)]
LU-16982 ldiskfs: Fix crash after "umount -d -f /mnt/..."
This patch adds an extra state check during the unmount process;
Since there was the following problem:
Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds4
kernel BUG at fs/jbd2/transaction.c:378!
CPU: 0 PID: 310834 Comm: kworker/0:2 4.18.0-477.15...
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: events flush_stashed_stats_work [ldiskfs]
RIP: 0010:start_this_handle+0x22c/0x520 [jbd2]
Call Trace:
jbd2__journal_start+0xee/0x1f0 [jbd2]
jbd2_journal_start+0x19/0x20 [jbd2]
flush_stashed_stats_work+0x36/0x90 [ldiskfs]
process_one_work+0x1a7/0x360
worker_thread+0x30/0x390
kthread+0x134/0x150
Fixes:
e27a7b33d6 ("LU-16298 ldiskfs: Periodically write ldiskfs superblock")
Signed-off-by: Vitaliy Kuznetsov <vkuznetsov@ddn.com>
Change-Id: I162d3416ca1fe9bd09f1102ccca892db05719016
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51760
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Patrick Farrell [Tue, 14 Feb 2023 18:29:09 +0000 (13:29 -0500)]
LU-13805 tests: add unaligned io to multiop
Add memory unaligned IO support to multiop.
This will be used by tests for unaligned DIO.
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I38c049690610d34564a15e57f37c052105ab2066
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/49990
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Qian Yingjin <qian@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Li Dongyang [Wed, 26 Jul 2023 10:52:24 +0000 (20:52 +1000)]
LU-16954 llite: do not set SB_I_CGROUPWB on super block
On clients with a more recent kernel e.g. ubuntu2204,
this makes the mount fails sometimes with
sysfs: cannot create duplicate filename '/devices/virtual/bdi/lustre-
ffff8dd549f3d000'
Change-Id: Ie15e41eb9d039829545e1d69f97ed9e13f89e53e
Fixes:
f5a75ea44d ("LU-16697 llite: Set BDI_CAP_* flags for lustre")
Test-Parameters: clientdistro=ubuntu2204 testlist=sanity,conf-sanity
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51701
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Qian Yingjin <qian@ddn.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Oleg Drokin [Tue, 1 Aug 2023 06:17:39 +0000 (02:17 -0400)]
New tag 2.15.57
Change-Id: Ice12bbb65d4d455b2beea14e83a9ab663bda237c
Mikhail Pershin [Tue, 25 Jul 2023 22:09:31 +0000 (01:09 +0300)]
LU-16983 mdc: check errcode prior mdc_fill_lvb() call
The mdc_enqueue_fini() can be called with negative
errcode parameter if request processing was failed.
In that case the mdc_fill_lvb() shouldn't be called.
Issue may occur with DoM files, old server (<2.14) and
new client. The problem is in new client code.
Test-Parameters: testlist=racer serverversion=EXA5.2.8
Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I884398beada4286bc07875247e15b41120f73a3e
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51761
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Alex Zhuravlev [Tue, 25 Jul 2023 15:07:19 +0000 (18:07 +0300)]
LU-16979 utils: enable throttling mirror extend
this can be useful in some scenarios like massive mirror
creation.
Test-Parameters: trivial
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Ia84054f3519cd5cef37aaabb2ae605fb6ea200e0
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51758
Reviewed-by: Timothy Day <timday@amazon.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Arshad Hussain [Sun, 23 Jul 2023 05:40:04 +0000 (11:10 +0530)]
LU-16796 obdclass: Change struct jobid_pid_map to use refcount_t
This patch changes struct jobid_pid_map to use
refcount_t(kref) instead of atomic_t
Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Change-Id: Ia3458d5605a8cff2bb65476495c321fa98cf01dc
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51747
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Timothy Day <timday@amazon.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Timothy Day [Wed, 19 Jul 2023 17:23:24 +0000 (17:23 +0000)]
LU-16969 build: check that pkg-config is installed
PKG_CHECK_MODULES macro fails in very annoying to debug ways.
Often, this will fail with:
syntax error near unexpected token `LIBNL3,'
` PKG_CHECK_MODULES(LIBNL3, libnl-genl-3.0 >= 3.1)'
and provide no indication that the real error is that
pkg-config is not installed. Adding an explicit check
for pkg-config will make the error more self-evident.
Test-Parameters: trivial
Signed-off-by: Timothy Day <timday@amazon.com>
Signed-off-by: James Simmons <jsimmons@infradead.org>
Change-Id: Ic2ee8e4c3ec3fa2e03c5ece03e6a9ce335133578
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51716
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Bobi Jam [Wed, 12 Jul 2023 15:05:27 +0000 (23:05 +0800)]
LU-16958 llite: migrate vs regular ops deadlock
When it need to lock inode in lov_conf_set(), it could have hold
inode's lli_layout_mutex, we need unlock the layout mutex before
taking its inode lock to keep the lock order.
Fixes:
51d62f2122f ("LU-16637 llite: call truncate_inode_pages() in inode lock")
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I7ee58039a6d31daefc625ac571a52baf112f8151
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51641
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
James Simmons [Thu, 20 Jul 2023 19:57:36 +0000 (15:57 -0400)]
LU-16878 tests: use RUNAS_UID / RUNAS_GID for NRS TBF
Some of the sanityn NRS TBF test hardcode the use of uid 500 and gid
500. They are not guaranteed to exist so use RUNAS_UID and RUNAS_GID
instead.
Test-Parameters: trivial testlist=sanityn env=ONLY=77
Change-Id: Ie987c70e94918c5cddadb632a4a3a3caac12c96f
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51238
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Arshad Hussain [Tue, 16 May 2023 03:00:49 +0000 (08:30 +0530)]
LU-16796 libcfs: Remove reference to LASSERT_ATOMIC_ZERO
This patch removes all reference to LASSERT_ATOMIC_ZERO macro.
Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Change-Id: I73259599d1dee6277fadf66181699f1282274a80
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51004
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Timothy Day <timday@amazon.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Andriy Skulysh [Thu, 24 Nov 2022 13:18:04 +0000 (15:18 +0200)]
LU-16430 ptlrpc: racy rq_obsolete bit modification
Racy bit modification causes assertion failure in
ptlrpc_at_remove_timed():
ASSERTION( !list_empty(&req->rq_srv.sr_timed_list) )
rq_obsolete is a bit field, so it's modification
isn't atomic and should be modified under rq_lock.
Change-Id: Ib1d3ad189a78b71ecf5b01585478922e984c9568
HPE-bug-id: LUS-11368
Fixes:
23773b32bf ("LU-11444 ptlrpc: resend may corrupt the data")
Signed-off-by: Andriy Skulysh <andriy.skulysh@hpe.com>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/49505
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Shaun Tancheff [Mon, 22 May 2023 12:36:21 +0000 (07:36 -0500)]
LU-16053 build: Update zfs configure checks
From Brian Behlendorf <behlendorf1@llnl.gov>:
update dmu_*_by_dnode checks
Provided as a feature since ZFS 0.7.0, convert to a fatal configure
error when unavailable.
update zap_*_by_dnode checks
Provided as a feature since ZFS 0.7.0, convert to a fatal configure
error when unavailable.
update multihost protection check
Provided as a feature since ZFS 0.7.0, convert to a fatal configure
error when unavailable. Drop the compatibility code required to
support OpenZFS releases older than 0.7.0.
update userobj accounting check
Provided as a feature since ZFS 0.7.0, convert to a fatal configure
error when unavailable.
update dmu_prefetch() check
Provided since at least ZFS 0.7.0, convert to a fatal configure
error when unavailable.
update dmu_object_alloc_dnsize() check
Provided since at least ZFS 0.7.0, convert to a fatal configure
error when unavailable.
update spa_maxblocksize() check
Provided since at least ZFS 0.7.0, convert to a fatal configure
error when unavailable.
update dsl_pool_config_enter/exit check
Convert to a fatal configure error, these functions have
been provided since at least ZFS 0.7.x.
replace sa_spill_block() check
The sa_spill_block() function was removed after the ZFS 0.6.x
release. Replace the check with one for use zio_buf_alloc/free
which have been available since 0.7.x.
The dsl_sync_task_do_nowait() function has not been provided
by since the 0.6.x releases. Furthermore, the results of this
check are unused by Lustre so let's just remove it.
Test-Parameters: trivial
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I3c1597e56100961178f9001e918ffb9aa3558706
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/48089
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Timothy Day [Fri, 21 Jul 2023 20:16:03 +0000 (20:16 +0000)]
LU-6142 obd: remove OBP and MDP macros
These macros save very little space, make it harder
to understand the code (by adding one more thing to
remember) and make it impossible to grep for
o_* and m_* functions. Luckily, they are only used in
a few places. So, remove them and all references.
Test-Parameters: trivial
Signed-off-by: Timothy Day <timday@amazon.com>
Change-Id: I4c23199ca53c906ca190a81ffdf916ff6cff9a0b
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51739
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Timothy Day [Fri, 21 Jul 2023 20:29:32 +0000 (20:29 +0000)]
LU-6142 obd: fix white space, header
Convert all of the remaining spaces to tabs. Also,
add SPDX text to file.
Test-Parameters: trivial
Signed-off-by: Timothy Day <timday@amazon.com>
Change-Id: I2d4e71646f7aaa286f7500564c817c76a4b716ed
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51738
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Lai Siyao [Mon, 10 Jul 2023 04:30:28 +0000 (00:30 -0400)]
LU-11036 test: race in sanity-lfsck test_8
In sanity-lfsck test_8, "sleep 1" is run after START_NAMESPACE,
but it still has chance that LFSCK status is complete but LFSCK
thread not quit yet, therefore the following START_NAMESPACE may fail
with -EALREADY. Just check the first lfsck started scanning.
And similarly use wait_update to check flags for DELAY3.
Test-Parameters: trivial MDSCOUNT=2 MDTCOUNT=4 testlist=sanity-lfsck env=ONLY=8,ONLY_REPEAT=10
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Ie1f612bebb52c4755e5b4e13d58ab8bf2aeb2832
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51720
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
James Simmons [Wed, 19 Jul 2023 16:07:26 +0000 (12:07 -0400)]
LU-9680 utils: add updating the key table for Netlink.
Currently lnetconfig implementation only sends the key table once
to construct a YAML document. Add the ability to update the key
table at a latter time. New keys will be used by the YAML
document.
Test-Parameters: trivial
Change-Id: Ie2201f91eb24d06c7e2a2d4abe3da3805f74e5a7
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51715
Tested-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
James Simmons [Mon, 24 Jul 2023 16:10:17 +0000 (12:10 -0400)]
LU-9680 contrib: share libyaml C code generator
Writing proper libyaml C code is not easy. So I wrote an
application that generates the C code to help the developer not
struggle starting from scratch. It wouldn't be a one to one
copy and paste but it greatly helps. The build the application
just do gcc -lyaml yaml-event-dump.c
Test-Parameters: trivial
Change-Id: I1b570dbfc3ea2e6a7ec77b3743aa4cd80aba2acb
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51508
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Timothy Day <timday@amazon.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Alex Zhuravlev [Tue, 23 May 2023 13:30:58 +0000 (16:30 +0300)]
LU-16843 ldiskfs: merge extent blocks
There are cases (e.g. file written synchronously with discontiguous
blocks that are later filled in) when a lot of extents are created
initially, then the extents get merged over time, but there is no
way to merge the index blocks. This can cause a very deep extent
index tree (above 5 levels) and cause problems like:
inode has invalid extent depth: 6
Merge leave/index blocks (one at each level at most) to right/left
when extents are removed from the index.
submitted to ext4@ maillist:
https://lore.kernel.org/linux-ext4/
7A2B8861-96AA-4815-BB58-
180F63F62436@whamcloud.com/
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I746c0917e746eb442d3c69a23f591d9cdade76fa
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51096
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Artem Blagodarenko <ablagodarenko@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Timothy Day [Fri, 14 Jul 2023 15:42:39 +0000 (15:42 +0000)]
LU-16965 obd: remove unused obd_evict_inprogress
Remove the atomic_t struct field obd_evict_inprogress
from 'struct obd_device'. This field was only ever
incremented in a unused function that was removed in
a previous patch. Hence, remove it altogther. This
patch also removes the associated wait queue.
Test-Parameters: trivial
Signed-off-by: Timothy Day <timday@amazon.com>
Change-Id: Id151c1e6a0adde8c1aeb6dbc903b9d98d00fd21d
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51681
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Lai Siyao [Sat, 8 Jul 2023 20:35:43 +0000 (16:35 -0400)]
LU-15553 test: mkdir_on_mdt0 in recovery-small.sh
Many subtests in recovery-small.sh requires test dir be created on
MDT0, replace mkdir with mkdir_on_mdt0.
Fixes:
b9c4dc3c33 ("LU-14792 llite: enable filesystem-wide default LMV")
Test-Parameters: trivial
Test-Parameters: testlist=recovery-small,recovery-small,recovery-small
Test-Parameters: MDSCOUNT=2 MDTCOUNT=4 testlist=recovery-small,recovery-small,recovery-small
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Ibc37b2dd25bcd94794392f5ff8a79df2e7932dcc
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51669
Tested-by: Maloo <maloo@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Hongchao Zhang [Thu, 6 Jul 2023 17:16:09 +0000 (01:16 +0800)]
LU-16951 test: don't call echo in function call
In sanity-quota, the call '$(get_slave_nr expr "foo")' will fail
if there is "echo" call in "wait_update_facet/wait_update_cond".
Test-Parameters: trivial
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Change-Id: Ib35bf8ccd7eb121a0a2852ba7ed69ad9b01f271a
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51615
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Arshad Hussain [Thu, 6 Jul 2023 10:25:04 +0000 (15:55 +0530)]
LU-16947 tests: On error correctly kill multiop
multiop_bg_pause under test-framework starts multiop
in background and waits for signal if "_" option is
provided. On 'verbose' mode the PAUSING string is
printed on console which is checked and if not found
error is reported by multiop_bg_pause function.
On error, it is required to kill the existing running
multiop binary and if not done will eventually timeout
and not exit the test.
Currently on error multiop_bg_pause function incorrectly
sends signal to wrong PID. This patch fixes this issue.
Test-Parameters: trivial testlist=replay-single mdscount=2 mdtcount=4
Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Change-Id: I3fb505302615512a891725e7339a6f0238c2cdab
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51589
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
James Simmons [Tue, 11 Jul 2023 17:28:13 +0000 (13:28 -0400)]
LU-16663 tests: use python to compare YAML files
For the sanity-lnet test we often compare different YAML files.
This is done with diff which is the wrong tool since two YAML
files that are the same can have different indentations. The
libyaml maintainer states using python tools for this is the
proper supported way to do this.
Test-Parameters: trivial testlist=sanity-lnet
Change-Id: Ie0ef623e8ec729aaad862fc3f33eb0a3b4172fad
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51507
Reviewed-by: Timothy Day <timday@amazon.com>
Reviewed-by: Feng Lei <flei@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Sebastien Buisson [Mon, 19 Jun 2023 11:38:07 +0000 (13:38 +0200)]
LU-16911 sec: quiet messages from identity upcall retry mech
Do not use CERROR to print messages about failed identity acquire
upcalls. And make a difference between initial attempt before retry,
and final failure.
Fixes:
61c3b3a9bb ("LU-16165 sec: retry mechanism for identity cache")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I35e04ca31b623d6037bb49e4ded4ea96d653f074
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51355
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Chris Horn [Thu, 4 May 2023 22:48:28 +0000 (17:48 -0500)]
LU-16483 tests: replay-single test_200 fixes
Modify test to ensure idle disconnect is enabled for all targets
except OST0000. This prevents an issue where an idle ping is sent to
another target instead of OST0000.
Re-work test to check the debug log for all relevant messages.
rcli is not set correctly when RCLIENTS contains multiple hostnames.
Fix it by not surrounding RCLIENTS with double quotes.
Added a debug statement to ptl_send_rpc(), and moved an existing one,
to faciliate debugging any future test failures.
Test-Parameters: trivial clients=3 testlist=replay-single env=ONLY=200,ONLY_REPEAT=100
Fixes: eb1f4a5 ("LU-16483 ptlrpc: Track highest reply XID")
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: If0a214092dad1e40f1b9e785e179ef67f686b85a
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50891
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Lai Siyao [Sat, 13 May 2023 08:25:53 +0000 (04:25 -0400)]
LU-15527 dne: refactor commit-on-sharing for DNE
Commit-on-sharing for DNE is different from the original
commit-on-sharing:
* the original commit-on-sharing is to eliminate dependency between
operations from different clients.
* while commit-on-sharing for DNE is to eliminate dependency between
operations handled by different MDTs, so that upon multiple MDT
failures, an operaiton replay won't fail because its dependent
operation is not replayed by another MDT yet.
Current CoS for DNE implementation checks dependency in MDT layer, and
it decides by checking whether current operation is a distributed
transaction, if so, it will trigger CoS upon conflicting locks.
Actually this may miss some cases that should trigger CoS (even local
transaction should trigger CoS if it depends on a distributed
transaction), and on the other hand it may trigger extra CoS because
if two operations are handled by the same MDT, the dependency is
ensured because they will always be replayed by transaction number.
And to avoid mixing the code of two different CoS, the following
changes are made:
* add new ldlm lock mode LCK_TXN. On DNE system, downgrade PW/EX locks
to this mode after transaction stop.
* add li_initiator_id in struct ldlm_inodebits, which is the index of
MDT where the lock is enqueued, i.e. where operation is handled. If
another operation handled by a different MDT requests a conflicting
PW|EX mode lock against this TXN mode lock, it will trigger commit
to ensure the dependent operation is committed to disk (NB, it
doesn't trigger commit on all involved MDTs, but only the MDT where
the conflict happens, which is enough to allow replay succeed).
* remove LDLM_FL_COS_INCOMPAT and LDLM_FL_COS_ENABLED.
* MDT layer doesn't need to check such dependency any more, since lock
itself knows.
* updated sanityn 33c, 33d and 33e since fewer CoS are triggered now.
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Ib0149fcdc0178afd2c6894d211480f3c6c9284a0
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/46641
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mikhail Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Patrick Farrell [Wed, 1 Mar 2023 15:43:49 +0000 (10:43 -0500)]
LU-13805 llite: Improve sync_io comments
Correct and improve comments on cl_sync_io_wait_recycle.
Test-parameters: trivial
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I7784aa75df46831d1b501c823ec1ada48376b227
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50167
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Qian Yingjin <qian@ddn.com>
Alex Zhuravlev [Mon, 3 Apr 2023 10:32:59 +0000 (13:32 +0300)]
LU-16405 osd: lookup cache
MDT may need to re-lookup just checked names (after locking).
introduce a trivial tiny per-thread cache in OSD in order to
make such a repeating lookup cheap.
the original issue is that ext4_add_entry() doesn't really
check for possible duplicate (that would be expensive as
a whole 4K block must be scanned).
important: the cache is reset upon request processing completion as
we don't update iversion on a disk (due to conflict with VBR).
Fixes:
79acb9a9e7 ("LU-10235 mdt: mdt_create: check EEXIST without lock")
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I40c3ee702f7895c3bda00b380f904cd587e0a1c4
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50521
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Vitaly Fertman [Fri, 31 Mar 2023 18:04:44 +0000 (21:04 +0300)]
LU-15535 llite: deadlock on lli_lsm_sem
it may happen that one process is doing lookup, and after reply while
holding the LDLM lock is trying to update LSM/default LSM under the
write lli_lsm_sem for a dir.
another process has taken the read lli_lsm_sem (taken for all the MD
ops in ll_prep_md_op_data()) and is waiting for a conflicting PW LDLM
lock on server for its modification for this dir.
it may happen on restriping with LSM, on changing the default LSM, but
even more often way is racer run even without striped dirs:
- racer does LFS mkdir -i $i <subdir> per each MDS, what creates a default
LSM on these subdirs inherited endlessly - to keep the MDS index;
- racer also does mkdir -p <path>, in which case we do:
ll_new_node - create a parent dir, no RMF_DEFAULT_MDT_MD in reply
ll_lookup parent it=open - no RMF_DEFAULT_MDT_MD in reply
ll_new_node - create a child
the default LSM is inherited on the parent creation, however as those RPCs
do not have lookup LDLM lock and no data - the default layout is not set
for the parent in inode at the time of a child creation. thus a parallel
lookup which gets the LSM deadlocks with this ll_new_node().
at the same time, similar to CLIO, we do not need to hold a sem nor an
LDLM lock over the whole operation to avoid LSM modification on server,
we just need to take an uptodate LSM (this is a subject for LU-16320)
and to guarantee this op will be working on the client on this LSM for
the whole operation.
the solution is to let MD ops to work on a copy of LSM therefore letting
others to modify LSM attached to inode in parallel if needed.
HPE-bug-id: LUS-10725
Signed-off-by: Vitaly Fertman <vitaly.fertman@hpe.com>
Change-Id: I3137300b5bcce2e890994ce8751cdf7fce2f3f54
Reviewed-on: https://es-gerrit.hpc.amslabs.hpecorp.net/161525
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Tested-by: Vitaly Fertman <c17818@cray.com>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50489
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Vitaly Fertman [Fri, 31 Mar 2023 17:30:27 +0000 (20:30 +0300)]
LU-15535 revert: "LU-15284 llite: access lli_lsm_md with lock in all places"
This reverts commit
1dfae156d1dbc11cfb77b2d35cbffb2da7f28137
as a prerequisite of a larger fix in this ticket which covers
this problem as well.
Signed-off-by: Vitaly Fertman <vitaly.fertman@hpe.com>
Change-Id: Ic1b0b6c963ea96e9f51324625deaa851245f8a7d
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50488
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Shaun Tancheff [Tue, 11 Jul 2023 12:42:27 +0000 (19:42 +0700)]
LU-16667 build: struct mnt_idmap, linux/filelock.h
Linux commit v6.2-rc3-9-g5970e15dbcfe
filelock: move file locking definitions to separate
header file
Add configure test for linux/filelock.h and include it
where needed.
linux kernel v6.2-rc1-4-gb74d24f7a74f
fs: port ->getattr() to pass mnt_idmap
linux kernel v6.2-rc1-3-gc1632a0f1120
fs: port ->setattr() to pass mnt_idmap
Add a configure test for mnt_idmap and fallback to using
user_namespace for older kernels.
Test-Parameters: trivial
HPE-bug-id: LUS-11557
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: Ib8cbb3157fb11b4f1fc55f1626c2998cb202bd8c
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/50420
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Alex Zhuravlev [Mon, 13 Dec 2021 11:47:59 +0000 (14:47 +0300)]
LU-15365 tests: conf-sanity/115 to cleanup properly
when /tmp can't fit large MDT filesystem the image
should be removed after all.
Test-Parameters: trivial testlist=conf-sanity env=ONLY=115
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Ifb0bd201156f4beb665f3c38aa02d44802b13bbf
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/45836
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Jian Yu [Fri, 14 Jul 2023 03:31:21 +0000 (11:31 +0800)]
LU-16916 tests: fix client_evicted() not to ignore EOPNOTSUPP
After RHEL 9.x or Ubuntu 22.04 client is evicted, "lfs df" returns
error code 95 (EOPNOTSUPP), which is ignored in check_lfs_df_ret_val()
and then causes client_evicted() to ingore that error.
This patch fixes client_evicted() to check the return value
from "lfs df" directly so as not to ignore EOPNOTSUPP.
Test-Parameters: trivial clientdistro=el9.2 testlist=replay-vbr
Test-Parameters: trivial clientdistro=el8.8 testlist=replay-vbr
Test-Parameters: trivial clientdistro=ubuntu2204 testlist=replay-vbr
Change-Id: I633ae8769fc563b8068f433e2afae29463ac5553
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51667
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Alex Deiter [Wed, 12 Jul 2023 20:58:02 +0000 (00:58 +0400)]
LU-15046 tests: skip sanity-flr/test_200c for old MDS
Skip sanity-flr test_200c for old MDS missing the fix
for synchronized replicas and its corresponding test.
Fixes:
b7ec0d2390 ("LU-15046 osp: precreate thread vs connect race")
Test-Parameters: trivial testlist=sanity-flr env=ONLY=200c
Signed-off-by: Alex Deiter <adeiter@tintri.com>
Change-Id: I87cd7d6b767086f993a27ce6905b05f87e325474
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51649
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sarah Liu <sarah@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Alex Deiter [Wed, 12 Jul 2023 20:36:37 +0000 (00:36 +0400)]
LU-16341 tests: skip sanity-quota/test_1b for old MDS
Skip sanity-quota test_1b for old MDS missing the fix
for LU-16341 kernel NULL in qmt_site_recalc_cb.
Fixes:
d965d63415 ("LU-16341 quota: fix panic in qmt_site_recalc_cb")
Test-Parameters: trivial testlist=sanity-quota env=ONLY=1b
Signed-off-by: Alex Deiter <adeiter@tintri.com>
Change-Id: I1b1bc3fdfa8f36b0c20a9a06721735c8e02c034c
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51648
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Sergey Cheremencev [Tue, 11 Jul 2023 14:28:12 +0000 (18:28 +0400)]
LU-16913 quota: fix ASSERTION(lqe->lqe_gl)
It is possible to add in a 2nd time lqe into qmt_reba_list while
handling of the 1st from the 1st time is not finished. There is a
small window in qmt_id_lock_glimpse when lqe_link is empty but
lqe_gl is not set.
Signed-off-by: Sergey Cheremencev <scherementsev@ddn.com>
Change-Id: I1168903bff88df7e5106186b082e8065a6480367
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51629
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Mikhail Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Lei Feng [Tue, 11 Jul 2023 00:13:35 +0000 (08:13 +0800)]
LU-16953 tests: wait longer in replay-dual/test_31
Wait until file was created in replay-dual/test_31.
Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial testlist=replay-dual env=ONLY=31,ONLY_REPEAT=100
Change-Id: I847beb51d53e667f1599c9693aa5eb099dcf9435
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51621
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Thomas Bertschinger [Wed, 5 Jul 2023 14:19:22 +0000 (10:19 -0400)]
LU-16944 utils: lfs find: handle multiple paths correctly
When lfs find is used with multiple paths and the first non-path
option is a '!' or an option without an argument like '-print',
the code skipped the final path because it assumed that the first
non-path option would be an option with an argument.
This commit resolves the bug by remembering the last-processed argv
index so that the index of the final path argument is correct whether
the next option consumes 2 indexes or just 1.
Signed-off-by: Thomas Bertschinger <bertschinger@lanl.gov>
Change-Id: I03133a43641af7a53a20d947b8ef82529e453251
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51578
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Timothy Day <timday@amazon.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Shaun Tancheff [Tue, 27 Jun 2023 08:10:19 +0000 (15:10 +0700)]
LU-16893 libcfs: Remove force_sig usage from lfsck
The lfsck pool of kernel threads uses force_sig() to signal
the worker threads to stop. A signal is used here as the
lfsck workers may be waiting in various, and possibly
nested, states.
As force_sig() has been removed let us simply enable SIGINT
to be passed to the worker threads using send_sig().
Test-parameters: testlist=sanity-lfsck,lfsck-performance
HPE-bug-id: LUS-11670
Fixes:
db9f9543ec ("LU-12634 libcfs: force_sig() removed task parameter")
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: Ibf6a67f43687960b3eff9cb9a7c7dc8b1be1da63
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51470
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Arshad Hussain [Mon, 29 May 2023 07:31:26 +0000 (03:31 -0400)]
LU-16796 libcfs: Remove reference to LASSERT_ATOMIC_GT_LT
This patch removes all reference to LASSERT_ATOMIC_GT_LT macro.
Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Change-Id: I9acf820b32855e54369c18470fb1b73d7f08c41a
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51157
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Bobi Jam [Wed, 21 Jun 2023 06:33:21 +0000 (14:33 +0800)]
LU-16831 lfs: limit stripe count for component size
If stripe count is larger than component_size/stripe_size, some
allocated OST objects are created but inaccessible. This patch
reduces the number of stripes in that case to avoid this.
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I117ed8a7696c6c6adcdd0c2c6531a958cc53bd51
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51143
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Mr NeilBrown [Wed, 7 Dec 2022 07:55:57 +0000 (18:55 +1100)]
LU-15969 llite: add support for ->fileattr_get/set
From Linux 5.13, FS_IOC_SETFLAGS and GETFLAGS aren't passed down to
the filesystem, we need ->fileattr_get/set inode_operations instead.
Signed-off-by: Mr NeilBrown <neilb@suse.de>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Change-Id: Ib3ffba3529ea32b702ad80abd4b9e4e3ad90b412
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/51107
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
git://git.whamcloud.com / fs / lustre-release.git / log