git://git.whamcloud.com - fs/lustre-release.git/log

EX-4166 lipe: lamigo test coverage for OSS

Add test for lamigo ALR with multiple OSS's
Add debug trace point to report update message
from ofd_access_log_reader

Test-Parameters: trivial testlist=hot-pools
Test-Parameters: trivial testlist=hot-pools mdscount=2 osscount=2 mdtcount=2 ostcount=8
Signed-off-by: Alexandre Ioffe <aioffe@ddn.com>
Change-Id: Iaae847190426ff34d8991e8a571b3e38616bc4c9
Reviewed-on: https://review.whamcloud.com/46686
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

EX-4866 lipe: don't unmount the local client

Exclude local client from unmount list when test is
completed in hot-pools test framework.

Test-Parameters: trivial testlist=hot-pools
Signed-off-by: Alexandre Ioffe <aioffe@ddn.com>
Change-Id: I6b12269b6af3d3b5465645cbc007c9a5302f64a1
Reviewed-on: https://review.whamcloud.com/46671
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

EX-4539 lipe: lipe_find3 print updates

Change -print-json to accept a comma separated list of
attributes. Optional attributes may be specified by placing them
inside brackets. For example "lipe_find3 DEVICE -print-json
'uid,gid,som,[size,blocks]' will only print JSON for inodes with a
valid UID, GID, and SoM atrribute. If in addition the size and blocks
attributes of the inode are valid then they will be included in the
object as well. Support a pass-through --list-json-attrs option.

Change the default action to print to relative path. Adjust
sanity-lipe-find3 accordingly.

Test-Parameters: trivial testlist=sanity-lipe-find3 serverextra_install_params="--packages lipe-scan"
Test-Parameters: trivial testlist=sanity-lipe-scan3 serverextra_install_params="--packages lipe-scan" facet=mds1
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Id380ca21e2b1aabf30f65fd3e14b7e2f7808d0a6
Reviewed-on: https://review.whamcloud.com/46630
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-4015 lipe: add -blocks, -crtime, -mirror-count, -stripe-count

Add -blocks, -crtime, -mirror-count, and -stripe-count to
lipe_find3. Add (crtime), (lov-mirror-count), and (lov-stripe-count)
to lipe_scan3.

Test-Parameters: trivial testlist=sanity-lipe-scan3 serverextra_install_params="--packages lipe-scan" facet=mds1
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I5b4314a9621309b00453fea637329d3de442544a
Reviewed-on: https://review.whamcloud.com/46607
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-4015 lipe: batching and threading improvements

Remove the current group descriptor mutex from struct
scan_control. Use __ATOMIC_RELAXED fetch and add to allocate the next
batch of groups. Report the correct start group of the batch in
debugging output and remove a redundant batch debug message.

Use atomic loads and stores for the ti_should_stop member which is
responsible for lipe-scan-break. Check if the current thread should
stop in the outer loop of ls3_scan_thread_start_scm() as well as the
inner loop of ldiskfs_scan_groups().

Reduce the default scanning thread count from _SC_NPROCESSORS_ONLN / 2
to _SC_NPROCESSORS_ONLN / 4.

Test-Parameters: trivial testlist=sanity-lipe-scan3 serverextra_install_params="--packages lipe-scan" facet=mds1
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Ic99c27504333f1d63a689e091d857e44062ef584
Reviewed-on: https://review.whamcloud.com/46605
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-4015 lipe: add lipe-scan RPM

Adding new dependencies to existing EXAScaler RPMs may create
headaches when distributing hotfixes to existing installs. So move
lipe_find3 and lipe_scan3 to a new RPM (lipe-scan). This also has the
benefit of explicitly severing the new scanning tools from any python2
RPM or pip dependencies.

Compile fid.scm and find.scm to (%site-ccache-dir)/lipe/.

Test-Parameters: trivial testlist=sanity-lipe-find3 serverextra_install_params="--packages lipe-scan"
Test-Parameters: trivial testlist=sanity-lipe-scan3 serverextra_install_params="--packages lipe-scan" facet=mds1
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Ifecb5ab1f399ba9be8cb395ded29d6394b13dc86
Reviewed-on: https://review.whamcloud.com/46572
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-4539 lipe: remove -coverage

Remove -coverage from CFLAGS for lipe_scan3 and lipe_find3.

Test-Parameters: trivial
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I8be003650574104d5eaa8298043ba789e6464fde
Reviewed-on: https://review.whamcloud.com/46532
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-4539 lipe: add -pool to lipe_find3

Add a -pool test to lipe_find3. Add sanity-lipe-find3 test_110() to
verify.

Test-Parameters: trivial testlist=sanity-lipe-find3
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I9649d7f80431d22223da17372ec4d64fa6ca2f37
Reviewed-on: https://review.whamcloud.com/46584
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-4539 lipe: add -perm to lipe_find3

Fill in the -perm test in lipe_find3. Populate sanity-lipe-find3
test_101() to verify.

Test-Parameters: trivial testlist=sanity-lipe-find3
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Ib201503247101619416c39ae97f5068230441863
Reviewed-on: https://review.whamcloud.com/46576
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-4539 lipe: add lipe_find3

Add a lipe_find3 wrapper around the lipe_scan3 scanner and test script
sanity-lipe-find3.sh.

Test-Parameters: trivial testlist=sanity-lipe-find3
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I2259170e8b71a94394009aeaf9878a17c2a3fa6d
Reviewed-on: https://review.whamcloud.com/46417
Tested-by: jenkins <devops@whamcloud.com>

EX-4015 lipe: add make-prompt-tag hack

Hack guile (make-prompt-tag ...) to return a list instead of a
gensym. This reduces catch overhead in multi-threaded code and is what
guile does eventually (see guile commit 283ab48d3f). Add this along
with some comments to ls3_scm_init() and rename that function to
ls3_module_init().

Test-Parameters: trivial testlist=sanity-lipe-scan3 facet=mds1
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I2ddd324b290ee4e985bba171681927b9434bbcc5
Reviewed-on: https://review.whamcloud.com/46571
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15559 tests: add do_node_vp() and do_facet_vp()

Add new test-framework functions (do_node_vp() and do_facet_vp())
which carefully escape and quote command lines for execution on the
local or remote node. Add sanityn test_0 to verify.

Lustre-change: https://review.whamcloud.com/46535

Test-Parameters: trivial env=ONLY="0" testlist=sanityn
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Ic491b0148e6ef11ecd0b3ccce983afcf4d1300e5
Reviewed-on: https://review.whamcloud.com/46537
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-4015 lipe: add lma, filter_fid, and hsm json attributes

Add "lma", "filter_fid", and "hsm" json attributes. Use struct
lustre_mdt_attrs rather than broken out fields in loa. Add
sanity-lipe-scan3 tests 111, 112, 113 to "verify".

Combine init_lipe_scan3_env and init_lipe_scan3_env_file.

Test-Parameters: trivial testlist=sanity-lipe-scan3 facet=mds1
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I1856a896d192d08f9e16b9ac764030907256f79c
Reviewed-on: https://review.whamcloud.com/46547
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-4015 lipe: cache layout in loa

In ldiskfs_read_attr_lov(), cache the decoded llapi_layout() in the
current object attrs. Then resue this layout in lov-pools. Add
lov-ost-indexes to return a list of all object OST indexes for the
current file.

Test-Parameters: trivial testlist=sanity-lipe-scan3 facet=mds1
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I3451857fc25f1f9507b6e185bad39fcb3f0e6f22
Reviewed-on: https://review.whamcloud.com/46546
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-4015 lipe: reduce ls3_object_attrs size

Reduce the size of struct ls3_object_attrs from 131KB to 360B by heap
allocating the link and lmv xattr buffers.

Test-Parameters: trivial testlist=sanity-lipe-scan3 facet=mds1
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I5b210cb46eb027dbc922675deb1231a544b93d6a
Reviewed-on: https://review.whamcloud.com/46544
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-4015 lipe: define lipe scheme module

Avoid spurious "possibly undefined symbol" warnings from the guile
compiler by placing all of the snarfed definitions into a "lipe"
module. Add the fid accessors to a "lipe fid" module.

Test-Parameters: testlist=sanity-lipe-scan3 facet=mds1
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Ifbfee81422b1a3df22ee23f1945577c29e485aec
Reviewed-on: https://review.whamcloud.com/46525
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-4015 lipe: handle trusted xattrs uniformly

Add a wrapper (ldiskfs_trusted_xattr_get()) around ext2fs_attr_get()
to do uniform error messages and error handling.

Test-Parameters: trivial testlist=sanity-lipe-scan3 facet=mds1
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I5ad82a56b7729354364afa594b3d8d9ee83a4b7f
Reviewed-on: https://review.whamcloud.com/46513
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-4015 lipe: add fid2path cache to lipe_scan3

Add a thread local directory fid2path cache to lipe_scan3. Without the
cache, as single scanning thread could expect to do about 3K fid2path
operations per second. After the cache the rate improves to about
70K. We set the max cache size to 1024 FIDs and use LRU to reclaim
slots. Based on this a full cache will use about 4MB of memory per
thread.

Test-Parameters: testlist=sanity-lipe-scan3 facet=mds1
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I8a022665de78e6b599f2b4c4f1e2b7400d4d8ffe
Reviewed-on: https://review.whamcloud.com/46509
Tested-by: jenkins <devops@whamcloud.com>

EX-4015 lipe: add lipe_scan3

Add a guile embedded lipe scanner (lipe_scan3) and test script
sanity-lipe-scan3.sh.

Test-Parameters: testlist=sanity-lipe-scan3 facet=mds1
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I059fb4044db5baff76a04247fb8e3cbec82e5448
Reviewed-on: https://review.whamcloud.com/46416
Tested-by: jenkins <devops@whamcloud.com>

LU-15068 ptlrpc: Do not unlink difficult reply until sent

If a difficult reply is queued in LNet, or the PUT for it is
otherwise delayed, then it is possible for the commit callback
to unlink the reply MD which will abort the send. This results in
client hitting "slow reply" timeout for the associated RPC and
an unnecessary reconnect (and possibly resend).

This patch replaces the rs_on_net flag with rs_sent and rs_unlinked.
These flags indicate whether the send event for the reply MD has
been generated, and whether the MD has been unlinked, respectively.

If rs_sent is set, but rs_unlinked has not been set, then ptlrpc_hr
is free to unlink the reply MD as a result of the commit callback.
The reply-ack will simply be dropped by the server.

If ptlrpc_hr is processing the reply because of commit callback, and
rs_sent has not been set, then ptlrpc_hr will not unlink the reply
MD. This means that the reply_out_callback must also be modified to
check for this case when the send event occurs. Otherwise, if the ACK
never arrives from the client, then the MD would never be unlinked.
Thus when the send event occurs, and rs_handled is set, the
reply_out_callback will schedule the reply for handling by ptlrpc_hr.

Lustre-change: https://review.whamcloud.com/45138
Lustre-commit: 5c156b48425aae245537aaf10229734166463347

HPE-bug-id: LUS-10505
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: Ib8f4853c7ab35d72624fce7ee3fba9e59a746e1f
Reviewed-by: Andriy Skulysh <andriy.skulysh@hpe.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Signed-off-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46751
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14865 utils: llog_reader.c printf type mismatch

Add (unsigned long long) cast to results of
__le64_to_cpu so that it matches the formatting (%llu)
of the enclosing printf call.

Build log message:
"llog_reader.c:887:9: error: format '%llu' expects
argument of type 'long long unsigned int', but
argument 3 has type '__u64' [-Werror=format=]"

Lustre-change: https://review.whamcloud.com/44346
Lustre-commit: 14b8276e06d6f4e3bfe785df1165458555e406f3

Test-Parameters: trivial
Fixes: 9962d6f84db5 LU-14617 utils: llog_reader updatelog support
Signed-off-by: Gian-Carlo DeFazio <defazio1@llnl.gov>
Change-Id: I9549e0a0bd21727dfcc42992b693bc39a779e1a1
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46757
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14617 utils: llog_reader updatelog support

The patch adds printing UPDATE_REC for llog_reader. It is usefull
for updatelog analyze. Here is an example of record

[0x50001a21b:0x1233d:0x0] type:xattr_set/7 params:3 p_0:0 p_1:1 p_2:2
[0x50001a211:0x475:0x0] type:xattr_set/7 params:3 p_0:0 p_1:1 p_2:2
[0x3800182e3:0x475:0x0] type:xattr_set/7 params:3 p_0:0 p_1:1 p_2:2
[0x200032c9a:0x245:0x0] type:xattr_set/7 params:3 p_0:0 p_1:1 p_2:2
[0x200000001:0x15:0x0] type:write/12 params:2 p_0:3 p_1:4
p_0 - 12/trusted.lov
p_1 - 0/
p_2 - 25972/\x0100000000000000000000000000000000000000000002000...
p_3 - 25974/\x0800000000000000P\xD1AB006x0000000400EC^\x000000...
p_4 - 1/

llog logic processing base on incrementing record index,
the fix adds checks for it. Also adds more info from header,
and drops useless - Bit X not set.

Lustre-change: https://review.whamcloud.com/43343
Lustre-commit: 9962d6f84db5fd587bbe13640a9361c2872f3728

Test-Parameters: trivial
Signed-off-by: Alexander Boyko <alexander.boyko@hpe.com>
Change-Id: Id50de15040526dc07ae708ac5db046832706be31
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46756
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14876 out: don't connect to busy MDS-MDS export

MDS-MDS connection is missing check for busy requests upon
reconnect, so resent can be executed concurrently with
original request.

- in ptlrpc_server_check_resend_in_progress() remove exception
  for bulk requests, they can be compared by XID nowadays.
  This prevents OUT requests vs resent execution as well.
- fix messages in target_handle_connect() to report correct
  information about connection details
- in out_handle() check for last_xid only once per OUT_UPDATE
- test 110m is added to recovery-small to reproduce the issue

Lustre-change: https://review.whamcloud.com/44390
Lustre-commit: 301d76a71176c186129231ddd1323bae21100165

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I2ad183674d59a2cdeab0037bd8551c607b10ffeb
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46762
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-4583 lipe: show all lpcc information in 'lpcc status' command

Collect lpcc config, lpcc_purge stats, and lustre stats related to
lpcc to together and show in 'lpcc status' command. 'lpcc status-all'
is an alias of 'lpcc status'. The output of 'lpcc status' looks like:
{
    "/mnt/lustre": {
        "pcc": [
            {
                "mount": "/mnt/lustre",
                "cache": "/mnt/pcc",
                ...
            },
            {
                "mount": "/mnt/lustre",
                "cache": "/mnt/pcc2",
                ...
            }
        ],
        "fs_stats": {
        }
}

Change-Id: I032763fb3b45646330b13f5cef34ce8658bddfe4
Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial testlist=sanity-pcc
Reviewed-on: https://review.whamcloud.com/46191
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-4433 pcc: add some statistics data

Add statictics of the number and total size of pcc attached files
and pcc hit files.

Change-Id: Ib0e429c636298d4c6ff06d84a416073895b86184
Signed-off-by: Lei Feng <flei@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/45976
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-4893 wrong dependencies for lustre-client-modules deb package

Fixed dependencies for DKMS deb package:
- added autocon, automake and libtool
- added bison and flex
- added required dev packages
- added linux-base and linux-image
- added python3-distutils-extra to fix build on Ubuntu 16.04

Change-Id: Ic1d05155cd8ad056dece1d22d0f040695d038652
Signed-off-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-on: https://review.whamcloud.com/46604
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15142 lctl: fixes for set_param -P and llog_print

- properly handle permanent param deletion
- don't print skipped parameters in llog_print output
- add --raw option to llog_print to output all entries
including markers

Lustre-change: https://review.whamcloud.com/45332
Lustre-commit: 2a5b50d207173ca1ac71be8dfc39f98a2773bc3a

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: Id93a206a255dc885343efa293e1ee2672493e5e5
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46638
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-13107 utils: clean up lctl command usage

The lctl usage is confusing because it lists a number of valid
commands after "testing (DANGEROUS)", such as LFSCK and llog.

Move the useful commands before the "testing" section so it is
not mis-interpreted as all following commands are dangerous.
Group some other commands together with more related commands,
rather than whatever order they happened to be imlpemented in.

Remove function prototypes for commands that no longer exist.

Lustre-change: https://review.whamcloud.com/37108
Lustre-commit: b0efebdaef52d8ac9b02857166ceb00079612ebc

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I469f9c92953762cc46a68e44238c4b67ebacab07
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Stephane Thiell <sthiell@stanford.edu>
Reviewed-on: https://review.whamcloud.com/46637
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>

LU-15316 tests: use integers in sanity test_255a

The [[ ... > ... ]] operator doesn't really compare floats, it
compares strings. That works as expected if the strings are
the same length, but fails for comparisons like [[ 32 > 123 ]].
Use (( ... > ... )) for comparisons, and only use integer values.

This test has been failing intermittently forever, but the error
was ignored because of running in a VM.

Lustre-change: https://review.whamcloud.com/46350
Lustre-commit: TBD (from a96a4a5894bef714b19086fa09918080f05a7674)

Test-Parameters: trivial
Fixes: f3b8f3fad502 ("tests: fix float comparison in sanity test_255a")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I6787082cd579ae3f1bdd43222a739c939d3ebbe5
Reviewed-on: https://review.whamcloud.com/46618
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-4889 configure script does not check for required build tools

- added check for flex and bison
- added requirement for build kernel modules

Change-Id: I4f4f19ea44f3cd8f69482d950970bf701e81f7ec
Signed-off-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-on: https://review.whamcloud.com/46602
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn38

New tag 2.14.0-ddn38

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Id8a05c166302a48b4553ec76922b70c665763277

LU-15340 llite: Delay dput in ll_dirty_page_discard_warn

Otherwise we can be final dput and need to wait for pages
to clear which is bad because this is called from ptlrpcd
that is not supposed to block esp. for network traffic as
it can cause livelocks if it happens to be needed to kill
the very same RPC we are waiting on.

Additionally pass in the inode from IO since the page
we are using might come from directio and that is
probably not even a valid inode.

Lustre-change: https://review.whamcloud.com/45784
Lustre-commit: a1d75780ba19cfca53cbacf0d38e8d7df540b209

Fixes: 624a3ac23393 ("LU-921 llite: warning in case of discarding dirty pages")
Change-Id: Ie2f1a34047145202c11a4e1a0b18b2e01d9e4601
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-on: https://review.whamcloud.com/46635
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-4408 lipe: collect some statistic data with lpcc_purge

Collect the number of cached files, min/max/avg file size,
min/max/avg age of files in LPCC. Scan cache device forcefully
even if it is not full enough to collect the statistics data
in time.

Change-Id: Id716d4689c83ecc5754e41734e44e7c051d36a8e
Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial testlist=sanity-pcc
Reviewed-on: https://review.whamcloud.com/45937
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15381 hsm: update size upon completion of data version

We found a HSM retore followed by a HSM release will set the
file size with 0 wrongly during the tests.
The reason is that the file size and blocks information is
incorrect obtained via @ll_merger_attr().
The data version operation will flush dirty pages from all
clients, the size and blocks information returns from the Lustre
OST is correct.
In this patch, we update the size and block attributes for a file
upon the completion of the data version operation accordingly.
By this way, HSM release will set the size and blocks information
correctly after data version ioctl operation.

Add sanity-hsm test_261.

lustre-change: https://review.whamcloud.com/45935
lustre-commit: dd3b5601ec6905b00d07cbcb8c139c46dd555b3b

Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: Ifdbf6b58ecd00dc9677a2328438ef68529b72882
Reviewed-on: https://review.whamcloud.com/45935
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Artem Blagodarenko <artem.blagodarenko@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46336
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14124 target: set OBD_MD_FLGRANT in read's reply

If tgt_grant_shrink() decides to not shrink grants - a client is
supposed to restore its cl_grant_avail in osc_update_grant(). In case
of read OBD_MD_FLGRANT is not set on reply's body->oa.o_valid, so
osc_update_grant() misses the cl_grant_avail update. As result server
keeps thinking that client has a lot of grants while a client thinks
that it is missing grants badly. That may lead to performance
degradation.

A test to illustrate the issue is included.

Lustre-change: https://review.whamcloud.com/43375
Lustre-commit: 4894683342d77964daeded9fbc608fc46aa479ee

Change-Id: Ibe7ce0af5701226c8be3ae3f9ad57c354791fa0f
HPE-bug-id: LUS-9943
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46468
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-12807 tests: fix intermittent runtests failure

Occasional runtests failures are seen in full testing on ldiskfs.
Increase the llog space limit to 72KB from 50KB due to seeing a
regular failures in the 52/64KB range.

Lustre-change: https://review.whamcloud.com/44614
Lustre-commit: 14d07b623731233a62a8acd021c8ccdcb2705371

Test-Parameters: trivial testlist=runtests
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I6e272fe9fec21a650110a42efe31a1dc48e35854
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Vikentsi Lapa <vlapa@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46463
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>

LU-12752 mdt: commitrw_write() - check dying object under lock

If process writes to unlinked file the following race between
mdt_commitrw_write() and mdd_close() may occur because
mdt_commitrw_write() checks whether an object is dying without lock:

mdt_commitrw_write() checks lu_object_is_dying(&mo->mot_header) and it
not yet

mdd_close() interposes and destroys the object via
  mdo_destroy()
    lod_destroy()
      lod_sub_destroy()
        osd_destroy()
          obj->oo_destroyed = 1;

mdt_commitrw_write() continues, locks the object and returns ENOENT
from

  dt_attr_get()
    osd_attr_get()
      if (unlikely(obj->oo_destroyed))
        return -ENOENT;

If the file is built of DoM and raid component ll_delete_inode() calls
cl_sync_file_range() which is to iterate over both mdt and raid
components via mdc_io_fsync_start() and osc_io_fsync_start().  As
mdc_io_fsync_start() fails with -ENOENT due to failed write rpc,
osc_io_fsync_start() does not get called. Then
truncate_inode_pages_final() finds not-discarded pages and fails with:

  (osc_page.c:183:osc_page_delete()) Trying to teardown failed: -16
  (osc_page.c:184:osc_page_delete()) ASSERTION( 0 ) failed:
  (osc_page.c:184:osc_page_delete()) LBUG

Test to illustrate the issue is added.

The fix is to call lu_object_is_dying() under object lock.

Lustre-change: https://review.whamcloud.com/41797
Lustre-commit: d48a0ebb5a8d5d49684325434b503e8aab085397

Change-Id: I463c8a6f85d4f5fd934b167c6194f50ae9d4b7d4
Signed-off-by: Vladimir Saveliev <c17830@cray.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46612
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-13055 libcfs: allow comma-separated masks

For debug and changelog mask names, allow a comma-separated list
of names to be given, so that the space-separated list does not
need to be quoted for use.

Change sanity-quota to use a comma-separated list to verify it works.

Fix a couple of test cases where the debug parameter is set and
printed overly verbosely during tests.

Lustre-change: https://review.whamcloud.com/43741
Lustre-commit: 6b6fde1026311a28595ea43af56392ca6ad24d79

Test-Parameters: trivial testlist=sanity-quota
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Icf1e3ebc74f0e48b38a65486b2275ec4c33ebbe5
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46759
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15218 quota: delete unused quota ID

Add lfs option '--delete' to delete unused quota ID.

Lustre-change: https://review.whamcloud.com/45548
Lustre-commit: 78be823f33396819724330d7154f054c52e11944

Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Change-Id: I0d8e6b61dc23c7b22b6054bcced087b8dc94a277
Reviewed-on: https://review.whamcloud.com/46610
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13799 llite: Move free user pages

It is incorrect to release our reference on the user pages
before we're done with them - We need to keep it until the
i/o is complete, otherwise we access them after releasing
our reference.  This has not caused any known bugs so far,
but it's still wrong.

So only drop these references when we free the aio struct,
which is only freed once i/o is complete.

Also rename free_user_pages to release_user_pages, because
it does not free them - it just releases our reference.

This also helps performance by moving free_user_pages to
the daemon threads.  This is a 5-10% boost.

This patch reduces i/o time in ms/GiB by:
Write: 18 ms/GiB
Read: 19 ms/GiB

Totals:
Write: 180 ms/GiB
Read: 178 ms/GiB

mpirun -np 1  $IOR -w -r -t 64M -b 64G -o ./iorfile --posix.odirect

With previous patches in series:
write     5183 MiB/s
read      5201 MiB/s

Plus this patch:
write        5702 MiB/s
read         5756 MiB/s

Lustre-change: https://review.whamcloud.com/39443
Lustre-commit: 7f9b8465bc1125e51aa29cdc27db9a9d6fdc0b89

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I5cf2201e5fd4eeee5b4c996de51d3a6a5394ae34
Reviewed-on: https://review.whamcloud.com/44685
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13799 llite: Remove unnecessary page get/put

Part of the aio cleanup code has the slightly strange
behavior of doing get on every page before calling page
cleanup, then doing a put after.

This was required because we call cl_page_list_del before
calling cl_page_delete, and cl_page_list_del was holding
the last reference on the page struct.

If we reverse the order, then we don't need the extra
get/put to keep the pages live. This should save
significant CPU time in the ptlrpcd threads when finishing
i/o, since this removes a get/put on every page.

Lustre-change: https://review.whamcloud.com/44293
Lustre-commit: c2e94f08cf3ff000b350faf61b6d25ebbad7970e

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Ia9e7bf4a331a5220bfb1eab43493ac5fda59c611
Reviewed-on: https://review.whamcloud.com/44688
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13799 llite: Do not get/put DIO pages

We've already told the kernel we're working with these pages
using the get/put_user_pages functions, and userspace must
hold references on them throughout the i/o anyway.

So getting/putting these vmpages is unnecessary.  This
saves around 7% of the time in DIO page submission, netting
about that much of a performance improvement.

This patch reduces i/o time in ms/GiB by:
Write: 22 ms/GiB
Read: 19 ms/GiB

Totals:
Write: 135 ms/GiB
Read: 143 ms/GiB

mpirun -np 1  $IOR -w -r -t 64M -b 64G -o ./iorfile --posix.odirect

With previous patches in series:
write     6470 MiB/s
read      6354 MiB/s

Plus this patch:
write     7531 MiB/s
read      7179 MiB/s

Lustre-change: https://review.whamcloud.com/39438
Lustre-commit: 881b4c722296ff7ac22c6fd7988363f2cdad9f1e

Signed-off-by: Patrick Farrel <pfarrell@whamcloud.com>
Change-Id: Icfd5bc73ba4254898d6051f11ab6aea624948763
Reviewed-on: https://review.whamcloud.com/44686
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13799 llite: Implement lower/upper aio

This patch creates a lower level aio struct for each set of
pages submitted, and attaches that to the llite level aio.

That means the completion of i/o (in the sense of
successful RPC/page completion) is associated with the
lower level aio struct, and the higher level aio waits for
the completion of these lower level structs. Previously,
all pages were associated with the upper level (and only)
aio struct.

This patch is a reorganization/cleanup, which is necessary
for the next patch, which moves release pages to aio_end.
The justification for this (correctness and performance)
will be provided in that patch.

Lustre-change: https://review.whamcloud.com/44209
Lustre-commit: 46ff76137160b66f1d4437b3443859027faae9c4

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I2d1c01875c5478c54d7cdd93af7ff12bb4928d94
Reviewed-on: https://review.whamcloud.com/44684
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13799 osc: Always set aio in anchor

We currently do not set csi_aio for DIO and use this to
control when we free the aio struct. (For AIO, we must
free it in cl_sync_io_note, but for other users, we have to
wait until after cl_sync_io_wait has been called.)

The lack of csi_aio causes trouble for the implementation
of the next patch, so instead we always set it and control
freeing by checking at that time if we are doing DIO.

Lustre-change: https://review.whamcloud.com/44153
Lustre-commit: eadccb33ac4bbe54a01da5168f6170702f9b2629

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I3590709c1cb891f1c111d611a438fd348e833787
Reviewed-on: https://review.whamcloud.com/44683
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13799 llite: Simplify cda_no_aio_complete use

It is better to handle AIO and DIO the same as much as
possible, limiting the difference to setup if possible.

In this spirit, move the check for DIO (is_sync_kiocb()) to
the setup function rather than cleanup and just use
no_aio_complete.

Lustre-change: https://review.whamcloud.com/44154
Lustre-commit: b60bd21ec5d5f34ed79c63158860b6f5e948dba2

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Iefc0865aee20958960f48539ad73c80c997ff0b4
Reviewed-on: https://review.whamcloud.com/44682
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13799 lov: Cache stripe offset calculation

Calculating the page offset relative to the stripe (etc)
in a file is surprisingly expensive.  Because i/o has
already been split up to stripes by the cl_io code,
calculating the stripe each time is unnecessary.

We cache most of the values requiring calculation.

This improves AIO/DIO page submission significantly,
improving performance by a bit over 10%.

Also remove lpg_generation, which isn't doing anything
useful.  This suggests the possibility of removing
lov_page, but that's for another patch.

This patch reduces i/o time in ms/GiB by:
Write: 17 ms/GiB
Read: 22 ms/GiB

Totals:
Write: 119 ms/GiB
Read: 121 ms/GiB

mpirun -np 1  $IOR -w -r -t 64M -b 64G -o ./iorfile --posix.odirect

With previous patches in series:
write        7531 MiB/s
read         7179 MiB/s

Plus this patch:
write        8637 MiB/s
read         8488 MiB/s

Lustre-change: https://review.whamcloud.com/39445
Lustre-commit: 14db1faa0fbe813fed616435303753d390f45827

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I046af5eda9560da0fb8daa8ca025484851538a50
Reviewed-on: https://review.whamcloud.com/44681
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15317 llite: Fix iotrace name

The ES6 port of the iotrace patch missed the name of the
debug flag, leaving it as tty. Correct this.

Fixes: 32142909 ("LU-15317 llite: Add D_IOTRACE")

test-parameters: trivial

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I0674fa54d6bf13ac5d436f8fd94c6559a3eef27d
Reviewed-on: https://review.whamcloud.com/46743
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15317 osc: Add RPC to iotrace

Add RPCs to iotrace debugging.

To avoid creating too much debug output, this debug
ignores the possiblity that an RPC contains non-contiguous
extents.  Thus the eventual visualization will act as
though the RPC is a continuous whole.  I judge this to be
superior to the amount of log data and complexity of
capturing each extent separately.  If that level of detail
is needed, a higher debug level can be used.

Lustre-change: https://review.whamcloud.com/45894/
Lustre-commit: 711182c0188b66c87c696fa11165de41c6e3675f (tbd)

Test-parameters: trivial

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I6fe416ba44be5572f130704ba9d3e9b85d09c656
Reviewed-on: https://review.whamcloud.com/46005
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15317 llite: Add COMPLETED iotrace messages

It's very useful to see how long an I/O call took. There
are other ways to do this, but the goal is for iotrace to
provide all necessary information for basic I/O performance
analysis, so we add COMPLETED messages to iotrace.

Lustre-change: https://review.whamcloud.com/46484/
Lustre-commit: 1c90d638a8d7993f5cbd70680d33052e888da6c3 (tbd)

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I17f52ebc87a31d5ba34f63dc8b6a279e83cd10ef
Reviewed-on: https://review.whamcloud.com/46703
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15317 llite: Add FID to async ra iotrace

IOtrace log entries need to include the FID of the file
concerned. Add this to async readahead.

test-parameters: trivial

Lustre-change: https://review.whamcloud.com/45912/
Lustre-commit: 4bd78fa2145efa1e73e42a0c650de8da3fb3887c (tbd)

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I8d788969f29412ce88f1cafa229977f6efa20962
Reviewed-on: https://review.whamcloud.com/46702
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15317 llite: Add strided readahead to iotrace

We need to capture some additional parameters to correctly
understand the behavior of strided readahead. Add these
parameters to the existing iotrace message.

test-parameters: trivial

Lustre-change: https://review.whamcloud.com/45888/
Lustre-commit: d09c4ccd93175470088835a414a7be19638b1e4c (tbd)

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I7caddf9dfaf9ba5f2645d045d5a4a50562cc1b54
Reviewed-on: https://review.whamcloud.com/46701
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15317 llite: Make iotrace logging quieter

Most of the time, we don't read any pages with readahead,
since we're moving through the window and aren't ready to
read more yet. That's important for readahead debug, but
there's no need to log it for iotrace. (This matters
because without this change, this messsage is the large
majority of iotrace messages.)

test-parameters: trivial

Lustre-change: https://review.whamcloud.com/45887/
Lustre-commit: c8e04d958073e15040691c03606c1eb7631b5aff (tbd)

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I58197acd1ef97c903320a2433ec1d5dcb0e46bd0
Reviewed-on: https://review.whamcloud.com/46700
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15452 utils: support lctl getattr for osc

In lctl:jt_obd_getattr(), support FIDs in addition to OIDs and print
whatever valid attributes were returned. Add a supporting
OBD_IOC_GETATTR case to osc_iocontrol().

  # function lctl_osc_device() {
    # Find osc device name for file and index.
    # lctl_osc_device /mnt/lustre/... 42 => lustre-OST002a-osc-ffff89cca1555000
    local path="$1"
    local index="$2"
    local fsname=$(lfs getname --fsname "$path")
    local instance=$(lfs getname --instance "$path")

    printf '%s-OST%04x-osc-%sn' "$fsname" "$index" "$instance"
  }
  # lfs getstripe /mnt/lustre/f0 | grep l_ost_idx
        - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x2:0x0] }
        - 1: { l_ost_idx: 2, l_fid: [0x100020000:0x2:0x0] }
        - 0: { l_ost_idx: 3, l_fid: [0x100030000:0x2:0x0] }
        - 1: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }
  # lctl --device $(lctl_osc_device /mnt/lustre 1) getattr '[0x100010000:0x2:0x0]'
  valid: 0x110000001008fff
  oi.oi.oi_id: 0x100020000
  oi.oi.oi_seq: 0x2
  oi.oi_fid: [0x100020000:0x2:0x0]
  atime: 0
  mtime: 1642178551
  ctime: 1642178551
  size: 0
  blocks: 0
  blksize: 4194304
  mode: 0107666
  uid: 0
  gid: 0
  flags: 2097152
  layout_version: 3
  projid: 0
  data_version: 4294967298

Lustre-change: https://review.whamcloud.com/46131
Lustre-commit: 4143c3bdec2a87319a6c9dee4a589cd41e85dad3

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I57d5778e9ac39030ae9477a0979f20b7f7460fc8
Reviewed-on: https://review.whamcloud.com/46390
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13621 lnet: utility to print cpt number

Added a command to lnetctl to print the cpt of the NID.
lnetctl cpt-of-nid --nid <nid> --ncpt <number of cpts>
ex:
lnetctl cpt-of-nid --nid 192.28.12.35@tcp9 --ncpt 7
This will return what cpt the NID will hash to within the 0-6 range.
If the NI is bound to specific set of CPTs, then the ncpts refers
to the number of CPTs the NI is bound to. The cpt value returned
will be an index into the list of bound CPTs.

For example if an NI is bound to [0,4,5,7], then the ncpt should be
4. And the returned value will be an index in the array:
ex:
lnetctl cpt-of-nid --nid 192.28.12.35@tcp9 --ncpt 4
cpt:
value: 1
therefore, the actual CPT the NID will be bound to is 4.

Lustre-change: https://review.whamcloud.com/39113
Lustre-commit: df6f17ee97ac47c949c1963ff8d57fb2d4becd06

Test-parameters: trivial testlist=sanity-lnet

Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: I3cb562842448bfb663c2d41007be65299a919300
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46397
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-3637 osd-ldiskfs: revalidate nonrotational state

Until the nonrotational state of the device is correctly exported by
the underlying storage, periodically recheck the nonrotational state
in case it is changed by the udev/tune_devices.sh script after the
OST is mounted. It would be possible to check less often (e.g. after
a time limit or some number of operations), but directly checking the
rotational state each time is not more expensive and is only checked
on statfs RPCs (sent about once per 5s from the MDS).

If the nonrotational state is changed and the flash-related cache
parameters have not been explicitly set, then tune them appropriately.

Rename the parameter functions and variables for read_cache_enable,
writethrough_cache_enable, and read_cache_max_filesize to match the
parameter names to make them easier to find in the code.

Lustre-change: https://review.whamcloud.com/45745
Lustre-commit: TBD (from cbc74314532632fbcab88531d43deed7555e125e)

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Iec78a5d5c22c0474eda84a5a793fbb00623ebbe5
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Gaurang Tapase <gtapase@ddn.com>
Tested-by: Gaurang Tapase <gtapase@ddn.com>
Reviewed-on: https://review.whamcloud.com/46514
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shuichi Ihara <sihara@ddn.com>

LU-15045 utils: fix lfs_migrate for files with spaces

Fix the lfs_migrate script to properly quote "$OLDNAME" so that
it works for filenames with spaces and other characters in them.

Lustre-change: https://review.whamcloud.com/45173
Lustre-commit: 0549031b991ec0bbf570a41c7d28b8a7fd4b41c3

Test-Parameters: trivial
Fixes: 8bedfa377fbd ("LU-11510 lfs: migrate a composite layout file correctly")
Fixes: 128137adfc53 ("LU-13090 utils: fix lfs_migrate -p for file with pool")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ic00f41f3a91ad9dfa491ff57768a3da0c6300c1e
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Reviewed-on: https://review.whamcloud.com/46685
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>

EX-4572 tests: fix setfattr stack_trap in init_hot_pools_env()

This patch fixes setfattr stack_trap in init_hot_pools_env() to
resolve the following error:

/usr/lib64/lustre/tests/test-framework.sh: line 2:
trusted.dmv=0xd00c......: command not found

Test-Parameters: trivial testlist=hot-pools
Test-Parameters: trivial mdscount=2 mdtcount=4 testlist=hot-pools

Fixes: 7ff7b9d8cf0 ("EX-3640 test: mkdir on MDT0 in hot-pools.sh")
Change-Id: Ia406a066455bd83c2199f96507a11b36aa80fea2
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46662
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexandre Ioffe <aioffe@ddn.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15492 build: fallthrough macro for pre/post gcc-7

gcc-7.5 on openSUSE 15:
error: this statement may fall through [-Werror=implicit-fallthrough=]

Use the __attribute__((fallthrough)) for gcc-7 and later.
and use a no op statement for earlier gcc where the fallthrough
attribute is not available.

Lustre-change: https://review.whamcloud.com/46357
Lustre-commit: TBD (from 48145f2eaed1537506bfffb1c0a44a8cfdb38254)

Test-Parameters: trivial
Fixes: 5549b1b9e0 ("LU-15220 lustre: use 'fallthrough' pseudo keyword for switch")
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: Ib72f5996149c738805f15e354e1e1606d981ce29
Reviewed-on: https://review.whamcloud.com/46643
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14553 changelog: eliminate mdd_changelog_clear warning

When handling a changelog_clear request, the user may specify a
range of indices which do not exist. Similarly, the user may
specify a changelog user which does not exist. Neither indicates
a problem within Lustre that justifies a a console warning.

Change those cases to CDEBUG.

Lustre-change: https://review.whamcloud.com/43125
Lustre-commit: 6b183927e19715d093c80a35ebc42a1cda5e70e2

Test-Parameters: trivial
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Change-Id: I64bab12ef4978c4bf7139f5f36a39f9b109616fb
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46758
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14413 test: test for overstriping for sanity 27M

The introduction of sanity 27M broke interop with 2.12 LTS since
over striping doesn't exist in that version. Adjust the test to
use over striping if the client supports it, otherwise just use
traditional striping.

Lustre-change: https://review.whamcloud.com/44340
Lustre-commit: 4e1f9c4bd1d96063a1fbb2dfaab41b15836167ab

Test-Parameters: trivial testlist=sanity env=ONLY=27M
Change-Id: I2d788a116cbb749a83d6cec36f97d06533b32421
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46760
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-3342 tests: correct Lustre version checks in sanity

Many patches land to the EXAScaler branches as ports from
other branches. Sometimes the tests that are included with
the ported patches check the version of Lustre to ensure
that the feature it tests exists in this version of Lustre.
These version values are not always changed when patches
are ported from one branch to another.

Modified b_es6_0 version of these tests for interop
testing.

Sanity tests 65n and 311 were modified.
Performing test steps on versions >= 2.12.59 or between
versions 2.12.3 and 2.12.50 makes no sense for b_es5_2
branch, because all Lustre tags on this branch are lower
than 2.12.50.
Checks in test 65n will be changed to >= 2.12.3.6 which
is a tag for the patch that added this test to b_es5_2
branch.
Version check in test 311 is not needed at all, because
it includes all possible versions on b_es5_2 branch.

Lustre-change: https://review.whamcloud.com/46068
Lustre-commit: 2d92fc888e12fb7b58d496952ccd69dd10299c09

Test-Parameters: env=ONLY="65n 311" serverversion=2.10.8 serverdistro=el7.6
Test-Parameters: env=ONLY="65n 311" serverversion=2.14
Test-Parameters: trivial env=ONLY="65n 311"
Test-Parameters: env=ONLY="65n 311" clientversion=2.12.6-ddn42
Test-Parameters: env=ONLY="65n 311" serverversion=2.12.6-ddn42

Fixes: 550e41edbcb ("LU-11739 lod: subdir under ROOT should honor default layout")
Fixes: 37a9077f103 ("LU-11208 tests: add version check to sanity tests")

Signed-off-by: Alena Nikitenko <anikitenko@ddn.com>
Change-Id: Ic6383fe5585e18646a365f49f501da52848c765a
Reviewed-on: https://review.whamcloud.com/46421
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-3342 tests: fix Lustre version in test skip checks in sanity

Many patches land to the EXAScaler branches as ports from
other branches. Sometimes the tests that are included with
the ported patches check the version of Lustre to ensure
that the feature it tests exists in this version of Lustre.
These version values are not always changed when patches
are ported from one branch to another.

Change Lustre test suite version checks to be relative to
this branch. Sanity tests 160j,432,810,812 were modified.

Lustre-change: https://review.whamcloud.com/45967
Lustre-commit: 60186cbf11d7cfa24e94721b8bf77e0dc5af0c50

Test-Parameters: env=ONLY="160j 432 810 812" serverversion=2.10.8 \
serverdistro=el7.6
Test-Parameters: env=ONLY="160j 432 810 812" clientversion=2.12.6-ddn42 \
mdscount=2 mdtcount=4 osscount=1 ostcount=8 clientcount=2
Test-Parameters: env=ONLY="160j 432 810 812" serverversion=2.12.6-ddn42 \
mdscount=2 mdtcount=4 osscount=1 ostcount=8 clientcount=2
Test-Parameters: env=ONLY="160j 432 810 812",SHARED_KEY=true mdscount=2 \
mdtcount=4 osscount=1 ostcount=8 clientcount=2 serverversion=2.12.6-ddn42
Test-Parameters: env=ONLY="160j 432 810 812",SHARED_KEY=true mdscount=2 \
mdtcount=4 osscount=1 ostcount=8 clientcount=2 clientversion=2.12.6-ddn42
Test-Parameters: trivial env=ONLY="160j 432 810 812"

Fixes: 9d88030b8a5 ("LU-11626 mdc: hold obd while processing changelog")
Fixes: 6a70c5ec116 ("LU-14804 nodemap: do not return error for improper ACL")
Fixes: bc38e0f6eaa ("LU-12784 llite: limit max xattr size by kernel value")
Fixes: 78ddbc59530 ("LU-11951 ptlrpc: reset generation for old requests")

Signed-off-by: Alena Nikitenko <anikitenko@ddn.com>
Change-Id: I91b45cc9a9d78ab96877edfef59cf6e8fe259452
Reviewed-on: https://review.whamcloud.com/46432
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14935 tests: Use FAIL_CHECK_QUIET for fake i/o

Logging to the console is relatively expensive and doing it
for fake i/o is very expensive in terms of CPU time.

If we use FAIL_CHECK_QUIET, a debug message is logged only once
to the console, and the rest at D_INFO level (probably not at all).

This should hugely reduce the CPU cost of the debugging.

Lustre-change: https://review.whamcloud.com/44651
Lustre-commit: 890466a32d3e8683a96d5a18e76c0cc704810f5f

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I46a5042efd116a4f5c80eaf0d5dae7fe132f6a79
Reviewed-on: https://review.whamcloud.com/46372
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15420 uapi: avoid gcc-11 -Werror=stringop-overread

GCC 11 warns about string and memory operations on fixed address:

In function ‘strlen’,
    inlined from ‘changelog_rec_sname’ at include/uapi/linux/lustre/lustre_user.h:1981:19,
    inlined from ‘mdd_changelog_rec_ext_rename’ at lustre/mdd/mdd_dir.c:932:2,
    inlined from ‘mdd_changelog_ns_store’ at lustre/mdd/mdd_dir.c:1061:3:
include/linux/fortify-string.h:25:33: error: ‘__builtin_strlen’
reading 1 or more bytes from a region of size 0 [-Werror=stringop-overread]
   25 | #define __underlying_strlen     __builtin_strlen

The reason is that we are looking for an address right after the end
of the chanelog record header which gcc thinks is an overrun. Rework
the code to allow us to index the memory right after the changelog
record header.

Also fix a long hidden bug in the lustre snmp code.

Lustre-change: https://review.whamcloud.com/46319
Lustre-commit: TBD (from 0ae7f1196ba14df9a2354f0803eb75eea92cb1ee)

Change-Id: I13479b9074a392330d39f01656b26f9e9a91a8ec
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/46498
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15587 kernel: kernel update RHEL7.9 [3.10.0-1160.59.1.el7]

Update RHEL7.9 kernel to 3.10.0-1160.59.1.el7.

Test-Parameters: trivial clientdistro=el7.9 serverdistro=el7.9

Change-Id: I4f445be6c7ed341e6e912c7df97775af7e46c1e1
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46644
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15196 kernel: kernel update RHEL8.4 [4.18.0-305.25.1.el8_4]

Update RHEL8.4 kernel to 4.18.0-305.25.1.el8_4 for server support.

Test-Parameters: trivial fstype=ldiskfs \
clientdistro=el8.4 serverdistro=el8.4 testlist=sanity

Test-Parameters: trivial fstype=zfs \
clientdistro=el8.4 serverdistro=el8.4 testlist=sanity

Change-Id: If05ec88e37faa8d61881b3987a3364544e39f8a9
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46343
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15286 build: only use baseonly option on el7

el7 baseonly option allow to build perf package while
in el8 does not.

Lustre-change: https://review.whamcloud.com/45677
Lustre-commit: 6eb5d3ae4aaab1662afc8ae50f56b0a0eaa6556a

Change-Id: Ie973c5cc816b4b98ef71ab7080bd11286bcd644a
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46056
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-11597 test: Fix sanityn 16a failed on arm

As now O_DIRECT expect IO aligned with PAGE SIZE,
x86_64 expect 4K size, but some other platform, it
could be 64K, use PAGE_SIZE here to make the test happy.

And macro O_DIRECT is defined if macro _GNU_SOURCE is defined
according to open man doc[1] and _GNU_SOURCE is defined at the
head of file fsx.c already. So set the value of OP_DIRECT to
O_DIRECT instead of hardcoding its value as O_DIRECT could have
different values for other platforms like Arm64[2].

[1]
https://man7.org/linux/man-pages/man2/open.2.html
"The O_DIRECT, O_NOATIME, O_PATH, and O_TMPFILE flags are Linux-
specific. One must define _GNU_SOURCE to obtain their definitions."
[2]
https://code.woboq.org/userspace/glibc/sysdeps/unix/sysv/linux/aarch64/bits/fcntl.h.html#_M/__O_DIRECT

Lustre-change: https://review.whamcloud.com/37589
Lustre-commit: eb704aecfaad2a6256d1e2e48cdfadbabb07e5cb

Fixes: 853d180121a6 ("LU-3606 fsx: Add fallocate operation to fsx")

Test-Parameters: testlist=sanityn envdefinitions=ONLY=16a
Change-Id: If72d434adaf91a960dfc50c557d8b50793fda575
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Signed-off-by: Xinliang Liu <xinliang.liu@linaro.org>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46430
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15487 mdd: print FID in mdd_dir_page_build() error

Print the MDT name and FID in mdd_dir_page_build() when an error
is hit. Because this changes the callback function signature,
also update dt_index_page_build() to print a more useful message.

Add OBD_FAIL_MDS_DIR_PAGE_WALK to allow triggering this codepath
to see if this is the source of problems in error handling.

Lustre-change: https://review.whamcloud.com/46368
Lustre-commit: TBD

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ic475f4a2c775871ff5af59a47e0966ba3eed7013
Reviewed-on: https://review.whamcloud.com/46335
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46370
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>

LU-11667 tests: Fix sanity test 317 for 64K PAGE_SIZE OST

When create a file, blocks are allocated with PAGE_SIZE aligned,
see function osd_ldiskfs_map_inode_pages(). E.g. for 64K PAGE_SIZE
Arm64 OST server, if create a file with size less than 64K, it
actually allocates 128 blocks each block 512 Bytes.

It needs to adjust the test for 64K PAGE_SIZE OST server.

Lustre-change: https://review.whamcloud.com/45395
Lustre-commit: 63d4d9ff2f5c8cc992ca6b2f698bb43a3257bfb3

Test-Parameters: trivial
Test-Parameters: clientdistro=el8.5 clientarch=aarch64 fstype=ldiskfs testlist=sanity \
env=PTLDEBUG=-1,ONLY=317
Test-Parameters: fstype=ldiskfs testlist=sanity \
env=PTLDEBUG=-1,ONLY=317

Change-Id: Iada701f4f424093e847fc70aa843873b75fe6b06
Signed-off-by: Xinliang Liu <xinliang.liu@linaro.org>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46482
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15513 lod: skip uninit component in lod_fill_mirrors

Do not iterate over the "objects" in lod_fill_mirrors() to check
for non-rotational OSTs if the component is uninitialized. In
cases where an OST is not present (e.g. sparse OST indexes used)
the lod_tgt_desc[] array has holes and OST_TGT() returns NULL.

Skip the loop entirely if the component is not initialized, but
also add some sanity checks to verify that the OST index values
are sane in case there are other problems in the future (e.g.
corrupt/invalid layout on disk).

Lustre-change: https://review.whamcloud.com/46435
Lustre-commit: 591a990c617f9b953d2e838427d45fa1de061a83

Fixes: 8507472dd37e ("LU-14996 lov: prefer mirrors on non-rotational OSTs")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I8ec23367059a4ec9e483adb768095b24f03ebbe5
Reviewed-on: https://review.whamcloud.com/46437
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>

LU-15021 quota: protect lqe_glbl_data in lqe

The lqe_glbl_data in "struct lquota_entry" is allocated in
qmt_lvbo_init and freed in qmt_lvbo_free, it could be freed
during qmt_seed_glbe called by qmt_set_id_notify, and cause
panic because of using freed memory.

Lustre-change: https://review.whamcloud.com/45098
Lustre-commit: TBD (from 124c60bd3ff7705a63a572f4619ecbc5cff13f4c)

Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Change-Id: I274f07ee8609c83852572be51625cc929a9130ec
Reviewed-on: https://review.whamcloud.com/46351
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15609 flr: Don't assume RDONLY implies SOM

In lov_io_slice_mirror_init, the client code assumes that
the LCM_FL_RDONLY flag in the layout implies SOM and skips
glimpse if it sees one. The RDONLY flag means the mirrors
are in sync, which has historically implied SOM is valid.

To start with, using LCM_FL_RDONLY to imply SOM is sort of
a layering violation. SOM is only communicated from the
MDS when it is valid, and the client already skips glimpse
in that case, so this duplicates functionality from the
higher layers.

More seriously, patch:
"LU-14526 flr: mirror split downgrade SOM"
(https://review.whamcloud.com/43168/)
Made it possible to have LCM_FL_RDONLY but not strict SOM,
so this assumption is no longer correct.

The fix is to not look at LCM_FL_RDONLY when deciding
whether to glimpse a file for size.

Lustre-change: https://review.whamcloud.com/46666/
Lustre-commit: 91db60601be1e3ac1f06dc79185f67884598acd2 (tbd)

Fixes: d87b6f4d1c ("LU-14526 flr: mirror split downgrade SOM")
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I5ed0a2124005bc58c1ed11681c9bd642cffcd1b5
Reviewed-on: https://review.whamcloud.com/46668
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn37

New tag 2.14.0-ddn37

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ia37f6b3a8348ab8e92ba73cfa2c0e56aa95de38d

EX-4890 configure script changes system header and config files

Remove the SUBDIRS target from configure tests.
Since Linux kernel version 2.6.x we can use target M.

Change-Id: I8e59bdaf2d0e4e08a659e08f63a14472fba72eb2
Signed-off-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-on: https://review.whamcloud.com/46603
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: new tag 2.14.0-ddn36

New tag 2.14.0-ddn36

Change-Id: I5bf382b0a6e1f51864804e5f1c859be423b6f1c2
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>

EX-4841 ofd: Return EOPNOTSUPP instead of EPROTO

Modify server to return -EOPNOTSUPP instead of
-EPROTO for unsupported fallocate modes

Lustre-commit: TBD (5ad69eae24a4f1a6bc02a9fa9d66b0ed1a57fe84)
Lustre-change: https://review.whamcloud.com/46516

Fixes: 7462e8cad730 ("LU-14160 fallocate: Add punch mode to fallocate")

Signed-off-by: arshad.hussain@aeoncomputing.com
Change-Id: Id203c0b9abbdd674af33f1f78e81ae7fe105e90f
Reviewed-on: https://review.whamcloud.com/46527
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>

EX-4822 build: dkms ptlrpc_gss in wrong location

Test-Parameters: trivial

Change-Id: Ie69f8313977a40613864eca763aa987d897d4e00
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-on: https://review.whamcloud.com/46542
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Gaurang Tapase <gtapase@ddn.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

EX-4822 build: fix ubuntu mofed dkms

MOFED is always installed via dkms.
The path /usr/src/ofa_kernel hasn't changed
after MOFED 5.5 so we will use the
new way to detect in non-ubuntu only

Test-Parameters: trivial

Change-Id: Ie76b90f61449aaa880c44c8184691711f4ccfb19
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46536
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

EX-4812 lipe: Always mkdir config directories

Always do "mkdir -p", not just when there isn't a client mount
existing. Client mounts can now be created via config-pacemaker.

Signed-off-by: Nathaniel Clark <nclark@whamcloud.com>
Change-Id: I813edd30157279a1164492e747fd154571a72b87
Reviewed-on: https://review.whamcloud.com/46518
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

RM-620 build: new tag 2.14.0-ddn35

New tag 2.14.0-ddn35

Change-Id: I7c94cc36ff7efa229077753625a50597c50ce27a
Signed-off-by: Oleg Drokin <green@whamcloud.com>

Revert "EX-3637 osd-ldiskfs: revalidate nonrotational state"

This reverts commit 5a1da772cf1721967d9dae55fae3d19aebba70f8.

Change-Id: I576d4cbd9794e4d4c4d941929e9c9e35ec01212a
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46473
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

EX-3342 tests: fix version skip checks in sanity-quota

Many patches land to the EXAScaler branches as ports from
other branches. Sometimes the tests that are included with
the ported patches check the version of Lustre to ensure
that the feature it tests exists in this version of Lustre.
These version values are not always changed when patches
are ported from one branch to another.

Change Lustre test suite version checks to be relative to
this branch. Sanity-quota test 78 was modified.

Fixes: 9abc60b6782 ("LU-15167 quota: fallocate send UID/GID for quota")

Test-Parameters: env=ONLY="78" serverversion=2.12.6-ddn42 \
testlist=sanity-quota
Test-Parameters: trivial env=ONLY="78" testlist=sanity-quota

Signed-off-by: Alena Nikitenko <anikitenko@ddn.com>
Change-Id: If9022a5006c6d595e12c037aecd5c6eff0096725
Reviewed-on: https://review.whamcloud.com/46443
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12585 mdt: Add read/write latency to MDT stats

The MDT does not currently record latency stats for reads
and writes.

Add this, and change the naming to be the same as for the
OFD.

Note on this:
Existing naming on the MDT uses "read/write" instead of
"{read,write}_bytes", which is inconsistent with OFD and
also inconsistent within the MDT, since other ops without
the "_bytes" suffix are latency.

It's not deal to change the names of existing stats, but I
decided this was less problematic than leaving them
inconsistent.

Lustre-change: https://review.whamcloud.com/46229/
Lustre-commit: ab1f7876b4b9e81738b1357395289aab76124255 (tbd)

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I7b7a5742678cbe0269086f37877833e877a5ca5f
Reviewed-on: https://review.whamcloud.com/46230
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12585 obdfilter: Use actual I/O bytes in stats

Currently the obdfilter stats note the number of bytes
requested by the client rather than the number of bytes
actually read or written.  This is particularly confusing
for reads because clients can request more data than
exists and some applications do this normally.

This results in statistics that can be off by almost any
amount from the actual number of bytes read.  This patch
moves the stats to be collected just before commit, which
allows the true number of bytes to be recorded but does not
include the commit time in the time stats.  (Since commit
time is not part of the operation latency as experienced by
the client.)

Lustre-change: https://review.whamcloud.com/46075/
Lustre-commit: 34cbc31a9e73498d7f9324e942c85fbc44791c83 (tbd)

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I81fe9a6afdad5b48e8421f4aa72b8ef10a0eee93
Reviewed-on: https://review.whamcloud.com/46076
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14074 scripts: automatic LNet unconfigure

After using the lnetctl utility a reference count is taken on the LNet
modules. lnetctl lnet unconfigure is called in order for
lustre_rmmod to remove the LNet module.

Lustre-change: https://review.whamcloud.com/41698
Lustre-commit: ebdf71a2df044c441ff633dbfc2f463520892bfb

Signed-off-by: Cyril Bordage <cbordage@whamcloud.com>
Change-Id: I7251a0c62c45da7b3cb0fddea97394b32cb6902a
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46426
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn34

New tag 2.14.0-ddn34

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ib79e7cb1fc5ae91aff6038db52979b1f17c78a59

EX-3342 tests: fix Lustre ver in skip checks in recovery-small

Many patches land to the EXAScaler branches as ports from
other branches. Sometimes the tests that are included with
the ported patches check the version of Lustre to ensure
that the feature it tests exists in this version of Lustre.
These version values are not always changed when patches
are ported from one branch to another.

Change Lustre test suite version checks to be relative to
this branch. Recovery-small tests 110k,136,138,139,143 were
modified.

Fixes: 80eb31f5367 ("LU-11761 fld: let's caller to retry FLD_QUERY")
Fixes: 5bf8c0eaf91 ("LU-12098 mdd: explicitly clear changelogs on deregister")
Fixes: 18a53eddc08 ("LU-12846 mdd: return error while delete failed")
Fixes: c2d5af902d8 ("LU-13061 osp: check catlog FID after reading in")
Fixes: 18a53eddc08 ("LU-12846 mdd: return error while delete failed")

Test-Parameters: env=ONLY="110k 136 138 139 143" serverversion=2.10.8 \
serverdistro=el7.6 testlist=recovery-small
Test-Parameters: env=ONLY="110k 136 138 139 143" \
clientversion=2.12.6-ddn42 testlist=recovery-small
Test-Parameters: env=ONLY="110k 136 138 139 143" \
serverversion=2.12.6-ddn42 testlist=recovery-small
Test-Parameters: trivial env=ONLY="110k 136 138 139 143" \
testlist=recovery-small

Lustre-change: https://review.whamcloud.com/45947
Lustre-commit: 4a981ce7975a499680bc53c374dd002bb89fb515

Signed-off-by: Alena Nikitenko <anikitenko@ddn.com>
Change-Id: Id86d1d0fbbb5eca5aec9b6fcbf5baae8941573e1
Reviewed-on: https://review.whamcloud.com/46418
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-3342 tests: fix Lustre ver in skip checks in replay-single

Many patches land to the EXAScaler branches as ports from
other branches. Sometimes the tests that are included with
the ported patches check the version of Lustre to ensure
that the feature it tests exists in this version of Lustre.
These version values are not always changed when patches
are ported from one branch to another.

Change Lustre test suite version checks to be relative to
this branch. Replay-single tests 2d,80c,80d were modified.

Lustre-change: https://review.whamcloud.com/45948
Lustre-commit: a5f7c423b685517a8ff805f3602f56aec3fdedda

Test-Parameters: env=ONLY="2d 80c 80d" serverversion=2.10.8 \
serverdistro=el7.6 testlist=replay-single
Test-Parameters: env=ONLY="2d 80c 80d" clientversion=2.12.6-ddn42 \
testlist=replay-single
Test-Parameters: env=ONLY="2d 80c 80d" serverversion=2.12.6-ddn42 \
testlist=replay-single
Test-Parameters: trivial env=ONLY="2d 80c 80d" testlist=replay-single

Fixes: c44b6a5f36c ("LU-10143 tests: Add version check for interop")

Signed-off-by: Alena Nikitenko <anikitenko@ddn.com>
Change-Id: I6491e8ef2828ed73d049dd97a240298c08670daf
Reviewed-on: https://review.whamcloud.com/46419
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-3342 tests: correct Lustre version in skip checks in sanity-flr

Many patches land to the EXAScaler branches as ports from
other branches. Sometimes the tests that are included with
the ported patches check the version of Lustre to ensure
that the feature it tests exists in this version of Lustre.
These version values are not always changed when patches
are ported from one branch to another.

Change Lustre test suite version checks to be relative to
this branch. Sanity-flr tests 44c and 207 were modified.

Lustre-change: https://review.whamcloud.com/45607
Lustre-commit: 3b31d79a53a7daebcd214b83b4d0a20d9a570eba

Test-Parameters: env=ONLY="44c 207" serverversion=2.12.6-ddn3 serverdistro=el7.7 testlist=sanity-flr
Test-Parameters: env=ONLY="44c 207" clientversion=2.12.6-ddn42 testlist=sanity-flr
Test-Parameters: env=ONLY="44c 207" serverversion=2.12.6-ddn42 testlist=sanity-flr
Test-Parameters: trivial env=ONLY="44c 207" testlist=sanity-flr

Fixes: e317b4ef681 ("LU-13730 lod: don't confuse stale with primary flag")
Fixes: 6c3a4fad094 ("LU-14526 flr: mirror split downgrade SOM")

Signed-off-by: Alena Nikitenko <anikitenko@ddn.com>
Change-Id: Ic479c21f0d6de644b466291ca82ceb4bbc498d93
Reviewed-on: https://review.whamcloud.com/46424
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-3342 tests: fix Lustre ver in skip checks in sanity-flr

Many patches land to the EXAScaler branches as ports from
other branches. Sometimes the tests that are included with
the ported patches check the version of Lustre to ensure
that the feature it tests exists in this version of Lustre.
These version values are not always changed when patches
are ported from one branch to another.

Sanity-flr tests 0b, 0h, 37 were modified.

Performing test steps on versions >= 2.12.56 or between
versions 2.12.2 and 2.12.50 makes no sense for b_es5_2
branch, because all Lustre tags on this branch are lower
than 2.12.50.
Similar checks happen in all three of the modified tests.
Checks in tests 0b and 0h will be changed to >= 2.12.3.1
which is a tag for the patches that added these tests to
this branch. Test 37 check doesn't need elif statement,
because original test excludes versions between 2.12.50
and 2.12.52, which doesn't make sense for b_es5_2 branch.
It will be replaced with else statement.

Fixes: 3a8d96a1ebf ("LU-11022 lfs: remove mirror by pool name")
Fixes: 79346930b34 ("LU-11485 lod: disallow setting the last non-stale mirror as stale")
Fixes: 24f181513fb ("LU-10258 lfs: lfs mirror copy command")

Lustre-change: https://review.whamcloud.com/45949
Lustre-commit: 896b18bf3dd169641147ad382314f86d482ad670

Test-Parameters: env=ONLY="0b 0h 37" serverversion=2.10.8 \
serverdistro=el7.6 testlist=sanity-flr
Test-Parameters: trivial env=ONLY="0b 0h 37" testlist=sanity-flr

Signed-off-by: Alena Nikitenko <anikitenko@ddn.com>
Change-Id: Ie417f72b7ccafff600a762f78f5c2c1605e00221
Reviewed-on: https://review.whamcloud.com/45949
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46414
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-3342 tests: fix Lustre ver in skip checks in sanity

Many patches land to the EXAScaler branches as ports from
other branches. Sometimes the tests that are included with
the ported patches check the version of Lustre to ensure
that the feature it tests exists in this version of Lustre.
These version values are not always changed when patches
are ported from one branch to another.

Change Lustre test suite version checks to be relative to
this branch. Sanity tests 24z,27M,36g were modified.

Fixes: 9cbefa99f23 ("LU-12037 mdt: add option for cross-MDT rename")
Fixes: 3c0733aea94 ("LU-9341 lod: Add special O_APPEND striping")
Fixes: 5456b1bdbf9 ("LU-10496 tgt: move FMD handling from OFD to target")

Test-Parameters: env=ONLY="24z 27M 36g" serverversion=2.10.8 \
serverdistro=el7.6
Test-Parameters: env=ONLY="24z 27M 36g" clientversion=2.12.6-ddn42
Test-Parameters: env=ONLY="24z 27M 36g" serverversion=2.12.6-ddn42
Test-Parameters: trivial env=ONLY="24z 27M 36g"

Lustre-change: https://review.whamcloud.com/45951
Lustre-commit: 1c8ed467169e27302ec53c43751496c31de2f51a

Signed-off-by: Alena Nikitenko <anikitenko@ddn.com>
Change-Id: I151d9a0a90a27f1b47a567b40c6f1f4ea641420a
Reviewed-on: https://review.whamcloud.com/46407
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15417 build: find the new path for MOFED 5.5

The path of the mofed header files has change to
/usr/src/ofa_kernel/x86_64/<kernel>,
so we cannot assume it's /usr/src/ofa_kernel/default.

Besides updating lbuild, we also need to update
lustre-lnet.m4 and lustre.spec.in.

Lustre-change: https://review.whamcloud.com/46383
Lustre-commit: TBD (from a770658032bfe2717a689b3d6346e6b98e5cdeff)

Test-Parameters: trivial

Change-Id: Iab42ce9e458f78b0dc0233ac6fd23a1760be5324
Fixes: 0dbbe482148 ("LU-15417 build: build MOFED 5.5")
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46412
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Colin Faber <cfaber@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn33

New tag 2.14.0-ddn33

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Id4409efdeeed893c50e9685ce6c1cb26478855fb

LU-13335 ldiskfs: add projid to debug logs

There is virtually no tracking of projid changes in osd-ldiskfs,
which makes it very difficult to debug operations therein.

Add some minimal debugging on the client and servers to print
the projid when it is changed, along with the affected FID.

Lustre-change: https://review.whamcloud.com/46369
Lustre-commit: TBD

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ibcf3f09ee243ebe052c8f9119383897072ce7057
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46373
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-13195 osp: track destroyed OSP object

retain destroyed OSP objects in memory to prevent races when
in-flight destroyed is passed by read or attr_get leading to
incorrect local states.
also block operations to such an object with -ENOENT.

Lustre-change: https://review.whamcloud.com/38385
Lustre-commit: f5a8f1bcf5563f96cf6ba0e5de5a99a1ea524cc6

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Ied59f1a95458e8890249b92d4efc38e258a7e3cf
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-on: https://review.whamcloud.com/46338
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-13195 osp: osp_send_update_req() should check generation

and don't send requests depending on just failed one

Lustre-change: https://review.whamcloud.com/45042
Lustre-commit: dff1e0d21c8c6bb20d63669252190795198bc49f

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I27a2b21130e33287168204ad829c0a53002b517e
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46337
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-3342 tests: fix Lustre version in skip checks in conf-sanity

Many patches land to the EXAScaler branches as ports from
other branches. Sometimes the tests that are included with
the ported patches check the version of Lustre to ensure
that the feature it tests exists in this version of Lustre.
These version values are not always changed when patches
are ported from one branch to another.

Change Lustre test suite version checks to be relative to
this branch. Conf-sanity test_126 was modified.

Fixes: efd1b764bac ("LU-11814 obd: Crashed while mount in parallel")

Test-Parameters: env=ONLY="126" serverversion=2.10.8 serverdistro=el7.6 \
testlist=conf-sanity
Test-Parameters: env=ONLY="126" clientversion=2.12.6-ddn42 testlist=conf-sanity
Test-Parameters: env=ONLY="126" serverversion=2.12.6-ddn42 testlist=conf-sanity
Test-Parameters: trivial env=ONLY="126" testlist=conf-sanity

Lustre-change: https://review.whamcloud.com/45942/
Lustre-commit: 5f2b47b4d9fb4625290c9fa498c776b3df15ceb9

Signed-off-by: Alena Nikitenko <anikitenko@ddn.com>
Change-Id: I953be8ec00e28b09eb492ffa24c03c50430f59df
Reviewed-on: https://review.whamcloud.com/46365
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-3253 iokit: correct device name in ec_using_srv_nid()

Patch changes in ec_using_srv_nid():
- use 'oscname' instead of 'ocsname', that also fixes typo
introduced by EX-1873
- use explicit --device parameter for echo_client setup, that
was missed in EX-1873 and completes its work

Test-Parameters: trivial testlist=obdfilter-survey env=ONLY=3a
Fixes: aba6201da5 (EX-1873 iokit: fix the obsolete usage of cfg_device)
Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I661a58299adfba09983ec73f9801726789d79f4a
Reviewed-on: https://review.whamcloud.com/46366
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>