LU-14439 build: require a newer version of e2fsprogs

Require a build version of e2fsprogs-1.44.3 for osd-ldiskfs builds to
get EXT2_FLAG_IGNORE_SB_ERRORS and ext2fs_has_feature_{mmp,quota}()
functions.

Require a minimum install version of ldiskfsprogs-1.45.6.wc1.

Test-Parameters: trivial
Fixes: 7dc8aa7e7848 ("LU-13241 utils: use libext2fs for ldiskfs operations")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I240823266c7342338c72126011485120543ebbe5
Reviewed-on: https://review.whamcloud.com/41682
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13609 mgs: fix config_log buffer handling

Fix buffer handling in mgs_list_logs() to list all MGS config_logs
using multiple ioctl calls when we have a large number of targets.

Fixes: 1d97a8b4cd3d ("LU-13609 llog: list all the log files correctly on MGS/MDT")
Signed-off-by: Stephane Thiell <sthiell@stanford.edu>
Change-Id: I1bf32e918e242f4da83c3d1624b7285a18a88d01
Reviewed-on: https://review.whamcloud.com/41478
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10391 socklnd: use sockaddr instead of __u32 addresses.

LNet/socklnd often uses __u32 to hold an ipv4 address.
As we want to extend socklnd to work with IPv6 addresses too,
this needs to change.

This patch changes many __u32s to variants of 'struct sockaddr'.

Library code from sunrpc is used for copying and comparing addresses
   rpc_copy_addr() and rpc_cmp_addr()
and for extracting or setting the port:
   rpc_get_port() and rpc_set_port().

The "%pIS" printf format is used for printing a sockaddr (works for
both IPv4 and IPv6), and "%pISp" for printing the address with the
port.

The __u32 is in host-byte-order, while addresses in sockaddr are
always network-order, so htonl and ntohl are used as needed.

When storing an address (e.g. in a structure), 'struct
sockaddr_storage' is used.  When passing an address to a function,
'struct sockaddr' is used.  When an address is known to be IPv4 (i.e.
when converting to or from __u32), 'struct sockaddr_in' is used.

The following functions are changed to take a 'struct sockaddr*'
argument:

 lnet_connect()
 lnet_connect_console_error()
 lnet_sock_getaddr()
 ksocknal_ip2iface()
 ksocknal_ip2index()
 ksocknal_create_route()
 ksocknal_connecting()
 ksocknal_close_peer_conns_locked()
 ksocknal_peer_del_interface_locked()

The following structures have had fields changed to 'struct
sockaddr_storage'

 struct ksock_interface:
      ksni_ipaddr -> ksni_addr
 struct ksock_conn
      ksnc_myipaddr -> ksnc_myaddr
      ksnc_ipaddr and ksnc_port -> ksnc_peeraddr
 struct ksock_route
      ksnr_myipaddr -> ksnr_myaddr
      ksnr_ipaddr and ksnr_port -> ksnr_addr

Assorted strings have been joined onto a single line, and various
indented have been changed from space to tabs.

There should be no behaviour changes, though the structures mentioned
above will now be a little larger.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I42d12260185638407b5b611391fc69bfd9f91754
Reviewed-on: https://review.whamcloud.com/37704
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Aurelien Degremont <degremoa@amazon.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13929 lnet: modify assertion in lnet_post_send_locked

Check that the pointer to the local interface is not NULL
before asserting. While checking if local ni is the destination,
the assertion may attempt to dereference pointer to local
interface after it has already been cleaned up on shutdown.

Test-Parameters: trivial
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: I0f4be04a728a7243823bec70f9efbe52bcb104b3
Reviewed-on: https://review.whamcloud.com/40749
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14362 tests: sanity-flr to prepare stuff before checks

otherwise /mnt/lustre may be missing and sanity-flr fails

Test-Parameters: trivial testlist=sanity-flr
Change-Id: Ic5502ee07bab557162fbf16718ca5fb5beed45e9
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/41307
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14423 osd: recognize holes in osd_is_mapped()

ldiskfs_fiemap() can return {0,0,0} for last non-allocated
region.  osd_is_mapped() should be able to recognize and
cache this state.

Fixes: 144b5a65c1 ("LU-7132 osd-ldiskfs: speedup rewrites")
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I03883038c2c0ec84754377a442c4947c7e3021a9
Reviewed-on: https://review.whamcloud.com/41481
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andrew Perepechko <andrew.perepechko@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14398 llapi: add llapi_fid2path_at()

Add llapi_fid2path_at() which works like llapi_fid2path() takes an
open FD on the moint point instead of a 'fsname or dirirectory path'
and a const struct lu_fid * instead of a const char *.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I76234bc28de231587b65c5d866954441e0893aac
Reviewed-on: https://review.whamcloud.com/41406
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14398 llapi: simplify llapi_fid2path()

Simplify llapi_fid2path(). Remove the fid_is_sane() check. Remove the
call to root_ioctl() and use get_root_path() directly.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Ib70b8c9e239c77da8b46408de8341fc8aaf4d1c3
Reviewed-on: https://review.whamcloud.com/41405
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>

LU-14390 gnilnd: Use DIV_ROUND_UP to calculate niov

Use DIV_ROUND_UP to calculate niov, also remove 'unlikely'
as this is the common case.

Fixes: 7a74d382 ("LU-13004 modules: replace lnet_kiov_t with struct bio_vec")
Test-Parameters: trivial
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: Iaf62326743ebe5e92bb3aa4b5780b47a5cfdfb18
Reviewed-on: https://review.whamcloud.com/41371
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14301 lustre: add ENOTSUPP to spelling.txt

Add a spelling check for ENOTSUPP to suggest use of EOPNOTSUPP
instead. Note:

ENOTSUPP (524) and defined only in the kernel errno.h and is a NFSv3
specific errno. If ENOTSUPP is returned to userspace then strerror()
will print "Unknown error 524".

EOPNOTSUPP (95) is defined in kernel and userspace errno.h.

ENOTSUP is defined in userspace errno.h as an alias for EOPNOTSUPP.

Test-Parameters: trivial
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I13b0389c9ec0853f43d8ab4a8f6538eb24c8a2ad
Reviewed-on: https://review.whamcloud.com/41280
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>

LU-12766 test: convert time to seconds properly

According to test logs, grace time could be 3m56s,
and it will be converted to 21305s by wrongly which
caused timeout of test sytem.

Actually, grace time could be something like 1w2d3h4m5s
Fix calculations for this.

Test-Parameters: trivial testlist=sanity-quota
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I1c02e83f5ec15de2a4a6b312dd6e36b55dd4a7bc
Reviewed-on: https://review.whamcloud.com/41259
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13903 build: make lustre-devel buildable for Linux client

Recently a new lustre-devel rpm was created which always has a
dependency on the kernel detected by the kmod macros. This doesn't
work in the case of running a custom Linux kernel. Addreess this
by not making use of the kmod macros when only building the Lustre
utilies i.e configure --disable-modules.

Test-Parameters: trivial
Fixes: 16af4e5ed634 ("LU-9215 build: Re-add the lustre-devel package")
Signed-off-by: James Simmons <jsimmons@infradead.org>
Change-Id: I324fbaa0f5b03e2095b493f3d8e00b74ca64298a
Reviewed-on: https://review.whamcloud.com/41188
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14313 utils: mount error when no server support

When the mount utility was built without server support,
and this causes the mount to fail, an error message should
say so.

The error values returned by main() should be positive as
they are exposed to the user.

Upon mount() failure, errno must be checked to determine
the reason for the failure.  The return value is -1.

Test-Parameters: trivial
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Change-Id: I49906340f6ddbe16b7503663cedabe0085daf113
Reviewed-on: https://review.whamcloud.com/41182
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14268 lod: fix layout generation inc for mirror split

Mirror split does not increase the layout generation properly.

Mirror split does not change FLR state of the file, even when it
contains 1 mirror afterwards, and FLR state should be LCM_FL_NONE
instead.

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I9c9621d67d901f2e9ca6ed3e0684cd308c396076
Reviewed-on: https://review.whamcloud.com/41068
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14098 obdclass: try to skip corrupted llog records

if llog's header or record is found corrupted, then
ignore the remaining records and try with the next one.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: If47ec1fc1e2eaf64be7ba08d3aa9c2b93903c0cf
Reviewed-on: https://review.whamcloud.com/40754
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9820 osd-ldiskfs: OI scrub speed limit fix

The OI scrub speed limit is set at start time, and shouldn't be
changed in the middle, otherwise lfsck is run with speed control,
while OI scrub may finish before it, thus sanity-scrub 9 failed.

Allow wider margin in speed control since the speed control is
heuristic.

Test-Parameters: testlist=sanity-scrub,sanity-scrub,sanity-scrub,sanity-scrub,sanity-scrub,sanity-scrub,sanity-scrub,sanity-scrub,sanity-scrub,sanity-scrub,sanity-scrub,sanity-scrub,sanity-scrub
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I995679db085a193c71fc7914f961b538b5930c69
Reviewed-on: https://review.whamcloud.com/40591
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14099 build: Fix for unconfigured arch_stackwalk

On aarch64 CONFIG_ARCH_STACKWALK is not defined and
print_stack_trace is not available.

Replace print_stack_trace with an open-coded variant
using %pB introduced in Linux v2.6.38-6557-g0f77a8d37825

This also fixes the symbols lookup of stack_trace_save_tsk
using kallsyms at module init time over the use of
symbol_get.

HPE-bug-id: LUS-9518
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I04c3a0a84bb1a05d813a90502d1ed0f5bb2e33ab
Reviewed-on: https://review.whamcloud.com/40503
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14044 llog: check fid after convert

We should convert from llog_id and then check fid. Also
change fid-lookup to error check instead LASSERT.

Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: I673d8f16ff9e57a0482d6a3ec3ee3db33699f57f
Reviewed-on: https://review.whamcloud.com/40294
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13620 tests: pool_add_targets() fix

Fix pool_add_targets to don't fail if number of
OSTs is >= 10 - lctl expects them in a hex view.
Check result of "lctl pool_add". Pass only if
the result is either 0 or 17(EEXIST).
Finally, make pool_add_targets to check that were
added only requested OSTs, i.e. don't fail to add
OST0 if OST1 is already in a pool.

Test-Parameters: trivial testlist=ost-pools
HPE-bug-id: LUS-8723
HPE-bug-id: LUS-8941
Change-Id: I841b3db3a89dbc86075cd23b7d71764ffb849181
Signed-off-by: Elena Gryaznova <elena.gryaznova@hpe.com>
Signed-off-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-on: https://review.whamcloud.com/38803
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13584 tests: gather_logs() fix

Fix gather_logs() to work on real HW where server nodes
do not have the access to clients.

Signed-off-by: Elena Gryaznova <elena.gryaznova@hpe.com>
HPE-bug-id: LUS-8888
Reviewed-by: Alexander Lezhoev <alexander.lezhoev@hpe.com>
Reviewed-by: Vladimir Saveliev <vladimir.saveliev@hpe.com>
Change-Id: Ifeea54e20d3123ee64582e32b92a4573e60ff33e
Reviewed-on: https://review.whamcloud.com/38660
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Vladimir Saveliev <c17830@cray.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13513 osp: make neterr not fatal for precreate_reserve

When OST_CREATE (not resendable rpc) sent by precreate thread fails
with network error, osp_pre_update_status() sets d->opd_pre_status to
EIO. osp_precreate_reserve() considers EIO as fatal and does not wait
for another attempt from precreate thread. That may make
mdt_intent_open() to return ENOSPC confusing a caller.  ENOSPC comes
from lod_alloc_rr().

osp_precreate_send(): in case of network error switch EIO to ENOTCONN.

Test to illustrate the issue is added.

Cray-bug-id: LUS-8811
Signed-off-by: Vladimir Saveliev <c17830@cray.com>
Change-Id: Iffaad9bd16f216f758c784b708e21b525c999b14
Reviewed-on: https://review.whamcloud.com/38472
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13453 osd-ldiskfs: do not leak inode if OI insertion fails

osd_create() should destroy just created inode if OI insertion
fails.

also fixes lustre_index_restore() to drop nlink for object to
be removed.

the patch adds two tests:
 - ENOSPC on OI insertion
 - ENOSPC on .. insertion, i.e. directory block allocation

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I2a5db657c7dab54b8dc2c50bc29365d5ee754a2e
Reviewed-on: https://review.whamcloud.com/38235
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

Merge "LU-9121 lnet: User Defined Selection Policy (UDSP)"

LU-12682 llite: fake symlink type of foreign file/dir

This patch implements a "fake symlink" specific usage of
"foreign" LOV/LMV format. It basically allows these
particular type of foreign files/dirs to behave as a
symlink from VFS point of view, by allowing to construct
a relative path from the LOV/LMV foreign content, to
complement it with a prefix, and then to expose it to
the VFS as a symlink destination. The default/internal
mechanism simply takes the full foreign free string as
the relative path, and for more complex internal formats
an upcall has been implemented to provide format's
details (presently just in terms of constant strings
and substrings positions in EA, but this can be enhanced)
to llite layer.
Using this feature, instead of real symlinks or user EA,
will permit to benefit from the special features (lock,
prefetch, caches) already implemented to handle both
LOV/LMV EAs.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: Id3c262e3b042447aa09aad25f682ff02787b350d
Reviewed-on: https://review.whamcloud.com/35856
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Ben Evans <beevans@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14444 gss: handle empty reqmsg in sptlrpc_req_ctx_switch

In sptlrpc_req_ctx_switch(), everything is already there to handle
the case of a ptlrpc_request that has an empty rq_reqmsg.
But assertions were left over at the beginning of the function, so
just remove them from here.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I6ae1f8b9da9600d3b57b9efc9018c2461114f2fe
Reviewed-on: https://review.whamcloud.com/41685
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: Oleg Drokin <green@whamcloud.com>

LU-14455 mdt: fix DoM lock prolong logic

- don't stop at the first found lock if it is not PW or EX lock
- add LCK_GROUP lock as valid mode of lock to check

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: If947f8565008953cc34146b6f0ac1e0f0a038bb5
Reviewed-on: https://review.whamcloud.com/41701
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14436 tgt: only use T10PI guard when doing full sector read

The T10PI guard was generated on full sectors, if we
do we partial read and still use the guard, the rpc
checksum won't match.

Test-Parameters: trivial
Change-Id: I40d481d703a46b9711021a162208b86a956bd8d1
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-on: https://review.whamcloud.com/41677
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14435 doc: include lfs-flushctx manpage inside packages

lfs manpage redirects to lfs-flushctx(1), so it has to be
included in the Lustre packages.

Test-Parameters: trivial
Fixes: c246a9ba04 ("LU-14263 gss: unlink revoked key")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I5c55f8a74eb6dac20fa85b6ea0663ad701341006
Reviewed-on: https://review.whamcloud.com/41676
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>

LU-14430 mdd: fix inheritance of big default ACLs

If the number of default ACLs in directory is more than 31, then
mdd_acl_init() fails to inherit them for a newly created file.
This limitation is caused by using a fixed-size def_acl_buf buffer
in the mdd_create()->mdd_acl_init() call chain. Instead, the
default ACL buffer should be increased when it is needed.

Patch adds check for -ERANGE after mdd_acl_init(), reallocates
default ACL buffer with required size and calls mdd_acl_init()
again. Thus big default ACL are processed as expected.

Fixes: 6350af100c20 ("LU-3437 mdd: Fix ACL/def_ACL during object creation")
Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I700da90c09f824955fcb8dc7ca0bc2f581f916a0
Reviewed-on: https://review.whamcloud.com/41494
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

New tag 2.14.50

Change-Id: I760c8197bafa0f7eb8d1f5d78863900d647ccddc
Signed-off-by: Oleg Drokin <green@whamcloud.com>

LU-9121 lnet: User Defined Selection Policy (UDSP)

User Defined Selection Policies (UDSP) are introduced to add
ability of fine traffic control. The policies are instantiated
on LNet constructs and allow preference of some constructs
over others as an extension of the selection algorithm.
The order of operation is defined by the selection
algorithm logical flow:

   1. Iterate over all the networks that a peer can be reached on
      and select the best local network
      - The remote network with the highest priority is examined
        (Network Rule)
      - The local network with the highest priority is selected
        (Network Rule)
      - The local NI with the highest priority is selected
        (NID Rule)
   2. If the peer is a remote peer and has no local networks,
      - then select the remote peer network with the highest
        priority (Network Rule)
      - Select the highest priority remote peer_ni on the network
        selected (NID Rule)
      - Now that the peer's network and NI are decided, select
        the router in round robin from the peer NI's preferred
        router list. (Router Rule)
      - Select the highest priority local NI on the local net of the
        selected route.
        (NID Rule)
   3. Otherwise for local peers, select the peer_ni from the peer.
      - highest priority peer NI is selected
        (NID Rule)
      - Select the peer NI which has the local NI selected on its
        preferred list.
        (NID Pair Rule)

   Accordingly, the User Interface allows for the following:
   - Adding a local network udsp: if multiple local networks are
     available, each one can have a priority.
   - Adding a local NID udsp: after a local network is chosen,
     if there are multiple NIs, each one can have a priority.
   - Adding a remote NID udsp: assign priority to a peer NID.
   - Adding a NID pair udsp: allows to specify local NIDs
     to be added on the list on the specified peer NIs
     When selecting a peer NI, the one with the
     local NID being used on its list is preferred.
   - Adding a Router udsp: similar to the NID pair udsp.
     Specified router NIDs are added on the list on the specified
     peer NIs. When sending to a remote peer, remote net is selected
     and the peer NID is selected. The router which has its nid on
     the peer NI list is preferred.

Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: I0d94d160bdb8c09c1cf467f1c155e14ab761a47e

LU-9121 lnet: Add info on udsp to lnetctl man page

Adding description of UDSP commands to lnetctl man page.
Listing each UDSP rule type with specific parameters.
Adding some examples of UDSP commands.

Test-Parameters: trivial testlist=lnet-selftest,sanity-lnet
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: I1bde8a4da217c9ba89f31b5d9a1e7d26658bfb40
Reviewed-on: https://review.whamcloud.com/38698
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>

LU-9121 lnet: add show udsp command

Add the show udsp command in liblnetconfig and lnetctl

Test-Parameters: trivial testlist=lnet-selftest,sanity-lnet
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: Ibddfa60a7b257b136a6be403e94d6f73fb444222
Reviewed-on: https://review.whamcloud.com/34580
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-9121 lnet: Delete a selection policy

This patch adds the function lustre_lnet_del_udsp()
which takes index of the udsp as input and calls
the IOC_LIBCFS_DEL_UDSP ioctl handler to
delete the udsp with that index value.

Change-Id: Ib81c79fd4e76f4e8fbd84709e08cb5d29a059e63
Test-Parameters: trivial testlist=lnet-selftest,sanity-lnet
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Signed-off-by: Sonia Sharma <sharmaso@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/34553
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-9121 lnet: Add a selection policy

This patch adds the function lustre_lnet_add_udsp()
which marshals the input rules and makes an ioctl
call to add those rules in the lnet structures.

Change-Id: I5fa1e34d401b9eca58381e23cac2eb49c4dd3575
Test-Parameters: trivial testlist=lnet-selftest,sanity-lnet
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Signed-off-by: Sonia Sharma <sharmaso@whamcloud.com>
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/34529
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-9121 lnet: Add the userspace Marshalling API

Given a UDSP and a memory block of size
enough to hold a marshalled udsp, marshal
the UDSP pointed to by udsp into the memory block.

Change-Id: I67e9c2cc0d7f3dab1e968019d5ee546e6fefeba3
Test-Parameters: trivial testlist=lnet-selftest,sanity-lnet
Signed-off-by: Sonia Sharma <sharmaso@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/34513
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>

LU-9121 lnet: Add the userspace De-Marshalling API

Given a bulk containing a single UDSP, De-Marshalling
API demarshals it and populates a udsp structure.

Change-Id: Id2502dffc6fd259ad552c018b0cab415a12a8cfe
Test-Parameters: trivial testlist=lnet-selftest,sanity-lnet
Signed-off-by: Sonia Sharma <sharmaso@whamcloud.com>
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/34501
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-9121 lnet: ioctl handler for get policy info

Add ioctl handler for GET_UDSP_SIZE and GET_UDSP

Test-Parameters: trivial testlist=lnet-selftest,sanity-lnet
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: I4e188cac92ca98c26b26e00b8917ad3632d99d06
Reviewed-on: https://review.whamcloud.com/34579
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>

LU-9121 lnet: ioctl handler for "delete policy"

The ioctl handler for "delete policy" deletes
a policy with the given index value. It
returns 0 if a policy with that index is found
else it returns -EINVAL.

Change-Id: I037c6808e71e5c13ccf46c243145b0ce6f1229cb
Test-Parameters: trivial testlist=lnet-selftest,sanity-lnet
Signed-off-by: Sonia Sharma <sharmaso@whamcloud.com>
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/34552
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>

LU-9121 lnet: Add the ioctl handler for "add policy"

The ioctl handler for "add policy" de-marshals the
udsp rules passed from userspace and then add the
rules if there is no copy of the same rules already
added. Apply the rules to the existing LNet
constructs.

Change-Id: Ia76ecdea6de94e1dbcfe71bfcb5a1753fb3c874d
Test-Parameters: trivial testlist=lnet-selftest,sanity-lnet
Signed-off-by: Sonia Sharma <sharmaso@whamcloud.com>
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/34514
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>

LU-9121 lnet: Add the kernel level De-Marshalling API

Given a bulk allocated from userspace containing a
single UDSP, De-Marshalling API demarshals it
and populate the provided udsp structure.

Change-Id: I0a5a9148f8b2abc64284cb7780c37cbc8063b828
Test-Parameters: trivial testlist=lnet-selftest,sanity-lnet
Signed-off-by: Sonia Sharma <sharmaso@whamcloud.com>
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/34488
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>

LU-9121 lnet: Add the kernel level Marshalling API

Given a UDSP, Marshal the UDSP pointed to by udsp
into the memory block that is allocated from userspace.

Change-Id: I325c977fd9c902f7ee31fceaaf07abf27ef7391e
Test-Parameters: trivial testlist=lnet-selftest,sanity-lnet
Signed-off-by: Sonia Sharma <sharmaso@whamcloud.com>
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/34403
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>

LU-9121 lnet: Apply UDSP on local and remote NIs

When a peer net, peer ni, local net or local ni are created
apply the UDSPs in the system on these constructs.

Test-Parameters: trivial testlist=lnet-selftest,sanity-lnet
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: I0ee427f1bc0d1c8d1ddc17072dcbd9442403fa0f
Reviewed-on: https://review.whamcloud.com/34355
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-9121 lnet: UDSP handling

This patch adds the following functionality:
1. Add UDSPs
2. Delete UDSPs
3. Apply UDSPs

- Adding a local network udsp: if multiple local networks are
available, each one can have a priority.
- Adding a local NID udsp: after a local network is chosen,
if there are multiple NIs, each one can have a priority.
- Adding a remote NID udsp: assign priority to peer NIDs.
- Adding a NID pair udsp: allows to specify local] NIDs
to be added to the list on the specified peer NIs. When
selecting a peer NI, the one with the local NID being used
on its list is preferred.
- Adding a Router udsp: similar to the NID pair udsp.
Specified router NIDs are added on the list on the specified
peer NIs. When sending to the remote peer, remote net is
selected and the peer NID is selected. The router which has
its nid on the peer NI list is preferred.
- Deleting a udsp: use the specified policy index to remove it
from the policy list.

Generally, the syntax is as follows
 lnetctl policy <add | del | show>
  --src: ip2nets syntax specifying the local NID to match
  --dst: ip2nets syntax specifying the remote NID to match
  --rte: ip2nets syntax specifying the router NID to match
  --priority: Priority to apply to rule matches
  --idx: Index of where to insert the rule. By default it appends
 to the end of the rule list

Test-Parameters: trivial testlist=lnet-selftest,sanity-lnet
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: If588b78f3d5fce286270afb02e0e7683185d6898
Reviewed-on: https://review.whamcloud.com/34354
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>

LU-9121 lnet: select best peer and local net

Select the healthiest and highest priority peer and local net when
sending a message.

Test-Parameters: trivial testlist=lnet-selftest,sanity-lnet
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: I42717e7fdc3226c6faa7c59c713f18422e27f2e5
Reviewed-on: https://review.whamcloud.com/34352
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>

LU-9121 lnet: Select NI/peer NI with highest prio

Modify the selection algorithm to select the highest priority
local and peer NI. Health always trumps all other selection
criteria

Test-Parameters: trivial testlist=lnet-selftest,sanity-lnet
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: I487a706f4da30311d0bd59fe03f72dbe68a52425
Reviewed-on: https://review.whamcloud.com/34351
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>

LU-9121 lnet: Preferred gateway selection

Add mechanism for managing preferred gateway lists.
When selecting a route through a gateway, if there exists
a preferred gateway list for the destination peer, then choose
the preferred gateway. If there are multiple preferred
gateways, to make the selection, use in order of decreasing
importance: route priority, number of hops, number of available
tx credits on the associated lpni and route sequence counters.
If there are no preferred routes, select the best route
available using the same criteria.

Test-Parameters: trivial testlist=lnet-selftest,sanity-lnet
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: If46920cf7b79aa8b211d6c0a35995edce9b1699a
Reviewed-on: https://review.whamcloud.com/34353
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>

LU-9121 lnet: foundation patch for selection mod

Add the priority and preferred NIDs fields in the lnet_ni,
lnet_net, lnet_peer_net and lnet_peer_ni. Switched
the implementation of the preferred NIDs list to list_head
instead of array, because the code is more straight forward.
There is more memory overhead due to list_head, but these lists
are expected to be small, so I chose code simplicity over memory.

Test-Parameters: trivial testlist=lnet-selftest,sanity-lnet
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: I0c75855b736345c25e1604083eee2b65d38ef28d
Reviewed-on: https://review.whamcloud.com/34350
Reviewed-by: Chris Horn <chris.horn@hpe.com>

LU-9121 lnet: UDSP liblnetconfig structure def

Definition of the UDSP structures used to store parsed UDSPs

Test-Parameters: trivial testlist=lnet-selftest,sanity-lnet
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: Ifc61a191de85d7904c70917d0c45648aecdd0310
Reviewed-on: https://review.whamcloud.com/34254
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>

LU-9121 lnet: UDSP storage and marshalled structs

Commit the structures which will be used by kernel space
to store the UDSPs. This commit also adds the IOCTL structures
which are used for marshalling the UDSPs between user and
kernel space.

Test-Parameters: trivial testlist=lnet-selftest,sanity-lnet
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: I490fa9a050cb1f8debc381be773cda4ce8abe29b
Reviewed-on: https://review.whamcloud.com/34253
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>

New release 2.14.0

Change-Id: I2eb99af8fbeaab80b6614e427b77949b1225b406
Signed-off-by: Oleg Drokin <green@whamcloud.com>

LU-14345 misc: update e2fsprogs to 1.45.6.wc5

Update Changelog to reference new e2fsprogs release.

4aea203f LU-5949 e2fsck: call delete_inode() properly
8725134d LU-5949 e2fsck: simplify inode badness handling
71b74579 LU-14345 e2fsck: fix check of directories over 4GB

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ie0845eeed410f6f9f8ef985342fc19d160aa8cb0
Reviewed-on: https://review.whamcloud.com/41433
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Artem Blagodarenko <artem.blagodarenko@hpe.com>
Reviewed-by: Peter Jones <pjones@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

New RC 2.14.0-RC3

Change-Id: I594b5c6d0da7f067bef69fa7a7027374d4434dd8
Signed-off-by: Oleg Drokin <green@whamcloud.com>

LU-14424 Revert "LU-9679 osc: simplify osc_extent_find()"

It looks like there are performance regressions atttributed to this patch.

This reverts commit 80e21cce3dd6748fd760786cafe9c26d502fd74f.

Change-Id: I55e0abd50573dd82a9d216f9c3b01483f99c3223
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/41498
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

New release candidate 2.14.0 RC2

Change-Id: Iad3d71e7dcf96173d192717ef4fef3f0dc12b051
Signed-off-by: Oleg Drokin <green@whamcloud.com>

LU-13751 tests: remove read of changelog sanity 160j

sanity test 160j tries to read the changelog after one of two
client mounts is unmounted.  In this case, we can fail to read
the changelog and get a "Cannot send after transport endpoint
shutdown" error.

The intention of sanity test 160j is to check that
there is no LBUG due to missed obd device.  So, do not try to
read from the changelog after file system unmount.

Test-Parameters: trivial testlist=sanity env=ONLY=160j,ONLY_REPEAT=200
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I1746a422b25d546b9aae38ae8438d9c08bce8827
Reviewed-on: https://review.whamcloud.com/41317
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-14355 ptlrpc: do not output error when imp_sec is freed

There is a race condition on client reconnect when the import is being
destroyed.  Some outstanding client bound requests are being processed
when the imp_sec has already been freed.
Ensure to output the error message in import_sec_validate_get() only
if import is not already in the zombie work queue.

Fixes: 135fea8fa9 ("LU-4423 obdclass: use workqueue for zombie management")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I4b431128e04f11b1e3ee7de47090af87538c3558
Reviewed-on: https://review.whamcloud.com/41310
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14299 test: sleep to enable quota acquire again

sanity-quota test_61 fails with incorrect quota exceeded
errors because quota acquire will be disabled for 5 seconds
after edquot flag is set.  The test should introduce some
delay between the test of over quota and normal one.

Test-Parameters: trivial fstype=zfs testlist=sanity-quota env=ONLY=61,ONLY_REPEAT=20
Fixes: 530881fe4ee20 ("LU-7816 quota: add default quota setting support")
Change-Id: I8040ba960f32cf01cb7cee3a77c06ad4bd732f0e
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/41389
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13449 tests: fix recovery-small test_140b check

The recovery timer is printed in MM:SS format, but the current test
is unhappy if the SS part is printed as "08" or "09" since that is
interpreted by bash as an invalid octal number.  Also, the current
check does not handle the case if recovery is longer than a minute.

Change the code to convert MM:SS back to seconds for the comparison.

Test-Parameters: trivial testlist=recovery-small env=ONLY=140
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ie1dc77a88bb0e8fd5025f2b5ca57d4a61d3ebbe5
Reviewed-on: https://review.whamcloud.com/39909
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14316 llite: quiet spurious ioctl warning

Calling "lfs setstripe" prints a suprious warning about using the old
ioctl(LL_IOC_LOV_GETSTRIPE) when that is not actually the case.

Remove the ioctl warning for now and deal with related issues later.

Fixes: 364ec95f3688 ("LU-9367 llite: restore ll_file_getstripe in ll_lov_setstripe")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I20f5a7adb60a30fce27e49827bd46229e2ce7057
Reviewed-on: https://review.whamcloud.com/41427
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-11607 tests: replace lustre_version/fstype in sanity-scrub

The routine get_lustre_env() is available to all Lustre test
suites and sets an environment variable for the Lustre
version and file system types of servers.

In sanity-scrub, sanity-hsm and sanity-lfsck, replace calls
to lustre_version_code() and facet_fstype() for all server
types with definitions from get_lustre_env().

Clean up around any modifications by removing calls to
return after skip() or skip_env().

Fixes: c8790ae52393 (LU-1538 tests: standardize test script init - dne-part-2)
Fixes: c54b6ca2bdb5 (LU-13718 tests: add LU numbers to skipped tests)
Test-Parameters: trivial fstype=ldiskfs testlist=sanity-scrub,sanity-hsm,sanity-lfsck
Test-Parameters: fstype=zfs testlist=sanity-scrub,sanity-hsm,sanity-lfsck
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I5aa2898f6efef127cf2d7f4a2f08838f503c51ab
Reviewed-on: https://review.whamcloud.com/35929
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Vikentsi Lapa <vlapa@whamcloud.com>

LU-14389 lov: avoid NULL dereference in cleanup

Running racer concurrently with file migration crashes easily
when the layout changes for a file in an unexpected way:

  lov_init_composite() lustre-clilov: DOM entries with different sizes
  lov_layout_change() lustre-clilov: cannot apply new layout on
      [0x200000402:0x3e6a:0x0] : rc = -22
  BUG: unable to handle kernel NULL pointer dereference at 0x00000014
  IP: [<ffffffffa08baef4>] lov_delete_composite+0x104/0x540 [lov]
  Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
  CPU: 1 PID: 20227 Comm: ln

Avoid the NULL dereference if the entry is not fully initialized
during cleanup.

Test-Parameters: testlist=racer env=DURATION=3600
Fixes: 61a002cd863 ("LU-13602 flr: skip unknown FLR component types")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I8fe17f1b49ca2bccc7a285febe47032d023ebbe5
Reviewed-on: https://review.whamcloud.com/41398
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13669 llite: make readahead aware of hints

Calling madvise(MADV_SEQUENTIAL) and madvise(MADV_RANDOM) sets the
VM_SEQ_READ and VM_RAND_READ hints in vma->vm_flags.  These should
be used to guide the Lustre readahead for better performance.

Disable the kernel readahead for mmap() pages and use the llite
readahead instead.  There was also a bug in ll_fault0() that would
set both VM_SEQ_READ and VM_RAND_READ at the same time, which was
confusing the detection of the VM_SEQ_READ case, since VM_RAND_READ
was being checked first.

This changes the readahead for mmap from submitting mostly 4KB RPCs
to a large number of 1MB RPCs for the application profiled:

  llite.*.read_ahead_stats     before        patched
  ------------------------     ------        -------
  hits                           2408         135924 samples [pages]
  misses                        34160           2384 samples [pages]

  osc.*.rpc_stats           read before    read patched
  ---------------          -------------  --------------
  pages per rpc            rpcs   % cum%   rpcs   % cum%
     1:                    6542  95  95     351  55  55
     2:                     224   3  99      76  12  67
     4:                      32   0  99      28   4  72
     8:                       2   0  99       9   1  73
    16:                      25   0  99      32   5  78
    32:                       0   0  99       8   1  80
    64:                       0   0  99       5   0  80
   128:                       0   0  99      15   2  83
   256:                       2   0  99     102  16  99
   512:                       0   0  99       0   0  99
  1024:                       1   0 100       3   0 100

Readahead hit rate improved from 6% to 98%, and 4KB RPCs dropped from
95% to 55% and 1MB+ RPCs increased from 0% to 16% (79% of all pages).

Add debug to ll_file_mmap(), ll_fault() and ll_fault_io_init() to
allow tracing VMA state functions for future IO optimizations.

Fixes: 62ef9c949753 ("add 2.6.27 kernel support")
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I4bbb028db05b21ae01dafe6a7bea398e9b74d8a4
Reviewed-on: https://review.whamcloud.com/41228
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10958 ofd: data corruption due to RPC reordering

Without read-only cache, it is possible that a client
resends a BRW RPC, receives a reply from the original
BRW RPC, modifies the same data and sends a new BRW
RPC, however, because of RPC reordering stale data
gets to disk.

Let's use range locking to protect against this race.

Change-Id: I35cbf95594601eacfc5f108b14e4c447962b0bbf
Signed-off-by: Andrew Perepechko <c17827@cray.com>
Cray-bug-id: LUS-5578,LUS-8943
Reviewed-on: https://review.whamcloud.com/32281
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

New release candidate 2.14.0-RC1

Change-Id: I54bced5067f605bae67faffce46d89383dc69a39
Signed-off-by: Oleg Drokin <green@whamcloud.com>

LU-13184 tests: wait for OST startup in test_112

Wait for OST0000 to finish mounting and connecting to the MDS before
forcing to create any files there.  Otherwise, intermittent failures
are seen because the OST has no objects and returns -ERANGE:

    lfs setstripe: 'f112.conf-sanity.0': Numerical result out of range
    problem creating f112.conf-sanity.0 on OST0000

Test-Parameters: trivial testlist=conf-sanity envdefinitions=ONLY=112,ONLY_REPEAT=100
Fixes: 416e67222b76 ("LU-12036 ofd: add 'no_precreate' mount option")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: If2d547a42c51a7028803ec25680931a7593ebbe5
Reviewed-on: https://review.whamcloud.com/37393
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14307 quota: fix NULL pointer dereference

Fix NULL pointer dereference at 0x20 in
qmt_trans_start_with_slv->lquota_lqe_debug.

HPE-bug-id: LUS-9662
Change-Id: Iead0df053ae0dcb7453c1910a4b4b7a3728da829
Signed-off-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-on: https://review.whamcloud.com/41264
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14356 utils: change position of the -lext2fs option

This patch changes the position of the -lext2fs option
in the gcc command line so as to resolve the following
issue:

mount_osd_ldiskfs.so: undefined symbol: unix_io_manager

Change-Id: I9ceaca867697c132b8d4a7800169101a024d17b8
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/41305
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13578 test: use a single read() in sanity test_39r

In sanity test_39r() ensure that the we only call read() once on the
file ot update it's atime. Otherwise the file atime may be greater
than the OST object atime due to a final read() done by dd which
returns no bytes and does not generate a BRW RPC to the OST. Even
though it returns 0 bytes, it requested a non-zero number of bytes and
is therefore required to update the file access time.

Fixes: 7c9ce8aac ("LU-13383 ofd: lazy atime update")
Test-Parameters: trivial testlist=sanity env=ONLY=39r,ONLY_REPEAT=400
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I1dd4e93ac150b34f4fc943fe25b92bf9119b0461
Reviewed-on: https://review.whamcloud.com/40973
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>

LU-14198 build: Use pax tar format for build of dist-targets

   The tar ustar format used during build of dist-* targets (which is
   used when building e.g. rpms and debs) has several limitations and
   will prevent e.g. users with uid > 2097151 (2^21) from building
   these targets (or from passing the autoconf test for tar).

   This commit changes format from ustar to POSIX.1-2001/pax which
   will remove this and several other limits (with path-length being
   one relevant example, see LU-12078).

Signed-off-by: Daniel Ahlin <ahlin@google.com>
Change-Id: Ic66ca696ede2e359a04c179c6d630baacaa9bcb1
Reviewed-on: https://review.whamcloud.com/40915
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>

LU-14286 osd-ldiskfs: enable fallocate by default

Enable fallocate on ldiskfs OSTs by default now that the known
problems have been resolved.  The default mode=0 is the standard
"allocate unwritten extents" behavior used by ext4.  This is by
far the fastest for space allocation, but requires the unwritten
extents to be split and/or zeroed when they are overwritten.

The OST fallocate mode=1 can also be set to use "zeroed extents",
which may be handled by "WRITE SAME", "TRIM zeroes data", or
other low-level functionality in the underlying block device.
This is somewhat slower at fallocate() time (especially for very
large allocations), but still avoids sending any data over the
network, avoids runtime overhead from managing the extents.  There
is not yet an FALLOC_FL_* flag to request this behavior from the
client on a per-file basis.

If problems are hit in the field, fallocate can also be disabled
with mode=-1 at runtime or persistently.

   lctl set_param [-P] osd-ldiskfs.*.fallocate_zero_blocks=<mode>

Ensure that all of the tests which currently use fallocate() are
enabling it for test runs, even if the default changes again.

Fixes: 4f18e08099e5 ("LU-14286 osd-ldiskfs: fallocate with unwritten extents")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Iefa71c525597d54fc82a3d6de27a50d4d2ce7057
Reviewed-on: https://review.whamcloud.com/41315
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14337 lov: return stripe_count=1 instead of 0 for DoM files

Return stripe_count=1 instead of 0 for DoM files to avoid
divide-by-zero for older userspace that calls this ioctl,
e.g. lustre ADIO driver.

Signed-off-by: Emoly Liu <emoly@whamcloud.com>
Change-Id: I43c5e01bdee834f9a05a669a3e6f3d5cd926cb87
Reviewed-on: https://review.whamcloud.com/41265
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14283 osc: avoid crash if ocd reset

Avoid divide-by-zero if OSC obd_connect_data is not fully initialized.
cl_ocd_grant_param is only set after cl_max_extent_pages is OK to use.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ibcee30b46e24ca3d4c2b571b27f3c0bb43f4bf71
Reviewed-on: https://review.whamcloud.com/41225
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14339 obdclass: add option %H for jobid

Add a option %H to avoid jobid too long in some cases.

Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: Iaf70da5de25fd321a21e6e6cd7f7d211dca1adf3
Reviewed-on: https://review.whamcloud.com/41262
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>

LU-14306 sec: get rid of bad rss-counter state messages

When doing O_DIRECT IOs on encrypted files, messages about bad
rss-counter state can be seen in the console. The mm get confused
because we twist the Lustre pages used for RPCs so that they are
suitable for llcrypt API.
In order to do this properly, the original mapping on these pages
must be preserved outside of the encryption/decryption needs.

Fixes: 728036f256 ("LU-12275 sec: O_DIRECT for encrypted file")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I80ebcd3f96c51a3d158d7ef66f23b8da13904c52
Reviewed-on: https://review.whamcloud.com/41199
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14286 osd-ldiskfs: fallocate with unwritten extents

The osd_fallocate() code should typically be allocating unwritten
extents with LDISKFS_GET_BLOCKS_CREATE_UNWRIT_EXT instead of actually
zeroing the blocks on disk with LDISKFS_GET_BLOCKS_CREATE_ZERO.

Writing zeroes during fallocate() is typically slower initially, and
is causing timeouts in sanity test_150e, which is trying to fill up
all OSTs to 90%.  In some cases, zeroing the underlying blocks can
use the underlying storage support for efficient zeroing (WRITE_SAME),
so it may be faster for later use than uninitialized extents that have
to be converted to initialized extents by (possibly) splitting them
into smaller extents and/or zero filling them when they are paritally
being overwritten.

Add a tunable parameter osd-ldiskfs.*.fallocate_zero_blocks to allow
selecting this behavior at runtime.  The default is -1, to disable
fallocate completely (return -EOPNOTSUPP) due to current bugs.

Test-Parameters: testlist=sanityn env=ONLY=16,ONLY_REPEAT=10
Fixes: 72617588ac8c ("LU-14286 osd-ldiskfs: fallocate() should zero new blocks")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ida3692c487fdc8918863fc5c99459caaba17d92e
Reviewed-on: https://review.whamcloud.com/41204
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14312 ldlm: don't change GROUP lock GID on client

GROUP lock GID is part of inodebits policy and is passed
to the server from client in policy li_gid field.
Meanwhile the ldlm_ibits_policy_wire_to_local() is used on
client also when server reply or completion AST is processed,
so client original GID can be overwritten by server value.
This is not problem if both server and client have the same
Lustre version but if server is older then it can have garbage
in li_gid field and client lock policy is updated with it.

Considering that GROUP lock GID is never changed and server should
not do that, the solution is to ignore returned li_gid from server
and never update original GID of GROUP lock on client from server
response.

Test-Parameters: testlist=sanity serverversion=2.12.6 env=ONLY=272b
Test-Parameters: testlist=sanity serverversion=2.13.0 serverdistro=el7.7 env=ONLY=272b

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I9a82f4b3513fd93d63b92a9527cb7b89c635e61b
Reviewed-on: https://review.whamcloud.com/41268
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14034 hsm: add PID file handling to lhsmtool_posix

Add pid-file handling to lhsmtool_posix to prevent accidentally
running concurrent instances of the copytool. (Multiple instances are
still allowed if you do not use PID files or use separate files.)

Use the PID file to avoid needing libtool when stopping, continuing,
or killing the copytool from test scripts.

Remove unnecessary libtool usage from test scripts.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I4f012a88c58f3a86a731df3b7d35ff32db047c2d
Reviewed-on: https://review.whamcloud.com/40253
Tested-by: Maloo <maloo@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Ben Evans <beevans@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14326 osc: correctly update size/kms for fallocate

* fallocate chose oa->o_size for falloc_offset and o->o_blocks for
  falloc_end, but forgot to change attr->cat_size and attr->cat_kms
  to use sa_attr.lvb_size to update osc's lvb and kms if it expands
  the file's size.

  Other setattr IO uses @size (sa_falloc_offset in fallocate case) to
  update the lvb and kms.

* lock request extent for fallocate should be
  [sa_falloc_offset, sa_falloc_end)

* calculate sa_attr.lvb_size correctly for osc objects
  (lov_io_sub_inherit())

Test-Parameters: testlist=sanityn env=ONLY=16,COUNT=50000,ONLY_REPEAT=10
Test-Parameters: testlist=sanity-benchmark env=ONLY=fsx,COUNT=50000,ONLY_REPEAT=10
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Fixes: 48457868a02a ("LU-3606 fallocate: Implement fallocate preallocate operation")
Change-Id: I7dbed3bc6899a3db53284c8aac3cb9476e7958f5
Reviewed-on: https://review.whamcloud.com/41272
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14303 tests: parallel-scale test rr_alloc fails

Correct the parameter for DNE environment. Else the
test case will fail on 'No such file or directory'.

Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: Ib94c5b17a3b49153ac229bfc4dfcee39bd9f60d4
Reviewed-on: https://review.whamcloud.com/41192
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14334 lnet: update changelog

Updated changelog to indicate changes in OFED/MOFED support
as well as new features added in this version of LNet.

Test-Parameters: trivial
Change-Id: I264f6566324da42fd51a8e159d172cbf0ae1a28b
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/41263
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14283 obdclass: connect vs disconnect race

There might be a possible race if setup (connect)
and cleanup (disconnect) are tangled together(similar
comments in osc_disconnect()):

  Thread1: Thread2:
   connecting  class_cleanup
     ptlrpc_connect_interpret
   obd->obd_setup = 0
obd_import_event
  if (obd->obd_set_up)
osc_init_grant() /*skipped*/
        ptlrpc_activate_import..

And If RPC was waked up and send out before
class_disconnect_exports(), It might hit divide zero crash
in osc_announce_cached() because @cl_max_extent_pages is zero.

The problem is we clear @obd_setup too early, It should be cleared
when OBD is really shutdown.

Fixes: 45900a ("LU-4134 obdclass: obd_device improvement")
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I898b6f53602c05221a3154a61615a0e270167ac6
Reviewed-on: https://review.whamcloud.com/41256
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-13602 flr: skip unknown FLR component types

Currently, in lov_init_composite() it will quit with error when
reading an unknown LOV pattern.
Since FLR will be used for upcoming new features, like PCC-RO,
an old client should be able to read the old format of the
component types, ignore and skip the new types of FLR component.

Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: Ica3fe98203d44b52cf25b085c34c83b1a4702464
Reviewed-on: https://review.whamcloud.com/39513
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12125 llite: send file mode with rename RPC

In preparation for parallel rename operations, send renamed file mode
to the MDS in order to allow the rename locking to be more efficient.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I4691f30d151a8ff81e443d24109234341b3ebbe5
Reviewed-on: https://review.whamcloud.com/41184
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14286 osd-ldiskfs: don't read unwritten blocks

which were allocated using fallocate(2), instead fill
the pages with zeroes.

Add a test to verify that fallocated blocks read as zeroes.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I9c0c90f93fd33f26f834144e225b2643cf9fffb7
Reviewed-on: https://review.whamcloud.com/41216
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14194 cksum: add lprocfs checksum support in MDC/MDT

Add missed support for checksum parameters in MDC and MDT
Handle T10-PI parameters in MDT similar to OFD, move all
functionality to target code and unify its usage in both
targets

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I7d397067304e028bf597d5c3ab16250731ccba9d
Reviewed-on: https://review.whamcloud.com/40971
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>

LU-13676 tools: find unique stacks from sysrq-t

Enhance the crash-find-unique-traces.awk script to be more useful:
- allow parsing output from sysrq-t
- skip leading fields from input (e.g. timestamps, syslog header)
- sort messages to avoid duplicates
- handle some variance in kernel function names
- print thread names as well as PIDs for all matching threads
- print number of threads matching a particular stack trace
- warn if parts of the stack traces were lost due to buffer overflow

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Icfd70fbf255f413ca1aab0b94c62754e4c00628d
Reviewed-on: https://review.whamcloud.com/40956
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14317 ldiskfs: ‘%llu’ mismatch with type ‘long int’ on arm64

This patch fixes build issues like this one on arm64:

lustre-release/ldiskfs/mballoc.c: In function ‘ldiskfs_mb_release’:
lustre-release/ldiskfs/mballoc.c:3180:4: warning:
format ‘%llu’ expects argument of type ‘long long unsigned int’,
but argument 4 has type ‘long int’ [-Wformat=]
    "mballoc: (%llu, %llu, %llu) useless c(0,1,2) loops",
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Change-Id: I07018302fae8da3b71c0cc649753bdcccdc0d34f
Test-Parameters: trivial
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/41191
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>

LU-14263 gss: unlink revoked key

When a GSS context is destroyed, it is unbound from its key, marking
the key as revoked.
The key also needs to be unlinked from the session keyring. This way,
a subsequent context initialization will manage to create a new valid
key and link it to the keyring.

Similarly, add a new '-r' flag to 'lfs flushctx', in order to reap the
revoked keys from the keyring when flushing the GSS context.

Test-Parameters: trivial
Test-Parameters: clientdistro=el7.9 testgroup=review-dne-ssk
Test-Parameters: clientdistro=el8.3 testgroup=review-dne-ssk
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ida4b4ea53202c1f40ad93816fb4ec96fec2bf8bc
Reviewed-on: https://review.whamcloud.com/41047
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14286 tests: reduce fallocate space sanity 150e

Changes to fallocate have increase the time to allocate
space.  The increased run time is making sanity test
150e run long enough to time out.  We need to reduce the
amount of space santiy test 150e fallocates from 90%
to 100 MB per OST or 60% of available space if running
in SLOW mode to allow the test to complete.

Test-Parameters: trivial env=ONLY=150e fstype=ldiskfs testlist=sanity
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: Ic6ab41daf8f451c31d092849ccf6906747baa72d
Reviewed-on: https://review.whamcloud.com/41211
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14324 tests: Fix fsx fallocate detection

Avoid kernel playing fake tricks on us by optimizing away small
increases in file size by fallocate, so to ensure the truth make
large fallocate call instead.
Additionally always truncate to zero after all call, as even a
failed call can result in increased file size.

Test-Parameters: trivial testlist=sanity env=ONLY=150
Test-Parameters: testlist=sanityn env=ONLY=16,COUNT=10000
Test-Parameters: testlist=sanity-dom env=ONLY=fsx,sanityn,SANITYN_ONLY=16
Test-Parameters: fstype=zfs testlist=sanity env=ONLY=150
Test-Parameters: fstype=zfs testlist=sanityn env=ONLY=16,COUNT=10000
Test-Parameters: fstype=zfs testlist=sanity-dom env=ONLY=fsx,sanityn,SANITYN_ONLY=16
Change-Id: I8cddb7d0d30bdf89bf13d18ae36a5f950adc392d
Fixes: 03247bda765c ("LU-14286 osd-ldiskfs: fallocate() should zero new blocks")
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/41202
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14128 lov: correctly set OST obj size

When extends a PFL file to a size locating at a boundary of a stripe
in a component, the truncate won't set the size of the OST object
in the prior stripe.

This patch record the prior stripe in
lov_layout_raid0::lo_trunc_stripeno and add the stripe in the
truncate IO and enqueue the lock covering it.

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Ic5d8e3c16f950003736cd6dbd5af404613f818c7
Reviewed-on: https://review.whamcloud.com/40581
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14296 lnet: use an unbound cred in kiblnd_resolve_addr()

In kiblnd_resolve_addr() call prepare_kernel_cred(NULL) rather than
prepare_creds() to get a cred with unbound capabilities.

Fixes: 1e4bd16acfa ("LU-14006 o2ib: raise bind cap before resolving address")
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Ib2500d82aa4eab5590fa95ae73fd75b2ab542bcd
Reviewed-on: https://review.whamcloud.com/41137
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14301 llite: return EOPNOTSUPP if fallocate is not supported

In ll_fallocate() if the server returns the NFSv3 specific error code
ENOTSUPP then replace it with EOPNOTSUPP to avoid confusing
applications.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I85952be27808704112a6902684900cd938d576ac
Reviewed-on: https://review.whamcloud.com/41148
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Wang Shilong <wshilong@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14281 tests: fix sanity-sec test_56 for ZFS

Skip sanity-sec test_56 in case of ZFS backend, as it does not
support FIEMAP.
Also simplify sanity-sec test_56, as all we care about is whether the
"encrypted" flag is set at all.

Fixes: 33322f3a2488 ("LU-14149 llite: fiemap set flags for encrypted files")
Test-Parameters: trivial
Test-Parameters: testlist=sanity-sec env=ONLY=56 clientdistro=el8.3 serverdistro=el8.3 fstype=ldiskfs
Test-Parameters: testlist=sanity-sec env=ONLY=56 clientdistro=el8.3 serverdistro=el8.3 fstype=zfs
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I69125d3e06acb89dbda421e573d18a2c22beac37
Reviewed-on: https://review.whamcloud.com/41130
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>

LU-13783 o2iblnd: make FMR-pool support optional.

Linux 5.8 removes the FMR-pool API.  This patch makes
all use for this API optional, selected only if the
support exists in the kernel.

Signed-off-by: Mr NeilBrown <neilb@suse.de>
Change-Id: I4c40f3a766f5b46ae4f26d7d3ecf8434a6e5a0cb
Reviewed-on: https://review.whamcloud.com/40287
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sergey Gorenko <sergeygo@nvidia.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-14286 osd-ldiskfs: fallocate() should zero new blocks

The ldiskfs osd_fallocate() does not correctly zero the
allocated blocks. This is not noticed during autotest and
local developer testing because the underlying storage is
zero-filled due to using a sparse backing file and will
usually read back as zero. Also, the fsx data correctness
test was not properly detecting fallocate() support, and
as a result this behavior was not detected properly.

'fsx' uses test_fallocate() to determine if fallocate()
along with its various FLAGS is enabled and could be
called. It was found that test_fallocate() always returned
false as a bug. Resulting is fallocate being skipped always
for all fsx runs.

This patches fixes test_fallocate() to properly determine
if fallocate calls could be supported or not and return
appropriately and correctly (1 or 0) true/false.

After patch:
~~~~~~~~~~~
$ MOUNT_2="yes" bash ./lustre/tests/llmount.sh

$ lfs setstripe -c -1 /mnt/lustre/f110

$ fsx -c 50 -p 100 -N 2500 -S 0 -l 10485760 /mnt/lustre*/f110
Chance of close/open is 1 in 50
Seed set to 4481
fd 0: /mnt/lustre/f110
fd 1: /mnt/lustre2/f110
fsx: test_fallocate: does not support fallocate mode 0x3, disabling!
fsx: test_fallocate: does not support fallocate mode 0x10, disabling!
skipping zero size read
:
fallocating to largest ever: 0x30ce
:
1900[0] 1609444486.316868 fallocate  0x7f6c thru 0xab3a (0x2bcf bytes)
:
2300[1] 1609444487.241910 fallocate  0x0697 thru 0xed8e (0xe6f8 bytes)
:
All operations completed A-OK!
$

Test-Parameters: trivial testlist=sanity,sanityn,sanity-dom env=COUNT=10000
Fixes: 48457868a02a ("LU-3606 fallocate: Implement fallocate preallocate operation")
Fixes: 853d180121a6 ("LU-3606 fsx: Add fallocate operation to fsx")
Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Change-Id: Idbb092db6bd09cfda129f6077dcbfae4f4c9d0d0
Reviewed-on: https://review.whamcloud.com/41119
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14298 tests: fix sanity-sec test_44 for aarch64 client

PAGE_SIZE on aarch64 client is 64KB instead of 4KB, so sizes of IOs
performed in sanity-sec test_44 must be adapted.

Test-Parameters: testgroup=review-ldiskfs-arm env=ONLY=44 clientdistro=el8.3 serverdistro=el8.3
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I3764a75f6df871d71d751764627ed618bedf8e0b
Reviewed-on: https://review.whamcloud.com/41146
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-14187 osd-ldiskfs: fix locking in write commit

Restart transaction in osd layer break rules that
locks are taken afer transaction start.

This patch try to fix by moving transaction restart
to OFD layer.

We will record how many extents we declare with
@oh_declared_ext. And during IO we check if it
run out before we restart whole transaction.

We track an average extent bytes per filesystem and
use it as a hint during declare write commit. This
will avoid us restart transaction ofen if filesystem
is heavily fragmented.

Fixes: 0271b17b ("LU-14134 osd-ldiskfs: reduce credits for new writing")
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I289a3a6775befe159b7fef29004eaaaff873e2c6
Reviewed-on: https://review.whamcloud.com/40991
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-13641 socklnd: announce deprecation of 'use_tcp_bonding'

Add warning to be printed if 'use_tcp_bonding' option is used
notifying the user that the feature is being deprecated.
It is suggested to use MR configuration with dynamic discovery
instead.

Multi-Rail feature doesn't need to be explicitly enabled.
To use MR instead of tcp bonding, group the interfaces
on the same network using the lnetctl utility:

lnetctl net add --net tcp --if eth2,eth3

or via the modprobe configuration file (/etc/modprobe.d/lnet.conf
or /etc/modprobe.d/lustre.conf):

        options lnet networks="tcp(eth2,eth3)"

and make sure dynamic discovery is enabled:

        lnetctl set discovery 1

MR will aggregate the throughput of all configured and available
networks/interfaces shared between peer nodes.

Test-Parameters: trivial
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: I34288ae1c8a1c4092f88b45a571312691f145218
Reviewed-on: https://review.whamcloud.com/41088
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>