Whamcloud - gitweb
fs/lustre-release.git
2 months agoLU-11426 llog: changelog records reordering 87/36187/5
Andrew Perepechko [Tue, 17 Sep 2019 07:34:44 +0000 (10:34 +0300)]
LU-11426 llog: changelog records reordering

Changelog records can get reordered because of a race
window between cr_index generation and llog file
space allocation. This can lead to llog records
loss.

llog_write() holds loghandle->lgh_lock semaphore,
so it seems an appropriate place to generate a
new changelog index.

Change-Id: I034d1a696bde1d0f780e494ab65073e4018ceec9
Signed-off-by: Andrew Perepechko <c17827@cray.com>
Reviewed-by: Alexander Boyko <c17825@cray.com>
Reviewed-by: Alexander Zarochentsev <c17826@cray.com>
Cray-bug-id: LUS-7691
Reviewed-on: https://review.whamcloud.com/36187
Reviewed-by: Alexandr Boyko <c17825@cray.com>
Reviewed-by: Olaf Weber <olaf.weber@hpe.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-9859 libcfs: move misc-device registration closer to related code. 18/36118/2
NeilBrown [Mon, 9 Sep 2019 17:53:56 +0000 (13:53 -0400)]
LU-9859 libcfs: move misc-device registration closer to related code.

The ioctl handler for the misc device is in  lnet/libcfs/module.c
but is it registered in lnet/libcfs/linux/linux-module.c.

Keeping related code together make maintenance easier, so move the
code.

Linux-commit: b4ded66db93bbe1f5323ad38ce51bb1be114934f

Test-Parameters: trivial

Change-Id: Ia2b3590a769214fe964dab7a63fd5edcfd6c5042
Signed-off-by: NeilBrown <neilb@suse.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-on: https://review.whamcloud.com/36118
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12690 llite: error handling of ll_och_fill() 13/35913/4
Bobi Jam [Sat, 24 Aug 2019 17:20:23 +0000 (01:20 +0800)]
LU-12690 llite: error handling of ll_och_fill()

The return error of ll_och_fill() should be handled.

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I4e750001cb124104836fa24e39ec8ae203b51a83
Reviewed-on: https://review.whamcloud.com/35913
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-11380 llapi: separate FID man pages 73/35673/11
Andreas Dilger [Tue, 9 Jul 2019 06:34:47 +0000 (00:34 -0600)]
LU-11380 llapi: separate FID man pages

Add separate man pages for the lfs commands and llapi functions.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I05abfaf888a5474d62feebab4e8db543ba3ebbe5
Reviewed-on: https://review.whamcloud.com/35673
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-11933 mdt: clear sp_cr_flags in migrate unpack 54/36154/3
Lai Siyao [Thu, 15 Aug 2019 14:31:17 +0000 (22:31 +0800)]
LU-11933 mdt: clear sp_cr_flags in migrate unpack

mdt_thread_info.mti_spec is not cleared after operation handling, so
mdt_migrate_unpack() should clear it in case the old values are used.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Ib3d5d39a4a072621c8da8b6ef7869cb4d8178aac
Reviewed-on: https://review.whamcloud.com/36154
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-6142 tests: Fix style issues for multiop.c 04/35904/2
Arshad Hussain [Tue, 6 Aug 2019 11:56:54 +0000 (17:26 +0530)]
LU-6142 tests: Fix style issues for multiop.c

This patch fixes issues reported by checkpatch
for file lustre/tests/multiop.c

Test-Parameters: trivial
Signed-off-by: Arshad Hussain <arshad.super@gmail.com>
Change-Id: I08c038f034e26eafacc99a3f3d7966704a79709c
Reviewed-on: https://review.whamcloud.com/35904
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-6142 ptlrpc: Fix style issues for llog_net.c 23/35823/2
Arshad Hussain [Mon, 5 Aug 2019 20:18:44 +0000 (01:48 +0530)]
LU-6142 ptlrpc: Fix style issues for llog_net.c

This patch fixes issues reported by checkpatch
for file lustre/ptlrpc/llog_net.c

Test-Parameters: trivial
Signed-off-by: Arshad Hussain <arshad.super@gmail.com>
Change-Id: Ieee60565e5332a1546999e6c8edaffc833074271
Reviewed-on: https://review.whamcloud.com/35823
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12570 mdt: request env for DT threads 79/36179/3
Alexey Zhuravlev [Fri, 13 Sep 2019 19:28:06 +0000 (22:28 +0300)]
LU-12570 mdt: request env for DT threads

as part of lock enqueue MDT thread can call ldlm_reclaim_full() to
cancel old unused LDLM locks and that scans all presented namespace
including OFD-originated (with extent locks). thus MDT ends with
calls into OFD code which needs own env marked with LCT_DT_THREAD.

Test-Parameters: testlist=sanity,sanity,sanity,sanity envdefinitions=ONLY="134a",SHARED_KEY=true
Test-Parameters: testlist=sanity,sanity,sanity,sanity envdefinitions=ONLY="134a",SHARED_KEY=true
Test-Parameters: testlist=sanity,sanity,sanity,sanity envdefinitions=ONLY="134a",SHARED_KEY=true
Signed-off-by: Alexey Zhuravlev <bzzz@whamcloud.com>
Change-Id: I231b88159978bc3ce7a3fa0f27e57eb32137c343
Reviewed-on: https://review.whamcloud.com/36179
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12103 ldiskfs: don't search large block range if disk full 80/35180/7
Artem Blagodarenko [Thu, 6 Jun 2019 13:50:11 +0000 (16:50 +0300)]
LU-12103 ldiskfs: don't search large block range if disk full

Block allocator tries to find:
1) group with the same range as required
2) group with the same average range as required
3) group with required amount of space
4) any group

For quite full disk step 1 is failed with higth
probability, but takes a lot of time.

Skip 1st step if disk space < 25%
Skip 2d step if disk space < 15%
Skip 3d step if disk space < 5%
Also check if group has any free space on step 4.

This three thresholds can be adjusted through added interface.

Variables added which counts unsuccessfull group processing loops.
This can show allocator effectiveness in different circumstances.

This statistics output through mb_alloc file. This file is
useful to track allocator activity.

Signed-off-by: Artem Blagodarenko <c17828@cray.com>
Change-Id: I18c7147e32951c49e12a2444803aa2995bb4ae2d
Cray-bug-id: LUS-6746
Reviewed-on: https://review.whamcloud.com/35180
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12776 doc: update Changelog to reflect primary kernels 14/36214/4
Joseph Gmitter [Tue, 17 Sep 2019 17:45:49 +0000 (13:45 -0400)]
LU-12776 doc: update Changelog to reflect primary kernels

This patch updates the lustre/Changelog file to distinguish
between primary kernels that are built and tested during the
current release cycle from other kernels that are also known
to have built at some point.

Test-Parameters: trivial
Signed-off-by: Joseph Gmitter <jgmitter@whamcloud.com>
Change-Id: I7b6b5ce2dc813da91e2d9cd7a499168443e41f59
Reviewed-on: https://review.whamcloud.com/36214
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Peter Jones <pjones@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12612 osd: add lnb size down to osd 01/35801/7
Alex Zhuravlev [Thu, 15 Aug 2019 18:33:08 +0000 (22:33 +0400)]
LU-12612 osd: add lnb size down to osd

so that each OSD can check for lnb array overflow.
the patch isn't final - there will be proper
implementation in osd-zfs and a new test.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I43683c84e48006b4075f9a8b3e87cdfeae28c02b
Reviewed-on: https://review.whamcloud.com/35801
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-11367 som: integrate LSOM with lfs find 67/35167/22
Qian Yingjin [Thu, 1 Nov 2018 08:49:53 +0000 (16:49 +0800)]
LU-11367 som: integrate LSOM with lfs find

The patch integrates LSOM functionality with lfs find so that it
is possible to use LSOM functionality directly on the client. The
MDS fills in the mbo_size and mbo_blocks fields from the LSOM
xattr, if the actual size/blocks are not available, and then set
new OBD_MD_FLLSIZE and OBD_MD_FLLBLOCKS flags in the reply so that
the client knows these fields are valid.

The lfs find command adds "-l|--lazy" option to allow the use of
LSOM data from the MDS.

Add a new version of ioctl(LL_IOC_MDC_GETINFO) call that also returns
valid flags from the MDS RPC to userspace in struct lov_user_mds_data
so that it is possible to determine whether the size and blocks are
returned by the call.  The old LL_IOC_MDC_GETINFO ioctl number is
renamed to LL_IOC_MDC_GETINFO_OLD and is binary compatible, but
newly-compiled applications will use the new struct lov_user_mds_data.

New llapi interfaces llapi_get_lum_file(), llapi_get_lum_dir(),
llapi_get_lum_file_fd(), llapi_get_lum_dir_fd() are added to fetch
valid stat() attributes and LOV info to the user.

Signed-off-by: Qian Yingjin <qian@ddn.com>
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I21dfae7c2633dead5d83b438ec340fea4d3ebbe5
Reviewed-on: https://review.whamcloud.com/35167
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
2 months agoLU-12621 o2iblnd: cache max_qp_wr 73/36073/5
Amir Shehata [Fri, 6 Sep 2019 01:15:10 +0000 (18:15 -0700)]
LU-12621 o2iblnd: cache max_qp_wr

When creating the device the maximum number of work requests per qp
which can be allocated is already known. Cache that internally,
and when creating the qp make sure the qp's max_send_wr does not
exceed that max. If it does then cap max_send_wr to max_qp_wr.
Recalculate the connection's queue depth based on the max_qp_wr.

Test-Parameter: nettype=o2ib
Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: I6d9a642d03633264f5f14445a051dd14515709c1
Reviewed-on: https://review.whamcloud.com/36073
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Doug Oucharek <dougso@me.com>
Reviewed-by: Olaf Weber <olaf.weber@hpe.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12751 tests: add missing error() 59/36159/2
Alex Zhuravlev [Wed, 11 Sep 2019 14:32:21 +0000 (17:32 +0300)]
LU-12751 tests: add missing error()

nothing else I can say

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I040771e57ec6f6c6bfbde5a21358c6747f4f20dc
Reviewed-on: https://review.whamcloud.com/36159
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
2 months agoLU-12753 tests: wait for mds2 recovery in sanity 278 67/36167/2
Andreas Dilger [Wed, 11 Sep 2019 21:47:03 +0000 (15:47 -0600)]
LU-12753 tests: wait for mds2 recovery in sanity 278

At the end of sanity.sh test_278() the mds2 facet is remounted.
However, test_300a() which runs immediately afterward expects
all MDTs to be available, and occasionally fails creating a
fully-striped directory with:

    stripe_count is 1, expect 2

because the mds2 facet has not finished mounting at mkdir time.

Add a wait at the end of test_278() to ensure that mds2 has finished
mounting so that it doesn't affect any later tests.

Test-Parameters: trivial envdefinitions=ONLY=270-300 mdscount=1 mdtcount=2
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I9421101be6394288ccd4ce5777d118523a3ebbe5
Reviewed-on: https://review.whamcloud.com/36167
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12734 misc: add bash completion for lctl set/get_param 05/36105/6
Dominique Martinet [Mon, 9 Sep 2019 14:46:45 +0000 (16:46 +0200)]
LU-12734 misc: add bash completion for lctl set/get_param

Add some start of bash completion for lctl, mainly set_param and
get_param, and modify build system to install it.

Test-Parameters: trivial
Signed-off-by: Dominique Martinet <dominique.martinet@cea.fr>
Change-Id: I16d2698e782702375c7fa3edf3bfde2e3b197297
Reviewed-on: https://review.whamcloud.com/36105
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Quentin Bouget <quentin.bouget@cea.fr>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12707 obdecho: avoid panic with partially object init 50/35950/5
Alexey Lyashkov [Wed, 28 Aug 2019 15:06:43 +0000 (18:06 +0300)]
LU-12707 obdecho: avoid panic with partially object init

in some cases (like ENOMEM) init function can't called, so
any init related code should placed in the object delete handler,
not in the object free.

Signed-off-by: Alexey Lyashkov <c17817@cray.com>
Change-Id: I1fca56423de9a045aac2c495fbc45069c3bbc97c
Reviewed-on: https://review.whamcloud.com/35950
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12569 o2iblnd: Make credits hiw connection aware 78/35578/8
Patrick Farrell [Sun, 21 Jul 2019 17:06:37 +0000 (13:06 -0400)]
LU-12569 o2iblnd: Make credits hiw connection aware

The IBLND_CREDITS_HIGHWATER mark check currently looks only
at the global peer credits tunable, ignoring the connection
specific queue depth when determining the threshold at
which to send a NOOP message to return credits.

This is incorrect because while connection queue depth
defaults to the same as peer credits, it can be less than
that global value for specific connections.

So we must check for this case when setting the threshold.

Test-Parameter: nettype=o2ib
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Ie028ae11cdbd0f75a38b265b7ab5830f92f08d90
Reviewed-on: https://review.whamcloud.com/35578
Reviewed-by: Chris Horn <hornc@cray.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12514 utils: add "lustre_tgt" filesystem type 65/36165/3
Andreas Dilger [Wed, 24 Jul 2019 19:36:52 +0000 (13:36 -0600)]
LU-12514 utils: add "lustre_tgt" filesystem type

Add a "lustre_tgt" filesystem type separate from the "lustre"
filesystem type for forward compatibility to allow the client and
server mount code to be disentangled.  This does not actually make
any changes to the mount code itself, but allows testing interop
and fallback after an upgrade if the new filesystem type is used.

Remove the check for the "lustre_lite" filesystem type from
llapi_is_lustre_mnttype() since that was unused since Lustre 1.4,
and replace it with a check for filesystem type "lustre_tgt".

Test-Parameters: trivial testlist=conf-sanity
Change-Id: If635834dfff502e6ecf8a85a2665de09efd52372
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/36165
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Neil Brown <neilb@suse.de>
Tested-by: Maloo <maloo@whamcloud.com>
2 months agoLU-11956 mdd: do not reset original lu_buf.lb_len 33/35333/3
Li Dongyang [Thu, 27 Jun 2019 03:25:45 +0000 (13:25 +1000)]
LU-11956 mdd: do not reset original lu_buf.lb_len

In mdd_iterate_xattrs(), we are resetting the xbuf.lb_len
to a smaller value returned by linkea_overflow_shrink().

If that's the last xattr we gonna process, we could deduct
less than originally allocated size from obd_memory stats,
failing the memleak check later.

Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Change-Id: I6175a91c61ceb0e37ab889d0cfd904f4993ab5cc
Reviewed-on: https://review.whamcloud.com/35333
Reviewed-by: Li Xi <lixi@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12745 build: Account for optional SPL for ZFS 0.8+ 61/36161/2
Nathaniel Clark [Wed, 11 Sep 2019 15:10:58 +0000 (11:10 -0400)]
LU-12745 build: Account for optional SPL for ZFS 0.8+

With ZFS 0.8.0 and later, SPL is not longer present.
Some zfs packages provide vestigial spl package contents, but zfs-dkms
does not.  This makes testing SPL directories optional depending on
version of ZFS, this also accounts for the new location of the spl
include directory under the zfs include directory.

Test-Parameters: trivial
Signed-off-by: Nathaniel Clark <nclark@whamcloud.com>
Change-Id: I8afcff079f25543a3c86df0c404146a859b226aa
Reviewed-on: https://review.whamcloud.com/36161
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-11485 lod: disallow setting the last non-stale mirror as stale 41/36141/6
Jian Yu [Mon, 16 Sep 2019 05:56:33 +0000 (22:56 -0700)]
LU-11485 lod: disallow setting the last non-stale mirror as stale

"lfs setstripe" allows setting stale flag on the last
non-stale mirror of a file, which makes the file have
no valid component to read and return IO error.

This patch fixes the above issue by disallowing that.
It also disallows "lfs mirror split" to destroy the
last non-stale mirror of a file.

Change-Id: I6934cfe0190cd1ea83de1cf28ddf840b9f96193a
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/36141
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
2 months agoLU-6142 lustre: introduce CONFIG_LUSTRE_FS_POSIX_ACL 85/36085/5
NeilBrown [Mon, 9 Sep 2019 16:45:51 +0000 (12:45 -0400)]
LU-6142 lustre: introduce CONFIG_LUSTRE_FS_POSIX_ACL

Lustre (or any file system) should not conditionally
compile code based on CONFIG_FS_POSIX_ACL.  This config
option enables library support.
A file system can define it's own config option,
which then selects CONFIG_FS_POSIX_ACL (if needed).  It should
act on its own option, not the library one.

This patch makes that change.

While lustre is out-of-tree, it cannot select CONFIG_FS_POSIX_ACL
and must work with the configruation of the kernel it is being built
against. So the new CONFIG_LUSTRE_FS_POSIX_ACL can only be selected
if the based kernel has CONFIG_FS_POSIX_ACL selected. With that
restriction this change becomes little more than an indirection.
However it means that the out-of-tree code can be closer to the
(eventual) in-tree code.

Test-Parameters: trivial

Signed-off-by: NeilBrown <neilb@suse.com>
Change-Id: I3ba283f295d1c7217e7ff9917573be0fe92bb2ca
Reviewed-on: https://review.whamcloud.com/36085
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12385 lnet: update opa defaults 72/36072/5
Amir Shehata [Thu, 5 Sep 2019 21:28:48 +0000 (14:28 -0700)]
LU-12385 lnet: update opa defaults

Testing reveals no significant performance improvements
when using peer_credits > 32. Adjusted the default
peer_credits, peer_credits_hiw and concurrent_sends
to take that into account.

This has the advantage of avoiding an issue observed
on multiple opa sites where the qp can not be created because
of large initial queue_depth. The queue depth is then
reduced gradually until the qp creation succeeds.

Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: I6036ec1da7063e30b567446e5db89040f21bc701
Reviewed-on: https://review.whamcloud.com/36072
Reviewed-by: Chris Horn <hornc@cray.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-11542 import: Fix missing spin_unlock() 99/35999/4
Mr NeilBrown [Wed, 11 Sep 2019 18:26:47 +0000 (14:26 -0400)]
LU-11542 import: Fix missing spin_unlock()

A recent patch moved the spin_unlock() down into
each branch of an 'if', but missed the final 'else'.
Add the spin_unlock in the else.

Fixes: 29904135df67 ("LU-11542 import: fix race between imp_state & imp_invalid")
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I6ee399050aad0fe9df9c0e3ddf8ec0be8eae1641
Reviewed-on: https://review.whamcloud.com/35999
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-10467 ptlrpc: fix indents in ptlrpc_recover_import() 76/35976/9
Mr NeilBrown [Thu, 29 Aug 2019 00:21:56 +0000 (10:21 +1000)]
LU-10467 ptlrpc: fix indents in ptlrpc_recover_import()

Next patch will make code changes in ptlrpc_recover_import(),
so fix up indenting to use TABs first.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I1d839e3087a5d12473b0428509455dbbb4b5d9e8
Reviewed-on: https://review.whamcloud.com/35976
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-10467 lustre: use TABs for indents in a few places. 70/35970/8
Mr NeilBrown [Thu, 29 Aug 2019 00:05:04 +0000 (10:05 +1000)]
LU-10467 lustre: use TABs for indents in a few places.

Each of the functions changed here will have code changes
in the next patch, so fix up all the indentation first.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: Ib10e999a8c58eb96d3312878be91b465da3a2df8
Reviewed-on: https://review.whamcloud.com/35970
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-10467 fid: style cleanups in seq_client_alloc_meta() 63/35963/4
Mr NeilBrown [Fri, 23 Aug 2019 06:10:32 +0000 (16:10 +1000)]
LU-10467 fid: style cleanups in seq_client_alloc_meta()

Prior to code changes in seq_client_alloc_meta, make indents
consistently tabs, and change "env != NULL" to "env"

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: If72d737f843a5f6e56ef82aabb7a779b358f4382
Reviewed-on: https://review.whamcloud.com/35963
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-9341 lod: Add special O_APPEND striping 17/35617/8
Patrick Farrell [Wed, 28 Aug 2019 16:54:37 +0000 (12:54 -0400)]
LU-9341 lod: Add special O_APPEND striping

Files opened with O_APPEND are almost always log files,
which generally stay small and do not benefit from being
striped widely.  Additionally, PFL files accessed with
O_APPEND are fully instantiated, meaning that because the
files usually stay small, these objects usually wasted.

This patch adds special striping for files created with
O_APPEND.  This is controlled on the MDS by two new proc
variables:
mdd_append_stripe_count
mdd_append_pool

If the stripe count is set to 0 and the pool is not set,
this functionality is disabled and files created with
O_APPEND will be striped like any other file.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I433d1b8c80488a851b8eb26c78cf5519a6cd75bf
Reviewed-on: https://review.whamcloud.com/35617
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-11213 lod: share object alloc QoS code with LMV 19/35219/7
Lai Siyao [Sat, 27 Apr 2019 18:33:06 +0000 (02:33 +0800)]
LU-11213 lod: share object alloc QoS code with LMV

Move object alloc QoS code to obdclass, so that LMV and LOD
can share the same code.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I451a43fa9a254ec709b2acd43538fdcba0be4a88
Reviewed-on: https://review.whamcloud.com/35219
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-11213 lmv: use lu_tgt_descs to manage tgts 18/35218/10
Lai Siyao [Tue, 23 Apr 2019 18:46:05 +0000 (02:46 +0800)]
LU-11213 lmv: use lu_tgt_descs to manage tgts

Like LOD, use lu_tgt_descs to manage tgts, so that they can
share tgt management code.

TODO: use the same tgt management code for LOV/LFSCK.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I843e85e0f8b05e21056567cebfea0a8fff1d4ef8
Reviewed-on: https://review.whamcloud.com/35218
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-11670 osc: glimpse - search for active lock 60/33660/31
Patrick Farrell [Mon, 9 Sep 2019 15:56:07 +0000 (11:56 -0400)]
LU-11670 osc: glimpse - search for active lock

When there are lock-ahead write locks on a file, the server
sends one glimpse AST RPC to each client having such (it
may have many) locks. This callback is sent to the lock
having the highest offset.

Client's glimpse callback goes up to the clio layers and
gets the global (not lock-specific) view of size.  The clio
layers are connected to the extent lock through the
l_ast_data (which points to the OSC object).

Speculative locks (AGL, lockahead) do not have l_ast_data
initialised until an IO happens under the lock. Thus, some
speculative locks may not have l_ast_data initialized.

It is possible for the client to do a write using one lock
(changing file size), but for the glimpse AST to be sent to
another lock without l_ast_data initialized.  Currently, a
lock with no l_ast_data set returns ELDLM_NO_LOCK_DATA to
the server.  In this case, this means we do not return the
updated size.

The solution is to search the granted lock tree for any lock
with initialized l_ast_data (it points to the OSC object
which is the same for all the extent locks) and to reach the
clio layers for the size through this lock instead.

cray-bug-id: LUS-6747
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I6c60f4133154a3d6652315f155af24bbc5752dd2
Reviewed-on: https://review.whamcloud.com/33660
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12634 lnet: for_ifa removed. Use in_dev_for_each_ifa_rtnl 44/35744/2
Shaun Tancheff [Fri, 9 Aug 2019 04:02:55 +0000 (23:02 -0500)]
LU-12634 lnet: for_ifa removed. Use in_dev_for_each_ifa_rtnl

Linux 5.3 removed for_ifa and replaced it with an _rntl and _rcu
versions for use with their respective locking primitives.

kernel-commit: ef11db3310e272d3d8dbe8739e0770820dd20e52

Test-Parameters: trivial
Cray-bug-id: LUS-7689
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Change-Id: Iea07222b9abb3f9c219d28fe2c660d9eaf21af80
Reviewed-on: https://review.whamcloud.com/35744
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12559 ptlrpc: Hold imp lock for idle reconnect 30/35530/4
Patrick Farrell [Tue, 16 Jul 2019 19:26:43 +0000 (15:26 -0400)]
LU-12559 ptlrpc: Hold imp lock for idle reconnect

Idle reconnect sets import state to IMP_NEW, then releases
the import lock before calling ptlrpc_connect_import.  This
creates a gap where an import in IMP_NEW state is exposed,
which can cause new requests to fail with EIO.

Hold the lock across the call so as not to expose imports
in this state.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I9f8509d11c4d5a8917a313349534d98b964cd588
Reviewed-on: https://review.whamcloud.com/35530
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12269 kernel: RHEL 8.0 server support 65/35665/13
Jian Yu [Tue, 3 Sep 2019 17:11:28 +0000 (10:11 -0700)]
LU-12269 kernel: RHEL 8.0 server support

This patch makes changes to support RHEL 8.0 release
for Lustre server.

The ldiskfs patch series was added by commit
5f7bd9a806d5ba5b0e70ae1b299a9b9a87fcc0a6.

Test-Parameters: trivial \
envdefinitions=SANITY_EXCEPT="421a 817" \
clientdistro=el8 serverdistro=el8 \
testlist=sanity

Change-Id: I6380c6cb6280e49bd1b193a639f1c5cd6f4e7ef1
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/35665
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-11729 obdclass: align to T10 sector size when generating guard 43/34043/11
Andreas Dilger [Wed, 21 Aug 2019 14:03:11 +0000 (10:03 -0400)]
LU-11729 obdclass: align to T10 sector size when generating guard

Otherwise the client and server would come up with
different checksum when the page size is different.

Improve test_810 to verify all available checksum types.

Test-Parameters: trivial envdefinitions=ONLY=810 testlist=sanity,sanity,sanity
Test-Parameters: clientarch=aarch64 envdefinitions=ONLY=810 testlist=sanity,sanity
Test-Parameters: clientarch=ppc64 envdefinitions=ONLY=810 testlist=sanity,sanity
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Change-Id: I24117aebb277d4ddcb7787b715587e33023ebbe5
Reviewed-on: https://review.whamcloud.com/34043
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
2 months agoLU-12579 tests: allow some margin in runtests 11/36011/3
Andreas Dilger [Fri, 30 Aug 2019 23:23:50 +0000 (17:23 -0600)]
LU-12579 tests: allow some margin in runtests

Allow some margin in the space used by runtests for internal
log files for Lustre and the underlying filesystem.

Test-Parameters: trivial testlist=runtests,runtests,runtests
Test-Parameters: mdtcount=4 mdscount=2 testlist=runtests,runtests,runtests
Test-Parameters: fstype=zfs testlist=runtests,runtests,runtests
Test-Parameters: fstype=zfs mdtcount=4 mdscount=2 testlist=runtests,runtests
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I34b47a8436c5718be311698a3f6e6d7af7ea45ad
Reviewed-on: https://review.whamcloud.com/36011
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
2 months agoLU-1957 tests: remove sanity test 180 from ALWAYS_EXCEPT 30/35930/2
Andreas Dilger [Mon, 26 Aug 2019 23:00:44 +0000 (17:00 -0600)]
LU-1957 tests: remove sanity test 180 from ALWAYS_EXCEPT

Remove test_180 from sanity ALWAYS_EXCEPT, since it should have been
fixed by landing LU-2803.

Fixes: e99f38594d2b ("LU-2803 osd: osd-zfs to handle echo sequence (2) properly")
Test-Parameters: trivial testlist=sanity fstype=zfs
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I7601164865baba8fe2db3ce7bb33fd4c81eb0291
Reviewed-on: https://review.whamcloud.com/35930
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-11607 tests: add a default definition for SINGLEMDS 93/36093/2
Andreas Dilger [Thu, 25 Jul 2019 18:35:20 +0000 (12:35 -0600)]
LU-11607 tests: add a default definition for SINGLEMDS

Move the $SINGLEMDS definition from the test config file into
get_lustre_env() so that it is always set.  It should use facet
"mds1", anything else probably doesn't work.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Iecbd6fe7d5102da7c2c14741c85986cf73054035
Reviewed-on: https://review.whamcloud.com/36093
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12686 lnet: change ln_mt_waitq to a completion. 74/35874/2
Mr NeilBrown [Thu, 22 Aug 2019 04:58:12 +0000 (14:58 +1000)]
LU-12686 lnet: change ln_mt_waitq to a completion.

ln_mt_waitq is only waited on by a call to
  wait_event_interruptible_timeout(..., false, timeout);

As 'false' is never 'true', this will always wait for the full
timeout to expire.  So the waitq is effectively pointless.

To acheive the apparent intent of the waitq, change it to a
completion.  The completion adds a 'done' flag to a waitq so we can
wait until a timeout or until a wakeup is requested.

With this, a longer timeout would could be used, but that is left to
a later patch.

Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I10df4b33ee59f579b965e44c022175b8ab007626
Reviewed-on: https://review.whamcloud.com/35874
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <hornc@cray.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-11626 mdc: hold obd while processing changelog 84/35784/3
Hongchao Zhang [Tue, 20 Aug 2019 09:58:25 +0000 (05:58 -0400)]
LU-11626 mdc: hold obd while processing changelog

During read/write changelog, the corresponding obd_device should
be held to protect it from being released by umount.

Change-Id: Ib5b528f178edcf73425587ea60335df640c1696d
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/35784
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12650 lib: fix strings comparison during mount searching 55/35755/2
Artem Blagodarenko [Fri, 9 Aug 2019 19:19:29 +0000 (22:19 +0300)]
LU-12650 lib: fix strings comparison during mount searching

get_root_path() returns path to "lustre" mount instead "lustre1"
because last symbol is not taking in account during comparison.
This bug has influence to get_root_path() users.

For example, fid2path use get_root_path().

lfs path2fid /mnt/lustre2/foodir3
[0x200000401:0x1:0x0]

lfs fid2path lustre2 [0x200000401:0x1:0x0]
lfs fid2path: cannot find '[0x200000401:0x1:0x0]': No such file or
directory

umount /mnt/lustre
lfs fid2path lustre2 [0x200000401:0x1:0x0]
foodir3

This fix adds strings length comparison.

Signed-off-by: Artem Blagodarenko <c17828@cray.com>
Cray-bug-id: LUS-7693
Change-Id: I3275d2182486d25389814f4c25b3f2a54ec29469
Reviewed-on: https://review.whamcloud.com/35755
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexander Zarochentsev <c17826@cray.com>
2 months agoLU-12634 llite: lm_compare_owner removed 47/35747/2
Shaun Tancheff [Fri, 9 Aug 2019 03:52:06 +0000 (22:52 -0500)]
LU-12634 llite: lm_compare_owner removed

Linux 5.3 removed lm_compare_owner

kernel-commit: f85d93385e9fe6886a751f647f6812a89bf6bee3

Test-Parameters: trivial
Cray-bug-id: LUS-7689
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Change-Id: I3aa33df0fcd2443ff8ac5dc8b1c893de92931352
Reviewed-on: https://review.whamcloud.com/35747
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12634 osd-ldiskfs: bi_phys_segments removed from struct bio 46/35746/3
Shaun Tancheff [Fri, 9 Aug 2019 05:51:41 +0000 (00:51 -0500)]
LU-12634 osd-ldiskfs: bi_phys_segments removed from struct bio

Linux 5.3 removed the bi_phys_segments field in struct bio
Use bio_segments() if bi_phys_segments is not available

kernel-commit: 14ccb66b3f585b2bc21e7256c96090abed5a512c

Test-Parameters: trivial
Cray-bug-id: LUS-7689
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Change-Id: I87f2b1dc34f47a4eaa5ab5d6e55a903afc317011
Reviewed-on: https://review.whamcloud.com/35746
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-11607 tests: replace version/fstype calls in recovery-small 23/35723/3
James Nunez [Wed, 7 Aug 2019 21:43:16 +0000 (15:43 -0600)]
LU-11607 tests: replace version/fstype calls in recovery-small

The routine get_lustre_env() is available to all Lustre test
suites and sets an environment variable for the file system
type for MDS1 and OST1 and sets a variable for the Lustre
version of servers.

In recovery-small, replace the calls to facet_fstype() and
lustre_version_code() for all server types with definitions
in get_lustre_env().

While doing this, replace facet_fstype $SINGLEMDS and
lustre_version_code $SINGLEMDS with facet_fstype mds1
and lustre_version_code mds1, respectively.

Clean up around any modifications by removing calls to
return after skip() or skip_env().

Test-Parameters: trivial testlist=recovery-small
Test-Parameters: fstype=zfs testlist=recovery-small
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I8a99324d3c8694b13064dd3cd6c589329b892bec
Reviewed-on: https://review.whamcloud.com/35723
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
2 months agoLU-12581 osc: prevent use after free 01/35601/2
Bobi Jam [Wed, 24 Jul 2019 13:24:01 +0000 (21:24 +0800)]
LU-12581 osc: prevent use after free

Clear aa_oa after it's been freed to prevent use after free.

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Idf122aa53fe5b13c07337745e5a26763e8712be2
Reviewed-on: https://review.whamcloud.com/35601
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12137 osd-ldiskfs: create locked and unlocked versions of osd lookup code 94/35594/3
James Simmons [Mon, 19 Aug 2019 13:17:29 +0000 (09:17 -0400)]
LU-12137 osd-ldiskfs: create locked and unlocked versions of osd lookup code

The inode_lock is expected to be taken when lookup_one_len() is
called. Rename osd_ios_lookup_one_len() to osd_lookup_one_len()
since it now doesn't take the inode_lock manually. Instead it
expects the called to have already called inode_lock(). The
osd_lookup_one_len_unlocked() function will always take the
inode_lock itself. Currently osd-ldiskfs always uses
osd_lookup_one_len_unlocked().

Change-Id: Ie406db5e24f983e52ea8a89bca66546ab9d25148
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/35594
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12137 osd-ldiskfs: trace the obd device used for osd lookup 82/35582/3
James Simmons [Mon, 19 Aug 2019 13:14:55 +0000 (09:14 -0400)]
LU-12137 osd-ldiskfs: trace the obd device used for osd lookup

If a osd lookup of a dentry fails a debug message is logged but
no information is given for which OSD device this happened on.
Add in the osd device information for debugging purposes.

Change-Id: If7719987682c4b9f8219c08431337306bd91ab67
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/35582
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12137 osd-ldiskfs: refactor osd_ios_ROOT_scan 03/35803/3
James Simmons [Thu, 15 Aug 2019 20:40:06 +0000 (16:40 -0400)]
LU-12137 osd-ldiskfs: refactor osd_ios_ROOT_scan

With the change of osd_ios_scan_one() to handle a NULL
dchild->d_inode we can migrate the scrub code after calling
osd_ios_scan_one(). This can help reduce the code indentation
as well as make the code clearer.

Change-Id: Ieb59aa85c8199a1f1a1bee5cbe633e52b8e92a9a
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/35803
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12400 ptlrpc: Sun RPC changes for RCU locking 99/35499/3
Shaun Tancheff [Sun, 14 Jul 2019 11:38:28 +0000 (06:38 -0500)]
LU-12400 ptlrpc: Sun RPC changes for RCU locking

In kernel 4.20 SUNRPC cache_detail->hash_lock changed to spinlock_t

  Now that the reader functions are all RCU protected, use a regular
  spinlock rather than a reader/writer lock.

Linux-commit: 1863d77f15da0addcd293a1719fa5d3ef8cde3ca

Test-Parameters: trivial
Cray-bug-id: LUS-7600
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Change-Id: If0df38337d5a2bb0ac4b8cb645dbe89f65e0f352
Reviewed-on: https://review.whamcloud.com/35499
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12518 readahead: convert stride page index to byte 29/35829/5
Wang Shilong [Fri, 16 Aug 2019 06:42:32 +0000 (14:42 +0800)]
LU-12518 readahead: convert stride page index to byte

This is a prepared patch to support unaligned stride readahead.
Some detection variables are converted to byte unit to be aware
of possible unaligned stride read.

Since we still need read pages by page index, so those variables
are still kept as page unit. to make things more clear, fix them
to use pgoff_t rather than unsigned long.

Change-Id: Ic0d0b1b2f11cb6ef5753814a5b8fc6d34afb25ea
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/35829
Reviewed-by: Li Xi <lixi@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12535 lov: Move page index to top level 70/35470/9
Patrick Farrell [Thu, 8 Aug 2019 17:14:51 +0000 (13:14 -0400)]
LU-12535 lov: Move page index to top level

When doing readahead, we see an amazing amount of time
(~5-8%) just looking up the page index from the lov layer.

In particular, this is more than half the time spent
submitting pages:
         - 14.14% cl_io_submit_rw
            - 13.40% lov_io_submit
               - 8.24% lov_page_index

This requires several indirections, all of which can be
avoided by moving this up to the cl_page struct.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I99bd7eb4d6556ac89c1aa9aeb4b7afc99774b212
Reviewed-on: https://review.whamcloud.com/35470
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12533 llite: Improve readahead RPC issuance 58/35458/10
Patrick Farrell [Thu, 8 Aug 2019 17:13:29 +0000 (13:13 -0400)]
LU-12533 llite: Improve readahead RPC issuance

lov_io_submit receives a range of pages, then adds pages in
to a batch until it hits a page which is not in the stripe
associated with this lov object.  This means that if a
readahead page range hits the same stripe more than once,
we will issue multiple I/Os, even if the pages would fit in
one RPC.

This is unnecessary - Just submit all these pages at once.

mpirun -n 2 $IOR -s 2000 -t 47K -b 47K -k -r -E -o $FILE

Without patch:
osc.lustre-OST0001-osc-ffff8fe82c952000.rpc_stats=

                        read                    write
pages per rpc         rpcs   % cum % |       rpcs   % cum %
1:                     118  56  56   |          0   0   0
2:                       0   0  56   |          0   0   0
4:                       0   0  56   |          0   0   0
8:                       0   0  56   |          0   0   0
16:                      5   2  58   |          0   0   0
32:                      0   0  58   |          0   0   0
64:                      0   0  58   |          0   0   0
128:                    21  10  68   |          0   0   0
256:                    25  11  80   |          0   0   0
512:                    10   4  85   |          0   0   0
1024:                   31  14 100   |          0   0   0

osc.lustre-OST0002-osc-ffff8fe82c952000.rpc_stats=
                        read                    write
pages per rpc         rpcs   % cum % |       rpcs   % cum %
1:                       5   6   6   |          0   0   0
2:                       0   0   6   |          0   0   0
4:                       0   0   6   |          0   0   0
8:                       0   0   6   |          0   0   0
16:                      0   0   6   |          0   0   0
32:                      0   0   6   |          0   0   0
64:                      0   0   6   |          0   0   0
128:                    19  23  29   |          0   0   0
256:                    19  23  52   |          0   0   0
512:                     5   6  58   |          0   0   0
1024:                   34  41 100   |          0   0   0

With patch:
osc.lustre-OST0001-osc-ffff8fe7a7227800.rpc_stats=
                        read                    write
pages per rpc         rpcs   % cum % |       rpcs   % cum %
1:                      12  17  17   |          0   0   0
2:                       0   0  17   |          0   0   0
4:                       0   0  17   |          0   0   0
8:                       0   0  17   |          0   0   0
16:                      5   7  24   |          0   0   0
32:                      0   0  24   |          0   0   0
64:                      5   7  31   |          0   0   0
128:                     6   8  40   |          0   0   0
256:                     1   1  42   |          0   0   0
512:                     2   2  44   |          0   0   0
1024:                   38  55 100   |          0   0   0

osc.lustre-OST0002-osc-ffff8fe7a7227800.rpc_stats=
                        read                    write
pages per rpc         rpcs   % cum % |       rpcs   % cum %
1:                       0   0   0   |          0   0   0
2:                       0   0   0   |          0   0   0
4:                       0   0   0   |          0   0   0
8:                       0   0   0   |          0   0   0
16:                      0   0   0   |          0   0   0
32:                      0   0   0   |          0   0   0
64:                      4   7   7   |          0   0   0
128:                     7  13  21   |          0   0   0
256:                     0   0  21   |          0   0   0
512:                     3   5  26   |          0   0   0
1024:                   38  73 100   |          0   0   0

Note the much larger # of smaller RPC issued without the patch.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Ic10c138628c269afe57fbc57ec8c91ce990717f9
Reviewed-on: https://review.whamcloud.com/35458
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12043 llite: extend readahead locks for striped file 38/35438/7
Wang Shilong [Thu, 8 Aug 2019 17:07:21 +0000 (13:07 -0400)]
LU-12043 llite: extend readahead locks for striped file

Currently cl_io_read_ahead() can not return locks
that cross stripe boundary at one time, thus readahead
will stop because of this reason.

This is really bad, as we will stop readahead every
time we hit stripe boundary, for example default stripe
size is 1M, this could hurt performances very much
especially with async readahead introduced.

So try to use existed locks aggressivly if there is no
lock contention, otherwise lock should be not
less than requested extent.

Change-Id: I8b2dcd0e80138ea530272cab6a665981aa00cca8
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/35438
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-11780 tests: add server version check to replay-single 90/34590/8
James Nunez [Fri, 28 Jun 2019 15:34:01 +0000 (09:34 -0600)]
LU-11780 tests: add server version check to replay-single

replay-single test 132a was added to Lustre 2.12.0. Thus,
this test should be skipped for all servers with version less
than 2.12.0.

Fixes: e5abcf83c057 (LU-11158 mdt: grow lvb buffer to hold layout)
Test-Parameters:trivial envdefinitions=ONLY=132a serverjob=lustre-b2_10 serverbuildno=170 testlist=replay-single
Test-Parameters:envdefinitions=ONLY=132a testlist=replay-single
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I8fb69c4ab6e08a4be7619b70c0a54d49ebfebf64
Reviewed-on: https://review.whamcloud.com/34590
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12719 obdclass: serialize lwp list access 03/36003/3
Lai Siyao [Sun, 11 Aug 2019 06:34:04 +0000 (14:34 +0800)]
LU-12719 obdclass: serialize lwp list access

lustre_sb_info.lsi_lwp_list should be acessed with lock, and
some place may sleep, change lsi_lwp_lock from spinlock to mutex.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Ifc3622eb28cd6cf49661b14fc10e98aa689a58dc
Reviewed-on: https://review.whamcloud.com/36003
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-6142 mdt: remove unused field mti_wait_info. 21/35921/5
Mr NeilBrown [Mon, 26 Aug 2019 06:10:16 +0000 (16:10 +1000)]
LU-6142 mdt: remove unused field mti_wait_info.

This field is neither set nor accessed.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I7b8ccd97ad1228bbd4e8577a6d6003a9888f9282
Reviewed-on: https://review.whamcloud.com/35921
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-6142 mtd: use tabs to indent struct mdt_thread_info 20/35920/4
Mr NeilBrown [Mon, 26 Aug 2019 06:09:04 +0000 (16:09 +1000)]
LU-6142 mtd: use tabs to indent struct mdt_thread_info

Prior to a (small) code change, change indenting of
struct mtd_thread_info to consistently use TABs.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: Ic0f29a8b84640862294610e37a941b77292446e7
Reviewed-on: https://review.whamcloud.com/35920
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
2 months agoLU-6142 socklnd: remove ksnp_sharecount 92/35892/4
Mr NeilBrown [Thu, 22 Aug 2019 05:54:55 +0000 (15:54 +1000)]
LU-6142 socklnd: remove ksnp_sharecount

This field is never set, though its value is printed.
Remove it.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: Ie1d8987d8c4981b6ac9204c27124a536bef969b5
Reviewed-on: https://review.whamcloud.com/35892
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-6142 socklnd: Fix some white-space issues ... 19/35919/3
Mr NeilBrown [Fri, 23 Aug 2019 02:39:43 +0000 (12:39 +1000)]
LU-6142 socklnd: Fix some white-space issues ...

... in ksocknal_debug_peerhash()

Prior to a code change, fix up the white-space here.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I1d492bbb8be608a297d641111eb6238a4fd2f38e
Reviewed-on: https://review.whamcloud.com/35919
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Tested-by: Maloo <maloo@whamcloud.com>
2 months agoLU-6142 o2iblnd: remove some unused fields. 91/35891/3
Mr NeilBrown [Thu, 22 Aug 2019 05:50:13 +0000 (15:50 +1000)]
LU-6142 o2iblnd: remove some unused fields.

Fields kib_min_reconnect_interval kib_max_reconnect_interval kib_ntx
are never used or set.

ibh_mr_shift is set but never used;
rx_status is used (in a debug message) but never set.

Remove them all.

We could possibly remove ibh_mr_size too. It is only used
for an error message.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: If1ff73c15f8e712be4e3d77e9572a4d4e741c75d
Reviewed-on: https://review.whamcloud.com/35891
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-6142 osc: remove oe_next_page 90/35890/3
Mr NeilBrown [Thu, 22 Aug 2019 05:41:39 +0000 (15:41 +1000)]
LU-6142 osc: remove oe_next_page

As the comment says, this field is unused.  So remove it.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: Ibf4ccc9c09c4efcf5c768e71e0fd448958a943f3
Reviewed-on: https://review.whamcloud.com/35890
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
2 months agoLU-6142 osc: remove oti_descr oti_handle oti_plist 89/35889/3
Mr NeilBrown [Thu, 22 Aug 2019 05:40:26 +0000 (15:40 +1000)]
LU-6142 osc: remove oti_descr oti_handle oti_plist

These three fields in 'struct osc_thread_info' are
unused, so remove them.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I8642ebd692f44eca9d9ddad3b8184cabb27d27b3
Reviewed-on: https://review.whamcloud.com/35889
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
2 months agoLU-6142 llite: remove sub_reenter field. 88/35888/3
Mr NeilBrown [Thu, 22 Aug 2019 05:37:47 +0000 (15:37 +1000)]
LU-6142 llite: remove sub_reenter field.

This field is never set or accessed, so
remove it.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I34e8d59d068e18ff65660924b5b984f0ab45510f
Reviewed-on: https://review.whamcloud.com/35888
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
2 months agoLU-6142 llite: remove ft_mtime field 87/35887/3
Mr NeilBrown [Thu, 22 Aug 2019 05:36:09 +0000 (15:36 +1000)]
LU-6142 llite: remove ft_mtime field

This field is set but never accessed, so remove it.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: If6f96e90309235a61c833e262e50b03cf36132b0
Reviewed-on: https://review.whamcloud.com/35887
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
2 months agoLU-6142 llite: remove lti_iter field 86/35886/3
Mr NeilBrown [Thu, 22 Aug 2019 05:34:52 +0000 (15:34 +1000)]
LU-6142 llite: remove lti_iter field

This field is never used, so remove it.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I26489ce4200969b746b2214eaa5c07bdaddf2baf
Reviewed-on: https://review.whamcloud.com/35886
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
2 months agoLU-6142 llite: remove ll_umounting field 85/35885/3
Mr NeilBrown [Thu, 22 Aug 2019 05:33:59 +0000 (15:33 +1000)]
LU-6142 llite: remove ll_umounting field

This field is set but never accessed, so remove it.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I27d8388b73fea3821ff2ecfd85b28c3cdd13193f
Reviewed-on: https://review.whamcloud.com/35885
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
2 months agoLU-6142 llite: remove lli_readdir_mutex 84/35884/3
Mr NeilBrown [Thu, 22 Aug 2019 05:33:40 +0000 (15:33 +1000)]
LU-6142 llite: remove lli_readdir_mutex

This mutex is initialized but never used, so
remove it.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I8df34923c44892dbf3a4e0bb603209cfe28adb1b
Reviewed-on: https://review.whamcloud.com/35884
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
2 months agoLU-6142 ldlm: remove unused ldlm_server_conn 83/35883/3
Mr NeilBrown [Thu, 22 Aug 2019 05:29:13 +0000 (15:29 +1000)]
LU-6142 ldlm: remove unused ldlm_server_conn

This field is never set or used, so remove it.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I1421914e487372eecb501e2483aeed07f56788e8
Reviewed-on: https://review.whamcloud.com/35883
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
2 months agoLU-6142 ptlrpc: remove scp_nthrs_stopping field. 82/35882/3
Mr NeilBrown [Thu, 22 Aug 2019 05:24:06 +0000 (15:24 +1000)]
LU-6142 ptlrpc: remove scp_nthrs_stopping field.

This field is unused, so remove it.
If "shrinking threads" is ever needed, any extra fields
required can be added then.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: Ie727033133aa846d92fdc52f6545bc1cd5efe153
Reviewed-on: https://review.whamcloud.com/35882
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
2 months agoLU-6142 ptlrpc: remove srv_threads from struct ptlrpc_service 81/35881/3
Mr NeilBrown [Thu, 22 Aug 2019 05:22:20 +0000 (15:22 +1000)]
LU-6142 ptlrpc: remove srv_threads from struct ptlrpc_service

The threads are not stored here - nothing is.
Threads are stored in svcpt->scp_threads.
So remove the field and update the comment.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I46875f77bb017bec47f901eca8dc946f11488cbe
Reviewed-on: https://review.whamcloud.com/35881
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
2 months agoLU-6142 ptlrpc: remove bd_import_generation field. 80/35880/3
Mr NeilBrown [Thu, 22 Aug 2019 05:19:20 +0000 (15:19 +1000)]
LU-6142 ptlrpc: remove bd_import_generation field.

This field is set, but never accessed. So remove it.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I26791e9fbc50676a705a94755674945eea9bf262
Reviewed-on: https://review.whamcloud.com/35880
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
2 months agoLU-6142 ptlrpc: remove struct ptlrpc_bulk_page 79/35879/3
Mr NeilBrown [Thu, 22 Aug 2019 05:18:04 +0000 (15:18 +1000)]
LU-6142 ptlrpc: remove struct ptlrpc_bulk_page

This structure is never used, so remove it.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I2a4fbfe2701636670801adcd0c2f2d991598f8f6
Reviewed-on: https://review.whamcloud.com/35879
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
2 months agoLU-6142 llog: remove olg_cat_processing field. 78/35878/3
Mr NeilBrown [Thu, 22 Aug 2019 05:16:08 +0000 (15:16 +1000)]
LU-6142 llog: remove olg_cat_processing field.

This mutex is initialized but never used.
Remove it.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: Id6fbbfdc5344435627b88104c46785e8d95e4ab1
Reviewed-on: https://review.whamcloud.com/35878
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
2 months agoLU-6142 lustre: remove imp_no_timeout field 77/35877/3
Mr NeilBrown [Thu, 22 Aug 2019 05:11:58 +0000 (15:11 +1000)]
LU-6142 lustre: remove imp_no_timeout field

This field is never set and never used.  Remove it.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I1dddbe620405b49e90530adcb9a7dec7f9ece64f
Reviewed-on: https://review.whamcloud.com/35877
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
2 months agoLU-6142 lustre: remove ldt_obd_type field of lu_device_type 76/35876/4
Mr NeilBrown [Thu, 22 Aug 2019 05:05:50 +0000 (15:05 +1000)]
LU-6142 lustre: remove ldt_obd_type field of lu_device_type

This field is never set, so it is always NULL.
So remove it,
 and the one place it is used,
 and a variable that now will now never be set.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I5530dd92b653b4259ddcbf8dc21af03a79ed255a
Reviewed-on: https://review.whamcloud.com/35876
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Tested-by: Maloo <maloo@whamcloud.com>
2 months agoLU-6142 fld: remove fci_no_shrink field. 75/35875/3
Mr NeilBrown [Thu, 22 Aug 2019 05:03:37 +0000 (15:03 +1000)]
LU-6142 fld: remove fci_no_shrink field.

This field is never set, so is always zero.
Remove it, and the one place where it is tested.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I4e576cc1984e0fac829a65c6b53d6ddac0d0ee88
Reviewed-on: https://review.whamcloud.com/35875
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
2 months agoLU-11421 dom: manual OST-to-DOM migration via mirroring 59/35359/5
Mikhail Pershin [Fri, 28 Jun 2019 10:54:04 +0000 (13:54 +0300)]
LU-11421 dom: manual OST-to-DOM migration via mirroring

Allow DOM mirroring, update LOV/LOD code to check not just
first component for DOM pattern but cycle through all mirrors
if any. Sanity checks allows one DOM component in a mirror
and it should be the first one. Multiple DOM components are
allowed only with the same for now.

Do OST file migration to MDT by using FLR. That can't be done
by layout swapping, because MDT data will be tied to temporary
volatile file but we want to keep data with the original file.
The mirroring allows that with the following steps:
- extent layout with new mirror on MDT, no data is copied but
  new mirror stays in 'stale' state. The reason is the same
  problem with volatile file.
- resync mirrors, now new DOM layout is filled with data.
- remove first mirror

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I1d4213196a16d6aec70861c5530910cac062e34f
Reviewed-on: https://review.whamcloud.com/35359
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12443 ptlrpc: fix reply buffers shrinking and growing 43/35243/7
Mikhail Pershin [Mon, 17 Jun 2019 08:00:31 +0000 (11:00 +0300)]
LU-12443 ptlrpc: fix reply buffers shrinking and growing

The req_capsule_shrink() doesn't update capsule itself with
new buffer lenghts after the shrinking. Usually it is not
needed because reply is packed already. But if reply buffers
are re-allocated by req_capsule_server_grow() then non-updated
lenghts from capsule are used causing bigger reply message.
That may cause client buffer re-allocation with resend.

Patch does the following:
- update capsule lenght after the shrinking
- introduce lustre_grow_msg() to grow msg field in-place
- update req_capsule_server_grow() to use generic
  lustre_grow_msg() and make it able to grow reply without
  re-allocation if reply buffer is big enough already
- update sanity test 271f to use bigger file size to exceed
  current maximum reply buffer size allocated on client.

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I154c55d98f41406d0c932c7e8705e0ecf3dfa935
Reviewed-on: https://review.whamcloud.com/35243
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Tested-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-10931 tests: resume testing of recovery-small 136 49/35949/2
James Nunez [Wed, 28 Aug 2019 14:48:47 +0000 (08:48 -0600)]
LU-10931 tests: resume testing of recovery-small 136

recovery-small test 136 was skipped, added to the
ALWAYS_EXCEPT list, due to excessive failures.  A fix
for failures has landed and we need to start running
test 136 again.

Test-Parameters: trivial
Test-Parameters: fstype=zfs mdscount=2 mdtcount=4 testlist=recovery-small
Test-Parameters: mdscount=2 mdtcount=4 testlist=recovery-small
Test-Parameters: fstype=zfs mdscount=1 mdtcount=1 testlist=recovery-small
Test-Parameters: mdscount=1 mdtcount=1 testlist=recovery-small
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I2ae8578e85024b31e226807e743a154707989ad4
Reviewed-on: https://review.whamcloud.com/35949
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12705 build: fix building fail against Power9 little endian 07/36007/5
Gu Zheng [Fri, 30 Aug 2019 07:27:30 +0000 (03:27 -0400)]
LU-12705 build: fix building fail against Power9 little endian

We use "%ll[dux]" for __u64 variable as an input/output modifier,
this may cause building error on some architectures which use "long"
for 64-bit types, for example, Power9 little endian.
Here add necessary typecasting (long long/unsigned long long) to
make the build correct.

Test-Parameters: trivial
Change-Id: I2e8569f4ac14f7d328a29d153ff57c7834cabc46
Signed-off-by: Gu Zheng <gzheng@ddn.com>
Reviewed-on: https://review.whamcloud.com/36007
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
2 months agoLU-12602 mdt: more EA size check in mdt_getxattr_pack_reply() 03/36103/3
Emoly Liu [Mon, 9 Sep 2019 08:10:29 +0000 (16:10 +0800)]
LU-12602 mdt: more EA size check in mdt_getxattr_pack_reply()

While the RMF_EAVALS field size can be arbitrary length,
the RMF_EAVALS_LENS field definition specifies
the RMF_F_STRUCT_ARRAY flag, so the passed size must be a multiple
of sizeof(__u32) or the internal LBUG() will trigger.

Signed-off-by: Emoly Liu <emoly@whamcloud.com>
Change-Id: I767e1b1496298e9a66274fc324f9c34daaed4a09
Reviewed-on: https://review.whamcloud.com/36103
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
2 months agoLU-12613 ptlrpc: check buffer length in lustre_msg_string() 32/35932/7
Emoly Liu [Thu, 29 Aug 2019 06:15:15 +0000 (14:15 +0800)]
LU-12613 ptlrpc: check buffer length in lustre_msg_string()

Check buffer length in lustre_msg_string() in case of any invalid
access.

Change-Id: I286000db16384938a594bd8d104e5f3d0fff585a
Reported-by: Alibaba Cloud <yunye.ry@alibaba-inc.com>
Signed-off-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/35932
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Yunye Ry <yunye.ry@alibaba-inc.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
2 months agoLU-12635 lnet: Fix deceptive indenting on for_each 86/36086/2
Shaun Tancheff [Fri, 6 Sep 2019 17:25:57 +0000 (12:25 -0500)]
LU-12635 lnet: Fix deceptive indenting on for_each

This patch fixes some deceptive indentation not reported by checkpatch
introduced by the previous cleanup

Test-Parameters: trivial
Cray-bug-id: LUS-7690
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Change-Id: I568637c508c2c5fcb84810e8e6deae2fb082e069
Reviewed-on: https://review.whamcloud.com/36086
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-9897 utils: remove llverfs dependency on libext2fs 16/36016/2
Andreas Dilger [Fri, 30 Aug 2019 22:47:45 +0000 (16:47 -0600)]
LU-9897 utils: remove llverfs dependency on libext2fs

The llverfs tool can be built with or without libext2fs.

If built without libext2fs then it estimates how many subdirectories
are needed based on the total filesystem size and default settings.
In this case, it calls "chattr" directly to set the "TOPDIR" flag so
directories are still spread across the filesystem, if possible.

Don't unnecessarily seek() the file descriptor if this is not needed.

Improve stats output so it isn't negative if the IO rate changes, and
always print at least _something_ for read and write when run.

Test-Parameters: trivial testlist=conf-sanity
Signed-off-by: Andreas Dilger <adilger@dilger.ca>
Change-Id: Ia1d51ec8d68d3068a9c31e966935149f123ebbe5
Reviewed-on: https://review.whamcloud.com/36016
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12428 tests: fix sanity-sec wait_nm_sync 09/36009/3
Sebastien Buisson [Fri, 30 Aug 2019 15:21:40 +0000 (17:21 +0200)]
LU-12428 tests: fix sanity-sec wait_nm_sync

There are 2 different versions of wait_nm_sync in sanity-sec.sh,
because of a bad patch rebase.

Test-Parameters: trivial
Test-Parameters: combinedmdsmgs=true mdscount=2 mdtcount=4 osscount=1 ostcount=8 testlist=sanity-sec,sanity-sec,sanity-sec
Test-Parameters: combinedmdsmgs=true mdscount=2 mdtcount=4 osscount=1 ostcount=8 testlist=sanity-sec,sanity-sec,sanity-sec
Test-Parameters: combinedmdsmgs=true mdscount=2 mdtcount=4 osscount=1 ostcount=8 testlist=sanity-sec,sanity-sec,sanity-sec
Fixes: bb0a10752850 ("LU-11883 nodemap: make deny_unknown visible on default nodemap")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I25edb19ce29b9380ad5721b550a6816899d9f95b
Reviewed-on: https://review.whamcloud.com/36009
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12608 kernel: kernel update RHEL7.6 [3.10.0-957.27.2.el7] 38/35638/5
Jian Yu [Tue, 3 Sep 2019 17:06:45 +0000 (10:06 -0700)]
LU-12608 kernel: kernel update RHEL7.6 [3.10.0-957.27.2.el7]

Update RHEL7.6 kernel to 3.10.0-957.27.2.el7.

Test-Parameters: clientdistro=el7.6 serverdistro=el7.6

Change-Id: I7b065a0476fafd46a3bf2e8f9712e214c30da271
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/35638
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-11915 tests: use trusted.* xattr for conf-sanity test_115 48/34948/8
Artem Blagodarenko [Thu, 23 May 2019 11:49:54 +0000 (14:49 +0300)]
LU-11915 tests: use trusted.* xattr for conf-sanity test_115

conf-sanity test 115 is always skipped because of patch
"LU-8569 linkea: linkEA size limitation". Links could not be
used for creating external inode for xattrs.

Let's use "trusted.*" xattrs for creating external xattr inode.

Fixes: 048a8740ae26 ("LU-8569 lfsck: handle linkEA overflow")

Signed-off-by: Artem Blagodarenko <c17828@cray.com>
Cray-bug-id: LUS-6966
Change-Id: If4c74c5baaeec6503edb3c5e756b9a55ff1bd90a
Test-Parameters: trivial testlist=conf-sanity envdefinitions=FLAKEY=false
Reviewed-on: https://review.whamcloud.com/34948
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
2 months agoLU-12336 build: Revert Update ZFS version to 0.8.1 37/36137/3
James Nunez [Mon, 9 Sep 2019 20:13:13 +0000 (14:13 -0600)]
LU-12336 build: Revert Update ZFS version to 0.8.1

Revert "LU-12336 build: Update ZFS version to 0.8.1"

We've seen an uptick in autotest testing failures for ZFS
testing and are investigating if moving to ZFS 0.8.1 is a
cause of these failures.  Thus, run the ZFS test sessions
multiple times to get a baseline of test failures with
ZFS 0.7.13.

This reverts commit dba32e417635359b1d68180b77193e1c9ddd1e8f.

Test-Parameters: trivial testgroup=review-dne-zfs-part-1
Test-Parameters: testgroup=review-dne-zfs-part-2
Test-Parameters: testgroup=review-dne-zfs-part-3
Test-Parameters: testgroup=review-dne-zfs-part-4
Test-Parameters: testgroup=review-zfs

Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: Iedb5c5411c721cfe738a7d3db455b222e7b4050d
Reviewed-on: https://review.whamcloud.com/36137
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 months agoNew tag 2.12.58 2.12.58 v2_12_58
Oleg Drokin [Mon, 9 Sep 2019 21:07:22 +0000 (17:07 -0400)]
New tag 2.12.58

Change-Id: I9cd549d130a84eda7697046feabcc3a818ddaafc
Signed-off-by: Oleg Drokin <green@whamcloud.com>
3 months agoLU-12368 ptlrpc: make DEBUG_REQ messages consistent 11/35311/13
Andreas Dilger [Tue, 25 Jun 2019 04:41:12 +0000 (22:41 -0600)]
LU-12368 ptlrpc: make DEBUG_REQ messages consistent

Remove linefeed from DEBUG_REQ() messages, since this results in
debug logs that are split across multiple lines and do not start
with the proper timestamp or other standard fields.  This makes
post-processing difficult.

Some error and debug messages are checked for explicitly in tests.
Add a comment by those lines in the code to alert the reader that
changes to those messages may cause test failures, and make the
tests more forgiving in case of minor changes to the formatting.

Fix several tests to check for actual error message.  Some tests
have been broken for so long (1.5/1.8) that there is no point to
also check for the old messages, so use only the new messages.

The EINPROGRESS messages should not use D_ERROR, since they can be
hit under normal usage (e.g. LFSCK running), so D_WARNING at most.
Don't print every one to the console, that would be too verbose.

Fix code style of affected lines.

Fixes: f2a7347e4ceb ("LU-1943 filter: remove obdfilter from tree")
Fixes: 473a4adc4ec2 ("LU-7787 mdd: clean up orphan object handling")
Fixes: 9f79d4488fbb ("LU-10048 ofd: take local locks within transaction")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I3cfc333ccfbf2a9bfc9e93dcfa7dbb99443ebbe5
Reviewed-on: https://review.whamcloud.com/35311
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 months agoLU-9964 llite: prevent mulitple group locks 91/35791/5
Alexander Boyko [Wed, 14 Aug 2019 09:06:14 +0000 (05:06 -0400)]
LU-9964 llite: prevent mulitple group locks

The patch adds mutex for group lock enqueue. It also adds waiting
of group lock users on a client side for a same node. This prevents
mulitple locks on the same resource and fixes a bugs when two locks
cover the same dirty pages.

The patch adds test sanity 244b. It creates threads which
opens file, takes group lock, writes data, puts group lock, closes.
It recreates the problem when a client has two or more group locks
for a single file. This leads to a wrong behaviour for a flush etc.
osc_cache_writeback_range()) ASSERTION( hp == 0 && discard == 0 )
failed
One more test for group lock with open file and fork. It checks that
childs doesn't unlock file until the last close.

Cray-bug-id: LUS-7232
Signed-off-by: Alexander Boyko <c17825@cray.com>
Change-Id: Iaf65cf26219211b76daaa1a91eeb80092c328f2d
Reviewed-on: https://review.whamcloud.com/35791
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 months agoLU-9897 utils: remove dependency on libblkid 10/36010/2
Andreas Dilger [Thu, 29 Aug 2019 07:46:24 +0000 (01:46 -0600)]
LU-9897 utils: remove dependency on libblkid

There was a weak dependency in llverdev on libblkid in order to
find the size of the underlying block device.  However, in the
absence of libblkid there is fallback code that can find the
block device size, so we don't really need libblkid at all.

Similarly, llverdev had a weak dependency on libext2fs to check
if the block device was currently in use, but all kernels since
v2.6.34-rc4-22-g6b4517a7913a allow using open(O_EXCL) to ensure
a block device is not mounted, so there is no need for llverdev
to depend on libext2fs for this.

Remove the conditional code depending on the external libraries
from llverdev and the corresponding configuration checks, which
were causing problems in various build environments.

Test-Parameters: trivial testlist=conf-sanity
Fixes: 6bc323747706 ("LU-9897 build: use pkgconf for detecting libblkid")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I90d2f0c99115ce07168c185dec6628be813ebbe5
Reviewed-on: https://review.whamcloud.com/36010
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 months agoLU-12094 ptlrpc: protect obd_import in lprocfs_import_seq_write 53/35853/3
James Simmons [Thu, 22 Aug 2019 23:52:12 +0000 (19:52 -0400)]
LU-12094 ptlrpc: protect obd_import in lprocfs_import_seq_write

The obd_import can disappear so take the cl_sem to protect it.

Change-Id: I36f2c4c606a0470781a692631265938848173f8b
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/35853
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 months agoLU-12674 osp: handle -EINPROGRESS on llog objects 44/35844/4
Alexey Zhuravlev [Wed, 21 Aug 2019 08:32:56 +0000 (11:32 +0300)]
LU-12674 osp: handle -EINPROGRESS on llog objects

if llog object is corrupted and OI doesn't allow access to that
OSP panics being asked to declare new llog record (e.g. for unlink).
Instead OSP should complain in the logs, skip llogging and suggest
to run LFSCK to fix orphans.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I18d4d68811833c08cdc1937d147ac6e8c3408a30
Reviewed-on: https://review.whamcloud.com/35844
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 months agoLU-11607 tests: replace version/fstype in sanity-dom/quota 66/35766/2
James Nunez [Sun, 11 Aug 2019 16:44:16 +0000 (10:44 -0600)]
LU-11607 tests: replace version/fstype in sanity-dom/quota

The routine get_lustre_env() is available to all Lustre test
suites and sets an environment variable for the file system
type for MDS1 and OST1 and sets variables for the Lustre
version of servers.

In sanity-dom and sanity-quota, replace the calls to
facet_fstype() and lustre_version_code() for all server
types with definitions in get_lustre_env().

While doing this, replace 'facet_fstype $SINGLEMDS' and
'lustre_version_code $SINGLEMDS' with $mds1_FSTYPE
and $MDS1_VERSION, respectively.

Clean up around any modifications by removing calls to
return after skip() or skip_env() and convert spaces
to tabs.

Test-Parameters: trivial testlist=sanity-dom,sanity-quota
Test-Parameters: fstype=zfs testlist=sanity-dom,sanity-quota
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I4dffae5ad69aa2dec5b85e285a9b314b9b35b2ba
Reviewed-on: https://review.whamcloud.com/35766
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 months agoLU-11607 tests: replace lustre_version calls in sanity-flr/pfl 60/35760/2
James Nunez [Fri, 9 Aug 2019 23:58:32 +0000 (17:58 -0600)]
LU-11607 tests: replace lustre_version calls in sanity-flr/pfl

The routine get_lustre_env() is available to all Lustre test
suites and sets an environment variable for the Lustre
version of servers for MDS1.

In sanity-flr and sanity-pfl, replace the calls to
lustre_version_code() for all server types with
definitions in get_lustre_env().

While doing this, replace ‘lustre_version_code $SINGLEMDS’
with ‘MDS1_VERSION’.

Clean up around any modifications by removing calls to
return after skip() or skip_env().

Test-Parameters: trivial testlist=sanity-flr,sanity-pfl
Test-Parameters: fstype=zfs testlist=sanity-flr,sanity-pfl
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: Ic80ea80268ba28f52d96680e304838b6e8c3a6fd
Reviewed-on: https://review.whamcloud.com/35760
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 months agoLU-12236 tests: add tests for LNET network namespace 66/35666/10
Aurelien Degremont [Thu, 1 Aug 2019 12:48:05 +0000 (12:48 +0000)]
LU-12236 tests: add tests for LNET network namespace

This patch adds tests for LNET for this feature.

Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Aurelien Degremont <degremoa@amazon.com>
Change-Id: I2320e5da1beef30be5dcca9529fa838fc9304876
Reviewed-on: https://review.whamcloud.com/35666
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <hornc@cray.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 months agoLU-12422 lnet: discovery off route state update 99/35199/4
Amir Shehata [Wed, 12 Jun 2019 00:41:23 +0000 (17:41 -0700)]
LU-12422 lnet: discovery off route state update

When discovery is off rely on the discovery ping response
only, rather than the internal peer database to determine
route state. With discovery off the internal peer database
is not updated with all the gateway's interfaces.

Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: If1b5347629ffeb92208097c06d708e1105d27679
Reviewed-on: https://review.whamcloud.com/35199
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Olaf Weber <olaf.weber@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
3 months agoLU-12367 llite: Fix page count for unaligned reads 15/35015/7
Patrick Farrell [Fri, 31 May 2019 23:52:31 +0000 (19:52 -0400)]
LU-12367 llite: Fix page count for unaligned reads

When a read is unaligned on both the first and last page,
the calculation used to determine count of pages for
readahead misses that we access both of those pages.

Increase the calculated count by 1 in this case.

This case is covered by the generic readahead tests added
in LU-12645.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Idb28d300a232026a9d916d010ee8ebf209283242
Reviewed-on: https://review.whamcloud.com/35015
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>