Whamcloud - gitweb
fs/lustre-release.git
12 months agoLU-1732 utils: allow ldiskfs wide striping with ea_inode 15/4315/16
Patrick Farrell [Sat, 25 May 2019 13:48:06 +0000 (09:48 -0400)]
LU-1732 utils: allow ldiskfs wide striping with ea_inode

Format the MDT filesystem with the "ea_inode" option by default
so that files can have more than 160 stripes, and large xattrs
over one filesystem block in size (normally 4096 bytes).

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I71589c13e59a9d13db3bf075282cf6334b86be30
Reviewed-on: https://review.whamcloud.com/4315
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
12 months agoLU-12462 llite: Remove old fsync versions 39/35339/2
Patrick Farrell [Thu, 27 Jun 2019 15:41:29 +0000 (11:41 -0400)]
LU-12462 llite: Remove old fsync versions

The old two arg and three arg versions of fsync were last
used in 2.6.3X kernels, and we no longer support those.

Remove it to clean up our fsync defines, and add to debug
to capture all the arguments of the current fsync.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I3ab18fa5a7a4a6d3b0714570a8ff3f2ad820e5ad
Reviewed-on: https://review.whamcloud.com/35339
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-9971 lnet: fix peer ref counting 46/35446/2
Amir Shehata [Mon, 8 Jul 2019 19:51:05 +0000 (12:51 -0700)]
LU-9971 lnet: fix peer ref counting

Exit from the loop after peer ref count has been incremented
to avoid wrong ref count.

The code makes sure that a peer is queued for discovery at most
once if discovery is disabled. This is done to use discovery
as a standard ping for gateways which do not have discovery feature
or discovery is disabled.

Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: I2cc4c8f9d780f5c438d9b51bb2d1106fec553f39
Reviewed-on: https://review.whamcloud.com/35446
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Olaf Weber <olaf.weber@hpe.com>
Reviewed-by: Chris Horn <hornc@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoRevert "LU-11760 ofd: formatted OST recognition change" 88/35388/4
Andreas Dilger [Sun, 30 Jun 2019 07:54:13 +0000 (07:54 +0000)]
Revert "LU-11760 ofd: formatted OST recognition change"

This is causing conf-sanity test_69 failures in LU-12404 due to
the increased limit on the number of objects precreated after
recovery.  The issue will be fixed in a different way.

This reverts commit d07d9c5ed0aa1d6614944c7d1e0ca55cba301dc4.

Change-Id: I437889f20699207fa15eff6685b0992292555f19
Reviewed-on: https://review.whamcloud.com/35388
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Sergey Cheremencev <c17829@cray.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12494 kernel: kernel update SLES12 SP4 [4.12.14-95.19.1] 91/35391/2
Jian Yu [Sun, 30 Jun 2019 07:01:18 +0000 (00:01 -0700)]
LU-12494 kernel: kernel update SLES12 SP4 [4.12.14-95.19.1]

Update SLES12 SP4 kernel to 4.12.14-95.19.1 for Lustre client.

Test-Parameters: trivial clientdistro=sles12sp4 \
envdefinitions=LNET_SELFTEST_EXCEPT=smoke,SANITY_EXCEPT=103a

Change-Id: I6a101dc2637945192cf8aca661e23c3bccb47609
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/35391
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12458 kernel: kernel update RHEL7.6 [3.10.0-957.21.3.el7] 68/35268/5
Jian Yu [Wed, 19 Jun 2019 20:19:53 +0000 (13:19 -0700)]
LU-12458 kernel: kernel update RHEL7.6 [3.10.0-957.21.3.el7]

Update RHEL7.6 kernel to 3.10.0-957.21.3.el7.

Test-Parameters: clientdistro=el7.6 serverdistro=el7.6

Change-Id: I10c5708a412022e6066ff7ce2801375049f188d9
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/35268
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12456 kernel: kernel update RHEL 8.0 [4.18.0-80.4.2.el8_0] 67/35267/4
Jian Yu [Wed, 19 Jun 2019 18:42:53 +0000 (11:42 -0700)]
LU-12456 kernel: kernel update RHEL 8.0 [4.18.0-80.4.2.el8_0]

Update RHEL 8.0 kernel to 4.18.0-80.4.2.el8_0 for Lustre client.

Change-Id: I1ff433f6ef3433dae54def0e89bc035d25ff15a4
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/35267
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12474 tests: Do not run check_progs_installed for racer 27/35327/2
Oleg Drokin [Wed, 26 Jun 2019 02:22:23 +0000 (22:22 -0400)]
LU-12474 tests: Do not run check_progs_installed for racer

it's run from within racer so racer is already there for sure

Change-Id: Ifd78cd051842c9663130b650c6e35d60332250e7
Test-Parameters: testlist=racer
Test-Parameters: trivial
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/35327
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
13 months agoLU-9792 tests: t-f do not create empty files for ZFS 05/34105/7
Alex Zhuravlev [Thu, 24 Jan 2019 13:24:26 +0000 (16:24 +0300)]
LU-9792 tests: t-f do not create empty files for ZFS

as zpool doesn't like empty device files.

Change-Id: I3a224bf2e60c6f20e13013caf827fe29641a8c5c
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/34105
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12395 build: build mpitests for el8 74/35374/9
Minh Diep [Fri, 28 Jun 2019 21:28:39 +0000 (14:28 -0700)]
LU-12395 build: build mpitests for el8

RHEL8 has rpm-mpi-hooks which requires binaries
to be in specific mpi bin to generate the correct
requires

See https://fedoraproject.org//wiki/Changes/RpmMPIReqProv
and https://fedoraproject.org/wiki/Packaging:MPI

Test-Parameters: trivial clientdistro=el8 serverdistro=el7.6 testgroup=regression-mpi

Change-Id: Id9fa50e15b48b9da846083b9e9cd894ad1eac967
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/35374
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12483 tests: fix sanity test 60h running conditions 55/35355/7
Oleg Drokin [Fri, 28 Jun 2019 01:59:37 +0000 (21:59 -0400)]
LU-12483 tests: fix sanity test 60h running conditions

The test is supposed to run in DNE mode on 2.12.52 or above,
but the conditions are somehow reversed.

Change-Id: I322941a6098b0dbfbabe2f5c70f40f8e81d1bbab
Test-Parameters: trivial
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/35355
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
13 months agoRevert "LU-12328 flr: preserve last read mirror" 50/35450/2
Oleg Drokin [Tue, 9 Jul 2019 17:27:41 +0000 (17:27 +0000)]
Revert "LU-12328 flr: preserve last read mirror"

This is causing somewhat frequent crashes tracked in
LU-12525

This reverts commit 810f2a5fef577b4f0f6a58ab234cf29afd96c748.

Change-Id: If7604ad4ca1d4ddc63a20fa2ec7d9467ee7bb5f9
Reviewed-on: https://review.whamcloud.com/35450
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12495 obdclass: generate random u64 max correctly 94/35394/5
Lai Siyao [Fri, 28 Jun 2019 17:36:27 +0000 (01:36 +0800)]
LU-12495 obdclass: generate random u64 max correctly

Generate random u64 max number correctly, and make it an obdclass
function lu_prandom_u64_max().

Fixes: 7a707d4828 (libcfs: replace cfs_rand() with prandom_u32_max())

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I2b94a42b42539be319f358d7af2a82dc8b26117c
Reviewed-on: https://review.whamcloud.com/35394
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-9859 libcfs: remove libcfs_debug_vmsg2 25/35225/3
NeilBrown [Thu, 13 Jun 2019 19:10:21 +0000 (15:10 -0400)]
LU-9859 libcfs: remove libcfs_debug_vmsg2

Now that libcfs_debug_vmsg2 has no (external) users, we can remove it.
It is used to implement libcfs_debug_msg(), so simply move
the body of the function (suitably modified) into that one caller.

Linux-commit: d42a3aded317c97594c19995879999428de53c46

Signed-off-by: NeilBrown <neilb@suse.com>
Change-Id: I80d24abcc23a8f6e2f8b995d1337ba5038318d5a
Reviewed-on: https://review.whamcloud.com/35225
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12328 flr: preserve last read mirror 11/35111/2
Jinshan Xiong [Sat, 8 Jun 2019 05:34:03 +0000 (22:34 -0700)]
LU-12328 flr: preserve last read mirror

This patch preserves the mirror that has been read successfully
so that all subsequent I/O can take this advantage it and avoid
trying to read unavailable OSTs.

Signed-off-by: Jinshan Xiong <jinshan.xiong@gmail.com>
Change-Id: I806f936340db7c73228048edf21d5ecbed4b3c6c
Reviewed-on: https://review.whamcloud.com/35111
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-9971 lnet: use after free in lnet_discover_peer_locked() 44/28944/5
Olaf Weber [Tue, 12 Sep 2017 12:07:50 +0000 (14:07 +0200)]
LU-9971 lnet: use after free in lnet_discover_peer_locked()

When the lnet_net_lock is unlocked, the peer attached to an
lnet_peer_ni (found via lnet_peer_ni::lpni_peer_net->lpn_peer)
can change, and the old peer deallocated. If we are really
unlucky, then all the churn could give us a new, different,
peer at the same address in memory.

Change the reference counting on the lnet_peer lp so that it
is guaranteed to be alive when we relock the lnet_net_lock for
the cpt. When the reference count is dropped lp may go away if
it was unlinked, but the new peer is guaranteed to have a
different address, so we can still correctly determine whether
the peer changed and discovery should be redone.

Signed-off-by: Olaf Weber <olaf.weber@hpe.com>
Change-Id: Ia44dce20074b27ec0e77d7c1908c6a44ec73d326
Reviewed-on: https://review.whamcloud.com/28944
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-930 doc: update MAINTAINERS file 41/35241/4
Andreas Dilger [Mon, 17 Jun 2019 03:23:50 +0000 (05:23 +0200)]
LU-930 doc: update MAINTAINERS file

Update Patrick's email address, and remove John Hammond from the list.

Add some existing files to existing subsystems, and add a few new
subsystems for recently-landed code.

Re-order a couple of the entries to be in alphabetical order.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I297ec31abf65d54dc363b5d5fd460b7b3e3ebbe5
Reviewed-on: https://review.whamcloud.com/35241
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12045 tests: honor EXCEPT tests when using ONLY list 38/34938/9
James Nunez [Wed, 22 May 2019 16:22:19 +0000 (10:22 -0600)]
LU-12045 tests: honor EXCEPT tests when using ONLY list

The Lustre test framework allows a user to specify a subset
of tests to run using the ONLY parameter or --only flag.
The test framwork also allows the user to specify a list of
tests to skip using the EXCEPT or ALWAYS_EXCEPT parameters.
By default, if the ONLY parameter or --only flag is used,
the EXCEPT and ALWAYS_EXCEPT lists are ignored.

Add a flag to auster, -H, and an environment variable,
HONOR_EXCEPT, to skip the tests on the ALWAYS_EXCEPT,
EXCEPT and SLOW lists when using the ONLY/--only parameter.

Test-Parameters: trivial
Test-Parameters: envdefinitions=ONLY="40-43" testlist=sanity
Test-Parameters: envdefinitions=ONLY="40-43" austeroptions=-H testlist=sanity
Test-Parameters: envdefinitions=SLOW="no",ONLY="27" testlist=sanity
Test-Parameters: envdefinitions=SLOW="no",ONLY="27" austeroptions=-H testlist=sanity
Test-Parameters: envdefinitions=SLOW="yes",ONLY="27" testlist=sanity
Test-Parameters: envdefinitions=SLOW="yes",ONLY="27" austeroptions=-H testlist=sanity
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I173e48e1d2dc3b404d148146639a13148bc48a3d
Reviewed-on: https://review.whamcloud.com/34938
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-11518 ptlrpc: don't reset lru_resize on idle reconnect 85/35285/7
Andriy Skulysh [Tue, 11 Jun 2019 14:44:32 +0000 (17:44 +0300)]
LU-11518 ptlrpc: don't reset lru_resize on idle reconnect

ptlrpc_disconnect_idle_interpret() clears imp_remote_handle,
so reconnect has pcaa_initial_connect set to 1.

Update only changed ns_connect_flags bits.

Fixes: 5a6ceb664f0 ("LU-7236 ptlrpc: idle connections can disconnect")
Change-Id: I2368708b6381c1d772c47dc6e61c8fb39a14a2cc
Cray-bug-id: LUS-7471
Signed-off-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Alexander Boyko <c17825@cray.com>
Reviewed-by: Andrew Perepechko <c17827@cray.com>
Reviewed-on: https://review.whamcloud.com/35285
Reviewed-by: Alexandr Boyko <c17825@cray.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Gu Zheng <gzheng@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
13 months agoLU-11444 ptlrpc: Add increasing XIDs CONNECT2 flag 13/35113/3
Andriy Skulysh [Fri, 28 Sep 2018 11:02:22 +0000 (14:02 +0300)]
LU-11444 ptlrpc: Add increasing XIDs CONNECT2 flag

This patch reserves the OBD_CONNECT2 flag
for increasing XIDs.

Change-Id: Id825ce1c86a345884c9d4cd90cd2e13839a268f0
Cray-bug-id: LUS-6272
Signed-off-by: Andriy Skulysh <c17819@cray.com>
Reviewed-on: https://review.whamcloud.com/35113
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexandr Boyko <c17825@cray.com>
13 months agoLU-4423 lustre: don't declare extern variables in C files. 94/35294/3
NeilBrown [Sun, 23 Jun 2019 13:34:50 +0000 (09:34 -0400)]
LU-4423 lustre: don't declare extern variables in C files.

'extern' declarations should only appear in .h files.
All these names are declared in .h files as needed,
and these duplicate declarations in .c files can
be removed.

Test-Parameters: trivial

Change-Id: Ic563789f350fd21fd033f1d3c49cdac2125b86c5
Signed-off-by: NeilBrown <neilb@suse.com>
Reviewed-on: https://review.whamcloud.com/35294
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
13 months agoLU-4423 mgc: remove llog_process_lock 93/35293/2
NeilBrown [Sun, 23 Jun 2019 13:28:24 +0000 (09:28 -0400)]
LU-4423 mgc: remove llog_process_lock

This mutex is never used, so remove it.

Test-Parameters: trivial

Change-Id: I53ce68beafcf8b895c1db138a6eff2ade2994832
Signed-off-by: NeilBrown <neilb@suse.com>
Reviewed-on: https://review.whamcloud.com/35293
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
13 months agoLU-1538 tests: standardize test script init - lnet 54/35254/2
Andreas Dilger [Mon, 17 Jun 2019 20:59:39 +0000 (14:59 -0600)]
LU-1538 tests: standardize test script init - lnet

Standardize the initial Lustre test script initialization for clarity
and consistency.

The LUSTRE path is already normalized in init_test_env(), so this
doesn't need to be done in the caller.  Use $(...) subshells instead
of `...` in the affected lines.  Remove NAME, PATH, SAVE_PWD,
RLUSTRE, MULTIOP variable initialization, since it is already done in
init_test_env() or not needed in the test script.

Move all definitions of ALWAYS_EXCEPT and SLOW to after
init_test_env() and init_logging() and call build_test_filter()
immediately after the ALWAYS_EXCEPT and SLOW definitions.

Test-Parameters: trivial
Test-Parameters: testgroup=review-dne-zfs-part-3
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I491cf3e282a1f31fb8c87440554445d708c6da1e
Reviewed-on: https://review.whamcloud.com/35254
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-10931 tests: stop running recovery-small 136 25/35325/5
James Nunez [Tue, 25 Jun 2019 21:46:15 +0000 (15:46 -0600)]
LU-10931 tests: stop running recovery-small 136

recovery-small test 136 hangs on MDS mount with multiple
MDSs.  We need to stop running this test until we find a
solution for this problem.

Test-Parameters: trivial mdscount=2 mdtcount=4 testlist=recovery-small

Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I5364efc8b6fe1ea9b3c3920121a8fab6ac03bd05
Reviewed-on: https://review.whamcloud.com/35325
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
13 months agoLU-12349 llite: console message for disabled flock call 86/34986/13
Li Xi [Thu, 13 Jun 2019 03:27:50 +0000 (11:27 +0800)]
LU-12349 llite: console message for disabled flock call

When flock option is disabled on a Lustre client, any call to
flock() or lockf() would cause a return value with failure.
For applications that don't print proper error message, it is
hard to know the root cause is the missing flock option on Lustre
file system. Thus this patch prints following error message to
the tty that calls flock()/lockf():

"Lustre: flock disabled, mount with '-o [local]flock' to enable"

Such message will print to each file descriptor no more than
once to avoid message flood.

In order to do so, this patch adds support for CDEBUG_LIMIT(D_TTY).
It prints the message to tty. When using this macro, please
note that "\r\n" needs to be the end of the line. Otherwise,
message like "format at $FILE:$LINO:$FUNC doesn't end in '\r\n'"
will be printed to the system message for warning.

Change-Id: I4eeb3ea219848ebbbca9d14e3d2b8a23237105b5
Signed-off-by: Li Xi <lixi@ddn.com>
Reviewed-on: https://review.whamcloud.com/34986
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-10070 lod: layout_del memleak 70/35270/4
Vitaly Fertman [Wed, 19 Jun 2019 21:28:40 +0000 (00:28 +0300)]
LU-10070 lod: layout_del memleak

A component may have been declared on a pool or on a specific set
of OSTs, however only the first component is INIT'ed at the beginning.
lod_layout_del should take care about these allocations independently
of the existent striping

Cray-bug-id: LUS-2528
Signed-off-by: Vitaly Fertman <c17818@cray.com>
Change-Id: I7069154ec6b3e64cd945231829c19c4c6920c030
Reviewed-on: https://review.whamcloud.com/35270
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-11775 obdclass: protect imp_sec using rwlock_t 61/33861/7
Li Dongyang [Fri, 14 Dec 2018 02:16:39 +0000 (13:16 +1100)]
LU-11775 obdclass: protect imp_sec using rwlock_t

We've seen spinlock contention on imp_lock in
sptlrpc_import_sec_ref(), introduce a new rwlock
imp_sec_lock to protect imp_sec instead of using imp_lock.

This patch also removes imp_sec_mutex from obd_import,
which is not needed, to avoid confusion between
imp_sec_lock/mutex.

Test-Parameters: testlist=sanity-sec envdefinitions=SHARED_KEY=true
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Change-Id: I3be3a6d1225888134bf7dd58e4b05b864f8415b4
Reviewed-on: https://review.whamcloud.com/33861
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexey Lyashkov <c17817@cray.com>
Reviewed-by: Alexandr Boyko <c17825@cray.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-6142 ldlm: Fix style issues for ldlm_lib.c 95/34495/10
Arshad Hussain [Thu, 21 Mar 2019 01:10:18 +0000 (06:40 +0530)]
LU-6142 ldlm: Fix style issues for ldlm_lib.c

This patch fixes issues reported by checkpatch for
file lustre/ldlm/ldlm_lib.c

Change-Id: I7555974ea311af4e3d4eb64a24f810e4699a8690
Signed-off-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-on: https://review.whamcloud.com/34495
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
13 months agoLU-6142 obdecho: Fix style issues for echo.c 90/34490/5
Arshad Hussain [Wed, 20 Mar 2019 15:08:32 +0000 (20:38 +0530)]
LU-6142 obdecho: Fix style issues for echo.c

This patch fixes issues reported by checkpatch
for file lustre/obdecho/echo.c

Change-Id: I4a828de8c0761d5e9058beae38009055f1030c81
Signed-off-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-on: https://review.whamcloud.com/34490
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <maloo@whamcloud.com>
13 months agoLU-6142 utils: Fix style issues for mount_utils.c 39/34439/7
Arshad Hussain [Sat, 9 Mar 2019 20:55:10 +0000 (02:25 +0530)]
LU-6142 utils: Fix style issues for mount_utils.c

This patch fixes issues reported by checkpatch
for file lustre/utils/mount_utils.c

Change-Id: Ic60df860a15d8b66ec37c979eec66b2bc0c52dea
Signed-off-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-on: https://review.whamcloud.com/34439
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
13 months agoLU-11867 osd-ldiskfs: FID in LMA mismatch won't block create 52/34052/5
Lai Siyao [Mon, 7 Jan 2019 03:37:48 +0000 (11:37 +0800)]
LU-11867 osd-ldiskfs: FID in LMA mismatch won't block create

Sometimes two OST objects may be mapped to the same inode, so the
second object FID mismatch with FID in inode LMA, in this case,
if this inode was not written yet, it's safe to set object inode
to NULL to let it create a new inode.

Another case is if the mapped inode doesn't exist, it's also safe
to not initialize inode and return 0, so that create can succeed.

Add sanity-scrub.sh 4d for this.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Ic84cdeaca2ea202ab0c01a0075a2f9ee8627f508
Reviewed-on: https://review.whamcloud.com/34052
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-4423 ptlrpc: make ptlrpc_bulk_frag_ops always const. 95/35295/2
NeilBrown [Sun, 23 Jun 2019 13:41:25 +0000 (09:41 -0400)]
LU-4423 ptlrpc: make ptlrpc_bulk_frag_ops always const.

There is one place where a non-const pointer to this struct
exists, and that causes a cast to be required.

Make it always const, and discard the cast.

Test-Parameters: trivial

Change-Id: Ic48ff6a1f759abdddc75298feee2b68c60bf3e7b
Signed-off-by: NeilBrown <neilb@suse.com>
Reviewed-on: https://review.whamcloud.com/35295
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-4423 lprocfs: use log2.h macros instead of shift loop. 74/35274/3
NeilBrown [Thu, 20 Jun 2019 03:43:30 +0000 (23:43 -0400)]
LU-4423 lprocfs: use log2.h macros instead of shift loop.

These shift loops seem to be trying to avoid doing a
multiplication.
The same effect can be achieved more transparently using
rounddown_pow_of_two().  Even though there is a multiplication
in the C code, the resulting machine code just does a single shift.

As rounddown_pow_of_two() is not defined for 0, and as we cannot be
positively use the blk_size is non-zero, use blk_size ?: 1.

Change-Id: Ie4cec1ca3f30617df0022a9c0dd80fe7e755ed64
Signed-off-by: NeilBrown <neilb@suse.com>
Reviewed-on: https://review.whamcloud.com/35274
Tested-by: Jenkins
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-4423 ldlm: discard varname in ldlm_pool. 73/35273/3
NeilBrown [Thu, 20 Jun 2019 03:38:40 +0000 (23:38 -0400)]
LU-4423 ldlm: discard varname in ldlm_pool.

This allocated buffer serves no purpose.
A constant string is copied into it, it is passed to some
function which copies it out again, then the buffer is freed.
Instead, we can pass the constant string to that function.

Change-Id: I83ea8fc839da3933ebc34756fc5c06b86427ce7c
Signed-off-by: NeilBrown <neilb@suse.com>
Reviewed-on: https://review.whamcloud.com/35273
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
13 months agoLU-9859 libcfs: replace cfs_rand() with prandom_u32_max() 72/35272/3
NeilBrown [Thu, 20 Jun 2019 02:57:13 +0000 (22:57 -0400)]
LU-9859 libcfs: replace cfs_rand() with prandom_u32_max()

All occurrences of
   cfs_rand() % X
are replaced with
   prandom_u32_max(X)

cfs_rand() is a simple Linear Congruential PRNG. prandom_u32_max()
is at least as random, is seeded with more randomness, and uses
cpu-local state to avoid cross-cpu issues.

This is the first step is discarding the libcfs prng with
the standard linux prng.

Linux-commit: bcfa98a50763a0f781a8441d1994ae1456816219

Change-Id: I63679c269b72f4c4860cb3a47225178edf2d7892
Signed-off-by: NeilBrown <neilb@suse.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-on: https://review.whamcloud.com/35272
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-1538 tests: standardize test script init - dne-part-4 55/35255/3
Andreas Dilger [Mon, 17 Jun 2019 22:20:41 +0000 (16:20 -0600)]
LU-1538 tests: standardize test script init - dne-part-4

Standardize the initial Lustre test script initialization for
clarity and consistency.

The LUSTRE path is already normalized in init_test_env(), so
this doesn't need to be done in the caller.  Use $(...) subshells
instead of `...` in the affected lines.  Remove NAME, CHECKSTAT,
TMP, SAVE_PWD,SRCDIR, PATH, MULTIOP, SETUP, CLEANUP variable
initialization, since it is already done in init_test_env() or
not needed in the test script.  Remove all calls to get_lustre_env()
in the test scripts since this is called in init_test_env().

Move all definitions of ALWAYS_EXCEPT and SLOW to after
init_test_env() and init_logging() and call build_test_filter()
immediately after the ALWAYS_EXCEPT and SLOW definitions.

Test-Parameters: trivial
Test-Parameters: testgroup=review-dne-part-4
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I9d2a7e6bedd2d66e5ee564405b86b6206226769f
Reviewed-on: https://review.whamcloud.com/35255
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12446 tests: Fix use of undefined variable under sanityn.sh/16b 46/35246/5
Arshad Hussain [Mon, 17 Jun 2019 18:49:02 +0000 (00:19 +0530)]
LU-12446 tests: Fix use of undefined variable under sanityn.sh/16b

Under sanityn.sh/16b variable STRIPE_BYTES is used to define
the block size within 'dd' command. However STRIPE_BYTES is
used undefined. This results in 'dd' command failing silently.
Although this does not affect the outcome of the test, the 'dd'
command used this way is no-op.

This patch fixes use of undefined variable under sanityn.sh:16b
by replacing variable STRIPE_BYTES with variable stripe_size.

Test-Parameters: trivial testlist=sanityn
Change-Id: I8bc947d93e339573759d5d37e800aae9bc3b4b18
Signed-off-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-on: https://review.whamcloud.com/35246
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
13 months agoLU-12424 lnet: prevent loop in LNetPrimaryNID() 91/35191/6
Amir Shehata [Tue, 11 Jun 2019 18:25:27 +0000 (11:25 -0700)]
LU-12424 lnet: prevent loop in LNetPrimaryNID()

If discovery is disabled locally or at the remote end, then attempt
discovery only once. Do not update the internal database when
discovery is disabled and do not repeat discovery.

This change prevents LNet from getting hung waiting for
discovery to complete.

Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: I4543b0f71e6cf297a1a5f058ebcc6bf74b8ac328
Reviewed-on: https://review.whamcloud.com/35191
Reviewed-by: Olaf Weber <olaf.weber@hpe.com>
Tested-by: Jenkins
Reviewed-by: Chris Horn <hornc@cray.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-11673 tests: add space before ']' in test-framework 79/35079/4
James Nunez [Thu, 6 Jun 2019 13:48:13 +0000 (07:48 -0600)]
LU-11673 tests: add space before ']' in test-framework

The test command '[' expects spaces before all arguments
including the closing ']'.

Add a space before the closing ']' in the function
print_summary() in test-framework.sh.

Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: If2365cb5f2b9c003949c6224997644c61341fe35
Reviewed-on: https://review.whamcloud.com/35079
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
13 months agoLU-8066 obd: collect all resource releasing for obj_type. 16/34716/4
NeilBrown [Wed, 5 Jun 2019 16:36:23 +0000 (12:36 -0400)]
LU-8066 obd: collect all resource releasing for obj_type.

Now that obj_type is managed as a kobject, move all
the freeing and deregistering into class_sysfs_release().

Change-Id: I784287ea17e010206b5fa256c7a224d01085be92
Signed-off-by: NeilBrown <neilb@suse.com>
Reviewed-on: https://review.whamcloud.com/34716
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-11838 scrub: handle s_uuid change to uuid_t 89/34689/8
James Simmons [Thu, 20 Jun 2019 20:57:01 +0000 (16:57 -0400)]
LU-11838 scrub: handle s_uuid change to uuid_t

The 4.12 kernel changed the s_uuid field in struct super_block from
an character array to an uuid_t. While ldiskfs uses it own s_uuid
field in struct ext4_super_block that field is a char array instead
of an uuid. Currently on going effort are being down in the linux
kernel to move to uuid_t so I suspect this will change in the future.
Since this is the case change all the character arrays for uuid
handling to uuid_t located in the scrubbing code. Change osd-ldiskfs
to use the struct super_block uuid, which is equivalent to s_es
version, to handle the uuid_t changes now.

Change-Id: I40643d342b5bc17a6ef922e99b3e8524930822de
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/34689
Tested-by: Jenkins
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
13 months agoLU-9859 libcfs: replace cfs_get_random_bytes calls with get_random_byte() 34/35234/4
NeilBrown [Sat, 15 Jun 2019 01:21:15 +0000 (21:21 -0400)]
LU-9859 libcfs: replace cfs_get_random_bytes calls with get_random_byte()

The cfs_get_random_bytes() interface adds nothing of value
to get_random_byte() (which it uses internally).  So just use the
standard interface.

Linux-commit: e904f839cdb04d1b314753a83a6e58146e315c66

Change-Id: I48e153d7658f0f616afe4e884faeb09c2dbdcd03
Signed-off-by: NeilBrown <neilb@suse.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-on: https://review.whamcloud.com/35234
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-9859 libcfs: deprecate libcfs_debug_vmsg2 24/35224/4
NeilBrown [Fri, 14 Jun 2019 01:56:34 +0000 (21:56 -0400)]
LU-9859 libcfs: deprecate libcfs_debug_vmsg2

Since 2.6.36, Linux' vsprintf has supported %pV
which supports "recursive sprintf" - exactly the task
that libcfs_debug_vmsg2 aims to provide.

Instead of calling libcfs_debug_vmsg2(), we can put the fmt and
args in a 'struct va_format', and pass the address of that structure
to the "%pV" format.

So do this to remove all users of libcfs_debug_vmsg2().

Linux-commit: 0fe922e1eca8e2850f0e6c535a14ba7414ca73c2

Change-Id: I6952ca8fdb619423639734aab1a30f4635b089cc
Signed-off-by: NeilBrown <neilb@suse.com>
Reviewed-on: https://review.whamcloud.com/35224
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <hornc@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12226 doc: recommend e2fsprogs 1.45.2.wc1 02/35202/2
Li Dongyang [Wed, 12 Jun 2019 06:17:00 +0000 (16:17 +1000)]
LU-12226 doc: recommend e2fsprogs 1.45.2.wc1

Update the recommended e2fsprogs version to 1.45.2.wc1

Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Change-Id: I0eea35a0bcc24a6109d0c90254e9d071f70e8e9d
Reviewed-on: https://review.whamcloud.com/35202
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
13 months agoLU-8066 tests: use lod / osp tunables on servers 85/35185/2
James Simmons [Tue, 11 Jun 2019 15:39:17 +0000 (11:39 -0400)]
LU-8066 tests: use lod / osp tunables on servers

Before the lustre 2.4 OSD work the lov and osc code was used on
both servers and clients. With the OSD layer work we saw the new
lod and osp layers created that are server specific. To avoid
breakage symlinks were created that went from the lod / osp to
lov / osc directories in the proc tree on the server side.

It has been a very long time since that change so we can now
safely start to unwind that handling. The first step taken here
is to migrate the maloo test from using lov / osc for the server
tunables to using lod / osp instead.

Change-Id: I9dd562cd74d68aaa0226d5ab93042b52193604a1
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/35185
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12420 utils: llog_reader handles uninitialized mountdata 78/35178/2
Li Xi [Tue, 11 Jun 2019 12:28:30 +0000 (20:28 +0800)]
LU-12420 utils: llog_reader handles uninitialized mountdata

When reading an mountdata that has never been used, "llog_reader
CONFIGS/mountdata" command crashes with following output:

Header size : 500170753
Time : Wed Sep  4 00:57:37 6869
Number of records: 65534
Target uuid :
-----------------------
Segmentation fault

After apply this patch, llog_reader will print following message
and quit under this circumstance:

Header size : 500170753
Time : Wed Sep  4 00:57:37 6869
Number of records: 65534
Target uuid :
-----------------------
uninitialized llog record at index 0

Change-Id: I25147f7fd09c6d59ff0049bdb20ac1979cf43ee4
Signed-off-by: Li Xi <lixi@ddn.com>
Reviewed-on: https://review.whamcloud.com/35178
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12420 utils: llog_reader handles uninitialized llog properly 77/35177/3
Li Xi [Tue, 11 Jun 2019 08:26:22 +0000 (16:26 +0800)]
LU-12420 utils: llog_reader handles uninitialized llog properly

When reading an empty LLOG, llog_reader would crash because
of record number of zero. E.g. "llog_reader CONFIGS/nodemap" on
a MGS without nodemap configuration would cause failure of:

llog_reader: Error allocating -16 bytes for recs_buf: Cannot allocate memory (12)
llog_reader: Could not pack buffer.: Cannot allocate memory (12)

After apply this patch, llog_reader will print following message
and quit if the LLOG is unintialized:

uninitialized llog: zero record number

Change-Id: I87246672e9fc992c99126134236c2e8d304df74b
Signed-off-by: Li Xi <lixi@ddn.com>
Reviewed-on: https://review.whamcloud.com/35177
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12382 llite: fix deadloop with tiny write 58/35058/7
Wang Shilong [Tue, 4 Jun 2019 12:54:01 +0000 (20:54 +0800)]
LU-12382 llite: fix deadloop with tiny write

For a small write(<4K), we will use tiny write and
__generic_file_write_iter() will be called to handle it.

On newer kernel(4.14 etc), the function is exported and will
do something like following:

|->__generic_file_write_iter
  |->generic_perform_write()

If iov_iter_count() passed in is 0, generic_write_perform() will
try go to forever loop as bytes copied is always calculated as 0.

The problem is VFS doesn't always skip IO count zero before it comes
to lower layer read/write hook, and we should do it by ourselves.

To fix this problem, always return 0 early if there is no
real any IO needed.

Change-Id: I765a723da79eb5fd09317c3fad47fe479b1dd4fb
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/35058
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jenkins
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12330 obdclass: allow per-session jobids. 95/34995/9
NeilBrown [Thu, 23 May 2019 00:15:43 +0000 (10:15 +1000)]
LU-12330 obdclass: allow per-session jobids.

Lustre includes a jobid in all RPC message sent to the server.  This
is used to collected per-job statistics, where a "job" can involve
multiple processes on multiple nodes in a cluster.

Nodes in a cluster can be running processes for multiple jobs, so it
is best if different processes can have different jobids, and that
processes on different nodes can have the same job id.

The current mechanism for supporting this is to use an environment
variable which the kernel extracts from the relevant process's address
space.  Some kernel developers see this to be an unacceptable design
choice, and the code is not likely to be accepted upstream.

This patch provides an alternate method, leveraging the concept of a
"session id", as set with setsid().  Each login session already gets a
unique sid which is preserved for all processes in that session unless
explicitly changed (with setsid(1)).
When a process in a session writes to
     /sys/fs/lustre/jobid_this_session
the string becomes the name for that session.
If jobid_var is set to "session", then the per-session jobid is used
for the jobid for all requests from processes in that session.

When a session ends, the jobid information will be purged within 5
minutes.

Change-Id: I6fb1a75f8f60f824e402706de0b1439464bfa05c
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Reviewed-on: https://review.whamcloud.com/34995
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12043 llite: improve single-thread read performance 95/34095/35
Wang Shilong [Mon, 21 Jan 2019 12:23:47 +0000 (20:23 +0800)]
LU-12043 llite: improve single-thread read performance

Here is whole history:

Currently, for sequential read IO, We grow up window
size very quickly, and once we cached @max_readahead_per_file
pages. For following command:

  dd if=/mnt/lustre/file of=/dev/null bs=1M

We will do something like following:
...
64M bytes cached.
fast io for 16M bytes
readahead extra 16M to fill up window.
fast io for 16M bytes
readahead extra 16M to fill up window.
....

In this way, we could only use fast IO for 16M bytes and
then fall through non-fast IO mode. this is also reason
that why increasing @max_readahead_per_file don't give us
performances up, since this value only changes how much
memory we cached in memory, during my testing whatever
I changed the value, i could only get 2GB/s for single thread
read.

Actually, we could do this better, if we have used
more than 16M bytes readahead pages, submit another readahead
requests in the background. and ideally, we could always
use fast IO.

Test Patched Unpatched
dd if=file of=/dev/null bs=1M.   4.0G/s 1.9G/s
ior -np 192 r -t 1m -b 4g -F -e -vv -o /cache1/ior -k 11195.97 10817.02 MB/sec

Tested with drop OSS and client memory before every run.
max_readahead_per_mb=128M, RPC size is 16M.
dd file's size is 400G which is double of memory or so.

Change-Id: I9b6be078ca24c256198488a9c1635791dafbd7e7
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/34095
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12043 llite,readahead: don't always use max RPC size 33/35033/4
Wang Shilong [Sun, 2 Jun 2019 15:17:26 +0000 (23:17 +0800)]
LU-12043 llite,readahead: don't always use max RPC size

Since 64M RPC landed, @PTLRPC_MAX_BRW_PAGES will be 64M.
And we always try to use this max possible RPC size to check
whether we should avoid fast IO and trigger real context IO.

This is not good for following reasons:

(1) Since current default RPC size is still 4M,
most of system won't use 64M for most of time.

(2) Currently default readahead size per file is still 64M,
which makes fast IO always run out of all readahead pages
before next IO. This breaks what users really want for readahead
grapping pages in advance.

To fix this problem, we use 16M as a balance value if RPC smaller
than 16M, patch also fix the problem that @ras_rpc_size could not
grow bigger which is possibe in the following case:

1) set RPC to 16M
2) Set RPC to 64M

In the current logic ras->ras_rpc_size will be kept as 16M which is wrong.

Change-Id: Ida9f839f7c692cd88d32dc0909503f6ae991d909
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/35033
Tested-by: Jenkins
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-9010 lnet: Change static defines to use macro for module.c 32/33932/12
Arshad Hussain [Thu, 27 Dec 2018 12:26:17 +0000 (07:26 -0500)]
LU-9010 lnet: Change static defines to use macro for module.c

This patch replaces mutex which are defined statically
in file lnet/lnet/module.c with kernel provided macro.

Signed-off-by: Arshad Hussain <arshad.super@gmail.com>
Change-Id: I59de4514dc332c3c59e0d816720a81394521881c
Reviewed-on: https://review.whamcloud.com/33932
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-11775 osc: reduce lock contention in osc_unreserve_grant 58/33858/6
Li Dongyang [Fri, 14 Dec 2018 01:19:09 +0000 (12:19 +1100)]
LU-11775 osc: reduce lock contention in osc_unreserve_grant

In osc_queue_async_io() the cl_loi_list_lock is acquired to reserve
and consume the grant and released, right after we expand the extent
the same lock is used to unreserve the grant.
We can keep the spinlock when we are done with the grant to improve
the throughput.

mpirun  -np 32 /root/ior-openmpi/src/ior -w -t 1m -b 8g -F -e -vv
-o /scratch0/file -i 1
master:
Max Write: 13799.70 MiB/sec (14470.04 MB/sec)
master with 33858:
Max Write: 14339.57 MiB/sec (15036.13 MB/sec)

Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Change-Id: Ic61af84c7b98b5a189d7adabe33ae687954b2ed4
Reviewed-on: https://review.whamcloud.com/33858
Tested-by: Jenkins
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Alexey Lyashkov <c17817@cray.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12411 lnet: Do not allow gateways on remote nets 98/35198/2
Chris Horn [Tue, 11 Jun 2019 19:59:31 +0000 (14:59 -0500)]
LU-12411 lnet: Do not allow gateways on remote nets

A gateway needs to be reachable over some local interface.

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: Ib66d4f8fd48d8863097280c480648ab8e29d2767
Reviewed-on: https://review.whamcloud.com/35198
Tested-by: Jenkins
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sonia Sharma <sharmaso@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-10070 ldlm: layout lock fixes 32/35232/7
Vitaly Fertman [Tue, 11 Jun 2019 23:18:45 +0000 (02:18 +0300)]
LU-10070 ldlm: layout lock fixes

as the intent_layout operation becomes more frequent with SEL,
cancel existent layout locks in advance and reuse ELC to deliver
cancels to MDS

as clients are given LCK_EX layout locks, take into account this
mode as well in ldlm_lock_match

Cray-bug-id: LUS-2528
Signed-off-by: Vitaly Fertman <c17818@cray.com>
Change-Id: I1525153b3a07385fc17ef5416ded7b6d4378b2ec
Reviewed-on: https://review.whamcloud.com/35232
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12447 utils: specify correct size for lfs project buffer 57/35257/2
Wang Shilong [Tue, 18 Jun 2019 00:38:17 +0000 (20:38 -0400)]
LU-12447 utils: specify correct size for lfs project buffer

Enviorment:
Fedora release 28 (Twenty Eight)

gcc (GCC) 8.0.1 20180324 (Red Hat 8.0.1-0.20)
Copyright (C) 2018 Free Software Foundation, Inc.

Hit build failure:
lfs_project.c: In function ‘lfs_project_item_alloc’:
lfs_project.c:72:2: error: ‘strncpy’ specified bound 4096
equals destination size [-Werror=stringop-truncation]
  strncpy(lpi->lpi_pathname, pathname, sizeof(lpi->lpi_pathname));
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Test-Parameters: trivial testlist=sanity-quota
Change-Id: Ia6429c47391bf503546609ec6a262fe24664bdd4
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/35257
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Gu Zheng <gzheng@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12392 utils: specify correct size for the buffer 70/35070/6
Alex Zhuravlev [Wed, 5 Jun 2019 13:38:27 +0000 (16:38 +0300)]
LU-12392 utils: specify correct size for the buffer

otherwise gcc8 makes a warning which interrupts build.

Change-Id: I6a94c6cd63473df9fc88b1867bbda1353fa10247
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/35070
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
13 months agoLU-12355 ldiskfs: bio_phys_segments symbol is not exported 40/35040/3
Shaun Tancheff [Sat, 8 Jun 2019 17:05:04 +0000 (12:05 -0500)]
LU-12355 ldiskfs: bio_phys_segments symbol is not exported

As of kenrel 5.0 bio_phys_segments not exported
It is only used in one CDEBUG(D_INODE so use bio->bi_phys_segments
directly.

Linux-commit: 6c210aa596d0ecf6f3eea65c02ac807877385a18

Test-Parameters: trivial
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Change-Id: I19cf7cab86ccebe4fccf7a34a945a4150069d18b
Reviewed-on: https://review.whamcloud.com/35040
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-8066 obd: cleanup server sysfs symlinks handling 15/34715/14
James Simmons [Wed, 5 Jun 2019 16:32:38 +0000 (12:32 -0400)]
LU-8066 obd: cleanup server sysfs symlinks handling

Rename class_setup_tunables() to class_add_symlinks(). Move all the
special sysfs and debugfs symlink handling into the function
class_add_symlinks(). Now that the obd_type is created using sysfs
handling we can complete the initializion if the real obd device is
registered later. For example if lod is registered first and it
creates the the "lov" obd_type. Then if the lov module is loaded
later then class_register_type() will use the "lov" obd_type created
by the lod module.

Change-Id: I754ec15a88458b170422b988d783efbe20141b87
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/34715
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12333 ptlrpc: Add more flags to DEBUG_REQ_FLAGS macro 90/35090/5
Vitaly Fertman [Wed, 5 Jun 2019 21:17:34 +0000 (00:17 +0300)]
LU-12333 ptlrpc: Add more flags to DEBUG_REQ_FLAGS macro

Add rq_no_reply flag to the DEBUG_REQ_FLAGS macro for debug purposes
Also, add another debug message to check_write_rcs

Test-Parameters: trivial
Signed-off-by: Vitaly Fertman <c17818@cray.com>
Change-Id: I39ea7e9359a377ad46f7600edad14375f9935793
Reviewed-on: https://review.whamcloud.com/35090
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Chris Horn <hornc@cray.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
13 months agoLU-11264 llapi: reduce llapi_stripe_limit_check() overhead 91/35091/2
Andreas Dilger [Fri, 7 Jun 2019 03:34:49 +0000 (21:34 -0600)]
LU-11264 llapi: reduce llapi_stripe_limit_check() overhead

There is no need to check PAGE_SIZE in llapi_stripe_limit_check()
every time, since this cannot change between calls.

Always set errno if an error is returned.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ib377c1cc734c9e683f75eeb509e220c4ea3ebbe5
Reviewed-on: https://review.whamcloud.com/35091
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-11089 obd: rename lu_keys_guard to lu_context_remembered_guard 73/33673/8
NeilBrown [Wed, 5 Jun 2019 13:26:36 +0000 (09:26 -0400)]
LU-11089 obd: rename lu_keys_guard to lu_context_remembered_guard

The only remaining use of lu_keys_guard is to protect the
lu_context_remembers linked list, and always write_lock()
is used.
So rename it to reflect this, and change to a spinlock.
We move keys_fini() out of the locked region in
lu_context_fini() - once we have removed the context from
the lc_remembers list, there can no longer be a race.

Linux-commit: dc8aaaca0062878c2fbad9df1b9ac3e85cad8630

Change-Id: Id66930e073d5351a96b139f2fc1a8007841de728
Signed-off-by: NeilBrown <neilb@suse.com>
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/33673
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Gu Zheng <gzheng@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-11213 lod: default LMV can't be deleted 07/35207/3
Lai Siyao [Wed, 12 Jun 2019 10:33:12 +0000 (18:33 +0800)]
LU-11213 lod: default LMV can't be deleted

When 'space' hash type was introduced, default LMV deletion added
check for hash type, but it only checks whether type is 'none',
while by default it's 'fnv_1a_64', which caused default LMV can't
be deleted.

Change check to !LMV_HASH_TYPE_SPACE and update test 413b.

Fixes: a24f6153292 ("LU-11213 dne: add new dir hash type 'space'")

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I88c630dc8d339ddeb9dc03d6f8987d8783062a13
Reviewed-on: https://review.whamcloud.com/35207
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-11213 doc: update lfs-setdirstripe man 'space' hash 08/35208/3
Lai Siyao [Wed, 12 Jun 2019 10:54:06 +0000 (18:54 +0800)]
LU-11213 doc: update lfs-setdirstripe man 'space' hash

Add 'space' hash description for lfs-setdirstripe man.

Test-Parameters: trivial
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I847618c5c98925ebc7b1bd124173ffb5adfe90dd
Reviewed-on: https://review.whamcloud.com/35208
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Joseph Gmitter <jgmitter@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12355 lnet: ib_fmr_pool_unmap returns void 17/35017/3
Shaun Tancheff [Thu, 13 Jun 2019 19:04:54 +0000 (14:04 -0500)]
LU-12355 lnet: ib_fmr_pool_unmap returns void

Historically ib_fmr_pool_unmap only ever returned 0
Linux kernel 4.20 changed the return for ib_fmr_pool_unmap to void.

Linux-commit: 3eeeb7a59acddaa326b03efdf6dce61c120449a3

Test-Parameters: trivial
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Change-Id: I49d91a49c452dad5c7d9b153fdbc011f2f25743a
Reviewed-on: https://review.whamcloud.com/35017
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Chris Horn <hornc@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12355 llite: vfs atomic_open change with FMODE_CREATED 20/35020/6
Shaun Tancheff [Thu, 13 Jun 2019 13:57:33 +0000 (08:57 -0500)]
LU-12355 llite: vfs atomic_open change with FMODE_CREATED

Kernel 4.19 introduced FMODE_CREATED and switched to it while
the last argument to vfs atomic_open was removed and the f_mode
flags are used to indicate the created state on return.

Linux-commit: 73a09dd94377e4b186b300bd5461920710c7c3d5

Test-Parameters: trivial
Change-Id: I26d4aadb123bb1d1bc0aa1d78a64a75b94276ffb
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-on: https://review.whamcloud.com/35020
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Thomas Stibor <t.stibor@gsi.de>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12355 ldiskfs: Added ext4_iget_flags to ext4_iget 23/35023/5
Shaun Tancheff [Thu, 13 Jun 2019 14:14:41 +0000 (09:14 -0500)]
LU-12355 ldiskfs: Added ext4_iget_flags to ext4_iget

Kernel 4.19 introduced ext4_iget_flags and changed ext4_iget
to require a flags argument.

Use EXT4_IGET_HANDLE for stale inode checking by ext4/ldiskfs

Linux-commit: 8a363970d1dc38c4ec4ad575c862f776f468d057

Test-Parameters: trivial
Change-Id: Ic6cbe3eabcdbb1a506813d145fb2980dbed95e66
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-on: https://review.whamcloud.com/35023
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12355 osd-ldiskfs: timespec_trunc removed 50/35050/7
Shaun Tancheff [Thu, 13 Jun 2019 18:27:19 +0000 (13:27 -0500)]
LU-12355 osd-ldiskfs: timespec_trunc removed

As part of the y2038 changes:

timespec64_trunc was added in kernel 4.17
inode i_Xtime values are timespec64 in kernel 4.18
timespec_trunc was removed in kernel 4.19

Harden the existing LC_INODE_TIMESPEC64 check and stop
using the deprecated timespec_trunc().

oti_time is set but never used, remove oti_time.

Linux-commit: 8efd6894ff089adeeac7cb9f32125b85d963d1bc
Linux-commit: 95582b00838837fc07e042979320caf917ce3fe6
Linux-commit: 976516404ff3fab2a8caa8bd6f5efc1437fed0b8

Test-Parameters: trivial
Change-Id: Idadbf6ddd9aedbd251d7bbbb5b1486d4aa757ac5
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-on: https://review.whamcloud.com/35050
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
13 months agoLU-12430 build: remove misleading help message 05/35205/3
Li Xi [Wed, 12 Jun 2019 10:41:31 +0000 (18:41 +0800)]
LU-12430 build: remove misleading help message

build/README.kernel-source has been deleted for nine years, but
the build help message still says: "Consult build/README.kernel-source
for details."

This patch removes these messages.

Test-Parameters: trivial
Change-Id: I2ff3afc363205b13f7f5ef645ade2854bfc3fa47
Signed-off-by: Li Xi <lixi@ddn.com>
Reviewed-on: https://review.whamcloud.com/35205
Reviewed-by: Gu Zheng <gzheng@ddn.com>
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
13 months agoLU-12436 lov: return error if cl_env_get fails 29/35229/2
Shaun Tancheff [Thu, 13 Jun 2019 22:51:38 +0000 (17:51 -0500)]
LU-12436 lov: return error if cl_env_get fails

When cl_env_get() fails with an error return the error.

Test-Parameters: trivial
Cray-bug-id: LUS-7310
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Change-Id: Ia065aeb142a772f4d620b84111af423e27c06b90
Reviewed-on: https://review.whamcloud.com/35229
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-11434 tests: add version check conf-sanity 109a/b 54/33954/17
James Nunez [Wed, 2 Jan 2019 23:02:29 +0000 (16:02 -0700)]
LU-11434 tests: add version check conf-sanity 109a/b

conf-sanity test 109a and 109b were added to Lustre with tag 2.10.59.
Thus, we need to check that the server is 2.10.59 or later before
running conf-sanity test 109a and 109b.

Test-Parameters: trivial envdefinitions=ONLY=109 serverjob=lustre-b2_10 serverbuildno=170 testlist=conf-sanity
Test-Parameters: envdefinitions=ONLY=109 testlist=conf-sanity
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I4a653d1374973f00d283af885183621ec14628e1
Reviewed-on: https://review.whamcloud.com/33954
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoNew tag 2.12.55 2.12.55 v2_12_55
Oleg Drokin [Sun, 16 Jun 2019 03:38:29 +0000 (23:38 -0400)]
New tag 2.12.55

Change-Id: I6e76cc7c06092f54b778f6e45932e21427991dcf

13 months agoLU-12395 build: require python2 for lustre-iokit 94/35094/2
Li Dongyang [Fri, 7 Jun 2019 07:34:23 +0000 (17:34 +1000)]
LU-12395 build: require python2 for lustre-iokit

RHEL8 has splitting python2 and python3 rpms,
and none of them provdes python anymore.
We can just require python2 in the spec, other
distros all have python rpm providing python2.

Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Change-Id: I881c90a4e66d1a431d11d16b9e89781de2f87a7d
Reviewed-on: https://review.whamcloud.com/35094
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12383 utils: only check project inherit bit for dir 76/35076/2
Wang Shilong [Thu, 6 Jun 2019 02:36:39 +0000 (10:36 +0800)]
LU-12383 utils: only check project inherit bit for dir

Currently, ZFS won't set inherit bit on regular files, but
ext4 always set it, it doesn't make sense for regular files
have this bit, but own it won't do any harm as well.

To make test happy and give a consistent view on users,
let's fix project check only complain erros for Direcotry.

Test-Parameters: trivial testlist=sanity-quota
Change-Id: I194f3ed9d6ded69313a683995295ab8c07b4fb3a
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/35076
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12267 tests: filter trailing '.' for SELinux 26/35026/6
James Nunez [Fri, 31 May 2019 21:28:20 +0000 (15:28 -0600)]
LU-12267 tests: filter trailing '.' for SELinux

When SELinux is enforced, sanity test 420 fails due to
the "ls -n" command producing an extra '.' at the end of
the file/directory permissions to indicate extra security
attributes are set.

We need to filter out the trailing '.' in the 'ls -n'
output for testing to pass when SELinux is enabled.

Test-Parameters: trivial envdefinitions=ONLY=420 testlist=sanity
Test-Parameters: clientselinux envdefinitions=ONLY=420 testlist=sanity
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I4a2f199d2ef4a7b1b6a1b381041b384bb0077cc6
Reviewed-on: https://review.whamcloud.com/35026
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
13 months agoLU-12399 tests: avoid 'pdsh localhost' in sanity test_420 76/35176/2
Sebastien Buisson [Tue, 11 Jun 2019 09:50:01 +0000 (11:50 +0200)]
LU-12399 tests: avoid 'pdsh localhost' in sanity test_420

sanity test_420 needs a clean env to execute openfile, ie not
inherited from root user.
Replace 'pdsh localhost' with simpler 'su - $uname -c' alternative
to achieve this.

Test-Parameters: trivial envdefinitions=ONLY=420 testlist=sanity
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ifeba7fc1eba86d74a64cca187e286adb23147e2e
Reviewed-on: https://review.whamcloud.com/35176
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
13 months agoLU-12412 recovery: wake all waiters of trd_finishing 41/35141/2
Sergey Cheremencev [Wed, 20 Jan 2016 20:57:01 +0000 (23:57 +0300)]
LU-12412 recovery: wake all waiters of trd_finishing

There is a small window where lctl --device abort_recovery
and umount->...->stop_recovery_thread may be called before
recovery finish. In such case all threads need to be
waked up, so change complete to complete_all.

Cray-bug-id: LUS-2000
Change-Id: I01ef163e72c7691a2c2d5449adf55b55ec734c4d
Signed-off-by: Sergey Cheremencev <c17829@cray.com>
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Vladimir Saveliev <c17830@cray.com>
Reviewed-on: https://review.whamcloud.com/35141
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12396 utils: lfs should not output 'nul' char 37/35137/3
Patrick Farrell [Mon, 10 Jun 2019 16:29:06 +0000 (12:29 -0400)]
LU-12396 utils: lfs should not output 'nul' char

If lfs prints a nul char, it breaks parsing of the output.

Fixes: 68635c3d9b31 ("LU-11963 osd: Add nonrotational flag to statfs")
Signed-off-by: Yang Sheng <ys@whamcloud.com>
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Ibfa77920adf3a6c62b01efb005d02ca81db7f7c1
Reviewed-on: https://review.whamcloud.com/35137
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
13 months agoLU-12355 lnet: Adjust checks for ib_device_ops 16/35016/4
Shaun Tancheff [Tue, 11 Jun 2019 12:29:49 +0000 (07:29 -0500)]
LU-12355 lnet: Adjust checks for ib_device_ops

RDMA/core: Introduce ib_device_ops

The ib_device_ops structure defines all the InfiniBand device
operations in one place

Linux-commit: 521ed0d92ab0db3edd17a5f4716b7f698f4fce61

Test-Parameters: trivial
Change-Id: Ia2a617597c75ec819f485b93a1deb368d4b5e873
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-on: https://review.whamcloud.com/35016
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Chris Horn <hornc@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12381 ko2iblnd: ignore down interfaces 98/35098/4
James Simmons [Mon, 10 Jun 2019 13:58:29 +0000 (09:58 -0400)]
LU-12381 ko2iblnd: ignore down interfaces

The for_each_netdev() loop in kiblnd_create_dev() scans for all
network devices on a system. Currently the code exit when an
network device is down but the device could be something besides
an IB device. Instead of exiting just ignore any device that is
down.

Test-Parameters: trivial

Fixes: c4b39bf56bbc ("LU-11893 o2iblnd: add secondary IP address handling")
Change-Id: I0a3bf808d849cd00711b6ef2e4e5bbd876b64903
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/35098
Tested-by: Jenkins
Reviewed-by: Chris Horn <hornc@cray.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-1538 tests: standardize test script init - sanity 63/34863/3
Andreas Dilger [Mon, 3 Jun 2019 14:39:19 +0000 (08:39 -0600)]
LU-1538 tests: standardize test script init - sanity

Standardize the initial Lustre test script initialization of the
test-framework.sh for clarity and consistency.

The LUSTRE path is already normalized in init_test_env(), so this
doesn't need to be done in the caller.  Use $(...) subshells instead
of `...` in the affected lines.  Remove PATH, NAME, TMP, LFS, LCTL
variable initialization, since it is already done in init_test_env().

Move MACHINEFILE into init_test_env().

Move get_lustre_env() to the end of init_test_env(). All test scripts
currently call init_test_env() and this move will allow all test
scripts to use the variables defined in get_lustre_env() without
having to modify the individual test scripts.

Move all definitions of ALWAYS_EXCEPT to after init_test_env()
and init_logging() and call build_test_filter() immediately
after these and SLOW definitions.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I1ef6639bcb3eb5179bd44da13b35fd843c267156
Reviewed-on: https://review.whamcloud.com/34863
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
13 months agoLU-11758 osp: remove assertion from statfs 32/33832/6
Sergey Cheremencev [Fri, 6 Jul 2018 19:51:14 +0000 (22:51 +0300)]
LU-11758 osp: remove assertion from statfs

Sequence can't be changed or overflowed
in case of IDIF. Thus don't tigger kernel
panic for below case:
last_created [0x100000001:0x15:0x0], next_fid [0x100000000:0xfffffff6:0x0]
The same assertion that excepts IDIFs exists
in osp_fid_diff.
Also the patch is adding several optimizations
in osp_precreate_send.

Change-Id: I3966dfc621999d065c9b485d387938085fccb140
Cray-bug-id: LUS-2386
Signed-off-by: Sergey Cheremencev <c17829@cray.com>
Reviewed-on: https://es-gerrit.dev.cray.com/153571
Reviewed-by: Alexander Boyko <c17825@cray.com>
Reviewed-by: Alexey Lyashkov <c17817@cray.com>
Reviewed-on: https://review.whamcloud.com/33832
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexandr Boyko <c17825@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-9846 test: a test number fix 89/35089/5
Vitaly Fertman [Mon, 3 Jun 2019 16:30:06 +0000 (19:30 +0300)]
LU-9846 test: a test number fix

A wrong test number was specified originally

Test-Parameters: trivial
Signed-off-by: Vitaly Fertman <c17818@cray.com>
Change-Id: I8ea31bb2e613c6e225fa7f41f405d5ee2d396a61
Reviewed-on: https://review.whamcloud.com/35089
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-10948 mdt: Remove openlock compat code with 2.1 39/35039/4
Oleg Drokin [Mon, 3 Jun 2019 06:39:41 +0000 (02:39 -0400)]
LU-10948 mdt: Remove openlock compat code with 2.1

Checking openlock when doing a create does not allow us to create
a file if we want to also get openlock from it right away.

Since 2.1 is no longer something we care about wrt compatibility,
ok to kill it.

Change-Id: Ic4327be5c45ae856dfbe20291a59c5b1654dbf8f
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/35039
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
13 months agoLU-12387 tests: Validate l_tunedisk max_sectors_kb tuning 81/35081/10
Chris Horn [Thu, 6 Jun 2019 15:59:18 +0000 (10:59 -0500)]
LU-12387 tests: Validate l_tunedisk max_sectors_kb tuning

Add test to ensure that l_tunedisk only tunes the max_sectors_kb
value of OST devices, and that it properly tunes any slave devices.

Test-parameters: trivial
Test-parameters: fstype=ldiskfs testlist=conf-sanity \
 envdefinitions=ONLY=125
Cray-bug-id: LUS-7358
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I414526e71fd7ac2811d7c0e8a6afd80a50788258
Reviewed-on: https://review.whamcloud.com/35081
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Gu Zheng <gzheng@ddn.com>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-12387 utils: Avoid passing symlink to tune_block_dev 65/35065/4
Chris Horn [Wed, 5 Jun 2019 00:14:47 +0000 (19:14 -0500)]
LU-12387 utils: Avoid passing symlink to tune_block_dev

In tune_block_dev_slaves we iterate over the directories inside the
slaves subdirectory for the multipath device that is being tuned. For
example:

 $ /usr/sbin/l_tunedisk /dev/mapper/mpathc

Suppose mpathc maps to /dev/dm-2. tune_block_dev will initially set
the value of
/sys/devices/virtual/block/dm-2/queue/max_sectors_kb
equal to the value of
/sys/devices/virtual/block/dm-2/queue/max_hw_sectors_kb

Then it looks at the entries in /sys/devices/virtual/block/dm-2/slaves
Suppose the slave devices are as follows:

 $ ls /sys/devices/virtual/block/dm-2/slaves
 sdc  sdh  sdm  sdr
 $

It then calls tune_block_dev recursively, passing
/sys/devices/virtual/block/dm-2/slaves/sdc,
/sys/devices/virtual/block/dm-2/slaves/sdh, etc. However, these are
symlinks that point to directories and as such tune_block_dev will not
tune them because stat does not identify them as block devices.

Instead we should contruct the path argument for these recursive calls
as /dev/<d_name>. In this example, /dev/sdc, /dev/sdh, etc.

Cray-bug-id: LUS-7358
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I63bc073a82384d68648ff23a56b7d43d6656159b
Reviewed-on: https://review.whamcloud.com/35065
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Gu Zheng <gzheng@ddn.com>
13 months agoLU-12387 utils: Read existing ldd data in l_tunedisk 66/35066/5
Chris Horn [Tue, 4 Jun 2019 19:34:01 +0000 (14:34 -0500)]
LU-12387 utils: Read existing ldd data in l_tunedisk

Read the lustre_disk_data from the device passed to l_tunedisk, so
we can determine whether the device is an MGT or MDT and thus skip
the tuning of the device.

Fixes: 892280742a2b ("LU-9551 utils: add l_tunedisk to fix disk tunings")
Fixes: 2f8d7b4679de ("LU-11736 utils: don't set max_sectors_kb on MDT/MGT")
Cray-bug-id: LUS-7358
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I193fe008d5777b0e83f2be9a500eaffb1d3ca615
Reviewed-on: https://review.whamcloud.com/35066
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Gu Zheng <gzheng@ddn.com>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
13 months agoLU-12375 scripts: Start lnet after opa 32/35032/2
Nathaniel Clark [Sun, 2 Jun 2019 13:50:53 +0000 (09:50 -0400)]
LU-12375 scripts: Start lnet after opa

Ensure ordering of lnet after opa for startup and lnet before opa on
shutdown.

Signed-off-by: Nathaniel Clark <nclark@whamcloud.com>
Change-Id: I4c2cad2381349f866bdc08e2a81e3d8990d8752e
Reviewed-on: https://review.whamcloud.com/35032
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Artur Novik <anovik@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-6142 ptlrpc: Fix style issues for pinger.c 01/34701/5
Arshad Hussain [Thu, 18 Apr 2019 13:42:17 +0000 (19:12 +0530)]
LU-6142 ptlrpc: Fix style issues for pinger.c

This patch fixes issues reported by checkpatch
for file lustre/ptlrpc/pinger.c

Change-Id: I048a7ab7d31bc468a410ec1704c5d79a34feebb4
Signed-off-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-on: https://review.whamcloud.com/34701
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
13 months agoLU-11623 tests: Fix sanity 27E to ensure getattr RPC 67/35067/2
Oleg Drokin [Wed, 5 Jun 2019 06:28:24 +0000 (02:28 -0400)]
LU-11623 tests: Fix sanity 27E to ensure getattr RPC

While cat does perform fstat() on the file it opens,
I guess it's not guaranteed.
More importantly, we really need to ensure the locks
that the file has after creation are dropped before
we issue our stat() to ensure the RPC is actually made,
since it's this GETATTR RPC that is ensuring easize
update from MDT response.

Change-Id: Ic86229ac514e1385c665c6c0d9f6eef13d9748f5
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Test-Parameters: trivial
Reviewed-on: https://review.whamcloud.com/35067
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
13 months agoLU-12438 llite: vfs_read/write removed, use kernel_read/write 23/35223/5
Shaun Tancheff [Thu, 13 Jun 2019 18:48:10 +0000 (13:48 -0500)]
LU-12438 llite: vfs_read/write removed, use kernel_read/write

As of Linux 4.14 the vfs_read() is no longer available
to kernel modules. The kernel_read() function calls
vfs_read() and will continue to be available.

Adding a configure test to use kernel_read() as the
function signature changed in 4.14 to match the other file I/O
helpers.

Also remove vfs_write() in favor of kernel_write() wrapper
cfs_kernel_write().

Fixes: f172b1168857 ("LU-10092 llite: Add persistent cache on client")
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I5e5fce0e6644ba750169f3bf11ac5c98525da0a7
Reviewed-on: https://review.whamcloud.com/35223
Tested-by: Jenkins
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
13 months agoLU-10092 First phase of persistent client cache project merging 14/35214/1
Oleg Drokin [Thu, 13 Jun 2019 04:36:36 +0000 (00:36 -0400)]
LU-10092 First phase of persistent client cache project merging

Merge remote-tracking branch 'origin/pcc'

Change-Id: I87a681c54712926d336c983dd8e56b58ebf4b612
Signed-off-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-10092 pcc: change detach behavior and add keep option 44/33844/19
Qian Yingjin [Thu, 13 Dec 2018 02:41:14 +0000 (10:41 +0800)]
LU-10092 pcc: change detach behavior and add keep option

After introduce the feature of auto-attach at open, when the PCC
cached file is detach by "pcc detach" command, it will be attached
automatically at the next open. This may be not what the user wants.

To solve this problem, we change the defualt detach behavior and
add an option "--keep|-k" for the detach of RW-PCC.
The manual "lfs pcc detach" command will detach the file from PCC
permanently. And it will also remove the PCC copy by default.
When the file is detached with "keep" option, it only unmaps the
relationship between the file inode and PCC copy, but keep the
PCC copy. The file is allowed to be attached automatically at
the next open when the file is still valid in cache.

Note here that currently auto detach caused by inode reclaim or
revocation of the layout lock would not delete the PCC copy too.

Test-Parameters: clientcount=3 testlist=sanity-pcc,sanity-pcc,sanity-pcc
Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I010df54177ae4cfeddcc0a9982c1aee58ee683de
Reviewed-on: https://review.whamcloud.com/33844
Tested-by: Jenkins
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-10092 pcc: auto attach during open for valid cache 87/33787/21
Qian Yingjin [Wed, 5 Dec 2018 03:22:11 +0000 (11:22 +0800)]
LU-10092 pcc: auto attach during open for valid cache

In current PCC implementation, all PCC state information is
stored in the in-memory data structure named pcc_inode (a member
of data structure ll_inode_info). Once the file inode is reclaimed
due to the memory pressure or memory shrinking, the corresponding
in-memory pcc_inode will be released too, and the PCC-cached file
will be detached automatically. And the revocation of layout lock
will also trigger the detach of the PCC-cached file. These all lead
that the still valid PCC-cached file can not be used.

To solve this problem, we introduce an auto-attaching mechanism
during open. During PCC attach, the L.Gen will be stored as
extented attribute of the local copy file on PCC device. When the
in-memory inode is reclaimed or the layout lock is revoked, and
the file is opend again, it can check whether the stored L.Gen on
the PCC copy is same as the Lustre file current L.Gen on MDT. If
they are consistent, it means the cached copy on PCC device is still
valid, we can continue to use it after auto-attach.

Test-Parameters: testlist=sanity-pcc,sanity-pcc,sanity-pcc
Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I63be96f8d83816529983d0f97af0aaca81703fed
Reviewed-on: https://review.whamcloud.com/33787
Tested-by: Jenkins
Reviewed-by: Li Xi <lixi@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-10918 llite: Rule based auto PCC caching when create files 51/34751/15
Qian Yingjin [Wed, 24 Apr 2019 09:51:25 +0000 (17:51 +0800)]
LU-10918 llite: Rule based auto PCC caching when create files

Configurable rule based auto PCC caching for newly created files
can significantly benefit users for readwrite PCC. It can
determine which file can use a cache on PCC directly without any
admission control for high priority user/group/project or filename
with wildcard support. Meanwhile, we can enforce a quota limitation
of capacity usage for each user/group/project to providing caching
isolation.

Similar to NRS TBF command line, it supports logical conditional
conjunction and disjunction operations among different user/group/
project or filename with the wildcard support.

The command line to add this kind of rule is as follow:
lctl pcc add /mnt/lustre /mnt/pcc
"projid={500 1000}&fname={*.h5},uid={1001} rwid=1 roid=1"
It means that Project ID of 500, 1000 AND file suffix name is "h5"
OR User ID is 1001 can be auto cached on PCC for newly create file
on the client. "rwid" means RW-PCC attach ID (which is
usually archive ID); "roid" means RO-PCC attach ID. By defualt,
RO-PCC attach id is setting same with RW-PCC attach ID for a
shared PCC backend.

Test-Parameters: clientcount=3 testlist=sanity-pcc,sanity-pcc,sanity-pcc
Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I628975b3e097e98d6b93f1c6acd855aaacdaa8b3
Reviewed-on: https://review.whamcloud.com/34751
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Tested-by: Jenkins
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-10092 pcc: security and permission for non-root user access 37/34637/20
Qian Yingjin [Thu, 11 Apr 2019 02:41:38 +0000 (10:41 +0800)]
LU-10092 pcc: security and permission for non-root user access

For current PCC, if a file is left on the PCC cache, it may be
accessible to other jobs/users who would not normally be able to
access it. (That is,  they access it directly on the PCC mount via
FID as the local PCC mount is basically just a normal local file
system.)

This patch solves this by restricting access on the PCC side and
just depending on the Lustre side permissions for opening a file.
So PCC files on the local mount fs are created with some minimal
(zero) set of permissions. Then, when accessing a PCC cached
file, we do the permission check on the Lustre file, then do not
do it on the PCC file. This should render the PCC files
inaccessible except to root or via Lustre.

Test-Parameters: clientcount=3 testlist=sanity-pcc,sanity-pcc,sanity-pcc
Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I059fa3e479fe97ef6b65db1cbeb8b7f3ea611880
Reviewed-on: https://review.whamcloud.com/34637
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-10092 pcc: Non-blocking PCC caching 66/32966/37
Qian Yingjin [Thu, 5 Jul 2018 06:43:46 +0000 (14:43 +0800)]
LU-10092 pcc: Non-blocking PCC caching

Current PCC uses refcount of PCC inode to determine whether a
previous PCC-attached file can be detached. If a file is open
(refcount > 1), the detaching will return -EBUSY.

When another client accesses the PCC-cached file, it will trigger
the restore process as the file is HSM released. During restore,
the Agent needs to detach the PCC-cached file.
Thus, if a PCC-attached file is keeping opened but not closed
for a long time, the restore request will always return failure.

In this patch, we implement a non-blocking PCC caching mechanism
for Lustre. After attaching the file into PCC, the client acquires
the layout lock for the file, and the layout generation is
maintained in the PCC inode. Under the layout lock protection, the
PCC caching state is valid and all I/O will direct into PCC. When
the layout lock is revoked, in the blocking AST it will invalidate
the PCC caching state and detach the file automatically.

This patch is also helpful to handle the ENOSPC error for PCC
write by fallback to normal I/O path which will restore the file
data into OSTs (The file is in HSM released state) and redo the
write again.

Change-Id: I9130c04dc0e6eae879ea2ff3fdda65726e74d177
Test-Parameters: clientcount=3 testlist=sanity-pcc,sanity-pcc,sanity-pcc
Signed-off-by: Qian Yingjin <qian@ddn.com>
Reviewed-on: https://review.whamcloud.com/32966
Tested-by: Jenkins
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-10092 llite: Add persistent cache on client 63/32963/38
Li Xi [Tue, 27 Jun 2017 12:18:14 +0000 (20:18 +0800)]
LU-10092 llite: Add persistent cache on client

PCC is a new framework which provides a group of local cache
on Lustre client side. No global namespace will be provided
by PCC. Each client uses its own local storage as a cache for
itself. Local file system is used to manage the data on local
caches. Cached I/O is directed to local filesystem while
normal I/O is directed to OSTs.

PCC uses HSM for data synchronization. It uses HSM copytool
to restore file from local caches to Lustre OSTs. Each PCC
has a copytool instance running with unique archive number.
Any remote access from another Lustre client would trigger
the data synchronization. If a client with PCC goes offline,
the cached data becomes inaccessible for other client
temporarilly. And after the PCC client reboots and the copytool
restarts, the data will be accessible again.

ToDo:
1) Make PCC exclusive with HSM.
2) Strong size consistence for PCC cached file among clients.
3) Support to cache partial content of a file.

Change-Id: I188ed36c48aae223380739f607cc6caf2f789298
Test-Parameters: clientcount=3 testlist=sanity-pcc,sanity-pcc,sanity-pcc
Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Signed-off-by: Qian Yingjin <qian@ddn.com>
Reviewed-on: https://review.whamcloud.com/32963
Tested-by: Jenkins
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
13 months agoLU-11838 osd-ldiskfs: inode times switched to timespec64 75/34675/4
Li Dongyang [Tue, 16 Apr 2019 01:14:13 +0000 (11:14 +1000)]
LU-11838 osd-ldiskfs: inode times switched to timespec64

Since kernel 4.18 inode times swtich from struct timespec
to timespec64 to make it y2038 safe.

Linux-commit: 95582b00838837fc07e042979320caf917ce3fe6

Test-Parameters:trivial
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Change-Id: Iaddb2f2be27ec348fb97e13371aa3d7e6f6e5c9f
Reviewed-on: https://review.whamcloud.com/34675
Reviewed-by: Gu Zheng <gzheng@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
13 months agoLU-11838 ldiskfs: add rhel8 server support 74/34674/6
Li Dongyang [Mon, 15 Apr 2019 08:05:58 +0000 (18:05 +1000)]
LU-11838 ldiskfs: add rhel8 server support

This patch adds ldiskfs patch series for rhel8
kernel 4.18.0-32.el8.

Fix lustre-build-ldiskfs.m4, make
CONFIG_LDISKFS_FS_ENCRYPTION consistent with
kernel's config CONFIG_EXT4_FS_ENCRYPTION.
Otherwise ldiskfs won't build
on kernels with CONFIG_EXT4_FS_ENCRYPTION
disabled.

Note: this contains a small clean up for
ubuntu18/ext4-kill-dx-root.patch

Test-Parameters:trivial
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Change-Id: Ib500bff2f6688405b912620c5217586c8420c6e1
Reviewed-on: https://review.whamcloud.com/34674
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>