Whamcloud - gitweb
fs/lustre-release.git
17 months agoLU-11919 llite: Initialize cl_dirty_max_pages 73/34173/4
Patrick Farrell [Mon, 4 Feb 2019 03:47:10 +0000 (22:47 -0500)]
LU-11919 llite: Initialize cl_dirty_max_pages

cl_dirty_max_pages must be initialized to zero before
calling client_adjust_max_dirty.

Test-Parameters: trivial

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Ie34306ae329e377520a7a4858ab969f901c6154d
Reviewed-on: https://review.whamcloud.com/34173
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
17 months agoLU-9859 libcfs: replace cfs_srand() calls with add_device_randomness(). 70/34170/5
NeilBrown [Mon, 4 Feb 2019 17:54:41 +0000 (12:54 -0500)]
LU-9859 libcfs: replace cfs_srand() calls with add_device_randomness().

The only places that cfs_srand is called, the random bits are
mixed with bits from get_random_bytes().  So it is equally effective
to add entropy to either pool.
So we can replace calls to cfs_srand() with calls that add the
entropy with add_device_randomness().  That function adds time-based
entropy, so we can discard the ktime_get_ts64 calls.

One location in lustre_handles.c only adds time based
entropy. This cannot improve the entropy provided by
get_random_bytes(), so just discard that call.

Linux-commit: 30f4236aafa81722490e74ded48a9fb2aff013ab

Change-Id: If1a8ffad05fcc89272136949300f6512dca39704
Signed-off-by: NeilBrown <neilb@suse.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-on: https://review.whamcloud.com/34170
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-11917 tests: wait for test_257 to finish recovery 65/34165/4
Andreas Dilger [Fri, 1 Feb 2019 23:35:47 +0000 (16:35 -0700)]
LU-11917 tests: wait for test_257 to finish recovery

The sanity test_257 restarts the MDS, but it may still be in
recovery at the start of test_260a and cause failures.  Wait for
recovery to complete before the end of test_257.

Fixes: 1cb9e85039c ("LU-7433 ldlm: xattr locks are lost on mdt")
Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I00f183938a208608155ca71b08ca7977903ebbe5
Reviewed-on: https://review.whamcloud.com/34165
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-8346 tests: remove spaces around fail_val 55/34155/3
James Nunez [Fri, 1 Feb 2019 03:40:59 +0000 (20:40 -0700)]
LU-8346 tests: remove spaces around fail_val

conf-sanity test 93 tries to set fail_loc and fail_val
with the command 'lctl set_param fail_val = 10 fail_loc...'.
fail_val should have no spaces before and after the
equals sign.

Test-Parameters: trivial mdscount=2 mdtcount=4 envdefinitions=ONLY=93 testlist=conf-sanity
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: Iaa2bff1750a2afa96a73a452a0c098ae92f7616c
Reviewed-on: https://review.whamcloud.com/34155
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
17 months agoLU-11911 lov: fix lov_iocontrol for inactive OST case 48/34148/2
Vladimir Saveliev [Fri, 1 Feb 2019 00:16:29 +0000 (03:16 +0300)]
LU-11911 lov: fix lov_iocontrol for inactive OST case

For inactive OSTs lov->lov_tgts[index]->ltd_exp is
NULL. lov_iocontrol() is to check that before dereferencing to
lov->lov_tgts[index]->ltd_exp->exp_obd.

Signed-off-by: Vladimir Saveliev <c17830@cray.com>
Cray-bug-id: LUS-6937
Test-Parameters: trivial
Change-Id: I4bb332ee2c50b07a1471035556f4d77a3559847f
Reviewed-on: https://review.whamcloud.com/34148
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexandr Boyko <c17825@cray.com>
17 months agoLU-930 utils: fix --verbose option for lfs-migrate.1 75/34075/2
Andreas Dilger [Fri, 18 Jan 2019 02:44:39 +0000 (21:44 -0500)]
LU-930 utils: fix --verbose option for lfs-migrate.1

Fix the "lfs migrate --verbose" option.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I66c02926023fd3f08ddb6d5d10ee5836b83ebbe5
Reviewed-on: https://review.whamcloud.com/34075
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Joseph Gmitter <jgmitter@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-11401 tests: add version check sanity-flr tests 55/33955/5
James Nunez [Tue, 15 Jan 2019 00:48:17 +0000 (17:48 -0700)]
LU-11401 tests: add version check sanity-flr tests

sanity-flr test 48 and 203 was added to Lustre tag 2.11.55.
Thus, we need to check that the server version is 2.11.55 or
later before running test 48 and 203.

sanity-flr test 0h checks for a file inheriting the directory
layout. sanity-flr test 37 added ‘lfs mirror write’ functionality.
Inheritance was fixed and ‘lfs mirror write’ was added in Lustre
tag 2.11.57. Thus, we need to check that the server version is
2.11.57 or later before running test 0h and 37.

Test-Parameters: trivial testlist=sanity-flr
Test-Parameters: serverjob=lustre-b2_11 serverbuildno=2 testlist=sanity-flr
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I94c68e900d60e2b97d7f74c6629ee54bcb3a5480
Reviewed-on: https://review.whamcloud.com/33955
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-11770 osc: allow build without blk_integrity or crc-t10pi 23/33923/9
Andreas Dilger [Wed, 26 Dec 2018 09:05:37 +0000 (02:05 -0700)]
LU-11770 osc: allow build without blk_integrity or crc-t10pi

Allow the client to build if blk_integrity or crc-t10pi is not
enabled in the kernel.

Fixes: ccf3674c9ca ("LU-10472 osd-ldiskfs: T10PI between RPC and BIO")
Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I97c4e75ad084e99927bcb41cf0df8a680525a5b1
Reviewed-on: https://review.whamcloud.com/33923
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-10171 lmv: avoid gratuitous 64-bit modulus 22/33922/9
Andreas Dilger [Wed, 26 Dec 2018 10:45:52 +0000 (03:45 -0700)]
LU-10171 lmv: avoid gratuitous 64-bit modulus

Fix the pct() calculation to use unsigned long arguments, since this
is what callers use.  Remove duplicate pct() definition in lproc_mdc.

Don't do a 64-bit modulus of the LNet NID to find the starting MDT
index when this isn't really needed.

Similarly, don't compute the FLD cache usage percentage for a debug
message that is never used.

Fixes: 9b924e86b27d ("LU-10171 headers: define pct(a,b) once")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I34cefd269cb83f563d2f08c32dc3fa1ed5c5a5b1
Reviewed-on: https://review.whamcloud.com/33922
Reviewed-by: Ben Evans <bevans@cray.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-11797 tests: improve sanity test 272a checking 89/33889/3
Andreas Dilger [Tue, 18 Dec 2018 18:57:46 +0000 (11:57 -0700)]
LU-11797 tests: improve sanity test 272a checking

Improve sanity.sh test_272a() to check data integrity before it
checks the stripe count.  Also, print out the stripe count if it
does not match our expectations.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I75397f251f10ed00c3dea9f80243d7ba9eacab07
Reviewed-on: https://review.whamcloud.com/33889
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-11634 tests: sanityn/test_77 improvements 07/33607/5
Vladimir Saveliev [Mon, 9 Apr 2018 09:18:50 +0000 (12:18 +0300)]
LU-11634 tests: sanityn/test_77 improvements

sshd limits number of simultaneous unauthenticated connections via
MaxStartups configuration parameter. By default, 10 connections are
allowed. nrs_write_read() tries to run up to 32 do_nodes() in
parallel, causing sshd to drop some of connections.

The fix is to have do_nodes() to start required number of dd-s in
parallel.

Minor changes which were probably meant at the development:
- Test filenames include $HOSTNAME so that each client worked with its
own file, it seems. Add missing escaping backslashes so that $HOSTNAME
worked as expected.
- Add conv=notrunc parameter for dd-s which write lustre file at
  different seeks.
- Have reading dd-s to read files which were especially created for
  that.
- use /dev/null instead on /dev/zero to throw read data away.

Signed-off-by: Vladimir Saveliev <c17830@cray.com>
Change-Id: I496b0f6b50811351ac8e0e606cf5a20843fab5d4
Cray-bug-id: LUS-2493
Test-Parameters: testlist=sanityn envdefinitions=ONLY=77
Reviewed-on: https://review.whamcloud.com/33607
Reviewed-by: Elena Gryaznova <c17455@cray.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andrew Perepechko <c17827@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-11495 tests: zconf_mount_clients() fix to work with FILESET 38/33338/11
Elena Gryaznova [Fri, 1 Feb 2019 16:41:22 +0000 (19:41 +0300)]
LU-11495 tests: zconf_mount_clients() fix to work with FILESET

If FILESET is set zconf_mount_clients() fails to mount clients
because of missing $mnt dir on the clients.

When FILESET is set the following tests are to be skipped:
- tests accessing .lustre directly
- tests involving lustre_rsync (which needs .lustre to do
  open by FID)
- tests involving copytool (lhsmtool_posix needs .lustre)
- tests taking use of llapi_open_by_fid (sanity.sh:test_405()
  ->swap_lock_test, sanity.sh:test_807()->llsom_sync)
- tests require Lustre root mounted on $MOUNT
  (sanity.sh:test_65n() needs root to test default layout
  inheritance)

Test-Parameters: trivial envdefinitions=FILESET=/subdir
Signed-off-by: Elena Gryaznova <c17455@cray.com>
Cray-bug-id: LUS-6546, LUS-6561
Reviewed-by: Vladimir Saveliev <c17830@cray.com>
Reviewed-by: Andrew Perepechko <c17827@cray.com>
Reviewed-by: Alexander Boyko <c17825@cray.com>
Change-Id: I68081fbff808abbaddb314e36283e09d151a81db
Reviewed-on: https://review.whamcloud.com/33338
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexandr Boyko <c17825@cray.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-10885 llite: enable flock mount option by default 91/32091/10
Andreas Dilger [Tue, 2 Oct 2018 21:52:28 +0000 (15:52 -0600)]
LU-10885 llite: enable flock mount option by default

The "flock" mount option has been optional for many years, initially
because of potential stability issues, and also to provide a choice
for administrators to select between "flock" and "localflock" options.

However, from the large number of problems that users report when
trying to use applications that depend on this feature (typically
databases and other cloud stacks) that disabling flock by default
causes more problems than it solves.

Enable the "flock" (distributed coherent userspace locking) feature
by default.  If applications do not need this functionality, then it
will not affect them.  If applications *do* need this functionality,
they will get it.  If administrators really know what they are doing,
then they can use the "localflock" feature to enable client-local
flock functionality, possibly only on select nodes that need this.

Users wanting to disable this functionality should mount with the
existing "-o noflock" mount option, or build the client with the
"configure --disable-flock" option.

If clients are already using "-o {flock|localflock|noflock}" then
their existing options will be handled appropriately.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I182637604fa22573b1da6b6b86d8915e3c3ebbe5
Reviewed-on: https://review.whamcloud.com/32091
Reviewed-by: Patrick Farrell <paf@cray.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-10885 tests: fix up flocks_test bugs and code style 92/32092/10
Andreas Dilger [Tue, 2 Oct 2018 21:52:02 +0000 (15:52 -0600)]
LU-10885 tests: fix up flocks_test bugs and code style

Fix the flocks_test test program:
- don't segfault if run without any command-line arguments
- return errors from test2 to the caller

Clean up flocks_test code style:
- tabify the whole file
- don't put assignments inside conditionals
- print error messages where needed.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I2555c741b0c170a43c47c16425cba3186e3ebbe5
Reviewed-on: https://review.whamcloud.com/32092
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-11947 scripts: handle ZFS targets in Lustre RA 17/34217/3
Nathaniel Clark [Fri, 8 Feb 2019 18:02:28 +0000 (13:02 -0500)]
LU-11947 scripts: handle ZFS targets in Lustre RA

Fixes a regression introduced in LU-11461
This handles the case of realpath of target being an empty string.

Fixes: c36d70272541 ("LU-11461 scripts: Support symlink target")
Test-Parameters: trivial
Signed-off-by: Nathaniel Clark <nclark@whamcloud.com>
Change-Id: I1bcb85908019e968ac0d69e437db217594a6565e
Reviewed-on: https://review.whamcloud.com/34217
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
17 months agoLU-11897 ost: improve memory allocation for ost 27/34127/7
Andrew Perepechko [Fri, 26 Oct 2018 08:29:03 +0000 (11:29 +0300)]
LU-11897 ost: improve memory allocation for ost

OST_BUFSIZE is defined as 17 KiB. Lustre uses
OBD_CPT_ALLOC_LARGE() to allocate buffers, which,
in turn, uses kmalloc_node(). kmalloc_node(8192+) falls
back to the traditional buddy allocator kmalloc_large_node().

In the end, 32 KiB is allocated using a 17 KiB allocation
request.

This patch changes OST_BUFSIZE to 32 KiB so we can
effectively use the whole allocated buffer.

Change-Id: I93ce5b26eff4a6a1a17b2a9bfb83161528570197
Signed-off-by: Andrew Perepechko <c17827@cray.com>
Reviewed-by: Alexey Lyashkov <c17817@cray.com>
Reviewed-by: Alexander Zarochentsev <c17826@cray.com>
Tested-by: Alexander Lezhoev <c17454@cray.com>
Cray-bug-id: LUS-6657
Reviewed-on: https://review.whamcloud.com/34127
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
17 months agoLU-11803 tests: don't assume obd device name 94/33894/8
James Simmons [Thu, 24 Jan 2019 18:20:48 +0000 (13:20 -0500)]
LU-11803 tests: don't assume obd device name

Several tests created to exercise lustre were developed on the
x86 platform and it was assumed the device name exposed in the
sysfs tree are the same across all platforms. Additionally
we can update the test to handle the case of using an uuid
format for the sysfs directory naming instead of an internal
address pointer.

Test-Parameters: clientdistro=ubuntu1804

Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Change-Id: I704d13059f76337fa49aab77f3e748a70a74f1bc
Reviewed-on: https://review.whamcloud.com/33894
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
17 months agoLU-11591 llog: add synchronization for the last record 83/33683/6
Alexander Boyko [Thu, 29 Nov 2018 13:58:30 +0000 (08:58 -0500)]
LU-11591 llog: add synchronization for the last record

The initial problem was a race between llog_process_thread
and llog_osd_write_rec for a last record with lgh_last_idx.
The catalog should be wrapped for the problem. The lgh_last_idx
could be increased with a modification of llog bitmap, and a writing
record happen a bit later. When llog_process_thread processing
lgh_last_idx after modification and before a write it operates
with old record data.

The patch adds synchronization when lgh_last_idx is processed.

The patch changes llog_test 10h to check race between
llog_process_thread and llog_osd_write_rec.

1 Thread with write                  2 Thread with read
llog_osd_write_rec()                llog_process_thread()
lgh_last_idx++
lock lgh_hdr_mutex
ext2_set_bit()
dt_write_rec (write header)         ext2_test_bit()
         check lgh_last_idx was changed
         dt_read_rec()
         reread the record, and here we
         got the old value of record
unlock lgh_hdr_mutex
dt_write_rec (write the record)

Signed-off-by: Alexander Boyko <c17825@cray.com>
Cray-bug-id: LUS-6683
Change-Id: I642b488655940b9456ca8e2f2174c98a966ba242
Reviewed-on: https://review.whamcloud.com/33683
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alexander Zarochentsev <c17826@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-11111 lfsck: skip orphan processing 59/32959/20
Alexey Lyashkov [Wed, 8 Aug 2018 04:47:51 +0000 (07:47 +0300)]
LU-11111 lfsck: skip orphan processing

LFSCK can reconnect a recently-deleted orphan object back
into the normal namespace when it shouldn't.  This can
cause access to the deleted data (potential security risk),
and sometimes cause an assertion if orphan is later deleted.

Skip LFSCK on orphan objects.  Fix the handling of the
LUSTRE_ORPHAN_FL in both osd-zfs and osd-ldiskfs so that
la_valid |= LA_FLAGS is set when la_flags |= LUSTRE_ORPHAN_FL
is set, otherwise the upper layers cannot properly detect it.

Clean up alignment of flags values and provide hex equivalents
so that they are more easily referenced during debugging.

Signed-off-by: Alexey Lyashkov <c17817@cray.com>
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I9b2809b95efa4b3c3e3b2c7d0a501624ed743ede
Reviewed-on: https://review.whamcloud.com/32959
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-11616 llite: replace smp_wb() with full memory barrier 71/33571/2
NeilBrown [Sun, 4 Nov 2018 20:26:57 +0000 (15:26 -0500)]
LU-11616 llite: replace smp_wb() with full memory barrier

While porting the smp_mb() patch from LU-9210 to the linux clients
Neil asked that the smp_wb() be replaced with the full memory
barrier functions smp_store_release() and smp_load_acquire(). For
this case _sa_make_ready() sets the entry->se_state and
revalidate_statahead_dentry() tests the value of entry->se_state
after waiting on sai_waitq. This change will make it obvious which
variable was important, and would show the paired synchronization
points. An additional benefit is that code will work on platforms
that are not "total store order" TSO architectures like PowerPC.

Change-Id: I687177bf1697a21db624a289c136215af4d90506
Signed-off-by: NeilBrown <neilb@suse.com>
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/33571
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Patrick Farrell <paf@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-11058 tests: cleanup persistent checksum= in 77k 35/34035/4
Alex Zhuravlev [Tue, 15 Jan 2019 12:13:50 +0000 (15:13 +0300)]
LU-11058 tests: cleanup persistent checksum= in 77k

this trivial patch let me pass 77k locally. can be used
till actual solution is landed

Test-Parameters: trivial
Change-Id: I5c4b4cd15d8e02dd96d918c07aacd184014ade0c
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/34035
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-11881 utils: silence error message 86/34086/2
Alexander Zarochentsev [Sun, 15 Jul 2018 21:17:20 +0000 (00:17 +0300)]
LU-11881 utils: silence error message

llapi_get_poollist prints an error message
in case of reallocating the buffer
and successful completion.

Cray-bug-id: LUS-6185
Test-Parameters: testlist=ost-pools
Change-Id: I0ca1f25edf3f4c89525d41f2deab8d25ec9e0516
Signed-off-by: Alexander Zarochentsev <c17826@cray.com>
Reviewed-on: https://review.whamcloud.com/34086
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Alexandr Boyko <c17825@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-11838 socklnd: use for_each_netdev() instead of lnet_ipif_enumerate() 66/33966/4
NeilBrown [Mon, 7 Jan 2019 19:13:21 +0000 (14:13 -0500)]
LU-11838 socklnd:  use for_each_netdev() instead of lnet_ipif_enumerate()

for_each_netdev() is a more direct interface and doesn't require
library support.

Also get the ip address directly from the net_device, rather than
using lnet_ipif_query().

Linux-commit: f703f71afd98e6e7ec70f92ffc52ef3ffffcd849
Linux-commit: 9eb957b98aa6322abde33240bf50dd483c5d1190

Change-Id: I82894991b9a4a250d0560af31325b6c765cc0620
Signed-off-by: NeilBrown <neilb@suse.com>
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/33966
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sonia Sharma <sharmaso@whamcloud.com>
Reviewed-by: Doug Oucharek <dougso@me.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-11825 llite: Remove ptask framework 13/33913/4
Patrick Farrell [Sun, 23 Dec 2018 21:04:33 +0000 (16:04 -0500)]
LU-11825 llite: Remove ptask framework

With the removal of pio, the ptask framework no longer has
any users.

Signed-off-by: Patrick Farrell <paf@cray.com>
Change-Id: Id3237740c251b01bc176967f8fe5fd12bb82f553
Reviewed-on: https://review.whamcloud.com/33913
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@gmail.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-1365 tests: createmany outputs stat after 2 seconds 56/33756/6
Artem Blagodarenko [Wed, 28 Nov 2018 19:55:30 +0000 (22:55 +0300)]
LU-1365 tests: createmany outputs stat after 2 seconds

Createmany outputs current operation information every 10 seconds or
every 10k operations. For fast  devices, this overflows screen.

Change createmany to output every 10k operations or every 10 seconds
in case it has not been finished after 2 seconds of execution.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Artem Blagodarenko <c17828@cray.com>
Change-Id: I8bebff021e53e2338d372336a9ed14da6a3b45c7
Reviewed-on: https://review.whamcloud.com/33756
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
17 months agoLU-11701 ptlrpc: eviction of lwp connection aborts requests 19/33719/6
Andriy Skulysh [Mon, 8 Oct 2018 15:43:20 +0000 (18:43 +0300)]
LU-11701 ptlrpc: eviction of lwp connection aborts requests

Lwp connection isn't replayed but ongoing request
should be resent.
Modify recovery-small/106 to check for
LUSTRE_IMP_RECOVER state instead of LUSTRE_IMP_EVICTED.

Change-Id: I50cf85405de588de8499d8fad8a4fe30923f348e
Cray-bug-id: LUS-6509
Signed-off-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Alexander Boyko <c17825@cray.com>
Reviewed-by: Vladimir Saveliev <c17830@cray.com>
Reviewed-on: https://review.whamcloud.com/33719
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexandr Boyko <c17825@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-10913 llog: add startcat for wrapped catalog 49/33749/3
Alexander Boyko [Thu, 29 Nov 2018 08:20:00 +0000 (03:20 -0500)]
LU-10913 llog: add startcat for wrapped catalog

The osp_sync_thread loop for a llog_cat_process has a mistake.
When llog_cat_process has reached a bottom of catalog, the processing
restarts with 0. Which means a default processing. In this case a
catalog is wrapped and processing starts from a llh_cat_idx. But
records at the bottom were processed already, and were not cancelled
yet. The next message appears at log.
osp_sync_interpret()) reply req ffff8800123e3600/1, rc -2, transno 0

llog_cat_process support startcat index for processing catalog.
In this case the processing starts from startcat index. But if
catalog is wrapped startcat index is ignored.

The patch adds supporting of startcat index for wrapped catalog.

Signed-off-by: Alexander Boyko <c17825@cray.com>
Change-Id: Ie4e3ecf2532878578ae0463969115664e3589188
Cray-bug-id: LUS-6765
Reviewed-on: https://review.whamcloud.com/33749
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sergey Cheremencev <c17829@cray.com>
Reviewed-by: Alexander Zarochentsev <c17826@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-8130 libcfs: port working hash from upstream 89/33789/5
James Simmons [Tue, 15 Jan 2019 16:25:54 +0000 (11:25 -0500)]
LU-8130 libcfs: port working hash from upstream

The hash_[32|64] function in pre-4.6 kernels produce hashes
with poor distributions which result in high collision rates.
Backport those improvements for the pre-4.6 kernels Lustre
supports. Details can be read here:

https://lwn.net/Articles/687494

Test-Parameters: trivial
Change-Id: Id2436ba8be2d3ed482c5386b79710f594d5b3e59
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/33789
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexey Lyashkov <c17817@cray.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-11862 lfsck: avoid unexpected repairing 33/34033/2
Alexey Lyashkov [Tue, 15 Jan 2019 13:08:08 +0000 (16:08 +0300)]
LU-11862 lfsck: avoid unexpected repairing

lfsck can called with create in parallel,
so some OST objects will don't have a right owner.
Don't treat it as wrong owner issue.

Cray-bug-id: LUS-6181
Signed-off-by: Alexey Lyashkov <c17817@cray.com>
Change-Id: Id9c067a67d558dcfcb0a2f0fdd790b446b4f54d0
Reviewed-on: https://review.whamcloud.com/34033
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexander Zarochentsev <c17826@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-4664 libcfs: allow file/func/line passed to CDEBUG() 88/33588/7
Andreas Dilger [Tue, 6 Nov 2018 11:23:00 +0000 (04:23 -0700)]
LU-4664 libcfs: allow file/func/line passed to CDEBUG()

Allow the file, function, and line number to be passed to CDEBUG()
messages so that they are not duplicated in helper functions that
may be called from multiple places.

This patch is largely a no-op in terms of code, with the exception
of one call in osc_extent_sanity_check0() to OSC_EXTENT_DUMP() that
is changed to OSC_EXTENT_DUMP_WITH_LOC().

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I0bd866b94daff904c16523d11c710f562a3ebbe5
Reviewed-on: https://review.whamcloud.com/33588
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-9868 llite: handle DCACHE_PAR_LOOKUP in ll_dcompare 86/28486/24
NeilBrown [Mon, 1 Oct 2018 18:19:05 +0000 (14:19 -0400)]
LU-9868 llite: handle DCACHE_PAR_LOOKUP in ll_dcompare

ll_dcompare is used in two slightly different contexts.
It is called (from __d_lookup, __d_lookup_rcu, and d_exact_alias)
to compare a name against a dentry that is already in the dcache.
It is also called (from d_alloc_parallel) to compare a name against
a dentry that is not in the dcache yet, but is part of an active
"lookup" or "atomic_open" call.

In the first case we need to avoid matching against "invalid" dentries
as a match implies something about ldlm locks which is not accurate.
In the second case we need to allow matching against "invalid" dentries
as the dentry will always be invalid (set by ll_d_init()) but we still
want to guard against multiple concurrent lookups of the same name.
d_alloc_parallel() will repeat the call to ll_dcompare() after
the lookup has finished, and if the dentry is still invalid, the whole
d_alloc_parallel() process is repeated.  This assures us that it is safe
to report success whenever d_in_lookup().

With this patch, there will never be two threads concurrently in
ll_lookup_nd(), looking up the same name in the same directory.

Linux-commit: a22c3d41d187dc3cdaf41166ef0a20b8663fdfee

Change-Id: If489a6f2bbc5c0974570583e3d5083cf77a3b950
Signed-off-by: NeilBrown <neilb@suse.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/28486
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
17 months agoLU-11811 mdt: Add a proc entry to set MDT to readonly 92/33892/3
Jinshan Xiong [Tue, 18 Dec 2018 23:36:33 +0000 (15:36 -0800)]
LU-11811 mdt: Add a proc entry to set MDT to readonly

It is sometimes desirable to be able to mark the filesystem read-only
directly on the server, rather than remounting the clients and setting
the option there.  This can be useful if there is a rogue client
that is deleting files, or when decommissioning a system to
prevent already-mounted clients from modifying it anymore.

Add the mdt.*.readonly parameter to allow setting the MDT read-only
immediately if set to 1.  All future MDT access will immediately
return -EROFS until the parameter is set to 0 again.

Signed-off-by: Jinshan Xiong <jinshan.xiong@uber.com>
Change-Id: I6d8d529ed4ba797012ded6920a9d9e7db6659efc
Reviewed-on: https://review.whamcloud.com/33892
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11568 ldlm: Remove use of SLAB_DESTROY_BY_RCU for ldlm lock slab 47/34147/5
Oleg Drokin [Thu, 31 Jan 2019 18:42:43 +0000 (13:42 -0500)]
LU-11568 ldlm: Remove use of SLAB_DESTROY_BY_RCU for ldlm lock slab

Whatever it was doing does not appear to be necessary anymore
as evidenced with newer kernels where the define was removed,
but disabled in Lustre instead.
Another important reason to remove it is because rhel7.3+ seems
to have broken this RCU functionality and leads to frequent use
after frees.

Change-Id: I50991b9daf4ef06b24cb65d7a04a5e9b86706d36
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/34147
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <maloo@whamcloud.com>
18 months agoLU-11861 obdclass: fix build with debug kernel 30/34030/6
Alexey Lyashkov [Tue, 15 Jan 2019 12:24:42 +0000 (15:24 +0300)]
LU-11861 obdclass: fix build with debug kernel

Move declaration before usage.

Test-Parameters: trivial
Signed-off-by: Alexey Lyashkov <c17817@cray.com>
Change-Id: I9a6c451bb5454b1542f0b06041f6938702e20b36
Reviewed-on: https://review.whamcloud.com/34030
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
18 months agoLU-11930 llite: Restore pio flags 87/34187/4
Oleg Drokin [Tue, 5 Feb 2019 19:42:07 +0000 (14:42 -0500)]
LU-11930 llite: Restore pio flags

flags could never be removed from the LL_SBI_FLAGS list,
since the order is dependent on actual bits in the
structure.

Test-Parameters: trivial
Change-Id: I601ad0df478b17075658706c830f4a615c418104
Fixes: 2b0a34fe43bf ("LU-11825 clio: Remove pio feature")
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/34187
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
18 months agoLU-11899 utils: Possible resource/memory leaks 29/34129/2
Thomas Stibor [Tue, 29 Jan 2019 15:09:07 +0000 (16:09 +0100)]
LU-11899 utils: Possible resource/memory leaks

Static checker revealed possible resource/memory leaks
of file handle and process pipe.

Signed-off-by: Thomas Stibor <t.stibor@gsi.de>
Change-Id: I4ee05a8325408370bcfb5e9b8d30253107115c7b
Reviewed-on: https://review.whamcloud.com/34129
Tested-by: Jenkins
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Shilong Wang <wshilong@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11892 hsm: fix memory leak when scheduling HSM requests 23/34123/2
Aurelien Degremont [Mon, 28 Jan 2019 13:54:14 +0000 (13:54 +0000)]
LU-11892 hsm: fix memory leak when scheduling HSM requests

After creating a big HSM request candidate list, coordinator thread
allocates an update list which is 16 bytes times the number of
candidates files. In some cases, they could be more than 300,000
candidate files and so several MB will be needed.

Memory allocation failure was skipping candidate list freeing
which led to memory leak.

Test-Parameters: trivial testlist=sanity-hsm
Signed-off-by: Aurelien Degremont <degremoa@amazon.com>
Change-Id: Ibb4833f4430cc50cc014f6d14caf0551e0fce161
Reviewed-on: https://review.whamcloud.com/34123
Tested-by: Jenkins
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11878 tests: don't fork-bomb sanity test_103b 82/34082/5
Andreas Dilger [Tue, 22 Jan 2019 07:53:44 +0000 (00:53 -0700)]
LU-11878 tests: don't fork-bomb sanity test_103b

Running sanity test_103b may start up to 512 parallel threads for
running the test, each of which starts two bash processes and lfs
or rm processes.

For the VMs running in our testbed (esp. ARM with 64KB PAGE_SIZE)
this can trigger the OOM killer and cause the test to fail if bash
is killed.  Limit the number of started bash processes to avoid this.

Fixes: 543f1fbe260 ("LU-10830 utils: fix create mode for lfs")
Test-Parameters: trivial clientarch=aarch64
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I82c322013da91d717924e2c664fa57ad4e3ebbe5
Reviewed-on: https://review.whamcloud.com/34082
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Elena Gryaznova <c17455@cray.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-10143 osd-zfs: allocate sequence in advance 69/34069/12
Alex Zhuravlev [Sun, 20 Jan 2019 05:38:26 +0000 (08:38 +0300)]
LU-10143 osd-zfs: allocate sequence in advance

on the controller, so that we have it ready before any potential
read-only makeup. this is what osd-ldiskfs is doing already.

Change-Id: I3d27f112b0d013ac923c5d250b296b5528b8112d
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/34069
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11876 dkms: Add missing Requires for zlib-devel 65/34065/2
Nathaniel Clark [Fri, 18 Jan 2019 22:21:12 +0000 (17:21 -0500)]
LU-11876 dkms: Add missing Requires for zlib-devel

lustre.spec has BuildRequires for zlib-devel, and lustre-*-dkms should
also Requires zlib-devel so build completes successfully.

Test-Parameters: trivial
Signed-off-by: Nathaniel Clark <nclark@whamcloud.com>
Change-Id: Icf1c3f4342823502200d3326c63fe991c3140aab
Reviewed-on: https://review.whamcloud.com/34065
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Joe Grund <jgrund@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-10447 tests: replace $SETSTRIPE/$GETSTRIPE with $LFS 17/33917/2
James Nunez [Wed, 26 Dec 2018 23:44:30 +0000 (16:44 -0700)]
LU-10447 tests: replace $SETSTRIPE/$GETSTRIPE with $LFS

$SETSTRIPE and $GETSTRIPE were needed when we used the
standalone 'lstripe' utility. 'lstripe' hasn't been used
for years and we need to clean up all remnants of it.

Replace all instances of $SETSTRIPE with '$LFS setstripe'
and $GETSTRIPE with '$LFS getstripe' in the conf-sanity,
ost-pools and sanity-quota test suites.

Test-Parameters: trivial testlist=conf-sanity
Test-Parameters: testlist=ost-pools,sanity-quota
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I9c5a3fd3a7ba8bf1e65da34880dff8df8d4e7b75
Reviewed-on: https://review.whamcloud.com/33917
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Elena Gryaznova <c17455@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11749 tests: sanity-sec 23b exec commands on right node 46/33846/3
Sebastien Buisson [Thu, 13 Dec 2018 09:02:47 +0000 (18:02 +0900)]
LU-11749 tests: sanity-sec 23b exec commands on right node

In sanity-sec test 23b, make sure commands are executed on
client 1, then client 2.
Otherwise, ACL mapping cannot be correctly demonstrated.

Test-Parameters: trivial clientcount=2 envdefinitions=ONLY=23b testlist=sanity-sec,sanity-sec,sanity-sec,sanity-sec,sanity-sec,sanity-sec
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Iffa3946be149313af696907fa6c83a6ea58cb3f6
Reviewed-on: https://review.whamcloud.com/33846
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11586 tests: test_10() fix for FILESET set 14/33514/6
Elena Gryaznova [Tue, 30 Oct 2018 13:43:17 +0000 (16:43 +0300)]
LU-11586 tests: test_10() fix for FILESET set

Patch fixes sanity-pfl test_10() to set root layout
properly: subdirectory could be mounted on $MOUNT if FILESET
is set, test now mounts root on $MOUNT2 for this case and
sets the root layout via $MOUNT2.

Test-Parameters: trivial testlist=sanity-pfl envdefinitions=ONLY=10
Signed-off-by: Elena Gryaznova <c17455@cray.com>
Cray-bug-id: 6557
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Alexander Boyko <c17825@cray.com>
Change-Id: If94474d92686a37518c1dd51793e742967fdfbc1
Reviewed-on: https://review.whamcloud.com/33514
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-10496 tests: disable 39k for DoM for a while 15/34115/2
Alex Zhuravlev [Sat, 26 Jan 2019 10:46:55 +0000 (13:46 +0300)]
LU-10496 tests: disable 39k for DoM for a while

as too many runs fail due to missing FMD.

Test-Parameters: trivial

Change-Id: Ie391917fd7e251283289a4d6930d26c977810a47
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/34115
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11720 spec: srpm should be free of kernel requiements 71/33771/2
Nathaniel Clark [Mon, 3 Dec 2018 19:04:37 +0000 (14:04 -0500)]
LU-11720 spec: srpm should be free of kernel requiements

This moves the fix for LU-9731 into spec file and out of lbuild.
This lets "make rpms" benefit from the fix.
This also prevents the srpm from being incorrectly locked to the
kernel present when lbuild was used to create it (via
kmp-lustre.preamble).

Test-Parameters: trivial
Signed-off-by: Nathaniel Clark <nclark@whamcloud.com>
Change-Id: I15f61c0e37182c0efbea3566d43b1e89f180d3e5
Reviewed-on: https://review.whamcloud.com/33771
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Gu Zheng <gzheng@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11915 tests: stop running conf-sanity test 110 66/34166/2
James Nunez [Sat, 2 Feb 2019 00:12:56 +0000 (17:12 -0700)]
LU-11915 tests: stop running conf-sanity test 110

conf-sanity test 115 started hanging about the time
test 110 and 111 were added to conf-sanity. Stop running
conf-sanity test 110 by adding it to the ALWAYS_EXCEPT
list.

Test-Parameters: trivial
Test-Parameters: fstype=ldiskfs mdscount=2 mdtcount=4 testlist=conf-sanity
Test-Parameters: fstype=ldiskfs mdscount=2 mdtcount=4 testlist=conf-sanity
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I5b36df4f0e852ba4f00945a0718bb2f6f89cc925
Reviewed-on: https://review.whamcloud.com/34166
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
18 months agoNew tag 2.12.51 2.12.51 v2_12_51
Oleg Drokin [Mon, 4 Feb 2019 17:07:36 +0000 (12:07 -0500)]
New tag 2.12.51

Change-Id: Icfc5df666e4f1d3bd384d6519069bdbeb0b07e7c

18 months agoLU-11721 tests: wait for statfs to update on DNE 64/34164/2
Andreas Dilger [Fri, 1 Feb 2019 21:07:01 +0000 (14:07 -0700)]
LU-11721 tests: wait for statfs to update on DNE

Wait for the statfs to update properly when there are multiple
MDTs so that the test doesn't gratuitously fail.

Fixes: 757403191c3 ("LU-11721 utils: print used inodes ratio ...")
Test-Parameters: trivial testlist=sanity mdscount=2 mdtcount=4 ostcount=7
Test-Parameters: testlist=sanity fstype=zfs mdscount=2 mdtcount=4
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ia75f7bd4d3027c91f10ce990730b2bd7123ebbe5
Reviewed-on: https://review.whamcloud.com/34164
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
18 months agoLU-11605 osp: max_create_count and create_count changes 59/33559/9
Sergey Cheremencev [Wed, 29 Aug 2018 19:20:36 +0000 (22:20 +0300)]
LU-11605 osp: max_create_count and create_count changes

Setting max_create_count to 0 causes setting create_count
to 0. Set create_count to OST_MIN_PRECREATE when setting
back max_create_count.
Without the patch create_count remains equal to 0 despite
on changing max_create_count to something != 0.
This causes create to stuck in osp_precreate_reserve
because osp_precreate_send doesn't send new request to OST.
To understand the number of objects to precreate(grow) it
uses opd_pre_create_count that is equal to 0.

Cray-bug-id: LUS-6435
Change-Id: I940c48f91e9c7d49b766bd85ea271ce229424c7f
Signed-off-by: Sergey Cheremencev <c17829@cray.com>
Reviewed-on: https://es-gerrit.dev.cray.com/153888
Reviewed-by: Alexander Zarochentsev <c17826@cray.com>
Reviewed-by: Artem Blagodarenko <c17828@cray.com>
Tested-by: Elena Gryaznova <c17455@cray.com>
Reviewed-on: https://review.whamcloud.com/33559
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-10059 tests: sanityn 32a restore parameters 70/34070/5
Patrick Farrell [Sun, 20 Jan 2019 15:31:02 +0000 (10:31 -0500)]
LU-10059 tests: sanityn 32a restore parameters

Test sanityn test 32a does not restore parameters on error,
making other tests fail because lockless truncate is still
enabled.

Add cleanup.

Also fix use of stack_trap with restore_params in two other
tests to clean up temporary file.

Test-Parameters: trivial testlist=sanityn

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I315eb5c53cc1722833b1159c5ad908c3b7bf5550
Reviewed-on: https://review.whamcloud.com/34070
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
18 months agoLU-11868 mdc: Improve xattr buffer allocations 59/34059/2
Patrick Farrell [Thu, 17 Jan 2019 20:26:09 +0000 (15:26 -0500)]
LU-11868 mdc: Improve xattr buffer allocations

Many of the xattr related buffers in the mdc/mdt code are
allocated at max_easize, but they are used for normal POSIX
xattrs (primarily ACLs) and so they are guaranteed not to
exceed XATTR_SIZE_MAX.

HSM xattrs should also be less than XATTR_SIZE_MAX.

Reduce allocations to MIN(XATTR_SIZE_MAX, max_easize).

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I228486ebd79b35358baeff490d6181cefa5a4586
Reviewed-on: https://review.whamcloud.com/34059
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
18 months agoLU-11871 doc: man page for lctl nodemap_set_fileset 57/34057/4
Sebastien Buisson [Thu, 17 Jan 2019 15:53:50 +0000 (00:53 +0900)]
LU-11871 doc: man page for lctl nodemap_set_fileset

Man page for lctl nodemap_set_fileset.

Test-Parameters: trivial
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Icce0b1558621bca84b14f76037a5000002855881
Reviewed-on: https://review.whamcloud.com/34057
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
18 months agoLU-11161 tests: start running sanity 160g again 94/33994/3
James Nunez [Mon, 7 Jan 2019 18:06:36 +0000 (11:06 -0700)]
LU-11161 tests: start running sanity 160g again

sanity test 160g was failing when run in a DNE configuration
and we stopped running this test meaning added to the
ALWAYS_EXCEPT list. The problem is that the test did not
write enough files to exceed changelog idle index threshold
for deregistering users.

Start running sanity test 160g with DNE testing again.

Test-Parameters: trivial
Test-Parameters: ostfilesystemtype=ldiskfs mdtfilesystemtype=ldiskfs clientcount=2 mdscount=2 mdtcount=4 osscount=1 ostcount=8 testlist=sanity
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I286ef8eb7c4638ff8f357db54c4926d5a2f20ac4
Reviewed-on: https://review.whamcloud.com/33994
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11658 tests: fix typo in previous patch 83/33983/3
Yang Sheng [Tue, 8 Jan 2019 12:39:10 +0000 (20:39 +0800)]
LU-11658 tests: fix typo in previous patch

Fixes: 42e83c44eb ("LU-11658 lov: cl_cache could miss initialize")

Test-Parameters: trivial testlist=conf-sanity envdefinitions=ONLY=50h

Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: I9981b60293f95bbc96ca838dc157f98685a17338
Reviewed-on: https://review.whamcloud.com/33983
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
18 months agoLU-11838 o2iblnd: get IP address more directly. 70/33970/4
NeilBrown [Mon, 7 Jan 2019 19:25:57 +0000 (14:25 -0500)]
LU-11838 o2iblnd: get IP address more directly.

Use dev_get_by_name() and for_primary_ifa() to
get IP address for a named device.  This is more
direct.

Linux-commit: 10e138e41a4343fd1a88e4543990205d134e562a
Linux-commit: 9eb957b98aa6322abde33240bf50dd483c5d1190

Change-Id: Ic4562c3948934bacb8613e9f6f57f609ecc04de7
Signed-off-by: NeilBrown <neilb@suse.com>
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/33970
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sonia Sharma <sharmaso@whamcloud.com>
Reviewed-by: Doug Oucharek <dougso@me.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11739 lod: don't inherit default layout from root directory 56/33956/7
Jian Yu [Wed, 23 Jan 2019 01:44:26 +0000 (17:44 -0800)]
LU-11739 lod: don't inherit default layout from root directory

There is no need to inherit the default directory layout from
the root directory when subdirectories are created therein.
This consumes xattr space on the subdirectories, and makes it
more complex to change the filesystem default layout in the future.

This patch fixes the above issue in lod_ah_init() to check if
the parent directory is the root directory and not copy
the default layout xattr to the new subdirectory.

Change-Id: Ie0d286785bdbcd73e2ae60b429e66d5d54b44eef
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/33956
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jenkins
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
18 months agoLU-930 doc: move server man pages to server package 52/33952/2
Andreas Dilger [Wed, 2 Jan 2019 22:07:47 +0000 (15:07 -0700)]
LU-930 doc: move server man pages to server package

Don't install the ll_decode_filter_fid.8 and llverdev.8 man pages
with the client build.  These utilities are only on the server.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I713107f6e8f09866650245d5835f41db6abcab07
Reviewed-on: https://review.whamcloud.com/33952
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-8066 ldlm: match store macro prototype 20/33920/3
Andreas Dilger [Wed, 26 Dec 2018 09:18:45 +0000 (02:18 -0700)]
LU-8066 ldlm: match store macro prototype

Use size_t in the LDLM_POOL_SYSFS_WRITER_STORE() function declaration
to match the struct lustre_attr (*store) function prototype.

Test-Paramters: trivial
Fixes: 1196bd199e7 ("LU-8066 ldlm: move /proc/fs/lustre/ldlm to /sys")

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I41012a66507a048c63bfe6b6e732cf858ba5a5b1
Reviewed-on: https://review.whamcloud.com/33920
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11828 clio: fix incorrect invariant in cl_io_iter_fini() 15/33915/4
James Simmons [Wed, 2 Jan 2019 17:13:50 +0000 (12:13 -0500)]
LU-11828 clio: fix incorrect invariant in cl_io_iter_fini()

It was discovered during PFL testing that if LINVRNT() is enabled
that cl_io_iter_fini() will crash with the following backtrace:

    kernel: LustreError: 16009:0:(cl_io.c:439:cl_io_iter_fini())
            ASSERTION( io->ci_state == CIS_UNLOCKED ) failed
    kernel: cl_io_iter_fini+0x10c/0x110 [obdclass]
    kernel: cl_io_loop+0x46/0x220 [obdclass]
    kernel: cl_setattr_ost+0x1ed/0x2a0 [lustre]
    kernel: ll_setattr_raw+0x7b0/0x9a0 [lustre]
    kernel: notify_change+0x1dc/0x430
    kernel: do_truncate+0x72/0xc0
    kernel: do_sys_ftruncate+0xf5/0x160

This is due to the incorrect assumption that the ci_state will
always be CIS_UNLOCKED, but by looking at the behavior of
cl_io_loop() it can be seen that is not the case with PFL.
We do want to make sure the IO state is not in the middle of
some other action (up to CIS_IT_STARTED or CIS_IO_FINISHED or
later) when cl_io_iter_fini() is called.

Change-Id: I41c4d3ae5618d0e138f850e63edfd2dc6d4df19a
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/33915
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11770 osd-ldiskfs: clean up inline macro usage 41/33841/7
Andreas Dilger [Wed, 12 Dec 2018 23:28:38 +0000 (16:28 -0700)]
LU-11770 osd-ldiskfs: clean up inline macro usage

Remove inline macro usage as much as possible in the osd-ldiskfs
code, per Linux and Lustre code style.  Balance start/end braces
for functions to keep syntax highlighting working properly.

Change the autoconf macros to be checking for the new structure
fields instead of the old fields, and add compat macros for older
kernels so that they can later be removed without code churn.

Remove direct inode->truncate() usage and autoconf check, since
ldiskfs_truncate() has been exported since at least RHEL6.3.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ifc3187275c1841918228fac2368b18a9fa89a47d
Reviewed-on: https://review.whamcloud.com/33841
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11566 utils: fix lctl llog_print for large configs 15/33815/8
Andreas Dilger [Mon, 10 Dec 2018 10:18:19 +0000 (03:18 -0700)]
LU-11566 utils: fix lctl llog_print for large configs

If "lctl llog_print" is called for a large configuration, it will
overflow the 8KB buffer limit for OBD ioctl commands.  The kernel
snprintf calls try to overflow the supplied buffer.  Avoid that.
If the configuration is large, fetch the configuration records in
chunks and print them incrementally.

Add --start and --end options to llog_print and deprecate the use of
positional parameters, since positional parameters are increasingly
complex to parse as options are added, and are harder to use.

The callback for the configuration records will allow "lctl pool_*"
commands to be processed directly on the MGS.

Move existing llog_print test_60aa, test_60ab to conf-sanity as
test_123aa and test_123ab (rename set_param -F test_123 to test_123F).
Add new test_123ac and test_123ad for the new llog_print --start and
--end param, and update test_123aa to test old positional parameters.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ib7d2ae893033bd4594646c980b7d0ddbd2b3a089
Reviewed-on: https://review.whamcloud.com/33815
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11665 tests: check number of pages correctly 72/33772/5
Andreas Dilger [Tue, 4 Dec 2018 01:30:23 +0000 (18:30 -0700)]
LU-11665 tests: check number of pages correctly

On ARM or other 64KB PAGE_SIZE systems, check the read cache size
against the actual PAGE_SIZE instead of just checking the number
of pages being read.

Test-Parameters: trivial clientarch=aarch64 testlist=sanity
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I8fb90325a18b343c5f5af01df603a25fe33ebbe5
Reviewed-on: https://review.whamcloud.com/33772
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11721 utils: print used inodes ratio when using "lfs df -i" 58/33758/6
Nikitas Angelinas [Thu, 29 Nov 2018 23:23:05 +0000 (15:23 -0800)]
LU-11721 utils: print used inodes ratio when using "lfs df -i"

"lfs df -i" prints the used blocks percentage, instead of the used
inodes percentage. Fix this by allowing obd_statfs_ratio() to
distinguish when "-i" is used.

Round up the ratio returned from obd_statfs_ratio() in a ceiling
manner, to match the output of df(1). Add a sanity test to check
that the outputs from df(1) and lfs df match.

Signed-off-by: Nikitas Angelinas <nangelinas@cray.com>
Cray-bug-id: LUS-6748
Test-Parameters: trivial
Change-Id: I0b31ecb7371875c93bc07dda1f1c89e04d5b4576
Reviewed-on: https://review.whamcloud.com/33758
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
18 months agoLU-8955 sec: create new function sptlrpc_get_sepol() 21/24421/31
Sebastien Buisson [Fri, 16 Jun 2017 12:36:23 +0000 (14:36 +0200)]
LU-8955 sec: create new function sptlrpc_get_sepol()

Create new function sptlrpc_get_sepol() in ptlrpc/sec.c to compute
SELinux policy info, by calling new userland command l_getsepol.

The SELinux policy info syntax is the following:
<mode>:<name>:<version>:<hash>
where:
- <mode> is a digit telling if SELinux is in Permissive mode (0)
  or Enforcing mode (1)
- <name> is the name of the SELinux policy
- <version> is the version of the SELinux policy
- <hash> is the computed hash of the binary representation of the
  policy, as exported in /etc/selinux/<name>/policy/policy.<version>

Userland command l_getsepol can be called on the command line by a
security administrator to get SELinux status information to store into
'sepol' field of nodemap.

SELinux status information is reported by Lustre client only if
new 'send_sepol' ptlrpc kernel module's parameter is not zero, and
SELinux is enabled on the client.
'send_sepol' accepts various values:
- 0: do not send SELinux policy info;
- -1: send SELinux policy info for every request;
- N > 0: only send SELinux policy info every N seconds. Use max value
  2^31-1 (signed int on 32 bits) to make sure SELinux policy info is
  only checked at mount time.
Independently from 'send_sepol' value, SELinux policy info has an
associated mtime. l_getsepol checks mtime and recalculates whole
SELinux policy info (including SHA) only if mtime changed.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I7a4b955f36c5e0f4eca1561beae2b40fec88d570
Reviewed-on: https://review.whamcloud.com/24421
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-8955 nodemap: add SELinux policy info to nodemap 20/24420/27
Sebastien Buisson [Thu, 6 Dec 2018 09:40:45 +0000 (10:40 +0100)]
LU-8955 nodemap: add SELinux policy info to nodemap

Give the ability to set SELinux policy information on a nodemap,
in a new nodemap field named 'sepol'.
When set, a client pertaining to this nodemap will be allowed to
connect only if the SELinux policy information it sends matches
the one stored in the nodemap.

Expected 'sepol' string format is:
<1-digit>:<policy name>:<policy version>:<policy hash>
1-digit is 0 for SELinux Permissive mode, 1 for Enforcing mode.

SELinux policy info of nodemap is stored permanently by using
'lctl set_param -P' commands. It means MDS and OSS will be able to
retrieve SELinux policy info of nodemap after a restart (when they
read params llog).
MGS will not see SElinux policy info after restart, but this does
not prevent the feature from working.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ia1d178dd34f05ede020b490b1797a71dbae15d7b
Reviewed-on: https://review.whamcloud.com/24420
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-1365 tests: LARGEDIR and 3-level htree for local ldiskfs 09/22009/23
Artem Blagodarenko [Thu, 18 Aug 2016 17:36:29 +0000 (20:36 +0300)]
LU-1365 tests: LARGEDIR and 3-level htree for local ldiskfs

Add conf-sanity.sh test for the LARGEDIR feature with Lustre + ldiskfs
to exceed both 2-level htree (with 1024-byte blocksize to reduce the
number of entries per htree level) and 2GB directory size (marked SLOW
since this takes a while to run). This is done using a smaller number
of hard-linked inodes (nlink_max = 65000), to avoid the overhead of
accessing and caching a large number of different inodes.

Running multiple threads on a single directory is serialized by the
VFS directory mutex, so the tests are run in parallel on all available
clients and mountpoints to speed up the test as much as possible.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Artem Blagodarenko <c17828@cray.com>
Change-Id: If52ef346a39a1c3236871edbb0327bf44fe1a392
Test-Parameters: trivial testlist=conf-sanity envdefinitions=SLOW=yes,FLAKEY=false
Reviewed-on: https://review.whamcloud.com/22009
Tested-by: Jenkins
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
18 months agoLU-1365 utils: allow set block size for ldiskfs backend 57/33757/8
Artem Blagodarenko [Wed, 28 Nov 2018 20:37:53 +0000 (23:37 +0300)]
LU-1365 utils: allow set block size for ldiskfs backend

Add “-b” option to mkfs.lustre that allows to set backend block size.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Artem Blagodarenko <c17828@cray.com>
Change-Id: I83fc76f64ce2a0b4bf500841b695d64d3dea78de
Reviewed-on: https://review.whamcloud.com/33757
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
18 months agoLU-11790 ldiskfs: add terminating u32 when expanding inodes 93/33893/2
Li Dongyang [Wed, 19 Dec 2018 03:03:14 +0000 (14:03 +1100)]
LU-11790 ldiskfs: add terminating u32 when expanding inodes

In ext4_expand_extra_isize_ea(), we calculate the total size of the
xattr header, plus the xattr entries so we know how much of the
beginning part of the xattrs to move when expanding the inode extra
size.  We need to include the terminating u32 at the end of the xattr
entries, or else if there is uninitialized, non-zero bytes after the
xattr entries and before the xattr values, the list of xattr entries
won't be properly terminated.

Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Change-Id: I247b935b3cf315481dc4658133a7eee02b6350e9
Reviewed-on: https://review.whamcloud.com/33893
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
18 months agoLU-11889 tests: disable sanity/156 for ZFS OST 14/34114/3
Alex Zhuravlev [Sun, 27 Jan 2019 05:15:25 +0000 (08:15 +0300)]
LU-11889 tests: disable sanity/156 for ZFS OST

disable sanity/156 with ZFS OST as osd-zfs doesn't
maintain cache hit/miss stats yet.

Test-Parameters: trivial

Change-Id: I34eac3dcc49962ead0c0d6b17d8d605036872f4f
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/34114
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
18 months agoLU-11607 tests: create routine to get Lustre env 38/33938/5
James Nunez [Fri, 28 Dec 2018 16:23:32 +0000 (09:23 -0700)]
LU-11607 tests: create routine to get Lustre env

The Lustre tests in the test suites make repeated calls
to a small number functions that relate to the Lustre
environment. Some examples are the version of the Lustre
server or client and the file system type of the server.

Collect these calls into a routine called get_lustre_env()
in the test-framework.sh library and replace calls in
sanity with the global variables.

Test-Parameters: trivial
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I01dd00dd50cca39c964c5fd8abc3f51ab3c8e6b8
Reviewed-on: https://review.whamcloud.com/33938
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-9859 libcfs: add watchdog for ptlrpc service threads. 18/33018/24
NeilBrown [Mon, 19 Nov 2018 15:32:47 +0000 (10:32 -0500)]
LU-9859 libcfs: add watchdog for ptlrpc service threads.

If a ptlrpc thread takes longer than expected, it
can be used to get a stack trace to see where the
problem is.

This patch moves this functionality out of libcfs
by using a delayed work struct to report a back
trace of the thread if it ever fires.

Change-Id: I8f54df430322f0751b179f67284018e576defc08
Signed-off-by: NeilBrown <neilb@suse.com>
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/33018
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-8939 nodemap: remove deprecated lproc files 52/24352/8
Kit Westneat [Fri, 4 Jan 2019 15:02:21 +0000 (10:02 -0500)]
LU-8939 nodemap: remove deprecated lproc files

Before the MGS was able to transfer the nodemap configuration to the
MDSes and OSSes, people had to use lproc interfaces in order to load
the configuration. This patch removes those deprecated interfaces.

Signed-off-by: Kit Westneat <kit.westneat@gmail.com>
Change-Id: I8c8f38755cf2870fbabca9e96d7d51a9ac290193
Reviewed-on: https://review.whamcloud.com/24352
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-7239 mdd: make mdd_attr_set() synchronous less often 99/16699/7
Olaf Faaland [Tue, 6 Nov 2018 18:24:27 +0000 (10:24 -0800)]
LU-7239 mdd: make mdd_attr_set() synchronous less often

In permission_needs_sync(), test mode changes for S_ISUID or S_ISVTX
rather than simply checking whether the new mode in the incoming
request has those bits set.  Similarly check for uid and gid changes.

Prior to these changes, copying a directory tree into lustre, whose
directories' modes have S_ISUID | S_ISGID | S_ISVTX set, is
unnecessarily slow because chmod() system calls are synchronous
whether they change the actual permissions or not.

Change-Id: I41d3297a007eb4b796790b3b3de0ad355d17cb58
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Reviewed-on: https://review.whamcloud.com/16699
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-7159 tests: fix 224c fault injection 26/16426/20
Artem Blagodarenko [Tue, 15 Sep 2015 09:18:41 +0000 (12:18 +0300)]
LU-7159 tests: fix 224c fault injection

Sanity 224c test always passes because no fault injection happened.

This Patch adds additional step in fault injection and fix this test.

Test-Parameters: clientarch=aarch64 testlist=node-provisioning-1,lustre-initialization-1,lnet-selftest,sanity

Signed-off-by: Artem Blagodarenko <artem.blagodarenko@seagate.com>
Xyratex-bug-id: MRP-2472
Change-Id: I5f637e98b46ddda6acbafb956f9b85fccb7cae15
Reviewed-on: https://review.whamcloud.com/16426
Reviewed-by: Alexander Zarochentsev <c17826@cray.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
18 months agoLU-11852 osc: skip cache entry wakeup without grants 13/34013/2
Alexander Boyko [Fri, 11 Jan 2019 07:49:51 +0000 (02:49 -0500)]
LU-11852 osc: skip cache entry wakeup without grants

No need to wakeup osc_entry_cache waiters with EDQUOT or insufficent
grants, cause they will sleep again.
osc_enter_cache could skip queue of waiters if client has enough
grants. The fix adds list_empty checking and improves FIFO.

Signed-off-by: Alexander Boyko <c17825@cray.com>
Cray-bug-id: LUS-6119
Change-Id: I0a2415a19b2c8ecd7df3411d88cab9fee2470a10
Reviewed-on: https://review.whamcloud.com/34013
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Andrew Perepechko <c17827@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11845 zfs: put configure checks in version order 00/34000/2
Andreas Dilger [Wed, 9 Jan 2019 21:35:10 +0000 (14:35 -0700)]
LU-11845 zfs: put configure checks in version order

Put the ZFS feature checks in release version order, so that it is
easier to track when they apply and when they can be removed in
the future.

Make the configure checks use decrypt=B_TRUE just to illustrate
more correct usage.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I22053638d72b41b51b6f56dea5668e78535cab07
Reviewed-on: https://review.whamcloud.com/34000
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11716 tests: fix scrub tests 50/33750/3
Alexander Boyko [Mon, 1 Oct 2018 12:47:34 +0000 (08:47 -0400)]
LU-11716 tests: fix scrub tests

scrub_prep set MDT failloc to 0x198. It prevents the inserting
of osd_obj to OI, and brokes the logic of osd_fid_lookup. The
creating and searching quota slv file could happen during this
failloc. osd_fid_lookup() returned osd object with oo_inode equal
to zero and a bit later, osd_index_try got BUG cause it tried to
access to &LDISKFS_I(inode)->i_es_lock.

The patch adds OST object creation for quota synchronization before
setting failloc.

Test-Parameters: trivial testlist=sanity-scrub
Signed-off-by: Alexander Boyko <c17825@cray.com>
Cray-bug-id: LUS-6051
Change-Id: I8a37c476c2c95b235b0f0ecc7ec1ab0066ae3883
Reviewed-on: https://review.whamcloud.com/33750
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11625 ofd: handle upgraded filter_fid properly 27/33627/5
Andreas Dilger [Wed, 7 Nov 2018 02:40:18 +0000 (19:40 -0700)]
LU-11625 ofd: handle upgraded filter_fid properly

Since there have been several iterations of struct filter_fid stored
on disk, the current code wasn't checking for all of the possible
cases when trying to decide what action to take when accessing and
upgrading the xattr for new capabilities.

Properly check for the various different struct filter_fid sizes and
handle them appropriately.  Add a more verbose description of the
various cases so that this is more clear to others in the future.

Add decoding of filter_fid fields added for FLR in 2.11.

We should already be testing for upgrading the filter_fid xattr
from different OST versions in conf-sanity test_32d.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ifef2292296236cb06ff7e8cd50caff4b133ebbe5
Reviewed-on: https://review.whamcloud.com/33627
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11555 utils: ZFS check multihost enabled in read_ldd() 91/33491/5
Nathaniel Clark [Sat, 3 Nov 2018 04:03:43 +0000 (00:03 -0400)]
LU-11555 utils: ZFS check multihost enabled in read_ldd()

For ZFS check that multihost is enabled if failover host is defined.
Print a warning if it's not.

Test-Parameters: trivial mdtfilesystemtype=zfs ostfilesystemtype=zfs
Signed-off-by: Nathaniel Clark <nclark@whamcloud.com>
Change-Id: Iddb5871afc6fb6808a25921c8d3e8516d675f15c
Reviewed-on: https://review.whamcloud.com/33491
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-10953 tests: disable sanity/300k on small setups 48/32148/3
Alex Zhuravlev [Wed, 25 Apr 2018 13:49:26 +0000 (16:49 +0300)]
LU-10953 tests: disable sanity/300k on small setups

sanity/300k simulates creation of 512-stripes directory and
fails on small setup causing all subsequent tests to fail as
well.

Change-Id: I5e1d8d25ea364c0f49d9e114cc447cc40f951e2f
Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-on: https://review.whamcloud.com/32148
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11838 lnet: change lnet_ipaddr_enumerate() to use for_each_netdev() 69/33969/3
NeilBrown [Mon, 7 Jan 2019 19:23:19 +0000 (14:23 -0500)]
LU-11838 lnet: change lnet_ipaddr_enumerate() to use for_each_netdev()

for_each_netdev() is a more direct interface than
lnet_ipif_enumerate(), so use it instead.  Also get
address and 'up' status directly from the device.

This means we need to possible re-allocate the storage
space if there are lots of IP addresses.

However there is no need to resize the allocation down if we
over-allocated.  This is only used once, and is freed soon
after it is allocated, so that is a false optimization.

Linux-commit: 0400cf406c32ac3968241cd528747d922b6c55c3

Change-Id: I1c1e7722c7b2b267dcb8134ae295a54f976d96ad
Signed-off-by: NeilBrown <neilb@suse.com>
Reviewed-on: https://review.whamcloud.com/33969
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Doug Oucharek <dougso@me.com>
Reviewed-by: Sonia Sharma <sharmaso@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11838 kernel: harden current_time autoconf test 63/33963/2
James Simmons [Mon, 7 Jan 2019 16:46:38 +0000 (11:46 -0500)]
LU-11838 kernel: harden current_time autoconf test

In newer kernels CURRENT_TIME was replaced by current_time(). The
return value of current_time() was struct timespec but to support
time after 2038 the return value was changed to struct timespec64.
This change broke the autoconf test. The solution is to use one
of the struct iattr field in the autoconf test since it hides
the return value type.

Test-Parameters: trivial

Change-Id: I95abd2cd2b777f99cbf6ab78370ee2171e5fca67
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/33963
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Ben Evans <bevans@cray.com>
18 months agoLU-9325 msg: replace obsolete simple_strtoul use 59/33959/4
James Simmons [Tue, 8 Jan 2019 14:59:26 +0000 (09:59 -0500)]
LU-9325 msg: replace obsolete simple_strtoul use

The kernel function simple_strtoul() has been superseded by the
kstrto* family of functions. Replace all uses of simple_strto*()
with the equivalent kstrto*() function in the msg subsystem.

Change-Id: I5ca5df29b8b2f34107ee40628ab3b77d57547b99
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/33959
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-10447 tests: replace $SET/$GETSTRIPE in replay tests 18/33918/6
James Nunez [Tue, 8 Jan 2019 21:04:06 +0000 (14:04 -0700)]
LU-10447 tests: replace $SET/$GETSTRIPE in replay tests

$SETSTRIPE and $GETSTRIPE were needed when we used the
standalone 'lstripe' utility. 'lstripe' hasn't been used
for years and we need to clean up all remnants of it.

Replace all instances of $SETSTRIPE with '$LFS setstripe'
and $GETSTRIPE with '$LFS getstripe' in the replay-dual,
replay-ost-single, and replay-single test suites.

Test-Parameters: trivial testlist=replay-dual,replay-ost-single,replay-single
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I3902142061ddb3c8860b6983d5ec55383aeb25a9
Reviewed-on: https://review.whamcloud.com/33918
Reviewed-by: Elena Gryaznova <c17455@cray.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-10966 utils: Fix `lfs check` documentation and arguments 75/33775/6
George Melikov [Sun, 13 Jan 2019 22:50:58 +0000 (17:50 -0500)]
LU-10966 utils: Fix `lfs check` documentation and arguments

Several bugs exist for 'lfs check'. Add checking for MGS servers.
Add 'lfs check mdts' which is inline with the other options but
still keep mds for backwards compatibility. Do the same with the
'all' option which is equivalent to older 'servers' that is also
kept for backwards compatibility. Update man pages to reflect
these changes

Change-Id: Ifffec8fd1498616036122f9af311cd6366d693bc
Signed-off-by: George Melikov <mail@gmelikov.ru>
Reviewed-on: https://review.whamcloud.com/33775
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11418 mdd: delete name if orphan doesn't exist 61/33661/7
Lai Siyao [Tue, 23 Oct 2018 11:17:20 +0000 (19:17 +0800)]
LU-11418 mdd: delete name if orphan doesn't exist

mdd_orphan_destroy() should delete name if orphan object doesn't
exist, otherwise the orphan clean thread will try to destroy this
orphan in dead loop.

add sanity test_811.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Id22b2fab0ac87dfb81ca9f01d8ed0338f1b12120
Reviewed-on: https://review.whamcloud.com/33661
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11402 tests: add version check sanity-quota 60 18/33418/8
James Nunez [Wed, 2 Jan 2019 22:38:58 +0000 (15:38 -0700)]
LU-11402 tests: add version check sanity-quota 60

sanity-quota test 60 was added to Lustre tag 2.11.53. Thus,
we need to check that the server is 2.11.53 or later before
running test 60.

Test-Parameters: trivial serverjob=lustre-b2_10 serverbuildno=152 testlist=sanity-quota
Test-Parameters: testlist=sanity-quota
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I5db738f0776b5615165977df1708f197f215b994
Reviewed-on: https://review.whamcloud.com/33418
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11843 tests: customise sanity-quota 97/33997/3
Elena Gryaznova [Wed, 9 Jan 2019 11:57:04 +0000 (14:57 +0300)]
LU-11843 tests: customise sanity-quota

Patch adds the possibility to not turn project quotas OFF
at the end of sanity-quota if it is ON initially.
ENABLE_PROJECT_QUOTA=false is ignored in this case.

Signed-off-by: Elena Gryaznova <c17455@cray.com>
Cray-bug-id: LUS-6162
Reviewed-by: Alexander Boyko <c17825@cray.com>
Reviewed-by: Vladimir Saveliev <c17830@cray.com>
Test-Parameters: trivial testlist=sanity-quota
Change-Id: Ic7d58f4a077acdbeec426613777e3c7c0778f0a0
Reviewed-on: https://review.whamcloud.com/33997
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexandr Boyko <c17825@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11838 lustre: discard LTIME_S macro 84/33984/3
NeilBrown [Tue, 8 Jan 2019 17:44:35 +0000 (12:44 -0500)]
LU-11838 lustre: discard LTIME_S macro

Rather than using a macro, just access the required
field directly.

Linux-commit: 5b7cc4e4ce3cc15f67462ae75c55eecc7edc3a40

Change-Id: I325cac7458265d1cf6ad9f195a513f2612865906
Signed-off-by: NeilBrown <neilb@suse.com>
Reviewed-on: https://review.whamcloud.com/33984
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Ben Evans <bevans@cray.com>
18 months agoLU-11834 llite: fix temporary instance buffer size 51/33951/2
Andreas Dilger [Wed, 2 Jan 2019 22:12:11 +0000 (15:12 -0700)]
LU-11834 llite: fix temporary instance buffer size

The formatting of the cfg_instance variable was changed in LU-11809
to always use a fixed "%016llu" format, but the temporary buffer
allocations for the instance string were not changed to match the
printed value.

This results in string truncation in some situations.  Change the temp
buffer size to always have 16 bytes for the instance instead of using
sizeof(cfg_instance).

Fixes: cd294a1255 ("LU-11809 llite: don't use %p for cfg_instance")

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I5eca99afa2787cc57e739489b252b12af68cab07
Reviewed-on: https://review.whamcloud.com/33951
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11825 clio: Remove pio feature 12/33912/2
Patrick Farrell [Sun, 23 Dec 2018 21:04:08 +0000 (16:04 -0500)]
LU-11825 clio: Remove pio feature

The pio feature was an interesting concept, but never
achieved real use in production, primarily because it was
actually slower than normal i/o for some common workloads,
so it could not be on by default.

It has had a trivial deadlock in it (with truncate) since
release, and this has never been reported by a customer.

The attemp to rewrite readahead to use ptasks was never
fully successful, and is being superseded by a different
implementation.

Given all this, there is no reason to keep the extra
complexity in the already complicated clio code.

Signed-off-by: Patrick Farrell <paf@cray.com>
Change-Id: Ie558db946af94189e35e985e8894706c14ed5f39
Reviewed-on: https://review.whamcloud.com/33912
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Wang Shilong <wshilong@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11810 misc: allow Fixes: tag in commit signoff block 88/33888/4
Andreas Dilger [Tue, 18 Dec 2018 19:01:04 +0000 (12:01 -0700)]
LU-11810 misc: allow Fixes: tag in commit signoff block

Allow the "Fixes:" tag in the signoff block of the commit message.
The Fixes: tag should contain a valid git commit hash, and may
optionally be followed by a description of the original patch.
For example:

Fixes: 5760b34c48b ("LU-1145 test: add Test-Parameters tag")
Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I49f712dd8da173510e5941b66eb050e53a1cab07
Reviewed-on: https://review.whamcloud.com/33888
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <paf@cray.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11683 build: Update ZFS to 0.7.12 87/33687/3
Nathaniel Clark [Mon, 19 Nov 2018 22:07:31 +0000 (17:07 -0500)]
LU-11683 build: Update ZFS to 0.7.12

0.7.12

ZFS
 * Add BuildRequires gcc, make, elfutils-libelf-devel #8095
 * Fix flake8 "invalid escape sequence 'x'" warning #8056
 * ZTS: Update O_TMPFILE support check #7528
 * Allow use of pool GUID as root pool #8052
 * Fix arc_release() refcount #8000
 * Prefix all refcount functions with zfs_ #7963
 * Linux 4.19-rc3+ compat: Remove refcount_t compat #7885
 * Zpool iostat: remove latency/queue scaling #7945
 * Fix statfs(2) for 32-bit user space #7122
 * Skip import activity test in more zdb code paths #7797
 * Reduce taskq and context-switch cost of zio pipe #7736
 * Fix race in dnode_check_slots_free() #7147

SPL
 * Add BuildRequires gcc, make, elfutils-libelf-devel #8095
 * Linux 4.20 compat: current_kernel_time() #8074
 * Backport vnode.h changes from zfs:93ce2b4c
 * Define timestruc_t for Lustre compatibility #8014

0.7.11

ZFS
 * tx_waited -> tx_dirty_delayed in trace_dmu.h #7096
 * Revert "zpool reopen should detect expanded devices"

SPL
 * Linux 4.18 compat: Kconfig #7890)

0.7.10

ZFS
 * Correctly handle errors from kern_path #7764
 * Fix build with CONFIG_GCC_PLUGIN_RANDSTRUCT {{f4ee628}}
 * Fix 'zfs recv' of non large_dnode send streams #7617
 * Fix object reclaim when using large dnodes #7097
 * Fix problems receiving reallocated dnodes #6821
 * Fedora 28: Fix misc bounds check compiler warnings #7826
 * Fix libaio-devel requirement for Debian-based distributions #7829
 * Add libaio-devel BuildRequires #7821
 * Add missing zfs-dracut RPM dependencies #7729
 * Don't modify argv] in user tools #7760
 * Fix arcstat.py handling of unsupported options #7799
 * Allow inherited properties in zfs_check_settable() #7755
 * Fix zfs incremental send remove '-o' properties #7478
 * OpenZFS 8906 - uts: illumos rootfs should support salted cksum
 * Fix zpl_mount() deadlock #7598
 * Fix kernel unaligned access on sparc64 #7642
 * Default ashift for Amazon EC2 NVMe devices #7676
 * Linux 4.14 compat: blk_queue_stackable() #7645
 * Add pool state /proc entry, "SUSPENDED" pools #7331
 * zpool reopen should detect expanded devices #7546
 * Support Debian DKMS builds #7358
 * module param callbacks check for initialized spa #7496
 * Trim new line from zfs_vdev_scheduler #3356
 * Fix ENOSPC in "Handle zap_add() failures in ..." #7401
 * Fix divide-by-zero in mmp_delay_update() #7391
 * OpenZFS 8997 - ztest assertion failure in zil_lwb_write_issue #7084
 * Linux compat 4.18: check_disk_size_change() #7611
 * Linux 4.18 compat: inode timespec -> timespec64 #7643
 * zv_suspend_lock in zvol_open()/zvol_release() #6342

SPL
 * Add rwsem_tryupgrade for 4.9.20-rt16 kernel (SPL) #7589
 * Add pool state /proc entry, "SUSPENDED" pools (SPL) #7331
 * Linux 4.18 compat: inode timespec -> timespec64 #7643

Signed-off-by: Nathaniel Clark <nclark@whamcloud.com>
Change-Id: I7b108d659d9d6cd6df3d4c55d8b8380f68ccaa6e
Reviewed-on: https://review.whamcloud.com/33687
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11661 test: improve sanityn test_47g 47/33647/4
Lai Siyao [Mon, 22 Oct 2018 11:51:49 +0000 (19:51 +0800)]
LU-11661 test: improve sanityn test_47g

'stat' may be run before 'mkdir', to avoid this, sync data before
test and wait longer time after 'mkdir' in background.

Test-Parameters: trivial testlist=sanityn,sanityn,sanityn,sanityn mdtfilesystemtype=zfs mdscount=2 mdtcount=4 envdefinitions=ONLY=47g
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I314bc9d36629a5185efc5ef8281a03337ea77776
Reviewed-on: https://review.whamcloud.com/33647
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <paf@cray.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11579 llite: remove cl_file_inode_init() LASSERT 05/33505/4
Andreas Dilger [Mon, 29 Oct 2018 06:42:46 +0000 (00:42 -0600)]
LU-11579 llite: remove cl_file_inode_init() LASSERT

If there is some corruption or other reason that the file layout
cannot be used, the first call to cl_file_inode_init() will fail.
If it is called a second time on the same file then it will hit
an LASSERT() since I_NEW is no longer set on the inode.

It would be good to handle the error in lov_init_raid0() better,
but we still want to avoid this LASSERT() if there is an error.

Convert the LASSERT() in cl_file_inode_init() into a CERROR() and
error return.  This is being triggered due to corruption on the
server, but that shouldn't cause the client to assert.

    lov_dump_lmm_common() oid 0xdf4e:311367, magic 0x0bd10bd0
    lov_dump_lmm_common() stripe_size 1048576, stripe_count 4
    lov_dump_lmm_objects() stripe 0 idx 10 subobj 0x0:151194471
    lov_dump_lmm_objects() stripe 1 idx 12 subobj 0x0:152477530
    lov_dump_lmm_objects() stripe 2 idx 25 subobj 0x0:151589797
    lov_dump_lmm_objects() stripe 3 idx 2 subobj 0x0:150332564
    lov_init_raid0() fsname-clilov: OST0019 is not initialized
    cl_file_inode_init() Failure to initialize cl object
        [0x20004c047:0xdf4e:0x0]: -5

    cl_file_inode_init() ASSERTION(inode->i_state & (1 << 3) ) failed
    cl_file_inode_init() LBUG
    Pid: 37233, comm: ll_sa_4709 3.10.0-862.14.4.el7.x86_64 #1 SMP
    Call Trace:
    libcfs_call_trace+0x8c/0xc0 [libcfs]
    lbug_with_loc+0x4c/0xa0 [libcfs]
    cl_file_inode_init+0x2ac/0x300 [lustre]
    ll_update_inode+0x315/0x600 [lustre]
    ll_iget+0x163/0x350 [lustre]
    ll_prep_inode+0x232/0xc80 [lustre]
    sa_handle_callback+0x3a4/0xf70 [lustre]
    ll_statahead_thread+0x40e/0x2080 [lustre]

Instead, return an IO error instead of killing the client.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: If8a6eb24df09e7e158b61f02e2517132893ebbe5
Reviewed-on: https://review.whamcloud.com/33505
Reviewed-by: Patrick Farrell <paf@cray.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11496 tests: start_client_loads () fix 39/33339/10
Elena Gryaznova [Wed, 10 Oct 2018 15:55:17 +0000 (18:55 +0300)]
LU-11496 tests: start_client_loads () fix

recovery-*-scale tests are not functional with LOADS=IOR
because of MPIRUN and MPIRUN_OPTIONS are not set.
Patch fixes this defect.

Test-Parameters: trivial
Signed-off-by: Elena Gryaznova <c17455@cray.com>
Cray-bug-id: LUS-6220
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Alexander Boyko <c17825@cray.com>
Change-Id: I9b6d3810978260c0cec4a5482f093f788651aac0
Reviewed-on: https://review.whamcloud.com/33339
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexandr Boyko <c17825@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-10447 tests: deprecate $SETSTRIPE/$GETSTRIPE in sanity 09/32809/8
James Nunez [Wed, 26 Dec 2018 20:25:27 +0000 (13:25 -0700)]
LU-10447 tests: deprecate $SETSTRIPE/$GETSTRIPE in sanity

$SETSTRIPE and $GETSTRIPE were needed when we used the
standalone 'lstripe' utility. 'lstripe' hasn't been used
for years and we need to clean up all remnants of it.

Replace all instances of $SETSTRIPE with '$LFS setstripe'
and $GETSTRIPE with '$LFS getstripe' in the sanity and
sanityn test suites.

Test-Parameters: trivial testlist=sanityn
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I492d261c7568634c8f0dd0d5a100ef5bb8433ad8
Reviewed-on: https://review.whamcloud.com/32809
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
18 months agoLU-8955 cfg: reserve flags for SELinux status checking 97/33797/7
Sebastien Buisson [Thu, 6 Dec 2018 09:35:47 +0000 (10:35 +0100)]
LU-8955 cfg: reserve flags for SELinux status checking

Reserve LCFG_NODEMAP_SET_SEPOL config flag that will be used to
define sepol parameter on nodemap entries.
Reserve OBD_CONNECT2_SELINUX_POLICY connection flag that will be set
(in ocd_connect_flags2) if a client supports sending the SELinux
policy status info.
Add checks for all lcfg_command_type constants, along with lustre_cfg
and cfg_record_type.

Test-Parameters: trivial
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I009eb8f1597b51acce7dc21498a43096e41cd034
Reviewed-on: https://review.whamcloud.com/33797
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <farr0186@gmail.com>
18 months agoLU-10937 sptlrpc: split sptlrpc_process_config() 60/33760/4
James Simmons [Fri, 4 Jan 2019 12:37:52 +0000 (07:37 -0500)]
LU-10937 sptlrpc: split sptlrpc_process_config()

Make sptlrpc_process_config() more than a single line wapper
exporting function. Instead migrate the lcfg parsing out of
__sptlrpc_process_config() so that we can use this function
for both LCFG_PARAM and LCFG_SET_PARAM handling.

The first field parsed from struct lustre_cfg *lcfg is the
target. This can be "_mgs", file system name, or an obd target
i.e fsname-MDT0000. We can move to extracting the file system
name out of the target string using server_name2fsname().

Test-Parameters: testlist=sanity-sec envdefinitions=SHARED_KEY=true

Change-Id: If94ee8164ab2b49fe10d098b32faea9e331f04ac
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/33760
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
18 months agoLU-11647 ptlrpc: always unregister bulk 78/22378/6
Hongchao Zhang [Thu, 15 Nov 2018 16:21:15 +0000 (11:21 -0500)]
LU-11647 ptlrpc: always unregister bulk

In ptlrpc_check_set, the bulk should be unregistered before
ptl_send_rpc in any case.

Change-Id: Icf963002f934b43ccbb9d6ef02ba7f9d11f297f8
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/22378
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>