Whamcloud - gitweb
fs/lustre-release.git
3 years agoNew tag 2.10.57 2.10.57 v2_10_57 v2_10_57_0
Oleg Drokin [Wed, 17 Jan 2018 07:25:01 +0000 (02:25 -0500)]
New tag 2.10.57

Change-Id: Ic2704bf2256afdf0800a30c1a979e3d99f2c208a
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-7004 obd: make LCFG_SET_PARAM functional 90/28590/26
James Simmons [Wed, 13 Dec 2017 20:05:56 +0000 (15:05 -0500)]
LU-7004 obd: make LCFG_SET_PARAM functional

The LCFG_SET_PARAM infrastructure was meant to replace the
class_process_proc_param() functionality but various software
bugs have prevented its adoption. This patch does the following:

1) Take the better print_lustre_cfg() of the mgs module and use
   that in llog_swab.c instead with the intent of exporting this
   function. I add to process_param2_config() a call to
   print_lustre_cfg() for debugging purposes.

2) Move obdname2fsname to obd_mount.c and make it exportable.
   Expanded the functionality to work for both lctl conf_param
   and lctl set_parm -P.

3) Split mgs_setparam() into two functions since the difference
   in LCFG_SET_PARAM and LCFG_PARAM are large enough.

Currently virtual attributes failover.nid, sptlrpc, and quota
are not fully supported. They will be addressed in later patches.

Change-Id: Iced6505f39a3270139c1630270cfe1dc4a2e49ed
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/28590
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10455 kernel: kernel update RHEL7.4 [3.10.0-693.11.6.el7] 34/30734/3
Bob Glossman [Thu, 4 Jan 2018 15:57:37 +0000 (07:57 -0800)]
LU-10455 kernel: kernel update RHEL7.4 [3.10.0-693.11.6.el7]

update RHEL 7.4 kernel to 3.10.0-693.11.6.el7

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: Id3428aa00e4b1501b642587db7911b6adafd51ef
Reviewed-on: https://review.whamcloud.com/30734
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-8999 test: ignore unrelated quota id 30/30730/3
Hongchao Zhang [Wed, 6 Dec 2017 04:59:23 +0000 (12:59 +0800)]
LU-8999 test: ignore unrelated quota id

In test_38 of sanity_quota, the quota id larger than 9999
should be ignored.

Change-Id: I12e7936c0c1abc2dcaad7646a048c98bb37de254
Signed-off-by: Hongchao Zhang <hongchao.zhang@intel.com>
Reviewed-on: https://review.whamcloud.com/30730
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-9859 libcfs: delete libcfs/linux/libcfs.h 06/30706/4
James Simmons [Tue, 9 Jan 2018 05:52:16 +0000 (00:52 -0500)]
LU-9859 libcfs: delete libcfs/linux/libcfs.h

Lustre uses libcfs.h as a the header to include all headers. This
approach has drawbacks like colliding with MOFED compat headers
that do the same thing. This patch is the first step to unwind
including libcfs.h everywhere. This starts with eliminating
linux/libcfs.h.

Test-Parameters: trivial

Change-Id: Id2040d4295c16135561c8251e160cb2117ee21b8
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/30706
Reviewed-by: Doug Oucharek <dougso@me.com>
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
3 years agoLU-10052 tests: wait for OST objects to be deleted 78/30678/4
Hongchao Zhang [Mon, 4 Dec 2017 19:47:30 +0000 (03:47 +0800)]
LU-10052 tests: wait for OST objects to be deleted

In test_20b of replay-single, the used space difference after
the file creation and deletion shows that a block is not freed,
wait for OST objects to be destroyed after recovery is done.

Test-Parameters: trivial testlist=replay-single.sh ostfilesystemtype=zfs mdtfilesystemtype=zfs
Signed-off-by: Hongchao Zhang <hongchao.zhang@intel.com>
Change-Id: I6311d8b8fa4cea713a9755cfb6a3d63e693c8344
Reviewed-on: https://review.whamcloud.com/30678
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
3 years agoLU-10383 hsm: flatten mdt_cdt_started_cb() 61/30561/4
John L. Hammond [Fri, 15 Dec 2017 20:19:46 +0000 (14:19 -0600)]
LU-10383 hsm: flatten mdt_cdt_started_cb()

Rewrite mdt_cdt_started_cb() to avoid creating a fake progress kernel
for mdt_hsm_update_request_state() and handle the cleanup from the
timedout action directly. Cancel cancel actions that have timedout
rather than leaving them in the log indefinitely. The code is improved
in several places to clean up all resources associated with the action
rather than having the clean up depend on unnecessary assumptions.

Since mdt_hsm_coordinator_update() in then only called from the
MDS_HSM_PROGRESS handler, the update_record parameter can be removed
aw well as the now useless wrapper function
mdt_hsm_coordinator_update().

Test-Parameters: trivial testlist=sanity-hsm
Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ic6663b29b2a87de0da59085ccbe297b50abd049d
Reviewed-on: https://review.whamcloud.com/30561
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10383 hsm: consolidate CDT restore handle handling 57/30557/4
John L. Hammond [Fri, 15 Dec 2017 19:24:32 +0000 (13:24 -0600)]
LU-10383 hsm: consolidate CDT restore handle handling

Consolidate duplicated HSM coordinator restore handle handling into
new functions cdt_restore_handle_{add,del_(). Rename
mdt_hsm_restore_hdl_find() and some struct members for consistency.

Test-Parameters: trivial testlist=sanity-hsm
Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I9798ed93ea26a9d61d4786540c6dae95cdc38c4b
Reviewed-on: https://review.whamcloud.com/30557
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10383 hsm: refactor mdt_coordinator_cb() 52/30552/3
John L. Hammond [Fri, 15 Dec 2017 16:14:11 +0000 (10:14 -0600)]
LU-10383 hsm: refactor mdt_coordinator_cb()

Split the ARS_WAITING and ARS_STARTED cases of mdt_coordinator_cb()
into subfunctions, mdt_cdt_waiting_cb() and mdt_cdt_started_cb().

Test-Parameters: trivial testlist=sanity-hsm
Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I734e10e4db72f76a6b0de76c383ad0b03efd76d8
Reviewed-on: https://review.whamcloud.com/30552
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-6051 lfs: Update lfs_migrate man page for in-use files 50/29950/2
Steve Guminski [Mon, 6 Nov 2017 17:46:30 +0000 (12:46 -0500)]
LU-6051 lfs: Update lfs_migrate man page for in-use files

Update man page to state that it is safe to use the script on in-use
files for versions at or above 2.5.

Test-Parameters: trivial
Signed-off-by: Steve Guminski <stephenx.guminski@intel.com>
Change-Id: I412ee7681db3860ca395c4afc2a30c87f1f49d6d
Reviewed-on: https://review.whamcloud.com/29950
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-5541 build: move libcfs and liblustreapi over to libtool 62/30562/6
James Simmons [Mon, 8 Jan 2018 22:30:23 +0000 (17:30 -0500)]
LU-5541 build: move libcfs and liblustreapi over to libtool

Change libcfs into a convenience library using libtool. This allows
use to embbed libcfs library into both liblnetconfig and liblustreapi
so their is no longer a need to link applications to libcfs.a
anymore. With this change we need migrate liblustreapi to libtool.

libtool knows how to build both static and dymanic libraries for
liblusteapi, so no need to hack the Makefile. As two added benefits,
the utilities will now use the dynamic version, thus reducing their
footprint, and calling make twice in a row won't rebuild objects
already built.

Test-Parameters: trivial

Change-Id: Icc1e5d42df503b9bf393396fe09f4e4f1f242486
Signed-off-by: frank zago <fzago@cray.com>
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/30562
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-8358 vvp: Print discarded page warning on -EIO 11/21111/3
Patrick Farrell [Thu, 27 Jul 2017 15:18:08 +0000 (10:18 -0500)]
LU-8358 vvp: Print discarded page warning on -EIO

On client eviction, the client sometimes has dirty pages
outstanding, which are then discarded.  The client is
supposed to print an error when this happens,
from vvp_vmpage_error->ll_dirty_page_discard_warn.

However, the client looks for specific errors, and newer
Lustre clients will sometimes return -EIO to I/O requests
on eviction, instead of -EINTR.  Since they can still
return -EINTR, we must add -EIO as a new condition and
keep -EINTR.

Signed-off-by: Patrick Farrell <paf@cray.com>
Change-Id: I22ac82570a3840782c3fc6db40281b4a2c1cba1c
Reviewed-on: https://review.whamcloud.com/21111
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Chris Horn <hornc@cray.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-3846 test: Fix sanity test_56* with different layouts 79/7479/13
Andreas Dilger [Fri, 7 Jul 2017 16:51:17 +0000 (10:51 -0600)]
LU-3846 test: Fix sanity test_56* with different layouts

Fix a bug in "lfs getstripe --obd" and "lfs find --obd" where they
tried to access objects of uninitialized components to check if
they were on specified OSTs.  Those components have no objects, so
skip uninitialized components when searching for specific OSTs.

Sanity test_56s and test_56u will fail when the default stripe count
is not 1 and the test is run with more than one OST.  Explicitly
set stripe_count=1 for the default directory layout for these tests.

For PFL layout testing, test_56a needs to fix its output parsing, as
it is using this to verify that lfs getstripe is returning valid data
by counting occurrences of "obdidx" and not the new "l_ost_idx".

Do not delete the filesystem default striping in 56a, 56g, 56h, as
this will silently cause failures with default PFL layout testing.

The sanity test_56* subdirectories were allowed to be shared long ago,
but when $tdir became a per-subtest directory this was lost.  Allow
the subdirectories to be shared again, where possible, to avoid
duplicate setup of the test directory for each subtest if not needed.

Pass test directory explicitly to setup_56() and not via global $TDIR.

Clean up test_56* to better match modern test code style.

Test-Parameters: trivial mdscount=2 mdtcount=4 testlist=sanity
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I1bcdeb80fc6e39227a87365f823879db70eec652
Reviewed-on: https://review.whamcloud.com/7479
Tested-by: Jenkins
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
3 years agoLU-10444 utils: Don't remount debugfs every time 75/30675/3
Oleg Drokin [Sat, 30 Dec 2017 03:16:30 +0000 (22:16 -0500)]
LU-10444 utils: Don't remount debugfs every time

Check if debugfs is mounted at /sys/kernel/debug and only
mount if it is not.

Change-Id: Ib31bd8f7c5c93ab942c6708ed3a4d17a11159e95
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: https://review.whamcloud.com/30675
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
3 years agoLU-9019 osp: migrate to 64 bit time 74/30674/3
James Simmons [Thu, 4 Jan 2018 03:35:41 +0000 (22:35 -0500)]
LU-9019 osp: migrate to 64 bit time

Change opd_statfs_maxage from int to time64_t to make it clear
this field is in units of seconds. Change the last libcfs specific
cfs_time_t which maps to jiffies to ktime_t since it give better
than second resolution which is needed in this case.

Change-Id: I31baa73d5f6bd53dbcce4fc9f90462b11c6457a3
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/30674
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-9019 osc: migrate to 64 bit time 07/30607/4
James Simmons [Thu, 4 Jan 2018 03:24:57 +0000 (22:24 -0500)]
LU-9019 osc: migrate to 64 bit time

Change od_contention_time from int to time64_t to make it clear
this field is in units of seconds. Change the *_contention_time
fields from jiffies to ktime_t to make it clear we are dealing
with time and ktime_t is consistent on any platform unlike
jiffies.

Change-Id: Ieb240e40cc4d56050607314db057004db00aae13
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/30607
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-9892 test: fix SuSe nfsserver setup 76/30476/23
Minh Diep [Mon, 11 Dec 2017 18:12:20 +0000 (10:12 -0800)]
LU-9892 test: fix SuSe nfsserver setup

Checking for SuSE-release and use nfsserver
Add export info to a /etc/exports

Test-Parameters: trivial testlist=parallel-scale-nfsv4

Change-Id: Id12370ae35d878e51bdf6f71a77b1b82b5e82c33
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/30476
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10350 tests: make parsing routines pattern aware 36/30636/2
James Nunez [Thu, 21 Dec 2017 21:23:40 +0000 (14:23 -0700)]
LU-10350 tests: make parsing routines pattern aware

'lfs getstripe' now returns the pattern for each component
of a directory and files. The routines that parse
parameters, parse_layout_param() and parse_plain_param(),
need to look for the component pattern when parsing the output
of 'lfs getstripe'.

Test-Parameters: trivial testlist=sanity-pfl,ost-pools
Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: Iab605f58e9c8f501fa0889806c511e3310cb6dd7
Reviewed-on: https://review.whamcloud.com/30636
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10237 mdc: interruptable during RPC retry for EINPROGRESS 66/30166/2
Fan Yong [Sun, 19 Nov 2017 05:55:11 +0000 (13:55 +0800)]
LU-10237 mdc: interruptable during RPC retry for EINPROGRESS

Sometimes, some system resource may be inaccessible temporarily,
for example, related OI mapping is crashed and has yet not been
rebuilt. Under such case, the server will reply the client with
"-EINPROGRESS", then client will retry the RPC some time later.

Currently, the client will retry infinitely until related RPC
succeed or get other failure. But we do not know how long it
will be before related resource becoming available. It may be
very long time as to the RPC sponsor - the application or the
user does not want to retry any more, then we need to make the
logic to be interruptable. This patch is for such purpose.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I4f939f9a350d3a99ce3d3af37d0dea8ab8030fee
Reviewed-on: https://review.whamcloud.com/30166
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10468 tests: sync zfs dataset before reading blocks 28/30828/5
Jinshan Xiong [Thu, 11 Jan 2018 01:21:25 +0000 (17:21 -0800)]
LU-10468 tests: sync zfs dataset before reading blocks

Before reading blocks it should synchronize zfs dataset therefore
the block number will be accurate.

Test-Parameters: trivial envdefinitions=SLOW=yes,ENABLE_QUOTA=yes mdtfilesystemtype=zfs ostfilesystemtype=zfs mdscount=2 mdtcount=4 testlist=sanity-flr,sanity-flr,sanity-flr
Test-Parameters: trivial envdefinitions=SLOW=yes,ENABLE_QUOTA=yes mdtfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs mdscount=2 mdtcount=4 testlist=sanity-flr,sanity-flr,sanity-flr
Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I663d689296e5847b460de9a491b551a56bfbc77d
Reviewed-on: https://review.whamcloud.com/30828
Tested-by: Jenkins
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
3 years agoLU-10488 tests: saved and restore layout for dom test 36/30836/2
Jinshan Xiong [Thu, 11 Jan 2018 17:47:07 +0000 (17:47 +0000)]
LU-10488 tests: saved and restore layout for dom test

Some features like FLR and quota are still not supported by dom.
It will cause failures when dom is enabled when the corresponding
test cases are running.

This patch saves and restores default layout before and after
sanity-dom runs.

Test-Parameters: trivial envdefinitions=SLOW=yes,ENABLE_QUOTA=yes testlist=sanity-dom,sanity-flr,sanity-quota
Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I69752c3c48e3edf32ff22399af32b67c718f8e0e
Reviewed-on: https://review.whamcloud.com/30836
Tested-by: Jenkins
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
3 years agoLU-5170 lfs: Standardize error messages in lfs_check() 66/30666/2
Steve Guminski [Wed, 12 Jul 2017 18:56:58 +0000 (14:56 -0400)]
LU-5170 lfs: Standardize error messages in lfs_check()

Error messages in lfs_check() are updated to a standard format.
Messages are prefixed with the name of the utility and the command
that caused the error.  User-provided values are delimited with
single quotes.

Test-Parameters: trivial
Signed-off-by: Steve Guminski <stephenx.guminski@intel.com>
Change-Id: Ife0fa0b5f22f1099757c38d07b1827e26182426e
Reviewed-on: https://review.whamcloud.com/30666
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
3 years agoLU-10425 kernel: kernel update [SLES12 SP3 4.4.103-6.33] 47/30647/2
Bob Glossman [Thu, 21 Dec 2017 18:13:08 +0000 (10:13 -0800)]
LU-10425 kernel: kernel update [SLES12 SP3 4.4.103-6.33]

Update target and kernel_config files for new version

Test-Parameters: clientdistro=sles12sp3 testgroup=review-ldiskfs \
  mdsdistro=sles12sp3 ossdistro=sles12sp3 \
  mdtfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: Ibdaf6edeb34388d3d9bf39dc7f84639d5fb9992f
Reviewed-on: https://review.whamcloud.com/30647
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Cliff White <cliff.white@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10392 fid: improve seq allocation error messages 23/30623/2
Emoly Liu [Wed, 20 Dec 2017 09:32:37 +0000 (17:32 +0800)]
LU-10392 fid: improve seq allocation error messages

When MDTs are waiting for MDT0000 to start the master sequence
server and be granted a meta sequence for the first time, in case
of "-EINPROGRESS", the limited console messages would be clearer
and better than many error messages.

Change-Id: I64e80508cf9fb837a328b4f8228f29bfb764845d
Signed-off-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-on: https://review.whamcloud.com/30623
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
3 years agoLU-10327 test: use nogroup instead of nobody 59/30559/8
Minh Diep [Fri, 15 Dec 2017 20:48:58 +0000 (12:48 -0800)]
LU-10327 test: use nogroup instead of nobody

In Ubuntu, there isn't any nobody group

Change-Id: I2d4607b3d1384d2d619dd9640363533f92397356
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/30559
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10316 tests: skip checksum check for sanity 77c 02/30402/7
James Nunez [Wed, 6 Dec 2017 00:17:16 +0000 (17:17 -0700)]
LU-10316 tests: skip checksum check for sanity 77c

sanity test 77c does not need to verify the checksum for
some versions of Lustre servers. The code to skip the
checksum verification is in place, but it is missing a
'retun 0;' to exit the test.

Test-Parameters: trivial
Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: I9ea19f699d1142475f84d59b9c31880b7daf7f52
Reviewed-on: https://review.whamcloud.com/30402
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Cliff White <cliff.white@intel.com>
3 years agoLU-10308 misc: update Intel copyright messages for 2017 41/30341/4
Andreas Dilger [Fri, 1 Dec 2017 10:10:07 +0000 (03:10 -0700)]
LU-10308 misc: update Intel copyright messages for 2017

Update copyright messages for files updated in 2016, excluding
trivial patches.

Add trivial patches to updatecw.sh script exclude list.

Revert some changes that were incorrectly attributed to the
2016 (d10200a80770f0029d1d665af954187b9ad883df) and
2015 (0754bc8f2623bea184111af216f7567608db35b6) copyright
update patches themselves, since they were not in the exclude
list when the subsequent script was run.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I82f21c30c4dac75792bb49fc139bee2ca51f5545
Reviewed-on: https://review.whamcloud.com/30341
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-5170 utils: Add utility and command name to llapi messages 71/28571/8
Steve Guminski [Thu, 3 Aug 2017 13:33:39 +0000 (09:33 -0400)]
LU-5170 utils: Add utility and command name to llapi messages

Allow error and info messages printed by llapi functions to
include the name of the utility and the command that caused
the error.  The command name is provided by the caller before
invoking any llapi functions through the use of the new function
llapi_set_command_name().  After the command has completed, the
utility should reset the name using llapi_clear_command_name().
If no command name is provided, only the utility name is printed.

The lfs and lctl utilities have been updated with the new functions.
They will print both the utility name and the command name while
in non-interactive mode.  In interactive mode, only the utility
name is printed.

Test-Parameters: trivial
Signed-off-by: Steve Guminski <stephenx.guminski@intel.com>
Change-Id: Ifaa1549f9bac6fdb25120f5721f69a8e1a7a52e1
Reviewed-on: https://review.whamcloud.com/28571
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-5170 lfs: Standardize error messages in lfs_setquota() 86/28286/4
Steve Guminski [Wed, 12 Jul 2017 16:16:30 +0000 (12:16 -0400)]
LU-5170 lfs: Standardize error messages in lfs_setquota()

Error and warning messages in lfs_setquota() and the llapi functions
it calls are updated to a standard format.  Messages are prefixed
with the name of the utility and the command that caused the error.
User-provided values are delimited with single quotes.

Messages that duplicate information printed elsewhere have been
removed.

Test-Parameters: trivial
Signed-off-by: Steve Guminski <stephenx.guminski@intel.com>
Change-Id: I2d23f3dfc897047ac1c0803f7da9b1e5f2e5d719
Reviewed-on: https://review.whamcloud.com/28286
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-5170 lfs: Standardize error messages in lfs_quota() 53/28253/4
Steve Guminski [Wed, 12 Jul 2017 15:05:50 +0000 (11:05 -0400)]
LU-5170 lfs: Standardize error messages in lfs_quota()

Error and warning messages in lfs_setquota() and the llapi functions
it calls are updated to a standard format.  Messages are prefixed
with the name of the utility and the command that caused the error.
User-provided values are delimited with single quotes.

Test-Parameters: trivial
Signed-off-by: Steve Guminski <stephenx.guminski@intel.com>
Change-Id: I598be2d73a28675c06b77ca6f9fa0544ecaecc7e
Reviewed-on: https://review.whamcloud.com/28253
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-9594 tests: remove sanity tests from ALWAYS_EXCEPT 13/27413/19
James Nunez [Wed, 25 Oct 2017 14:01:45 +0000 (08:01 -0600)]
LU-9594 tests: remove sanity tests from ALWAYS_EXCEPT

Remove the following sanity tests from the ALWAYS_EXCEPT list:

45 - bz3561 'osc io page accounting' because associated bugzilla
ticket contain no activity.

68b - bz5188 test no longer exists.

Signed-off-by: dilip krishnagiri <dilipx.krishnagiri@intel.com>
Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: Ia6aa47872ffa7fe9e54131de7f7d3a6f8a70bd27
Reviewed-on: https://review.whamcloud.com/27413
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Cliff White <cliff.white@intel.com>
Reviewed-by: Saurabh Tandan <saurabh.tandan@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-9145 nodemap: new_init_ucred doesn't do nodemapping 24/26624/4
Kit Westneat [Fri, 14 Apr 2017 15:06:02 +0000 (11:06 -0400)]
LU-9145 nodemap: new_init_ucred doesn't do nodemapping

The new_init_ucred path was missed in the original nodemap
implementation. This patch adds the mapping calls to new_init_ucred.

WIP: There are some issues/questions:
 - some of the new_init_ucred code should be merged with
old_init_ucred_common.
 - Why does new_init check for setuid/setgid/setgrp, but not old_init?
 - What is ptlrpc_user_desc and should it interact with nodemap?

Signed-off-by: Kit Westneat <kit.westneat@gmail.com>
Change-Id: Ia0e07078f32c43ec319be5a4cea753667056d645
Reviewed-on: https://review.whamcloud.com/26624
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-8235 scripts: pass unrecognized options to lfs migrate 21/20621/6
Steve Guminski [Thu, 19 Oct 2017 14:52:07 +0000 (10:52 -0400)]
LU-8235 scripts: pass unrecognized options to lfs migrate

Pass through any unrecognized options to the "lfs migrate" command,
allowing the script to support migrate options without any special
handling.

Add new options "--rsync" and "--no-rsync" to specify how rsync
is used as a fallback alternative for the "lfs migrate" command.
The "--rsync" option forces the use of rsync instead of lfs migrate,
while the "--no-rsync" option prevents falling back to rsync in
the case where lfs migrate fails.

Add new "--dry-run" option.  The current "-n" option for a dry-run
duplicates the "-n" option to designate non-block for "lfs migrate".
The script's usage of "-n" is therefore deprecated, so that a future
patch can instead pass it through.

Add new "-v" option to increase verbosity, to help test/debug/monitor
what is being done by the script.  This option is also passed
through to "lfs migrate".

Test-Parameters: trivial
Signed-off-by: Nathan Dauchy <Nathan.Dauchy@noaa.gov>
Signed-off-by: Steve Guminski <stephenx.guminski@intel.com>
Change-Id: Ia3f56bf528d2ac8155f08d93d01abd8bb9168cc4
Reviewed-on: https://review.whamcloud.com/20621
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
3 years agoLU-5680 tests: Remove use of /etc/motd from sanity-hsm 21/14021/6
James Nunez [Tue, 5 Dec 2017 17:49:26 +0000 (10:49 -0700)]
LU-5680 tests: Remove use of /etc/motd from sanity-hsm

In sanity-hsm tests, /etc/motd is copied to the file system
and archived. /etc/motd may not exist in some Linux distributions
and, if exists, may be of size 0 and may defeat the use of
bandwidth control for timing. Uses of /etc/motd are replaced
with a generated file of known suitable size.

Test-Parameters: trivial testlist=sanity-hsm

Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: I2f287db6d422ce20bca037ca69e24179c7e48144
Reviewed-on: https://review.whamcloud.com/14021
Reviewed-by: Quentin Bouget <quentin.bouget@cea.fr>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-7372 test: Remove replay-dual test_26 from ALWAYS_EXCEPT 77/30677/2
Jim Casper [Sun, 31 Dec 2017 23:27:04 +0000 (18:27 -0500)]
LU-7372 test: Remove replay-dual test_26 from ALWAYS_EXCEPT

The following patch to fix test 26 was checked into master and b2_10, but test 26 was
not added back to the test set:
  https://review.whamcloud.com/17853/
  https://review.whamcloud.com/28323/

Test-Parameters: trivial testlist=replay-dual

Change-Id: I5c72537b1b62b2a29882c8e03ce18f5a7766301a
Signed-off-by: Jim Casper <jamesx.casper@intel.com>
Reviewed-on: https://review.whamcloud.com/30677
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Cliff White <cliff.white@intel.com>
Reviewed-by: Saurabh Tandan <saurabh.tandan@intel.com>
3 years agoLU-8649 recovery: print some useful messages in recovery 56/30656/3
Emoly Liu [Mon, 25 Dec 2017 03:34:36 +0000 (11:34 +0800)]
LU-8649 recovery: print some useful messages in recovery

To make it more clear to the admins that recovery won't start
until the first client connects, this patch prints the following
useful messages in recovery, in case the admins are waiting for
recovery to complete:
- a console message every 10 minutes or so, and
- adding status WAITING_FOR_CLIENTS to /proc recovery_status file

Change-Id: I03d37b4c00a799a1fd651b8d60cdbceed807cea1
Signed-off-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-on: https://review.whamcloud.com/30656
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
3 years agoLU-10199 tests: Re-enable sanity test 56xb 16/30616/2
Steve Guminski [Fri, 15 Dec 2017 20:18:34 +0000 (15:18 -0500)]
LU-10199 tests: Re-enable sanity test 56xb

Re-enable sanity test 56xb now that LU-10199 has been resolved.

Test-Parameters: trivial
Signed-off-by: Steve Guminski <stephenx.guminski@intel.com>
Change-Id: I02502e802be89c3fdfd559e1113193af568bc33c
Reviewed-on: https://review.whamcloud.com/30616
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
3 years agoLU-8616 dne: allow mkdir with specific MDTs 66/30566/4
Lai Siyao [Sat, 16 Dec 2017 08:22:10 +0000 (16:22 +0800)]
LU-8616 dne: allow mkdir with specific MDTs

Silimiar to create file on specific OSTs, allow 'lfs mkdir' to mkdir
on specific MDTs. This is achieved by allowing 'lfs mkdir -i' specify
multiple MDTs, and adding LMV_USER_MAGIC_SPECIFIC.

Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Change-Id: I3876707103465d1659afc80914ed6f9b58da25eb
Reviewed-on: https://review.whamcloud.com/30566
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
3 years agoLU-9845 test: add failmode=panic to zpool import options 49/30649/4
John L. Hammond [Fri, 22 Dec 2017 16:21:14 +0000 (10:21 -0600)]
LU-9845 test: add failmode=panic to zpool import options

In test-framework.sh add failmode=panic to the zpool import options.

Test-Parameters: trivial
Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I3b82f54034d57d99ed1f548bd3ac3736ae98d9f2
Reviewed-on: https://review.whamcloud.com/30649
Tested-by: Jenkins
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
3 years agoLU-10406 tests: stop running sanity-lfsck 31c 94/30694/3
James Nunez [Tue, 2 Jan 2018 22:56:11 +0000 (15:56 -0700)]
LU-10406 tests: stop running sanity-lfsck 31c

sanity-lfsck is failing about half the time it is run.
Add sanity-lfsck test 31c to the ALWAYS_EXCEPT list until
we understand and fix the issue.

Test-Parameters: trivial
Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: Iace77e1b1ea59971548497063974628be9e70733
Reviewed-on: https://review.whamcloud.com/30694
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
3 years agoLU-10399 test: use /dev/urandom in sanity-hsm test_1b() 56/30556/3
John L. Hammond [Fri, 15 Dec 2017 18:48:21 +0000 (12:48 -0600)]
LU-10399 test: use /dev/urandom in sanity-hsm test_1b()

In sanity-hsm test_1b() use /dev/urandom instead of /dev/random so we
won't have to wait for the second law of thermodynamics to become
true.

Test-Parameters: trivial testlist=sanity-hsm
Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ib64691fd6ee1a172c631ffa55956e1d69f24d349
Reviewed-on: https://review.whamcloud.com/30556
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Quentin Bouget <quentin.bouget@cea.fr>
3 years agoLU-10357 hsm: open file to be archived before starting 56/30456/2
John L. Hammond [Fri, 8 Dec 2017 19:59:43 +0000 (13:59 -0600)]
LU-10357 hsm: open file to be archived before starting

In the archive case of llapi_hsm_action_begin(), open the file to be
archived before calling the LL_IOC_HSM_COPY_START ioctl. Store the
open FD in struct hsm_copyaction_private, return a dup of it
llapi_hsm_action_get_fd(), and close it in
llapi_hsm_action_end(). Calling open() first avoids 3 extra RPCs
(MDS_GETATTR, LDLM_ENQUEUE for layout to get data version,
LDLM_ENQUEUE for fstat()) when archiving.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Idea0a7f4cb63bd9d712ff2ce9fcb59a3b278d0f2
Reviewed-on: https://review.whamcloud.com/30456
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Quentin Bouget <quentin.bouget@cea.fr>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10341 hsm: filter kkuc write by client UUID 19/30419/2
John L. Hammond [Thu, 7 Dec 2017 02:21:48 +0000 (20:21 -0600)]
LU-10341 hsm: filter kkuc write by client UUID

Add a struct obd_uuid kr_uuid member to struct kkuc_reg to hold the
UUID of the client (super block) that owns the kkuc pipe. Modify
libcfs_kkuc_group_{put,rem,foreach}() to accept a UUID pointer which
filters the kkuc pipes operated on. Modify mdc_hsm_copytool_send() to
pass the UUID of the MDC device when calling
libcfs_kkuc_group_put(). The effect of all this is that HALs received
by a given MDC will only be delivered to copytools registered on the
corresponding mount point.

Remove the cluuid member of struct lmv_obd since it is always the same
as the obd_uuid member of the corresponding struct obd_device. Remove
the kcd_uuid member of struct kkuc_ct_data as it is no longer needed.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ibbda253979739a1d56d3e132a51a482a02a0ec27
Reviewed-on: https://review.whamcloud.com/30419
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10310 utils: change l_getidentity log level 38/30338/5
Sergey Cheremencev [Fri, 1 Dec 2017 14:47:35 +0000 (17:47 +0300)]
LU-10310 utils: change l_getidentity log level

With current error level(NOTICE) l_getidentity error
messages don't appear in kernel messages. Changing
this level to WARNING may help to better and faster
understanding that issue relates to l_getidentity.

Change error level for errlog->vsyslog from LOG_NOTICE
to LOG_WARNING.

Test-Parameters: trivial
Cray-bug-id: MRP-2132
Change-Id: If72a5ee52de89b15b4bb7fc0dda77915531dbc39
Signed-off-by: Sergey Cheremencev <c17829@cray.com>
Reviewed-on: https://review.whamcloud.com/30338
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
3 years agoLU-10210 tests: Add lustre_routes_conversion script in PATH 35/30335/3
Sonia Sharma [Fri, 1 Dec 2017 11:25:48 +0000 (03:25 -0800)]
LU-10210 tests: Add lustre_routes_conversion script in PATH

When running out of build tree test_67 in conf-sanity.sh cannot
find lustre_routes_conversion script because it's not in PATH.

Add lustre_routes_conversion script in test-framework.sh
init_test_env() so that it is visible when running out of
build tree.

Test-Parameters: trivial testlist=conf-sanity
Change-Id: Id5ed93e0176f0b42aa704511c72fbe14902ea42f
Signed-off-by: Sonia Sharma <sonia.sharma@intel.com>
Reviewed-on: https://review.whamcloud.com/30335
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10211 tests: conf_sanity test 102 don't call lustre_rmmod 01/30001/2
Oleg Drokin [Wed, 8 Nov 2017 05:29:15 +0000 (00:29 -0500)]
LU-10211 tests: conf_sanity test 102 don't call lustre_rmmod

When testing from build tree lustre/scripts is not in the
PATH, so need to call $LUSTRE_RMMOD that knows the actual
path to that script.

Change-Id: I7e8fa2b4ac8c2d03d1a9a6865c50dbae6c139a30
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: https://review.whamcloud.com/30001
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
3 years agoLU-10030 utils: add lfs tool to change/list project of file 90/29190/22
Wang Shilong [Mon, 16 Oct 2017 06:58:04 +0000 (14:58 +0800)]
LU-10030 utils: add lfs tool to change/list project of file

Currently, we are using chattr/lsattr for project quota
interface, this have some problems:

1)Client side need patched e2fsprogs or latest upstream e2fsprogs.
2)Project quota will be no longer osd-ldiskfs based, ZFS
too, zfs guys might dislike ldiskfs tool dependency for them.
3)customers argue chattr might be a little dangerous.

So this patch add native lfs tools for project quota.
usage: project [-p id] [-s] [-r] <file|directory..>
          set project ID and/or inherit flag for specified
          file(s) or directory.
       project [-d|-r [-0]] <file|directory...>
          list project ID and flags on file(s) or directory,
          print outliers
       project -c [-d|-r [-p id] [-0]] <file|directory..>
          check project ID and flags on file(s) or directory,
          print outliers
       project -C [-r] [-k] <file|directory..>
          clear the project inherit flag and ID on the file
          or directory

Test-Parameters: testlist=sanity-quota,sanity-quota,sanity-quota,\
    sanity-quota clientdistro=el7 serverdistro=el7 \
    ostfilesystemtype=ldiskfs mdtfilesystemtype=ldiskfs
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I45960fb8fbd12e22a654792fba517896c0447447
Reviewed-on: https://review.whamcloud.com/29190
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10192 lfsck: verify agent entry 85/29985/7
Fan Yong [Wed, 29 Nov 2017 04:27:12 +0000 (12:27 +0800)]
LU-10192 lfsck: verify agent entry

Originally, we only support agent entry for ldiskfs backend,
and the osd-ldiskfs will maintain agent entry from cross-MDTs
directory, NOT corss-MDTs regular file. So if someone create
cross-MDTs hard link or renames regular file cross-MDTs, then
related object will become invisible to userspace when mount
the MDT as 'ldiskfs' directly.

On the other hand, old ZFS based MDT also did not support
agent entry. When upgraded from the old ZFS based device,
or migrated from old ldiskfs based MDT to new ZFS based MDT,
some (or all) agent entries need to be created.

So we enhance the namespace LFSCK logic to check whether the
agent entry is properly setup or not. If not, the LFSCK will
trigger the lower layer agent entry verify mechanism via set
xattr operation.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I0aa83aff8b39b894dbde19f573c078faf0ef249c
Reviewed-on: https://review.whamcloud.com/29985
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-9796 kernel: improve metadata performaces for RHEL7 76/28276/9
Wang Shilong [Thu, 21 Dec 2017 07:02:20 +0000 (15:02 +0800)]
LU-9796 kernel: improve metadata performaces for RHEL7

Port following upstream patch for RHEL7:

commit de92c8caf16ca84926fa31b7a5590c0fb9c0d5ca
Author: Jan Kara <jack@suse.cz>
Date:   Mon Jun 8 12:46:37 2015 -0400

    jbd2: speedup jbd2_journal_get_[write|undo]_access()

    jbd2_journal_get_write_access() and jbd2_journal_get_create_access() are
    frequently called for buffers that are already part of the running
    transaction - most frequently it is the case for bitmaps, inode table
    blocks, and superblock. Since in such cases we have nothing to do, it is
    unfortunate we still grab reference to journal head, lock the bh, lock
    bh_state only to find out there's nothing to do.

    Improving this is a bit subtle though since until we find out journal
    head is attached to the running transaction, it can disappear from under
    us because checkpointing / commit decided it's no longer needed. We deal
    with this by protecting journal_head slab with RCU. We still have to be
    careful about journal head being freed & reallocated within slab and
    about exposing journal head in consistent state (in particular
    b_modified and b_frozen_data must be in correct state before we allow
    user to touch the buffer).

Signed-off-by: Jan Kara <jack@suse.cz>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
From 087ffd4eae9929afd06f6a709861df3c3508492a Mon Sep 17 00:00:00 2001
Date: Fri, 4 Dec 2015 12:29:28 -0500
Subject: [PATCH] jbd2: fix null committed data return in undo_access

     introduced jbd2_write_access_granted() to improve write|undo_access
     speed, but missed to check the status of b_committed_data which caused
     a kernel panic on ocfs2.
     ...

     Fixes: de92c8caf16c("jbd2: speedup jbd2_journal_get_[write|undo]_access()")
Cc: <stable@vger.kernel.org>
Signed-off-by: Junxiao Bi <junxiao.bi@oracle.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
This patches improve 10% of file create, 17% of file lustre unlink
performances

Change-Id: I8082be396209d8f658e3265cedf32670e15a53f5
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/28276
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Reviewed-by: Gu Zheng <gzheng@ddn.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-9543 ofd: fiemap deadlock 24/27224/4
Andriy Skulysh [Mon, 22 May 2017 12:43:19 +0000 (15:43 +0300)]
LU-9543 ofd: fiemap deadlock

lock_zero_regions() locks all zero regions by acquiring
a set of independent locks.
It can deadlock with a PW lock for the whole file from
a client.

Indeed it isn't required to have all zero regions locked
at once, we need only force clients to flush data for
these regions.

Change-Id: Ib48e2bd9e6f715eb54a7821acde7b38b0de6650c
Seagate-bug-id: MRP-4393
Signed-off-by: Andriy Skulysh <andriy.skulysh@seagate.com>
Reviewed-on: http://es-gerrit.xyus.xyratex.com:8080/15734
Reviewed-by: Artem Blagodarenko <artem.blagodarenko@seagate.com>
Reviewed-by: Vitaly Fertman <vitaly.fertman@seagate.com>
Tested-by: Elena V. Gryaznova <elena.gryaznova@seagate.com>
Reviewed-on: https://review.whamcloud.com/27224
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Artem Blagodarenko <c17828@cray.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-8895 target: limit grant allocation 96/24096/4
Vladimir Saveliev [Fri, 15 Dec 2017 09:33:17 +0000 (12:33 +0300)]
LU-8895 target: limit grant allocation

tgt_grant_alloc() is missing a check for amount of space already
granted to a client. If the client submits number of RPCs
simultaneously when the client's grant is below its max amount of
grants then the server may grant the client with amount of grants
substantially exceeding the amount of grants requested in one RPC. In
case of decent number of clients that may lead to ENOSPC long before
the lack of disk space is really achieved.

Limit grants given to a client to asked amount plus grants for 2 full
write RPCs.

A test to illustrate the issue is included.
The test needs to lower debug level so that dd provided sufficient I/O
throughput.

Signed-off-by: Vladimir Saveliev <c17830@cray.com>
Seagate-bug-id: MRP-4013
Change-Id: Ie6a8abbad28a06bc1d55ff2fd042b9664a29e9e4
Reviewed-on: https://review.whamcloud.com/24096
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-6609 test: wait for import state FULL 43/14843/11
Sergey Cheremencev [Wed, 18 Dec 2013 13:56:38 +0000 (17:56 +0400)]
LU-6609 test: wait for import state FULL

recovery-small 26a sometimes couldn't remove sub-test dirs.
Decrement of export number may be caused by net issues.
So, now test is passed only when import state becomes EVICTED.
And in the end it waits for state FULL before removing sub-test dirs.

Test-Parameters: trivial envdefinitions=SLOW=yes,ONLY=26a testlist=recovery-small

Change-Id: Ib6156f4761bc79d89b42654898b51cc86c2ef40a
Reviewed-on: http://es-gerrit.xyus.xyratex.com:8080/1277
Signed-off-by: Sergey Cheremencev <Sergey_Cheremencev@xyratex.com>
Xyratex-bug-id: MRP-1168
Tested-by: Jenkins
Tested-by: Elena Gryaznova <elena_gryaznova@xyratex.com>
Reviewed-by: Alexey Lyashkov <alexey_lyashkov@xyratex.com>
Reviewed-by: Andrew Perepechko <andrew_perepechko@xyratex.com>
Reviewed-by: Vitaly Fertman <vitaly_fertman@xyratex.com>
Reviewed-on: https://review.whamcloud.com/14843
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alexey Lyashkov <c17817@cray.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-9019 llite: change lli_glimpse_time to ktime 01/30601/2
James Simmons [Tue, 19 Dec 2017 18:20:45 +0000 (13:20 -0500)]
LU-9019 llite: change lli_glimpse_time to ktime

Currently lli_glimpse_time is in jiffies which can vary between
platforms. Migrate to ktime since we need more than second
time resolution that is consistent on any platform. Replace the
last cfs_time_current_sec() with ktime_get_real_seconds().

Change-Id: I352c3adbd07d9dadb7e5dbe180447a1cb18a48d2
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/30601
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-5637 tests: set filefrag blocksize to 1024 91/30391/3
Sergey Cheremencev [Wed, 6 Dec 2017 11:08:21 +0000 (14:08 +0300)]
LU-5637 tests: set filefrag blocksize to 1024

If blocksize for filefrag is unspecified it defaults to 1024 bytes.
But for example at SL7 blocksize defaults is 4096.
Set it to 1024 to have everywhere the same results.

Test-Parameters: trivial envdefinitions=ONLY=130 testlist=sanity
Change-Id: I550446c6c7c0b85aa769f0f8a7575a6d33b2dc4b
Cray-bug-id: MRP-2933
Signed-off-by: Sergey Cheremencev <c17829@cray.com>
Reviewed-on: https://review.whamcloud.com/30391
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
3 years agoLU-10269 ldlm: allow trybits in waiting queue 43/30343/6
Mikhal Pershin [Sat, 2 Dec 2017 08:42:11 +0000 (11:42 +0300)]
LU-10269 ldlm: allow trybits in waiting queue

Lock trybits can be kept while lock is waiting and each new
lock filters trybits of locks in the waiting queue. When lock
if granted finally remaining trybits are added to the granted
bits. Therefore trybits can be granted for blocking lock if
no other locks take these bits while lock is waiting.

Test-Parameters: mdscount=1 mdtcount=1 testlist=racer,racer,racer,racer
Signed-off-by: Mikhal Pershin <mike.pershin@intel.com>
Change-Id: I775f776f4cf8b581e32e4a1585e862e1764b5bed
Reviewed-on: https://review.whamcloud.com/30343
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-8626 hsm: expose the number of active hsm requests per type 36/30336/8
Quentin Bouget [Fri, 1 Dec 2017 11:41:44 +0000 (11:41 +0000)]
LU-8626 hsm: expose the number of active hsm requests per type

This patch creates 3 new proc files under the hsm directory:
 - archive_count
 - restore_count
 - remove_count

These should help monitor the coordinator's health and allow
policy engine to adapt their request flow.

Test-Parameters: trivial testlist=sanity-hsm
Signed-off-by: Quentin Bouget <quentin.bouget@cea.fr>
Change-Id: I30c9fb658e8c14a181b094b51408c92df609c3ca
Reviewed-on: https://review.whamcloud.com/30336
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Henri Doreau <henri.doreau@cea.fr>
Reviewed-by: Stephan Thiell <sthiell@stanford.edu>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-8271 nodemap: wait before getting large conf if changed 81/26781/6
Kit Westneat [Fri, 21 Apr 2017 18:57:42 +0000 (14:57 -0400)]
LU-8271 nodemap: wait before getting large conf if changed

If a nodemap configuration spans multiple RPCs, it's possible for the
nodemap config to change between RPCs. Previously, the MGC would
immediately retry to get the nodemap config. This patch modifies the
behavior so the nodemap config lock is readded to the wait queue,
which will delay retrying by ~10s.

Test-Parameters: envdefinitions=SLOW=yes \
testlist=sanity-sec,sanity-sec,sanity-sec

Signed-off-by: Kit Westneat <kit.westneat@gmail.com>
Change-Id: Ie4e70def712e5eaa38adecc450e39c0380e34b69
Reviewed-on: https://review.whamcloud.com/26781
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10321 lfsck: not start lfsck during umount 13/30513/2
Fan Yong [Wed, 13 Dec 2017 05:35:39 +0000 (13:35 +0800)]
LU-10321 lfsck: not start lfsck during umount

There is race condition bewtween lfsck_start and umount:
the LFSCK may be triggered just after the LFSCK stopped
during umount the target, then nobody will stop the new
started LFSCK, as to the umount may be blocked.

This patch sets flag on the lfsck instance when umount
that will prevent subsequent lfsck_start.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I779f862d4195d4289bb9dd96575cd7746ac4b35b
Reviewed-on: https://review.whamcloud.com/30513
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10321 lfsck: allow to stop the in-starting lfsck 20/30420/2
Fan Yong [Thu, 7 Dec 2017 07:36:47 +0000 (15:36 +0800)]
LU-10321 lfsck: allow to stop the in-starting lfsck

The LFSCK start logic will hold li_mutex on the lfsck instance
during LFSCK start processing. The LFSCK stop logic also needs
to take the li_mutex on the lfsck instance when stop the LFSCK.
If someone triggers lfsck_stop (such as when umount the target)
before the lfsck_start return, then lfsck_stop will be blocked
on the li_mutex. And if the li_mutex holder is blocked by other
things, for example, it may be waiting for the LFSCK RPC to be
handled by remote server (MDT/OST) but the connection or remote
server is not ready yet, then the lfsck_stop will be blocked.

To avoid such cascade block trouble, the patch makes lfsck_stop
can go ahead without taking li_mutex, then it can directly tell
related LFSCK engines the stop event even if former lfsck_start
does not complete yet.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I6e168d955db33d74778142235a8ed2802d3577d9
Reviewed-on: https://review.whamcloud.com/30420
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10212 ldlm: fix prolong for destroyed lock 92/29992/6
Alexander Boyko [Wed, 8 Nov 2017 19:30:05 +0000 (14:30 -0500)]
LU-10212 ldlm: fix prolong for destroyed lock

For a IO request ofd_prolong_extent_locks use
a fast path if the lock is found by handle. If the lock
has LDLM_FL_DESTROYED, prolong should try a general path.

No lock was accounted for IO request with destroyed lock
and ESTALE error happaned for a client.

operation ost_read to node x.x.x.x@o2ib failed: rc = -116

Signed-off-by: Alexander Boyko <c17825@cray.com>
Change-Id: I63e619d0330279bb2ae678ed98b1c0e899ad4e08
Reviewed-on: https://review.whamcloud.com/29992
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
3 years agoLU-5163 mdd: migrated entry may not exist 20/26620/6
Lai Siyao [Thu, 13 Apr 2017 09:54:53 +0000 (17:54 +0800)]
LU-5163 mdd: migrated entry may not exist

During dirent migration, we shouldn't assert file exists.

Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Change-Id: I11bbc5556007ec045b7a5d57a250981082ef6d70
Reviewed-on: https://review.whamcloud.com/26620
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-6142 gnilnd: handle LNet core typedef removal 74/30474/3
James Simmons [Fri, 22 Dec 2017 07:17:56 +0000 (02:17 -0500)]
LU-6142 gnilnd: handle LNet core typedef removal

The LNet layer has removed all the typedefs. Update the gnilnd
driver to handle the removals.

Test-parameters: trivial

Change-Id: I96377a977bc9ef689a0c48afb77106aa2a8993a1
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/30474
Reviewed-by: Chris Horn <hornc@cray.com>
Reviewed-by: Doug Oucharek <dougso@me.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-9943 lnd: correct WR fast reg accounting 11/30311/4
Amir Shehata [Tue, 28 Nov 2017 04:18:02 +0000 (20:18 -0800)]
LU-9943 lnd: correct WR fast reg accounting

Ensure that enough WRs are allocated for the fast reg
case which needs two additional WRs per transfer:
the first for memory window registration and the second for
memory window invalidation.

Failure to allocate these causes the following problem:
mlx5_warn:mlx5_0:begin_wqe:4085(pid 9590): work queue overflow

Test-Parameters: trivial
Signed-off-by: Alexey Lyashkov <alexey.lyashkov@seagate.com>
Signed-off-by: Amir Shehata <amir.shehata@intel.com>
Change-Id: Icf98b6bbb3d98fb29794173da84412070f13541b
Reviewed-on: https://review.whamcloud.com/30311
Reviewed-by: Alexey Lyashkov <c17817@cray.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Doug Oucharek <dougso@me.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10185 gnilnd: Change gnc_tx_bits to unsigned long 97/29897/4
Chuck Fossen [Wed, 6 Sep 2017 13:18:36 +0000 (08:18 -0500)]
LU-10185 gnilnd: Change gnc_tx_bits to unsigned long

gnc_tx_bits declared as __u8. Change to unsigned long.

The goal is to align gnc_tx_bits[] to a 32 bit boundary.
The initial declaration I believe was not a good choice.
The use of the gnc_tx_bits makes more sense as a long than as a 8 bit
value.
It is used by:
static inline int test_and_clear_bit(int nr, unsigned long *addr);
unsigned long find_next_zero_bit(unsigned long *addr, unsigned long
size, unsigned long offset);
static inline int test_and_set_bit(int nr, unsigned long *addr);
all of which takes a unsigned long pointer.

Test-parameters: trivial

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I559e2a356182f253716d30f69bf675c485fe1b72
Reviewed-on: https://review.whamcloud.com/29897
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Doug Oucharek <dougso@me.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10185 gnilnd: change default credits 96/29896/4
James Shimek [Fri, 14 Jul 2017 21:48:09 +0000 (16:48 -0500)]
LU-10185 gnilnd: change default credits

It has been found that a credit value of 64 reduces likelihood of file
io induced congestion without producing performance degradation when a
many router single compute pattern occurs.
Change default compute credits to 64 based on this finding.

Test-parameters: trivial

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I0a1f46bdb2c327a7e6dc9eeb145fe1418d691da1
Reviewed-on: https://review.whamcloud.com/29896
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Shimek <knathrak@gmail.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Doug Oucharek <dougso@me.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10185 gnilnd: Use consistent size for GNI MBOX. 95/29895/4
Russell Cattelan [Thu, 15 Jun 2017 19:57:02 +0000 (14:57 -0500)]
LU-10185 gnilnd: Use consistent size for GNI MBOX.

KMALLOC_MAX_SIZE is not consistent when changing
the allocator. For example when using the SLUB allocator
the size is set to 30 bits. For slab allocator the max size
is 25 bits or 32MB.

For SLAB this is a no-op change for other allocators the size
is also set to 32MB.

Test-parameters: trivial

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: Icab4c69f9afd276d55360a76991ba9783f83ac8b
Reviewed-on: https://review.whamcloud.com/29895
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Doug Oucharek <dougso@me.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10185 gnilnd: Add mod param to adjust vmalloc's retry flag 94/29894/4
James Shimek [Wed, 31 May 2017 20:15:36 +0000 (15:15 -0500)]
LU-10185 gnilnd: Add mod param to adjust vmalloc's retry flag

In kernel 3.12 gnilnd had issues with the memory allocator
getting stuck trying to find pages for long periods of time
so __GFP_NORETRY was added to fail fast. The memory subsystem
has changed enough that it is beleived that vmalloc will fail
if it can't get pages.

Add modparam to allow no_retry flag to be enabled or disabled
dynamically.

Test-parameters: trivial

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I067a8d7b52d237e464fceae7e2220f2488b68957
Reviewed-on: https://review.whamcloud.com/29894
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Shimek <knathrak@gmail.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Doug Oucharek <dougso@me.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10185 gnilnd: treat seq_printf as void function 93/29893/4
Chris Horn [Wed, 14 Jun 2017 19:04:37 +0000 (14:04 -0500)]
LU-10185 gnilnd: treat seq_printf as void function

Starting with the 4.3 linux kernel seq_printf is a void
function not returning the character count. Earlier cleanups
did not update all of gnilnd to accomodate this change
when they updated the rest of Lustre. This change takes
care of what was missed.

Test-parameters: trivial

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: Ie76c1cc7578cb5919a1f2aca6fb15fd8b75882c1
Reviewed-on: https://review.whamcloud.com/29893
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Doug Oucharek <dougso@me.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10185 gnilnd: SLES 12 SP2 support and cleanup 92/29892/4
Chuck Fossen [Wed, 21 Sep 2016 20:31:43 +0000 (15:31 -0500)]
LU-10185 gnilnd: SLES 12 SP2 support and cleanup

Handle a couple changes for SLES 12 SP2:
 - Handle rename of set_mb() to smp_store_mb()
 - Handle rename of __GFP_WAIT to __GFP_RECLAIM
 - Handle parameter change to sock_create_kern()

Also remove various dead code.

Test-parameters: trivial

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I9b5799a492b347fe7961f2d70e24bed5cc2e7021
Reviewed-on: https://review.whamcloud.com/29892
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Doug Oucharek <dougso@me.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10371 ptlrpc: check for posix_acl a_entries 95/30495/4
Thomas Stibor [Tue, 12 Dec 2017 14:27:30 +0000 (15:27 +0100)]
LU-10371 ptlrpc: check for posix_acl a_entries

Since commit 2211d5ba5c6c4e972ba6dbc912b2897425ea6621
posix_acl_xattr_entry a_entries[0] was removed.
Make sure the LASSERTF test works with Kernels
after this commit.

Test-Parameters: trivial
Signed-off-by: Thomas Stibor <t.stibor@gsi.de>
Change-Id: If2404c89775a5f9077c7d9379d73c8187b796a3a
Reviewed-on: https://review.whamcloud.com/30495
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10302 ldlm: destroy lock if LVB init fails 77/30477/3
John L. Hammond [Mon, 11 Dec 2017 19:03:20 +0000 (13:03 -0600)]
LU-10302 ldlm: destroy lock if LVB init fails

In ldlm_cli_enqueue_local(), destroy the newly created lock if
ldlm_lvbo_init() fails. Rename sanity test_232() to test_232a(), fix
the fail_loc setting to happen on the correct node, and check that the
OST can be unmounted after the failed write. Add sanity test_232b() to
do the same but for data version since that uses a different LDLM
path.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I18e9594b9a2461afbc66128f477d3185a6627bc0
Reviewed-on: https://review.whamcloud.com/30477
Reviewed-by: Quentin Bouget <quentin.bouget@cea.fr>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10149 llite: avoid live-lock when concurrent mmap()s 65/30465/2
Bruno Faccini [Mon, 11 Dec 2017 09:55:32 +0000 (10:55 +0100)]
LU-10149 llite: avoid live-lock when concurrent mmap()s

Patch is an attempt to delay page-fault retry from Client side for
inode with extents being mmap()'ed, to prevent live-lock situation
to occur with other page's competitors.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: Id04664fce1a5dad4dbdd7ad4b183dffb8e38b844
Reviewed-on: https://review.whamcloud.com/30465
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10301 kernel: kernel update RHEL7.4 [3.10.0-693.11.1.el7] 01/30401/2
Bob Glossman [Thu, 30 Nov 2017 15:06:47 +0000 (07:06 -0800)]
LU-10301 kernel: kernel update RHEL7.4 [3.10.0-693.11.1.el7]

update RHEL 7.4 kernel to 3.10.0-693.11.1.el7

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: Iea7a4604c333c26fdc1b2439e6a00f27472d2410
Reviewed-on: https://review.whamcloud.com/30401
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10291 lnd: remove concurrent_sends tunable 12/30312/4
Amir Shehata [Tue, 28 Nov 2017 19:08:31 +0000 (11:08 -0800)]
LU-10291 lnd: remove concurrent_sends tunable

Concurrent sends tunable was intended to limit the number of in-flight
transfers per connection. However queue depth does the exact same job.
So for example if the queue depth is negotiated to 16 and
concurrent_sends is set to 32, the maximum number of in-flight
transfers doesn't exceed 16. There is no need to keep concurrent_sends
around since it doesn't add any unique functionality

Test-Parameters: trivial
Signed-off-by: Amir Shehata <amir.shehata@intel.com>
Change-Id: I4dd7f5ecac5a7d10403c4f0ab0f01374a2f10206
Reviewed-on: https://review.whamcloud.com/30312
Reviewed-by: Alexey Lyashkov <c17817@cray.com>
Tested-by: Jenkins
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Doug Oucharek <dougso@me.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-9998 libcfs: split single NUMA node into partitions 45/29645/2
Dmitry Eremin [Tue, 17 Oct 2017 09:02:35 +0000 (12:02 +0300)]
LU-9998 libcfs: split single NUMA node into partitions

For machines with single NUMA node change default behavior
and slpit it with cpu_npartitions as it was before 2.8.59.
See LU-5050 libcfs: default CPT matches NUMA topology

Change-Id: I7f9122931a88fd5770628d7ae21b764efc21c134
Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-on: https://review.whamcloud.com/29645
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Sonia Sharma <sonia.sharma@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10192 osd-zfs: create agent entry for remote entry 17/29617/21
Fan Yong [Wed, 6 Dec 2017 13:54:45 +0000 (21:54 +0800)]
LU-10192 osd-zfs: create agent entry for remote entry

In DNE environment, the object (in spite of regular file
or directory) and its name entry may reside on different
MDTs. Under such case, we will create an agent entry on
the MDT where the object resides. The agent entry references
the object locally, that makes the object to be visible to
the userspace when mounted as 'zfs' directly. Then the
userspace tools, such as 'tar' can handle the object properly.
That is compatibile between ldiskfs backend and ZFS backend.

We handle the agent entry during set linkEA that is the common
interface for both regular file and directroy, can handle kinds
of cases, such as create/link/unlink/rename, and so on.

NOTE: we can NOT do that when ea_{insert,delete} that is only
for directory.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Icc4a63027221edf279994fbecda4d47cc121b799
Reviewed-on: https://review.whamcloud.com/29617
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10192 osd-ldiskfs: create agent entry for remote entry 84/29984/11
Fan Yong [Wed, 6 Dec 2017 03:55:33 +0000 (11:55 +0800)]
LU-10192 osd-ldiskfs: create agent entry for remote entry

In DNE environment, the object (in spite of regular file
or directory) and its name entry may reside on different
MDTs. Under such case, we will create an agent entry on
the MDT where the object resides. The agent entry references
the object locally, that makes the object to be visible to
the userspace when mounted as 'ldiskfs' directly. Then the
userspace tools, such as 'tar' can handle the object properly.

We handle the agent entry during set linkEA that is the common
interface for both regular file and directroy, can handle kinds
of cases, such as create/link/unlink/rename, and so on.

NOTE: we can NOT do that when ea_{insert,delete} that is only
for directory.

There are two known issues:
1. For one object, we will create at most one agent entry even if
   there may be more than one cross-MDTs hard links on the object.
   So the local e2fsck may claim that the object's nlink is larger
   than the name entries that reference such inode. And in further,
   the e2fsck will fix the nlink attribute to match the local
   references. Then it will cause the object's nlink attribute to
   be inconsistent with the global references. it is bad but not
   fatal. The ref_del() can handle the zero-referenced case. On the
   other hand, the global namespace LFSCK can repair the object's
   attribute according to the linkEA.
2. There may be too many hard links on the object as to its linkEA
   overflow, then the linkEA entry for cross-MDTs reference may be
   discarded. If such case happened, then at this point, we do not
   know whether there are some cross-MDTs reference. But there are
   local references, it guarantees that object is visible to
   userspace when mounted as 'ldiskfs'. That is enough.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Ic9cc68a4dc864307dd5dd5fdb930bfac699c8379
Reviewed-on: https://review.whamcloud.com/29984
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-9019 libcfs: add ktime_compare for platforms lacking it. 75/30475/2
James Simmons [Mon, 11 Dec 2017 18:18:42 +0000 (13:18 -0500)]
LU-9019 libcfs: add ktime_compare for platforms lacking it.

The function ktime_compare() was added to the linux kernel
starting with verison 3.19. Add to libcfs support for older
platforms lacking this function.

Test-Parameters: trivial

Change-Id: I7826b8e78d0dc2c633490a2949210176a0003d9a
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/30475
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
3 years agoLU-1757 brw: Fix short i/o and enable for mdc 35/30435/3
Patrick Farrell [Thu, 7 Dec 2017 13:00:58 +0000 (07:00 -0600)]
LU-1757 brw: Fix short i/o and enable for mdc

The short i/o flag was left out of the OST flags in the
original patch, meaning it was not really on.  Also, the
short_io_size value was used uninitialized, meaning it
was sometimes non-zero, which coudl lead to several issues.

Also add the short i/o flag to the MDC/MDT for data on MDT.
Quick testing suggests this works fine with no further
changes.

Cray-bug-id: LUS-187
Signed-off-by: Patrick Farrell <paf@cray.com>
Change-Id: I4154b87d5ad73b53467b0382368fad7c5ba177fe
Reviewed-on: https://review.whamcloud.com/30435
Tested-by: Jenkins
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Alexandr Boyko <c17825@cray.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10213 lnd: calculate qp max_send_wrs properly 10/30310/4
Amir Shehata [Tue, 28 Nov 2017 01:52:20 +0000 (17:52 -0800)]
LU-10213 lnd: calculate qp max_send_wrs properly

The maximum in-flight transfers can not exceed the
negotiated queue depth. Instead of calculating the
max_send_wrs to be the negotiated number of frags *
concurrent sends, it should be the negotiated number
of frags * queue depth.

If that value is too large for successful qp creation then
we reduce the queue depth in a loop until we successfully
create the qp or the queue depth dips below 2.

Due to the queue depth negotiation protocol it is guaranteed
that the queue depth on both the active and the passive
will match.

This change resolves the discrepancy created by the previous
code which reduces max_send_wr by a quarter.

That could lead to:
mlx5_ib_post_send:4184:(pid 26272): Failed to prepare WQE
When the o2iblnd transfers a message which requires more
WRs than the max that has been allocated.

Test-Parameters: trivial
Signed-off-by: Alexey Lyashkov <alexey.lyashkov@seagate.com>
Signed-off-by: Amir Shehata <amir.shehata@intel.com>
Change-Id: I88f96f950bf4c0a8efd4df812d44e5e20d5907dc
Reviewed-on: https://review.whamcloud.com/30310
Reviewed-by: Alexey Lyashkov <c17817@cray.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Doug Oucharek <dougso@me.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-9019 sec: migrate to 64 bit time 59/29859/14
James Simmons [Tue, 5 Dec 2017 20:46:18 +0000 (15:46 -0500)]
LU-9019 sec: migrate to 64 bit time

Replace cfs_time_current_sec() to avoid the overflow
issues in 2038 with ktime_get_real_seconds(). Mirgate
the rest of the gss code to time64_t to avoid the 2038
overflow issue.

Currently in encrypt_page_pools we are reporting the jiffy
cycles for "max wait time" which not only doesn't make
sense but can vary from platform to platform. Instead we
will report in terms of milliseconds. That requires changing
epp_st_max_wait into ktime_t since we need better than
seconds precision. Lastly the time in encrypt_page_pools for
"last access" and "last shrink" was showing up negative.
This was due to epp_last_* field being set to the number of
seconds since the epoch instead of the number of seconds
since the node booted. Change epp_last_* to being set by
ktime_get_seconds() instead of ktime_get_real_seconds()
resolves this problem.

Test-Parameters: trivial

Change-Id: Ia2d559454287675699a067121760543a2e6877da
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/29859
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-9810 lnd: use less CQ entries for each connection 79/28279/10
Alexey Lyashkov [Mon, 27 Nov 2017 12:44:56 +0000 (15:44 +0300)]
LU-9810 lnd: use less CQ entries for each connection

Currently we have a 2 work requests chains per transfer.
It mean OFED stack will generate only 2 events if transfer will
faild. Reduce number CQ entries to avoid extra resource consumption.

Test-Parameters: trivial
Seagate-bug-id: MRP-4508
Signed-off-by: Alexey Lyashkov <alexey.lyashkov@seagate.com>
Change-Id: I0c06fef9589478f40ef7e1eeacff2aec7013e562
Reviewed-on: https://review.whamcloud.com/28279
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Doug Oucharek <dougso@me.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10129 lnd: rework map_on_demand behavior 95/29995/8
Amir Shehata [Wed, 8 Nov 2017 18:10:49 +0000 (10:10 -0800)]
LU-10129 lnd: rework map_on_demand behavior

map_on_demand use is ambiguous. In kernels which supported global
memory regions, map-on-demand was used to limit the number of RDMA
fragments transferred between peers. That tuanble didn't impact the
memory allocation since the maximum allowed was always allocated. It
was used however as a variable in determining max_send_wr. With the
introduction of FMR if the number of fragments exceed the negotiated
number of fragments, which is determined by the map-on-demand value,
then FMR is used because we can transfer all the fragments in one FMR
fragment.

The latest kernels have removed support for global memory regions so
the use of map-on-demand has become ineffectual. However, we need to
keep it for backwards compatibility.  The behavior has changed as
follows:

map_on_demand is a flag used to determine if we can use FMR or
FastReg.  This is applicable for kernels which support global memory
regions. For later kernels this flag is always enabled, since we will
always either use FMR or FastReg For kernels which support global
memory regions map_on_demand defaults to 0 which means we will be
using global memory regions exclusively.  If it is set to a value
other than 0, then we will behave as follows:
  1. Always default the number of fragments to IBLND_MAX_RDMA_FRAGS
  2. Create FMR/FastReg pools
  3. Negotiate the supported number of fragments per connection
  4. Attempt to transmit using global memory regions only if
     map-on-demand is off, otherwise use FMR or FastReg.
  5. In case of transmitting tx with GAPS over FMR we will need to
     transmit it with multiple fragments. Look at the comments in
     kiblnd_fmr_map_tx() for an explanation of the behavior.

For later kernels we default map_on_demand to 1 and not allow it to be
set to 0, since there is no longer support for global memory regions.
Behavior:
  1. Default the number of fragments to IBLND_MAX_RDMA_FRAGS
  2. Create FMR/FastReg pools
  3. Negotiate the supported number of fragments per connection
  4. Look at the comments in kiblnd_fmr_map_tx() for an explanation of
     the behavior when transmit with GAPS verses contiguous.

Test-Parameters: trivial
Signed-off-by: Amir Shehata <amir.shehata@intel.com>
Change-Id: I20b0ea57ec394a050603b5a638f515dfc4ac9446
Reviewed-on: https://review.whamcloud.com/29995
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Doug Oucharek <dougso@me.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-9714 llog: fix llog_process_thread race 38/27838/15
Alexander Boyko [Tue, 19 Sep 2017 09:52:54 +0000 (12:52 +0300)]
LU-9714 llog: fix llog_process_thread race

lgh_cur_offset is used at llog_write_osd_rec,
and modified at llog_process_thread.
When two threads of llog_process_thread are processing the
same llog, it could happen that one thread do llog_write,
and second modify lgh_cur_offset. This situation drives to
wrong modification of llog.

Signed-off-by: Alexander Boyko <alexander.boyko@seagate.com>
Seagate-bug-id: MRP-4455
Change-Id: I7a63850c876146b14118a7d395cf3cfb3a40dd67
Reviewed-on: https://review.whamcloud.com/27838
Reviewed-by: Sergey Cheremencev <c17829@cray.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoRevert "LU-5216 hsm: cancel hsm actions running on CT when killed" 15/30615/3
John L. Hammond [Wed, 20 Dec 2017 15:56:06 +0000 (15:56 +0000)]
Revert "LU-5216 hsm: cancel hsm actions running on CT when killed"

This is too complicated given what it accomplishes and review
comments to that effect were not addressed.

This reverts commit 462c7aae05dfc9cd730f44ffdc661c4c36294012.

Change-Id: I7ab12e62780a8c3f4d4428980d9ce5be02101761
Reviewed-on: https://review.whamcloud.com/30615
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Tested-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoRevert "LU-9019 osc: migrate to time64_t" 71/30571/2
Oleg Drokin [Mon, 18 Dec 2017 15:40:38 +0000 (15:40 +0000)]
Revert "LU-9019 osc: migrate to time64_t"

This causes frequent failures tracked in LU-10403

This reverts commit 4f2a5d5887492da9abe320074511811415e0a06c.

Change-Id: I7d255fcee654508b6df233728624a39917853b98
Reviewed-on: https://review.whamcloud.com/30571
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Tested-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10232 lov: call cl_object_attr_get under cl_attr lock 52/30052/2
Mikhail Pershin [Sun, 12 Nov 2017 21:20:26 +0000 (00:20 +0300)]
LU-10232 lov: call cl_object_attr_get under cl_attr lock

cl_object_attr_get() must be called under cl_object_attr_lock
get. There is place in lov_getstripe where it is called
without that lock.

Signed-off-by: Mikhail Pershin <mike.pershin@intel.com>
Change-Id: Ia0a2322ba4ff0ff4affb081375cb108fbf2988c4
Reviewed-on: https://review.whamcloud.com/30052
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10005 osp: cache non-exist EA 78/29078/9
Di Wang [Fri, 27 Oct 2017 15:08:11 +0000 (23:08 +0800)]
LU-10005 osp: cache non-exist EA

OSP should also cache non-exist EA, otherwise
it will keep sending RPC to try to get remote
non-exist EA.

So if default stripe EA does not exist on root
MDT, file creation on non-root MDT will always
try to send extra OSP RPC to get the EA.

This patch also fixes a LFSCK bug that may cause
the LFSCK repaired items to be counted repeately.

Test-Parameters: mdtfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs mdscount=2 mdtcount=4 testlist=sanity-lfsck,sanity-lfsck,sanity-lfsck
Signed-off-by: Di Wang <di.wang@intel.com>
Change-Id: Ie9c8cb5fd54e404f1af97de47e809f6f96de8d86
Reviewed-on: https://review.whamcloud.com/29078
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
3 years agoLU-10331 out: use OBD_ALLOC_LARGE() for update buffers 55/30455/2
John L. Hammond [Fri, 8 Dec 2017 19:10:21 +0000 (13:10 -0600)]
LU-10331 out: use OBD_ALLOC_LARGE() for update buffers

In out_handle() the update buffers may be up to 100KB so use
OBD_ALLOC_LARGE() to avoid high order page allocation errors.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ie9970eb219ff0e5357132e0858cb16db8f703209
Reviewed-on: https://review.whamcloud.com/30455
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
3 years agoLU-10038 test: ignore readdir races in sanity 133g 51/30451/3
John L. Hammond [Fri, 8 Dec 2017 16:19:48 +0000 (10:19 -0600)]
LU-10038 test: ignore readdir races in sanity 133g

In sanity test_133g() writing to some files (for example
mdt/*/exports/clear) may cause cause the proc tree to change while
find is running. Avoid errors in these cases by adding the
-ignore_readdir_race option to the find invocations.

Test-Parameters: trivial testlist=sanity

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I3ce61ee0f4e1041041d6872a0fe03488a9df363c
Reviewed-on: https://review.whamcloud.com/30451
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
3 years agoLU-10346 utils: use the correct lib path in mount_utils.c 31/30431/3
Quentin Bouget [Thu, 7 Dec 2017 15:48:28 +0000 (15:48 +0000)]
LU-10346 utils: use the correct lib path in mount_utils.c

The backfs_load_module() function of mount_utils.c used an outdated
to link itself against the osd library. This patch fixes it.

Test-Parameters: trivial
Signed-off-by: Quentin Bouget <quentin.bouget@cea.fr>
Change-Id: I90a523b45b5772c05f7520e810eec7eac14b75be
Reviewed-on: https://review.whamcloud.com/30431
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10338 test: handle empty CLIENTS in mountcli() 13/30413/2
John L. Hammond [Wed, 6 Dec 2017 21:27:22 +0000 (15:27 -0600)]
LU-10338 test: handle empty CLIENTS in mountcli()

In mountcli() use if [ -n "$CLIENTS" ]; ... rather than
[ -n "$CLIENTS" ] && ... to avoid immediate exit on error when CLIENTS is
the empty string.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I440c3eefb4bee134caee1db5ff2ecdbdf9c1ee5c
Reviewed-on: https://review.whamcloud.com/30413
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10330 tests: fix version check in sanity.sh test_101g 89/30389/2
Emoly Liu [Wed, 6 Dec 2017 07:46:11 +0000 (15:46 +0800)]
LU-10330 tests: fix version check in sanity.sh test_101g

The 16MB RPC patch is available in 2.7.x, so one more version
check needs to be added to 2.7.17 or later to avoid interop
failure.

Test-Parameters: trivial

Change-Id: I6eeefbf74017aaeeb8998accd6da04d8de75606c
Signed-off-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-on: https://review.whamcloud.com/30389
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-9535 tests: check server version in sanityn 77j 85/30385/3
James Nunez [Tue, 5 Dec 2017 23:31:21 +0000 (16:31 -0700)]
LU-9535 tests: check server version in sanityn 77j

The QoS feature in the TBF policy was added to Lustre with
tag 2.9.53 and a patch with commit hash d2c403363f6. Thus,
sanityn test 77j should only be run for servers with version
2.9.53 and later.

Test-Parameters: trivial mdsjob=lustre-b2_9 ossjob=lustre-b2_9 serverbuildno=2 testlist=sanityn
Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: Ia2a04b5ae688809d4a84a2dc8459598f02932119
Reviewed-on: https://review.whamcloud.com/30385
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Saurabh Tandan <saurabh.tandan@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10267 llog: fix EOF handling in llog_client_next_block() 13/30313/2
John L. Hammond [Wed, 29 Nov 2017 14:45:47 +0000 (08:45 -0600)]
LU-10267 llog: fix EOF handling in llog_client_next_block()

In llog_client_next_block() update *cur_idx and *cur_offset in the
special case that the handler has returned -EIO after reaching the end
of the log without finding the desired record. This fixes client side
EOF detection in llog_process_thread().

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: If30ebec065ddd38fb7b06c3e16f96bb3cd76fa1b
Reviewed-on: https://review.whamcloud.com/30313
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10248 mdd: set PFID for swap and merge layout 92/30292/5
Jinshan Xiong [Thu, 16 Nov 2017 22:05:06 +0000 (22:05 +0000)]
LU-10248 mdd: set PFID for swap and merge layout

It should update PFID for the OST objects if the layout has been
moved to another file. So far only swap and merge layout would
have this requirement.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: Ic2fd7d7a88cd0ef4501d2f02baf58dd177e07973
Reviewed-on: https://review.whamcloud.com/30292
Tested-by: Jenkins
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-8999 test: check chown result 43/30243/4
Hongchao Zhang [Fri, 17 Nov 2017 15:05:17 +0000 (23:05 +0800)]
LU-8999 test: check chown result

For test_38 in sanity-quota, the chown operation should be
checked whether it succeeds or not. the patch also collects
the output of quota users if the test failed.

Change-Id: I6c04f9519f3f097af7126064380faf1bdc4fff6a
Signed-off-by: Hongchao Zhang <hongchao.zhang@intel.com>
Reviewed-on: https://review.whamcloud.com/30243
Tested-by: Jenkins
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10257 clio: remove unused cl_lock layers 92/30192/2
John L. Hammond [Fri, 17 Nov 2017 21:20:02 +0000 (15:20 -0600)]
LU-10257 clio: remove unused cl_lock layers

Remove the struct vvp_lock and struct lovsub_lock, omitting these
layers from cl_lock. Adjust cl_lock_enqueue() to allow for empty locks
(corresponding to unstriped files).

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ib1478d0cfb9604540ffc38eb9b01da4f4f96212a
Reviewed-on: https://review.whamcloud.com/30192
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-9474 tests: rewrite copytool_setup to use stack_trap 97/30097/7
Quentin Bouget [Tue, 14 Nov 2017 14:20:27 +0000 (14:20 +0000)]
LU-9474 tests: rewrite copytool_setup to use stack_trap

Replace copytool_setup with its rewrite (copytool setup) as much
as possible. There are some interaction between copytool_setup()
and other test functions (mainly import_file() and its derivatives).
Later patch(es) will handle those interactions and completely
remove copytool_setup() from sanity-hsm.

Among the benefits of using the new copytool() function:
 - it has a nice command line interface
 - it uses stack_trap() to register clean up actions
 - it makes it easier to run sanity-hsm with other copytools

Test-Parameters: trivial clientcount=3 mdscount=2 testlist=sanity-hsm,sanity-hsm
Change-Id: I81635da39f19d001e0f5c6348b233b45c4298fd0
Signed-off-by: Quentin Bouget <quentin.bouget@cea.fr>
Reviewed-on: https://review.whamcloud.com/30097
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Henri Doreau <henri.doreau@cea.fr>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
3 years agoLU-10227 ptlrpc: simplify struct ptlrpc_request_set 60/30060/3
John L. Hammond [Fri, 10 Nov 2017 20:03:30 +0000 (14:03 -0600)]
LU-10227 ptlrpc: simplify struct ptlrpc_request_set

Remove obd_statfs_rqset(), replacing its only use with
obd_statfs(). Collapse lov_statfs_async() and lov_statfs() into a
single function, removing the need for lov_statfs_interpret().

Remove the then unused set_wakeup_ptr, set_cblist, set_interpret, and
set_arg members of struct ptlrpc_request_set. Remove struct
ptlrpc_set_cbdata and ptlrpc_set_add_cb(). On x86_64 this reduces the
size of struct ptlrpc_request_set from 152 bytes to 112.

Add sanity test_118n() to ensure that OST_STATFS requests are still
async.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Iec8aa378157367f03de96f82d67158b281ec374c
Reviewed-on: https://review.whamcloud.com/30060
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>