LU-7412 osp_md_read() may pass an ERR_PTR() to osp_update_request_destroy()

In osp_md_read() if osp_update_request_create() fails with ERR_PTR()
it should return rather than passing on ERR_PTR() to the
osp_update_request_destroy()

Change-Id: Id4c0c5b3e0619a4e657c22bf27a5679e02164007
Signed-off-by: akam kumar bharathi <azurelustre@gmail.com>
Reviewed-on: http://review.whamcloud.com/17522
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>

LU-3953 build: Only chmod dkms.mkconf once

With AC_CONFIG_FILES, the "commands" parameter (the second one) is
applied for _each_ command, not just once.  That means the existing
chmod command was run many times, and several of the times it runs
it complains because the dkms.mkconf file does not yet exist.

This patch fixes that by giving the dkms.mkconf file its own
AC_CONFIG_FILES macro.

Change-Id: Ic71cc5d8c3555d28ff16efa23d564dce28662443
Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>
Reviewed-on: http://review.whamcloud.com/17516
Tested-by: Jenkins
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7503 utils: add “--verbose|-v” option to “lfs migrate -m”

“lfs mv” has -v option to track the migration progress, which is
very useful for migrating big directory. However, the option is
missing while we changing “lfs mv” to “lfs migrate -m” in commit
849d7d5b1b4cabb7578c3ab5aaf271e90dd33864. This patch adds the option.

Signed-off-by: Jian Yu <jian.yu@intel.com>
Change-Id: I0729f74f46943736c6ed6ade46ca26aee905f550
Reviewed-on: http://review.whamcloud.com/17420
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>

LU-6710 tests: Constrain stripe index conf-sanity test 82a

conf-sanity test 82a specifies the OSTs to stripe a file over.
The OST index is computed as RANDOM * 2 for a maximum of 65534.
Yet, the maximum stripe count is 65532. Thus, the OST index in
conf-sanity test 82a needs to be limited.

Also, change the single use of the deprecated
llapi_stripe_offset_is_valid() to llapi_stripe_index_is_valid().

Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: I4ceb44d639b88527105c1e8812cbd7590d041316
Reviewed-on: http://review.whamcloud.com/15824
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Saurabh Tandan <saurabh.tandan@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-5717 ptlrpc: fix deadlock problem of nrs_tbf_timer_cb

When callback of TBF timer is triggered, nrs_lock could be
held by the current CPU which will cause dead lock. This
patch removes unnecessary nrs_lock to fix this problem.

Signed-off-by: Li Xi <lixi@ddn.com>
Change-Id: I6329e3e71da30a415dbb35b37d79ade118917c6a
Reviewed-on: http://review.whamcloud.com/12228
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>

LU-7210 lnet: Change connect peer failed cleanup order

A race condition has been found where connd is cleaning up failed
connections, the peer ref counter goes to zero, but we stil have
a connecting counter > 0.

One possible race is when we are retrying a connection by
calling kiblnd_connect_peer() which itself fails and decrements
the peer ref counter and gets swapped out before it can decrement
the connecting counter.  connd swaps in and cleans up the
connection where it sees a peer ref counter of 1 and a connecting
counter of 1.  This will trigger the assert seen in LU-7210 when
it decrements the peer counter.

The solution: be sure to decrement the connecting counter
before decrementing the peer counter in the peer connect
failure path.

Signed-off-by: Doug Oucharek <doug.s.oucharek@intel.com>
Change-Id: I2d6ddeae80ac72492a4323a730e3e61c876ebb36
Reviewed-on: http://review.whamcloud.com/17004
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Amir Shehata <amir.shehata@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5718 o2iblnd: Revert original fix

The original fix for this ticket introduced a regression
where bit flags could interfere with each other triggering
asserts.  Also, the focus was on addressing connection
races, but the fix should be expanded to include all
reconnects.

The updated fix is being done under ticket: LU-7569.

Signed-off-by: Doug Oucharek <doug.s.oucharek@intel.com>
Change-Id: I455e43f8a5134f7896ad14c3cd0888b8c08d38d2
Reviewed-on: http://review.whamcloud.com/17699
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Reviewed-by: Amir Shehata <amir.shehata@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7577 mdt: root inode checking for migration

Do not migrate root inode, and add test case
to verify it.

Signed-off-by: Di Wang <di.wang@intel.com>
Change-Id: I8b7a4211d76cbfc1e1b095c6e8f94841d42bc50f
Reviewed-on: http://review.whamcloud.com/17669
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7623 Add __user to seq_write buffer arguments

Updates whole tree and adds forgotten __user attribute,
syncs up prototypes and such.
This keeps sparse happy and helps to ensure user/kernel pointers
correctness.

Change-Id: I54cf7479fffbd8ce211b28f9f3a9de81f600a32e
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/17788
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Frank Zago <fzago@cray.com>

LU-7623 libcfs: Properly map page for vfs_write

Apparently it's possible that the page might not be mapped before
vfs_write, so map it. Also forcefully mark it __user to make
sparse happy.

This was part of upstream kernel commit 7ebcb731 that somehow
never made it back even though vfs_write changes did.

Change-Id: I88494eac08c02d9cac2476eeb56be1eb65cfc3bf
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/17787
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-7623 lnet: Remove __user attribute from lnet_copy_flat2* functions

These functions do not take a user pointer, so __user attribute was a mistake

This was introduced by http://review.whamcloud.com/11819,
commit 4ea13c63f3d919282b12c64d0c014bfe9a67a5e6

Change-Id: I97c1a9b17e3146f9f5f37ac92f7769cfdcf085cb
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/17785
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Frank Zago <fzago@cray.com>

LU-7623 gss: Update upcall structure with user attributes

This helps to better track userspace pointers and keeps sparse happy.

Also fix gss_do_ctx_init_rpc definition and sync up the forward
declaration

Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Change-Id: I10c1a7dd58531886bf8cfa8c684ea6fba321c511
Reviewed-on: http://review.whamcloud.com/17790
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Jeremy Filizetti <jeremy.filizetti@gmail.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Frank Zago <fzago@cray.com>

LU-7085 lov: trying smaller memory allocations

Reduce struct lov_io_sub to smaller memory usage
on wide-stripe file systems.

Signed-off-by: Yang Sheng <yang.sheng@intel.com>
Change-Id: Ie92cec079f6514efca71663af1ae6ae1e34c73cb
Reviewed-on: http://review.whamcloud.com/17476
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5030 libcfs: create cfs_get_paths() function

With the move of the upstream client to sysfs the
lustre and lnet utilites no long should directly
access the proc file system. To avoid this a new
function is created to support both the traditional
procfs lustre/lnet parameters as well as the new
sysfs lustre/lnet parameters as well. Besides the
creation of these functions the LNet utilites are
converted to use these new functions.

Change-Id: I78788f41955073e1e48126103ef2aec13d64812b
Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Signed-off-by: Wang Chao <chao.ornl@gmail.com>
Reviewed-on: http://review.whamcloud.com/17459
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Ryan Haasken <haasken@cray.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>

LU-7506 lfs: "lfs quota -h" should support petabytes output

This patch adds human-readable petabytes output and allows to set
fraction block limit, e.g. 1.5P.
Also, it adds sanity-quota.sh test_27d to verify these new functions,
and replaces some sprintf() with snprintf() to meet the code style
requirement.

Signed-off-by: Emoly Liu <emoly.liu@intel.com>
Change-Id: I1dced352687e8e53988f85d2781396e647b62865
Reviewed-on: http://review.whamcloud.com/17441
Tested-by: Jenkins
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-6401 uapi: change uapi_kernelcomm.h to LGPLv2.1 license

The header uapi_kernelcomm.h need their license to be
changed so it can be used by LGPL library HSM code.

Change-Id: I4ea4fd3017e9e8e286ad9d13ff6b6979cf71359b
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: http://review.whamcloud.com/17720
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Nathan Rutman <nathan.rutman@seagate.com>
Reviewed-by: Henri Doreau <henri.doreau@cea.fr>
Reviewed-by: Frank Zago <fzago@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6613 lnet: change lib-dlc.h to LGPLv2.1 license

Change the lib-dlc.h header to the proper LGPLv2.1 license

Change-Id: Ic152d0722ec775d57cf7702058e6772ce2f5ff8a
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: http://review.whamcloud.com/17697
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-7604 gss: handle revoked keys properly

Revoked keys are no longer returned by request_key kernel
function. So it is now necessary to remove revoked keys from
keyring when flushing context.
Moreover, if a revoked key is present, do not consider it
matches when searching for a valid key with request_key. That
way it will be replaced with a valid, newly created one.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I0fbaf01f6a8f50e9fb06eef96c74e73c25de257f
Reviewed-on: http://review.whamcloud.com/17721
Tested-by: Jenkins
Reviewed-by: Jeremy Filizetti <jeremy.filizetti@gmail.com>
Tested-by: James Nunez <james.a.nunez@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

Revert "LU-7030 security: put imp_sec after all requests drained off"

This is causing non-NULL security flavors to fail during cleanup.

This reverts commit 504ca288d99779812495a91345421ad4ad8f7d95.

Change-Id: I2f370a465c3b2bb49983d13ff98c9e7dec3d15a5
Reviewed-on: http://review.whamcloud.com/17709
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7147 obdclass: remove unused lu_site_stats_print

This should have been removed when lu_site_stats_seq_print was added.

Change-Id: I6003153f6da26b04fd98a3718f44a46b43c67a65
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/16391
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>

LU-7596 kernel: kernel update [SLES11 SP4 3.0.101-68]

Update SLES11 SP4 kernel to 3.0.101-68

Test-Parameters: mdsdistro=sles11sp4 ossdistro=sles11sp4 \
  clientdistro=sles11sp4 mdsfilesystemtype=ldiskfs \
  mdtfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs \
  testgroup=review-ldiskfs

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: Ia9c4ddfe29c172a6dc4d85ac99317ecfa1c5cb56
Reviewed-on: http://review.whamcloud.com/17710
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7421 utils: getparam_display() should reset popt

otherwise it's randomly set leading to wrong output for
lctl get_param. this may break testing.

Change-Id: Ice1ab3ce2ba09a3c6bf6843ff4381ccb73483e42
Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-on: http://review.whamcloud.com/17135
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6824 ldiskfs: give warning with dir htree growing

Currently without large dir feature, ldiskfs directory hash tree
will be limited 2 height, this means directory size is limited about
1GB, and in fact users are likely to hit ENOSPC when reaching half of
limit because of bad hash. tested by following scripts.

i=0
filename="aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaabbbbbbbbbbbbbbbbbbbbbbbb"
while [ 1 ]
do
        touch $filename"$i"
        if [ $? -ne 0 ];then
                break
        fi
        ((i++))
done

When directoy size grow about 590M, we hit ENOSPC. Better way is to
add support to e2fsprogs so that we can use large dir feature.
As a walkaround way, this patch try to give warning messages to
console when 10/16 and 11/16 of limit reach.

So this patch will try to give following messages when warning
limit or hash index tree limit reach:

Directory (inode: 8388610 FID: [0x200000401:0x1:0x0]) has approached
maximum limit.

Directory (inode: 8388610 FID: [0x200000401:0x1:0x0]) is approaching
maximum limit.

FID output here is useful for administrators to locate lustre
file path

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I7f78c421bbb89f76298e0174cc46d774ea82eb06
Reviewed-on: http://review.whamcloud.com/15548
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>

LU-7543 obd: reserve connection flag OBD_CONNECT_FLAGS2

This is a feature for the client and server to use
obd_connect_flags2 to communicate future feature flags. The
client should set this flag whenever any flags in that field
are requested, and the server should mask unsupported features
from this field (assuming it understands OBD_CONNECT_FLAGS2).
When checking if an OBD_CONNECT2_xxxx feature is supported,
the client/server needs to firstly check if OBD_CONNECT_FLAGS2
is supported, since this field is also beyond the end of the
old obd_connect_data.

Land the connection flags to master earlier for reserving the
slot to avoid potential conflict with others.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I466a4f858326d79f8c732116915b078902aa0736
Reviewed-on: http://review.whamcloud.com/17647
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Patrick Farrell <paf@cray.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-7543 obd: reserve connection flag OBD_CONNECT_LOCK_AHEAD

The connection flag OBD_CONNECT_LOCK_AHEAD will be used for the
following the patch: LU-6917 LDLM lock ahead
http://review.whamcloud.com/13564

Land the connection flags to master earlier for reserving the
slot to avoid potential conflict with others.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Ifd5d26d52336b45c962cb34a83bc3582646cfdf4
Reviewed-on: http://review.whamcloud.com/17646
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Patrick Farrell <paf@cray.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-7543 obd: reserve connection flag OBD_CONNECT_OBDOPACK

The connection flag OBD_CONNECT_OBDOPACK will be used for the
following the patch: LU-4215 optimize OUT protocol
http://review.whamcloud.com/15336

Land the connection flags to master earlier for reserving the
slot to avoid potential conflict with others.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I75171f5e9095d40481b4462de74c6a63e1cf3084
Reviewed-on: http://review.whamcloud.com/17645
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Patrick Farrell <paf@cray.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-7356 build: handle library configure options for SNMP/LNet

Currently if you build lustre with configure options
--disable-shared and/or --disable-static it will fail
to build in the case of the mount plugins or just plain
ignore the configure request. This work enables the
ability to create only static or only dynamic libraries
for lustre SNMP and libnetconfig.

Change-Id: I8b526acd50a7b8eb4f65e7a3bdd0a664e9144376
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: http://review.whamcloud.com/17576
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7543 obd: reserve connection flag OBD_CONNECT_SUBTREE

The connection flag OBD_CONNECT_SUBTREE will be used for the
following the patch: LU-28 mounting of filesystem from MDS
http://review.whamcloud.com/5007

Land the connection flags to master earlier for reserving the
slot to avoid potential conflict with others.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I61dad4ff22caf6ab933f38112f2f6e30fc5b9f3e
Reviewed-on: http://review.whamcloud.com/17644
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Patrick Farrell <paf@cray.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-7535 mdt: clear the lock handle in mdt_intent_layout()

In mdt_intent_layout() if the object cannot be found then clear the
lock handle. In recover-small add test_130c() to check that this is
handled condition is handled safely.

In recovery-small test_130[ab]() add a synt to avoid having an
unintended layout intent RPC (from writeback) hit the fault injection
point.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ic2f401ff2dc8a83467b12d146f4d7ffdf70ef5da
Reviewed-on: http://review.whamcloud.com/17735
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7084 obd: correct some OBD allocator macro defines

Use protective parentheses around instances of the GFP mask parameters
in the definitions of __OBD_MALLOC_VERBOSE() and
__OBD_SLAB_ALLOC_VERBOSE() to prevent unexpected expansions.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I0c3307ae488fb4f79a07db8a3b870dbcb609da0b
Reviewed-on: http://review.whamcloud.com/17755
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>

LU-7404 zfs: reset ZFS baseline to 0.6.4.2

ZFS 0.6.5.2 is known to introduce I/O problems with the following
stack backtrace:

Call Trace:
 [] ? vdev_mirror_child_done+0x0/0x30 [zfs]
 [] io_schedule+0x73/0xc0
 [] cv_wait_common+0xaf/0x130 [spl]
 [] ? autoremove_wake_function+0x0/0x40
 [] __cv_wait_io+0x18/0x20 [spl]
 [] zio_wait+0x10b/0x1e0 [zfs]
 [] dbuf_read+0x439/0x850 [zfs]
 [] __dbuf_hold_impl+0x1a1/0x4f0 [zfs]
 [] dbuf_hold_impl+0x7d/0xb0 [zfs]
 [] dbuf_hold+0x20/0x30 [zfs]
 [] dmu_buf_hold_noread+0x87/0x140 [zfs]
 [] dmu_buf_hold+0x3b/0x90 [zfs]
 [] ? dbuf_rele_and_unlock+0x268/0x400 [zfs]
 [] zap_lockdir+0x5a/0x770 [zfs]
 [] ? kmem_cache_alloc_node_trace+0x1cd/0x200
 [] zap_lookup_norm+0x4a/0x190 [zfs]
 [] zap_lookup+0x33/0x40 [zfs]
 [] dmu_tx_hold_zap+0x146/0x210 [zfs]
 [] osd_declare_object_create+0x2a5/0x440 [osd_zfs]
 [] ofd_precreate_objects+0x4e4/0x19d0 [ofd]
 [] ? libcfs_debug_msg+0x41/0x50 [libcfs]
 [] ? ofd_grant_create+0x23b/0x3e0 [ofd]
 [] ofd_create_hdl+0x56e/0x2640 [ofd]
 [] ? lustre_pack_reply_v2+0x220/0x280 [ptlrpc]
 [] tgt_request_handle+0x8bc/0x12e0 [ptlrpc]
 [] ptlrpc_main+0xe41/0x1910 [ptlrpc]
 [] ? ptlrpc_main+0x0/0x1910 [ptlrpc]
 [] kthread+0x9e/0xc0
 [] child_rip+0xa/0x20
 [] ? kthread+0x0/0xc0
 [] ? child_rip+0x0/0x20

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: Ia10427523535ba82cc374e2eae52014cadfac0de
Reviewed-on: http://review.whamcloud.com/17712
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7223 tests: print more information when mmp.sh failed

Print more information when mmp feature failed to be enabled
on OST device.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: Ia049261661202e14fdef838cb8d28b90ce14af84
Reviewed-on: http://review.whamcloud.com/17628
Tested-by: Maloo <hpdd-maloo@intel.com>
Tested-by: Jenkins
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5844 get rid of IOC_LIBCFS_MEMHOG and IOC_LIBCFS_PANIC ioctls

Removing unused IOC_LIBCFS_MEMHOG and IOC_LIBCFS_PANIC ioctls
from the libcfs/ lnet/ code. This cleanup allows us to remove
the cfs_psdev abstraction.

Change-Id: Ifa9b444279f10f0dedf6e31d9f55a5c8cdc6f1af
Signed-off-by: Parinay Kondekar <parinay.kondekar@seagate.com>
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: http://review.whamcloud.com/17492
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7114 ldiskfs: corrupted bitmaps handling patches

Fork patch for SLES12 server support.

Signed-off-by: Yang Sheng <yang.sheng@intel.com>
Change-Id: I56345d71857b2fe19f4e9f137532a0c9285044ae
Reviewed-on: http://review.whamcloud.com/17374
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7451 kernel: kernel upgrade RHEL7.2 [3.10.0-327.3.1.el7]

with this mod we switch our supported el7 version to RHEL 7.2
original release RHEL 7.2 kernel version is 3.10.0-327.el7
current RHEL 7.2 kernel update version is 3.10.0-327.3.1.el7

Test-Parameters: clientdistro=el7 mdsdistro=el7 ossdistro=el7 \
  mdsfilesystemtype=ldiskfs mdtfilesystemtype=ldiskfs \
  ostfilesystemtype=ldiskfs testgroup=review-ldiskfs

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: Idcfbee3875e7114962c92401f782bb035a6e3221
Reviewed-on: http://review.whamcloud.com/17305
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7550 tests: wait OSTs up in check_seq_oid()

This patch fixes check_seq_oid() to wait for OST up after it's
restarted.

Test-Parameters: envdefinitions=ONLY=27 \
testlist=sanity,sanity,sanity,sanity

Test-Parameters: envdefinitions=ONLY=27 \
mdtfilesystemtype=zfs mdsfilesystemtype=zfs ostfilesystemtype=zfs \
testlist=sanity,sanity,sanity,sanity

Test-Parameters: envdefinitions=ONLY=27 \
mdtcount=4 testlist=sanity,sanity,sanity,sanity

Signed-off-by: Jian Yu <jian.yu@intel.com>
Change-Id: I1bab1749f041b7fdbd7f78fa5f5e312586b5cc8c
Reviewed-on: http://review.whamcloud.com/17691
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5017 idl: clean up file attribute flags

Remove unused file attribute flag LUSTRE_BFLAG_UNCOMMITTED_WRITES
that was used internally on the client at one point.

Remove unused file attribute LUSTRE_BFLAG_EXT_FLAGS from wireshark
which was used to distinguish between S_* (in-memory) flags and
*_FL (on-disk) flags.

Add flags from the kernel which may be useful in the near future.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I39d6f26aa0fbe7b5f6ddcf31d97c959556660310
Reviewed-on: http://review.whamcloud.com/10274
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5030 tests: delete old quota test script

The sanity-quota-old.sh test script was for testing new clients
against pre-2.4 servers.  However, testing those old servers is
no longer needed and we can delete this script entirely.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I383949f36dbcfcfab34ec17c687aec0071ea6073
Reviewed-on: http://review.whamcloud.com/17700
Tested-by: Jenkins
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Wei Liu <wei3.liu@intel.com>
Reviewed-by: Ryan Haasken <haasken@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7549 mdt: cancel the atomic lock in the same thread

If cross-MDT lock has ATOMIC_CB flag, let's cancel it in
the same thread as blocking_ast(), to make sure the lock
is cancelled completely after unlock().

This will be used when MDT tries to cancel remote locks
during migration (see mdt_lock_objects_in_linkea()).
Only if the remote lock is cancelled completely, then
the next trylock will succeed.

Signed-off-by: Di Wang <di.wang@intel.com>
Change-Id: Ib23eb8eb312252e6eca680d08f8fcd89cf29339c
Reviewed-on: http://review.whamcloud.com/17577
Tested-by: Jenkins
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7210 o2iblnd: take extra refcount in kiblnd_connreq_done

refcount taken by cmid is not reliable after kiblnd_connreq_done
released the glock because this connection is visible to other
threads, another thread can find and close this connection right
after kiblnd_connreq_done released the glock, if kiblnd_cm_callback
for RDMA_CM_EVENT_DISCONNECTED is called, it can release the
connection refcount taken by cmid. It means the connection could be
destroyed before kiblnd_connreq_done() finish operations on it.

Signed-off-by: Liang Zhen <liang.zhen@intel.com>
Change-Id: Ic49b63551c13abc8c874732de5fd4ea5cef4c6b7
Reviewed-on: http://review.whamcloud.com/17527
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7349 lfsck: skip auto resume lfsck when mount

Usually, if during the LFSCK running the server is umounted or
crashed, then when the server is mounted again, the LFSCK will
resume from the latest checkpoint automatically.

But if the server corruption is caused by the LFSCK, such auto
resuming LFSCK when mount may cause the server to crash again.
To avoid such trouble, introduce new server side mount option
"skip_lfsck" which will prevent unfinished LFSCK auto resuming
when mount.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I773dcde9dd239abe1e82127004ffef28ff1f1ae3
Reviewed-on: http://review.whamcloud.com/16972
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7581 ldiskfs: wrong EA inode backpointer check

EA inode is linked back to the parent inode using
i_mtime.tv_sec filed. An inode number bigger 2G gets
mangled due to sign bit extension over the high bits
of tv_sec. It causes parent backpointer checks to fail.
Add an explicit integer type conversion to ignore high
bits of i_mtime.tv_sec.

Change-Id: I4e086ca5bf13ec77ef0af73fa1f88846e278de23
Seagate-bug-id: MRP-3215
Signed-off-by: Alexander Zarochentsev <alexander.zarochentsev@seagate.com>
Reviewed-on: http://review.whamcloud.com/17675
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7573 test: remove rmmod -w option

In some distro there is no rmmod -w option availale so
just remove -w option.

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I5c22079d4ce9c7ca6b510e7e816408de47b6cd63
Signed-off-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-on: http://review.whamcloud.com/17659
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7534 build: Allow lustre to build against ZFS/SPL DKMS install

The current method to build against DKMS installs of ZFS and SPL
assumes directories like this:

  /var/lib/dkms/spl/${VERSION}/build
  /var/lib/dkms/zfs/${VERSION}/build

Unfortunately, that "build" directory was really just a temporary
step in the DKMS build process.  Older versions of DKMS forgot to
clean up that directory, but in RHEL7 at least, DKMS no longer
leaves that aroudn.

Instead we need to look in these places for source code:

  /var/lib/dkms/spl/${VERSION}/source
  /var/lib/dkms/zfs/${VERSION}/source

and in these places for the per-kernel object info:

  /var/lib/dkms/spl/${VERSION}/${LINUXRELEASE}/${target_cpu}
  /var/lib/dkms/zfs/${VERSION}/${LINUXRELEASE}/${target_cpu}

The changes in this patch update the Lustre build system to
look in those corrected directories for DKMS builds of SPL
and ZFS.

Change-Id: I614b5fc6a6a506cb3f927b2c39ce27160979aaf7
Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>
Reviewed-on: http://review.whamcloud.com/17536
Tested-by: Jenkins
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7555 tests: add load_module to mount_facet

Since commit 2a223541d299bc8 there has not been automated loading
of lustre modules in the log() command, since this command only
needs libcfs loaded to use the "lctl mark" functionality.

When testing from a local build tree, conf-sanity.sh was failing
because it unloads the modules during cleanup and is not able to
automatically load the modules on demand if they are not installed,
and does not always manually load the modules for each running test.

Add a call to load_modules() in mount_facet() where the modules are
actually needed for the filesystem to mount.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: Ie7655c04a7bf42ec9969cb5c11fab3ac92f3dc0b
Reviewed-on: http://review.whamcloud.com/17603
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Jenkins
Reviewed-by: wangdi <di.wang@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Wei Liu <wei3.liu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7301 tests: delete old lfsck tests

The lfsck utility is no longer supported and the corresponding
test scripts and functions can be removed.

Instead of deleting the run_lfsck() routine and LFSCK_ALWAYS check
to run after every test script, update it to run the new lfsck with
all available checks on all targets.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: If7fb8f61e02e09ba346030a3d04d74b9ed3b0c4c
Reviewed-on: http://review.whamcloud.com/16237
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7173 mdt: intent vs unlink race

a race between intent and unlink results in working on stale object,
the intent resend finds the lock and !existent object and LBUG:
    ASSERTION( ((o)->lo_header->loh_attr & LOHA_EXISTS) != 0 )
in mdt_getattr_name_lock()->lu_object_attr(). a check for getattr is
added for !existent object in RESENT case.

another case is when resend finds inode cached on thread,
osd_iget_check() checking the inode returns ESTALE as nlinks==0.
ldlm_lock_enqueue() gets an error from ns_policy() and tries to
destroy the found lock, which is granted, getting "lock still on
resource" LBUG.

It is unclear if the lock reached the client in the original reply,
just leave the lock on server, not returning it again to the client.
Due to LU-6529, the server will not OOM in case the 2nd reply will
be handled on the client.

Signed-off-by: Vitaly Fertman <vitaly.fertman@seagate.com>
Change-Id: I128cd6eeda579c6477bf4564db5e551a46a74d71
Reviewed-on: http://es-gerrit.xyus.xyratex.com:8080/8849
Tested-by: Jenkins
Seagate-bug-id: MRP-3042
Reviewed-by: Alexander Nikolaevich Boyko <alexander.boyko@seagate.com>
Reviewed-by: Andriy Skulysh <andriy.skulysh@seagate.com>
Reviewed-by: Alexey Leonidovich Lyashkov <alexey.lyashkov@seagate.com>
Tested-by: Elena V. Gryaznova <elena.gryaznova@seagate.com>
Reviewed-on: http://review.whamcloud.com/17501
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alexander Boyko <alexander.boyko@seagate.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7531 osp: allow few requests during recovery

Allow OSP requests during recovery, so recovery
threads will not be blocked if the remote target
is also in recovery status, otherwise it might
cause deadlock.

Add replay-single.sh 117 to verify this case.

Signed-off-by: Di Wang <di.wang@intel.com>
Change-Id: Iad3b6fd382d76c9bc042096c51cfac0a0d33091d
Reviewed-on: http://review.whamcloud.com/17539
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7192 tests: conf-sanity test 32c supports DNE on ZFS

1. ZFS needs import pool before tunefs.
2. previous 2.4 ZFS disk image contains 100M file, which will cause
   'migrate' used up all OST space and fail, recreate the image.
3. generate test images for 2.7-formatted ldiskfs and zfs with
   DNE striped directories feature enabled and update t32_test()
   to test that.

Test-Parameters: alwaysuploadlogs envdefinitions=SLOW=yes,ENABLE_QUOTA=yes,ONLY=32 mdtfilesystemtype=zfs mdsfilesystemtype=zfs ostfilesystemtype=zfs mdtcount=4 testlist=conf-sanity
Change-Id: Ib35e4cb80d6beee9695b26844c4e6a99e88adb89
Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Signed-off-by: Jian Yu <jian.yu@intel.com>
Reviewed-on: http://review.whamcloud.com/16663
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>

LU-5710 all: fourth set of corrected typos and grammar errors

All the typos are in comments or logs.

Signed-off-by: frank zago <fzago@cray.com>
Change-Id: If7973c31e73759f7e16c6f8d6a93c6453de8cb2b
Reviewed-on: http://review.whamcloud.com/17306
Tested-by: Jenkins
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7274 gss_do_ctx_fini_rpc() passes an already freed request

ptlrpc_request_free(req) calls either __ptlrpc_free_req_to_pool()
OR ptlrpc_request_cache_free() depending on request from pool /
memory.

ptlrpc_req_finished()-> __ptlrpc_req_finished()->__ptlrpc_free_req()
calls either __ptlrpc_free_req_to_pool() OR ptlrpc_request_cache_free()
too, but depending on reference and request count.

Thus its safe to removed the call to ptlrpc_request_free(req).

Change-Id: Ie2dd647af6b07d2810c458707b791b71c92bc3df
Signed-off-by: akam kumar bharathi <azurelustre@gmail.com>
Reviewed-on: http://review.whamcloud.com/17574
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Jeremy Filizetti <jeremy.filizetti@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6866 hsm: prevent migration of HSM archived files

The reference copytool cannot handle migration of HSM archive
files. In the MDT migration path check for HSM attributes and fail if
they are present. In the LMV layer allow creation of volatile files
with any MDT index. Add a test to sanity-hsm to ensure that attempting
to migrate an HSM archive file is handled safely.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ia133bfb345ce17473f54bd9a0ea0fbff5e5a5c58
Reviewed-on: http://review.whamcloud.com/17511
Tested-by: Jenkins
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6662 utils: allow overriding default mountopts

during formatting the device for Lustre, there are two kinds of
mount options to be used by "mkfs.lustre", which is default and
wanted options. the wanted options is mandatory but the default
should only be used if user doesn't specify it.

Signed-off-by: Hongchao Zhang <hongchao.zhang@intel.com>
Change-Id: I7d8093082455e0a08eb40df63dbc938d6dd8a2a6
Reviewed-on: http://review.whamcloud.com/15870
Tested-by: Jenkins
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-6895 lfsck: drop bad OI files after MDT file-level restore

For a new formatted Lustre system, it does not guarantee that all
the on-disk inodes/blocks are initialized. Instead, they are marked
as free in related inode/block bitmaps. When the inode is allocated
to some object, it will be initialized at that time. Such process
accelerates the format. But it may cause trouble for MDT file-level
backup/restore. For example: the sysadmin backup the MDT via server
side file level tar, then reformat the MDT device, and then restore
the MDT from the backup. Assume some object_A, its FID is the FID_A,
and it was mapped to the inode_B before the backup, such OI mapping
was recorded in the OI file. After the restore, another inode_C is
assigned to the object_A, but before OI scrub rebuilding related
OI mapping, the stale mapping "FID_A => inode_B" is still in the
OI file, and can be found by OI lookup. Generally, even if someone
found the stale OI mapping, it is not trouble, because the OSD will
verify whether FID-in-LMA for the indoe_B matches the FID_A or not.
But if the inode_B is NOT allocated after the restore, and because
we did not initialize inode_B during reformat, then the FID-in-LMA
for the indoe_B is still kept as the same before the backup, means
it matches the FID_A, then the OSD will think that the inode_B is
still assigned to the object_A after the restore. That is wrong.
In futher, although the inode_B is not allocated, but some of the
blocks that were assigned to it may have been allocated to others.
Then accessing the blocks via the inode_B may access some invalid
data, and may trigger some assertion, such as this issue.

So the key issues are two:

1) Some FID based operation may access stale OI mapping after MDT
   file-level backup/restore.

2) The OSD-ldiskfs may get some non-allocated inode with the give
   ino#/generation.

So as long as we can resovle one of them, then the trouble in the
ticket can be resolved.

The solutions for 1):

1.1) Avoid FID based operation before OI scrub done. That is not
     easy, because fid2path cannot be covered by some name based
     operation.

1.2) Remove the OI files after MDT file-level backup/restore. It
     is more direct solution. Another benefit is that even if OI
     scrub rebuilt the OI files, it only guarantees that all the
     FIDs' OI mappings have been refrshed. But it does not clean
     the stale FIDs' OI mappings. Because the OI scrub only does
     inode-table based scanning, not OI files scanning. Removing
     the OI files can resolve related trouble completely.

The solutions for 2):

2.1) New ldiskfs patch to make ldiskfs_iget() to return "-ENOENT"
     for the case of loading non-allocated inode by checking the
     inode bitmap.

2.2) Check the inode's valid inside OSD via related inode bitmap.

Generally, less ldiskfs patches is better. It will safe a lot of
effort when kernel upgrade. So 2.1) is not the best solution. As
for 2.2), it is not good to access the inode bitmap directly in
OSD unless we have to.

Relatively, the solution 1.2) is more efficient and benefit. That
is the current patch does.

On the other hand, this patch also makes some effort for the 1.1):

When start LFSCK, to hold the potential orphans in subsequent system
check, the start process will verify the .lustre/lost+found/ and its
sub-directories firstly. For each MDT, there is one sub-dir named as
.lustre/lost+found/MDTxxxx, which FID is recorded in the LFSCK trace
bookmark file. Originally, the start process uses such FID to locate
the MDTxxxx object. It is not problem for most of cases, but if the
MDT is just restored from file-level backup, and the low layer (OSD)
OI mapping may be invalid, then locating the MDTxxxx object will hit
-EREMCHG failure, and then may cause more troubles:

1) It will try to start OI scrub because of bad OI mapping detected,
   but because we are already in starting the LFSCK, then the logic
   of triggering OI scrub because of LFSCK is confused and difficult
   to be handled.

2) It will misguide the up layer LFSCK to think that the MDTxxxx obj
   does not exist or crashed, then will take some unexpected actions
   to repair such "fake" inconsistency.

The patch make some adjustment of the LFSCK start:

a) Trigger low layer OI scrub before verifying .lustre/lost+found.

b) To verify the .lustre/lost+found/ and its sub-dirs, scanning the
   .lustre/lost+found/ directory via namespace dt_lookup, and check
   the returned FID with the FID in the LFSCK trace bookmark file.
   With such way, even though the MDT is restored from file-level
   backup, the low layer still can locate the right object (inode).

Test-Parameters: alwaysuploadlogs envdefinitions=SLOW=yes,ENABLE_QUOTA=yes mdtfilesystemtype=ldiskfs mdsfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs clientdistro=el7 ossdistro=el7 mdsdistro=el7 mdtcount=1 testlist=sanity-lfsck,sanity-lfsck,sanity-lfsck
Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Icfdab19e69b75400bb39279924c0c3e0b400b4e0
Reviewed-on: http://review.whamcloud.com/17403
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-7430 mdt: better handle MDT recovery error path

This fix ensures the lu_target structure is still valid before
updating export related data in tgt_client_free().

In case of recovery error, it may happen that lu_target and mdt_device
structures are freed before exports are destroyed.

Signed-off-by: Gregoire Pichon <gregoire.pichon@bull.net>
Change-Id: I0559d42f40d4ecc9078717aa07b988be56ce3bc5
Reviewed-on: http://review.whamcloud.com/17424
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7465 llite: Do not track memory leak for kernel function

Do not use OBD_FREE to track memory leak if memory
allocation is from kernel function iov_iter_get_pages_alloc().

Signed-off-by: Wally Wang <wang@cray.com>
Change-Id: If723123bffceb46e2a3b3738043e54c62b8c96cd
Reviewed-on: http://review.whamcloud.com/17342
Reviewed-by: Frank Zago <fzago@cray.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7394 obd: remove HAVE_SELINUX_IS_ENABLED

In obd_mount_server.c unconditionally include linux/selinux.h. Include
linux/selinux.h in xattr_security.c.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Iea7a64d4e66d4c78494da40642111085b95700f9
Reviewed-on: http://review.whamcloud.com/17054
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5030 utils: add -R parameter to lctl get_param

To allow printing all parameters under a specified directory.
This is needed to replace hard-coded /proc pathnames in sanity-sec.sh
test_24, but would also be useful for normal usage.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Signed-off-by: Emoly Liu <emoly.liu@intel.com>
Change-Id: I4391afce06fd63a87f556f7b95bd0cb2883ebbe5
Reviewed-on: http://review.whamcloud.com/17081
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Frank Zago <fzago@cray.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

Revert "LU-6910 osp: add procfs values for OST reserved size"

This is causing LU-7550 and LU-7552 test failures in sanity.

This reverts commit 0585b0fb5895a24f07ca32e830d1fa72b75f4f2b.

Change-Id: Ic332a54ace4998acc4ba2ceab6f76ef733f85be5
Reviewed-on: http://review.whamcloud.com/17617
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-7381 e2fsck: update recommended e2fsprogs version

Update the recommended e2fsprogs version to 1.42.13.wc4.
This includes a number of important fixes to e2fsck, which
fix corruption problems in a number of cases:

   LU-7381 e2fsck: fix e2fsck -fD directory truncation
   - http://review.whamcloud.com/17153
   LU-7267 e2fsck: remove duplicated ea value size check
   - http://review.whamcloud.com/16779
   LU-7368 e2fsck: skip quota update when interrupted
   - http://review.whamcloud.com/17150
   LU-7267 e2fsck: remove duplicated ea value size check
   - http://review.whamcloud.com/16779/
   LU-6722 jbd: double minimum journal size for RHEL7
   - http://review.whamcloud.com/15401

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I1183394eec7717af1a163dbe7e4d93aa453ebbe5
Reviewed-on: http://review.whamcloud.com/17572
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7053 osd: don't lookup object at insert

the idea is to cache FID->ino/type mapping in per-thread cache
at declaration/object creation. then insert can find that information
and don't lookup object in LU/OI. this should avoid potential deadlock
with lu_object_find() and iget(). also, this should improve performance
as in the majority of cases required data is filled locally by create.

stats collected for sanity-benchmark:
lustre-MDT0000: 448306 created, lookups: 8910 in OI, 8910 in FLD
meaning we have to lookup ino 448K times and only 9K times we had
to use OI, in 439K cases we found ino in the cache.

Change-Id: Ifa66c2d074f04e47d0d85b735f57dc506aa65f4c
Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-on: http://review.whamcloud.com/17092
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7408 target: declare write for reply data

declare reply_data at max possible offset - this ensures
enough credits reserved.

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Change-Id: I08466452e1e95b803f316abae777a8c8f4a8626e
Reviewed-on: http://review.whamcloud.com/17086
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7144 tests: skip scrub/lfsck test under interoperation

Since the scrub/lfsck test are only for server side logic,
it is unnecessary to test scrub/lfsck under interoperation
mode, skip them.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I044030b3bace787809d7cfd5622000b44a8be789
Reviewed-on: http://review.whamcloud.com/17520
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7450 osd: call commit_callback if no write updates

If it does not need write updates in some failure cases,
top_trans_stop should also call commit_callback to help
release the top_thandle in the commit list. Otherwise
it will stay in the commit list forever, as well as the
following top thandle, then update logs will be culmulated,
and cause long time recovery.

Signed-off-by: Di Wang <di.wang@intel.com>
Change-Id: I1feaf0bd6d20f14dfabb4572f49818083e697dbb
Reviewed-on: http://review.whamcloud.com/17268
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7209 doc: more accurate documentation for obdfilter-survey

Make the the description of obdfilter-survey accurate and
precise.

Signed-off-by: Richard Henwood <richard.henwood@intel.com>
Change-Id: Icdd4adf53643e91dc8a2539f63977aae5fe28fe0
Reviewed-on: http://review.whamcloud.com/16646
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Omkar Kulkarni <omkar.kulkarni@intel.com>
Tested-by: Omkar Kulkarni <omkar.kulkarni@intel.com>
Reviewed-by: Cliff White <cliff.white@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7419 llog: lock new llog object creation

Lock the new llog object creation to avoid two
process create the same object at the same time.

Signed-off-by: Di Wang <di.wang@intel.com>
Change-Id: Icdc0eec534ca2f15cd0e195df951416953195346
Reviewed-on: http://review.whamcloud.com/17132
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7475 lnet: ensure buffer config symmetry

When showing the configuration, make sure to add a buffers block
in the YAML output, if routing is configured, in order to allow
the same YAML block to be fed back for configuration to LNet.

Signed-off-by: Amir Shehata <amir.shehata@intel.com>
Change-Id: I3b269edf5b3688b500bbb3656c367bc82fff6b68
Reviewed-on: http://review.whamcloud.com/17370
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6020 gss: properly map buffers to sg

A lot of buffer pointers passed to buf_to_sg() as input are coming
from vmalloc(), e.g. OBD_ALLOC_LARGE() in ptlrpc_add_rqs_to_pool().
sg_set_buf() uses virt_to_page() to map virtual addresses to
struct page, which does not work for vmalloc addresses.

The original code for buf_to_sg() caused the following crash:

BUG: unable to handle kernel paging request at ffffeb040057c040
IP: [<ffffffff81300367>] scatterwalk_pagedone+0x27/0x70
PGD 0
Oops: 0000 [#1] SMP
CPU 1
Pid: 2374, comm: ptlrpcd_3 Tainted: G           O 3.6.10-030610-generic
RIP: 0010:[<ffffffff81300367>]  [<ffffffff81300367>] scatterwalk_pagedone+0x27/0x70
RSP: 0018:ffff8801a3c178a8  EFLAGS: 00010282
RAX: ffffeb040057c040 RBX: ffff8801a3c17938 RCX: ffffeb040057c040
RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8801a3c17970
RBP: ffff8801a3c178a8 R08: 00000000000005a8 R09: ffff8801a3c17a40
R10: ffff8801a30370d0 R11: 0000000000000a68 R12: 0000000000000010
R13: ffff8801a3c17a08 R14: ffff8801a3c17970 R15: ffff88017d1c2c80
FS:  0000000000000000(0000) GS:ffff8801afa40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffeb040057c040 CR3: 0000000001c0c000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process ptlrpcd_3 (pid: 2374, threadinfo ffff8801a3c16000, task ffff8801a44e0000)
Stack:
 ffff8801a3c178b8 ffffffff813004bd ffff8801a3c17908 ffffffff8130303f
 ffff880100000000 ffffffff00000000 ffff8801a3c17908 ffff8801a3c17b18
 ffffc90015f015a8 0000000000000000 0000000000000010 0000000000000010
Call Trace:
 [<ffffffff813004bd>] scatterwalk_done+0x3d/0x50
 [<ffffffff8130303f>] blkcipher_walk_done+0x8f/0x230
 [<ffffffff8130a39f>] crypto_cbc_encrypt+0xff/0x190
 [<ffffffffa0688660>] ? aes_decrypt+0x80/0x80 [aesni_intel]
 [<ffffffffa0a1a1e4>] krb5_encrypt_bulk+0x164/0x5b0 [ptlrpc_gss]
 [<ffffffffa0a1a812>] gss_wrap_bulk_kerberos+0x1e2/0x490 [ptlrpc_gss]
 [<ffffffffa0a1600e>] lgss_wrap_bulk+0x2e/0x100 [ptlrpc_gss]
 [<ffffffffa0a0d98e>] gss_cli_ctx_wrap_bulk+0x44e/0x650 [ptlrpc_gss]
 [<ffffffffa0ab867c>] sptlrpc_cli_wrap_bulk+0x3c/0x70 [ptlrpc]
 [<ffffffffa0aba2d0>] sptlrpc_cli_wrap_request+0x60/0x360 [ptlrpc]
 [<ffffffffa0a8cde4>] ptl_send_rpc+0x164/0xc30 [ptlrpc]
 [<ffffffffa07be957>] ? libcfs_debug_msg+0x47/0x50 [libcfs]
 [<ffffffffa0a80ee0>] ptlrpc_send_new_req+0x3b0/0x940 [ptlrpc]
 [<ffffffffa0a86530>] ptlrpc_check_set+0x8e0/0x1d50 [ptlrpc]
 [<ffffffff816ac9f6>] ? schedule_timeout+0x146/0x260
 [<ffffffffa0ab0c9b>] ptlrpcd_check+0x4eb/0x5d0 [ptlrpc]
 [<ffffffffa0ab105f>] ptlrpcd+0x2df/0x420 [ptlrpc]
 [<ffffffff8108efa0>] ? try_to_wake_up+0x200/0x200
 [<ffffffffa0ab0d80>] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc]
 [<ffffffff8107c5f3>] kthread+0x93/0xa0
 [<ffffffff816b8d04>] kernel_thread_helper+0x4/0x10
 [<ffffffff8107c560>] ? flush_kthread_worker+0xb0/0xb0
 [<ffffffff816b8d00>] ? gs_change+0x13/0x13

Change-Id: I346d50568b65ed10da2762ca34562fc2858a05d8
Signed-off-by: Andrew Perepechko <andrew.perepechko@seagate.com>
Xyratex-bug-id: SNT-15
Reviewed-on: http://review.whamcloud.com/17319
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5690 mount: fix lmd_parse() to handle commas in expr_list

The lmd_parse() function parses mount options with comma as
delimiter without considering commas in expr_list as follows
is a valid LNET nid range syntax:

<expr_list>  :== '[' <range_expr> [ ',' <range_expr>] ']'

This patch fixes the above issue by using cfs_parse_nidlist()
to parse nid range list instead of using class_parse_nid_quiet()
to parse only one nid.

Signed-off-by: Jian Yu <jian.yu@intel.com>
Change-Id: I8ba6ee6eb31b4bb078a83d9db213cfca27b0fe66
Reviewed-on: http://review.whamcloud.com/17036
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6714 llog: test on-disk llog header values

llog_test_2():
- re-enable the disabled llog_open() test cases.
- Checks that llog_log_hdr values are written atomically
  with the llog record addition/cancelling.

Patch contains also minor fixes:
- verify_handle() does header sanity checks at first then
  checks amount of records against expected value.
- llog_test_3: rename test_3 static variables to show
  that they are related to the test_3.

Signed-off-by: Mikhail Pershin <mike.pershin@intel.com>
Change-Id: Iedcf15c8f365f9c2021abae3325edcaf08efc4c9
Reviewed-on: http://review.whamcloud.com/16287
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7030 security: put imp_sec after all requests drained off

imp_sec should be put after all requests being drained off.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I35f572fcc79b2bd1991db14577226a3ea735630d
Reviewed-on: http://review.whamcloud.com/16071
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Sebastien Buisson <sebastien.buisson@bull.net>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7530 mdt: Do not leak identity when no nodemap is present

It looks like sometimes nodemap structure on the export is not there
due to a race in old_init_ucred_common.
Move the nodemap check to the start not to leak identity reference
in such a case.

The bug was introduced in commit 2aea469a3a6e214d from LU-7199

Also silence the warning as there's nothing sysadmins could do when
it happens.

Change-Id: I5329ccb16201a71a263eb586e3a486b26ff238db
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/17519
Tested-by: Jenkins
Reviewed-by: Kit Westneat <kit.westneat@gmail.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-6910 osp: add procfs values for OST reserved size

osp_pre_status=-ENOSPC is used to skip OST from object allocation.
The error was set when OST available space is less than 0.1% of total
OST size. This value is not configurable, so procfs files was
added:
reserved_mb_low - low watermark, if available space is less
  than it, object allocation is stopped.
reserved_mb_high - highw watermark, if available space is more
   than it, object allocation is enabled.

By default ~0.1% is reserved as low watermark. The high watermark
is twice bigger than the low by default.
High and low watermark could be changed by:
lctl set_param osp.lustre-OST0000-osc-MDT0000.reserved_mb_high=1024

When object allocation is disabled, a clients could appened to
existing files. And 0.1% is too low for them. For example, OST size
is 8TB, 0.1% is 8GB, if cluster has 1k clients, reserved space is
~8MB per client. The main reason of the patch is ability to increase
reserved space.

Signed-off-by: Alexander Boyko <alexander.boyko@seagate.com>
Xyratex-bug-id: MRP-2606
Change-Id: Ie48cc1a232f64aa7dc922000861004277fb47340
Reviewed-on: http://review.whamcloud.com/15731
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@seagate.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7136 test: allow more time for copytools to stop

In sanity-hsm allow up to 200 seconds for the copytool to stop. This
is needed to prevent sporadic failures due to slow NFS (for the
copytool log file) delaying the termination of the copytool.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Icafc4e9c5a00c849dcb479233826de058d2ede62
Reviewed-on: http://review.whamcloud.com/17499
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6767 osd-zfs: Track readonly status of ZFS

Return READONLY from osd_statfs() if underlying ZFS has been set to
READONLY, or if osd_ro() has been called.  This adds a callback for
ZFS_PROP_READONLY for when it's changed.

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Ib7f35925904b1d93f9a457936585e9783635c849
Reviewed-on: http://review.whamcloud.com/15400
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6229 utils: fix lustre_rsync bug of cascade move

When replaying the changelog, destination files have to be
put into a special directory, if their parent directory is
possessing a different path other than the ultimate path
because of renaming. With the replaying process going on,
when the parent directory is being moved to the ultimate path,
the child files should be moved under the parent directory
which is called cascade move.

As long as a directory has child files under sepcial direcoty,
cascade move should happen, no matter the direcotry is being
renamed from sepcial direcoty or not. This patch fixes the problem
that cascade move is missing when the direcotry is being renamed
from ordinary path.

Signed-off-by: Li Xi <lixi@ddn.com>
Change-Id: I2d21604b81fe0cf08df1af2bfccc90a32986bf05
Reviewed-on: http://review.whamcloud.com/14914
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7515 obdclass: add export for lprocfs_stats_alloc_one()

 When compiling Lustre without optimization, when using GCOV,
 the lprocfs_stats_alloc_one() symbol is not properly exported to other modules
 and causes the ptlrpc module to fail loading with an unknown symbol.
 Added EXPORT_SYMBOL(lprocfs_stats_alloc_one) so that this works properly.

Seagate-bug-id: MRP-3188
Signed-off-by: Chennaiah Palla <chennaiah.palla@seagate.com>
Change-Id: I8ef02a0e0bf519fa93f85cb162a6340e3feeb736
Reviewed-on: http://review.whamcloud.com/17443
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3569 utils: remove ll_recover_lost_found_obj

remove obsolete tool ll_recover_lost_found_obj.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Change-Id: I5d5f33f5c9d68bb1f05d7ab0da6fb2986e873501
Reviewed-on: http://review.whamcloud.com/16477
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7446 clio: lov_io_init() should return error code

lov_io_init_empty/release() should returns error code instead of
true on error case.

Fault IO need handle restart in the case of accessing HSM released
file.

Add a test case.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I4953c12c1e9b82a16aed9b8b1e3fe6e38d783b24
Reviewed-on: http://review.whamcloud.com/17240
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6732 llite: ll_write_begin/end not passing on errors

Because of a implementation of generic_perform_write(), write(2)
may return 0 with no errno even if EDQUOT or ENOSPC actually
happend in it.
This patch fixes the issue with setting a proper errno to
ci_result and get it in ll_file_io_generic.

Signed-off-by: Hiroya Nozaki <nozaki.hiroya@jp.fujitsu.com>
Change-Id: I3fc986b57d703ad5fbf41e1ea8182d2d561e8005
Reviewed-on: http://review.whamcloud.com/15302
Tested-by: Maloo <hpdd-maloo@intel.com>
Tested-by: Jenkins
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1606 misc: clean up DFID related error messages

Improve the error messages related to DFID output and parsing left
over from removal of LPU64/LPX64 usage in userspace.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I4b4fcb3cc389b8d8ec4375fa92bfee9b353ebbe5
Reviewed-on: http://review.whamcloud.com/6156
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Frank Zago <fzago@cray.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2524 test: Clean up sanity-quota

Conduct miscellaneous cleanup to sanity-quota including:
Removing the `-p` (parents) option from many calls to mkdir
Replace `lfs setstripe` with $SETSTRIPE
Added check for and call to `error` with error messages for a variety
of common routines ,like mkdir, or for functions that return a value.
Replace `…` with $(...)
Removed linefeed escape after |, ||, & and && operators.
Modified parameters in test 4b so that the --inode-grace value exceeds
the valid range.
Removed unused variables
Removed $ from variables inside $(())

Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: Iadedea0bd0a0f85235e0bb908ee0a6ed36503eb3
Reviewed-on: http://review.whamcloud.com/14680
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Wei Liu <wei3.liu@intel.com>
Reviewed-by: Saurabh Tandan <saurabh.tandan@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7148 osc: Remove remains of osc_ast_guard

osc_ast_guard has been removed by the clio simplification.
Remove the extern declaartion and lock class definition.

Change-Id: Ibcf14e7aebe1dab8b586d3cd8d81560f6d3dcc81
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/16392
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>

LU-5951 ptlrpc: track unreplied requests

The request xid was used to make sure the ost object timestamps
being updated by the out of order setattr/punch/write requests
properly. However, this mechanism is broken by the multiple rcvd
slot feature, where we deferred the xid assignment from request
packing to request sending.

This patch moved back the xid assignment to request packing, and
the manner of finding lowest unreplied xid is changed from scan
sending & delay list to scan a unreplied requests list.

This patch also skipped packing the known replied XID in connect
and disconnect request, so that we can make sure the known replied
XID is increased only on both server & client side.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: Ic98e1599085871c0ac08d28609a044c79d5af75d
Reviewed-on: http://review.whamcloud.com/16759
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Grégoire Pichon <gregoire.pichon@bull.net>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7508 ldlm: Don't check opcode with NULL rq_reqmsg

When GSS is enabled it's possible to have a NULL rq_reqmsg
if a bad signature or no context is returned during the unwrap
of the request.  Don't check the opcode in this case.

Signed-off-by: Jeremy Filizetti <jeremy.filizetti@gmail.com>
Change-Id: I3a74dff7638b318190c5c4ad73acbe7ec299aa80
Reviewed-on: http://review.whamcloud.com/17414
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-7268 scrub: NOT assign LMA for EA inode

Originally, when OI scrub scans the device, if the target inode has
no FID-in-LMA EA, then it will generate an IGIF mode FID and store
it in the LMA EA. Such behavior is not suitable if the target inode
is used for large EA. The OI scrub should skip the EA inode that is
marked as "LDISKFS_EA_INODE_FL".

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I52b05b864ef8a2797a2f3dda0f80f95227809c34
Reviewed-on: http://review.whamcloud.com/17043
Tested-by: Jenkins
Reviewed-by: Kalpak Shah <kalpak.shah@seagate.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-6298 hsm: shutdown HSM CDTs in parallel

In sanity-hsm.sh rewrite copytool_cleanup() to shutdown and restart
the MDT HSM coordinators in parallel. This saves about 8 * (MDSCOUNT -
1) seconds per call.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I75445ad126dc73251a3d056611133e3ab6b83362
Reviewed-on: http://review.whamcloud.com/13901
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5921 tests: enhance server target mount race testing

This patch is a follow on to LU-5299 to strengthen and enhance
concurrent server target mount race testing.
It uses OBD_RACE() feature to better set a concurrent/racy
situation, and also allow to handle all mount errors instead
of only EALREADY.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: I16a94e5aa046e15096d2e55d57e22899a93fa03f
Reviewed-on: http://review.whamcloud.com/17302
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7383 mdt: retry for busy lock during migration

In migration, if the lock of the migrating object
is being cached on other node, it should revoke
the lock and retry, instead of return -EBUSY.

Signed-off-by: Di Wang <di.wang@intel.com>
Change-Id: I1317681a892b9a21f2c78d7696ca6f94d43bd9bc
Reviewed-on: http://review.whamcloud.com/17048
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

New tag 2.7.64

Change-Id: I79fb95af8bc9e979edf3214315219f786eb12599
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7428 test: disable conf-sanity, test_84

Add failing test to ALWAYS_EXCEPT.
This is a temprorary workaround until a real
fix for the test failure is developed.

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I73e00d658a9e7728ce52b5dc90741e9a18ce15f9
Reviewed-on: http://review.whamcloud.com/17482
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-7437 lctl: list_param -R can't work correctly

We shouldn't call lprocfs_param_pattern() inside listparam_display(),
otherwise it will add the prefix "/proc/{fs,sys}/{lnet,lustre}" each
time, so that the parameters can be listed recursively. The similar
issue in {set/get}param is fixed as well.

Also, this patch adds sanity.sh test_401 to verify this function.

Signed-off-by: Emoly Liu <emoly.liu@intel.com>
Change-Id: I21acec364cdbdfc025979153f66a87d44c9136e8
Reviewed-on: http://review.whamcloud.com/17223
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7297 osd-zfs: initialize oh_lock

The ZFS osd was not initializing od_brw_stats.hist[].oh_lock.
This rectifies that.

Change-Id: I3f637b73c77908c2297bfab97e33eca63b0d5986
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Reviewed-on: http://review.whamcloud.com/16919
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>

LU-1026 ldiskfs: make bitmaps corruption not fatal

We still hit bitmaps problems for rhel6 series kernel,
corruptions happen because ext4_mb_check_ondisk_bitmap()
check failed and FS become RO again:

ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group
294corrupted: 20180 blocks free in bitmap, 20181 - in gd
Aborting journal on device dm-6-8.
LDISKFS-fs (dm-6): Remounting filesystem read-only
ldiskfs_mb_new_blocks: Updating bitmap error: [err -30]
 [pa ffff880d9d6e4d68] [phy 14974976] [logic 8192] [len 3072]
 [free 3072] [error 1] [inode 278678]
ldiskfs_ext_new_extent_cb: Journal has aborted

this might be caused by some ext4 internal bugs, this patch
did the following things:

1.Inside ext4_read_block_bitmap() have gaven reasons
why it failed, so caller don't need call ext4_error() again.
2. mark block group corrupt and use ext4_warning() instead
of ext4_error().

There are still some bitmaps corruption places not handling,
let's keep it for now, and if it really hurt, let's add the
same handling codes logic later.

Tested by following scripts:

TEST_DEV="/dev/sdb"
TEST_MNT="/mnt/ext4"

mkdir -p $TEST_MNT
mkfs.ext4 -F $TEST_DEV >&/dev/null

mount -t ldiskfs $TEST_DEV $TEST_MNT
dd if=/dev/zero of=$TEST_MNT/largefile
oflag=direct bs=10485760 count=200
umount $TEST_MNT
dd if=/dev/zero of=$TEST_DEV bs=4096 seek=641
count=10 oflag=direct
mount -t ldiskfs $TEST_DEV $TEST_MNT
rm -f $TEST_MNT/largefile
dd if=/dev/zero of=$TEST_MNT/largefile oflag=direct
bs=10485760 count=200 && echo
  "FILESYSTEM still usable after bitmaps corrupts happen"
dmesg | tail
umount $TEST_MNT
e2fsck $TEST_DEV -y

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: Iabb6ebf719d80d9ba4f41bee0b237e304212832b
Reviewed-on: http://review.whamcloud.com/16679
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7315 osd-ldiskfs: handle pdo lock properly

Inside the osd_dirent_check_repair(), if the logic comes to
"goto again", it only unlock the "hlock" but without seting
the variable @hlock as NULL. Althouth it will not cause any
logic failure, it may make the readers to be confused. This
patch will set "hlock = NULL;" explicitly to avoid trouble.

On the other hand, inside ldiskfs, the pdo lock users need
to check whether the lock handler is NULL or not properly.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I9db9dc758a2976849c299f76e06723e796da235d
Reviewed-on: http://review.whamcloud.com/16924
Tested-by: Jenkins
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6856 zfs: handle non existing file in osd_object_ref_del

Remove false assertion in zfs:osd_object_ref_del() because this
may be in the cleanning path of error handling.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: Ib7b9d80816bdab7f68b36a33e95140ea7f3eae8c
Reviewed-on: http://review.whamcloud.com/15611
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>

LU-6802 ptlrpc: reset imp_replay_cursor

At client side, the replay cursor using to speed up the lookup
of committed open requests in its obd_import should be resetted
for normal connection (not reconnection) during recovery.

Change-Id: I68816780a5d79053d9109cb68ae1c3b8ea13ede8
Signed-off-by: Hongchao Zhang <hongchao.zhang@intel.com>
Reviewed-on: http://review.whamcloud.com/17351
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6693 out: not return NULL in object_update_param_get

Return ERR_PTR in object_update_param_get() for all cases to
avoid unnecessary confusion to callers.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: Idfcc19d99bbf308759481b3d60d95341745d19e8
Reviewed-on: http://review.whamcloud.com/16417
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7316 build: Update ZFS/SPL version to 0.6.5.3

Bug Fixes

* Fix CPU hotplug zfsonlinux/spl#482
* Disable dynamic taskqs by default to avoid deadlock
  zfsonlinux/spl#484
* Don't import all visible pools in zfs-import init script
  zfsonlinux/zfs#3777
* Fix use-after-free in vdev_disk_physio_completion
  zfsonlinux/zfs#3920
* Fix avl_is_empty(&dn->dn_dbufs) assertion zfsonlinux/zfs#3865

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I36347630be2506bee4ff0a05f1b236ba2ba7a0ae
Reviewed-on: http://review.whamcloud.com/16877
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-4423 lnet: don't use iovec instead of kvec

Replace struct iovec with struct kvec.

Linux commit: f351bad2b4b4bb74810ad4f127f6602e2d2ae403

Change-Id: Ib7bb49069e42ca82d66a149617361c73ee4d710d
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-on: http://review.whamcloud.com/17205
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Frank Zago <fzago@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>