LU-7543 obd: reserve connection flag OBD_CONNECT_OBDOPACK

The connection flag OBD_CONNECT_OBDOPACK will be used for the
following the patch: LU-4215 optimize OUT protocol
http://review.whamcloud.com/15336

Land the connection flags to master earlier for reserving the
slot to avoid potential conflict with others.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I75171f5e9095d40481b4462de74c6a63e1cf3084
Reviewed-on: http://review.whamcloud.com/17645
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Patrick Farrell <paf@cray.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-7356 build: handle library configure options for SNMP/LNet

Currently if you build lustre with configure options
--disable-shared and/or --disable-static it will fail
to build in the case of the mount plugins or just plain
ignore the configure request. This work enables the
ability to create only static or only dynamic libraries
for lustre SNMP and libnetconfig.

Change-Id: I8b526acd50a7b8eb4f65e7a3bdd0a664e9144376
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: http://review.whamcloud.com/17576
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7543 obd: reserve connection flag OBD_CONNECT_SUBTREE

The connection flag OBD_CONNECT_SUBTREE will be used for the
following the patch: LU-28 mounting of filesystem from MDS
http://review.whamcloud.com/5007

Land the connection flags to master earlier for reserving the
slot to avoid potential conflict with others.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I61dad4ff22caf6ab933f38112f2f6e30fc5b9f3e
Reviewed-on: http://review.whamcloud.com/17644
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Patrick Farrell <paf@cray.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-7535 mdt: clear the lock handle in mdt_intent_layout()

In mdt_intent_layout() if the object cannot be found then clear the
lock handle. In recover-small add test_130c() to check that this is
handled condition is handled safely.

In recovery-small test_130[ab]() add a synt to avoid having an
unintended layout intent RPC (from writeback) hit the fault injection
point.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ic2f401ff2dc8a83467b12d146f4d7ffdf70ef5da
Reviewed-on: http://review.whamcloud.com/17735
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7084 obd: correct some OBD allocator macro defines

Use protective parentheses around instances of the GFP mask parameters
in the definitions of __OBD_MALLOC_VERBOSE() and
__OBD_SLAB_ALLOC_VERBOSE() to prevent unexpected expansions.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I0c3307ae488fb4f79a07db8a3b870dbcb609da0b
Reviewed-on: http://review.whamcloud.com/17755
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>

LU-7404 zfs: reset ZFS baseline to 0.6.4.2

ZFS 0.6.5.2 is known to introduce I/O problems with the following
stack backtrace:

Call Trace:
 [] ? vdev_mirror_child_done+0x0/0x30 [zfs]
 [] io_schedule+0x73/0xc0
 [] cv_wait_common+0xaf/0x130 [spl]
 [] ? autoremove_wake_function+0x0/0x40
 [] __cv_wait_io+0x18/0x20 [spl]
 [] zio_wait+0x10b/0x1e0 [zfs]
 [] dbuf_read+0x439/0x850 [zfs]
 [] __dbuf_hold_impl+0x1a1/0x4f0 [zfs]
 [] dbuf_hold_impl+0x7d/0xb0 [zfs]
 [] dbuf_hold+0x20/0x30 [zfs]
 [] dmu_buf_hold_noread+0x87/0x140 [zfs]
 [] dmu_buf_hold+0x3b/0x90 [zfs]
 [] ? dbuf_rele_and_unlock+0x268/0x400 [zfs]
 [] zap_lockdir+0x5a/0x770 [zfs]
 [] ? kmem_cache_alloc_node_trace+0x1cd/0x200
 [] zap_lookup_norm+0x4a/0x190 [zfs]
 [] zap_lookup+0x33/0x40 [zfs]
 [] dmu_tx_hold_zap+0x146/0x210 [zfs]
 [] osd_declare_object_create+0x2a5/0x440 [osd_zfs]
 [] ofd_precreate_objects+0x4e4/0x19d0 [ofd]
 [] ? libcfs_debug_msg+0x41/0x50 [libcfs]
 [] ? ofd_grant_create+0x23b/0x3e0 [ofd]
 [] ofd_create_hdl+0x56e/0x2640 [ofd]
 [] ? lustre_pack_reply_v2+0x220/0x280 [ptlrpc]
 [] tgt_request_handle+0x8bc/0x12e0 [ptlrpc]
 [] ptlrpc_main+0xe41/0x1910 [ptlrpc]
 [] ? ptlrpc_main+0x0/0x1910 [ptlrpc]
 [] kthread+0x9e/0xc0
 [] child_rip+0xa/0x20
 [] ? kthread+0x0/0xc0
 [] ? child_rip+0x0/0x20

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: Ia10427523535ba82cc374e2eae52014cadfac0de
Reviewed-on: http://review.whamcloud.com/17712
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7223 tests: print more information when mmp.sh failed

Print more information when mmp feature failed to be enabled
on OST device.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: Ia049261661202e14fdef838cb8d28b90ce14af84
Reviewed-on: http://review.whamcloud.com/17628
Tested-by: Maloo <hpdd-maloo@intel.com>
Tested-by: Jenkins
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5844 get rid of IOC_LIBCFS_MEMHOG and IOC_LIBCFS_PANIC ioctls

Removing unused IOC_LIBCFS_MEMHOG and IOC_LIBCFS_PANIC ioctls
from the libcfs/ lnet/ code. This cleanup allows us to remove
the cfs_psdev abstraction.

Change-Id: Ifa9b444279f10f0dedf6e31d9f55a5c8cdc6f1af
Signed-off-by: Parinay Kondekar <parinay.kondekar@seagate.com>
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: http://review.whamcloud.com/17492
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7114 ldiskfs: corrupted bitmaps handling patches

Fork patch for SLES12 server support.

Signed-off-by: Yang Sheng <yang.sheng@intel.com>
Change-Id: I56345d71857b2fe19f4e9f137532a0c9285044ae
Reviewed-on: http://review.whamcloud.com/17374
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7451 kernel: kernel upgrade RHEL7.2 [3.10.0-327.3.1.el7]

with this mod we switch our supported el7 version to RHEL 7.2
original release RHEL 7.2 kernel version is 3.10.0-327.el7
current RHEL 7.2 kernel update version is 3.10.0-327.3.1.el7

Test-Parameters: clientdistro=el7 mdsdistro=el7 ossdistro=el7 \
  mdsfilesystemtype=ldiskfs mdtfilesystemtype=ldiskfs \
  ostfilesystemtype=ldiskfs testgroup=review-ldiskfs

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: Idcfbee3875e7114962c92401f782bb035a6e3221
Reviewed-on: http://review.whamcloud.com/17305
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7550 tests: wait OSTs up in check_seq_oid()

This patch fixes check_seq_oid() to wait for OST up after it's
restarted.

Test-Parameters: envdefinitions=ONLY=27 \
testlist=sanity,sanity,sanity,sanity

Test-Parameters: envdefinitions=ONLY=27 \
mdtfilesystemtype=zfs mdsfilesystemtype=zfs ostfilesystemtype=zfs \
testlist=sanity,sanity,sanity,sanity

Test-Parameters: envdefinitions=ONLY=27 \
mdtcount=4 testlist=sanity,sanity,sanity,sanity

Signed-off-by: Jian Yu <jian.yu@intel.com>
Change-Id: I1bab1749f041b7fdbd7f78fa5f5e312586b5cc8c
Reviewed-on: http://review.whamcloud.com/17691
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5017 idl: clean up file attribute flags

Remove unused file attribute flag LUSTRE_BFLAG_UNCOMMITTED_WRITES
that was used internally on the client at one point.

Remove unused file attribute LUSTRE_BFLAG_EXT_FLAGS from wireshark
which was used to distinguish between S_* (in-memory) flags and
*_FL (on-disk) flags.

Add flags from the kernel which may be useful in the near future.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I39d6f26aa0fbe7b5f6ddcf31d97c959556660310
Reviewed-on: http://review.whamcloud.com/10274
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5030 tests: delete old quota test script

The sanity-quota-old.sh test script was for testing new clients
against pre-2.4 servers.  However, testing those old servers is
no longer needed and we can delete this script entirely.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I383949f36dbcfcfab34ec17c687aec0071ea6073
Reviewed-on: http://review.whamcloud.com/17700
Tested-by: Jenkins
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Wei Liu <wei3.liu@intel.com>
Reviewed-by: Ryan Haasken <haasken@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7549 mdt: cancel the atomic lock in the same thread

If cross-MDT lock has ATOMIC_CB flag, let's cancel it in
the same thread as blocking_ast(), to make sure the lock
is cancelled completely after unlock().

This will be used when MDT tries to cancel remote locks
during migration (see mdt_lock_objects_in_linkea()).
Only if the remote lock is cancelled completely, then
the next trylock will succeed.

Signed-off-by: Di Wang <di.wang@intel.com>
Change-Id: Ib23eb8eb312252e6eca680d08f8fcd89cf29339c
Reviewed-on: http://review.whamcloud.com/17577
Tested-by: Jenkins
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7210 o2iblnd: take extra refcount in kiblnd_connreq_done

refcount taken by cmid is not reliable after kiblnd_connreq_done
released the glock because this connection is visible to other
threads, another thread can find and close this connection right
after kiblnd_connreq_done released the glock, if kiblnd_cm_callback
for RDMA_CM_EVENT_DISCONNECTED is called, it can release the
connection refcount taken by cmid. It means the connection could be
destroyed before kiblnd_connreq_done() finish operations on it.

Signed-off-by: Liang Zhen <liang.zhen@intel.com>
Change-Id: Ic49b63551c13abc8c874732de5fd4ea5cef4c6b7
Reviewed-on: http://review.whamcloud.com/17527
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7349 lfsck: skip auto resume lfsck when mount

Usually, if during the LFSCK running the server is umounted or
crashed, then when the server is mounted again, the LFSCK will
resume from the latest checkpoint automatically.

But if the server corruption is caused by the LFSCK, such auto
resuming LFSCK when mount may cause the server to crash again.
To avoid such trouble, introduce new server side mount option
"skip_lfsck" which will prevent unfinished LFSCK auto resuming
when mount.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I773dcde9dd239abe1e82127004ffef28ff1f1ae3
Reviewed-on: http://review.whamcloud.com/16972
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7581 ldiskfs: wrong EA inode backpointer check

EA inode is linked back to the parent inode using
i_mtime.tv_sec filed. An inode number bigger 2G gets
mangled due to sign bit extension over the high bits
of tv_sec. It causes parent backpointer checks to fail.
Add an explicit integer type conversion to ignore high
bits of i_mtime.tv_sec.

Change-Id: I4e086ca5bf13ec77ef0af73fa1f88846e278de23
Seagate-bug-id: MRP-3215
Signed-off-by: Alexander Zarochentsev <alexander.zarochentsev@seagate.com>
Reviewed-on: http://review.whamcloud.com/17675
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7573 test: remove rmmod -w option

In some distro there is no rmmod -w option availale so
just remove -w option.

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I5c22079d4ce9c7ca6b510e7e816408de47b6cd63
Signed-off-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-on: http://review.whamcloud.com/17659
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7534 build: Allow lustre to build against ZFS/SPL DKMS install

The current method to build against DKMS installs of ZFS and SPL
assumes directories like this:

  /var/lib/dkms/spl/${VERSION}/build
  /var/lib/dkms/zfs/${VERSION}/build

Unfortunately, that "build" directory was really just a temporary
step in the DKMS build process.  Older versions of DKMS forgot to
clean up that directory, but in RHEL7 at least, DKMS no longer
leaves that aroudn.

Instead we need to look in these places for source code:

  /var/lib/dkms/spl/${VERSION}/source
  /var/lib/dkms/zfs/${VERSION}/source

and in these places for the per-kernel object info:

  /var/lib/dkms/spl/${VERSION}/${LINUXRELEASE}/${target_cpu}
  /var/lib/dkms/zfs/${VERSION}/${LINUXRELEASE}/${target_cpu}

The changes in this patch update the Lustre build system to
look in those corrected directories for DKMS builds of SPL
and ZFS.

Change-Id: I614b5fc6a6a506cb3f927b2c39ce27160979aaf7
Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>
Reviewed-on: http://review.whamcloud.com/17536
Tested-by: Jenkins
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7555 tests: add load_module to mount_facet

Since commit 2a223541d299bc8 there has not been automated loading
of lustre modules in the log() command, since this command only
needs libcfs loaded to use the "lctl mark" functionality.

When testing from a local build tree, conf-sanity.sh was failing
because it unloads the modules during cleanup and is not able to
automatically load the modules on demand if they are not installed,
and does not always manually load the modules for each running test.

Add a call to load_modules() in mount_facet() where the modules are
actually needed for the filesystem to mount.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: Ie7655c04a7bf42ec9969cb5c11fab3ac92f3dc0b
Reviewed-on: http://review.whamcloud.com/17603
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Jenkins
Reviewed-by: wangdi <di.wang@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Wei Liu <wei3.liu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7301 tests: delete old lfsck tests

The lfsck utility is no longer supported and the corresponding
test scripts and functions can be removed.

Instead of deleting the run_lfsck() routine and LFSCK_ALWAYS check
to run after every test script, update it to run the new lfsck with
all available checks on all targets.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: If7fb8f61e02e09ba346030a3d04d74b9ed3b0c4c
Reviewed-on: http://review.whamcloud.com/16237
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7173 mdt: intent vs unlink race

a race between intent and unlink results in working on stale object,
the intent resend finds the lock and !existent object and LBUG:
    ASSERTION( ((o)->lo_header->loh_attr & LOHA_EXISTS) != 0 )
in mdt_getattr_name_lock()->lu_object_attr(). a check for getattr is
added for !existent object in RESENT case.

another case is when resend finds inode cached on thread,
osd_iget_check() checking the inode returns ESTALE as nlinks==0.
ldlm_lock_enqueue() gets an error from ns_policy() and tries to
destroy the found lock, which is granted, getting "lock still on
resource" LBUG.

It is unclear if the lock reached the client in the original reply,
just leave the lock on server, not returning it again to the client.
Due to LU-6529, the server will not OOM in case the 2nd reply will
be handled on the client.

Signed-off-by: Vitaly Fertman <vitaly.fertman@seagate.com>
Change-Id: I128cd6eeda579c6477bf4564db5e551a46a74d71
Reviewed-on: http://es-gerrit.xyus.xyratex.com:8080/8849
Tested-by: Jenkins
Seagate-bug-id: MRP-3042
Reviewed-by: Alexander Nikolaevich Boyko <alexander.boyko@seagate.com>
Reviewed-by: Andriy Skulysh <andriy.skulysh@seagate.com>
Reviewed-by: Alexey Leonidovich Lyashkov <alexey.lyashkov@seagate.com>
Tested-by: Elena V. Gryaznova <elena.gryaznova@seagate.com>
Reviewed-on: http://review.whamcloud.com/17501
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alexander Boyko <alexander.boyko@seagate.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7531 osp: allow few requests during recovery

Allow OSP requests during recovery, so recovery
threads will not be blocked if the remote target
is also in recovery status, otherwise it might
cause deadlock.

Add replay-single.sh 117 to verify this case.

Signed-off-by: Di Wang <di.wang@intel.com>
Change-Id: Iad3b6fd382d76c9bc042096c51cfac0a0d33091d
Reviewed-on: http://review.whamcloud.com/17539
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7192 tests: conf-sanity test 32c supports DNE on ZFS

1. ZFS needs import pool before tunefs.
2. previous 2.4 ZFS disk image contains 100M file, which will cause
   'migrate' used up all OST space and fail, recreate the image.
3. generate test images for 2.7-formatted ldiskfs and zfs with
   DNE striped directories feature enabled and update t32_test()
   to test that.

Test-Parameters: alwaysuploadlogs envdefinitions=SLOW=yes,ENABLE_QUOTA=yes,ONLY=32 mdtfilesystemtype=zfs mdsfilesystemtype=zfs ostfilesystemtype=zfs mdtcount=4 testlist=conf-sanity
Change-Id: Ib35e4cb80d6beee9695b26844c4e6a99e88adb89
Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Signed-off-by: Jian Yu <jian.yu@intel.com>
Reviewed-on: http://review.whamcloud.com/16663
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>

LU-5710 all: fourth set of corrected typos and grammar errors

All the typos are in comments or logs.

Signed-off-by: frank zago <fzago@cray.com>
Change-Id: If7973c31e73759f7e16c6f8d6a93c6453de8cb2b
Reviewed-on: http://review.whamcloud.com/17306
Tested-by: Jenkins
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7274 gss_do_ctx_fini_rpc() passes an already freed request

ptlrpc_request_free(req) calls either __ptlrpc_free_req_to_pool()
OR ptlrpc_request_cache_free() depending on request from pool /
memory.

ptlrpc_req_finished()-> __ptlrpc_req_finished()->__ptlrpc_free_req()
calls either __ptlrpc_free_req_to_pool() OR ptlrpc_request_cache_free()
too, but depending on reference and request count.

Thus its safe to removed the call to ptlrpc_request_free(req).

Change-Id: Ie2dd647af6b07d2810c458707b791b71c92bc3df
Signed-off-by: akam kumar bharathi <azurelustre@gmail.com>
Reviewed-on: http://review.whamcloud.com/17574
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Jeremy Filizetti <jeremy.filizetti@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6866 hsm: prevent migration of HSM archived files

The reference copytool cannot handle migration of HSM archive
files. In the MDT migration path check for HSM attributes and fail if
they are present. In the LMV layer allow creation of volatile files
with any MDT index. Add a test to sanity-hsm to ensure that attempting
to migrate an HSM archive file is handled safely.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ia133bfb345ce17473f54bd9a0ea0fbff5e5a5c58
Reviewed-on: http://review.whamcloud.com/17511
Tested-by: Jenkins
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6662 utils: allow overriding default mountopts

during formatting the device for Lustre, there are two kinds of
mount options to be used by "mkfs.lustre", which is default and
wanted options. the wanted options is mandatory but the default
should only be used if user doesn't specify it.

Signed-off-by: Hongchao Zhang <hongchao.zhang@intel.com>
Change-Id: I7d8093082455e0a08eb40df63dbc938d6dd8a2a6
Reviewed-on: http://review.whamcloud.com/15870
Tested-by: Jenkins
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-6895 lfsck: drop bad OI files after MDT file-level restore

For a new formatted Lustre system, it does not guarantee that all
the on-disk inodes/blocks are initialized. Instead, they are marked
as free in related inode/block bitmaps. When the inode is allocated
to some object, it will be initialized at that time. Such process
accelerates the format. But it may cause trouble for MDT file-level
backup/restore. For example: the sysadmin backup the MDT via server
side file level tar, then reformat the MDT device, and then restore
the MDT from the backup. Assume some object_A, its FID is the FID_A,
and it was mapped to the inode_B before the backup, such OI mapping
was recorded in the OI file. After the restore, another inode_C is
assigned to the object_A, but before OI scrub rebuilding related
OI mapping, the stale mapping "FID_A => inode_B" is still in the
OI file, and can be found by OI lookup. Generally, even if someone
found the stale OI mapping, it is not trouble, because the OSD will
verify whether FID-in-LMA for the indoe_B matches the FID_A or not.
But if the inode_B is NOT allocated after the restore, and because
we did not initialize inode_B during reformat, then the FID-in-LMA
for the indoe_B is still kept as the same before the backup, means
it matches the FID_A, then the OSD will think that the inode_B is
still assigned to the object_A after the restore. That is wrong.
In futher, although the inode_B is not allocated, but some of the
blocks that were assigned to it may have been allocated to others.
Then accessing the blocks via the inode_B may access some invalid
data, and may trigger some assertion, such as this issue.

So the key issues are two:

1) Some FID based operation may access stale OI mapping after MDT
   file-level backup/restore.

2) The OSD-ldiskfs may get some non-allocated inode with the give
   ino#/generation.

So as long as we can resovle one of them, then the trouble in the
ticket can be resolved.

The solutions for 1):

1.1) Avoid FID based operation before OI scrub done. That is not
     easy, because fid2path cannot be covered by some name based
     operation.

1.2) Remove the OI files after MDT file-level backup/restore. It
     is more direct solution. Another benefit is that even if OI
     scrub rebuilt the OI files, it only guarantees that all the
     FIDs' OI mappings have been refrshed. But it does not clean
     the stale FIDs' OI mappings. Because the OI scrub only does
     inode-table based scanning, not OI files scanning. Removing
     the OI files can resolve related trouble completely.

The solutions for 2):

2.1) New ldiskfs patch to make ldiskfs_iget() to return "-ENOENT"
     for the case of loading non-allocated inode by checking the
     inode bitmap.

2.2) Check the inode's valid inside OSD via related inode bitmap.

Generally, less ldiskfs patches is better. It will safe a lot of
effort when kernel upgrade. So 2.1) is not the best solution. As
for 2.2), it is not good to access the inode bitmap directly in
OSD unless we have to.

Relatively, the solution 1.2) is more efficient and benefit. That
is the current patch does.

On the other hand, this patch also makes some effort for the 1.1):

When start LFSCK, to hold the potential orphans in subsequent system
check, the start process will verify the .lustre/lost+found/ and its
sub-directories firstly. For each MDT, there is one sub-dir named as
.lustre/lost+found/MDTxxxx, which FID is recorded in the LFSCK trace
bookmark file. Originally, the start process uses such FID to locate
the MDTxxxx object. It is not problem for most of cases, but if the
MDT is just restored from file-level backup, and the low layer (OSD)
OI mapping may be invalid, then locating the MDTxxxx object will hit
-EREMCHG failure, and then may cause more troubles:

1) It will try to start OI scrub because of bad OI mapping detected,
   but because we are already in starting the LFSCK, then the logic
   of triggering OI scrub because of LFSCK is confused and difficult
   to be handled.

2) It will misguide the up layer LFSCK to think that the MDTxxxx obj
   does not exist or crashed, then will take some unexpected actions
   to repair such "fake" inconsistency.

The patch make some adjustment of the LFSCK start:

a) Trigger low layer OI scrub before verifying .lustre/lost+found.

b) To verify the .lustre/lost+found/ and its sub-dirs, scanning the
   .lustre/lost+found/ directory via namespace dt_lookup, and check
   the returned FID with the FID in the LFSCK trace bookmark file.
   With such way, even though the MDT is restored from file-level
   backup, the low layer still can locate the right object (inode).

Test-Parameters: alwaysuploadlogs envdefinitions=SLOW=yes,ENABLE_QUOTA=yes mdtfilesystemtype=ldiskfs mdsfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs clientdistro=el7 ossdistro=el7 mdsdistro=el7 mdtcount=1 testlist=sanity-lfsck,sanity-lfsck,sanity-lfsck
Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Icfdab19e69b75400bb39279924c0c3e0b400b4e0
Reviewed-on: http://review.whamcloud.com/17403
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-7430 mdt: better handle MDT recovery error path

This fix ensures the lu_target structure is still valid before
updating export related data in tgt_client_free().

In case of recovery error, it may happen that lu_target and mdt_device
structures are freed before exports are destroyed.

Signed-off-by: Gregoire Pichon <gregoire.pichon@bull.net>
Change-Id: I0559d42f40d4ecc9078717aa07b988be56ce3bc5
Reviewed-on: http://review.whamcloud.com/17424
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7465 llite: Do not track memory leak for kernel function

Do not use OBD_FREE to track memory leak if memory
allocation is from kernel function iov_iter_get_pages_alloc().

Signed-off-by: Wally Wang <wang@cray.com>
Change-Id: If723123bffceb46e2a3b3738043e54c62b8c96cd
Reviewed-on: http://review.whamcloud.com/17342
Reviewed-by: Frank Zago <fzago@cray.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7394 obd: remove HAVE_SELINUX_IS_ENABLED

In obd_mount_server.c unconditionally include linux/selinux.h. Include
linux/selinux.h in xattr_security.c.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Iea7a64d4e66d4c78494da40642111085b95700f9
Reviewed-on: http://review.whamcloud.com/17054
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5030 utils: add -R parameter to lctl get_param

To allow printing all parameters under a specified directory.
This is needed to replace hard-coded /proc pathnames in sanity-sec.sh
test_24, but would also be useful for normal usage.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Signed-off-by: Emoly Liu <emoly.liu@intel.com>
Change-Id: I4391afce06fd63a87f556f7b95bd0cb2883ebbe5
Reviewed-on: http://review.whamcloud.com/17081
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Frank Zago <fzago@cray.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

Revert "LU-6910 osp: add procfs values for OST reserved size"

This is causing LU-7550 and LU-7552 test failures in sanity.

This reverts commit 0585b0fb5895a24f07ca32e830d1fa72b75f4f2b.

Change-Id: Ic332a54ace4998acc4ba2ceab6f76ef733f85be5
Reviewed-on: http://review.whamcloud.com/17617
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-7381 e2fsck: update recommended e2fsprogs version

Update the recommended e2fsprogs version to 1.42.13.wc4.
This includes a number of important fixes to e2fsck, which
fix corruption problems in a number of cases:

   LU-7381 e2fsck: fix e2fsck -fD directory truncation
   - http://review.whamcloud.com/17153
   LU-7267 e2fsck: remove duplicated ea value size check
   - http://review.whamcloud.com/16779
   LU-7368 e2fsck: skip quota update when interrupted
   - http://review.whamcloud.com/17150
   LU-7267 e2fsck: remove duplicated ea value size check
   - http://review.whamcloud.com/16779/
   LU-6722 jbd: double minimum journal size for RHEL7
   - http://review.whamcloud.com/15401

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I1183394eec7717af1a163dbe7e4d93aa453ebbe5
Reviewed-on: http://review.whamcloud.com/17572
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7053 osd: don't lookup object at insert

the idea is to cache FID->ino/type mapping in per-thread cache
at declaration/object creation. then insert can find that information
and don't lookup object in LU/OI. this should avoid potential deadlock
with lu_object_find() and iget(). also, this should improve performance
as in the majority of cases required data is filled locally by create.

stats collected for sanity-benchmark:
lustre-MDT0000: 448306 created, lookups: 8910 in OI, 8910 in FLD
meaning we have to lookup ino 448K times and only 9K times we had
to use OI, in 439K cases we found ino in the cache.

Change-Id: Ifa66c2d074f04e47d0d85b735f57dc506aa65f4c
Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-on: http://review.whamcloud.com/17092
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7408 target: declare write for reply data

declare reply_data at max possible offset - this ensures
enough credits reserved.

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Change-Id: I08466452e1e95b803f316abae777a8c8f4a8626e
Reviewed-on: http://review.whamcloud.com/17086
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7144 tests: skip scrub/lfsck test under interoperation

Since the scrub/lfsck test are only for server side logic,
it is unnecessary to test scrub/lfsck under interoperation
mode, skip them.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I044030b3bace787809d7cfd5622000b44a8be789
Reviewed-on: http://review.whamcloud.com/17520
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7450 osd: call commit_callback if no write updates

If it does not need write updates in some failure cases,
top_trans_stop should also call commit_callback to help
release the top_thandle in the commit list. Otherwise
it will stay in the commit list forever, as well as the
following top thandle, then update logs will be culmulated,
and cause long time recovery.

Signed-off-by: Di Wang <di.wang@intel.com>
Change-Id: I1feaf0bd6d20f14dfabb4572f49818083e697dbb
Reviewed-on: http://review.whamcloud.com/17268
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7209 doc: more accurate documentation for obdfilter-survey

Make the the description of obdfilter-survey accurate and
precise.

Signed-off-by: Richard Henwood <richard.henwood@intel.com>
Change-Id: Icdd4adf53643e91dc8a2539f63977aae5fe28fe0
Reviewed-on: http://review.whamcloud.com/16646
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Omkar Kulkarni <omkar.kulkarni@intel.com>
Tested-by: Omkar Kulkarni <omkar.kulkarni@intel.com>
Reviewed-by: Cliff White <cliff.white@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7419 llog: lock new llog object creation

Lock the new llog object creation to avoid two
process create the same object at the same time.

Signed-off-by: Di Wang <di.wang@intel.com>
Change-Id: Icdc0eec534ca2f15cd0e195df951416953195346
Reviewed-on: http://review.whamcloud.com/17132
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7475 lnet: ensure buffer config symmetry

When showing the configuration, make sure to add a buffers block
in the YAML output, if routing is configured, in order to allow
the same YAML block to be fed back for configuration to LNet.

Signed-off-by: Amir Shehata <amir.shehata@intel.com>
Change-Id: I3b269edf5b3688b500bbb3656c367bc82fff6b68
Reviewed-on: http://review.whamcloud.com/17370
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6020 gss: properly map buffers to sg

A lot of buffer pointers passed to buf_to_sg() as input are coming
from vmalloc(), e.g. OBD_ALLOC_LARGE() in ptlrpc_add_rqs_to_pool().
sg_set_buf() uses virt_to_page() to map virtual addresses to
struct page, which does not work for vmalloc addresses.

The original code for buf_to_sg() caused the following crash:

BUG: unable to handle kernel paging request at ffffeb040057c040
IP: [<ffffffff81300367>] scatterwalk_pagedone+0x27/0x70
PGD 0
Oops: 0000 [#1] SMP
CPU 1
Pid: 2374, comm: ptlrpcd_3 Tainted: G           O 3.6.10-030610-generic
RIP: 0010:[<ffffffff81300367>]  [<ffffffff81300367>] scatterwalk_pagedone+0x27/0x70
RSP: 0018:ffff8801a3c178a8  EFLAGS: 00010282
RAX: ffffeb040057c040 RBX: ffff8801a3c17938 RCX: ffffeb040057c040
RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8801a3c17970
RBP: ffff8801a3c178a8 R08: 00000000000005a8 R09: ffff8801a3c17a40
R10: ffff8801a30370d0 R11: 0000000000000a68 R12: 0000000000000010
R13: ffff8801a3c17a08 R14: ffff8801a3c17970 R15: ffff88017d1c2c80
FS:  0000000000000000(0000) GS:ffff8801afa40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffeb040057c040 CR3: 0000000001c0c000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process ptlrpcd_3 (pid: 2374, threadinfo ffff8801a3c16000, task ffff8801a44e0000)
Stack:
 ffff8801a3c178b8 ffffffff813004bd ffff8801a3c17908 ffffffff8130303f
 ffff880100000000 ffffffff00000000 ffff8801a3c17908 ffff8801a3c17b18
 ffffc90015f015a8 0000000000000000 0000000000000010 0000000000000010
Call Trace:
 [<ffffffff813004bd>] scatterwalk_done+0x3d/0x50
 [<ffffffff8130303f>] blkcipher_walk_done+0x8f/0x230
 [<ffffffff8130a39f>] crypto_cbc_encrypt+0xff/0x190
 [<ffffffffa0688660>] ? aes_decrypt+0x80/0x80 [aesni_intel]
 [<ffffffffa0a1a1e4>] krb5_encrypt_bulk+0x164/0x5b0 [ptlrpc_gss]
 [<ffffffffa0a1a812>] gss_wrap_bulk_kerberos+0x1e2/0x490 [ptlrpc_gss]
 [<ffffffffa0a1600e>] lgss_wrap_bulk+0x2e/0x100 [ptlrpc_gss]
 [<ffffffffa0a0d98e>] gss_cli_ctx_wrap_bulk+0x44e/0x650 [ptlrpc_gss]
 [<ffffffffa0ab867c>] sptlrpc_cli_wrap_bulk+0x3c/0x70 [ptlrpc]
 [<ffffffffa0aba2d0>] sptlrpc_cli_wrap_request+0x60/0x360 [ptlrpc]
 [<ffffffffa0a8cde4>] ptl_send_rpc+0x164/0xc30 [ptlrpc]
 [<ffffffffa07be957>] ? libcfs_debug_msg+0x47/0x50 [libcfs]
 [<ffffffffa0a80ee0>] ptlrpc_send_new_req+0x3b0/0x940 [ptlrpc]
 [<ffffffffa0a86530>] ptlrpc_check_set+0x8e0/0x1d50 [ptlrpc]
 [<ffffffff816ac9f6>] ? schedule_timeout+0x146/0x260
 [<ffffffffa0ab0c9b>] ptlrpcd_check+0x4eb/0x5d0 [ptlrpc]
 [<ffffffffa0ab105f>] ptlrpcd+0x2df/0x420 [ptlrpc]
 [<ffffffff8108efa0>] ? try_to_wake_up+0x200/0x200
 [<ffffffffa0ab0d80>] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc]
 [<ffffffff8107c5f3>] kthread+0x93/0xa0
 [<ffffffff816b8d04>] kernel_thread_helper+0x4/0x10
 [<ffffffff8107c560>] ? flush_kthread_worker+0xb0/0xb0
 [<ffffffff816b8d00>] ? gs_change+0x13/0x13

Change-Id: I346d50568b65ed10da2762ca34562fc2858a05d8
Signed-off-by: Andrew Perepechko <andrew.perepechko@seagate.com>
Xyratex-bug-id: SNT-15
Reviewed-on: http://review.whamcloud.com/17319
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5690 mount: fix lmd_parse() to handle commas in expr_list

The lmd_parse() function parses mount options with comma as
delimiter without considering commas in expr_list as follows
is a valid LNET nid range syntax:

<expr_list>  :== '[' <range_expr> [ ',' <range_expr>] ']'

This patch fixes the above issue by using cfs_parse_nidlist()
to parse nid range list instead of using class_parse_nid_quiet()
to parse only one nid.

Signed-off-by: Jian Yu <jian.yu@intel.com>
Change-Id: I8ba6ee6eb31b4bb078a83d9db213cfca27b0fe66
Reviewed-on: http://review.whamcloud.com/17036
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6714 llog: test on-disk llog header values

llog_test_2():
- re-enable the disabled llog_open() test cases.
- Checks that llog_log_hdr values are written atomically
  with the llog record addition/cancelling.

Patch contains also minor fixes:
- verify_handle() does header sanity checks at first then
  checks amount of records against expected value.
- llog_test_3: rename test_3 static variables to show
  that they are related to the test_3.

Signed-off-by: Mikhail Pershin <mike.pershin@intel.com>
Change-Id: Iedcf15c8f365f9c2021abae3325edcaf08efc4c9
Reviewed-on: http://review.whamcloud.com/16287
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7030 security: put imp_sec after all requests drained off

imp_sec should be put after all requests being drained off.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I35f572fcc79b2bd1991db14577226a3ea735630d
Reviewed-on: http://review.whamcloud.com/16071
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Sebastien Buisson <sebastien.buisson@bull.net>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7530 mdt: Do not leak identity when no nodemap is present

It looks like sometimes nodemap structure on the export is not there
due to a race in old_init_ucred_common.
Move the nodemap check to the start not to leak identity reference
in such a case.

The bug was introduced in commit 2aea469a3a6e214d from LU-7199

Also silence the warning as there's nothing sysadmins could do when
it happens.

Change-Id: I5329ccb16201a71a263eb586e3a486b26ff238db
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/17519
Tested-by: Jenkins
Reviewed-by: Kit Westneat <kit.westneat@gmail.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-6910 osp: add procfs values for OST reserved size

osp_pre_status=-ENOSPC is used to skip OST from object allocation.
The error was set when OST available space is less than 0.1% of total
OST size. This value is not configurable, so procfs files was
added:
reserved_mb_low - low watermark, if available space is less
  than it, object allocation is stopped.
reserved_mb_high - highw watermark, if available space is more
   than it, object allocation is enabled.

By default ~0.1% is reserved as low watermark. The high watermark
is twice bigger than the low by default.
High and low watermark could be changed by:
lctl set_param osp.lustre-OST0000-osc-MDT0000.reserved_mb_high=1024

When object allocation is disabled, a clients could appened to
existing files. And 0.1% is too low for them. For example, OST size
is 8TB, 0.1% is 8GB, if cluster has 1k clients, reserved space is
~8MB per client. The main reason of the patch is ability to increase
reserved space.

Signed-off-by: Alexander Boyko <alexander.boyko@seagate.com>
Xyratex-bug-id: MRP-2606
Change-Id: Ie48cc1a232f64aa7dc922000861004277fb47340
Reviewed-on: http://review.whamcloud.com/15731
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@seagate.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7136 test: allow more time for copytools to stop

In sanity-hsm allow up to 200 seconds for the copytool to stop. This
is needed to prevent sporadic failures due to slow NFS (for the
copytool log file) delaying the termination of the copytool.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Icafc4e9c5a00c849dcb479233826de058d2ede62
Reviewed-on: http://review.whamcloud.com/17499
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6767 osd-zfs: Track readonly status of ZFS

Return READONLY from osd_statfs() if underlying ZFS has been set to
READONLY, or if osd_ro() has been called.  This adds a callback for
ZFS_PROP_READONLY for when it's changed.

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Ib7f35925904b1d93f9a457936585e9783635c849
Reviewed-on: http://review.whamcloud.com/15400
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6229 utils: fix lustre_rsync bug of cascade move

When replaying the changelog, destination files have to be
put into a special directory, if their parent directory is
possessing a different path other than the ultimate path
because of renaming. With the replaying process going on,
when the parent directory is being moved to the ultimate path,
the child files should be moved under the parent directory
which is called cascade move.

As long as a directory has child files under sepcial direcoty,
cascade move should happen, no matter the direcotry is being
renamed from sepcial direcoty or not. This patch fixes the problem
that cascade move is missing when the direcotry is being renamed
from ordinary path.

Signed-off-by: Li Xi <lixi@ddn.com>
Change-Id: I2d21604b81fe0cf08df1af2bfccc90a32986bf05
Reviewed-on: http://review.whamcloud.com/14914
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7515 obdclass: add export for lprocfs_stats_alloc_one()

 When compiling Lustre without optimization, when using GCOV,
 the lprocfs_stats_alloc_one() symbol is not properly exported to other modules
 and causes the ptlrpc module to fail loading with an unknown symbol.
 Added EXPORT_SYMBOL(lprocfs_stats_alloc_one) so that this works properly.

Seagate-bug-id: MRP-3188
Signed-off-by: Chennaiah Palla <chennaiah.palla@seagate.com>
Change-Id: I8ef02a0e0bf519fa93f85cb162a6340e3feeb736
Reviewed-on: http://review.whamcloud.com/17443
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3569 utils: remove ll_recover_lost_found_obj

remove obsolete tool ll_recover_lost_found_obj.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Change-Id: I5d5f33f5c9d68bb1f05d7ab0da6fb2986e873501
Reviewed-on: http://review.whamcloud.com/16477
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7446 clio: lov_io_init() should return error code

lov_io_init_empty/release() should returns error code instead of
true on error case.

Fault IO need handle restart in the case of accessing HSM released
file.

Add a test case.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I4953c12c1e9b82a16aed9b8b1e3fe6e38d783b24
Reviewed-on: http://review.whamcloud.com/17240
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6732 llite: ll_write_begin/end not passing on errors

Because of a implementation of generic_perform_write(), write(2)
may return 0 with no errno even if EDQUOT or ENOSPC actually
happend in it.
This patch fixes the issue with setting a proper errno to
ci_result and get it in ll_file_io_generic.

Signed-off-by: Hiroya Nozaki <nozaki.hiroya@jp.fujitsu.com>
Change-Id: I3fc986b57d703ad5fbf41e1ea8182d2d561e8005
Reviewed-on: http://review.whamcloud.com/15302
Tested-by: Maloo <hpdd-maloo@intel.com>
Tested-by: Jenkins
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1606 misc: clean up DFID related error messages

Improve the error messages related to DFID output and parsing left
over from removal of LPU64/LPX64 usage in userspace.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I4b4fcb3cc389b8d8ec4375fa92bfee9b353ebbe5
Reviewed-on: http://review.whamcloud.com/6156
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Frank Zago <fzago@cray.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2524 test: Clean up sanity-quota

Conduct miscellaneous cleanup to sanity-quota including:
Removing the `-p` (parents) option from many calls to mkdir
Replace `lfs setstripe` with $SETSTRIPE
Added check for and call to `error` with error messages for a variety
of common routines ,like mkdir, or for functions that return a value.
Replace `…` with $(...)
Removed linefeed escape after |, ||, & and && operators.
Modified parameters in test 4b so that the --inode-grace value exceeds
the valid range.
Removed unused variables
Removed $ from variables inside $(())

Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: Iadedea0bd0a0f85235e0bb908ee0a6ed36503eb3
Reviewed-on: http://review.whamcloud.com/14680
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Wei Liu <wei3.liu@intel.com>
Reviewed-by: Saurabh Tandan <saurabh.tandan@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7148 osc: Remove remains of osc_ast_guard

osc_ast_guard has been removed by the clio simplification.
Remove the extern declaartion and lock class definition.

Change-Id: Ibcf14e7aebe1dab8b586d3cd8d81560f6d3dcc81
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/16392
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>

LU-5951 ptlrpc: track unreplied requests

The request xid was used to make sure the ost object timestamps
being updated by the out of order setattr/punch/write requests
properly. However, this mechanism is broken by the multiple rcvd
slot feature, where we deferred the xid assignment from request
packing to request sending.

This patch moved back the xid assignment to request packing, and
the manner of finding lowest unreplied xid is changed from scan
sending & delay list to scan a unreplied requests list.

This patch also skipped packing the known replied XID in connect
and disconnect request, so that we can make sure the known replied
XID is increased only on both server & client side.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: Ic98e1599085871c0ac08d28609a044c79d5af75d
Reviewed-on: http://review.whamcloud.com/16759
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Grégoire Pichon <gregoire.pichon@bull.net>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7508 ldlm: Don't check opcode with NULL rq_reqmsg

When GSS is enabled it's possible to have a NULL rq_reqmsg
if a bad signature or no context is returned during the unwrap
of the request.  Don't check the opcode in this case.

Signed-off-by: Jeremy Filizetti <jeremy.filizetti@gmail.com>
Change-Id: I3a74dff7638b318190c5c4ad73acbe7ec299aa80
Reviewed-on: http://review.whamcloud.com/17414
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-7268 scrub: NOT assign LMA for EA inode

Originally, when OI scrub scans the device, if the target inode has
no FID-in-LMA EA, then it will generate an IGIF mode FID and store
it in the LMA EA. Such behavior is not suitable if the target inode
is used for large EA. The OI scrub should skip the EA inode that is
marked as "LDISKFS_EA_INODE_FL".

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I52b05b864ef8a2797a2f3dda0f80f95227809c34
Reviewed-on: http://review.whamcloud.com/17043
Tested-by: Jenkins
Reviewed-by: Kalpak Shah <kalpak.shah@seagate.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-6298 hsm: shutdown HSM CDTs in parallel

In sanity-hsm.sh rewrite copytool_cleanup() to shutdown and restart
the MDT HSM coordinators in parallel. This saves about 8 * (MDSCOUNT -
1) seconds per call.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I75445ad126dc73251a3d056611133e3ab6b83362
Reviewed-on: http://review.whamcloud.com/13901
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5921 tests: enhance server target mount race testing

This patch is a follow on to LU-5299 to strengthen and enhance
concurrent server target mount race testing.
It uses OBD_RACE() feature to better set a concurrent/racy
situation, and also allow to handle all mount errors instead
of only EALREADY.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: I16a94e5aa046e15096d2e55d57e22899a93fa03f
Reviewed-on: http://review.whamcloud.com/17302
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7383 mdt: retry for busy lock during migration

In migration, if the lock of the migrating object
is being cached on other node, it should revoke
the lock and retry, instead of return -EBUSY.

Signed-off-by: Di Wang <di.wang@intel.com>
Change-Id: I1317681a892b9a21f2c78d7696ca6f94d43bd9bc
Reviewed-on: http://review.whamcloud.com/17048
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

New tag 2.7.64

Change-Id: I79fb95af8bc9e979edf3214315219f786eb12599
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7428 test: disable conf-sanity, test_84

Add failing test to ALWAYS_EXCEPT.
This is a temprorary workaround until a real
fix for the test failure is developed.

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I73e00d658a9e7728ce52b5dc90741e9a18ce15f9
Reviewed-on: http://review.whamcloud.com/17482
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-7437 lctl: list_param -R can't work correctly

We shouldn't call lprocfs_param_pattern() inside listparam_display(),
otherwise it will add the prefix "/proc/{fs,sys}/{lnet,lustre}" each
time, so that the parameters can be listed recursively. The similar
issue in {set/get}param is fixed as well.

Also, this patch adds sanity.sh test_401 to verify this function.

Signed-off-by: Emoly Liu <emoly.liu@intel.com>
Change-Id: I21acec364cdbdfc025979153f66a87d44c9136e8
Reviewed-on: http://review.whamcloud.com/17223
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7297 osd-zfs: initialize oh_lock

The ZFS osd was not initializing od_brw_stats.hist[].oh_lock.
This rectifies that.

Change-Id: I3f637b73c77908c2297bfab97e33eca63b0d5986
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Reviewed-on: http://review.whamcloud.com/16919
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>

LU-1026 ldiskfs: make bitmaps corruption not fatal

We still hit bitmaps problems for rhel6 series kernel,
corruptions happen because ext4_mb_check_ondisk_bitmap()
check failed and FS become RO again:

ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group
294corrupted: 20180 blocks free in bitmap, 20181 - in gd
Aborting journal on device dm-6-8.
LDISKFS-fs (dm-6): Remounting filesystem read-only
ldiskfs_mb_new_blocks: Updating bitmap error: [err -30]
 [pa ffff880d9d6e4d68] [phy 14974976] [logic 8192] [len 3072]
 [free 3072] [error 1] [inode 278678]
ldiskfs_ext_new_extent_cb: Journal has aborted

this might be caused by some ext4 internal bugs, this patch
did the following things:

1.Inside ext4_read_block_bitmap() have gaven reasons
why it failed, so caller don't need call ext4_error() again.
2. mark block group corrupt and use ext4_warning() instead
of ext4_error().

There are still some bitmaps corruption places not handling,
let's keep it for now, and if it really hurt, let's add the
same handling codes logic later.

Tested by following scripts:

TEST_DEV="/dev/sdb"
TEST_MNT="/mnt/ext4"

mkdir -p $TEST_MNT
mkfs.ext4 -F $TEST_DEV >&/dev/null

mount -t ldiskfs $TEST_DEV $TEST_MNT
dd if=/dev/zero of=$TEST_MNT/largefile
oflag=direct bs=10485760 count=200
umount $TEST_MNT
dd if=/dev/zero of=$TEST_DEV bs=4096 seek=641
count=10 oflag=direct
mount -t ldiskfs $TEST_DEV $TEST_MNT
rm -f $TEST_MNT/largefile
dd if=/dev/zero of=$TEST_MNT/largefile oflag=direct
bs=10485760 count=200 && echo
  "FILESYSTEM still usable after bitmaps corrupts happen"
dmesg | tail
umount $TEST_MNT
e2fsck $TEST_DEV -y

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: Iabb6ebf719d80d9ba4f41bee0b237e304212832b
Reviewed-on: http://review.whamcloud.com/16679
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7315 osd-ldiskfs: handle pdo lock properly

Inside the osd_dirent_check_repair(), if the logic comes to
"goto again", it only unlock the "hlock" but without seting
the variable @hlock as NULL. Althouth it will not cause any
logic failure, it may make the readers to be confused. This
patch will set "hlock = NULL;" explicitly to avoid trouble.

On the other hand, inside ldiskfs, the pdo lock users need
to check whether the lock handler is NULL or not properly.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I9db9dc758a2976849c299f76e06723e796da235d
Reviewed-on: http://review.whamcloud.com/16924
Tested-by: Jenkins
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6856 zfs: handle non existing file in osd_object_ref_del

Remove false assertion in zfs:osd_object_ref_del() because this
may be in the cleanning path of error handling.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: Ib7b9d80816bdab7f68b36a33e95140ea7f3eae8c
Reviewed-on: http://review.whamcloud.com/15611
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>

LU-6802 ptlrpc: reset imp_replay_cursor

At client side, the replay cursor using to speed up the lookup
of committed open requests in its obd_import should be resetted
for normal connection (not reconnection) during recovery.

Change-Id: I68816780a5d79053d9109cb68ae1c3b8ea13ede8
Signed-off-by: Hongchao Zhang <hongchao.zhang@intel.com>
Reviewed-on: http://review.whamcloud.com/17351
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6693 out: not return NULL in object_update_param_get

Return ERR_PTR in object_update_param_get() for all cases to
avoid unnecessary confusion to callers.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: Idfcc19d99bbf308759481b3d60d95341745d19e8
Reviewed-on: http://review.whamcloud.com/16417
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7316 build: Update ZFS/SPL version to 0.6.5.3

Bug Fixes

* Fix CPU hotplug zfsonlinux/spl#482
* Disable dynamic taskqs by default to avoid deadlock
  zfsonlinux/spl#484
* Don't import all visible pools in zfs-import init script
  zfsonlinux/zfs#3777
* Fix use-after-free in vdev_disk_physio_completion
  zfsonlinux/zfs#3920
* Fix avl_is_empty(&dn->dn_dbufs) assertion zfsonlinux/zfs#3865

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I36347630be2506bee4ff0a05f1b236ba2ba7a0ae
Reviewed-on: http://review.whamcloud.com/16877
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-4423 lnet: don't use iovec instead of kvec

Replace struct iovec with struct kvec.

Linux commit: f351bad2b4b4bb74810ad4f127f6602e2d2ae403

Change-Id: Ib7bb49069e42ca82d66a149617361c73ee4d710d
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-on: http://review.whamcloud.com/17205
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Frank Zago <fzago@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7462 mdd: check object existence

Check object existence in mdd_is_parent()
and _mdd_lookup(), so the following retrieving
attributes will not panic if object does not
exist.

Signed-off-by: Di Wang <di.wang@intel.com>
Change-Id: Icae243949e202f9d6a4d38b9823373101a093c74
Reviewed-on: http://review.whamcloud.com/17323
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Andreas Dilger <andreas.dilger@intel.com>

LU-7375 lbuild: add missing case to lbuild

Add the missing case of el6.7 to autodetect_target

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I6a8032eb03b2fb3002798aa49ee585879cf711c3
Reviewed-on: http://review.whamcloud.com/17024
Tested-by: Jenkins
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7103 test: avoid cat of /dev/urandom

Use a different command to generate a random number.
Using cat /dev/urandom doesn't work in all cases.

Test-Parameters: clientdistro=el7
testlist=ost-pools,ost-pools,ost-pools,ost-pools,ost-pools,ost-pools,ost-pools,
ost-pools,ost-pools,ost-pools,ost-pools,ost-pools,ost-pools,ost-pools

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I63cfad2654cf654460263529457538965e373f82
Reviewed-on: http://review.whamcloud.com/17350
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7447 lfsck: correct nlink attr for new created dir

For new created directory object, there are three main operations:
1) insert dot entry
2) insert dotdot entry
3) ref_add on the directory object.

Sometimes, the 3rd step maybe ahead of 1st, sometimes maybe between
1st and 2nd. Usually, the developers would think that such order is
not important. But in fact, such assumption isn't true, because the
ldiskfs will set the new created directory object's nlink attr as 2
when inserting dotdot entry. Then if ref_add() is called after ".."
inserted, the empty directory object's nlink attr is 3.

To avoid above trouble and make the OSD API to be order-independent,
we will make the osd-ldiskfs to backup the new created dir object's
nlink attr before calling ldiskfs_add_dot_dotdot(), then restore it
after ldiskfs_add_dot_dotdot() called.

On the other hand, this patch also adds some missed logic for lfsck
declaring to insert dot/dotdot entries, remove some redundant logic.

Add e2fsck check for on-disk consistency verification after LFSCK.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I065f683388417f3e23d47471c065419bd9bfcb19
Reviewed-on: http://review.whamcloud.com/17269
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-7371 test: wrong read length over isize

This patch adds tests to check read length is correct if reading
a file of size 4095.

Signed-off-by: Li Xi <lixi@ddn.com>
Change-Id: I0c0f6b378fa4af053ed54f2f5dea2418191a7b69
Reviewed-on: http://review.whamcloud.com/17060
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-6851 lnet: Ignore hops if not explicitly set

Since the # of hops is not a mandatory parameter the LU-6060
patch will cause problems to already existing systems since it
changes the behavior by which a route is determined down.

To fix this case the # of hops now defaults to LNET_UNDEFINED_HOPS
if no hop count is specified.

LNET_UNDEFINED_HOPS is defined to ((__u32)-1). When it's printed as
%d, it displays as -1.

__u32 is used through out the call stack for hop count to explicitly
define the size of the hop count and to avoid any sizing issues when
passing data to and from the kernel.

To keep existing behavior both lnet_compare_routes() and LNetDist()
will treat undefined hop count as hop count 1.

When executing the logic in lnet_parse_rc_info() there is no
longer an assumption that the default hop count is 1. If
the hop count is 1 then it must've been explicitly set by
the user.

Signed-off-by: Amir Shehata <amir.shehata@intel.com>
Change-Id: I1a28a35a4edc2437cf95cb9d455e59c8102736fa
Reviewed-on: http://review.whamcloud.com/15719
Tested-by: Jenkins
Reviewed-by: Olaf Weber <olaf@sgi.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7400 lod: register stop callbacks at create

In some cases we stop just created transaction (i.e. it's
empty, not started), but then top_trans_stop() waits
indefinitely for stop callbacks which were supposed to be
register at top_trans_start(). instead register them at
top_trans_create().

Instead of register the commit callback in top_trans_start(),
we should register the commit callback once the top thandle
is added into commit list, because in some error cases,
top_trans_start() might not be called, then the top thandle
will stay in the commit list forever, and also blocking
llog cancellation.

Change-Id: I1ca528e2f8f5e4d9cf6b5dd484653a055e17cc6c
Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Signed-off-by: wang di <di.wang@intel.com>
Reviewed-on: http://review.whamcloud.com/17059
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7199 nodemap: assign nodemap to export before connecting

This patch moves the nodemap assignment to be before the connection
is active, and the nodemap deassignemnt to be after the connection
is made inactive. It also checks for null nodemaps and returns
-EACCES if the nodemap is not set in order to avoid a kernel panic.

Signed-off-by: Kit Westneat <kit.westneat@gmail.com>
Change-Id: Id4dd02cf5b4208bda6d6f17aed6adb13b18c7731
Reviewed-on: http://review.whamcloud.com/16802
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7428 tests: write superblock in conf-sanity test_84

In conf-sanity.sh test_84() a newly-formatted MDS filesystem's block
device is set read-only immediately after mount via replay_barrier(),
which may result in initial formatting or configs to be discarded,
resulting in a wide variety of different failure modes for this test.

Ensure that the superblock and other configuration logs are flushed
to disk before replay_barrier() is called, so that the MDS can mount
properly again later in the test.

Test-Parameters: testlist=conf-sanity,conf-sanity,conf-sanity,conf-sanity
Test-Parameters: testlist=conf-sanity,conf-sanity,conf-sanity,conf-sanity
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I95894b600ed7596c2014cba4f35fef4b443ebbe5
Reviewed-on: http://review.whamcloud.com/17371
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7329 obdclass: sync device to flush journal callbacks

After llog_test_10() has been added to check correct Catalog
wrap-around as part of LU-6556, frequent auto-tests failures
have been encountered due to OOM situation on VMs caused by
a lot of journal commit callbacks backlog induced by the huge
LLOG activity generated by new test.
This patch forces frequent device sync to flush these journal
commit callbacks.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: If06914a395f5bacbbb55fcdc229ffcd47d742843
Reviewed-on: http://review.whamcloud.com/17052
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7098 osd-ldiskfs: don't alloc inode directly

We should alloc ldiskfs_inode_info instead alloc inode
directly. Else will overflow in follow ldiskfs_add_entry.

Signed-off-by: Yang Sheng <yang.sheng@intel.com>
Change-Id: Ib88f847ea8f160f6f0d0c7e7e680247d7ec96fb6
Reviewed-on: http://review.whamcloud.com/16804
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Frank Zago <fzago@cray.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>

LU-1095 mdc: remove console spew from mdc_ioc_fid2path

In some cases with a very long pathname, such as with sanity.sh
test_154c, mdc_ioc_fid2path() would spew long debug messages to
the log, because libcfs_debug_vmsg2() refuses to log messages over
one page in size.

Truncate the debug message to only log the last 512 characters
of the pathname, which is sufficient for most debugging, saves a
bit of space in the debug log, and will prevent the debug logging
from printing to the console in the first place.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I3cad19d02e8065574b8baf5694e9894e43112318
Reviewed-on: http://review.whamcloud.com/17078
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7396 llite: check request != NULL in ll_migrate

Check if the request is NULL, before retrieve reply body
from the request.

Signed-off-by: Di Wang <di.wang@intel.com>
Change-Id: Ifec9caf270b938b7583de0315610f930fa52649d
Reviewed-on: http://review.whamcloud.com/17079
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-7276 utils: make llog_reader consistent with kernel

Handle the CM_SKIP records in the same manner as the kernel,
by printing "SKIP" only until the first CM_END record, rather
until the first CM_END|CM_SKIP record.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: Ie0c35ecd9d74a1e0b378bcf6642bca32e4dfa35e
Reviewed-on: http://review.whamcloud.com/16788
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Frank Zago <fzago@cray.com>
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7068 mdd: object leak in mdd_migrate_entries

mdd_migrate_entries should release the object
if mdd_trans_start() fails.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: If10dce8b450e8c19bb93465beb08118e98c4ed96
Reviewed-on: http://review.whamcloud.com/16384
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>

LU-6666 osc: Do not merge extents with partial pages

After range lock is introduced to Lustre, it's possible for
multiple threads to submit osc_extents with partial pages, and
finally I/O engine may try to merge these extents, which will
end up with assert in osc_build_rpc().

In this patch, osc_extent::oe_no_merge is introduced, and this flag
is set if osc_extent submitted via osc_io_submit() includes partial
pages. This flag is used by I/O engine to stop merging this kind
of extents.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I0c851912beb0370553bc2e2b05f80dee175a2f1f
Reviewed-on: http://review.whamcloud.com/15468
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Patrick Farrell <paf@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7463 osd: Change existence assert into error

In osd_declare_xx(), some objects might not existent,
especially when calling out_handler(). Let's change
these assert into -ENOENT error to avoid panic on
MDS.

Signed-off-by: Di Wang <di.wang@intel.com>
Change-Id: If17a013a6939a1ebe0519406d39e405fd915110f
Reviewed-on: http://review.whamcloud.com/17324
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>

LU-7461 lod: retry to get remote update log

If the remote MDT is also in recovery status,
then retrieving update logs in lod_sub_recovery_thread()
might return -EAGAIN or -EIO or -EBUSY, let's
retry in this case until the recovery is aborted or
the local MDT is umounted.

Signed-off-by: Di Wang <di.wang@intel.com>
Change-Id: Iee945942bd01925cdcfe75c4e59dccbd63b34498
Reviewed-on: http://review.whamcloud.com/17322
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7416 osp: check rq_repmsg in osp_request_commit_cb

Check if rq_repmsg is NULL before retrieving
last committed transno from reply message.

Signed-off-by: Di Wang <di.wang@intel.com>
Change-Id: Ibf1e110e33df333934c65dfcf52870954e936180
Reviewed-on: http://review.whamcloud.com/17130
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7343 osd-ldiskfs: handle ldiskfs_append failure

In new linux kernel (linux-3.1x, x>=0), the ldiskfs exported
function ldiskfs_append() return error# via the return value,
instead of via the output parameter @err as it does on other
kernels (linux-2.6). Under such case, the caller should not
assume non-NULL returned value is valid buffer head, it can
stands for error#. So check that properly.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I4dca43bcfd31aafd999f54934a51d258071dab22
Reviewed-on: http://review.whamcloud.com/17148
Tested-by: Jenkins
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-7376 tests: sanity-hsm/59 should skip old servers.

there is no poin to crash vulnerable versions.

Change-Id: Iacafd10d2a3d04ba1bb9ca70d8e343809490a349
Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-on: http://review.whamcloud.com/17029
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7436 tests: skip conf-sanity/91 with old servers

due to missing functionality.

Change-Id: I2be72820413aee9a9d7082c2d8c6f308eeb6e141
Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-on: http://review.whamcloud.com/17222
Tested-by: Jenkins
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7415 kernel: kernel update RHEL 6.7 [2.6.32-573.8.1.el6]

Update RHEL6.7 kernel to 2.6.32-573.8.1.el6

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I1b3c4b144b06e5e96f818e35c08f490e574ed798
Reviewed-on: http://review.whamcloud.com/17119
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7164 osc: osc_extent should hold refcount to osc_object

To avoid a race that osc_extent and osc_object destroy happens on the
same time, which causes kernel crash.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I3e3237f0d1cff4bd992bef4e4c01355a1d5c8d9f
Reviewed-on: http://review.whamcloud.com/16433
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Patrick Farrell <paf@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7384 lfsck: check transaction stop status

The LFSCK modification will be sent to remote server when the
transaction stop, for sync transaction case, we can check the
dt_trans_stop() result.

If the lfsck_namespace_create_orphan_dir() failed, but we may
ignored that before because of ignoring dt_trans_stop result.
Then it may cause subsequent lfsck_namespace_insert_normal()
failed at LASSERT(lu_object_exists(o) != 0);

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: If897b7bd479ecdb61e6435f3177211f865a4e303
Reviewed-on: http://review.whamcloud.com/17042
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>