LU-6446 ldiskfs: remove WARN_ON from ldiskfs_orphan_add{del}

RHEL7.1 kernel carefully check i_mutex whether locked in
many places. It will be triggered while ldiskfs_truncate()
was invoked in osd layer. A dead lock would occured if we
just locked i_mutex around it since Lustre using a reverse
order for start journal==>lock i_mutex. Consider Lustre has
own ldlm lock. So just remove such messages in ldiskfs patches.

Signed-off-by: Yang Sheng <yang.sheng@intel.com>
Change-Id: I79520b0a1d013722a5a27e71318416608bc25285
Reviewed-on: http://review.whamcloud.com/14690
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6802 ptlrpc: reset replay cursor on reconnection

In this way, clients can make sure every single replayable ptlrpc
requests will be replayed and replies will be received.

Test-Parameters: envdefinitions=PTLDEBUG=+inode mdtfilesystemtype=ldiskfs mdsfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs clientdistro=el6.6 ossdistro=el6.6 mdsdistro=el6.6 mdscount=2 mdtcount=4 testlist=sanity,sanity,sanity,sanity,sanity,sanity
Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I32daf787d141c84376879fb0fb3b3eb8424f91ad
Reviewed-on: http://review.whamcloud.com/15669
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Hongchao Zhang <hongchao.zhang@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

New tag 2.7.57

Change-Id: I7802951fb411c6f40a2d7ee14ba0759c2036c227
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6872 lov: avoid infinite loop in lsm_alloc_plain()

In lsm_alloc_plain() use a signed loop index to avoid an infinite loop
in the error path.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I084bfadd8a6bf44bbcbe60624e31926ec6cdc04e
Reviewed-on: http://review.whamcloud.com/15644
Tested-by: Jenkins
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6790 build: handle external Intel OFED stack

Besides the Mellanox stack Intel also releases their own
OFED stack. Just like the Mellanox stack this Intel OFED
stack has a compatibility layer to support distributions
kernels. This layer provided stomps over the default
kernel headers so like the Mellanox stack compact-2.6.h
has to be placed before all other headers. The reason
it didn't show up before with the Mellanox stacks is the
Intel stack additionally stomps the pci layer of the
kernel.

Change-Id: If565f25432cf87723b69e5d65e00a7d5d301dc4f
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: http://review.whamcloud.com/15582
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Chris Horn <hornc@cray.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6835 build: add -Wall to CFLAGS for test/ and utils/

Add -Wall to the CFLAGS used for lustre/{test,utils}/. Fixup missing
includes where needed.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I19d0739e0a9b5b079665d5d24d54c6dcaad93b0c
Reviewed-on: http://review.whamcloud.com/15392
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Jenkins
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Chris Horn <hornc@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6501 libcfs: removal last entry of libcfs_netstrfns[]

Currently NID string handling test for the last entry,
and last entry has .nf_type == (__u32) -1. If we ask
for a non existent LND we hit the last entry which then
calls a strlen on a NULL which causes a error. We can
avoid this problem if we just remove the last entry
since it is not used for anything except as a last
entry marker.

Signed-off-by: Frederic Saunier <frederic.saunier@atos.net>
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Change-Id: Ia8e3bea0f01bb5fa78e88bfbca698b0aa0d148ea
Reviewed-on: http://review.whamcloud.com/15424
Tested-by: Jenkins
Reviewed-by: frank zago <fzago@cray.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6261 gnilnd: Clean up cfs abstractions from gnilnd

Running the script contrib/scripts/libcfs_cleanup.sed gets us ready
for cfs changes.

find lnet/klnds/gnilnd -name "*.[ch]" -exec \
sed -i -f contrib/scripts/libcfs_cleanup.sed {} ;

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: Ie7aa1616d33d9f60870f4b8c2a946ce66d3c348c
Reviewed-on: http://review.whamcloud.com/15437
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: James Shimek <jshimek@cray.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6261 gnilnd: Hold shared MDD for gnilnd.

Creating and destroying shared MDDs can cause OS noise. This affects
benchmarks that are sensitive to OS noise like fwq.
Allocate and register some memory so that we always have a shared MDD
available.
Moved kgnilnd_check_kgni_version() into gnilnd.c to simplify
gemini/aries headers.

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: Ibc1f4b9a44035c6fb25e88d30552136486c260d6
Reviewed-on: http://review.whamcloud.com/15436
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: James Shimek <jshimek@cray.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6261 gnilnd: Fix LIBCFS_ALLOC_POST incompatibility.

master has changed LIBCFS_ALLOC to use vzalloc so LIBCFS_ALLOC_POST
does not contain a memset anymore.
kgnilnd_vmalloc() used LIBCFS_ALLOC_POST to zero memory allocated but
master removed the memset in favor of using vzalloc.
Use the __GFP_ZERO flag in (now named) kgnilnd_vzalloc to zero memory
allocated.

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: Ia014c18f13e4e263eccb51d49d375fc4b3bc8b61
Reviewed-on: http://review.whamcloud.com/15435
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: James Shimek <jshimek@cray.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6261 gnilnd: Use trylock for conn mutex.

When converting to thread safe implementation, I missed the need for
the conn mutex to trylock because we may be holding the kgn_peer_conn
lock.
Change conn mutex to use a trylock in kgnilnd_sendmsg_trylock().
Add the module parameter thread_safe.
Disable thread safe on gemini.
In kgnilnd_create_conn(), use NOFS flag when vmalloc the tx_ref_table
to avoid possible hangs because of OOM condition.
Clean up info message for down event.

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I1896c12b421ae35f65d1816bbe3eb5599b664498
Reviewed-on: http://review.whamcloud.com/15434
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: James Shimek <jshimek@cray.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6261 gnilnd: Add pkey module parameter

lustre:19561 changed the pkey to the value reserved in gni_pub.h and
the max_immediate size. These need to be the same on both the service
and compute nodes for gnilnd to establish a connection.
Add the module parameter pkey so that a node running a dev version of
software will be able to connect to a node running an older version
with appropriate changes to modprobe.conf.
With this change, add the following parameters to
IMAGE_PATH/compute/etc/modprobe.conf:

options kgnilnd max_immediate=2048
options kgnilnd pkey=0xa3579

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I08ea2d66e745c37f41931a53e599bc572664dd9a
Reviewed-on: http://review.whamcloud.com/15433
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: James Shimek <jshimek@cray.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6261 gnilnd: Add ability to bind scheduler threads to cpus.

Added the module parameter thread_affinity which when enabled, will
bind the kgnilnd_sd_xx threads to cpus 1 through xx.
thread_affinity defaults to disabled.
Testing shows that enabling thread affinity will allow us to get
better small message performance with more scheduler threads.

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I5298c25cc21a1a856e4d84f1bca4f484449cd055
Reviewed-on: http://review.whamcloud.com/15432
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: James Shimek <jshimek@cray.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6679 ldlm: do not send blocking ast for group locks

Group locks are acquired and released manually on client so
it doesn't make sense to send blocking AST to client when
there is an incompatible lock enqueued.

Currently client will set CBPENDING on the group lock when
it receives a blocking AST. Having the CBPENDING flag set
will make ldlm_lock_match to fail and there will be two
group locks granted on the same resource on the client.

Signed-off-by: Li Dongyang <dongyang.li@anu.edu.au>
Change-Id: I7af89e957528b3ed9771d86243ac8271084ee81f
Reviewed-on: http://review.whamcloud.com/15119
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6847 kernel: kernel update RHEL 6.6 [2.6.32-504.30.3.el6]

Update RHEL6.6 kernel to 2.6.32-504.30.3.el6

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I4f60d43dea248ce57290579c021838d0f731c332
Reviewed-on: http://review.whamcloud.com/15605
Tested-by: Jenkins
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6186 tests: avoid errors in using >&- bash close syntax

Seems that recent EL7 bash version include changes in handling
the ">&-" redirection/close syntax, which cause "Bad file
descriptor" error. Use >/dev/null to avoid this.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: Iac3e54ef69f0c5c241b87a9e307e059974a4caf3
Reviewed-on: http://review.whamcloud.com/13857
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6822 nrs: remove obsolete assertion in nrs_orr_start()

kmem_cache_destroy() doesn't return any value, so we should remove
this obsolete assertion, otherwise LBUG will happen when orr policy
is enabled.

Signed-off-by: Emoly Liu <emoly.liu@intel.com>
Change-Id: I240e7bd660a6960b1e29da3575388a966ce8dca9
Reviewed-on: http://review.whamcloud.com/15540
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Hongchao Zhang <hongchao.zhang@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6540 test: fix no %s specifier for the file

The %m prints strerror(errno), but there is no %s specifier for the file.

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: Id9a8f486521b94e98986140fa374c40e08a00dac
Reviewed-on: http://review.whamcloud.com/14815
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6442 utils: "-G <value>" can be passed through mkfs options

mkfs.lustre util checks if parametrs already contain "flex_bg"
option (number of block groups that will be packed together
to create a larger virtual block group)  and adds this option
if it doesn't exists. But second parameter "-G" (number of block
groups that will be packed together to create a larger virtual
block group) can be added twice: one passed through mkfs options
and another one - default value. In this case this parameter
is not changed actually and default value is applied.

This patch adds extra check. Default option "-G" is added
only if no "-G" option passed through mkfs options.

Xyratex-bug-id: MRP-2046
Signed-off-by: Artem Blagodarenko <artem_blagodarenko@xyratex.com>
Reviewed-by: Sergey Cheremencev <sergey_cheremencev@xyratex.com>
Tested-by: Elena Gryaznova <elena.gryaznova@seagate.com>
Reviewed-by: Vitaly Fertman <vitaly.fertman@seagate.com>
Change-Id: I14fb5f8d10fa369428efcbbcb4f638f388979818
Reviewed-on: http://review.whamcloud.com/14400
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6198 test: fix large-lun test_1, test_2 defects

setupall()->start ost called at the end of large-lun.sh
fails to start the ost devices which leads test_1() and
test_2() fail if run separately.

The reasons are:
1) osts data are destroyed by test_1()->llverdev
2) osts are reformatted by test_2, but mds config not changed
Patch fixes the following:
- format Lustre at the end of test_1() and test_2()
- add the "REFORMAT" option check to avoid killing
filesystems that do not want a reformat.

Signed-off-by: Elena Gryaznova <elena.gryaznova@seagate.com>
Xyratex-bug-id: MRP-1984
Reviewed-by: Vladimir Saveliev <vladimir.saveliev@seagate.com>
Change-Id: I7f1465d4777b7a5d5538e019226334dc090e3f50
Reviewed-on: http://review.whamcloud.com/13601
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6261 gnilnd: Changes for small message rate improvment

Change number of threads on Aries service nodes to 7 for better
performance of small messages.
Change the max_immediate value to 8k instead of 2k. FMA has better
performance up to 8k.
lnet passes an niov of 256 reflecting the size of the router buffer
number of pages in the buffer pool when receiving data from the ib
interface when the message size is over one page. Adjust the niov to
reflect the message size/offset for sends.
Use the cookie defined in gni_pub.h instead of value from gnilnd.h.

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I84270653efcff56f06da7de7fb9674ae319800dd
Reviewed-on: http://review.whamcloud.com/15431
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: James Shimek <jshimek@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6261 gnilnd: Thread-safe optimizations.

Take advantage of improved gni threading.
Do not use gnd_cq_mutex lock for kgni versions that support thread
safe gni api.
Check if version is greater than code rev 0xb9 and use locking of
smsg and rdma on a per connection instead of the global cq lock.
Changed gnc_tx_seq and gnc_rx_seq to atomics.
Added gnc_smsg_mutex and gnc_rdma_mutex per conn to protect the lists
that the messages are placed on.

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: Ic03f1877ab7b9632ca5517cd74f7e7fa25ba171b
Reviewed-on: http://review.whamcloud.com/15430
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: James Shimek <jshimek@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6668 test: regression tests for NRS TBF policy

This patch adds fundamental regression tests for NRS TBF policy.

Signed-off-by: Li Xi <lixi@ddn.com>
Change-Id: I37371845c2920f65d38df03bc42b40fd9dea5bb0
Reviewed-on: http://review.whamcloud.com/15102
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6661 test: add version check for tests

add version check for sanity 27e and 162c, so that interop test
won't fail.

Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Change-Id: I45cc25d71e2e7b361455c9dcde1c9a3e1c797e9b
Reviewed-on: http://review.whamcloud.com/15521
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6683 osd: declare enough credits for generating LMA

Usually, the LMA EA is set first after the object created. But if the
system is upgraded from 1.8 release or older, then there is no LMA EA
stored in the object, and then generating LMA EA for the object maybe
not the first EA in the object's inode.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I2bdbf5ba56db1ea08edf0a8a4d724df4ad97e071
Reviewed-on: http://review.whamcloud.com/15361
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6690 tests: start nfsserver service for SLES

The NFS server service name in SLES distro is "nfsserver"
instead of "nfs". This patch fixes setup-nfs.sh to start
nfsserver service for SLES.

Test-Parameters: alwaysuploadlogs \
envdefinitions=EXCEPT=compilebench \
mdtfilesystemtype=ldiskfs mdsfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs \
clientdistro=sles11sp3 ossdistro=sles11sp3 mdsdistro=sles11sp3 \
mdtcount=1 testlist=parallel-scale-nfsv3,parallel-scale-nfsv4

Test-Parameters: alwaysuploadlogs \
mdtfilesystemtype=ldiskfs mdsfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs \
clientdistro=el6.6 ossdistro=el6.6 mdsdistro=el6.6 \
mdtcount=1 testlist=parallel-scale-nfsv3,parallel-scale-nfsv4

Signed-off-by: Jian Yu <jian.yu@intel.com>
Change-Id: Ifb200400c0fa072fee6f563431be0429e2c79890
Reviewed-on: http://review.whamcloud.com/15149
Reviewed-by: Wei Liu <wei3.liu@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6834 lod: fix idx_array overwritten

Compare index with stripe_count -1 , otherwise it will
overwrite the idx_array[] in lod_prep_md_striped_create().

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: Ib116821fb9e9b800752d760c262ded725c55ce0e
Reviewed-on: http://review.whamcloud.com/15572
Tested-by: Jenkins
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6529 ldlm: reclaim granted locks defensively

To avoid ldlm lock exhausting server memory, two global parameters:
ldlm_watermark_low & ldlm_watermark_high are used for reclaiming
granted locks and rejecting incoming enqueue requests defensively.

ldlm_watermark_low: When the amount of granted locks reaching this
threshold, server start to revoke locks gradually.

ldlm_watermark_high: When the amount of granted locks reaching this
threshold, server will return -EINPROGRESS to any incoming enqueue
request until the lock count is shrunk below the threshold again.

ldlm_watermark_low & ldlm_watermark_high is set to 20% & 30% of the
total memory by default. It is tunable via proc entry, when it's set
to 0, the feature is disabled.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I2fab39ac0ab6f269b7f1a40f3e08b8a51807cc69
Reviewed-on: http://review.whamcloud.com/14931
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2778 tests: UNEXCEPT conf-sanity/57b

Due to fixes for LU-2059 this test should now pass consistantly on
ZFS.

Test-Parameters: mdsfilesystemtype=zfs mdtfilesystemtype=zfs ostfilesystemtype=zfs testlist=conf-sanity
Test-Parameters: mdsfilesystemtype=zfs mdtfilesystemtype=zfs ostfilesystemtype=zfs testlist=conf-sanity
Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Ia9e567a5692109458b8cbf9fc034d571c52ea130
Reviewed-on: http://review.whamcloud.com/10357
Tested-by: Jenkins
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6541 utils: fix potential memory leaks with realloc()

The semantics of realloc are to return NULL when the allocation fails,
but won't free the original allocation:

"The realloc() function returns a pointer to the newly allocated memory,
which is suitably aligned for any built-in type and may be different
from ptr, or NULL if the request fails. If size was equal to 0, either
NULL or a pointer suitable to be passed to free() is returned. If
realloc() fails, the original block is left untouched; it is not freed
or moved."

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: Ib348acad13c866a2904e023ac5822f23d13b38ca
Reviewed-on: http://review.whamcloud.com/14814
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6539 quota: fix incorrect format string used for short type

The %u in format string requires 'unsigned int' but the argument type
is 'short'.

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: Ie84b5f2d46809eb1a840d07ffa0b45f6ada1379e
Reviewed-on: http://review.whamcloud.com/14810
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6245 libcfs: remove last of user land code in libcfs module

Currently user land code still exist in libcfs source for
workitems and some of the linux specific debug code. We
can now safely remove the last bit of this code.

Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Change-Id: Id137071f52cbce3a4c6de323b5412976fb687c54
Reviewed-on: http://review.whamcloud.com/15197
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: frank zago <fzago@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5829 libcfs: remove unnecessary EXPORT_SYMBOL

A lot of symbols don't need to be exported at all because they are
only used in the module they belong to.

Signed-off-by: frank zago <fzago@cray.com>
Change-Id: Ie386a75bb8d9b5b77360ea2c714aba886dd8f780
Reviewed-on: http://review.whamcloud.com/13319
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6800 obdclass: change spinlock of key to rwlock

Most of the time, keys are never changed. So rwlock might be
better for the concurrency of key read.

Signed-off-by: Li Xi <lixi@ddn.com>
Change-Id: I5c86c7333884a44ec8e9017d532d6eea5a119cfa
Signed-off-by: Gu Zheng <gzheng@ddn.com>
Reviewed-on: http://review.whamcloud.com/15558
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6837 update: re-lookup the dtrq in the replay list.

In insert_update_records_to_replay_list(), if the
dtrq request has been added to the list by another
thread, it should lookup the request in the list
again, otherwise it will cause the following sub req
lookup panic, because the current dtrq was just freed.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I766cddbe5abcd96e97a611985032208585573b43
Reviewed-on: http://review.whamcloud.com/15573
Tested-by: Jenkins
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4340 tests: Fix test_69 of conf-sanity test

Since 100% inodes were used with default value of MDSSIZE and OSTSIZE,
test_69 of conf-sanity test was failing with ENOSPC error.

Use decreased value of num_create to create 20000*3 files.

Also added umount of client before OST replace operation so that
eviction/recovery of client is not seen.

Seagate-bug-id: MRP-1698
Signed-off-by: Ashish Purkar <ashish.purkar@seagate.com>
Change-Id: I4568cfb3d259ae248f26d5463cd3291db60f2d5b
Reviewed-on: http://review.whamcloud.com/15487
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6660 rename: DNE2 should return -EXDEV upon remote rename

DNE2 MDS should return -EXDEV upon remote rename, so that old
client can do rename with copy and delete, instead of fail
with -EREMOTE.

Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Change-Id: I68e8e99259065922f31bee5343be309380715674
Reviewed-on: http://review.whamcloud.com/15323
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>

LU-5946 lprocfs: free expired jobstats after /proc read

Currently, jobstats for expired jobids are only freed when a new
jobid arrives.  If no new jobids have arrived in a long time, the
expired jobstats won't be freed even though the /proc/.../job_stats
file is read repeatedly.  This shouldn't be a problem for steady job
submission patterns, but if there is a big gap in new jobid arrival
then expired jobids shouldn't be left around such a long time.

Free expired stats in lprocfs_jobstats_seq_release() after job_stats
is read, and move normal expiry to the arrival of new jobids instead
of on every stat to reduce overhead.  If jobs arrive frequently then
expired jobs will also be freed frequently, and if jobs arrive
slowly then there wan't be many expired jobs accumulating.

Rename job_iter_callback() to job_cleanup_iter_callback() for name
consistency with its caller and to make it more clear what it is for.

Instead of just passing a "bool force" param to lprocfs_job_cleanup(),
pass the number of seconds before which stats should be expired.
That allows lprocfs_job_cleanup() to be used at all callers instead
of having to have custom iterators in several places.

Avoid having multiple threads do cleanup of the jobstats concurrently.
This is not itself dangerous, since hash locking avoids races, but it
is inefficient to have multiple threads contending on the hash locks.

Add comment blocks for modified functions.

Remove some LASSERT() calls and replace with checks and error returns,
preferably at initialization time rather than during later usage.

Fix miscellaneous coding style issues.

Add a test for jobstats expiry after /proc read, and add a DNE test.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: Icfd65ed7cd7beb26a9b566667e2686afe23ebbe5
Reviewed-on: http://review.whamcloud.com/12821
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-5638 tests: Skip sanity-quota tests for ZFS

sanity-quota tests 11, 33, 34 and 35 are failing frequently
in review-zfs-part-1 test groups. We are temporarily disabling
these tests until a fix is submitted.

Note that LU-5638 is thought to be caused by
LU-2435 inode accounting in osd-zfs is racy.

Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: I3eaa3f89ca82123f8578b54006451143ce005349
Reviewed-on: http://review.whamcloud.com/15590
Tested-by: Jenkins
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6325 ptlrpc: make ptlrpcd threads cpt-aware

On NUMA systems, the placement of worker threads relative to the
memory they use greatly affects performance. The CPT mechanism can be
used to constrain a number of Lustre thread types, and this change
makes it possible to configure the placement of ptlrpcd threads in a
similar manner.

To simplify the code changes, the global structures used to manage
ptlrpcd threads are changed to one per CPT. In particular this means
there will be one ptlrpcd recovery thread per CPT.

To prevent ptlrpcd threads from wandering all over the system, all
ptlrpcd thread are bound to a CPT. Note that some CPT configuration
is always created, but the defaults are not likely to be correct for
a NUMA system. After discussing the options with Liang Zhen we
decided that we would not bind ptlrpcd threads to specific CPUs,
and rather trust the kernel scheduler to migrate ptlrpcd threads.

With all ptlrpcd threads bound to a CPT, but not to specific CPUs,
the load policy mechanism can be radically simplified:

- PDL_POLICY_LOCAL and PDL_POLICY_ROUND are currently identical.
- PDL_POLICY_ROUND, if fully implemented, would cost us the locality
  we are trying to achieve, so most or all calls using this policy
  would have to be changed to PDL_POLICY_LOCAL.
- PDL_POLICY_PREFERRED is not used, and cannot be implemented without
  binding ptlrpcd threads to individual CPUs.
- PDL_POLICY_SAME is rarely used, and cannot be implemented without
  binding ptlrpcd threads to individual CPUs.

The partner mechanism is also updated, because now all ptlrpcd
threads are "bound" threads. The only difference between the various
bind policies, PDB_POLICY_NONE, PDB_POLICY_FULL, PDB_POLICY_PAIR, and
PDB_POLICY_NEIGHBOR, is the number of partner threads. The bind
policy is replaced with a tunable that directly specifies the size of
the groups of ptlrpcd partner threads.

Ensure that the ptlrpc_request_set for a ptlrpcd thread is created on
the same CPT that the thread will work on. When threads are bound to
specific nodes and/or CPUs in a NUMA system, it pays to ensure that
the datastructures used by these threads are also on the same node.

Visible changes:

* ptlrpcd thread names include the CPT number, for example
  "ptlrpcd_02_07". In this case the "07" is relative to the CPT, and
  not a CPU number.

Tunables added:

* ptlrpcd_cpts (string): A CPT string describing the CPU partitions
  that ptlrpcd threads should run on. Used to make ptlrpcd threads
  run on a subset of all CPTs.

* ptlrpcd_per_cpt_max (int): The maximum number of ptlrpcd threads
  to run in a CPT.

* ptlrpcd_partner_group_size (int): The desired number of threads
  in each ptlrpcd partner thread group. Default is 2, corresponding
  to the old PDB_POLICY_PAIR. A negative value makes all ptlrpcd
  threads in a CPT partners of each other.

Tunables obsoleted:

* max_ptlrpcds: The new ptlrcpd_per_cpt_max can be used to obtain the
  same effect.

* ptlrpcd_bind_policy: The new ptlrpcd_partner_group_size can be used
  to obtain the same effect.

Internal interface changes:

* pdb_policy_t and related code have been removed. Groups of partner
  ptlrpcd threads are still created, and all threads in a partner
  group are bound on the same CPT. The ptlrpcd threads bound to a
  CPT are typically divided into several partner groups. The partner
  groups on a CPT all have an equal number of ptlrpcd threads.

* pdl_policy_t and related code have been removed. Since ptlrpcd
  threads are not bound to a specific CPU, all the code that avoids
  scheduling on the current CPU (or attempts to do so) has been
  removed as non-functional. A simplified form of PDL_POLICY_LOCAL
  is kept as the only load policy.

* LIOD_BIND and related code have been removed. All ptlrpcd threads
  are now bound to a CPT, and no additional binding policy is
  implemented.

* ptlrpc_prep_set(): Changed to allocate a ptlrpc_request_set
  on the current CPT.

* ptlrpcd(): If an error is encountered before entering the main loop
  store the error in pc_error before exiting.

* ptlrpcd_start(): Check pc_error to verify that the ptlrpcd thread
  has successfully entered its main loop.

* ptlrpcd_init(): Initialize the struct ptlrpcd_ctl for all threads
  for a CPT before starting any of them. This closes a race during
  startup where a partner thread could reference a non-initialized
  struct ptlrpcd_ctl.

Signed-off-by: Olaf Weber <olaf@sgi.com>
Change-Id: I3ac40ea56f9c792c3e7c36967e2e1f20105c566c
Reviewed-on: http://review.whamcloud.com/13972
Tested-by: Jenkins
Reviewed-by: Grégoire Pichon <gregoire.pichon@bull.net>
Reviewed-by: Stephen Champion <schamp@sgi.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6816 utils: remove libzfs_load_module() call

Remove libzfs_load_module() call from the OSD.  The function is no
longer exported by libzfs2 in ZFS master, as it was intended as an
internal function to begin with.  libzfs_init(), which is already
being used to set up the OSD, loads the module if necessary.

The intent of the call to libzfs_load_module apparently was to make
zfs_init() fail silently if ZFS was improperly installed (e.g. the
userspace libraries were missing), but fail with an error message if
ZFS initialization failed.  However it's not clear how this could ever
work.

Handling such error conditions will be dealt with in a separate ticket
as it's a larger concern.

Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Change-Id: I97a2de7cd1a32c8251d805192aaf557f48840584
Reviewed-on: http://review.whamcloud.com/15539
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>

LU-6602 osp: change lgh_hdr_lock to mutex

Change lgh_hdr_lock from spinlock to mutex, because in
osp_md_write_obj(), it might be blocked by memory
allocation.

Reorganize the llog_cache_rec process, so lgh_hdr_mutex
will protect both setbit and update llog header, to avoid
conflicts between add records and cancel llog records.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: Ic12d7d768546c67a858cb235929de8ae6aeaa1aa
Reviewed-on: http://review.whamcloud.com/15274
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6467 test: skip s-l test_9 for interop with b2_5

Master uses layout LFSCK for sanity-lfsck test_9, that is not
support old version, should be skipped.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I41497b27d669813d321208e291792f200f7ac863
Reviewed-on: http://review.whamcloud.com/14500
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-4045 mdd: use same attr to declare setattr

in mdd_attr_set, different lu_attr ("la" and "la_copy") is used
for declaring "setattr" and the "setattr" itself, the "LA_MODE"
could be added to "la_copy" by "mdd_fix_attr" in some case and
cause the "setxattr" in "mdd_attr_set_internal" is processed
without declaration.

Change-Id: Ic4441e0c72e8d8209d8f5e60f71ab09dc7987163
Signed-off-by: Hongchao Zhang <hongchao.zhang@intel.com>
Reviewed-on: http://review.whamcloud.com/14566
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6794 nfs: ASSERTION( atomic_read(&d->ld_ref) == 0 ) failed

ll_iget_for_nfs() can call unbalanced iput() causing memory
leaks. This patch removes this iput() call.

Also, avoid unhashing disconnected dentries in
d_lustre_invalidate(), which is another source of memory
leaks.

One of the symptoms of the leak is the following crash pattern:
<0>LustreError: 14812:0:(lu_object.c:1251:lu_device_fini())
ASSERTION( atomic_read(&d->ld_ref) == 0 ) failed: Refcount is 1
<0>LustreError: 14812:0:(lu_object.c:1251:lu_device_fini()) LBUG
<4>Pid: 14812, comm: umount
<4>
<4>Call Trace:
<4> [<ffffffffa11bc895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
<4> [<ffffffffa11bce97>] lbug_with_loc+0x47/0xb0 [libcfs]
<4> [<ffffffffa1458a48>] lu_device_fini+0xb8/0xc0 [obdclass]
<4> [<ffffffffa08e9ab2>] lovsub_device_free+0x52/0x220 [lov]
<4> [<ffffffffa145c64e>] lu_stack_fini+0x7e/0xc0 [obdclass]
<4> [<ffffffffa146356e>] cl_stack_fini+0xe/0x10 [obdclass]
<4> [<ffffffffa08bc1a8>] lov_device_fini+0x58/0x120 [lov]
<4> [<ffffffffa145c619>] lu_stack_fini+0x49/0xc0 [obdclass]
<4> [<ffffffffa146356e>] cl_stack_fini+0xe/0x10 [obdclass]
<4> [<ffffffffa0e1279d>] cl_sb_fini+0x6d/0x190 [lustre]
<4> [<ffffffffa0dd34bc>] ll_put_super+0x1bc/0x11e0 [lustre]
<4> [<ffffffff811cd0f2>] ? fsnotify_clear_marks_by_inode+0x32/0xf0
<4> [<ffffffff811a59df>] ? destroy_inode+0x2f/0x60
<4> [<ffffffff811a5eac>] ? dispose_list+0xfc/0x120
<4> [<ffffffff811a62a6>] ? invalidate_inodes+0xf6/0x190
<4> [<ffffffff8118b35b>] generic_shutdown_super+0x5b/0xe0
<4> [<ffffffff8118b446>] kill_anon_super+0x16/0x60
<4> [<ffffffffa144e7ba>] lustre_kill_super+0x4a/0x60 [obdclass]
<4> [<ffffffff8118bbe7>] deactivate_super+0x57/0x80
<4> [<ffffffff811aabef>] mntput_no_expire+0xbf/0x110
<4> [<ffffffff811ab73b>] sys_umount+0x7b/0x3a0
<4> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b

Xyratex-bug-id: MRP-2414
Signed-off-by: Andrew Perepechko <andrew.perepechko@seagate.com>
Change-Id: Ie8bc428014f3897dc108568f55db9a9ed93c932b
Reviewed-on: http://review.whamcloud.com/15480
Tested-by: Jenkins
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4416 ldiskfs: enable support for SLES12

This patch adds support for the 3.12-based SLE12 kernel. Additonally
this work separates out the basic needed patches for the upstream
3.12 kernel which can be shared with other distros as well as future
upstream kerrnels. This will help with book keeping ldiskfs patches
that need to be pushed upstream.

kernel version SLES12 [3.12.43-52.6]

Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Change-Id: I2b8d83cddf8fd27617657348ea6d7c3a48e9472c
Signed-off-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-on: http://review.whamcloud.com/10165
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6805 ptlrpc: use smp unsafe at_init only for initialization

at_init() is not smp safe, so it is not supposed to be used anywhere
but in at initialization.
Add at_reinit() - safe version of at_init().

Xyratex-bug-id: MRP-2726
Signed-off-by: Vladimir Saveliev <vladimir.saveliev@seagate.com>
Change-Id: I90f872bd217d1e8008f8d35bb4606d168c9bf2ad
Reviewed-on: http://review.whamcloud.com/15522
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Chris Horn <hornc@cray.com>

LU-6747 osd-zfs: initialize obd_statfs in osd_statfs()

The osd-zfs osd_statfs() method wasn't initializing all of the fields
of struct obd_statfs, and in some cases could be returning random
data to the client.  For most of the fields this doesn't matter, but
the os_state field could contain OS_STATE_READONLY and prevent the
MDS from allocating objects there.

Reorder the assignment of struct obd_statfs fields to be in struct
order so that it is more clear which fields have already been set.

Also fixup osd-ldiskfs obd_statfs() to initialize os_maxbytes as
osd-zfs does, even though it currently isn't used.

Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: If4ec6b5fc9c31676267e63be34d00275b4ac8dfb
Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-on: http://review.whamcloud.com/15346
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-5092 nodemap: convert member hash to a list

The members of a nodemap were stored in a hash, but they were only
iterated against, not accessed by key. This patch converts the hash
to a list in order to simplify things.

Signed-off-by: Kit Westneat <kit.westneat@gmail.com>
Change-Id: Ic1df4035d1c53a7c7966c4c528d20cefc7258d0e
Reviewed-on: http://review.whamcloud.com/14885
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6775 ldlm: reduce mem footprint of ldlm_resource

- Removing the unused lr_most_restr, reduced 4 bytes;
- Allocating lr_itree only for LDLM_EXTENT resource, reduced
  120 bytes;
- Moving fields around to eliminate holes, eliminated 3 holes,
  reduced 4 bytes;
- Making lr_contention_time and lr_lvb_inode into union,
  reduced 8 bytes;

Reduced 136 bytes in total.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I232a2669d9f5eaee35eb9ff8313d86d4ac297bc1
Reviewed-on: http://review.whamcloud.com/15485
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6766 utils: remove llapi_quotachown() and cb_quotachown()

The SYS_chown syscall is not available on arm64, but the functions
llapi_quotachown() and cb_quotachown() are not needed anymore and
should have been removed in the patch
http://review.whamcloud.com/4022

Change-Id: I11fa4ab901327c1c8bcfb7e6dfe14a955e2f2618
Signed-off-by: Arnaud Guignard <arnaud.guignard@cea.fr>
Reviewed-on: http://review.whamcloud.com/15396
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6755 kernel: kernel update RHEL7.1 [3.10.0-229.7.2.el7]

Update RHEL7.1 kernel to 3.10.0-229.7.2.el7.

Signed-off-by: Yang Sheng <yang.sheng@intel.com>
Change-Id: Ie922ed7c04394d4ffb02d28fd213c8a1da884bdd
Reviewed-on: http://review.whamcloud.com/15388
Tested-by: Jenkins
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6763 lnet: test for sk_sleep presence in compact-2.6.h

Like Lustre external infiniband stacks create a compability
layer to handle various distributions and kernel versions.
Due to this besides the linux kernel sk_sleep can also be
defined by the external infiniband stack. We need to examine
the infiniband stack's headers to see if sk_sleep is available
there as well.

Change-Id: I0317349cfe13f281c56a8aec17dc330a0e3c759d
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: http://review.whamcloud.com/15386
Tested-by: Jenkins
Reviewed-by: Chris Horn <hornc@cray.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Amir Shehata <amir.shehata@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6714 llog: fix wrong offset in llog_process_thread()

- llh_cat_idx may become bigger than llog bitmap size in
  llog_cat_set_first_idx() function
- it is wrong to use previous cur_offset as new buffer offset,
  new offset should be calculated from value returned by
  llog_next_block().
- optimize llog_skip_over() to find llog entry offset by index
  for llog with fixed-size records.

Signed-off-by: Mikhail Pershin <mike.pershin@intel.com>
Change-Id: I0207ddd7e09161e25a2f5d56d69db4c993e650c3
Reviewed-on: http://review.whamcloud.com/15316
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5988 obdclass: drop malloc fault and page accounting

Remove the OBD_ALLOC() memory allocation fault injection since this
is also possible with existing kernel mechanisms and it is no longer
useful to have Lustre-specific code to do this.

Remove the page allocation counters and OBD_PAGE_{ALLOC,FREE} helpers
that are almost entirely unused since the move to CLIO.

Update the max memory allocation counter on /proc read, so that it
doesn't show as lower than the current allocation, in case it hasn't
been updated since the previous ping.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I024919d1e81730a9388e7999938a4cca1e1cab07
Reviewed-on: http://review.whamcloud.com/15110
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4423 lnet: fixed space coding style issues for lib-msg.c

Linux commit: a1caff6de83acca44faecea14b72724cebb9972e

staging: lustre: lnet: lnet: fixed space coding style issues

Fixed a coding style issue with spaces between function names and arguments:

func (x) -> func(x)

Signed-off-by: Samuel Dominguez <samuel.dominguez.lorenzo@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Change-Id: I0952bcf15e93705e77abe5a3fe0d181581df9973
Reviewed-on: http://review.whamcloud.com/15092
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-6543 utils: fix memory leak in lfs_setstripe

Allocated buffer param should be freed before return.

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: I22d21fb12c0f62330a7c2ca6296f571db71084f2
Reviewed-on: http://review.whamcloud.com/14809
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5956 o2iblnd: wrong uses of kib_tx_t::tx_nfrags

The kib_tx_t::tx_nfrags field is the # entries in
the kib_tx_t::tx_frags array, rather than # DMA
mapped entries. So kiblnd_send/kiblnd_recv should
use kib_rdma_desc_t::rd_nfrags instead.

Signed-off-by: Isaac Huang <he.huang@intel.com>
Change-Id: Ib395636af8bcb0b448f4bb3338b0ddd9c5cfa903
Reviewed-on: http://review.whamcloud.com/12857
Tested-by: Jenkins
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5794 osd: additional checks to verify credits calculation

every operation counts credits consumed, so we can check when
credits are exceeded. now the number of declarations doesn't
matter for the checks, it's only credits. i.e. it's OK to
declare a single write for a specific range and then make many
writes to that range within a single transaction.

the patch fixes few issues on osd-zfs making it impossible to
declare change to an object being created in a same transaction.

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Change-Id: I714ccae7ce957e639938df3bed0c4e8fb49d3454
Reviewed-on: http://review.whamcloud.com/12169
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5580 ptlrpc: policy switch directly in tbf

Former version can't change nid to jobid(or reversely) directly in
tbf, it need to change to FIFO(or any other policy) first. This patch
fix this problem. It save the policy argument(like "nid" in tbf),
and if the policy start with a new argument(like "jobid"), it will
stop the former policy and start a new one.

This is also adapt to any other nrs policies if they have arguments.

Signed-off-by: Wu Libin <lwu@ddn.com>
Change-Id: I645af94d774f60e2081e3716b2f249fcab4930d4
Reviewed-on: http://review.whamcloud.com/11749
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4306 tests: improve test_4a of s-q

test_4a of s-q used "sync; sleep 1; sync" to sync all data, that
could stall for an unexpected long time, this patch changes it
to sync_all_data for better time-efficiency.

This patch also modified the script to report proper error when
the grace has already expired before trying create new file.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I4aa2f69933be37d5f98af8172ed6a20781fe6e38
Reviewed-on: http://review.whamcloud.com/15483
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6245 gss: remove libcfs.h from gss utilities code

The gss utility code has nearly all the user land headers
it needs. No need to include libcfs.h anymore.

Change-Id: I2d08d8bf7d4f6165134eb8d445563497db3bf65d
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: http://review.whamcloud.com/15368
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Sebastien Buisson <sebastien.buisson@bull.net>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6735 lnet: select appropriate optimization options for ko2iblnd

In order to support different IB cards in the best way we need to have
ability to select appropriate optimization options for ko2iblnd
according IB cards installed on the system.

The different defaults for different IB cards are selected by aliases
from /etc/modprobe.d/ko2iblnd.conf and customer can change them manually.

In this patch the general mechanism of selecting tunables is proposed.
If TrueScale IB cards are inserted a new optimal tunables are selected.

With proposed tunables the performance with TrueScale IB cards gets up
from 900 MB/sec to 2.2 GB/sec.

Change-Id: Ie296599f8b0c923100aac41994cbb9b49f99decf
Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-on: http://review.whamcloud.com/15313
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Amir Shehata <amir.shehata@intel.com>

LU-6275 obdclass: don't forget to remove LWP mgc config

We may take an MGC lock, but failed with take a config from MGS.
mgc will cancel a lock in that situation and but lock added to
re-enqueue list.

Shutdown (cleanup) started as result lustre_start_lwp failed, but
cleanup forget to call lustre_end_log as lsi_lwp_started don't set
so - lock/cld still on enqueue list, but super block is destroyed.

Signed-off-by: Alexey Lyashkov <alexey.lyashkov@seagate.com>
Change-Id: I9e72d2526f5a21e66a326532d4770c07e2b7cca2
Reviewed-on: http://review.whamcloud.com/13863
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6433 quota: handle QUOTA_DQACQ in READPAGE portal

MDS sends quota request to quota master, which can be
the same host. Deadlock is possible if all other MDS
threads are waiting for lock taken by first thread.

Lets use MDS_READPAGE portal to avoid the deadlock.

Xyratex-bug-id: MRP-2433
Change-Id: Ib2266a6771cfc5a70bae23206b1144bfd45f9ff3
Signed-off-by: Andriy Skulysh <andriy.skulysh@seagate.com>
Reviewed-on: http://review.whamcloud.com/14369
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Ann Koehler <amk@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6570 tests: fix 40a in sanityn.sh

In sanityn.sh 40a, it should cache the parent lookup LOCK
Before check parallel directory lock, otherwise parent
revalidation might cause lock conflicts on MDT.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: Ib4c7c667d4e1dd821df96497e06459db0d5c33f9
Reviewed-on: http://review.whamcloud.com/15366
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6825 ofd: statfs needs NOT check layout LFSCK status

The ofd_statfs() needs NOT to check whether the layout LFSCK is
rebuilding the LAST_ID file(s) on the OST or not, because:
1) Such checking will be done when the MDT wants to pre-allocate
   OST-objects via ofd_create_hdl().
2) The STATFS request may be from client, if ofd_statfs() returns
   -ENOSPC to the client under the case of layout LFSCK rebuilding
   the LAST_ID file(s), the client may be confused.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Iae9eb8485f3b971fda3bcd299df28db7ad4d80a0
Reviewed-on: http://review.whamcloud.com/15349
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-6415 utils: deny non-root user for changelog operations

To avoid potential security problems, non-privileged users should
have no permission to run 'lfs changelog' & 'lfs changelog_clear'.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I5f38ba5b139f2f3b6495d3c97d82a47daecf8187
Reviewed-on: http://review.whamcloud.com/14280
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6073 tests: use sys/xattr.h in tests/mpi/*.c

In lustre/tests/mpi/mdsrate.c include sys/xattr.h rather than
attr/xattr.h since the former is more likely to be installed that the
latter and provides the needed declarations.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I3b3f6a2cc2a5a7153bb3dbff535d7c1b3472f9b8
Reviewed-on: http://review.whamcloud.com/15391
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-6245 utils: remove libcfs.h from lustre utilities code

Since user land libcfs is mostly gone libcfs.h can be
removed from the code that is used to create the lustre
utilities such as lfs and mount.lustre.

Change-Id: I5e0dd7d9ad44e988b2045906d2179dc3e2538d38
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: http://review.whamcloud.com/15217
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: frank zago <fzago@cray.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6705 lod: re-order lodsub recovery cleanup

Before MDT stop recovery thread, it should disconnect
the OSP and stop the sub_recovery thread, otherwise
the recovery thread will be blocked by sub_recovery
thread.

The the right steps to stop the MDT recovery thread is
(see mdt_device_fini())

1. Do stack pre_cleanup (by LCFG_PRE_CLEANUP), to disconnect
OSP and stop sub_recovery thread (used for collecting update
records).
2. Stop the recovery thread.
3. Cleanup the update llog ctxt in mdt_stack_fini()
(by LCFG_CLEANUP).

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I4e938cb87ab02d388bce16bd15d181fd7fcc783b
Reviewed-on: http://review.whamcloud.com/15210
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6245 tests: remove libcfs.h from lctl code

Since user land libcfs is mostly gone libcfs.h can be
removed from the code that is used by lctl.

Change-Id: Ieb2f3873177dfc2c7fa3036568e263f4ba27e1ce
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: http://review.whamcloud.com/15208
Reviewed-by: frank zago <fzago@cray.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6245 lnet: Avoid nid range related forward declarations in nidstring.c

Since forward declarations are frowned on upstream we move
the NID range handling to near the start of the nidstring.c
file.

Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Change-Id: I03961e1f5a89a4bf60873fcc3fc1522542e4931a
Reviewed-on: http://review.whamcloud.com/15086
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6489 osd-ldiskfs: check max_page_index correctly

This patch fixes a issue that original code check
max_page_index against incorrect variable.

Signed-off-by: Yang Sheng <yang.sheng@intel.com>
Change-Id: I4b164d6187ef4a1c343d4b5803b8915283394037
Reviewed-on: http://review.whamcloud.com/14731
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-977 lod: Patch to protect lqr_start_idx

Protect lqr_start_idx to avoid the imblance in allocating objects
on OSTs with round-robin algorithm

Signed-off-by: Rahul Deshmukh <rahul.deshmukh@seagate.com>
Change-Id: I689ba1c4e8c9224cc67badba7fd6cf45e64dd7b6
Seagate-bug-id: MRP-2471
Reviewed-on: http://review.whamcloud.com/14636
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6750 zfs: add stop trans callback

Add stop transaction callback in OSD-ZFS, so when the
osd transaction is stopped, the top transaction
will be notified.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I56b8d964b3c4dd24e3c56429b7f1a8960d3202cb
Reviewed-on: http://review.whamcloud.com/15367
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6717 llog: Create update llog synchronously

Create remote update llog object synchronously, so
it will make sure the llog object exists, before
the cross-MDT operations will write any update llog
records.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I690e94e68b8b64dfe94412f58448f3867f61ff3e
Reviewed-on: http://review.whamcloud.com/15278
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6674 utils: fix of uninitilized lmm structure usage

Fill with "fake" the "use the default" values since the lmm struct
is zeroed out at this point and not filled by ioctl.

Change-Id: I9df005446c5d03bdf561823842afe98ab8d352c1
Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-on: http://review.whamcloud.com/15109
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: jacques-Charles Lafoucriere <jacques-charles.lafoucriere@cea.fr>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6109 osd-ldiskfs: handle no fid-in-dirent correctly

The commit: 4d408c9aed9adaf1f4e2ea87851728a1cf662594
introduced a issue in '..' no fid-in-dirent case.
Since it just deal with remote case. So the '..'
would disappeared if it is a local object and
no fid-in-dirent.

Signed-off-by: Yang Sheng <yang.sheng@intel.com>
Change-Id: Ib2b0be1c43986472cb674dd86d32f023a4b26a34
Reviewed-on: http://review.whamcloud.com/13987
Tested-by: Jenkins
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6207 osd: add osd_ost_fini in osd_obj_map_init

In case of ENOSPC in osd_mdt_init->simple_mkdir("REMOTE_PARENT_DIR")
missed dput(for "O") causes kernel panic with msg:
BUG: Dentry ffff88005d6b3e40{i=b2,n=O} still in use (1)

Change-Id: Ibd5391e781e19efe48e9a9700ee4565b46c2f0b5
Signed-off-by: Sergey Cheremencev <sergey.cheremencev@seagate.com>
Xyratex-bug-id: MRP-2059
Reviewed-on: http://es-gerrit.xyus.xyratex.com:8080/4431
Tested-by: Jenkins
Reviewed-by: Alexander Zarochentsev <alexander_zarochentsev@xyratex.com>
Reviewed-by: Rahul Deshmukh <rahul_deshmukh@xyratex.com>
Tested-by: Elena Gryaznova <elena.gryaznova@seagate.com>
Reviewed-by: Vitaly Fertman <vitaly.fertman@seagate.com>
Reviewed-on: http://review.whamcloud.com/13649
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>

New tag 2.7.56

Change-Id: Ib5d0eb7a06314f347f87eb0ed103435c61fc7033

LU-6722 ldiskfs: declare credits for quota when destroy inode

When destroy the inode, if it has large EA that occupies another
separated block, then it calls ldiskfs_xattr_delete_inode(). But
such function does not consider the quota modification as to that
before making the group descriptor dirty, the persevered credits
becomes zero, so jbd2_journal_dirty_metadata() failed as "-ENOSPC",
then the subsequent journal abort makes the system as RDONLY.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I0567d6b6d54cd57b25524c3d58190926e3505867
Reviewed-on: http://review.whamcloud.com/15334
Tested-by: Jenkins
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6454 test: double default OSTSIZE

Double the default test OST size from 200000KB to 400000KB.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I76a19086e789fe60519b301ee2b1d8f186e13062
Reviewed-on: http://review.whamcloud.com/14650
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-6380 scrub: handle update_log during initial scrub

The normal OI scrub will not run until recovery finished. But the
update logs (update_log, and sub-items under update_log_dir) will
be accessed during the recovery. So if the the OI mapping for the
update log cannot be handled before the recovery, then the normal
OI scrub cannot be started/resumed. To resolve above dependency,
the initial OI scrub (that will be triggered by the mount process
with sync mode before the recovery) will handle the update logs'
OI mappings in advance.

In DNE2, each MDT will have a special FID for update log, during
recovery, the failover MDT can retrieve update logs on remote MDT
by this FID, that said the FID is special and it also has to be
located globally (not local FID). So the MDT index is packed into
the FID (as f_oid), then FLD can locate the special FID. To handle
such special FIDs, OI scrub needs to know the target index.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Iac71c20c1f1631efbe19a03e380822cead6c00d1
Reviewed-on: http://review.whamcloud.com/15192
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6780 ptlrpc: Do not resend req with allow_replay

If the request is allowed to be sent during recovery,
and it is not timeout yet, then we do not need to
resend it in the final stage of recovery.

Unnecessary resend will cause the bulk request to resend the
request with different mbit, but same xid, and on the remote
server side, it will drop such request with same xid, so it
will never get the bulk in this case.

re-enable the multiple MDT failover cases in replay-single.sh

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I96900448fc16b0e2a336a42c8ff977ec56941427
Reviewed-on: http://review.whamcloud.com/15458
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6602 update: split update llog record

If the update llog record size exceeds the limit (llog
chunk size), which usually happens when creating striped
directory with large stripe count , then it will be split
into mulitple records, and these update records will have
the same batchid.

During recovery, these records will be combined and
constructed into the updates of one operation, then
do update replay.

Allow multiples stripes in a single MDT, so it can verify
creating large stripes in small scale test environment.

Add sanity 300j/300k and replay-single 116 to verify it.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I86ca2594fe62d5b921e794de4cd88981d91f7677
Reviewed-on: http://review.whamcloud.com/15162
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6602 llog: increase update llog chunk size

Increase DNE update llog chunk size from 8KB to 32KB, so
one cross-MDT operation can includes more update records,
and it can create striped directory with more stripe count.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I01d1435f1249c3048ac0a412bdb53f710e6e98b3
Reviewed-on: http://review.whamcloud.com/15161
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3534 osp: transfer updates with bulk RPC

Send update request with bulk RPC(iovec), and update request
will be split into multiple buffers, so OSP does not need to
allocate big contingous buffer to store the update request.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I70995a552ee6fdd77f4c0a84d2435e5963e4fd6d
Reviewed-on: http://review.whamcloud.com/13786
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6744 lov: init LOV stripe type beforehand

When lu_object_alloc() reaches to LOV object init, we need initialize
its stripe type beforehand, so that if something wrong in the conf
buffer, the object chain need to be traversed to free what has been
allocated, with LOV object type be set as LLT_EMPTY, and when the LOV
part is reached, it won't panic without knowing what stripe type it
is.

This patch also improves debug messages in lsm_unpackmd_common(), and
does not return error if the LOV device is still processing config
log while trying to verify a layout buffer.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: Ic814f89ebc960cc4c04835141d915f86c35ca39d
Reviewed-on: http://review.whamcloud.com/15362
Tested-by: Jenkins
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-3881 tests: single-node minimal config in sanity-hsm

Patch to allow for a minimal configuration to be
automatically setup in sanity-hsm.sh, and for example
to permit the single-node sessions runs like :
./lustre/tests/llmount.sh
./lustre/tests/sanity-hsm.sh

Also changed default mount-point to be monitored by agent
to become $MOUNT2 if $MOUNT_2==y.

Created get_agent_uuid() function to dynamically discover
agent's monitored mount-point.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: I194d4a321ea3bcc169c3811e19b1069749b99e72
Reviewed-on: http://review.whamcloud.com/12982
Tested-by: Maloo <hpdd-maloo@intel.com>
Tested-by: Jenkins
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6356 mgs: fix security flavor setting for connection to mgs

Security flavor setting for connections to MGS is done with special
target '_mgs'. If set, only allow connections to MGS with the given
flavor. If not set, all flavors must be allowed.

Signed-off-by: Sebastien Buisson <sebastien.buisson@bull.net>
Change-Id: Id47aae25f6d5591870656d9926a64e52ab789c56
Reviewed-on: http://review.whamcloud.com/14349
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Jeremy Filizetti <jeremy.filizetti@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3778 sptlrpc: OSP and LWP don't know sptlrpc

The sptlrpc subsystem must be initialized for OSP and LWP
connections, by calling sptlrpc_lprocfs_cliobd_attach().
Moreover, GSS related functions must not return an LBUG when
dealing with OSP and LWP OBDs.

Signed-off-by: Sebastien Buisson <sebastien.buisson@bull.net>
Change-Id: I3d4bc635a7076addea7bbb52ac13f74a25f625ad
Reviewed-on: http://review.whamcloud.com/14040
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5319 mdt: support multiple modify RCPs in parallel

This patch implements the server part of the feature that allows
support of multiple modify RPCs in parallel on MDT targets.

Each target export is able to store several in-memory reply
data so that it can reconstruct several requests issued by
the client in parallel.

Additionally, a new internal file REPLY_DATA is created on
the target to store on-disk reply data. The reply data slots
in that file are managed by a bitmap (lut_reply_bitmap) and
can be used to store reply data of any client.

When target recovers, the on-disk reply data is used to
restore the in-memory reply data and ensure reconstruction
of committed operations.

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Signed-off-by: Gregoire Pichon <gregoire.pichon@bull.net>
Change-Id: I8f91666f5b0b4f7b9445a01c520d73f56d059ff3
Reviewed-on: http://review.whamcloud.com/14860
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5319 mdc: manage number of modify RPCs in flight

This patch is the main client part of a new feature that supports
multiple modify metadata RPCs in parallel. Its goal is to improve
metadata operations performance of a single client, while maintening
the consistency of MDT reply reconstruction and MDT recovery
mecanisms.

It allows to manage the number of modify RPCs in flight within
the client obd structure and to assign a virtual index (the tag) to
each modify RPC to help server side cleaning of reply data.

The mdc component uses this feature to send multiple modify RPCs
in parallel.

Signed-off-by: Gregoire Pichon <gregoire.pichon@bull.net>
Change-Id: Ia707e39770e479648627611a99d0724e7070baec
Reviewed-on: http://review.whamcloud.com/14374
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5560 llite: basic support of SELinux in CLIO

Bring the ability to properly initiate security context
on SELinux-enabled client and store it on server side via
extended attribute.

Security context initialization is not atomic, but that would
require a wire protocol change to send security label in the
creation request.

Filter out security.selinux from xattr cache as it is
already cached in system slab.

Signed-off-by: Sebastien Buisson <sebastien.buisson@bull.net>
Change-Id: Ie6db97be6f78b0b33da2ae54883a4051a2576af5
Reviewed-on: http://review.whamcloud.com/11648
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3534 ptlrpc: mbits is sent within ptlrpc_body

ptlrpc is using rq_xid as matchbits of bulk data, which means it
has to change rq_xid for bulk resend to avoid several bulk data
landing into the same buffer from different resends.

This patch uses one of reserved __u64 of ptlrpc_body to transfer
mbits to peer, matchbits is now separated from xid. With this change,
ptlrpc can keep rq_xid unchanged on resend, it only updates matchbits
for bulk data.

This protocol change is only applied if both sides of connection have
OBD_CONNECT_BULK_MBITS, otherwise, ptlrpc still uses old approach and
update xid while resending bulk.

Signed-off-by: Liang Zhen <liang.zhen@intel.com>
Change-Id: Ide0b3f490368babeb6ac1b4ea6953f8f9aacf81a
Reviewed-on: http://review.whamcloud.com/15421
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5823 llite: Remove access of stripe in ll_setattr_raw

In ll_setattr_raw(), it needs to know if a file is released
when the file is being truncated. It used to get this information
by accessing lov_stripe_md. This turns out not necessary. This
patch removes the access of lov_stripe_md and solves the problem
in lov_io_init_released().

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I05b532fd137889d13e8ecb35585f4d1b09958583
Reviewed-on: http://review.whamcloud.com/13514
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Henri Doreau <henri.doreau@cea.fr>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6377 llite: clear LLIF_DATA_MODIFIED in atomic

This flag should be cleared atomically after the op_data flag
MDS_DATA_MODIFIED is packed. Otherwise, if there exists an
operation to dirty the file again, the state may be missed on
the MDT.

Stop using spin lock lli_lock to protect operations of changing
file flags; using bit operations instead.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Change-Id: I3a3129e24cea89b0ebaae1b7816d892b5623b3b4
Reviewed-on: http://review.whamcloud.com/14100
Tested-by: Maloo <hpdd-maloo@intel.com>
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6409 nodemap: don't destroy nodemaps in cfs_hash_for_each

nodemap_destroy might sleep, but cfs_hash_for_each iterators aren't
allowed to sleep. This patch changes nodemap_cleanup_all to avoid
destroying nodemaps in its cfs_hash_for_each iterator. Instead, the
iterator creats a list of nodemaps, and then all the nodemaps in the
list are destroyed.

Signed-off-by: Kit Westneat <kit.westneat@gmail.com>
Change-Id: I1d16f61ba7739af0494944f6515508b3f0d19ba1
Reviewed-on: http://review.whamcloud.com/14275
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Andrew Perepechko <andrew.perepechko@seagate.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-5319 ptlrpc: embed highest XID in each request

Atomically assign XIDs and put request and sending list so
we can learn the lowest unreplied XID at any point.

This allows to embed in every resquests the highest XID for
which a reply has been received and does not have an unreplied
lower-numbered XID.

This will be used by the MDT target to release in-memory
reply data corresponding to XIDs of reply received by the client.

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Signed-off-by: Gregoire Pichon <gregoire.pichon@bull.net>
Change-Id: Ic88fb6db704d8e9a78a34fe16f64abb2cdffc4c4
Reviewed-on: http://review.whamcloud.com/14793
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>