LU-6450 mdd: Remove unused MAY_ constants

Remove unused MAY_ constants from lustre_idl.h
Remove dead code from mdd_permission()

Signed-off-by: Ben Evans <bevans@cray.com>
Change-Id: I722f55eac88c7143555e680f72971b96791b00ae
Reviewed-on: http://review.whamcloud.com/15398
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5319 utils: update lr_reader to display additional data

The lr_reader command currently displays only the server area
of the LAST_RCVD file of a Lustre ldiskfs target device.

This patch enhances the command to allow diplaying :
- the per-client area of the LAST_RCVD file
  when --client or -c option is specified
- the reply data of the REPLY_DATA file
  when --reply or -r option is specified
It still only support ldiskfs target device.

Signed-off-by: Gregoire Pichon <gregoire.pichon@bull.net>
Change-Id: Ifc2b7f9591eef0da5e4a5dc00f37e58484d4ea97
Reviewed-on: http://review.whamcloud.com/14862
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2261 test: Re-enable sanity test 156 for zfs

With the landing of LU-4259 zfs now supports brw
stats like ldiskfs. Earlier sanity test 156 was
disabled for zfs but now it can be re-enabled so
we can actually verify brw stats on zfs works.

Apply additional clean up to test 156: Spaces to tabs.

Change-Id: I0a0014bee4c666faadce96177c44953573d85ffd
Signed-off-by: James Simmons <uja.ornl@gmail.com>
Reviewed-on: http://review.whamcloud.com/13535
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-5246 tests: create OST objects on correct MDT

The sanity.sh test_220 is checking that precreated objects on the MDT
can be used even when the OST has subsequently run out of inodes.
The test was not correctly working if the directory was striped over
multiple MDTs since the number of precreated objects on other MDTs
may be fewer than the number of precreated objects $SINGLEMDS and
run out of objects.

Change the test to only create files on $SINGLEMDS so to avoid this.

Also fix the return code of lod_alloc_specific() to be -ENOSPC if
the stripe count is 1, since -EFBIG doesn't make sense in that case.
Fix other code comments for lod_alloc_specific() and lod_alloc_qos().

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I788706c3b212b426aa15424d02913aedcdb98b2a
Reviewed-on: http://review.whamcloud.com/12937
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6215 llite: handle new_sync_[read|write] removal

In newer linux kernel version the file read/write
API have been moving to using struct iov_iter. To
continue supporting the old API new wrapper
functions were created, new_sync_*, which allowed
the use of the new API's to support the old API's.
For the linux 4.1 kernel those new_sync_* are used
internally so they are no longer exposed to any
file systems. If a file system wants to use the
"old" api they need to call vfs_[write|read].
Update lustre to handle this change. The upstream
commit that made new_sync_* internal was:

Linux commit: 5d5d568975307877e9195f5305f4240e506a2807

Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Change-Id: Ibf69d25af8a7d5cda00c5b1be4757ec369e8e814
Reviewed-on: http://review.whamcloud.com/15165
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6365 obd: Eliminate hash bucket scans in lu_cache_shrink

The lu_cache_shrink slab shrinker is too slow, accounting for > 90% of
the time spent in shrink_slab when allocating huge pages. Most of its
time is spent iterating over the buckets in each site's object hash
table to compute the number of freeable objects. This iteration is
eliminated by adding an lru length count to the lu_site struct. A
percpu counter is used to maintain the lru length, so that the
lu_site does not need to be locked when an object is accessed through
the hash table. A counter is updated whenever an object is added to
or deleted from any of the hash table buckets. The number of freeable
objects is the sum of the counter values across all cpus.

Signed-off-by: Ann Koehler <amk@cray.com>
Change-Id: I5c70dd8e8cc1fe024292076e118774d9299bf40b
Reviewed-on: http://review.whamcloud.com/14066
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5554 ptlrpc: race at req processing

First commit 56e5a6bb4fc0329092f85016b8664da0b6ffa8a8
decrease race window, but does not remove it. Disable rq_resend
right after MSG_REPLAY flag set. Import lock protects two threads
from race between set/clear MSG_REPLAY and rq_resend flags.

Signed-off-by: Alexander Boyko <alexander_boyko@xyratex.com>
Xyratex-bug-id: MRP-1888
Change-Id: I64d7c997976ff73956ebe97d7ea0775e0a54bc50
Reviewed-on: http://review.whamcloud.com/10735
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6896 llog: update log initialization needs to be sync

During initialization, update llog catlist and cat file creation
needs to be synchronized, because the following update records
will be written to these files, if these files are missing during
update recovery, then update recovery will fail.

Because update llog is initialized in a separate thread, it should
set loc_handle until the catlog is written to catlist to make sure
the udpate llog is fully initialized because MDT can handle cross
MDT operation. see lod_obd_get_info() to check the healthy of the
target.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: Iccd09749bedde46c01ff89b5be4fa6f4327ec9b9
Reviewed-on: http://review.whamcloud.com/15721
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6972 llite: get rid of unused ll_super_blocks list

ll_super_blocks became unused quite a while ago with switch
to the new CLIO code.
So this patch removes the list, ll_sb_lock spinlock that guards it
and superblock info ll_list linkage.

Change-Id: If0af4f4c05f45feaedc54ca57f0d1106ad7cabe3
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/15922
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>

LU-6831 lmv: revalidate the dentry for striped dir

If there are bad stripe during striped dir revalidation,
most likely due the race between close(unlink) and
getattr, then let's revalidate the dentry, instead of
return error, like normal directory.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I410940473d123796f75607d39153240b7d3f737b
Reviewed-on: http://review.whamcloud.com/15720
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6124 test: skip tests require remote server with nodsh set

Patch fixes the following tests to be skipped for remote
servers with nodsh set:
- sanity 116a, 116b, 160c, 205, 220, 225a, 225b, 230c;
- sanityn 34, 40a, 40b, 40c, 40d, 40e.

Signed-off-by: Elena Gryaznova <elena.gryaznova@seagate.com>
Xyratex-bug-id: MRP-1720
Reviewed-by: Vladimir Saveliev <vladimir.saveliev@seagate.com>
Change-Id: I5567e752f0d05c2447de595cb72c754e06e2fe83
Reviewed-on: http://review.whamcloud.com/13406
Tested-by: Jenkins
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3163 tests: Fix test_61 from conf-sanity test

When large xattr is enabled and fstype is ldiskfs,
use tune2fs to enable/disable large_xattr.

Also replaced $TUNE2FS in all places instead of tune2fs.

Seagate-bug-id: MRP-1191
Signed-off-by: Ashish Purkar <ashish.purkar@seagate.com>
Change-Id: I86605d980343a234cd6f4e2101cac409e017af62
Reviewed-on: http://review.whamcloud.com/15486
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6126 test: skip sanity test_187a, test_187b on old MDS

Don't run sanity test_187a, test_187b in interop mode for
MDS's versions older than 2.3.0 (per file data_version
implementation was included in 2.3, LU-827).

Signed-off-by: Elena Gryaznova <elena.gryaznova@seagate.com>
Xyratex-bug-id: MRP-1826
Reviewed-by: Vladimir Saveliev <vladimir_saveliev@xyratex.com>
Change-Id: I6d42b51c769be80486c84fe436f6f3f71616e1cb
Reviewed-on: http://review.whamcloud.com/13408
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6490 gss: handle struct key_type match replacement

Starting with the 3.17 kernel the function match
used by struct key_type was replaced by a new
function match_preparse. This patch added support
for this change.

Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Change-Id: I2db0860cf089b0ba8f70bad9155f48e52ec8c4b5
Reviewed-on: http://review.whamcloud.com/15804
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Sebastien Buisson <sebastien.buisson@bull.net>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6096 ldiskfs: mark dir's inode dirty

mark directory's inode dirty in ldiskfs_add_dot_dotdot()
as ldiskfs_init_new_dir() doesn't do this internally.
otherwise, the attributes set by ldiskfs_append() aren't
transferred to the buffer cache.

Change-Id: Ice93b75e291736f83c0d9b1047f7a627fef0141e
Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-on: http://review.whamcloud.com/15581
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6938 osd: fix error handling in osd_xattr_list()

in case of buffer overflow, osd_xattr_list() should leave
properly: release the semaphore and the cursor.

Change-Id: I750183cb083cb87b3d8adae8fb0f41b61e6689e5
Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-on: http://review.whamcloud.com/15828
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6542 lnet: handle cYAML out of memory error

in cYAML_build_error() there is a potential on run out of memory
scenario, that it is ignored and no error is built.
In this case print out a properly formated YAML error message
indicating fatal error

Signed-off-by: Amir Shehata <amir.shehata@intel.com>
Change-Id: Id9159044acafb38bd022f421a997fa2d2f1841e8
Reviewed-on: http://review.whamcloud.com/15778
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6906 mgc: IR log failure should not stop mount

If clients or other targets can not get IR config lock
or lock, the mount should continue, instead of failing.
Because timeout mechanism will handle the recovery anyway.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: Ie717cb363180907b510593ee4a6caaec6f07a5f3
Reviewed-on: http://review.whamcloud.com/15728
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6544 mkfs: Improve MDT inode size calculation

This patch reduces mkfs.lustre "--stripe-count-hint" limits when
calculating MDT inode size, so that there is a proper amount of
space reserved for different kinds of EAs and ACL in the inode.

This would allow files with N stripes to fit the lov, lma, link
EAs into the inode rather than storing it in an external xattr
block, which reduces performance and significantly increases the
space used per file on the MDT.

Also, this patch adds conf-sanity.sh test_87 to verfiy this
calculation with different stripe count.

Signed-off-by: Emoly Liu <emoly.liu@intel.com>
Change-Id: Idfdd9a064f0ac07c383a3af79c61c1ff973fb3f7
Reviewed-on: http://review.whamcloud.com/15643
Tested-by: Jenkins
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6904 mdd: prepare linkea before declare

In mdd_rename, it should prepare linkea before journal
start, otherwise, it may send RPC to other MDT while
holding the journal(see mdd_prepare_linkea). So if the
other MDT is in recovery, then this thread will be
blocked with the started journal handle, then it will
block other journal thread.

For the same reason, mdd_rename_order, which will send
RPC to the remote MDT (mdd_is_parent()), should also
move ahead of mdd_trans_start().

Because mdd_sobj can not be null in mdd_rename, this
patch also cleanup this API a bit.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: Id2426ef5d0d83c704403aa3f762596284576df6d
Reviewed-on: http://review.whamcloud.com/15724
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6874 out: missing noop in out_update_ops

NOOP is missing in out_update_ops, though this NOOP
operation will only exit inside the update log, i.e.
out_handler should never handle noop update. But
if we add new update in out_update_ops table, missing
noop there might cause problems.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I967ffefcd1499801fa1890ed8d97ce70190c1902
Reviewed-on: http://review.whamcloud.com/15692
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6490 build: enable gss build on sles12

Now that the build failures of gss on sles12 are fixed
change lbuild to stop disabling the gss build there.

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: Icf25c038c28b8e230550de0bbe5b0b2fbe0c0b78
Reviewed-on: http://review.whamcloud.com/15354
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Sebastien Buisson <sebastien.buisson@bull.net>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6490 gss: 3.1x kernels adjustments for gssapi code

There are a number of changes in 3.1x kernels concerning the GSSAPI:
- libgssapi and libgssglue do not exist anymore, so call krb5
  primitives directly, and remove associated config checks;
- struct cred has no tgcred member anymore, so use cred directly;
- struct key_type instantiate and update function prototypes
  have changed;
- add new config checks on struct cred and struct key_type;
- u_int is BSD specific, so it is replaced with unsigned int.

Signed-off-by: Sebastien Buisson <sebastien.buisson@bull.net>
Change-Id: I3b13c2afcb4b800bdcffb3b8713048f8e39f6866
Reviewed-on: http://review.whamcloud.com/15342
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6356 ptlrpc: add a 'enc_pool_max_memory_mb' module param

Create a new 'enc_pool_max_memory_mb' module param for ptlrpc.
It is used to set max memory used by encoding pools, in MB.
By default encoding pools will use at max 1/8 of total physical
memory, but it appears to be too few under some circumstances.

Signed-off-by: Sebastien Buisson <sebastien.buisson@bull.net>
Change-Id: If431ba082eec4bc2c4f2131d4b6b5b7028a3a569
Reviewed-on: http://review.whamcloud.com/15069
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5888 utils: limit max_sectors_kb tunable setting

Properly limit the max_sectors_kb tunable if larger than 32MB, as
added in commit 73bd456e896 (http://review.whamcloud.com/12723).
It was using the old value in "buf" instead of the updated "newval".

Change write_file() to use a file descriptor directly instead of
streams, since it only writes a single value before closing again.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I43a5b4c9d0e45649e0666fd58634286de53ebbe5
Reviewed-on: http://review.whamcloud.com/13240
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6916 target: fix ted_lcd_lock init

Move ted_lcd_lock init from tgt_client_new()
to tgt_client_alloc(), otherwise if error happens
in client_new_export(), the error handler path
will cause panic like this

<1>BUG: unable to handle kernel NULL pointer dereference at (null)
<1>IP: [<ffffffff812957f6>] __list_add+0x26/0xa0
<4> [<ffffffff8152adef>] __mutex_lock_slowpath+0xcf/0x180
<4> [<ffffffffa047e087>] ? cfs_hash_putref+0x2e7/0x480 [libcfs]
<4> [<ffffffff8152acfb>] mutex_lock+0x2b/0x50
<4> [<ffffffffa083db72>] tgt_client_free+0x62/0x610 [ptlrpc]
<4> [<ffffffffa0f59f84>] mdt_destroy_export+0x74/0x220 [mdt]
<4> [<ffffffffa05862c5>] class_new_export+0x315/0x9c0 [obdclass]
<4> [<ffffffffa058745e>] class_connect+0xae/0x250 [obdclass]
<4> [<ffffffffa0f57fd1>] mdt_obd_connect+0xb1/0x720 [mdt]
<4> [<ffffffffa07aaf78>] target_handle_connect+0xe58/0x2d30 [ptlrpc]
<4> [<ffffffff8128c1f0>] ? string+0x40/0x100
<4> [<ffffffffa084edf2>] tgt_request_handle+0x5b2/0x1230 [ptlrpc]
<4> [<ffffffffa07f7611>] ptlrpc_main+0xe41/0x1920 [ptlrpc]
<4> [<ffffffffa07f67d0>] ? ptlrpc_main+0x0/0x1920 [ptlrpc]
<4> [<ffffffff8109abf6>] kthread+0x96/0xa0
<4> [<ffffffff8100c20a>] child_rip+0xa/0x20
<4> [<ffffffff8109ab60>] ? kthread+0x0/0xa0
<4> [<ffffffff8100c200>] ? child_rip+0x0/0x20

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: If09c019384f53a63107aeaa293ef93d6415f81b5
Reviewed-on: http://review.whamcloud.com/15770
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6827 osd: trans credit insufficient for EA object create

EA object consumes more credits than regular object: osd_mk_index vs.
osd_mkreg, this patch reserves more credits for OSD_OT_CREATE
operation.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I40ec58ca467474ec4d96e5a1d24164fae03fc227
Reviewed-on: http://review.whamcloud.com/15694
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6768 osd: unmap reallocated blocks

call unmap_underlying_metadata() on every allocated
block. otherwise metadata blocks released in a previous
transaction can be written by the kernel corrupting
user's data. pblock != 0 should be a good sign of that.

Change-Id: I0d06611feb384a3f7ef5d8e5b34951822369ed0f
Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-on: http://review.whamcloud.com/15593
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6427 osd: race between destroy and load LMV

Do not get LMVEA if the object has been destroyed,
because it needs to iterate the master object, which
is dangerous for ZFS OSD.

Add oo_destroyed flags to avoid accessing destroyed
directory.

Add sanityn.sh 83 to verify accessing destroyed striped
directory will not cause panic.

Signed-off-by: wang di <di.wang@intel.com>
Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I1ef6bf197eac0917fd649868235c96af2f64207c
Reviewed-on: http://review.whamcloud.com/14346
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6502 lnet: remove unnecessary NULL check

In LNetCtl():IOC_LIBCFS_GET_NET there is a check for config == NULL
This is not necessary as it'll never be NULL.  That's ensured before
the call to LNetCtl.

Signed-off-by: Amir Shehata <amir.shehata@intel.com>
Change-Id: Ib4d1ca5ca4a342f134e3102bcaf167e60f978b5f
Reviewed-on: http://review.whamcloud.com/15779
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6833 build: gerrit_checkpatch.py improvements

In contrib/scripts/gerrit_checkpatch.py:
  Add LASSERT and LCONSOLE, to the list of ignored types.
  Use a 60 second HTTP request timeout to avoid hangs.
  Catch any Exception around HTTP methods.
  Refactor some methods for to facilitate unit testing.
  Add timestamps to log messages.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ic8ddf815e58744af1cfa5a03986d869b790d91ae
Reviewed-on: http://review.whamcloud.com/15560
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5092 nodemap: add structure to hold nodemap config

This patch moves global state variables into a configuration
structure so that new configurations can be more easily loaded and
swapped into the active role.

Signed-off-by: Kit Westneat <kit.westneat@gmail.com>
Change-Id: Ib0d51d56154d5e831b13f2935feab9bd73944bcc
Reviewed-on: http://review.whamcloud.com/14254
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6908 lfsck: initalize name before use

lfsck_create_lpf() should initialize name before using
that to enqueue a hashed lock.

Change-Id: I32ade3231111e5ce687eff348a0454a98c34d101
Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-on: http://review.whamcloud.com/15727
Tested-by: Jenkins
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6803 gss: add printf format checking to printerr()

Add printf format checking to the gss error reporting functions
printerr(). Fixup all errors exposed by this change.

Change-Id: I2b7a2d65cb3ee81b11eb6af45297dad2e6cbb796
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: http://review.whamcloud.com/15751
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Sebastien Buisson <sebastien.buisson@bull.net>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6857 tests: TF_FAIL is not initialised

TF_FAIL is initialised in auster only,
this leads to t-f failure "touch: missing file operand"
if the suites run not under auster control.
DEFAULT_SUITES is initialised in acceptance-small only,
this leads to garbage summary message printed.

Patch adds export TF_FAIL to init_test_env() and
skips useless summary printing for the case when suites run
not under acceptance-small control.

Signed-off-by: Elena Gryaznova <elena.gryaznova@seagate.com>
Xyratex-bug-id: MRP-2116
Reviewed-by: Vladimir Saveliev <vladimir.saveliev@seagate.com>
Change-Id: I047aae3b7c6f7d75e98e7862a7f00fd2d75d8b1f
Reviewed-on: http://review.whamcloud.com/15617
Tested-by: Jenkins
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Wei Liu <wei3.liu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6261 gnilnd: kgnilnd_check_rdma_cq race in error path.

It is possible for multiple threads to both be processing a
transaction error that causes the connection to close.
If the close path stalls after changing the conn state to CLOSING,
the tx_done path will put a tx in purgatory since the conn state is
not ESTABLISHED anymore. When the close path continues, it expects to
find that the conn->gnc_mdd_list is empty and assert if it isn't
empty.
Change the error code for the tx_done path so that we don't put this
tx in purgatory since we are closing the connection anyway.

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I4458522f16508eb53d380f62320c65c7bf84657a
Reviewed-on: http://review.whamcloud.com/15439
Tested-by: Jenkins
Reviewed-by: James Shimek <jshimek@cray.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6261 gnilnd: Use kgnilnd_vzalloc to allocate fma blocks.

In low memory situations we may not be able to allocate memory.
vmalloc tries forever to get memory.
Use kgnilnd_vzalloc which uses the GFP_NOFS flag for allocating
memory for fma blocks.

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I18f8564424abfc9c63e675ac98ca61487b5a2b34
Reviewed-on: http://review.whamcloud.com/15438
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6460 llite: clear LLIF_FILE_RESTORING when done

Clear LLIF_FILE_RESTORING if restore done to ensure to start again
to glimpse new attrs.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: I0f2f9532e06965273efa10e3f69e26d00676b8a6
Reviewed-on: http://review.whamcloud.com/14609
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6260 llite: add support for new iter functionality

For the 3.16+ kernels struct file_operations added new read
and write methods; read_iter and write_iter; to use struct
iov_iter as a parameter since it contains all the iovec
data needed. This avoid having the file system managing
iovec data like transversing the iovec page list. Now we
can use supplied iov_iter helper functions that does this
work for us. In later kernels only this back end is
supported. Backported from the upstream lustre client.

-----------------------------------------------------------------------
Linux-commit: b42b15fdad3ebb790250041d1517acebb9bd56d9

lustre: get rid of messing with iovecs

* switch to ->read_iter/->write_iter
* keep a pointer to iov_iter instead of iov/nr_segs
* do not modify iovecs; use iov_iter_truncate()/iov_iter_advance() and
  a new primitive - iov_iter_reexpand() (expand previously truncated
  iterator) istead.
* (racy) check for lustre VMAs intersecting with iovecs kept for now as
  for_each_iov() loop.

Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
-----------------------------------------------------------------------
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Change-Id: I71928beaa7b1c87f3e2c689e1dee96052eaef872
Reviewed-on: http://review.whamcloud.com/15028
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6341 llite: Use ll_dir_getstripe to get default LMVEA

Use ll_dir_getstripe to get default stripeEA in ll_new_node(),
Because ll_getxattr_common requires admin rights for retrieving
default LMVEA (because of trusted- prefix), which might cause
mkdir (from normal user) failure.

If parent does not have default stripeEA, then child should always
be in the same MDT for mkdir. Otherwise MDT should return -EREMOTE,
then client will refresh the default stripe index, and recreate
the object.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I0d8884b78fbc8b8d1930b1133150686e65d20c54
Reviewed-on: http://review.whamcloud.com/13990
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6052 utils: change "lfs mv" to "lfs migrate"

"lfs mv" causes some confusion between mv and migration, change
"lfs mv" to "lfs migrate -m", and mark "lfs mv" deprecated.

Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Change-Id: I72ead14854eb6038ba13fddec81b867bc0542b46
Reviewed-on: http://review.whamcloud.com/13754
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: frank zago <fzago@cray.com>

LU-6836 tests: Skip sanity-quota test 4a for ZFS

sanity-quota test 4a is failing frequently in
review-zfs-part-1 test groups. We are temporarily disabling
this test until a fix is submitted.

Note that LU-6836 is thought to be caused by bad ZFS sync
performance. We should reenable test 4a when ZIL support
is ready (LU-4009).

Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: I07867458e4365c0b1d31654be08e0fef9092c9eb
Reviewed-on: http://review.whamcloud.com/15679
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6769 build: Test for kthread_worker support

The OFED compatibility layer will backport kthread_worker unless
CONFIG_COMPAT_IS_KTHREAD is defined. kthread_worker is available in
the SLES 11 SP3 kernel, but not RHEL 6.5 or RHEL 6.6. This mod adds a
test to check if kthread is available in the kernel and defines
CONFIG_COMPAT_IS_KTHREAD appropriately.

Later versions of MLNX_OFED (2.4, 3.0) require that we similarly
define HAVE_KTHREAD_WORK.

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: Ia89b73e4c8a3ad8549efdea70f0ebd96a1151dba
Reviewed-on: http://review.whamcloud.com/15428
Tested-by: Jenkins
Reviewed-by: Shuichi Ihara <sihara@ddn.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6828 lod: fix memory leak in lod_connect_to_osd

We would replace '$fsname-mdtlov' to '$fsname-MDT0000-osd'
when upgrade from 1.x. But buffer alloced as former length.

Signed-off-by: Yang Sheng <yang.sheng@intel.com>
Change-Id: I8c92e3cb4472527fa5ad5da7f0e4cb498eaccb6c
Reviewed-on: http://review.whamcloud.com/15635
Tested-by: Jenkins
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5597 build: Ensure MOFED Module symvers are used

Change way extra symbols are included in build.  Use
KBUILD_EXTRA_SYMBOLS to directly included Modules.symvers
files instead of buildling Modules.symvers in buildroot
directory.

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I4a6ee59f4e4eed9f878ad52993b8b17426f19d4a
Reviewed-on: http://review.whamcloud.com/15498
Tested-by: Jenkins
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6907 contrib: make gerrit_checkpatch.py less spammy

Currently every checkpatch message is broadast to everybody
on the patch and then to all watchers.
Use the REST API to indicate that on failure the message
should only be sent to the patch owner and in case of success
the emails should not be sent at all.

Change-Id: I65b2017c1cc9558eed3707d81f936acac4af37f5
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/15729
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-6803 gss: add printf format checking to __logmsg{,_gss}()

Add printf format checking to the gss error reporting functions
__logmsg() and __logmsg_gss(). Fixup all errors so discovered.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ie1526e0189f0b4e43d6bd3dcecb53574945f40d1
Reviewed-on: http://review.whamcloud.com/15524
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6713 lmv: lock necessary part of lmv_add_target

Release lmv_init_mutex once the new target is added
into lmv_tgt_desc, so lmv_obd_connect will not be
serialized.

New target should be allowed to added to fld client
lists, so FLD can always choose new added target to
do the FLD lookup request, and also remove some noise
error messages in this process.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I1aa2e31108f2da672b41e2a5c1c544605328ceea
Reviewed-on: http://review.whamcloud.com/15269
Tested-by: Jenkins
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6446 ldiskfs: remove WARN_ON from ldiskfs_orphan_add{del}

RHEL7.1 kernel carefully check i_mutex whether locked in
many places. It will be triggered while ldiskfs_truncate()
was invoked in osd layer. A dead lock would occured if we
just locked i_mutex around it since Lustre using a reverse
order for start journal==>lock i_mutex. Consider Lustre has
own ldlm lock. So just remove such messages in ldiskfs patches.

Signed-off-by: Yang Sheng <yang.sheng@intel.com>
Change-Id: I79520b0a1d013722a5a27e71318416608bc25285
Reviewed-on: http://review.whamcloud.com/14690
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6802 ptlrpc: reset replay cursor on reconnection

In this way, clients can make sure every single replayable ptlrpc
requests will be replayed and replies will be received.

Test-Parameters: envdefinitions=PTLDEBUG=+inode mdtfilesystemtype=ldiskfs mdsfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs clientdistro=el6.6 ossdistro=el6.6 mdsdistro=el6.6 mdscount=2 mdtcount=4 testlist=sanity,sanity,sanity,sanity,sanity,sanity
Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I32daf787d141c84376879fb0fb3b3eb8424f91ad
Reviewed-on: http://review.whamcloud.com/15669
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Hongchao Zhang <hongchao.zhang@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

New tag 2.7.57

Change-Id: I7802951fb411c6f40a2d7ee14ba0759c2036c227
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6872 lov: avoid infinite loop in lsm_alloc_plain()

In lsm_alloc_plain() use a signed loop index to avoid an infinite loop
in the error path.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I084bfadd8a6bf44bbcbe60624e31926ec6cdc04e
Reviewed-on: http://review.whamcloud.com/15644
Tested-by: Jenkins
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6790 build: handle external Intel OFED stack

Besides the Mellanox stack Intel also releases their own
OFED stack. Just like the Mellanox stack this Intel OFED
stack has a compatibility layer to support distributions
kernels. This layer provided stomps over the default
kernel headers so like the Mellanox stack compact-2.6.h
has to be placed before all other headers. The reason
it didn't show up before with the Mellanox stacks is the
Intel stack additionally stomps the pci layer of the
kernel.

Change-Id: If565f25432cf87723b69e5d65e00a7d5d301dc4f
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: http://review.whamcloud.com/15582
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Chris Horn <hornc@cray.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6835 build: add -Wall to CFLAGS for test/ and utils/

Add -Wall to the CFLAGS used for lustre/{test,utils}/. Fixup missing
includes where needed.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I19d0739e0a9b5b079665d5d24d54c6dcaad93b0c
Reviewed-on: http://review.whamcloud.com/15392
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Jenkins
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Chris Horn <hornc@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6501 libcfs: removal last entry of libcfs_netstrfns[]

Currently NID string handling test for the last entry,
and last entry has .nf_type == (__u32) -1. If we ask
for a non existent LND we hit the last entry which then
calls a strlen on a NULL which causes a error. We can
avoid this problem if we just remove the last entry
since it is not used for anything except as a last
entry marker.

Signed-off-by: Frederic Saunier <frederic.saunier@atos.net>
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Change-Id: Ia8e3bea0f01bb5fa78e88bfbca698b0aa0d148ea
Reviewed-on: http://review.whamcloud.com/15424
Tested-by: Jenkins
Reviewed-by: frank zago <fzago@cray.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6261 gnilnd: Clean up cfs abstractions from gnilnd

Running the script contrib/scripts/libcfs_cleanup.sed gets us ready
for cfs changes.

find lnet/klnds/gnilnd -name "*.[ch]" -exec \
sed -i -f contrib/scripts/libcfs_cleanup.sed {} ;

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: Ie7aa1616d33d9f60870f4b8c2a946ce66d3c348c
Reviewed-on: http://review.whamcloud.com/15437
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: James Shimek <jshimek@cray.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6261 gnilnd: Hold shared MDD for gnilnd.

Creating and destroying shared MDDs can cause OS noise. This affects
benchmarks that are sensitive to OS noise like fwq.
Allocate and register some memory so that we always have a shared MDD
available.
Moved kgnilnd_check_kgni_version() into gnilnd.c to simplify
gemini/aries headers.

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: Ibc1f4b9a44035c6fb25e88d30552136486c260d6
Reviewed-on: http://review.whamcloud.com/15436
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: James Shimek <jshimek@cray.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6261 gnilnd: Fix LIBCFS_ALLOC_POST incompatibility.

master has changed LIBCFS_ALLOC to use vzalloc so LIBCFS_ALLOC_POST
does not contain a memset anymore.
kgnilnd_vmalloc() used LIBCFS_ALLOC_POST to zero memory allocated but
master removed the memset in favor of using vzalloc.
Use the __GFP_ZERO flag in (now named) kgnilnd_vzalloc to zero memory
allocated.

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: Ia014c18f13e4e263eccb51d49d375fc4b3bc8b61
Reviewed-on: http://review.whamcloud.com/15435
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: James Shimek <jshimek@cray.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6261 gnilnd: Use trylock for conn mutex.

When converting to thread safe implementation, I missed the need for
the conn mutex to trylock because we may be holding the kgn_peer_conn
lock.
Change conn mutex to use a trylock in kgnilnd_sendmsg_trylock().
Add the module parameter thread_safe.
Disable thread safe on gemini.
In kgnilnd_create_conn(), use NOFS flag when vmalloc the tx_ref_table
to avoid possible hangs because of OOM condition.
Clean up info message for down event.

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I1896c12b421ae35f65d1816bbe3eb5599b664498
Reviewed-on: http://review.whamcloud.com/15434
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: James Shimek <jshimek@cray.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6261 gnilnd: Add pkey module parameter

lustre:19561 changed the pkey to the value reserved in gni_pub.h and
the max_immediate size. These need to be the same on both the service
and compute nodes for gnilnd to establish a connection.
Add the module parameter pkey so that a node running a dev version of
software will be able to connect to a node running an older version
with appropriate changes to modprobe.conf.
With this change, add the following parameters to
IMAGE_PATH/compute/etc/modprobe.conf:

options kgnilnd max_immediate=2048
options kgnilnd pkey=0xa3579

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I08ea2d66e745c37f41931a53e599bc572664dd9a
Reviewed-on: http://review.whamcloud.com/15433
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: James Shimek <jshimek@cray.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6261 gnilnd: Add ability to bind scheduler threads to cpus.

Added the module parameter thread_affinity which when enabled, will
bind the kgnilnd_sd_xx threads to cpus 1 through xx.
thread_affinity defaults to disabled.
Testing shows that enabling thread affinity will allow us to get
better small message performance with more scheduler threads.

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I5298c25cc21a1a856e4d84f1bca4f484449cd055
Reviewed-on: http://review.whamcloud.com/15432
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: James Shimek <jshimek@cray.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6679 ldlm: do not send blocking ast for group locks

Group locks are acquired and released manually on client so
it doesn't make sense to send blocking AST to client when
there is an incompatible lock enqueued.

Currently client will set CBPENDING on the group lock when
it receives a blocking AST. Having the CBPENDING flag set
will make ldlm_lock_match to fail and there will be two
group locks granted on the same resource on the client.

Signed-off-by: Li Dongyang <dongyang.li@anu.edu.au>
Change-Id: I7af89e957528b3ed9771d86243ac8271084ee81f
Reviewed-on: http://review.whamcloud.com/15119
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6847 kernel: kernel update RHEL 6.6 [2.6.32-504.30.3.el6]

Update RHEL6.6 kernel to 2.6.32-504.30.3.el6

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I4f60d43dea248ce57290579c021838d0f731c332
Reviewed-on: http://review.whamcloud.com/15605
Tested-by: Jenkins
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6186 tests: avoid errors in using >&- bash close syntax

Seems that recent EL7 bash version include changes in handling
the ">&-" redirection/close syntax, which cause "Bad file
descriptor" error. Use >/dev/null to avoid this.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: Iac3e54ef69f0c5c241b87a9e307e059974a4caf3
Reviewed-on: http://review.whamcloud.com/13857
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6822 nrs: remove obsolete assertion in nrs_orr_start()

kmem_cache_destroy() doesn't return any value, so we should remove
this obsolete assertion, otherwise LBUG will happen when orr policy
is enabled.

Signed-off-by: Emoly Liu <emoly.liu@intel.com>
Change-Id: I240e7bd660a6960b1e29da3575388a966ce8dca9
Reviewed-on: http://review.whamcloud.com/15540
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Hongchao Zhang <hongchao.zhang@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6540 test: fix no %s specifier for the file

The %m prints strerror(errno), but there is no %s specifier for the file.

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: Id9a8f486521b94e98986140fa374c40e08a00dac
Reviewed-on: http://review.whamcloud.com/14815
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6442 utils: "-G <value>" can be passed through mkfs options

mkfs.lustre util checks if parametrs already contain "flex_bg"
option (number of block groups that will be packed together
to create a larger virtual block group)  and adds this option
if it doesn't exists. But second parameter "-G" (number of block
groups that will be packed together to create a larger virtual
block group) can be added twice: one passed through mkfs options
and another one - default value. In this case this parameter
is not changed actually and default value is applied.

This patch adds extra check. Default option "-G" is added
only if no "-G" option passed through mkfs options.

Xyratex-bug-id: MRP-2046
Signed-off-by: Artem Blagodarenko <artem_blagodarenko@xyratex.com>
Reviewed-by: Sergey Cheremencev <sergey_cheremencev@xyratex.com>
Tested-by: Elena Gryaznova <elena.gryaznova@seagate.com>
Reviewed-by: Vitaly Fertman <vitaly.fertman@seagate.com>
Change-Id: I14fb5f8d10fa369428efcbbcb4f638f388979818
Reviewed-on: http://review.whamcloud.com/14400
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6198 test: fix large-lun test_1, test_2 defects

setupall()->start ost called at the end of large-lun.sh
fails to start the ost devices which leads test_1() and
test_2() fail if run separately.

The reasons are:
1) osts data are destroyed by test_1()->llverdev
2) osts are reformatted by test_2, but mds config not changed
Patch fixes the following:
- format Lustre at the end of test_1() and test_2()
- add the "REFORMAT" option check to avoid killing
filesystems that do not want a reformat.

Signed-off-by: Elena Gryaznova <elena.gryaznova@seagate.com>
Xyratex-bug-id: MRP-1984
Reviewed-by: Vladimir Saveliev <vladimir.saveliev@seagate.com>
Change-Id: I7f1465d4777b7a5d5538e019226334dc090e3f50
Reviewed-on: http://review.whamcloud.com/13601
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6261 gnilnd: Changes for small message rate improvment

Change number of threads on Aries service nodes to 7 for better
performance of small messages.
Change the max_immediate value to 8k instead of 2k. FMA has better
performance up to 8k.
lnet passes an niov of 256 reflecting the size of the router buffer
number of pages in the buffer pool when receiving data from the ib
interface when the message size is over one page. Adjust the niov to
reflect the message size/offset for sends.
Use the cookie defined in gni_pub.h instead of value from gnilnd.h.

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I84270653efcff56f06da7de7fb9674ae319800dd
Reviewed-on: http://review.whamcloud.com/15431
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: James Shimek <jshimek@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6261 gnilnd: Thread-safe optimizations.

Take advantage of improved gni threading.
Do not use gnd_cq_mutex lock for kgni versions that support thread
safe gni api.
Check if version is greater than code rev 0xb9 and use locking of
smsg and rdma on a per connection instead of the global cq lock.
Changed gnc_tx_seq and gnc_rx_seq to atomics.
Added gnc_smsg_mutex and gnc_rdma_mutex per conn to protect the lists
that the messages are placed on.

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: Ic03f1877ab7b9632ca5517cd74f7e7fa25ba171b
Reviewed-on: http://review.whamcloud.com/15430
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: James Shimek <jshimek@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6668 test: regression tests for NRS TBF policy

This patch adds fundamental regression tests for NRS TBF policy.

Signed-off-by: Li Xi <lixi@ddn.com>
Change-Id: I37371845c2920f65d38df03bc42b40fd9dea5bb0
Reviewed-on: http://review.whamcloud.com/15102
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6661 test: add version check for tests

add version check for sanity 27e and 162c, so that interop test
won't fail.

Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Change-Id: I45cc25d71e2e7b361455c9dcde1c9a3e1c797e9b
Reviewed-on: http://review.whamcloud.com/15521
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6683 osd: declare enough credits for generating LMA

Usually, the LMA EA is set first after the object created. But if the
system is upgraded from 1.8 release or older, then there is no LMA EA
stored in the object, and then generating LMA EA for the object maybe
not the first EA in the object's inode.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I2bdbf5ba56db1ea08edf0a8a4d724df4ad97e071
Reviewed-on: http://review.whamcloud.com/15361
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6690 tests: start nfsserver service for SLES

The NFS server service name in SLES distro is "nfsserver"
instead of "nfs". This patch fixes setup-nfs.sh to start
nfsserver service for SLES.

Test-Parameters: alwaysuploadlogs \
envdefinitions=EXCEPT=compilebench \
mdtfilesystemtype=ldiskfs mdsfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs \
clientdistro=sles11sp3 ossdistro=sles11sp3 mdsdistro=sles11sp3 \
mdtcount=1 testlist=parallel-scale-nfsv3,parallel-scale-nfsv4

Test-Parameters: alwaysuploadlogs \
mdtfilesystemtype=ldiskfs mdsfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs \
clientdistro=el6.6 ossdistro=el6.6 mdsdistro=el6.6 \
mdtcount=1 testlist=parallel-scale-nfsv3,parallel-scale-nfsv4

Signed-off-by: Jian Yu <jian.yu@intel.com>
Change-Id: Ifb200400c0fa072fee6f563431be0429e2c79890
Reviewed-on: http://review.whamcloud.com/15149
Reviewed-by: Wei Liu <wei3.liu@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6834 lod: fix idx_array overwritten

Compare index with stripe_count -1 , otherwise it will
overwrite the idx_array[] in lod_prep_md_striped_create().

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: Ib116821fb9e9b800752d760c262ded725c55ce0e
Reviewed-on: http://review.whamcloud.com/15572
Tested-by: Jenkins
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6529 ldlm: reclaim granted locks defensively

To avoid ldlm lock exhausting server memory, two global parameters:
ldlm_watermark_low & ldlm_watermark_high are used for reclaiming
granted locks and rejecting incoming enqueue requests defensively.

ldlm_watermark_low: When the amount of granted locks reaching this
threshold, server start to revoke locks gradually.

ldlm_watermark_high: When the amount of granted locks reaching this
threshold, server will return -EINPROGRESS to any incoming enqueue
request until the lock count is shrunk below the threshold again.

ldlm_watermark_low & ldlm_watermark_high is set to 20% & 30% of the
total memory by default. It is tunable via proc entry, when it's set
to 0, the feature is disabled.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I2fab39ac0ab6f269b7f1a40f3e08b8a51807cc69
Reviewed-on: http://review.whamcloud.com/14931
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2778 tests: UNEXCEPT conf-sanity/57b

Due to fixes for LU-2059 this test should now pass consistantly on
ZFS.

Test-Parameters: mdsfilesystemtype=zfs mdtfilesystemtype=zfs ostfilesystemtype=zfs testlist=conf-sanity
Test-Parameters: mdsfilesystemtype=zfs mdtfilesystemtype=zfs ostfilesystemtype=zfs testlist=conf-sanity
Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Ia9e567a5692109458b8cbf9fc034d571c52ea130
Reviewed-on: http://review.whamcloud.com/10357
Tested-by: Jenkins
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6541 utils: fix potential memory leaks with realloc()

The semantics of realloc are to return NULL when the allocation fails,
but won't free the original allocation:

"The realloc() function returns a pointer to the newly allocated memory,
which is suitably aligned for any built-in type and may be different
from ptr, or NULL if the request fails. If size was equal to 0, either
NULL or a pointer suitable to be passed to free() is returned. If
realloc() fails, the original block is left untouched; it is not freed
or moved."

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: Ib348acad13c866a2904e023ac5822f23d13b38ca
Reviewed-on: http://review.whamcloud.com/14814
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6539 quota: fix incorrect format string used for short type

The %u in format string requires 'unsigned int' but the argument type
is 'short'.

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: Ie84b5f2d46809eb1a840d07ffa0b45f6ada1379e
Reviewed-on: http://review.whamcloud.com/14810
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6245 libcfs: remove last of user land code in libcfs module

Currently user land code still exist in libcfs source for
workitems and some of the linux specific debug code. We
can now safely remove the last bit of this code.

Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Change-Id: Id137071f52cbce3a4c6de323b5412976fb687c54
Reviewed-on: http://review.whamcloud.com/15197
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: frank zago <fzago@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5829 libcfs: remove unnecessary EXPORT_SYMBOL

A lot of symbols don't need to be exported at all because they are
only used in the module they belong to.

Signed-off-by: frank zago <fzago@cray.com>
Change-Id: Ie386a75bb8d9b5b77360ea2c714aba886dd8f780
Reviewed-on: http://review.whamcloud.com/13319
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6800 obdclass: change spinlock of key to rwlock

Most of the time, keys are never changed. So rwlock might be
better for the concurrency of key read.

Signed-off-by: Li Xi <lixi@ddn.com>
Change-Id: I5c86c7333884a44ec8e9017d532d6eea5a119cfa
Signed-off-by: Gu Zheng <gzheng@ddn.com>
Reviewed-on: http://review.whamcloud.com/15558
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6837 update: re-lookup the dtrq in the replay list.

In insert_update_records_to_replay_list(), if the
dtrq request has been added to the list by another
thread, it should lookup the request in the list
again, otherwise it will cause the following sub req
lookup panic, because the current dtrq was just freed.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I766cddbe5abcd96e97a611985032208585573b43
Reviewed-on: http://review.whamcloud.com/15573
Tested-by: Jenkins
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4340 tests: Fix test_69 of conf-sanity test

Since 100% inodes were used with default value of MDSSIZE and OSTSIZE,
test_69 of conf-sanity test was failing with ENOSPC error.

Use decreased value of num_create to create 20000*3 files.

Also added umount of client before OST replace operation so that
eviction/recovery of client is not seen.

Seagate-bug-id: MRP-1698
Signed-off-by: Ashish Purkar <ashish.purkar@seagate.com>
Change-Id: I4568cfb3d259ae248f26d5463cd3291db60f2d5b
Reviewed-on: http://review.whamcloud.com/15487
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6660 rename: DNE2 should return -EXDEV upon remote rename

DNE2 MDS should return -EXDEV upon remote rename, so that old
client can do rename with copy and delete, instead of fail
with -EREMOTE.

Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Change-Id: I68e8e99259065922f31bee5343be309380715674
Reviewed-on: http://review.whamcloud.com/15323
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>

LU-5946 lprocfs: free expired jobstats after /proc read

Currently, jobstats for expired jobids are only freed when a new
jobid arrives.  If no new jobids have arrived in a long time, the
expired jobstats won't be freed even though the /proc/.../job_stats
file is read repeatedly.  This shouldn't be a problem for steady job
submission patterns, but if there is a big gap in new jobid arrival
then expired jobids shouldn't be left around such a long time.

Free expired stats in lprocfs_jobstats_seq_release() after job_stats
is read, and move normal expiry to the arrival of new jobids instead
of on every stat to reduce overhead.  If jobs arrive frequently then
expired jobs will also be freed frequently, and if jobs arrive
slowly then there wan't be many expired jobs accumulating.

Rename job_iter_callback() to job_cleanup_iter_callback() for name
consistency with its caller and to make it more clear what it is for.

Instead of just passing a "bool force" param to lprocfs_job_cleanup(),
pass the number of seconds before which stats should be expired.
That allows lprocfs_job_cleanup() to be used at all callers instead
of having to have custom iterators in several places.

Avoid having multiple threads do cleanup of the jobstats concurrently.
This is not itself dangerous, since hash locking avoids races, but it
is inefficient to have multiple threads contending on the hash locks.

Add comment blocks for modified functions.

Remove some LASSERT() calls and replace with checks and error returns,
preferably at initialization time rather than during later usage.

Fix miscellaneous coding style issues.

Add a test for jobstats expiry after /proc read, and add a DNE test.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: Icfd65ed7cd7beb26a9b566667e2686afe23ebbe5
Reviewed-on: http://review.whamcloud.com/12821
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-5638 tests: Skip sanity-quota tests for ZFS

sanity-quota tests 11, 33, 34 and 35 are failing frequently
in review-zfs-part-1 test groups. We are temporarily disabling
these tests until a fix is submitted.

Note that LU-5638 is thought to be caused by
LU-2435 inode accounting in osd-zfs is racy.

Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: I3eaa3f89ca82123f8578b54006451143ce005349
Reviewed-on: http://review.whamcloud.com/15590
Tested-by: Jenkins
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6325 ptlrpc: make ptlrpcd threads cpt-aware

On NUMA systems, the placement of worker threads relative to the
memory they use greatly affects performance. The CPT mechanism can be
used to constrain a number of Lustre thread types, and this change
makes it possible to configure the placement of ptlrpcd threads in a
similar manner.

To simplify the code changes, the global structures used to manage
ptlrpcd threads are changed to one per CPT. In particular this means
there will be one ptlrpcd recovery thread per CPT.

To prevent ptlrpcd threads from wandering all over the system, all
ptlrpcd thread are bound to a CPT. Note that some CPT configuration
is always created, but the defaults are not likely to be correct for
a NUMA system. After discussing the options with Liang Zhen we
decided that we would not bind ptlrpcd threads to specific CPUs,
and rather trust the kernel scheduler to migrate ptlrpcd threads.

With all ptlrpcd threads bound to a CPT, but not to specific CPUs,
the load policy mechanism can be radically simplified:

- PDL_POLICY_LOCAL and PDL_POLICY_ROUND are currently identical.
- PDL_POLICY_ROUND, if fully implemented, would cost us the locality
  we are trying to achieve, so most or all calls using this policy
  would have to be changed to PDL_POLICY_LOCAL.
- PDL_POLICY_PREFERRED is not used, and cannot be implemented without
  binding ptlrpcd threads to individual CPUs.
- PDL_POLICY_SAME is rarely used, and cannot be implemented without
  binding ptlrpcd threads to individual CPUs.

The partner mechanism is also updated, because now all ptlrpcd
threads are "bound" threads. The only difference between the various
bind policies, PDB_POLICY_NONE, PDB_POLICY_FULL, PDB_POLICY_PAIR, and
PDB_POLICY_NEIGHBOR, is the number of partner threads. The bind
policy is replaced with a tunable that directly specifies the size of
the groups of ptlrpcd partner threads.

Ensure that the ptlrpc_request_set for a ptlrpcd thread is created on
the same CPT that the thread will work on. When threads are bound to
specific nodes and/or CPUs in a NUMA system, it pays to ensure that
the datastructures used by these threads are also on the same node.

Visible changes:

* ptlrpcd thread names include the CPT number, for example
  "ptlrpcd_02_07". In this case the "07" is relative to the CPT, and
  not a CPU number.

Tunables added:

* ptlrpcd_cpts (string): A CPT string describing the CPU partitions
  that ptlrpcd threads should run on. Used to make ptlrpcd threads
  run on a subset of all CPTs.

* ptlrpcd_per_cpt_max (int): The maximum number of ptlrpcd threads
  to run in a CPT.

* ptlrpcd_partner_group_size (int): The desired number of threads
  in each ptlrpcd partner thread group. Default is 2, corresponding
  to the old PDB_POLICY_PAIR. A negative value makes all ptlrpcd
  threads in a CPT partners of each other.

Tunables obsoleted:

* max_ptlrpcds: The new ptlrcpd_per_cpt_max can be used to obtain the
  same effect.

* ptlrpcd_bind_policy: The new ptlrpcd_partner_group_size can be used
  to obtain the same effect.

Internal interface changes:

* pdb_policy_t and related code have been removed. Groups of partner
  ptlrpcd threads are still created, and all threads in a partner
  group are bound on the same CPT. The ptlrpcd threads bound to a
  CPT are typically divided into several partner groups. The partner
  groups on a CPT all have an equal number of ptlrpcd threads.

* pdl_policy_t and related code have been removed. Since ptlrpcd
  threads are not bound to a specific CPU, all the code that avoids
  scheduling on the current CPU (or attempts to do so) has been
  removed as non-functional. A simplified form of PDL_POLICY_LOCAL
  is kept as the only load policy.

* LIOD_BIND and related code have been removed. All ptlrpcd threads
  are now bound to a CPT, and no additional binding policy is
  implemented.

* ptlrpc_prep_set(): Changed to allocate a ptlrpc_request_set
  on the current CPT.

* ptlrpcd(): If an error is encountered before entering the main loop
  store the error in pc_error before exiting.

* ptlrpcd_start(): Check pc_error to verify that the ptlrpcd thread
  has successfully entered its main loop.

* ptlrpcd_init(): Initialize the struct ptlrpcd_ctl for all threads
  for a CPT before starting any of them. This closes a race during
  startup where a partner thread could reference a non-initialized
  struct ptlrpcd_ctl.

Signed-off-by: Olaf Weber <olaf@sgi.com>
Change-Id: I3ac40ea56f9c792c3e7c36967e2e1f20105c566c
Reviewed-on: http://review.whamcloud.com/13972
Tested-by: Jenkins
Reviewed-by: Grégoire Pichon <gregoire.pichon@bull.net>
Reviewed-by: Stephen Champion <schamp@sgi.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6816 utils: remove libzfs_load_module() call

Remove libzfs_load_module() call from the OSD.  The function is no
longer exported by libzfs2 in ZFS master, as it was intended as an
internal function to begin with.  libzfs_init(), which is already
being used to set up the OSD, loads the module if necessary.

The intent of the call to libzfs_load_module apparently was to make
zfs_init() fail silently if ZFS was improperly installed (e.g. the
userspace libraries were missing), but fail with an error message if
ZFS initialization failed.  However it's not clear how this could ever
work.

Handling such error conditions will be dealt with in a separate ticket
as it's a larger concern.

Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Change-Id: I97a2de7cd1a32c8251d805192aaf557f48840584
Reviewed-on: http://review.whamcloud.com/15539
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>

LU-6602 osp: change lgh_hdr_lock to mutex

Change lgh_hdr_lock from spinlock to mutex, because in
osp_md_write_obj(), it might be blocked by memory
allocation.

Reorganize the llog_cache_rec process, so lgh_hdr_mutex
will protect both setbit and update llog header, to avoid
conflicts between add records and cancel llog records.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: Ic12d7d768546c67a858cb235929de8ae6aeaa1aa
Reviewed-on: http://review.whamcloud.com/15274
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6467 test: skip s-l test_9 for interop with b2_5

Master uses layout LFSCK for sanity-lfsck test_9, that is not
support old version, should be skipped.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I41497b27d669813d321208e291792f200f7ac863
Reviewed-on: http://review.whamcloud.com/14500
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-4045 mdd: use same attr to declare setattr

in mdd_attr_set, different lu_attr ("la" and "la_copy") is used
for declaring "setattr" and the "setattr" itself, the "LA_MODE"
could be added to "la_copy" by "mdd_fix_attr" in some case and
cause the "setxattr" in "mdd_attr_set_internal" is processed
without declaration.

Change-Id: Ic4441e0c72e8d8209d8f5e60f71ab09dc7987163
Signed-off-by: Hongchao Zhang <hongchao.zhang@intel.com>
Reviewed-on: http://review.whamcloud.com/14566
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6794 nfs: ASSERTION( atomic_read(&d->ld_ref) == 0 ) failed

ll_iget_for_nfs() can call unbalanced iput() causing memory
leaks. This patch removes this iput() call.

Also, avoid unhashing disconnected dentries in
d_lustre_invalidate(), which is another source of memory
leaks.

One of the symptoms of the leak is the following crash pattern:
<0>LustreError: 14812:0:(lu_object.c:1251:lu_device_fini())
ASSERTION( atomic_read(&d->ld_ref) == 0 ) failed: Refcount is 1
<0>LustreError: 14812:0:(lu_object.c:1251:lu_device_fini()) LBUG
<4>Pid: 14812, comm: umount
<4>
<4>Call Trace:
<4> [<ffffffffa11bc895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
<4> [<ffffffffa11bce97>] lbug_with_loc+0x47/0xb0 [libcfs]
<4> [<ffffffffa1458a48>] lu_device_fini+0xb8/0xc0 [obdclass]
<4> [<ffffffffa08e9ab2>] lovsub_device_free+0x52/0x220 [lov]
<4> [<ffffffffa145c64e>] lu_stack_fini+0x7e/0xc0 [obdclass]
<4> [<ffffffffa146356e>] cl_stack_fini+0xe/0x10 [obdclass]
<4> [<ffffffffa08bc1a8>] lov_device_fini+0x58/0x120 [lov]
<4> [<ffffffffa145c619>] lu_stack_fini+0x49/0xc0 [obdclass]
<4> [<ffffffffa146356e>] cl_stack_fini+0xe/0x10 [obdclass]
<4> [<ffffffffa0e1279d>] cl_sb_fini+0x6d/0x190 [lustre]
<4> [<ffffffffa0dd34bc>] ll_put_super+0x1bc/0x11e0 [lustre]
<4> [<ffffffff811cd0f2>] ? fsnotify_clear_marks_by_inode+0x32/0xf0
<4> [<ffffffff811a59df>] ? destroy_inode+0x2f/0x60
<4> [<ffffffff811a5eac>] ? dispose_list+0xfc/0x120
<4> [<ffffffff811a62a6>] ? invalidate_inodes+0xf6/0x190
<4> [<ffffffff8118b35b>] generic_shutdown_super+0x5b/0xe0
<4> [<ffffffff8118b446>] kill_anon_super+0x16/0x60
<4> [<ffffffffa144e7ba>] lustre_kill_super+0x4a/0x60 [obdclass]
<4> [<ffffffff8118bbe7>] deactivate_super+0x57/0x80
<4> [<ffffffff811aabef>] mntput_no_expire+0xbf/0x110
<4> [<ffffffff811ab73b>] sys_umount+0x7b/0x3a0
<4> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b

Xyratex-bug-id: MRP-2414
Signed-off-by: Andrew Perepechko <andrew.perepechko@seagate.com>
Change-Id: Ie8bc428014f3897dc108568f55db9a9ed93c932b
Reviewed-on: http://review.whamcloud.com/15480
Tested-by: Jenkins
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4416 ldiskfs: enable support for SLES12

This patch adds support for the 3.12-based SLE12 kernel. Additonally
this work separates out the basic needed patches for the upstream
3.12 kernel which can be shared with other distros as well as future
upstream kerrnels. This will help with book keeping ldiskfs patches
that need to be pushed upstream.

kernel version SLES12 [3.12.43-52.6]

Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Change-Id: I2b8d83cddf8fd27617657348ea6d7c3a48e9472c
Signed-off-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-on: http://review.whamcloud.com/10165
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6805 ptlrpc: use smp unsafe at_init only for initialization

at_init() is not smp safe, so it is not supposed to be used anywhere
but in at initialization.
Add at_reinit() - safe version of at_init().

Xyratex-bug-id: MRP-2726
Signed-off-by: Vladimir Saveliev <vladimir.saveliev@seagate.com>
Change-Id: I90f872bd217d1e8008f8d35bb4606d168c9bf2ad
Reviewed-on: http://review.whamcloud.com/15522
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Chris Horn <hornc@cray.com>

LU-6747 osd-zfs: initialize obd_statfs in osd_statfs()

The osd-zfs osd_statfs() method wasn't initializing all of the fields
of struct obd_statfs, and in some cases could be returning random
data to the client.  For most of the fields this doesn't matter, but
the os_state field could contain OS_STATE_READONLY and prevent the
MDS from allocating objects there.

Reorder the assignment of struct obd_statfs fields to be in struct
order so that it is more clear which fields have already been set.

Also fixup osd-ldiskfs obd_statfs() to initialize os_maxbytes as
osd-zfs does, even though it currently isn't used.

Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: If4ec6b5fc9c31676267e63be34d00275b4ac8dfb
Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-on: http://review.whamcloud.com/15346
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-5092 nodemap: convert member hash to a list

The members of a nodemap were stored in a hash, but they were only
iterated against, not accessed by key. This patch converts the hash
to a list in order to simplify things.

Signed-off-by: Kit Westneat <kit.westneat@gmail.com>
Change-Id: Ic1df4035d1c53a7c7966c4c528d20cefc7258d0e
Reviewed-on: http://review.whamcloud.com/14885
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6775 ldlm: reduce mem footprint of ldlm_resource

- Removing the unused lr_most_restr, reduced 4 bytes;
- Allocating lr_itree only for LDLM_EXTENT resource, reduced
  120 bytes;
- Moving fields around to eliminate holes, eliminated 3 holes,
  reduced 4 bytes;
- Making lr_contention_time and lr_lvb_inode into union,
  reduced 8 bytes;

Reduced 136 bytes in total.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I232a2669d9f5eaee35eb9ff8313d86d4ac297bc1
Reviewed-on: http://review.whamcloud.com/15485
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6766 utils: remove llapi_quotachown() and cb_quotachown()

The SYS_chown syscall is not available on arm64, but the functions
llapi_quotachown() and cb_quotachown() are not needed anymore and
should have been removed in the patch
http://review.whamcloud.com/4022

Change-Id: I11fa4ab901327c1c8bcfb7e6dfe14a955e2f2618
Signed-off-by: Arnaud Guignard <arnaud.guignard@cea.fr>
Reviewed-on: http://review.whamcloud.com/15396
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6755 kernel: kernel update RHEL7.1 [3.10.0-229.7.2.el7]

Update RHEL7.1 kernel to 3.10.0-229.7.2.el7.

Signed-off-by: Yang Sheng <yang.sheng@intel.com>
Change-Id: Ie922ed7c04394d4ffb02d28fd213c8a1da884bdd
Reviewed-on: http://review.whamcloud.com/15388
Tested-by: Jenkins
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>