LU-5047 tests: correct cleanup files in sanity.sh

Don't remove all files from Lustre share. Create files that will be
cleaned up at the end tests session in check_and_cleanup_lustre().

Fix cleanup for test_54c.

Luster-Commit: c42bb35bc32f472d69276475de3cd186ca5f5247
Lustre-Review: http://review.whamcloud.com/10291

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: I23624b9d14e7de3423b82b4a5b8508c4820c55d4
Reviewed-on: http://review.whamcloud.com/10291
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/10633

LU-5132 tests: Add version check to sanity/160c

Account for what versions of Lustre this test should work for.
This test was added in http://review.whamcloud.com/9723 and that
was included in 2.5.58 and prior to 2.5.2 in the b2_5 branch.

Lustre-change: http://review.whamcloud.com/10557

Test-Parameters: alwaysuploadlogs envdefinitions=ONLY=160c \
ossjob=lustre-b2_4 mdsjob=lustre-b2_4 ossbuildno=73 mdsbuildno=73 \
testlist=sanity

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Signed-off-by: Jian Yu <jian.yu@intel.com>
Change-Id: I560a9900de983b06963d49946e9188649dd5e7fc
Reviewed-on: http://review.whamcloud.com/10558
Tested-by: Jenkins
Reviewed-by: Patrick Farrell <paf@cray.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3386 lproc: improve osc/mdc "imports" connect data

Improve /proc/fs/lustre/{osc,mdc}/*/import files to print
the struct obd_connect_data data fields as "connect_data:"
in addition to the "connect_flags:" field.
Lustre-commit: 62660c799be886e557ae0426126fc3e4b61927d4
Lustre-change: http://review.whamcloud.com/6588
Also, this patch includes the fix in
http://review.whamcloud.com/#/c/10344

Signed-off-by: Liu Ying <emoly.liu@intel.com>
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: Iabf2231341f3d674960626fb7230e2ba5cf4f113
Reviewed-on: http://review.whamcloud.com/9264
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5133 tests: Add version check in sanity/238

Fix for LU-4659 was not included until after 2.5.57 and after 2.5.1.
Ensure MDS is running one of these valid branches before allowing the
test to run.

Lustre-change: http://review.whamcloud.com/10559

Test-Parameters: alwaysuploadlogs envdefinitions=ONLY=238 \
ossjob=lustre-b2_4 mdsjob=lustre-b2_4 ossbuildno=73 mdsbuildno=73 \
testlist=sanity

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Signed-off-by: Jian Yu <jian.yu@intel.com>
Change-Id: I25787874866b38f290b21baf76a09a2aec3dc7f4
Reviewed-on: http://review.whamcloud.com/10560
Tested-by: Jenkins
Reviewed-by: Patrick Farrell <paf@cray.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5100 llite: set dir LOV xattr length variable

the LOV xattr of directory could be either lov_user_md_v1
(size is 32) or lov_user_md_v3 (size is 48), then the actual
size of the LOV xattr should be return.

this patch also changes lod_verify_stripe to make it to accept
lov stripe from user and it's "pattern" is "LOV_PATTERN_RAID0".

Change-Id: I1ccba6da19aef0b7464c43b1b50871df44ce0eb7
Signed-off-by: Hongchao Zhang <hongchao.zhang@intel.com>
Reviewed-on: http://review.whamcloud.com/10453
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: jacques-Charles Lafoucriere <jacques-charles.lafoucriere@cea.fr>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/10575

LU-4780 lnet: NI shutdown may loop forever

lnet_shutdown_lndnis() may enter endless loop if there is a busy NI,
this is injected by LNet SMP improvements. It's fixed in this patch.

Lustre-change: http://review.whamcloud.com/9706
Lustre-commit: 3efa31eacc612261ea3019fbd7243e573714042c

Signed-off-by: Liang Zhen <liang.zhen@intel.com>
Reviewed-by: Isaac Huang <he.huang@intel.com>
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Change-Id: If6bf6a8362d8808cd924d6fe79dbe43151f41f30
Reviewed-on: http://review.whamcloud.com/10489
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-4102 doc: recommend newer e2fsprogs version

Recommend newer e2fspogs version for 2.5.2 release so that it gets
the latest fixes.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: Ica1e47b4242228334f9d673ee5159471f4e1ca8c
Reviewed-on: http://review.whamcloud.com/10561
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4573 tests: check all MDTs for open files

sanityn.sh test_76 verifying mdt.*.exports.*.open_files content
did not check all of the MDTs. It was broken by commit 370de927fc5,
which created all directories striped across all MDTs by default.

There is actually no need to special-case striped or remote
directories in this test, and it should be possible to just check
for open files on all MDTs, and this also simplifies the test.

Lustre-change: http://review.whamcloud.com/10107
Lustre-commit: fed7a99441c2dd078a671cf5b3dcaf27996bf448

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Signed-off-by: Jian Yu <jian.yu@intel.com>
Change-Id: I2b7b434052946c8d84bffef1fc7358cc29b00b75
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-on: http://review.whamcloud.com/10499
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2524 test: Modify tdir to be single directory

Currently, the tdir variable is a directory with a subdirectory.
This requires the '-p' option when calling mkdir. We've made tdir
be a single directory so calls to mkdir and test_mkdir do not
require the '-p' option in most cases.

tdir was changed from d0.${TESTSUITE}/d${base} to
d${testnum}.${TESTSUITE} and tfile was changed from
f.${TESTSUITE}.${testnum} to f${testnum}.${TESTSUITE}. Now tdir and
tfile are consistent in their format and the call to remove files
and directories at the beginning of many test scripts will remove
these files and directories.

Once this patch lands, we can remove the "-p" option from many
of the calls to mkdir.

Lustre-commit: a98d44e0439ad5b20d0d7f273b29e6ea990312f0
Lustre-change: http://review.whamcloud.com/8123

LU-2524 tests: run sanity test_51ba in test_51b dir

Run the test_51ba directory cleanup in the same directory as the
test_51b subtest created its subdirectories.

Lustre-commit: 47c581b8155f865ade5c9d067f85930146361c2a
Lustre-change: http://review.whamcloud.com/9021

Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: Ib49d7102a49ff6b5f3ec539a5b2f2f5186231a04
Reviewed-on: http://review.whamcloud.com/8123
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-on: http://review.whamcloud.com/10349
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4830 tests: only deactivate MDTs of Lustre FSNAME

Problem has been discovered using a very specific copytool Lustre
root/archive usage. All MDCs, including archive one, were
deactivated.

Lustre-commit: 09cc4025440f7e4f9f887f4a2a9bdceef0da7689
Lustre-change: http://review.whamcloud.com/9843

Change-Id: Ifab4bd5a7d6a88ae0e0b95d5a6774dbfed8822e3
Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Reviewed-on: http://review.whamcloud.com/10232
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4676 hsm: Fix return value error of ct_run()

ct_run() returns 0 after find out that fs_name is invalid.
lhsmtool_posix will exists with 0 after that error happens.
That might mislead scripts calling lhsmtool_posix.

Lustre-change: http://review.whamcloud.com/9412
Lustre-commit: d5ad6ff819e635c6ba337586cf70698b7728ff7f

Change-Id: Id182e989011e02104b5960c6692995825d939ad6
Signed-off-by: Li Xi <lixi@ddn.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: jacques-Charles Lafoucriere <jacques-charles.lafoucriere@cea.fr>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/10234

LU-4852 osc: osc_extent_truncate()) ASSERTION( !ext->oe_urgent ) failed

The bug was caused by race between truncate & fsync.
osc_extent_wait() doesn't takes into account oe_trunc_pending
during setting oe_urgent. The race arises after
osc_object_unlock().
osc_extent_wait() should ignore extents with oe_trunc_pending
while waiting for OES_INV. osc_cache_truncate_end() will set
oe_urgent and call osc_io_unplug_async()

Change-Id: I6f4c8420faa67bf4b14b1c5c2bc37fa282a7b072
Xyratex-bug-id: LELUS-239
Signed-off-by: Andriy Skulysh <Andriy_Skulysh@xyratex.com>
Reviewed-on: http://review.whamcloud.com/10204
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4704 test: enable acl 974 and 2561 series test

Add acl/{974,974_remote,2561,2561_zfs}.test to the
lustre/tests/Makefile.am nobase_noinst_DATA list, and fix acl/run
test to fail if the test scripts are missing.
acl/2561_zfs.test is a newly created test file for zfs. Since zfs
allocates a 512 byte dnode for empty file, its "ls -s" result is
different from ldiskfs'.

Lustre-change: http://review.whamcloud.com/9541
Lustre-commit: 1116d8a6233e307abab579e2ee0a63b450087413

Signed-off-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Change-Id: Ifba5b76fa7302f8b129b979aeccf3844f4fe334d
Reviewed-on: http://review.whamcloud.com/10512
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4811 mdt: remove eadatasize check in xattr packing

Remove client eadatasize check in mdt xattr packing because
client can handle -EOVERFLOW.

Originally part of Lustre-change: http://review.whamcloud.com/6339

Test-Parameters: clientjob=lustre-master clientbuildno=1963
Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I47536fa0926804630315e5b78da852f6239497fd
Reviewed-on: http://review.whamcloud.com/9812
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4403 mds: extra lock during resend lock lookup.

1. If the request does not require open lock, MDT does not
need to search the lock in exp_hash_lock list, because the MDT
will release the lock anyway, and it can always re-enqueue the
request during resend.

2. Lock the resource when the MDS is trying to lookup the lock for
resend request(in mdt_intent_fixup_resent). Otherwise, if the
lock is being released, fixup_resent will return a released lock,
which will cause LBUG when this lock is being released later.

Lustre-change: http://review.whamcloud.com/#/c/8680/
Lustre-commit: 96dba12553f1fab5344f5eb7346181bee8e1ea59

Change-Id: Ie106ca46b2b390d178b454655cda54c702cc244e
Signed-off-by: wang di <di.wang@intel.com>
Reviewed-on: http://review.whamcloud.com/9779
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4279 mdd: Fixed OBF of the FS .lustre

Added a check to prevent fid access to the filesystem .lustre
directory from returning EINVAL.
Added a sanity check accordingly.

Lustre-change: http://review.whamcloud.com/8360
Lustre-commit: 09c7765939789b0505644f8d220e5be8262c7e06

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Sebastien Buisson <sebastien.buisson@bull.net>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Change-Id: I4895648a6e1a99b4e3964ff86fc020ef7b8dc9a4
Reviewed-on: http://review.whamcloud.com/10490
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-5107 osp: only check FID for OSP connecting to OST

The backport of http://review.whamcloud.com/8997 to b2_5
(commit ed82a26746d22) didn't need MDT OSPs to allocate FIDs.
The call to osp_prepare_fid_client(d) should be moved after
the check if the OSP connecting to OST, because only such OSP
will intialize its FID client.

Test-Parameters: mdtcount=2 mdscount=2 testlist=sanity
Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I0306a086d9274d520aa43b82ccdb39966ef2ae97
Reviewed-on: http://review.whamcloud.com/10476
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4980 tests: enhance sanity-scrub test_11

If there are new objects created during the OI scrub re-scanning,
then the new created objects will not be skipped by OI scrub this
time. The test should record the total checked objects count for
each cycle scanning to avoid unnecessary check failure.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I2bde83a5c03910216070cc23d01785624d829af4
Reviewed-on: http://review.whamcloud.com/10214
Tested-by: Jenkins
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4398 mdt: acquire an open lock for write or execute

In mdt_object_open_lock() opens for write or execute will always
acquire an open lock of the appropriate mode so that any conflicting
cached open locks on other clients will be canceled. Add a regression
test to sanityn.sh.

Lustre-commit: 708d85a652a77f85153790e6cca1b7a2b91947cf

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I8092bca4c418ec99a25584abdfb635ffec19a26e
Reviewed-on: http://review.whamcloud.com/9063
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/10218

LU-5066 build: fix compile issues with newer gcc

Fixed the error: 'name' may be used uninitialized in
the osd_compact code. Also added padding to struct
shrinker for the posix layer otherwise gcc complies
it away and we end up with a build failure.

Signed-off-by: James Simmons <uja.ornl@gmail.com>
Change-Id: I164957525d572d9d7a58196e5b38fb1349745d44
Reviewed-on: http://review.whamcloud.com/9678
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5088 kernel: kernel update [SLES11 SP3 3.0.101-0.29]

update target and config files for new version

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I9f43b5f2f23ec06c0cda03aa30f23744d643f1af
Reviewed-on: http://review.whamcloud.com/10392
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4924 llite: Only kill SGID/SUID bits

Check that attr mode is valid before using it when determining if to
clear SGID and SUID bits in ll_setattr.
Add regression test.

Lustre-commit: 1504473051df3d20e824f1064ee134501f3856e8
Lustre-change: http://review.whamcloud.com/10153

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I3c8382d6f0899ce9190ccb2993af4c508942b316
Reviewed-on: http://review.whamcloud.com/10405
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4028 quota: fix output of 'lfs quota'

Fix the output of 'lfs quota' when '-h' option is specified:
- total allocated block limit should be converted after calculation;
- show 1~3 digit after the decimal point;

Lustre-commit: 69dda0cf5fb34a1a7ff93468ca1c155dd3a53060
Lustre-change: http://review.whamcloud.com/8407

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: Ibed84a73b87cf741c9122216dc8c45f3a33a78d8
Reviewed-on: http://review.whamcloud.com/10412
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4028 quota: improve lfs quota output

- print the total allocated block/inode limit for all OSTs/MDTs
  when the verbose option is specified;
- add 'h' option to print used space in human readable format;

Lustre-commit: 50a11ef2b232f21ebf81a71c71233889d852ae99
Lustre-change: http://review.whamcloud.com/8152

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>
Change-Id: Ied4cf2131524311f8666da7488108cefa8177141
Reviewed-on: http://review.whamcloud.com/10411
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4659 mdd: rename forgets updating target linkea

MDD rename forgets updating target LinkEA if target is not unlinked,
which will cause LinkEA inconsistency.

Lustre-commit: 2d6315fbeab3c815efa3ea7b953ff34e6f3fa3fc
Lustre-change: http://review.whamcloud.com/9745

Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Change-Id: I348c58e4a11e7b0ba31befef8922dfe47cee755c
Reviewed-on: http://review.whamcloud.com/10409
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-4619 llite: prevent buffer overflow in fiemap

lov_fiemap() does not take consider its @vallen parameter, which is
the max buffer size the caller can hold for the fiemap extents.

This patch fixes this and limits the max mapped fiemap extent count
to fit in the preallocted buffer.

This patch also fixes a memory out of bound write issue when the
fiemap call is only for detecting the number of existing extent.

Lustre-commit: 19326174e264802b91fb4dfc0c1ea16d8bf3ef7e
Lustre-change: http://review.whamcloud.com/9834

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Patrick Farrell <paf@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Change-Id: Ia5aadfa38c3a3d12cdb6e0773ebffb6e213a9abf
Reviewed-on: http://review.whamcloud.com/10408
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-4611 osd: improve credits calculation

- llog catalog do not declare records twice (for old and new objects)
  this might be an issue on ZFS with full debug enabled..
- llog to specify append by pos=-1, so OSD can take this into account
- object create/destroy should not include OI, this is calculated yet
- EA declaration to improve few specific cases
- osd_declare_write() to recognize overwrite optimistically,
  using inode size and i_blocks
- osd_declare_write() to optimize very specific cases, like legacy
  blockmaps with small offsets and allocated indirects
- index delete modify just a single block

preliminary testing on a local setup with 7 OSTs:
1360 credits before and 436 credits after.

llog declarations and index inserts (part of llog object creation)
still consume about 70% (7 OSTs):
   create: 7/28, destroy: 1/4
   attr_set: 2/2, xattr_set: 8/21
   write: 37/179, punch: 14/56, quota 2/2
   insert: 8/135, delete: 2/5
   ref_add: 1/1, ref_del: 3/3

Lustre-commit: 3902ff4c54925b2f1fcb732a32ed7ee5428e9f77
Lustre-change: http://review.whamcloud.com/9258

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: Ia7f0f08e0a4fc19acb11f2dd030c0a9879bc75f7
Reviewed-on: http://review.whamcloud.com/10407
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4863 lloop: avoid panic during blockdev_info

Change the LL_IOC_LLOOP_INFO ioctl in the lustre lloop
device driver to return an error instead of causing
panics with LASSERT().

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: Ib50afa94960dee6fb580038d6e96ea648252d9ea
Reviewed-on: http://review.whamcloud.com/9888
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-4690 tests: disable test_4 in sanity.sh

Disable test_4 in sanity.sh because of LU-4690.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I1acac5ad5c049eed7618aebf473b385a150316d7
Reviewed-on: http://review.whamcloud.com/9440
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-4245 ldlm: do not fill empty lvbo buffer

Don't fill empty lvbo buffer if server does not prepare it.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I22a2ddd016dcc43c00b2b3d6be5d18921bf26d67
Reviewed-on: http://review.whamcloud.com/8285
Tested-by: Jenkins
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4237 lfs: Include lfs mkdir in lfs man page

This patch adds the usage information for lfs mkdir to the lfs
man page.

Signed-off-by: Patrick Farrell <paf@cray.com>
Change-Id: I33a7ad1f48d2a14ad6b0c212a38140622f47c8a7
Reviewed-on: http://review.whamcloud.com/8221
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>

LU-4108 tests: decrease load for zfs in performance-sanity/4

Given the poor performance of metadata on ZFS (cf LU-2600), this patch
decreases the number of files created (similar to what happens in
perforamnce-scale.sh).

Test-Parameters: mdtfilesystemtype=zfs ostfilesystemtype=zfs  mdsfilesystemtype=zfs testlist=performance-sanity
Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I998c8525bf218dacad07d98ac842f4c12102cd9a
Reviewed-on: http://review.whamcloud.com/9725
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>

LU-3992 libcfs: Fix NUMA emulated mode

Kernel commit c1c3443c9c5e9be92641029ed229a41563e44506
assigns all allowed cpus to emulated node.

End cpt initialization loop when all CPUs are assigned.

Change-Id: I4cf22e4cf58cd8377cf19af30217fec72503be69
Signed-off-by: Andriy Skulysh <Andriy_Skulysh@xyratex.com>
Reviewed-on: http://review.whamcloud.com/7724
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3858 test: fix sanity 27A failure caused by stripe delay

When clearing the stripe of root directory, the updating of stripe
might be delayed because of the processing delay of config log.
This will cause the failure of sanity 27A test.

This makes the test suit wait for the update of root directory
stripe.

Signed-off-by: Li Xi <lixi@ddn.com>
Change-Id: Ib4f96d9d4d0ae8a0b54e3e306a7abed758dbdf39
Reviewed-on: http://review.whamcloud.com/7495
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4858 ofd: ofd_osd is always valid

ofd_osd is valid in its life time because ofd is released before
osd, so it should never be cleared.

Lustre-commit: I3cb6d82b71b3018b4fc9e885eb6533b064179238
Lustre-change: http://review.whamcloud.com/10082

Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Change-Id: I2ef4ba4a8faeaf8b718c9888332fa180c144d68b
Reviewed-on: http://review.whamcloud.com/10084
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4748 test: get one single qos_threshold_rr number

In sanity.sh test_116b, if there are multiple MDTs on the same MDS,
we should get one single qos_threshold_rr number instead of the array,
otherwise it will cause resetting qos_threshold_rr failure.

This is a backport of
Lustre-commit: b4a5e9545bcc9c4558fc8a564a8daa2ede997aa6
Lustre-change: http://review.whamcloud.com/9580/

Signed-off-by: Emoly Liu <emoly.liu@intel.com>
Change-Id: I6d7c143713dac809b2f308f7b4f0882345599638
Reviewed-on: http://review.whamcloud.com/9636
Tested-by: Jenkins
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4707 build: Don’t deploy the "lustre" init script on clients

Don’t deploy init scripts if they are not supported,
for example on none RedHat systems.

Don’t deploy the "lustre" and "ha.d" init scripts on clients.

Fix typo in lustre/scripts/lnet script.

Lustre-commit: 109e2da1b156cf038615513b838e18c8d6edf56f
Lustre-change: http://review.whamcloud.com/9535

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: I8ce4ce7aba8e8040ab1dc6b58efb43dcc9bf51ba
Reviewed-on: http://review.whamcloud.com/9940
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4653 fid: only compare non-IDIF seq in lu_fid_diff

Because it will pack OST index into IDIF FID in 2.6, and
in 2.4/2.5, the OST index in IDIF FID is always 0. So we
only compare non-IDIF sequence in lu_fid_diff.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: Ia40f3e0196fdc8cd71050ca070e2bc310af18b55
Reviewed-on: http://review.whamcloud.com/10058
Tested-by: Jenkins
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4569 hsm: Specify import source by FID.

Make the posix CT also accept a FID as import source parameter (from
which the path can be rebuilt) to allow easy "undeletion" of archived
files:

lhsmtool_posix --import <fid> /mnt/lustre/undelete ...

This patch also fixes a typo and replaces a bulky path construction by
a single asprintf() statement.

Lustre-change: http://review.whamcloud.com/9075
Lustre-commit a847e69c3b3853f547f99fcbb9c77259c4f3aa05

Change-Id: Id77841fe7c118872dba9654e88980585485e0ee4
Signed-off-by: Henri Doreau <henri.doreau@cea.fr>
Reviewed-on: http://review.whamcloud.com/10132
Tested-by: Jenkins
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4509 ptlrpc: re-enqueue ptlrpcd worker

osc_extent_wait can be stuck in scenario like this:

1) thread-1 held an active extent
2) thread-2 called flush cache, and marked this extent as "urgent"
   and "sync_wait"
3) thread-3 wants to write to the same extent, osc_extent_find will
   get "conflict" because this extent is "sync_wait", so it starts
   to wait...
4) cl_writeback_work has been scheduled by thread-4 to write some
   other extents, it has sent RPCs but not returned yet.
5) thread-1 finished his work, and called osc_extent_release()->
   osc_io_unplug_async()->ptlrpcd_queue_work(), but found
   cl_writeback_work is still running, so it's ignored (-EBUSY)
6) thread-3 is stuck because nobody will wake him up.

This patch allows ptlrpcd_work to be rescheduled, so it will not
miss request anymore

Lustre-commit: d1dcff3084e929f5768dc733cdc104cddf168c06
Lustre-change: http://review.whamcloud.com/8922

Signed-off-by: Liang Zhen <liang.zhen@intel.com>
Signed-off-by: James Simmons <uja.ornl@gmail.com>
Change-Id: I4929d52b2d409c2ce081147bb5ee3dd380a86c43
Reviewed-on: http://review.whamcloud.com/8922
Tested-by: Jenkins
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/9705

LU-4413 osp: move seq allocation out of osp_import_event

Because seq allocation(osp_init_pre_fid) might be stuck
during RPC, move it out of osp_import_event, which is
inside ptlrpcd_rcv. Otherwise, some other import RPCs(like
connect req)might be blocked in ptlrpcd_rcv.

Lustre-commit: 9b643f4f54c3e9e0c07c2211904a72ad9c4f823f
Lustre-change: http://review.whamcloud.com/8997

Signed-off-by: wang di <di.wang@intel.com>
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Change-Id: Ia41487d7859ec2cf8f0033dd97c7df98a18dcded
Reviewed-on: http://review.whamcloud.com/9875
Tested-by: Jenkins
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-4405 mdc: use ibits_known mask for lock match

Before revalidating a lock on the client, mask the lock bits against
the lock bits supported by the server (ibits_known), so newer clients
will find valid locks given by older server versions.

Xyratex-bug-id: MRP-1583

Lustre-commit: 6b557f04e55b14e11b7fce310094e71331c8b204
Lustre-change: http://review.whamcloud.com/8636

Signed-off-by: Patrick Farrell <paf@cray.com>
Signed-off-by: Alexey Lyashkov <alexey_lyashkov@xyratex.com>
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Signed-off-by: Jian Yu <jian.yu@intel.com>
Change-Id: I7088217fe28e45f1be6c8d1145b42de0f6228372
Reviewed-on: http://review.whamcloud.com/8847
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-4384 ofd: add OBD_INCOMPAT_FID to the OST

OFD may work with normal FIDs since Lustre 2.4, add this flag for
backward compatibility with future use of normal FIDs on OST

Signed-off-by: Mikhail Pershin <mike.pershin@intel.com>
Change-Id: I24d7398388a8146da1e953d97d6c5776b97caa86
Reviewed-on: http://review.whamcloud.com/9411
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-4300 ldlm: ELC picks locks in a safer policy

Change the policy of ELC to pick locks that have no dirty pages,
no page in writeback state, and no locked pages.

Lustre commit: bfae5a4e4a37e4057200bcc70dbff14e18a00797
Lustre change: http://review.whamcloud.com/9175

Change-Id: I05f82a16bf10662c8242adc23e8c2582f0a2d866
Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Signed-off-by: James Simmons <uja.ornl@gmail.com>
Reviewed-on: http://review.whamcloud.com/9451
Tested-by: Jenkins
Reviewed-by: Ann Koehler <amk@cray.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4266 build: fix lbuild to work with OFED-3.5-x

* Remove OFED backport headers for all components except lnet.
* Move all OFED related autoconf testing into lnet/autoconf.
* After OFED built is done, there should only be one -devel rpm.
  We don't really need to filter out any version string.

Lustre-commit: a48df9fbf812cd0fc00f5201a437f38b138c6df2
Lustre-change: http://review.whamcloud.com/8451

Change-Id: Ia5745b7dcd700a35231cf75404807d823952106d
Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Signed-off-by: James Simmons <uja.ornl@gmail.com>
Reviewed-on: http://review.whamcloud.com/9109
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Shuichi Ihara <sihara@ddn.com>
Reviewed-by: Chris Horn <hornc@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4063 tests: use $DIR to check file at agent host

12a used to use DIR2 which is not mounted at agent host.

Lustre-change: http://review.whamcloud.com/#/c/7869/
Lustre-commit: f0f0a4951493859d213ae41b7818722e7c37e58a

Change-Id: Icad9a478700f4365d80233515931f9be28a78332
Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-on: http://review.whamcloud.com/9860
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3498 kernel: remove uses of IS_ERR_VALUE()

Remove most uses of IS_ERR_VALUE(). This macro was often given an int
argument coming from PTR_ERR(). This invokes implementation defined
behavior since the long value gotten by applying PTR_ERR() to a kernel
pointer will usually not be representable as an int. Moreover it may
be just plain wrong to do this since the expressions IS_ERR(p) and
IS_ERR_VALUE((int) PTR_ERR(p)) are not equivalent for a general
pointer p.

Lustre-commit: 5cd7c3f2ab53a07cf98aef83685dafddb13077ed
Lustre-change: http://review.whamcloud.com/6759

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: Ib76592a1fc491539d28a51668c7f06347e988f9f
Reviewed-on: http://review.whamcloud.com/10363
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4877 mdt: tolerate zero md_packed in mdt_fix_reply

The assertion for md_packed value is not really needed there,
if it is zero then buffer is not packed in reply and is not
needed, it is enough just unmark big buffer usage.

Signed-off-by: Mikhail Pershin <mike.pershin@intel.com>
Change-Id: I4db92ce2ef492069db4dbd4d07c0d69770130fad
Reviewed-on: http://review.whamcloud.com/10116
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4578 ptlrpc: Early replies need to honor at_max

When determining whether an early reply can be sent the server will
calculate the new deadline based on an offset from the request
arrival time. However, when actually setting the new deadline
the server offsets the current time. This can result in deadlines
being extended more than at_max seconds past the request arrival
time. Instead, the server should offset the arrival time when updating
its request timeout.

When a client receives an early reply it doesn't know the server side
arrival time so we use the original sent time as an approximation.

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I1b7b04bd20ea5d471ab3777f987e89add744b813
Reviewed-on: http://review.whamcloud.com/9100
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Christopher J. Morrone <chris.morrone.llnl@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4952 osd: wrong paramters for dmu_tx_hold_write

osd_declare_write_commit may pass wrong parameters to
dmu_tx_hold_write when there are multiple IO fragments.

Signed-off-by: Liang Zhen <liang.zhen@intel.com>
Change-Id: Ic7bd5ce1f42e7777c1a07eb4b9d19f8477da66a0
Reviewed-on: http://review.whamcloud.com/10079
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Isaac Huang <he.huang@intel.com>

LU-4711 libcfs: Always clamp cdls_delay between min and max

In libcfs_debug_vmsg2, cdls_delay is only clamped between the minimum
and the maximum when it is increased by multiplying by the backoff
factor.  It is not clamped when it is decreased by dividing by the
backoff factor.  This allows it to achieve values less than the
minimum, which allows a console message to be printed that should have
been skipped.  This patch moves the clamping outside of the else
statement, ensuring that cdls_delay is always between the min and the
max after the first time through libcfs_debug_vmsg2.

Signed-off-by: Ryan Haasken <haasken@cray.com>
Change-Id: I5e587292b73b4c61ef306908fa10d324da5ce069
Reviewed-on: http://review.whamcloud.com/9503
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Chris Horn <hornc@cray.com>
Reviewed-by: Ann Koehler <amk@cray.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-3254 llite: Update layout_gen only if layout change succeed

Move layout generation update to ll_layout_conf() so that it will
change layout at LLITE only if layout update has succeeded.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I407d617be184b1c888d6b16b11f7e33bf10add80
Reviewed-on: http://review.whamcloud.com/9858
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4131 mdd: Changelog not logging file truncate.

Use of truncate(2) does not produce any change log entries,
and the action is invisible. This patch introduces CL_TRUNC
event and adds the corresponding changelog record when the
size has changed.

Signed-off-by: Rahul Deshmukh <rahul_deshmukh@xyratex.com>
Change-Id: I6621e726a82a8105a959b77b4f505f42990f5d64
Reviewed-on: http://review.whamcloud.com/9723
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3868 osc: Update inode timestamp for lockless IO as well

Removed the checks for oi_lockless from osc_io_read_start() and
osc_io_write_start(). This patch also removes the unnecessary call to
cl_object_attr_get() in osc_io_write_start() before calling
cl_object_attr_set()

Signed-off-by: Swapnil Pimpale <spimpale@ddn.com>
Change-Id: Ic96694e84444cd19fafe0af72848250e5b1a3d14
Reviewed-on: http://review.whamcloud.com/8797
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-4397 utils: fix lfs_df loop for disconnected client

If the client is disconnected from the MDT on which "lfs df" is
run (typically MDT0 for the mountpoint), then the file open can
fail with -ESHUTDOWN, causing mntdf() to loop forever trying to
find other MDTs or OSTs to get statfs data from.

The caller of llapi_obd_statfs() cannot tell the difference between
an error from open() and ioctl(IOC_OBD_STATFS), so llapi_obd_statfs()
should return a fatal error if the open failed since all of the
later opens will fail as well.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I05d48d5b6680c382c9e05c02fd82a9a200500c1e
Reviewed-on: http://review.whamcloud.com/8949
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4799 llite: Do not rate limit dirty page discard warning

Messages which are printed by ll_dirty_page_discard_warn() should not
be rate limited.  If they are rate limited, some files which may be
corrupted on client eviction will not be reported to the user.

This patch changes the CWARN to a CDEBUG to disable console message
rate limiting for this message.  The dirty page discard warnings are
already limited on a per-file basis by the function vvp_vmpage_error
which calls ll_dirty_page_discard_warn only if the ccc_object's
cob_discard_page_warned == 0.

Signed-off-by: Ryan Haasken <haasken@cray.com>
Change-Id: Ia8a13c7add89c890014170254e462ae861de0fc5
Reviewed-on: http://review.whamcloud.com/9752
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Cory Spitz <spitzcor@cray.com>
Reviewed-by: Ann Koehler <amk@cray.com>
Reviewed-by: Chris Horn <hornc@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4065 mdt: missing NL in CERROR format

Add a missing 'n' at the end of the CERROR format parameter to
avoid annoying "format at mdt_coordinator.c:977:mdt_hsm_cdt_start
doesn't end in newline" msg in syslog and the consequence of 2
merged lines/traces in Lustre debug-log.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: Ic8baf81886b5be3de32ee595be99e20428729f6d
Reviewed-on: http://review.whamcloud.com/9597
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-3254 llite: access layout version under a lock

We used to access layout version under the protection of ldlm
lock, this introduces extra overhead for dlm lock matching.

In this patch, lli_layout_lock is introduced to access the layout
version. Also, when a layout lock is losing, we should tear down
mmap of the correspoding inode to avoid stale data accessing in the
future.

This is part of technical verification of replication.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I657c6a8b27f2cdaf468dc208c43c1d92f3843705
Reviewed-on: http://review.whamcloud.com/8689
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4778 llog: update catlog when plain log destroyed

When the empty plain log is destroyed while processing
catlog, we should call llog_cat_cleanup() to close the
plain log and update in catlog.

Intel-bug-id: INTL-64

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I97b1d5cc59e84fdb4888274b48f223a063293647
Reviewed-on: http://review.whamcloud.com/9702
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4743 osp: continue llog processing

when unexpected record type is met

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Change-Id: I3dd45f501192bc6dd3bfddb550fd455ae319b923
Reviewed-on: http://review.whamcloud.com/9574
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>

LU-5005 utils: handle ROOT path correctly

MDT replies path '\0' to the request of fid2path on ROOT,
client utils should handle it properly.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: Ic7064866b0e8ef33fc0a17e273a120d6d6dae4fe
Reviewed-on: http://review.whamcloud.com/10211
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4783 mgs: start params log during mgs initialization

Params log was introduced by LU-3155 to set parameters permanently
with lctl "set_param -P" instead of "conf_param". But in a newly
mounted filesystem, the log won't be created automatically until "lctl
set_param -P" is called. This causes that the other servers can't make
a local copy successfully during their setup, and the misleading
console error message "LustreError: 13a-8: Failed to get MGS log
params and no local copy." is printed.
This patch is to start params log during mgs intialization so that the
error message is quieted.
Also, test_76b is added to conf-sanity.sh to verify params log setup.

Signed-off-by: Emoly Liu <emoly.liu@intel.com>
Change-Id: Ifc3abc1e35ef20230a1be1cdf1c4cc8f57b2788e
Reviewed-on: http://review.whamcloud.com/9737
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2099 osd: clear iobuf up on I/O completion

it was a bad idea to clear iobuf on buffer release because the
same buffers can be used few times in case of transient ENOSPC.

Change-Id: Ic1d2ddaa0794f2f6f7d1ba71620e06b4ea7a99d2
Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>

LU-5025 kernel: kernel update [RHEL6.5 2.6.32-431.17.1.el6]

update RHEL6.5 kernel to 2.6.32-431.17.1.el6

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I0c89b09f40cae082dd03a1effb80fe36c3ea0897
Reviewed-on: http://review.whamcloud.com/10283
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Tested-by: Jenkins
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5048 ldiskfs: SLES ext4-mballoc-extra-checks patch fix

Original SLES patch version, from LU-1812, was missing a latest hunk.
This caused OSTs to be put in read-only mode due to wrong
inconsistencies detected.

This patch is back-ported from the following one:
Lustre-commit: 62993e6261f552baddb1035ffda5a037427bbcc8
Lustre-change: http://review.whamcloud.com/10294

Signed-off-by: Jian Yu <jian.yu@intel.com>
Change-Id: Ic7de0e7f957da2d0a8ebecf7a3f15f39bbec3db5
Reviewed-on: http://review.whamcloud.com/10304
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4791 lod: subtract xattr overhead when calculate max EA size.

1. It needs to subtract the xattr overhead when calculate the max
EA size, otherwise, it might get oversize stripe_count, which cause
ENOSPC error in the following xattr_set.
2. Release stripe information, if xattr set fails.

Lustre-change: http://review.whamcloud.com/9835
Lustre-commit: a11c69ad81ca8071b9d97d5032539dd7b36873ee

Change-Id: I4a4fe0681f2cd9dd6f242a03c75326acde02ae05
Signed-off-by: wang di <di.wang@intel.com>
Reviewed-on: http://review.whamcloud.com/10020
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4708 ldiskfs: do not copy NUL terminator from direntry

Do not try to copy a NUL terminator from the source dirent if there
is no dir_data for that record.  This can happen with 1.x filesystems
that do not have a dirdata field in the dirent to store the FID.

The target buffer is expected to be NUL terminated, and this is
handled by using kzalloc() to allocate it and zero out the buffer,
but the source buffer may not have a trailing NUL byte if it is a
multiple of 4 bytes in length.

If the dirent is aligned at the end of the directory block/page this
can cause an oops by accessing beyond the end of the page, or will
otherwise copy a garbage byte to the target buffer.

Lustre commit: a101a95a3f58cde3ed912d074dc283b4083672ff
Lustre change: http://review.whamcloud.com/9510

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: Iab52f916d7571b3d2587255280c72a52c60399c9
Reviewed-on: http://review.whamcloud.com/9510
Tested-by: Jenkins
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/9662

LU-4008 mdt: Use actual size for layout lock intents

The intent for layout looks up the size of the lov xattr. That size
should be used for the reply buffer instead of the maximum size ever
seen.

Lustre-commit: 586a609c9d5a53e75174b66269c685f594ded7c2
Lustre-change: http://review.whamcloud.com/#/c/9571

Signed-off-by: Matt Ezell <ezellma@ornl.gov>
Signed-off-by: James Simmons <uja.ornl@gmail.com>
Change-Id: Id5d560001df3d058d189339ede33347f8353990a
Reviewed-on: http://review.whamcloud.com/10018
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4008 mdt: Shrink default LOVEA reply buffers

Instead of allocating maximum possible LOVEA for intent requests,
allocate a sane default to fit up to 100 stripes, should the size
be bigger, the buffer reallocation code will kick in.

Additionally, since quite awhile ago we no longer send
unlink cookies or EA data to the clients from unlinks/renames,
so we no longer need to allocate reply buffer for them in
mdt_close() and other such places.
Also clean up unused fields related to these buffers in
md_attr structure.

Lustre-commit: 00260466f9c84f21e6db55d5dd6e87e8a9906438
Lustre-change: http://review.whamcloud.com/9322

Change-Id: I06b9f5665ec2616cd9a5d483872369d710400d22
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Signed-off-by: James Simmons <uja.ornl@gmail.com>
Reviewed-on: http://review.whamcloud.com/9942
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>

LU-4719 mdt: Don't try to print non-existent objects

mdt_dump_lmm() needs to make sure it's not trying to print
objects for a default striping from a directory, since there
are not objects in this case.

Return early if the D_INFO debug mask is disenabled, so it
doesn't iterate through the LOV/LMV layout at all.  That is
just a waste of effort.

Lustre-change: http://review.whamcloud.com/#/c/9520
Lustre-commit: ebfc6f4b54961e5e0762af7d2555f2dc5d1e1ec3

Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: Ifbda7c0bfdec74c5ee4c097ea24613a4ce692572
Reviewed-on: http://review.whamcloud.com/10021
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Matt Ezell <ezellma@ornl.gov>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-4257 clio: replace semaphore with mutex

According https://www.kernel.org/doc/Documentation/mutex-design.txt:
 - the mutex subsystem is slightly faster and has better scalability
   for contended workloads. In terms of 'ops per CPU cycle', the
   semaphore kernel performed 551 ops/sec per 1% of CPU time used,
   while the mutex kernel performed 3825 ops/sec per 1% of CPU time
   used - it was 6.9 times more efficient.
 - there are no fastpath tradeoffs, the mutex fastpath is just as
   tight as the semaphore fastpath. On x86, the locking fastpath is
   2 instructions.
 - 'struct mutex' semantics are well-defined and are enforced if
   CONFIG_DEBUG_MUTEXES is turned on. Semaphores on the other hand
   have virtually no debugging code or instrumentation.

One more benefit of mutex is optimistic spinning. It try to spin for
acquisition when there are no pending waiters and the lock owner is
currently running on a (different) CPU. The rationale is that if the
lock owner is running, it is likely to release the lock soon.

This significantly reduce amount of context switches when locked
region is small and we have high contention.

Lustre commit: 96f482b509867bb70cc26e94e60903df82b09f30
Lustre change: http://review.whamcloud.com/9095

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Signed-off-by: Shuichi Ihara <sihara@ddn.com>
Change-Id: I29cb202468610636ebd45d49281c4e503930bde6
Reviewed-on: http://review.whamcloud.com/9943
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4351 tests: skip loop test in SLES11 clients

Avoid known failure in test 54c due to SLES specific
incompatible implementation flaw in kernel loop devices.

Lustre-commit: f5683b151be0e8ba0d41e85a8f99850c7d00e55c
Lustre-change: http://review.whamcloud.com/9955

Test-Parameters: clientdistro=sles11sp3 testlist=sanity
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Change-Id: If39aa85e8c1f468df532aa93610d3eb7cd8d32f5
Reviewed-on: http://review.whamcloud.com/10085
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-4808 tests: sanity cleanup to work on Xeon Phi

* Made the test_74c robust to different platforms.
* Fix an issue with values more than 2^32 (for example 9.36473e+09).
* Fix an integer comparison in case of empty variable.
* Check for tools presents (rsync and getfattr).
* Check for CLIENTONLY mode.
* Remove "-x" options from grep (it's not always supported).
* Coding style cleanup.

Lustre-commit: e217648d50da55366ef819e78bb8d7601a109077
Lustre-change: http://review.whamcloud.com/9766

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Change-Id: I8e553da96afd78e96c9c534898db7173aa268223
Reviewed-on: http://review.whamcloud.com/10086
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-3333 ptlrpc: Protect request buffer changing

*_enlarge_reqbuf class of functions can change request body location
for a request that's already in replay list, as such a parallel
traverser of the list (after_reply -> ptlrpc_free_committed) might
access freed and scrambled memory causing assertion.

Since all such users only can get to this request under imp_lock, take
imp_lock to protect against them in *_enlarge_reqbuf

Change-Id: I0fa690aabd8696a9f05b94c66e06e30eefb5c759
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/10074
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2479 ldiskfs: do not check dir max size for regular files

ldiskfs_append() is used not only to extend directory but also to
maintain iam container. In later case ldiskfs_append() should not
check for max directory size. The iam container is distunguished as
regular file.

This is a backport of
Lustre-commit: 4b9e64aec7cfdf859f1f931e1ee44056db050bb9
Lustre-change: http://review.whamcloud.com/8137

Signed-off-by: Vladimir Saveliev <vladimir_saveliev@xyratex.com>
Signed-off-by: Emoly Liu <emoly.liu@intel.com>
Change-Id: Ibc3e96078e9af3eb854c6666f087b1049b68f409
Reviewed-on: http://review.whamcloud.com/10043
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4422 quota: fix s-q test_6

s-q test_6 should not fail by checking whether the file is
growing, because in most time, the test is waiting for quota
slave reconnecting to quota master.

This patch is back-ported from the following one:
Lustre-commit: a1cf492e3632e2930b9593817d9397ba1b378fda
Lustre-change: http://review.whamcloud.com/9339

Signed-off-by: Jian Yu <jian.yu@intel.com>
Change-Id: I5c40e704266672af270e8306dff1fb867af1b8b5
Reviewed-on: http://review.whamcloud.com/9946
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4915 kernel: kernel update [SLES11 SP3 3.0.101-0.21]

update target and config files for new version

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: Ia2f2659e3f57d2fccacf4afe2c50efe7a656be7c
Reviewed-on: http://review.whamcloud.com/9982
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4381 lov: to not hold sub locks at initialization

Otherwise, it will cause deadlock because it essentially holds
some sub locks and then to request others in an arbitrary order.

Lustre-commit: c6ab1fcc056778b18f685ec591ce27907e887617
Lustre-change: http://review.whamcloud.com/9152

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Change-Id: I9bdfa2339c83396efa5d16763a5329d06e232ddd
Reviewed-on: http://review.whamcloud.com/9994
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-4558 clio: Solve a race in cl_lock_put

It's not atomic to check the last reference and state of cl_lock
in cl_lock_put(). This can cause a problem that an using lock is
freed, if the process is preempted between atomic_dec_and_test()
and (lock->cll_state == CLS_FREEING).

This problem can be solved by holding a refcount by coh_locks. In
this case, it can be sure that if the lock refcount reaches zero,
nobody else can have any chance to use it again.

Lustre-commit: ec1a5d4f1f5b52ee5031ddc11a862c82996541f7
Lustre-change: http://review.whamcloud.com/9881

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I5e16396156e1c7b8b86f7aa74b7b4735bb774a0f
Reviewed-on: http://review.whamcloud.com/9887
Tested-by: Jenkins
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4591 lov: cancel ungranted sub lock

When the top lock is canceled due to error, we should cancel
ungranted sub lock otherwise the sublock state is undefined.

backport from http://review.whamcloud.com/9524

Signed-off-by: Shuichi Ihara <sihara@ddn.com>
Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: Ifa68743cd5bf3a9d69258014d066124fd7fc87c9
Reviewed-on: http://review.whamcloud.com/9851
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4721 obdclass: handle local storage init/fini properly

1) In local_oid_storage_fini(), take the mutex on ls_device
   before decreasing the 'los' reference to avoid others to
   obtain the mutex earlier and freed the 'los' by race.

2) When llog init the local stroage for FID_SEQ_LLOG and
   FID_SEQ_LLOG_NAME, it should record the handlers which
   can be used to fini them to avoid releasing the handler
   which is in using by others.

3) NOT forget the llog_ctxt_put() if something wrong during
   the llog_osd_setup().

4) NOT put the object in lastid_compat_check() until all the
   usages on such object have been done.

Test-Parameters: envdefinitions=SLOW=yes,ENABLE_QUOTA=yes testlist=sanity-scrub,sanity-scrub,sanity-scrub,sanity-scrub
Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: If7d9241147870e202b6a1082a398ee1c48f1a6d8
Reviewed-on: http://review.whamcloud.com/9572
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4596 lprocfs: mdt/*/exports/*/uuid is empty after remount

need to assign exp_nid_stats even if old stats are present

Change-Id: I9b0ac4c7ccee6cbd457827b20123bf40d91aaf33
Xyratex-bug-id: MRP-1641
Signed-off-by: Andriy Skulysh <Andriy_Skulysh@xyratex.com>
Reviewed-on: http://review.whamcloud.com/9171
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4853 ldiskfs: fix race in ldiskfs_ext_new_extent_cb

In ldiskfs_ext_calc_credits_for_insert(), we should use the 'depth'
stored in the 'path' instead from inode, because the extent tree
could have been changed when ldiskfs_ext_calc_credits_for_insert()
is called (by ldiskfs_ext_new_extent_cb()).

It was fixed in LU-2555, but the fix is missed in
sles11sp2/ext4-misc.patch

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I9ca349ba52060f10fc980721317ba47e10572473
Reviewed-on: http://review.whamcloud.com/9868
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4252 osd: remove locking i_mutex in osd_punch

This piece of code intent to calm down the kernel
WARN_ON messsage show up since 3.9 upstream. So we
can just remove it for now. We'll add a ldiskfs
patch to resolve this issue while 3.9 server support
come up.

Signed-off-by: yang sheng <yang.sheng@intel.com>
Change-Id: I0d8a13829ebb1d18c672a09729be603c923bc79c
Reviewed-on: http://review.whamcloud.com/9742
Tested-by: Jenkins
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-992 doc: remove old kernels from lustre/ChangeLog

Remove the old and obsolete kernels from the current list of
kernel versions that Lustre works with, since this is confusing
to list kernels that are no longer supported at all.

Update recommended e2fsprogs version to include "or newer".

Move ldiskfs README file to top-level directory.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: Ia4859b3b2bf5a8aa9faabeb43d554130103ebbe5
Reviewed-on: http://review.whamcloud.com/8628
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4109 tests: raise out-of inode LOV EA detection threshold

Sanity sub-test 57b sometimes failed because the threshold (< 8K
of MDT space) it used to detect if LOV EA has been stored out-of
inode is too short (vs space also allocated for llog files, ChangeLog,
etc). So raising threshold to 16.

Lustre commit: f52da9f69bf08e82a3f4ddaf174e3a6318258f1f
Lustre change: http://review.whamcloud.com/8156

Change-Id: I4599d8b45828cee607937b387c1b46ae0df4f6b3
Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Reviewed-on: http://review.whamcloud.com/9652
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4728 mdt: fix NULL deference of mdt_fid_lock

When enabling hsm_control, mti_exp field of struct mdt_thread_info
could be NULL.  ldlm_cli_enqueue_local will crash the kernel when
dereference it.

Lustre commit:  0eca7d92e4a84cc6ea2ca9975d2d3b4cef17a686
Lustre change: http://review.whamcloud.com/9543

Change-Id: Iba41a8e646a2fad7f009934d8bf1b67930a98f70
Signed-off-by: Li Xi <lixi@ddn.com>
Reviewed-on: http://review.whamcloud.com/9707
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4545 hsm: Always report errors to coordinator.

Make sure feedback on processed items gets properly delivered to the
coordinator even if errors occur between action item delivery and
item processing initialization phase.

Lustre commit: 344a261080d462c29cf45b075deacf793c92e197
Lustre change: http://review.whamcloud.com/9310

Change-Id: I9b79f049728d011480dd86ef6baa23e39a124634
Signed-off-by: Henri Doreau <henri.doreau@cea.fr>
Reviewed-on: http://review.whamcloud.com/9708
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: jacques-Charles Lafoucriere <jacques-charles.lafoucriere@cea.fr>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4770 tests: tests scripts simplification

1. Xeon Phi don't support "mount -t lustre", we can use only
   "mount.lustre" utility.
2. A "head" utility don't support shortcut "-1" option.
   It should be specified as "-n 1".
3. A "sed" utility don't support expression "s/[-.]/ /3"

Lustre-commit: 4d96e960d574387e8ac4a31249ef1d30c859480b
Lustre-change: http://review.whamcloud.com/9668

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Change-Id: I24804d7bde55ac3a70d99a582a1635ee9bfe62b8
Reviewed-on: http://review.whamcloud.com/9735
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>

New tag 2.5.1-RC4

Change-Id: Ide0da772d7fd739fce878691e8820e19848534b9

LU-4751 hsm: Fix sanity-hsm tests for non-mrsh $PDSH

The workaround for starting backgrounded copytool
monitors via pdsh -Rmrsh is not compatible with non-mrsh
values of $PDSH. This commit adds a branch such that
the workaround is only used when required by the test
cluster environment.

Lustre-commit: 0c261a1fd22a0c867b02baa275ae5023f27d6f5f
Lustre-change: http://review.whamcloud.com/#/c/9587/

Signed-off-by: Michael MacDonald <michael.macdonald@intel.com>
Change-Id: Iab7a94180313170a968c7acbe66919a532f08cfd
Reviewed-on: http://review.whamcloud.com/9593
Tested-by: Jenkins
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

New tag 2.5.1-RC3

Change-Id: I9e62744390dc3207a56fee8ad28535249af356ba

LU-4209 utils: fix O_TMPFILE/O_LOV_DELAY_CREATE conflict

In kernel 3.11 O_TMPFILE was introduced, but the open flag value
conflicts with the O_LOV_DELAY_CREATE flag 020000000 added to fix
LU-812 in Lustre 2.4.  O_LOV_DELAY_CREATE allows applications
to defer file layout and object creation from open time (the default)
until it can instead be specified by the application using an ioctl.

Instead of trying to find a non-conflicting O_LOV_DELAY_CREATE flag
or define a Lustre-specific flag that isn't of use to most/any other
filesystems, use (O_NOCTTY|FASYNC) as the new value.  These flags
are not meaningful for newly-created regular files and should be
ok since O_LOV_DELAY_CREATE is only meaningful for new files.

I looked into using O_ACCMODE/FMODE_WRITE_IOCTL, which allows calling
ioctl() on the minimally-opened fd and is close to what is needed,
but that doesn't allow specifying the actual read or write mode for
the file, and fcntl(F_SETFL) doesn't allow O_RDONLY/O_WRONLY/O_RDWR
to be set after the file is opened.

For 2.5.1 and later, only check for the 020000000 flag in the kernel
for compatibility with applications compiled against 2.5.0 headers,
since this is needed for SLES11 SP2/SP3 clients on 3.0 kernels.

We will keep the 0100000000 flag in O_LOV_DELAY_CREATE for backward
compatibility until 3.13 is the oldest supported client kernel, but
drop the conflicting __O_TMPFILE value of 02000000 since that will
cause an error when running on 3.11+ kernels.  The 020000000 has only
been used in Lustre 2.4.0-2.4.2 and 2.5.0 and always in conjunction
with 0100000000, so any apps that used O_LOV_DELAY_CREATE directly
instead of calling llapi_file_create*() will still work until Linux
3.13 is used.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I565f3454616edc60c6acee01034aa5d7733ebbe5
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Peng Tao <bergwolf@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/9492
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-4643 hsm: Make sanity-hsm test_60 more robust

The first version of this test was fragile and could fail
intermittently when test infrastructure was not capable of
providing 1MB/sec in lustre bandwidth. This commit changes the
test to validate that a progress update occurs within the expected
window, rather than testing for a specific amount of data copied
under ideal conditions.

Lustre-commit: 1c9fbda98bf2274d63517ad72ef81484081834b9
Lustre-change: http://review.whamcloud.com/#/c/9376/

Signed-off-by: Michael MacDonald <michael.macdonald@intel.com>
Change-Id: I335e4cd9a3671f60be5a9236f322705b68664c8c
Reviewed-on: http://review.whamcloud.com/9378
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Jenkins
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4689 hsm: count NULL terminator in hai_first/hal_size

If fsname is 8-byte aligned, hai_first fails to count the ending NULL
terminator causing hai to directly attached after fsname and future
hai_first will return a different position for first hai.

Signed-off-by: Peng Tao <bergwolf@gmail.com>
Change-Id: I5e9e1f48f99b4743b2d5b93397e06f6becabeb26
Reviewed-on: http://review.whamcloud.com/9431
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/9468

LU-4724 hsm: Safe copytool event logging

Protect against concurrent event log writes by multiple
threads within a copytool process. Fixes sanity-hsm test_71
failures.

Lustre-commit: f28e754dcbf36e159386038bdebfaef50135715f
Lustre-change: http://review.whamcloud.com/#/c/9553/

Signed-off-by: Michael MacDonald <michael.macdonald@intel.com>
Change-Id: Ic83b162c60074bde7235bb9ae44f8f28797c1dfd
Reviewed-on: http://review.whamcloud.com/9568
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

New tag 2.5.1-RC2

Change-Id: Ie7c87fe160c2cdd0bc4ca225ea07dc4cbdb9b7e2

LU-4703 mdd: do not skip xattr sanity check for all cases

xattr sanity check should be done at all times.
Otherwise we are risking letting a non root user setting
access acl on any file.

Lustre-commit: 6d7066989c06bdf6c86f5b8b61d542fda8e5739e
Lustre-change: http://review.whamcloud.com/9469

Signed-off-by: Li Dongyang <dongyang.li@anu.edu.au>
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: Ia669a20139abc5f4b64688cccd4b07a5bcc6910b
Reviewed-on: http://review.whamcloud.com/9513
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4020 hsm: allow copytool event monitoring with JSON

Adds hooks into various llapi_hsm_* functions to emit JSON-formatted
event messages for consumption by a monitoring agent. The copytool
needs to be supplied with an optional --event-fifo argument to
enable this feature.

Incorporates the following work done by Bruno Faccini:
* Put all JSON routines in a separate file/lib to allow for
  LGPL licensing.
* Major code cleanup to follow best practices and address review
  concerns.

Lustre-commit: 45c9ef1dfff207086665c764c7d500c00aa03c7f
Lustre-change: http://review.whamcloud.com/7790

Signed-off-by: Michael MacDonald <michael.macdonald@intel.com>
Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Change-Id: Icfad0f2093f24fcfec22408d9819d438c6a5e7f1
Reviewed-on: http://review.whamcloud.com/9512
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-4704 acl: fix permission problem of setfacl

Setxattr does not check the permission when setting ACL xattrs. This
will cause security problem because any user can walk around
permission checking by changing ACL rules.

Lustre-commit: 42f504ecada81d7a2a8e2244f345e8dbf73fd157
Lustre-change: http://review.whamcloud.com/9473

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: If6bda8ec6b3d91f31bef812acb9f9f636ed56291
Reviewed-on: http://review.whamcloud.com/9514
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>