LU-10165 llite: disable statahead if starting statahead fail

Once starting statahead thread fails, it should disable statahead.
Current code only does this when "sai != NULL", instead it should
check whether current process is opening this dir, so for cases
like current file is not the first dirent, or sai allocation fail,
it won't retry statahead.

Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Change-Id: Iaedddd3659cdffeab51800f45b02f0b39c4a1ec1
Reviewed-on: https://review.whamcloud.com/29817
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5170 lfs: Standardize error messages in macros

Error messages in ARG2INT() and ARG2ULL() are updated to a standard
format.  Messages are prefixed with the name of the utility and the
command that caused the error.  User-provided values are delimited
with single quotes.

Test-Parameters: trivial
Signed-off-by: Steve Guminski <stephenx.guminski@intel.com>
Change-Id: I31a0a30ac15681826e7e25b8a44d56174fb23e08
Reviewed-on: https://review.whamcloud.com/28250
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-9727 lustre: Add an additional set of 64 changelog flags.

This adds a new changelog extension containing 64 additional flag
bits, to be used for future changelog extensions.
The presence of the extension is signalled using the last remaining
unused changelog flag bit.
The new extension is present in all changelog records by default, but
will be removed from records read by legacy changelog consumers.

Signed-off-by: Matthew Sanderson <matthew.sanderson@anu.edu.au>
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Iad05880e82400d7e84d927e84e44d1454e240a80
Reviewed-on: https://review.whamcloud.com/28045
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10271 hsm: don't release with wrong size

The result of ll_merge_attr operation was ignored during
ll_hsm_release. For this case the released file could have
the wrong size on MDS, like
File: ‘released.file’
Size: 0  Blocks: 0  IO Block: 4194304 regular empty file

Patch adds test_253 sanity-hsm, to check hsm release
operation when cl_object_attr_get failed. It produces
the wrong size-on-mds for a released file.

Signed-off-by: Alexander Boyko <c17825@cray.com>
Change-Id: I94df1889265e5b6d03b16745de93e52af95d5b7c
Reviewed-on: https://review.whamcloud.com/30240
Reviewed-by: Henri Doreau <henri.doreau@cea.fr>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Sergey Cheremencev <c17829@cray.com>
Reviewed-by: Quentin Bouget <quentin.bouget@cea.fr>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10176 tests: limit DoM tests to new servers only

Run DoM tests only if server has support for DoM by checking
its version is 2.10.55 at least.

Signed-off-by: Mikhal Pershin <mike.pershin@intel.com>
Change-Id: I79f26cd3c80fbfc4bf48ddc3fcc762248d491034
Reviewed-on: https://review.whamcloud.com/30085
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>

LU-6142 uapi: Get rid of lustre_fid typedef

Replace it with struct lu_fid. Update the userland code and man
pages to reflect this change.

Linux-commit: d8f6bc9a53f97d1ea4b2b955672904338643308b

Test-Parameters: trivial

Change-Id: I0b7e0770dd9da9bdac55c02c2ec98aea7cea7100
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: https://review.whamcloud.com/29849
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-10220 mdd: fix buf alloc in mdd_changelog_data_store_by_fid

Fix allocation of mti_big_buf by call to lu_buf_check_and_alloc()
in mdd_changelog_data_store_by_fid().
reclen must take the header size of struct llog_changelog_rec into
account.

Maybe no memory corruptions were seen before because the buffer size
allocated in a previous call to mdd_declare_changelog_store() was
covering the need. But audit will add more information in changelog
records, provoking memory corruptions without this fix.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Id0a06c412b54c0ae12c15d53f3e166e3e5d9ed68
Reviewed-on: https://review.whamcloud.com/30014
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Henri Doreau <henri.doreau@cea.fr>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-9887 tests: adjust lfsck speek test error range

To make sanity-lfsck test_9{a,b} to be more robust for kinds
of environment.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Ie0740f2b205f319cbefe54c86734b889e6d03cc6
Reviewed-on: https://review.whamcloud.com/30247
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>

LU-10255 tests: Fix wrong comparison operator in sanity 129

Currently sanity 129 uses numerica (-eq) operator to check whether
the has_warning variable is true or false depending on whether
expected warnings have been detected. This is wrong since bash
doesn't really have bolean types and true/false are not evaluated to
1/0. This results in the following bash error when the test is run:

    ./sanity.sh: line 9244: [: true: integer expression expected

Fix it by using the correct string operator '=' to check for
correct value.

Test-Paramters: clientdistro=sles12sp2 testlist=sanity
Test-Paramters: clientdistro=sles12sp3 testlist=sanity

Signed-off-by: Nikolay Borisov <nborisov@suse.com>
Change-Id: I071dcb3001ea9b2e5806d346dcd48bfcaf3297e9
Reviewed-on: https://review.whamcloud.com/30149
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Tested-by: Jenkins
Reviewed-by: Saurabh Tandan <saurabh.tandan@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10229 test: Add sanity-sec test_27 to ALWAYS_EXCEPT

Received request to ALWAYS_EXCEPT test_27 since it has been
failing regularly on master.  This will keep it from affecting
other patches.

Test-Parameters: trivial testlist=sanity-sec

Change-Id: I630b5cd1f978025e605ecad38cd5e276afdab694
Signed-off-by: Jim Casper <jamesx.casper@intel.com>
Reviewed-on: https://review.whamcloud.com/30135
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Saurabh Tandan <saurabh.tandan@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-10241 kernel: kernel update RHEL6.9 [2.6.32-696.16.1.el6]

Update RHEL6.9 kernel to 2.6.32-696.16.1.el6

Test-Parameters: clientdistro=el6.9 mdsdistro=el6.9 \
  ossdistro=el6.9 mdtfilesystemtype=ldiskfs \
  ostfilesystemtype=ldiskfs testgroup=review-ldiskfs

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: Ib2215b56e647fb8e19bc8b7e344a94564c7a8c7f
Reviewed-on: https://review.whamcloud.com/30130
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9135 osp: do not fail if last_rcvd update failed

just ignore an error in this case and hope the next time
OST will be able to serve well and OSP cancel the record.

Change-Id: Ib1bbe77132e81322f353909620b6a413981ae342
Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-on: https://review.whamcloud.com/30129
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10038 test: improve error reporting in sanity test_133g()

In sanity test_133g() remove the output redirection to /dev/null so
that errors from do_facet or find go to the test logs. Suppress error
messages in badarea_io.c.

Test-Parameters: trivial envdefinitions=ONLY=133 testlist=sanity,sanity,sanity,sanity,sanity,sanity,sanity,sanity,sanity,sanity
Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ia51ccdacb995ff51a6467cbdd280c8965cfc5f47
Reviewed-on: https://review.whamcloud.com/30105
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10217 test: Make sanity 248 work with newer dds

This patch fixes 2 problems when working with value from dd.

Firstly, extraction of the readings from dd rely on them always
being in field number 6. However, that's not true for newer dd
versions where the output has changed and the time reading is now in
field number 10. So to make the code more generic and work with both
old and new outputs switch to using an egrep regular expression to
extract the correct field.

Secondly, dd can output its time in scientific notation, with
decimal and floating point portion being separated by a comma rather
than a dot:
   4096 bytes (4,1 kB, 4,0 KiB) copied, 9,1229e-05 s, 44,9 MB/s

Unforuntately this is not understood by bc so it needs to be
processed by substituting the comma for a dot and also expanding
the scientific notation to multiplication of 10 to the correct power.

Test-Parameters: trivial
Test-Parameters: clientdistro=sles12sp2 testlist=sanity
Test-Parameters: clientdistro=sles12sp3 testlist=sanity

Signed-off-by: Nikolay Borisov <nborisov@suse.com>
Change-Id: I83ec659874ffe702c68f94f828827ff93325db72
Reviewed-on: https://review.whamcloud.com/30005
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10204 tests: Fix line continuation

There is a simple  missing in one of the error paths in
ladvise_willread_performance which results in an unknown command
error. Fix it by adding the missing backslash.

Test-Parameters: trivial
Signed-off-by: Nikolay Borisov <nborisov@suse.com>
Change-Id: Ifff854c6cf5c1add71fd1571cb4de01d56204fde
Reviewed-on: https://review.whamcloud.com/29981
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>

LU-8950 tests: standardize file creation in sanity-hsm

sanity-hsm exposed several functions to create files:
 - make_small()
 - make_small_sync()
 - make_archive()

This patch renames all these functions (respectively):
 - create_small_file()
 - create_small_sync_file()
 - create_archive()

And adds a new one: create_empty_file() which creates an empty file

The actual file_creation is now abstracted in the create_file()
function used by all the aforementionned functions.

Test-Parameters: trivial testlist=sanity-hsm
Signed-off-by: Quentin Bouget <quentin.bouget@cea.fr>
Change-Id: Ib4267827eb2e31ed396da1d711419f45f9e0ef16
Reviewed-on: https://review.whamcloud.com/24704
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Henri Doreau <henri.doreau@cea.fr>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3285 mdt: remove assertion on mdt_pack_size2body()

The assertion on result of mdt_pack_size2body() is not
always correct, e.g. in case of resend.

Test-Parameters: mdssizegb=20 testlist=sanity-dom,dom-performance
Signed-off-by: Mikhal Pershin <mike.pershin@intel.com>
Change-Id: Iff2c9723959db591fdd29074b0186d8ddf0752e9
Reviewed-on: https://review.whamcloud.com/30058
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3285 mdt: get lu_object_attr for existent object

In mdt_punch_hdl() lu_object_attr() may be taken for
non-existent object with LBUG() as result.
Take lu_object_attr() after mdt_object_exists()check.

Test-Parameters: mdssizegb=20 testlist=sanity-dom,dom-performance
Signed-off-by: Mikhal Pershin <mike.pershin@intel.com>
Change-Id: If1181caa94b28cf1f251f57fa6f7102dc40c329d
Reviewed-on: https://review.whamcloud.com/30057
Tested-by: Jenkins
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3285 tests: disable kernel flusher in discard tests

Test 101 checks that client discards data instead of flush,
but it is not reliable because write RPC may happens before
unlink command is executed due to kernel flusher.
Set /proc/sys/vm/dirty_writeback_centisecs to 0 for the test
time and return it back to the default value later.

Signed-off-by: Mikhail Pershin <mike.pershin@intel.com>
Change-Id: I2c4da039a456519e25c6d9ffcc015c5baa479740
Reviewed-on: https://review.whamcloud.com/30016
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9872 tests: modify check space requirements for NFS test

In parallel-scale-nfs testing, the Lustre clients NFS mounts
the Lustre file system. In some cases, the file system looks
like it has enough space (du) to run NFS testing, but one or
more OSTs is full or is almost full. Since the default striping
for Lustre is a single stripe, some NFS tests will fail because
the file it is writing gets written to that almost full OST.

Increase the default stripe to all OSTs for all
parallel-scale-nfs tests.

Test-Parameters: trivial testlist=parallel-scale,parallel-scale-nfsv3,parallel-scale-nfsv4
Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: Iff187dc6c158d9063cd36ced0801b88cdc8814ae
Reviewed-on: https://review.whamcloud.com/29786
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Casper <jamesx.casper@intel.com>
Reviewed-by: Saurabh Tandan <saurabh.tandan@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10234 tests: use wait correctly in sanity-hsm test_3[4-6]

Test 34, 35 and 36 did not properly check the return code of wait
and potentially missed regressions.

Test-Parameters: trivial testlist=sanity-hsm
Signed-off-by: Quentin Bouget <quentin.bouget@cea.fr>
Change-Id: I21e8b16b55b946a8e5837549c3a3698379132e48
Reviewed-on: https://review.whamcloud.com/30102
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dominique Martinet <dominique.martinet@cea.fr>
Reviewed-by: Henri Doreau <henri.doreau@cea.fr>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10218 mdc: propagate changelog errors to readers

Store errors encountered by the changelog llog reader thread in the
crs_err field of struct changelog_reader_state so that they can be
propageted to changelog readers. In chlg_read() improve the error and
EOF reporting. Return -ERESTARTSYS when the blocked reader is
interrupted. Replace uses of l_wait_event() with
wait_event_interruptible().

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I279ec8ca11bfeb67f844acf12e3f7cba8f8aa1e6
Reviewed-on: https://review.whamcloud.com/30040
Tested-by: Jenkins
Reviewed-by: Quentin Bouget <quentin.bouget@cea.fr>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Henri Doreau <henri.doreau@cea.fr>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10226 tests: properly check for fileset clear and error

Make sure fileset info on nodemap is cleared everywhere before
continuing test.
Also, properly exit on this error.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I1d117ffef6c4a1002d3eba47a301e64830be4b4f
Reviewed-on: https://review.whamcloud.com/30037
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Stephan Thiell <sthiell@stanford.edu>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10182 utils: change MDT name too long error

Change the error message to something more generic.
fsname-MDT00 failed with "Name too long", which was wrong

Test-parameters: trivial

Signed-off-by: Ben Evans <bevans@cray.com>
Change-Id: If7c7de080c75dfdc853ff6d06e98a8ed074279cb
Reviewed-on: https://review.whamcloud.com/29884
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Patrick Farrell <paf@cray.com>
Reviewed-by: Steve Guminski <stephenx.guminski@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10168 kernel: kernel update [SLES12 SP2 4.4.90-92.45]

Update target and kernel_config files for new version

Test-Parameters: clientdistro=sles12sp2 testgroup=review-ldiskfs \
  mdsdistro=sles12sp2 ossdistro=sles12sp2 \
  mdtfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: If24eddeeadc825b1b3073439304c0958c0ae1b74
Reviewed-on: https://review.whamcloud.com/29839
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10166 mdc: invalid free in changelog reader

Use kthread_stop() to instruct the producer thread to exit
when the device is closed, and only then: release the CRS
structure.

The previous implementation left small time windows open,
during which the producer threads could free a structure
before the consumer thread was completely done with it.

Change-Id: Id2038aa9b7fcfd2c3347f628e749f9d2c265ac6e
Signed-off-by: Henri Doreau <henri.doreau@cea.fr>
Reviewed-on: https://review.whamcloud.com/29818
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Quentin Bouget <quentin.bouget@cea.fr>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10161 lbuild: fix the problem of determine default_iface

The script of determining the default interface is broken.
Remove complexity and replace with hostname

Test-Parameters: trivial
Signed-off-by: Li Xi <lixi@ddn.com>
Change-Id: I0d662e9e24bed17078e31b6b5ca305cff2c0548e
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/29776
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Gu Zheng <gzheng@ddn.com>

LU-10136 test: Lockahead test fixes

Two fixes:
1. Add version check for interop testing
2. Reduce count of locks requested in test15.  At 5000
locks, we would sometimes get some locks reclaimed during
the test due to memory pressure.  Reduce count to 500.

Signed-off-by: Patrick Farrell <paf@cray.com>
Change-Id: I693f8543b3579819aa64101ef30453627637ead6
Reviewed-on: https://review.whamcloud.com/29684
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Jenkins
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-9474 tests: add the stack_trap() utility function

This patch implements the stack_trap() function and uses it in
test_24d of sanity-hsm (where the need for it first arose).

The stack_trap() function is meant to be used to register multiple
cleanup actions on a single sigspec (most likely EXIT). Over time,
any call to bash's trap builtin in tests should be replaced by a call
to stack_trap().

Note that cleanup functions do not need to start with "trap - EXIT".
Each test is run in a subshell of its own (in run_one_logged()) and
therefore traps do not leak from one test to another. This is true
with or without this patch.

Test-Parameters: trivial testlist=sanity-hsm,sanity-hsm
Change-Id: I47f92a963f094d3d7055592a748e81ce357bff75
Signed-off-by: Quentin Bouget <quentin.bouget@cea.fr>
Reviewed-on: https://review.whamcloud.com/29653
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Dominique Martinet <dominique.martinet@cea.fr>
Reviewed-by: James Nunez <james.a.nunez@intel.com>

LU-10028 obdclass: Export Lustre mount flags and options

This patch help to export the information about Lustre special
mount flags and options via '/proc/mounts'.

The output for Lustre mounts by '/proc/mounts' is:
/dev/loop0 /mnt/lustre-mds1 lustre ro,svname=lustre-MDT0000,
mgs,osd=osd-ldiskfs,user_xattr,errors=remount-ro 0 0
/dev/loop1 /mnt/lustre-ost1 lustre ro,svname=lustre-OST0000,
mgsnode=192.168.59.129@tcp,osd=osd-ldiskfs,errors=remount-ro 0 0
/dev/loop2 /mnt/lustre-ost2 lustre ro,svname=lustre-OST0001,
mgsnode=192.168.59.129@tcp,osd=osd-ldiskfs,errors=remount-ro 0 0
192.168.59.129@tcp:/lustre /mnt/lustre lustre rw,flock,
user_xattr,lazystatfs 0 0

Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I6cdbcaf4fd5a6a3621ed675b1c4485073e774d8c
Reviewed-on: https://review.whamcloud.com/29228
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-10030 llite: extend inode flags into GET/SETXATTR ioctl

This is very useful for projet quota tools to save RPC call
to change project inherit attribute together with project ID.

Change-Id: I5b545bf2c5c601b552a85322c80994dabbca0b12
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/30072
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>

LU-10011 utils: refactor lfs quota codes

'lfs quota' default output logic is a bit hard to
understand, refactor codes to make it eaiser to
read.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I9b2a5867ed0416ccb92856538ef029bc59ac2c4b
Reviewed-on: https://review.whamcloud.com/29165
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Gu Zheng <gzheng@ddn.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>

LU-5024 mdc: don't assert on name pack

For old version of MDTs which don't support ATTRFID, open will pack
dentry name in the request, but this name may change if there is
concurrent rename, as a result an assertion will be triggerred.

It's inconvenient to copy name in advance or take dentry lock in the
call chain, so It's better to remove this assert, which is safe.

Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Change-Id: I58d8b8fbbd8057aa2e8618e8923d0432f31dac96
Reviewed-on: https://review.whamcloud.com/29161
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-9796 ldiskfs: improve inode allocation performance

Backport following upstream patches:

------
ext4: cleanup goto next group

avoid duplicated codes, also we need goto
next group in case we found reserved inode.
------

ext4: reduce lock contention in __ext4_new_inode

While running number of creating file threads concurrently,
we found heavy lock contention on group spinlock:

FUNC                           TOTAL_TIME(us)       COUNT        AVG(us)
ext4_create                    1707443399           1440000      1185.72
_raw_spin_lock                 1317641501           180899929    7.28
jbd2__journal_start            287821030            1453950      197.96
jbd2_journal_get_write_access  33441470             73077185     0.46
ext4_add_nondir                29435963             1440000      20.44
ext4_add_entry                 26015166             1440049      18.07
ext4_dx_add_entry              25729337             1432814      17.96
ext4_mark_inode_dirty          12302433             5774407      2.13

most of cpu time blames to _raw_spin_lock, here is some testing
numbers with/without patch.

Test environment:
Server : SuperMicro Sever (2 x E5-2690 v3@2.60GHz, 128GB 2133MHz
         DDR4 Memory, 8GbFC)
Storage : 2 x RAID1 (DDN SFA7700X, 4 x Toshiba PX02SMU020 200GB
          Read Intensive SSD)

format command:
        mkfs.ext4 -J size=4096

test command:
        mpirun -np 48 mdtest -n 30000 -d /ext4/mdtest.out -F -C \
                -r -i 1 -v -p 10 -u #first run to load inode

        mpirun -np 48 mdtest -n 30000 -d /ext4/mdtest.out -F -C \
                -r -i 3 -v -p 10 -u

Kernel version: 4.13.0-rc3

Test  1,440,000 files with 48 directories by 48 processes:

Without patch:

File Creation   File removal
79,033          289,569 ops/per second
81,463          285,359
79,875          288,475

With patch:
File Creation   File removal
810669          301694
812805          302711
813965          297670

Creation performance is improved more than 10X with large
journal size. The main problem here is we test bitmap
and do some check and journal operations which could be
slept, then we test and set with lock hold, this could
be racy, and make 'inode' steal by other process.

However, after first try, we could confirm handle has
been started and inode bitmap journaled too, then
we could find and set bit with lock hold directly, this
will mostly gurateee success with second try.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I234ff3027c8d96155d374c56b12aab7c4dc0dafd
Reviewed-on: https://review.whamcloud.com/29032
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Gu Zheng <gzheng@ddn.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9201 tests: don't check network for localhost

Don't try to check if the network to the local host is up, since
this just wastes time and adds noise to the test output logs.

Remove ancient 1.8 compatibility check.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I6c1f8df613afd657155da1309525e75b9d1c5bae
Reviewed-on: https://review.whamcloud.com/29011
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Steve Guminski <stephenx.guminski@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9899 tests: mount client on MGS for tests with pools

When a Lustre file system has the MGS and MDS on separate
nodes, the file system must be mounted on the MGS to allow
OST pools to work properly.

There are several tests that deal with creating, adding
to and deleting OST pools. Create functions to mount and
unmount the file system on the MGS in test-framework and call
these functions when necessary for the Lustre test suites
conf-sanity test 82b, 103
sanity tests 27D, 200, 220, 406 and
replay-single test 85b.

Test-Parameters: combinedmdsmgs=false testlist=sanity,replay-single,conf-sanity

Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: I4b3e331b8d1ea6c3f8c9ea8a571e26f66f4535f8
Reviewed-on: https://review.whamcloud.com/28806
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9894 lfsck: Print correct lfsck start info

When printing which scan type was started, lfsck iterates
through an array of types and compares the given type to
the types in the array.  Type numbers are a bit mask.

However, LFSCK_TYPES_SUPPORTED and LFSCK_TYPES_DEF are the
full mask, and they are the first in the array.  This means
we always match LFSCK_TYPES_SUPPORTED, which results in
always printing "all" as the type of scan.

Reorder the struct to increasing bit order to fix this.

Test-Parameters: trivial testlist=sanity-scrub,sanity-lfsck
Signed-off-by: Patrick Farrell <paf@cray.com>
Change-Id: Ibc82369ff6531b56ceda4b1c0a833720dc61f3f4
Reviewed-on: https://review.whamcloud.com/28604
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-8837 utils: make tools lightweight for lustre clients

Currently the lustre tools are built for the server target even
for the case of clients rpms. Rework the tools so server specific
code is not built for the client only case. The UAPI headers
lustre_log_user.h, lustre_barrier_user.h, and lustre_disk.h are
server specific and are not available in the upstream client.
So make our tools not to depend on those headers in the client
case. With this change it is simple step to make our tools
buildable against upstream. Last move user land specific code
from lustre_disk.h to mount_utils.h. Don't install server UAPI
headers on client targets.

Change-Id: I6b2b780ac0e7338a085ee6a8747097f0097efca1
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/28315
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5170 lfs: Standardize error messages in adjust_first_extent

Error and warning messages in adjust_first_extent() are updated
to a standard format.  Messages are prefixed with the name of the
utility and the command that caused the error.  User-provided values
are delimited with single quotes.

Test-Parameters: trivial
Signed-off-by: Steve Guminski <stephenx.guminski@intel.com>
Change-Id: I9240fbce1f8fb065a17eaff1e4bbda70b4a5b0c9
Reviewed-on: https://review.whamcloud.com/28304
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-5170 lfs: Standardize error messages in lfs_component_del()

Error messages in lfs_component_del() are updated to a standard
format.  Messages are prefixed with the name of the utility and the
command that caused the error.  User-provided values are delimited
with single quotes.

Test-Parameters: trivial
Signed-off-by: Steve Guminski <stephenx.guminski@intel.com>
Change-Id: I02f8b25226efbbe8bd7a96bd2495a98b8d733183
Reviewed-on: https://review.whamcloud.com/28255
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-9343 utils: update the usage of hsm_remove

Update the usage of hsm_remove in lfs to describe it more correctly.

Change-Id: I2934f0d7038f418dbad3a7e7365c7cd655ee28da
Signed-off-by: Hongchao Zhang <hongchao.zhang@intel.com>
Reviewed-on: https://review.whamcloud.com/26882
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Quentin Bouget <quentin.bouget@cea.fr>

LU-9052 lod: accept lfs mkdir from old client

MDS should also accept create striped directory request
from old client, but it will only honour stripe index in
this case, since the old client can only create remote
directory.

And also move the magic check from LOD to MDD to validate
LMV as early as possible.

Directory creation for echo client should go with lmv_user_md,
otherwise validation on user md will fail.

For striped directory, echo client should first choose stripe
before MD operation.

And also remove some unecessary lmv audit.

Re-enable 225b tests for DNE to test echo operations on
striped directory.

In mds-survey, if the current environment has more than one MDS,
let's skip the test. Because echo test will attach echo_client
individually on each MDSs(all of MDDs on the MDS), so each echo
client will not be able to access all of MDTs. But the test will
create a striped dir by MAX MDS count internally in mds_survey,
so echo client can not resolve the directory path in this case.

Signed-off-by: Di Wang <di.wang@intel.com>
Change-Id: I74a3712edfa36fd178f92965f8e35f5ff77ad422
Reviewed-on: https://review.whamcloud.com/25091
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>

LU-10199 utils: strip // from fid2path

There are a few ways to get a path with a // back from
fid2path, make sure we strip them all out

Signed-off-by: Ben Evans <bevans@cray.com>
Change-Id: I30591585ccf62e4d793ee658d2772ce1db242034
Reviewed-on: https://review.whamcloud.com/30078
Tested-by: Jenkins
Reviewed-by: Steve Guminski <stephenx.guminski@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-6142 lnet: remove last of the typedefs from LNet core

Change the remaining typedefs in LNet core to proper structures.
Several other style changes to fix checkpatch issues with code
impacted by typedef change.

Test-Parameters: trivial

Change-Id: Ic72b31698d1b1e7c612181a50aa7fa921fd417d7
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/29847
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Doug Oucharek <dougso@me.com>
Reviewed-by: Sonia Sharma <sonia.sharma@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10040 nodemap: add nodemap idmap correctly

There are 3 situations when adding an idmap {id_client:id_fs} to
a nodemap tree:
 - both id_client and id_fs are matched, that means this idmap
   already exists, so return -EEXIT;
 - neither id_client nor id_fs is matched, that means this is a
   new idmap, so insert it;
 - only "id_client" or "id_fs" is matched, since idmap uses
   "id_client" as index key, we need to delete that old idmap and
   its index by its id_client, and then insert it again. In the
   original implementation, idmap_insert() calls rb_replace_node()
   to replace the old idmap without re-sorting its both id_client
   and id_fs, so that this new added idmap can't be found by
   idmap_search() due to its wrong left and right nodes, and can't
   be deleted either.

Also, this patch improves the following code:
 - nm_idmap_lock: use type rw_semaphore instead of rwlock, because
   nodemap_idx_idmap_del() may sleep on FS operations while this
   lock is being held.
 - Add "update_idmaps" to sanity-sec.sh test_15 to verify this fix,
   and improve other part of this test case as well to make it run
   correctly.

Test-Parameters: testlist=sanity-sec
Signed-off-by: Stephan Thiell <sthiell@stanford.edu>
Signed-off-by: Emoly Liu <emoly.liu@intel.com>
Change-Id: Icf777f14c2e1dd56fa5cd0eb56666240e206d199
Reviewed-on: https://review.whamcloud.com/29364
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9019 mdt: migrate to 64 bit time

Port the remaining time code from 32 bits to 64 bits.
Replace cfs_time_current_sec() to avoid the overflow
issues in 2038 with ktime_get_real_seconds().

Change-Id: Ia97dbfe00cd90dc6ef04583ba2eb512a8feca8c8
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/28788
Reviewed-by: Quentin Bouget <quentin.bouget@cea.fr>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9983 llog: skip non-exist log cancellation

Let's skip the record cancellation if the llog does
not exist at all, which usually happens when remote
log creation failed.

Signed-off-by: Di Wang <di.wang@intel.com>
Change-Id: If76d29d8fd72f3c5b619b72a899b18459c608dbb
Reviewed-on: https://review.whamcloud.com/29218
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>

LU-8998 test: Test 'lfs find' composite file options

The Progressive File Layout feature expanded the options
to find files using the 'lfs find' utility. Files can be
searched for with the component start, end, count and flags
options.

These 'lfs find' options need to be exercised.

Test-Parameters: trivial
Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: I36a6e774fa77b4089642ff5f2f8d482b3196aa62
Reviewed-on: https://review.whamcloud.com/26926
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Steve Guminski <stephenx.guminski@intel.com>

LU-3285 merge: 'dom' branch merging

Merge remote-tracking branch 'origin/dom'

Signed-off-by: Mikhal Pershin <mike.pershin@intel.com>
Change-Id: I8c20ac4f5fdae6ec7ad034fbb7f5fda656f03c8b

LU-3285 mdc: remove wrong assertion from mdc_object_ast_clear

It was added wrongly during DoM development but this
assertion is not correct due to AGL locks,
check LU-6042.

Signed-off-by: Mikhail Pershin <mike.pershin@intel.com>
Change-Id: Id14d408cbb7365cd9bb973c48e076abac4cd878f
Reviewed-on: https://review.whamcloud.com/30021
Tested-by: Jenkins
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-3285 tests: disable DOM files in racer by default

Use RACER_ENABLE_DOM=true to enable dom files in racer
explicitly.

Test-Parameters: mdssizegb=20 mdscount=1 mdtcount=1 testlist=racer
Signed-off-by: Mikhal Pershin <mike.pershin@intel.com>
Change-Id: Ibcbbff0dd995903915a2404c1076108e2016a4cb
Reviewed-on: https://review.whamcloud.com/30020
Tested-by: Jenkins
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-10174 osd-zfs: Fix build against ZFS 0.6.5.x

Ensure all values used in https://review.whamcloud.com/29242 are
actually used conditionally.  This allows us to build against
ZFS 0.6.5.7 again.

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Ide560e21f7b7fedc0c107b1943602e5af99b478c
Reviewed-on: https://review.whamcloud.com/29866
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>

LU-10151 lnet: report better config error with lnetctl

Configuring net or route currently reports generic
message on error. Adding specific error message for
the missing parameters.

With importing config from yaml file, specific
error messages help in finding the exact error.

Change-Id: I7d9bdeb2280f4548897d9958aab7d451b981e8bd
Signed-off-by: Sonia Sharma <sonia.sharma@intel.com>
Test-Parameters: trivial
Reviewed-on: https://review.whamcloud.com/29820
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Amir Shehata <amir.shehata@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Doug Oucharek <dougso@me.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10067 mdt: reinit lock when fail to try lock

In mdt_lock_objects_in_linkea, the mdt_lock_handle should be
re-initialized when the first try (mdt_object_lock_try) failed.

Change-Id: I8232c40c3e9e239391b7a43d9d16f5563bd794e5
Signed-off-by: Hongchao Zhang <hongchao.zhang@intel.com>
Reviewed-on: https://review.whamcloud.com/29597
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9855 obdclass: cleanup obd_class.h defines

Get rid of OBT and CTXTP defines.
Remove OBD_CHECK_DT_OP
Convert EXP_CHECK_MD_OP into a function.
Remove CTXT_CHECK_OP.

Signed-off-by: Ben Evans <bevans@cray.com>
Change-Id: Ia2adc579602818ebdba0a40c99da9e3af1575d24
Reviewed-on: https://review.whamcloud.com/29286
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Steve Guminski <stephenx.guminski@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10041 tests: sanity/51 to cleanup properly

sanity/51 should cleanup properly in case of failure

Change-Id: Ib1c0e1a116a6802f2bf25d665be5c076ae07c8b8
Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-on: https://review.whamcloud.com/29246
Tested-by: Jenkins
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Wei Liu <wei3.liu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1757 brw: add short io osc/ost transfer.

There's no need to do target bulk io for small amount of
data, and it requires extra network operations.

For this case we add short i/o.  When the i/o size is less
than or equal to some number of pages (default 3), we
encapsulate the data in the ptlrpc request.

With this patch, 4k direct i/o read latency on a Cray Aries
network (data is on flash on another node on the Aries)
drops from ~280 microseconds to ~200 microseconds.  Write
latency drops from ~370 microseconds to ~350 microseconds
(much more of write latency is waiting for write commit).

This translates to about a 25-30% performance improvement
on 4k direct i/o reads and 4k random reads.  (Write
performance improvement was small to non-existent.)

Improvement was similar with 8k i/o.

Buffered sequential i/o sees no improvement, because it
does not perform small i/os.

Performance data:
        access             = file-per-process
        pattern            = segmented (1 segment)
        ordering in a file = random offsets
        ordering inter file= no tasks offsets
        xfersize           = 4096 bytes
        blocksize          = 100 MiB

nprocs  xfsize  shortio dio     random  Read (MB/s)
1       4k      no      yes     no      15.0
8       4k      no      yes     no      73.4
16      4k      no      yes     no      81.1
1       4k      yes     yes     no      16.5
8       4k      yes     yes     no      95.2
16      4k      yes     yes     no      107.3
1       4k      no      no      yes     15.5
8       4k      no      no      yes     73.4
16      4k      no      no      yes     81.2
1       4k      yes     no      yes     16.8
8       4k      yes     no      yes     95.0
16      4k      yes     no      yes     106.5

Note even when individual i/o performance is not improved,
this change reduces the # of network operations required
for small i/o, which can help on large systems.

Signed-off-by: Patrick Farrell <paf@cray.com>
Change-Id: I70050935eaa0a5e98ca437e18e730be4aa0e4700
Reviewed-on: https://review.whamcloud.com/27767
Tested-by: Jenkins
Reviewed-by: Alexey Lyashkov <c17817@cray.com>
Reviewed-by: Alexandr Boyko <c17825@cray.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7991 quota: project quota against ZFS backend

This patch enables the project quota feature against ZFS
backend. The project quota patch for ZFS itself will be
landed via https://github.com/zfsonlinux/zfs/pull/6290.
Such patch may be released via zfs-0.7.x (x >= 5).

This patch also cleanup the quota logic for ZFS backend
and sanity-quota.sh

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Ic488efe19a6e34a3615811f79fb9ea7954ad9263
Reviewed-on: https://review.whamcloud.com/27093
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

New tag 2.10.55

Change-Id: I999629a7b3939fb827729e053fc9b52c24ca69ab
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10199 utils: disable sanity test 56xb

The new sanity test 56xb introduced by changeset 25851 fails due
to a mismatch in the returned vs. expected value.  Disable the test
until it can be fixed.

Test-Parameters: trivial
Signed-off-by: Steve Guminski <stephenx.guminski@intel.com>
Change-Id: I12b88eccdf7954ae854133b48cadf7054c8f683e
Reviewed-on: https://review.whamcloud.com/29954
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-10164 kernel: kernel update [SLES12 SP3 4.4.92-6.18]

Update target and kernel_config files for new version

Test-Parameters: clientdistro=sles12sp3 testgroup=review-ldiskfs \
  mdsdistro=sles12sp3 ossdistro=sles12sp3 \
  mdtfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I4b046f0ac64b75545cd0e2a3a44e9117f4d79ef4
Reviewed-on: https://review.whamcloud.com/29793
Tested-by: Jenkins
Reviewed-by: Minh Diep <minh.diep@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10142 kernel: kernel update RHEL7.4 [3.10.0-693.5.2.el7]

update RHEL 7.4 kernel to 3.10.0-693.5.2.el7

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I6f22d9d96b86d7bd860a400cd803fad26a6b5d1a
Reviewed-on: https://review.whamcloud.com/29727
Tested-by: Jenkins
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10152 tests: reformat after conf-sanity test 99

conf-sanity test 99 reformats an OST with options that need
to be cleared to allow future tests to complete successfully.

Reformatting the file system will clear the OST format options.

Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: I32be4bab5a22b9e5f939c6478c96255db30dc0eb
Reviewed-on: https://review.whamcloud.com/29717
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Casper <jamesx.casper@intel.com>
Reviewed-by: Wei Liu <wei3.liu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10150 build: Build with ZFS 0.7.3

Update ZFS and SPL version to 0.7.3

Changelog: https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.7.3

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I88f0f2601262c0c535431d256a159d9e9d6ebd75
Reviewed-on: https://review.whamcloud.com/29712
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9946 nodemap: have fileset on default nodemap.

When a directory is defined as fileset on the default nodemap entry,
all clients that fall into default will be imposed subdirectory
mount on this directory.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I3491bce4f8c2011629e56494bf1a0de57f19829d
Reviewed-on: https://review.whamcloud.com/28858
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-8174 mdt: remove use of typedef mds_reint_t

Replace the use of mdt_reint_t with enum mds_reint_op.

Remove the old mds_reint_t and mdt_reint_t since they were only used
in one place in the Lustre code and are unlikely to cause additional
patch breakage.  The majority of references are in wireshark.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I3932db8e6dc9ac83325900d3f382c648ac57b8c0
Reviewed-on: https://review.whamcloud.com/27494
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Ben Evans <bevans@cray.com>

LU-6051 utils: allow lfs_migrate to handle hard links

Detect files that are hard links to files that have already been
migrated, and skip them.  This prevents unnecessary work and speeds
up the migration process.  Sanity test 56xb has been added to test
the new behavior.  A new option '-S <stripe_size>' has been added
to lfs_migrate, and allows the test to be run with a single OST.

Previously, the default behavior was to skip migration of any
file name that contained hard links other than itself. Therefore,
the file in the link set would not be migrated at all.  The '-l'
option could be used to override.  However, there was no attempt to
detect which file names were hard linked to already migrated files,
so the file would be migrated multiple times.

The new behavior automatically migrates the file once and skips
other file names in the link set.

Test-Parameters: trivial
Signed-off-by: Steve Guminski <stephenx.guminski@intel.com>
Signed-off-by: Evan Felix <evan.felix@pnnl.gov>
Change-Id: I43678b34931507c0fe30c1c40ca2dc2d47f86aeb
Reviewed-on: https://review.whamcloud.com/25851
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Henri Doreau <henri.doreau@cea.fr>

LU-7813 lov: rename LOV_MAGIC_V*_DEF to *_DEFINED

Rename LOV_MAGIC_V1_DEF to LOV_MAGIC_V1_DEFINED so that it is not
confused with "_DEFAULT" as one might expect.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: Idb98fc24974a105077bf75bb122006fad34d69e2
Reviewed-on: https://review.whamcloud.com/18888
Tested-by: Jenkins
Reviewed-by: Steve Guminski <stephenx.guminski@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10086 libcfs: use dynamic minors for /dev/{lnet,obd}

Request dynamic minor allocation when registering /dev/lnet and
/dev/obd. Remove the obsolete create-device-if-not-found code from
register_ioc_dev().

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I59c70912b4729f58a76dc6107b3e1d7379c6d7a3
Reviewed-on: https://review.whamcloud.com/29741
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10007 pacemaker: Use lctl and load lustre

When scripts are started, load lustre module.
Use lctl instead of directly accessing health_check file.

Test-Parameters: trivial
Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I4a81248939464e498006dc2c4072d44685add018
Reviewed-on: https://review.whamcloud.com/29144
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-9611 lov: allow lov.*.stripe{size,count}=-1 param

Since LU-7344 patch http://review.whamcloud.com/16930 was landed,
lov_stripeoffset_seq_write() and lov_stripecount_seq_write() have
incorrectly checked that lov.*.stripecount and lov.*.stripeoffset
are not negative.  In fact they can both be "-1" to indicate that
the filesystem-wide default value should be used. These parameters
can also be set internally if using "lfs setstripe -c -1 $MOUNT"
or "lfs setstripe -i -1 $MOUNT" to set the system wide default,
generating console errors on the MDS from class_process_proc_param():

    lov.: error writing proc entry 'stripecount': rc = -34
    lov.: error writing proc entry 'stripeoffset': rc = -34

Fix these functions to allow "-1" as a valid value.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I295d2591d535b039634689524a29725e963ebbe5
Reviewed-on: https://review.whamcloud.com/27946
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7988 hsm: wake up cdt when requests are empty

The coordinator only runs once per second, we need a mechanism
to send more work when everything is done (cdt_request_count
goes to zero)

Without this, there is a hard limit of max_requests per sec
requests that can be processed, causing performance issues
with small files.

Signed-off-by: Ben Evans <bevans@cray.com>
Change-Id: I563666a1a3e53f0ec5908de593de71ff4d925467
Reviewed-on: https://review.whamcloud.com/29742
Reviewed-by: Frank Zago <fzago@cray.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Sergey Cheremencev <cherementsev@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10098 scripts: Fix mounted check in Lustre RA

The "Lustre" resource agent for pacemaker can mis-identify a resource
as being mounted if it's mountpoint is a substring match for anything
else in /proc/mounts.  Change the lustre_is_mounted() function to
check to make sure it's a lustre fs mounted at mountpoint and the
"source" (i.e. device) is the target we expect.

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Ib877b0dc3d3ce0d93fd4663aa2418ac21d670428
Reviewed-on: https://review.whamcloud.com/29351
Tested-by: Jenkins
Reviewed-by: Malcolm Cowe <malcolm.j.cowe@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9994 obdclass: fix llog_cat_id2handle() error handling

Patch for LU-9153 ("llog: consolidate common error checking") has
introduced a regression in llog_cat_id2handle() error handling
path by adding llog_cat_process_common() common routine additional
call in sequence and allowing it to return zero even when catalog
entry was invalid and it had cleared it instead to populate
llog_handle, thus causing an exception when handle was later
dereferenced in llog_process_thread().

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: I50153b931e3c1567bfe9c15564ba29fabe3a2d4c
Reviewed-on: https://review.whamcloud.com/29370
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>

LU-10132 llite: handle xattr cache refill race

In ll_xattr_cache_refill() if the xattr cache was invalid (and no
request was sent) then return -EAGAIN so that ll_getxattr_common()
caller will fetch the xattr from the MDT.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ia9ec7424e8786d92bdecf4897fafcf71d5061fb1
Reviewed-on: https://review.whamcloud.com/29654
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9951 lustre_compat: add wrapper function for posix_acl_update_mode

posix_acl_update_mode is introduced in kernel 4.9, add the precheck
of it, if not exists, use inline wrapper function instead.

Change-Id: I8a1476d611c387a88efef5d5b8707edf5feacca8
Signed-off-by: Gu Zheng <gzheng@ddn.com>
Reviewed-on: https://review.whamcloud.com/28871
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Jenkins
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9558 llite: port lustre to unified handling of bdi

For the linux 4.12 kernel the bdi handling was unified for all
file systems. This was done by allocating struct backing_dev_info
separately instead of embedding it inside superblock. For older
kernels we move all the bdi handling lustre does to the function
super_setup_bdi_name() which is what exist in the latest kernels.

Linux-commit: 9594caf216dc0fe3e318b34af0127276db661241

Change-Id: I5af60ea3661e3d3a97973fd99a79c28dcd1ce1cc
Signed-off-by: Jan Kara <jack@suse.cz>
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/28511
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7251 osp: do not assign commit callback to every thandle

with OSP there is a risk of getting a lot of commit callbacks.
say, 10K unlinks/sec on 4-striped files could result in 4*10K*5
= 200K commit callbacks. this patch implements another schema:
every OSP registers own callback every second. this should result
in 4*5 commit callbacks in the same situation. in case of forced
sync the commit callback is registered unconditionally.

the patch removes th_tags and th_ctx from struct thandle as they
are not used anymore. this elimintates 3 allocations from every
transaction:
(lu_object.c:1714:keys_init()) kmalloced 'ctx->lc_value': 320
(update_records.c:1217:update_key_init()) kmalloced 'value': 408
(osp_dev.c:1807:osp_txn_key_init()) kmalloced 'value': 4

Change-Id: I460d5eccb585b166423d84d5c142af2e27751d8b
Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-on: https://review.whamcloud.com/17270
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>

LU-9019 ldlm: migrate the rest of the code to 64 bit time

Replace the last cfs_time_current_sec() to avoid the overflow
issues in 2038 with ktime_get_real_seconds(). Reduce the jiffies
usage to the bare minimum which is useage for mod_timer() and
schedule_timeout(). This makes the ldlm totally 64 bit time
compliant.

Change-Id: Iaee92c17d51fdfc55bd26e9e813e30a6ce794856
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/29295
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10015 o2iblnd: fix race at kiblnd_connect_peer

cmid will be destroyed at OFED if kiblnd_cm_callback return error.
if error happen before the end of kiblnd_connect_peer, it will touch
destroyed cmid and fail as
(o2iblnd_cb.c:1315:kiblnd_connect_peer())
            ASSERTION( cmid->device != ((void *)0) ) failed:

Seagate-bug-id: MRP-4592
Signed-off-by: Alexander Boyko <alexander.boyko@seagate.com>
Change-Id: I83eb5bceeb567acef0316498b936d25d6c6ccd95
Reviewed-on: https://review.whamcloud.com/29134
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alexey Lyashkov <c17817@cray.com>
Reviewed-by: Doug Oucharek <dougso@me.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-8066 obd: migrate to ksets

Lustre's sysfs only uses kobjects but with the introduction of
ksets we can use functionality like kset_find_kobj() and uevents.
Currently lustre is layered as lustre_kobj -> class -> obd device.
This patch changes the obd devices and the top level lustre_kobj
into ksets. The class level is kept as kobjects but are bound to
the top level lustre kset so they searchable and uevents can be
created for them. Also much of the class functionality can be
replaced with what ksets can do. With obd devices now ksets we
can replace lprocfs_kset_register with lprocfs_obd_setup. Some
of the sysfs attributes were not cleaned up so added proper
removal. Reversed what the default sysfs attributes are. This
will be needed for the replacement functionality for
class_process_proc_param().

Change-Id: I3ced5f69ace6a0a9a6bc51957f20a0caecdbafc9
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/28948
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Sonia Sharma <sonia.sharma@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3285 mdc: implement own mdc_io_fsync_start()

DoM lock cancelation may happen in llite and call
cl_sync_file_range() function to flush related data and
it uses DOM component end as limit for data to be flushed.
However related lock and extent are expanded to EOF and
this is asserted in osc_cache_writeback_range().
To avoid this a MDC uses own version of cio_start for FSYNC
and osc_cache_writeback_range() is called on whole DoM object
no matter what start/end are supplied by upper layers.

Test-Parameters: mdscount=1 mdtcount=1 mdssizegb=20 testlist=sanity-dom,dom-performance,racer
Signed-off-by: Mikhal Pershin <mike.pershin@intel.com>
Change-Id: I044faa95d3664a0448c21e9a2f1c2e5dd3a69cde
Reviewed-on: https://review.whamcloud.com/29813
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-3285 osc: remove wrongly applied assertion.

This assertion was removed from master by commit
4311cdaa832 but kept wrongly in dom branch.

Test-Parameters: mdssizegb=20 testlist=sanity-dom,dom-performance
Signed-off-by: Mikhal Pershin <mike.pershin@intel.com>
Change-Id: I403eaca68fd334feb80c5db8c295eef9dbe171ac
Reviewed-on: https://review.whamcloud.com/29811
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-3285 llite: check layout size after cl_object_layout_get

Check that layout size is not zero before doing flush for
DoM object. Also patch does that:
- initializes all values in cl_layout explicitly
  in lov_object_layout_get().
- changes cl_dom_comp_size to u64 for future needs, e.g.
  importing an existing filesystem image directly.

Test-Parameters: mdssizegb=20 testlist=sanity-dom,dom-performance
Signed-off-by: Mikhal Pershin <mike.pershin@intel.com>
Change-Id: Ic6ff06ad4022f85686fda238eae34ad38ff2dcd0
Reviewed-on: https://review.whamcloud.com/29810
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-3285 lod: don't initialize OSP objects for DoM

Skip OSP object initialization for DoM component in
lod_use_defined_striping().

Signed-off-by: Mikhal Pershin <mike.pershin@intel.com>
Change-Id: I720eb100198b46974187b6f9d7931650e4f8742b
Reviewed-on: https://review.whamcloud.com/29812
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-3285 lfs: handle -L option alphabetically

Move -L option handling in lfs.c to handle it
alphabetically.

Signed-off-by: Mikhal Pershin <mike.pershin@intel.com>
Change-Id: I6ade41eb8e75df82f0306c89d446955e546f810c
Reviewed-on: https://review.whamcloud.com/29809
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-3285 tests: style fixes in DoM tests

Several style fixes in DoM tests and code from review debts.

Signed-off-by: Mikhail Pershin <mike.pershin@intel.com>
Change-Id: I4ee011025a008cfa474db3d1c2f86b2c250483c4
Reviewed-on: https://review.whamcloud.com/29681
Tested-by: Jenkins
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-10041 osd: osd-zfs to choose dnode size

depending on dnodesize property it can be:
legacy (512 bytes), auto (512 bytes to 16K) or absolute
size (512, 1024, 2048, 4096, 8192, 16384).

Change-Id: Iea35d8ae850523440272467320410850821f484c
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/29242
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>

LU-10131 llite: Update i_nlink on unlink

Currently, the client inode link count is not updated on
last unlink.  This is fine because the dentries are all
gone and the inode is eligible for reclaim, but it's still
incorrect.  This causes two problems:

1. Inode is not immediately reclaimed
2. i_nlink count is > 0 for a fully unlinked file, which
confuses wrapfs

On last unlink, the MDT sends back attributes.  Use the
nlink count from these to update the client inode.

Remove null check inherited from ll_get_child_fid, because
the inode should never be null on an unlink.

Re-enabled test 76, which passes with this patch.
Removed slab allocator tuning from test_76, because slab is
no longer the default Linux allocator.

Signed-off-by: Patrick Farrell <paf@cray.com>
Change-Id: Ib253b5cf3d35188554cf8fc33a8a3d4b8bb237e8
Reviewed-on: https://review.whamcloud.com/29651
Tested-by: Jenkins
Reviewed-by: Ben Evans <bevans@cray.com>
Reviewed-by: Alexey Lyashkov <c17817@cray.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andrew Perepechko <c17827@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10089 o2iblnd: use IB_MR_TYPE_SG_GAPS

When allocating fastreg buffers use IB_MR_TYPE_SG_GAPS
instead of IB_MR_TYPE_MEM_REG, since the fragments we provide
the fast registration API can have gaps. MEM_REG doesn't handle
that case.

Test-Parameters: trivial
Signed-off-by: Amir Shehata <amir.shehata@intel.com>
Change-Id: I36dba6f676fdbc60730aed7c50d71f2a6b7c2549
Reviewed-on: https://review.whamcloud.com/29551
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Doug Oucharek <dougso@me.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-8475 target: use slab allocation

The patch adds kmem slabs for target threads and session info
to improve allocation and better accounting.

Signed-off-by: Alexander Boyko <alexander.boyko@seagate.com>
Signed-off-by: Alexey Lyashkov <alexey.lyashkov@seagate.com>
Change-Id: Ia0a93d410618c5d7724f2dcc86f1bcb9ae32e572
Seagate-bug-id: MRP-2836
Reviewed-on: https://review.whamcloud.com/21654
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alexey Lyashkov <c17817@cray.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10101 tests: correct sanity-quota call to quota_error

sanity-quota test 7e calls quota_error with a "-u"
argument. Input to quota_error should not be prefaced
with  "-".

Test-Parameters: trivial testgroup=review-zfs-part-1,review-dne-part-2
Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: Iac84e3fb2348588a157beefcf4d554b1ac3171ed
Reviewed-on: https://review.whamcloud.com/29358
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Wei Liu <wei3.liu@intel.com>
Reviewed-by: Saurabh Tandan <saurabh.tandan@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10046 misc: replace LASSERT() with CLASSERT()

Some code consistency checks are being done at runtime with LASSERT()
when they could be done at compile time with CLASSERT(). This might
miss defects introduced into the code if that particular code path is
not exercised during testing.

Replace LASSERT() with CLASSERT() in such cases.

Style cleanup for related code.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I8dca903109b7151de59afe17fe9ca311119d1b36
Reviewed-on: https://review.whamcloud.com/29256
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10141 llapi: integer overflow in llapi_changelog_start

Use the appropriate type to store and check the return value from lseek.
This prevents from misinterpreting high offsets as errors.

Change-Id: I15e92be3454af20ee6611c2a7ddfc1b597d639c2
Signed-off-by: Henri Doreau <henri.doreau@cea.fr>
Reviewed-on: https://review.whamcloud.com/29674
Tested-by: Jenkins
Reviewed-by: Thomas LEIBOVICI <thomas.leibovici@cea.fr>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Stephan Thiell <sthiell@stanford.edu>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10023 kernel: kernel update [SLES12 SP3 4.4.82-6.9]

Update target and kernel_config files for new version

Test-Parameters: clientdistro=sles12sp3 testgroup=review-ldiskfs \
  mdsdistro=sles12sp3 ossdistro=sles12sp3 \
  mdtfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs

Change-Id: Ie65afb3d7356c4679f4f37f4af324e955261b5af
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-on: https://review.whamcloud.com/29560
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4705 mdc: improve mdc_enqueue() error message

Include the parent/child FIDs and name in the mdc_enqueue()
debug message.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I7b84921a52a4650be70fe87eea691ba2217bb3a6
Reviewed-on: https://review.whamcloud.com/28978
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Steve Guminski <stephenx.guminski@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9897 utils: remove libcfsutils.a and libptlctl.a

Currently lustre creates many libraries and combines them in
redudant ways. Broke up libptlctl.a and merged debug.c and
portals.c into lctl. The application lustre_rsync pulled in
way too much extra code that is not needed just for the function
obd_initialize(). Instead just call register_ioc_dev() directly
for lustre_rsync. This removes the libptlctl.a/portals.c
dependency. In time the portals.c code can be replaced by the
work from liblnetconfig. Integrated cyaml into liblnetconfig
instead of directly linking cyaml.c into lnetctl. This way
we can take advantage of YAML in the future for lnet selftest
and lustre utilities. Only a small change was needed for lnet
selftest to be dependent on liblnetconfig instead of libptlctl.a.

Change-Id: Ic1caaa01b3faedf90dc7ae8bc26ee40396a52a07
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/28752
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9578 llite: use security context if it's enabled in the kernel

if it's disabled, then Lustre stop to work properly (can not create
files, etc)

Change-Id: I1e431ec95a2b0613b43893567eb6d1a64ec832de
Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-on: https://review.whamcloud.com/27364
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Reviewed-by: Chris Horn <hornc@cray.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9452 ldlm: remove MSG_CONNECT_LIBCLIENT support

Remove old server code that handled liblustre client connections,
marked with MSG_CONNECT_LIBCLIENT and associated code checking for
exp_libclient.  Servers will now outright refuse connections from
liblustre clients with a clear message, rather than allowing the
connection and pretending to work.  Liblustre client support was
broken and removed years ago.

There are still some liblustre remnants in the code (e.g. blocked
lock handling for LDLM_FL_CANCEL_ON_BLOCK), but that has more
complex semantics and should be removed in a separate patch.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: Ifbea507e82d758f849db24094c5cc0a8003ebbe5
Reviewed-on: https://review.whamcloud.com/26972
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7990 llite: increase whole-file readahead to RPC size

Increase the default whole-file readahead limit to match the current
RPC size.  That ensures that files smaller than the RPC size will be
read in a single round-trip instead of sending multiple smaller RPCs.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I3bdb1c7f92c546d58951a9e6b783af23c83ebbe5
Reviewed-on: https://review.whamcloud.com/26955
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Patrick Farrell <paf@cray.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>