LU-790 tests: remove old get_lustre_version() calls

Replace old get_lustre_version() calls with lustre_build_version(),
since the former was deleted in commit 2a223541d2.  The few places
using get_lustre_version() were non-critical (just advisory messages)
but may generate errors messages in the test output.

Fix sanity-sec.sh to check the version on the mgs facet instead of
the mds1 facet, in case they are on separate nodes at some point.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I448bc13abbd36e86d0aa8cd6e22bacc9b6def27c
Reviewed-on: http://review.whamcloud.com/14768
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>

LU-6599 build: warn on likely uses of GPLv3

Add a warning to checkpatch.pl to warn on likely uses of GPLv3
headers.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I5fc48c9aa7495d949734db432b6a0b413379e99f
Reviewed-on: http://review.whamcloud.com/14863
Tested-by: Jenkins
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6517 lmv: fix potential null pointer dereference

In lmv_disconnect_mdc do procfs removal only if we actually know the name.

Change-Id: Icb658a8a2007eea254ea5ce70b95c5d73df89914
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/14605
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Jenkins
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-5919 build: Make "make rpms" use "make srpm" output

We rework the rpms and srpm Make targets.  The srpm
target generates the source rpm package much as it did
before, but it puts the results in a known temporary
rpm build tree.  This allows the "make rpms" target to
simply depend on the srpm target, and then build the
binary rpms from the source rpm.

Previously the rpm target _also_ generated a source rpm
package, which could be redudant, and failed to verify that
the source rpm package was itself buildable.

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>
Change-Id: I9b2e83e000aca62b5d4841e2ebe9e91fa1db300f
Reviewed-on: http://review.whamcloud.com/12961
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6529 ldlm: cancel aged locks for LRUR

It doesn't make sense to keep the very aged lock even with the
LRUR policy. This patch decreased the default ns_max_age from 10
hours to 65 minutes and changed LRUR policy to cancel very aged
locks.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: Iabf6f50a7837821afcf2a75b6c17c7484b34667f
Reviewed-on: http://review.whamcloud.com/14856
Tested-by: Jenkins
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5814 lov: use obd_get_info() to get def/max LOV EA sizes

Use obd_get_info() to get the default and maximum LOV EA sizes (along
with maximum cookiesize) from LOV. Remove the then unused function
obd_size_diskmd() and the unused get info key KEY_LOVDESC. When
computing the maximum LOV EA size use the active OST count
(ld_active_tgt_count) rather than the OST count (ld_tgt_count).

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I4a7eb158d9cb8f4828f5cb272df8d1c843f2eb6b
Reviewed-on: http://review.whamcloud.com/13695
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6573 test: fix sanity test_224c

It should always wait for the changing to the at_max/obd_timeout
taking effect.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I681ecb4a72c0b5f4f69e92341e7bde1b554c12a5
Reviewed-on: http://review.whamcloud.com/14844
Tested-by: Jenkins
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Andreas Dilger <andreas.dilger@intel.com>

LU-6593 kernel: kernel update [RHEL7.1 3.10.0-229.4.2.el7]

kernel update [RHEL7.1 3.10.0-229.4.2.el7]

Test-Parameters: clientdistro=el7 testgroup=review-ldiskfs \
  mdtfilesystemtype=ldiskfs mdsfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: Id2e221c70d591f6ca852f87a9bdb317c7bd6a2c8
Reviewed-on: http://review.whamcloud.com/14804
Tested-by: Jenkins
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4772 mgs: free MGS fsdb before export barrier

The mgs_fsc keeps export reference and should be cleaned up
before the waiting for exports cleanup. Patch moves FSDB cleanup
functions which cleanups also mgs_fsc ahead of obd_export_barried()

Signed-off-by: Mikhail Pershin <mike.pershin@intel.com>
Change-Id: I98aa2e669a7ed2cba3b31df44fd3e96dcbbb6807
Reviewed-on: http://review.whamcloud.com/14443
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6557 llite: remove unused ll_max_rw_chunk

ll_max_rw_chunk seems to be unused ever since we implemented CLIO in 2.0,
so remove it and all supporting infrastructure.

Change-Id: Ie4d4142c3b70aa4fa757770b03b220de2253a681
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/14657
Tested-by: Jenkins
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-6557 tests: ll_max_rw_chunk is going away, remove from test framework

ll_max_rw_chunk is being removed already, but it's used by
parallel group lock test too, so need to remove it from there.

The parameter itself was a noop since Lustre 2.0

Change-Id: Ib4df4d916348b408fd09ec09fba7ae2d9ae8e2d1
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/14746
Tested-by: Jenkins
Reviewed-by: Jian Yu <jian.yu@intel.com>

LU-6505 libcfs: Use after free in __proc_cpt_table

Only free the too small buffer if it's really too small, not always.

Change-Id: I3fbdcb74dd0091c43aeb3e5b98460c203595bc19
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/14602
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: frank zago <fzago@cray.com>

LU-6215 libcfs: shrinker code needs atomic.h

In the latest linux kernel code struct shrinker now
includes a new atomic_long_t so atomic.h is required
for the libcfs autoconf test will pass.

Change-Id: I7b5a5cc030bc8e8fc4cbd0c47bc75f73b07742f7
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: http://review.whamcloud.com/14662
Tested-by: Jenkins
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Thomas Stibor <t.stibor@gsi.de>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5407 tests: Disable replay-single test 58c

Replay-single test 58c is failing review-zfs with enough
frequency that is it interfering with landing patches.

Test 58c is temporarily disabled for zfs only.

Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: Ief1b4d5c9ab6bf163bdae9737c1bffcdbef763cf
Reviewed-on: http://review.whamcloud.com/14864
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5683 clio: add CIT_DATA_VERSION

Add a new cl_io type CIT_DATA_VERSION to get file data version. Remove
the old data version code (cl_object_obd_info_get() and
cl_object_data_version()). Add test_37() to sanity-hsm.sh to ensure
that data version is computed under the correct layout.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: If371c8e8c18e91d476f94de2e0170199252c088d
Reviewed-on: http://review.whamcloud.com/14649
Tested-by: Jenkins
Reviewed-by: Henri Doreau <henri.doreau@cea.fr>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

New tag 2.7.54

Change-Id: I43dc353a8463eae5a2cc0d21c335e8fb74a9d575

LU-6599 header: Change erroneous GPLv3 header to GPLv2

The template for lustre_dlm_flags.h was including default
gpl definition which happened to be v3, and should have been v2.

Update the header to reflect that.
Also while we are at it, drop the whole autogeneration completely,
it was not really used and is no longer valid.

Change-Id: Id249566b71f77ecbf79a13c711f8d3ab72ca3679
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/14797
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-6068 misc: update old URLs to hpdd.intel.com

Update a URLs in several places to more modern equivalents.  Some
were in patches landed after the last mass URL update, others are
so old they were missed on the last update (CVS URL, sigh).

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I18418078ba8923b71c78f3de87aac997dda2eecc
Reviewed-on: http://review.whamcloud.com/13193
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6389 llite: restart short read/write for normal IO

If normal IO got short read/write, we'd restart the IO from where
we've accomplished until we meet EOF or error happens.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I4582affe8d2030df669b6e22b598b0c321e81466
Reviewed-on: http://review.whamcloud.com/14123
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5657 man: manpages for Lustre HSM user API

Added 2 man pages describing the following HSM user API functions:

  llapi_hsm_action_begin
  llapi_hsm_action_end
  llapi_hsm_action_get_dfid
  llapi_hsm_action_get_fd
  llapi_hsm_action_progress
  llapi_hsm_copytool_get_fd
  llapi_hsm_copytool_recv
  llapi_hsm_copytool_register
  llapi_hsm_copytool_unregister

Change-Id: Icd9c738c7b5287b5cfac2db4b697acb5a9e758ba
Signed-off-by: frank zago <fzago@cray.com>
Reviewed-on: http://review.whamcloud.com/12061
Reviewed-by: Henri Doreau <henri.doreau@cea.fr>
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6017 obd: remove destroy cookie handling

Clients no longer need to track the max and default MDS cookiesizes
so remove
  the obsolete obod o_valid flag OBD_MD_FLCOOKIE,
  the struct client_obd members cl_{default,max}_mds_cookiesize,
  the struct obd_trans_info and parameters of this type,
  the get info keys KEY_{DEFAULT,MAX}_COOKIESIZE,
  the cookiesize parameters from md_init_ea_size(),
  the files llite/*/{default,max}_cookiesize, and
  any code that needlessly handled these values.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I95f0dae370998006ec366a252891db531759ea7d
Reviewed-on: http://review.whamcloud.com/12922
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>

LU-6503 lnet: info leak in lnet_ping()

This patch is a backport from upstream
751a624adbe47145d3667c4b504858ffd3e8f175

On 64 bit systems there is a 4 byte hole after
the last member of the struct. We should clear
it to avoid disclosing stack information.

Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Yang Sheng <yang.sheng@intel.com>
Change-Id: I15849b3b35034d6c0c2a0f4c5cd0210ec453084e
Reviewed-on: http://review.whamcloud.com/14706
Reviewed-by: Amir Shehata <amir.shehata@intel.com>
Tested-by: Jenkins
Reviewed-by: Isaac Huang <he.huang@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5242 osd-zfs: umount hang in sanity 133g

It's not optimal to free large/sparse object directly by
dmu_object_free(). The dmu_tx_count_free() may over estimate the
memory overhead and cause the TX to fail in arc_tempreserve_space()
and eventually hang.

The correct way is to truncate the object with dmu_free_long_range()
before freeing it. This patch moves large object to ZPL unlinked set
and then later truncates and frees it, with as many TXs as needed.

Change-Id: I077dbcafff5abaea4a42dc9bc3f9ae980bdcdc54
Signed-off-by: Isaac Huang <he.huang@intel.com>
Test-Parameters: envdefinitions=SLOW=yes mdtfilesystemtype=zfs mdsfilesystemtype=zfs ostfilesystemtype=zfs
Reviewed-on: http://review.whamcloud.com/13630
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-5975 quota: remove obsolete quota code

Remove the obsolete quotacheck, quotaon and quotaoff, which
were retained for the interoprability with old (< 2.4) client
and server.

Some other obsolete quota code related to LL_IOC_QUOTACTL_18,
Q_INVLIDATE and Q_FINVALIDATE are removed as well.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I21cd14d9fb783c1bfa09668b318189a24bf72224
Reviewed-on: http://review.whamcloud.com/14705
Tested-by: Jenkins
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6534 tests: Fix build issue, missing mode on an open with O_CREAT

When building with gcc 4.9.2, we hit the following build
error:

In function ‘open’,
    inlined from ‘sendfile_copy’ at sendfile_grouplock.c:229:9:
/usr/include/x86_64-linux-gnu/bits/fcntl2.h:50:4: error: call
to ‘__open_missing_mode’ declared with attribute error: open
with O_CREAT in second argument needs 3 arguments
    __open_missing_mode ();

..fix this with a mode of 0644

Change-Id: Ia06392228c960584e4a8262fac775078152d78d8
Signed-off-by: Colin Ian King <colin.king@canonical.com>
Reviewed-on: http://review.whamcloud.com/14631
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4672 test: use large files in sanity-hsm tests 54 and 55

In sanity-hsm test_54() and test_55() use make_large_for_progress()
rather than make_small() so that the archives do not complete too
quickly for the tests to function correctly.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ic46ff5e2f92497808be791201759e250ebbbfe7c
Reviewed-on: http://review.whamcloud.com/14759
Tested-by: Jenkins
Reviewed-by: Henri Doreau <henri.doreau@cea.fr>
Reviewed-by: frank zago <fzago@cray.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Andreas Dilger <andreas.dilger@intel.com>

LU-6468 tests: skip conf-sanity.sh test_83 for 2.6.90 or older

Skip conf-sanity.sh test_83 if OST server version is older than
2.6.91.

Test-Parameters: alwaysuploadlogs testlist=conf-sanity envdefinitions=ONLY=83 ossjob=lustre-b2_5 mdsjob=lustre-b2_5 ossbuildno=86 mdsbuildno=86
Signed-off-by: Emoly Liu <emoly.liu@intel.com>
Change-Id: I38fe3a52586dcb4d5706b4d7c379685e09e031cc
Reviewed-on: http://review.whamcloud.com/14534
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>

LU-6558 ptlrpc: handle NULL export in nrs_orr_key_fill()

In nrs_orr_key_fill() the request export may be NULL so avoid
dereferencing it.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I7f0a3a898586eb49ae7c1fadd9118759179280b9
Reviewed-on: http://review.whamcloud.com/14737
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Chris Horn <hornc@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6406 tests: fix undersized fsname and FID strings

A Lustre filesystem name is up to 8 characters, plus 1 for the NUL
terminator. If a filesystem name is 8 characters, then the
llapi_layout_test, llapi_hsm_test, group_lock_test and llapi_fid_test
may have their stack scribbled or overran when they call
llapi_search_mounts(). Add one extra byte for the NUL character.

Same issue with a FID string, which also needs an extra character for
NUL termination.

Signed-off-by: frank zago <fzago@cray.com>
Change-Id: Ic7757ebce051e12f4723e27b5b00d4db1637b384
Reviewed-on: http://review.whamcloud.com/14213
Tested-by: Jenkins
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6484 build: remove unused LIBLUSTRE stuff

- Remove unused LIBLUSTRE stuff
- Fix packaging for ptlrpc module
- Remove liblustre.h
- Remove LIBLUSTRE_TEST
- Remove LLCPPFLAGS and LLCFLAGS
- Remove AC_LUSTRE_LIB_VER_OFFSET_WARN

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: Ib1735e75a8e1b043eacca861448cecdfc84283cd
Reviewed-on: http://review.whamcloud.com/14543
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: frank zago <fzago@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2667 ptlrpc: Move NRS structures out of lustre_net.h

NRS specific structures are not needed in the rest of the PtlRPC code.
It is more appropriate for these structures to be defined in a
separate header. This commit creates a lustre_nrs.h header for the
generic NRS structures, and policy-specific headers for the various
NRS policies.

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: If7ef5e7eebf629a8102c8a199926cbb394a374bd
Reviewed-on: http://review.whamcloud.com/13966
Tested-by: Jenkins
Reviewed-by: Henri Doreau <henri.doreau@cea.fr>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6152 osd-zfs: ZFS large block compat

This patch adds the required compatibility code to leverage the
new ZFS large block feature.  This includes supporting older versions
of ZFS which not contain large block support (ZFS v0.6.3 and earlier).

At the heart of this change is the SPA_MAXBLOCKSIZE macro which has
been increased from 128K to 16M.  This macro has always defined the
maximum possible blocksize support by ZFS.  However, just because
the ZFS code may now support this new large block it doesn't mean
that the feature is enabled on the pool.  Therefore, all consumers
of SPA_MAXBLOCKSIZE have been changed to use spa_maxblocksize() which
consults the pools feature flags to determine the supported maximum.

In addition, the ZFS OSD has been updated to to properly integrate
with a datasets recordsize property.  This allows the maximum block
size of a dataset to be changed with the 'zfs set' command.  This
has no effect on existing objects but new objects will be written
using the larger block size.  For example:

  zfs set recordsize=1M <dataset>

Finally, all references to SPA_MAXBLOCKSIZE and SPA_MAXBLOCKSHIFT
were removed and replaced with portable equivilants.  Going forward
the OSD layer needs to take case to no longer make any assumtions
about the maximum block size.

Test-Parameters: mdtfilesystemtype=zfs ostfilesystemtype=zfs mdsfilesystemtype=zfs
Change-Id: Iae782ff80ff185ca8c0f38e1002ff90e457c3ca1
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-on: http://review.whamcloud.com/13544
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5868 nodemap: Add nodemap_info to lctl

This patch adds nodemap_info to lctl, which can list all defined
nodemaps and display information about them, such as idmaps, ranges,
and flags. It is implemented as a wrapper to get_param/list_param,
and adds a -D parameter to list_param to only show directories.

Signed-off-by: Kit Westneat <kit.westneat@gmail.com>
Change-Id: I5d5ac44ec30bad49878be76168954dd231656421
Reviewed-on: http://review.whamcloud.com/13044
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6546 rpm: exclude duplicate binaries

Exclude /usr/sbin/wirecheck and /usr/sbin/wiretest binaries from lustre
package to avoid duplicate binaries in lustre and lustre-tests packages.

Change-Id: I59674fa5d1263a7e8022e6008556bf35a2de477b
Signed-off-by: Arnaud Guignard <arnaud.guignard@cea.fr>
Reviewed-on: http://review.whamcloud.com/14646
Tested-by: Jenkins
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Isaac Huang <he.huang@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6533 utils: Include unistd.h in wirecheck

Include unistd.h which is needed due to the use of
pipe, fork, dup2 and close.

Remove libcfs.h that became unneeded.

Signed-off-by: Henri Doreau <henri.doreau@cea.fr>
Change-Id: I123e0051b6988445e2e56c17ad582ece6758e3a2
Reviewed-on: http://review.whamcloud.com/14626
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6426 lustre: remove EIOCBRETRY handling

With linux commit 41003a7bcfed1255032e1e7c7b487e505b22e298
AIO retry handling was removed due to it being buggy and no
one using it, including lustre. Since this is the case
remove EIOCBRETRY since it no longer in the kernel starting
with version 3.18.

Change-Id: Icdc2de3fe252a6315914b906ef29c80343aa93e8
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: http://review.whamcloud.com/14507
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Jenkins
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Blake Caldwell <blakec@ornl.gov>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Thomas Stibor <thomas@stibor.net>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6476 lnet: avoid race during acceptor thread termination

This patch will avoid potential race, around socket sleepers
wait list, during acceptor thread termination and using
sk_callback_lock RW-Lock protection.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: Ib0a06bd8b3947648cb3ab7e1148b5073f68316fe
Reviewed-on: http://review.whamcloud.com/14503
Reviewed-by: Amir Shehata <amir.shehata@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5939 hsm: make HSM modification requests replayable

There are several HSM requests which modify data on server and
reply on Lustre recovery, e.g. they should replay changes in
case of recovery.

Patch allows such requests to be replayed in recovery time and
they are issued from client using mdc_rpc_lock to serialize them
and avoid concurrent last_rcvd update on server.

Signed-off-by: Mikhail Pershin <mike.pershin@intel.com>
Change-Id: I39810f99e99fadb76a96facf903405f6c85cc71f
Reviewed-on: http://review.whamcloud.com/13684
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5814 llite: add cl_object_maxbytes()

Add cl_object_maxbytes() to return the maximum supported size of a
cl_object. Remove the lli_maxbytes member from struct
ll_inode_info. Change the lsm_maxbytes member of struct lov_stripe_md
from __u64 to loff_t. Correct the computation of lsm_maxbytes in the
released layout case.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: Idfa6d38ed1192d562f7d99bf21b924fd711f77dc
Reviewed-on: http://review.whamcloud.com/13694
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>

LU-6536 llapi: lmm_stripe_count used unswabbed

The function llapi_layout_get_by_fd() uses the value
lum->lmm_stripe_count without first checking if it needs
to be swabbed.  This causes a false error on PowerPC clients
because the lov_user_md data appears corrupted.

Signed-off-by: Ned Bass <bass6@llnl.gov>
Change-Id: I5aacbd5804c784ec8231d51f8c93645fda077102
Reviewed-on: http://review.whamcloud.com/14633
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6559 test: use local tmp for HSM archive

In sanity-hsm.sh place the HSM archive in the tmp directory local to
the copytool.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ied319cfca2481f19f726f91f065148c5675e006d
Reviewed-on: http://review.whamcloud.com/14659
Tested-by: Jenkins
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-6245 tests: remove libcfs.h from test applications

Since user land libcfs is mostly gone libcfs.h can be
removed in the appilcations used for the test suite.

Change-Id: I8104655a750533ece9c74b2ee06cd197abfd3354
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: http://review.whamcloud.com/14629
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: frank zago <fzago@cray.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6527 ext4: journal_commit_callback optimization

Remove spin lock/unlock s_md_lock - it gives too
much overhead with huge transactions. This place
is not racy because txn is closed to the moment
when callbacks are called.

Change-Id: Ieaf76cce52854b9e5481c729bbb539eb7f97213f
Signed-off-by: Sergey Cheremencev <sergey.cheremencev@seagate.com>
Xyratex-bug-id: MRP-2466
Reviewed-on: http://es-gerrit.xyus.xyratex.com:8080/5690
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@seagate.com>
Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@seagate.com>
Tested-by: Jenkins
Tested-by: Elena Gryaznova <elena.gryaznova@seagate.com>
Reviewed-on: http://review.whamcloud.com/14610
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4698 target: check for NULL tgt before deref

In the case of and echo server during obdfilter-survey tests a target
can be missing from the obd_device.
Check for a valid pointer before dereferencing it, return error if
it's NULL.
Check for valid obt in class_exp2tgt otherwise return NULL target.
Add checks for all places that call class_exp2tgt for NULL return
value.

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I27a3020dc7ef9943fb600c3ddb6432f7f67f8b39
Reviewed-on: http://review.whamcloud.com/9936
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6419 fld: fld client lookup should retry

If FLD client lookup fails because of the remote target
is shutdown (or deactive), it should retry another target,
otherwise it will cause the application failure.

And FLD client should stop retry if the import has
been deactive.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: Ic5934054bf0aff3158b6fe040fcb26699336f038
Reviewed-on: http://review.whamcloud.com/14313
Tested-by: Jenkins
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4366 test: sync didn't return ENOMEM

The fail_loc has been intercepted by earlier write, this patch add
a sync before setting the fail_loc.

Test-Parameters: alwaysuploadlogs testlist=sanity
Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I0517ccc591651e275dc4d4cd2a3de179c2c2b3f1
Reviewed-on: http://review.whamcloud.com/14658
Tested-by: Jenkins
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Andreas Dilger <andreas.dilger@intel.com>

LU-5420 mgc: MGC should retry for invalid import

After http://review.whamcloud.com/#/c/9967/ is landed, mgc does
not wait the import connected(state = FULL), then enqueue and
retrieve config log, which will cause the mount process to fail,
especially if the mgc is shared by multiple targets.
So once mgc enqueue is failed, it will give another chance to
wait the import to recover, if the import comes back in time,
it will try to enqueue again. Otherwise it will use local config log.

Signed-off-by: Wang Di <di.wang@intel.com>
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I8dbcafbda362ebbd3370e7527a3c14c594500c1e
Reviewed-on: http://review.whamcloud.com/11258
Tested-by: Jenkins

Revert "LU-6206 ptlrpc: start minimum service threads"

This is causing test failures in conf-sanity test_53b,
as seen in LU-6563.

This reverts commit cafa669a43062c5097d40803b9ba14e66edbae25.

Change-Id: I9935bb4d84fda7a9219818c20daa683954dcd0d7
Reviewed-on: http://review.whamcloud.com/14682
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Andreas Dilger <andreas.dilger@intel.com>

Revert "New tag 2.7.53"

Actually some quota code causes us build failures here.
revert for now, as this is gettign fixed.

This reverts commit 984ee697332660a9e1d67965db26060e35a9ab32.

New tag 2.7.53

Change-Id: I503736234daf8af75cf4e3cbdac32535ef01b7f4

LU-6465 osd: NO OI scrub because of cached invalid OI mapping

On server side, the RPC service thread may cache one OI mapping
on its stack, such OI mapping will become invalid if some other
removed the object by race. If the RPC service thread uses the
cached OI mapping and finds some inconsistent object, it should
re-lookup the OI files to verify whether the OI mapping that is
cached on the stack is valid or not. If the OI mapping has been
removed from the OI file, then means the OI mapping is invalid,
should NOT trigger OI scrub under such case.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Ifbc653fcc635d77447747f91de79bcc16f2dcbbc
Reviewed-on: http://review.whamcloud.com/14499
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5814 obd: remove unused LSM parameters

Remove unused struct lov_stripe_md * parameters from obd_get_info(),
mgc_enqueue(), and osc_brw_prep_request().

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: Iee0be72393c609967ee8dad06f878026ca09821d
Reviewed-on: http://review.whamcloud.com/13426
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6319 tests: Clean up replay-single ALWAYS_EXCEPT list

Several tests in replay-single were added to the ALWAYS_EXCEPT list
when a bug was making the test fail repeatedly. For tests 33a, 33b,
and 62, the tickets were closed, but the tests were never
re-activated.

Remove replay-single tests 33a, 33b, and 62 from the
ALWAYS_EXCEPT list.

Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: I872af58ca6fb7b362b4ffe146132d98262a280a4
Reviewed-on: http://review.whamcloud.com/14149
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Wei Liu <wei3.liu@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6477 build: Update SPL/ZFS to 0.6.4.1

Version 0.6.4 Relase Notes
https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.6.4

New Functionality

* Compatible with kernels up to Linux 4.0.
* New feature flags (additional details below):
        spacemap_histogram
        extensible_dataset
        bookmarks
        enabled_txg
        hole_birth
        embedded_data
* New asynchronous I/O (AIO) support.
* New fallocate() FALLOC_FL_PUNCH_HOLE support.
* New fragmentation metric in 'zpool list'.
* New LZ4 compression of meta data.
* New "redundant_metadata" property controls desired redundancy
level.
* New "overlay" property controls behavior for non-empty mount
points.
* New 'zpool list -v' shows individual disk capacity.
* New 'zpool get -H' (scripted mode) support.
* New 'zpool create -t' creates a pool with a temporary name.
* New arc_summary.py script from FreeNAS.
* New bash completion support.
* New DTRACE_PROBES integrated with Linux tracepoints.
* New compressed block histograms with zdb.
* New verbatim pool imports with zdb.

New Feature Flags

*   spacemap_histogram

    This features allows ZFS to maintain more information about how
free space is organized within the pool. If this feature is enabled,
ZFS will set this feature to active when a new space map object is
created or an existing space map is upgraded to the new format.

*   extensible_dataset

    This feature allows more flexible use of internal ZFS data
structures, and exists for other features to depend on. This feature
will be active when the first dependent feature uses it.

*   bookmarks

    This feature enables use of the zfs bookmark subcommand. Bookmarks
mark the point in time when a snapshot was created, they can be used
as the incremental source for a zfs send command. All bookmarks in the
pool can be listed by running zfs list -t bookmark -r poolname.

*   enabled_txg

    Once this feature is enabled ZFS records the transaction group
number in which new features are enabled. This has no user-visible
impact, but other features may depend on this feature.

*   hole_birth

    This feature improves performance of incremental sends ("zfs send
-i") and receives for objects with many holes. The most common case of
hole-filled objects is zvols.

*   embedded_data

    This feature improves the performance and compression ratio of
highly-compressible blocks. Blocks whose contents can compress to 112
bytes or smaller can take advantage of this feature.

Bug Fixes

    Fixed I/O error on fs/vol delete corrupting space map.
    Fixed corruption during spacemap reallocation.
    Fixed corruption due to faulty logic when undirtying spill block.
    Fixed stale bonus buffer in recycled dnode_t data corruption.
    Fixed SA header size accounting.
    Fixed O_APPEND flag for open(2).
    Fixed deadlocks caused by direct reclaim by setting PF_FSTRANS.
    Fixed deadlocks on suspended pools.
    Fixed deadlock in zio pipeline caused by mutex_exit() race.
    Fixed deadlock between 'zpool export' and 'zfs list'.
    Fixed deadlock related to 'zfs rename'.
    Fixed deadlock related to z_teardown_inactive_lock.
    Fixed deadlock related to zfs_putpage().
    Fixed deadlock for meta data intensive workloads.
    Fixed panic when removing log device.
    Fixed panic in metaslab init when space_map_open returned ENXIO.
    Fixed panic due to dirtying inodes in a snapshot.
    Fixed panic in dbufstat.py.
    Fixed panic when performing ACL-to-mode translation on empty ACL.
    Fixed SEEK_HOLE misreporting hole at end of file.
    Fixed discrepancies in futimens() timestamps.
    Fixed pool free space leak.
    Fixed L2ARC compressed buffer leak.
    Fixed 'zpool history -i' hang.
    Fixed 'zpool import -t' it should not update the cache file.
    Fixed multiple 'zfs send/recv' failure modes.
    Fixed dracut to export ZFS root pool on shutdown.
    Fixed restore_object now performed in a single transaction.
    Fixed zvol symbolic link handling.
    Fixed per-filesystem memory reclaim.
    Fixed removal of SA in sa_modify_attrs().
    Fixed readdir for .zfs/snapshot directory.
    Fixed maximum zvol transfer size.
    Fixed dmu_sync'ed holes should retain birth time.
    Fixed ctor/dtor called on each alloc/free not once per slab.
    Fixed ZED io-spare.sh script.
    Fixed spurious timeouts when create large pools.
    Improved 'zpool add' dry-run mode.
    Improved 'zfs send -p' to only send properties for sent snapshots.
    Improved 'zpool import' hostid behavior.
    Improved 'zpool import -XF' behavior.
    Improved 'zpool import' when multiple duplicate labels are found.
    Improved 'zfs receive' performance by increasing pipe buffer size.
    Improved 'zfs send' for small blocks by increasing prefetch.
    Improved SPL kmem implementation.
    Improved zvol_get_stats() performance.
    Improved ashift auto-detect and management.
    Improved documentation in man pages.
    Improved handling of damaged block pointers.
    Improved ZED logging
    Rate limited debug backtraces to avoid impacting performance.
    Assorted performance improvements.
    Substantial changes to realign code base with illumos.
    Over 200 additional bug fixes.

Version 0.6.4.1 Relase Notes
https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.6.4.1

Bug Fixes:
  Fixed io-spare.sh script for ZED.
  Fixed multiple deadlocks which might occur when reclaiming memory.
  Fixed excessive CPU usage for meta data heavy workloads when
reclaiming the ARC.

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Ifebb473ba5508cc9a105a5dd08a6598bbda17db3
Reviewed-on: http://review.whamcloud.com/14505
Tested-by: Jenkins
Reviewed-by: Isaac Huang <he.huang@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6437 doc: enable lustre to be built without manpages

Add configure option --disable-manpages, to prevent the building and
packaging of the manpages. This can be used for an embedded
environment, or when manpages can't be build because python-docutils
(rst2man) isn't available.

Do not distribute (with make dist) manpages generated from ReST, but
distribute the ReST sources.

Signed-off-by: frank zago <fzago@cray.com>
Change-Id: I47405d466ac35eb9d295c5a6dc0bdeb0d4d75b9e
Reviewed-on: http://review.whamcloud.com/14382
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Jenkins
Reviewed-by: Patrick Farrell <paf@cray.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6422 tests: tests group lock on volatiles

Add some tests to check whether a group lock can be taken on a
volatile file.

Fixes an invalid ASSERTF.

Signed-off-by: Frank Zago <fzago@cray.com>
Change-Id: I6f37b5c9818aad74ecaeb1427f0dbe4a9d5f5bae
Reviewed-on: http://review.whamcloud.com/14344
Reviewed-by: Patrick Farrell <paf@cray.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6412 build: fix typo in lustre.spec.in

Fix a copy-paste error which should be zfs rather than ldiskfs.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I7eee4d847478903a4c913439090b090c4e9c7dce
Reviewed-on: http://review.whamcloud.com/14257
Tested-by: Jenkins
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6428 llite: Add check of d_alias and d_child

In some specific kernel versions like debian wheezy 3.2 or Ubuntu
14.04, d_alias has been moved to d_u.d_alias and d_u.d_child has
been moved to d_child in struct dentry in <linux/dcache.h>. This
patch add a check of this two members to make the llite be
compatible with those versions.

Signed-off-by: Wu Libin <lwu@ddn.com>
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Change-Id: I8ae63fdd887609e2049ac5e90f2bf4ff2ab14fd4
Reviewed-on: http://review.whamcloud.com/14060
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Thomas Stibor <t.stibor@gsi.de>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6323 build: fix debian to compile with external IB

Debian seems compiling with fixed configurations, if users
want to make debian package with external IB, it won't work.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I0ecb5c79d5948c0e8ab821c175ca89afee9cb97d
Reviewed-on: http://review.whamcloud.com/13949
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Blake Caldwell <blakec@ornl.gov>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6285 libcfs: get rid of deprecated cpumask function usage

In current kernel cpus_* functions are deprecated, replace them
with cpumask_* functions instead.
Also get rid of most of NR_CPUS

Change-Id: I573faadb2f60cdfb523158d32427e9cb456a9456
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/13926
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>

LU-6206 ptlrpc: start minimum service threads

If the ptlrpc_min_threads parameter is changed via /proc after the
service has started, then at least the requested number of service
threads should be started.  Otherwise this parameter would only be
used at initial thread startup and ignored if changed via /proc.

Fix conf-sanity.sh test_53[ab] to verify that at least the minimum
number of threads has been started when threads_min parameter is
changed, instead of just checking the parameter itself.  Also fix
test code style for 80-column line wrapping and tabs for indents.

The head utility does not always support shortcut "-1" option. It
should be specified as "-n1".

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I6e4bb4131d7500a93952b64102f885c765500c1e
Reviewed-on: http://review.whamcloud.com/13823
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6266 libcfs: add kstrtoul() compat function

The kstrtoul() function was only added to the upstream kernel in
commit v2.6.38-6934-g33ee3b2. While it is backported to RHEL6 2.6.32
kernels, it does not exist for vanilla kernels before that point.

The patch http://review.whamcloud.com/5700 added uses of kstrtoul()
to Lustre that break the build for older kernels.  While it would be
possible to convert those calls to simple_strtoul(), that function
is deprecated in newer kernels, and checkpatch.pl will generate a
warning, so using kstrtoul() is the right thing for newer kernels.

Add a wrapper function for kstrtoul() for older kernels.  There are
other kstrto*() functions in newer kernel not added by this patch.
Only the one function call is currently used, though it would be good
to convert Lustre over to using this family of functions instead of
simple_strtoul() in the future.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: Id30313e51b8d240aacc09a6e86a22460fa2540e5
Reviewed-on: http://review.whamcloud.com/13820
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5744 obdclass: fix problem of muitple MGTs on same node

When a MGT is mounted on a node which already has an running MGT
it will fail as expected since only one MGS can run on a node.
However, the MGS will be stopped by mistake during this process.
This patch fixes the problem.

Signed-off-by: Li Xi <pkuelelixi@gmail.com>
Change-Id: Ibebf59be3d41849d02801a4ac29fca31dc06c408
Reviewed-on: http://review.whamcloud.com/12301
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3266 test: regression tests for nrs policies

Tests make IO for each policy with different parametrs.

Change-Id: Iacf4721015be4cf9abdd2f41f5dd0614a8553756
Signed-off-by: Sergey Cheremencev <Sergey_Cheremencev@xyratex.com>
Signed-off-by: Wu Libin <lwu@ddn.com>
Reviewed-on: http://review.whamcloud.com/9286
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>

LU-6177 lfsck: calculate the phase2 time correctly

When the second-stage scanning finished, the assistant thread will
calculate how long the phase2 scanning has consumed by calling the
la_double_scan_result(). It should use com->lc_time_last_checkpoint,
(unfortunately it used lfsck->li_time_last_checkpoint by wrong) for
that.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I96cfdf615038f52aea0df33930226155ca61ce9b
Reviewed-on: http://review.whamcloud.com/14014
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6350 lfsck: lock object based on prediction for bad linkEA

Originally, when the namespace LFSCK verifies the object's linkEA,
it will check the linkEA without related ldlm/dt lock firstly. If
finds bad linkEA (or lost linkEA), it will acquire the ldlm/dt lock
on the object, then re-check the object; if still some inconsistent,
it will repair the inconsistency with the lock.

For the system that contains many inconsistent objects, such double
check mechanism is some inefficient. Be as improvement, the LFSCK can
make some prediction, for example: if the former object contains bad
linkEA, then when verifies current object, the namespace LFSCK will
acquire the ldlm/dt lock on the object in advance; if the prediction
is wrong, it will NOT take ldlm/dt lock in advance for next object.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I16b317305894a07bb372f5ca5e5f22052ff7cd66
Reviewed-on: http://review.whamcloud.com/14008
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6441 ptlrpc: ptlrpc_bulk_abort unlink all entries in bd_mds

desc->bd_md_count is passed as bd_mds entries count that should
be unlinked in mdunlink_iterate_helper() function. But active
bd_md_count entries not always placed from first array element.
So there are cases when active bd_mds entries are not unlinked.
This problem happened, for example, with enabled 4MB IO, if not
all parts of bulk transfered.

bd_md_max_brw should be passed to mdunlink_iterate_helper() so all
bd_mds entries are unlinked.

This patch adds test. Last MD in RPC is not received. target_bulk_io()
terminates waiting by timeout and call ptlrpc_abort_bulk() to unlink
mds. If not all active MDs unlinked on this step, then OST finally
hangs.

Xyratex-bug-id: MRP-2472
Signed-off-by: Artem Blagodarenko <artem_blagodarenko@xyratex.com>
Reviewed-by: Alexey Lyashkov <alexey.lyashkov@seagate.com>
Tested-by: Elena Gryaznova <elena.gryaznova@seagate.com>
Reviewed-by: A. Zarochentsev <alexander.zarochentsev@seagate.com>
Change-Id: Ia8ea555e947f5fae06bee6e0c38f7dbc09570edc
Reviewed-on: http://review.whamcloud.com/14399
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Alexander Zarochentsev <alexander_zarochentsev@xyratex.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-6496 ptlrpc: Fix wrong code indentation in plain_authorize

Dont hide the assignment with ,isleading indentation.

Change-Id: If15fb88e4d31962d07a48c2077179cbfe38adfd8
Signed-off-by: Oleg Drokin <green@linuxhacker.ru>
Reviewed-on: http://review.whamcloud.com/14591
Tested-by: Jenkins
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4189 mdc: Removed unneeded NULL check

Do not bother checking the return value of changelog_kuc_hdr()
against NULL since this value was dereferenced earlier.

Signed-off-by: Henri Doreau <henri.doreau@cea.fr>
Change-Id: I34a81e840838e3b50bca0099fb810815b540875e
Reviewed-on: http://review.whamcloud.com/12919
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6356 gss: call out info must include 'self nid'

When using interconnect different from eth, svc_princ_verify_host()
fails because remote node uses nid of target to build principal host
information.
Call out info must include a self nid so that svc_princ_verify_host()
can successfully match the host info built by the remote node.
For that purpose we add a new kup_selfnid field in struct
keyring_upcall_param, and a new lc_self_nid field in struct
lgss_cred.

Signed-off-by: Sebastien Buisson <sebastien.buisson@bull.net>
Change-Id: Ia44503a188feb162b32dea56333de6bafdfcfcd2
Reviewed-on: http://review.whamcloud.com/14042
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jeremy Filizetti <jeremy.filizetti@gmail.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6047 lustre: remove Size on MDS support

Remove unused definitions related to Size on MDS support from
lustre/include/lustre/lustre_idl.h. Remove unused code from several
places in lustre/.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I1673cd918539c264ea65e2384d72b32e34e8063b
Reviewed-on: http://review.whamcloud.com/13443
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>

LU-790 tests: clean up lustre version checking

Since commit 48bad5d9db9baa7bca093de5c54294adf1cf8303, the check in
sanity-scrub.sh to verify if the MDS had a new enough Lustre version
was being run before modules are loaded, if sanity-scrub.sh was run
stand-alone.  In addition to using "lctl get_param version" to get
the version, use "lctl lustre_build_version" or "lctl --version",
which will always work, even if the modules are not loaded on the
node.  This also avoids the need to load the modules just to find
that the test should not be run.

While cleaning up this code, move the lustre_version_code() earlier in
test-framework.sh, so that it is declared before calling functions,
and is co-located with the other version checking code.  Not strictly
necessary for bash scripts, but nicer when reading the code.

Fix test regressions in sanity.sh test_48[ab] since commit 13b269ab77
that incorrectly used a full path for test_mkdir() opertations instead
of using a relative filename as the test originally intended.

Also remove the obsolete check_kernel_version() code, which was only
used when we had a patched client kernel, which is no longer the case,
and remove a few overly verbose logging lines from tests.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I8a8aec11370d395dcb4f4b250001ecdb40500c1e
Reviewed-on: http://review.whamcloud.com/4173
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6387 build: lustre-dev debain package missing shared libraries

While testing on our Ubuntu platform I discovered several tools
didn't work due to missing shared libraries from lustre. This
was due to the build system packaging only the static libraries.
This patch ensure that dynamic libraries are packaged as well.

Change-Id: Ic8923fc8aa552d5afb95813f52526c5663eb9af6
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: http://review.whamcloud.com/14544
Tested-by: Jenkins
Reviewed-by: Blake Caldwell <blakec@ornl.gov>
Reviewed-by: Thomas Stibor <t.stibor@gsi.de>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6482 lov: calculate file offset correctly

In lov_stripe_pgoff(), it calls lov_stripe_size() to calculate the
file size by ost_size, which will be wrong if the stripe_index
happens to be stripe aligned.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I87f33027a835121e7b45687e3642ca50bfbf94da
Reviewed-on: http://review.whamcloud.com/14462
Tested-by: Maloo <hpdd-maloo@intel.com>
Tested-by: Jenkins
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-6448 tests: fix network config

Correct LND load during startup time, send a network parameters
to target node.

Signed-off-by: Alexey Lyashkov <alexey.lyashkov@seagate.com>
Change-Id: Ib37b90b480e5c74e7f2d55932b3d849254bad097
Reviewed-on: http://review.whamcloud.com/14433
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6356 ptlrpc: dont take unwrap in req_waittime calculation

Do not take unwrap time in req_waittime calculation on client part
as decryption is not related to request service time.

Signed-off-by: Sebastien Buisson <sebastien.buisson@bull.net>
Change-Id: I17986905f4d67323cd7e0644ab25f73dc00f23fb
Reviewed-on: http://review.whamcloud.com/14404
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Jeremy Filizetti <jeremy.filizetti@gmail.com>
Reviewed-by: Andrew Perepechko <andrew.perepechko@seagate.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5683 llite: Report first encountered error

Failures in ll_ioc_copy_{start,end} are reported to coordinator.
The return code delivered to the caller should indicate what this
error was, not whether coordinator was succesfully notified.

Signed-off-by: Henri Doreau <henri.doreau@cea.fr>
Change-Id: Iffcfe060be25ac071c7c314f33607292ed314d6a
Reviewed-on: http://review.whamcloud.com/14335
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6378 osd-ldiskfs: unlock inode before attr_init

The ldiskfs_create_inode() returns a locked inode to the caller (OSD)
that with I_NEW state, before further initializing the inode's attr
via osd_attr_init(), we should unlock the inode, otherwise, it will
cause some unnecessary dqget operations and affect the performance.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I36755cacc4895775130377da4ca486da08ede895
Reviewed-on: http://review.whamcloud.com/14102
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6349 ptlrpc: remove LUSTRE_MSG_MAGIC_V1 support

Remove the remains of LUSTRE_MSG_MAGIC_V1 support from ptlrpc.
It has not been supported since 1.8 and is not functional since 2.0.

In lustre_msg_check_version(), return an error for unsupported RPC
versions so that the server will reject such RPCs early.  Otherwise
the server only prints an error message and continue on.

Fix up whitespace and code style in lustre_msg_*() accessor functions.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I001def16e242d6ab64122912ac69fb0a072c312f
Reviewed-on: http://review.whamcloud.com/14007
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6349 ptlrpc: remove old protocol compatibility

Some old protocol compatibility workarounds are still present in
master that should have been removed when LUSTRE_MSG_MAGIC_V1 was.

In particular, the process for upgrading LUSTRE_MSG_MAGIC_V1 to
LUSTRE_MSG_MAGIC_V2 had the client to connect to the server with the
V1 protocol with op_flag=MSG_CONNECT_NEXT_VER set, and if the server
supported the V2 protocol it would reply with LUSTRE_MSG_MAGIC_V2.
This ensured that if the new client connected to an old server the
connection would be allowed.  However, even with V1 protocol support
removed, the 2.x clients are still connecting with NEXT_VER set.
In b1_8 this flag was contingent on LUSTRE_MSG_MAGIC_V1 being used,
which is how it should have been in 2.x as well.

A few other cleanups are be done at the same time:
- disallow 1.8 clients (or at least those that don't understand
  OBD_CONNECT_FULL20) so we can remove workarounds for 1.8 clients
- remove support for pre-2.1 DLM flock lock handling
- don't workaround the lack of MDS_ATTR_xTIME_SET flags in setattr
- always set MSGHDR_CKSUM_INCOMPAT18 (it can eventually be removed)

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: Id0a7a5724f2f84decaa3444fe278d0fc05500c1e
Reviewed-on: http://review.whamcloud.com/14006
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6245 libcfs: remove bitops wrappers for libcfs

The libcfs layer contains wrappers to provide bitop
wrappers to user land. Since libcfs is no longer
built for user land we can remove the wrappers. Also
remove user-crypto.h since it is not used at all.

Signed-off-by: James Simmons <uja.ornl@gmail.com>
Change-Id: I310cfc99d0cff3f527f656f4127e105883127338
Reviewed-on: http://review.whamcloud.com/13899
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: frank zago <fzago@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6245 libcfs: remove prim wrappers for libcfs

The libcfs layer contains wrappers to handle primitive
management for both kernel space and user land. Since
libcfs is no longer built for user land we can remove
the wrappers. For linux-prim.h we kept only what is
needed to deal with different kernel versions. Also
we separated out libcfs_string.h to util/string.h so
we can remove the wrappers no longer needed.

Signed-off-by: James Simmons <uja.ornl@gmail.com>
Change-Id: I0bad836f20ef29bf2ed17f1f4e5e1234d193868e
Reviewed-on: http://review.whamcloud.com/13874
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6245 libcfs: remove mem wrappers for libcfs

The libcfs layer contains wrappers to handle memory
management for both kernel space and user land. Since
libcfs is no longer built for user land we can remove
the wrappers. For linux-mem.h we kept only what is
needed to deal with different kernel versions. The
rest was moved to libcfs_prim.h.

Signed-off-by: James Simmons <uja.ornl@gmail.com>
Change-Id: Ie198500cdfa28f5d4dc8635ce0a41726ab158cbb
Reviewed-on: http://review.whamcloud.com/13841
Reviewed-by: frank zago <fzago@cray.com>
Tested-by: Jenkins
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6245 libcfs: remove libcfsutil.h

Remove libcfsutil.h, fixup some headers, rename
libcfs/util/libcfsutil_ioctl.h to libcfs/util/ioctl.h, remove
libcfs/util/platform.h, remove libcfs/util/util.c, do some other
stuff. With these changes libcfs.h can be removed from libcfs
userland library.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Signed-off-by: James Simmons <uja.ornl@gmail.com>
Change-Id: I23c0531cd1c50a652799ad807d506fda35b22b3b
Reviewed-on: http://review.whamcloud.com/14180
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: frank zago <fzago@cray.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6227 vvp: Use range lock for direct IO reads

Direct IO does not use the page cache like normal IO, so
concurrent direct IO reads of the same pages are not safe.

As a result, direct IO reads must take the range lock
in ll_file_io_generic, otherwise they will attempt to work
on the same pages and hit assertions like:
(osc_request.c:1219:osc_brw_prep_request())
ASSERTION( i == 0 || pg->off > pg_prev->off ) failed:
i 3 p_c 10 pg ffffea00017a5208 [pri 0 ind 2771] off 16384
prev_pg ffffea00017a51d0 [pri 0 ind 2256] off 16384

Also added a test for dio vs dio reads.

Signed-off-by: Patrick Farrell <paf@cray.com>
Change-Id: I5af524a42891bfa4c52d0fe8d2f7cc94c1ba5c57
Reviewed-on: http://review.whamcloud.com/14385
Tested-by: Jenkins
Reviewed-by: Alexander Boyko <alexander.boyko@seagate.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6351 lfsck: check object existence before using it

Under some cases, when the LFSCK locate the object via its FID,
it does not check whether it exists or not, then further using
such object may access NULL-pointed local object (inode).

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I637eb9bc8c3e283394c8bf9e1250d83659c5e5ba
Reviewed-on: http://review.whamcloud.com/14009
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6343 lfsck: locate object only when necessary

Currently, for every item to be verified by the LFSCK assistant
thread, the LFSCK assistant thread will try to locate its parent
directory (for namespace LFSCK) or its parent MDT-object (for
layout LFSCK) firstly via the given parent FID. But under most
cases, the system is consistent, such locating parent object is
unnecessary. For example, for namespace LFSCK, the LFSCK main
engine has put the <parent_FID, child_name, child_FID> in the
pre-loaded request, then the assistant thread will locate the
child object with the child_FID and search its linkEA to find
the <parent_FID, child_name> pairs.

Be as some improvement, we will make the LFSCK to locate related
object (especially for parent object) only when really used.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: If6a3677a6ad3fa1c694ee06f9688c1f38374f8de
Reviewed-on: http://review.whamcloud.com/13993
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6368 ldlm: Do not use cbpending for group locks

Currently, the CBPENDING flag is set on group locks when
the osc lock above them is released (osc_cancel_base).

This results in a situation where a new group lock request
on a resource does not match an existing group lock because
LDLM_FL_CBPENDING is set on the existing lock.

So two group locks are granted on the same resource, which
is not valid, since a given client can only have one group
lock on a particular resource.

Since group locks are manually released and not called back
like other LDLM locks, the CBPENDING flag doesn't make
sense.  Since they must be manually released, they also
cannot go in the LDLM LRU cache and must be fully released
immediately once they are no longer in use.

This was previously accomplished by setting CBPENDING when
the corresponding osc lock is released, but as noted above,
this prevents the group lock matching some future lock
requests.

This patch uses the fact that group locks have an l_writers
reference which they keep until they are manually released,
so we remove them when they have no more reader or writer
references, without checking cbpending.

Additionally, this patch adds several sendfile tests,
courtesy of Frank Zago <fzago@cray.com>.

Signed-off-by: Patrick Farrell <paf@cray.com>
Change-Id: I2845750777cbc9849b18999f1b77f791034c50b0
Reviewed-on: http://review.whamcloud.com/14093
Reviewed-by: frank zago <fzago@cray.com>
Tested-by: Jenkins
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6421 osc: fix bug when setting max_pages_per_rpc

After setting like "lctl set_param -P osc.*.max_pages_per_rpc", it
is possible that the function osc_obd_max_pages_per_rpc_seq_write
will be called before ocd_brw_size has been set when mount.
ocd_brw_size is meaningless when it is zero. So it should not be
the limit at that time.

Signed-off-by: Wu Libin <lwu@ddn.com>
Change-Id: I1abbd1bfc089ca1fea2b7ec71a6d45ee70f1ba20
Reviewed-on: http://review.whamcloud.com/14333
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2524 test: Test suite clean up for replay-single

The patch modifying the tdir variable to a single directory
has landed; http://review.whamcloud.com/#/c/8123/. We can
now conduct miscellaneous cleanup including:

Remove the `-p` (parents) option from many calls to mkdir
Modified directory and file names to use $tdir and $tfile
Add check for and call `error` and/or add error messages for a
variety of common functions.
Replace `…` with $(...)
Removed linefeed escape after |, ||, & and && operators.
Remove test dependencies between tests 4a and 4b

Test-Parameters: testlist=replay-single alwaysuploadlogs envdefinitions=SLOW=yes mdtfilesystemtype=ldiskfs mdsfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs
Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: Ib09102e50f855550db801180be3f7fc42911191a
Reviewed-on: http://review.whamcloud.com/13170
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6407 lnet: set task state before scheduling

In 0b868add80281c085ce1b297d1cb078deaab802a when libcfs_sock_accept()
was made into lnet_sock_accept() a call to
set_current_state(TASK_INTERRUPTIBLE) got dropped. Restore it.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I132d6a611e5677ad52358178aa566ca805536e61
Reviewed-on: http://review.whamcloud.com/14265
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Amir Shehata <amir.shehata@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5579 tests: Add test for resend enqueue vs lock destroy

This test is split out from the actual patch because
some other test infrastructure issues make it always
fail right now.

Xyratex-bug-id: MRP-2094

Signed-off-by: Vitaly Fertman <vitaly_fertman@xyratex.com>
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Signed-off-by: Liang Zhen <liang.zhen@intel.com>
Change-Id: I6f764fd863d6046bde8c6336f003e602fc07e59f
Reviewed-on: http://review.whamcloud.com/12210
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>

LU-6486 ptlrpc: export lustre_swab_lov_mds_md

For ppc64 clients lustre would not mount due to an Unknown
symbol error. This is due to lustre_swab_lov_mds_md not
being exported. A simple export of this function resolves
this issue.

Change-Id: Ib924045988291efbc5ac129ce58354bbc72aa7b6
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: http://review.whamcloud.com/14545
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6481 kernel: kernel update RHEL 6.6 [2.6.32-504.16.2.el6]

Update RHEL6.6 kernel to 2.6.32-504.16.2.el6

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I3d439ad2338b7940292fb584e09ca2c0b4fce4ef
Reviewed-on: http://review.whamcloud.com/14546
Tested-by: Jenkins
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6471 obdclass: fix llog_cat_cleanup() usage on Client

With patch/commit 3a83b4b9 for LU-5195, LLOG code has been
strengthen against catalog inconsistency by detecting a
referenced plain LLOG is missing and by clearing its
associated entry by calling llog_cat_cleanup(), which now
needs to handle the case where it is also executed on a Client
(ie, cathandle->lgh_obj == NULL) and thus must not attempt to
update on-disk catalog.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: Ie6fb9240ac76810358a91c7410046c626f42c2b9
Reviewed-on: http://review.whamcloud.com/14489
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6245 utils: split kernel comm between user and kernel

The kernel communication code used for HSM and changelog is
entangled. Move the user space bits into the liblustreapi. This will
also help for a possible relicensing. The kernel portion is also moved
from libcfs to obdclass.

The original libcfs_kernelcomm.h header is split into three parts:

  * lustre_kernelcomm.h, a new header for the kernel parts;
  * uapi_kernelcomm.h, a new header for the data structures shared
    between userspace and kernelspace;
  * lustreapi_internal.h receives the private liblustreapi prototypes.

The original code in kernel_user_comm.c is split into two parts:

  * obdclass/kernelcomm.c for the kernel part. filp_user_write() was
    moved there, and linux-fs.c deleted;
  * liblustreapi_kernelconn.c for the user part. The calls to CDEBUG
    have been removed, and calls to CERROR have been transformed to
    llapi_err_noerrno. The type lustre_kernelcomm has been removed and
    replace by struct lustre_kernelcomm.

Various names and filenames have been harmonized to *kernelcomm*.

The unused symbol KUC_FL_BLOCK has been removed.

Signed-off-by: frank zago <fzago@cray.com>
Change-Id: Id15ab6ea23a160e0aa0517a1a890266b2e971982
Reviewed-on: http://review.whamcloud.com/14270
Tested-by: Jenkins
Reviewed-by: Nathan Rutman <nathan.rutman@seagate.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6300 build: Kernel string mismatch for ppc64 in RPMs

The macro in the kernel RPMs for ppc64 provided
by Redhat describing the kernel version supported,
mismatch the content of the macro of the Lustre kernel
client modules for ppc64 describing the required
kernel. This leads to unresolved dependency errors
during installation of the Lustre client modules
although the kernel versions match actually.

While Redhat uses the convention:

kernel = <major-number>-<minor-release-number>.el6

the following string is used in Lustre client RPM
for the required kernel version:

kernel = <major-number>-<minor-release-number>.el6.ppc64

The patch will remove the trailing string '.ppc64' in
macro 'krequires' defined in the lustre specfile
describing the software dependeny.

Signed-off-by: Frank Heckes <frank.heckes@intel.com>
Change-Id: I930d2fe67da819db57a6591341c5a55f0374ad43
Reviewed-on: http://review.whamcloud.com/14012
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Tested-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5814 llite: remove lli_has_smd

Remove the lli_has_smd flag from struct ll_inode_info. The empty
layout case will be handled by the LOV layer. Remove the unused
function cl_local_size().

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I50b73482fdc4c415f8d800e9a848099117be8f74
Reviewed-on: http://review.whamcloud.com/13690
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6413 lod: set FLAGS all stripes of striped dir

Except permission and time, flags should be set on
each stripe of striped dir as well, otherwise
chattr can not set special flags (immutable etc)
correctly on striped dir.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I8b3274b674bd1b8b41f95bb9299245aa8c2905c3
Reviewed-on: http://review.whamcloud.com/14278
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6320 lod: Use target index as the master index

Because lum_stripe_offset might be -1, if it comes
from the default stripe, and also the master stripe
always in the same MDT as the master object, so
it should use master object MDT index as the master
stripe index.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I4a700580dd7a035b243d2314231e50e12742100e
Reviewed-on: http://review.whamcloud.com/13941
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>