LU-4689 hsm: count NULL terminator in hai_first/hal_size

If fsname is 8-byte aligned, hai_first fails to count the ending NULL
terminator causing hai to directly attached after fsname and future
hai_first will return a different position for first hai.

Signed-off-by: Peng Tao <bergwolf@gmail.com>
Change-Id: I5e9e1f48f99b4743b2d5b93397e06f6becabeb26
Reviewed-on: http://review.whamcloud.com/9431
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3951 lfsck: OST-object inconsistency self detect/repair

When client sends object-based RPC to the OST, the RPC service
thread on the OST needs to verify whether the given parent FID
in the client RPC matches the parent FID information stored in
the OST-object.

When the client given PFID does not match the OST local stored
PFID, then the OST will return "-EINPROGRESS" to the client for
retry later to avoid the RPC service thread to be blocked for
long time. On the other hand, there will be a dedicated thread
to talk with the LFSCK for the PFID verification. If the client
given parent FID information is incorrect, then deny the access;
otherwise, if the OST local stored PFID attribute is invalid,
then the OST local stored PFID xattr will be repaired.

Other fixes:
1) Hold update lock on the .lustre/lost+found/MDTxxxx object
   when add new name entry for handling orphan OST-object.
2) Hold dt_write_lock on the OST-object to be destroyed before
   transaction start, the same as normal ofd_object_destroy does.
3) Simplify the lfsck_layout_recreate_lovea() implementation.
4) Make sanity-lfsck test_18 to be workable under both DNE and
   non-DNE cases.
5) Other code cleanup.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Iefd26fe1782761ab16954a00aea0788c39534580
Reviewed-on: http://review.whamcloud.com/7667
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>

LU-4690 tests: disable test_4 in sanity.sh

Disable test_4 in sanity.sh because of LU-4690.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I1acac5ad5c049eed7618aebf473b385a150316d7
Reviewed-on: http://review.whamcloud.com/9440
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

Revert "LU-4659 mdd: rename forgets updating target linkea"

This now breaks the build due to conflict with some DNE patch.

This reverts commit f6c64625b87c06749e04a0a74960852bb9e50750.

Change-Id: Ie47a36b5e9df70183f3f27cec0f47402cebb102c
Reviewed-on: http://review.whamcloud.com/9445
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Tested-by: Oleg Drokin <oleg.drokin@intel.com>

Revert "LU-4199 llite: Init lli_lmv_md field of struct ll_inode_info"

In fact this member was already removed by some DNE patch.
So this causes build failures now.

This reverts commit c330f3c9d58ce507b2111ab3744dbf71353374bc.

Change-Id: I7d99d727fd2ef1f5b8f17ff6e3d9e9da79eae577
Reviewed-on: http://review.whamcloud.com/9442
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Tested-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3953 build: configure output improvments

Cleanup from mess in configure output and multi-line messages.

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: If90b0962f1b44f28af41e30015cc2ab253d6b83e
Reviewed-on: http://review.whamcloud.com/9309
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Brian J. Murrell <brian.murrell@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4300 ldlm: ELC picks locks in a safer policy

Change the policy of ELC to pick locks that have no dirty pages,
no page in writeback state, and no locked pages.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: Ie73130c4100e3b91c211635f6a72fe5dad994426
Reviewed-on: http://review.whamcloud.com/9175
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3974 llite: invalidatepage api changed

Until recently invalidating pages from the buffer cache
was dependent only on the page passed in and the start
in the page to invalidate. Starting with the 3.11 kernel
you can also specify the length of the data in the page
to invalidate. This patch enables us to handle the new
case.

Signed-off-by: James Simmons <uja.ornl@gmail.com>
Change-Id: Iedf458b20b2604bc3099d5ae38bf0ad07df83bd3
Reviewed-on: http://review.whamcloud.com/7729
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Peng Tao <bergwolf@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3974 llite: use new struct dir_context

The readdir and nfs code over time has added more
parameters to be passed to be processed. For the 3.11
kernel a new struct dir_context was introduced to
minimize the impact of future expansion. This patch
addresses this change.

Signed-off-by: James Simmons <uja.ornl@gmail.com>
Change-Id: Ib42bf8cb06635a2a64e63b294d79e66ac82a1a5b
Reviewed-on: http://review.whamcloud.com/7747
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Peng Tao <bergwolf@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4416 utils: loop_control device support

FC19 use loop_control device since 3.12.5 kernel.

Signed-off-by: yang sheng <yang.sheng@intel.com>
Change-Id: I40b241243507648b7dd9c6ddaf924c2324396e3f
Reviewed-on: http://review.whamcloud.com/8799
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Jenkins
Reviewed-by: Peng Tao <bergwolf@gmail.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2675 obdclass: remove uses of lov_stripe_md

Remove the unused function llog_obd_add(). Remove the unused count and
lsm parameters from llog_cancel(). Move dump_lsm() from obdclass to
the only module that uses it (lov). Remove obd_lov.h.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I38a5fdae210b6317a07d1601c1c93ae8071631ae
Reviewed-on: http://review.whamcloud.com/8545
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3319 lprocfs: client side cleanups

Now that all the client side seq_file patches it is
time to handle the issue that people pointed out but
were not severe enough to prevent landing. This patch
addresses all the concerns as well and move all struct
lprocfs_seq_var to be initialized C99 style.

Change-Id: I89e8b719bd067ecf4e3cab481a2d4c62d5052af0
Signed-off-by: James Simmons <uja.ornl@gmail.com>
Reviewed-on: http://review.whamcloud.com/8943
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Peng Tao <bergwolf@gmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4129 libcfs: Only dump log once per sec. to avoid EEXIST

Since the log file name contains the current time in seconds, dumping
the logs more than once per second causes EEXIST errors to be emitted.
Add a static variable to libcfs_debug_dumplog_internal that records
the time of the last Lustre log dump.  If the current time in seconds
is equal to the last time, do not dump logs again.

Note that this is not thread-safe.  However, in the rare case that two
threads try to access last_dump_time simultaneously, the worst thing
that could happen is that one of the threads will get an EEXIST error
when trying to write the log file.  This is no worse than the current
situation, and it is not likely to happen.

Signed-off-by: Ryan Haasken <haasken@cray.com>
Change-Id: I7345635ab84333d6c1b455de6059d9d72e5a88f5
Reviewed-on: http://review.whamcloud.com/8964
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>

LU-4522 llite: quiet console permission error messages

Quiet some common console error messages for permission errors
that can be hit in common cases.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I8ef2eec8c466e41bdaf08779440a186e442540e5
Reviewed-on: http://review.whamcloud.com/8988
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4199 llite: Init lli_lmv_md field of struct ll_inode_info

lli_lmv_md field of structure ll_inode_info is not inited to NULL,
which causes lmv_read_entry() to crash the kernel when trying to
access it.

Signed-off-by: Li Xi <lixi@ddn.com>
Change-Id: I84e932a4246b4637d5ba240ea9193c8b347d62e4
Reviewed-on: http://review.whamcloud.com/9042
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-4199 libcfs: Handle nodemask on UMP machines

For UMP and SMP machines the struct cfs_cpt_table are
defined differently. In the case handled by this patch
nodemask is defined as a integer for the UMP case and
as a pointer for the SMP case. This will cause a problem
for ost_setup which reads the nodemask directly. Instead
we create a UMP version of cfs_cpt_nodemask and use that
in ost_setup.

Change-Id: I0dd80f79cf4589fc98182786d3f970432388f1fe
Signed-off-by: James Simmons <uja.ornl@gmail.com>
Reviewed-on: http://review.whamcloud.com/9219
Tested-by: Jenkins
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Li Xi <pkuelelixi@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-4629 liblustre: remove suspicious check

Suspicious dereference of pointer 'inode' by passing argument 1 to
function 'llu_i2info' at line 387 before NULL check at line 391.

The pointer 'inode' can not be NULL. This checked on upper layer.

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: Iea1e4d46f9faccf74a895002a8640959d63c395a
Reviewed-on: http://review.whamcloud.com/9324
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-4629 lmv: fix issue found by Klocwork Insight tool

'plock.cookie' might be used uninitialized in this function.

sscanf format specification '%d' expects type 'int *' for 'd',
but parameter 3 has a different type '__u32*'

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: Ie800e69f345e815dc7437b89707cbaabac1af5f7
Reviewed-on: http://review.whamcloud.com/9390
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4629 gss: fix incorrect memset size

Function 'memset' is called for 'sizeof' applied to pointer
instead of 'sizeof' applied to pointed type.

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: Icd1583cb2de90617f6d96baa5d39d28c0d5167ae
Reviewed-on: http://review.whamcloud.com/9345
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4629 utils: fix rsources leak

Resource acquired to 'fp' at line 3540 may be lost here.
Resource acquired to 'debug_log' at line 1690 may be lost here.
Resource acquired to 'mattr' at line 471 may be lost here.
Resource acquired to 'conn->oc_fd' at line 208 may be lost here.

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: I47a1c44d8b1cfd8500ca71958814b8a23d6fff2b
Reviewed-on: http://review.whamcloud.com/9343
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4629 gss: fix uninitialized variable

'sg->page_link' is used uninitialized in many functions.

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: I7f6d2870a8e95ce0614d617d08cf12f657a680ca
Reviewed-on: http://review.whamcloud.com/9325
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3831 llite: Remove touch_atime() call in ll_readdir()

In 2.6.32 vfs_readdir() calls file_accessed() which checks for
O_NOATIME and calls touch_atime() appropriately. In 3.11, the same is
true of iterate_dir(). Hence removed the call to touch_atime() in
ll_readdir(). Also, removed the HAVE_TOUCH_ATIME_1ARG and
HAVE_F_PATH_MNT autocrud since ll_readdir() was the last caller of
touch_atime() in the lustre tree.

Signed-off-by: Swapnil Pimpale <spimpale@ddn.com>
Change-Id: Iedd3be29adae80427d71a7eb5a0f8c0d1f1d2924
Reviewed-on: http://review.whamcloud.com/8664
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4643 hsm: Make sanity-hsm test_60 more robust

The first version of this test was fragile and could fail
intermittently when test infrastructure was not capable of
providing 1MB/sec in lustre bandwidth. This commit changes the
test to validate that a progress update occurs within the expected
window, rather than testing for a specific amount of data copied
under ideal conditions.

Signed-off-by: Michael MacDonald <michael.macdonald@intel.com>
Change-Id: Ief40a27452bc09d94101c7df00a46aa171a572f8
Reviewed-on: http://review.whamcloud.com/9376
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4659 mdd: rename forgets updating target linkea

MDD forgets updating target LinkEA if target is not unlinked, and
this will cause LinkEA inconsistency.

Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Change-Id: I92b9019b9517513d0cf3f92cf09e7a546cee6027
Reviewed-on: http://review.whamcloud.com/9366
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Li Wei <wei.g.li@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Henri Doreau <henri.doreau@cea.fr>

LU-4516 mdc: missing lexxx_to_cpu in mdc_read_entry

A few missing lexx_to_cpu on some accesses to lu_dirpage/lu_dirent
in MDC.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I50d9d8916004ef460bd7a49376306e2d6bda2b99
Reviewed-on: http://review.whamcloud.com/9331
Tested-by: Jenkins
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3527 nodemap: idmap management functions

This adds support for uid/gid mapping management to the nodemap
module. This mapping is added as four rbtrees in the nodemap
structure for forward and reverse mapping of UIDs and GIDs.
Postorder traversal for rbtrees have been added for cleanup
upon nodemap removal.

lctl commands have been added for adding, removing, and testing
maps.

Unit tests have been added to test the maps in all the relevant
modes (allow root access, trust client ids).

Signed-off-by: Joshua Walgenbach <jjw@iu.edu>
Change-Id: I9a8672f2883185a35d3553e5c2e91ae1d2cb094d
Reviewed-on: http://review.whamcloud.com/8125
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Andrew Perepechko <andrew_perepechko@xyratex.com>
Reviewed-by: Ken Hornstein <kenh@cmf.nrl.navy.mil>

LU-3527 nodemap: add NID range management

The nodemap module defines a nodemap as a set of NID ranges
and a policy that is applied to them. This adds the range
management functions to nodemap. The definition of a range
is kept in the range_node structure.

The range_node structures are kept in a global rb_tree for
ease and speed of managing the range_nodes (each range_node
can be a member of eexactly one nodemap). The range_node
contains a pointer back to the nodemap of which it is a
member. The range_node also keeps the (inclusive) start and
end of the NID range is represents.

Additionally, each nodemap contains a linked list of the
pointers to the range_node in encompasses, for use in
management of the nodemap (deleting, checking ranges, etc)
to prevent walking the range_node tree.

Commands have been added to lctl to control the range management
and other nodemap policy features (squash ids, admin, trusted)
that will control its behavior with regards to ID mapping.

Unit tests have been added to sanity-sec.sh to check the
range management and policy controls.

Signed-off-by: Joshua Walgenbach <jjw@iu.edu>
Change-Id: I2b93f15de8f07a2bcef16fde27fb03a41a2063bc
Reviewed-on: http://review.whamcloud.com/8057
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andrew Perepechko <andrew_perepechko@xyratex.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Ken Hornstein <kenh@cmf.nrl.navy.mil>

LU-2430 mdt: Add global rename lock.

Add global rename lock to protect migration process.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I363ab58150350e5bc558fcd655c5e7e7b653af10
Reviewed-on: http://review.whamcloud.com/8826
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4371 gss: Generalize testing of GSSAPI code

Rename sanity-gss.sh to sanity-krb5.sh and make sanity-gss.sh a
mechanism-agnostic test suite for just the GSSAPI code itself using
the gssnull security flavor, which in turn uses the null GSSAPI
mechanism.

Test-Parameters: testlist=sanity-gss
Signed-off-by: Andrew Korty <ajk@iu.edu>
Change-Id: I149ce66c4b044e20b9e8e1b93058ec4f62aba881
Reviewed-on: http://review.whamcloud.com/8525
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-2430 mdd: add lfs mv to migrate inode.

Add lfs mv to migrate the individual inode from one MDT to
another MDT, and this function will only migrate inode layout
on MDT but not touch data object on OST.

lfs mv -M 1 /mnt/lustre/test1 #move test1 to MDT1.

The directory will be migrated from top to the bottom, i.e.
migrating parent first, then migrating the child.

Add migrate into sanity 17n to check on-disk format.

Add sanity 230(c,d,e,f) sanityn 80 to for migration.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: Ib4456a1db8909bd96260c67fff48922081948dcd
Reviewed-on: http://review.whamcloud.com/6662
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3539 protocol: Change UPDATE_OBJ RPC format

This patch updates struct update for DNE phase II.  See the DNE phase
II HLD for more information.

The comment describing the UPDATE_OBJ RPC format is already a bit out
of date.  This patch simplies and updates it so that the overall
structure is easier to see while the details are commented in-line.

The related type names are modified to be more consistent with the
terminology used in the HLD and the comment.

Change OBJ_XXX-->OUT_XXX, and refine a few update structures to
object_update_xxx. And there are no logic changes in this patch.

Change-Id: I4355b4c863abb448713115f17b80402ec96009f8
Signed-off-by: Di Wang <di.wang@intel.com>
Signed-off-by: Li Wei <wei.g.li@intel.com>
Reviewed-on: http://review.whamcloud.com/7128
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4604 lfsck: LFSCK async updates RPC flow control

Control the max in flight async updates RPCs count in OSP layer to
avoid too much in flight RPCs causes memory issue and OST overload.

Currently, we use a semaphore for that which is initialized as 512.

It also contains the layout LFSCK performance tests:
1) lfsck_layout performance under single MDS mode with kinds of
   stripe counts and kinds of file sets, for routine check case
   and repairing case.

2) lfsck_layout performance under DNE mode with kinds of file
   sets, for routine check case and repairing case.

3) lfsck_layout impact on create performance.

This patch also fixes some bugs related with lu_device_type
operations smooth the lfsck performance test.

There was no protection when inc/dec lu_device_type::ldt_device_nr,
which may caused the ldt_device_nr to be wrong and trigger assert.
This patch redefine lu_device_type::ldt_device_nr as atomic type.

There was no protection when add/del lu_device_type::ldt_linkage
into/from the global lu_device_types list, which may caused bad
address accessing. This patch uses the existing obd_types_lock
to protect related operations.

We do NOT need lu_types_stop() any longer. Such function scans
the global lu_device_types list, and for each type item on it
which has zerod lu_device_type::ldt_device_nr, call its stop()
method. In fact, the lu_device_type::ldt_device_nr only will be
zero when the last lu_device_fini() is called, and at that time,
inside the lu_device_fini(), its stop() method will be called.
So it is unnecessary to call the stop() again via lu_types_stop().

Test-Parameters: mdtcount=2 ostcount=2 envdefinitions=SLOW=yes testlist=lfsck-performance
Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Ib9bae6ad0446e8705fa2767d080150e82a495e2f
Reviewed-on: http://review.whamcloud.com/8694
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3336 lfsck: recreate the lost MDT-object

If the MDT-object does not exist but related OST-obejct still
alive, then when handle the orphan OST-object, the MDT-object
will be re-created under /.lustre/lost+found/MDTxxxx with the
given OST-object stripe and owner information. The other part
of the MDT-obejct layout EA will be empty which may be filled
by other orphan OST-object(s) as the LFSCK processing.

The name for the above re-created MDT-object will be named as
${MOUNT_POINT}/.lustre/lost+found/MDTxxxx/N-${FID}, the "FID"
is the MDT-object's fid. If the OST-object has zero PFID (for
some reason), then LFSCK will assign new FID to the parent.

If the MDT-object exists, but related layout EA slot is occupied
by other OST-object which may be created by former LFSCK running
for repairing dangling referenced MDT-object or may be not.

For the latter case, the LFSCK will create a new MDT-object with
the given OST-object under /.lustre/lost+found/MDTxxxx with name
"C-${FID1}-${FID2}-${idx}":
FID1: the new MDT-object's fid.
FID2: the original MDT-object's fid.
idx: index in the layout EA.

For the former one, then means the former LFSCK guess was wrong.
If such new created OST-object has never been modified which can
be indicated with lu_attr::la_ctime is zero and lu_attr::la_mode
has no "S_ISUID" set, then the LFSCK will destroy it and replace
it with the orphan OST-object; otherwise the LFSCK will keep the
new data, and create a new MDT-object to reference the orphan as
describe above with name "C-${FID1}-${FID2}-${idx}".

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Ia7700f7b2e4e09d6b576d025295fdb5c10ca90e0
Reviewed-on: http://review.whamcloud.com/7811
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3336 lfsck: namespace visible lost+found directory

The LFSCK will create the lost MDT-objects under a namespace visible
directory, then the admin can future handle those orphans with human
knowledge later.

The directory is named as "lost+found" under ".lustre", which is not
the same as backend (for ldiskfs case) "/lost+found". The latter one
is invisible from Lustre namespace. Under ".lustre/lost+found/", the
LFSCK will create sub-directory "MDTxxxx" for each MDT. (xxxx is the
MDT number)

The ".lustre/lost+found" uses special FID:

enum dot_lustre_oid {
        FID_OID_DOT_LUSTRE      = 1UL,
        FID_OID_DOT_LUSTRE_OBF  = 2UL,
        FID_OID_DOT_LUSTRE_LPF  = 3UL,
};

const struct lu_fid LU_LPF_FID = { .f_seq = FID_SEQ_DOT_LUSTRE,
                                   .f_oid = FID_OID_DOT_LUSTRE_LPF,
                                   .f_ver = 0x0000000000000000 };

Both ".lustre" and its child "lost+found" reside on the MDT0. As for
the "lost+found"'s children "MDTxxxx" uses normal FID and resides on
the corresponding MDT. The orphans will be linked under the "MDTxxx"
by the LFSCK on such MDT when it find the orphans.

The ".lustre/lost+found" is created by the MDT0 when mdd_prepare().
The ".lustre/lost+found/MDTxxxx" is created by the LFSCK on related
MDT when it needs to process orphans. (create on demand)

The permission for ".lustre/lost+found" is "r-x------" for root user.
Only the LFSCK can create sub-directories "MDTxxxx" under it. All the
other modifications are denied.

The permission for ".lustre/lost+found/MDTxxxx" is "rwx------" for
root user. Any modification is the same as operated on normal file.

The lost+found mechanism is not only for LFSCK phase II but also for
other LFSCK orphan handling. Since it is visible to Lustre namespace,
the admin can operate the orphans as other normal files easily.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Idd37c87c2517175022127a2a605ebb3ddf1f223a
Reviewed-on: http://review.whamcloud.com/8305
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3336 lfsck: regenerate lost layout EA

To find out orphan OST-objects, the LFSCK on OST side maintains
two bitmaps in RAM for the OST-object accessed during the LFSCK.
After the first cycle system scanning, the LFSCK got the bitmap
for the known OST-objects, and got another bitmap for which OST
objects have been referenced by MDT-objects. Then the LFSCK can
know which OST-objects are not referenced by any MDT-object via
comparing the two bitmaps.

If the MDT-object exists, then check the layout EA. If related
layout EA is empty, then fill the layout EA slot with the given
OST-object stripe information. If the given OST-object stripe
index exceeds current layout EA size, then extend the layout EA
and fill the gap slot(s) as empty, which can be filled by others.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Ie542289b397576be457b04792e845324d6926836
Reviewed-on: http://review.whamcloud.com/7810
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3336 lfsck: orphan OST-objects iteration

During the second stage scanning, the LFSCK on the MDT(s) will scan
the orphan OST-objects via OSP level iteration which fetches remote
orphan OST-objects information via OBD_IDX_READ RPC, and shares the
existing framework/functions with others, such as quota.

Implement the sponsor (the master LFSCK engine on the MDT) logic
for the orphan OST-objects iteration.

Implement LFSCK layout rbtree iteration - lfsck_orphan_index_ops,
for slave LFSCK on OST. The lfsck_orphan_index_ops is registered
onto the rbtree object. The incoming OBD_IDX_READ RPC for orphan
OST-object scanning will iterate the rbtree via dt_index_read to
call the registered lfsck_orphan_index_ops.

Others:
1) Speed control during the second-phase scanning.
2) The LFSCK layout trace file (on the MDT) flags should be set
   with LF_INCOMPLETE if LFSCK slave on OST restart or failed.
3) Some code cleanup.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I67d5d870dbf9b80530f4d61ed1a3e5b5df70b1a0
Reviewed-on: http://review.whamcloud.com/8303
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4357 libcfs: restore __GFP_WAIT flag to memalloc calls

In 2.4, the flags passed to the memory allocation functions are
translated from CFS enumeration values types to the kernel GFP
values by calling cfs_alloc_flags_to_gfp(). This function adds
__GFP_WAIT to all flags except CFS_ALLOC_ATOMIC. In 2.5, when
the cfs wrappers were dropped, cfs_alloc_flags_to_gfp() was
removed and the CFS_ALLOC_xxxx was simply replaced with __GFP_xxxx.
This means that most memory allocation calls are missing the
__GFP_WAIT flag. The result is that Lustre experiences more ENOMEM
errors, many of which the higher levels of Lustre do not handle
robustly.
Notes GFP_NOFS = __GFP_WAIT | __GFP_IO. So the patch replaces
__GFP_IO with GFP_NOFS.
Patch does not add __GFP_WAIT to GFP_IOFS. GFP_IOFS was not used in
2.4 so it has never been used with __GFP_WAIT.

Signed-off-by: Ann Koehler <amk@cray.com>
Signed-off-by: Emoly Liu <emoly.liu@intel.com>
Change-Id: Ib241b39674129a27fea53c23c8ce3e74d165372a
Reviewed-on: http://review.whamcloud.com/9223
Tested-by: Jenkins
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-4460 mount: fix lmd_parse() to handle comma-separated NIDs

This patch reverts commit 3917e62018878dfffac59ceed70f20b0419945d3,
which cannot handle the upgrade situation that old mountdata already
contains comma-separated NIDs. The correct way to fix the original
issue is to parse comma-separated NIDs in lmd_parse().

The patch also updates disk2_4-ldiskfs.tar.bz2 to make the mountdata
of ost contain comma-separated NIDs so as to verify the patch under
upgrade situation.

Test-Parameters: alwaysuploadlogs \
envdefinitions=SLOW=yes,ENABLE_QUOTA=yes testlist=conf-sanity

Signed-off-by: Jian Yu <jian.yu@intel.com>
Change-Id: If179618c9c89dc2168f748aeba59384ea31197ff
Reviewed-on: http://review.whamcloud.com/8918
Tested-by: Maloo <hpdd-maloo@intel.com>
Tested-by: Jenkins
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: Sebastien Buisson <sebastien.buisson@bull.net>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4471 mdd: mdd_unlink: do trans_start after sanity check

Currently, mdd_trans_start is called before
mdd_unlink_sanity_check. This means a remote directory
which has files in it can be removed on MDT0 before the
sanity check on MDT1 finds the files and errors, which
orphans the files on MDT1. This patch moves the sanity
check before mdd_trans_create and mdd_trans_start.

Signed-off-by: Patrick Farrell <paf@cray.com>
Change-Id: I08882b682b9f0016577214821efec4759ee5c184
Reviewed-on: http://review.whamcloud.com/8827
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>

New tag 2.5.56

Change-Id: I9f095e4102fc46f7ed9829a6056ed83c0ac3f692
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3531 mdt: delete striped directory

Add delete striped directory, it includes

1. enable sync log between MDTs, so slave objects will
be delete by unlink log, which is similar as deleting ost
object.

2. retrieve layout information of striped directory on MDT,
then lock all of the slave objects before unlink.

3. remove a few unnecessary cfs_size_round, because update_size
and update_buf_size already do size_around inside.

4. add sanity 300 for striped dir test

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: Ib461156bbff9e416ac9d2500a0b9491427542340
Reviewed-on: http://review.whamcloud.com/7445
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3950 lfsck: control all LFSCK nodes via single command (2)

The single command should work for not only layout LFSCK, but also for
other LFSCK components, such as namespace LFSCK, OI scrub on each node
and DNE LFSCK in the future.

Introduce another lfsck_start option "-o" for enable orphan handling.
Currently it is used for orphan OST-objects handling. When enable it,
the layout LFSCK will be triggered on all servers by default.

Code cleanup and more log information.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Iaed9ee61d3d0fced32f9dd6b2a7f6663de6d2dc7
Reviewed-on: http://review.whamcloud.com/9257
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3336 lfsck: use rbtree to record OST-object accessing

To find out orphan OST-objects, the LFSCK on OST side maintains
two bitmaps in RAM for the OST-object accessed during the LFSCK.
After the first cycle system scanning, the LFSCK got the bitmap
for the known OST-objects, and got another bitmap for which OST
objects have been referenced by MDT-objects. Then the LFSCK can
know which OST-objects are not referenced by any MDT-object via
comparing the two bitmaps.

Above two bitmaps are organized via a single rbtree. The rbtree
is maintained by LFSCK on the OST side. For every LFSCK scanned
OST-object, it will be recorded in the known-bitmap, for every
OST-object accessed by any RPC during the scanning, it will be
recoreded in the accessed-bitmap.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: If399584a8617e7c368e48922a3582294ac98d5f4
Reviewed-on: http://review.whamcloud.com/7743
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3594 lfsck: repair inconsistent owner and multiple referenced cases

Sometimes, the OST-object owner information is inconsistent with the
MDT-object owner information because of incompleted chown/chgrp, or
other system crash. Under such case, the MDT-object owner information
is trusted over the OST-object's. Because the chown/chgrp processing
order is: client => MDT => OST, it is possible that the OST-object
owner information is stale rather than the MDT-object's. Also, the
MDT-object's owner information is visible to users and can be directly
repaired by the system administrator, while the OST-object's owner
information is only used internally by quota. So the LFSCK will update
the OST-object owner information according to the MDT-object's owner.

If both MDT-object1 and MDT-object2 claim the OST-object1 as one
of its child OST-object, but the OST-object1 only recognizes the
MDT-object1, then the LFSCK will create new a OST-object and fix
the MDT-object2's layout information to reference the new created
OST-object.

Replace is_remote_th() with is_only_remote_trans(), then drop the
compat patch http://review.whamcloud.com/9361

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I6b148180b5a2d68650b291250c03aac651e5f6e9
Reviewed-on: http://review.whamcloud.com/7524
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3591 lfsck: repair unmatched MDT-OST objects pairs

Sometimes, the MDT-object1 claims that the OST-object1 is one of its
child objects. But the OST-object1 says inconsistent information:

1. It claims invalid parent information, such as empty or bad parent
   FID information.
2. It claims that its parent is the MDT-object2, but the MDT-object2
   does not exist, or
3. The MDT-object2 exists, but it does not recognize the OST-object1.

Under such cases, the MDT-object layout information is trusted over
the OST-object back-pointer because it relates to user visible file
data. The OST-object back-pointer is only used for internal recovery
purposes and is not visible to the user, so does not affect proper
file usage information, nor was kept consistent for Lustre 1.8.x MDT
file-level backup/restore. The LFSCK will update the OST-object to
make it recognize the new parent.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I01e67baf661b0a9e1c3de37a35de86699b07d049
Reviewed-on: http://review.whamcloud.com/7519
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-3590 lfsck: repair MDT-object with dangling reference

If the OST-object referenced by the MDT-object is lost, then the
LFSCK needs to recreate the OST-object with the specified FID and
initialize it with the given parent MDT-object FID and owner attr.
Although the new created OST-object is initialized, the SUID+SGID
mode will be kept, which will be dropped by the first modification
RPC, like write/punch/setattr. Then we can distinguish whether the
recreate OST-object has been modified or not.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Ic45254695e7b1902020c133bb23fd32685b9a414
Reviewed-on: http://review.whamcloud.com/7517
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Tested-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3590 osp: compat macro for is_remote_trans

There is a clash between a recently landed DNE patch and
a ready to land LFSCK series.

Change-Id: I0b3f9805fbf892e4ad0eb4b7fc736a871c438d77
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/9361
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-4629 gss: fix few issues found by Klocwork Insight tool

Array 'message_buf' of size 500 may use index value(s) -1

Object 'enc_key.data' was freed at line 164 after being freed
by calling 'free' at line 150. Also there are 3 similar errors
on line(s) 164.

Suspicious dereference of pointer 'vmsg' before NULL check at
line 187. Also there are 2 similar errors on line(s) 196, 205.

Suspicious dereference of pointer 'rmsg' before NULL check at
line 191. Also there are 2 similar errors on line(s) 200, 209.

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: I50905ea99d904123df30ba7078b180b44b8a6e06
Reviewed-on: http://review.whamcloud.com/9274
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4629 llite: fix suspicious dereference (merge issue)

Suspicious dereference of pointer 'lfd' before NULL check at line 286

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: I64c652279abb8fa1e720d23d645f74f07e5237ca
Reviewed-on: http://review.whamcloud.com/9273
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3963 libcfs: remove cfs_hash_long

Replace the name with Linux defined hash_long.
The similar patch has already been submitted upstream.

Signed-off-by: Peng Tao <bergwolf@gmail.com>
Change-Id: Ia96bb703284ed4843c4433a1a50539d9c68ed6d1
Reviewed-on: http://review.whamcloud.com/9268
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-4598 quota: fix s-q test_30

After LU-4139 landed, the block grace time isn't accurate as before,
the s-q test_30 should now write more bytes to make sure the spare
quota allocated on slave be used up.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: Iee14721051b9e41074a13f11afb11a7b286352c2
Reviewed-on: http://review.whamcloud.com/9177
Tested-by: Jenkins
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>

LU-4423 obdclass: fix return value check in capa_hmac()

In case of error, the function crypto_alloc_hash() returns ERR_PTR()
and never returns NULL. The NULL test in the return value check
should be replaced with IS_ERR().

Signed-off-by: Wei Yongjun <yongjun_wei@trendmicro.com.cn>
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Change-Id: I4889387752d1eb5400649cd5f4da172d64c054e2
Reviewed-on: http://review.whamcloud.com/8681
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1538 tests: delete test files from /tmp after use

Delete files created for tests in /tmp after testing has finished.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I8c3fb62f844cb50d82eba81f274d86c73e3d2e08
Reviewed-on: http://review.whamcloud.com/8615
Tested-by: Jenkins
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3943 utils: fix lfs df -i summary line

If the number of free objects on the OSTs is fewer than on the MDT,
use the number of free OST objects in the filesystem summary, so
it matches "df -i" (limited internally by ll_statfs_internal()).

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: Icfaaba3f9c39f12174e6681e9fb68c1f7a2540e5
Reviewed-on: http://review.whamcloud.com/8614
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4413 ptlrpc: don't try to recover no_recov connection

If a connection has been stopped with ptlrpc_pinger_del_import() and
marked obd_no_recov, don't reconnect in ptlrpc_disconnect_import() if
the import is already disconnected.  Otherwise, without the pinger it
will just wait there indefinitely for the reconnection that will never
happen.

Put the obd_no_recov check inside ptlrpc_import_in_recovery() so that
any threads waiting on the connection to recover would also be broken
out of their sleep if obd_no_recov is set.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: Icd8041be0ce344add8d67b026353df1b1e0cab07
Reviewed-on: http://review.whamcloud.com/8996
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4515 tests: set fail_loc only once per node

Since fail_loc and fail_val are common for all services on a node,
it is only necessary to set it once per node instead of once per
facet.  That avoids a bunch of extra remote commands and spew in
the test output.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I9a69eb325fa80a90d929ab9a258cce21973ebbe5
Reviewed-on: http://review.whamcloud.com/8978
Tested-by: Jenkins
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4356 kernel: kernel update [SLES11 SP2 3.0.101-0.7]

update target and config files for new version

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: Ia5a9472029e7134b3f6b997506371f5e5e624797
Reviewed-on: http://review.whamcloud.com/8951
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4102 utils: add dry-run to ll_recover_lost_found_objs

Add the dry-run (-n) option to ll_recover_lost_found_objs.  This
allows scanning an OST filesystem without modifying it.

It is now possible to test both the "lost+found/" directory, as well
as the "O/" directory to verify the LMA and FID xattrs on existing
objects that are in the normal filesystem hierarchy.

Fix verbose (-v) option to print all of the inodes being checked.
This option previously did nothing.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I8e2b427046c6acf5cf7429f41ccd57496c500c1e
Reviewed-on: http://review.whamcloud.com/8061
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4597 clio: clear nowait flag agl lock re-enqueue

The LDLM_FL_BLOCK_NOWAIT flag should be cleared when re-enqueue
the agl lock as normal glimpse, otherwise, it won't get size back
if there is conflicting locks on other client.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: Ifd311606e824d6574bfbf3256841061e8867214a
Reviewed-on: http://review.whamcloud.com/9249
Tested-by: Jenkins
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Ned Bass <bass6@llnl.gov>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4529 quota: call qsd_op_end() after trasn stop

qsd_op_end() shouldn't be called before the transaction stopped,
because qsd_op_end() is a quite heavy operation which could
probably allocate memory with standard allocator flag (__GFP_IO),
and allocating memory could result in dirty flush on other
filesystems, that will lead to opening transaction on different
journal and trigger the assert in jbd2_journal_start():
J_ASSERT(handle->h_transaction->t_journal == journal) at the end.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I4ea3ff011fa7e44460b9912050e90b174813e01a
Reviewed-on: http://review.whamcloud.com/8968
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>

LU-4488 build: fix compilation with --enable-invariants

Fix the build which was broken since the following commit:

    commit 0a259bd7dbac76d75b89a389bc317720153aa452
    Author: Jinshan Xiong <jinshan.xiong@intel.com>
    Date:   Mon Sep 30 15:00:38 2013 -0700

    LU-3321 clio: collapse layer of cl_page

    Move radix tree to osc layer to for performance improvement.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
    Change-Id: I93e3cb8352f7be41c23465b12945874316aa1809
    Reviewed-on: http://review.whamcloud.com/7892
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: Ie8543513852d98d0aa82bca0f227d286cdf8ebd2
Reviewed-on: http://review.whamcloud.com/8853
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Jenkins
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4422 tests: disable sanity-quota test_6 temporary

To avoid other patches to be failed for LU-4422 under DNE.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Ia48f7bb14733f6bdc5ef34bde1ce91b5762f9192
Reviewed-on: http://review.whamcloud.com/9203
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-4482 grant: don't use cache data in osd_statfs()

osd_statfs() shouldn't cache statfs data anymore: the statfs data
is already cached in ofd layer, put another cache in osd layer
looks redundant, and what more important is: grant mechanism relies
on dt_statfs() returning fresh statfs data, caching statfs data in
osd layer would just break grant.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I89b6384cc59d77b1edb0412f24b5c8e823532170
Reviewed-on: http://review.whamcloud.com/8911
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>

LU-3593 lfsck: repair inconsistent layout EA

The layout EA storing on the MDT-object records not only the file
layout but also some information which indicates the layout owner,
such as lov_mds_md.lmm_oi. They are generated from MDT-object FID,
with them we can know which file the layout EA belongs to. In the
LFSCK phase II, we need to verify whether such information in the
layout EA is correct or not by re-caculating from the MDT-object
FID. If inconsistency is found, trust the MDT-object FID rather
than the FID information in the layout EA, and repair the later.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I3c31e19e9fabe66fe7ffdba2fe8569795ae49b4a
Reviewed-on: http://review.whamcloud.com/7456
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Tested-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4472 mdc: Fix mdc_page_locate ASSERT

Storing hash 0 at same as hash 1 should be okay, don't ASSERT in this
case.

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I9b9488e5374d22dcfc9e7f2c969da3b02778097a
Reviewed-on: http://review.whamcloud.com/8821
Tested-by: Jenkins
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Peng Tao <bergwolf@gmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3531 llite: fix "lfs getdirstripe" to show stripe info

Fix "lfs getdirstripe", so it can show layout information
of striped directory

[root@testnode tests]# ../utils/lfs getdirstripe /mnt/lustre/test1
/mnt/lustre/test1
lmv_stripe_count: 2
lmv_stripe_offset: 0
mdtidx  FID[seq:oid:ver]
     0  [0x280000400:0x1:0x0]
     1  [0x2c0000400:0x1:0x0]

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I586f78ee2e0c35d8c3ed10726d5f5e12a4b543e7
Reviewed-on: http://review.whamcloud.com/7228
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3529 lod: create striped directory

1. Add "lfs setdirstripe -i -c" to create striped
directory.

2. client send create request to the master MDT, which
will allocate FIDs and create slaves. for all of slaves.

3. Client needs to revalidate slaves during intent getattr
and open request.

4. lmv_stripe_md will include attributes(size, nlink etc)
from all of stripe, which will be protected by UPDATE lock.
client needs to merge these attributes when update inode.

5. send create request to the MDT where the file is located,
which can help creating master stripe of striped directory.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I7ac560e39dcb415e310dc5e6ade531d76227ffae
Reviewed-on: http://review.whamcloud.com/7196
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-4196 build: Reenable OFED-3.5 support on SLES11

With the merger of LU-4266 support for SLES11 with
OFED-3.5 was accidentally removed. This patch restores
this support.

Change-Id: If70a8815d90e7d1fa998ba82c4dac5a384216353
Signed-off-by: James Simmons <uja.ornl@gmail.com>
Reviewed-on: http://review.whamcloud.com/8884
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4269 ldlm: Hold lock when clearing flag

This patch moves lock's skip flag clearing from lru-delete to
lru-add code to prevent clearing lock's flag without resource lock
proection.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I5cce4699833c2a935e418bdd7181a2151612a8be
Reviewed-on: http://review.whamcloud.com/8772
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4290 llog: discard unavailable records and keep going

if llog can't process some records due to I/O errors or
corruption, just discard them from the header and keep
going. a new test added to verity this behavior.

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Change-Id: Id0dc83ae6239cd55a43eec128b3c750bb9f0894a
Reviewed-on: http://review.whamcloud.com/9281
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-4505 quota: race of edquot updating

The slave edquot flag could be set mistakenly as following:

- slave A acquires quota from master, master found that the
  user is running out of quota, set edquot in reply;
- another slave deletes files and release quota to master,
  master clears edquot and notify all slaves by glimpse;
- glimpse reaches slave A before the reply of dqacq, so
  edquot flag will be set on slave A at the end.

Given that edquot can't be fully trusted, it should only be
revalidated every 5 seconds on the sync acquire path.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: Id4db47462bdf620a42cd31f75726fbcaff869179
Reviewed-on: http://review.whamcloud.com/8954
Tested-by: Jenkins
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-4620 kernel: kernel update [RHEL6.5 2.6.32-431.5.1.el6]

update RHEL6.5 kernel to 2.6.32-431.5.1.el6

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I5b11d0b0e1c3749232caf8d21d365ae351f538aa
Reviewed-on: http://review.whamcloud.com/9253
Tested-by: Jenkins
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Cliff White <cliff.white@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4613 tests: purge older request result in test_12o

sanity-hsm/test_12o sub-test, which has been introduced as part
of LU-3834, submits 2 RESTORE requests for the same FID and thus
needs to purge 1st result from log before to check 2nd.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: Ia2a0ead487b29a68c8a920bae2aa1d654eac4051
Reviewed-on: http://review.whamcloud.com/9235
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: jacques-Charles Lafoucriere <jacques-charles.lafoucriere@cea.fr>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2687 test: add b2_4 zfs image for conf-sanity test_32a

In order to ensure that we do not break ZFS upgrades
in the future, add 2.4.0 zfs filesystem test image for
conf-sanity.sh test_32a.

Test-Parameters: mdtfilesystemtype=zfs \
ostfilesystemtype=zfs mdsfilesystemtype=zfs \
envdefinitions=SLOW=yes testlist=conf-sanity

Change-Id: Iae560e05b428907409dc7069d30b601b52750cca
Signed-off-by: Wei Liu <wei3.liu@intel.com>
Reviewed-on: http://review.whamcloud.com/7193
Tested-by: Jenkins
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1778 libcfs: add a service that prints a nidlist

The libcfs already provides services to parse a string into a nidlist
and to match a nid into a nidlist. This patch implements a service
that prints a nidlist into a buffer.

This is required for instance to print the nosquash_nids parameter
of the MDT procfs component.

Additionally, this patch fixes a bug in return code of
parse_addrange() routine, so that parsing of nids including
a * character works fine ('*@elan' for instance).

Signed-off-by: Gregoire Pichon <gregoire.pichon@bull.net>
Change-Id: I518845b03b34ab5a1e2cbc673c58c5a384702930
Reviewed-on: http://review.whamcloud.com/9221
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1032 build: Honor --disable-modules option in spec file

All the way back to 2004 Lustre has supported an option to
disable the compilation of the kernel modules.  This can be useful
because there are situations where only the user space componets
are required.

For example, when the Lustre kernel modules are either a) provided
by the kernel, or b) proivded as a dkms package.  In both of these
cases it's desirable to be able to build the lustre package without
building lustre-modules subpackage.

The patch adds that missing functionality to the existing lustre
spec file by leveraging the existing --disable-modules configure
option.

Additionally, a small fix was made to lustre/quota/autoMakefile.am
because it didn't properly support the --disable-modules option.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: Ic4f4f7f19da9951b47c587399a71c42fb0e720d0
Reviewed-on: http://review.whamcloud.com/6020
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Brian J. Murrell <brian.murrell@intel.com>

LU-4577 lnet: Dropped messages are not accounted correctly

LNET messages that are dropped are not accounted for correctly in
/proc/sys/lnet/stats. What I assume to be a simple typo is causing
drop_length to be double-counted and drop_count to never be
incremented.

Signed-off-by: Matt Ezell <ezellma@ornl.gov>
Change-Id: I761c62b1f3c4c4ceffbe47008b79692a6e643458
Reviewed-on: http://review.whamcloud.com/9096
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Tested-by: Jenkins
Reviewed-by: Isaac Huang <he.huang@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4382 ldiskfs: add quota credit for ldiskfs_delete_inode

In ldiskfs_delete_inode() we missed possible journal credits
for journaled quota change, this patch makes it up.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: Ic4ef8030b5d9743b18f0417dde702f60ccdaf5d7
Reviewed-on: http://review.whamcloud.com/9187
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Jenkins
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4293 mdd: Allow layout swap for IGIF FIDs

Patch to also allow layout swap for pre-2.x migrated
files (ie, IGIF FID with linkEA).

Root user special case has also been added to lfs/migrate
command to map owner/group of original file to
volatile, in order to comply with other layout_swap rules.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: Ia7e7cb2e6e36ba67a57474b8a806a53257a3e014
Reviewed-on: http://review.whamcloud.com/8737
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-4543 osd: return actual hash value for a record

hash value should be fetched only once we've got a record.

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Change-Id: If02a5ba0c85c0230dea799445a8b985ed1a6fbae
Reviewed-on: http://review.whamcloud.com/9218
Tested-by: Jenkins
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Isaac Huang <he.huang@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4386 osc: don't activate deactivated obd_import

In ptlrpc_activate_import(), obd_import->imp_deactive should
be checked if it is deactivated, otherwise it will trigger an
LBUG in ptlrpc_invalidate_import():

  ptlrpc_invalidate_import() ASSERTION(imp->imp_invalid) failed

Change-Id: I4c16f166c0c2cf60664119bf438dfd8606d71a2f
Signed-off-by: Hongchao Zhang <hongchao.zhang@intel.com>
Reviewed-on: http://review.whamcloud.com/8747
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4625 gss: fixup for shared key mechanism & flavors

Fixup for Commit 6323d52abfe4cf1eda06b4ac3a5b325d9fa13276
The new file lustre/ptlrpc/gss/gss_sk_mech.c was added but
not in Makefile.

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: I7cc1f0df848877e2ad07ad89b0ad1a0182374a96
Reviewed-on: http://review.whamcloud.com/9287
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-4319 build: Clean up rpms/srpm Make targets

The "rpms" and "srpm" targets were unnecessarily complicated.  The rpms
target in particular has a very long shell script embedded in the
autoMakefile, which is not especially desirable.  Because of the embedded
shell script with its associated backslashes, we didn't use standard
autoconf/automake macros because we didn't want shell comments to appear
after line continuation.  To get around that, we need another layer of
variables to convert autoconf/automake variables into Make variables.

It gets rather difficult to read and modify.

Instead we move the scripting into autoconf m4 files, where scripting
is much easier (little line continuations necessary, far fewer escapes needed).
We also have direct access to the original variables, so we don't need
to hop through two or three files before we eventually find where
a variable gets set.

All of the decisions are made at configure time anyway, so constructing
the command line options for rpmbuild at configure time is the Right Thing
to do.

A nice side effect of this change is that one can now easily look at
the autoMakefile after running "./configure" and see exactly the command
line that will be passed to rpmbuild.

Change-Id: I10fcfa740d9e901805615c2262263cc1ea8552bf
Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>
Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-on: http://review.whamcloud.com/8426
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4525 lfsck: distinguish objects visibility by LFSCK

Originally, the ldiskfs backend otable-based iteration only returned
namespace visible FIDs. That means that the OSD needs to distinguish
related objects visibility. But the OSD should not has the knowledge
about the objects visibility. It is the iteration caller - LFSCK, to
distinguish that by itself.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I1eef4041170e856af00a4b222d053ccb3d8d0023
Reviewed-on: http://review.whamcloud.com/9186
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4369 build: make --disable-ldiskfs option workable

Building ldiskfs is enabled by default, so we need to disable it
if --disable-ldiskfs option is specified to ./configure

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: I7ffb013b976e870de32d38b669a1437f8388bbda
Reviewed-on: http://review.whamcloud.com/8883
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Christopher J. Morrone <chris.morrone.llnl@gmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4612 lvfs: correct call to pop_ctxt

Earlier commit 3e7573cc14a331f01150814495e2345793e22f06 that converted
a call of osd_pop_ctxt() to pop_ctxt() ignored the fact the argument
order of these routines were different. This led to panics.

This patch fixes the pop_ctxt() call by putting the arguments
in the correct order.

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I352250fe3ed91cb8a23a5b8e88b944dc8309b481
Reviewed-on: http://review.whamcloud.com/9224
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3373 osd-ldiskfs: export ext4_truncate

The latest kernel removes inode_operations::truncate member, while
SLES11 kernel still keep the member but does not fill that member
for regular file.

This patch exports symbol for ext4_truncate, and calls it directly
in osd_punch() if the regular file inode operation does not fill its
truncate function.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I42477ca3f1a56e9c0870a641431936298f6d71b5
Reviewed-on: http://review.whamcloud.com/9174
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3289 gss: Shared key mechanism & flavors

Implement security flavors and GSSAPI mechanism to perform shared key
authentication (ski) and encryption (skpi).

Signed-off-by: Andrew Korty <ajk@iu.edu>
Change-Id: I48855c098965fcf527b3949c6dfb181d457b4ca5
Reviewed-on: http://review.whamcloud.com/8629
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Reviewed-by: Ken Hornstein <kenh@cmf.nrl.navy.mil>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-1267 lfsck: enhance API for MDT-OST consistency

Introduce new dt_object method ::do_declare_attr_get(). The caller
can use such method to notify low layer that "It will need the OST
object attribute very soon, please help to prepare in advance". For
the LFSCK layout consistency verification, the osp_declare_attr_get()
will use UPDATE_OBJ RPC with sub_opcode OBJ_ATTR_GET.

Similarly, another new new dt_object method ::do_declare_xattr_get()
is used to notify low layer that "It will need the OST object xattr
very soon, please help to prepare in advance", which uses UPDATE_OBJ
RPC with sub_opcode OBJ_XATTR_GET.

These idempotent requests can be batched together during the phase
of declaration and sent out via single OUT RPC. It can be shared by
any thread that wants to send idempotent requests to the same OST.

Introduce cache in OSP for remote object's attribute and extended
attribute. Currently, it is mainly used to hold those pre-fetched
OST-objects' kinds of attr/parent FID. But it also can be used by
DNE for other purposes in the future.

For performance, the batched idempotent OUT RPC uses asynchronous
mode. Every sub operation needs to register its own interpterer.
These interpterers will be called one by one after the batched OUT
RPC is replied by the OST.

Implement do_attr_get() against OSP-object for the MDT to get OST
object attribute.

Implement do_xattr_get() against OSP-object for the MDT to get OST
object parent FID attribute.

Implement do_xattr_set() against OSP-object for the MDT to set OST
object parent FID extended attribute.

Some code cleanup and re-organization, such as moving transaction
related code from osp/osp_md_object.c to osp/osp_trans.c, moving
common OUT code from osp/osp_md_object.c to target/out_lib.c.

Originally, only DNE operations will use OUT RPCs, so they use sync
mode transaction. But with LFSCK phase II introduced, sync mode OUT
RPC processing is bad performance. The patch improves related funcs
to allow LFSCK to use async mode transaction for OUT RPC processing,
and DNE related operations still use sync mode transaction.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I4fe99f96ad24d43c1edea3a4a16b7ed206c38c4f
Reviewed-on: http://review.whamcloud.com/7156
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4209 utils: fix O_TMPFILE/O_LOV_DELAY_CREATE conflict

In kernel 3.11 O_TMPFILE was introduced, but the open flag value
conflicts with the O_LOV_DELAY_CREATE flag 020000000 added to fix
LU-812 in Lustre 2.4.  O_LOV_DELAY_CREATE allows applications
to defer file layout and object creation from open time (the default)
until it can instead be specified by the application using an ioctl.

Instead of trying to find a non-conflicting O_LOV_DELAY_CREATE flag
or define a Lustre-specific flag that isn't of use to most/any other
filesystems, use (O_NOCTTY|FASYNC) as the new value.  These flags
are not meaningful for newly-created regular files and should be
ok since O_LOV_DELAY_CREATE is only meaningful for new files.

I looked into using O_ACCMODE/FMODE_WRITE_IOCTL, which allows calling
ioctl() on the minimally-opened fd and is close to what is needed,
but that doesn't allow specifying the actual read or write mode for
the file, and fcntl(F_SETFL) doesn't allow O_RDONLY/O_WRONLY/O_RDWR
to be set after the file is opened.

We will keep the 0100000000 flag for backward compatibility until
3.13 is the oldest client kernel that is supported, but drop the
conflicting __O_TMPFILE value of 02000000 since that will cause an
error when running on newer kernels.  The 020000000 has only been
used since Lustre 2.4.0 and always in conjunction with 0100000000,
so any apps that used O_LOV_DELAY_CREATE directly instead of calling
llapi_file_create*() will still work until Linux 3.13 is used.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I565f3454616edc60c6acee01034aa5d773500c1e
Reviewed-on: http://review.whamcloud.com/8312
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Peng Tao <bergwolf@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4554 lfsck: old single-OI MDT always scrubbed

Old ldiskfs MDT's that contain a single OI container named "oi.16"
trigger an automatic OI scrub on each restart.  This is because
osd_oi_table_open() gets ENOENT opening "oi.16.0" and consequently
sets bit 0 in scrub_file::sf_oi_bitmap.  This bit indicates the OI
container 0 needs to be recreated, and it triggers a scrub in
osd_fid_lookup() for lookups that fail with ENOENT.  Fix this by
clearing the bit in osd_oi_init() after a successful open of
"oi.16".

Signed-off-by: Ned Bass <bass6@llnl.gov>
Change-Id: Ie69223d3f8289c90de46f9afe0a2de0e0625b0f6
Reviewed-on: http://review.whamcloud.com/9067
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>

LU-1032 build: Add Lustre DKMS spec file

Add a lustre-dkms.spec file which can be used to distribute dkms
style Lustre modules.  The spec file is originally based on the
generic dkms template and the default behavior is as follows:

* Disable ldiskfs osd support.  The ldiskfs packages currently
  cannot be built reliably against arbitrary kernels and are
  therefore disabled by default.

* Enable zfs osd support.  ZFS dkms packages are hosted at
  http://archive.zfsonlinux.org/{epel,fedora}/{release}/ and
  are compatible once LU-3117 is merged in to the Lustre source.

* Some of the default Lustre build options can be changed by
  setting parameted in the /etc/sysconfig/lustre config file.
  Going forward the options can be extended as needed.  The
  currently supported options are:

    * LUSTRE_DKMS_DISABLE_CDEBUG=y|N
    * LUSTRE_DKMS_DISABLE_TRACE=y|N
    * LUSTRE_DKMS_DISABLE_ASSERT=y|N
    * LUSTRE_DKMS_DISABLE_STRIP=y|N

* A build target was not added for the lustre-dkms.spec file.
  To create lustre dkms packages you must manually invoke rpmbuild.

    ./configure --enable-dist
    make dist
    rpmbuild -bs lustre-dkms.spec lustre-x.y.z.tar.gz
    rpmbuild --rebuild lustre-dkms-x.y.z-r.dist.src.rpm

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: I870f362b8948d5cd28a8dccd98b565e38ad2da7c
Reviewed-on: http://review.whamcloud.com/6019
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Brian J. Murrell <brian.murrell@intel.com>

LU-4512 hsm: Fix lhsmtool_posix --report option

The --report option is intended to allow an override of the
default copytool progress reporting interval, but it doesn't
work. This commit implements the intended functionality and
renames the option to "--update-progress", or "-u" for short.

Also fixes the progress display in hsm/active_requests to
reflect the change from percentage complete to bytes moved.

Signed-off-by: Michael MacDonald <michael.macdonald@intel.com>
Change-Id: Id6ead1b33868e3454f00053165944bc3900cabb4
Reviewed-on: http://review.whamcloud.com/9034
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Henri Doreau <henri.doreau@cea.fr>
Reviewed-by: jacques-Charles Lafoucriere <jacques-charles.lafoucriere@cea.fr>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4287 kernel: kernel update RHEL6.5 [2.6.32-431.3.1.el6]

Add RHEL6.5 support [2.6.32-431.3.1.el6]

ext4 in RHEL6.5's kernel version 2.6.32-431.3.1.el6 no longer contains
the required function ext4_ext_walk_space(). We start a new rhel6.5
ldiskfs patch series and reintroduce ext4_ext_walk_space() through an
new patch, copying ext4_ext_walk_space() from older kernel rhel6.4
2.6.32-358.23.2.el6.

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Signed-off-by: yang sheng <yang.sheng@intel.com>
Change-Id: I7112747970343b1264910aa21d7a62c45b5ca1ea
Reviewed-on: http://review.whamcloud.com/8549
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4571 fld: resend seq lookup RPC if it is on LWP

Because Light Weight connection might be evicted after
restart, then cause inflight RPC fails, to avoid this,
we need resend seq lookup RPC.

remove "-f" from "stop mdt" in sanity 17m, so umount can
keep the the connection, and otherwise the OSP might be
evicted.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I032dfb95e65da56b198129c6d6d6039bad08ab9c
Reviewed-on: http://review.whamcloud.com/9106
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>

LU-4590 ptlrpc: Remove log message about export timer update

Function ptlrpc_update_export_timer generates lots of D_HA level log
messages whenever the export timer gets updated. Those log messages
are found little use for issue investigations, and it will take space
in the Lustre log buffer. We are removing it now.

Xyratex-bug-id: MRP-733
Signed-off-by: Cheng Shao <cheng_shao@xyratex.com>
Change-Id: I3699e81fd4bf0b8677c1fbd09ced5d81ffba3f81
Reviewed-on: http://review.whamcloud.com/9147
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4589 kernel: kernel update [SLES11 SP3 3.0.101-0.15]

update target and config files for new kernel version

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I91a003a5b1947287265b06f16eb5cb9c817d5758
Reviewed-on: http://review.whamcloud.com/9149
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4586 ptlrpc: cast type in the swith op

should allow to build with gcc-4.7.2

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Change-Id: I489ea927d3dc87a7b01f57c5d390612c015b8c47
Reviewed-on: http://review.whamcloud.com/9130
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Li Xi <pkuelelixi@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

Revert "LU-1778 libcfs: add a service that prints a nidlist"

Whoops, this patch broke build: http://build.whamcloud.com/job/lustre-master/arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/1879/changes

So I am reverting it.

This reverts commit 874f67c06da8304a194df5fc0dd5a2c61937076c.

Change-Id: Ieb36ba5c909bc3731dc4a925d89773be89ab64ec
Reviewed-on: http://review.whamcloud.com/9178
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Tested-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1778 libcfs: add a service that prints a nidlist

The libcfs already provides services to parse a string into a nidlist
and to match a nid into a nidlist. This patch implements a service
that prints a nidlist into a buffer.

This is required for instance to print the nosquash_nids parameter
of the MDT procfs component.

Additionally, this patch fixes a bug in return code of
parse_addrange() routine, so that parsing of nids including
a * character works fine ('*@elan' for instance).

Signed-off-by: Gregoire Pichon <gregoire.pichon@bull.net>
Change-Id: I5dbc405e02b8f0f90d45e1a7e44589d5972cc384
Reviewed-on: http://review.whamcloud.com/8479
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>