LU-2469 test: Skip failing ZFS tests

Skip tests that are currently failing for ZFS.  Associate with bugs so
that they can be unskipped when they are fixed.

Test-Parameters: mdsfilesystemtype=zfs ostfilesystemtype=zfs   mdtfilesystemtype=zfs
Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I7fe3a2ad51659c4e31240594702a4cdc87a67336
Reviewed-on: http://review.whamcloud.com/5472
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2280 tests: wait osc import to be FULL after sanityn 34.

1. It should wait osc import to be FULL before proceeding to the
following test.

2. fixes in osc_wait_import_event for waiting multiple osc imports.

Signed-off-by: Wang Di <di.wang@intel.com>
Change-Id: Ia054264954a6e0aacadc7c8062c2f353cd87f834
Reviewed-on: http://review.whamcloud.com/4468
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2624 ptlrpc: improve stop of ptlrpcd threads

This fix improves the stop of ptlrpcd threads so that
it occurs in parallel. On machines with a large number
of cores this significantly reduce the time of unload of
ptlrpc module.

Signed-off-by: Gregoire Pichon <gregoire.pichon@bull.net>
Change-Id: I744de5d9887bfaa4c7a742d34923680e3a536ee7
Reviewed-on: http://review.whamcloud.com/5039
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2720 clio: remove CILR_PEEK optimization

some issues with that optimization in mkwrite are found:
1) ENODATA handling wrote the page and invalidated it, but the page
could have been re-added to the cache in between these 2 steps;
2) lock could be busy, so we get ENODATA on trylock, whereas a racy
ll_fault waits for lock mutex and re-adds the page to the cache;

The patch also adds a loop over trying to take a cached lock while
it is used.

Signed-off-by: Vitaly Fertman <vitaly_fertman@xyratex.com>
Change-Id: I7799b45ff3a0a60f307e05ffd1618d23e0e77b72
Xyratex-bug-id: MRP-752
Reviewed-on: http://review.whamcloud.com/5222
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2591 lov: race between ptlrpc_rcv and umount/umount thread

The race which is refered here happens in the following scenario

1) mount runs but fails to communicate with some OSTs. Then the
   import objects which represents the OSTs are registered to
   a pinger list.
2) pinger succeeds to communicate with, at least, ONE OST. Then
   ptlrpc_rcv calls lov_set_osc_active to activate the OST and
   holds lov_refcount.
4) For some reason ... possibly mount finally fails or umount runs,
   ll_put_super is called
5) ll_put_super tries to disconnect all OSTs with lov_disconnect
   and this func calls lov_del_target to set all OSC's
   target->ltd_reap flags in order for lov_putref to handle all
   of them.
6) ptlrpc_rcv thread puts lov_refcount and if lov_refcount becomes
   0 here, the thread has to disconnect all the OSCs whose
   ltd->reap has been set by __lov_del_obd.
7) Some OSCs' imports have still been LUSTRE_IMP_CONNECTING state
   because of (2), so ptlrpc_rcv thread has to wait for these
   import states to be changed to non-recovery states, such as FULL,
   CLOSED or DISCON at ptlrpc_disconnect_import.

Now that ptlrpc_rcv thread is waiting for the import states to be
changed to non-recovery states but ptlrpc_rcv is the one who is
supposed to change a recovery state to a non-recovery state, So
ptlrpc_rcv mush hung, And mount/umount thread which has called
ll_put_super has to also wait for changing state at
ptlrpc_disconnect_import, so umount must hung too.

Signed-off-by: Hiroya Nozaki <nozaki.hiroya@jp.fujitsu.com>
Change-Id: Idcf6831d1ee6b72332c943dfde5316fddba6c13f
Reviewed-on: http://review.whamcloud.com/4979
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2792 fld: add dot_seq into in_fldb check

Add FID_SEQ_DOT_LUSTRE into fid_seq_in_fldb, because the
dot_seq is inserted into fldb cache list in fld_server_init.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I6417bab77d48c9d9c991119b29f3d2ab04e88ef2
Reviewed-on: http://review.whamcloud.com/5318
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2801 mdt: checking remote object in mdt_swap_layouts

It should use mdt_remote_object to check remote object in
mdt_swap_layouts, instead of mdt_object_exists < 0.

Send swap layout req to the right MDT.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: Ia24635e16cb67e5683abc9797ec6c9e61f648c62
Reviewed-on: http://review.whamcloud.com/5332
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2739 mdt: Deny non-DNE client to access remote directory

Return EIO to old clients, if they try to access the remote
directory.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I1cc4e8fa6e0f3a4b31b5610642e204bd100c1743
Reviewed-on: http://review.whamcloud.com/5425
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2729 mgs: fix cleanup of nonexistant MGSSELF_NAME proc file

The mgs teardown code gets invoked in the failure case but assumes
that the MGSSELF_NAME proc file has been created. If it hasn't been
created, the proc code issues a WARN_ON.

The setup code clears the FSDB_MGS_SELF from &fsdb->fsdb_flags when it
creates the proc file, so we can key off of that to decide whether to
remove it.

Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Change-Id: Idf58d1fa572eeced5dbaa72f03e38ee46a708166
Reviewed-on: http://review.whamcloud.com/5122
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2642 osd-ldiskfs: Use __REQ_WRITE instead of BIO_RW.

Commit 7b6d91da removed BIO_RW and replaced it with __REQ_WRITE.
(v2.6.35)

Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Change-Id: I33f87d1e7d8ffe90efcd8b173ad56d9ecf723d7a
Reviewed-on: http://review.whamcloud.com/5121
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2158 lvfs: move obdclass related functions to obclass

Currently a lot of functions exist in the lvfs layer
that really belongs to different parts of the stack.
This patch moves obdclass specific code that is located
in the lvfs layer into the proper location.

Signed-off-by: James Simmons <uja.ornl@gmail.com>
Change-Id: I5be4dede1afdbbc4af0157cbcd13a5133ce7a1db
Reviewed-on: http://review.whamcloud.com/5250
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2158 lvfs: move ldiskfs only functions to osd-ldiskfs

Currently a lot of functions exist in the lvfs layer
that really belongs to different parts of the stack.
This patch moves osd-ldiskfs specific code that
is located in the lvfs layer into the proper
location.

Signed-off-by: James Simmons <uja.ornl@gmail.com>
Change-Id: Ia197391599cc7fffad396d487873edd9d6c0b1c4
Reviewed-on: http://review.whamcloud.com/5246
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2789 lod: initialize objects in a detached stripe array

Initializing the ldo_stripe member of lod_device in place exposes some
races with attribute setting so instead we use a detached array and
assign that to ldo_stripe once it's complete.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I63d57a21ed7402829d4b3889c1f32d4ddec1f096
Reviewed-on: http://review.whamcloud.com/5422
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2770 llite: specify ea_size for md_getattr_name

For nfs re-exported case, miss to set ea_size for md_getattr_name(),
then causes the ll_get_parent failure.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I99c8301f9b00cafdf79a6c15541f868f18e4a109
Reviewed-on: http://review.whamcloud.com/5455
Tested-by: Hudson
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2620 tests: Flush changes from test 5 in replay-ost-single

Use wait_delete_completed_mds to flush out and wait for all the
effects on kbytesavail from test 5 to finish before test 6 starts.

Test-Parameters: testlist=replay-ost-single envdefinitions=SLOW=yes

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: Id37cf23402dd609e6743c36a1aff8e71dd246776
Reviewed-on: http://review.whamcloud.com/5042
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2675 cleanup: make bitfield (un)signedness explicit

Fix this common sparse warning (dubious one-bit signed bitfield) in
case ones-complement machines come back or something.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I24ea468efedd8a1269ea92022d13e802f4838ba1
Reviewed-on: http://review.whamcloud.com/5166
Tested-by: Hudson
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Sebastien Buisson <sebastien.buisson@bull.net>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2786 llite: allow setting pool name via fsetxattr()

Setting the pool name in the striping data of a file fails when the
fsetxattr() interface is used to store the lustre extended attributes,
while using ioctl() works as expected.  This is because ll_setxattr()
assumes the size of the lustre user metadata structure to be that of
the older v1 format which doesn't include a pool name field.  Fix this
by using the magic number to detect the format version and setting
the size accordingly when passing the user data down the stack.

Signed-off-by: Ned Bass <bass6@llnl.gov>
Change-Id: I6f6969dfaf2149f268d52272f538fee05459697b
Reviewed-on: http://review.whamcloud.com/5313
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2227 build: fix 'memory illegal access' errors

Fix 'memory illegal access' defects found by Coverity version
6.0.3:
Buffer not null terminated (BUFFER_SIZE_WARNING)
Calling strncpy with a certain maximum size argument on
destination array of same size might leave the destination string
unterminated.
Out-of-bounds read (OVERRUN_STATIC)
Overrunning static array with n elements, at position n.
String not null terminated (STRING_NULL)
Function does not terminate string.

Signed-off-by: Sebastien Buisson <sebastien.buisson@bull.net>
Change-Id: I5ded09054b2e0e5296ba17d7528518fed298fc89
Reviewed-on: http://review.whamcloud.com/4391
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2548 quota: don't apply migration on global index copy

Quota migration shouldn't be applied on the global index copy of
quota slave on mdt.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I9f36b79e867ede20562223765dfc56358a4f3831
Reviewed-on: http://review.whamcloud.com/5259
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>

LU-2391 build: separate lustre-osd-[ldiskfs,zfs] RPMs

Add support for separate lustre-osd-[ldiskfs,zfs] RPMs creation in
lustre.spec.in template. This will prevent osd_[ldiskfs,zfs].ko
modules from being shipped as part of lustre-modules RPM and thus
avoids having many unsatisfied external references error messages
reported when either lustre-ldiskfs or zfs-modules RPMs are missing.
Users willing to build Lustre FSs on top of ldiskfs/zfs back-ends
will have to install the corresponding RPM.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: Ib406443186026ed5ac3dc7b3ec30db41aedac2a2
Reviewed-on: http://review.whamcloud.com/4869
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Brian J. Murrell <brian.murrell@intel.com>
Reviewed-by: Christopher J. Morrone <chris.morrone.llnl@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1812 ldiskfs: detect generated utsrelease.h

Fix lustre-ldiskfs.spec.in to use SLES11's generated header files.

Signed-off-by: James Simmons <uja.ornl@gmail.com>
Signed-off-by: chas williams - CONTRACTOR <chas@cmf.nrl.navy.mil>
Change-Id: I0eb198274015bfd38c48e9f68ceb15fe9e601d17
Reviewed-on: http://review.whamcloud.com/5188
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2634 ldiskfs: clear extent flag for fast symlink

Fast symlinks on MDT filesystems formatted with the "extents" feature
are created with EXT4_EXTENTS_FL. e2fsck considers this a corruption
and deletes the symlink.

The patch is to clear LDISKFS_INODE_EXTENTS flag in
osd_ldiskfs_writelink() just as ext4_symlink() does.

Signed-off-by: Liu Ying <emoly.liu@intel.com>
Change-Id: Ib7c1eee7becfaf2b80805d3f6c13658686ad5699
Reviewed-on: http://review.whamcloud.com/5154
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>

LU-582 test: fix bug in get_ost_param in sanity.sh

in get_ost_param(), the ouput should be gl_sum, which is
the sum of 'gl' in all OSS.

Signed-off-by: Hongchao Zhang <hongchao.zhang@whamcloud.com>
Change-Id: I51d3aa591c3ca875f8a559974cf05d250b0c032a
Reviewed-on: http://review.whamcloud.com/3693
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Jian Yu <jian.yu@intel.com>
Tested-by: Hudson
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2775 osp: enable fid-on-OST only for DNE.

1. MDT0 will only create IDIF fids, so old clients, who
only know IDIF, can still access files on MDT0.

2. Using fid_ostid_pack to pack the FID into ostid, otherwise
IDIF will be packed wrongly and caused the panic on the OST0.

3. Add dd into compatible tests of conf-sanity.sh(32) to
check OST fids on the upgrade FS.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: Ic9322bd1c66e5fedec68c10aa2f300d5ecb3b6db
Reviewed-on: http://review.whamcloud.com/5307
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>

LU-2625 scripts: prevent several errors in lustre_rmmod

This fix prevents lustre_rmmod scripts from returning
errors when lustre modules are partially or not loaded.

Signed-off-by: Gregoire Pichon <gregoire.pichon@bull.net>
Change-Id: Iefb4166b9b14315319858031f53ecb148c589474
Reviewed-on: http://review.whamcloud.com/5040
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2608 tests: check_catastrophe success on failure to connect

Allow check_catastrophe to pass on failure to connect to host.
Otherwise LBUG assertions crop up when hosts are overloaded.

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Ie873b817911613369a226341d8efe06707b08bc4
Reviewed-on: http://review.whamcloud.com/5200
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2785 mdd: do not use mti_fid in mdd_find_or_create_root()

as in the subsequent callchain it can be used by
mdd_create_sanity_check()

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Change-Id: Ia357f587192c54e3c406a35054dca47580077056
Reviewed-on: http://review.whamcloud.com/5311
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>

LU-2494 tests: add interop support to framework check

Modify the test-framework wait_mds_ost_sync() function to work
with mds_sync when old_sync_processed doesn't exist.
This should cover both old and new servers.

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I2e5c430a96ac40a1ca87990696e7daaad5a69102
Reviewed-on: http://review.whamcloud.com/5021
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2734 tests: Remove XXXDEV overrides in init_facets_vars()

init_facets_vars() assigns outputs of xxxdevname() calls to XXXDEVs.
This is incorrect because XXXDEVs are supposed to contain device names
while xxxdevname() calls may print dataset names for ZFS-based facets.
Actually, XXXDEV should not be changed at all.  This patch removes the
XXXDEV assignments.

Change-Id: Id484390cbd814c6fa919c02e390700f914561434
Signed-off-by: Li Wei <wei.g.li@intel.com>
Reviewed-on: http://review.whamcloud.com/5243
Tested-by: Hudson
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

New tag 2.3.61

Change-Id: If171f34ebcec4d651ca091af4197c71d017ee645

LU-1940 test: use default resend count for 118c

in sanity, the 118c could fail with the small resend count
for SLOW=no case, using the default value instead.

Signed-off-by: Hongchao Zhang <hongchao.zhang@intel.com>
Change-Id: I3e5ecc6af2eba560f82a80eb75223f8ab299f397
Reviewed-on: http://review.whamcloud.com/4694
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1812 ldiskfs: don't use sb->s_qf_inums

ext4-quota-first-class.patch was finally accepted into the mainline
Linux kernel via commit 7c319d32 (v3.6). The final version doesn't
include the cached inode numbers so they aren't available on newer
kernels that include this patch (or that are using an ldiskfs
patched using the usptream version).

Since the file system will refuse mounting when the inodes can't
be read, the lookup code isn't invoked without valid inode numbers
anyway.

Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Change-Id: Ie1b824183cf734a0f242bb6483a192f1919dc4be
Reviewed-on: http://review.whamcloud.com/5264
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>

LU-2675 cleanup: remove User Mode Linux specific code

The UML specific code has been uncompilable for some time and no one
has complained so it goes.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ie91c6e4484fee64fec4a2e38effd8a1021e8cddb
Reviewed-on: http://review.whamcloud.com/5175
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>

LU-2673 procfs: call lprocfs_free_xxx_stats() later

Defer calls to lprocfs_free_{obd,md}_stats() and
lprocfs_job_stats_fini() until after the corresponding proc entries
have been removed. Change the return of some proc fini()s to void.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ic410a4cf12eeb084270f5e679222c54b12d80266
Reviewed-on: http://review.whamcloud.com/5160
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2642 quota: update quota APIs

Commit 871a2931 renamed vfs_dq_init to dquot_initialize. (v2.6.34)
Commit 9f754758 renamed vfs_dq_drop to dquot_drop. (v2.6.34)
Commit b43fa828 renamed vfs_dq_transfer to dquot_transfer. (v2.6.34)
Commit 0f0dd62f killed vfs_dq_off and vfs_dq_quota_on_remount and
replaced them with dquot_suspend and dquot_resume. (v2.6.34)

Signed-off-by: James Simmons <uja.ornl@gmail.com>
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Change-Id: I61e5a4d6c4a519457146b4a65ce78bfddebe2899
Reviewed-on: http://review.whamcloud.com/5119
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>

LU-2642 osd-ldiskfs: use struct fs_disk_quota for quotas

Commit b9b2dd36 modified ->get_dqblk to use struct fs_disk_quota.
(v2.6.34)

Signed-off-by: James Simmons <uja.ornl@gmail.com>
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Change-Id: Ifb8a79db37e3f88046c00eb906b8e7264005481b
Reviewed-on: http://review.whamcloud.com/5118
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>

LU-1199 ldiskfs: Remove HAVE_{EXT4,JBD2}_JOURNAL_CALLBACK_{SET,ADD}

Lustre servers now only support kernels/ldiskfs with
{ext4,ldiskfs}_journal_callback_add().  Additionally, the
HAVE_EXT4_JOURNAL_CALLBACK_ADD define is not in the upstream
3.3 kernel where the ext4_journal_callback_add() patch landed.

Therefore we remove the HAVE_{LDISKFS,EXT4}_JOURNAL_CALLBACK_{SET,ADD}
checks and just assume that it is set.

Change-Id: I10cc79fcf07dd05a10ae83014f188c613ab1620c
Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>
Reviewed-on: http://review.whamcloud.com/5055
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-1431 ptlrpc: Support for over 1MB bulk I/O RPC

Increase the maximum size of BRW RPCs between OSCs and OSTs to 4MB
by increasing the number of LNet Memory Descriptors assosiated with
a bulk.  Each bulk transfer is still limited to LNET_MTU_SIZE, so
that it can be passed through routers and does not break RDMA limits.

The client and server negotiate the maximum BRW RPC size at connect
time via ocd_brw_size, with the client sending the maximum size it
supports, and the server returning min(client_max, server_max) back
to the client.

For each RPC, the number of bulk MDs that are registered depends on
the actual RPC size at sending time.  The (max_brw_count - 1) is sent
to the OST in the high 16 bits of obd_ioobj.ioo_max_brw and forms a
mask for the bulk RPC XIDs.  The actual number of bulk transfers
is encoded in the BRW RPC XID.  The masked value of the request XID
determines the starting bulk transfer match bits, and the last bulk
match is the RPC XID.

For older clients this means that the starting and ending bulk match
bits are the RPC XID itself (as it always was) and the old ioo_type
(now ioo_max_brw) field being set to zero would result in a mask of
"1", which just keeps the same RPC XID as it was before.

It is important to note that the client and server do NOT share the
value of PTLRPC_BULK_OPS_MASK, or this would incorrectly mask the
RPC XID if the PTLRPC_BULK_OPS_BITS was ever increased on the server.

The actual BRW RPC size can be controlled for each client separately
by changing the client's max_pages_per_rpc.  For example, this will
set it to 4MB:

        lctl set_param osc.*.max_pages_per_rpc=1024

It is also possible to specify a units suffix, so that the RPC size
can be set independently of the client PAGE_SIZE:

        lctl set_param osc.*.max_pages_per_rpc=4M

By default, the size is 1MB for for OST BRW RPCs until more testing
is done. All other bulk I/O (e.g. MDC<->MDS) is left at 1MB.

With PTLRPC_MAX_BRW_SIZE growing larger, it doesn't make sense for
the readahead to increase the window size so dramatically at one time
Instead, limit readahead growth by the current inode i_blkbits value,
which currently defaults to 4MB.

If we want to tune the readahead growth so that it matches the actual
cl_max_pages_per_rpc value, then this can be done much more easily by
changing the per-inode i_blkbits value, since this is tied to specific
OSTs with specific RPC settings (which may be different for e.g. local
and remote OSTs), and only needs to be checked once.

Change the ras_*() functions to take the inode as an argument, and
always put it first in the argument list for consistency.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Signed-off-by: Sergii Glushchenko <sergii_glushchenko@xyratex.com>
Reviewed-by: Alexey Lyashkov <alexey_lyashkov@xyratex.com>
Reviewed-by: Andrew Perepechko <andrew_perepechko@xyratex.com>
Change-Id: I757b14a04e5d4cc053576e41e47864c743c35b8b
Xyratex-bug-id: MRP-319
Reviewed-on: http://review.whamcloud.com/4993
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>

LU-2710 build: avoid function resolution mistakes by Coverity

Remove function resolution mistakes made by Coverity version
6.0.3.
The .hs_keycmp function is incorrectly resolved by Coverity,
making the variable passed as its first 'key' parameter to
be considered of the wrong type. Data structures being of
different sizes, it ends up with this Coverity defect:
Out-of-bounds access (OVERRUN_STATIC)
Overrunning static array by passing it as an argument to a
function which indexes it at a higher byte position.

Signed-off-by: Sebastien Buisson <sebastien.buisson@bull.net>
Change-Id: Ie062b05a73db16d63c63db89dc4d0e264f3b1f8a
Reviewed-on: http://review.whamcloud.com/5211
Tested-by: Hudson
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-642 lov: make up obd_connect for inactive OSC

When OSC is inactivated before lov tries to connect it, lov_connect()
miss the chance to connect it to OST devices even when it is
activated later.

We need make up the connection for the initially inactive OSC when it
is activated.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I4aca442bc4a77baa2bc14adaf2b0f8f5c7bfa7df
Reviewed-on: http://review.whamcloud.com/4691
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2444 build: fix 'error handling' issues

Fix 'error handling issues' defects found by Coverity version
6.0.3:
Unchecked return value (CHECKED_RETURN)
Value returned from a function is not checked for errors before
being used.
Argument cannot be negative (NEGATIVE_RETURNS)
Negative value used as argument to a function expecting a
positive value.

Signed-off-by: Sebastien Buisson <sebastien.buisson@bull.net>
Change-Id: I44022a7d7227ee46345b77701b4c5199a74c386d
Reviewed-on: http://review.whamcloud.com/4772
Tested-by: Hudson
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2302 scripts: make lfs_migrate compatible with 1.8

Earlier versions of lfs setstripe/getstripe use option -s instead
of -S. Make sure lfs_migrate can be used with any version.

Signed-off-by: Daniel Kobras <d.kobras@science-computing.de>
Signed-off-by: Keith Mannthey <keith.mannthey@intel.com>
Change-Id: Ic3caaeaf3d6c764afbd75cd7a8a7279462d21d47
Reviewed-on: http://review.whamcloud.com/4702
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2590 obdclass: correct swab lov_mds_md

For caller's convenience, the 2nd parameter of obd_free_diskmd()
could be host endianness, it needs swab to LE if necessary, while just
lov_mds_md header needs it for figuring out how much memory needs to
be freed.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I537e83b56b4be6e08a7431bfda1de71efefa33ee
Reviewed-on: http://review.whamcloud.com/5145
Tested-by: Hudson
Reviewed-by: Prakash Surya <surya1@llnl.gov>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2158 fsfilt: Remove unused opcodes

This change removes the unused fsfilt opcode definitions and the
accompanying code in fsfilt_ext3_start. This is strictly cleanup.

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Change-Id: Ifccb6e7c9c728bc98e4ddfa52093e660dd117b8b
Reviewed-on: http://review.whamcloud.com/5192
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>

LU-2158 fsfilt: Remove lvfs_common.c

This change removes the lvfs_common.c file, and moves it's only
remaining functionality directly into the obd_lfs_fid2dentry function.
This should have no functional impact, and is merely a code cleanup
exercise.

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Change-Id: I5f8588a17f6e7659a7d33c2a5f79792826976de4
Reviewed-on: http://review.whamcloud.com/5193
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2360 ptlrpc: don't clobber rq_history_seq with CPT_ANY

In ptlrpc_req_add_history() don't allow service_parts with cpt CPT_ANY
(-1) to clobber the timestamp and sequence bits of rq_history_seq.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I52161d6f36dcd70dd569b0be294e6d23cc5ac4a9
Reviewed-on: http://review.whamcloud.com/5207
Tested-by: Hudson
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2764 tests: Remove obsolete comment in local.sh

The comment above LOAD_MODULES_REMOTE in local.sh was written for the
special case of the 2.3 OSS stack and is no longer applicable to
current master.

Signed-off-by: Li Wei <wei.g.li@intel.com>
Change-Id: I75518e9cebfa717ccfb625e2786c8d191c6bddbc
Reviewed-on: http://review.whamcloud.com/5277
Tested-by: Hudson
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2749 ofd: check return value of ofd_seq_load

Return value ofd_seq_load was not always checked for both NULL and
errno.  This caused a NULL pointer dereference.

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Ib2e1658f16ba2012c9410966c08201cab7a5aa5c
Reviewed-on: http://review.whamcloud.com/5263
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>

LU-2414 tests: cleanup replay-single/0c and 44c

Switch from return codes to error messages in test_0c and test_44c.

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I61895442fa52e2f7bff421e0b8309f0f11c97de8
Reviewed-on: http://review.whamcloud.com/4856
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Hudson
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-1434 utils: extend wireshark parsing

Add support for MGS_CONFIG_READ
Add capabilities where they're defined in layout.c
Fix processing of MGS_BULK_PORTAL (not a Lustre packet)
Make use of LUSTRE_BUFFER_LEN everywhere.
Add dissecting of capabilities field
Add dissecting of fld field
Add dissecting of MGS_CONFIG_BODY
Fixed all compiler warnings.
Fixed parsing of get/set (x)attrs

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Ic75bb4814570b30a1f1dca19d2f49311294ff2fe
Reviewed-on: http://review.whamcloud.com/5155
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2689 interop: use stripe md in ll_update_inode()

When connected to servers that do not support LL_SBI_LAYOUT_LOCK, if
stripe md is available in ll_inode_update() then use it.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I949a0129c3fc92c5434288df0549a54ccd440f97
Reviewed-on: http://review.whamcloud.com/5289
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2562 test: cleanup previous test result

in sanity.sh, test_51ba only depends on test_51b, but test_51a
also create files under $DIR/$tdir, which causes test_51ba fail
for it also takes those files into account.

Change-Id: I42f491f64c8d2d65bee656c167c9523274cd7200
Signed-off-by: Hongchao Zhang <hongchao.zhang@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/5183
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2449 osd: osd-zfs to handle ./.. properly

ZFS does not store directory records . and .. on a disk.
Instead they are generated upon request (like in zfs_readdir()).
with the patch osd-zfs does the same for namespace directories.

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Change-Id: I910f9bad807c0e5a2bee1fd481470576ceaf4444
Reviewed-on: http://review.whamcloud.com/4990
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>

LU-2240 mds: Assign special fid sequence to root.

In current implementation, we still use IGIF as the ROOT FID,
but ZFS does not have IGIF at all, so it will move root ROOT fid
to a new sequence.

But for compatible with clients, which are being mounted
during upgrade, it will still keep the old IGIF root FID.

Signed-off-by: Wang Di <di.wang@intel.com>
Change-Id: I2e6b81cd359436d2e5b96c8a315a56c415f8e631
Reviewed-on: http://review.whamcloud.com/5257
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>

LU-1866 lfsck: LFSCK for namespace consistency (3)

It is the third part of LFSCK 1.5 for namespace consistency
check and repair. It mainly implements the LFSCK namespace
component functionality lfsck_operations::lfsck_double_scan:
process multiple-linked objects.

1) Find out and repair crashed linkEA entry.

2) Find out and remove invalid linkEA entry

Test-Parameters: envdefinitions=ENABLE_QUOTA=yes testlist=sanity-lfsck

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Ia38fa343ed4da10635c1af4d90b9039c5e01688c
Reviewed-on: http://review.whamcloud.com/4914
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2726 procfs: Invalid assertion for counter_{add,sub}

lprocfs_stats::lc_sum_irq is only used for obd_memory for accurate
memory usage counting, other procfs stats counters do not need it,
even though they can be used in RCU callbacks.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I4c2dec50eecc0c63631f1915ba30a54a5da2fec1
Reviewed-on: http://review.whamcloud.com/5237
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: John Hammond <johnlockwoodhammond@gmail.com>
Tested-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2017 mdc: add layout swap between 2 objects

This patch adds the client and MDT code to swap layouts
between 2 files:
- a lfs command
- a llapi call, based on a new ioctl
- the mdc/mdt codes for the ioctl (with layout locking)

Signed-off-by: JC Lafoucriere <jacques-charles.lafoucriere@cea.fr>
Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I7e710a1ab3ca38e8b26582e49d08e9943aa445cd
Reviewed-on: http://review.whamcloud.com/4507
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1866 lfsck: LFSCK for namespace consistency (2)

It is the second part of LFSCK 1.5 for namespace consistency
check and repair. It mainly implements the LFSCK namespace
component functionality lfsck_operations::lfsck_exec_dir:
for linkEA verification and repair.

The basic functionality is to find out missed linkEA entries
and add them to the object. It is also part of handling IGIF
objects upgrading.

Test-Parameters: envdefinitions=ENABLE_QUOTA=yes testlist=sanity-lfsck

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I9df2a1fddfa4b15838ecbdfb6279c240d01d0f95
Reviewed-on: http://review.whamcloud.com/4913
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-1866 lfsck: FID-in-{dirent,LMA} check and repair

As part of LFSCK 1.5, it implementes the following functionalities:

1) Verify and rebuild FID-in-dirent, mainly for MDT file-level
   backup/restore.

2) (Partly) verify and rebuild FID-in-LMA: if the FID-in-dirent
   is valid but FID-in-LMA is missed, then the later one will be
   repaired; but if both are missed, then it is OI scrub's duty
   to repair the inconsistency.

3) (Partly) handle IGIF objects upgrading: build IGIF-in-dirent
   for IGIF objects. The other two parts work for IGIF objects
   upgrading are:

3.1) Generate IGIF-in-LMA, which has been done by OI scrub in
     former patch(es).

3.2) Generate linkEA, which will be done by up layer LFSCK in
     other patch(es).

Test-Parameters: envdefinitions=ENABLE_QUOTA=yes testlist=sanity-lfsck

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Ibb317f0f3f3e9cf6db56bfa25f55d2017b0e1937
Reviewed-on: http://review.whamcloud.com/4912
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1866 lfsck: LFSCK 1.5 user space control

Include the user space command to start/stop the LFSCK for
namespace consistency check and repair. And proc interface
to query the LFSCK status, statistical informtion, and ect.

Test-Parameters: envdefinitions=ENABLE_QUOTA=yes testlist=sanity-lfsck

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Icf4046b313dc1653b5b8b1a3add7fa0329ca7ad0
Reviewed-on: http://review.whamcloud.com/4911
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1866 lfsck: LFSCK for namespace consistency (1)

It is the first part of LFSCK 1.5 for namespace consistency
check and repair. It mainly implements the LFSCK namespace
component internal framework and external APIs.

The functionality parts for the LFSCK namespace component,
such as lfsck_operations::lfsck_{exec_dir,dump,double_scan},
are not included. They will be implemented in other patches.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I011e06c991577e632eadf364b87be74f1f5535c6
Reviewed-on: http://review.whamcloud.com/4910
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1866 lfsck: LFSCK main engine

There are two engines for LFSCK 1.5:

1) One is low layer object table based iteration, which can
   guarantee that all the objects can be scanned at least once.

2) The other is namespace based directory traversal. All the
   namespace based components, such as FID-in-dirent and linkEA,
   can be handled during that.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Ic5a404cde35fdb20c24f6522faa8b21117cf4254
Reviewed-on: http://review.whamcloud.com/4909
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1866 lfsck: general framework for LFSCK 1.5

LFSCK 1.5 main data structure definations, LFSCK
component APIs, setup/cleanup/start/stop interfaces.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I4507b9432289ca8d54cbf686893b706052d0eeb3
Reviewed-on: http://review.whamcloud.com/4908
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-1866 lfsck: ancillary work for namespace LFSCK

1) Share related linkEA functions with LFSCK.

2) Multiple linked objects are rare, and the objects with
   a lot of linkea entries are more rare. It is unnecessary
   to worry about performance under such cases. So drop the
   linkea entries count limitation: LINKEA_MAX_COUNT.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I75f90c9619c56f2f62063435aea26ed01337d6dd
Reviewed-on: http://review.whamcloud.com/4907
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2733 osd: osd-zfs to initialize existing sequence

osd_find_or_add_seq() to use osd_oi_find_or_create() so that
existing sequence is loaded if exists

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Change-Id: I9f173615ef047dd4696d8ed51687c7c268b32dbf
Reviewed-on: http://review.whamcloud.com/5247
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2728 fld: check for NULL param in fldb_seq_stop()

Check for NULL param in fldb_seq_stop().

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I44ed10f9ce8bceabb203e6ef0b843b23340ced2b
Reviewed-on: http://review.whamcloud.com/5238
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2737 osc: double unlock in osc_lru_shrink()

It could unlock twice in osc_lru_shrink() if the cl_io_init() failed.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: Idf3ebafced7f7d3a68c13ea20d12cf221e9d74ce
Reviewed-on: http://review.whamcloud.com/5258
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-1187 mdt: Add MDS_INODELOCK_PERM lock for remote dir

Add MDS_INODELOCK_PERM for remote directory on remote MDT to
protect permission changes on remote MDT. So, Master MDT, where
the parent is, will grant LOOKUP lock to the client. Remote MDT,
where the child is, will grant UPDATE|PERM lock to the client.

And client will clear the dcache during, when either PERM or LOOKUP
lock is being revoked.

Change-Id: Ie11d14bbe7c706a76f7c4bd8f75b07807e2dac02
Signed-off-by: Wang Di <di.wang@intel.com>
Reviewed-on: http://review.whamcloud.com/4346
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2722 clio: directIO thread races with completion thread

ll_direct_IO_26()) ASSERTION( obj->cob_transient_pages == 0 ) failed

The directio thread puts pages to the transfer in osc_io_submit(), and
waits for its completion. Upon completion, osc_completion() is called,
which notifies waiters IO is completed and only after that put the
page. the original directIO thread wakes up and may succeed to assert
on the last page which is not yet put by osc_completion().

Signed-off-by: Vitaly Fertman <vitaly_fertman@xyratex.com>
Change-Id: Ib912e913885a9ff2cb1e9c865ae2003a676c2b5b
Xyratex-bug-id: MRP-838
Reviewed-on: http://review.whamcloud.com/5223
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2467 ptlrpc: Allow OBD_PINGs to be suppressed

This patch introduces a new ptlrpc module parameter, "suppress_pings",
to provide an option for reducing excessive OBD_PINGs in large
clusters.  The parameter affects all MDTs and OSTs on a node.  It is
off (zero) by default, giving a behavior identical to current
implementation.  If it is on (non-zero), all clients of the affected
targets who understand OBD_CONNECT_PINGLESS will know, at connect
time, that pings are not required.  When suppressing pings, there must
be an external mechanism to notify the targets of client deaths, via
the targets' "evict_client" procfs entries.  In addition, a highly
available standalone MGS is also recommended when suppressing pings,
so that clients are notified (through Imperative Recovery) of target
recoveries.

The changes do two basically independent things.  One is initializing
import and export states (i.e., imp_connect_data and
exp_obd_chain_timed) according to "suppress_pings", since whether to
ping or not is a property of each import-export pair.  MGC pings can
not be suppressed, because maintaining MGS connections is dictated by
the reliance on Imperative Recovery.

The other thing is changing pinger and import routines to respect the
import property set earlier.  (The export side does not need any
change at all.)  Pings are still needed to query last committed
transactions if there are uncommitted requests on an import, so that
resources pinned for replays can be released even when applications
become idle.  An early version of this patch removes imports that do
not need to be pinged from pinger_imports and add them back when last
committed transactions are needed or recoveries must be initiated.
This version does not do that because a) the overheads of iterating
through 10,000 imports are not prohibitively large---around 10 ms, b)
adding imports back to pinger_imports requires the global pinger_mutex
to be held, and c) the imp_lock contention added on each import is
small.

Change-Id: Iabc84d395c978c3f156c52aebfad83621facb4fe
Signed-off-by: Li Wei <wei.g.li@intel.com>
Reviewed-on: http://review.whamcloud.com/5009
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Tested-by: Hudson
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2467 protocol: Add OBD_CONNECT_PINGLESS

Reserve a bit for OBD_CONNECT_PINGLESS, which indicates a client is
capable of suppressing keep-alive OBD_PINGs.  If granted by a server,
it means the server does not require (but still allows) pings.

Signed-off-by: Li Wei <wei.g.li@intel.com>
Change-Id: I7575082244a84f2e633a1936fcf894da3c6ee7dd
Reviewed-on: http://review.whamcloud.com/5229
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2723 clio: a client hangs on osc_extent_wait in dio

If dio overlaps with mapped region, first overlapped pages are sent
by generic_file_direct_write->filemap_write_and_wait_range->writepages
After that the dio pages are sent by write_begin/write_end, writepages
However, write_end->osc_page_cache_add does not release extent because
the IO is not considered as synchronous. as the result, the following
writepages->osc_cache_writeback_range() does not send pages as the
extent is still ACTIVE and osc_cache_wait_range() hangs forever.

The patch make directIO to be considered as sync by lustre so that
extent was released at write_end.

Signed-off-by: Vitaly Fertman <vitaly_fertman@xyratex.com>
Xyratex-bug-id: MRP-839
Change-Id: I0c0e1599ee180f467826e4b6715c3cbfd0c6eb5c
Reviewed-on: http://review.whamcloud.com/5224
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2194 tests: add debugging to recovery-small/19

Add error messages to 19a and 19b

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Ief35c063734654c5a6f3dab0d283c4b7a32706e8
Reviewed-on: http://review.whamcloud.com/5141
Reviewed-by: Li Wei <wei.g.li@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-1187 mdt: directory remote open fix

Directory remote open continue as normal open, i.e. return the
remote fid to client, and client will send the open request to
the right MDT.

Set correct it disposition in mdt_open_by_fid.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I32d5dee9cd1eac47787e2d85a5db1e164d2846d3
Reviewed-on: http://review.whamcloud.com/4934
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>

LU-1187 mdd: a few missing stuff in MD stack for DNE.

1. Assign the index operation for the directory create
in declare phase, which is needed for creating the
object in OUT.

2. Declare dotdot insertion for remote directory creation,
so the insert update can be packed into RPC and send to
the remote MDT (out) to be executed.

3. Add hint to the object declare phase, by which OSP can
get some info during update RPC packing, like parent fid.

4. Separate lu_object_exists into lu_object_remote and
lu_object_exists, so it can check whether remote object
exists as well.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: Icb359443d9982ee8567af933c5def42dc51a3a7a
Reviewed-on: http://review.whamcloud.com/4930
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>

LU-2654 ldlm: ldlm_resource_dump should be called with resource locked

Fix ldlm_resource_complain to lock resource before calling
ldlm_resource_dump

Change-Id: I9dfbe84bd63de3093c99e7379daeca30d02fc875
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/5254
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Tested-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1187 mdt: add sanity check for rename and link

Add sanity check for rename/link, so remote rename/link
will return EXDEV.

Signed-off-by: wang di <di.wang@whamcloud.com>
Change-Id: I841c8ffaa74b162fcca5cd1661c383215350cccc
Reviewed-on: http://review.whamcloud.com/4348
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1187 mdt: unlink remote directory

Send unlink req to the slave MDT, so both open/close
and unlink will send request to the slave MDT, where
the remote object resides. Then it would be able to
check orphan (unlink open file) locally.

Add lock_object api for enqueue remote object.

Change-Id: I7483b0f023e4e3de6597da58d3d9f3e96c0d53b7
Signed-off-by: Wang Di <di.wang@intel.com>
Reviewed-on: http://review.whamcloud.com/4339
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-1187 utils: add lfs setdirstripe/getdirstripe

1.Add lfs setdirstripe/mkdir/getdirstripe to create
remote directory.

2.Add lfs rm_entry to just remove the entry of remote
directory.

Change-Id: Idaecfbb9dd97e3ae9fbc70e5b902a91a7b2ed18e
Signed-off-by: Wang Di <di.wang@intel.com>
Reviewed-on: http://review.whamcloud.com/4341
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>

LU-1866 lfsck: enhance otable-based iteration

1) Use ::load() method to unplug OSD layer iteration.

2) Use ::put() method to wakeup the LFSCK main engine
   if it is blocked at low layer for ::next() call.

Test-Parameters: envdefinitions=ENABLE_QUOTA=yes testlist=sanity-scrub
Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I28e5689ad3a4ee96a3f17a64e8d6dca553a86d77
Reviewed-on: http://review.whamcloud.com/4906
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Tested-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>

LU-1866 osd: FID-in-LMA and OI files

Generate FID-in-LMA not only for normal objects, but also for
IGIF objects, server-side local files, and other non-OST/LLOG
objects.

LAST_ID has FID-in-LMA but no related mappings in the OI file.

At the same time, OI mapping usage is also extended to above
objects. Only the objects marked as "LDISKFS_STATE_LUSTRE_NO_OI"
or OST objects have no OI mappings the OI files.

It is part of 1.8 IGIF objects upgrage handling. OI scrub will
automatically generate IGIF-in-LMA and add related OI mappings
in the OI files.

Test-Parameters: envdefinitions=ENABLE_QUOTA=yes testlist=sanity-scrub

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: If1e1434c238f2ab737e35433bfad1abbb00fcb00
Reviewed-on: http://review.whamcloud.com/4904
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>

LU-2733 osd: not return "-ENOENT" for zfs osd_object_init

For zfs backend, the osd_object_init() should not return
"-ENOENT" if osd_fid_lookup() cannot find the object,
which is normal for new created cases.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I76e5d802d0caf8ac3a9992278ec2f7c6c6e2cf07
Reviewed-on: http://review.whamcloud.com/5242
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-1187 osd: add agent/local inode for remote directory

A local inode will be created for remote directory entry, so
to make e2fsck check pass.

The remote directory on the remote MDT will be put under the
agent directory.

Remove LA_TYPE checking during osd_inode_set, because when
initialize the remote object, it needs to get TYPE information
from remote attr get, where we need type information anyway,
(osp_md_attr_get ->out_attr_get->osd_attr_get | LA_TYPE).
So osd_attr_get will always return LA_TYPE valid information
here, then if it do attr_get/modify/attr_set might hit this
ASSERT, So during osd_attr_set, we can just ignore type.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I7f177fa8ae2ab450cde78a8c2da9a2770ec6b0cb
Reviewed-on: http://review.whamcloud.com/4931
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-1866 scrub: initial OI scrub

We will extend OI mapping usage (in OI directories/files) to
all the objects, except for the objects which are marked as
"LDISKFS_STATE_LUSTRE_NO_OI" or OST object.

The OI mappings for the server-side local files may become
invalid because of server-side file-level back/restore, or
server crash. Those files are necessary for server mount,
so related OI mappings must be checked and rebuilt before
they are really used. That is the initial OI scrub work.

Initial OI scrub is used for quickly checking and rebuilding
OI mappings for server-side local files during server mount.

Another change is about MDT_LAST_RECV_OID/OFD_LAST_RECV_OID.
Currently, we can just use the same OID for the "last_rcvd"
files on different targets. So merge the two OIDs as single
OID: LAST_RECV_OID, which also simplifies OI scrub work.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Iddca4812d3c82ca43f1e852fd74475b11b6e6dea
Reviewed-on: http://review.whamcloud.com/4903
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-1866 osd: ancillary work for initial OI scrub

1) Abstract some general interfaces that can be shared by common
   OSD operations, by normal OI scrub, and by initial OI scrub.

2) Mark the object as "LDISKFS_STATE_LUSTRE_NO_OI" if it is not
   unnecessary to add OI mapping for the object, such as backend
   local root, OI directories/files, and others created before OI
   files created.

3) Do not check fld in OI scurb, which may be not ready yet.

4) Reorganize OI scrub code to be more readable.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Ife8168df2454437ad18550b73d26f7372c21776f
Reviewed-on: http://review.whamcloud.com/4902
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2725 ost: ost_rw_hpreq_check should return 0 or 1

ost_rw_hpreq_check was returning number of matched locks which is wrong.

Change-Id: Ic4d3dd4a186b616c13ffd9f897254da3c19f84f8
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/5233
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-1431 ptlrpc: PTLRPC_BRW_MAX_SIZE usage cleanup.

Clean-up the layering in current code by eliminating direct
usage of PTLRPC_BRW_MAX_SIZE macro outside of the ptlrpc
module. This should help us acheive "floating" max brw size
value across the cluster, which in turn should help with
4MB IO task.

Signed-off-by: Sergii Glushchenko <sergii_glushchenko@xyratex.com>
Reviewed-by: Alexey Lyashkov <alexey_lyashkov@xyratex.com>
Reviewed-by: Alexander Zarochentsev <Alexander_Zarochentsev@xyratex.com>
Change-Id: Ib9f6265bd28f25c34b27eb6ad1a56a752b71a4ac
Xyratex-bug-id: MRP-687
Reviewed-on: http://review.whamcloud.com/4876
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1866 fid: cleanup object visibility definition and check

1) The normal fid, IGIF fid, "ROOT" fid, ".lustre" fid are
   used for client-mdt visible objects.

2) Remove redundan osd_igif.{c,h}

3) Hide server-side local root for up layer dir readpage.

4) It is unnecessary to append FID-in-dient for dot name entry.
   Because we can get it from the directory object directly.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: If98de0c15c0cd3a0e93b666a8d48366b0d8a2cb0
Reviewed-on: http://review.whamcloud.com/4901
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-1866 misc: fix some issues found during LFSCK

1) Test scripts issues for sanity.sh and sanity-scrub.sh

2) Incorrectly use open flag 'MDS_OPEN_OWNEROVERRIDE" for
   permission check for set_attr.

3) Other code clean, such as mdt_body, md_op_spec, wiretest.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I52fbfe8481e35edba75e8fe7e0ab2fec094eabae
Reviewed-on: http://review.whamcloud.com/5046
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>

LU-1187 mdt: Only create remote dir on MDT0

1.Add checking for only creating remote dir on MDT0.
2.Add enable_remote_dir parameter to enalbe remote dir creation
on other MDTs.

Change-Id: I272d4763f07115c5aa0432eb4bbdda5349711d25
Signed-off-by: Wang Di <di.wang@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/4336
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-1187 lod: prepare default stripe EA for remote directory

During remote directory create, it should prepare the default
stripe EA in declare, so the default EA can be set correctly
on the remote directory.

Change-Id: Ief93e72cc033f3553865f2abba53e81df0f3fa89
Signed-off-by: Wang Di <di.wang@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/4335
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-1187 mdt: Add pre_cleanup phase in MDT stack cleanup

Add pre_cleanup phase in MDT stack cleanup, so MD OSP can be
disconnected before cleanup MDT stack.

Signed-off-by: wang di <di.wang@whamcloud.com>
Change-Id: Id1a56c393c22b4fc526c81c05b11bfb656a80559
Reviewed-on: http://review.whamcloud.com/4929
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-1187 mdt: add out handler for object update

Add object update handler on remote MDT. Because the RPC
only includes object updates, (no metadata operation), the
out_handler will skip MDD layer and call osd layer directly.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I9d553145cedfffcbb26d839ed8c7d9cc0887ed86
Reviewed-on: http://review.whamcloud.com/4928
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-1187 osp: add osp_md_object for remote directory.

In declare phase, the update for remote object will be added
in the remote list of the transaction, which will then being
sent to the remote MDT during trans_start.

Add osp_md_object for remote directory, any updates for
remote operation will be packed into RPC by api in
the osp_md_object

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I5fac24e21e6663dca0e2d2fa5d90d4a07a002136
Reviewed-on: http://review.whamcloud.com/4927
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-2466 lnet: Use Hash Table for Remote Route List

This change updates the remote route list (the_lnet.ln_remote_nets)
to be a hash table (the_lnet.ln_remote_nets_hash) to speed up
access when dealing with a large number of routes (over 1000).

Signed-off-by: Doug Oucharek <doug.s.oucharek@intel.com>
Change-Id: I2dac8075ff38d29bd38905b5b1b002603c5a56b6
Reviewed-on: http://review.whamcloud.com/5023
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Tested-by: Hudson
Reviewed-by: John Hammond <john.hammond@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-2700 mdt: serializing the rename

BFL is needed in the rename process even though the source and
target are under the same directory, because we still need
serialize the locking of these two objects.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I61593e3902f1f81ac282c020d912fdf28ce2cc2c
Reviewed-on: http://review.whamcloud.com/5203
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Tested-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>

LU-1876 hsm: Avoid deadlock on i_mutex and layout change

The deadlock is as follows:
Proc 1 (write): start an IO -> try to acquire i_mutex in
            generic_file_aio_write()
Proc 2 (setattr): Acquire i_mutex in ll_setattr_raw -> initialise IO
            -> ll_layout_refresh() -> wait for IO above.

It's actually not necessary to acquire i_mutex in ll_setattr_raw()
until setattr really starts.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I1988d67eab8bcac2ee63b9ec0ecab06d0b7cfc66
Reviewed-on: http://review.whamcloud.com/5159
Tested-by: Hudson
Reviewed-by: jacques-Charles Lafoucriere <jacques-charles.lafoucriere@cea.fr>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>

LU-1876 hsm: bugfix for deadlock in glimpse and layout change

This dead lock is as follows: glimpse calls ccc_inode_lsm_get()
inside glimpse IO, it will be blocked at grabbing lo_type_guard
sem, because another thread is changing layout which holds
lo_type_guard but waiting for all active IO to complete.

The basic rule: any attempt to grab lo_type_guard inside an IO is
generically risky.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: Ib41c03f0a73e2ef7cd37e7689c2d55f7fd0484bf
Reviewed-on: http://review.whamcloud.com/5158
Reviewed-by: jacques-Charles Lafoucriere <jacques-charles.lafoucriere@cea.fr>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>

LU-1282 lprocfs: reduce lprocfs stats memory use

* Move percpu counter common data out, do not need to store them
  redundantly in percpu counter.
* LPROCFS_STATS_FLAG_IRQ_SAFE flag implies three things:
  1. lprocfs_counter needs lc_sum_irq.
  2. when a stats is a percpu counter stat, all its percpu counter
     gets allocated along with the stats itself.
  3. when a stats is a non-percpu stat, lprocfs_stats_lock() needs
     disable irq.
* change lprocfs_counter to make non-irq-safe stats counter do not
  use lc_sum_irq counter, which can save memory.
* Right now, only obd_memory stats use LPROCFS_STATS_FLAG_IRQ_SAFE
  flag.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I2eb037f9dcda983844857fc068c428c4fa387e7a
Reviewed-on: http://review.whamcloud.com/3246
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2221 ptlrpc: kerberos support for kernel>=2.6.24

Since kernel 2.6.24 the scatterlist struct has no field "page".
Then related Lustre kerberos code cannot work anymore.

So do not access scatterlist::page directly, instead, use the
scatterlist functions sg_set_page and sg_assign_page.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Signed-off-by: Thomas Stibor <thomas@stibor.net>
Change-Id: I446925bb42c1e018a55a69948383c8f71976f1fa
Reviewed-on: http://review.whamcloud.com/4394
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>