git://git.whamcloud.com - fs/lustre-release.git/log

LU-2634 ldiskfs: clear extent flag for fast symlink

Fast symlinks on MDT filesystems formatted with the "extents" feature
are created with EXT4_EXTENTS_FL. e2fsck considers this a corruption
and deletes the symlink.

The patch is to clear LDISKFS_INODE_EXTENTS flag in
osd_ldiskfs_writelink() just as ext4_symlink() does.

Signed-off-by: Liu Ying <emoly.liu@intel.com>
Change-Id: Ib7c1eee7becfaf2b80805d3f6c13658686ad5699
Reviewed-on: http://review.whamcloud.com/5154
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>

LU-582 test: fix bug in get_ost_param in sanity.sh

in get_ost_param(), the ouput should be gl_sum, which is
the sum of 'gl' in all OSS.

Signed-off-by: Hongchao Zhang <hongchao.zhang@whamcloud.com>
Change-Id: I51d3aa591c3ca875f8a559974cf05d250b0c032a
Reviewed-on: http://review.whamcloud.com/3693
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Jian Yu <jian.yu@intel.com>
Tested-by: Hudson
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2775 osp: enable fid-on-OST only for DNE.

1. MDT0 will only create IDIF fids, so old clients, who
only know IDIF, can still access files on MDT0.

2. Using fid_ostid_pack to pack the FID into ostid, otherwise
IDIF will be packed wrongly and caused the panic on the OST0.

3. Add dd into compatible tests of conf-sanity.sh(32) to
check OST fids on the upgrade FS.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: Ic9322bd1c66e5fedec68c10aa2f300d5ecb3b6db
Reviewed-on: http://review.whamcloud.com/5307
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>

LU-2625 scripts: prevent several errors in lustre_rmmod

This fix prevents lustre_rmmod scripts from returning
errors when lustre modules are partially or not loaded.

Signed-off-by: Gregoire Pichon <gregoire.pichon@bull.net>
Change-Id: Iefb4166b9b14315319858031f53ecb148c589474
Reviewed-on: http://review.whamcloud.com/5040
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2608 tests: check_catastrophe success on failure to connect

Allow check_catastrophe to pass on failure to connect to host.
Otherwise LBUG assertions crop up when hosts are overloaded.

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Ie873b817911613369a226341d8efe06707b08bc4
Reviewed-on: http://review.whamcloud.com/5200
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2785 mdd: do not use mti_fid in mdd_find_or_create_root()

as in the subsequent callchain it can be used by
mdd_create_sanity_check()

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Change-Id: Ia357f587192c54e3c406a35054dca47580077056
Reviewed-on: http://review.whamcloud.com/5311
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>

LU-2494 tests: add interop support to framework check

Modify the test-framework wait_mds_ost_sync() function to work
with mds_sync when old_sync_processed doesn't exist.
This should cover both old and new servers.

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I2e5c430a96ac40a1ca87990696e7daaad5a69102
Reviewed-on: http://review.whamcloud.com/5021
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2734 tests: Remove XXXDEV overrides in init_facets_vars()

init_facets_vars() assigns outputs of xxxdevname() calls to XXXDEVs.
This is incorrect because XXXDEVs are supposed to contain device names
while xxxdevname() calls may print dataset names for ZFS-based facets.
Actually, XXXDEV should not be changed at all. This patch removes the
XXXDEV assignments.

Change-Id: Id484390cbd814c6fa919c02e390700f914561434
Signed-off-by: Li Wei <wei.g.li@intel.com>
Reviewed-on: http://review.whamcloud.com/5243
Tested-by: Hudson
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

New tag 2.3.61

Change-Id: If171f34ebcec4d651ca091af4197c71d017ee645

LU-1940 test: use default resend count for 118c

in sanity, the 118c could fail with the small resend count
for SLOW=no case, using the default value instead.

Signed-off-by: Hongchao Zhang <hongchao.zhang@intel.com>
Change-Id: I3e5ecc6af2eba560f82a80eb75223f8ab299f397
Reviewed-on: http://review.whamcloud.com/4694
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1812 ldiskfs: don't use sb->s_qf_inums

ext4-quota-first-class.patch was finally accepted into the mainline
Linux kernel via commit 7c319d32 (v3.6). The final version doesn't
include the cached inode numbers so they aren't available on newer
kernels that include this patch (or that are using an ldiskfs
patched using the usptream version).

Since the file system will refuse mounting when the inodes can't
be read, the lookup code isn't invoked without valid inode numbers
anyway.

Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Change-Id: Ie1b824183cf734a0f242bb6483a192f1919dc4be
Reviewed-on: http://review.whamcloud.com/5264
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>

LU-2675 cleanup: remove User Mode Linux specific code

The UML specific code has been uncompilable for some time and no one
has complained so it goes.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ie91c6e4484fee64fec4a2e38effd8a1021e8cddb
Reviewed-on: http://review.whamcloud.com/5175
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>

LU-2673 procfs: call lprocfs_free_xxx_stats() later

Defer calls to lprocfs_free_{obd,md}_stats() and
lprocfs_job_stats_fini() until after the corresponding proc entries
have been removed. Change the return of some proc fini()s to void.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ic410a4cf12eeb084270f5e679222c54b12d80266
Reviewed-on: http://review.whamcloud.com/5160
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2642 quota: update quota APIs

Commit 871a2931 renamed vfs_dq_init to dquot_initialize. (v2.6.34)
Commit 9f754758 renamed vfs_dq_drop to dquot_drop. (v2.6.34)
Commit b43fa828 renamed vfs_dq_transfer to dquot_transfer. (v2.6.34)
Commit 0f0dd62f killed vfs_dq_off and vfs_dq_quota_on_remount and
replaced them with dquot_suspend and dquot_resume. (v2.6.34)

Signed-off-by: James Simmons <uja.ornl@gmail.com>
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Change-Id: I61e5a4d6c4a519457146b4a65ce78bfddebe2899
Reviewed-on: http://review.whamcloud.com/5119
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>

LU-2642 osd-ldiskfs: use struct fs_disk_quota for quotas

Commit b9b2dd36 modified ->get_dqblk to use struct fs_disk_quota.
(v2.6.34)

Signed-off-by: James Simmons <uja.ornl@gmail.com>
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Change-Id: Ifb8a79db37e3f88046c00eb906b8e7264005481b
Reviewed-on: http://review.whamcloud.com/5118
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>

LU-1199 ldiskfs: Remove HAVE_{EXT4,JBD2}_JOURNAL_CALLBACK_{SET,ADD}

Lustre servers now only support kernels/ldiskfs with
{ext4,ldiskfs}_journal_callback_add(). Additionally, the
HAVE_EXT4_JOURNAL_CALLBACK_ADD define is not in the upstream
3.3 kernel where the ext4_journal_callback_add() patch landed.

Therefore we remove the HAVE_{LDISKFS,EXT4}_JOURNAL_CALLBACK_{SET,ADD}
checks and just assume that it is set.

Change-Id: I10cc79fcf07dd05a10ae83014f188c613ab1620c
Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>
Reviewed-on: http://review.whamcloud.com/5055
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-1431 ptlrpc: Support for over 1MB bulk I/O RPC

Increase the maximum size of BRW RPCs between OSCs and OSTs to 4MB
by increasing the number of LNet Memory Descriptors assosiated with
a bulk.  Each bulk transfer is still limited to LNET_MTU_SIZE, so
that it can be passed through routers and does not break RDMA limits.

The client and server negotiate the maximum BRW RPC size at connect
time via ocd_brw_size, with the client sending the maximum size it
supports, and the server returning min(client_max, server_max) back
to the client.

For each RPC, the number of bulk MDs that are registered depends on
the actual RPC size at sending time.  The (max_brw_count - 1) is sent
to the OST in the high 16 bits of obd_ioobj.ioo_max_brw and forms a
mask for the bulk RPC XIDs.  The actual number of bulk transfers
is encoded in the BRW RPC XID.  The masked value of the request XID
determines the starting bulk transfer match bits, and the last bulk
match is the RPC XID.

For older clients this means that the starting and ending bulk match
bits are the RPC XID itself (as it always was) and the old ioo_type
(now ioo_max_brw) field being set to zero would result in a mask of
"1", which just keeps the same RPC XID as it was before.

It is important to note that the client and server do NOT share the
value of PTLRPC_BULK_OPS_MASK, or this would incorrectly mask the
RPC XID if the PTLRPC_BULK_OPS_BITS was ever increased on the server.

The actual BRW RPC size can be controlled for each client separately
by changing the client's max_pages_per_rpc.  For example, this will
set it to 4MB:

        lctl set_param osc.*.max_pages_per_rpc=1024

It is also possible to specify a units suffix, so that the RPC size
can be set independently of the client PAGE_SIZE:

        lctl set_param osc.*.max_pages_per_rpc=4M

By default, the size is 1MB for for OST BRW RPCs until more testing
is done. All other bulk I/O (e.g. MDC<->MDS) is left at 1MB.

With PTLRPC_MAX_BRW_SIZE growing larger, it doesn't make sense for
the readahead to increase the window size so dramatically at one time
Instead, limit readahead growth by the current inode i_blkbits value,
which currently defaults to 4MB.

If we want to tune the readahead growth so that it matches the actual
cl_max_pages_per_rpc value, then this can be done much more easily by
changing the per-inode i_blkbits value, since this is tied to specific
OSTs with specific RPC settings (which may be different for e.g. local
and remote OSTs), and only needs to be checked once.

Change the ras_*() functions to take the inode as an argument, and
always put it first in the argument list for consistency.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Signed-off-by: Sergii Glushchenko <sergii_glushchenko@xyratex.com>
Reviewed-by: Alexey Lyashkov <alexey_lyashkov@xyratex.com>
Reviewed-by: Andrew Perepechko <andrew_perepechko@xyratex.com>
Change-Id: I757b14a04e5d4cc053576e41e47864c743c35b8b
Xyratex-bug-id: MRP-319
Reviewed-on: http://review.whamcloud.com/4993
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>

LU-2710 build: avoid function resolution mistakes by Coverity

Remove function resolution mistakes made by Coverity version
6.0.3.
The .hs_keycmp function is incorrectly resolved by Coverity,
making the variable passed as its first 'key' parameter to
be considered of the wrong type. Data structures being of
different sizes, it ends up with this Coverity defect:
Out-of-bounds access (OVERRUN_STATIC)
Overrunning static array by passing it as an argument to a
function which indexes it at a higher byte position.

Signed-off-by: Sebastien Buisson <sebastien.buisson@bull.net>
Change-Id: Ie062b05a73db16d63c63db89dc4d0e264f3b1f8a
Reviewed-on: http://review.whamcloud.com/5211
Tested-by: Hudson
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-642 lov: make up obd_connect for inactive OSC

When OSC is inactivated before lov tries to connect it, lov_connect()
miss the chance to connect it to OST devices even when it is
activated later.

We need make up the connection for the initially inactive OSC when it
is activated.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I4aca442bc4a77baa2bc14adaf2b0f8f5c7bfa7df
Reviewed-on: http://review.whamcloud.com/4691
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2444 build: fix 'error handling' issues

Fix 'error handling issues' defects found by Coverity version
6.0.3:
Unchecked return value (CHECKED_RETURN)
Value returned from a function is not checked for errors before
being used.
Argument cannot be negative (NEGATIVE_RETURNS)
Negative value used as argument to a function expecting a
positive value.

Signed-off-by: Sebastien Buisson <sebastien.buisson@bull.net>
Change-Id: I44022a7d7227ee46345b77701b4c5199a74c386d
Reviewed-on: http://review.whamcloud.com/4772
Tested-by: Hudson
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2302 scripts: make lfs_migrate compatible with 1.8

Earlier versions of lfs setstripe/getstripe use option -s instead
of -S. Make sure lfs_migrate can be used with any version.

Signed-off-by: Daniel Kobras <d.kobras@science-computing.de>
Signed-off-by: Keith Mannthey <keith.mannthey@intel.com>
Change-Id: Ic3caaeaf3d6c764afbd75cd7a8a7279462d21d47
Reviewed-on: http://review.whamcloud.com/4702
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2590 obdclass: correct swab lov_mds_md

For caller's convenience, the 2nd parameter of obd_free_diskmd()
could be host endianness, it needs swab to LE if necessary, while just
lov_mds_md header needs it for figuring out how much memory needs to
be freed.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I537e83b56b4be6e08a7431bfda1de71efefa33ee
Reviewed-on: http://review.whamcloud.com/5145
Tested-by: Hudson
Reviewed-by: Prakash Surya <surya1@llnl.gov>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2158 fsfilt: Remove unused opcodes

This change removes the unused fsfilt opcode definitions and the
accompanying code in fsfilt_ext3_start. This is strictly cleanup.

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Change-Id: Ifccb6e7c9c728bc98e4ddfa52093e660dd117b8b
Reviewed-on: http://review.whamcloud.com/5192
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>

LU-2158 fsfilt: Remove lvfs_common.c

This change removes the lvfs_common.c file, and moves it's only
remaining functionality directly into the obd_lfs_fid2dentry function.
This should have no functional impact, and is merely a code cleanup
exercise.

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Change-Id: I5f8588a17f6e7659a7d33c2a5f79792826976de4
Reviewed-on: http://review.whamcloud.com/5193
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2360 ptlrpc: don't clobber rq_history_seq with CPT_ANY

In ptlrpc_req_add_history() don't allow service_parts with cpt CPT_ANY
(-1) to clobber the timestamp and sequence bits of rq_history_seq.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I52161d6f36dcd70dd569b0be294e6d23cc5ac4a9
Reviewed-on: http://review.whamcloud.com/5207
Tested-by: Hudson
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2764 tests: Remove obsolete comment in local.sh

The comment above LOAD_MODULES_REMOTE in local.sh was written for the
special case of the 2.3 OSS stack and is no longer applicable to
current master.

Signed-off-by: Li Wei <wei.g.li@intel.com>
Change-Id: I75518e9cebfa717ccfb625e2786c8d191c6bddbc
Reviewed-on: http://review.whamcloud.com/5277
Tested-by: Hudson
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2749 ofd: check return value of ofd_seq_load

Return value ofd_seq_load was not always checked for both NULL and
errno. This caused a NULL pointer dereference.

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Ib2e1658f16ba2012c9410966c08201cab7a5aa5c
Reviewed-on: http://review.whamcloud.com/5263
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>

LU-2414 tests: cleanup replay-single/0c and 44c

Switch from return codes to error messages in test_0c and test_44c.

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I61895442fa52e2f7bff421e0b8309f0f11c97de8
Reviewed-on: http://review.whamcloud.com/4856
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Hudson
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-1434 utils: extend wireshark parsing

Add support for MGS_CONFIG_READ
Add capabilities where they're defined in layout.c
Fix processing of MGS_BULK_PORTAL (not a Lustre packet)
Make use of LUSTRE_BUFFER_LEN everywhere.
Add dissecting of capabilities field
Add dissecting of fld field
Add dissecting of MGS_CONFIG_BODY
Fixed all compiler warnings.
Fixed parsing of get/set (x)attrs

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Ic75bb4814570b30a1f1dca19d2f49311294ff2fe
Reviewed-on: http://review.whamcloud.com/5155
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2689 interop: use stripe md in ll_update_inode()

When connected to servers that do not support LL_SBI_LAYOUT_LOCK, if
stripe md is available in ll_inode_update() then use it.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I949a0129c3fc92c5434288df0549a54ccd440f97
Reviewed-on: http://review.whamcloud.com/5289
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2562 test: cleanup previous test result

in sanity.sh, test_51ba only depends on test_51b, but test_51a
also create files under $DIR/$tdir, which causes test_51ba fail
for it also takes those files into account.

Change-Id: I42f491f64c8d2d65bee656c167c9523274cd7200
Signed-off-by: Hongchao Zhang <hongchao.zhang@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/5183
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2449 osd: osd-zfs to handle ./.. properly

ZFS does not store directory records . and .. on a disk.
Instead they are generated upon request (like in zfs_readdir()).
with the patch osd-zfs does the same for namespace directories.

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Change-Id: I910f9bad807c0e5a2bee1fd481470576ceaf4444
Reviewed-on: http://review.whamcloud.com/4990
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>

LU-2240 mds: Assign special fid sequence to root.

In current implementation, we still use IGIF as the ROOT FID,
but ZFS does not have IGIF at all, so it will move root ROOT fid
to a new sequence.

But for compatible with clients, which are being mounted
during upgrade, it will still keep the old IGIF root FID.

Signed-off-by: Wang Di <di.wang@intel.com>
Change-Id: I2e6b81cd359436d2e5b96c8a315a56c415f8e631
Reviewed-on: http://review.whamcloud.com/5257
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>

LU-1866 lfsck: LFSCK for namespace consistency (3)

It is the third part of LFSCK 1.5 for namespace consistency
check and repair. It mainly implements the LFSCK namespace
component functionality lfsck_operations::lfsck_double_scan:
process multiple-linked objects.

1) Find out and repair crashed linkEA entry.

2) Find out and remove invalid linkEA entry

Test-Parameters: envdefinitions=ENABLE_QUOTA=yes testlist=sanity-lfsck

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Ia38fa343ed4da10635c1af4d90b9039c5e01688c
Reviewed-on: http://review.whamcloud.com/4914
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2726 procfs: Invalid assertion for counter_{add,sub}

lprocfs_stats::lc_sum_irq is only used for obd_memory for accurate
memory usage counting, other procfs stats counters do not need it,
even though they can be used in RCU callbacks.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I4c2dec50eecc0c63631f1915ba30a54a5da2fec1
Reviewed-on: http://review.whamcloud.com/5237
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: John Hammond <johnlockwoodhammond@gmail.com>
Tested-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2017 mdc: add layout swap between 2 objects

This patch adds the client and MDT code to swap layouts
between 2 files:
- a lfs command
- a llapi call, based on a new ioctl
- the mdc/mdt codes for the ioctl (with layout locking)

Signed-off-by: JC Lafoucriere <jacques-charles.lafoucriere@cea.fr>
Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I7e710a1ab3ca38e8b26582e49d08e9943aa445cd
Reviewed-on: http://review.whamcloud.com/4507
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1866 lfsck: LFSCK for namespace consistency (2)

It is the second part of LFSCK 1.5 for namespace consistency
check and repair. It mainly implements the LFSCK namespace
component functionality lfsck_operations::lfsck_exec_dir:
for linkEA verification and repair.

The basic functionality is to find out missed linkEA entries
and add them to the object. It is also part of handling IGIF
objects upgrading.

Test-Parameters: envdefinitions=ENABLE_QUOTA=yes testlist=sanity-lfsck

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I9df2a1fddfa4b15838ecbdfb6279c240d01d0f95
Reviewed-on: http://review.whamcloud.com/4913
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-1866 lfsck: FID-in-{dirent,LMA} check and repair

As part of LFSCK 1.5, it implementes the following functionalities:

1) Verify and rebuild FID-in-dirent, mainly for MDT file-level
   backup/restore.

2) (Partly) verify and rebuild FID-in-LMA: if the FID-in-dirent
   is valid but FID-in-LMA is missed, then the later one will be
   repaired; but if both are missed, then it is OI scrub's duty
   to repair the inconsistency.

3) (Partly) handle IGIF objects upgrading: build IGIF-in-dirent
   for IGIF objects. The other two parts work for IGIF objects
   upgrading are:

3.1) Generate IGIF-in-LMA, which has been done by OI scrub in
     former patch(es).

3.2) Generate linkEA, which will be done by up layer LFSCK in
     other patch(es).

Test-Parameters: envdefinitions=ENABLE_QUOTA=yes testlist=sanity-lfsck

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Ibb317f0f3f3e9cf6db56bfa25f55d2017b0e1937
Reviewed-on: http://review.whamcloud.com/4912
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1866 lfsck: LFSCK 1.5 user space control

Include the user space command to start/stop the LFSCK for
namespace consistency check and repair. And proc interface
to query the LFSCK status, statistical informtion, and ect.

Test-Parameters: envdefinitions=ENABLE_QUOTA=yes testlist=sanity-lfsck

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Icf4046b313dc1653b5b8b1a3add7fa0329ca7ad0
Reviewed-on: http://review.whamcloud.com/4911
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1866 lfsck: LFSCK for namespace consistency (1)

It is the first part of LFSCK 1.5 for namespace consistency
check and repair. It mainly implements the LFSCK namespace
component internal framework and external APIs.

The functionality parts for the LFSCK namespace component,
such as lfsck_operations::lfsck_{exec_dir,dump,double_scan},
are not included. They will be implemented in other patches.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I011e06c991577e632eadf364b87be74f1f5535c6
Reviewed-on: http://review.whamcloud.com/4910
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1866 lfsck: LFSCK main engine

There are two engines for LFSCK 1.5:

1) One is low layer object table based iteration, which can
   guarantee that all the objects can be scanned at least once.

2) The other is namespace based directory traversal. All the
   namespace based components, such as FID-in-dirent and linkEA,
   can be handled during that.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Ic5a404cde35fdb20c24f6522faa8b21117cf4254
Reviewed-on: http://review.whamcloud.com/4909
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1866 lfsck: general framework for LFSCK 1.5

LFSCK 1.5 main data structure definations, LFSCK
component APIs, setup/cleanup/start/stop interfaces.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I4507b9432289ca8d54cbf686893b706052d0eeb3
Reviewed-on: http://review.whamcloud.com/4908
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-1866 lfsck: ancillary work for namespace LFSCK

1) Share related linkEA functions with LFSCK.

2) Multiple linked objects are rare, and the objects with
   a lot of linkea entries are more rare. It is unnecessary
   to worry about performance under such cases. So drop the
   linkea entries count limitation: LINKEA_MAX_COUNT.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I75f90c9619c56f2f62063435aea26ed01337d6dd
Reviewed-on: http://review.whamcloud.com/4907
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2733 osd: osd-zfs to initialize existing sequence

osd_find_or_add_seq() to use osd_oi_find_or_create() so that
existing sequence is loaded if exists

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Change-Id: I9f173615ef047dd4696d8ed51687c7c268b32dbf
Reviewed-on: http://review.whamcloud.com/5247
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2728 fld: check for NULL param in fldb_seq_stop()

Check for NULL param in fldb_seq_stop().

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I44ed10f9ce8bceabb203e6ef0b843b23340ced2b
Reviewed-on: http://review.whamcloud.com/5238
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2737 osc: double unlock in osc_lru_shrink()

It could unlock twice in osc_lru_shrink() if the cl_io_init() failed.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: Idf3ebafced7f7d3a68c13ea20d12cf221e9d74ce
Reviewed-on: http://review.whamcloud.com/5258
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-1187 mdt: Add MDS_INODELOCK_PERM lock for remote dir

Add MDS_INODELOCK_PERM for remote directory on remote MDT to
protect permission changes on remote MDT. So, Master MDT, where
the parent is, will grant LOOKUP lock to the client. Remote MDT,
where the child is, will grant UPDATE|PERM lock to the client.

And client will clear the dcache during, when either PERM or LOOKUP
lock is being revoked.

Change-Id: Ie11d14bbe7c706a76f7c4bd8f75b07807e2dac02
Signed-off-by: Wang Di <di.wang@intel.com>
Reviewed-on: http://review.whamcloud.com/4346
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2722 clio: directIO thread races with completion thread

ll_direct_IO_26()) ASSERTION( obj->cob_transient_pages == 0 ) failed

The directio thread puts pages to the transfer in osc_io_submit(), and
waits for its completion. Upon completion, osc_completion() is called,
which notifies waiters IO is completed and only after that put the
page. the original directIO thread wakes up and may succeed to assert
on the last page which is not yet put by osc_completion().

Signed-off-by: Vitaly Fertman <vitaly_fertman@xyratex.com>
Change-Id: Ib912e913885a9ff2cb1e9c865ae2003a676c2b5b
Xyratex-bug-id: MRP-838
Reviewed-on: http://review.whamcloud.com/5223
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2467 ptlrpc: Allow OBD_PINGs to be suppressed

This patch introduces a new ptlrpc module parameter, "suppress_pings",
to provide an option for reducing excessive OBD_PINGs in large
clusters.  The parameter affects all MDTs and OSTs on a node.  It is
off (zero) by default, giving a behavior identical to current
implementation.  If it is on (non-zero), all clients of the affected
targets who understand OBD_CONNECT_PINGLESS will know, at connect
time, that pings are not required.  When suppressing pings, there must
be an external mechanism to notify the targets of client deaths, via
the targets' "evict_client" procfs entries.  In addition, a highly
available standalone MGS is also recommended when suppressing pings,
so that clients are notified (through Imperative Recovery) of target
recoveries.

The changes do two basically independent things.  One is initializing
import and export states (i.e., imp_connect_data and
exp_obd_chain_timed) according to "suppress_pings", since whether to
ping or not is a property of each import-export pair.  MGC pings can
not be suppressed, because maintaining MGS connections is dictated by
the reliance on Imperative Recovery.

The other thing is changing pinger and import routines to respect the
import property set earlier.  (The export side does not need any
change at all.)  Pings are still needed to query last committed
transactions if there are uncommitted requests on an import, so that
resources pinned for replays can be released even when applications
become idle.  An early version of this patch removes imports that do
not need to be pinged from pinger_imports and add them back when last
committed transactions are needed or recoveries must be initiated.
This version does not do that because a) the overheads of iterating
through 10,000 imports are not prohibitively large---around 10 ms, b)
adding imports back to pinger_imports requires the global pinger_mutex
to be held, and c) the imp_lock contention added on each import is
small.

Change-Id: Iabc84d395c978c3f156c52aebfad83621facb4fe
Signed-off-by: Li Wei <wei.g.li@intel.com>
Reviewed-on: http://review.whamcloud.com/5009
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Tested-by: Hudson
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2467 protocol: Add OBD_CONNECT_PINGLESS

Reserve a bit for OBD_CONNECT_PINGLESS, which indicates a client is
capable of suppressing keep-alive OBD_PINGs. If granted by a server,
it means the server does not require (but still allows) pings.

Signed-off-by: Li Wei <wei.g.li@intel.com>
Change-Id: I7575082244a84f2e633a1936fcf894da3c6ee7dd
Reviewed-on: http://review.whamcloud.com/5229
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2723 clio: a client hangs on osc_extent_wait in dio

If dio overlaps with mapped region, first overlapped pages are sent
by generic_file_direct_write->filemap_write_and_wait_range->writepages
After that the dio pages are sent by write_begin/write_end, writepages
However, write_end->osc_page_cache_add does not release extent because
the IO is not considered as synchronous. as the result, the following
writepages->osc_cache_writeback_range() does not send pages as the
extent is still ACTIVE and osc_cache_wait_range() hangs forever.

The patch make directIO to be considered as sync by lustre so that
extent was released at write_end.

Signed-off-by: Vitaly Fertman <vitaly_fertman@xyratex.com>
Xyratex-bug-id: MRP-839
Change-Id: I0c0e1599ee180f467826e4b6715c3cbfd0c6eb5c
Reviewed-on: http://review.whamcloud.com/5224
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2194 tests: add debugging to recovery-small/19

Add error messages to 19a and 19b

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Ief35c063734654c5a6f3dab0d283c4b7a32706e8
Reviewed-on: http://review.whamcloud.com/5141
Reviewed-by: Li Wei <wei.g.li@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-1187 mdt: directory remote open fix

Directory remote open continue as normal open, i.e. return the
remote fid to client, and client will send the open request to
the right MDT.

Set correct it disposition in mdt_open_by_fid.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I32d5dee9cd1eac47787e2d85a5db1e164d2846d3
Reviewed-on: http://review.whamcloud.com/4934
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>

LU-1187 mdd: a few missing stuff in MD stack for DNE.

1. Assign the index operation for the directory create
in declare phase, which is needed for creating the
object in OUT.

2. Declare dotdot insertion for remote directory creation,
so the insert update can be packed into RPC and send to
the remote MDT (out) to be executed.

3. Add hint to the object declare phase, by which OSP can
get some info during update RPC packing, like parent fid.

4. Separate lu_object_exists into lu_object_remote and
lu_object_exists, so it can check whether remote object
exists as well.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: Icb359443d9982ee8567af933c5def42dc51a3a7a
Reviewed-on: http://review.whamcloud.com/4930
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>

LU-2654 ldlm: ldlm_resource_dump should be called with resource locked

Fix ldlm_resource_complain to lock resource before calling
ldlm_resource_dump

Change-Id: I9dfbe84bd63de3093c99e7379daeca30d02fc875
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/5254
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Tested-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1187 mdt: add sanity check for rename and link

Add sanity check for rename/link, so remote rename/link
will return EXDEV.

Signed-off-by: wang di <di.wang@whamcloud.com>
Change-Id: I841c8ffaa74b162fcca5cd1661c383215350cccc
Reviewed-on: http://review.whamcloud.com/4348
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1187 mdt: unlink remote directory

Send unlink req to the slave MDT, so both open/close
and unlink will send request to the slave MDT, where
the remote object resides. Then it would be able to
check orphan (unlink open file) locally.

Add lock_object api for enqueue remote object.

Change-Id: I7483b0f023e4e3de6597da58d3d9f3e96c0d53b7
Signed-off-by: Wang Di <di.wang@intel.com>
Reviewed-on: http://review.whamcloud.com/4339
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-1187 utils: add lfs setdirstripe/getdirstripe

1.Add lfs setdirstripe/mkdir/getdirstripe to create
remote directory.

2.Add lfs rm_entry to just remove the entry of remote
directory.

Change-Id: Idaecfbb9dd97e3ae9fbc70e5b902a91a7b2ed18e
Signed-off-by: Wang Di <di.wang@intel.com>
Reviewed-on: http://review.whamcloud.com/4341
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>

LU-1866 lfsck: enhance otable-based iteration

1) Use ::load() method to unplug OSD layer iteration.

2) Use ::put() method to wakeup the LFSCK main engine
if it is blocked at low layer for ::next() call.

Test-Parameters: envdefinitions=ENABLE_QUOTA=yes testlist=sanity-scrub
Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I28e5689ad3a4ee96a3f17a64e8d6dca553a86d77
Reviewed-on: http://review.whamcloud.com/4906
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Tested-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>

LU-1866 osd: FID-in-LMA and OI files

Generate FID-in-LMA not only for normal objects, but also for
IGIF objects, server-side local files, and other non-OST/LLOG
objects.

LAST_ID has FID-in-LMA but no related mappings in the OI file.

At the same time, OI mapping usage is also extended to above
objects. Only the objects marked as "LDISKFS_STATE_LUSTRE_NO_OI"
or OST objects have no OI mappings the OI files.

It is part of 1.8 IGIF objects upgrage handling. OI scrub will
automatically generate IGIF-in-LMA and add related OI mappings
in the OI files.

Test-Parameters: envdefinitions=ENABLE_QUOTA=yes testlist=sanity-scrub

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: If1e1434c238f2ab737e35433bfad1abbb00fcb00
Reviewed-on: http://review.whamcloud.com/4904
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>

LU-2733 osd: not return "-ENOENT" for zfs osd_object_init

For zfs backend, the osd_object_init() should not return
"-ENOENT" if osd_fid_lookup() cannot find the object,
which is normal for new created cases.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I76e5d802d0caf8ac3a9992278ec2f7c6c6e2cf07
Reviewed-on: http://review.whamcloud.com/5242
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-1187 osd: add agent/local inode for remote directory

A local inode will be created for remote directory entry, so
to make e2fsck check pass.

The remote directory on the remote MDT will be put under the
agent directory.

Remove LA_TYPE checking during osd_inode_set, because when
initialize the remote object, it needs to get TYPE information
from remote attr get, where we need type information anyway,
(osp_md_attr_get ->out_attr_get->osd_attr_get | LA_TYPE).
So osd_attr_get will always return LA_TYPE valid information
here, then if it do attr_get/modify/attr_set might hit this
ASSERT, So during osd_attr_set, we can just ignore type.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I7f177fa8ae2ab450cde78a8c2da9a2770ec6b0cb
Reviewed-on: http://review.whamcloud.com/4931
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-1866 scrub: initial OI scrub

We will extend OI mapping usage (in OI directories/files) to
all the objects, except for the objects which are marked as
"LDISKFS_STATE_LUSTRE_NO_OI" or OST object.

The OI mappings for the server-side local files may become
invalid because of server-side file-level back/restore, or
server crash. Those files are necessary for server mount,
so related OI mappings must be checked and rebuilt before
they are really used. That is the initial OI scrub work.

Initial OI scrub is used for quickly checking and rebuilding
OI mappings for server-side local files during server mount.

Another change is about MDT_LAST_RECV_OID/OFD_LAST_RECV_OID.
Currently, we can just use the same OID for the "last_rcvd"
files on different targets. So merge the two OIDs as single
OID: LAST_RECV_OID, which also simplifies OI scrub work.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Iddca4812d3c82ca43f1e852fd74475b11b6e6dea
Reviewed-on: http://review.whamcloud.com/4903
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-1866 osd: ancillary work for initial OI scrub

1) Abstract some general interfaces that can be shared by common
   OSD operations, by normal OI scrub, and by initial OI scrub.

2) Mark the object as "LDISKFS_STATE_LUSTRE_NO_OI" if it is not
   unnecessary to add OI mapping for the object, such as backend
   local root, OI directories/files, and others created before OI
   files created.

3) Do not check fld in OI scurb, which may be not ready yet.

4) Reorganize OI scrub code to be more readable.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Ife8168df2454437ad18550b73d26f7372c21776f
Reviewed-on: http://review.whamcloud.com/4902
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2725 ost: ost_rw_hpreq_check should return 0 or 1

ost_rw_hpreq_check was returning number of matched locks which is wrong.

Change-Id: Ic4d3dd4a186b616c13ffd9f897254da3c19f84f8
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/5233
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-1431 ptlrpc: PTLRPC_BRW_MAX_SIZE usage cleanup.

Clean-up the layering in current code by eliminating direct
usage of PTLRPC_BRW_MAX_SIZE macro outside of the ptlrpc
module. This should help us acheive "floating" max brw size
value across the cluster, which in turn should help with
4MB IO task.

Signed-off-by: Sergii Glushchenko <sergii_glushchenko@xyratex.com>
Reviewed-by: Alexey Lyashkov <alexey_lyashkov@xyratex.com>
Reviewed-by: Alexander Zarochentsev <Alexander_Zarochentsev@xyratex.com>
Change-Id: Ib9f6265bd28f25c34b27eb6ad1a56a752b71a4ac
Xyratex-bug-id: MRP-687
Reviewed-on: http://review.whamcloud.com/4876
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1866 fid: cleanup object visibility definition and check

1) The normal fid, IGIF fid, "ROOT" fid, ".lustre" fid are
used for client-mdt visible objects.

2) Remove redundan osd_igif.{c,h}

3) Hide server-side local root for up layer dir readpage.

4) It is unnecessary to append FID-in-dient for dot name entry.
Because we can get it from the directory object directly.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: If98de0c15c0cd3a0e93b666a8d48366b0d8a2cb0
Reviewed-on: http://review.whamcloud.com/4901
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-1866 misc: fix some issues found during LFSCK

1) Test scripts issues for sanity.sh and sanity-scrub.sh

2) Incorrectly use open flag 'MDS_OPEN_OWNEROVERRIDE" for
permission check for set_attr.

3) Other code clean, such as mdt_body, md_op_spec, wiretest.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I52fbfe8481e35edba75e8fe7e0ab2fec094eabae
Reviewed-on: http://review.whamcloud.com/5046
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>

LU-1187 mdt: Only create remote dir on MDT0

1.Add checking for only creating remote dir on MDT0.
2.Add enable_remote_dir parameter to enalbe remote dir creation
on other MDTs.

Change-Id: I272d4763f07115c5aa0432eb4bbdda5349711d25
Signed-off-by: Wang Di <di.wang@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/4336
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-1187 lod: prepare default stripe EA for remote directory

During remote directory create, it should prepare the default
stripe EA in declare, so the default EA can be set correctly
on the remote directory.

Change-Id: Ief93e72cc033f3553865f2abba53e81df0f3fa89
Signed-off-by: Wang Di <di.wang@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/4335
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-1187 mdt: Add pre_cleanup phase in MDT stack cleanup

Add pre_cleanup phase in MDT stack cleanup, so MD OSP can be
disconnected before cleanup MDT stack.

Signed-off-by: wang di <di.wang@whamcloud.com>
Change-Id: Id1a56c393c22b4fc526c81c05b11bfb656a80559
Reviewed-on: http://review.whamcloud.com/4929
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-1187 mdt: add out handler for object update

Add object update handler on remote MDT. Because the RPC
only includes object updates, (no metadata operation), the
out_handler will skip MDD layer and call osd layer directly.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I9d553145cedfffcbb26d839ed8c7d9cc0887ed86
Reviewed-on: http://review.whamcloud.com/4928
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-1187 osp: add osp_md_object for remote directory.

In declare phase, the update for remote object will be added
in the remote list of the transaction, which will then being
sent to the remote MDT during trans_start.

Add osp_md_object for remote directory, any updates for
remote operation will be packed into RPC by api in
the osp_md_object

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I5fac24e21e6663dca0e2d2fa5d90d4a07a002136
Reviewed-on: http://review.whamcloud.com/4927
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-2466 lnet: Use Hash Table for Remote Route List

This change updates the remote route list (the_lnet.ln_remote_nets)
to be a hash table (the_lnet.ln_remote_nets_hash) to speed up
access when dealing with a large number of routes (over 1000).

Signed-off-by: Doug Oucharek <doug.s.oucharek@intel.com>
Change-Id: I2dac8075ff38d29bd38905b5b1b002603c5a56b6
Reviewed-on: http://review.whamcloud.com/5023
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Tested-by: Hudson
Reviewed-by: John Hammond <john.hammond@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-2700 mdt: serializing the rename

BFL is needed in the rename process even though the source and
target are under the same directory, because we still need
serialize the locking of these two objects.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I61593e3902f1f81ac282c020d912fdf28ce2cc2c
Reviewed-on: http://review.whamcloud.com/5203
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Tested-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>

LU-1876 hsm: Avoid deadlock on i_mutex and layout change

The deadlock is as follows:
Proc 1 (write): start an IO -> try to acquire i_mutex in
generic_file_aio_write()
Proc 2 (setattr): Acquire i_mutex in ll_setattr_raw -> initialise IO
-> ll_layout_refresh() -> wait for IO above.

It's actually not necessary to acquire i_mutex in ll_setattr_raw()
until setattr really starts.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I1988d67eab8bcac2ee63b9ec0ecab06d0b7cfc66
Reviewed-on: http://review.whamcloud.com/5159
Tested-by: Hudson
Reviewed-by: jacques-Charles Lafoucriere <jacques-charles.lafoucriere@cea.fr>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>

LU-1876 hsm: bugfix for deadlock in glimpse and layout change

This dead lock is as follows: glimpse calls ccc_inode_lsm_get()
inside glimpse IO, it will be blocked at grabbing lo_type_guard
sem, because another thread is changing layout which holds
lo_type_guard but waiting for all active IO to complete.

The basic rule: any attempt to grab lo_type_guard inside an IO is
generically risky.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: Ib41c03f0a73e2ef7cd37e7689c2d55f7fd0484bf
Reviewed-on: http://review.whamcloud.com/5158
Reviewed-by: jacques-Charles Lafoucriere <jacques-charles.lafoucriere@cea.fr>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>

LU-1282 lprocfs: reduce lprocfs stats memory use

* Move percpu counter common data out, do not need to store them
  redundantly in percpu counter.
* LPROCFS_STATS_FLAG_IRQ_SAFE flag implies three things:
  1. lprocfs_counter needs lc_sum_irq.
  2. when a stats is a percpu counter stat, all its percpu counter
     gets allocated along with the stats itself.
  3. when a stats is a non-percpu stat, lprocfs_stats_lock() needs
     disable irq.
* change lprocfs_counter to make non-irq-safe stats counter do not
  use lc_sum_irq counter, which can save memory.
* Right now, only obd_memory stats use LPROCFS_STATS_FLAG_IRQ_SAFE
  flag.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I2eb037f9dcda983844857fc068c428c4fa387e7a
Reviewed-on: http://review.whamcloud.com/3246
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2221 ptlrpc: kerberos support for kernel>=2.6.24

Since kernel 2.6.24 the scatterlist struct has no field "page".
Then related Lustre kerberos code cannot work anymore.

So do not access scatterlist::page directly, instead, use the
scatterlist functions sg_set_page and sg_assign_page.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Signed-off-by: Thomas Stibor <thomas@stibor.net>
Change-Id: I446925bb42c1e018a55a69948383c8f71976f1fa
Reviewed-on: http://review.whamcloud.com/4394
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>

LU-2592 tests: do not remove $TMP/*active in rpc.sh

The $TMP/*active files record the current active facets info
under failover test configuration. They are removed from
init_test_env() initially. If failover tests use do_rpc_nodes()
which performs rpc.sh and in which init_test_env() is used,
then the active facets info will be lost during the testing.

This patch fixes the above issue by introducing an RPC_MODE
variable which controls that the $TMP/*active files will not
be removed from init_test_env() in rpc.sh.

Test-Parameters: envdefinitions=SLOW=yes,ENABLE_QUOTA=yes clientcount=4 osscount=2 mdscount=2 austeroptions=-R failover=true useiscsi=true testlist=replay-vbr
Signed-off-by: Jian Yu <jian.yu@intel.com>
Change-Id: Iab464449e7c41ffd76256e9ec60b48b326f3ea9e
Reviewed-on: http://review.whamcloud.com/5127
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Reviewed-by: Wei Liu <wei3.liu@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-2693 tests: check Lustre version in conf-sanity test 66

Lustre starts supporting the "replace nids" feature
(commit 6aa30ff) since version 2.3.59. We need add
a Lustre version check into conf-sanity test 66 to
make sure the test does not run on unsupported
Lustre version.

Test-Parameters: envdefinitions=SLOW=yes \
serverjob=lustre-b2_1 serverbuildno=164 \
testlist=conf-sanity

Signed-off-by: Jian Yu <jian.yu@intel.com>
Change-Id: I7eee38664e66059bbc028c3eb7d9741b1351b883
Reviewed-on: http://review.whamcloud.com/5198
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Reviewed-by: Wei Liu <wei3.liu@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-2697 osp: remove trailing newline on thread names

The trailing newline on the thread name causes confusion with some
utilities that examine this string.

Signed-off-by: chas williams - CONTRACTOR <chas@cmf.nrl.navy.mil>
Change-Id: I031aafd2541b1829bc721ebcb9f52cb5e73101b3
Reviewed-on: http://review.whamcloud.com/5189
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-2709 build: fix 'memory corruption' errors

Fix 'memory corruption' defects found by Coverity version 6.5.0:
Out-of-bounds access (OVERRUN_STATIC)
Overrunning static array by passing it to a
function which indexes it at a higher byte position.
Unbounded source buffer (STRING_SIZE)
Passing string of unknown size to a function that expects
a string of a particular size.

Signed-off-by: Sebastien Buisson <sebastien.buisson@bull.net>
Change-Id: Iabc9bd7609d285a3709737748f2100b90769d795
Reviewed-on: http://review.whamcloud.com/5210
Tested-by: Hudson
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-1897 test: replay-single test_70b dbench not found

dbench is exiting before test_70b completes is work.

dbench is being used as a background load in this test.
In one observed case test was taking a little under 90s but
dbench was exiting at 62s. Duration was set at 90.

Presently dbench starts for 60s then the system sleeps
for 12s, then we start a new 60s timer in which we expect
dbench to be running. There is a large window in which
this test will fail.

The main issue is the use of Duration for use as both the
timer of the real test and the timer of the background load.

I raised the real Duration and started the timer for the real
test before starting dbench such that the real test should
finish before dbench does or about the same time.

Signed-off-by: Keith Mannthey <keith.mannthey@intel.com>
Change-Id: Iba243fe3132a5fa677e5a7bcd09e491727ba092a
Reviewed-on: http://review.whamcloud.com/4973
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Hudson
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-2101 clio: cl_sync_io_wait() need a barrier

cl_sync_io_note() has a small window between cfs_atomic_dec_and_test
and cfs_waitq_broadcast, another thread might have destroyed anchor
between this window, then cl_sync_io_note() is calling wakeup on a
poisoned waitq and crash.

Signed-off-by: Liang Zhen <liang.zhen@intel.com>
Change-Id: I04bfc738c8c5bec3a1e8b51131fae4199ef5ec9a
Reviewed-on: http://review.whamcloud.com/5199
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>
Tested-by: Hudson
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2010 tests: replay-single/44a,b,45 find only tested mdc

If multiple filesystems are mounted, then test_45 would have found
mdcdevs for each filesystem. This corrects that to only look for the
filesystem that is being tested against.

Also correctly delete old test files.

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I0297cf9b21e9c97ee468cd81d5d4ac409c59d1e0
Reviewed-on: http://review.whamcloud.com/4724
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-2576 osc: various fix in osc_enter_cache()

- It should not go to sleep if there isn't any inflight write
  RPC, otherwise, it could not be waked up for a long time,
  until dirty flush triggered on other OSCs or other objects.

- If the page can't be granted due to too many dirty pages
  (> obd_max_dirty_pages), osc_enter_cache() should return
  -EDQUOT.

- It should pass NULL osc object to  osc_io_unplug(), otherwise
  dirty flush can't be triggered if the passed object is
  clean.

- It should not wait in a loop of "while (cli->cl_dirty > 0)",
  trigger the dirty flush once is enough. If there are too
  many threads that consumed grants immediately, then we
  should turn to sync write, but not trigger flush and wait
  for grant again (that'll block io process on grant).

- Refuse setting max_dirty_mb as 0 via proc file.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I449cba07bd427749ab023c249d9e200aba1b406a
Reviewed-on: http://review.whamcloud.com/4963
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Prakash Surya <surya1@llnl.gov>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-2663 tests: don't remove objects by debugfs

In lfsck.sh, we shouldn't use debugfs to remove objects, that'll
cause quota usage inconsistence warning in e2fsck.

Fix improper LBUG() in mdd_create_data(), MDS_OPEN_HAS_OBJS is
still used by lfsck.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I93c6f5c40e359d0d8402868be548088d6e2ac5da
Reviewed-on: http://review.whamcloud.com/5186
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Tested-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2640 osd: enhance transaction debug to trace rollback

Some modifications may need to rollback because of failures during
the transactions. The original transaction trace mechanism did not
consider rollback cases, so cannot correctly record sub-operations
and the rollback cases.

In this patch, not only the specified sub-operation will be traced,
but also its rollback operation (if may rollback) will be recorded
also. For example:

When declare for ref_add, it will record the ref_add sub-operation
by "osd_thandle::ot_declare_ops[OSD_OT_REF_ADD]"; when consume the
ref_add credit, the possible rollback will be recorded through the
"osd_thandle::ot_declare_ops_rb[OSD_OT_REF_ADD]"; and when need to
rollback the ref_add (by ref_del) in subsequent processing, it can
be traced by the "osd_thandle::ot_declare_ops_rb[OSD_OT_REF_ADD]";
and once rollback starts, only "osd_thandle::ot_declare_ops_rb[X]"
can be consumed.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Iade214eff468c2298f84920b997040e06f1a204f
Reviewed-on: http://review.whamcloud.com/5138
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2061 hsm: send HSM resquest to CDT

This patch implements the user interface to ask for hsm requests

It implements:
- lfs hsm_archive
- lfs hsm_restore
- lfs hsm_cancel
- lfs hsm_remove
- lfs hsm_release

The matching llapi call:
- llapi_hsm_user_request_alloc()
- llapi_hsm_request()

related MDS functions:
- mdt_hsm_request()

and the MDS RPC:
- MDS_HSM_REQUEST

Signed-off-by: JC Lafoucriere <jacques-charles.lafoucriere@cea.fr>
Change-Id: I635f6e0518ec12c359dbb528c3a61216650d952d
Reviewed-on: http://review.whamcloud.com/5030
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2061 hsm: get list of HSM actions

This patch implements a MDS RPC to get
the list of registered and running hsm actions on files

It implements:
- lfs hsm_action

The matching llapi call:
- llapi_hsm_current_action()

The matching MDS function:
- mdt_hsm_action()

and MDS RPC:
- MDS_HSM_ACTION

Signed-off-by: JC Lafoucriere <jacques-charles.lafoucriere@cea.fr>
Change-Id: If4b00949ca8c94c8572cc81a7f7b8db32914542c
Reviewed-on: http://review.whamcloud.com/5029
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-1338 hsm: HSM flags feature

This extends lustre_hsm_attrs struct in MDD to store HSM per-file
information. It adds infrastructure (function, RPCs, ...) to read and
modify it, from MDT code or from user-space on clients through llapi
and lfs command.

This patch implements:
- lfs hsm_state
- lfs hsm_set
- lfs hsm_clear

matching llapi functions:
- llapi_hsm_state_get()
- llapi_hsm_state_set()

matching mdt functions:
- mdt_hsm_state_get()
- mdt_hsm_state_set()

and MDS RPC:
- MDS_HSM_STATE_GET
- MDS_HSM_STATE_SET

Signed-off-by: Aurelien Degremont <aurelien.degremont@cea.fr>
Signed-off-by: JC Lafoucriere <jacques-charles.lafoucriere@cea.fr>
Change-Id: Ic8f82ddc9a56206307c2e5be2523fb7ce42b8638
Reviewed-on: http://review.whamcloud.com/3035
Tested-by: Hudson
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2638 ldiskfs: update dotdot entry for rename

On master, when rename a directory, its old dotdot entry will
be removed firstly, then insert the new dotdot entry, and try
to append FID-in-dirent. But the space for dotdot entry maybe
not enough to hold the new dotdot with FID-in-dirent, such as
the MDT device restored from file-level backup, or the device
is upgraded from 1.8. If we do not move data in the directory
block, the FID-in-dirent will overwrite the dx_root and cause
the directory crashed. Currently, we do not want to introduce
complex logic to handle directory data moving, instead, under
such case, ignore the FID-in-dirent for the new dotdot entry,
and just insert the new dotdot entry.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Id38f5136fb2bc607344c37ca10a32e63ab427e3d
Reviewed-on: http://review.whamcloud.com/5179
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>

LU-1187 lod: move seq client init from MDT to LOD

Move seq client init from MDT to LOD, so to unify
config log for adding OSP and MD OSP.

Change-Id: I93f91c0dd6f3568196a99eecf422e0b409abda31
Signed-off-by: Wang Di <di.wang@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/4337
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-1187 lod: Add remote object for DNE

LOD will do FLD lookup to check whether the object
is for OST or remote MDT object.

Because there are remote object invlove, it might
return some error other than ENOMEM for object allocation.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: Ie226b3c475524fc7e705665c6200e4c66d50acf0
Reviewed-on: http://review.whamcloud.com/4924
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-2062 hsm: llapi hsm calls for CopyTool

llapi hsm calls for CopyTool
- llapi_hsm_copytool_start()
- llapi_hsm_copytool_fini()
- llapi_hsm_copytool_recv()
- llapi_hsm_copytool_free()
- llapi_hsm_copy_start()
- llapi_hsm_copy_end()
- llapi_hsm_progress()
- llapi_hsm_import()

and related MDT functions
- mdt_hsm_ct_register()
- mdt_hsm_ct_unregister()
- mdt_hsm_progress()

The new RPC are:
- MDS_HSM_CT_REGISTER
- MDS_HSM_CT_UNREGISTER
- MDS_HSM_PROGRESS

Signed-off-by: JC Lafoucriere <jacques-charles.lafoucriere@cea.fr>
Change-Id: I93617ac6e66a3324344b57f1a22d7c851d715262
Reviewed-on: http://review.whamcloud.com/4736
Tested-by: Hudson
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

New tag 2.3.60

Also another can kicking for LU-2449, hopefully last this time.

Change-Id: I1455e48f1b557bc8787ef234e1894b0f9d7f4843
Signed-off-by: Oleg Drokin <green@whamcloud.com>

LU-2183 osp: cleanup osp-on-ost

With new quota design and FIDonOST introduced, a new connection
from OST/MDTn to MDT0 is required for sending quota and sequence
RPCs. This connection was implemented by adding a new special
osp device named osp-on-ost, which introduced complexity to osp
code, and many hacks of checking the osp device name in both
client and server side code.

In this patch, we removed the osp-on-ost, and introduced a new
device type lwp (Light Weight Proxy) to manage the light weight
connection established from OSTs/MDTs to MDT0.

Signed-off-by: Niu Yawei <niu@whamcloud.com>
Signed-off-by: Wang Di <di.wang@whamcloud.com>
Change-Id: Ic1ba25f941a4f971a542dc4be8a81e28ee0f477b
Reviewed-on: http://review.whamcloud.com/4923
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-398 ptlrpc: Add the NRS framework and FIFO policy

The Network Request Scheduler (NRS) allows a user to control the way
in which RPCs are dispatched from PTLRPC services. This can be used to
various effects, most importantly improved fairness across the
cluster, better I/O performance under certain workloads, and allowing
for QoS semantics to be used at the filesystem level. Central to the
operation of NRS are NRS policies, each of which implements handling
of RPCs in a different way, aiming to achieve a particular goal.

This patch adds the core NRS framework and the NRS FIFO policy, which
is a logical wrapper around previous, non-NRS functionality, and is
used as the default policy for handling all types of RPCs.

Signed-off-by: Nikitas Angelinas <nikitas_angelinas@xyratex.com>
Co-authored-by: Liang Zhen <liang@whamcloud.com>
Change-Id: I7fcd7885cc89b653d9bc482da533aae2f129bdf9
Oracle-bug-id: b=13634
Xyratex-bug-id: MRP-73
Reviewed-on: http://review.whamcloud.com/4411
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>