LU-2909 osc: flush until no dirty in osc_enter_cache()

In osc_enter_cache(), when there is high contention on grant, the
returned grant could be consumed by other process immediately, then
we should repeat flush until get enough grant or no dirty to be
flushed. Otherwise, mmap writer could easily get -EDQUOT on the
osc_enter_cache() and result in SIGBUS at the end.

Because we now changed to async flush in osc_enter_cache(), the
wakeup condition is changed accordingly.

This patch also temporarily disabled osc lru shrinker in
osc_io_unplug() to avoid the potential stack overrun problem.
See LU-2859.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I0c7c90ffe27dab6ded7ad07ed78017acb8665d59
Reviewed-on: http://review.whamcloud.com/5732
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2388 statahead: race in do_sa_entry_fini()

Two fixes:
* When iterating sa_entry list in do_sa_entry_fini(), there is
  no lock, as may cause one entry put twice. To fix this, all
  entries are put in one list, and only 'scanner' will drop
  entry from this list.
* sa_entry may be linked to sai_sent_entries, but not hashed
  yet, if ll_sa_entry_fini() is called at this moment, this
  sa_entry may be unhashed.

Also include minor cleanup:
* rename do_sai_entry_fini() to do_sa_entry_fini().
* rename do_sai_entry_to_stated() to do_sa_entry_to_stated().
* rename do_statahead_interpret() to ll_post_statahead() to
  distinguish from ll_statahead_interpret().
* ll_post_statahead() always post handle statahead from received
  list to simplify logic.

Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Change-Id: I3d0911d0bd3b940c9650473099604646408200c4
Reviewed-on: http://review.whamcloud.com/5842
Tested-by: Hudson
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Peng Tao <bergwolf@gmail.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1199 build: Relocate missed lbuild-fc18

lbuild-fc18 was missed in the move of the other lbuild files
to contrib/lbuild.  This relocates that file as well.

Change-Id: I2775f1e0aa4c7d17d2e1d8a114f2bea3702fec68
Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>
Reviewed-on: http://review.whamcloud.com/5872
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3024 test: use facet name in init_param_vars

When configuring jobstats init_parm_vars should call
set_conf_param_and_check with the facet name (client) rather then the
hostname.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I00b99ec3172d3e108fb6dd6d94825badf81e84df
Reviewed-on: http://review.whamcloud.com/5835
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2826 tests: Re-enable sanity/18

Due to ZFS work in LU-2449, sanity/18 should now work correctly.

Test-Parameters: mdtfilesystemtype=zfs mdsfilesystemtype=zfs  ostfilesystemtype=zfs testlist=sanity
Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I5d9d986b00a36a2b5f30ef9b4ff3d52779070498
Reviewed-on: http://review.whamcloud.com/5720
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2900 llite: Null deref in ll_fsync:mkdir on NFSmounted Lus

When a Lustre file system is mounted via NFS and a mkdir
operation is attempted, a null pointer dereference occurs
in ll_fsync.

With 2.x, Lustre added support for different VFS fsync APIs
that do not include a dentry parameter.

To make the logic the same in all cases, the old ll_fsync
interface was changed to pull the inode from f_dentry
in the *file parameter.

In some cases when using the old ll_fsync interface, the
caller does not set the f_dentry part of the *file parameter
resulting in a NULL dereference. The fix to this is to
restore the old logic in those cases: when a dentry
parameter is provided, get the inode from that parameter
rather than the file parameter.

Signed-off-by: Patrick Farrell <paf@cray.com>
Change-Id: I93ecf04e23121c76e571383d74e2fc902565614e
Reviewed-on: http://review.whamcloud.com/5585
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Reviewed-by: Cory Spitz <spitzcor@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3038 mgs: clear mti in the end of marker

The target information should be cleared when it met
the end marker of each mdc/osc, otherwise, the information
left over in mti will be brought into the config log of
next target, even worse some target might be identfied
wrongly as failover pair.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I67838914825caf0a9b4da8bc3cdfbb779a6eadd4
Reviewed-on: http://review.whamcloud.com/5851
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2995 osd: Delete the ref of remote parent in del ..

Try to delete remote dir from REMOTE PARENT during delete ..,
because the object might not be destroyed if it is being
opened by someone else, instead it will be inserted to the
ORPHAN dir, so if it does not delete the {.., REMOTE_PARENT},
it will be two parents for this remote dir.

Add remote object FID into op_data->op_fid3, so the stale check
will not fail during remote open.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I6bbac9174869ae02f385ac52740bb2046f0f1bab
Reviewed-on: http://review.whamcloud.com/5791
Tested-by: Hudson
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2926 ldiskfs: crash in is_bad_inode() on SLES11 SP2

Fix error handling in ldiskfs_xattr_inode_iget
for SLES11 SP2 platform

Signed-off-by: James Simmons <uja.ornl@gmail.com>
Change-Id: I05568cac5f4cc8a12cfd95034a99b06541289673
Reviewed-on: http://review.whamcloud.com/5709
Tested-by: Hudson
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1199 build: Clean out the build directory

Clean up the build directory.  Move in the direction of reserving
"build/" for the special-purpose Makefile that lives there.
Eventually we could rewrite the autoconf tests to eliminate that
Makfile, and the build directory could disappear altogether (after
finding homes for anything else that is left).

The autoconf m4 file move into a top level "config" directory.

Most other things that have moved are put in the new "contrib"
top-level directory.  For instance, "contrib/lbuild" contains
all of the lbuild-related files, and "contrib/git-hooks" contains
the git hooks for lustre developers.

Most of the moved files were unchanged, however the lbuild scripts
needed some tweaking to deal with the new location.

Because of the way that Intel's build farm and git hooks expect to
find certain files in fixed locations I have had to leave a few
symlinks in place that point to the new locations.

Change-Id: I04dc529d4f4060b892e1e4eaa8613bbc3337c414
Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>
Reviewed-on: http://review.whamcloud.com/5035
Reviewed-by: Brian J. Murrell <brian.murrell@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2682 fid: cleanup direct _id and _seq access

Access/copying the individual _id and _seq fields should be
replaced with the _oi field, and compatibility macros for
accessing the individual fields should be removed.

Signed-off-by: Wang Di <di.wang@intel.com>
Change-Id: Iedce5cfb308c1b6de6aa24872b8c71e7b616b67c
Reviewed-on: http://review.whamcloud.com/5601
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2970 lov: wait for active IOs before deleting the object

lov_layout_operations::llo_delete will be called when the file's
layout is going to change or file is being taken out of cache so we
must make sure there is no active IOs on this object.

For layout change case, it's lov_layout_change()'s responsibility to
make sure there is no active IOs before entering into llo_delete.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: Id56a66548f9f05555e6d112355d2a71da5ab4705
Reviewed-on: http://review.whamcloud.com/5812
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1187 osd: add remote entry insert for ZFS DNE.

Add remote entry insert in osd_index_insert for DNE on ZFS.

Signed-off-by: Wang Di <di.wang@intel.com>
Change-Id: I4c2337e69e4c4c4754b8ff7995dd69318f3d674d
Reviewed-on: http://review.whamcloud.com/4933
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>

LU-2783 ofd: Implement OBD_IOC_GET_OBJ_VERSION

This patch implements OBD_IOC_GET_OBJ_VERSION for ofd, so that tests
can use "lctl getobjversion" on OST objects.  jt_get_obj_version() now
takes ID and group pairs, as well as FIDs, for OST objects.  Although
the code are for testing purposes only, dereferencing an ioc_inlbuf
without checking its size is still unacceptable.  Hence,
mdt_ioc_version_get() is changed to check sizes first.

Signed-off-by: Li Wei <wei.g.li@intel.com>
Change-Id: If259aeca281dcb3f4ee4dc719719f722cf72cad4
Reviewed-on: http://review.whamcloud.com/5467
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3025 hsm-llapi: bad type cast in llapi_hsm_copytool_fini()

Correct a wrong type cast in llapi_hsm_copytool_fini()

Signed-off-by: JC Lafoucriere <jacques-charles.lafoucriere@cea.fr>
Change-Id: Iebe4a36178afdfad8b99140efd6b3afc8b9eb942
Reviewed-on: http://review.whamcloud.com/5837
Reviewed-by: John Hammond <johnlockwoodhammond@gmail.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2646 osd: add special flag for agent inode.

1. Add special flag into the LMA of agent inode of remote entry,
so lfsck check can skip this inode.

2. Assign a local FID for Agent dir, so lfsck can skip this agent
dir as well.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: Ifc50fbc531a4306a76d5263dd6135c613d7a2a30
Reviewed-on: http://review.whamcloud.com/5625
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3009 tests: failure on conf-sanity test_17

Test 17 is failing due to the failure of the debugfs command
in this subtest.  The command fails because MGSDEV is undefined.
Using an alternative that is available in the current test framework.

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I93407dd3c349d91a2ea4690b70dd67c68d65582c
Reviewed-on: http://review.whamcloud.com/5816
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3002 tests: Fix an OST index issue in replay-single

When run with 16 OSTs, replay-single 90 happened to pick ost14 as
ostfail and complained:

  lfs find does not report the affected lustre-OST000d_UUID for fd

The "d" in "fd" is a hexadecimal OST index, taken from
"lustre-OST000d_UUID".  Because the actual file names use decimal OST
indices, the test failed to match the file "f13" with "fd".  Moreover,
the test assumes the OST index is a single-digit number, which may not
be case with more than 15 OSTs.  This patch changes the test to simply
calculate the index from the facet name.

Test-Parameters: ostcount=32 testlist=replay-single envdefinitions=ONLY=90
Change-Id: I0c95d28f83c41cabf4ed8e5ed618897d6000da3b
Signed-off-by: Li Wei <wei.g.li@intel.com>
Reviewed-on: http://review.whamcloud.com/5796
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3003 tests: Do "lctl conf_param" on mds1 in t32_test()

With DNE, conf-sanity 32c inovkes "lctl conf_param" on mds2's host,
which may not be where the MGS is running, causing errors like:

  CMD: c05 /usr/sbin/lctl conf_param
  t32fs-MDT0001.mdc.max_rpcs_in_flight=9
  c05: No device found for name MGS: Invalid argument
  c05: This command must be run on the MGS.
  c05: error: conf_param: No such device
   conf-sanity test_32c: @@@@@@ FAIL: Setting MDT1
  "max_rpcs_in_flight"

This patch changes these "lctl conf_param" commands to set the same
parameters but on mds1's host, which always hosts the MGS.

Change-Id: Ibdca2802eaf33db504ed2b41a356d7a18be33f10
Signed-off-by: Li Wei <wei.g.li@intel.com>
Reviewed-on: http://review.whamcloud.com/5829
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1897 test: replay-single test_70b dbench not found

dbench is taking too long to start. It was observed
that a 12 seconds sleep was not long enough to start
dbench on all clients.

I added a loop to rally all clients before starting
the testing and improved the error messages.

Signed-off-by: Keith Mannthey <keith.mannthey@intel.com>
Change-Id: I7b0fce4c39c003c4db23686b295fb5e45a6f8a92
Reviewed-on: http://review.whamcloud.com/5761
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2556 osp: count requests in osp_sync_interpret() only

as it's called by ptlrpc always, so any counting in
osp_sync_request_commit_cb() is extra and wrong.

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Change-Id: Ib67ba11d2c382b1ff9cd19522ce9e15b57ebca94
Reviewed-on: http://review.whamcloud.com/5453
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1330 obdclass: splits server-side object stack from client

1. removes client-side dependencies of server object stack:
md_object.h/md_local_object.c, and dt_object.h/dt_object.c.
2. removes client-side dependencies of lustre_idmap.h/idmap.c
3. splits some server-side acl handlings

Signed-off-by: Liu Xuezhao <xuezhao.liu@emc.com>
Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ie3a3cd998ef7b2d7d1090673a7c2d8f1152ca976
Reviewed-on: http://review.whamcloud.com/2676
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>

LU-2951 mdt: Increase bc_req_max_size for MDS_REQUEST_PORTAL

Large EA tests triggered error messages like this on MDSs:

  10:38:21:LNetError: 3022:0:(lib-ptl.c:190:lnet_try_match_md())
  Matching packet from 12345-10.10.17.9@tcp, match 1429230968490588
  length 65928 too big: 117674 left, 49386 allowed

These were the REINT_SETXATTR requests carrying large EA values.  They
were dropped because the MDSs did not expect request buffers larger
than 49386 bytes.  This patch increases bc_req_max_size (and
bc_rep_max_size) for MDS_REQUEST_PORTAL (and MDS_REPLY_PORTAL) to
accommodate REINT_SETXATTR requests (and MDS_GETXATTR replies) with
255-byte names and 65536-byte values (or 65536-byte lists of EA
names).

Change-Id: Ifdcda6d3e91aa6115e5cafd5abd6c89b15485020
Signed-off-by: Li Wei <wei.g.li@intel.com>
Reviewed-on: http://review.whamcloud.com/5703
Tested-by: Hudson
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2445 utils: lfs migrate support

Add a new command migrate to lfs utility.
migrate takes sames args as setstripe and a --block option
It:
- create a volatile file based on new stripe information
- copy file data to this volatile file
- swap the 2 layouts
- if --block is set, take the grouplock to block other access
  during copy

So after migrate the file is restriped (mtime, atime are not
changed).
lfs_migrate is also updated to use the new call and
keeps the rsync mode as a fallback.

Signed-off-by: JC Lafoucriere <jacques-charles.lafoucriere@cea.fr>
Change-Id: I96bafb0be9bc273295c4c900c65b4028864fcbaa
Reviewed-on: http://review.whamcloud.com/5620
Tested-by: Hudson
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2895 osp: return proper error in osp_precreate_reserve

Make sure the osp_precreate_reserve() always return proper error
when reserving object failed.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I9d16135091ebe5ef1243416a15722c4859026e46
Reviewed-on: http://review.whamcloud.com/5683
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2960 lmv: typo fix for output of "lfs mdts"

Using ltd_idx(in lmv_tgt_descs) directly for the output of
lctl get_params lustre.lmv.*-clilmv-*.target_obd.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I6a4ff4b82e4d413baf1f36f85764d3c11cd82435
Reviewed-on: http://review.whamcloud.com/5782
Tested-by: Hudson
Reviewed-by: Li Wei <wei.g.li@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2756 ost: buffer utilization rate of OST rqbd

This patch covered a couple of things:
- different request buffer size for different OST services
  only requests on OST_IO_PORTAL need large buffers

- add extra 46K to OST_IO_BUFSIZE (buffer size is increased to 64K)
  size of OST_IO_MAXREQSIZE is about 18K, if OST_IO_BUFSIZE is only
  a little bigger than OST_IO_MAXREQSIZE, then each rqbd can only fit
  in 2-3 request even there are 18K bytes left in this rqbd, memory
  utilization rate is low. Extra bytes can make buffer utilization
  rate increase to 72%. Also, it can avoid version compatibility
  issue in the future if we want to increase request size again.

Signed-off-by: Liang Zhen <liang.zhen@intel.com>
Change-Id: Iaf8b45520b000dd5219a4af2ec3f5b2ea7f53134
Reviewed-on: http://review.whamcloud.com/5626
Tested-by: Hudson
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2927 mdt: Not return -EREMOTE for open lock.

Instead of returning -EREMOTE for open_lock, it will check
the open lock by the DISP_OPEN_LOCK, otherwise it will cause
transno being wrongly set to 0.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I4f3bccea87872ca191b0b6c2142e8897be3d731a
Reviewed-on: http://review.whamcloud.com/5694
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2920 autoconf: delete unused LB_LINUX_MODPOST

LB_LINUX_MODPOST does not seem to be used anywhere.

Signed-off-by: Minh Diep <minh.diep@intel.com>
Change-Id: Ibec2002ae8b74b9c6f1f6df82a35d44bd00916dc
Reviewed-on: http://review.whamcloud.com/5615
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Hudson
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

New tag 2.3.63

Change-Id: I02e9d699d3a7c5dc4253ad24dee0b10651cb4ba7
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2398 osc: ignore layout for osc_lru_shrink()

osc_lru_shrink() doesn't need to check verify layout change
because layout can't be destroyed if a page already exists.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I034c15cbecfaf1491d7d7fcb992b74e3d94d9fa5
Reviewed-on: http://review.whamcloud.com/4701
Tested-by: Hudson
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2955 tests: make replay-ost-single/8b SLOW for ZFS

This test runs dd with osync, which is currently a VERY slow
operation, so mark it slow until LU-2887 is addressed.

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I3f55278490bd3c2e3bce8a2de58828a3a716b2fe
Reviewed-on: http://review.whamcloud.com/5767
Tested-by: Hudson
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2405 tests: account for inter-op testing sanity/17g

Account for where patch hasn't been applied and skip known broken
servers.

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I1dd354c36c38e2569ce0cfaf57d4cd35a65117ff
Reviewed-on: http://review.whamcloud.com/5754
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2813 hsm: Add user credential information to copy tool RPC

Add MDT_BODY data to the RPC used for coytool, so the
coordinator can make some check based on the user

Signed-off-by: JC Lafoucriere <jacques-charles.lafoucriere@cea.fr>
Change-Id: Ia45aacfdc78411f7c202bafb6bd6411f686606a6
Reviewed-on: http://review.whamcloud.com/5436
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2783 include: Fix a conditional indentation

Just an indentation fix to make the conditional expression in
fid_is_sane() easier to read.

Signed-off-by: Li Wei <wei.g.li@intel.com>
Change-Id: I1112d7d43c29f032c79277c6434d4c02e96e49bd
Reviewed-on: http://review.whamcloud.com/5466
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2743 mgs: Add a missing rc reset

Current mgs_write_log_target() does not reset rc when EALREADY is
found.  This may cause the positive errno to leak to upper layers and
eventually to wire.  This patch adds the missing rc reset.

Change-Id: I0ba8a0decdb9583224a5de02f630742ae94fff39
Signed-off-by: Li Wei <wei.g.li@intel.com>
Reviewed-on: http://review.whamcloud.com/5730
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2743 llite: Fix a negative hpk_errval assignment

The hpk_errval field should store positve errnos.  This patch fixes a
place where errnos are assigned to hpk_errval as negative numbers.

Change-Id: I77b1f9e6a19a64aedcdcc5fe38b08babb810a472
Signed-off-by: Li Wei <wei.g.li@intel.com>
Reviewed-on: http://review.whamcloud.com/5729
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2929 build: fix unused/uninitilized virables error

As reported by gcc (GCC) 4.6.3 20120306 (Red Hat 4.6.3-2).

Signed-off-by: Peng Tao <tao.peng@emc.com>
Signed-off-by: James Simmons <uja.ornl@gmail.com>
Change-Id: I05aeb88dbd00b6b005ddde543a527339a9d66ce0
Reviewed-on: http://review.whamcloud.com/5656
Reviewed-by: John Hammond <johnlockwoodhammond@gmail.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Alexey Shvetsov <alexxy@gentoo.org>

LU-2335 lnet: remove unnecessary libcfs include

libcfs_pack/unpack.h are only needed for WINNT.
Only include them when necessary.

Signed-off-by: Peng Tao <tao.peng@emc.com>
Change-Id: Ibe6230d37534fd087c2afe6166d10259fd59c076
Reviewed-on: http://review.whamcloud.com/4647
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: John Hammond <johnlockwoodhammond@gmail.com>

LU-2675 cleanup: remove unused recov_thread.c and llog code

Remove recov_thread.c and the following exported functions:
  llog_cat_process_thread
  llog_obd_origin_add
  llog_catalog_list
  llog_receptor_accept
  llog_origin_connect
  llog_handle_connect
  llog_obd_repl_cancel
  llog_obd_repl_sync
  llog_obd_repl_connect
  llog_get_cat_list
  llog_put_cat_list
  llog_recov_thread_start
  llog_recov_thread_stop

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I620c641646294fcf697f1e2d4d1f389d71bb3ddd
Reviewed-on: http://review.whamcloud.com/5190
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Liu Xuezhao <xuezhao.liu@emc.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>

LU-2396 ofd: Remove unmaintained brw_stats

Remove unused nid_brw_stats member from struct nid_stat and the files
/proc/fs/lustre/obdfilter/*/{brw_stats,exports/*/brw_stats}.  If the
corresponding osd has a brw_stats proc entry then add a symlink from
the ofd proc dir.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I79a10816c612f67b80172d679b875ca94157b5f4
Reviewed-on: http://review.whamcloud.com/4689
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>

LU-2961 build: Fix commit-msg to love new files

It is annoying to receive errors like this when the changes being
committed include new files:

  $ git commit -asv --amend
  line 65: diff --git a/lustre/include/lustre_errno.h
  b/lustre/include/lustre_errno.h
  new file mode 100644
  error: commit message invalid signoff section line
  line 66: index 0000000..89b8d1b
  error: commit message invalid signoff section line
  line 67: --- /dev/null
  error: commit message invalid signoff section line
  ...

This patch adds the "new file mode" pattern to commit-msg.  Hopefully,
this is enough.

Signed-off-by: Li Wei <wei.g.li@intel.com>
Change-Id: I39c985d51afb04184322a8fc725bcef3938bf419
Reviewed-on: http://review.whamcloud.com/5712
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Reviewed-by: Bruce Korb <bruce_korb@xyratex.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-1434 llog: remove unused llog_create_rec struct

Remove the unused llog_create_rec and llog_orphan_rec definitions
from wireshark.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I21c1215b627fb972d8da89a7ff1965848027cab0
Reviewed-on: http://review.whamcloud.com/2877
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: John Hammond <johnlockwoodhammond@gmail.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-812 ldiskfs: super_operations->dirty_inode now takes a flag

Currently this flag is unused by ext4, so just pass in 0.  This change
happened in kernel commit aa38572954ade525817fe88c54faebf85e5a61c0.
Apparently the flag is used to tell the difference between timestamp
updates and anything else.

Signed-off-by: James Simmons <uja.ornl@gmail.com>
Signed-off-by: chas williams - CONTRACTOR <chas@cmf.nrl.navy.mil>
Change-Id: I24536546256f5f043c1f53e15220b0c956be343f
Reviewed-on: http://review.whamcloud.com/4966
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-1994 kernel: fix reference counting with l_dentry_open

Commit 78b1d1bd (LU-1994 kernel: 3.6 dentry_open uses struct path
as first arg) added support for the new dentry_open call that
accepts struct path instead of a dentry/vfsmount pair, but missed
the new reference counting rules that go along with it.

Upstream commit 765927b2 also makes dentry_open grab references itself
so it no longer frees references that weren't passed to it.

On failure, we'll end up with an extra reference to the dentry
that was passed in.

Since new dentry_open is the one that will be around for the
foreseeable future, let's just map to that directly for the path case.

For the other two cases, we'll take the references ourselves in
ll_dentry_open, free them on failure, and adjust callers to expect
that it won't free any references passed to it.

Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Change-Id: I05a95cf735a5b2d70273a485335d571fcda7a6b0
Reviewed-on: http://review.whamcloud.com/5330
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Reviewed-by: Peng Tao <bergwolf@gmail.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-1798 tests: don't set jobid if not changing

Don't set jobid_var conf_param if the value would not change.  This
avoids setting the same parameter in the config log multiple times,
and avoids the extra delay on each startup waiting for the conf_param
to propagate from the MGS to the client each time.

Fix the console error printing if the jobid_env does not exist.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I36ed60451a875bcd46fd0d6f3d7068d1b1398df5
Reviewed-on: http://review.whamcloud.com/3867
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2484 mgs: remove unused md_stats

Do not allocate md_stats or create /proc/.../MGS/md_stats for mgs
devices since they do not implement the md_ops interface.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I8b417bc2433484a0e008ff9bf7fef69ba1e62416
Reviewed-on: http://review.whamcloud.com/4813
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2513 osc: compute grant targets in bytes

In osc_shrink_grant() and osc_shrink_grant_to_target() convert page
unit target values to bytes before comparing to cl_avail_grant.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ie8e6e8b4b3245efa3b14777608f3a48bbab7e4e2
Reviewed-on: http://review.whamcloud.com/5495
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2675 cleanup: remove obsolete llog-test crud

Do not create the llog-test.c symlink as this Linux 2.4-ism is no
longer needed. Also remove the now defunct llog-test.sh.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I72ba710659e168ead206a2b9b28633724d0e44c5
Reviewed-on: http://review.whamcloud.com/5239
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2916 llite: call ll_permission rather than inode_permission

So that we can build on kernels older than 2.6.27.

Signed-off-by: Peng Tao <tao.peng@emc.com>
Change-Id: I288030c3ee37ccb909d45121d457adb4dccafe0a
Reviewed-on: http://review.whamcloud.com/5607
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: jacques-Charles Lafoucriere <jacques-charles.lafoucriere@cea.fr>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2642 osd-ldiskfs: fix typo in REQ_WRITE check

Commit eecb3086 defines __REQ_WRITE as BIO_RW if __REQ_WRITE is
not defined. __REQ_WRITE is an enum. REQ_WRITE is a define.

Let's use that instead.

Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: James Simmons <uja.ornl@gmail.com>
Change-Id: I500c771e6f3e69e1be3fab872150019103515d30
Reviewed-on: http://review.whamcloud.com/5503
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2873 tests: t-f generates numerous dots garbage

Remove 'echo -n "."' from run_test t-f function when skipping test

Signed-off-by: Kyrylo Shatskyy <kyrylo_shatskyy@xyratex.com>
Reviewed-by: Roman Grigoryev <Roman_Grigoryev@xyratex.com>
Xyratex-bug-id: MRP-863
Change-Id: I2c6de9fff3676aee7a506a2c2f6b6f9fe7a3b472
Reviewed-on: http://review.whamcloud.com/5540
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2731 scripts: Speed up /etc/init.d/lustre stop

This patch parallelizes the shutdown of multiple services running on the
same node. This has been empirically shown to drastically reduce the
runtime of the script for an OSS with many OSTs.

This patch was tested on a Lustre 2.1 ldiskfs OSS node with 32 OSTs
attached, by recording startup and shutdown times for the OSS. The
number of OSTs used in varied, ranging from a single one, up to all 32,
incrementing by powers of two (i.e. timed startup/shutdown of 1 OST,
then of 2 OSTs, then 4, etc.).

Results of startup and shutdown times *without* this patch applied:

    +------------------------------------------------+
    | $ time /etc/init.d/lustre start # (w/o patch)  |
    +-----------+------------+-----------+-----------+
    | # of OSTs |    real    |    user   |    sys    |
    +-----------+------------+-----------+-----------+
    |      1    | 0m  2.184s | 0m 0.162s | 0m 0.077s |
    |      2    | 0m  4.285s | 0m 0.281s | 0m 0.148s |
    |      4    | 0m  8.508s | 0m 0.500s | 0m 0.302s |
    |      8    | 0m 16.961s | 0m 1.017s | 0m 0.568s |
    |     16    | 0m 33.884s | 0m 1.964s | 0m 1.176s |
    |     32    | 1m  7.744s | 0m 3.986s | 0m 2.280s |
    +-----------+------------+-----------+-----------+

    +------------------------------------------------+
    | $ time /etc/init.d/lustre stop # (w/o patch)   |
    +-----------+------------+-----------+-----------+
    | # of OSTs |    real    |    user   |    sys    |
    +-----------+------------+-----------+-----------+
    |     1     | 0m  4.758s | 0m 0.072s | 0m 0.030s |
    |     2     | 0m  9.018s | 0m 0.118s | 0m 0.049s |
    |     4     | 0m 18.813s | 0m 0.185s | 0m 0.083s |
    |     8     | 0m 37.586s | 0m 0.337s | 0m 0.141s |
    |    16     | 1m 16.092s | 0m 0.597s | 0m 0.263s |
    |    32     | 2m 37.550s | 0m 1.181s | 0m 0.403s |
    +-----------+------------+-----------+-----------+

Results of startup and shutdown time *with* this patch:

    +------------------------------------------------+
    | $ time /etc/init.d/lustre start # (w/ patch)   |
    +-----------+------------+-----------+-----------+
    | # of OSTs |    real    |    user   |    sys    |
    +-----------+------------+-----------+-----------+
    |      1    | 0m  2.183s | 0m 0.158s | 0m 0.083s |
    |      2    | 0m  4.282s | 0m 0.274s | 0m 0.153s |
    |      4    | 0m  8.519s | 0m 0.510s | 0m 0.303s |
    |      8    | 0m 16.966s | 0m 1.019s | 0m 0.583s |
    |     16    | 0m 33.878s | 0m 1.984s | 0m 1.154s |
    |     32    | 1m  7.745s | 0m 3.944s | 0m 2.322s |
    +-----------+------------+-----------+-----------+

    +------------------------------------------------+
    | $ time /etc/init.d/lustre stop # (w/ patch)    |
    +-----------+------------+-----------+-----------+
    | # of OSTs |    real    |    user   |    sys    |
    +-----------+------------+-----------+-----------+
    |      1    | 0m  4.566s | 0m 0.075s | 0m 0.023s |
    |      2    | 0m  4.857s | 0m 0.105s | 0m 0.070s |
    |      4    | 0m  4.777s | 0m 0.175s | 0m 0.064s |
    |      8    | 0m  5.449s | 0m 0.323s | 0m 0.153s |
    |     16    | 0m  5.862s | 0m 0.606s | 0m 0.208s |
    |     32    | 0m  6.307s | 0m 1.183s | 0m 0.811s |
    +-----------+------------+-----------+-----------+

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Change-Id: I90c1f6a265a8d86bbc8ddfb88aa635e5b96fd975
Reviewed-on: http://review.whamcloud.com/5235
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2760 ldiskfs: update rhel6.3 series to handle mkdir errors

ext4_mkdir can fail in several paths to dirtying an inode but
the errors aren't caught. This patch adds the upstream commit
that handles the errors and adjusts the dependent patches
accordingly.

Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: James Simmons <uja.ornl@gmail.com>
Change-Id: If86538e88c7386a06016ffae6893bacc8ba131e4
Reviewed-on: http://review.whamcloud.com/5279
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-913  test:  Framework needs to record the test filesystem.

Add a section

file_system: XXX

to the node info in yaml.sh

Because this runs on the node being recorded it can use
node_fstypes $HOSTNAME
to fetch the filesystemtype of the local machine

Signed-off-by: Chris Gearing <chris.gearing@intel.com>
Change-Id: I721e4084096c75b69290959190526ca27b573e1b
Reviewed-on: http://review.whamcloud.com/4591
Tested-by: Hudson
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Li Wei <wei.g.li@intel.com>
Reviewed-by: Wei Liu <wei3.liu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2675 cleanup: remove unused mkdirdeep.c and lltrace.h

That's all.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I547623a98541d545057776a01424fa6a362f06ee
Reviewed-on: http://review.whamcloud.com/5177
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2342 tests: account for log size in replay-single/20b

Account for larger ondisk log size in ZFS (256) vs. ldiskfs (50).

Test-Parameters: mdsfilesystemtype=zfs ostfilesystemtype=zfs   mdtfilesystemtype=zfs testlist=replay-single
Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I4150d58556240b128c02ba667c4c390c79b8a463
Reviewed-on: http://review.whamcloud.com/5666
Tested-by: Hudson
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2903 tests: calculation of available space

Better debug logging for statfs path.
Account for log size in space calculation.
Sleep slightly longer to wait for good statfs because zfs doesn't
count uncommitted changes in available space.

Test-Parameters: mdsfilesystemtype=zfs ostfilesystemtype=zfs   mdtfilesystemtype=zfs testlist=replay-ost-single
Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Ib4b5a7c645d6b2a630dcc729483422c8b3a095db
Reviewed-on: http://review.whamcloud.com/5662
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2911 llite: add obd_fid_init/fini() back to llite

If without obd_fid_init() in llite, when filesystem is upgraded from
branch 1.8 to 2.4, obd_fid_init() in lmv won't be triggered because
of no lmv in branch 1.8 based config log. This will cause LBUG during
running mkdir after upgrade, like

seq_client_alloc_fid()) ASSERTION( seq != ((void *)0) ) failed.
seq_client_alloc_fid()) LBUG
Call Trace:
[<ffffffffa0371895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
[<ffffffffa0371e97>] lbug_with_loc+0x47/0xb0 [libcfs]
[<ffffffffa080bea9>] seq_client_alloc_fid+0x379/0x440 [fid]
[<ffffffffa03822e1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[<ffffffffa082470b>] mdc_fid_alloc+0xbb/0xf0 [mdc]
[<ffffffffa0832b1c>] mdc_create+0xcc/0x780 [mdc]
[<ffffffffa09c487b>] ll_new_node+0x19b/0x6a0 [lustre]
[<ffffffffa09c50a7>] ll_mkdir+0x97/0x1f0 [lustre]

Signed-off-by: Liu Ying <emoly.liu@intel.com>
Change-Id: I0eab1298b8d02ca08ecd4ac8bb422a2de12b7f83
Reviewed-on: http://review.whamcloud.com/5733
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Tested-by: Hudson
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2948 osc: keep trying to shrink more LRU slots

It used to try once. It can enter into a livelock state as we can
see it in this ticket.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: If603686d1d9fa7cf6513143fcc6ef962cfea9863
Reviewed-on: http://review.whamcloud.com/5760
Tested-by: Hudson
Reviewed-by: Prakash Surya <surya1@llnl.gov>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2849 hsm: don't run sanity-hsm for ver < 2.3.61

Add version check in sanity-hsm.
Now that sanity-hsm has been added, we should consider running it by
default.

Test-Parameters: testlist=sanity-hsm

Signed-off-by: Johann Lombardi <johann.lombardi@intel.com>
Change-Id: I8cec304bed46ac24354a27716ad12f5233c75a3f
Reviewed-on: http://review.whamcloud.com/5502
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>

LU-2845 osp: fix osp precreate thread init error handling

If osp device hasn't connected OST, osp_precreate_thread() should
heed to that and bypass the normal quitting path.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I27d18dfd4d7d55c97eeb169b5d7dc7042a42fd33
Reviewed-on: http://review.whamcloud.com/5508
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2884 libcfs: SMP optimizations cleanups

Miscelaneous cleanups for the SMP optimizations code:
- Fixed typos.
- Fixed resource leak in lnet_create_locks().
- Fixed incorrect symbols in EXPORT_SYMBOL().

Signed-off-by: Isaac Huang <he.huang@intel.com>
Change-Id: I3b617367e5ed6b11ae327e477fc2c201c453e347
Reviewed-on: http://review.whamcloud.com/5547
Tested-by: Hudson
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1876 hsm: revise ll_setattr_raw to not check stripe data

It used to check if the file has stripe data, or it won't send file
size to the MDT. After layout lock is introduced, it's unreliable to
check stripe data at llite layer.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: Ib947c568232b4025210d1e2e4e05bcf3514fd36a
Reviewed-on: http://review.whamcloud.com/5291
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: John Hammond <johnlockwoodhammond@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2868 lfsck: finish otable iteration if no more objects

If there are no objects to be scanned just at the LFSCK beginning,
then the LFSCK should mark the otable-based iteration as finished
to avoid to access non-initialized data via dt_it_ops::rec().

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I67e880ccfa13e867c4ca7f1858d67909ba0415b3
Reviewed-on: http://review.whamcloud.com/5622
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2912 mdd: move fid2path from MDD to MDT

1. Move some linkEA API to obdclass, so both MDD(set linkea)
and MDT(fid2path) can access these LINKEA operation.

2. Move fid2path from MDD to MDT, so it can detect remote
object, and return -EREMOTE to the client. Then the client
will try fid2path on another MDT, and finally the client
will assemble the path fragments from different MDT.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I464cd1fdd44ebbe02c94821f910294829f3d9b94
Reviewed-on: http://review.whamcloud.com/5676
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2748 fsfilt: ext4_map_inode_page in osd and ldisk out of sync

The functon ext4_map_inode_page is mismatched in its use between
osd-ldiskfs and ldiskfs. The integer array is no longer used so
we remove its handling from the ldiskfs layer.

Signed-off-by: James Simmons <uja.ornl@gmail.com>
Change-Id: I9b5522ce187f06983f328408cbcd0ce077e72ea1
Reviewed-on: http://review.whamcloud.com/5708
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>

LU-2902 test: Debug for roc_hit v1

This will output all the stats and help sort out what
is happening on the systems.  I would like to know
what the proc values are the roc_hit is blank.

More debug may be needed but I want to start here.

This patch should be dropped when LU-2902 is closed.

I have used this patch with auster without issue it
just added noise to the test results.

Signed-off-by: Keith Mannthey <keith.mannthey@intel.com>
Change-Id: I3efe9aa87e8c51909667f9fbc3b9c2d6779d8d0d
Reviewed-on: http://review.whamcloud.com/5648
Tested-by: Hudson
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-1565 ldlm: make blocking threads async wherenever possible

There is no need to wait for the cancel lru lock completion in the
client side pool recalculation, make it asynchronous.

make all the ldlm_cli_cancel() calls from blocking callbacks async

Change-Id: Ie510c7361f1025a78c693a11b457baf1652f8c90
Xyratex-bug-id: MRP-690
Reviewed-by: Andrew Perepechko <andrew_perepechko@xyratex.com>
Reviewed-by: Bruce Korb <bruce_korb@xyratex.com>
Signed-off-by: Vitaly Fertman <vitaly_fertman@xyratex.com>
Reviewed-on: http://review.whamcloud.com/4181
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2802 lmv: get tgt by checking ltd_idx in lmv_find_target.

Currently, lmv_find_target return lmv->tgts[mds]
according to mdt_index, which is not correct. LMV
index is created by mount sequence, while mdt_index
by indicated by --index. So we should check ltd_idx
in lmv_find_target.

Signed-off-by: Wang Di <di.wang@intel.com>
Change-Id: I67a941cca00eb80ba91af6eb3f3441982d4fcab3
Reviewed-on: http://review.whamcloud.com/5412
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-1187 out: add resend check for update.

1. Add update resend check between MDTs.

2. Even during creating the new object, osp_init_object
   still needs to get remote object attrs, because the
   object might be created already by some partial failed
   operation.

3. During resend handling, OSP needs to delete the existing
   orphan objects first, then do remote create.

4. MDT will check whether the name has been deleted during
   the resend of remote unlink, and only delete the local
   directory if the name on the remote MDT has been delete.

5. Fix the fail_id assignment location to only fail real
   update RPC.

6. Some fixes for replay DNE test sctipts.

Change-Id: Ia9ba8091b6622b0e2fd1f1b4fd355b5ff3eb9758
Signed-off-by: Wang Di <di.wang@intel.com>
Reviewed-on: http://review.whamcloud.com/4343
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2571 lfsck: run lfsck on server node

Usually we run e2fsck on OST and MDS nodes, and run lfsck on clients,
but it's not necessary to install e2fsprogs rpm on clients.
So, if lfsck is not found on client, we will try server node instead.

Test-Parameters: testlist=lfsck
Signed-off-by: Liu Ying <emoly.liu@intel.com>
Change-Id: I3979dc0236e81163f3283eac3148c36c8ddccf63
Reviewed-on: http://review.whamcloud.com/5139
Tested-by: Hudson
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2194 test: avoid wrong eviction in recovery_small

in subtest 19a and 19b of recovery_small, the locks from OST/MDT
should be cancelled before "drop_ldlm_cancel" to avoid wrong eviction
from OST/MDT for it will also drop its lock cancellation request.

Change-Id: I49d1d189f6791224a3f132dc8cf2dd8b3d51d43e
Signed-off-by: Hongchao Zhang <hongchao.zhang@intel.com>
Reviewed-on: http://review.whamcloud.com/5679
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Hudson
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2640 osd: conditional enable transaction debug

Current transaction debug mechanism will be disabled
when Lustre-2.4 released.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I516a9026b2d887935c5714dc0b777d65d487dac7
Reviewed-on: http://review.whamcloud.com/5698
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2882 build: build broken with no zfs libraries installed

Reverting default values (originaly set in LU-2391 patch) for
ldiskfs/zfs OSD RPMs builds, to comply for more ways
("rpmbuild -tb <lustre_source_tarball>") used to build Lustre.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: I07eca22d03f597942284bbbc0bfd1b680ecb199b
Reviewed-on: http://review.whamcloud.com/5661
Tested-by: Hudson
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Alexey Lyashkov <alexey_lyashkov@xyratex.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2803 osd: osd-zfs to handle echo sequence (2) properly

use visible OI (/O directory) to map object from this sequence
into dnodes. this let obdecho to work with zfs.

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Change-Id: I63e50789df502d11863c69658c9524fbb3cd9f22
Reviewed-on: http://review.whamcloud.com/5414
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2910 clio: skip iov update when tot_nrsegs is zero

When tot_nrsegs is zero, we should skip the iov update too.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I52d4d52a65802f3967dd6a96ad46ec40fd4ef355
Reviewed-on: http://review.whamcloud.com/5747
Tested-by: Hudson
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1032 build: Support --disable-maintainer mode

We add "AM_MAINTAINER_MODE([enable])" to all configure
scripts to allow us to use --disable-maintainer-mode.

By default, without the AM_MAINTAINER_MODE macro, autotools
"maintainer mode" is enabled.  By specifying "enable" we
maintain our previous default behaviour.

Change-Id: I88366ad658795145af80ed96c6e708c385799ffa
Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>
Reviewed-on: http://review.whamcloud.com/5423
Tested-by: Hudson
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Prakash Surya <surya1@llnl.gov>
Reviewed-by: Brian J. Murrell <brian.murrell@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2468 libcfs: quiet spurious debug message

When cfs_trace_get_tage_try()->cfs_tage_alloc() is allocating a debug
buffer, since e2a2fab993d01597010cb2b44df44a522af0eec8 (b=21776) this
allocation is denied when the allocation is happening in a memory
freeing path.  This caused a spurious "cannot allocate a tage" message
on the console each time.  Quiet that message, since it is expected.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: Ic800c474dc33f62843b74e06d9ca642cad3ebbe5
Reviewed-on: http://review.whamcloud.com/5470
Tested-by: Hudson
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2424 ptlrpc: reduce initial buffers count

Separate buffers count for server and client services.
Reduce inititial allocation for client ("ldlm_cbd") because
its not required a lot. This reduced unreclaimable memory
usage after just mount from 233036kB to 97386kB.

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: I56fcbe6c45c61ba4876bce5482169ea06a03638c
Reviewed-on: http://review.whamcloud.com/5719
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2836 quota: improve test_3 & test_6 of s-q

When approaching quota limit, client turns to sync write, then the
tests which assume io can be finished quickly could fail, we need
to enlarge the time margin to make sure those tests can pass on slow
system.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: Ib14a905344eb78cdbd0cd79e3bfd8e50ab21a4d8
Reviewed-on: http://review.whamcloud.com/5539
Tested-by: Hudson
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2922 changelog: recommend e2fsprogs-1.42.6.wc2

This patch updates lustre/ChangeLog to recommend
a newer e2fsprogs-1.42.6.wc2 release instead of the
old 1.41.90.wc4 version.

Signed-off-by: Jian Yu <jian.yu@intel.com>
Change-Id: Ib56092500545f0373a307d7645d41d91079ae086
Reviewed-on: http://review.whamcloud.com/5718
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-2883 hsm: Mark file DIRTY as soon as pages are written

Since the dirty flag has to be packed in close, it should be set when
pages are written and not when building BRWs like SOM which relies on
MDS_DONE_WRITING.

Signed-off-by: Aurelien Degremont <aurelien.degremont@cea.fr>
Change-Id: I9cf0a71cf3228a7aadb8205cff2735a7abff5ef0
Reviewed-on: http://review.whamcloud.com/5543
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>

LU-2622 obdclass: Remove the global cl_env list

New cl_env structures are allocated using a SLAB cache specifically
created for allocating and freeing these structure. Without this patch,
when a thread is finished with a cl_env structure it places it on a
global list instead of freeing it back to the SLAB cache. With this
patch, this global list is completely removed, and cl_env structures are
released immediately back to the SLAB cache.

The motivation for this change essentially boils down to this secondary
global list cache being completely unnecessary, and only serving to
serialize any calls to cl_env_get and cl_env_put. This has proven to
cause a severe performance impact on large core count systems,
specifically during memory reclamation (i.e. ll_releasepage).

For example, on BG/Q Sequoia IO nodes, we've experienced nearly all
68 cores of a machine spinning on the lock protecting this global
list. Some example stack traces showcasing this problem were gathered
using sysrq-l, and are displayed below:

    CPU56:
    Call Trace:
    [c00000000fe3bb30] [c000000000008d1c] .show_stack+0x7c/0x184 (unreliable)
    [c00000000fe3bbe0] [c00000000027604c] .showacpu+0x64/0x94
    [c00000000fe3bc70] [c000000000068b30] .generic_smp_call_function_interrupt+0x10c/0x230
    [c00000000fe3bd40] [c00000000001d11c] .smp_message_recv+0x34/0x78
    [c00000000fe3bdc0] [c00000000002526c] .bgq_ipi_dispatch+0x118/0x18c
    [c00000000fe3be50] [c00000000007b20c] .handle_IRQ_event+0x88/0x18c
    [c00000000fe3bf00] [c00000000007dc90] .handle_percpu_irq+0x8c/0x100
    [c00000000fe3bf90] [c00000000001b808] .call_handle_irq+0x1c/0x2c
    [c0000003e1c4a4c0] [c0000000000059f0] .do_IRQ+0x154/0x1e0
    [c0000003e1c4a570] [c0000000000144dc] exc_external_input_book3e+0x110/0x114
    --- Exception: 501 at ._raw_spin_lock+0xd8/0x1a8
        LR = ._raw_spin_lock+0x104/0x1a8
    [c0000003e1c4a860] [8000000000b04f38] libcfs_nidstrings+0x2acc/0xfffffffffffe5824 [libcfs] (unreliable)
    [c0000003e1c4a910] [c00000000042d4cc] ._spin_lock+0x10/0x24
    [c0000003e1c4a980] [80000000024c2f4c] .cl_env_get+0xec/0x480 [obdclass]
    [c0000003e1c4aa60] [80000000024c336c] .cl_env_nested_get+0x8c/0xf0 [obdclass]
    [c0000003e1c4aaf0] [800000000692070c] .ll_releasepage+0xbc/0x200 [lustre]
    [c0000003e1c4aba0] [c000000000094110] .try_to_release_page+0x68/0x8c
    [c0000003e1c4ac10] [c0000000000a4190] .shrink_page_list.clone.0+0x3d8/0x63c
    [c0000003e1c4adc0] [c0000000000a47d8] .shrink_inactive_list+0x3e4/0x690
    [c0000003e1c4af90] [c0000000000a4f54] .shrink_zone+0x4d0/0x4d4
    [c0000003e1c4b0c0] [c0000000000a5a68] .try_to_free_pages+0x204/0x3d0
    [c0000003e1c4b220] [c00000000009d044] .__alloc_pages_nodemask+0x460/0x738
    [c0000003e1c4b3a0] [c000000000095af4] .grab_cache_page_write_begin+0x7c/0xec
    [c0000003e1c4b450] [8000000006920964] .ll_write_begin+0x94/0x270 [lustre]
    [c0000003e1c4b520] [c0000000000968c8] .generic_file_buffered_write+0x148/0x374
    [c0000003e1c4b660] [c000000000097050] .__generic_file_aio_write+0x374/0x3d8
    [c0000003e1c4b760] [c00000000009712c] .generic_file_aio_write+0x78/0xe8
    [c0000003e1c4b810] [800000000693ed4c] .vvp_io_write_start+0xfc/0x3e0 [lustre]
    [c0000003e1c4b8e0] [80000000024d9c6c] .cl_io_start+0xcc/0x220 [obdclass]
    [c0000003e1c4b980] [80000000024e1a84] .cl_io_loop+0x194/0x2c0 [obdclass]
    [c0000003e1c4ba30] [80000000068ba1d8] .ll_file_io_generic+0x498/0x670 [lustre]
    [c0000003e1c4bb30] [80000000068ba834] .ll_file_aio_write+0x1d4/0x3a0 [lustre]
    [c0000003e1c4bc00] [80000000068bab50] .ll_file_write+0x150/0x320 [lustre]
    [c0000003e1c4bce0] [c0000000000d1ba8] .vfs_write+0xd0/0x1c4
    [c0000003e1c4bd80] [c0000000000d1d98] .SyS_write+0x54/0x98
    [c0000003e1c4be30] [c000000000000580] syscall_exit+0x0/0x2c

    CPU63:
    Call Trace:
    [c00000000fe03b30] [c000000000008d1c] .show_stack+0x7c/0x184 (unreliable)
    [c00000000fe03be0] [c00000000027604c] .showacpu+0x64/0x94
    [c00000000fe03c70] [c000000000068b30] .generic_smp_call_function_interrupt+0x10c/0x230
    [c00000000fe03d40] [c00000000001d11c] .smp_message_recv+0x34/0x78
    [c00000000fe03dc0] [c00000000002526c] .bgq_ipi_dispatch+0x118/0x18c
    [c00000000fe03e50] [c00000000007b20c] .handle_IRQ_event+0x88/0x18c
    [c00000000fe03f00] [c00000000007dc90] .handle_percpu_irq+0x8c/0x100
    [c00000000fe03f90] [c00000000001b808] .call_handle_irq+0x1c/0x2c
    [c0000003c4f0a510] [c0000000000059f0] .do_IRQ+0x154/0x1e0
    [c0000003c4f0a5c0] [c0000000000144dc] exc_external_input_book3e+0x110/0x114
    --- Exception: 501 at ._raw_spin_lock+0xdc/0x1a8
        LR = ._raw_spin_lock+0x104/0x1a8
    [c0000003c4f0a8b0] [800000000697a578] msgdata.87439+0x20/0xfffffffffffccf88 [lustre] (unreliable)
    [c0000003c4f0a960] [c00000000042d4cc] ._spin_lock+0x10/0x24
    [c0000003c4f0a9d0] [80000000024c17e8] .cl_env_put+0x178/0x420 [obdclass]
    [c0000003c4f0aa70] [80000000024c1ab0] .cl_env_nested_put+0x20/0x40 [obdclass]
    [c0000003c4f0aaf0] [8000000006920794] .ll_releasepage+0x144/0x200 [lustre]
    [c0000003c4f0aba0] [c000000000094110] .try_to_release_page+0x68/0x8c
    [c0000003c4f0ac10] [c0000000000a4190] .shrink_page_list.clone.0+0x3d8/0x63c
    [c0000003c4f0adc0] [c0000000000a47d8] .shrink_inactive_list+0x3e4/0x690
    [c0000003c4f0af90] [c0000000000a4f54] .shrink_zone+0x4d0/0x4d4
    [c0000003c4f0b0c0] [c0000000000a5a68] .try_to_free_pages+0x204/0x3d0
    [c0000003c4f0b220] [c00000000009d044] .__alloc_pages_nodemask+0x460/0x738
    [c0000003c4f0b3a0] [c000000000095af4] .grab_cache_page_write_begin+0x7c/0xec
    [c0000003c4f0b450] [8000000006920964] .ll_write_begin+0x94/0x270 [lustre]
    [c0000003c4f0b520] [c0000000000968c8] .generic_file_buffered_write+0x148/0x374
    [c0000003c4f0b660] [c000000000097050] .__generic_file_aio_write+0x374/0x3d8
    [c0000003c4f0b760] [c00000000009712c] .generic_file_aio_write+0x78/0xe8
    [c0000003c4f0b810] [800000000693ed4c] .vvp_io_write_start+0xfc/0x3e0 [lustre]
    [c0000003c4f0b8e0] [80000000024d9c6c] .cl_io_start+0xcc/0x220 [obdclass]
    [c0000003c4f0b980] [80000000024e1a84] .cl_io_loop+0x194/0x2c0 [obdclass]
    [c0000003c4f0ba30] [80000000068ba1d8] .ll_file_io_generic+0x498/0x670 [lustre]
    [c0000003c4f0bb30] [80000000068ba834] .ll_file_aio_write+0x1d4/0x3a0 [lustre]
    [c0000003c4f0bc00] [80000000068bab50] .ll_file_write+0x150/0x320 [lustre]
    [c0000003c4f0bce0] [c0000000000d1ba8] .vfs_write+0xd0/0x1c4
    [c0000003c4f0bd80] [c0000000000d1d98] .SyS_write+0x54/0x98
    [c0000003c4f0be30] [c000000000000580] syscall_exit+0x0/0x2c

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Change-Id: Ief4b524784e07d7677ecb8a9ce97a7b54ccc6f75
Reviewed-on: http://review.whamcloud.com/5204
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2240 osd: change FID of /ROOT on zfs

Pre-production 2.4 code used FID_SEQ_LOCAL_FILE for /ROOT. With
ldiskfs that FID turns into IGIF which is mapped to MDT0 permanently.
With ZFS original local sequence, was used which makes existing setup
incompatibile with DNE. The intention of the patch is to fix this on
existing ZFS setups and replace FID with one from special FID_SEQ_ROOT
sequence which is mapped to MDT0 as well. For simplicity this is done
in few steps:

 - osd-zfs replaces direntry for /ROOT with the new FID and fixes OI
   so that the new FID is mapped to the same dnode
 - MDD finds all objects listed in /ROOT and updates linkEA properly
 - MDD removes ./.. which may be on disk for pre-production setups
 - finally MDD resets LMA on /ROOT with the new FID, which is later
   used to recognize already converted filesystems (or ones created
   with correct FID from the beginning) and skip this conversion code

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Signed-off-by: Di Wang <di.wang@intel.com>
Change-Id: I03062c6909146f9a3aed72f41c0708f9ef92bb82
Reviewed-on: http://review.whamcloud.com/5249
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Prakash Surya <surya1@llnl.gov>

LU-951 test: waiting import state in fail().

Anyway, There still has a rare chance that the request meet
a invalid import after fail() return. So we should waiting
import restore to a certain state and doing next operation.
Add wait_import_state_mount() to check import state while
client has a mount point.

Signed-off-by: yang sheng <yang.sheng@intel.com>
Change-Id: I55ccb0fd30d69eae651978804ef1e303d9939a71
Reviewed-on: http://review.whamcloud.com/5531
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>

LU-2712 tests: enable sanity/sanityn SLOW tests

Enable a number of subtests in sanity.sh and sanityn.sh that are
currently skipped for normal "review" builds because of "SLOW=no".
Checking the maloo test history for these tests, they are verified
as always passing for "full" builds in the SLOW=yes case.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I340ccdf1323d14a34dc87a5217b3256c8f62cab0
Reviewed-on: http://review.whamcloud.com/5218
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-1346 libcfs: remove cfs_ file wrappers

Replace file relevant wrappers with kernel API.

Affected primitives:
file, dentry, dirent, kstatfs, filp_size, filp_poff,
filp_open, do_fsync, filp_close, filp_read, filp_write,
filp_fsync, get_file, fget, fput, file_count, flock_t,
flock_type, flock_set_type, flock_pid, flock_set_pid,
flock_start, flock_set_start, flock_end, flock_set_end.

completion, init_completion, fini_completion,
wait_for_completion, complete

Change some API implementations of darwin/winnt to make it be
consistent with linux kernel API such as filp_open/filp_close etc.

Signed-off-by: Liu Xuezhao <xuezhao.liu@emc.com>
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ibe00df71c658aeb5dda854481f6ab5c181b3de7b
Reviewed-on: http://review.whamcloud.com/2830
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Peng Tao <bergwolf@gmail.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2910 clio: restore iov when restart io

The iovector needs be restored on restarted io. This patch also
added some LASSERT and debug message to make it easier to debug
this kind of problem later.

Test-Parameters: testlist=racer
Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I5d737cfa083ae3b3d9f040a2dc36d6b8693b548b
Reviewed-on: http://review.whamcloud.com/5652
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2871 lod: stripe data across the OSTs correctly

Since the ost-in-use array is initialized with 0 and isn't set with
OST index number correctly, OST0 is always skipped when allocating
objects on OSTs with specific stripe offset (offset > 0).

For example, when running command "lfs setstripe -c -1 -i 2 testfile"
on 4 OSTs, we will get a wrong layout, like
        obdidx           objid           objid           group
             2               3            0x3                0
             3               3            0x3                0
             1               4            0x4                0
             2               4            0x4                0

To fix the problem, we initialize the array with -1 instead, and store
the correct OST index number in it.

Signed-off-by: Liu Ying <emoly.liu@intel.com>
Change-Id: I5eeced6b66ae40771f8896204c5a6ed8e6663e57
Reviewed-on: http://review.whamcloud.com/5554
Tested-by: Hudson
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>

LU-2701 osp: wake up sync thread

osp_sync_process_committed() to wake up sync thread when it
is requested to stop (e.g. umount) and there is no pending
work left. the patch adds a sanity check to ensure this
process is not taking too long.

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Change-Id: I5251013afc2aee55627c806a11eb826a9d3dbec9
Reviewed-on: http://review.whamcloud.com/5463
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2925 out: increase reqbuf size for OUT

OUT service for DNE can have request size up to 9K, however, it's
using default definition of MDS request buffer size which is 5K.
This patch added individual definitions for OUT reqsize and
req_bufsize.
This patch also made some changes to MDS_BUFSIZE and MDS_LOV_BUFSIZE
to unify style of macros.

Signed-off-by: Liang Zhen <liang.zhen@intel.com>
Change-Id: I102d6b2aed1e0ed495f055fa3de0c7de7de8c28d
Reviewed-on: http://review.whamcloud.com/5653
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Reviewed-by: wangdi <di.wang@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2735 test: disable LRU to avoid page deletion

In sanity.sh, subtest 151 checks whether the page written previously
is in cache or not, this patch disable the LRU of objects in OST to
avoid the object and its content(pages) to be dropped.

Change-Id: Ie481d13215dac599e0b7e122fcc7e9819a053af5
Signed-off-by: Hongchao Zhang <hongchao.zhang@intel.com>
Reviewed-on: http://review.whamcloud.com/5475
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2852 script: sanity 27u should cleanup test dir

sanity 27u should cleanup test dir before test in case there
are remaining files from previous tests.

remove 27u from ALWAYS_EXCEPT since it can pass now.

Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Change-Id: Id21589c1e1ecf0f4143fad3574cd779c73adb7aa
Reviewed-on: http://review.whamcloud.com/5670
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2449 osd: lookup(..) to fetch fid from parent's LMA

in case LinkEA is not accessible for a reason, osd-zfs
will be trying to get fid for ".." using parent's dnode
which is stored in regular ZFS attributes. parent's
LMA can be used to get fid for ".."

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Change-Id: I34e9f884eb60f036c2f941013bf22e154efc2ff4
Reviewed-on: http://review.whamcloud.com/5629
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>

LU-2829 tests: sanityn/33a cleanup messages for zfs

Remove ldiskfs specific checks from running on zfs.
Account for lvm disks for ldiskfs.

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I15cdb33d095c465b117761d40d61579eb3fbf52a
Reviewed-on: http://review.whamcloud.com/5693
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>

LU-2926 ldiskfs: crash in is_bad_inode()

Fix error handling in ldiskfs_xattr_inode_iget

Xyratex-bug-id: MRP-883
Signed-off-by: Andriy Skulysh <Andriy_Skulysh@xyratex.com>
Change-Id: I9840b7bd32f2c96763cae402a7bcb51d5798ea6c
Reviewed-on: http://review.whamcloud.com/5631
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-2809 llite: Do not return layout_gen for getxattr

The problem is that layout_gen and stripe_offset are sharing the
same field in lov_user_md{}. Layout gen would be wrongly interpreted
as stripe_offset when a backup is restored.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: If2c120bf861eeffd2db5f92e1d23cb1b9a2f5c63
Reviewed-on: http://review.whamcloud.com/5664
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: John Hammond <johnlockwoodhammond@gmail.com>

LU-2816 llite: Set RAS_INCREASE_STEP correctly

RAS_INCREASE_STEP is by pages instead of by bytes.

However, I found it caused performance loss by setting it to be 4MB so
I set it back to 1MB. After 4MB RPC is enabled by default, more work
should be done to pick up a right value of max_read_ahead_mb to
maximize performance gain of 4MB RPC.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: Ic000f266cfefc827e112e03f76cf467c73ba88ad
Reviewed-on: http://review.whamcloud.com/5691
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2775 fid: allow FID-on-OST in fid_seq_is_mdt()

The LASSERT_SEQ_IS_MDT() macro used fid_seq_is_mdt() in several
places to verify that a FID was "sane" for where it was being used.
However, there should never be LASSERTs for data from the network
or disk.

The use of LASSERT_SEQ_IS_MDT() is removed from clients, since this
is "validating" data from the network.  This macro is no longer used,
so remove it.  The old CMD objseq_to_mdsno() and mdt_to_obd_objseq()
helpers are also no longer needed for DNE, and can be removed.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I74bc9198799045b8bd91510cb45e8f876012cab0
Reviewed-on: http://review.whamcloud.com/5456
Tested-by: Hudson
Reviewed-by: John Hammond <johnlockwoodhammond@gmail.com>
Reviewed-by: wangdi <di.wang@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>