LU-646 port bz23485 (clarification of lustre fsync behavior)

Add directory fsync operation.

Signed-off-by: Lai Siyao <laisiyao@whamcloud.com>
Change-Id: I9915bab70291c503ff1462328d5f2fbcff2b700e
Reviewed-on: http://review.whamcloud.com/1309
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Bobi Jam <bobijam@whamcloud.com>
Reviewed-by: Johann Lombardi <johann@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-553 build: fix commit-msg line width check

Fix the calculation of the commit message line width to skip the
trailing linefeed character.

Don't use the "git hash_object -t commit" option, since this makes
some versions of Git unhappy and generate an empty Commit-Id string.

Skip diffstat output from "commit -v" when validating commit comment.

Validate the Change-Id: line has the proper ID format.

Reported-by: Bobi Jam <bobijam@whamcloud.com>
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I0bcfdedbe6d33fd5f81381ca33862a10d6b41b38
Reviewed-on: http://review.whamcloud.com/1553
Reviewed-by: Bobi Jam <bobijam@whamcloud.com>
Tested-by: Bobi Jam <bobijam@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-629 ptlrpc: fix _debug_req to print opc/status

The 2.x _debug_req() function was changed in bug 16359/commit 5467a86021
to avoid problems with accessing unswabbed message buffers. Unfortunately,
this broke the printing of many/most _debug_req() messages, because it
didn't check whether swabbing was actually needed in the first place.

Also, in ptlrpc_expire_one_request() some extra debugging information was
added in bug 21636/commit 368689640 but never removed, making this common
message overly verbose.

Fix _debug_req() so that it prints opcode/flags/status, unless the
ptlrpc_body _needs_ to be swabbed, but isn't.  Also print out more
useful idenfifiers for the nodes (the obd_name and NID instead of
the connection UUID).  This removes some of the added verbosity from
ptlrpc_expire_one_request(), and most of the rest was already being
printed out (deadline, current, etc).

Change-Id: I88a78486becd19f5b38f5578e5cc30e649564908
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1286
Tested-by: Hudson
Reviewed-by: Mikhail Pershin <tappro@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Niu Yawei <niu@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-753 obdfilter: improper LASSERT in filter_commitrw_write()

In rare cases fsfilt_commit_wait() will wake up and return after the
transaction has finished its work and updated j_commit_sequence but
the commit callbacks have not been run yet. Which will trigger the
LASSERT(oti->oti_transno <= obd->obd_last_committed) improperly.

We should just wait for the commit callback finished instead of put
an improper LASSERT here.

Signed-off-by: Niu Yawei <niu@whamcloud.com>
Change-Id: Ibd5add8d352d2e7598be49b0bf8fa37d40ce6e1f
Reviewed-on: http://review.whamcloud.com/1583
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Hudson
Reviewed-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

ORNL-24 skip unnecessary client obj hash lookup

The client-side object will be initialized when inode is established
in memory. Such object will be inserted into global hash table for
further using. The caller follows "lookup-alloc-lookup-insert". It is
standard, but maybe unnecessary for some cases. If it is sure that
there is only one caller will insert the new established object, then
we can skip the two lookup, and "alloc-insert" directly.

Signed-off-by: Fan Yong <yong.fan@whamcloud.com>
Change-Id: Ie173c57627b2e5b4ed9b8a93f368d88ba8e54c31
Reviewed-on: http://review.whamcloud.com/1225
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15 slow IO with read-intense application

Align the readahead extent by 1M after when it is trimed by ra_max_pages.

Change-Id: I4102d2fe956fd01457949f0eb7c63654b5c2d095
signed-off-by: Wang Di <di.wang@whamcloud.com>
signed-off-by: Yang Sheng <ys@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1255
Tested-by: Hudson
Reviewed-by: Jinshan Xiong <jay@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-555 ll_have_md_lock() optimization to accelerate multiple bits locks

Change-Id: Ie300ad2abf285a7413553da2a86cc74216cbae7d
Signed-off-by: jcl <jacques-charles.lafoucriere@cea.fr>
Reviewed-on: http://review.whamcloud.com/1170
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-488 ptlrpc_connection_put() LASSERT(!cfs_hlist_unhashed(&conn->c_hash))

Connection hash may be rehashed while ptlrpc_connection_put() is
called, ASSERT &conn->c_refcount > 1 instead of this.

Signed-off-by: Lai Siyao <laisiyao@whamcloud.com>
Change-Id: Iec6d35419e0c4d8497bd0b84c6210abc8eb23882
Reviewed-on: http://review.whamcloud.com/1074
Tested-by: Hudson
Reviewed-by: Li Wei <liwei@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Prakash Surya <surya1@llnl.gov>
Reviewed-by: zhen liang <liang.zhen@live.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-513 Make cfs_wait_event_interruptible_exclusive really exclusive

Change-Id: Iea0556a006f8826f8597824131fb5110a848c434
Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>
Reviewed-on: http://review.whamcloud.com/1118
Tested-by: Hudson
Reviewed-by: Lai Siyao <laisiyao@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Liang Zhen <liang@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Increase lustre version to 2.1.51

Change-Id: I91b5cc909aae9434349ec1ddbd16fd1eedadc412
Signed-off-by: Oleg Drokin <green@whamcloud.com>

ORNL-28: remove lock contention in reconnecting

target_handle_connect() used to grab obd::obd_recovery_task_lock to update
obd_next_recovery_transno and obd_connected_clients. In this patch, I revised
this piece of code by:
- modify obd_connected_clients to cfs_atomic_t
- grab obd_recovery_task_lock only if ocd_transno is less than
  obd_next_recovery_transno

In this way, target_handle_connect() don't grab any global lock.

Change-Id: I971897de82566e239fe750cf95f2c3f1646325bc
Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1293
Tested-by: Hudson
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mikhail Pershin <tappro@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-735 Remove fixme() macro

This NULL export warning is the only consumer of the fixme()
macro.  Change it to a CWARN and remove the fixme() macro.

Change-Id: I194a9d92369a6a4dce35701fe261631420f9894d
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-on: http://review.whamcloud.com/1472
Tested-by: Hudson
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-707 Fix FID string quoting in replay-vbr

The FID string printed by the "path2fid" lfs command was passed to
do_facet() without any quoting.  Depending on the names of the files in
the remote shell's working directory, the FID string could be replaced,
by the shell's filename expansion, to a list of matching file names.
This patch adds the necessary quoting prevent the remote shell from
doing filename expansion on the FID string.

Change-Id: I0bc28eb7f29df717a661a4c1bce8c918117b997c
Signed-off-by: Li Wei <liwei@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1409
Tested-by: Hudson
Reviewed-by: Mikhail Pershin <tappro@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-443 LNet: Only squawk when md->start is NULL on non-zero length

Only squawk when md->start is NULL on non-zero length.
The md->start == NULL check prevents anyone from creating a ME/MD
with no buffer. These are used as backstop buffers to generate events
when traffic has exceeded the local buffer space.

Change-Id: I1389b0a45d3f8ff548f6400c66b30a69bafb4f39
Signed-off-by: Wally Wang <wang@cray.com>
Reviewed-on: http://review.whamcloud.com/989
Tested-by: Hudson
Reviewed-by: Cory Spitz <spitzcor@cray.com>
Reviewed-by: Liang Zhen <liang@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-448 add lst stat --count

This adds lst stat --count, making it easier to script up data
collection for performance tests.

The patch is from Oracle bug 22638 attachment 33001

Change-Id: I46aa1de12ea8e6c2d5eafff8ce5c31d7d618c06f
Signed-off-by: Wally Wang <wang@cray.com>
Reviewed-on: http://review.whamcloud.com/1004
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Liang Zhen <liang@whamcloud.com>
Reviewed-by: Lai Siyao <laisiyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-357 racer test cleanup
 1, increase the test time to 300s(900s for SLOW)
 2, fixing the problem of recursively calling racer.sh

Change-Id: I91ac7e5c42ed5bc98b3a647c30d7e37af0573f09
Signed-off-by: Hongchao Zhang <hongchao.zhang@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/905
Reviewed-by: Yu Jian <yujian@whamcloud.com>
Reviewed-by: Fan Yong <yong.fan@whamcloud.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-743 conf-sanity: test_46a failure

This failure is because client still didn't see the adding OSTs so it
met a problem when decoding lsm because the # of OSTs was over tgt count
at the client side.

Change-Id: I49ee22379734375a2a92f7495f6d849e47db0909
Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1494
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

ORNL-28: Set recovery timeout correctly

make sure recovery window uses timeout value from lustre config;
in current implementation this piece of code is totally wrong since
it just disregards timeout configuration.

Change-Id: I0cb0d777569cccd96f30da11834c6e333a673816
Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1292
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

ORNL-28: Solve reconnecting race between IR and SR

if there is a connecting request on the fly when client import is
notified by IR, it will set the corresponding conn uuid to a higher
prio and set imp_force_verify so that it will do reconnection
immediately in case RPC timeout happens.

Change-Id: I77e799e1f12b49f3c0271585c1ce812d16dc1ef6
Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1291
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

ORNL-27: Cancel on completion lock on the MGS

We should cancel the recover/config LCK_EX lock immediately
when they are granted to accelerate enqueue process.

Also, it doesn't make sense to add mgc recover/config lock into
LRU list because these kinds of lock would never be canceled
voluntarily. Restore LDLM_FL_NO_LRU flag and apply it for mgc
lock.

Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Change-Id: I369b57ca4780b0bfa07d33b4423b468481263ade
Reviewed-on: http://review.whamcloud.com/1261
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

ORNL-14: Configuration and Unit test cases

In this patch, some procfs and test cases are added for imperative recovery.

Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Change-Id: Ied49b622208157a74e8547e4609d61ee5041f624
Reviewed-on: http://review.whamcloud.com/1219
Tested-by: Hudson
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Yu Jian <yujian@whamcloud.com>

LU-687 clio: retry if fault page was truncated

In vvp_io_fault_start, if a page was truncated we should retry it in
ll_fault() instead of return -EFAULT because it will cause fake OOM.

Change-Id: Ia2ba40ca4ecd67170e6c2eb81ddc0ee34d9379a8
Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1453
Tested-by: Hudson
Reviewed-by: Niu Yawei <niu@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Fan Yong <yong.fan@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-699 tests: replay-dual test_1 failure

The root cause of this problem is due to data corruption on lustre_disk_data.
In ldd_write(), it just waits for log to be committed but NOT data to actually
write to disk. So in this test case, if the data is not written into disk when
we mark the device as readonly, the data will be lost and cause remount mdt
failed.

Change-Id: I8c46a925ce2df3d2db69b1e7fd8813eb0668401d
Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1557
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Hudson

ORNL-13: NonIR client support

In this task, NonIR support is added. NONIR clients means those clients who
doesn't know the protocol of imperative recovery. This means they won't be
notified for the restarting of target.

To support NonIR clients, the MGS has to record how many NonIR clients per
file system, and track those clients. In this way, if there are NonIR clients
for a specific file system, the MGS should tell the restarting target to disable
imperative recovery; otherwise, these `old' clients are to be evicted easily.

Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Change-Id: I3725c66b74d702aa213644ee9a6f89d59b8a8083
Reviewed-on: http://review.whamcloud.com/1218
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-462 Don't alloc/free client data for self export

Self export doesn't need client data and ldlm initialization.

Change-Id: I31307d2212e3d11c79f1ab215edbb840c3cfb8c6
Signed-off-by: Mikhail Pershin <tappro@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1023
Tested-by: Hudson
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

ORNL-23 avoid unnecessary CMD lock contention in LMV

There are some redundant lock operations in LMV layer which is only
useful under old CMD mode. We should avoid those lock operations to
reduce unnecessary lock contention.

Change-Id: I2ad1ba728da39ce69b0e521ab3a992dd2b253f2e
Signed-off-by: nasf <yong.fan@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1226
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Lai Siyao <laisiyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-515 canonicalize the devices names

Perform a readlink on the device name if path is /dev/disk/by-id...
See Oracle bug 24487.

Change-Id: I964b224d764677d60064901f4238ae77b9cfb5ea
Signed-off-by: Wally Wang <wang@cray.com>
Signed-off-by: Niu Yawei <niu@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1120
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Li Wei <liwei@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-760 obdecho: initialization and awk problem

echo_client registers obd type and then initializes kmem cache, this is
problematic because echo_key will be immidiately accessed after obd type
is registered. This will cause kernel fault.

Also, an awk problem is fixed. If the length of output buffer is longer
than 1024 bytes, awk will run into problem.

Change-Id: Ief287e8f4eeb6a39bc336e7a9f5c21e921b79a58
Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1521
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-389 update packet-lustre.c for master

Drop unused mds_body/mds_rec_xxx/mds_status_req. Add missing mdt_body/mdt_rec_xxx.

Change-Id: Ic530541f58d12c721fa6efd0bc9a1096a15d7e33
Signed-off-by: nasf <yong.fan@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/995
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Bobi Jam <bobijam@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

ORNL-10: Basic IR implementation

To support imperative recovery, there is a target status table defined for
each file system defined on the MGS. When a target registers itself to the
MGS, the MGS will change this table correspondingly.

In the status table, one important field is target NID. This NID information
is used by clients locating server where target lives. By transferring this NID
to clients, clients can know the restarting of targets earlier. This is
so-called imperative recovery - the MGS notifies clients to do recovery
imperatively instead of timeout based standard recovery.

To implement imperative recovery, clients are asked to cache a NID table, which
contains the location information of all servers. Clients need to hold a read
mode ldlm plain lock - recover lock - to cache this table. Whenever the MGS
wants to change this table, it will enqueue an EXCL recover lock so that all
clients will be notified for this change. Clients will request for a new read
recover lock and then query for the MGS for NID table updates.

Change-Id: I3b38ba142b810df507805b71972feeb1bade1ac2
Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1217
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-666 mdd depends on ldiskfs-only dynlocks feature

As discussed in Lustre bugzilla 22435 dynlocks are provided by
the ldiskfs and are not a generic facility.  They should be
removed from the mdd-level to allow building --without-ldiskfs
per Andreas's comment in 22435:

  https://bugzilla.lustre.org/show_bug.cgi?id=22435#c3

This change converts the instances of "struct dynlock_handle *"
to a "void *" pointer which is safe because they are never
dereferenced at this level.

It also wraps the "struct dynlock" entry in the mdd_object with
MDD_DISABLE_PDO_LOCK.  This effectively limits all direct dynlock
usage to the !MDD_DISABLE_PDO_LOCK case which is unconditionally
disabled in the code.  Presumable this code will simply be removed
at some point which will fully resolve the issue.

Change-Id: I7c8d2fb04b31736782e5f6ea2ff6b631b475fb91
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-on: http://review.whamcloud.com/1348
Reviewed-by: wangdi <di.wang@whamcloud.com>
Reviewed-by: Liang Zhen <liang@whamcloud.com>
Reviewed-by: Mikhail Pershin <tappro@whamcloud.com>
Tested-by: Hudson
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-499 grant/cancel_rate are static when OST is idle

ldlm_pool_recalc() shouldn't be skipped if namespace->ns_bref eqauls
zero, instead a flag ns_stopping is added to mark ldlm namespace is
being freed.

Signed-off-by: Lai Siyao <laisiyao@whamcloud.com>
Change-Id: Ic6485c34ec3e9868ae531a4dc25aee969c374eb5
Reviewed-on: http://review.whamcloud.com/1185
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Prakash Surya <surya1@llnl.gov>
Reviewed-by: Bobi Jam <bobijam@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-690 test: wait_osc_import_state() fixes

b=24498 wait_osc_import_state() fixes

i=Vladimir.Saveliev

-- increase maxtime to wait the timeout of 1st request;
   take into account at_min value;
-- cleanup wait_osc_import_state() to use _wait_import_state();
-- ost-pools test_1 fix: use local var instead of global NAME

Signed-off-by: Elena Gryaznova <elena.gryaznova@oracle.com>
Signed-off-by: Yu Jian <yujian@whamcloud.com>
Change-Id: I36b163a291d783b1d772f05898ccee783d815298
Reviewed-on: http://review.whamcloud.com/1499
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-731 ldlm: revise and export ldlm_lock_set_data

We can only reset ->l_ast_data if it's NULL; otherwise, returning an
error if ->l_ast_data is trying to be changed to a different value.

Change-Id: I8f999a98101daa2c465256792285582f581909c5
Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1466
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-580: update mgc llog process code

This patch does the following work:
1. fix a bug about setting mgc lock callback data;
2. define log instance types;
3. simplify and cleanup mgc_requeue_thread code

Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Change-Id: Ic8d25eda9f0a7884b4a8cbeccc89c1e66c19f0ff
Reviewed-on: http://review.whamcloud.com/1198
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-573: conf-sanity test_22 failed with 41

Make sure recovery on OST is finished before trying to create file

Change-Id: I4a36685a5cd9c55de729906bff50c29b1108c931
Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1192
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-433 remove jbd2-jcberr patch from kernel

In the upstream ext4 code there is a new functionality to allow a
per-commit callback to be set (j_commit_callback), which is used by
the mballoc code to manage free space bitmaps after deleted blocks
have been released. We expand it to contain multiple different
callbacks.

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I6397ccabd8d729658cf2ee13c9c3731a9eb31219
Reviewed-on: http://review.whamcloud.com/983
Tested-by: Hudson
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-553 build: Lustre style commit hooks for Git

Adds recommended commit hooks for Lustre commit messages, that
can/should be installed into lustre/.git/hooks for all Lustre trees.

prepare-commit-msg: adds the output of checkpatch.pl as a
    comment to the commit message, to "soft" validate the new
    patch is matching the Lustre coding style. It doesn't check
    100% of the Lustre coding style, but at least finds most of
    the obvious errors so that they can be fixed locally before
    any manual inspection time is spent on the patch.

commit-msg: verifies the format of the commit message (somewhat).
    Checks that there is a valid-looking Signed-off-by: line.
    Adds the Gerrit Change-Id: line for change tracking.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I35fdfb8206d1f0efb3bb401522e8412c62b7c02a
Reviewed-on: http://review.whamcloud.com/1296
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Start of 2.2 development, updated version to 2.1.50

Change-Id: Ibaca591c4db2b839ece2419b77404cdcd869d843
Signed-off-by: Oleg Drokin <green@whamcloud.com>

LU-459 quiet too noisy console messages at mount

Quiet a number of extra debug messages printed to the console after a
remount or recovery.  They provide no value and just add to the general
confusion of reading Lustre debug messages.

Change-Id: Ia0b43bbd337501169dec234ff66dd56f67f75131
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1275
Tested-by: Hudson
Reviewed-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

ORNL-14: debug: print recovery completion message

Print a proper "end of recovery" message to match the one printed
by 1.x so that it is clear when recovery has finished, how long it
took, and how many clients recovered.

This should have been part of the original patch committed for
eda8eab5abeef1117ae1082533b5888a91875e12.

Change-Id: I3a8793fdcb3c665e8df0d39ae82b4c56fb1bb527
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1278
Tested-by: Hudson
Reviewed-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-by: Niu Yawei <niu@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-258 tests: delete obsolete filter_survey script

The filter_survey.sh was a very early version of obdfilter-survey.sh
script, but is long obsolete (last updated 2003) and can be removed.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I2fa46d393355ab35352ac693272071424d1a57e1
Reviewed-on: http://review.whamcloud.com/1406
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-610 fix the wrong data type in find_value_cmp

in function find_value_cmp (lustre/utils/liblustreapi.c), the data type for
paramters "file" and "limit" is converted from "unsigned long long" type into
"unsigned int" type by mistake, and the file size should be defined as
"unsigned long long" type too.

Change-Id: Icacebccdf55d8ad761390bb03aa9aab0fdc67131
Signed-off-by: Hongchao Zhang <hongchao.zhang@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1289
Tested-by: Hudson
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Yu Jian <yujian@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Fan Yong <yong.fan@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-241 Support crc32c with hardware accelerated instruction as one of lustre checksums

Adding CRC32C as new lustre checksum algorithm. Because of crc32c,
the Intel Nehalem based CPU supports h/w accelerated crc32c
instruction as one of SSE4.2 instruction set. The new crc32c codes
in the lustre also use this h/w instruction. The lustre also
automatically detects whether crc32c instruction is available, if
not, use adler (fastest checksum other than crc32c).

Change-Id: I764851a46a94a879239cd127eac411e98342e67f
Signed-off-by: Shuichi Ihara <sihara@ddn.com>
Reviewed-on: http://review.whamcloud.com/1009
Tested-by: Hudson
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Johann Lombardi <johann@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-110 Recognize old mdt.quota_type param

Minor changes in mdt_process_config() to make it capable of recognizing old mdt.quota_type.

Issue:
Signed-off-by: Niu Yawei <niu@whamcloud.com>
Change-Id: Ie13851eee573eb633a68bf5ee3cd86dbbf1bd1a9
Reviewed-on: http://review.whamcloud.com/354
Tested-by: Hudson
Reviewed-by: Johann Lombardi <johann@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-570: Add function to find connect uuid by nid

In this patch, two functions are added:
- class_find_uuid(), find conn uuid by peer nid
- client_import_find_conn(), find a conn uuid in import connection list

Also, a code cleanup is performed.

Change-Id: I50e8e9392a39ef78719504cf083c0c22f5d39dcb
Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1189
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-641 obdclass: use rwlock for obd_dev_lock

In target_handle_connect() it grabs obd_dev_lock to find obd_device
by target uuid. This causes lock contention if there are two many connection
requests on the same time. Based on obd_dev_lock is a read-mostly lock, it
is much reasonable to revise it as a rwlock.

Change-Id: I7f936d6182df9e7ed46b8a1b6fb259226188acbc
Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1294
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-574: conf-sanity test_41b failed with 1

Do not run this case if the MGS and MDS are on different devices.

Change-Id: I3b3839a0860dbf55ba1e066e02358abc131f643a
Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1193
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Yu Jian <yujian@whamcloud.com>

LU-572 ptlrpc: make ptlrpc_recover_import() async

Add an async argument for ptlrpc_recover_import() and with async nonzero,
ptlrpc_recover_import() will return immediately after sending connecting
RPC.

This is needed for imperative recovery since it may recover multiple
targets meanwhile.

Change-Id: Iccc40fca4d3d5d3fb0b2965e62beda23e2fb3b4c
Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1191
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-571 ldlm: Remove parallel AST limitation

Imperative Recovery needs to notify clients as soon as possible.
However, limiting the AST count to 200 will definitely block it.
This may result in the recovery window has passed before the MGS
notifies all clients.

In this patch, I revised the above limitation to be configurable,
and it is no limit by default.

Change-Id: I141da9dbd8b282a76252da15b93c432ac5253bfd
Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1190
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-569: Make lu_object cache size adjustable

lu_object cache is specified to consume 20% of total memory. This
limits 200 clients at most can be mounted on one node. We should make
it adjustable so that customers have a chance to configure it by
their needs.

Change-Id: I8dcb993c88a3abd9fc8ba11ff1578aa3897c3933
Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1188
Tested-by: Hudson
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-721 llite: Parallel writes to same file results in zero file

Canceling lock may contain data being sent to OSTs. Change
find_cbdata iterator to take that into account.

Author: Vladimir Saveliev <vladimir.saveliev@oracle.com>
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I1b30fe13ee5303fe3b98e45ad969f5273c41d676
Reviewed-on: http://review.whamcloud.com/1460
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Update release date for 2.1 in Changelog to reflect RC2

Signed-off-by: Oleg Drokin <green@whamcloud.com>
Change-Id: Ifda31666cac43cb964184903a3d3de20f671cb7d

LU-683: handle short write in cl_io_commit_write

CLIO has to handle short write case where no bytes were
actually copied from user space. In this case, we should
avoid adding that page into cache.

Change-Id: I78c870e741d7bdb2a9c4c0bfba1adc50a6e814e7
Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1383
Tested-by: Hudson
Tested-by: Yu Jian <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@whamcloud.com>

LU-653 i_version shouldn't be used for VBR

RHEL6 plus EXT4 combination cause i_version is to be used to store
Lustre versions instead of i_fs_version. Meanwhile this field is used
by ext4 internally and may be changed breaking Lustre recovery.

Patch makes only i_fs_version to be used for versions.

Change-Id: Idf7cd257aa2d288abce2a7c18c4d8b139bc9c5c6
Signed-off-by: Mikhail Pershin <tappro@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1342
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Hudson
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Tested-by: Yu Jian <yujian@whamcloud.com>

LU-653 Ignore last transno from clients with no outstanding transactions

Clients that did not have any unreplied transactions should not
artificially reduce next_recovery_transno since those transactions are
never coming anyway and would just elay all replay activity
until all clients are reconnected.

Change-Id: Iafa32b315211ada93838f4b16fe0f2c800d619c4
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1318
Tested-by: Hudson
Reviewed-by: Mikhail Pershin <tappro@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-662 fix conflict between seq_client_flush and seq_client_alloc_fid

seq_client_flush modifies the lu_client_seq.lcs_space by holding the
lu_client_seq.lcs_sem, but seq_client_alloc_fid uses lcs_update to
exclude other concurrent modification.

Change-Id: Ibc3335841b9a045360283e15aab9b6c36b7a39af
Signed-off-by: Hongchao Zhang <hongchao.zhang@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1364
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Hudson
Reviewed-by: Fan Yong <yong.fan@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-650 Max possible file size is mutable

Move maxbytes fields out of wire data part of lov_stripe_md,
it can change when OSTs change their status, e.g. disconnected
or inactivated.

Change-Id: I9fb1f2597df1e98d993792706ac94f469d23a78c
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1327
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Hudson
Tested-by: Yu Jian <yujian@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-614 tests: speed up needlessly long tests

Fix unlinkmany to use the original $next_id instead of the one
seen after createmany was run.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I816081ab1ee04fe66d6c14b76eb076262215bc32
Reviewed-on: http://review.whamcloud.com/1298
Reviewed-by: Yu Jian <yujian@whamcloud.com>
Tested-by: Yu Jian <yujian@whamcloud.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-617 temporary fix for the setattr from open

The setattr from open(open(O_TRUNC)) is serialized with
'cl_setattr_lock' on client and goes to a dedicate portal, which is
different with other reint operations, consequently, setattr RPC
can be parallel with other reint RPCs, and that result in the race of
updating last_transno/last_xid on server.

This temporary fix removes the LASSERT(req_is_replay(req)) in
mdt_last_rcvd_update().

Signed-off-by: Niu Yawei <niu@whamcloud.com>
Change-Id: I7e07eaf9e718c2d75d9a503207523be9c700ea4a
Reviewed-on: http://review.whamcloud.com/1295
Tested-by: Hudson
Reviewed-by: Mikhail Pershin <tappro@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-594 Make sure sanity 27v fallout clears

We need wait long enough for OST to wake up from
OBD_FAIL_TGT_DELAY_PRECREATE sleep.

Change-Id: I8e4e8153a11a70b66a64f4e8a1deb3e741f58554
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1262
Reviewed-by: Yu Jian <yujian@whamcloud.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-622 Alloc enough cfg buffer space

obd device name needs space enough for "<dev_name>-<instance_name>\0".
This fixes the glitch in commit cbc4ca2e8dc37d54bb7d3c9b02ab20b63e60f592.

Change-Id: Ib8c3c8db49baad8a9ca068a1c8009aee72aa1e62
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1280
Tested-by: Hudson
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-by: Yu Jian <yujian@whamcloud.com>
Tested-by: Yu Jian <yujian@whamcloud.com>

LU-614 test: speed up needlessly long tests

Some tests are running needlessly long compared to the value that
they provide, and can be made much faster without any loss in value.

conf-sanity test_53a: run llverdev in partial mode always
conf-sanity test_53b: run llverfs in partial mode always
sanity test_220: use unlinkmany instead of "rm -rf", but could likely
                 be sped up dramatically via CFS_FAIL_LOC() on OST

Change-Id: Ifbe66bffc051b6163e41a9bffb1d6b1ab6304e94
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1266
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Yu Jian <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Update release date for 2.1.0 RC1

Change-Id: Ia71c1f4a4d6867e0e605675a1d57a541ef5f9c92
Signed-off-by: Oleg Drokin <green@whamcloud.com>

LU-365 Update lustre/Changelog

To indicate that the release is from Whamcloud.

Change-Id: I2e8f8e83c895ef1591d55b460727e04a695e1f29
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1269
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

Update version to 2.1 as a start of RC process.

Signed-off-by: Oleg Drokin <green@whamcloud.com>
Change-Id: I46e6405953581b5cc10140e8f25e2b4b345e1a4d

Update recommended e2fsprogs version

Signed-off-by: Oleg Drokin <green@whamcloud.com>
Change-Id: I00a587a4ab4e8fecb6d3def33ee952b2f793873e

LU-365 Update copyright for files modified by Whamcloud

I scoured through the commits logs since first whamcloud commit
and updated all the files that had non-trivial contributions by
Whamcloud.

Signed-off-by: Oleg Drokin <green@whamcloud.com>
Change-Id: I293a82b69a4ce02c028517771bd172ed94c4e44b
Reviewed-on: http://review.whamcloud.com/1267
Reviewed-by: Robert Read <rread@whamcloud.com>
Tested-by: Hudson

LU-598 open non-exist object should return ENOENT

When call ll_intent_file_open() against non-exist object, we cannot
obtain RPC request refcount after the call, so subsequent operation
should not use such internal (only) RPC request to avoid to access
freed (or re-allocated) memory.

Change-Id: I4c3763f460c6145e376538f392fde338d44e6c5e
Signed-off-by: nasf <yong.fan@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1242
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-473 llite: improve error for 2.x cli + 1.8 srv

Improve the error message printed to the console when a 2.x client
is trying to mount a 1.8 filesystem.  More specifically, have the 2.x
client check whether the MDS supports the OBD_CONNECT_FID and
OBD_CONNECT_FULL20 flags, without which the client will not be able
to cope.  The 2.x client also does not understand the RPC structure
of 1.8 servers (mdt_body, mdt_rec, etc), but this is harder to check.

For a 2.x server, the client will configure an LMV device, so
exp_connect_flags will be zero, but in that case we also know the
server is running 2.x.

Change-Id: I935571aa0623f9311ddf0545904230721d2fad32
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1036
Tested-by: Hudson
Reviewed-by: Yu Jian <yujian@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-592 1.8 <-> 2.1 interop: missing LMV on client

When the cluster is upgraded from 1.8, client will not have LMV layer,
and the "ll_md_exp" is pointing to MDC directly, so the code which
assuming "ll_md_exp" as LMV need be revised:

- ll_dir_ioctl(LL_IOC_GETOBDCOUNT) should return 1 for MDC case;
- mdc_iocontrol() needs to handle the OBD_IOC_QUOTACTL as well when the
  "ll_md_exp" is MDC;

Signed-off-by: Niu Yawei <niu@whamcloud.com>
Change-Id: I3659a5f8076125c9b2012826521c44540e65a75e
Reviewed-on: http://review.whamcloud.com/1236
Tested-by: Hudson
Reviewed-by: Fan Yong <yong.fan@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-599 verify "mi_ginfo" before accessing its member

If the target user only belong to one group, then we just set
"identity->mi_gid", and does not set "identity->mi_ginfo", to
save unnecessary memory allocation and groups processing.

So before accessing member inside "identity->mi_ginfo", make
sure "identity->mi_ginfo" is valid.

Change-Id: I880ccadddd93425c301115328d57c552d11bda8e
Signed-off-by: nasf <yong.fan@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1252
Tested-by: Hudson
Reviewed-by: Yu Jian <yujian@whamcloud.com>
Reviewed-by: hongchao.zhang <hongchao.zhang@whamcloud.com>
Tested-by: hongchao.zhang <hongchao.zhang@whamcloud.com>
Tested-by: Yu Jian <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-585 lower mdt reconstruct open debug message level

Reconstruct open is not an error case.

Change-Id: Ifb559616c148b01298196b4be7e398306a6b8e4c
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1233
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-325 using preallocated objects if OST has enough disk space

during creating files, if ENOSPC is returned by one OST, the preallocated
objects in the corresponding OSC should still can be used if there is enough
blocks but no inode available in the OST.

Change-Id: I9d57461698d1c0fb3a492188fb4ce1519d65efc7
Signed-off-by: Hongchao Zhang <hongchao.zhang@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1051
Tested-by: Hudson
Reviewed-by: Johann Lombardi <johann@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Fan Yong <yong.fan@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-586 1.8 <-> 2.1 interop: list_add corruption

In seq_client_alloc_fid(), client may list_add itself more than
once to waitqueue under race or error condition.

Signed-off-by: Lai Siyao <laisiyao@whamcloud.com>
Change-Id: Id7d35cc4ca3eafc4fc44d2d79a9ebc25fe56e62c
Reviewed-on: http://review.whamcloud.com/1216
Tested-by: Hudson
Reviewed-by: Johann Lombardi <johann@whamcloud.com>
Reviewed-by: Liang Zhen <liang@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-15 Strange slow IO messages and bad performance

b=24183 slow I/O on new files via mballoc.

upstream patch to avoid loading bitmaps from full groups

Change-Id: I9b4de1b4b1942b0f084b6199d5ab3e1267c9e8e3
Signed-off-by: Yang Sheng <ys@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/442
Tested-by: Hudson
Reviewed-by: Johann Lombardi <johann@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-544 speed up filesystem formatting time in conf-sanity.sh

Pass "-E lazy_itable_init" to mke2fs to speed up Lustre
filesystem formatting time in conf-sanity.sh.

Signed-off-by: Yu Jian <yujian@whamcloud.com>
Change-Id: I29070e6c4b95028b12573d552a32ca04189c2c1d
Reviewed-on: http://review.whamcloud.com/1210
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>

LU-455 Replace DIST_SOURCES with EXTRA_DIST

Resolve autoconf-2.63 warnings mainly by replacing DIST_SOURCES
with EXTRA_DIST.  Additionally, the non-portable $(shell) contruct
was removed from tests/module.mk and the sources simply enumated.
Finally, the incorrect EXTRA_SOURCES instance was removed from
the top level autoMakefile.am.

  Running automake-1.11 -a -c -W no-portability
  libcfs/libcfs/autoMakefile.am:92:
          variable `DIST_SOURCES' is defined but no program or
  libcfs/libcfs/autoMakefile.am:92:
          library has `DIST' as canonical name (possible typo)
  lnet/klnds/mxlnd/autoMakefile.am:44:
          variable `DIST_SOURCES' is defined but no program or
  lnet/klnds/mxlnd/autoMakefile.am:44:
          library has `DIST' as canonical name (possible typo)
  ...

Also, as a result of the above changes, libcfs/libcfs/autoMakefile.am had to be
modified in order to allow 'make dist' to succeed.

As it turns out, libcfsutils_a_SOURCES was incorrect. That list contained
references to nonexistent 'util/[parser|platform].h' files. The assumption is
it intended to reference libcfs/include/libcfs/util/[parser|platform].h.

To fix the issue, both [parser|platform].h references were removed from the
list. This produces a simple solution that maintains consistency with the rest
of the build system. The caveat being that libcfsutil.a won't automatically be
rebuilt if either of the intended [parser|platform].h files are modified.

Change-Id: Ia81eb1e3fc219f6dac4c7da234f7e736754c5440
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>
Signed-off-by: Prakash Surya <surya1@llnl.gov>
Reviewed-on: http://review.whamcloud.com/1092
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Richard Henwood <rhenwood@whamcloud.com>
Tested-by: Richard Henwood <rhenwood@whamcloud.com>
Reviewed-by: Brian J. Murrell <brian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-550 incorrect quota hash function

Quota code uses 'uid' or '(1 << 32) + gid' as the hash key, however,
the quota hash function lqs_hash() always try to interpret such a
unsigned long long key as a 'quota_adjust_qunit' struct, then calculate
the hash value by quota_adjust_qunit->qaq_id.

Signed-off-by: Niu Yawei <niu@whamcloud.com>
Change-Id: I48dc7f276eb97f34b22097b4edc529d5f76acf37
Reviewed-on: http://review.whamcloud.com/1162
Tested-by: Hudson
Reviewed-by: Fan Yong <yong.fan@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Johann Lombardi <johann@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-297 improve llverfs to display the current read/write performance

This patch changes show_filename() to show_rate() in llverfs.c to
display the current read/write file name and performance, along with
an estimate of how long the whole read/write operation will continue.

The patch also changes "fast" to "partial" and fixes some typos in
the comments of llverdev.c and llverfs.c.

Signed-off-by: Yu Jian <yujian@whamcloud.com>
Change-Id: I94cb112ae9a38b1128e53021aa0c1f1e8e5411af
Reviewed-on: http://review.whamcloud.com/1110
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-560 Fixup intent status on ENOENT in mdt_intent_reint

Clear -ENOENT from rep->lock_policy_res2 for non existent files so
that clients may cache negative dentries if they hold
MDS_INODELOCK_UPDATE on the parent directory.

Change-Id: I238f1001782923a595f9f7ee1c094d53ab1422ae
Signed-off-by: John Hammond <jhammond@tacc.utexas.edu>
Reviewed-on: http://review.whamcloud.com/1173
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Johann Lombardi <johann@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-574 Fix whitespace issues in conf-sanity.sh

Move "run_one test_41b" to immediately follow declaration of test_41b.
Fix some whitespace problems (space before tab, space at end of line)
that I noticed while inspecting LU-574.

Change-Id: I18a80a55c9895cf8a6647a01ffa646f5ca499795
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1200
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-321 Don't assume file system name for some test

Some of the test assume the file system name is lustre
which is not always the case.

Change-Id: I582aa81fce5d1a881837dfa28bb0d6d84cfb5547
Signed-off-by: James Simmons <uja.ornl@gmail.com>
Reviewed-on: http://review.whamcloud.com/253
Tested-by: Hudson
Reviewed-by: Brian J. Murrell <brian@whamcloud.com>
Reviewed-by: Yu Jian <yujian@whamcloud.com>
Tested-by: Yu Jian <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-556: Fix config_log_find() in mgc_request.c

This issue was imported by lu-411. Actually after checking
the code, I found we can make things a bit simpler.

Change-Id: Iad212782a5f0357b73014baf7766c77637fb47db
Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1168
Tested-by: Hudson
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mikhail Pershin <tappro@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>

LU-292 srv_n_queued_reqs is screwed up by ptlrpc_unregister_service

service::srv_n_queued_reqs is counter for pre-unpack request on
service::srv_req_in_queue, we shouldn't change it for unpacked request
returned by ptlrpc_server_request_get()

Change-Id: I770411fa8f986fa387a1790f87e6a265abc469f1
Signed-off-by: Liang Zhen <liang@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1172
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-553 add checkpatch.pl for Lustre coding style

Add a modified version of the Linux kernel checkpatch.pl script (from 2.6.38)
to Lustre for validating patches against the Lustre coding style. The Lustre
coding style is largely the same as the kernel, except using spaces instead of
tabs for indenting.

Change-Id: I2d116c6769f9211825dc8ddaead05914f2116c60
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1164
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-559 Keep the o_seq unchanged for 1.8 client

ost_validate_obdo() arbitrarily change the o_seq to 0 for the requests
from 1.8 client, which will make the 1.8 echo client use the 0 group
mistakenly. In fact, I think we needn't do anything special for the 1.8
client here.

Signed-off-by: Niu Yawei <niu@whamcloud.com>
Change-Id: I93cd3bdbf984f8aef0f5b68f5ee8640f31ded337
Reviewed-on: http://review.whamcloud.com/1182
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: wangdi <di.wang@whamcloud.com>

LU-540 misc patch for user identity upcall/downcall

1) l_getidentity forgets to allocate supplementary group info space, but access
   such non-allocated memoey without checking.
2) In structure "identity_downcall_data", "idd_gid" is same as "idd_groups[0]",
   remove such redundant information.
3) More sanity check for lprocfs_wr_identity_info().
4) Drop unnecessary memory allocation to simplify mdt_identity_do_upcall().
5) Add padding field in "identity_downcall_data" to resolve align issues.
6) Use new magic to distinguish old version/format downcall.

Change-Id: Ib0dfb37c39a7cb335e29557b68b885afa726789e
Signed-off-by: nasf <yong.fan@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1160
Tested-by: Hudson
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-540 quota: add compatibility for 1.8 userspace

The LL_IOC_QUOTACTL ioctl data structure if_quotactl changed size
in 2.x by adding new fields in the middle of the structure.  Add
compatibility for the old 1.8 quotactl userspace ioctl so that it
is possible for existing quotactl tools to run on both 1.x and 2.x.

This requires renumbering the LL_IOC_QUOTACTL ioctl, since it had
incorrectly used "struct if_quotactl *" as the "size" parameter,
but the _pointer_ size did not change.  ioctl() values should always
take a struct as the size parameter, not a pointer to the struct,
so that issues like this can be caught earlier.

Change-Id: I0bce98327a276c0c9a49ce78efd3f45e00df7168
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1152
Tested-by: Hudson
Reviewed-by: Fan Yong <yong.fan@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-533 Include posix-types.h in headers and distribution

Include libcfs/posix/posix-types.h in the main rpm, and
add #include statements in lustre_user.h and ll_fiemap.h.
This will allow user-space applications that include these
files to compile.

Change-Id: Ibe391bf28afe91b3914ca1eb07fe66e408b21b4a
Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>
Reviewed-on: http://review.whamcloud.com/1138
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-429 Fix and quiet debug message in filter_connect

The comment refers to mds_connect(), which no longer exits.  So I
just removed the comment.

The message says "Received MDS connection", but this function is
used when clients connect to OSTs as well.  We don't need to see
this message on the console all of the time, so I changed it to
D_INFO.

Change-Id: Iebfbaafe39df702862ade126b979528855173d5c
Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>
Signed-off-by: Yang Sheng <ys@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/963
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-350 fix bug in mgs to allow starting OST before MDT

in mgs_steal_llog_for_mdt_from_client, it forgets to add the "index" and
"svname" of the existing OST to the config log of the MDT's lov, then cause
Lustre fails to mount if the OSTs start before MDT.

Change-Id: I0e68bf51c88517dbd42790e7e4447a186cf268f3
Signed-off-by: Hongchao Zhang <hongchao.zhang@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/595
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Fan Yong <yong.fan@whamcloud.com>
Reviewed-by: Yu Jian <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-333: Make read_bytes in llite/*/stats report bytes read.

Move ll_stats_ops_tally(..., LPROC_LL_{READ,WRITE}_BYTES, ...) from
vvp_io_init() to the end of ll_file_io_generic(), and tally by the
actual number of bytes read rather than the number requested.

Test 127b, client stats are correctly tallied with this patch.

Change-Id: Icad0bc09630b8c1775f2ec59326902aa88215498
Signed-off-by: John Hammond <jhammond@tacc.utexas.edu>
Signed-off-by: Richard Henwood <rhenwood@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1094
Tested-by: Hudson
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-487 brw_[read|write] stats to correctly counts number of pages.

brw_[read|write] now correctly counts the number of pages. Before
this change, the number of pages multiplied by the page size was
counted.

Change-Id: I069c18cf46c67285de1ae2afc48f7fb4ab92df3c
Signed-off-by: Richard Henwood <rhenwood@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1056
Tested-by: Hudson
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-530 group quotas not enforced

In filter_quota_getflag(), we shouldn't break the loop when the file's owner
doesn't have quota limit, otherwise, the check for file's group will be skipped
and the out of quota flag will not be returned to client at the end.

Signed-off-by: Niu Yawei <niu@whamcloud.com>
Change-Id: I3e92ba4c061409d5a911d46916535c71f7843e0b
Reviewed-on: http://review.whamcloud.com/1156
Tested-by: Hudson
Reviewed-by: Fan Yong <yong.fan@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Johann Lombardi <johann@whamcloud.com>

LU-541 Fix liblustreapi.c usage of errno and rc.

* fix liblustreapi.c usage of errno and rc, llapi_xxx() functions should
  return negative value on error.
* enable sanity subtest 65.

Change-Id: I30d172c49b60f6f77ea57c3082d0c68f4c141dea
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1155
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: hongchao.zhang <hongchao.zhang@whamcloud.com>
Tested-by: hongchao.zhang <hongchao.zhang@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-540 remove file flags definition conflict between 1.8 and 2.x

There is file flags definition conflict between 1.8 "LL_FILE_LOCKED_DIRECTIO"
and 2.x "LL_FILE_RMTACL". Redefine the later one to erase such conflict.

Change-Id: I1f4b96382f865f8548bf1a54fb1c74d3649b0daf
Signed-off-by: nasf <yong.fan@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1159
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Johann Lombardi <johann@whamcloud.com>

LU-451 Workaround for lockd (nfsv3) locking

It seems that lockd works in a different way lock-wise compared
to everybody else. They actually don't use lock owner to determine
conflicts and install their own owner compare that compares specially
crafted pids (duh!).
As a workaround if a custom owner compare is detected, assign pid to
owner field (not intersecting with kernel pointer space anyway).
svclock is the only place using custom owner compare function ATM.

Change-Id: Ia89e2327705f2afbe200fdbadd0f86038fda936d
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1174
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Johann Lombardi <johann@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>

LU-290 Reconnects are not throttled

bz22423
Client should not try to reconnect to the same nid in a busy loop,
but instead rely on pinger to issue reconnect.

Signed-off-by: Lai Siyao <laisiyao@whamcloud.com>
Change-Id: I878093030d62173071192ca816e037464cc9248d
Reviewed-on: http://review.whamcloud.com/944
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Johann Lombardi <johann@whamcloud.com>
Reviewed-by: hongchao.zhang <hongchao.zhang@whamcloud.com>
Tested-by: hongchao.zhang <hongchao.zhang@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-407 Fix lustre-modules rpm name in spec file.

RHEL and Fedora kernels include ".%{_arch}" in the kernel's version
string.  Since Lustre includes the kernel version in its rpms, it
can appear that a double architecture name appears at the end.  Really
one is from the kernel, and one is from Lustre.  Trying to change that
is probably more trouble than it is worth.

This seems to cause problems with rpm's understanding of rpm names.  To
fix that this patch takes Andreas's suggestion of grep'ing the full
rpm name from "rpm -q kernel-modules".

We also add "|| true" to the last line of the %preun scriptlet so that
failure does not aport installation.

Finally, we also remove the redundant greps in the postun scriptlet.

Change-Id: I2c71e853e28ec6e0907eb4ea7c3205ca6e5dd873
Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>
Reviewed-on: http://review.whamcloud.com/934
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Brian J. Murrell <brian@whamcloud.com>