LU-8397 mgc: add comma-separated nids for primary MGS

In lustre_start_mgc(), if the primary MGS has multiple nids separated by
commas, then only the first nid is added, and the other nids are not added.
This patch fixes the above issue by adding the nids until the end of the
primary MGS nid list or hitting the first failover MGS nid.

Test-Parameters: combinedmdsmgs=false envdefinitions=ONLY=77 testlist=conf-sanity
Test-Parameters: envdefinitions=ONLY=77 testlist=conf-sanity

Signed-off-by: Vladimir Saveliev <vladimir.saveliev@seagate.com>
Signed-off-by: Jian Yu <jian.yu@intel.com>
Seagate-bug-id: MRP-2930
Change-Id: I3e84786bfc08767c75a133affb4f86325d789d6e
Reviewed-on: https://review.whamcloud.com/21308
Tested-by: Jenkins
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9354 utils: fix lfs.c compile for PPC

Fix the lfs_setstripe_args parameter to use "unsigned long long"
for lsa_comp_end so that when passed to llapi_parse_size() it is
the correct type.  PPC annoyingly uses "unsigned long" for __u64
while most other architectures use "unsigned long long".

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I27b724d5eab5f4669bde1f5fb44c75a1051ba7bd
Reviewed-on: https://review.whamcloud.com/26727
Tested-by: Jenkins
Reviewed-by: Minh Diep <minh.diep@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9311 pfl: shouldn't reprocess done/no-op resent request

When the LOVEA buffer is bigger than the request reply buffer, the
client will resend the layout write intent RPC, and
mdt_layout_change() should not reprocess it since the 2nd process
will try to cancel the 1st granted CR lock, while client has not
get it granted yet because of the reply buffer shortage resend the
RPC.

There is another layout change resent case: the client's job has been
done by another client, referring lod_declare_layout_change -EALREADY
case, and it became a operation w/o transaction, so we should not do
the layout change, otherwise mdt_layout_change() will try to cancel
the granted server CR lock whose remote counterpart is still in hold
on the client, and a deadlock ensues.

This patch also adjusts some debug messages, makes dump_lsm() dump
uninstantiated component stripe info.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I9b063ee54d57c233eca3250502a2707997892898
Reviewed-on: https://review.whamcloud.com/26474
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Tested-by: Jenkins
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9324 test: get layout parameters

Add test framework functions get_layout_param()/parse_layout_param
to get plain/PFL file/dir layout parameters, so that the layout
parameters can be used later.

Test-Parameters: trivial testlist=sanity-pfl
Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I4f43a5b4d23e353660114b7ff299a97afc405a07
Reviewed-on: https://review.whamcloud.com/26578
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-9168 spec: New lustre-resource-agents rpm

This rpm bundles up all resource agents for Lustre into a single RPM.

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Ifc4422f11e9d6b4c1e03ef85fc78e35d6fcb0316
Reviewed-on: https://review.whamcloud.com/26229
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Zhiqi Tao <zhiqi.tao@intel.com>
Reviewed-by: Gabriele Paciucci <gabriele.paciucci@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-8840 osp: handle EA cache properly (2)

For success case, dt_xattr_get() should return the EA size
instead of zero. If such EA does not exist, return -ENODATA.

More code cleanup for OSP EA cache to avoid potential reference
leak, buffer overflow, and so on.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I85d7c7a2cafd50334f2ea0634f5e2b21c0b3908e
Reviewed-on: https://review.whamcloud.com/25207
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-2155 utils: Enable ldiskfs mmp on tunefs failover add

Enable Multi-Mount Protection in ldiskfs when adding a failover peer
via tunefs.  MMP is enabled in mkfs if failover is configured
initially, but it wasn't if failover was added later.

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Ie21ef5324de240afe0fd760cb4a9b3e1b4165064
Reviewed-on: https://review.whamcloud.com/26758
Tested-by: Jenkins
Reviewed-by: Zhiqi Tao <zhiqi.tao@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-8998 utils: allow "--component-*" to be abbreviated

Allow "--component-foo" arguments to be abbreviated to "--comp-foo"
to simplify userspace, especially for options like "--comp-flags"
that do not have a short option.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I8c434dd68c042efefc2df93d0f2d7b9bc33ebbe5
Reviewed-on: https://review.whamcloud.com/26699
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9351 pfl: setstripe to existing file

It's ok to setstripe to an existing file when the file has no
LOVEA created yet, so we should not use exlusive open for the
'lfs setstripe'.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I2090ba2940b391f7853cc06ccdb9cd842ad6f984
Reviewed-on: https://review.whamcloud.com/26655
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>

LU-9335 pfl: calculate PFL file LOVEA correctly

PFL file could contain uninstantiated component, so it could still
keeps the specified -1 stripe count,
lov_mds_md_size()/lov_user_md_size() should heed this case,
otherwise its LOVEA size could be errneous big.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: Ied4bf4531f0b0ac9fdefc9efef3c97ae5ae449f4
Reviewed-on: https://review.whamcloud.com/26597
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-8922 lod: check master stripes properly

When creating striped directory, it should
first check if the stripe has been created
on the current MDT, otherwise it might create
duplicate stripes on the master MDT, especially
when one MDT is deactived, and specified stripes
is larger than the active MDTs.

Test-Parameters: envdefinitions=ONLY=50 testlist=conf-sanity,conf-sanity,conf-sanity
Test-Parameters: envdefinitions=ONLY=50 testlist=conf-sanity,conf-sanity,conf-sanity
Signed-off-by: Di Wang <di.wang@intel.com>
Change-Id: Id3bba2817c4c7c9584f9129e32555f0f676b3364
Reviewed-on: https://review.whamcloud.com/24776
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5718 o2iblnd: multiple sges for work request

In current protocol, lnet router cannot align buffer for rdma,
o2iblnd may run into "too fragmented RDMA" issue while routing
non-page-aligned IO larger than 512K, because each page will
be splited into two fragments by kiblnd_init_rdma().

With this patch, o2iblnd can have multiple sges for each work
request, and combine multiple remote fragments of the same page
into one work request to resovle the "too fragmented RDMA" issue.

Signed-off-by: Liang Zhen <liang.zhen@intel.com>
Change-Id: Id57a74dc92801b012956ab785233aa87cac14263
Reviewed-on: https://review.whamcloud.com/12451
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Amir Shehata <amir.shehata@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9334 lfsck: object leak in lfsck_load_one_trace_file

In lfsck_load_one_trace_file(), if we successfully load or
create the object via local_index_find_or_create(), but the
subsequent do_index_try() failed, then we need to release
such object, otherwise it will be left there.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Ic3c7db9239e0d10a5cf6fc2254a4c414f4cd007f
Reviewed-on: https://review.whamcloud.com/26703
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-1032 build: fix typo in lustre-dkms.spec changelog

Fix typo in the lustre-dkms.spec.in ChangeLog which has an
invalid date, and causes a build warning.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I6029c93928f84ff59c8798d2f8115e872c4cab07
Reviewed-on: https://review.whamcloud.com/26358
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9384 tests: skip 2.9 filesystem images on 2.10

The 2.9 disk images upgraded to 2.10 are causing common test failures
due to small differences in space usage.  Disable testing these images
until a proper fix is available.

Test-Parameters: trivial envdefinitions=ONLY=32 testlist=conf-sanity
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I64d14ac6b079c50ee234ff886c9a80d9213ebbe5
Reviewed-on: https://review.whamcloud.com/26789
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-6283 ptlrpc: Implement NRS Delay Policy

The NRS Delay policy seeks to perturb the timing of request processing
at the PtlRPC layer, with the goal of simulating high server load, and
finding and exposing timing related problems. When this policy is
active, upon arrival of a request the policy will calculate an offset,
within a defined, user-configurable range, from the request arrival
time, to determine a time after which the request should be handled.
The request is then stored using the cfs_binheap implementation,
which sorts the request according to the assigned start time.
Requests are removed from the binheap for handling once their start
time has been passed.

The behavior of the policy can be controlled via three proc files
which can be written to via lctl similar to other policies.

nrs_delay_min: Controls the minimum amount of time, in seconds, that a
request will be delayed by this policy. The default is 5 seconds.

nrs_delay_max: Controls the maximum amount of time, in seconds, that a
request will be delayed by this policy. The default is 300 seconds.

nrs_delay_pct: Control the percentage of requests that will be delayed
by this policy. The default is 100. Note, when a request is not
selected for handling by the delay policy due to this variable then
the request will be handled by whatever fallback policy is defined
for that service. If no other fallback policy is defined then the
request will be handled by the FIFO policy.

Some examples:

lctl set_param *.*.*.nrs_delay_min=reg_delay_min:5, to set the regular
request minimum delay on all PtlRPC services to 5 seconds.

lctl set_param *.*.*.nrs_delay_min=hp_delay_min:2, to set the
high-priority request minimum delay on all PtlRPC services to 2
seconds.

lctl set_param *.*.ost_io.nrs_delay_min=8, to set both the regular and
high-priority request minimum delay of the ost_io service to 8
seconds.

lctl set_param *.*.*.nrs_delay_max=reg_delay_max:20, to set the
regular request maximum delay on all PtlRPC services to 20 seconds.

lctl set_param *.*.*.nrs_delay_max=hp_delay_max:10, to set the
high-priority request maximum delay on all PtlRPC services to 10
seconds.

lctl set_param *.*.ost_io.nrs_delay_max=35, to set both the regular
and high-priority request maximum delay of the ost_io service to 35
seconds.

lctl set_param *.*.*.nrs_delay_pct=reg_delay_pct:5, to delay 5
percent of regular requests on all PtlRPC services.

lctl set_param *.*.*.nrs_delay_pct=hp_delay_pct:2, to delay 2 percent
of high-priority requests on all PtlRPC services.

lctl set_param *.*.ost_io.nrs_delay_pct=8, to delay 8 percent of both
regular and high-priority requests of the ost_io service.

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: Iab50a639900adf31893c7b1fe83658932fd59db1
Reviewed-on: https://review.whamcloud.com/14701
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Henri Doreau <henri.doreau@cea.fr>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9319 statahead: skip agl for the file in restoring

In case of restore, the MDT has the right size and has already sent
it back without granting the layout lock, inode is up-to-date. Then
AGL (async glimpse lock) is useless.

Also to glimpse we need the layout, in case of a running restore the
MDT holds the layout lock so the glimpse will block up to the end of
restore (statahead/agl will block).

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Iaaf138a28671c3eccfb05b08ce8d7364423256a1
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-on: https://review.whamcloud.com/26501
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9332 tests: fix pool_list issue in conf-sanity test_82b

In conf-sanity.sh test_82b, if we use separated MGS and MDS nodes,
MDS needs more time to list the OSTs in pool, so use wait_update
instead of the pool_list command.

Signed-off-by: Emoly Liu <emoly.liu@intel.com>
Change-Id: I8f7281d44a12731d7ae2d1f8fa9bad163086cecc
Reviewed-on: https://review.whamcloud.com/26582
Tested-by: Jenkins
Reviewed-by: Parinay Kondekar <parinay.kondekar@seagate.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9086 gss: handle specific mgc binding

Recent work has made mgc config log handle more than one MGC
existing on a client node which is the case when more than
one lustre file system is mounted. This broke sptlrpc
handling due to cfg_instance no longer being NULL. Since
this is the case we remove the cfg_instance NULL test in
class_config_llog_handler.

The second problem exposed was due to cfg_obdname changing from
the file system name to the actual mgc abstract client device.
Since target2fsname() was only checking fsname-[MDT|OST]XXXX we
could use this function with the obd_uuid field which has the
format of fsname-[MDT|OST]XXXX_UUID. For the case of obd names
with the format fsname-XXXXXXX those obd device have an obd_uuid
that is just some random long hexadecimal string. To handle the
obd_uuid case use the proper obd_uuid2fsname() function instead
of target2fsname(). Rename target2fsname() to obdname2fsname()
since this what it now does. Add handling the fsname-XXXXXXXX
obd name format to obdname2fsname().

Change-Id: I6c3235b3e2d41ef955b98d30aec76d47346c2369
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/25959
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-8458 pacemaker: Resource to manage Lustre Target

A new script to monitor health and mount/umount lustre target.

pcs resource create [Resource Name] ocf:lustre:Lustre
target=/dev/mapper/mpatha mountpoint=/mnt/MGS

This script should be located in /usr/lib/ocf/resource.d/lustre/
of both the Lustre servers with permission 755.

Test-Parameters: trivial
Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I1cfa42604af3c777c1b8d6ed344e22ceccfceb62
Reviewed-on: https://review.whamcloud.com/25664
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Malcolm Cowe <malcolm.j.cowe@intel.com>
Reviewed-by: Zhiqi Tao <zhiqi.tao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9083: Add scripts to collect and analyze Lustre RPC traces

A set of scripts to collect and analyze RPC traces in a Lustre file system.
These scripts turn the Lustre debug-daemon on and off, before and after
launching a job respectively. They also allow users to parse and visualize
the logged RPC traces.

Test-Parameters: trivial
Change-Id: I6ee598ff6e49b2a7406354172b10bd295e801cb0
Signed-off-by: CongXu <cong.xu@intel.com>
Reviewed-on: https://review.whamcloud.com/25395
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Omkar Kulkarni <omkar.kulkarni@intel.com>

LU-9075 mdt: avoid race causing mdt_coordinator_cb() err msgs

This patch mainly moves mdt_agent_record_update() call before
mdt_cdt_remove_request() in mdt_hsm_update_request_state(), to
avoid the frequent couple of
"(mdt_coordinator.c:1473:mdt_hsm_update_request_state()) ...
Cannot find running request for cookie ..."
and
"(mdt_coordinator.c:339:mdt_coordinator_cb()) ...
cannot cleanup timed out request ..."
error msgs, likely to concern active requests that have completed
and thus that have already been removed from memory in
mdt_hsm_update_request_state() (using mdt_cdt_remove_request() and
in the context of a MDT thread handling CT's MDS_HSM_PROGRESS
requests), but the corresponding action LLOG record update is stuck
awaiting for CDT to give-back cdt_llog_lock in
mdt_agent_record_update().

Others related but minor changes are, use of arr_req_change instead of
arr_req_create to more accuratelly determine if a request exceeds the
timeout, and change main debug msg in mdt_hsm_update_request_state()
to reflect if action LLOG record update will occur or not.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: I043813f1ff11a7e9e99c534fa8560a35e2c52543
Reviewed-on: https://review.whamcloud.com/25243
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Henri Doreau <henri.doreau@cea.fr>
Reviewed-by: Quentin Bouget <quentin.bouget@cea.fr>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-8641 tests: tests: speedup metabench

With huge number of nfiles created by metabench
rm -rf called at the end of run_metabench() takes a very long
time.
Sine metabench cleanup is also very slow, metabench
execution without -k option is useless to use also.
Patch adds mbench_CLEANUP mode, default value is true.
When set to false -- metabench testdir is just renamed to avoid
files/dirs deletion.

Seagate-bug-id: MRP-3454
Signed-off-by: Elena Gryaznova <elena.gryaznova@seagate.com>
Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@seagate.com>
Reviewed-by: Vitaly Fertman <vitaly.fertman@seagate.com>
Change-Id: Iaa130785d27dc3a87f642723fcaad6dc5e748440
Reviewed-on: https://review.whamcloud.com/22852
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-8832 tests: fix module loading for Parallels

It isn't possible to insmod modules directly from Parallels prl_fs
mounting a host filesystem when testing directly from a build tree
(insmod from an RPM installed in the root image filesystem works).
Instead, copy the locally-built module into /tmp before loading it.

Use load_module() in sanity.sh test_180b and run-llog.sh.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: Id0123059e8151e2a78086579150f04b1b05cab07
Reviewed-on: https://review.whamcloud.com/25660
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9280 osd-zfs: don't mark existing on failed creation

In osd_object_create(), existing flag shouldn't be set on failure.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I77dd96f0e3c562c6bef659dc0af405a1ce488805
Reviewed-on: https://review.whamcloud.com/26617
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

New tag 2.9.56

Change-Id: I20adf1083b0c081ce818941c87b40f64631ad8e4
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>

LU-8333 test: Add replay-dual 21b to ALWAYS_EXCEPT

Replay-dual's subtest test_21b has been failing more than half the time
(since June 2016) with a COS check failure.  Finding a fix for this
is taking longer than expected.  We would like to except this subtest
until a better COS check test can be developed.  This change only
affects the replay-dual.sh test set script.

Test-Parameters: trivial testlist=replay-dual

Change-Id: I575342ae5b6633efc198383b3351b4d9cfc1fbf0
Signed-off-by: Jim Casper <jamesx.casper@intel.com>
Reviewed-on: https://review.whamcloud.com/26631
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Saurabh Tandan <saurabh.tandan@intel.com>
Reviewed-by: Wei Liu <wei3.liu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9320 test: Skip sanity test_80[12]*

Skip sanity test_80[1-2]* if MDS or OSS version is before
2.9.55

Skipping following tests: test_801a, test_801b, test_801c,
test_802

Test-Parameters: trivial testlist=sanity,sanity
Signed-off-by: Saurabh Tandan <saurabh.tandan@intel.com>
Change-Id: I98ace61c312466381ffcc406644276d86faac782
Reviewed-on: https://review.whamcloud.com/26616
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>

LU-9383 test: use a subdir in sanityn test_71b()

In sanityn test_71b() use a test specific subdir to avoid changing the
default file striping on the filesystem root.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ic3bfbf24a9f6d00f07392014b48af0c5062756f6
Reviewed-on: https://review.whamcloud.com/26771
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>

LU-9318 test: sanity-lsnapshot skipped for all non-zfs configs

The problem with the script was that it tried to check
for ZFS file system after the script tried to add snapshot
to the debug list.

As part of solution the script just needs to check for ZFS
file system or the version is too old at the start for of
the script.

Test-Parameters: trivial testlist=sanity-lsnapshot,sanity-lsnapshot,sanity-lsnapshot
Signed-off-by: Saurabh Tandan <saurabh.tandan@intel.com>
Change-Id: Ie59dac7543d1e35b924c84d19c34a5ec5b903a14
Reviewed-on: https://review.whamcloud.com/26600
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-9271 tests: remove unnecessary sanity-scrub test delay

In sanity-scrub test_5, we want to check the OI scrub state machine.
For that, we inject some failure stub to delay the test for checking
status. But sometimes, we want to check the status within some short
time that may be not enough because of the injected test delay. This
patch removes one unnecessary status checking since it has been done
in former check. On the other hand, the subsequent checking also can
indicate the correctness of the skipped status.

Test-parameters: trivial

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I14b5096db3d7ca255f3de1a8853dde85ead54215
Reviewed-on: https://review.whamcloud.com/26457
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-9314 osd: cleaup qsd once commit callbacks are done

otherwise osd_trans_commit_cb() calling qsd_op_end() may
find qsd structure released.

Change-Id: Ifd64b17d636f5e728ed6a4ad1a2d4d43998c90ec
Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-on: https://review.whamcloud.com/26496
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9287  kernel: kernel update [SLES11 SP4 3.0.101-97]

Update SLES11 SP4 kernel to 3.0.101-97

Test-Parameters: mdsdistro=sles11sp4 ossdistro=sles11sp4 \
  clientdistro=sles11sp4 mdtfilesystemtype=ldiskfs \
  ostfilesystemtype=ldiskfs testgroup=review-ldiskfs

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I5214114b5c0bb0bfe5a68dbe8cf2a1f26705d352
Reviewed-on: https://review.whamcloud.com/26336
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9275 kernel: kernel update [SLES12 SP2 4.4.49-92.14]

Update target and kernel_config files for new version

Test-Parameters: clientdistro=sles12sp2 testgroup=review-ldiskfs \
  mdsdistro=sles12sp2 ossdistro=sles12sp2 \
  mdtfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I1d9c109904a61989b8eeb82c21ab99154db464cf
Reviewed-on: https://review.whamcloud.com/26263
Tested-by: Jenkins
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-930 doc: update LFSCK documentation

Update the LFSCK usage document to include information about logging
LFSCK changes to the console or separate log file.

Improve the description of some options, putting more commonly-used
options first.

Update URLs to reference lustre.org instead of other websites.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I980a43143297400c0a544281b716c32881d65172
Reviewed-on: https://review.whamcloud.com/26253
Tested-by: Jenkins
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Joseph Gmitter <joseph.gmitter@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9259 tests: set fail_loc on the right MDT

In sanity test_17o, we need to set the fail_loc on the MDT on which
the target file resides. So "getstripe -M" should be done after the
file created. In further client side ldlm locks should be cancelled
before failover the MDT.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I35acefb6b7a361f25593843e1f2a097e786e5f11
Reviewed-on: https://review.whamcloud.com/26225
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-9166 mdt: Add trailing newlines to proc files

Adds trailing newlines to two proc files.  A new sanity test ensures
that all non-empty readable proc files end with a newline.

Test-Parameters: trivial
Signed-off-by: Steve Guminski <stephenx.guminski@intel.com>
Change-Id: Id65dd5a43edc41be9f572d476b38975f0144fbaf
Reviewed-on: https://review.whamcloud.com/25977
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9076 ptlrpc: Prevent possible dereference of NULL pointers

Check pointers for NULL before passing to other functions.

Signed-off-by: Steve Guminski <stephenx.guminski@intel.com>
Change-Id: I5ad1e6675e9c5e41aee2dc59045ed67e9da5ea0d
Reviewed-on: https://review.whamcloud.com/25909
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9186 tests: fix sanityn test_76 and related code

Fix sanityn.sh test_76 to skip already open files when checking
whether mdt.*.exports.*.open_files accurately lists open files.
Otherwise, if some previous test leaves a file descriptor open
it will cause this test to fail spuriously.

Limit the number of open files to 256*MDSCOUNT rather than 2048,
so the test runs faster (80s vs. 215s in my VM with 2 MDTs).

Properly close the open shell FDs using "FD<&-" rather than
redirecting them to /dev/null, otherwise they stay open forever.

Print a '.' only once per 32 files opened/closed.

List open files in *.open_files in the order they were opened,
rather than reverse order, since this makes it easier to see which
files have been open longer.

Print out the pathname of the open files being skipped.

Don't explicitly mount $MOUNT2 in sanityn.sh.  This is already
handled inside check_and_setup_lustre() at the start of sanityn.sh.

Improve free_fd() to accept a "last_fd" argument to avoid searching
all open FDs on each call if it is known the fds are not closing.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I274b6db96f49a52c086f836c005558ef4b3ebbe5
Reviewed-on: https://review.whamcloud.com/25847
Tested-by: Jenkins
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Ashish Purkar <ashish.purkar@seagate.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9184 grants: move grant code to the target

Make grants code common for MDT and OST.
- move grants field from OFD to the lu_target
- same for statfs
- make ofd_grants.c independent from OFD
- move ofd_grants.c to the tgt_grants.c

This work was done for Data-on-MDT feature but it
is needed also for better space control on MDT as
well, e.g. for lfsck, HSM, etc.

Signed-off-by: Mikhal Pershin <mike.pershin@intel.com>
Change-Id: I43930b32b271c9e850aa2783795e7cce12830cd5
Reviewed-on: https://review.whamcloud.com/25820
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9183 libcfs: handle dump_trace() and related callbacks removal

In commit c8fe4609827aedc9c4b45de80e7cdc8ccfa8541b all previous users
of dump_trace() have been converted to use the new unwind interfaces,
so the dump_trace() and the related print_context_stack() and
print_context_stack_bp() callback functions were removed.

Test-Parameters: trivial
Change-Id: Ifa7a112d622b23f733d6daab05f9838afdf31a86
Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-on: https://review.whamcloud.com/25816
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3434 scripts: check for mdsfilesystemtype= option

Check if the Test-Parameters line is using the old mdsfilesystemtype=
option and ask for it to be removed.  The mdtfilesystemtype= option
should be used instead.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I52765fd94f4dec896593c86763737fa64f3ebbe5
Reviewed-on: https://review.whamcloud.com/25804
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Charlie Olmstead <charlie.olmstead@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9171 osc: GPF while doing ELC with no_wait_policy

osc_ldlm_weigh_ast() uses osc_object from ldlm_lock
without taking a reference.

It should take a reference like osc_ldlm_glimpse_ast()/
osc_dlm_blocking_ast0() does.

Change-Id: I33a817e4c08c2955f5395b13ffe8cd11f3bdf691
Seagate-bug-id: MRP-4179
Signed-off-by: Andriy Skulysh <andriy.skulysh@seagate.com>
Reviewed-on: https://review.whamcloud.com/25700
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7502 utils: add --mdt-count & --mdt-hash options

    lfs find --mdt-count and --mdt-hash options
    lfs getdirstripe --mdt-count and --mdt-hash options

Signed-off-by: Yang Sheng <yang.sheng@intel.com>
Change-Id: I55f2d7660c4e4fea5a197a57407b614c4004974c
Reviewed-on: https://review.whamcloud.com/25515
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4423 osc: use 64-bit timestamps in debugfs output

Some lustre debugfs files contain the current time. Make sure
we use a format here that does not overflow in 2038.

Linux-commit: 86655400c6e74751a73be82ea40d5ae6979c3225

Change-Id: I8a79e60bd15ce7c162b30025ca3b572fec2ee58f
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: Oleg Drokin <green@linuxhacker.ru>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/25407
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-8603 test: Separate tests in metadata_updates

The metadata-updates script is one continuous piece of code
that runs three different tests. These tests need to be broken
out into sub-tests and some of the comments echoed to the
console should be logged.

Test-Parameters: trivial testlist=metadata-updates
Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: I7e420ee8e0e02d2c2a66ac644301bb457a8b2256
Reviewed-on: https://review.whamcloud.com/24450
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-8703 libcfs: use int type for CPT identification.

This patch bring the lustre CPT code into alignment with the Linux kernel.

 * use int type for CPT identification like CPU identification.
 * fix coding style.

Change-Id: I5df33f37757eb6d47ab4e824ec1b800a40730f61
Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-on: https://review.whamcloud.com/23304
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7819 tests: fix checkfiemap for stripecount > 1

If sanityn test_71 is ran with stripecount > 1, it fails
with "error while ioctl 95" i.e. EOPNOTSUPP.

FIEMAP_FLAG_DEVICE_ORDER flag must be set while issuing
FS_IOC_FIEMAP ioctl.

New test case 71b to check fiemap for stripe count > 1
Renamed original test_71 to 71a

Test-Parameters: trivial testlist=sanityn envdefinitions=ONLY=71
Seagate-bug-id: MRP-2858
Signed-off-by: Ashish Purkar <ashish.purkar@seagate.com>
Change-Id: If62370fbcfd586607e0f0da2b348070641bed0d1
Reviewed-by: Artem Blagodarenko <artem.blagodarenko@seagate.com>
Reviewed-by: Ujjwal Lanjewar <ujjwal.lanjewar@seagate.com>
Tested-by: Parinay Vijayprakash Kondekar <parinay.kondekar@seagate.com>
Reviewed-on: https://review.whamcloud.com/18679
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>

LU-7529 test: fix tiny problems of tests in conf-sanity.sh

1)test_68:
Mount MGS in the end of test if MGS and MDS are seperated.
2)test_75:
Reformat all devices otherwise 76a might fail.

Change-Id: Iee8a5fa1c6acccbae86cdcbe14c8715a71b0ae2b
Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: teddy <jjkky@yahoo.com>
Reviewed-on: https://review.whamcloud.com/17507
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-9307 lov: NULL pointer deref in lov_delete_composite

When lov_init_composite() cannot allocate memory and bails out,
lov_delete_composite() is then called, and it should check the
composite entry's existence before trying to delete each component.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I013c596a19a114b763f3eebbaafa1559fa09d8d9
Reviewed-on: https://review.whamcloud.com/26456
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9019 quota : migrate to 64 bit time

Replace cfs_time_current_sec() to avoid the overflow
issues in 2038 with ktime_get_real_seconds(). The
quota uses the libcfs 64 bit time abstraction it
developed before the linux kernel has created its
own. Replace all the libcfs 64 bit time handle with
generic time64_t handling.

Change-Id: Ibf9475c7ba8312561a57d48516ffa9b0b3ae4112
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/26452
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9286 ptlrpc: fix wrong error handlers

If ptlrpc_request_pack finish with 0, we need to call
ptlrpc_req_finished for later errors.

Signed-off-by: Alexander Boyko <alexander.boyko@seagate.com>
Seagate-bug-id: MRP-4285
Change-Id: I97dec565c1aa95c4457b32cb5d72b4f8abdd3d1d
Reviewed-on: https://review.whamcloud.com/26319
Tested-by: Jenkins
Reviewed-by: Chris Horn <hornc@cray.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9207 test: Add new disk image for 2.9.0

Add new disk image for 2.9.0 of both ldiskfs and zfs
for conf-sanity test_32

Test-Parameters: trivial envdefinitions=ONLY=32 testlist=conf-sanity
Test-Parameters: trivial envdefinitions=ONLY=32 testlist=conf-sanity \
mdtfilesystemtype=zfs ostfilesystemtype=zfs

Change-Id: Iab57283e118cf92b19adda5794488cfe7bfcd6c0
Signed-off-by: Wei Liu <wei3.liu@intel.com>
Reviewed-on: https://review.whamcloud.com/26292
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-9065 osc: fix for cl_env_get in low memory

In low memory situation cl_env_get->cl_env_new->kmem_cache_alloc
could fail with ENOMEM error. Some parts doesn`t handle error
case, for example:
...(osc_lock_upcall()) ASSERTION( !IS_ERR(env) ) failed:
...(osc_lock.c:315:osc_lock_upcall()) LBUG

For osc_lock_upcall() the patch changes cl_env_get to
cl_env_percpu_peek, this prevents memory allocation and
couldn`t fail in low memory case.
For osc_extent_truncate() the patch adds error handle.

Signed-off-by: Alexander Boyko <alexander.boyko@seagate.com>
Change-Id: I4fbca9f3bdc77a5c52979eeb0e00e915ff9ad9b0
Seagate-bug-id: MRP-4120
Reviewed-on: https://review.whamcloud.com/25171
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andrew Perepechko <andrew.perepechko@seagate.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9019 obd: remove s2dhms time function

Lustre created the s2dhms function to report a time
lapse in a days:hour:minute:seconds format. This is
over kill so remove s2dhms and just report the number
of seconds that has lapse.

Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Change-Id: I834661e57bf14c5ae397a40784bdec69b9bd55a8
Reviewed-on: https://review.whamcloud.com/23269
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-8998 misc: minor clean ups

Use LCME_ID_INVAL instead of comparing component IDs to 0.

Swab layout fields in lfsck_lmm2layout() to match existing usage.

Handle the (potential) case where LCME_ID_MAX isn't ~LCME_ID_MASK.

Fix man page dates.

Test-Parameters: trivial testlist=sanity-pfl
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I45b10156ac0081b2246b15c6790e42764b5cab07
Reviewed-on: https://review.whamcloud.com/26486
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9187 lfsck: handle parameters properly

Currently, because of improper handling the parameters as to
related parameters (such as "failout", "dryrun") only work for
OI scrub, but do not work for namespace LFSCK and layout LFSCK.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Ib38f7480758a59ba6a042948a8b49c36b401f100
Reviewed-on: https://review.whamcloud.com/25849
Tested-by: Jenkins
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-8119 osd: remove dcb entry from the list prior processing.

The osd_trans_commit_cb() processes list of commit callbacks
but there is list_del_init() missed prior calling dcb_func().
The dcb_func() call frees dcb entry itself, so list head become
corrupted. Patch adds this list_del_init().

Signed-off-by: Mikhal Pershin <mike.pershin@intel.com>
Change-Id: I561add170f408a7d16631424218b2a049d81f4fd
Reviewed-on: https://review.whamcloud.com/24202
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-9339 ldiskfs: Make ldiskfs buildable on kernels with no project quota

Update ldiskfs patches to ensure buildability on unpatched rhel7.2/7.3
Also update lustre_user.h definition of LL_IOC_FSSETXATTR in the face
of missing define of FS_IOC_FSSETXATTR

Change-Id: I518e46e727086ed5e10b4820c4e69b2ea896fd43
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: https://review.whamcloud.com/26647
Tested-by: Jenkins
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-9339 quota: Replace MAXQUOTAS usage with LL_MAXQUOTAS

This is especially important when kernel idea of
MAXQUOTAS is smaller than LL_MAXQUOTAS

Change-Id: I0fb8b15a0bab34993e9570efb8de548955962590
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: https://review.whamcloud.com/26648
Tested-by: Jenkins
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>

LU-9140 tests: disable sanityn test_77j and test_77k

These tests are failing regularly, disable until fixed.

Test-Parameters: trivial testlist=sanityn
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I9aa88b0a564e6c74e021627bdc372d40df92c3f6
Reviewed-on: https://review.whamcloud.com/26615
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4017 quota: add project quota support to lfs quota/setquota

This patch adds project quota support to utils,
lfs quota/setquota now could be aware of project quota.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I274949293e098b94b9af4a1de7dd62c643993743
Reviewed-on: https://review.whamcloud.com/23955
Tested-by: Jenkins
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4017 quota: enable project quota limits

With this patch, lctl conf_param can be used to enable
or disable project quota limits

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I051c1395f909891f7e0ff60fa2f98195914cfa4b
Reviewed-on: https://review.whamcloud.com/23954
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4017 quota: add setting/getting project id function

Extend Attr RPC to support project ID attribute, new
ioctl is introduced to get/set project id, it is kept
same ioctl number as VFS, you could use:

lsattr -p <file>
chattr -p <file>

Redefine LL_MAXQUOTAS as 3, change
@qsd_acct_failed into qqi, otherwise osd-zfs
could not use USR/GRP quota normally.

Remove osd_quota_migration(), it only output
some messages in case upgrading from 1.8 which
give us extra efforts for project quota, remove
it indead.

also fix a typo bug in osp_attr_set().

Change-Id: I4b537be9efa29fda730592b6d5e41c6b60fb6c0d
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/26202
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Tested-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9278 quota: avoid crash when access failed quota proc file

If quota dose not have accounting objects because of failing to
load or unsupported type, @lqp_obj will be NULL, we could not access
directly, add proper handling to avoid following crash

[<ffffffffa0c60730>] ? lprocfs_quota_seq_show+0x460/0x460 [lquota]
[<ffffffff8126bd8f>] proc_reg_open+0x6f/0x100
[<ffffffff811fbfb7>] do_dentry_open+0x1a7/0x2e0
[<ffffffff812a8bfc>] ? security_inode_permission+0x1c/0x30
[<ffffffff8126bd20>] ? proc_reg_release+0x70/0x70
[<ffffffff811fc18f>] vfs_open+0x5f/0xe0
[<ffffffff81209b58>] ? may_open+0x68/0x110
[<ffffffff8120ccfd>] do_last+0x1ed/0x12a0
[<ffffffff8120de72>] path_openat+0xc2/0x490
[<ffffffff8118075b>] ? unlock_page+0x2b/0x30
[<ffffffff8120ffeb>] do_filp_open+0x4b/0xb0
[<ffffffff8121cc77>] ? __alloc_fd+0xa7/0x130
[<ffffffff811fd4f3>] do_sys_open+0xf3/0x1f0
[<ffffffff811fd60e>] SyS_open+0x1e/0x20
[<ffffffff81696d49>] system_call_fastpath+0x16/0x1b

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I8dfc2b35c1f38416ed0f933296bdaa455e117926
Reviewed-on: https://review.whamcloud.com/26277
Tested-by: Jenkins
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-4017 quota: add project id support

This patch is infrastrcuture change for Lustre project quota,
project ID is considered as file attribute like UID/GID,
MDS will return back project id to client upon open/getattr.

Extend llog_setattr64_rec_v2 to be able to store project
ID, upgrade from old format could be handled properly,
if you want to downgrade again, need to make sure llog
have been handled completely.

Change-Id: Id7d27188f22169d6cffb74ea8b792ad4b133ef1d
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: https://review.whamcloud.com/25812
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>

LU-4017 ldiskfs: export __ext4_ioctl_setproject for lustre

Export this helper so that lustre can save project id to
MDS and OST objects

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I5c374e6dc7a509c0cbd46ac383b8d99f7f67d6b6
Reviewed-on: https://review.whamcloud.com/23945
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4017 quota: add project quota support for Lustre

Thsi patch adds necessary support of project quota,
so that OSTs can be mounted successully

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I0c56b14b4ebb0b495497d2f05981a03218f686d7
Reviewed-on: https://review.whamcloud.com/23947
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4017 quota: add project quota support to system header

System header contains the quota macros. Project quota support
should be added to it so that user space utils can be compiled
successfully

This patch redefines some macros in the system header file,
which avoids to change the sytem header file directly.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I1ed8014b1050684891df43d4c3db3c2e90b7814c
Reviewed-on: https://review.whamcloud.com/23946
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4017 quota: cleanup codes of quota for new type

This patch cleanups codes of quota to prepare for
new type other than USRQUOTA/GRPQUOTA.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Change-Id: I5b6fadf4eead957c8c94c91a4993de8798d87d3d
Reviewed-on: https://review.whamcloud.com/18894
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>

LU-4017 ldiskfs: add project quota support

Pick following patchs from upstream:

commit 847aac644e92e5624f2c153bab409bf713d5ff9a
vfs: Add general support to enforce project quota limits
This patch adds support for a new quota type PRJQUOTA for
project quota enforcement. Also a new method get_projid()
is added into dquot_operations structure.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
Reviewed-by: Jan Kara <jack@suse.cz>
Signed-off-by: Jan Kara <jack@suse.cz>
commit 334e580a6f97e2e84d1c19a8679603956acaa622
fs: XFS_IOC_FS[SG]SETXATTR to FS_IOC_FS[SG]ETXATTR promotion
Hoist the ioctl definitions for the XFS_IOC_FS[SG]SETXATTR
API from fs/xfs/libxfs/xfs_fs.h to include/uapi/linux/fs.h
so that the ioctls can be used by all filesystems,
not just XFS. This enables (initially) ext4 to use
the ioctl to set project IDs on inodes.

Based-on-patch-from: Li Xi <lixi@ddn.com>
Signed-off-by: Dave Chinner <dchinner@redhat.com>
Above two chages are wrapped into vfs-project-quotas-rhel7.patch

commit 040cb3786d9b25293b8b0b05b90da0f871e1eb9b
ext4: adds project ID support

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reviewed-by: Andreas Dilger <adilger@dilger.ca>
Reviewed-by: Jan Kara <jack@suse.cz>
commit 689c958cbe6be4f211b40747951a3ba2c73b6715
ext4: add project quota support
This patch adds mount options for enabling/disabling project quota
accounting and enforcement. A new specific inode is also used for
project quota accounting.

[Includes fix from Dan Carpenter to correct error checking from dqget(). ]

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reviewed-by: Andreas Dilger <adilger@dilger.ca>
Reviewed-by: Jan Kara <jack@suse.cz>
commit 9b7365fc1c82038faa52d56173b20221cf422cbe
ext4: add FS_IOC_FSSETXATTR/FS_IOC_FSGETXATTR interface support
This patch adds FS_IOC_FSSETXATTR/FS_IOC_FSGETXATTR ioctl interface
support for ext4. The interface is kept consistent with
XFS_IOC_FSGETXATTR/XFS_IOC_FSGETXATTR.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reviewed-by: Andreas Dilger <adilger@dilger.ca>
Reviewed-by: Jan Kara <jack@suse.cz>
This patch add ldiskfs project quota support
Did following adjustment from original patches:

1.adds padding structure to superblock, so that @s_prj_num is
in the same position of superblock from upstream.

2.Patch aslo disable kabi check for rhel7, since
any of vfs patch's include files change will break the check

Also fix conf-sanity: test_55, because now debugfs output
Project ID, we should output file size.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I2392f402d9853af969e4cd37318a83f2c4941c3e
Reviewed-on: https://review.whamcloud.com/17770
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-8998 pfl: PFL feature implementation

Merge remote-tracking branch 'origin/pfl'

Please check out the HLD of PFL here:
        http://wiki.lustre.org/PFL_Prototype_High_Level_Design

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: Ide007c7fffc17dc6a51500b537f19d2391387f9d

LU-9300 test: need copytool setup for sanity-hsm 1b

sanity-hsm test_1b() needs setup/cleanup copytool in the test.

Test-Parameters: trivial testlist=sanity-hsm,sanity-pfl
Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I40b083db37eb4e764e06606d0912ac635dc8c2a1
Reviewed-on: https://review.whamcloud.com/26434
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>

LU-7659 mdc: expose changelog through char devices

Register one character device per MDT in order to allow non-llapi to
read them and to make delivery more efficient.

- open() spawns a thread to prefetch records and enqueue them into a
  local buffer (unless the device is open in write-only mode).
- lseek() can be used to jump to a specific record, in which case the
  offset is a record number (with SEEK_SET) or a number of records to
  skip (SEEK_CUR). Movement can only be done forward.
- read() copies records to userland. No truncation happens, so short
  reads are likely.
- write() is used to transmit control commands to the device.
  The only available one is changelog_clear, which is done by writing
  "clear:cl<user>:<recno>" into the device.
- close() terminates the prefetch thread if any, and releases resources.

It is possible to poll() on the device to get notified when new records
are available for read.

Signed-off-by: Henri Doreau <henri.doreau@cea.fr>
Change-Id: I14709fdbac76b5512e58099e4e536cf9c973868c
Reviewed-on: https://review.whamcloud.com/18900
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9165 pfl: MDS handling of write intent IT_LAYOUT RPC

* Instantiate the OST objects of the 1st component of the PFL when its
  data are being created or its layout being set.

* Hence component_add won't instantiate OST objects, the instantiation
  is relayed to later write/truncate access.

* When the client tries to write beyond OST-object-instantiated extent
  range, it will issue an IT_LAYOUT intent lock RPC, indicating the
  write range, MDS needs to handle this RPC to allocate all OST
  objects of components covering the range.

* MDS also needs to handle the replay of write intent IT_LAYOUT RPC,
  which contains the current layout objects which the client knows, so
  that MDS won't initialize those objects again.

Reviewed-on: https://review.whamcloud.com/25717

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: Idcb242af55788dd8d7f01a6ecec4fbe508375512
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>

LU-9008 pfl: dynamic layout modification with write/truncate

* in lov_init_composite(), skip init sub object without LCME_FL_INIT
  layout component.
* issue layout intent RPC during write/trunc ops when try to write to
  an un-init-ed component (even if at the lock stage).
* After layout intent RPC issued, restart the IO.
* get rid of unused lov_layout_operations::llo_install() interface.
* add an empty mdt_layout_change() interface to handle intent layout
  write RPC.

Reviewed-on: https://review.whamcloud.com/25317

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I2f79482187d2af2660dd86e55da3f5dc0138e94a
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>

LU-8998 pfl: test cases for lfsck on PFL

New test cases for check and repair inconsistent PFL
files, and related failures injections.

Reviewed-on: https://review.whamcloud.com/24884

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I102079f6a4bbdd4b1ca2739524e0e21ef5f79b98
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>

LU-8998 docs: add llapi_ man pages to Makefile.am

Reformat lustre/doc/Makefile.am to have one entry per line to
match master, so the merge is more straight forward.

Add the llapi_layout_comp_*.3 and llapi_layout_file_comp_*.3 man
pages to lustre/doc/Makefile.am so they are included into the RPMs.

Replace references to the non-existent "liblustreapi.7" man page
with "lustreapi.7" instead, and add all llapi_* man pages to it.

Remove the long-obsolete llbackup.8 man page.

Reviewed-on: https://review.whamcloud.com/26275

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@dilger.ca>
Change-Id: Ic97ea204f9635668fd5a4448687019f84f5cab07
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>

LU-8998 docs: add lfs-setripe manpage to dist

It's missing from commit a1e0fd6f.

Reviewed-on: https://review.whamcloud.com/26268

Test-Parameters: trivial
Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I8212cb1e48faaa6b19cd58d1ea6ab00a6f78644c
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-8998 pfl: layout LFSCK handles PFL file

Enhance the layout LFSCK logic to handle the PFL file,
including all the inconsistent cases corresponding to
the non-PFL file cases. In addition, for PFL file, the
LFSCK can also verify and recover the PFL extent range.

Reviewed-on: https://review.whamcloud.com/24883

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Iec5ac57fd556330130e33f187c6fbf8cc24e67b8
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>

LU-8998 pfl: enhance PFID EA for PFL

This is a misc patch that contains some adjustments to
store more stripe information in the OST-object's PFID
EA (XATTR_NAME_FID), including:

1) For regular file, store the stripe count and stripe
   size information in its OST-object's PFID EA. These
   information are necessary for the LFSCK to recover
   the MDT-object layout EA from orphan OST-object(s).

2) In addition, for PFL file, store the FPL component
   ID and extent information in its OST-object's PFID
   EA also. These information are also necessary for
   the LFSCK to recover the MDT-object PFL layout EA
   from orphan OST-object(s).

3) For ldiskfs backend, there may be not enough space
   in inode body (256 bytes by default) to hold the
   enlarged PFID EA. To avoid performance trouble if
   store the enlarged PFID EA in separated block, we
   make some hack (for the device with 256 bytes or
   smaller inode) inside osd-ldiskfs via merging the
   PFID EA into the LMA EA (XATTR_NAME_LMA) to save
   EA entry space, then allows both the original LMA
   EA and the PFID EA to be filled in the inode body
   without performance trouble. For that, introduce
   two compat flags: LMAC_STRIPE and LMAC_PFL.

4) It is client duty to transfer the stripe and PFL
   information to the OST via the write, setattr and
   punch RPC. Then OST will store these information
   in the PFID EA.

5) On the other hand, to recover the lost or crashed
   MDT-object's layout EA from orphan OST-object(s),
   the LFSCK on the OST needs to return the orphan's
   stripe and PFL (if have) information to the MDT.
   Originally, we only returned the MDT-object's FID
   (and owner) information. Now, we need to return
   related layout information also. For such purpose,
   new structure 'lu_orphan_rec_v2' replaces the old
   'lu_orphan_rec', and the idx_info::ii_recsize is
   used to indicates whether new 'lu_orphan_rec_v2'
   or old ''lu_orphan_rec' is used.

Reviewed-on: https://review.whamcloud.com/24882

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I5a69dbe91b1f83eeb394ce54453352d84a285193
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>

LU-8998 docs: man pages for composite layout APIs

Added man pages for the new layout component interfaces:

llapi_layout_comp_add(3)
llapi_layout_comp_del(3)
llapi_layout_comp_extent_get(3)
llapi_layout_comp_extent_set(3)
llapi_layout_comp_flags_get(3)
llapi_layout_comp_flags_set(3)
llapi_layout_comp_flags_clear(3)
llapi_layout_comp_id_get(3)
llapi_layout_comp_move(3)
llapi_layout_comp_move_at(3)
llapi_layout_file_comp_add(3)
llapi_layout_file_comp_del(3)

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I8fd466b9cb0352bb7812f0b748816801f20f5e71
Reviewed-on: https://review.whamcloud.com/25740
Tested-by: Jenkins
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>

LU-8054 lfsck: enhance lfsck notify API

Originally, the lfsck_in_notify() contains too much functionalities,
including both the remote LFSCK control and local LFSCK notification.
These LFSCK requirements/callers share the structure "lfsck_request"
to transfer kinds of data, as to it is very difficult to enhance the
"lfsck_request" for more local LFSCK requirements. Because enlarging
or adjusting the "lfsck_request" fields will affect the LFSCK RPC on
wire protocol and cause interoperability issues.

The patch splits the local LFSCK notifications from lfsck_in_notify(),
and introduce new LFSCK API lfsck_in_notify_local() for them. Then
even if we have more local LFSCK requirements in the future, we can
enhance such API freely without affecting LFSCK RPC protocol.

Reviewed-on: https://review.whamcloud.com/24881

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I8f285c6680de66a73050e167dc3874a4f6bf4ad0
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-8050 osp: not use obdo::o_lcookie for non-onwire RPC data

Originally, obdo::o_lcookie is for destroy with unlink cookie from
MDS, it is obsolete in 2.8. And then, it is reused by the OSP to
avoid RAM allocations. But now, we needs more fields in the obdo
structure to transfer more information from client to OST. The
obdo::o_lcookie is good candidate. So this patch adjust OSP
logic to NOT use such field for non-onwire RPC data usage.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I2bff0dce6f00f330657097a9ae9f775396e090cb
Reviewed-on: https://review.whamcloud.com/24876
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-8998 utils: enlarge default inode size

Composite layout consumes more EA space, to hold a typical 3
components composite layout in inode, the default inode size
on MDT needs be bumped from 512 bytes to 1024 bytes. The default
OST inode size is changed from 256 bytes to 512 bytes as well,
because there will be more composite layout information stored
in OST object EA.

Performance tests shows that increased default inode size doesn't
impact performance, following are few numbers from the mdtest
result:

                  create  stat  removal  tree-create tree-remove
512/256 (origin): 16543  47292  32826    48          22
1024/512 (new)  : 16903  47404  32114    55          20

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: Ic5c08fd514f1111343f11129b1503a5ba2265ec7
Reviewed-on: https://review.whamcloud.com/24873
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-8998 tests: test scripts for PFL

Added test scripts sanity-pfl.sh for PFL.

Reviewed-on: https://review.whamcloud.com/24872

Test-Parameters: testlist=sanity-pfl
Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I2a09282d7ce26fc2b683eba994cfc63cd1ce166d
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>

LU-8998 docs: man pages for tools of PFL

Added new options to 'lfs find' and 'lfs getstripe' for composite
files. Moved "lfs setstripe/migrate" into separate man page, and
added new options for the operations to composite file.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I9cafde834be5c3d890bf04a3efb2ea973364596a
Reviewed-on: https://review.whamcloud.com/25741
Tested-by: Jenkins
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>

LU-8998 lfs: user space tools for PFL

* 'lfs getstripe' to support composite file:

   --component-count:
   Display component count only;

   --component-id|-I [comp_id]:
   When 'comp_id' is provided, display components matching the
   provided 'copm_id', otherwise, display component ID only;

   --component-flags [flags]:
   When the 'flags' is provided, dispaly components contains the
   provided 'flags', otherwise, display component flags only;

   --component-start [[+-]start]:
   When '[+-]start' is provided, display components which have
   'at most' or 'at least' or exactly same extent start, otherwise,
   display component extent start only;

   --component-end|-E [[+-]end]:
   When '[+-]end' is provided, display components which have
   'at most' or 'at least' or exactly sme extent end, otherwise,
   display component extent end only;

* 'lfs find' to support composite file:

   --component-count <[+-]count>:
   Find composite files according to component count;

   --component-flags <flags>:
   Find composite files according to component flags;

   --component-start <[+-]start>:
   Find composite files according to component extent start;

   --component-end|-E <[+-]end>:
   Find composite files according to component extent end;

* 'lfs setstripe' to support composite file:

   --component-del [-I <comp_id> | --component-flags <flags>] filename
   Delete specified component(s) from existing composite file;

   --component-add -E <end1> [stripe options]
                              -E <end2> [stripe options] ...:
   Add component(s) to an existing composite file;

   -E <end1> [stripe options] -E <end2> [stripe options] ...:
   Create composite file with specifed layout components;

* 'lfs migrate' to support migrate composite files:
   - composite layout-> plain layout;
   - plain layout -> composite layout;
   - composite layout -> composite layout;

Reviewed-on: https://review.whamcloud.com/24867

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: Ib0d3eeb7927cf83a173099c474df6539d1264370
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>

LU-8998 lustreapi: composite llapi_layout

Extend the llapi_layout API to support composite layout.

- llapi_layout_comp_add():
  Add one component to an existing layout, the layout will
  be turned into composite if it was plain before adding.

- llapi_layout_comp_del():
  Delete current layout component from a composite layout,
  the component to be deleted must be the tail of layout
  component list, and it can't be the last one.

- llapi_layout_comp_extent_get/set():
  Get/Set current component extent of a layout. When set
  component extent, the layout will be turned into composite
  if it was plain.

- llapi_layout_comp_flags_get/set/clear():
  Get/Set/Clear current component flags of a composite layout.

- llapi_layout_comp_id_get():
  Get current component ID of a composite layout.

- llapi_layout_comp_move_at():
  Move the current component to a component with specified ID.

- llapi_layout_comp_move():
  Move the current component to a specified position: FIRST,
  NEXT, or LAST.

- llapi_layout_file_comp_add():
  Add component(s) to an existing composite file.

- llapi_layout_file_comp_del():
  Delete component(s) from an existing composite file.

- llapi_layout_file_comp_set():
  Set certain attribute of the specified layout component(s). Not
  implemented in PFL phase 2.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I44094149636be3162fc7f8b75a90c64abf2c3c02
Reviewed-on: https://review.whamcloud.com/24866
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>

LU-8998 clio: getstripe support comp layout

{get/set}stripe support composite layout

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: Iafe5205a79f721be1ec5ec83ab80a86a4b537e18
Reviewed-on: https://review.whamcloud.com/24851
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>

LU-8998 clio: Client side implementation for PFL

Make client layer support composite layout.

Plain layout will be stored in LOV layer as a composite layout
containing a single component.

Reviewed-on: https://review.whamcloud.com/24850

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: Ic3b85a4b10c66745e5c72ff02ea313baa0b12bb5
Reviewed-by: Lai Siyao <lai.siyao@intel.com>

LU-8998 lov: add composite layout unpacking

Update struct lov_stripe_md to accommodate composite layouts. Add
methods to unpack composite layouts.

Reviewed-on: https://review.whamcloud.com/24849

Change-Id: Ife0b2ae7671dd0ec0020a1da04e7261ef9f7e5f3
Signed-off-by: John L. Hammond <john.hammond@intel.com>
Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-8998 lod: accomodate to composite layout

Modify the LOD to make it support the composite layout:

- Introduced lod_layout_component structure to represent each
  individual layout component, and added ldo_comp_entries and
  ldo_comp_cnt in the lod_object structure to hold the striping
  information for composite layout.

- Object allocation code is adjusted to not only check the used
  OSTs in this round of allocation, but also the used OSTs in
  the existing layout components.

- Support special xattr names to add/set/delete layout component(s).

- Store default layout xattr on the filesystem root inode.

Reviewed-on: https://review.whamcloud.com/24823

Change-Id: I73736df1cd529a3b0e32b2bb6a09b5c436e08c86
Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>

LU-8998 pfl: Basic data structures for composite layout

Added basic structures and magic numbers for composite layout.

Test-Parameters: trivial
Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I02b4347d4959a935d4120d23f3c7b73511883a3f
Reviewed-on: https://review.whamcloud.com/24822
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-9281 test: stop using invalid ps option

Use of the '-q' option in a ps commend was introduced in sanity.sh
test_161d by the recent commit ee25babfe72378f9496a9732742984f26eb7d4a5.
Thst option isn't valid in all versions of ps.  It doesn't exist in any
sles version.

Use the '-p' option which exists everywhere instead.

Test-Parameters: clientdistro=sles12sp2 testgroup=review-ldiskfs

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I143be0c1ad633889c9ed3ff7f5cb38bf54feb19a
Reviewed-on: https://review.whamcloud.com/26298
Tested-by: Jenkins
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-9245 utils: fix string copy handling within lgss_sk

Strings are not copied correctly when the fsname or nodemap field of
a shared key is replaced. Instead of being zero-terminated, the new
string may overlap the original if it is smaller.

Fixed by using the buffer length instead of the string length in
strncpy().

Change-Id: I17bb5aa0ca7c25b2545c17c6f23e69045730a547
Test-Parameters: trivial
Signed-off-by: Chris Hanna <hannac@iu.edu>
Reviewed-on: https://review.whamcloud.com/26165
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Kit Westneat <kit.westneat@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-9236 kernel: new kernel RHEL 6.9 [2.6.32-696.el6]

Changes to support new kernel version in new el6.9 release distro
New kernel-config and target files.
Revised base kernel and ldiskfs patches.
Small adjustments to lbuild and autoconf.

Test-Parameters: clientdistro=el6.9 mdsdistro=el6.9 ossdistro=el6.9

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: Idd341e4db98c3e6e91c9d8696a9a5e7f03ded6fe
Reviewed-on: https://review.whamcloud.com/26111
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-8066 obd: make version_show sysfs compliant

A patch was pushed upstream that added "lustre: "
to the lustre version sysfs file. This was
rejected due to the one item per sysfs rule.
Recently a patch was landed to update the
lustre tools to handle "lustre: " missing. Since
this is the case we can bring the version
sysfs file into compliance.

Change-Id: I74568d581b176c802f132cef3c022fe0fdded164
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/25632
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-8843 build: fix all remaining less than 0 comparison for unsigned values

Remove the rest of the test of less than zero for unsigned values
found with -Wtype-limits.

Change-Id: I4ca764f359605a452cf4934e6a93106e91d2cc32
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/25384
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-8236 nrs: Add wildcard support for JOBID TBF rule

This patch adds wildcard support for JOBID TBF rule.
The following kind of wildcard matching rules can be
used:
start ruleName jobid={dd.*} rate=20
start ruleName jobid={*.500} rate=10
start ruleName jobid={prog*.10*} rate=15

Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: I0a4c44fd15533dfccd9f6ab7374a0a7a24b1403e
Reviewed-on: https://review.whamcloud.com/24523
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>