LU-10634 kernel: kernel update [SLES12 SP3 4.4.114-94.11]

Update target and kernel_config files for new version

Test-Parameters: clientdistro=sles12sp3 testgroup=review-ldiskfs \
  mdsdistro=sles12sp3 ossdistro=sles12sp3 \
  mdtfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I3ffcd4c368b2976cffa6a517f9fabcf674781ac9
Reviewed-on: https://review.whamcloud.com/31224
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10603 ptlrpc: export req_buffers_max via procfs

after LU-9372 gcc7 complains:
lustre/ptlrpc/lproc_ptlrpc.c:382:16: error: ‘ptlrpc_lprocfs_req_buffers_max_fops’ defined but not used [-Werror=unused-const-variable=]
 LPROC_SEQ_FOPS(ptlrpc_lprocfs_req_buffers_max);
                 ^

Change-Id: Ie4806b79d104c7ea9aa34b6a8a280587fccef689
Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-on: https://review.whamcloud.com/31162
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10560 libcfs: handle rename to wait_queue_entry_t

The 4.13 kernel renames wait_queue_t to wait_queue_entry_t.

Add a probe and handle rename across the code base and have
a define to translate to the new name when indicated.

Test-Parameters: trivial

Change-Id: I8f0f5ec4d02ccb270acb72ccffe13f0ecf6bd2f7
Signed-off-by: Mike Marciniszyn <mike.marciniszyn@intel.com>
Reviewed-on: https://review.whamcloud.com/31153
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-10560 lustre_compat: Convert GFP_TEMPORARY to GFP_KERNEL

The 4.14 kernel removes this gfp.h define.

Adjust the code to use GFP_KERNEL as the upstream
patch does.

Change-Id: I40fff2724499fa17aa285507e0fd9b21f4afc070
Signed-off-by: Mike Marciniszyn <mike.marciniszyn@intel.com>
Reviewed-on: https://review.whamcloud.com/31152
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-10574 tests: remove useless check from sanity-dom.sh

Tests test_sanity() and test_sanityn() are skipped if started
not from lustre/tests directory because of incorrect check
that ./sanity.sh exists.
Patch removes the check of the files which are part of
lustre/tests.

Signed-off-by: Elena Gryaznova <c17455@cray.com>
Cray-bug-id: LUS-2594
Reviewed-by: Vladimir Saveliev <c17830@cray.com>
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Test-Parameters: testlist=sanity-dom
Change-Id: I51ad517fbf3ff653d9a11994eb280daee589a886
Reviewed-on: https://review.whamcloud.com/31074
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10449 nrs: Generic TBF policy can't be shown correctly

After setting TBF NID/OPCode/JobID policy and switch to generic
policy, the output of "lctl get_param ost.OSS.ost.nrs_policies"
can not display correctly.

Change-Id: If8dcb7ae6ade634ec7ec4dfcb5887501cda90cdf
Signed-off-by: Qian Yingjin <qian@ddn.com>
Reviewed-on: https://review.whamcloud.com/30696
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9452 tests: remove sanityn test_29 from ALWAYS_EXCEPT

There is no longer a sanityn.sh test_29() so it shouldn't be
listed in ALWAYS_EXCEPT.

Test-Parameters: trivial testlist=sanityn
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: Ia8112e20cbd3203b69e85e586e2400551b94de81
Reviewed-on: https://review.whamcloud.com/30646
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Saurabh Tandan <saurabh.tandan@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10278 utils: allow to migrate without direct io

Using direct i/o to copy file contents during migration minimizes
cache interference, but may significatly reduce performance.
Introduce new option -D/--non-direct to lfs migrate/lfs_migrate that
leaves the tradeoff at the discretion of the caller.

Signed-off-by: Daniel Kobras <d.kobras@science-computing.de>
Signed-off-by: Emoly Liu <emoly.liu@intel.com>
Change-Id: I9c2935ff204ea5385bfc38006c5476b956deb6a7
Reviewed-on: https://review.whamcloud.com/30301
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-6142 uapi: remove remaining typedef in lustre UAPI headers

Remove remaining tyepdef in lustre UAPI headers to make them
linux kernel compliant.

Test-Parameters: trivial

Change-Id: I13a24deed348e06c1c63bd0c332f63d4f77a0d76
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/31175
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7004 quota: make lctl set_param -P functional for quota

Currently setting up quota permanently can only be done with a
command like lctl conf_param $FSNAME.quota.ost=ug. To see if those
settings take hold we examine the 'enabled' proc file located in
the quota_slave directory in the proc tree. To make this workable
with lctl set_param -P we can make the 'enabled' proc file
writable and lustre can treat the config log change from set_param
-P for quota like any other tunable to be set permanetly.

Change-Id: I6a4c1fdc9d16658930f48d21e4f79e6f36047511
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/31081
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10576 tests: sleep seconds to avoid using cached statfs

In sanity test_803, we check the object usage via "lfs df -i".
But the MDT may return cached statfs if two "df" calls arrive
too close each other (about 1 second). Sleep 3 seconds between
two "df" calls to avoid such trouble.

Test-Parameters: trivial envdefinitions=SLOW=yes testlist=sanity mdtfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs mdscount=2 mdtcount=4
Signed-off-by: Fan Yong <fan.yong@intel.com>
Signed-off-by: Ben Evans <bevans@cray.com>
Change-Id: I9ce4cb6c069a88fe2b93d2d5a6304c96bdb5a0c1
Reviewed-on: https://review.whamcloud.com/31102
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-8444 tests: test for unsigned xattr inode number

The patch for "MRP-3025 Hitting "LDISKFS-fs error (device md66):
ldiskfs_xattr_inode_iget: error while reading EA inode -2147483347" on
large MDT volumes with large_xattr feature enabled."

Added test:
1. MDS should have more than 2G inodes
2. mdt fs should be created with large_xattr flag.
3. set inode_goal to get higher inode number allocated:
   echo 2147483947 > /sys/fs/ldiskfs/<device>/inode_goal
3. create a file
4. start adding hard links to that file

Test-Parameters: trivial testlist=conf-sanity
Signed-off-by: Artem Blagodarenko <artem.blagodarenko@seagate.com>
Seagate-bug-id: MRP-3378
Change-Id: Id9c0fe9d8047935e5cf5be1b9209a74588565f2e
Reviewed-on: https://review.whamcloud.com/21547
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alexander Zarochentsev <c17826@cray.com>
Reviewed-by: Alexander Lezhoev <garson2@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-8602 gss: autoconf check missing "test" keyword

Change https://review.whamcloud.com/31095 introduced an error in the
autoconf, omitting the command "test" in an autoconf check.  Add it.

Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Change-Id: I525805801d9e8166ec1064dccbf6cec6f97efdfa
Reviewed-on: https://review.whamcloud.com/31191
Reviewed-by: Jeremy Filizetti <jeremy.filizetti@gmail.com>
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10611 autoconf: check zlib library and zlib.h header file

After landing commit f1daa8fc6575e5b9e4a2f1f2ae4ceaefb889a694,
zlib library and zlib.h header file are required to compile lfs.c.
This patch adds the check in configure script.

Change-Id: Id3a8acfc780fb4fcdec0bb99b79b550c5c9e957a
Test-Parameters: trivial testlist=sanity-flr
Signed-off-by: Jian Yu <jian.yu@intel.com>
Reviewed-on: https://review.whamcloud.com/31186
Tested-by: Jenkins
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Quentin Bouget <quentin.bouget@cea.fr>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10577 tests: fix lfsck-performance for separate MGT and MDT

lfsck-performance 0,1,2,3 tests run stopall and then mount MDT
which cause the tests failures on configuration with not
combined MGS/MDS.
Patch fixes these tests defects.

Signed-off-by: Elena Gryaznova <c17455@cray.com>
Cray-bug-id: LUS-2534
Test-Parameters: testlist=lfsck-performance
Reviewed-by: Vladimir Saveliev <c17830@cray.com>
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Change-Id: I24c15b9998511bab3dc6fdd3445793e70281c890
Reviewed-on: https://review.whamcloud.com/31075
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10482 flr: enhance "lfs find" to add mirror options

This patch adds the following mirror related search
options to "lfs find" command:

[[!] --mirror-count|-N [+-]n]
[[!] --mirror-state <[^]state>]

--mirror-count|-N indicates mirror count.
--mirror-state indicates mirrored file state.

A mirrored file can be one of the following states:
ro indicates the mirrored file is in read-only state.
   All of the mirrors contain the up-to-date data.
wp indicates the mirrored file is being written.
sp indicates the mirrored file is being resynchronized.

Change-Id: I3c8f5c8bb6518ba4bd73fc2f164dd52afdfac211
Signed-off-by: Jian Yu <jian.yu@intel.com>
Reviewed-on: https://review.whamcloud.com/31069
Tested-by: Jenkins
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9727 doc: update llog_reader man page for Changelogs

Add new paragraph in llog_reader's man page to explain how to read
Changelogs with llog_reader, and add an example.

Test-Parameters: trivial
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I3e1123b9a5ac88334a370fd69c1d9d63597e16f7
Reviewed-on: https://review.whamcloud.com/30970
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Olaf Weber <olaf.weber@hpe.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9906 osd: use pagevec for putting pages

Using a pagevec instead of individual page puts is much
more efficient.  This should reduce contention on the page
cache allocation/freeing, which becomes a bottleneck with
high speed OSTs.

Cray-bug-id: LUS-5670
Signed-off-by: Patrick Farrell <paf@cray.com>
Change-Id: Ic15cb8e30887ec55e9348e50af307bfd7108c7e4
Reviewed-on: https://review.whamcloud.com/30531
Tested-by: Jenkins
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Alexey Lyashkov <c17817@cray.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10377 build: Update ZFS Version to 0.7.6

Update SPL and ZFS version that is built against.

https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.7.6

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: If010d3a7e78b66a2acbd70242fe517218a438c02
Reviewed-on: https://review.whamcloud.com/30522
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9727 utils: make llog_reader decode changelog fields

Make llog_reader decode all Changelogs fields and extra fields.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Idfb41607fc5664cb99b254aece4625d1796331af
Reviewed-on: https://review.whamcloud.com/30315
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-9727 lustre: record denied OPEN in Changelogs

Record denied OPEN events in Changelogs, in the same format as
successful OPEN events.
Recording denied OPEN events is useful for security audit,
in order to find out who tried to get access to some data.
An NOPEN changlog entry is in the form:
4 24NOPEN 15:45:44.947406626 2017.08.31 0x2 t=[0x200000402:0x1:0x0]
ef=0xf u=500:500 nid=10.128.11.158@tcp m=-w-
By default, disable recording of NOPEN events in Changelogs.
NOPEN entries in Changelogs are rate limited: no more than one
entry per user per file per minute, configurable via
/proc/fs/lustre/mdd/<fsname>-MDTXXX/changelog_deniednext

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ib33651dda63735e21fffeed34cb1adc803ff7eca
Reviewed-on: https://review.whamcloud.com/28812
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Matthew S <matthew.sanderson@anu.edu.au>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9727 lustre: limit OPEN and CLOSE rates in Changelogs

Record OPEN only once in the Changelogs per UID/GID, for a given
open mode, as long as the file is not closed by this UID/GID.
Similarly, only record the last CLOSE per UID/GID.
For instance, it avoids flooding the Changelogs if there is an MPI
job opening the same file thousands of times from different threads.
It reduces the ChangeLog load significantly, without significantly
affecting the audit information.

To achieve this, add a list to struct mdd_object, containing uid/gid
of clients opening files. Record OPEN only if client uid/gid is not
already in list. And record CLOSE only if client uid/gid was just
removed from list.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I0fa08d11f0284d63e531ab48c03a8af6f3928487
Reviewed-on: https://review.whamcloud.com/28299
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9727 lustre: add CL_GETXATTR for Changelogs

Record GETXATTR events in Changelogs, and add a new changelog
extension named changelog_ext_xattr to hold xattr name.
A GETXATTR changlog entry is in the following form:
8 23GXATR 09:22:55.886793012 2017.07.27 0x0
t=[0x200000402:0x1:0x0] ef=0xf u=500:500 nid=10.128.11.159@tcp
x=user.name0
Also, rename CL_XATTR type to CL_SETXATTR.
By default, disable recording of GETXATTR events in Changelogs.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ia02e870ca162c7d2b97eb0ce80e99fe7145b7601
Reviewed-on: https://review.whamcloud.com/28251
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-9409 llite: Add tiny write support

If a page is already dirty in the page cache, we can write
to it without a full i/o.  This improves performance for
writes of < 1 page dramatically.

Append writes are a bit tricky, requiring us to take the
range lock (which we can normally avoid), but they are
still much faster than the normal i/o path.

Performance numbers with dd, on a VM with an older Xeon.

All numbers in MiB/s.

                8 bytes 1KiB
Without patch:  .75     75
With patch:     6.5     153

Cray-bug-id: LUS-1705
Signed-off-by: Patrick Farrell <paf@cray.com>
Change-Id: I75cc72ceb5f174a5394af8ffe5df4fe9583f19a3
Reviewed-on: https://review.whamcloud.com/27903
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Alexey Lyashkov <c17817@cray.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10418 flr: replace llapi_lease_get with llapi_lease_acquire

After commit 8f1c7c1e44a4cc8870eb2b2a71da323e265881b4 landed,
we need replace llapi_lease_{get,put} with llapi_lease_{acquire,release}.

Change-Id: Ie36abeb3c71f2d0c3345de9b4830549af3bb56a3
Test-Parameters: trivial testlist=sanity-flr
Signed-off-by: Jian Yu <jian.yu@intel.com>
Reviewed-on: https://review.whamcloud.com/31221
Tested-by: Jenkins
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10448 lod: pick primary mirror for write

As a mirrored file being written for the first time, MDS will choose
a mirror to write the data, a primary choosing policy function is
defined in this patch (lod_primary_pick()) to avoid the mirror with
unavailable OSTs.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I5d6d0459e96583294c3040a7994c33114be1e439
Reviewed-on: https://review.whamcloud.com/30711
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10181 mdt: high-priority request handling for DOM

Implement high-priority request handling and lock
prolongation for Data-on-MDT BRW requests to avoid incorrect
timeouts and client eviction under heavy MDS load.

Test-Parameters: mdssizegb=20 testlist=sanity-dom,dom-performance
Signed-off-by: Mikhal Pershin <mike.pershin@intel.com>
Change-Id: I589efd2774d739f3a0b471d7a6e4d6be7c6a7c2c
Reviewed-on: https://review.whamcloud.com/29968
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10438 flr: layout truncate compatibility

In PFL design, client issues [0, size) intent to MDS to instantiate
objects in this extent. While in FLR design, the intent serves two
purposes: 1) make objects across [size, EOF) on other mirrors stale,
and 2) instantiate objects in the chosen write mirror. And original
FLR chose to use [size, EOF) as the extent of truncate write intent
request.

This patch reverts the choice, and still uses [0, size) as the
truncate write intent extent.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I12d90320ba704f01457670f864d78fe764233000
Reviewed-on: https://review.whamcloud.com/30786
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9771 util: rename LCM_FL_NOT_FLR to LCM_FL_NONE

Having "lfs getstripe" print out "lcm_flags: not_flr" is not very
useful, as the composite files may not relate to FLR (e.g. PFL).
Rename "LCM_FL_NOT_FLR" to "LCM_FL_NONE" so it is more clear there
are no composite flags on the layout.

Print out "0" for flags if no flags are set, to match old behaviour.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: Ib3701da8368253969567b927300cd42bc33ebbe5
Reviewed-on: https://review.whamcloud.com/31047
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Tested-by: Jenkins
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10286 mdt: deny 2.10 clients to open mirrored files

2.10 clients would manipulate mirrored layout as PFL layout, which
would damage mirrored files.

This patch only allows mirrored files to be opened by clients who
understand mirror layout.

It also fixes the problem that it should check OBD_CONNECT_FLAGS2
first before checking OBD_CONNECT2_XXX flags.

Test-Parameters: envdefinitions=SLOW=yes clientjob=lustre-b2_10 clientbuildno=68 testlist=runtests,sanity
Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I3431211ad30a1edd07f0f583d573328d6308779d
Reviewed-on: https://review.whamcloud.com/30957
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Tested-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-10560 llite: remove extra headers from rw26.c

Remove headers from llite_mmap.c that are no longer (or were possibly
never) needed.  This avoids a compile problem with 4.14 kernels:

CC [M]  lustre/llite/rw26.o
In file included from lustre/llite/rw26.c:43:0:
./arch/x86/include/asm/uaccess.h: In function ‘set_fs’:
./arch/x86/include/asm/uaccess.h:31:9:
error: dereferencing pointer to incomplete type
current->thread.addr_limit = fs;

It turns out <asm/uaccess.h> was included twice but we don't need
anything from that header.  Same for <linux/stat.h> and a number of
other extraneous headers.

Sort headers alphabetically so it is easier to avoid duplicates.

Change-Id: I4cbe2c1c7289466396b2bb2eac3c475d1041a283
Signed-off-by: Mike Marciniszyn <mike.marciniszyn@intel.com>
Reviewed-on: https://review.whamcloud.com/31151
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>

LU-10560 llite: remove extra headers from llite_mmap.c

Remove headers from llite_mmap.c that are no longer (or were possibly
never) needed.  This avoids a compile problem with 4.14 kernels:

CC [M]  lustre/llite/llite_mmap.o
In file included from lustre/llite/llite_mmap.c:41:0:
./arch/x86/include/asm/uaccess.h: In function ‘set_fs’:
./arch/x86/include/asm/uaccess.h:31:9:
error: dereferencing pointer to incomplete type
current->thread.addr_limit = fs;

It turns out <asm/uaccess.h> was included twice but we don't need
anything from that header.  Same for <linux/stat.h> and a number of
other extraneous headers.

Sort headers alphabetically so it is easier to avoid duplicates.

Test-Parameters: trivial
Change-Id: Ic24a58bdaec16f92d7bea9c24172796031cad471
Signed-off-by: Mike Marciniszyn <mike.marciniszyn@intel.com>
Reviewed-on: https://review.whamcloud.com/31150
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>

LU-10536 build: add path for libnvpair to zfslib

For the case of building lustre against the ZFS source tree
instead of an ZFS installation on the node we need to add
the path $zfssrc/lib/libnvpair/.libs so mount.lustre can
link to the library.

Test-Parameters: trivial

Change-Id: Id08cc3aed1bc201611fb9382d2f53e40e69f9544
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/31128
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10580 lfsck: GPF in lfsck_namespace_repair_dirent

chlid object can be NULL if we want just to remove a dirent

Change-Id: I330fc690b36c48c4be7fa4062304c39028f0b878
Cray-bug-id: MRP-4606
Signed-off-by: Andriy Skulysh <c17819@cray.com>
Reviewed-on: https://review.whamcloud.com/31086
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Alexandr Boyko <c17825@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9466 tests: Error message for empty "error" calls

Adding error message to empty "error" calls made
without any message for a few tests.

Made changes to tests under sanity.sh , sanityn.sh
and sanity-hsm.sh

Test-Parameters: trivial envdefinitions="SLOW=yes" testlist=sanityn.sh,sanity-hsm.sh
Signed-off-by: Saurabh Tandan <saurabh.tandan@intel.com>
Change-Id: Idb3440561e47f6caaab455f2c6d7e0d2a2651f95
Reviewed-on: https://review.whamcloud.com/31078
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Wei Liu <wei3.liu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10508 utils: use callvpe() in lustre_rsync

Add callvpe() to lustre/utils as a replacement for system() in cases
where the command string includes arbitrary pathnames. In
lustre_rsync, replace calls to system() with calls to callvpe().

Test-Parameters: trivial testlist=lustre-rsync-test
Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Id5ae7e25e14346a1293497c2caa221513d0ee9f3
Reviewed-on: https://review.whamcloud.com/30869
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10507 tests: use {save,restore}_layout() in test

Revised test cases sanity:test_{27A,65i,65j,65m,406}(),
sanity-pfl:test_10() to use new interfaces to save and restore
layout.

Test-Parameters: trivial testlist=sanity-pfl,sanity,sanity-dom
Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I11f4e5dcd486d4f7d08666c462d056041e125365
Reviewed-on: https://review.whamcloud.com/30858
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-6353 contrib: Remove wireshark plugin

Wireshark dissection has been pushed upstream:
https://code.wireshark.org/review/24795 [lnet]
https://code.wireshark.org/review/24800 [lustre]

Both patches have landed to wireshark master.

Dissectors were ported to wireshark master and significantly expaneded
and cleaned up.

Test-Parameters: trivial
Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I0a9d54634599cdb7f9169f1186c58fa96666b246
Reviewed-on: https://review.whamcloud.com/30602
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Doug Oucharek <dougso@me.com>

LU-10347 tests: suspend the copytool in sanity-hsm/test_252

The copytool does not have to, and should not, process the archive
request emitted in test_252 of sanity-hsm: the test needs the request
to time out.

This patch suspends the copytool until the request times out.

Test-Parameters: trivial testlist=sanity-hsm
Signed-off-by: Quentin Bouget <quentin.bouget@cea.fr>
Change-Id: I54d114eafb85b75e7f9160806cd3298bfa186b19
Reviewed-on: https://review.whamcloud.com/30492
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10364 test: Add version check to test_255b

Skip test if server version is older than 2.8.54 since
it does not support ladvise.

Change-Id: Ie613c1b8d9d082b78529a3d72fd59150431f65ea
Test-Parameters: trivial
Signed-off-by: Wei Liu <wei3.liu@intel.com>
Reviewed-on: https://review.whamcloud.com/30481
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Saurabh Tandan <saurabh.tandan@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9664 hsm: protect cdt_state

In hsm_cancel_all_actions in mdt_coordinator.c, the cdt_state
could be set to wrong state if there are more than one
hsm_cancel_all_actions at the same time.

Assume the state is CDT_ENABLED before hsm_cancel_all_actions

the first call               the second call
CDT_ENABLED is saved
cdt_state = CDT_DISABLED
                             CDT_DISABLED is saved
...                          cdt_state remains CDT_DISABLED

cdt_state = CDT_ENABLED      ...
                             cdt_state = CDT_DISABLED

This patch introduces cdt_state_lock to protect the state.

Test-Parameters: trivial testlist=sanity-hsm
Change-Id: I7c976a3a506300de7cf9f5fa1d53741b2e28b654
Signed-off-by: Hongchao Zhang <hongchao.zhang@intel.com>
Reviewed-on: https://review.whamcloud.com/27634
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Quentin Bouget <quentin.bouget@cea.fr>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9474 tests: fix quoting in stack_trap

stack_trap() mishandled single quotes. This patch is not the cleanest
of fixes, but it works.

(sanity-hsm is the only test suite that uses the function, for now)

Test-Parameters: trivial testlist=sanity-hsm
Signed-off-by: Quentin Bouget <quentin.bouget@cea.fr>
Change-Id: Ia43219e57079abdbfc75485105d572bbfa85caba
Reviewed-on: https://review.whamcloud.com/30490
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9019 lnd: remove remaining cfs_time wrappers

Remove remaining libcfs time wrappers from ko2iblnd. Also fix bug
in ksocklnd to use cfs_time_seconds for calling schedule_timeout
instead of cfs_durations_sec. That was the opposite of the
conversion we needed. the remaining jiffy use it moved to
time64_t.

Change-Id: I5847d7260ac8a9be1b165423adb7b8e9a53998d2
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/31042
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10420 flr: split a mirror from mirrored file

Splits a mirror with mirror_id out of a mirrored file.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: Ib9c2ca7deb329ba0f95880ebeee77563317d0fca
Reviewed-on: https://review.whamcloud.com/30388
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10394 lnd: default to using MEM_REG

There is a performance drop when using IB_MR_TYPE_SG_GAPS. To
mitigate this, we added a module parameter, use_fastreg_gaps, which
defaults to 0. When allocating the memory region if this parameter
is set to 1 and the hw has gaps support then use it and output
a warning that performance may drop. Otherwise always use
IB_MR_TYPE_MEM_REG.

Test-Parameters: trivial
Signed-off-by: Amir Shehata <amir.shehata@intel.com>
Change-Id: I08a8b72756b9b5b5bcb391bf3e979f6d28eb5cbb
Reviewed-on: https://review.whamcloud.com/30749
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Doug Oucharek <dougso@me.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

New tag 2.10.58

Change-Id: I8dc17a8e4d8182548ccaafc83d643f5a0a09de9f
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10554 lnet: Remove LASSERT on userspace data

If the net information is not provided while adding
NI, it results in an LBUG. Remove the LASSERT on
userspace input and handle it gracefully.

Test-Parameters: trivial
Change-Id: I9d2b6f94cb35e94bc81d5c52936d32cbf833e597
Signed-off-by: Sonia Sharma <sonia.sharma@intel.com>
Reviewed-on: https://review.whamcloud.com/31100
Tested-by: Jenkins
Reviewed-by: Amir Shehata <amir.shehata@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-8602 gss: Fix autoconf check for crypto_hash

If earlier crypto_hash checks resulted in enable_gss=no, do not enable
GSS when gss_conf_test = success.

Fixes regression introduced by https://review.whamcloud.com/27823/
LU-9073 gss: remove newer kernel support

Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Change-Id: I6c135e638ec6b8350b916f18de73b83cc7dbfb09
Reviewed-on: https://review.whamcloud.com/31095
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10563 kernel: kernel update RHEL7.4 [3.10.0-693.17.1.el7]

update RHEL 7.4 kernel to 3.10.0-693.17.1.el7

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I1302c2de6d8eebb33ca741dd357b65579ce71d7d
Reviewed-on: https://review.whamcloud.com/31034
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10565 mdc: add __GFP_COLD for back compatible

The __GFP_COLD has been removed in upstream. Add
it for compatible.

Signed-off-by: Yang Sheng <yang.sheng@intel.com>
Change-Id: I70d0978fa1cdfb9ea22f0b1511a26b28523048b6
Reviewed-on: https://review.whamcloud.com/31028
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10565 osd: use readdir while iterate is kabi_extend

Sometime iterate interface is not initialized in ldiskfs.
So we use readdir in such case.

Signed-off-by: Yang Sheng <yang.sheng@intel.com>
Change-Id: I43260a6d27003895b0ddd1bdf7b0539cc2ad64c5
Reviewed-on: https://review.whamcloud.com/31018
Tested-by: Jenkins
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10560 libcfs: remove extra headers from linux-debug.c

Remove headers from linux-debug.c that are no longer (or were possibly
never) needed.  This avoids a compile problem with 4.14 kernels:

  CC [M] lustre/libcfs/libcfs/linux/linux-debug.o
  In file included from lustre/libcfs/libcfs/linux/linux-debug.c:50:0:
    ./arch/x86/include/asm/uaccess.h:31:9: In function set_fs():
      error: dereferencing pointer to incomplete type
      current->thread.addr_limit = fs;

It turns out <asm/uaccess.h> was included twice but we don't need
anything from that header.  Same for <linux/stat.h> and a number of
other extraneous headers.

Sort headers alphabetically so it is easier to avoid duplicates.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I751e796913624cd8c9c95052abe4ecbb823ebbe5
Reviewed-on: https://review.whamcloud.com/31007
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10497 tests: remove tests from sanity-dom

sanity test 42a, 42b, and 42c are known to fail and are
not run during normal testing due to LU-6493 and LU-9693.
Thus, remove these sanity tests from running in sanity-dom.

Test-Parameters: trivial testlist=sanity-dom
Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: I2c99f44bcae845fd5f9fb0a17bb09a2b4b254ec9
Reviewed-on: https://review.whamcloud.com/31004
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>

LU-8028 build: fix make dependencies for --disable-modules

When modules are excluded from a build there is no rule to build undef.h.
As result build fails with following:
/usr/include/stdc-predef.h:40:1: fatal error:
        /work/lustre-release/undef.h: No such file or directory

Test-Parameters: trivial
Change-Id: I4bf031933964c7d11f9e1c4a88016e1827d11762
Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-on: https://review.whamcloud.com/30965
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-4277 scripts: ofd status integrated with zpool status

Add zedlet to ZFS ZED that markes OFD as degraded/undegraded,
when a zpool is degraded or online, respectivly.

Test-Parameters: trivial
Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Ia8ec3cf3a31ce24d8598d690bcb0356245712858
Reviewed-on: https://review.whamcloud.com/30907
Tested-by: Jenkins
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10452 lnet: cleanup YAML output

The level of verbosity is high when exporting the YAML configuration
for the purposes of storing it to reconfigure a node. This patch
eliminates the unnecessary YAML lines which are not needed when
reconfiguring a node, such as statistics, status, etc.

Test-Parameters: trivial
Signed-off-by: Amir Shehata <amir.shehata@intel.com>
Change-Id: Ie57c761415cfb0ceee8b2dbc0b293e85ae415685
Reviewed-on: https://review.whamcloud.com/30845
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Doug Oucharek <dougso@me.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9158 quota: adjust quota ASAP

In qsd_upd_thread, the quota adjust request will only be
scheduled to run when the current time (seconds) is larger
than the queued time (seconds). The transactions in subtest 12b
of sanity_quota are committed in one second simultaneously,
which cause the quota is not freed.

Test-Parameters: alwaysuploadlogs \
envdefinitions=ENABLE_QUOTA=yes,DEBUG_SIZE=64,PTLDEBUG=rpctrace \
clientcount=2 osscount=2 mdscount=2 mdtcount=4 \
austeroptions=-R mdtfilesystemtype=zfs ostfilesystemtype=zfs \
testlist=sanity-quota,sanity-quota,sanity-quota,sanity-quota

Change-Id: I9310237d58a21ee8d47daab8901892bd12016339
Signed-off-by: Hongchao Zhang <hongchao.zhang@intel.com>
Reviewed-on: https://review.whamcloud.com/30765
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9378 utils: split getstripe and find from lfs.1

Split the getstripe and find commands from the lfs.1 man page into
their own lfs-getstripe.1 and lfs-find.1 man pages.

While updating the lfs-find.1 man page I realized that the short
options for "-print" and "-print0" were incorrectly documented
in both the usage message as well as the man page, which implies
that the short options were rarely, if ever, used.

Fix the "--print" option to be correctly documented as "-P" instead
of "-p", and deprecate the usage of "-p" for "--print0" in favour
of "-0".  This gives us the opportunity to reclaim "-p" for "--pool",
which is already used as such for "lfs df", "lfs getstripe", and
"lfs setstripe", after some period of printing a deprecation warning.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I9aa7a415d109d269c646fd034ea77785a94cab07
Reviewed-on: https://review.whamcloud.com/30464
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5152 quota: enforce block quota for chgrp

When an unprivileged user calls chgrp to change the group
of one of his files, the block quota limit of that new group
should be checked to ensure it not exceeds the limit.

The side effect of this patch could be,
1.The performance of chgrp from non-privileged user will be
very slow, no matter if quota is enabled. Since we assume that
chgrp issued from non-privileged user is very rare, the performance
impact possibly is acceptable.
2.If MDT crash while performing chgrp, inconsistency (group ownership
among MDT and OST objects) will be created. It should be acceptable.

This patch has fixed the bug while calculating the disk space of
some file for ldiskfs and zfs, the block unit is always 512.

Change-Id: I4b781e94493fe63c8cbd5700dc68293b2504c2ac
Signed-off-by: Hongchao Zhang <hongchao.zhang@intel.com>
Reviewed-on: https://review.whamcloud.com/30146
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10193 tests: test migration between ldiskfs and zfs

New test cases in conf-sanity.sh

test_108a: migrate from ldiskfs to zfs
test_108b: migrate from zfs to ldiskfs

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Ibb4749c316f51b0820648e59235a03a9656f762e
Reviewed-on: https://review.whamcloud.com/30106
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10193 osd-ldiskfs: backup index object with plain format

This patch is mainly for migrating filesyste between ZFS
backend and ldiskfs backend via server file level backup
and restore. It will dumps the ldiskfs special formatted
index object to the local '/index_backup' directory with
the name of source index's FID string and ".lbx" postfix
when umount device.

The format of the backup is as following (same ZFS case):
1) header: 512 bytes, including:
   magic:       4 bytes
   count:       4 bytes
   keysize:     4 bytes
   recsize:     4 bytes
   owner_fid:   16 bytes
   padding:     480 bytes

2) body: after the header, <key, rec> pairs one by one.

The backup will be done when server umount. The backup behavior
is controlled via new OSD lproc interface "index_backup". It is
off by default. You can turn it on to enable backup when server
umount via writing non-zero value to such lproc interface.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I5ac81dd470f3cb29eb3c9ec0e01935c9b1a0fda9
Reviewed-on: https://review.whamcloud.com/30911
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10193 osd-zfs: backup index object with plain format

Lustre uses ZAP to implement index object. When tar the index
object via backend ZPL for backup, it is explained as regular
file, then when untar it, it is not ZAP formatted again, then
the Lustre cannot recognize the 'bad' formatted index object.

On the other hand, each backend FS has its own special format
for index object. Then we cannot migrate the index files from
one backend to another directly.

To resolve such issue, the patch will backup the index object
with plain format to the local '/index_backup' directory with
the name of source index's FID string and ".lbx" postfix when
umount the device.

The format of the backup is as following:
1) header: 512 bytes, including:
   magic:       4 bytes
   count:       4 bytes
   keysize:     4 bytes
   recsize:     4 bytes
   owner_fid:   16 bytes
   padding:     480 bytes

2) body: after the header, <key, rec> pairs one by one.

The backup will be done when server umount. The backup behavior
is controlled via new OSD lproc interface "index_backup". It is
off by default. You can turn it on to enable backup when server
umount via writing non-zero value to such lproc interface.

Test-Parameters: envdefinitions=SLOW=yes testlist=sanity-scrub mdtfilesystemtype=zfs ostfilesystemtype=zfs mdscount=2 mdtcount=4
Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I01730bc9cfa3ae597f2d8652df9fb76418cf55ce
Reviewed-on: https://review.whamcloud.com/30910
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9564 build: Add server-build for Ubuntu with Kernel 4.4.0

This enables compatibility with the current LTS flavours of Ubuntu.
Do note that you need the Xenial HWE Kernel for Ubuntu 14.04.5, as
that distribution originally used a 3.x series Kernel.

The patches have been developed to apply cleanly to the kernel versions
4.4.0-45.66 to 4.4.0-85.108 from the Ubuntu Xenial (and its Trusty backports).

This change also adjusts the Debian scripting to produce the
ldiskfs modules and the server utilities.

To create the server modules run "./configure" with "--enable-server"
and specify "--enable-ldiskfs" and "--with-zfs/-spl" as appropriate.

The call to "make debs" will then produce the server modules and
utils instead of their client versions.

NOTE: This contains a small hack taken from LU-9995 / #29130

Test-Parameters: trivial
Signed-off-by: Martin Schroeder <martin.h.schroeder@intel.com>
Change-Id: I02cd5e9314367ad4e1f8f3d81712f84441a8bc71
Reviewed-on: https://review.whamcloud.com/29215
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Thomas Stibor <t.stibor@gsi.de>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9919 lnet: safe access in debug print

Move debug print within the cpt lock to keep
peer access safe.

Test-Parameters: trivial
Signed-off-by: Amir Shehata <amir.shehata@intel.com>
Change-Id: Ic37ff0973367b3eb9cbc0059ffee9c31ecf98c34
Reviewed-on: https://review.whamcloud.com/28771
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7501 utils: clean up lfs argument handling/docs

Change "mdt-hash" option in lfs_setstripe() and lfs_setdirstripe
to use C99 formatting as used for other options.

Add comments for already-used options to lfs_find(), lfs_getstripe(),
and lfs_setstripe() to avoid conflicts in the future.

A few initializers can fit onto the same line with minor formatting
changes, better to be more compact than a slave to exact formatting.

Remove options that are obsoleted by LUSTRE_VERSION_CODE after 2.10.
Remove over-zealous deprecation of "lfs mkdir -c".

Sort options to be in mostly alphabetical order, unless the long
option parsing would return a deprecated short option.

Add deprecation warnings for short/long options that were deprecated
already in commit cdeb2f3a56e8 (http://review.whamcloud.com/22581).

Fix up lfs-setdirstripe.1 and lfs-getdirstripe.1 man pages to list
preferred option names.  Also, lfs-getdirstripe.1 listed some options
that never existed, and others that were named incorrectly.

Move test scripts over to use preferred command and option names.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I74a59ce372115ae0906d0feb37c539a450bed6bd
Reviewed-on: https://review.whamcloud.com/28592
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9727 nodemap: add audit_mode flag to nodemap

Give the ability to specify an audit_mode flag on a nodemap.
When set to 1, a client pertaining to this nodemap will be able to
record file system access events to the Changelogs, if Changelogs are
otherwise activated.
When set to 0, events are not logged into the Changelogs, no matter
Changelogs are activated or not.
By default, audit_mode flag is set to 1 in newly created nodemap
entries. And it is also set to 1 on 'default' nodemap.

The idea of disabling audit on a per-nodemap basis is that it would
be possible to have some nodes (e.g. backup, HSM agent nodes) that do
not flood the audit logs.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ieb6c461c443b1734312afef44680d903deee5398
Reviewed-on: https://review.whamcloud.com/28313
Reviewed-by: Jean-Baptiste Riaux <riaux.jb@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10513 acl: prepare small buffer for ACL RPC reply

For most of files, their ACL entries are very limited, under
such case, it is unnecessary to prepare very large reply buffer
to hold unknown-sized ACL entries for the getattr/open RPCs.
Instead, we can prepare some relative small buffer, such as the
LUSTRE_POSIX_ACL_MAX_SIZE_OLD (260) bytes, that is equal to the
ACL size before patch 64b2fad22a4eb4727315709e014d8f74c5a7f289.
If the target file has too many ACL entries and exceeds the
prepared reply buffer, then the MDT will reply -ERANGE failure
to the client, and then the client can prepare more large buffer
and try again. Since the file with large ACL is rare case, such
retrying getattr/open RPCs will not affect the real performance
too much.

The advantage is that it reduces the client side RAM pressure.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I4c01b19520cab1cc712e36f3b0225973fba00410
Reviewed-on: https://review.whamcloud.com/28116
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9727 lustre: record CLOSE if OPEN was recorded

Record CL_CLOSE events in changelogs only if file was opened in
write mode, or if CL_OPEN was recorded.
Changelogs mask may change between open and close operations,
but this is not a big deal if we have a CL_CLOSE entry with no
matching CL_OPEN. Plus Changelogs mask may not change often.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I5984a4b07b84d84c3860b9b21abc3b19b7fd9b1a
Reviewed-on: https://review.whamcloud.com/27929
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Matthew S <matthew.sanderson@anu.edu.au>
Reviewed-by: Li Xi <lixi@ddn.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-9727 lustre: implement CL_OPEN for Changelogs

Record OPEN events in Changelogs, and add a new changelog
extension named changelog_ext_openmode to hold open mode.
An OPEN changlog entry is in the form:
7 10OPEN  13:38:51.510728296 2017.07.25 0x242
t=[0x200000401:0x2:0x0] ef=0x7 u=500:500 nid=10.128.11.159@tcp m=-w-
By default, disable recording of OPEN events in Changelogs.
Note that CREAT are still recorded even if OPEN are disabled.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I72c479938ab4782523f1b16aef19fbbc96f43c7f
Reviewed-on: https://review.whamcloud.com/28214
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9727 lustre: add client NID to Changelogs entries

Add a new changelog extension named changelog_ext_nid to hold
client's NID information.
NID info is added to every Changelog entry type except MARK, in
the form 'nid=<nid>':
1 01CREAT 15:50:20.834838318 2017.07.24 0x0 t=[0x200000401:0x2:0x0]
ef=0x3 u=500:500 nid=10.128.11.159@tcp p=[0x200000007:0x1:0x0] fileA

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I1049a699c17d3829d38abfade3187a28ca457bd1
Reviewed-on: https://review.whamcloud.com/28213
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Quentin Bouget <quentin.bouget@cea.fr>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-3397 lprocfs: create "export" /proc file on server

Similar to the "import" file on the client for each client-to-server
connection, it would be useful to have a file on the server in the
per-nid directory obdfilter/*/exports/$NID/export. This contains
export connection information as in the "import" file, like:
 a793e354-49c0-aa11-8c4f-a4f2b1a1a92b:
     name: MGS
     client: 10.211.55.10@tcp
     connect_flags: [ version, barrier, adaptive_timeouts, ... ]
     connect_data:
        flags: 0x2000011005002020
        instance: 0
        target_version: 2.10.51.0
        export_flags: [ ... ]

Also, sanity.sh test_0d is added to verify this patch.

Signed-off-by: Emoly Liu <emoly.liu@intel.com>
Signed-off-by: James Simmons <uja.ornl@gmail.com>
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I60896090e3a8ad872141a8d4299f0698f0a5636a
Reviewed-on: https://review.whamcloud.com/6713
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>

LU-10003 lnet: clarify lctl deprecation message

Print out the lctl command which is deprecated

Signed-off-by: Amir Shehata <amir.shehata@intel.com>
Change-Id: I2d07a609718c205fba172530202e6f0c1b1d2119
Reviewed-on: https://review.whamcloud.com/31030
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Olaf Weber <olaf.weber@hpe.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-6349 idl: add PTLRPC definitions to enum

Add PTLRPC definitions to enums for cleanliness, and use them:

    enum lustre_msg_version for LUSTRE_*_VERSION
    enum lustre_msg_magic for LUSTRE_MSG_MAGIC_*
    enum lustre_msghdr for MSGHDR_*

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: Ie17619bfc2433339ab32e257d596adf64c2c4ff0
Reviewed-on: https://review.whamcloud.com/30958
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10418 flr: revise lease API

Introduce two lease API llapi_lease_{acquire,release}() to replace
confusing llapi_lease_{get,put}(). Rename llapi_lease_ext_get() to
llapi_lease_set(). Implement new mirror_extend_layout() to separate
lfs_migrate().

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I88ab1f7f27aa81c44418aecf31c9e89e494fc01e
Reviewed-on: https://review.whamcloud.com/30363
Tested-by: Jenkins
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-9972 osd: cache OI mapping in dt_declare_ref_add

so that subsequent calls to manipulate /PENDING don't need to
consult with OI.
use OIC in osd_index_ea_delete() to optimize FLDB lookups.

Change-Id: I779bbddb429b577aecf1ad092d74d0802e43d567
Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-on: https://review.whamcloud.com/29709
Tested-by: Jenkins
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7934 tests: compatibility check

The patch adds compatibility check for test 103 replay-single.

Signed-off-by: Alexander Boyko <alexander.boyko@seagate.com>
Test-Parameters: trivial testlist=replay-single envdefinitions="ONLY=103"
Change-Id: I8c3457a2d18c32d9d1e701003c282e893692f5fc
Reviewed-on: https://review.whamcloud.com/20847
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10531 gss: fix GSS support for DNE

Part of the patch https://review.whamcloud.com/24236 was
inadvertently reverted by patch https://review.whamcloud.com/27823.
So re-apply missing part to have full DNE support in GSS code.

This patch is necessary because with DNE, an OSP can be used on MDT
for connections to other MDTs. So to determine exactly every
connection's purpose, use client_obd's cl_sp_to in
import_to_gss_svc().

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I029dbfc52eeabb2c6f0f6d2c972ceea1879405e7
Reviewed-on: https://review.whamcloud.com/30984
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>

LU-10562 tests: fix non-portable bash redirection

Replace non-portable bash redirection with a simple(r) array
assignment.  This was reporting for at least bash 4.4.26:

   line 5211: syntax error near unexpected token `<'

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I8289a4a3a1518c3cb6f33e2d8e52ba22db3ebbe5
Reviewed-on: https://review.whamcloud.com/31050
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Cliff White <cliff.white@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10536 build: add -lnvpair to ZFS LDFLAGS

The ZFS mount utils plugin directly depends on libnvpair so add
-lnvpair to the relevant LDFLAGS. This makes building with
--disable-shared succeed.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ie7d7043b011dab3e139786ed8fffb09e1078a34b
Reviewed-on: https://review.whamcloud.com/30947
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10531 obd: handle case tgt equals fsname for obdname2fsname

The function obdname2fsname() was updated to handle the case of
when tgt was for the format:

llite.lustre*.xattr_cache
llite.lustre-*.xattr_cache

but the case of tgt being exactly equal to the fsname is also
valid as in the case of sptlrpc: fsname.srpc.flavor.default.cli2ost

Change-Id: I9bb4082ee5f194842c7d72ae25e14f6998c0b8ed
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/30937
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9859 libcfs: merge UMP and SMP libcfs cpu header code

Currently we have two headers, linux-cpu.h that contains the SMP
version and libcfs_cpu.h contains the UMP version. We can simplify
the headers into a single header which handles both cases. Many
cleanups for checkpatch violations are done as well.

Change-Id: Ifb92e59ad370e991220b5957538b6bab89423e2f
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/30873
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10485 lustre: move LA_* flags to lustre_user.h

The LA_* flags are written to disk as part of the ChangeLog records
in mdd_attr_set_changelog(), which means they are now part of the
on-disk and network protocol, and cannot be changed (at least not the
first 12 bits that are written).
They need to be moved to lustre_user.h.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I9fc92e01301e70f0f4e5cd74135b9b2079d63658
Reviewed-on: https://review.whamcloud.com/30825
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Quentin Bouget <quentin.bouget@cea.fr>

LU-10464 kernel: kernel update [SLES12 SP2 4.4.103-92.56]

Update target and kernel_config files for new version

Test-Parameters: clientdistro=sles12sp2 testgroup=review-ldiskfs \
  mdsdistro=sles12sp2 ossdistro=sles12sp2 \
  mdtfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I0eb5dc783bcbcd4ee8643431345daefb9cb89eb1
Reviewed-on: https://review.whamcloud.com/30748
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10155 recovery: support setstripe replay

Regular file open will always reserve space for LOV ea, which is used
to store user specified lov_user_md, or lov_mds_md for replay, but if
this open is the first open in 'lfs setstripe', it doesn't have
lov_user_md specified, or lov_mds_md for replay because
O_LOV_DELAY_CREATE is set, but MDT will treat the EA field in the
request as valid one, so fails in magic check in this open replay.

This patch fixes this issues on both sides:
1. client doesn't reserve space for LOV ea in
   open(O_LOV_DELAY_CREATE), this change is not necessary, but to
   make clean of the code.
2. server doesn't create OST objects for open(O_LOV_DELAY_CREATE)
   replay.

Add setstripe/setdirstripe replay test in replay-single.sh, which
are 2c, 2d.

Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Change-Id: Ia7971523710822308328239f36f7d690314e0e45
Reviewed-on: https://review.whamcloud.com/30704
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10287 flr: lfs mirror verify command

This patch adds "lfs mirror verify" command to verify
that each SYNC mirror of a mirrored file contains exactly
the same data. It supports specifying multiple mirrored
files in one command line.

Usage:

lfs mirror verify [--only <mirror_id,mirror_id2[,...]>]
  [--verbose|-v]
  <mirrored_file> [<mirrored_file2> ...]

Options:

--only <mirror_id,mirror_id2[,...]>
Only verify the mirrors specified by mirror_ids contain
exactly the same data. This option cannot be used when
multiple mirrored files are specified.

--verbose|-v
With this option, the command will print where the
differences are if the data do not match. Otherwise,
the command will just return an error in that case.

Test-Parameters: testlist=sanity-flr
Signed-off-by: Jian Yu <jian.yu@intel.com>
Change-Id: Ib436dea73a8e7a0f8e30b246bb0062023a97cb81
Reviewed-on: https://review.whamcloud.com/30387
Tested-by: Jenkins
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10123 lnet: ensure peer put back on dc request queue

Upon async PUT request received from peer already in discovery
process, lnet_peer_push_event() was not handling the case where
peer could be on working/ln_dc_working queue. This could lead
for peer not to be re-dsicovered as expected, but left on
working queue and to be finally timed-out.

Also ensure that peer will not be put back on request queue by
event handler if discovery is already completed.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: Ic74a313c00edc1b8fdd14794d2c88411d12e0979
Reviewed-on: https://review.whamcloud.com/30147
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Amir Shehata <amir.shehata@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Olaf Weber <olaf.weber@hpe.com>
Reviewed-by: Doug Oucharek <dougso@me.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-7585 zfs: OI scrub for ZFS

The ZFS backend OI scrub is used for verifying OI mappings
consistency. ZFS has some mechanism to maintion the data
integrity, but there is still possible data corruption,
especially consider the data migration from other backend,
such as ldiskfs, or server side data backup and restore.
The OI scrub can check OI mappings consistency and rebuild
them when needed.

The ZFS backend OI scrub shares the same control interface
as ldiskfs backend. It can be triggered manually via the
lctl command:
lctl lfsck_start -M $device -t scrub

It also can be triggered automatically when inconsistency
detected if you do not disable 'auto_scrub' that can be
controlled via:
lct set_param -n osd-zfs.*.auto_scrub_interval=xxx

You can check the OI scrub status similar as you do for
ldiskfs backend:
lctl get_param -n osd-zfs.*.oi_scrub

Test-Parameters: envdefinitions=SLOW=yes testlist=sanity-scrub mdtfilesystemtype=zfs ostfilesystemtype=zfs mdscount=2 mdtcount=4
Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I59ae3142ecd7b27f48b14f2a2d1d110d9c8296e3
Reviewed-on: https://review.whamcloud.com/30909
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9372 ptlrpc: allow to limit number of service's rqbds

This patch provides a way to limit the number of rqbds per
service.
This should help to avoid OOM during heavy clients
requests load, like during target failover/recovery for
thousands of Clients.
This change has been required, even after first patch for
LU-9372 (ptlrpc: drain "ptlrpc_request_buffer_desc" objects)
which already allowed to drain unused rqbds previously
allocated during heavy load, but was not efficient during
too long period of load.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: Ib43f3e07741b9fcecdfae24a3753128a939d2196
Reviewed-on: https://review.whamcloud.com/29064
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9735 compat: heed the fs_struct::seq

2.6.37 kernel uses a seqlock in the fs_struct to enable us to take
an atomic copy of the complete cwd and root paths.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I35384b8f5c468a8c142a59032f3148b698a0c79e
Reviewed-on: https://review.whamcloud.com/28907
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10045 obdclass: multiple try when register target

It is possible that the connection between MGC and MGS has not
been established when register target to MGS for server mount.
At that time, the ptlrpcd may be trying to (re-)connect to MGS
at background. Under such case, the mount process should not
report failure (-ESHUTDOWN -r -EIO), instead, it can retry the
MGS_TARGET_REG RPC after sometime (such as 2 seconds).

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I44e53a9d1de037907bdb5148b8c44d332439a50c
Reviewed-on: https://review.whamcloud.com/30761
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10504 flr: check layout pointer before using it

This patch fixes mirror_create() to check layout pointer
before using it.

Change-Id: Ia1454b5c7fcfcee227d0b954a477cefe5d7bb5f7
Test-Parameters: trivial testlist=sanity-flr
Signed-off-by: Jian Yu <jian.yu@intel.com>
Reviewed-on: https://review.whamcloud.com/30915
Tested-by: Jenkins
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10503 flr: fix error handling in mirror_extend_file()

This patch fixes the error handling issues in mirror_extend_file().

Change-Id: I388295886657cf9b9b072017002be937a9e657c0
Test-Parameters: trivial testlist=sanity-flr
Signed-off-by: Jian Yu <jian.yu@intel.com>
Reviewed-on: https://review.whamcloud.com/30914
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Tested-by: Jenkins
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10515 utils: remove obd.c dependencies from lustre_rsync

lustre_rsync does not depend on /dev/obd and therefore should not call
register_ioc_dev(OBD_DEV_ID, OBD_DEV_PATH). So remove the
call. Replace erroneous uses of D_TRACE with the locally defined
DTRACE constant. Remove unneeded includes.

Test-Parameters: trivial testlist=lustre-rsync-test

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I3251469d48f4c60106dd14d1bf97e159a147688c
Reviewed-on: https://review.whamcloud.com/30870
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-5541 lustreapi: only export the API symbols

By default, all kind of symbols are exported from the library (dump,
libcfs_ukuc_start, l_ioctl, set_ioctl_dump, ...), which may create
external conflicts. Use the linker version-script options to only
export the API symbols, and prevent the export of internal symbols.

Only the symbols declared in the global section of liblustreapi.map
will be seen by applications.

Test-Parameters: trivial

Change-Id: I4fc2febd2528fc85f546426e08e3ab67e1305c40
Signed-off-by: frank zago <fzago@cray.com>
Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-on: https://review.whamcloud.com/30865
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10462 lod: Correct lfs --component-add striping

lfs --component-add -E <end> -c -1 would apply the default
stripe count to a file when it should be striping across
all OSTs. This was due to incorrect handling of sentinel
values in various portions of lod_object.

Signed-off-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Change-Id: I178f13be81ee546f44edcdadcf7086a1a57f7e9a
Reviewed-on: https://review.whamcloud.com/30790
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10456 kernel: kernel update RHEL6.9 [2.6.32-696.18.7.el6]

Update RHEL6.9 kernel to 2.6.32-696.18.7.el6

Test-Parameters: clientdistro=el6.9 mdsdistro=el6.9 \
  ossdistro=el6.9 mdtfilesystemtype=ldiskfs \
  ostfilesystemtype=ldiskfs testgroup=review-ldiskfs

Change-Id: Ib0b7d5ff7f50f96c8bbc064541ae5c8bf2a4367f
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-on: https://review.whamcloud.com/30736
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10505 scripts: avoid version dependent shell script syntax

Stop using -1 as an array index in lfs_migrate.
This isn't a valid construct in all supported bash versions.

Test-Parameters: trivial clientdistro=el6.9

Change-Id: I5500af40926e0fdb2a432c6bae7fbbe05097ec7c
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-on: https://review.whamcloud.com/30860
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-9411 tests: remove spaces around '+='

In sanity test 27D, the variable that collects tests to
skip needs to have spaces removed when the variable is
being concatenated with new tests to skip.

Test-Parameters: trivial
Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: Ie57efecd2867b748f56998bc2fc375ea9d566611
Reviewed-on: https://review.whamcloud.com/30725
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Saurabh Tandan <saurabh.tandan@intel.com>
Reviewed-by: Wei Liu <wei3.liu@intel.com>
Reviewed-by: Cliff White <cliff.white@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

LU-10405 lov: fill no-extent fiemap on object with no stripe.

This is useful for cp of large sparse files with no stripe info,
as cp relies on fiemap to detect what to read.

Add test 130f: fiemap for unstriped file

Change-Id: I12802c5b1cdec6baf5ee3cec6c706d92d9be4f63
Signed-off-by: Dominique Martinet <dominique.martinet@cea.fr>
Reviewed-on: https://review.whamcloud.com/30591
Tested-by: Jenkins
Reviewed-by: Quentin Bouget <quentin.bouget@cea.fr>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-9934 build: address issues raised by gcc7

Starting with gcc version 7 several platforms have enabled new
flags to report potential problems when compling code. For lustre
much of the reported problems deal with potential buffer overruns.
Also we have unused data structures and are not properly
initializing some data structures.

Change-Id: I10243ea88f2c726032d179febdbf26f28de13715
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/30376
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>