LU-12137 osd-ldiskfs: create locked and unlocked versions of osd lookup code

The inode_lock is expected to be taken when lookup_one_len() is
called. Rename osd_ios_lookup_one_len() to osd_lookup_one_len()
since it now doesn't take the inode_lock manually. Instead it
expects the called to have already called inode_lock(). The
osd_lookup_one_len_unlocked() function will always take the
inode_lock itself. Currently osd-ldiskfs always uses
osd_lookup_one_len_unlocked().

Change-Id: Ie406db5e24f983e52ea8a89bca66546ab9d25148
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/35594
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12137 osd-ldiskfs: trace the obd device used for osd lookup

If a osd lookup of a dentry fails a debug message is logged but
no information is given for which OSD device this happened on.
Add in the osd device information for debugging purposes.

Change-Id: If7719987682c4b9f8219c08431337306bd91ab67
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/35582
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12137 osd-ldiskfs: refactor osd_ios_ROOT_scan

With the change of osd_ios_scan_one() to handle a NULL
dchild->d_inode we can migrate the scrub code after calling
osd_ios_scan_one(). This can help reduce the code indentation
as well as make the code clearer.

Change-Id: Ieb59aa85c8199a1f1a1bee5cbe633e52b8e92a9a
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/35803
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12400 ptlrpc: Sun RPC changes for RCU locking

In kernel 4.20 SUNRPC cache_detail->hash_lock changed to spinlock_t

  Now that the reader functions are all RCU protected, use a regular
  spinlock rather than a reader/writer lock.

Linux-commit: 1863d77f15da0addcd293a1719fa5d3ef8cde3ca

Test-Parameters: trivial
Cray-bug-id: LUS-7600
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Change-Id: If0df38337d5a2bb0ac4b8cb645dbe89f65e0f352
Reviewed-on: https://review.whamcloud.com/35499
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12518 readahead: convert stride page index to byte

This is a prepared patch to support unaligned stride readahead.
Some detection variables are converted to byte unit to be aware
of possible unaligned stride read.

Since we still need read pages by page index, so those variables
are still kept as page unit. to make things more clear, fix them
to use pgoff_t rather than unsigned long.

Change-Id: Ic0d0b1b2f11cb6ef5753814a5b8fc6d34afb25ea
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/35829
Reviewed-by: Li Xi <lixi@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12535 lov: Move page index to top level

When doing readahead, we see an amazing amount of time
(~5-8%) just looking up the page index from the lov layer.

In particular, this is more than half the time spent
submitting pages:
         - 14.14% cl_io_submit_rw
            - 13.40% lov_io_submit
               - 8.24% lov_page_index

This requires several indirections, all of which can be
avoided by moving this up to the cl_page struct.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I99bd7eb4d6556ac89c1aa9aeb4b7afc99774b212
Reviewed-on: https://review.whamcloud.com/35470
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12533 llite: Improve readahead RPC issuance

lov_io_submit receives a range of pages, then adds pages in
to a batch until it hits a page which is not in the stripe
associated with this lov object.  This means that if a
readahead page range hits the same stripe more than once,
we will issue multiple I/Os, even if the pages would fit in
one RPC.

This is unnecessary - Just submit all these pages at once.

mpirun -n 2 $IOR -s 2000 -t 47K -b 47K -k -r -E -o $FILE

Without patch:
osc.lustre-OST0001-osc-ffff8fe82c952000.rpc_stats=

                        read                    write
pages per rpc         rpcs   % cum % |       rpcs   % cum %
1:                     118  56  56   |          0   0   0
2:                       0   0  56   |          0   0   0
4:                       0   0  56   |          0   0   0
8:                       0   0  56   |          0   0   0
16:                      5   2  58   |          0   0   0
32:                      0   0  58   |          0   0   0
64:                      0   0  58   |          0   0   0
128:                    21  10  68   |          0   0   0
256:                    25  11  80   |          0   0   0
512:                    10   4  85   |          0   0   0
1024:                   31  14 100   |          0   0   0

osc.lustre-OST0002-osc-ffff8fe82c952000.rpc_stats=
                        read                    write
pages per rpc         rpcs   % cum % |       rpcs   % cum %
1:                       5   6   6   |          0   0   0
2:                       0   0   6   |          0   0   0
4:                       0   0   6   |          0   0   0
8:                       0   0   6   |          0   0   0
16:                      0   0   6   |          0   0   0
32:                      0   0   6   |          0   0   0
64:                      0   0   6   |          0   0   0
128:                    19  23  29   |          0   0   0
256:                    19  23  52   |          0   0   0
512:                     5   6  58   |          0   0   0
1024:                   34  41 100   |          0   0   0

With patch:
osc.lustre-OST0001-osc-ffff8fe7a7227800.rpc_stats=
                        read                    write
pages per rpc         rpcs   % cum % |       rpcs   % cum %
1:                      12  17  17   |          0   0   0
2:                       0   0  17   |          0   0   0
4:                       0   0  17   |          0   0   0
8:                       0   0  17   |          0   0   0
16:                      5   7  24   |          0   0   0
32:                      0   0  24   |          0   0   0
64:                      5   7  31   |          0   0   0
128:                     6   8  40   |          0   0   0
256:                     1   1  42   |          0   0   0
512:                     2   2  44   |          0   0   0
1024:                   38  55 100   |          0   0   0

osc.lustre-OST0002-osc-ffff8fe7a7227800.rpc_stats=
                        read                    write
pages per rpc         rpcs   % cum % |       rpcs   % cum %
1:                       0   0   0   |          0   0   0
2:                       0   0   0   |          0   0   0
4:                       0   0   0   |          0   0   0
8:                       0   0   0   |          0   0   0
16:                      0   0   0   |          0   0   0
32:                      0   0   0   |          0   0   0
64:                      4   7   7   |          0   0   0
128:                     7  13  21   |          0   0   0
256:                     0   0  21   |          0   0   0
512:                     3   5  26   |          0   0   0
1024:                   38  73 100   |          0   0   0

Note the much larger # of smaller RPC issued without the patch.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Ic10c138628c269afe57fbc57ec8c91ce990717f9
Reviewed-on: https://review.whamcloud.com/35458
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12043 llite: extend readahead locks for striped file

Currently cl_io_read_ahead() can not return locks
that cross stripe boundary at one time, thus readahead
will stop because of this reason.

This is really bad, as we will stop readahead every
time we hit stripe boundary, for example default stripe
size is 1M, this could hurt performances very much
especially with async readahead introduced.

So try to use existed locks aggressivly if there is no
lock contention, otherwise lock should be not
less than requested extent.

Change-Id: I8b2dcd0e80138ea530272cab6a665981aa00cca8
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/35438
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-11780 tests: add server version check to replay-single

replay-single test 132a was added to Lustre 2.12.0. Thus,
this test should be skipped for all servers with version less
than 2.12.0.

Fixes: e5abcf83c057 (LU-11158 mdt: grow lvb buffer to hold layout)
Test-Parameters:trivial envdefinitions=ONLY=132a serverjob=lustre-b2_10 serverbuildno=170 testlist=replay-single
Test-Parameters:envdefinitions=ONLY=132a testlist=replay-single
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I8fb69c4ab6e08a4be7619b70c0a54d49ebfebf64
Reviewed-on: https://review.whamcloud.com/34590
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12719 obdclass: serialize lwp list access

lustre_sb_info.lsi_lwp_list should be acessed with lock, and
some place may sleep, change lsi_lwp_lock from spinlock to mutex.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Ifc3622eb28cd6cf49661b14fc10e98aa689a58dc
Reviewed-on: https://review.whamcloud.com/36003
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Li Xi <lixi@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-6142 mdt: remove unused field mti_wait_info.

This field is neither set nor accessed.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I7b8ccd97ad1228bbd4e8577a6d6003a9888f9282
Reviewed-on: https://review.whamcloud.com/35921
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-6142 mtd: use tabs to indent struct mdt_thread_info

Prior to a (small) code change, change indenting of
struct mtd_thread_info to consistently use TABs.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: Ic0f29a8b84640862294610e37a941b77292446e7
Reviewed-on: https://review.whamcloud.com/35920
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>

LU-6142 socklnd: remove ksnp_sharecount

This field is never set, though its value is printed.
Remove it.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: Ie1d8987d8c4981b6ac9204c27124a536bef969b5
Reviewed-on: https://review.whamcloud.com/35892
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-6142 socklnd: Fix some white-space issues ...

... in ksocknal_debug_peerhash()

Prior to a code change, fix up the white-space here.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I1d492bbb8be608a297d641111eb6238a4fd2f38e
Reviewed-on: https://review.whamcloud.com/35919
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-6142 o2iblnd: remove some unused fields.

Fields kib_min_reconnect_interval kib_max_reconnect_interval kib_ntx
are never used or set.

ibh_mr_shift is set but never used;
rx_status is used (in a debug message) but never set.

Remove them all.

We could possibly remove ibh_mr_size too. It is only used
for an error message.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: If1ff73c15f8e712be4e3d77e9572a4d4e741c75d
Reviewed-on: https://review.whamcloud.com/35891
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-6142 osc: remove oe_next_page

As the comment says, this field is unused.  So remove it.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: Ibf4ccc9c09c4efcf5c768e71e0fd448958a943f3
Reviewed-on: https://review.whamcloud.com/35890
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>

LU-6142 osc: remove oti_descr oti_handle oti_plist

These three fields in 'struct osc_thread_info' are
unused, so remove them.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I8642ebd692f44eca9d9ddad3b8184cabb27d27b3
Reviewed-on: https://review.whamcloud.com/35889
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>

LU-6142 llite: remove sub_reenter field.

This field is never set or accessed, so
remove it.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I34e8d59d068e18ff65660924b5b984f0ab45510f
Reviewed-on: https://review.whamcloud.com/35888
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>

LU-6142 llite: remove ft_mtime field

This field is set but never accessed, so remove it.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: If6f96e90309235a61c833e262e50b03cf36132b0
Reviewed-on: https://review.whamcloud.com/35887
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>

LU-6142 llite: remove lti_iter field

This field is never used, so remove it.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I26489ce4200969b746b2214eaa5c07bdaddf2baf
Reviewed-on: https://review.whamcloud.com/35886
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>

LU-6142 llite: remove ll_umounting field

This field is set but never accessed, so remove it.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I27d8388b73fea3821ff2ecfd85b28c3cdd13193f
Reviewed-on: https://review.whamcloud.com/35885
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>

LU-6142 llite: remove lli_readdir_mutex

This mutex is initialized but never used, so
remove it.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I8df34923c44892dbf3a4e0bb603209cfe28adb1b
Reviewed-on: https://review.whamcloud.com/35884
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>

LU-6142 ldlm: remove unused ldlm_server_conn

This field is never set or used, so remove it.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I1421914e487372eecb501e2483aeed07f56788e8
Reviewed-on: https://review.whamcloud.com/35883
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>

LU-6142 ptlrpc: remove scp_nthrs_stopping field.

This field is unused, so remove it.
If "shrinking threads" is ever needed, any extra fields
required can be added then.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: Ie727033133aa846d92fdc52f6545bc1cd5efe153
Reviewed-on: https://review.whamcloud.com/35882
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>

LU-6142 ptlrpc: remove srv_threads from struct ptlrpc_service

The threads are not stored here - nothing is.
Threads are stored in svcpt->scp_threads.
So remove the field and update the comment.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I46875f77bb017bec47f901eca8dc946f11488cbe
Reviewed-on: https://review.whamcloud.com/35881
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>

LU-6142 ptlrpc: remove bd_import_generation field.

This field is set, but never accessed. So remove it.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I26791e9fbc50676a705a94755674945eea9bf262
Reviewed-on: https://review.whamcloud.com/35880
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>

LU-6142 ptlrpc: remove struct ptlrpc_bulk_page

This structure is never used, so remove it.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I2a4fbfe2701636670801adcd0c2f2d991598f8f6
Reviewed-on: https://review.whamcloud.com/35879
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>

LU-6142 llog: remove olg_cat_processing field.

This mutex is initialized but never used.
Remove it.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: Id6fbbfdc5344435627b88104c46785e8d95e4ab1
Reviewed-on: https://review.whamcloud.com/35878
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>

LU-6142 lustre: remove imp_no_timeout field

This field is never set and never used.  Remove it.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I1dddbe620405b49e90530adcb9a7dec7f9ece64f
Reviewed-on: https://review.whamcloud.com/35877
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>

LU-6142 lustre: remove ldt_obd_type field of lu_device_type

This field is never set, so it is always NULL.
So remove it,
 and the one place it is used,
 and a variable that now will now never be set.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I5530dd92b653b4259ddcbf8dc21af03a79ed255a
Reviewed-on: https://review.whamcloud.com/35876
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-6142 fld: remove fci_no_shrink field.

This field is never set, so is always zero.
Remove it, and the one place where it is tested.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I4e576cc1984e0fac829a65c6b53d6ddac0d0ee88
Reviewed-on: https://review.whamcloud.com/35875
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>

LU-11421 dom: manual OST-to-DOM migration via mirroring

Allow DOM mirroring, update LOV/LOD code to check not just
first component for DOM pattern but cycle through all mirrors
if any. Sanity checks allows one DOM component in a mirror
and it should be the first one. Multiple DOM components are
allowed only with the same for now.

Do OST file migration to MDT by using FLR. That can't be done
by layout swapping, because MDT data will be tied to temporary
volatile file but we want to keep data with the original file.
The mirroring allows that with the following steps:
- extent layout with new mirror on MDT, no data is copied but
  new mirror stays in 'stale' state. The reason is the same
  problem with volatile file.
- resync mirrors, now new DOM layout is filled with data.
- remove first mirror

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I1d4213196a16d6aec70861c5530910cac062e34f
Reviewed-on: https://review.whamcloud.com/35359
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12443 ptlrpc: fix reply buffers shrinking and growing

The req_capsule_shrink() doesn't update capsule itself with
new buffer lenghts after the shrinking. Usually it is not
needed because reply is packed already. But if reply buffers
are re-allocated by req_capsule_server_grow() then non-updated
lenghts from capsule are used causing bigger reply message.
That may cause client buffer re-allocation with resend.

Patch does the following:
- update capsule lenght after the shrinking
- introduce lustre_grow_msg() to grow msg field in-place
- update req_capsule_server_grow() to use generic
  lustre_grow_msg() and make it able to grow reply without
  re-allocation if reply buffer is big enough already
- update sanity test 271f to use bigger file size to exceed
  current maximum reply buffer size allocated on client.

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I154c55d98f41406d0c932c7e8705e0ecf3dfa935
Reviewed-on: https://review.whamcloud.com/35243
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Tested-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10931 tests: resume testing of recovery-small 136

recovery-small test 136 was skipped, added to the
ALWAYS_EXCEPT list, due to excessive failures.  A fix
for failures has landed and we need to start running
test 136 again.

Test-Parameters: trivial
Test-Parameters: fstype=zfs mdscount=2 mdtcount=4 testlist=recovery-small
Test-Parameters: mdscount=2 mdtcount=4 testlist=recovery-small
Test-Parameters: fstype=zfs mdscount=1 mdtcount=1 testlist=recovery-small
Test-Parameters: mdscount=1 mdtcount=1 testlist=recovery-small
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I2ae8578e85024b31e226807e743a154707989ad4
Reviewed-on: https://review.whamcloud.com/35949
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12705 build: fix building fail against Power9 little endian

We use "%ll[dux]" for __u64 variable as an input/output modifier,
this may cause building error on some architectures which use "long"
for 64-bit types, for example, Power9 little endian.
Here add necessary typecasting (long long/unsigned long long) to
make the build correct.

Test-Parameters: trivial
Change-Id: I2e8569f4ac14f7d328a29d153ff57c7834cabc46
Signed-off-by: Gu Zheng <gzheng@ddn.com>
Reviewed-on: https://review.whamcloud.com/36007
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>

LU-12602 mdt: more EA size check in mdt_getxattr_pack_reply()

While the RMF_EAVALS field size can be arbitrary length,
the RMF_EAVALS_LENS field definition specifies
the RMF_F_STRUCT_ARRAY flag, so the passed size must be a multiple
of sizeof(__u32) or the internal LBUG() will trigger.

Signed-off-by: Emoly Liu <emoly@whamcloud.com>
Change-Id: I767e1b1496298e9a66274fc324f9c34daaed4a09
Reviewed-on: https://review.whamcloud.com/36103
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>

LU-12613 ptlrpc: check buffer length in lustre_msg_string()

Check buffer length in lustre_msg_string() in case of any invalid
access.

Change-Id: I286000db16384938a594bd8d104e5f3d0fff585a
Reported-by: Alibaba Cloud <yunye.ry@alibaba-inc.com>
Signed-off-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/35932
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Yunye Ry <yunye.ry@alibaba-inc.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-12635 lnet: Fix deceptive indenting on for_each

This patch fixes some deceptive indentation not reported by checkpatch
introduced by the previous cleanup

Test-Parameters: trivial
Cray-bug-id: LUS-7690
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Change-Id: I568637c508c2c5fcb84810e8e6deae2fb082e069
Reviewed-on: https://review.whamcloud.com/36086
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9897 utils: remove llverfs dependency on libext2fs

The llverfs tool can be built with or without libext2fs.

If built without libext2fs then it estimates how many subdirectories
are needed based on the total filesystem size and default settings.
In this case, it calls "chattr" directly to set the "TOPDIR" flag so
directories are still spread across the filesystem, if possible.

Don't unnecessarily seek() the file descriptor if this is not needed.

Improve stats output so it isn't negative if the IO rate changes, and
always print at least _something_ for read and write when run.

Test-Parameters: trivial testlist=conf-sanity
Signed-off-by: Andreas Dilger <adilger@dilger.ca>
Change-Id: Ia1d51ec8d68d3068a9c31e966935149f123ebbe5
Reviewed-on: https://review.whamcloud.com/36016
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12428 tests: fix sanity-sec wait_nm_sync

There are 2 different versions of wait_nm_sync in sanity-sec.sh,
because of a bad patch rebase.

Test-Parameters: trivial
Test-Parameters: combinedmdsmgs=true mdscount=2 mdtcount=4 osscount=1 ostcount=8 testlist=sanity-sec,sanity-sec,sanity-sec
Test-Parameters: combinedmdsmgs=true mdscount=2 mdtcount=4 osscount=1 ostcount=8 testlist=sanity-sec,sanity-sec,sanity-sec
Test-Parameters: combinedmdsmgs=true mdscount=2 mdtcount=4 osscount=1 ostcount=8 testlist=sanity-sec,sanity-sec,sanity-sec
Fixes: bb0a10752850 ("LU-11883 nodemap: make deny_unknown visible on default nodemap")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I25edb19ce29b9380ad5721b550a6816899d9f95b
Reviewed-on: https://review.whamcloud.com/36009
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12608 kernel: kernel update RHEL7.6 [3.10.0-957.27.2.el7]

Update RHEL7.6 kernel to 3.10.0-957.27.2.el7.

Test-Parameters: clientdistro=el7.6 serverdistro=el7.6

Change-Id: I7b065a0476fafd46a3bf2e8f9712e214c30da271
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/35638
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-11915 tests: use trusted.* xattr for conf-sanity test_115

conf-sanity test 115 is always skipped because of patch
"LU-8569 linkea: linkEA size limitation". Links could not be
used for creating external inode for xattrs.

Let's use "trusted.*" xattrs for creating external xattr inode.

Fixes: 048a8740ae26 ("LU-8569 lfsck: handle linkEA overflow")

Signed-off-by: Artem Blagodarenko <c17828@cray.com>
Cray-bug-id: LUS-6966
Change-Id: If4c74c5baaeec6503edb3c5e756b9a55ff1bd90a
Test-Parameters: trivial testlist=conf-sanity envdefinitions=FLAKEY=false
Reviewed-on: https://review.whamcloud.com/34948
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12336 build: Revert Update ZFS version to 0.8.1

Revert "LU-12336 build: Update ZFS version to 0.8.1"

We've seen an uptick in autotest testing failures for ZFS
testing and are investigating if moving to ZFS 0.8.1 is a
cause of these failures.  Thus, run the ZFS test sessions
multiple times to get a baseline of test failures with
ZFS 0.7.13.

This reverts commit dba32e417635359b1d68180b77193e1c9ddd1e8f.

Test-Parameters: trivial testgroup=review-dne-zfs-part-1
Test-Parameters: testgroup=review-dne-zfs-part-2
Test-Parameters: testgroup=review-dne-zfs-part-3
Test-Parameters: testgroup=review-dne-zfs-part-4
Test-Parameters: testgroup=review-zfs

Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: Iedb5c5411c721cfe738a7d3db455b222e7b4050d
Reviewed-on: https://review.whamcloud.com/36137
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

New tag 2.12.58

Change-Id: I9cd549d130a84eda7697046feabcc3a818ddaafc
Signed-off-by: Oleg Drokin <green@whamcloud.com>

LU-12368 ptlrpc: make DEBUG_REQ messages consistent

Remove linefeed from DEBUG_REQ() messages, since this results in
debug logs that are split across multiple lines and do not start
with the proper timestamp or other standard fields.  This makes
post-processing difficult.

Some error and debug messages are checked for explicitly in tests.
Add a comment by those lines in the code to alert the reader that
changes to those messages may cause test failures, and make the
tests more forgiving in case of minor changes to the formatting.

Fix several tests to check for actual error message.  Some tests
have been broken for so long (1.5/1.8) that there is no point to
also check for the old messages, so use only the new messages.

The EINPROGRESS messages should not use D_ERROR, since they can be
hit under normal usage (e.g. LFSCK running), so D_WARNING at most.
Don't print every one to the console, that would be too verbose.

Fix code style of affected lines.

Fixes: f2a7347e4ceb ("LU-1943 filter: remove obdfilter from tree")
Fixes: 473a4adc4ec2 ("LU-7787 mdd: clean up orphan object handling")
Fixes: 9f79d4488fbb ("LU-10048 ofd: take local locks within transaction")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I3cfc333ccfbf2a9bfc9e93dcfa7dbb99443ebbe5
Reviewed-on: https://review.whamcloud.com/35311
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9964 llite: prevent mulitple group locks

The patch adds mutex for group lock enqueue. It also adds waiting
of group lock users on a client side for a same node. This prevents
mulitple locks on the same resource and fixes a bugs when two locks
cover the same dirty pages.

The patch adds test sanity 244b. It creates threads which
opens file, takes group lock, writes data, puts group lock, closes.
It recreates the problem when a client has two or more group locks
for a single file. This leads to a wrong behaviour for a flush etc.
osc_cache_writeback_range()) ASSERTION( hp == 0 && discard == 0 )
failed
One more test for group lock with open file and fork. It checks that
childs doesn't unlock file until the last close.

Cray-bug-id: LUS-7232
Signed-off-by: Alexander Boyko <c17825@cray.com>
Change-Id: Iaf65cf26219211b76daaa1a91eeb80092c328f2d
Reviewed-on: https://review.whamcloud.com/35791
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9897 utils: remove dependency on libblkid

There was a weak dependency in llverdev on libblkid in order to
find the size of the underlying block device.  However, in the
absence of libblkid there is fallback code that can find the
block device size, so we don't really need libblkid at all.

Similarly, llverdev had a weak dependency on libext2fs to check
if the block device was currently in use, but all kernels since
v2.6.34-rc4-22-g6b4517a7913a allow using open(O_EXCL) to ensure
a block device is not mounted, so there is no need for llverdev
to depend on libext2fs for this.

Remove the conditional code depending on the external libraries
from llverdev and the corresponding configuration checks, which
were causing problems in various build environments.

Test-Parameters: trivial testlist=conf-sanity
Fixes: 6bc323747706 ("LU-9897 build: use pkgconf for detecting libblkid")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I90d2f0c99115ce07168c185dec6628be813ebbe5
Reviewed-on: https://review.whamcloud.com/36010
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12094 ptlrpc: protect obd_import in lprocfs_import_seq_write

The obd_import can disappear so take the cl_sem to protect it.

Change-Id: I36f2c4c606a0470781a692631265938848173f8b
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/35853
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12674 osp: handle -EINPROGRESS on llog objects

if llog object is corrupted and OI doesn't allow access to that
OSP panics being asked to declare new llog record (e.g. for unlink).
Instead OSP should complain in the logs, skip llogging and suggest
to run LFSCK to fix orphans.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I18d4d68811833c08cdc1937d147ac6e8c3408a30
Reviewed-on: https://review.whamcloud.com/35844
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-11607 tests: replace version/fstype in sanity-dom/quota

The routine get_lustre_env() is available to all Lustre test
suites and sets an environment variable for the file system
type for MDS1 and OST1 and sets variables for the Lustre
version of servers.

In sanity-dom and sanity-quota, replace the calls to
facet_fstype() and lustre_version_code() for all server
types with definitions in get_lustre_env().

While doing this, replace 'facet_fstype $SINGLEMDS' and
'lustre_version_code $SINGLEMDS' with $mds1_FSTYPE
and $MDS1_VERSION, respectively.

Clean up around any modifications by removing calls to
return after skip() or skip_env() and convert spaces
to tabs.

Test-Parameters: trivial testlist=sanity-dom,sanity-quota
Test-Parameters: fstype=zfs testlist=sanity-dom,sanity-quota
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I4dffae5ad69aa2dec5b85e285a9b314b9b35b2ba
Reviewed-on: https://review.whamcloud.com/35766
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-11607 tests: replace lustre_version calls in sanity-flr/pfl

The routine get_lustre_env() is available to all Lustre test
suites and sets an environment variable for the Lustre
version of servers for MDS1.

In sanity-flr and sanity-pfl, replace the calls to
lustre_version_code() for all server types with
definitions in get_lustre_env().

While doing this, replace ‘lustre_version_code $SINGLEMDS’
with ‘MDS1_VERSION’.

Clean up around any modifications by removing calls to
return after skip() or skip_env().

Test-Parameters: trivial testlist=sanity-flr,sanity-pfl
Test-Parameters: fstype=zfs testlist=sanity-flr,sanity-pfl
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: Ic80ea80268ba28f52d96680e304838b6e8c3a6fd
Reviewed-on: https://review.whamcloud.com/35760
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12236 tests: add tests for LNET network namespace

This patch adds tests for LNET for this feature.

Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Aurelien Degremont <degremoa@amazon.com>
Change-Id: I2320e5da1beef30be5dcca9529fa838fc9304876
Reviewed-on: https://review.whamcloud.com/35666
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <hornc@cray.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12422 lnet: discovery off route state update

When discovery is off rely on the discovery ping response
only, rather than the internal peer database to determine
route state. With discovery off the internal peer database
is not updated with all the gateway's interfaces.

Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: If1b5347629ffeb92208097c06d708e1105d27679
Reviewed-on: https://review.whamcloud.com/35199
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Olaf Weber <olaf.weber@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12367 llite: Fix page count for unaligned reads

When a read is unaligned on both the first and last page,
the calculation used to determine count of pages for
readahead misses that we access both of those pages.

Increase the calculated count by 1 in this case.

This case is covered by the generic readahead tests added
in LU-12645.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Idb28d300a232026a9d916d010ee8ebf209283242
Reviewed-on: https://review.whamcloud.com/35015
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-11681 lmv: disable remote file statahead

Remote file statahead is not supported, because such file needs
two RPCs to fetch both LOOKUP and GETATTR lock, on LOOKUP success
we only know file FID, thus can't prepare an inode correctly.

Disable this to avoid noise messages and confusion.

Update sanity.sh test_60g.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Ied99dbce320f678ffd94c461bef9d67e107e673c
Reviewed-on: https://review.whamcloud.com/33930
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-11213 uapi: Remove unused CONNECT flag

The plain layout connect flag was added as part of an
earlier implementation of LU-11213, but the design was
improved before landing and the flag was not needed.

Let's remove it.  Since it was never actually marked as
supported in any client/server version, we can just remove
it entirely, leaving the flag bit open for future use.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I30946a50299268d00cbfee2081607effd0e4fb47
Reviewed-on: https://review.whamcloud.com/36008
Reviewed-by: Shilong Wang <wshilong@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10467 llite: fix indentation

Prior to making code changes, fix up the indentation in
ll_umount_begin()

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: Iac84c5e0b97e6c07d6a591f3526e7bbc66f3726a
Reviewed-on: https://review.whamcloud.com/35967
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-6142 contrib: fix typo in spelling.txt file

Any patch that mentions cfs_time_seconds() gets a
weird warning from checkpatch due to this typo.

Test-Parameters: trivial
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: Ifeb54992c991f7f6cbcd0a9fd5aca5158adb2a4f
Reviewed-on: https://review.whamcloud.com/35965
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>

LU-12694 quota: display correct group quota information

GETQUOTA is not executed when oqctl->qc_dqblk.dqb_curspace
not 0. So, qctl.qc_dqblk need to be cleared
to display correct.

Signed-off-by: Tatsushi Takamura <takamr.tatsushi@jp.fujitsu.com>
Change-Id: I5163c00fbc58ec1dd04c912e36abd01a7d2239ad
Reviewed-on: https://review.whamcloud.com/35917
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Shilong Wang <wshilong@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-11549 mdd: set LUSTRE_ORPHAN_FL for non-dirs

mdd_mark_orphan_object() sets LUSTRE_ORPHAN_FL only for
directories, which is not correct, causing the important bit of
orphan object state not transferring across OSP link and
allowing a distributed link operation to succeed
for an orphan source object , causing a dangling reference
on one mdt and an unconnected inode on another mdt.
mdd_open_sanity_check() conditions had to be relaxed in
case of replay.

Signed-off-by: Alexander Zarochentsev <c17826@cray.com>
Change-Id: If0d868b3de4d68406e1a3b371827f354566d3e42
Reviewed-on: https://review.whamcloud.com/35776
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andrew Perepechko <c17827@cray.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9019 llite: fix timeout to not be zero

The timeout in ll_kill_super() is intended to be
1/8 of a second, while it loops waiting on other
threads to finish.  However, this was incorrectly
calculated to always be 0 (infinite wait) due to
integer division.

Instead, convert seconds to jiffies before division.

Fixes: 0c2cc920370e ("LU-9019 libcfs: avoid using HZ and msecs_to_jiffies()")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I33efbfc2d6c0a9c4ae6404c1974c7593a72540e5
Reviewed-on: https://review.whamcloud.com/35992
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-11675 hsm: don't allow new HSM requests during CDT_INIT

When the HSM CDT is shut down and restarted, it resets cdt_last_cookie
using ktime_get_real_seconds() and examines the CDT llog for existing
requests, in order to set cdt_last_cookie to the highest known value,
so that newly-assigned cookies are unique. There is a window between
CDT_INIT and CDT_RUNNING during which new requests can arrive, and if
the CDT llog has not been fully examined, cookies can be reused. This
can cause the following two assertions to be triggered in
cdt_agent_record_hash_add():

LASSERT(carl0->carl_cat_idx == carl1->carl_cat_idx);
LASSERT(carl0->carl_rec_idx == carl1->carl_rec_idx);

Fix this by not allowing new HSM requests during CDT_INIT.

Also, cookie values are incremented on a separate line, which causes
one value to be skipped at CDT startup time. This is not an issue, but
there does not seem to be a need for it; fix this post-incrementing
and assigning cookie values in the same line.

Signed-off-by: Nikitas Angelinas <nangelinas@cray.com>
Cray-bug-id: LUS-6589
Test-Parameters: trivial testlist=sanity-hsm
Change-Id: I18a1c3e85de6c50a9bf1ce598e21d83d893ad0ca
Reviewed-on: https://review.whamcloud.com/33671
Reviewed-by: Quentin Bouget <quentin.bouget@cea.fr>
Reviewed-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-11239 lfs: fix mirror resync error handling

This patch returns error for partially successful mirror resync.

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I9d6c9ef5aca1674ceb7a9cbc6b790f3f7276ff5d
Reviewed-on: https://review.whamcloud.com/33537
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12603 ldlm: Check cancel lock count for correctness

Make sure the number of locks we are going to cancel fits into
the supplied buffer first.

Change-Id: I93887133532bf7ee2be27114b1972aa64e06623c
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reported-by: Alibaba Cloud <yunye.ry@alibaba-inc.com>
Reviewed-on: https://review.whamcloud.com/35806
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Yunye Ry <yunye.ry@alibaba-inc.com>

LU-12590 ptlrpc: check lm_bufcount and lm_buflen

Check lm_bufcount to be used by lustre_msg_hdr_size_v2() and
validate individual and total buffer lengths in
lustre_unpack_msg_v2() in case of any out-of-bound read.

Change-Id: I4905e0665c7770443684cffe504935d27473d7c6
Reported-by: Alibaba Cloud <yunye.ry@alibaba-inc.com>
Signed-off-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/35783
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yunye Ry <yunye.ry@alibaba-inc.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-11546 tests: enable large_dir support for tests

Enable the ldiskfs large_dir feature by default for all test
filesystems, so that we can verify it is not causing any issues
in regular testing.  If this is successful, it can be enabled
in mkfs.lustre permanently.

Existing conf-sanity.sh test_110() and test_111() exercise this
feature specifically, this patch is to ensure other tests have
no problems with this feature enabled.  There are other problems
with test_110() so it cannot be enabled at this time.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ifff578551b57b05753fc10abb2d5294730254035
Reviewed-on: https://review.whamcloud.com/35358
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Artem Blagodarenko <artem.blagodarenko@gmail.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12614 ldlm: ldlm_cancel_hpreq_check should check lock count

Make sure the number of locks we are going to cancel fits into
the supplied buffer first.
This is similar to LU-12603, just in a different place.

Change-Id: Ifa2aa976ce8613217c739ef609de54538c57b5e9
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reported-by: Alibaba Cloud <yunye.ry@alibaba-inc.com>
Reviewed-on: https://review.whamcloud.com/35807
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Yunye Ry <yunye.ry@alibaba-inc.com>

LU-12685 llite: fix check for mem-alloc failure.

A change to allocation of op_data in ll_statahead_thread()
appears to have assumed that OBD_ALLOC_PTR() would set the
pointer to ERR_PTR(-ENOMEM) on failure.  It actually sets
it to NULL, so the test needs to be changed accordingly.

Test-Parameters: trivial
Fixes: ae828cd3b092 ("LU-4684 llite: add lock for dir layout data")
Signed-off-by: Mr NeilBrown <neilb@suse.com>
Change-Id: I37521d2da50d71ed6fa0f9f05b7cfb848f0d47d9
Reviewed-on: https://review.whamcloud.com/35873
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: Maloo <maloo@whamcloud.com>

LU-12635 lnet: Fix style issues for module.c conctl.c

This patch fixes issues reported by checkpatch for the file
selftest/module.c and selftest/conctl.c.
Linux 5.3 enforces the use of 'fallthrough' which is also
suggested by checkpatch

Test-Parameters: trivial
Cray-bug-id: LUS-7690
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Change-Id: If650375f63f27c01c40e251059fa242f919854be
Reviewed-on: https://review.whamcloud.com/35802
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12635 lnet: Fix style issues for selftest/rpc.c

This patch fixes issues reported by checkpatch for the file
selftest/rpc.c.
Linux 5.3 enforces the use of 'fallthrough' which is also
suggested by checkpatch

Test-Parameters: trivial
Cray-bug-id: LUS-7690
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Change-Id: I049e32c0b0cf1002166445a89ac39110442d28bd
Reviewed-on: https://review.whamcloud.com/35800
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12634 build: kbuild changes in 5.3 drop subdir-m

Several changes in kbuild affect the way external modules
can be built. In the Linux 5.3-rc4 series subdir-m has
been removed.

Linux commit: c07d8d47bca1b325102fa2be3a463075f7b051d9

Test-Parameters: trivial
Cray-bug-id: LUS-7689
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Change-Id: Id1f248ac4ccdee8d2a2d177b4fdff4444d2084d1
Reviewed-on: https://review.whamcloud.com/35786
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-11607 tests: replace version/fstype calls in replay-single

The routine get_lustre_env() is available to all Lustre test
suites and sets an environment variable for the file system
type for MDS1 and OST1 and sets a variable for the Lustre
version of servers.

In replay-single, replace the calls to facet_fstype() and
lustre_version_code() for all server types with definitions
in get_lustre_env().

While doing this, replace facet_fstype $SINGLEMDS and
lustre_version_code $SINGLEMDS with facet_fstype mds1
and lustre_version_code mds1, respectively.

Clean up around any modifications by removing calls to
return after skip() or skip_env().

Test-Parameters: trivial testlist=replay-single
Test-Parameters: fstype=zfs testlist=replay-single
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I14d90f934401c9f0504d42adc9f6e59b12149b0b
Reviewed-on: https://review.whamcloud.com/35724
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12635 build: Support for gcc -Wimplicit-fallthrough

Linux 5.3 enables -Wimplicit-fallthrough
Add decorators for implicit-fallthrough compiler checks.

Test-Parameters: trivial
Cray-bug-id: LUS-7690
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Change-Id: I5bccb2cfd6b5900ff7f0e21b5546eec9ffa83c19
Reviewed-on: https://review.whamcloud.com/35708
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>

LU-12523 ptlrpc: Stop sending ptlrpc_body_v2

ptlrpc_body_v2 does not include space for jobids, that
means that when we added jobid to the RPC debug messages,
we started getting errors like this:

LustreError: 6817:0:(pack_generic.c:425:lustre_msg_buf_v2()) msg
000000005c83b7a2 buffer[0] size 152 too small (required 184, opc=-1)

This happened every time we tried to print a ptlrpc_body_v2
message.

body_v2 is still sent on some RPCs for compatibility with
very old versions of Lustre, but we no longer support
interop with those versions (latest reported is 2.3).

So, stop sending ptlrpc_body_v2 on any RPCs.

Note that we need to retain the ptlrpc_body_v2 definitions
and parsing capability for interop with servers which still
use them for some messages, which is all prior to this
patch.

One further note:
This does *not* fix the case of newer clients collecting
rpctrace with older servers.  They will still see the
error message for some RPCs.  That could be fixed with
tweaks to the debug printing code.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I54fc9a174788235c43fe9101a1b42adc7f547847
Reviewed-on: https://review.whamcloud.com/35583
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9859 libcfs: double copy bug

The problem is that we copy hdr.ioc_len, we verify it, then we copy it
again without checking to see if it has changed in between the two
copies.

This could result in an information leak.

Linux-commit: 76bdaa161cd93d9c033bf6fe2b0a5661c8204441

Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Change-Id: Ic9ae8c19d90a5547600f3775ed337394717b94e3
Reviewed-on: https://review.whamcloud.com/35574
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-4423 ptlrpc: incorporate BUILD_BUG_ON into ptlrpc_req_async_args()

Every call to ptlrpc_req_async_args() should be preceded by a
BUILD_BUG_ON() (aka LASSERT()), though a few aren't.

To improve maintainability, include the BUILD_BUG_ON into the
ptlrpc_req_async_args() macro.

Change-Id: I07481921379930b4b2d9329aefb47068fd0e07f0
Signed-off-by: NeilBrown <neilb@suse.com>
Reviewed-on: https://review.whamcloud.com/35509
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Petros Koutoupis <pkoutoupis@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12075 mdt: commit migrate transaction with locks held

in normal conditions migration (being a distributed transaction)
saves LDLM locks in the reply to implement CoS semantics, but if
 migrate process has got too many LDLM locks and we can't save
them in the reply, then we should commit the transaction and
then we can release the locks.

Change-Id: I5e5f0516bdca973a72e43d63ecde79c792558abd
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/34431
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9859 mdt: replace CLASSERT with BUILD_BUG_ON

Replace the lustre specific CLASSERT in mdt layer with what the
linux kernel provides. Note the logic is reverse.

Test-Parameters: trivial

Change-Id: I1e4d75a6643204c55c888a5c2e95a8b12251f5b7
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/32219
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12616 obclass: fix MDS start/stop race

The MDS unload happen when type of MDT has no reference.
The MDT drop it during obd_cleanup. So race window located
between obd_cleanup and server_stop_servers.
Lustre can lost MDS obd_device during server_start_targets
between MDS checking and taking the type reference, if another MDT
stops.

The patch takes one more reference for a MDT type at
server_start_targets, and put it at server_stop_servers.

This patch adds sanity test 278. It reproduces the next race
   started cleanup of MDT01
   started cleanup of MDT00
   finished cleanup of MDT00
   started MDT00 mount, checked MDS exist
   finished cleanup of MDT01, and cleanup of MDS also
   asserted during MDT00 initialization

Cray-bug-id: LUS-7275
Signed-off-by: Alexander Boyko <c17825@cray.com>
Change-Id: I9ae3bc2ec1d23c8d436f143d12e26209fdb6b083
Reviewed-on: https://review.whamcloud.com/35652
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Alexander Zarochentsev <c17826@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12015 build: update changelog for ubuntu kernel

Update Ubuntu18.04 client build kernel

Test-Parameters: trivial

Change-Id: I02137fd813239d5d19a9dc0d74be49f302652e8e
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/34331
Reviewed-by: James Simmons <jsimmons@infradead.org>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Peter Jones <pjones@whamcloud.com>
Reviewed-by: Joseph Gmitter <jgmitter@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12671 mdd: rename mdd/sync_perm to sync_permissions

Commit e783bbff accidentally renamed a sysfs variable when moving.
Change the sysfs file to it proper name

Test-Parameters: trivial testlist=replay-vbr

Change-Id: I56e0534506271cf6760f775a9c8fa99b12683861
Fixes: e783bbff ("LU-8066 mdd: migrate from proc to sysfs")
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/35851
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9859 libcfs: switch to cpumask_var_t

So that we can use the common cpumask allocation functions,
switch to cpumask_var_t.
We need to be careful not to free a cpumask_var_t until the
variable has been initialized, and it cannot be initialized
directly.
So we must be sure either that it is filled with zeros, or
that zalloc_cpumask_var() has been called on it.

Linux-commit: 3872fb73cabdd47fd4abf7b6eff21d06e57297eb

Change-Id: I58d3f2e1fb1c71e1bd094a60ec4eb49b477e69e3
Signed-off-by: NeilBrown <neilb@suse.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-on: https://review.whamcloud.com/35809
Reviewed-by: Neil Brown <neilb@suse.de>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12457 kernel: RHEL 7.7 server support

This patch makes changes to support new RHEL 7.7 release
for Lustre server.

Test-Parameters: trivial clientdistro=el7.7 serverdistro=el7.7

Change-Id: Ic56e087e6c89f1bbd1ab247c44b2e979828f34f9
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/35727
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-11768 test: limit at_max to timeout in time

In test_6 of sanity-quota, if the AT is enabled, the timeout of
the QUOTA_DQACQ request could be longer than OBD_TIMEOUT*2, which
cause the watchdog to be triggered.

Change-Id: I7e3a976a004259f5c956fc48f4d8d63c751ee2c0
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/35651
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12400 osd-ldiskfs: support multi-page bvec

Mutli-page bvec support to enable bio_for_each_segment_all
needs users to supply bvec_iter_all as an iterator

Abstract the old (int) and new (bvec_iter_all) iterator types

Linux-commit: 6dc4f100c175dd0511ae8674786e7c9006cdfbfa

Test-Parameters: trivial
Cray-bug-id: LUS-7600
Signed-off-by: Shaun Tancheff <stancheff@cray.com>
Change-Id: I5a6decae9a4d470e268e20e86c29623b98e97205
Reviewed-on: https://review.whamcloud.com/35498
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>

LU-12455 osd: Correct readcache_max_filesize proc

Readcache max file size is displayed as unsigned, but
parsed as signed.  This causes confusion when users try to
set it back to the default, which is 2^64-1 (or -1), and
they can't use 2^64-1.

Easy enough to add an unsigned parser and use it instead.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I15f5677c61f4f12a8448a665c5d52a8c94d062f3
Reviewed-on: https://review.whamcloud.com/35266
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Ann Koehler <amk@cray.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-11883 nodemap: make deny_unknown visible on default nodemap

deny_unknown can be set on the 'default' nodemap, but its value
cannot be read, neither with 'lctl get_param' nor by reading the
file /proc/fs/lustre/nodemap/default/deny_unknown directly.

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Idc8db01a8d32f0ae071f92307843379f4c02571c
Reviewed-on: https://review.whamcloud.com/34090
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>

LU-12017 ldlm: DoM truncate deadlock

setxattr takes inode lock and sends reint to MDS.
truncate takes MDS_INODELOCK_DOM lock and wants
to acquire inode lock.

MDS locks are for different bits
MDS_INODELOCK_UPDATE|MDS_INODELOCK_XATTR vs
MDS_INODELOCK_DOM but they blocks each other if
some blocking lock was present earlier.

If IBITS waiting lock has no conflicts with any lock in the
granted queue or any lock ahead in the waiting queue then
it can be granted.

Use separate waiting lists for each ibit to eliminate full
lr_waiting list scan.

Cray-bug-id: LUS-6970
Change-Id: I95b2ed0b1a0063b7ece5277a5ee06e2511d44e5f
Signed-off-by: Andriy Skulysh <c17819@cray.com>
Reviewed-on: https://review.whamcloud.com/35057
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10048 ofd: take local locks within transaction

The patch (with companion patch LU-10048 osd: async truncate)
addresses long outstanding technical debt resulting in different
locking order on MDT and OST. With OUT introduction this mismatch
lead to deadlocks as OUT is used by the both sides and can't easily
support two different locking models simultanously.

The unified locking rules are:
- open transaction (dt_trans_start()),
- then take a lock (dt_{read|write}_lock()).

Change-Id: I1eaeb58ae3d61869914293a9fea2d0a1faefe76b
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/31293
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12602 mdt: check EA size in mdt_getxattr_pack_reply()

Check EA data size(non-positive or excessively large) in case of
any corruption.

Change-Id: I8ccea214f8d7c0403a9df180acf487ee381b8d77
Reported-by: Alibaba Cloud <yunye.ry@alibaba-inc.com>
Signed-off-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/35768
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12605 tgt: check client data size in target_handle_connect()

Check client data size (negtive or excessively large) in case of
memcpy corruption.

Change-Id: Ided26dea0e2bbb79e607c626810834ca947497d4
Reported-by: Alibaba Cloud <yunye.ry@alibaba-inc.com>
Signed-off-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/35711
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12675 mdt: release object reference upon error

LBUG ("(lu_object.c:1196:lu_device_fini()) ASSERTION(
atomic_read(&d->ld_ref) == 0) failed: Refcount is <x>") can
intermitently occur during umount of MDT0000, upon specific
use cases (playing with file/dir having foreign LOV/LMV), and
due to object reference set/leaked on server side.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: Ic49b2bb0402b1a6e51d7ba656f9957eeda1bd0fb
Reviewed-on: https://review.whamcloud.com/35845
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-9859 libcfs: fix cfs_print_to_console()

The original code for cfs_print_to_console() used printk() and
used tricks to select which printk level to use. Later a cleanup
patch landed the just converted printk directly to pr_info which
is not exactly correct. Instead of converting back to printk lets
move everything to pr_* type functions which simplify the code.
This allows us to fold both dbghdr_to_err_string() and the
function dbghdr_to_info_string() into cfs_print_to_console().

Linux-commit: f030d88558e77bbf07fab388c341af1cf86135c9

Fixes: 003096c7e3e ("LU-6142 libcfs: Fix style issues for linux-tracefile.c")

Change-Id: I44646b1ff41505faa05eeea7bfcb6911e893fb73
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/35847
Reviewed-by: Shaun Tancheff <stancheff@cray.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-11011 osc: add preferred checksum type support

Some checksum types might not work correctly even though they are
available options and have the best speeds during test. In these
circumstances, users might want to use a certain checksum type which
is known to be functional. However, "lctl conf_param XXX-YYY.osc.
checksum_type=ZZZ" won't help to enforce a certain checksum type,
because the selected checksum type is determined during OSC
connection, which will overwrite the LLOG parameter.

To solve this problem, whenever a valid checksum type is set by "lctl
conf_param" or "lctl set_param", it is remembered as the perferred
checksum type for the OSC. During connection process, if that
checksum type is available, that checksum type will be selected as
the RPC checksum type regardless of its speed.

The semantics of interface /proc/fs/lustre/osc/*/checksum_type is
changed for a little bit. If a wrong checksum name is being written
into this entry, -EINVAL will be returned as before. If the written
string is a valid checksum name, even though the checksum type is
not supported by this OSC/OST pair, the checksum type will still be
remembered as the perferred checksum type, and return value will be
-ENOTSUPP. Whenever connecting/reconnecting happens, if perferred
checksum type is availabe, it will be used for the RPC checksum.

Change-Id: Ie6fdc1d8ed6c55531ad6b7c926659d644fefccaf
Signed-off-by: Li Xi <lixi@ddn.com>
Reviewed-on: https://review.whamcloud.com/32349
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12313 llite: Mark lustre_inode_cache as reclaimable

This is required for proper kernel memory available accounting.
Without it memory allocated to lustre_inode_cache appears as
SUnreclaim where in reality it should apper as SReclaimable.
This affect MemAvailable as well (it is lower than it should be).

Signed-off-by: Jacek Tomaka <jacek.tomaka@poczta.fm>
Change-Id: Iac526a62d0e063b82eea451d1fafa42f2bb4d3b8
Reviewed-on: https://review.whamcloud.com/35790
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Neil Brown <neilb@suse.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12660 kernel: kernel update SLES12 SP4 [4.12.14-95.29.1]

Update SLES12 SP4 kernel to 4.12.14-95.29.1 for Lustre client.

Test-Parameters: trivial clientdistro=sles12sp4 \
envdefinitions=LNET_SELFTEST_EXCEPT=smoke,SANITY_EXCEPT="103a 817"

Change-Id: I4cc8564c4044f551a075ee0d41fd393844f4b760
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/35774
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10070 lod: SEL inheritance fix

a sub-dir cannot be created in a dir with a SEL layout template
due to problem layout verification

Fixes: 94e9699878e ("LU-10070 lod: SEL: interoperability support")

Signed-off-by: Vitaly Fertman <c17818@cray.com>
Cray-bug-id: LUS-2528
Change-Id: I2c439e44d8f183f4057f548ee38e530c578371e4
Reviewed-on: https://review.whamcloud.com/35704
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexey Lyashkov <c17817@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12611 lnet: continue adding routes

Continue adding routes specified even if the current route
exists or the gateway specified in the current route is unreachable.

Signed-off-by: Amir Shehata <ashehata@whamcloud.com>
Change-Id: I71257cd444c29d4641d9d27f05d9160871316b02
Reviewed-on: https://review.whamcloud.com/35641
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Chris Horn <hornc@cray.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexandr Boyko <c17825@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-12589 llite: swab LOV EA data in ll_getxattr_lov()

On PPC client, the LOV EA data returned by getfattr from x86_64 server
was not swabbed to the host endian. While running setfattr, the data was
swabbed in ll_lov_setstripe_ea_info(), which caused magic mis-match in
ll_lov_user_md_size() and then ll_setstripe_ea() returned -ERANGE.

This patch fixed the above issue by swabbing LOV EA data in ll_getxattr_lov().

Test-Parameters: clientarch=ppc64 \
envdefinitions=ONLY="24D 102a" testlist=sanity

Change-Id: I8069df0c8f07c0bedba2e27db7c3a5553f11afb4
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/35626
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>

LU-11873 tests: Increase barrier freeze time

Barrier freeze times of 10 seconds or less are roughly the
same length as ZFS commit intervals, and because barriers
generate sync ops, they have to wait for those.  This means
that a 10 second barrier will occassionally expire before
the commit has finished.

Switch to barriers of at least 20 seconds.

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I50fc8315c791ed444ccf39755441fdbe3aa1db6c
Reviewed-on: https://review.whamcloud.com/35361
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Li Xi <lixi@ddn.com>