git://git.whamcloud.com - fs/lustre-release.git/log

LU-11424 lnet: copy the correct amount of cpts to lnet_cpts

The incorrect size was used in the memory copy of the requested
cpts to net->lnet_cpts. This lead to the following in testing
RIP: 0010:lnet_match2mt.isra.8+0x2b/0x40 [lnet]

lnet_mt_of_attach+0x72/0x1b0 [lnet]
LNetMEAttach+0x60/0x1f0 [lnet]
ptl_send_rpc+0x26f/0xbb0 [ptlrpc]
libcfs_debug_msg+0x57/0x80 [libcfs]
ptlrpc_send_new_req+0x4c9/0x860 [ptlrpc]
ptlrpc_check_set.part.21+0x855/0x18b0 [ptlrpc]
? try_to_del_timer_sync+0x4d/0x80
? del_timer_sync+0x35/0x40
ptlrpcd_check+0x3ae/0x3f0 [ptlrpc]
ptlrpcd+0x2be/0x320 [ptlrpc]
? wait_woken+0x80/0x80

Changing the size from ncpts to ncpts * sizeof(*net->net_cpts)

Lustre-change: https://review.whamcloud.com/33229
Lustre-commit: 5afe99cac9a7be5bf776d68c65b2fe51b31591ae

Change-Id: I10832430e53ccc5b40ebce3ddfd2cf9ea330b0df
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: Sonia Sharma <sharmaso@whamcloud.com>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/33312
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-10584 tests: sanityn test 51d cancel layout lock

Modify sanityn test 51d to cancel layout locks directly
using cancel_lru_locks.

Author: Bobi Jam <bobijam@whamcloud.com>

Test-Parameters: trivial testlist=sanityn
Test-Parameters: mdsjob=lustre-master ossjob=lustre-master serverbuildno=3807 testlist=sanityn
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I603e99ad847d2726ff18c742db1f2d0b9f20e98b
Reviewed-on: https://review.whamcloud.com/33393
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Revert "LU-5152 quota: enforce block quota for chgrp"

This reverts commit 07412234ec60de20cb8d8e45d755297fe6da2d61.

This can cause deadlocks between the MDT and OST as seen in
LU-11465 and other related problems. Reverting this patch will
mean file group changes to not be updated in group quota.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I5ce48424f4d2011ce62e69047ace7f0b7c11a93b
Reviewed-on: https://review.whamcloud.com/33682
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-11563 build: Only add l_tunedisk udev rule to server

Split LU-9551 patch off into server only udev rules.
It just spits errors on the client since l_tunedisk is a server-side
only tool.

Lustre-change: https://review.whamcloud.com/33466
Lustre-commit: 0d11a314787bc795797a016262e9bcfe86e2193e

Test-Parameters: trivial
Signed-off-by: Nathaniel Clark <nclark@whamcloud.com>
Change-Id: Iee426588bcce611dc913cf89a4bcb733c364482b
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Jay J Lan <jay.j.lan@nasa.gov>
Reviewed-on: https://review.whamcloud.com/33688
Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LU-11525 kernel: new kernel [RHEL7.6 3.10.0-957.el7]

This patch makes changes to support new RHEL 7.6 release.

Test-Parameters: clientdistro=el7.6 ossdistro=el7.6 mdsdistro=el7.6

Change-Id: I1c6f1d065404fdfecf9ea978a2b161b4f90d3ab2
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/33621
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-11448 kernel: kernel update RHEL7.5 [3.10.0-862.14.4.el7]

Update RHEL7.5 kernel to 3.10.0-862.14.4.el7.

This patch is back-ported from the following one:
Lustre-commit: e734d9e803443139ccb160bbd4eae05ecc9627e5
Lustre-change: https://review.whamcloud.com/33254

Change-Id: I4901102347a14d23645547efc84857868acec0f7
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/33496
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-11215 tests: replace "large_xattr" with "ea_inode"

Change the test scripts over to using the "ea_inode" name, since
this is what the upstream e2fsprogs is using. The "large_xattr"
feature name was only ever used in the Lustre-patched e2fsprogs.

Don't try to turn off "ea_inode" feature on the targets anymore,
it's not supported by upstream e2fsprogs.

e2fsprogs commit: 5b72578279fe2470e682692a15d70a43d9289e0f

This patch is back-ported from the following one:
Lustre-commit: 900a1b6e4271a6c1903d6723082a01c98defe7b2
Lustre-change: https://review.whamcloud.com/33012

Test-Parameters: trivial testlist=conf-sanity
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Change-Id: I83bd303827fa28050d1d6d2416b2d630dc94ec12
Reviewed-on: https://review.whamcloud.com/33503
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-11412 kernel: kernel update [SLES12 SP3 4.4.155-94.50]

Update SLES12 SP3 kernel to 4.4.155-94.50.

Test-Parameters: envdefinitions=CONF_SANITY_EXCEPT=103 \
mdtfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs \
clientdistro=sles12sp3 ossdistro=sles12sp3 mdsdistro=sles12sp3 \
testlist=sanity,conf-sanity,sanity-sec

Lustre-change: https://review.whamcloud.com/33236
Lustre-commit: 937d72acf48f9d7cc2e382ac98c1d37c1f5fb1df

Change-Id: I47b72420664fe614a2da7c864c401a9729d96c55
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/33313
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-11500 kernel: kernel update RHEL6.10 [2.6.32-754.6.3.el6]

Update RHEL6.10 kernel to 2.6.32-754.6.3.el6 for Lustre client.

Test-Parameters: clientdistro=el6.10

Change-Id: I65f5c77f2fb9c086ce756ff4895fda71772128ba
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/33348
Tested-by: Jenkins
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-10579 mdd: emit changelogs for security and trusted xattrs

In mdd_xattr_changelog_type() include security.* and trusted.* among
the xattr names for which we emit a changelog record. This was done in
2.11 by https://review.whamcloud.com/28251 LU-9727 lustre: add
CL_GETXATTR for Changelogs.

Test-Parameters: trivial testlist=lustre-rsync-test
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I08cf6a293ac33dfb4b15a04000e301c516d7bd95
Reviewed-on: https://review.whamcloud.com/33048
Tested-by: Jenkins
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: Maloo <hpdd-maloo@intel.com>

New tag 2.10.6-RC1.

First release candidate for 2.10.6 release.

Change-Id: I60b19bda5c102d980638555eacd725cb3c712538
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>

LU-11160 build: Fix uuid / blkid dependency

UUID dependency stems from libblkid, so only link with uuid if blkid
is present.

Lustre-change: https://review.whamcloud.com/32842
Lustre-commit: b6c2ffa661f5aceb50fe03fbc8cfd7cdf7966ae2

Signed-off-by: Nathaniel Clark <nclark@whamcloud.com>
Change-Id: If1cc293cc48210a065f8910ea655615b11268b5c
Reviewed-by: Quentin Bouget <quentin.bouget@cea.fr>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/33147
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-11304 misc: update all url links to whamcloud

Even old links could redirect to whamcloud automatically,
We'd better update to use new whamcloud links to
avoid any further confusions.

Test-parameters: trivial

Lustre-change: https://review.whamcloud.com/33094
Lustre-commit: 605a139a193e4ef212560673b4bb93eec763d088

Change-Id: Ida7161a062d822141bf0c1fdf20b2098a21ea9e7
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Peter Jones <pjones@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/33109
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-8200 test: improve sanityn.sh 33c

If transaction is committed before unlock, lock won't be saved. So
trigger Sync-Lock-Cancel twice in sanityn.sh 33c, it's unlikely to
fail both.

This patch is back-ported from the following one:
Lustre-commit: 04c4b9c2a121e76881f8b387a4721069418c92f0
Lustre-change: https://review.whamcloud.com/31673

Test-Parameters: trivial clientcount=2 mdscount=2 mdtcount=4 \
osscount=1 ostcount=8 mdtfilesystemtype=zfs ostfilesystemtype=zfs \
testlist=sanityn

Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Change-Id: I5c99e5a8261df8c9f463aea7ed67df95baaf3e6c
Reviewed-on: https://review.whamcloud.com/33057
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-11254 kernel: kernel update RHEL6.10 [2.6.32-754.3.5.el6]

Update RHEL6.10 kernel to 2.6.32-754.3.5.el6 for Lustre client.

Test-Parameters: clientdistro=el6.10

Change-Id: Ib9534e98ddc37ff567a8b548e24d575b00162cdc
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/33014
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-4923 osd-ldiskfs: dirdata is not needed on MGS

Don't print a warning message if the "dirdata" feature is not enabled
on MGS devices. It is only needed for ldiskfs MDT devices.

Lustre-commit: 383ef1a93b67778704c009aa3c1b0e2c15a5bb76
Lustre-change: https://review.whamcloud.com/29274

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I1d8ccfc9c60eff128b480ea8efa298c1212c041a
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: https://review.whamcloud.com/32961
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-10575 build: make lustre-test dpkg depend on mpi-default-dev

Currently Ubuntu testing fails to run MPI based applications in
the lustre test suite. This is due improper dpkg setup. The
solution is to make lustre-test to depend on mp-default-dev
meta-package which will pull MPICH or OpenMPI depending on the
platform.

Test-Parameters: trivial testlist=performance-sanity clientdistro=ubuntu1604

Lustre-change: https://review.whamcloud.com/32179
Lustre-commit: 212e35bebef55339979cff9edfb3581249b2df32

Change-Id: I6b162d4d7b2485289d3e1c4dc23bcd6263fee682
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Wei Liu <wei3.liu@intel.com>
Reviewed-by: Thomas Stibor <t.stibor@gsi.de>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/32387
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-10174 osd-zfs: Fix build against ZFS 0.6.5.x

Ensure all values used in https://review.whamcloud.com/29242 are
actually used conditionally. This allows us to build against
ZFS 0.6.5.7 again.

Lustre-change: https://review.whamcloud.com/29866
Lustre-commit: 0cfb879608e5036723337ac0f8f01336ecf6b7fd

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Ide560e21f7b7fedc0c107b1943602e5af99b478c
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-on: https://review.whamcloud.com/33070
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jenkins
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-11125 ofd: decrease message level

The "destroys_in_progress already cleared" message
in ofd_create_hdl() may be result of high load on OST
server prior failover. It is not an error, so decrease
its level to D_HA from D_ERROR.

Lustre-change: https://review.whamcloud.com/32985
Lustre-commit: c25c4d60ac5fbede50a41da12f91de6e402b3c68

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: Id5142672a61244a6362be3778d0769baafc87b86
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/33100
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-11201 lfsck: check linkea entry validity

Invalid linkea data may lead to dead loop in linkea iteration, check
linkea entry validity on unpack, and if entry is not unpacked, check
entry length validity.

Test-Parameters: trivial mdscount=2 mdtcount=4 testlist=sanity-lfsck

Lustre-change: https://review.whamcloud.com/32958
Lustre-commit: a5441a717c3a97071494ff51cfb72a117d12d96e

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I8e1890ed64fab38b85149ebbfecce04caaf41e17
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/33078
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-9273 tests: disable random I/O in replay-ost-single/5

disable random I/O in replay-ost-single/5 as it's very slow
on ZFS - this is due to grants as the client consume them
way too quickly: 1MB blocksize + ~0.5MB metadata overhead
for each random 4K written by iozone.

This is a backport to b2_10 of master patch
Lustre-change: https://review.whamcloud.com/#/c/31671
Lustre-commit: e3bc6e681666aa2c60ada5f997966efa31fae68c

Test-Parameters: trivial ostcount=7 clients=2 ostfilesystemtype=zfs mdtfilesystemtype=zfs testlist=replay-ost-single envdefinitions=SLOW=yes,ONLY=5
Test-Parameters: trivial ostcount=7 clients=2 ostfilesystemtype=zfs mdtfilesystemtype=zfs testlist=replay-ost-single envdefinitions=SLOW=yes,ONLY=5
Test-Parameters: trivial ostcount=7 clients=2 ostfilesystemtype=zfs mdtfilesystemtype=zfs testlist=replay-ost-single envdefinitions=SLOW=yes,ONLY=5
Test-Parameters: trivial ostcount=7 clients=2 ostfilesystemtype=zfs mdtfilesystemtype=zfs testlist=replay-ost-single envdefinitions=SLOW=yes,ONLY=5
Test-Parameters: trivial ostcount=7 clients=2 ostfilesystemtype=zfs mdtfilesystemtype=zfs testlist=replay-ost-single envdefinitions=SLOW=yes,ONLY=5

Change-Id: Ic49429b8c681fdc16e5f95f483d78198b6f4804c
Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-on: https://review.whamcloud.com/31671
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
(cherry picked from commit e3bc6e681666aa2c60ada5f997966efa31fae68c)
Reviewed-on: https://review.whamcloud.com/33053
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-7372 tests: stop running replay-dual test 26

replay-dual test 26 fails frequently. We need to add
this test to the ALWAYS_EXCEPT list and, thus, stop
running the test until we fix the issue.

This is a backport to b2_10 of master patch
Lustre-change: https://review.whamcloud.com/#/c/32902
Lustre-commit: 77fc116fb230f330761767e240882c2141b30c1a

Test-Parameters: trivial testlist=replay-dual
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I1432d544068bfbd98c912c8b7699e04c6d067492
Reviewed-on: https://review.whamcloud.com/33052
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Wei Liu <sarah@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-1895 tests: don't fail mmp test_5 due to race

In the mmp.sh test_5() mount_after_unmount() testing, it is possible
that the first filesystem unmounts successfully before the second
one starts, and there is no contention for the MMP block.

This caused the test to fail on a regular basis. However, there is
still value in running this test, since non-MMP race conditions have
previously been seen in this area (OBD device refcount, etc).

Make mount_after_unmount() more robust, only failing if the first
filesystem is still mounted at the same time as the second one.

This is a backport to b2_10 of master patch
Lustre-change: https://review.whamcloud.com/#/c/32355/
Lustre-commit: a8148c56ad4f97142679559344a7da17f709a5b8

Author: Andreas Dilger <adilger@whamcloud.com>

Test-Parameters: trivial mdtfilesystemtype=ldiskfs failover=true ostfilesystemtype=ldiskfs osscount=2 mdscount=2 mdtcount=1 austeroptions=-R iscsi=1 testlist=mmp
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I186b9ce0a5a0e1ed6f2b46895fec4a32e73ebbe5
Reviewed-on: https://review.whamcloud.com/33007
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-11227 lod: lod_sync: don't attempt sync to inactive targets

chgrp on a client triggers lod_sync() which in turn loops over OST/MDT
targets with dt_sync(). dt_sync() fails with -ENOTCONN when targets
have been deactivated (ie. set to active=0). The client retries
infinitely causing the client process to hang and considerably MDS
network traffic, load, and disk i/o.

the fix is to not attempt dt_sync() to ost/mdt targets that have been
deactivated and also (because of possible races) to ignore connection
errors.

tested with Lustre 2.10.4.

Signed-off-by: Robin Humble <plaguedbypenguins@gmail.com>
Change-Id: I617509cf7944541489f4fd9762c233b771132165
Reviewed-on: https://review.whamcloud.com/32991
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

New release 2.10.5.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I9e8e692189153c7417f61646e25d1ae8d7a0d78a

LU-11266 build: update changelog for Ubuntu

Record the version that we are building

Test-Parameters: trivial

Change-Id: Ic4ef5e906bffcdebc82a3db44beff0bfd95b4b77
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/33021
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

New tag 2.10.5-RC2.

Second release candidate for 2.10.5 release.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I32a1fadf6a48d4ef871ecc3e408e4759c458186e

LU-11062 libcfs: use save_stack_trace for stack dump

The stacktrace_ops has been removed recently. So we
have to use save_stack_trace_tsk for stack trace
dump.

Lustre-change: https://review.whamcloud.com/#/c/32952/
Lustre-commit: 574e63fc86553510d87d02cd6d72785f341e48dc

Test-Parameters: trivial
Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: Icb3d0dbd62c35fdd9b8de925aec9358a2208814f
Reviewed-on: https://review.whamcloud.com/32972
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-11176 systemd: use univeral path for modprobe

The program modprobe is not the same on all platforms. On RHEL
systems it is located in /usr/sbin. For Ubuntu/Debian which is
busybox based /sbin/modprobe is a symlink to /bin/kmod. On all
platforms to keep some sort of standard a symlink for modprobe
exist in /sbin. Update the lnet.service script to use the hard
patch /sbin/modprobe

Test-Parameters: trivial

Lustre-change: https://review.whamcloud.com/32520
Lustre-commit: 574e63fc86553510d87d02cd6d72785f341e48dc

Change-Id: I54342971a6ee1aa4ce86a9fae0ac4dcb167b1510
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/32944
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/32970
Reviewed-by: Thomas Stibor <t.stibor@gsi.de>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-11176 build: properly package the shared libraries for debian/Ubuntu

The patch for LU-10569 when merged to the lustre LTS branch lost some
parts that needed to included for lustre to be properly supported.
This patch makes lustre properly package the shared libraries.

Test-Parameters: trivial

Change-Id: I63b77e2ef6816e36b1c340dec88db73bc15669df
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/32943
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-7763 tests: improve sanityn 33d

Sanityn.sh 33d verifies that CoS (Commit-on-Sharing) for DNE, but if
transaction commits before unlock, CoS will not be triggered. To
avoid this, do all tests twice, it's unlikely that both will fail.

This patch is back-ported from the following one:
Lustre-commit: 9d49453b675fba97002a2553769be89800783db3
Lustre-change: https://review.whamcloud.com/31946

Test-Parameters: trivial testlist=sanityn mdscount=2 mdtcount=4

Change-Id: If20670321e4bda9763b3794145d62a7264e7cef6
Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-on: https://review.whamcloud.com/32937
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-11168 kernel: kernel update RHEL6.10 [2.6.32-754.2.1.el6]

Update RHEL6.10 kernel to 2.6.32-754.2.1.el6 for Lustre client.

Test-Parameters: clientdistro=el6.10

Change-Id: I7379565713adfd5ae2c8ee604c7838d6e6aba086
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/32936
Tested-by: Jenkins
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-11079 llite: control concurrent statahead instances

It is found that if there are too many concurrent statahead
instances, then related statahead RPCs may accumulate on the
client import (for MDT) RPC lists
(imp_sending_list/imp_delayed_list/imp_unreplied_lis), as to
seriously affect the efficiency of spin_lock under the case
of MDT overloaded or in recovery. Be as the temporarily solution,
restrict the concurrent statahead instances.

If want to support more concurrent statahead instances, please
consider to decentralize the RPC lists attached on related import.

Lustre-change: https://review.whamcloud.com/32690
Lustre-commit: 164231a7a29b7203cd27ab624eddd5fb7c557b90

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I7251cc536f11d184f768e3d3704ba6717644541e
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/32909
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-10898 tests: enable conf-sanity 32a/32d

Stop zed service because it hold zfs module open.
Export zpools before trying to rmmod zfs.

Test-Parameters: trivial testlist=conf-sanity
Test-Parameters: trivial testlist=conf-sanity mdtfilesystemtype=zfs ostfilesystemtype=zfs

Lustre-change: https://review.whamcloud.com/31991
Lustre-commit: e8e6e3210e5eae78eefcc3f05e078a60c04dc80e

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Id1b891cadb91d9e3631a2d067c9d76a2965c34ff
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/32520
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: James Nunez <jnunez@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

New tag 2.10.5-RC1.

First release candidate for 2.10.5 release.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Ic94d4e47f37fab7400725c7651ebdb3fc94d67a7

LU-11069 llite: correct file position after appending writes

In ll_file_io_generic() use the position returned in the kiocb to set
the returned file position. This ensures that the file position is set
correctly after an appending write. Add sanity test_23d() to check
that calling lseek() for the current offset returns the correct value
in this situation.

Lustre-change: https://review.whamcloud.com/32641

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ic76ce49db6e87d5294e18546d5b75a12793aa99c
Reviewed-on: https://review.whamcloud.com/32661
Reviewed-by: Jinshan Xiong <jinshan.xiong@gmail.com>
Tested-by: Jenkins
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-11149 build: enable KMP for Mellanox build

* We need to build Mellanox KMP to avoid error
in symbol dependency when installing lustre
* Remove all Mellanox config parameters and use
default
* We won't build with rhel6

Test-Parameters: trivial

Change-Id: I4676d01bd5f788581e1be6df98d2d787a5419c07
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/32833
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/32913
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-11107 mdt: handle nonexistent xattrs correctly

In mdt_getxattr_pack_reply() propagate -ENODATA returns from
mo_xattr_list() to mdt_getxattr(). Add sanity test_102s() to ensure
that getting a nonexistint xattr will fail.

Lustre-change: https://review.whamcloud.com/32753
Lustre-commit: 39c7ac4c0dce3c62795814cd12a78bec1877520b

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Ic7a01feb3fcac66d39f84b4ebdfc86025c3e2779
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/32910
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-11074 mdc: set correct body eadatasize for getxattr()

In mdc_intent_getxattr_pack() set mbo_eadatasize to the size of the
xattr values buffer rather than the size of the xattr names buffer.

Lustre-change: https://review.whamcloud.com/32739
Lustre-commit: dea1cde92014545d97406bf8adba20840abdb1a9

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Ibbed6aba6718f50eed1a08d506d526b1e0e042c8
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Andrew Perepechko <c17827@cray.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/32901
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-11167 kernel: new kernel [RHEL6.10 2.6.32-754.el6]

This patch makes changes to support new RHEL 6.10 release
for Lustre client.

Test-Parameters: clientdistro=el6.10

Change-Id: I4cec46b0189f1fc6779e15b6dca76bcf9d607bbc
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/32873
Tested-by: Jenkins
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-10992 llite: decrease sa_running if fail to start statahead

Otherwise the counter of ll_sb_info::ll_sa_running will leak as
to the umount process will be blocked for ever.

Lustre-change: https://review.whamcloud.com/32287
Lustre-commit:◁6b8638bf792086d9e4b249e95091095bba4ada02

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I6b3a0dfb39f120a0c58184a22211b52376b41bd9
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/32526
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-10165 llite: disable statahead if starting statahead fail

Once starting statahead thread fails, it should disable statahead.
Current code only does this when "sai != NULL", instead it should
check whether current process is opening this dir, so for cases
like current file is not the first dirent, or sai allocation fail,
it won't retry statahead.

Lustre-change: https://review.whamcloud.com/29817
Lustre-commit: 8b1bd1b88ae915c89fcbdc83abebe217a681a9e7

Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Change-Id: Iaedddd3659cdffeab51800f45b02f0b39c4a1ec1
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/32366
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-10683 osd_zfs: set offset in page correctly

In osd_bufs_get_write, the offset in the first page should
Lustre-change the offset parameter instead of zero.

Lustre-change: https://review.whamcloud.com/32788
Lustre-commit: 83cb17031913ba2f33a5b67219a03c5605f48f27

Change-Id: I6592d8b5b0162b92953d59e2662a4381ba3e89ba
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/32899
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-11115 lod: skip max_create_count=0 OST in QoS and RR algorithms

While choosing OST to create object, both lod_alloc_qos() and
lod_alloc_rr() functions use lod_statfs_and_check() function
to check whether the OST is available for new OST objects or not.
However, OST with max_create_count=0 is not checked in that
function and just returned as an available OST.

This patch fixes the above issue by detecting OST with
max_create_count=0 in lod_statfs_and_check() and skip it.

Change-Id: Ic2129b483ead16f8971a062e326d5ecd537b7e6c
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/32859
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-11159 kernel: kernel update RHEL7.5 [3.10.0-862.9.1.el7]

Update RHEL7.5 kernel to 3.10.0-862.9.1.el7.

Change-Id: I7fb8ac89a4e153f47c3339f4047c356e57bd4f99
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/32846
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-11129 kernel: kernel update RHEL7.5 [3.10.0-862.6.3.el7]

Update RHEL7.5 kernel to 3.10.0-862.6.3.el7

Test-Parameters: clientdistro=el7.5 mdsdistro=el7.5 ossdistro=el7.5 mdtfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs
Signed-off-by: Yang Sheng <ys@whamcloud.com>
Change-Id: I59b362135b5c235ac76848afb2d48014b7a4e928
Reviewed-on: https://review.whamcloud.com/32795
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-11066 systemd: Add IB dependencies to lnet.service

Add ordering for inkernel (rdma.server) and Mellanox MOFED
(openibd.service).

This ensures that systemd will shutdown lnet prior to IB, thus
preventing it from hanging.

Lustre-change: https://review.whamcloud.com/32646
Lustre-commit: 45bb717b9d80bbe95559b45b6a8a6a49ec28e071

Test-Parameters: trivial
Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Ia0be1ca60eb8f54edd2f4f6bfbca10cbc01cc638
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-on: https://review.whamcloud.com/32793
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-9551 utils: add l_tunedisk to fix disk tunings

This adds l_tunedisk utility to utilize osd_tune_lustre call for
mount_utils.h. This can be called from udev.
This adds a udev rule to fix disk tunings.
This in some ways duplicates LU-9132, which sets this value at mount
time, but if a multipath component is removed then re-added, the
multipath's max_sectors_kb will not propgate to the newly added device
and this now will cause an error for I/Os that would violate this.

Lustre-change: https://review.whamcloud.com/31464
Lustre-commit: 892280742a2b6347df1464379b3ed223b2961ed4

Test-Parameters: trivial
Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I8fc7b296c0add29e98b625ac6395a0dee2e376df
Reviewed-on: https://review.whamcloud.com/32583
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-10989 utils: correct lustre_rsync changelog clear logic

In the non-extended rename case of lr_replicate() copy the record
number from ext to info. Then remove the spurious rename record
handling from lr_clear_cl().

Lustre-change: https://review.whamcloud.com/32247
Lustre-commit: 62ac9c6572e5d42183a9bf5fbbd6a8ead62408cd

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I615ec2f384f5f9d7807156acb3ce66ac47ca1e77
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/32525
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-10843 mgs: allow snapshot after MGS remount

If the MGS is unmounted/mounted without the MDS reconnecting,
the fsdb FSNAME-barrier would not be created.

This change allows mgs_barrier_freeze (called from snapshot_create)
to attempt the create the required fsdb.

This adds a test to sanity-lsnapshot.sh for this issue.

Test-Parameters: testlist=sanity-lsnapshot mgtfilesystemtype=zfs ostfilesystemtype=zfs mdtfilesystemtype=zfs combinedmdsmgs=false ostcount=2 standalonemgs=true

Lustre-change: https://review.whamcloud.com/31779
Lustre-commit: 2d63d5bc74a5b2477d795bf9e9b0eba94f7fc19f

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I2432cc0bdaddb07f024744065ca2ced77288fd7b
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/32519
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-10214 lnet: allow expressions for route config

Support the ip2nets syntax for route gateway configuration. Only support
a maximum of 128 gateways per configuration command. This upper limit is
to prevent a large number of routes to be configured by mistake. This
feature is available from both command line and YAML interfaces.

Command line examples:
lnetctl route add --net tcp1 --gateway 12.3.4.[2-6]@tcp
lnetctl route del --net tcp1 --gateway 12.3.4.[2-6]@tcp

YAML examples:
route:
- net: tcp1
gateway: 12.3.4.[2-20]@tcp

Lustre-change: https://review.whamcloud.com/30511
Lustre-commit: 235e8dea46f6f9ec0021c789cb5b9214cee94e38

Test-Parameters: trivial
Signed-off-by: Amir Shehata <amir.shehata@intel.com>
Change-Id: I0d8a0f94cce7140602a64f13f0401ef209f3ca57
Reviewed-by: Sonia Sharma <sonia.sharma@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/32248
Reviewed-by: Sonia Sharma <sharmaso@whamcloud.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-10825 lnet: add ip2nets syntax handling for peer

Allow peers to be added using ip2nets syntax, from
both command line and YAML block.

Command line example:
lnetctl peer add --ip2nets 10.10.10.[3-6,9]@tcp
lnetctl peer del --ip2nets 10.10.10.[3-6,9]@tcp

YAML example:
peer:
    ip2nets:
      - nid: 30.10.10.[3-8]@tcp
      - nid: 40.10.10.[9-40]@tcp

Lustre-change: https://review.whamcloud.com/31786
Lustre-commit: 70c95457f6836a9c0a9e95ae0c4bdd20f99a8747

Test-Parameters: trivial
Signed-off-by: Amir Shehata <amir.shehata@intel.com>
Change-Id: I904be8d496ad2be277c3d21dc7f72cbc7ed02b50
Reviewed-by: Sonia Sharma <sonia.sharma@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/32250
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Sonia Sharma <sharmaso@whamcloud.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-10825 libcfs: generate ip addresses

Add infrastructure API cfs_ip_addr_range_gen() to generate a
maximum of 'count' IP addresses from an expression.

Lustre-change: https://review.whamcloud.com/31785
Lustre-commit: 4c5f788397213aa41356df1f96f7ade58653973a

Test-Parameters: trivial
Signed-off-by: Amir Shehata <amir.shehata@intel.com>
Change-Id: I479dd7128eef404106a7863124a38c501150ba9e
Reviewed-by: Sonia Sharma <sonia.sharma@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/32249
Reviewed-by: Sonia Sharma <sharmaso@whamcloud.com>
Reviewed-by: Amir Shehata <ashehata@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-10419 lfsck: no delay for notify RPC

It is impossible that current MDT has trouble on the connection
with some other MDT(s) or OST(s). Under such case, the LFSCK on
current MDT should skip related MDT(s) or OST(s) to avoid whole
LFSCK process being blocked by the trouble connection or remote
targets via setting the LFSCK notify RPC as rq_no_delay.

This is back ported from master:
lustre-change: http://review.whamcloud.com/30768
lustre-commit: d8827a8ce44db121f80223dc7189e32f5bf3fd45

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Ib35080cedcbe49f4ae8c4b3690a4743d5afe41b1
Reviewed-on: https://review.whamcloud.com/30831
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-10988 lfsck: load object attr when prepare LFSCK request

It will avoid empty attrs in LFSCK request (lfsck_namespace_req).
The patch also shows invalid mode for dt_mode_to_dft() for debug.

Other cleanup for lfsck_namespace_striped_dir_rescan().

Lustre-change: https://review.whamcloud.com/32245
Lustre-commit: f7c354096a810df0a9333dace2f538d6dfbe486f

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I2874160d9a0c9e3084d0d3d7f365940108c82018
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/32522
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-8293 lnet: Add insserv header to lnet init script

The lnet init script does not contain header information as described
by the insserv man page. This header information is needed to ensure
the lnet init script is not run until openibd has been able to start

Lustre-change: https://review.whamcloud.com/20835
Lustre-commit: c99b645470427c74b0ba6ea9a031d747b92030b5

Signed-off-by: Chris Horn <hornc@cray.com>
Change-Id: I6f778e827f88ce34199dff70be5d5089f0ba51b9
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/32518
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-11065 kernel: kernel update [SLES12 SP3 4.4.132-94.33]

Update target, kernel_config, and ldiskfs files for new version
One ldiskfs patch revised for ext4 changes.
Old unchanged ldiskfs patch kept to use for sles12sp2.

Test-Parameters: clientdistro=sles12sp3 testgroup=review-ldiskfs \
mdsdistro=sles12sp3 ossdistro=sles12sp3 \
mdtfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: Ic6d0219a7133825d1dba0b2bfadf8354442cddb3
Reviewed-on: https://review.whamcloud.com/32600
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Tested-by: Jenkins
Reviewed-by: Minh Diep <minh.diep@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-11043 kernel: kernel update RHEL7.5 [3.10.0-862.3.2.el7]

update RHEL 7.5 kernel to 3.10.0-862.3.2.el7

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I0defa14e83ce098c48b3228b4867afa73a2d9185
Reviewed-on: https://review.whamcloud.com/32542
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Cliff White <cliff.white@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-11042 kernel: kernel update RHEL6.9 [2.6.32-696.30.1.el6]

Update RHEL6.9 kernel to 2.6.32-696.30.1.el6

Test-Parameters: clientdistro=el6.9 mdsdistro=el6.9 \
ossdistro=el6.9 mdtfilesystemtype=ldiskfs \
ostfilesystemtype=ldiskfs testgroup=review-ldiskfs

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I7b02f48f05c18e0b9e29f302150e8386a2b15064
Reviewed-on: https://review.whamcloud.com/32512
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Cliff White <cliff.white@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-10869 build: package configuration files for Ubuntu / Debian

For a long time Lustre never added /etc configuration files to its
debian packages. It could get away with but now you can see it
fail conf-sanity test 76a. This test the lctl set_param -P which
uses udev events to set the tunables for lustre. In order for it
to work a default udev rule has to be added to 99-lustre.rules.
Beside the missing 99-lustre.rules and in all the other files used
for configuration. Lastly create conffile which is the way Debian
handles potential stomping of configuration files. When installing
with apt-get install it will ask the person installing if they
want to over ride specific files.

Test-Parameters: clientdistro=ubuntu1604 trivial testlist=conf-sanity
Test-Parameters: envdefinitions=ONLY=76a testlist=conf-sanity

Lustre-change: https://review.whamcloud.com/31850
Lustre-commit: 8f221cf65b644d798493da489674abd2e2b7f23f

Change-Id: Ic0aaf2bba531ce23a3e23ef070a1501032ad1c9f
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Thomas Stibor <t.stibor@gsi.de>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-on: https://review.whamcloud.com/32471
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-11009 test: add version check to test_102

Skip test_102 if server is equal or less than 2.9.53

Signed-off-by: Wei Liu <wei3.liu@intel.com>
Change-Id: Ib9702ee9b2d174742f229f0c3493524606eacca7
Reviewed-on: https://review.whamcloud.com/32393
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-9796 ldiskfs: improve inode allocation performance

Backport following upstream patches:

------
ext4: cleanup goto next group

avoid duplicated codes, also we need goto
next group in case we found reserved inode.
------

ext4: reduce lock contention in __ext4_new_inode

While running number of creating file threads concurrently,
we found heavy lock contention on group spinlock:

FUNC                           TOTAL_TIME(us)       COUNT        AVG(us)
ext4_create                    1707443399           1440000      1185.72
_raw_spin_lock                 1317641501           180899929    7.28
jbd2__journal_start            287821030            1453950      197.96
jbd2_journal_get_write_access  33441470             73077185     0.46
ext4_add_nondir                29435963             1440000      20.44
ext4_add_entry                 26015166             1440049      18.07
ext4_dx_add_entry              25729337             1432814      17.96
ext4_mark_inode_dirty          12302433             5774407      2.13

most of cpu time blames to _raw_spin_lock, here is some testing
numbers with/without patch.

Test environment:
Server : SuperMicro Sever (2 x E5-2690 v3@2.60GHz, 128GB 2133MHz
         DDR4 Memory, 8GbFC)
Storage : 2 x RAID1 (DDN SFA7700X, 4 x Toshiba PX02SMU020 200GB
          Read Intensive SSD)

format command:
        mkfs.ext4 -J size=4096

test command:
        mpirun -np 48 mdtest -n 30000 -d /ext4/mdtest.out -F -C \
                -r -i 1 -v -p 10 -u #first run to load inode

        mpirun -np 48 mdtest -n 30000 -d /ext4/mdtest.out -F -C \
                -r -i 3 -v -p 10 -u

Kernel version: 4.13.0-rc3

Test  1,440,000 files with 48 directories by 48 processes:

Without patch:

File Creation   File removal
79,033          289,569 ops/per second
81,463          285,359
79,875          288,475

With patch:
File Creation   File removal
810669          301694
812805          302711
813965          297670

Creation performance is improved more than 10X with large
journal size. The main problem here is we test bitmap
and do some check and journal operations which could be
slept, then we test and set with lock hold, this could
be racy, and make 'inode' steal by other process.

However, after first try, we could confirm handle has
been started and inode bitmap journaled too, then
we could find and set bit with lock hold directly, this
will mostly gurateee success with second try.

Lustre-commit: 3f0a7241c434d9556308299eea069628715816c2
Lustre-change: https://review.whamcloud.com/29032

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I234ff3027c8d96155d374c56b12aab7c4dc0dafd
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Gu Zheng <gzheng@ddn.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: https://review.whamcloud.com/32295
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

New release 2.10.4.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I55a68a7fab1c633055854b24e297b399c49cb41e

New tag 2.10.4-RC2.

Second release candidate for 2.10.4 release.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I8759e4ca828947e4d22c7aa5abbcddba64b5a186

LU-11024 osd-zfs: properly detect ZFS dnode accounting

Properly detect if native ZFS dnode accounting is available for
ZFS 0.7.x releases that do not contain ZFS project quota. The
function signature changed after ZFS Project Quota was landed,
but we still need to check for the old function for 0.7.x.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Icd40e0bdaa0b7738e9aa761836167780843ebbe5
Reviewed-on: https://review.whamcloud.com/32422
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-11008 kernel: kernel update RHEL6.9 [2.6.32-696.28.1.el6]

Update RHEL6.9 kernel to 2.6.32-696.28.1.el6

Test-Parameters: clientdistro=el6.9 mdsdistro=el6.9 \
ossdistro=el6.9 mdtfilesystemtype=ldiskfs \
ostfilesystemtype=ldiskfs testgroup=review-ldiskfs

Change-Id: I3f42c0e9236cbe220486a858c8b5c22918756d78
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-on: https://review.whamcloud.com/32352
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-11026 lustre-dkms should require patch or quilt

Add patch requirement to lustre-dkms.spec.in
as it (or quilt) are needed for lustre-build-ldiskfs.

- Add requires patch.

Lustre-change: https://review.whamcloud.com/#/c/32431/
Lustre-commit: ea52ed90e1ea079b269351a98d86ad6b06273df1

Change-Id: I640bae382511502c02a0237694c93c304047f339
Signed-off-by: Joe Grund <joe.grund@intel.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/32450
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

New tag 2.10.4-RC1.

First release candidate for 2.10.4 release.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ia04fdb3d4d16999bd326d446a85312080b990b5e

LU-10569 build: properly package lustre for Debian/Ubuntu

Remove the obsolete linux-patch since patched kernels for lustre
clients have been long gone. Place only the static libraries and
*.so symlinks for the dynamic libraries in lustre-dev. The normal
dynamic libraries are placed into the utilities packages. Add in
all the missing dependencies and fix how the lustre debs are
dependent on each other. Lastly add in the missing lustre-iokit
that is present for rpm packages. Only thing missing is a package
for lustre resources which can be done at a latter time.

Test-Parameters: trivial

Lustre-change: https://review.whamcloud.com/31348
Lustre-commit: 020277f596168ed80d77ab50ef6a85a6e6e54d0a

Change-Id: I5fd2a23bc1ae73434cef8dcf3679b50878256ab3
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Thomas Stibor <t.stibor@gsi.de>
Tested-by: James Nunez <james.a.nunez@intel.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/32277
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-10897 kernel: kernel upgrade RHEL7.5 [3.10.0-862.2.3.el7]

With this mod we switch our supported el7 version to RHEL 7.5

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: Iedcea9498591d15eab69187274e4c32c57879e4e
Reviewed-on: https://review.whamcloud.com/32371
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-11019 build: Update ZFS/SPL to 0.7.9

This updates the ZFS version to 0.7.9.

https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.7.9

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I9452a589d9dc719de7a63d3ed287dec8b6f7c0b6
Reviewed-on: https://review.whamcloud.com/32389
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-10944 kernel: kernel update [SLES12 SP3 4.4.126-94.22]

Update target, kernel_config, and ldiskfs files for new version
ldiskfs for sles12sp2 and sles12sp3 are no longer identical,
so revise some build files for sles12sp2 here too.

Test-Parameters: clientdistro=sles12sp3 testgroup=review-ldiskfs \
mdsdistro=sles12sp3 ossdistro=sles12sp3 \
mdtfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: If554b24644f763462002d81309f8edfa016210e8
Reviewed-on: https://review.whamcloud.com/32146
Tested-by: Jenkins
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-10912 mdc: use large xattr buffers for old servers

Pre 2.10.1 MDTs will crash when they receive a listxattr (MDS_GETXATTR
with OBD_MD_FLXATTRLS) RPC for an orphan or dead object. So for
clients connected to these older MDTs, try to avoid sending listxattr
RPCs by making the bulk getxattr (MDS_GETXATTR with OBD_MD_FLXATTRALL)
more likely to succeed and thereby reducing the chances of falling
back to listxattr.

Lustre-change: https://review.whamcloud.com/31990
Lustre-commit: a95b85efdc97136d0fbf6959acf62919319901ed

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ia96323c47c91a44495b73be2d95705298c7f7ac9
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/32251
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-5490 tests: Sanity/133d ensure stats read is on correct MDT

Ensure directories used to collect rename_stats are on the MDT
that is checked. This ensures directories are created on
MDT0 and not striped and then rename_stats is read from MDT0.

Lustre-change: https://review.whamcloud.com/31585
Lustre-commit: 1d80fd72bf42f653a5d6a4a31fc2c5df571f1afc

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Ib27f5c531f2d8bd664ec3a4732c512b0c389dc43
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/32149
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-10030 utils: add lfs tool to change/list project of file

Currently, we are using chattr/lsattr for project quota
interface, this have some problems:

1)Client side need patched e2fsprogs or latest upstream e2fsprogs.
2)Project quota will be no longer osd-ldiskfs based, ZFS
too, zfs guys might dislike ldiskfs tool dependency for them.
3)customers argue chattr might be a little dangerous.

So this patch add native lfs tools for project quota.
usage: project [-p id] [-s] [-r] <file|directory..>
          set project ID and/or inherit flag for specified
          file(s) or directory.
       project [-d|-r [-0]] <file|directory...>
          list project ID and flags on file(s) or directory,
          print outliers
       project -c [-d|-r [-p id] [-0]] <file|directory..>
          check project ID and flags on file(s) or directory,
          print outliers
       project -C [-r] [-k] <file|directory..>
          clear the project inherit flag and ID on the file
          or directory

Test-Parameters: testlist=sanity-quota,sanity-quota,sanity-quota,\
    sanity-quota clientdistro=el7 serverdistro=el7 \
    ostfilesystemtype=ldiskfs mdtfilesystemtype=ldiskfs

Lustre-change: https://review.whamcloud.com/29190
Lustre-commit: 022c5a9a77d4612dce28b76a7691b7af1cefd058

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I45960fb8fbd12e22a654792fba517896c0447447
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/32098
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-9950 build: add support for Ubuntu(debian) arm64

Add arm64 into the support arch list of debian control file.

Lustre-change: https://review.whamcloud.com/28870
Lustre-commit: ba4b4eaacc37bc29b6d99f24b0f6451c758f9c4a

Change-Id: I9c39a4d8c1896c1255432380bd956330c2edf476
Signed-off-by: Gu Zheng <gzheng@ddn.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Thomas Stibor <t.stibor@gsi.de>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/32101
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-10030 llite: extend inode flags into GET/SETXATTR ioctl

This is very useful for projet quota tools to save RPC call
to change project inherit attribute together with project ID.

Lustre-change: https://review.whamcloud.com/30072
Lustre-commit: 311e8d539b1932593462de428c46ea176b8905db

Change-Id: I5b545bf2c5c601b552a85322c80994dabbca0b12
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/32099
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-7001 osp: fix llog processing

The osp_sync_thread base on fact that llog_cat_process
will not end until umount. This is worng when processing reaches
bottom of catalog, or if catalog is wrapped.
The patch fixes this issue.

For wrapped catalog llog_process_thread could process old
record.
1 thread llog_process_thread read chunk and proccing first record
2 thread add rec to this catalog at this chunk and
update bitmap
1 check bitmap for next idx and process old record

Test conf-sanity 106 was added.

Lustre-change: https://review.whamcloud.com/26132
Lustre-commit: 8da9fb0cf14cc79bf1985d144d0a201e136dfe51

Signed-off-by: Alexander Boyko <alexander.boyko@seagate.com>
Seagate-bug-id: MRP-4235
Change-Id: Ifc983018e3a325622ef3215bec4b69f5c9ac2ba2
Reviewed-by: Andriy Skulysh
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/32097
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-10887 lfsck: offer shard's mode when re-create it

The namespace will re-create the lost shard of a broken stripe
directory if "-C" option specified. Under such case, the @mode
parameter should be given properly.

Lustre-change: https://review.whamcloud.com/31915
Lustre-commit: 7d48050b7a3ba0b9db2ff823bc6fbc3091506597

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I2441d8ca83c932a34ef1971d334f54ecd7343b27
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/32076
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-10368 mdc: resend quotactl if needed

In mdc_quotactl, it is better to resend the quotactl request
if reconnection or failover is triggered during the process.

Lustre-change: https://review.whamcloud.com/31773
Lustre-commit: d511918e8eb725abba2561cc493e30651a89ac27

Change-Id: I64f96863a6f10026aa69cba3c59095966b58b98d
Signed-off-by: Hongchao Zhang <hongchao.zhang@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/32075
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-10901 build: Update ZFS/SPL to 0.7.8

This skips 0.7.7 due to regression fixed in 0.7.8

https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.7.8
https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.7.7

Lustre-change: https://review.whamcloud.com/31949
Lustre-commit: 33fa3ca05b5bb6abbcef01444c3f383d876ecba9

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Icd6a21793a5f7502c88121def8d86d9aa48c6ae8
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-on: https://review.whamcloud.com/32070
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-10864 build: update changelog for ubuntu 16.04

update chanage to the kernel we are building

Test-Parameters: trivial

Lustre-change: https://review.whamcloud.com/31826
Lustre-commit: ab4e054fca6c8e300d2fac96c5c882ed188a5237

Change-Id: Ic3a6accda4fc19d56676e2fb84f65942bc107539
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Peter Jones <peter.a.jones@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Joseph Gmitter <joseph.gmitter@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/32059
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-10859 ldiskfs: fix deadlock with heavy memory preassure

On one Customer site, we hit following deadlock:

Thread 1:
ofd_object_punch
osd_punch
  ldiskfs_truncate
   ldiskfs_inode_attach_jinode
     ...
     do_try_to_free_pages
      lu_cache_shrink
       mutex_lock -->try to hold @lu_sites_guard

kswapd thread2:
kthread
shrink_slab
  lu_cache_shrink
    mutex_lock ---->hold already.
     ...
     dqget
      ldiskfs_acquire_dquot
       jbd2__journal_start-->blocked to wait for more credits.

Thread3:
kthread
kjournald2
  jbd2_journal_commit_transaction-->blocked to wait Thread2 finished,
since Thread1 add a handle into transaction.

So deadlock happens because of Thread1 wait Thread2, Thread2 wait Thread3..
but Thread3 wait Thread1....

This problem still exists even we have switched @lu_sites_guard
into a read/write lock, sine we hold write lock at lu_cahce_shrink().

Fixed the problem by making ldiskfs_inode_attach_jinode() use
GFP_NOFS.

Test-Parameters: testgroup=review-ldiskfs \
mdtfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs

Lustre-change: https://review.whamcloud.com/31806
Lustre-commit: 0506e1bd6a6d5fafe7fc5e558aa1b75e456c2642

Change-Id: I0ab143fc0cdb8e1b0c490c2c25e8af483c491a81
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/32058
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-10350 tests: make parsing routines pattern aware

'lfs getstripe' now returns the pattern for each component
of a directory and files. The routines that parse
parameters, parse_layout_param() and parse_plain_param(),
need to look for the component pattern when parsing the output
of 'lfs getstripe'.

Lustre-change: https://review.whamcloud.com/30636
Lustre-commit: 503a78bde8a59e176356a02b2d078332e3201575

Test-Parameters: trivial testlist=sanity-pfl,ost-pools
Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: Iab605f58e9c8f501fa0889806c511e3310cb6dd7
Reviewed-on: https://review.whamcloud.com/30636
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
(cherry picked from commit 503a78bde8a59e176356a02b2d078332e3201575)
Reviewed-on: https://review.whamcloud.com/32048
Reviewed-by: Wei Liu <wei3.liu@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-10334 tests: add tests to ALWAYS_EXCEPT for Ubuntu

Several tests are known to fail when running on Ubuntu clients:
tests 103a, 130a, 130b, 130c, 130d, 130e, 400a, and 410.

Add these tests to the ALWAYS_EXCEPT list to allow Ubuntu
testing to pass.

Lustre-change: https://review.whamcloud.com/31828
Lustre-commit: 68be7b634d8e46c110f11916dd87b9f1c36a90e1

Test-Parameters: trivial clientdistro=ubuntu1604 serverdistro=el7 testlist=sanity
Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: I096db598e9d8f5e1722f22a3213da5b3553cf89c
Reviewed-on: https://review.whamcloud.com/32047
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Wei Liu <wei3.liu@intel.com>
Reviewed-by: Saurabh Tandan <saurabh.tandan@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-10795 quota: fix wrong skipping of reintegration

There are two problems addressed by this patch:
1)In qsd_prepare(), if @qqi_acct_failed is true,
that only means one type of quota failed, Quota
should continue to handle.
2)In qsd_config(), only trigger reintegration if
this type of quota is newly enabled, this could
fix annoying messages when admin running

$ lctl conf_param lustre.quota.mdt=ug

LustreError: 0-0: lustre-MDT0000: can't enable
quota enforcement since space accounting isn't
functional. Please run tunefs.lustre --quota on
an unmounted filesystem if not done already

Lustre-change: https://review.whamcloud.com/31607
Lustre-commit: 7a216e31b74ef0b825adf36eed5f636da8b07533

Change-Id: I9bad618e7e8fa836902cac9f446714cd6c03f98a
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Hongchao Zhang <hongchao.zhang@intel.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/32013
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-10364 test: Add version check to test_255b

Skip test if server version is older than 2.8.54 since
it does not support ladvise.

Lustre-change: https://review.whamcloud.com/30481
Lustre-commit: 043f5976d087a87bace840653c99e10cde13bb6d

Change-Id: Ie613c1b8d9d082b78529a3d72fd59150431f65ea
Test-Parameters: trivial
Signed-off-by: Wei Liu <wei3.liu@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Saurabh Tandan <saurabh.tandan@intel.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/32002
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-9019 libcfs: add ktime_compare for platforms lacking it.

The function ktime_compare() was added to the linux kernel
starting with verison 3.19. Add to libcfs support for older
platforms lacking this function.

Lustre-commit: 4bbc835dc87cf947e87bd687553602781381fc81
Lustre-change: https://review.whamcloud.com/30475

Test-Parameters: trivial

Change-Id: I7826b8e78d0dc2c633490a2949210176a0003d9a
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-on: https://review.whamcloud.com/31994
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-10707 lnet: revert to cfs_time functions

In 2.10 the entire set of changes to using ktime function is not in
creating a situation where routers consider peers dead, essentially
breaking routing functionality.

Revert the changes that were made to LNet as part of LU-6245.

Signed-off-by: Amir Shehata <amir.shehata@intel.com>
Change-Id: Id0765e8332a8b97167d3a602e6410f5bb6a48137
Reviewed-on: https://review.whamcloud.com/32082
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-10707 ksocklnd: revert back to jiffies

It was found for the case of routers which have LNet enabled
ethernet and infiniband that they couldn't communicate properly
with other. This was due to ko2iblnd still using jiffies and
the socklnd driver using time64_t. After some discussion it was
deceided that the best thing to do is roll back instead of
additional changes.

Revert "LU-9397 ksocklnd: move remaining time handling to 64 bits"

This reverts commit 59c25356208706b4dd0920bc7754f21e2db14a0f.

Change-Id: I42fd5620ab8131cd5fdf4bf0ef15553f6eabe550
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-on: https://review.whamcloud.com/32015
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-9444 tests: replace SINGLEMDS1 with SINGLEMDS

In conf-sanity test 87, we use the global variable SINGLEMDS1
to get the version of the MDS. SINGLEMDS1 is not defined and
the test should use SINGLEMDS to check the version of the MDS.

Lustre-change: https://review.whamcloud.com/31420
Lustre-commit: 5f00d6f5fce28c7cee3b400601e265358e3cdc66

Test-Parameters: trivial testlist=conf-sanity
Test-Parameters: envdefinitions=ONLY=87 mdsjob=lustre-b2_9 ossjob=lustre-b2_9 serverbuildno=22 testlist=conf-sanity
Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: Ic8b1f32b87cc596fcc2e98d5b6095b6e4171bfd7
Reviewed-by: Wei Liu <wei3.liu@intel.com>
Reviewed-by: Saurabh Tandan <saurabh.tandan@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/31967
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-10541 llite: setxattr directly in ll_set_acl

Call md_setxattr() directly from ll_set_acl().

Lustre-change: https://review.whamcloud.com/31588
Lustre-commit: 682939ffb60e9afff9a7d61cc1112611b2197b78

Test-Parameters: alwaysuploadlogs clientdistro=sles12sp3 testlist=parallel-scale-nfsv3
Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ie266ee4fe7a67338122a6a3effb545d3dbaee008
Reviewed-by: Hongchao Zhang <hongchao.zhang@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/31966
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>

LU-10837 ldiskfs: skip bitmap check if block bitmap is uninitialized

See comments in ext4_free_clusters_after_init:
/* Return the number of free blocks in a block group. It is used when
* the block bitmap is uninitialized, so we can't just count the bits
* in the bitmap. */
So extra check we enhanced here is wrong if this block group
bitmap is uninitialized, since we only check bitmaps here.

Further, Looking at EXT4_BG_BLOCK_UNINIT clear codes, Kernel
will reinit free_clusters_count when tried to clear the flag, so
extra check for uninited block bitmaps dosen't make much sense.

Let's skip uninited block bitmap check if EXT4_BG_BLOCK_UNINIT
is set, whatever free count group desc recorded is untrustable somehow

Lustre-change: https://review.whamcloud.com/31720
Lustre-commit: f7672143843a86656eaed5b9a8b89ce171b99d7a

Change-Id: I845f2e0e17e53b7e3073399bd8b0a85e3db66ef8
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/31957
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-10802 nrs: mismatch problem for wildcard in jobid TBF

When set the NRS JOBID rule
"start runas jobid={*.500} rate=10", run the dd with user 500,
the RPC rate is not under control.
This patch fix this mismatch problem for wildcard in TBF JOBID.

Test-Parameters: trivial testlist=sanityn
Lustre-change: https://review.whamcloud.com/29162
Lustre-commit: 96f5cc0eb36c0bc799016cb3c6f11c3ddf475157

Change-Id: I39a8e691c9dc8273ed9fce686eeef71be1ac3e43
Signed-off-by: Qian Yingjin <qian@ddn.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/31956
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-10560 lustre_compat: Convert GFP_TEMPORARY to GFP_KERNEL

The 4.14 kernel removes this gfp.h define.

Adjust the code to use GFP_KERNEL as the upstream
patch does.

Lustre-change: https://review.whamcloud.com/31152
Lustre-commit: f01bed03905d424fab81bee84a7b622f141e4f79

Change-Id: I40fff2724499fa17aa285507e0fd9b21f4afc070
Signed-off-by: Mike Marciniszyn <mike.marciniszyn@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/31926
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>

LU-10560 llite: remove extra headers from rw26.c

Remove headers from llite_mmap.c that are no longer (or were possibly
never) needed.  This avoids a compile problem with 4.14 kernels:

CC [M]  lustre/llite/rw26.o
In file included from lustre/llite/rw26.c:43:0:
./arch/x86/include/asm/uaccess.h: In function ‘set_fs’:
./arch/x86/include/asm/uaccess.h:31:9:
error: dereferencing pointer to incomplete type
current->thread.addr_limit = fs;

It turns out <asm/uaccess.h> was included twice but we don't need
anything from that header.  Same for <linux/stat.h> and a number of
other extraneous headers.

Sort headers alphabetically so it is easier to avoid duplicates.

Lustre-change: https://review.whamcloud.com/31151
Lustre-commit: b8e87b5a1c17048a760ce9809fde904ca6dc449d

Change-Id: I4cbe2c1c7289466396b2bb2eac3c475d1041a283
Signed-off-by: Mike Marciniszyn <mike.marciniszyn@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/31925
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>