LU-15513 lod: iterate initialized stripe

Fix an array out of bounds access.

Do not iterate over the component objects in lod_fill_mirrors() to
check non-rotational OSTs if the component is unintialized or it is
a MDT or RELEASED component.

Lustre-change: https://review.whamcloud.com/47028
Lustre-commit: TBD (from b9445c05576da11cc230b290b1ec53131a0e1c2a)

Fixes: 591a990c61 ("LU-15513 lod: skip uninit component in lod_fill_mirrors")
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I8530cd30fab750c47f15ee9278ec6d68871367cc
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47590
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-13363 lod: do object allocation in OST pool

Currently, the ltd->ltd_qos.lq_same_space boolean that decides
whether the LOD QOS allocator is active for an allocation or not
is tracked for the entire LOV. But when a pool is specified, this
judgement should be tracked on a per-pool basis.

sanity.sh test_116c is added to verify this patch.

Lustre-change: https://review.whamcloud.com/38136
Lustre-commit: e642e75cde0248eee30ca94aaeb81653db7f8d03

Test-Parameters: ostcount=6 testlist=sanity env=ONLY=116c
Signed-off-by: Emoly Liu <emoly@whamcloud.com>
Change-Id: I463d5927c7a9c9171483615d2cec629ec10dc666
Reviewed-on: https://review.whamcloud.com/47334
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15952 doc: improvement on lfs-project doc

Describe 'lfs project -C' clearly.

Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial
Change-Id: I1e0f70a9116265edac993b12b775bd57c8587d40
Reviewed-on: https://review.whamcloud.com/47645
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14765 test: disable sanity-flr/44c

file blocks number accounting does not include blocks occupied by
multiple mirrors, disable the test for now.

Lustre-commit: a45fe93cd8d1e941d58e0f11e21649e1956ba2c7
Lustre-change: https://review.whamcloud.com/44074

Test-Parameters: trivial testlist=sanity-flr
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I1a5d7e0c203d5c7dc0da53ce6c020795bf17b9fc
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47647
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14583 llapi: handle symlinks in llapi_file_get_stripe()

In llapi_file_get_stripe(), if the IOC_MDC_GETFILESTRIPE ioctl handler
returns -ENOTTY or -ENODATA then try to resolve any symlinks in the
path and try again.

Lustre-change: https://review.whamcloud.com/43229
Lustre-commit: cd4caef54f63b6a0d26a90d879f378b7fe3d4862

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Ic046d6ef77d8342d47336144e3066cab3a940a96
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47588
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-12022 tests: error on resync failure sanity-flr

In sanity-flr test 200, we should error if the final resync
fails.  Replace all calls to 'mirror_io resync' that does
not inject an error to  '$LFS mirror resync'.

Lustre-change: https://review.whamcloud.com/35754
Lustre-commit: 9e1e29406c3886bce0db6623be40d3edf6834117

Test-Parameters: trivial testlist=sanity-flr
Signed-off-by: James Nunez <jnunez@whamcloud.com>
Change-Id: I9b2ec1beb7060086808b7529467bef80c8e9659f
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Emoly Liu <emoly@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47617
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-5364 tests: add sanity-flr/38 to ALWAYS_EXCEPT

Add sanity-flr.sh test_38 to ALWAYS_EXCEPT as it is failing too often.

Test-Parameters: trivial testlist=sanity-flr
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ia27e32d7309bc3597ee4024e465959c9dbd58b4a
Reviewed-on: https://review.whamcloud.com/47610
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn45

New tag 2.14.0-ddn45

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ic13969b93a939490962ecc13c950d19ba19f1a76

LU-15838 autoconf: fix use of obsolete macros

This patch fixes the following warnings when using autoconf 2.71:

configure.ac:2: warning: AC_INIT: not a literal:
                "m4_esyscmd(sh -c "./LUSTRE-VERSION-GEN | tr -d '\n'")"
configure.ac:10: warning: The macro `AC_CANONICAL_SYSTEM' is obsolete.
configure.ac:16: warning: The macro `AC_PROG_LIBTOOL' is obsolete.
configure.ac:24: warning: The macro `AC_HELP_STRING' is obsolete.

Like m4_esyscmd, macro m4_esyscmd_s (introduced in autoconf 2.64)
expands to the result of running command in a shell. The difference
is that any trailing newlines are removed.

Since autoconf 2.50, macro 'AC_CANONICAL_TARGET' has been the new name
of 'AC_CANONICAL_SYSTEM':
AU_ALIAS([AC_CANONICAL_SYSTEM], [AC_CANONICAL_TARGET])

Since autoconf 2.58, macro 'AS_HELP_STRING' has been added to replace
'AC_HELP_STRING'.

Since libtool 2.0, new 'LT_INIT' interface has been added to replace
'AC_PROG_LIBTOOL'.

Lustre-change: https://review.whamcloud.com/47288
Lustre-commit: TBD (from a94a730df48089e3ffbf853af78a2444dc5a3042)

Change-Id: I3c06c21460d7a2cf643fe825e72a26a5416609cf
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47539
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-5347 lipe: update lpcc to python3

Update the lpcc utility to use python 3.

Test-Parameters: trivial clientdistro=ubuntu2204 env=SANITY_PCC_EXCEPT="15 16 27 39 101a 201" testlist=sanity-pcc
Test-Parameters: trivial clientdistro=ubuntu2004 env=SANITY_PCC_EXCEPT="101a" testlist=sanity-pcc
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Signed-off-by: Lei Feng <flei@whamcloud.com>
Change-Id: I690ef4362cbdd0118419ab37a15127619a860178
Reviewed-on: https://review.whamcloud.com/47553
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-5360 lipe: lpcc handles unmounted lustre fs

If lustre fs is unmounted, lpcc daemon crashes while serving
'lpcc status' command. Fix it.

Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial testlist=sanity-lipe
Change-Id: I5304745aed001fbe26b77ed3cebfaabfddfce11f
Reviewed-on: https://review.whamcloud.com/47572
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

EX-5045 lipe: use LDADD

Use foo_LDADD rather than foo_LDFLAGS to avoid future errors with
strict linkers. Remove AC_SEARCH_LIBS() uses for ext2fs and com_err,
since we already use PKG_CHECK_MODULES() to check for ext2fs
libraries, and not all lipe utilities need -lext2fs.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I1de960fcec5a8770984d9b1437423a854948eb84
Reviewed-on: https://review.whamcloud.com/47443
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Feng, Lei <flei@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15900 hsm: don't return error on state change during mount

HSM coordinator is started in stopped state always, but
mount may have hsm_control parameters 'disabled'. Such
parameter cause wrong state change so mount would fail
with error.

Treat parameter change from 'stopping/stopped' to 'disabled'
as not critical error, keep state unchanged and report
no error back to a caller.

Lustre-change: https://review.whamcloud.com/47473
Lustre-commit: TBD (from c3fc6fef304d052da011c5840134af197eaefd94)

Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I9d1366423391971b9511c46b6aed39d21ebf637c
Reviewed-on: https://review.whamcloud.com/47474
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15484 tests: Increase timeout to wait for child

Sanity-pcc 101a is failing because the remote 'sleep 600'
process is not always started in time, meaning the NS
doesn't exist.

Increase the wait time for this process to start.

Lustre-change: https://review.whamcloud.com/46764
Lustre-commit: TBD (20f794e7e8b736a944c9e1e87970e7232abd6a19)

Fixes: f3be560031 "LU-15170 llite: Switch pcc to lookup_one_len"

Test-Parameters: trivial
Test-Parameters: testlist=sanity-pcc env="ONLY=101a,ONLY_REPEAT=50"
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I57175ec2f0e948045cf88d4a13a1e56cc34f8407
Reviewed-on: https://review.whamcloud.com/47557
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14785 build: changelog updates should not dirty version

When building lustre debs the final version should not
include 'dirty' due to an update of the changelog

Lustre-change: https://review.whamcloud.com/44069
Lustre-commit: a91839eb57103e6fbf5a3f6bacfcc0c471620057

HPE-bug-id: LUS-10152
Test-Parameters: trivial
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I59f4da4b3006302e3598cfa56a0364b052f885ef
Reviewed-on: https://review.whamcloud.com/47535
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-5347 build: update libssl3

In Ubuntu 22.04 libssl1.1 has been superceded by libssl3

Test-Parameters: trivial clientdistro=ubuntu2004

Change-Id: Ic2504c3a40d5c756c2da7b81edb1501fb6b44712
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47556
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Colin Faber <cfaber@ddn.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15918 build: update debhelper compatibility level to 11

This patch updates debhelper compatibility level to 11 (which is
currently the recommended level) to resolve the following build
warnings while running "make debs":

Compatibility levels before 10 are deprecated (level 9 in use)

Test-Parameters: trivial clientdistro=ubuntu2004 \
testlist=sanity

Change-Id: Ie5c63f6a3eb29e9aeace1f008d8d492e7587adf2
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47551
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15573 build: remove mpi, CC deps from config cache

Also drop mpi and CC dependencies compiler type from the
initial values in the configure cache as these may be
changed during the setup in the rpm spec or debian build rules

Lustre-change: https://review.whamcloud.com/46562
Lustre-commit: TBD (from 497cea4815a3e08f3ffd88ecba0c129a9e9ca5b8)

Test-Parameters: trivial
Fixes: a5084c2f2e ("LU-14937 build: re-use config cache in 'make rpms/debs'")
Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I986c2ae3653deae08b9da8d64d0b3c02fdc8fa2b
Reviewed-on: https://review.whamcloud.com/47536
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15908 build: fix debian rules

A comment in debian/rules file breaks a commands-line continuation,
causing exported env vars to be dropped.

Lustre-change: https://review.whamcloud.com/47512
Lustre-commit: TBD (from d1d5b35306bc164afe444164a40fdd3940a2abcd)

Fixes: a5084c2f2e ("LU-14937 build: re-use config cache in 'make rpms/debs'")
Test-Parameters: trivial
Signed-off-by: Ake Sandgren <ake.sandgren@hpc2n.umu.se>
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I0894199301c489b1961e043651c74d7c6c0089d2
Reviewed-on: https://review.whamcloud.com/47537
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15796 build: require python2 for lustre-iokit deb

Package python is not available on Ubuntu 22.04.
We can just require python2 in the control files.

Lustre-change: https://review.whamcloud.com/47164
Lustre-commit: TBD (from d46e59a541a4a2db73ccae95707f08d5d9c40ccd)

Test-Parameters: trivial clientdistro=ubuntu2004 \
testlist=obdfilter-survey

Change-Id: Id37c9babba31c575855238592cbac4ce5c9ee79a
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47538
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15420 build: fixes to support building on Ubuntu 22.04 LTS

Lustre uses the glibc stdarg.h instead of the kernel's version which
causes the following build issue.

lustre/include/lu_object.h:35,
/usr/lib/gcc/x86_64-linux-gnu/11/include/stdarg.h:52: note: this is the
location of the previous definition
        #define va_copy(d,s)    __builtin_va_copy(d,s)

The solution is to use the kernels version of stdarg.h

The second build issue :
update_trans.c:1608:30: error: ‘struct task_struct’ has no member named
                        ‘state’; did you mean ‘__state’?

is due Linux commit 2f064a59a11ff9bc22e52e9678bc601404c7cb34
(sched: Change task_struct::state). The state field was
changed and the barrier macros READ_ONCE()/WRITE_ONCE()
are used to access it now which is the proper thing to do.
Since the check in update_trans.c is equivalent to testing
if the kernel thread is not running, since TASK_RUNNING == 0,
we can just change the code to use task_is_running(). The
task_is_running() was introduced in 5.13.

Lustre-change: https://review.whamcloud.com/47133
Lustre-commit: TBD (from 6b096f55bad9bd169b2aa01237764cf97b58ee56)

Test-Parameters: trivial
Change-Id: Ib5985b187c3013fbc513e9962a5f27bed4996f5b
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/47540
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-10632 tests: recovery-small test_26 idle_timeout

In recovery-small test_26() use "lfs df" instead of plain "df"
since statfs may be fetched from the MDS cache and will not
ensure that the client->OST connections are currently active.

Also, check a few entries further back in the OSC state log for an
EVICTED message, in case the client idle disconnects from the server
again while checking all of the imports.

Lustre-change: https://review.whamcloud.com/42006
Lustre-commit: b4391fcdaf392a50bd1419342eca3b730c077ed2

Test-Parameters: trivial testlist=recovery-small env=ONLY=26a,ONLY_REPEAT=100
Fixes: 5a6ceb664f07 ("LU-7236 ptlrpc: idle connections can disconnect")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I8c370cb75f4e06258ef3c032630fc20354a15dcc
Reviewed-on: https://review.whamcloud.com/45296
Tested-by: jenkins <devops@whamcloud.com>

LU-15220 lipe: avoid gcc-11 -Werror=stringop-truncation warning

This patch fixes -Werror=stringop-truncation warnings
in lipe codes.

Test-Parameters: trivial testlist=sanity-lipe

Change-Id: Ie007083b1fe94d81e3772d7c0947457d2c3042ac
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47510
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn44

New tag 2.14.0-ddn44

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ia5ec26041bac7cc62fcef79fd7c8ff2d5ca170da

LU-15896 gss: ignore OpenSSLv3 deprecated API

Lustre GSS code makes use of some OpenSSL API that has been
deprecated in v3, namely all the functions in the DH_* family.
Those would need to be replaced with their EVP_PKEY_* counterparts
but in the meantime we just use the OPENSSL_SUPPRESS_DEPRECATED
define to continue using the deprecated API.

Lustre-change: https://review.whamcloud.com/47485
Lustre-commit: TBD (from 05d5b73ebb253c0e3ed5a7954b751ebbdb2d61bf)

Test-Parameters: trivial testlist=sanity-sec
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I413a2af1299123b29ed2043188da0348ce9d0a5e
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47496
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15776 tgt: fix transaction handling in tgt_brw_write()

Hotfix to prevent possible data loss during WRITE replay.
Since commit f0f92773ee18 from LU-14187 the obd_commitrw()
may restart write transaction in OFD and MDT. That causes
transaction number to be assigned multiple times if such
restart happens. Without flag tti_mult_trans the first
transaction number is stored only so later one could remain
not applied causing data loss after recovery.

Patch sets tti_mult_trans for tgt_brw_write() so the latest
transaction number will be used as request transno.

Lustre-change: https://review.whamcloud.com/47371
Lustre-commit: TBD (from ccce4f23c773d9b9cc98cb15ed31861803167bb9)

Fixes: f0f92773ee ("LU-14187 osd-ldiskfs: fix locking in write commit")
Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Change-Id: I364b478591942be5562c3e98ee6e6aa487f3e0c5
Reviewed-on: https://review.whamcloud.com/47428
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15840 lov: return st_blocks=1 for HSM released files

The MDT will return st_blocks=1 for a HSM released file.
In the call ->coo_attr_get in LOV layer, the client should also
return st_blocks=1 for a HSM released file.

Otherwise, the client may get 0 block count. It is very easy to
reproduce this problem via the following commands for a archived
file:
# $LFS hsm_restore $file
# $LFS hsm_release $file
# $LFS hsm_release $file
After release a file twice, the reported block count via stat()
call will become 0.

Lustre-change: https://review.whamcloud.com/47291
Lustre-commit: TBD (from 974320c61ee7d165073bb2810795575261996dc7)

Change-Id: Id1841147e40a7df0ca615e887f324cff8e613f11
Signed-off-by: Qian Yingjin <qian@ddn.com>
Reviewed-on: https://review.whamcloud.com/47290
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-5249 utils: show pin information in 'lfs pcc state' command

If xattr lustre.pin exists, show its content in 'lfs pcc state'
command.

Change-Id: I867ea3b246fc3b7d10e166d754c081c8afc462d5
Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial
Reviewed-on: https://review.whamcloud.com/47287
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15841 lod: iterate component to collect avoid array

In newly created file, the mirror information hasn't been established
as LOD is trying allocate OST for its components, so we need to
iterate component instead of mirror to collect the avoid guidance
information.

Lustre-change: https://review.whamcloud.com/47293
Lustre-commit: TBD (from d04b04056e45edc95117f0761359ddc4637d3f37)

Test-Parameters: testlist=sanity-flr env=ONLY=47,ONLY_REPEAT=40
Fixes: fabf3fe7 ("LU-9007 lod: improve obj alloc for FLR file")
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I6bbe5f6b6dfea06c5213b77b7ebb6a5d28aa0d17
Reviewed-on: https://review.whamcloud.com/47295
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-5176 tests: re-enable sanity test_400a

Re-enable test_400a after it was disabled due to json-c dependency
during build test.  This library is now installed on client nodes.

Test-Parameters: trivial testlist=sanity env=ONLY=400 clientdistro=sles15sp3
Test-Parameters: testlist=sanity env=ONLY=400 clientdistro=ubuntu2004
Test-Parameters: testlist=sanity env=ONLY=400 clientdistro=el8.5
Fixes: fbfd2d0755 ("EX-5176 pcc: use JSON string for trusted.pin xattr")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I3daca219cedb447127f9402a79e6124afb3ebbe5
Reviewed-on: https://review.whamcloud.com/47363
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Feng, Lei <flei@whamcloud.com>

LU-15848 ldiskfs: escape encrypted file names

When a Lustre MDT is mounted as ldiskfs, the names of the encrypted
files have to be escaped to avoid breaking the shell.
On CentOS 7, the LDISKFS_ENCRYPT_FL flag does not exist. So we add it,
and when the target is mounted as ldiskfs (LDISKFS_MOUNT_DIRDATA flag
not present) we critical-encode encrypted file names before
presentation. And conversely, we critical-decode names upon lookup.
On CentOS 8, the LDISKFS_ENCRYPT_FL flag exists. The fscrypt functions
from kernel 4.18 are also wired up, but they all refer to -EOPNOTSUPP
or equivalent, so they cannot be used to present usable names. So when
the target is mounted as ldiskfs (LDISKFS_MOUNT_DIRDATA flag not
present) we proceed to critical-encoding of encrypted file names
before presentation, and to critical-decoding upon lookup.

Lustre-change: https://review.whamcloud.com/47309
Lustre-commit: fa7c81e81442c8d1ed0ce1c457769f2e144ff2c3

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Iaca467eaa233be8142356efa822962953754c2ce
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47482
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>

LU-15706 llog: deal with "SKIP" pool llog records correctly

If the requested "start" llog records by one ioctl are just the
useful ones "protected" by one marker, they will don't know their
cfg_flags(e.g. "SKIP" or not), and then will be mis-labeled in
class_config_yaml_output().
To fix this issue, this patch does the following changes:
- In kernel space, remember the marker cfg_flags for 10 records
  earlier than the requested "start" in function llog_print_cb(),
  so that the "start" can be output with its correct cfg_flags.
- In user space, since the pool_new/add record will be marked as
  "SKIP" if its corresponding pool_destroy/remove record is logged
  later in mgs_pool_cmd(), these "SKIP" records won't be printed
  by function jt_llog_print_iter(), so lpd_ost_num doesn't need to
  be decreased in callback function llog_search_pool_cb() as well,
  otherwise, it will cause "lctl pool_destroy" error.

Lustre-change: https://review.whamcloud.com/46951/

Test-Parameters: standalonemgs=true testlist=ost-pools
Test-Parameters: standalonemgs=true testlist=conf-sanity env=ONLY=123,HONOR_EXCEPT=y
Fixes: 2a5b50d20717 ("LU-15142 lctl: fixes for set_param -P and llog_print")
Signed-off-by: Emoly Liu <emoly@whamcloud.com>
Change-Id: Ie45127ac8b80a75eaeb7158559c690da52eef103
Reviewed-on: https://review.whamcloud.com/47463
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15800 ofd: take a read lock for fallocate

there is no need to take an write (exclusive) object's
lock for fallocate - we just need to serialize fallocate
vs destroy, all internal structures should be protected
by OSD and disk filesystem like the write path does.

Lustre-change: https://review.whamcloud.com/47268
Lustre-commit: TBD (from 64e269c800533c4ce15235b21597038c29f0040a)

Fixes: cdaaa87f6b ("LU-14214 ofd: fix locking in ofd_object_fallocate()")
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I65986745865ee329c5257a7efca5e79403830608
Reviewed-on: https://review.whamcloud.com/47477
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14541 llite: Check vmpage in releasepage

We cannot release a page if the vmpage reference count is
>1, otherwise we will detach a vmpage from Lustre when the
page is still referenced in the VM.

This creates a situation where page discard for lock
cancellation will not find the page, so we can get stale
data reads.

This re-introduces the LU-12587 issue where direct I/O on
a client falls back to buffered I/O if there are pages in
cache, since it cannot flush them.  This is annoying but
not a huge problem.

This allows us to safely re-enable fast reads.

Lustre-change: https://review.whamcloud.com/47262/
Lustre-commit: c524079f4f59a39b99467d9868ee4aafdcf033e9

Fixes: e59f0c9a245f ("LU-12587 llite: don't check vmpage refcount in ll_releasepage()")
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I3aa1cd7330f5e7d1ba2ddb0c12779aa22f3d70b7
Reviewed-on: https://review.whamcloud.com/47273
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15875 kernel: kernel update RHEL7.9 [3.10.0-1160.66.1.el7]

Update RHEL7.9 kernel to 3.10.0-1160.66.1.el7.

Test-Parameters: trivial clientdistro=el7.9 serverdistro=el7.9

Change-Id: I9e8ab33edd6cacbbf895399962027827a1befd5b
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47423
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-15793 kernel: kernel update RHEL8.5 [4.18.0-348.23.1.el8_5]

Update RHEL8.5 kernel to 4.18.0-348.23.1.el8_5.

Test-Parameters: trivial fstype=ldiskfs clientdistro=el8.5 serverdistro=el8.5 testlist=sanity
Test-Parameters: trivial fstype=zfs clientdistro=el8.5 serverdistro=el8.5 testlist=sanity
Change-Id: I9724e304c2d32bb0ac2dcf4138438c79d4abca0f
Signed-off-by: Jian Yu <yujian@whamcloud.com>

LU-15795 kernel: new kernel [RHEL 8.6 4.18.0-372.9.1.el8]

This patch makes changes to support new RHEL 8.6 release
for Lustre client.

Test-Parameters: trivial clientdistro=el8.6

Change-Id: Id738259ed94104c3a3c7bb5c1b853cfabad49405
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47304
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Colin Faber <cfaber@ddn.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-14541 llite: add rw_seq_cst_vs_drop_caches

Add a reproducer (rw_seq_cst_vs_drop_caches) for the read/write vs
drop_caches sequnetial consistency violation described in
LU-14541. Add an always excepted test (sanityn test_16f) to run
rw_seq_cst_vs_drop_caches.

Lustre-change: https://review.whamcloud.com/47243/
Lustre-commit: 8b9185d500b1e3ccdf66c84c1cbba82a0e392e69

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I557ae7386b38214110a4d85ba0515e95fed7a11e
Reviewed-on: https://review.whamcloud.com/47272
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-5295 lipe: allow for missing changelog files

In lipe/scripts/stratagem-hp-config.sh, suppress errors from a missing
.chlg file.

Fixes: af041fccc1 ("EX-4973 lipe: remove changelog user registration from config")
Test-Parameters: trivial
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Ic00a9bb591de22036936eecd76d08ec9649e4a9e
Reviewed-on: https://review.whamcloud.com/47429
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15830 mdt: mkdir to lookup target name

distributed mkdir should lookup the target name to
avoid rollback as much as possible as the latter is
very expensive due to llog re-initialization.

Lustre-change: https://review.whamcloud.com/47226
Lustre-commit: 6bac96ed3b4a6bf514e4bd4c9c27c8ad89d627d1

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: If28760e0afb804dca11e1e7501e0a53ff9067ca1
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Signed-off-by: Mikhail Pershin <mpershin@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47324
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-5260 pumount: use O_PATH for mount point

In pumount, use O_PATH when opening the mount point to try to avoid
RPCs from permission or atomic_open.

Test-Parameters: trivial testlist=sanity-pumount clientextra_install_params="--packages pumount"
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Ibd66579d9b5ecef724334c01d90367af4bd93179
Reviewed-on: https://review.whamcloud.com/47323
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-3838 test: sanity-pumount races

In sanity-pumount, add functions wait_for_open(), wait_for_exec(), and
wait_for_mmap() to wait for the forked PID to exec, open, or mmap the
file on the Lustre FS. Use these to prevent fork exec check races.

Test-Parameters: trivial testlist=sanity-pumount,sanity-pumount,sanity-pumount,sanity-pumount clientextra_install_params="--packages pumount"
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I2a89989521cc4e61d35799c1d4fdb8ce37ae12d9
Reviewed-on: https://review.whamcloud.com/47344
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-3763 test: disable sanity-pcc test_44

Disable sanity-pcc test_44 until the failures are sorted out.

Test-Parameters: trivial testlist=sanity-pcc
Change-Id: I8f5ab5025e8d049e1488b3f48708057ac7523600
Signed-off-by: Qian Yingjin <qian@ddn.com>
Reviewed-on: https://review.whamcloud.com/47336
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-15827 osd: respect filldir buffer limits

In osd_ldiskfs_filldir() ensure that the encoded name also fits in the
buffer. In osd_ldiskfs_filldir() and obj_name2lu_name() remove a
superfluous and potentially incorrect check on names that appear to be
FIDs.

Lustre-change: https://review.whamcloud.com/47224
Lustre-commit: 5879647624c834d4a579f425a0fc8d408235e275

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I1540f058801b002474ceac1206317344cebb1084
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-on: https://review.whamcloud.com/47325
Tested-by: jenkins <devops@whamcloud.com>

LU-14541 llite: reenable fast_read by default

Reenable fast_read by default.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I9985beb88d13d21ee538c632fc288470a58b0257
Reviewed-on: https://review.whamcloud.com/47299
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-3442 test: disable hot-pools test_15

Disable hot-pools test_15 until the failures are sorted out.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I787f08161846ec0dbb29524aa60d6eec40007d9c

RM-620 build: New tag 2.14.0-ddn43

New tag 2.14.0-ddn43

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ibe83ff8c478cd7678297cc11a0e205b5ee7b892f

LU-15846 tests: don't use comma-separated debug flags

To avoid test interop issues between b_es6_0 clients and b_es5_2
servers, don't use comma-separated debug flags in sanity-quota.sh
quota_init() and quota_fini().

Lustre-change: https://review.whamcloud.com/47308
Lustre-commit: TBD (from 12282314e2567973b1a7cb4d6806e987bf1a6755)

Test-Parameters: trivial testlist=sanity-quota.sh serverversion=2.12.6-ddn66
Fixes: 6b6fde1026 ("LU-13055 libcfs: allow comma-separated masks")
Fixes: 78be823f33 ("LU-15218 quota: delete unused quota ID")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ifca39054d14292bca8bcff9b8e03ae58fd5cc3a8
Reviewed-on: https://review.whamcloud.com/47307
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15285 mdt: fix same-dir racing rename deadlock

With LU-12125 lifting the BFL for same directory rename,
a deadlock possibility opens up since we lock source and target
of rename in the source-target order, if there are two renames
racing to rename arguments in reverse order:
mv a b &
mv b a

a lock inversion happens and a deadlock has been observed.

To avert this - instill additional order requirement:
lower PDO hash value is to be locked ahead of the higher one.

Lustre-change: https://review.whamcloud.com/45676
Lustre-commit: 82ec537d8b4cc9261828f4efe6b03d8d33f38432

Fixes: d76cc65d5d68 ("LU-12125 mds: allow parallel regular file rename")
Fixes: b50bb830f92e ("LU-3538 dne: Commit-on-Sharing for DNE")
Fixes: 9f1711f3d7d1 ("LU-12081 mdt: rename shouldn't PDO lock if parent is remote")
Change-Id: I88dd3aebb394ea40e97e6029d6dcc161116f982e
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47278
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-4973 lipe: remove changelog user registration from config

In stratagem-hp-config.sh, remove the changelog user registration
code. If no chnagelog user is configured then lamigo will register a
new named changelog user with the required mask.

Test-Parameters: trivial
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I457f4539451e93bf7980fc7fb807f67e67abf247
Reviewed-on: https://review.whamcloud.com/47045
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>

EX-5176 utils: fix package requirements for json-c

SLES15.3 uses "libjson-c3" for json-c RPM vs. "json-c" for RHEL.
Ubuntu 20.04 uses "libjson-c4" and 22.04 uses "libjson-c5", but
the right one is depended on by "libjson-c-dev".

Test-Parameters: trivial testlist=sanity clientdistro=sles15sp3
Test-Parameters: testlist=sanity clientdistro=ubuntu2004
Fixes: fbfd2d075534 ("EX-5176 pcc: use JSON string for trusted.pin xattr")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ibb01cfb7dadda83572be68961cf98450536980cf
Reviewed-on: https://review.whamcloud.com/47267
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Feng, Lei <flei@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-5235 lipe: Fix lipe-server build

To compile against liblustreapi, we now also need json-c.

Signed-off-by: Nathaniel Clark <nclark@whamcloud.com>
Change-Id: I5e5323c556d1d3bdd4fd6d7f9bc59e33baf59310
Reviewed-on: https://review.whamcloud.com/47263
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Feng, Lei <flei@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15025 quota: stale edquot after clearing limits

When hard and soft limit set to 0, lqe enforced flag is also
set to false. As qmt_adjust_qunit does not handle not enforced
lqes, edquot set to the pool continues to be true and a user
gets -EDQUOT even if all pool limits are cleared. This was ok
for global pool lqe as since it turned off, zero limits are
sent to OSTs causing OSTs to release all granted space and
avoid EDQUOT. Fix this for PQ - set edquot and qunit to zero,
since appropriate lqe becomes "not enforced".

Lustre-change: https://review.whamcloud.com/45000
Lustre-commit: 6ed5d6ed3bd0a705c6ae8c37a0b7e8906ebc6079

HPE-bug-id: LUS-10146
Change-Id: I1e08929bae7e1b37b1e8cbbc44859a786b5fb090
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Tested-by: Elena Gryaznova <c17455@cray.com>
Signed-off-by: Sergey Cheremencev <sergey.cheremencev@hpe.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-on: https://review.whamcloud.com/46657
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15787 sec: document enc-unaware clients on enc files

Document the behavior of encryption-unaware clients when they access
encrypted files.

Lustre-change: https://review.whamcloud.com/47182
Lustre-commit: 751a8114ef3afe9abe7692b3974b070db6a705a2

Test-Parameters: trivial
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I0354e3051e10aa0542baeb8e34c6201d47e65710
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47228
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15803 sec: correctly handle page lock in ll_io_zero_page

In ll_io_zero_page(), we need to make sure we have locked the page,
and it is up-to-date, before zeroing. So modify ll_io_read_page()
behavior to not disown the clpage for our use case. It avoids being
exposed to concurrent modifications.

Lustre-change: https://review.whamcloud.com/47170
Lustre-commit: edcd05e5ac035dd1daf263105df33d7cfc6dbf6e

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I58e4cf80374a798c9c4302364cf2fb39da9033bb
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47229
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-14802 nodemap: return proper error code

In nodemap_add_range_helper() it was always returning -ENOMEM when
there was an error inserting a new range into the existing nodemap.

    nodemap_add_range_helper()) cannot insert nodemap range: rc = -17
    mgs_iocontrol_nodemap()) MGS: OBD_IOC_NODEMAP command: rc = -12

This was confusing because the error returned by range_insert() was
typically -EEXIST (i.e. the entry being inserted already was in the
nodemap).  Do not print an error to the console in this common case.

Return the actual error to the caller so that this is more clear
to the end user.  Have l_ioctl() always set errno on error, in
addition to returning the error, since many callers depend on this.

Lustre-change: https://review.whamcloud.com/45626
Lustre-commit: 7d6cddff24de2f79e539eae462554dc21e674511

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I2c80a11dfdf9e6e1c9a8235b8f74f5bcea68c08e
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: Etienne AUJAMES <eaujames@ddn.com>
Reviewed-on: https://review.whamcloud.com/47247
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15787 sec: block enc unaware clients on enc files

Prevent encryption unaware clients from manipulating encrypted files
and directories. Those can be old clients, or clients built without
encryption support (intentionally or because they run on an old
kernel).
In the mdt layer, check that clients have the OBD_CONNECT2_ENCRYPT
connection flag, and if not, block access if they try to manipulate
a file or directory that has the LUSTRE_ENCRYPT_FL flag.
The forbidden operations from encryption unaware clients are:
- open
- create
- link
- rename
- migrate
Improve sanity-sec test_54 to test this use case.

Lustre-change: https://review.whamcloud.com/47156
Lustre-commit: a31db2ec062ccc995527d37f0330edbca9d486a9

Test-Parameters: testlist=sanity-sec mdscount=2 mdtcount=4 osscount=1 ostcount=8 clientcount=2 serverdistro=el7.9
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ief0639e49c0a8e1a1a0cb19cb13c006edfdff6c4
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47227
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn42

New tag 2.14.0-ddn42

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ifcaf23d84abf5802ab305e242d6ec3b3936d837e

LU-14790 tests: Check NI status when link is downed

Add test to check that NI status is set to down when the
ni_fatal_error_on flag is set (i.e. when a link is down).

Lustre-change: https://review.whamcloud.com/44073
Lustre-commit: 412970c586ea189d442620316c5ed6f1fa43cc0e

HPE-bug-id: LUS-10167
Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: If98a899b0ee8dd9637c08774109668ad06244c60
Reviewed-on: https://review.whamcloud.com/46846
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14790 lnet: Reflect ni_fatal in NI status

If the ni_fatal_error_on flag is set on an NI then that NI should be
considered down.

Lustre-change: https://review.whamcloud.com/44072
Lustre-commit: d77e95cc6d4e947be60757a861c656d8b30d2972

HPE-bug-id: LUS-10167
Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I201bda7e06da1fb1cc23db70ce0cfa3118635d0f
Reviewed-on: https://review.whamcloud.com/46845
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-5176 pcc: use JSON string for trusted.pin xattr

Change the format of trusted.pin xattr to JSON.
For example:
trusted.pin=[{"hsm":2}]
trusted.pin=[{"hsm":2},{"hsm":3},{"pool":"ddn_ssd"}]

Change-Id: I38aeb4ad960cf8324e77ac467f5e5cd742f9ec86
Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial testlist=sanity-pcc
Reviewed-on: https://review.whamcloud.com/47161
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15593 tests: fix version checks for backported patches

Check for 2.14.0.x versions instead of 2.14.5x versions:
   conf-sanity.sh: test_123ag
   sanity.sh: test_27S, test_64i, test_160q, test_413e
   sanity-flr.sh: test_44e
   sanity-sec.sh: test_60

Test-Parameters: trivial testlist=conf-sanity env=ONLY=123
Test-Parameters: testlist=sanity-flr,sanity-sec
Fixes: 32f0bbe2646a ("LU-15176 sec: allow subdir mount of encrypted dir")
Fixes: d053ce30ca2f ("LU-15593 mdt: Add option to disable use of SOM")
Fixes: a35ae4ec46ec ("LU-15142 lctl: fixes for set_param -P and llog_print")
Fixes: d35aa94e79de ("LU-14448 lod: verify LOV before set/inherit")
Fixes: 276182ae9b95 ("LU-13055 changelog: use default mask if server has no mask")
Fixes: 930d4fe4952f ("LU-9704 grant: ignore grant info on read resend")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ibd0e01ff5ac3b4a810b3eb5b8052e8a727ca416d
Reviewed-on: https://review.whamcloud.com/47206
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>

LU-14889 lproc: Add server checksum_type

This patch adds 'checksum_type' lproc entries under server:
1. obdfilter.$FSNAME-OST${count}.checksum_type
2. mdt.$FSNAME-MDT${count}.checksum_type

Test-case: sanity/77o added

Lustre-change: https://review.whamcloud.com/44755
Lustre-commit: c18d5d892b62ccee3cc9ec20a2278f17bb2d2f2c

Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Change-Id: I12a26f5c8c8f2f93d57f377626b1753fc13ffbb3
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-on: https://review.whamcloud.com/47142
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

EX-5199 sec: reinstate null encryption for file names

Reinstate null encryption for file names by adding a new llite
parameter named 'enable_filename_encryption', set to 0 by default.
When this parameter is 0, new empty directories configured as
encrypted ignore the filenames_encryption_mode and use
LLCRYPT_MODE_NULL instead, which is a no-op. This LLCRYPT_MODE_NULL
mode is inherited for all subdirectories and files.
When this parameter is 1, new empty directories configured as
encrypted use the normal encryption mode.

To set this parameter globally for all clients, do on the MGS:
mgs# lctl set_param -P llite.*.enable_filename_encryption=0

Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I9d726ba26cc91a51690d59a81efe3eb98ee2995c
Reviewed-on: https://review.whamcloud.com/47184
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15810 sec: fix enc dir migration

Now that the encryption context is stored in an xattr named
"encryption.c" instead of "security.c", we need to fetch this xattr
explicitly in case of encrypted directory migration. Indeed, there is
no xattr handler in ldiskfs for this "encryption." xattr type, so it
is not returned when listing all xattrs to migrate.

Lustre-change: https://review.whamcloud.com/47201
Lustre-commit: TBD (540f9208a9f6edcf6fdd1711e86fc182dfb92298)

Fixes: 4231fab66e ("LU-13717 sec: make client encryption compatible with ext4")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I628f9b253e86343db0b71f6a5b1ad2c5728ca38d
Reviewed-on: https://review.whamcloud.com/47207
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-12373 utils: fix llapi_pcc_del_internal() flags

The pch_flags value used by llapi_pcc_del_internal() was declared
as a "bool" but is used to pass a __u32 flags value from the caller.
This coincidentally works in the case where "flags = 1" (which is
the only value currently in use), but would break in confusing ways
in the future.

Change pch_flags to be "enum lu_pcc_cleanup_flags" to make it clear
what possible flag values are being used in this interface.

Fixes: 871da4898405 ("LU-12373 pcc: uncache the pcc copies when remove a PCC backend")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I41a008856af396376303fb156e2a92bce53ebbe5
Reviewed-on: https://review.whamcloud.com/47123
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Feng, Lei <flei@whamcloud.com>

LU-15761 obdclass: fix locking in llog_cat_refresh()

the patch fixes two problems:
1) pairing up_write() should be used with cathandle
2) llog_read_header() manipulates loghandle's internal
   structures (header, last_idx, etc) which are supposed
   to stay consistent from another user's point of view
   (like llog_add_rec())

Lustre-change: https://review.whamcloud.com/47185
Lustre-commit: 69cdcbe3cf732124b3b5be1a1f235aa1b78f6c85

Fixes: 71f409c9b31b ("LU-11418 llog: refresh remote llog upon -ESTALE")
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Ib86e10a925b541d02c22d74e6ddbc4368345ac11
Reviewed-on: https://review.whamcloud.com/47190
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15815 llite: disable fast_read and workaround

Revert the fast_read stale data workaround from LU-14541 and disable
fast_read by default. The workaround causes applications to receive
spurious SIGBUGs when reclaim is concurrent with mmap page fault
handlers. We disable fast read to avoid the stale data issue entirely.

This reverts commit f2a16793fa4316fc9ccdc46bcfe54f6b8d1e442b and
re-exposes us to the consistency issues described in LU-14541.

Lustre-change: https://review.whamcloud.com/47204

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I716b1fd6ab22242b9267b8883f0371a360aaecef
Reviewed-on: https://review.whamcloud.com/47205
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

RM-620 build: New tag 2.14.0-ddn41 and kernel ddn17

Increase kernel build version to -ddn17.

New tag 2.14.0-ddn41

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I3ebbe53c8a8d3c02d2831b8ac6d3068c0b3ebbe5

RM-620 build: New tag 2.14.0-ddn40

New tag 2.14.0-ddn40

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I3a069f3c8a8d3c02d2831b8ac6d3068c0be50a64

DDN-1956 tests: exercise laudit/laudit-report

Add sanity-laudit.sh to exercise laudit and laudit-report command
line tools.

Test-Parameters: trivial
Test-Parameters: mdtcount=2 testlist=sanity-laudit serverextra_install_params="--packages lipe-client" facet=mds1
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I97fefecab6442d9031de6a7fcd86d07a517386c2
Reviewed-on: https://review.whamcloud.com/46907
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14156 utils: mirror split to check for last in-sync early

currently this check to prevent last in-sync component is done
once the file is open with O_RDWR which interrupts on-going
resync/extend process. instead we can do this check early once
the layout is fetched (after the first open with O_RDONLY).

Lustre-change: https://review.whamcloud.com/40782
Lustre-commit: TBD (from 9996c22e346c219df5b165eb5ceab09873303bc2)

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Iee08d23008b44d2a7b2127358116a95ace40b7dd
Reviewed-on: https://review.whamcloud.com/47058
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15733 kernel: kernel update RHEL7.9 [3.10.0-1160.62.1.el7]

Update RHEL7.9 kernel to 3.10.0-1160.62.1.el7.

Test-Parameters: trivial clientdistro=el7.9 serverdistro=el7.9

Change-Id: I063ebf0434dceb023847f49f37d6217f5c719b58
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47063
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-15536 utils: add lfs somsync utility

This patch adds lfs somsync utility to synchronize
SOM xattr(s) for given FILE(s) or FID(s).

lfs somsync FILE ...
lfs somsync --by-fid MOUNT FID ...

Test-Parameters: trivial env=ONLY=807 testlist=sanity

Change-Id: Ie9ee39625d56ec026c89dcc0f27025904ca354e3
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46942
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14110 obdclass: Protect cl_env_percpu[]

cl_env_percpu is not protected against multi client mounts on the
same node: "keys_fill" could be called with the same cl_env_percpu
context by several mount processes (race on lu_context.lc_value).

This patch add a mutex for cl_env_percpu to proctect contexts
"refill".

Lustre-change: https://review.whamcloud.com/40565
Lustre-commit: 881551fbb7335694b89a877072bcda0aeaf8705c

Signed-off-by: Etienne AUJAMES <eaujames@ddn.com>
Change-Id: Icfd6f3715899fa4ac5279e932f462e7cf29d98bd
Reviewed-by: Neil Brown <neilb@suse.de>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/47073
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15145 hsm: unlock the restore layout lock for a cancel

The HSM restore EX layout lock is not unlock by a HSM cancel action
or by "hsm_control=purge" parameter.

This patch call cdt_restore_handle_del() in mdt_cancel_all_cb() and
mdt_agent_record_update_cb() for restore action (when updating action
status to ARS_CANCELED).
The test "sanity-hsm test_103a" checks the "purge actions" with
blocking restore.

Lustre-change: https://review.whamcloud.com/45341
Lustre-commit: 6d4019281b392bcb6993d1cfca3d47d7fa5f7c56

Test-Parameters: testlist=sanity-hsm,sanity-hsm
Signed-off-by: Etienne AUJAMES <eaujames@ddn.com>
Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: Id891e06aacd2a2c5950048a2d2a5d1398eedfdd7
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46932
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14958 kernel: use rhashtable for revoke records in jbd2

resizable hashtable should improve journal replay time when
the latter has got million of revoke records

before:
1048576 records - 95 seconds
2097152 records - 580 seconds

after:
1048576 records - 2 seconds
2097152 records - 3 seconds
4194304 records - 7 seconds

Lustre-change: https://review.whamcloud.com/45122
Lustre-commit: TBD (from cc2996585c489632fc8fb02ebf61d755a339361a)

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I9a9e3801223fa9e36cbf6d2ef5ddbad5dff3e19d
Reviewed-on: https://review.whamcloud.com/46371
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13485 build: remove parallel configure dir

This patch removes the temporary directory for
parallel configure tests at the end of configure
to avoid checking the already generated modules
while configuring the same Lustre codes with
a different kernel.

Lustre-change: https://review.whamcloud.com/47077
Lustre-commit: TBD (from e0db865abbe8d7fe3a61a649a17f70a88741eb66)

Change-Id: I4505fb932954531d262b5380f17b3df75a80bfa0
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/47081
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: Nathaniel Clark <nclark@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-4080 build: Use -j nproc in lustre dkms install during make

Test-Parameter: trivial

Change-Id: I4f815a0be8900ad427e59cace9a6afc3297710d0
Signed-off-by: Gaurang Tapase <gtapase@ddn.com>
Reviewed-on: https://review.whamcloud.com/47132
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-5021 lipe: lpcc_purge recognize pcc pin flag

lpcc_purge check pcc pin flag on lustre file and do not purge
matching cache file in PCC RO.

Change-Id: Iebbe8d1a46a9aefbecd8fd0eb1bbdf71770b0499
Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial testlist=sanity-pcc
Reviewed-on: https://review.whamcloud.com/46920
Reviewed-by: Yingjin Qian <qian@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-5102 utils: Add --pin option for 'lfs pcc attach' command

If --pin option is provided, 'lfs pcc attach' command will
first pin then attach a file. So that user does not need
to run them separately.

Change-Id: I8e06af09abe7922399fd5e8e60132ff24ca31d1a
Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial testlist=sanity-pcc
Reviewed-on: https://review.whamcloud.com/47067
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-5047 pcc: tunable GID to pin/unpin files

In this patch, we adds a tunable "mdt.*.enable_pin_gid" that
controls the GID to pin/unpin files.
Default should be gid=0, so it is limited to root, gid=-1 so any
user can pin/unpin, or gid=<gid> to allow a specific GID (e.g.
wheel).
We add a virtual "lustre.pin" xattr to set/get the xattr for
non-root users, if mdt.*.enable_pin_gid is set correctly.
Add test case sanity-pcc test_104b to verify it.

Change-Id: I1be92f365cfe8dea278559b2c34fcee8e42d1c4d
Signed-off-by: Qian Yingjin <qian@ddn.com>
Reviewed-on: https://review.whamcloud.com/47103
Reviewed-by: Feng, Lei <flei@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-5128 pcc: avoid to specify attach ID for every pin

In this patch, it avoids the need to specify "-i <attach_id>" for
every pin/unpin operation as in the very common case there is only
a single cache for that client.
If attach ID is not specified, it will select the first dataset
on the client as PCC backend.

Add the test case: sanity-pcc test_205.

Change-Id: I04e2b3d9ca360ae40f3723d7cd9bd78b22602123
Signed-off-by: Qian Yingjin <qian@ddn.com>
Reviewed-on: https://review.whamcloud.com/47105
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Feng, Lei <flei@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-5113 utils: add pcc pin/unpin commands

Store pin information in xattr trusted.pin.
Add lfs pcc pin/unpin commands to set/clear pcc pin flag.
A pcc-pinned file will not be removed from local storage
cache by lpcc_purge automatically.

Examples of xattr trusted.pin:
trusted.pin=[hsm: 2]
trusted.pin=[hsm: 2,hsm: 3]

Change-Id: I17ddf5ac8dc4eae48c0f6bdd0f2a19240474b0f5
Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial testlist=sanity-pcc
Reviewed-on: https://review.whamcloud.com/47083
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yingjin Qian <qian@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

EX-5131 build: Add python-*yaml deps for lipe-lpcc

Test-Parameters: trivial

Change-Id: Idd05bd3ae4c12b82392d0b7f2dff826e1954773c
Signed-off-by: Gaurang Tapase <gtapase@ddn.com>
Reviewed-on: https://review.whamcloud.com/47106
Reviewed-by: Raphael Druon <rdruon@ddn.com>
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Feng, Lei <flei@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

EX-4087 lipe: no lamigo in hot-pools test_59()

In hot-pools test_59(), do not start lamigo. Instead, menaully create
a file with the desired striping (uptodate on fast pool, stale on slow
pool).

Test-Parameters: trivial testlist=hot-pools env=ONLY=59,ONLY_REPEAT=20
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I200b1e890a12d59a0ba5cec5f6450704f35b4df8
Reviewed-on: https://review.whamcloud.com/47031
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>

LU-15683 ofd: proper initialize filter_fid in ofd fallocate

Intialize filter_fid buffer and call xattr set XATTR_NAME_FID properly
in ofd_object_fallocate().

Lustre-change: https://review.whamcloud.com/46919
Lustre-commit: TBD (from c77a4635286e5b74034bf6ff65db75bc79e10564)

Fixes: 48457868a02 ("LU-3606 fallocate: Implement fallocate preallocate operation")
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Ied573c39dde77f935622e9fbedb2d71eb3bd8f5d
Reviewed-on: https://review.whamcloud.com/46924
Tested-by: jenkins <devops@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

EX-5092 tests: skip hot-pools if yq is not working

Skip hot-pools if yq is not working.

Test-Parameters: trivial testlist=hot-pools
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Ia8ff851ea7e06626812ca736a0eeec78ca11b930
Reviewed-on: https://review.whamcloud.com/47020
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>

EX-3293 tests: use shorter age for lamigo

to speed up the testing

Test-Parameters: trivial testlist=hot-pools
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I433a5ed798c6d4f0f817092a1d21281fab236f78
Reviewed-on: https://review.whamcloud.com/47021
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

EX-5087 lipe: lamigo_job_fini() debugging

In lamigo_job_fini(), only print a debug message for check jobs if the
state of the agent changed.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I6f848d6fc896aea8c45627f9ee542d38bf40d6b4
Reviewed-on: https://review.whamcloud.com/46995
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexandre Ioffe <aioffe@ddn.com>

LU-14900 tests: do not fail if kmemleak tunable is not writable

Change-Id: Id77430f1e8ff7a8fda6538211a0d36bbe973a889
Test-Parameters: trivial
Fixes: 15c0a21ea9 ("tests: Add kmemleak awareness to test-framework")
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/46936
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

EX-3002 lipe: rename lpurge free{lo,hi} to {max,min}-used

Expressing things in terms of used rather than free is much easier to
understand.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Test-Parameters: trivial testlist=hot-pools
Change-Id: I31ff88db97c0dd1b2ddbaa67aa27e8bec520c7d3
Reviewed-on: https://review.whamcloud.com/46903
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexandre Ioffe <aioffe@ddn.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>

LU-14644 vvp: wait for nrpages to be updated

truncate_inode_pages() says there still may be a page in a process
of deletion upon return. wait for another thread which is doing
__delete_from_page_cache() to get nrpages updated.

Lustre-change: https://review.whamcloud.com/43464
Lustre-commit: 7d5d004506650c3739898e70d72c9a86b8aeeb88

Signed-off-by: Vitaly Fertman <c17818@cray.com>
Change-Id: I165b3d0866efaf2eb7e977520ebba4ee831874ab
HPE-bug-id: LUS-8842
Reviewed-on: https://review.whamcloud.com/46954
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>

LU-15702 lov: remove lo_trunc_stripeno

Remove the lo_trunc_stripeno member of struct lov_layout_raid0 and add
an lis_trunc_stripe_index array to struct lov_io. This makes the
truncate stripe index information belong to the IO and not to the
concurrently accessed object. This is needed because we do not have
locking that protects it from its initialization in lov_io_iter_init()
to its use in lov_lock_sub_init(). Also remove the unused
lo_write_lock member of struct lov_object.

Fixes: 9801500451 ("LU-14128 lov: correctly set OST obj size")
Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I799e07059212629fe9d84c5e58c349035a40da9e
Reviewed-on: https://review.whamcloud.com/46953
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>

Update lipe version to 2.23.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: Ie3cdc735a30225dc8f14c3e67292e52057b2170d

RM-620 build: New tag 2.14.0-ddn39

New tag 2.14.0-ddn39

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Iddb0af12e9564eb50d5839285e48a9209ab5c030

EX-4333 sec: support supplementary groups from client

The usual way to support more than 2 supplementary groups is to
resort to the server side's identity upcall. This identity upcall
retrieves all user's credentials, including all supplementary groups,
and stores them in cache. But this access to user's credentials from
server side is not always an option.

As an alternative to the server side's identity upcall, we implement
a retry mechanism for intent locking. The client can provide at most
2 supplementary groups in the request sent to the MDS, but sometimes
it does not know which ones are useful for credentials calculation on
server side. For instance in case of lookup, the client does not have
the child inode yet when it sends the intent lock request.
Hopefully, the server can hint at the useful groups, by putting in the
request reply the target inode's GID, and also its ACL. So, in case
the server replies -EACCES, we check the user's credentials against
those, and try again the intent lock request if we find a matching
supplementary group.

On server side, we add a new identity_upcall value named "INTERNAL".
It implements a particular behavior which does not involve an actual
upcall, but instead the cache is filled with supplementary groups read
from the client request, cumulatively at each request.

A problem with the 'runas' utility was found during testing. If no
supplementary group is provided via the '-G' option, then it needs at
least to set the given GID as a supplementary group. Otherwise the
supplementary groups of the invoking user would be silently inherited.
For instance, if root user calls 'runas -u 500 -g 500 $CMD', we must
not execute $CMD with UID:GID 500:500 and supplementary group 0, as
it would make the user executing $CMD part of the superuser group.

Test-Parameters: testgroup=review-dne-part-1 env=L_GETIDENTITY=INTERNAL
Test-Parameters: testgroup=review-dne-part-2 env=L_GETIDENTITY=INTERNAL
Test-Parameters: testgroup=review-dne-selinux-ssk-part-1 env=L_GETIDENTITY=INTERNAL
Test-Parameters: testgroup=review-dne-selinux-ssk-part-2 env=L_GETIDENTITY=INTERNAL
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I4608bb766a70ca12a2142a0e2687813f3a4b9100
Reviewed-on: https://review.whamcloud.com/46493
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-13717 sec: fix handling of encrypted file with long name

The ciphertext representation of the name of an encrypted file or
directory can be up to 256 bytes of binary data, if the cleartext
name is up to NAME_MAX. But then this ciphertext is encoded via
critical_encode() before being sent to servers. Once encoded, the
length can exceed NAME_MAX because of the escaped critical
characters.
So make sure ll_prep_md_op_data() accepts those too long encoded names
if it is called for lookup or create of an encrypted file or
directory. In the other cases, the 'name' taken as input is the plain
text version, so it must conform to the NAME_MAX limit.

When carrying out operations on an encrypted file with long name, we
manipulate a digested form whose hash needs to be matched against the
content of the LinkEA. The name found in the LinkEA is not NUL
terminated, so this aspect must be taken care of.

Lustre-change: https://review.whamcloud.com/45163
Lustre-commit: 75414af6bf310244d38284958ecf037d61936726

Fixes: 4d38566a00 ("LU-13717 sec: filename encryption")
Fixes: ed4a625d88 ("LU-13717 sec: filename encryption - digest support")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I4b0e51eee5e549ab56292fe0fec3c1be1b487fc7
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46887
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

LU-15608 sec: fix DIO for encrypted files

With Direct IO, we do not have proper page cache pages. So we need to
retrieve by ourselves the page mapping and the page index of the page
to be encrypted/decrypted.

For the index, we need to use the offset of the page within the file,
and not the object.
So we rename cl_page's cp_osc_index to cp_page_index for that purpose.
cp_osc_index is redundant with osc_async_page's oap_obj_off and only
used by osc_index(), so we also adapt this function.
cp_page_index is initialized in cl_page_alloc(), and accessed in
the OSC layer where the llcrypt primitives are called.

For the mapping, problem is page->mapping is not set to NULL on page
allocation, so it cannot safely be used to see if a page is a direct
I/O page.
Use cl_page for direct I/O and page->mapping for buffered
I/O.  (clpage->cp_inode is only set for direct I/O and
cannot easily be always set.)
Without this, we sometimes get panics when page2inode is
used in the OSC layer.  (Note the remaining use in dom is
safe because ll_dom_readpage is a page cache helper and
will never see DIO pages.)

Lustre-commit: 966ca46e4aa2eb39c70e49648ffe6fcaaf475536
Lustre-change: https://review.whamcloud.com/46664/

Fixes: a71e0dd7f7 ("LU-14306 sec: get rid of bad rss-counter state messages")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: Icb53a4e45463b8d3febc2e6212b39dc25719d866
Reviewed-on: https://review.whamcloud.com/46819
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

LU-14676 lnet: improve hash distribution across CPTs

Change the nid-to-cpt allocation function to use
(sum-by-multiplication of nid bytes) mod (number of CPTs)
to match nid to a CPT. This patch only addresses IPV4 nids.

Make the matching change for the nid-to-cpt function
used by the 'lnetctl cpt-of-nid' utility.

Lustre-change: https://review.whamcloud.com/46233
Lustre-commit: 9b6e27755507b9bb47a1d7b4aede6302a876a14d

Test-parameters: trivial testlist=sanity-lnet

Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: I1052414947c4cae8c63993ffa21f67cb389bb463
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/46398
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>